At petabyte scale, telemetry ingestion becomes a data-centre architecture problem

Most OpenTelemetry architectures make a sensible trade.

Applications emit generic telemetry. Collectors receive, batch and export it. A managed backend takes responsibility for parsing the payloads, dealing with types and schema, forming columns, encoding them, compressing them and laying the result out for storage and query.

That boundary is one of the reasons OpenTelemetry works. The applications and collectors don’t need to know how each destination stores its data and the backend gets to change its storage engine without requiring every customer to redeploy an agent. At petabyte scale, though, the same boundary can become an expensive place to stand.

The cost is hiding inside “ingest”

Ingest sounds like transport: get the bytes from the data centre into the platform.

The actual path does much more. A backend has to receive a large number of relatively generic batches, parse them, reconcile their shapes, build columns, choose encodings, compress the result, assemble useful storage blocks and make those blocks visible to the query system. It needs enough compute and buffering to absorb bursts, enough durability to recover when part of the path fails, and enough coordination to keep the storage engine healthy. At terabytes, that work is an implementation detail the service should absorb and at tens of petabytes a month, it becomes a more noticeable line item.

We learned this while scaling Axiom alongside frontier model labs. The workloads weren’t one uniform stream called “AI logs.” They were Kubernetes logs, OpenTelemetry traces, inference systems, API activity, network and edge traffic, voice systems, model-serving infrastructure and internal developer systems with volume spread across dozens of datasets with very different row sizes and event rates.

An OpenTelemetry deployment sending tens of petabytes a month is already a data-centre architecture in its own right. Collectors near the racks feed aggregation layers, and at the top there can be hundreds of nodes whose only job is to consolidate batches and forward them on. All the compute in that tier really buys is bandwidth, because when a batch arrives we still have to unpack it, read it again and reorganise every byte into the shape our storage format and query system need. That is double compute running for no reason.

The store was only half the problem. Axiom’s EventDB is built to compress, retain and query machine data at this scale. But before EventDB can do that, the data has to reach the shape EventDB is good at storing.

If every raw byte requires a central loading tier to do all of that shaping, the customer pays for the path as well as the destination.

The predictable response is sampling, keep 10% of the spans, drop a field here or there, aggregate before export, or put older data somewhere cheap and accept the rehydration ticket later. Those choices make the bill survivable by making the dataset less complete, and observability becomes a negotiation at the keyboard: is this trace worth what it costs to retain? I don’t think that’s a useful end state.

Move knowledge, not the datastore

Axiom has spent years building knowledge about how machine data should be stored: our columnar formats, encodings, compression, batching, layouts and the compaction and query systems around them.

Bifrost, the high-volume data-loading path we introduced yesterday, started with a direct question: what happens if we move more of that knowledge out to the systems producing the telemetry?

Not the whole datastore. Customers shouldn’t have to run EventDB inside their data centre. The point of Axiom is that a platform team can point data at us and avoid becoming the team that operates storage and query infrastructure . The useful thing to move is the ability to prepare data for that store.

Bifrost is that move since it sits in the OpenTelemetry egress path. Instead of exporting only generic batches for a central service to reshape, it applies Axiom-aware encoding, compression, batching and storage-layout logic near the source. The compute these teams are already spending starts producing compact columnar batches in the shape our store expects, which stream into object storage for Axiom to validate and register.

The reason this can work at all is architectural. Object storage is Axiom’s hot store, not an archive, so the query engine reads from it directly. Once a batch has been validated and registered, there’s nothing left to happen to it: it joins the same body of data as everything that came through the standard pipeline, and queries can reach it. On a platform where object storage is a warm or cold tier, data prepared near the source would still owe a trip through a central pipeline before it could serve a query, and the economics of moving the shaping work would mostly evaporate.

Most workloads, including some very large ones, are exactly where they should be: on the standard managed pipeline, which exists to absorb this shaping work and keeps doing so unchanged.

The applications still emit OpenTelemetry, the customer still uses Axiom datasets, retention, and APL, and Axiom still owns the service. What changes is the middle of the path, and only at the scale where that middle becomes an expensive place to stand.

The shortest path to the answer

The team built the first Bifrost path to answer the economic question as directly as possible.

It created Axiom’s native storage blocks near the source, uploaded them into object storage and registered them with EventDB, and from there they entered the normal query and compaction system.

That was an intentionally sharp implementation boundary. It removed a large amount of central loading work and let us learn whether the cost model was real before committing to the supported product architecture.

Then the team pushed it, exercising logs and traces, small rows and wider events, a single dataset and multiple concurrent datasets. They ran representative high-volume traffic, stepped through a calibration ladder and later mirrored substantial production telemetry through Bifrost into staging. The registered data was queryable in Axiom.

The work validated the direction. Moving the shaping work changes the economics enough to support a materially lower data-loading price. It also found the things you only find when you stop treating the diagram as the system.

Buffers filled faster than blocks could finalise. Collectors ran out of memory. Adaptive routing could oscillate between paths. Registration became a bottleneck. A retry classification could leave data stuck until an operator intervened. Bursty registrations exposed assumptions in compaction. Accounting had to distinguish raw telemetry from compressed block bytes.

The team resolved them, and that is the part of the work I think is worth celebrating: they didn’t produce a benchmark slide and stop. Rather, they pushed the implementation hard enough to turn abstract risks into specific engineering knowledge, with fixes built into the path preview partners run today.

The product boundary matters

The cheapest theoretical path isn’t automatically the product we should ask a customer to operate, so where the supported boundary sits is a design decision we’ve treated with the same seriousness as the economics.

Bifrost’s design sits deliberately close to the source. The egress tier does the heavy shaping work (encoding, compression, batching and storage layout) so data arrives in the shape our store expects, because that relocation is what changes the loading economics, and returning the work to a central tier would return the cost with it.

What Axiom keeps is validation and finalisation. A malformed or semantically incorrect batch is a different class of problem from a malformed OTLP request and discovering it during later compaction is too late, so nothing becomes queryable until Axiom has validated and registered it. The customer gets the economics of shaping near the source, and Axiom keeps responsibility for what enters the store.

What a preview partner shapes with us are the operational specifics: how much data the deployment produces, how it’s distributed across datasets, how quickly it needs to be visible and where the egress tier runs. Those determine how a path is sized and rolled out, not whether the boundary holds.

Why private preview

Bifrost is now in private preview for selected petabyte-scale OpenTelemetry workloads.

Private preview is the right form because these workloads are rare, consequential and different enough that our usual self-service toggle would teach us less than direct work with the team operating one.

An evaluation starts with the actual shape: bytes and records by signal and dataset, event-size distribution, egress topology, retention, query patterns, current sampling decisions, and the percentage of traffic that can use a Bifrost path.

From there, we model the economics, choose an initial dataset, size the path, run beside the existing route and measure the result. We care about cost, but also data correctness, durability, backlog, recovery, operational load and when newly loaded data becomes queryable.

The proposed economics cut the data-loading component by more than half at petabyte scale, including against the lowest tier of Axiom’s standard loading path. Query and storage do not disappear, so the exact whole-bill result depends on the workload. We will model that with each partner rather than advertise one percentage as if every system were the same.

└ The test sample was representative of petabyte-scale AI workloads.

Complete data is an economic claim

“Keep every byte” can sound like a storage claim.

It’s really an architectural and economic one: A storage engine can be capable of retaining the data while the path into it makes retention irrational and a query language can be capable of finding the event while the pricing model encouraged an engineer to drop the field six months earlier.

Bifrost exists because the frontier workload forced us to look at the whole path.

At ordinary scale, keep the boundary simple and let Axiom’s standard loading tier do the work. At exceptional scale, bring more of the storage intelligence to the data and keep the datastore managed.

If your OpenTelemetry workload is measured in petabytes, I’d like to see its shape. Bifrost private preview is open to a small number of teams willing to build the next version of this path with us. Reach out to me at njp@axiom.co to connect.