Product / Engineering

Adaptive Tiering lowers the cost of keeping machine data you haven't queried recently

Adaptive Tiering moves settled event data between three tiers based on how recently it was queried, so keeping logs, traces and events for longer costs progressively less. Every tier has the same latency, and there’s no restore step or anything to configure.

Conor Curran · · 6 min read

Abstract representation of Adaptive Tiering technology

Most teams decide how much history to keep by looking at the bill

Ask a team how long they keep their logs and traces, and the answer is usually a number that came from a budget, or from the license terms of whatever tool they were using at the time. It is rarely a number that came from how long the data stays useful.

But machine data seldom arrives with its value attached. An event from six months ago might be unremarkable until a question lands that only it can answer: when a slow degradation began, why a failure only ever shows up on one set of instances, what happened in the window a security review or an audit is asking about. Then it is the piece of evidence that settles the matter. Data that was never kept can't be queried later, whatever the question turns out to be. Most teams know this and shorten retention anyway, because the alternative was a storage line nobody wanted to defend.

Adaptive Tiering makes data you haven't queried recently cost less to keep

Adaptive Tiering is enabled for every organization on Axiom Cloud. It moves "settled" event data between three S3 access tiers according to how recently that data was queried. Settled data means the immutable blocks we have already written to object storage. There is nothing to turn on and nothing to configure.

Access tier

What sits in it

Frequent Access

Newly ingested data, and anything queried in the last 30 days

Infrequent Access

Data that hasn't been queried for 30 consecutive days

Archive Instant Access

Data that hasn't been queried for 90 consecutive days

Every tier serves the same low latency and high throughput. You query data in Archive Instant Access exactly as you query data written this morning: there is no restore step, no rehydration and no wait. The only thing that changes as data ages is what it costs to keep. The Adaptive Tiering docs have the details.

The price falls without the data moving out of the query path

The reason this works without a restore step is architectural. We write every event to object storage, and we read from object storage always, whether the data is a minute old or ten years old. Older blocks stay in the ordinary query path, surrounded by the layout, metadata, planner and execution machinery needed to interrogate them. There is no separate place old data goes to live.

Some of the tools built for observability or log management treat object storage as an archive. Data goes there to be cheap, and has to be pulled back out before anyone can do anything with it. That is why cheaper history usually comes with a wait, a worse answer, and sometimes, unbelievably, a second charge to get the data back. Adaptive Tiering is the opposite move: it changes what a block of data costs to store, and nothing else about it.

Mechanically it is one storage class, applied on write. We write blocks to object storage and the tier transitions happen in the storage layer rather than in a lifecycle job we run. The block never moves anywhere you have to think about.

A longer retention window means something different to each function

Engineers are usually the first to ask for longer retention, but they aren't the only people who live with the policy. A longer window means something different to each team.

  • The engineer can answer a question about what happened six months ago instead of reasoning about what probably happened.

  • The engineering leader stops trading coverage against budget at every renewal.

  • Security can't investigate what was never kept. The time between a breach and its discovery is often measured in weeks, and in months when an intruder is deliberately quiet, so the window that matters is rarely the one the budget set.

  • For compliance and legal, a retention window is an obligation rather than a preference, and it is now cheaper to meet the longer the data sits untouched.

There is nothing to configure, because placement follows how the data is used

Tier placement follows query recency, and only query recency. Nobody writes a lifecycle policy, pins a dataset to a tier, or guesses in advance which history is going to matter. The price follows how the data is used.

Querying data that has aged into a lower tier returns it to Frequent Access, and it moves down again after 30 and 90 consecutive days without a query. You pay the higher rate only for history you are using, and you never have to predict up front which data that will be. Run the investigation you need to run; the tiers follow.

Adaptive Tiering applies to organizations on Axiom Cloud. The Personal plan is unaffected: its 30-day retention means data never ages out of Frequent Access, so there is nothing to tier.

Retention becomes a decision about what the business wants to be able to answer

Storage was already a small proportion of an Axiom bill. EventDB compresses event data about 25 times on average, and up to 50 times for more structured events, which works out to a footprint reduction of more than 95 percent. Adaptive Tiering is a separate lever on top of that. It doesn't change how much data there is; it lowers the price per gigabyte even further for history you aren't actively working with.

You can see the effect in Console. The dataset information panel shows how much of each dataset sits in each access tier, and the organization's usage page breaks storage down the same way. Storage is billed as three line items, one per access tier, at the rates on the pricing page. Retention is set once for the organization, with a different period for any dataset that needs one.

For any team whose retention window was set by a bill rather than by the questions it needs to answer, this changes the shape of the decision. It stops being how much history you can afford and becomes how far back the business needs to be able to see.