A metrics bill can rise without a matching rise in traffic. Someone traces the increase to a single label such as customer_id, and the proposed fix is to delete the label. Deleting it usually brings the bill down. The next incident is then harder to diagnose, because the dimension that separated affected tenants from healthy ones is gone.
Most observability stacks were not designed for the number of dimensions modern systems emit, so teams reduce the dimensions they keep in order to stay inside budget. This playbook covers how to measure cardinality, decide which dimensions justify their cost, shape the rest at the collector, and choose a store that prices ingest by volume rather than by series count.
Why high-cardinality metrics turn into a cost problem
Every unique tag combination is a new series
Cardinality is the number of unique combinations of attributes on a metric, and each combination is its own stored object. The Prometheus project's instrumentation and naming guidance describes the mechanism: every unique combination of key-value label pairs represents a new time series. Every one of those series usually has to be indexed, held in memory, and billed as its own object.
The multiplication is easy to underestimate. Take a single request counter tagged with 40 services, 25 routes, 4 regions, 8 status codes, and 500 tenants. Those five attributes describe one metric, and together they can produce up to 16 million distinct series. Adding a sixth attribute with ten values makes it 160 million.
The legacy metrics model assumed a small number of tags per series. Current systems often exceed that assumption. AI workloads add prompt IDs, model versions, and per-request token counts. Microservices add per-tenant labels, and per-request identifiers like trace IDs leak into metric labels along the way. Designs that were adequate at terabyte scale often fail at petabyte scale, which is the premise behind the modern machine data platform.
How per-series pricing charges for dimensions
Most metrics systems charge by active time series, so the more dimensions a team tracks, the more it pays. That pricing model encourages teams to pre-aggregate data, drop labels, and limit cardinality to control costs. Traditional systems often respond in the same way, either charging per dimension or silently dropping high-cardinality tags.
The short-term saving is usually temporary. Costs rise as volume rises, and engineers reduce instrumentation to keep the bill predictable. The missing label then becomes a problem during an incident, when it would have been the one that isolated the blast radius. At petabyte scale, ingestion becomes an architecture problem, and the economics follow from the architecture.
A practical playbook for high-cardinality metrics
Step 1: Measure cardinality before you cut it
Auditing dimensions is the step most teams skip, and it is also the least expensive. Start there, because a limit may already be trimming the data before anyone decides anything. The OpenTelemetry metrics SDK specification sets a default cardinality limit of 2,000 data points per instrument per collection cycle. Measurements past that limit collapse into a single synthetic series marked otel.metric.overflow. If a dashboard has looked oddly flat, check that limit before changing application code.
Start by ranking metrics by active series count, then attribute each metric's series count to the specific dimensions driving it. One or two attributes usually account for most of the total, which turns a vague cost problem into a short list.
Step 2: Keep the dimensions that answer a question
Triage each dimension against a question someone has asked in an incident review. Customer, route, region, model, and deployment version tend to carry the context needed to isolate impact, so they usually stay. A unique request identifier nobody groups by is a different case, and it usually belongs on a trace or an event rather than on a metric.
The test is whether removing the dimension would change an answer. If nobody can group by pod_name after a pod is gone, that attribute is stored for a question the team never asks. If tenant_id is how support confirms one customer is affected while everyone else is fine, cutting it turns a short check into a much longer investigation.
AI workloads sharpen this test. Token spend can't be attributed on an average. Knowing which model version, which tenant, and which feature drove yesterday's inference bill requires exactly the dimensions per-series pricing punishes. For LLM-backed products, per-model and per-tenant tags aren't nice-to-have diagnostics; they're literally the FinOps ledger.
Write the decision down per dimension. That record is what stops the same argument from restarting at the next budget review, and it gives the next engineer a reason for the decision.
Step 3: Shape metrics at the collector
Dimension shaping belongs in the OpenTelemetry Collector pipeline rather than scattered through service code, because the pipeline is one place you can change without a redeploy. The transform processor rewrites or removes attributes before export, and the filter processor drops whole metrics or data points. The OpenTelemetry specification also enforces cardinality limits after attribute filtering, which makes "views" the intended place to shed dimensions inside the SDK.
While you're in the pipeline, check the limits on ingested metrics you're exporting into. The 4 MiB maximum uncompressed request is the one to watch, so batching doesn't turn into 413 responses.
Step 4: Move what you keep onto a store that doesn't charge per dimension
Steps 1 through 3 reduce unused series. Once a team has cut everything nobody asks about, the remaining dimensions are the ones people use, and the question becomes what they cost. At that point the answer depends on how the destination prices metrics.
Axiom's metrics datastore is purpose-built for high-cardinality time series, and it treats high-cardinality tags as a design principle, not a cost-control penalty. In practice that means capturing every dimension your systems emit and tagging per request, per tenant, and per model.
Migration doesn't have to be a cutover. Most teams stand up the OpenTelemetry Collector first, run both stacks in parallel for one workload, then switch at the next renewal seam. Teams coming from a Prometheus-shaped stack can translate PromQL queries to MPL, Axiom's query language for metric series, with minimal changes.
What changes when high cardinality is a design principle
Placement follows series pressure
Cardinality still consumes real resources, so a store has to place that load somewhere. Axiom's architecture answers with per-metric adaptive placement inside MetricsDB, Axiom's metrics store. Each metric gets a "subring," which is a subset of ingest nodes sized to its active series.
As Dynamic subrings: Consistent hashing without the tradeoff describes, each subring grows or shrinks on the actual number of series. A 1,000-series metric and a 1,000,000-series metric each get the right spread automatically.
One usage-based dial across every signal
Pricing is the other change. As Metrics are generally available describes, Axiom prices metrics ingest per gigabyte, starting at $0.12/GB with volume discounts. There's no active time series count to track. That's one usage-based dial across logs, traces, metrics, and events.
Two consequences follow. First, you can query metrics in Axiom with MPL, the Metrics Processing Language, right next to your logs, events, and traces. A cardinality question and a log question live in the same investigation. Second, the dimensions you keep stay useful across signals, because logs, traces, metrics, and events on one platform meet in the same Console dashboard.
That shared surface comes with two trade-offs to plan for before a migration. MetricsDB truncates timestamps to second precision, and it flattens resource, scope, and metric tags into a single namespace.
Where agents fit in cardinality work
Cardinality auditing is repetitive work, which makes it a reasonable thing to hand to an agent. Axiom's metrics skill exposes high-cardinality metrics to agent reasoning in MPL. The Query metrics skill can rank series counts by dimension, or compare a metric's shape before and after a collector change. Axiom's MCP server gives that agent the same governed access a person has, under the permissions the caller carries.
An agent can report which five attributes produce most of a team's series. A person still decides which of those five to keep.
High-cardinality metrics turn into a cost problem because most systems charge for series identity. Dropping labels reduces the bill, and it also removes the dimension that later investigations need. The sequence that works is the same in every stack: measure which dimensions drive series counts, keep the ones that answer questions people ask, shape the rest in the OpenTelemetry Collector, and then compare how the store underneath charges.
When metrics are priced per gigabyte instead of per active time series, tagging per request, per tenant, and per model is an instrumentation choice rather than a budget choice.
FAQs
Cardinality is the number of unique combinations of attributes on a metric, and each combination becomes its own stored time series. No single number marks the line, so the practical test is multiplication. A request counter tagged with 40 services, 25 routes, 4 regions, 8 status codes, and 500 tenants can produce up to 16 million series. Attributes with open-ended value sets are what get you there, and tenant IDs, pod names, model versions, and per-request identifiers are the usual candidates.
Often, yes. The OpenTelemetry metrics SDK specification sets a default cardinality limit of 2,000 data points per instrument per collection cycle. Past that limit, measurements collapse into a single synthetic series marked otel.metric.overflow instead of being discarded silently. If a dashboard looks unexpectedly flat, or a dimension stops splitting the way you'd expect, look for that overflow series before you start reading application code.
In the collector, because the pipeline is the one place you can change without a redeploy. The OpenTelemetry Collector gives you a transform processor to rewrite or remove attributes before export, and a filter processor to drop whole metrics or data points. Inside the SDK, cardinality limits are enforced after attribute filtering, which makes views the intended place to shed dimensions.
Size batches against the limits on ingested metrics at the destination too. Requests above the 4 MiB uncompressed maximum come back as 413s.
It changes what you're optimizing. Under per-series pricing, every added dimension multiplies the bill, so each instrumentation decision doubles as a budget decision. Metrics are generally available describes the alternative. Axiom prices metrics ingest per gigabyte, starting at $0.12/GB with volume discounts. There's no active time series count to track, so cost follows the volume of data you send on one usage-based dial shared with logs, traces, and events.
Teams do not have to rewrite PromQL from scratch. Axiom queries metrics in MPL, the Metrics Processing Language, and teams coming from a Prometheus-shaped stack can translate PromQL queries to MPL with minimal changes. Selectors, rate calculations, and aggregations carry over. The work is closer to a translation pass over saved queries and dashboards than a rewrite of how you think about the data.
Yes. Coexistence is the usual path. Most teams stand up the OpenTelemetry Collector first, run both stacks in parallel for one workload, then switch at the next renewal seam. Metrics ship through the standard OTel Collector with no proprietary agent, so adding a second destination is a pipeline change rather than a re-instrumentation project. Running both for a while also gives you a window to compare series counts and query results before anything depends on the new store.
Two trade-offs show up in the docs for querying metrics in Axiom. MetricsDB truncates timestamps to second precision, so sub-second resolution isn't preserved. It also flattens resource, scope, and metric tags into a single namespace, which means an attribute name reused across those scopes needs a rename before ingest. Neither is hard to work around, and both are easier to handle before a migration than after.
Axiom treats high-cardinality tags as a design principle, not a cost-control penalty, so teams are expected to tag per request, per tenant, and per model. There's no cardinality ceiling on Axiom Cloud for typical workloads. The soft limits that remain cover things like datasets and dimensions per metric, and they're liftable on request. Each dimension still has to belong in someone's investigation, which is what the triage step in the playbook is for.
Related Reading
See it on your own cardinality
If you're deciding which dimensions to keep and which to drop because of cost, a walkthrough on a live metric will show the series count and the per-gigabyte ingest cost on the same data. Bring the tags you want to keep and see Axiom on your data.