Most teams don't choose a "data lake" on purpose. It accumulates: a logging cluster here, a metrics store there, a security pipeline no one on the team owns, each sized for last year's volume and already behind this year's. By the time you need a six-month-old trace or a security event from a source you never indexed, the answer is often "we don't have that anymore," or "we have it, but it'll take a restore ticket and a day."
A "machine data lake" keeps logs, traces, metrics, and security events at full fidelity in one place, queryable however far back an investigation goes, without a platform team rebuilding infrastructure every time volume grows.
Events and metrics are different kinds of data
We split Axiom into two dataset kinds: events, which cover logs and traces, and metrics. That split reflects two different kinds of data. A log line or a trace span is a flexible record that can matter on its own. A metric is a point in a named series, identified by its exact dimensions, where the value only means something next to the points before and after it. Forcing both kinds of data into one generic table works until you need to ask a metric-shaped question of event-shaped storage, or the other way around. We back metrics with a purpose-built metrics datastore, so a metric isn't treated as a log event with a number attached. That's why dimension-heavy time series stay fast to query as the number of series grows.
If you're sending OpenTelemetry data, we recommend a dedicated dataset for each OTel component: one dataset each for logs, traces, and metrics. You set that structure once, in the collector's exporter configuration, regardless of team or service. A new service that adopts the same collector inherits the same dataset layout automatically, so you don't decide where its logs belong each time you ship something new.
That structure pays off the moment an investigation starts. Correlation groups link the logs, traces, and metrics datasets that share context, so you can move from a log line to its matching trace without copying an ID into a second tool or rebuilding a query from scratch. You don't need to split logs further by service to get there. A dataset's kind sets its event or metric semantics regardless of the service that wrote it, so the same two kinds cover every service a team ships.
Put security events in one OCSF dataset
Security telemetry doesn't fit the OpenTelemetry shape, and it shouldn't try to. OpenTelemetry semantic conventions describe observability data: requests, spans, resource attributes. OCSF (the Open Cybersecurity Schema Framework) describes security events: authentications, network connections, file activity, detections. Different questions need different schemas, and OCSF is the one security teams and their tools already expect.
Our OCSF implementation promotes every documented attribute into a typed column up to a fixed depth, so fields like class, activity, and severity are queryable as structured data from the moment they land. OCSF hasn't standardized every attribute; the schema spans more than a hundred thousand possible paths. We don't drop the attributes it hasn't standardized, and we don't force a schema migration every time a new source shows up with a field you haven't seen yet. Everything else remains in a single unmapped field, still present, still searchable, not promoted to its own column.
One OCSF dataset also sets up a practical path for teams running a SIEM today. A source that already speaks OCSF (Cribl is a common one) sends data into that dataset in Axiom. From there, Axiom for Splunk and Axiom Portal for Splunk let a SOC query the same dataset in SPL, so Splunk ES or SOAR keeps doing detection and case work without re-ingesting the data a second time. You can also query the dataset directly in APL. You don't have to choose between keeping Splunk's workflows and keeping the data affordable. The pattern we've seen work is to land high-volume or orphaned sources beside the SIEM first, prove the model on the data you weren't fully covering anyway, then expand at the renewal seam if the measured result supports it.
Let the schema come from the data
A machine data lake shouldn't ask you to design a schema before you can send it data, and we don't. The schema is derived from the data you ingest, not declared ahead of time in a mapping file you have to maintain.
For structured collectors, OpenTelemetry chief among them, that parsing happens at ingest. The fields in an OpenTelemetry log record or span arrive already named and typed, and we turn them into queryable columns as part of landing the event, ahead of any later batch job. There's no separate modeling step between "we started sending data" and "we can query it."
Query-time parsing still has a narrower job: pulling structure out of older, unstructured data that predates a cleaner pipeline, like extracting fields from a raw syslog message field that's a string today. Virtual fields are similar in spirit and optional in practice: deriving a browser and device from a user agent at query time, for instance, so you don't have to store those as separate ingested fields. Map fields exist for genuinely custom attributes that fall outside the OpenTelemetry schema; standard OpenTelemetry attributes already fit comfortably within Axiom's field limits, so you won't reach for a map field at all.
None of this means leaving every record unstructured until you write a parser. The schema exists, and it's useful the moment data lands: it's assembled from the data that arrives, not declared in advance. We built Axiom as a schema-less event store for that reason: no upfront modeling gate sits between a new source and a queryable dataset, and the structure a collector already provides still gets captured and typed automatically.
Keep every event
The single biggest compromise in most logging setups is sampling: keeping a fraction of events to stay inside a budget, and hoping the kept fraction includes the event the incident later needs. It usually doesn't work out that way. The event that explains a bug, a security incident, or a billing dispute is as likely to be in the 90% you dropped as the 10% you kept, and you find out which on the day you need it. Reconstructing the incident after the fact, through logs from an adjacent service or a guess at the likely cause, costs more engineering time than storing the event would have in the first place.
Our position is to keep every event rather than predict in advance the events that matter. That's only a reasonable default if full-fidelity retention is affordable, and the reason comes down to the storage format, separate from the amount discarded before it lands. Columnar compression is the reason full-fidelity retention is affordable: values for the same field sit together, Axiom's EventDB picks an encoding that fits each column's actual shape, and the result is a substantial reduction in footprint, around 25x by Axiom's own measurements, up to 50x for more structured events like OpenTelemetry spans.
If you've heard "our volume outgrew the budget we had for keeping it," that's a sentence about tools priced around row-oriented storage and per-host indexing. Keeping every event is a one-time storage decision that removes the need to guess.
Keep every age of data on one query path
Full-fidelity retention only helps if old data is still easy to ask questions of. A lot of systems solve the cost problem by moving old data somewhere cheaper and harder to reach: a cold archive, a separate search index, a restore ticket that takes a day to resolve before you can run a query.
We keep one query path across every age of data. Events first land in a "write-ahead log" (a durable, columnar record of everything as it arrives) and become queryable from there immediately. As that live state accumulates, it's compacted into immutable, compressed columnar blocks in object storage. A query doesn't need to track an event's current stage. It reads the write-ahead log and the settled blocks as one path, and a question about data from six months ago returns the same way as a question about data from this morning. There's no restore step in between.
That continuity turns retention into a policy decision, free of any architecture constraint. You decide how long to keep data based on its value to you alone. Adaptive Tiering builds on that by tracking query recency per record. Data queried within the last 30 days stays in the faster Frequent tier. Data that goes unqueried for 30 days or more moves to the cheaper Infrequent tier, and data unqueried for 90 days or more moves to Archive Instant Access. Each move happens automatically, with no restore step. The query that hits six-month-old data runs the same way, whatever tier holds it.
Separate storage from query compute
The last piece of the structure is the relationship between storage and compute, because that relationship decides whether keeping more history costs a bigger standing cluster or only a bigger storage bill.
Once events compress into columnar blocks in object storage, that storage only holds the data durably and cheaply. It doesn't run queries. When a query comes in, serverless query workers spin up on demand and read only the columns that query needs, skipping the rest of the block. Compute is an entirely separate resource from storage, provisioned for the duration of the query and released when it's done. Compute scales to each query, so it never needs to run as a standing cluster sized for the worst-case hour.
That separation lets wide events and long retention share one path without a tradeoff. Adding more fields or keeping events longer grows the object storage footprint without adding standing query capacity. A broad investigation can recruit extra compute for the minutes it runs, then release it, without raising your standing compute footprint. Storage and compute each do their own job, so capacity follows current usage.
A machine data lake is a design choice
Building one means making these six decisions deliberately, so the event an investigation needs is still there and still fast to reach, no matter how old it is. Each decision is made once, in configuration, and then it keeps paying off as volume grows. That's the idea behind Axiom as a modern machine data platform: all six decisions come built in, so your team configures them instead of building and running the infrastructure behind them.
FAQs
No. The schema is derived from the data you ingest, not declared ahead of time. Structured collectors such as OpenTelemetry are parsed at ingest and turn into queryable columns immediately; query-time parsing and virtual fields only come into play for older, unstructured data like a raw syslog message.
Logs, traces, and metrics are different kinds of data: a log line or span is a standalone record, while a metric only means something next to the points around it. We recommend one dataset each for logs, traces, and metrics, set once in the collector's exporter config, so the layout doesn't need to be re-decided per team or per service.
They go into a dedicated OCSF dataset. We promote every documented OCSF attribute into a typed column up to a fixed depth, and everything else remains in a single unmapped field, never dropped and never forcing a schema migration when a new source shows up with a field you haven't seen yet.
Yes. An OCSF-producing source such as Cribl can send data into Axiom, and Axiom for Splunk or Axiom Portal for Splunk lets a SOC keep querying that dataset in SPL while Splunk ES or SOAR continues detection and case work. The usual path is to land high-volume or orphaned sources beside the SIEM first, then expand once the model is proven.
Through storage rather than filtering. Columnar compression is the reason full-fidelity retention is affordable: EventDB's encoding typically cuts footprint by around 25x, up to 50x for more structured data like OpenTelemetry spans, so nothing has to be sampled away to stay inside budget.
No. Events land in a write-ahead log and settle into compressed columnar blocks in object storage, but a query reads both as one path. Retention becomes a policy decision, and Adaptive Tiering moves data between access tiers with no restore step, so an old query and a recent one return the same way.
No. Storage and compute are separate resources. Wider events or longer retention grow the object storage footprint, but serverless query workers spin up only when a query runs and read only the columns it needs, so compute scales to each query, never running as a standing cluster sized for worst-case volume.
Related reading
See the structure on your own data
If you're deciding on homes for logs, traces, metrics, and security events, a walkthrough on your own sources will settle it faster than another architecture diagram. See Axiom on your data and query a week of it in APL.
