Understanding Splunk's Machine Data Lake

The sections below cover what sits in the lake, how static and streaming promotion differ, why Splunk indexes only a subset, and how Axiom keeps full event fields in one query path so older data does not need a promotion job.

What the Machine Data Lake is

The Machine Data Lake is a raw-table landing zone in managed object storage. Logs and events that take this path are stored without being indexed in the lake copy. You can still dual-route the same events to an existing Splunk index when current dashboards and alerts must keep using that index. You use a Catalog on the raw tables to see which sources exist, roughly what they contain, and where they sit in time. Catalog metadata refreshes on a background schedule, so it is a discovery layer, not a real-time index of every field.

Raw search against the lake locates and inspects data using a bounded time range plus source, sourcetype, host, or raw-text criteria. It is a poor fit for the repeated, high-performance investigation you run during an incident. Object storage is priced for capacity, not for the low-latency, field-rich search an incident requires. Splunk's raw search is built around that constraint: you can find and inspect events on those metadata fields, but you do not get the dashboard-ready, alerting-ready experience of an index.

SmartStore is easy to mix up with the lake. SmartStore moves warm indexed buckets to remote object storage; an uncached bucket still has to download before Splunk can search it. The Machine Data Lake holds a raw, unindexed copy. SmartStore is a storage tier for something already indexed. The lake is the layer data sits in before promotion, unless you also dual-routed it into an index at landing.

The two-state model: raw and promoted

Once raw data is sitting in the lake, you decide which of it needs to become active. That job is promotion.

Static promotion covers a historical range

Static promotion takes a defined window of raw history (a day, a month, a quarter) and promotes it into an index or analytics table. You use it when you already know which past period matters: a security review of a specific source, or a capacity exercise over a full quarter of a metric-adjacent log.

Streaming promotion covers what happens next

Once a streaming promotion job is active, matching events that arrive after that point join a Splunk index as they land. Streaming promotion does not target an analytics table, does not backfill events already sitting in the raw table, and counts against Ingest Processor capacity. You use a separate static job for history, and streaming for a source that needs dashboards and alerts going forward.

The raw source can remain in the lake after promotion. The promoted target gets its own representation in an index or analytics table, with its own retention. Those are two copies, sized and governed separately. Where Splunk enables it, Open Sharing can give an authorized external tool read-only access to selected lake data without a promotion job.

Why promotion exists at all

Splunk's indexed search (the fast, field-rich, dashboard-ready experience) is expensive to build and expensive to keep running at full breadth. Splunk extracts core metadata at index time; most field extraction happens at search time. Indexing every byte at ingest would cost about the same as classic indexed Splunk, which is why the lake exists as a cheaper raw tier.

You land volume in the lake and keep it narrowly searchable. You promote the subset that needs fast, repeated search, dashboards, alerts, scheduled searches, and correlation across promoted sources. That split is how Splunk currently offers full-fidelity retention at a lower cost than indexing everything. Dashboards, alerts, and correlation only run against data in a Splunk index, whether you promoted it or dual-routed it there.

You choose what to promote before you know which query you need

You have to decide, before the investigation starts, which data is worth promoting, on what schedule, and for how long. Those choices are made with incomplete information. A field that looked irrelevant at ingest can be the one that separates an affected customer segment. A source that seemed safe to leave in the lake can be the exact place a security review needs fast, repeated, cross-referenced search.

You can promote that source later. Until the static or streaming job finishes, the new promoted dataset is not searchable. Promotion does not add events to an existing index. Dual-route if current dashboards must keep using that index. The delay is the time between realizing you need the data operational and the promotion job completing.

Axiom's live and settled path

Axiom writes to object storage too. The difference is what happens to the data once it is there: each immutable block stays in the ordinary query path, surrounded by the layout, metadata, planner, and execution machinery needed to query it, so data from a minute ago and data from ten years ago are queried the same way, with every field intact.

Axiom also has two states, live and settled. Both stay in the same query path. Recent events are queryable through a live columnar path. As they age, they settle into compressed, field-aware blocks on object storage that keep their full field structure. You do not run a separate promotion job to reach an older event's fields with the query language you already use.

A field that turns out to matter three weeks after ingest is still there, with its full shape, answerable by the same query you would run against the last hour. There is no historical-range job to schedule and no streaming job to turn on going forward. You do not have to pick which data earns richer treatment in advance. Events you retain keep full fidelity from the moment they arrived.

Where Axiom sits next to an existing Splunk estate

You can keep the Machine Data Lake and add Axiom beside the existing Splunk estate. Data can reach Axiom through a Splunk HEC-compatible endpoint, so existing HEC clients and HEC-path forwarders keep working.

From there you choose how Splunk reaches that data. The Axiom for Splunk app adds ax commands to SPL, so analysts can query Axiom datasets, enrich existing Splunk events with Axiom context, or run APL from a familiar search. Axiom datasets appear as ordinary Splunk indexes through the Axiom Portal for Splunk, so existing dashboards, saved searches, and alerts keep working against data that never went through a promotion step. Aggregations are pushed down and computed in Axiom, so results are exact, and because the Portal runs at search time it consumes no Splunk ingest license.

The result is a full-fidelity machine data platform beside Splunk. Sources you send stay queryable from the search bar analysts already use. A common first step is to send sources that would otherwise sit un-promoted in a lake to Axiom, then decide at renewal what else to move, with the Splunk estate still running. How Axiom and Splunk have diverged explains the retention difference that makes that coexistence possible: Axiom keeps full fields in one query path, while Splunk's lake keeps raw data until you promote it.

Deciding in advance what becomes active data creates the same gap on any stack that discards or downsamples early. If a field or source later needs dashboards, alerts, or cross-referenced search, Axiom can query the retained events without waiting on a promotion job.

FAQs

Related Reading

• Sampling could be a costly mistake

• Teaching AI to speak Splunk, then proving it works

• A new look for the modern machine data platform

Try Axiom on a source that is sitting in the lake

Bring one source that currently sits un-promoted in a lake, or an index you have been meaning to expand. See Axiom on your data and query the same source in full, with no promotion step in between.