Monitor and troubleshoot the Splunk Portal
Learn how to read Axiom metrics in the Splunk Job Inspector, what each search banner means, and how to resolve common issues.
The Splunk Portal never degrades silently. Every degraded path surfaces a visible banner on the search job, and every search reports Axiom-side metrics in Splunk’s Job Inspector. Start there.
Inspect a search
On any federated search, click Job, and then select Inspect Job. The Execution costs section includes axiom.* components that show what happened on the Axiom side:
- Events scanned in Axiom. For pushed-down aggregations, this reflects the full scan, even though only compact result rows crossed the wire.
- Axiom query time and end-to-end time.
- Rows in and rows out.
If the axiom.* components are missing, the response was served from the provider’s short-lived cache. Run a fresh query to see them.
Banners
Banners on federated searches are prefixed with [axiom,<provider>].
| Banner | Meaning | What to do |
|---|---|---|
Axiom: aggregation pushed down (N events aggregated at the source) | INFO. The aggregation ran inside Axiom, and results are exact regardless of size. | Nothing. This is the good path. |
Axiom: aggregation computed over sampled events… | WARN. A command that runs on the search head needed more events than the response budget, so it ran over a sample. | Narrow the time range or filters, or use a pushed-down function. See SPL command support. |
Axiom: results limited to N of the M requested events… | INFO. Raw event retrieval reached the provider’s response budget before satisfying head N. | Narrow the search, or keep fewer columns with fields, to fit more events. |
Axiom: one or more fields are not present in the dataset… | INFO. A filter referenced a field the dataset doesn’t have. Splunk semantics: no matches. | Check the field name against the dataset schema. |
Axiom authentication failed… | FATAL. Axiom rejected the credential. | Set the provider’s service account password to the token value, the string starting with xaat-, not the token’s name. Check the token isn’t expired or revoked. |
Axiom Portal: no credentials received… | FATAL. The request arrived without a service account password. | Set the provider’s service account password to your Axiom API token, with your Axiom organization ID as the username. |
Axiom rate limit exceeded… | ERROR. The Axiom organization hit its query rate limit. | Retry, or reduce dashboard refresh frequency. |
Common issues
Test connection succeeds, but searches report an authentication failure.
Splunk’s Test connection verifies DNS, TLS, protocol compatibility, and that a credential reached the Portal, but it doesn’t run a query. A well-formed but expired, revoked, or mistyped token can pass the test. Run index=federated:<name> | stats count as the real credential check. If it shows the authentication banner, replace the service account password with the token value. The most common cause is entering the token’s name instead of the xaat- token string.
A search returns 0 rows, but the data exists.
Check the time range first: the search’s earliest and latest are applied to the Axiom query as-is. Then check the field name. Dotted names must be quoted in SPL: "service.name"=frontend. A bare index= search with a bad credential can also look like an empty result. Run | stats count to surface any authentication banner.
Counts differ between two runs of the same search. Continuously ingesting datasets see new data between runs. Compare with identical, closed time ranges.
Daily timechart bars look shifted by a few hours. Bins of one day and longer align to the provider’s configured time zone, UTC by default. This mirrors native Splunk, where a provider bins in its own time zone and the search head doesn’t re-bin. Sub-day spans are timezone-independent and never need attention. Contact Axiom to align daily bins to your users’ time zone.
A lookup or data model doesn’t work. Lookups and data models require transparent mode. Scripted, external, and KV store lookups can’t run on any federated provider. CSV lookups work fully.
Results look inconsistent across searches. Check that only one mode is registered. Pointing a standard and a transparent provider at the same endpoint yields inconsistent results. Use one mode per deployment.
case(f>=500, …) returns nothing.
This is native Splunk behavior when the field holds string values. The same search fails on local indexes. Use case(tonumber(f)>=500, …).
Get help
Visit axiom.co/support or email support@axiom.co. Include the Splunk version, the mode, the search you ran, and the text of any banner.