# Monitor and troubleshoot the Splunk Portal



The Splunk Portal never degrades silently. Every degraded path surfaces a visible banner on the search job, and every search reports Axiom-side metrics in Splunk’s Job Inspector. Start there.

## Inspect a search [#inspect-a-search]

On any federated search, click **Job**, and then select **Inspect Job**. The Execution costs section includes `axiom.*` components that show what happened on the Axiom side:

* Events scanned in Axiom. For pushed-down aggregations, this reflects the full scan, even though only compact result rows crossed the wire.
* Axiom query time and end-to-end time.
* Rows in and rows out.

If the `axiom.*` components are missing, the response was served from the provider’s short-lived cache. Run a fresh query to see them.

## Banners [#banners]

Banners on federated searches are prefixed with `[axiom,<provider>]`.

| Banner                                                               | Meaning                                                                                                            | What to do                                                                                                                                                        |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Axiom: aggregation pushed down (N events aggregated at the source)` | INFO. The aggregation ran inside Axiom, and results are exact regardless of size.                                  | Nothing. This is the good path.                                                                                                                                   |
| `Axiom: aggregation computed over sampled events…`                   | WARN. A command that runs on the search head needed more events than the response budget, so it ran over a sample. | Narrow the time range or filters, or use a pushed-down function. See [SPL command support](/splunk/portal/spl-support).                                           |
| `Axiom: results limited to N of the M requested events…`             | INFO. Raw event retrieval reached the provider’s response budget before satisfying `head N`.                       | Narrow the search, or keep fewer columns with `fields`, to fit more events.                                                                                       |
| `Axiom: one or more fields are not present in the dataset…`          | INFO. A filter referenced a field the dataset doesn’t have. Splunk semantics: no matches.                          | Check the field name against the dataset schema.                                                                                                                  |
| `Axiom authentication failed…`                                       | FATAL. Axiom rejected the credential.                                                                              | Set the provider’s service account password to the token value, the string starting with `xaat-`, not the token’s name. Check the token isn’t expired or revoked. |
| `Axiom Portal: no credentials received…`                             | FATAL. The request arrived without a service account password.                                                     | Set the provider’s service account password to your Axiom API token, with your Axiom organization ID as the username.                                             |
| `Axiom rate limit exceeded…`                                         | ERROR. The Axiom organization hit its query rate limit.                                                            | Retry, or reduce dashboard refresh frequency.                                                                                                                     |

## Common issues [#common-issues]

**Test connection succeeds, but searches report an authentication failure.**
Splunk’s **Test connection** verifies DNS, TLS, protocol compatibility, and that a credential reached the Portal, but it doesn’t run a query. A well-formed but expired, revoked, or mistyped token can pass the test. Run `index=federated:<name> | stats count` as the real credential check. If it shows the authentication banner, replace the service account password with the token value. The most common cause is entering the token’s name instead of the `xaat-` token string.

**A search returns 0 rows, but the data exists.**
Check the time range first: the search’s `earliest` and `latest` are applied to the Axiom query as-is. Then check the field name. Dotted names must be quoted in SPL: `"service.name"=frontend`. A bare `index=` search with a bad credential can also look like an empty result. Run `| stats count` to surface any authentication banner.

**Counts differ between two runs of the same search.**
Continuously ingesting datasets see new data between runs. Compare with identical, closed time ranges.

**Daily timechart bars look shifted by a few hours.**
Bins of one day and longer align to the provider’s configured time zone, UTC by default. This mirrors native Splunk, where a provider bins in its own time zone and the search head doesn’t re-bin. Sub-day spans are timezone-independent and never need attention. [Contact Axiom](https://axiom.co/contact) to align daily bins to your users’ time zone.

**A lookup or data model doesn’t work.**
Lookups and data models require [transparent mode](/splunk/portal/set-up-transparent). Scripted, external, and KV store lookups can’t run on any federated provider. CSV lookups work fully.

**Results look inconsistent across searches.**
Check that only one mode is registered. Pointing a standard and a transparent provider at the same endpoint yields inconsistent results. Use one mode per deployment.

**`case(f>=500, …)` returns nothing.**
This is native Splunk behavior when the field holds string values. The same search fails on local indexes. Use `case(tonumber(f)>=500, …)`.

## Get help [#get-help]

Visit [axiom.co/support](https://axiom.co/support) or email [support@axiom.co](mailto:support@axiom.co). Include the Splunk version, the mode, the search you ran, and the text of any banner.