Send Tremor logs to Axiom
Axiom provides a unique way of ingesting Tremor logs into Axiom. With your connector definitions, you can configure Tremor connectors and events components to interact with your external systems, such as databases, message queues, or APIs, and eventually ingest data from these sources into Axiom.
Installation
To install tremor grab the latest package from the runtime releases tag, and install it on your local machine.
Configuration using HTTP
To send logs via Tremor to Axiom, you need to create a configuration file, e.g., axiom-http.troy, with the following content (using a file as example data source):
define flow client_sink_only
flow
use std::time::nanos;
use tremor::pipelines;
define connector input from file
args
file = "in.json" # Default input file is 'in.json' in current working directory
with
codec = "json", # Data is JSON encoded
preprocessors = ["separate"], # Data is newline separated
config = {
"path": args.file,
"mode": "read"
},
end;
create connector input;
define connector http_client from http_client
args
dataset,
token
with
config = {
"url": "https://api.axiom.co/v1/datasets/#{args.dataset}/ingest",
"tls": true,
"method": "POST",
"headers": {
"Authorization": "Bearer #{args.token}"
},
"timeout": nanos::from_seconds(10),
"mime_mapping": {
"*/*": {"name": "json"},
}
}
end;
create connector http_client
with
dataset = "$DATASET_NAME",
token = "$API_TOKEN"
end;
create pipeline passthrough from pipelines::passthrough;
connect /connector/input to /pipeline/passthrough;
connect /pipeline/passthrough to /connector/http_client;
end;
deploy flow client_sink_only;
This assumes you have set TREMOR_PATH
in your environment pointing to tremor-runtime/tremor-script/lib
if you are using a src clone then you can execute it as follows tremor server run axiom-http.troy
- The
$DATASET_NAME
dataset you want to send logs to in Axiom, and the$API_TOKEN
is your Axiom API token for ingesting and quering your Tremor logs.
Configuration using Syslog
You can also send logs via Tremor to the Axiom Syslog endpoint, if you're on the basic plan (using a file as an example data source).
define flow client_sink_only
flow
use std::time::nanos;
use tremor::pipelines;
define connector input from file
args
file = "in.json" # Default input file is 'in.json' in current working directory
with
codec = "json", # Data is JSON encoded
preprocessors = ["separate"], # Data is newline separated
config = {
"path": args.file,
"mode": "read"
},
end;
create connector input;
define connector syslog_forwarder from tcp_client
args
endpoint_hostport,
with
tls = true,
codec = "syslog",
config = {
"url": "#{args.endpoint_hostport}",
"no_delay": false,
"buf_size": 1024,
},
reconnect = {
"retry": {
"interval_ms": 100,
"growth_rate": 2,
"max_retries": 3,
}
}
end;
create connector syslog_forwarder
with
endpoint_hostport = "tcp+tls://testsyslog.syslog.axiom.co:6514"
end;
create pipeline passthrough from pipelines::passthrough;
connect /connector/input to /pipeline/passthrough;
connect /pipeline/passthrough to /connector/syslog_forwarder;
end;
deploy flow client_sink_only;
- Replace the
url
with the URL of your Syslog endpoint generated from Settings --> Endpoints --> New endpoint in the Axiom UI.