Send data from Tremor to Axiom
This step-by-step guide will help you configure Tremor connectors and events components to interact with your databases, APIs, and ingest data from these sources into Axiom.
Axiom provides a unique way of ingesting Tremor logs into Axiom. With your connector definitions, you can configure Tremor connectors and events components to interact with your external systems, such as databases, message queues, or APIs, and eventually ingest data from these sources into Axiom.
Installation
To install tremor grab the latest package from the runtime releases tag, and install it on your local machine.
Configuration using HTTP
To send logs via Tremor to Axiom, you need to create a configuration file. For example, create axiom-http.troy
with the following content (using a file as example data source):
define flow client_sink_only
flow
use std::time::nanos;
use tremor::pipelines;
define connector input from file
args
file = "in.json" # Default input file is 'in.json' in current working directory
with
codec = "json", # Data is JSON encoded
preprocessors = ["separate"], # Data is newline separated
config = {
"path": args.file,
"mode": "read"
},
end;
create connector input;
define connector http_client from http_client
args
dataset,
token
with
config = {
"url": "https://api.axiom.co/v1/datasets/#{args.dataset}/ingest",
"tls": true,
"method": "POST",
"headers": {
"Authorization": "Bearer #{args.token}"
},
"timeout": nanos::from_seconds(10),
"mime_mapping": {
"*/*": {"name": "json"},
}
}
end;
create connector http_client
with
dataset = "$DATASET_NAME",
token = "$API_TOKEN"
end;
create pipeline passthrough from pipelines::passthrough;
connect /connector/input to /pipeline/passthrough;
connect /pipeline/passthrough to /connector/http_client;
end;
deploy flow client_sink_only;
This assumes you have set TREMOR_PATH
in your environment pointing to tremor-runtime/tremor-script/lib
if you are using a src
clone then you can execute it as follows tremor server run axiom-http.troy
The$DATASET_NAME
dataset you want to send logs to in Axiom, and the $API_TOKEN
is your Axiom API token for ingesting and quering your Tremor logs.
Configuration using Syslog
You can also send logs via Tremor to the Syslog endpoint using a file as an example data source.
- Click Settings > Endpoints.
- Click New endpoint.
- Click .
- Name the endpoint.
- Select the dataset where you want to send data.
- Copy the URL displayed for the newly created endpoint. This is the target URL where you send the data.
In the code below, replace url
with the URL of your Syslog endpoint.
define flow client_sink_only
flow
use std::time::nanos;
use tremor::pipelines;
define connector input from file
args
file = "in.json" # Default input file is 'in.json' in current working directory
with
codec = "json", # Data is JSON encoded
preprocessors = ["separate"], # Data is newline separated
config = {
"path": args.file,
"mode": "read"
},
end;
create connector input;
define connector syslog_forwarder from tcp_client
args
endpoint_hostport,
with
tls = true,
codec = "syslog",
config = {
"url": "#{args.endpoint_hostport}",
"no_delay": false,
"buf_size": 1024,
},
reconnect = {
"retry": {
"interval_ms": 100,
"growth_rate": 2,
"max_retries": 3,
}
}
end;
create connector syslog_forwarder
with
endpoint_hostport = "tcp+tls://testsyslog.syslog.axiom.co:6514"
end;
create pipeline passthrough from pipelines::passthrough;
connect /connector/input to /pipeline/passthrough;
connect /pipeline/passthrough to /connector/syslog_forwarder;
end;
deploy flow client_sink_only;