Send Fluent Bit logs to Axiom
Fluent Bit
Fluent Bit is an open-source Log Processor and Forwarder that allows you to collect any data like metrics and logs from different sources, enrich them with filters, and send them to multiple destinations like Axiom.
Installation
Visit the Fluent Bit download page to install Fluent Bit on your system.
You'd need to specify the org-id header if you are using Personal Token, it's best to use an API Token to avoid the need to specify the org-id header.
Learn more about API and Personal Token
Configuration
Fluent Bit configuration file supports four types of sections:
- Service: Defines global properties of your service using different keys available for a specific version.
- Input: Defines the input plugin and base configuration of your file.
- Filter: Defines the input plugin and configure the pattern tags for your configuration.
- Output: Specify a destination that certain records should follow after a Tag match.
All sections will be configured in your .conf
file.
Example
The example below shows fluent Bit configuration that sends data to Axiom:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
Fluent Bit Filters
Fluent Bit provides several filter plugins that can be used to modify the logs. These filters can be added to the configuration file in the [FILTER]
section.
Here's how you can do it:
AWS ECS Filter
For AWS ECS, you can use the grep
filter which enriches logs with Amazon ECS metadata:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[FILTER]
Name grep
Match *
Regex ecs_task_arn .*app1.*
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
Kubernetes Filter
The kubernetes
filter enriches logs with Kubernetes metadata:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[FILTER]
Name kubernetes
Match *
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
Wasm Filter
Fluent Bit allows the usage of WebAssembly (Wasm) based filters.
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[FILTER]
Name wasm
Match *
Path /path/to/wasm/filter.wasm
public_token xxxxxxxxxxx
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On