Send data from Fluent Bit to Axiom
This step-by-step guide will help you collect any data like metrics and logs from different sources, enrich them with filters, and send them to Axiom.
Fluent Bit
Fluent Bit is an open-source Log Processor and Forwarder that allows you to collect any data like metrics and logs from different sources, enrich them with filters, and send them to multiple destinations like Axiom.
Installation
Visit the Fluent Bit download page to install Fluent Bit on your system.
You’d need to specify the org-id header if you are using personal token, it’s best to use an API token to avoid the need to specify the org-id header.
Learn more about API and personal token
Configuration
Fluent Bit configuration file supports four types of sections:
- Service: Defines global properties of your service using different keys available for a specific version.
- Input: Defines the input plugin and base configuration of your file.
- Filter: Defines the input plugin and configure the pattern tags for your configuration.
- Output: Specify a destination that certain records should follow after a Tag match.
All sections are configured in your .conf
file.
Example
The example below shows fluent Bit configuration that sends data to Axiom:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
Fluent Bit filters
Fluent Bit provides several filter plugins that can be used to modify the logs. These filters can be added to the configuration file in the [FILTER]
section.
Here’s how you can do it:
AWS ECS filter
For AWS ECS, you can use the grep
filter which enriches logs with Amazon ECS metadata:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[FILTER]
Name grep
Match *
Regex ecs_task_arn .*app1.*
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
Kubernetes Filter
The kubernetes
filter enriches logs with Kubernetes metadata:
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[FILTER]
Name kubernetes
Match *
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
WASM Filter
Fluent Bit allows the usage of WebAssembly (WASM) based filters.
[SERVICE]
Flush 5
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag cpu
[FILTER]
Name wasm
Match *
Path /path/to/wasm/filter.wasm
public_token xxxxxxxxxxx
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an API token
Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
compress gzip
format json
json_date_key _time
json_date_format iso8601
tls On
Send logs from Docker Compose with Fluent Bit
This section outlines how to configure Fluent Bit with Docker Compose to forward logs to Axiom. It includes setting up fluent-bit.conf
for log processing and docker-compose.yaml
for deploying Fluent Bit as a container. The setup captures logs from various system metrics, logs, and forwards them to Axiom.
Create Fluent Bit configuration file (fluent-bit.conf)
Replace $DATASET
with your Axiom dataset name and $API_TOKEN
with your Axiom API token.
[SERVICE]
Flush 1
Daemon off
Log_Level debug
[INPUT]
Name cpu
Tag system.cpu
Interval_Sec 5
[INPUT]
Name mem
Tag system.mem
Interval_Sec 5
[INPUT]
Name forward
Listen 0.0.0.0
port 24224
[INPUT]
Name netif
Tag netif
Interval_Sec 1
Interval_NSec 0
Interface eth0
[INPUT]
Name disk
Tag disk
Interval_Sec 1
Interval_NSec 0
[FILTER]
Name record_modifier
Match *
Record hostname ${HOSTNAME}
[OUTPUT]
Name http
Match *
Host api.axiom.co
Port 443
URI /v1/datasets/$DATASET_NAME/ingest
Header Authorization Bearer $API_TOKEN
Compress gzip
Format json
JSON_Date_Key _time
JSON_Date_Format iso8601
TLS On
Create Docker Compose file (docker-compose.yaml)
Ensure the volumes
section correctly maps the fluent-bit.conf
file to /fluent-bit/etc/fluent-bit.conf
inside the container with read-only access.
version: '3'
services:
fluentbit:
image: fluent/fluent-bit:latest
container_name: fluent-bit
user: root # Required for accessing host log files
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
- /var/lib/docker/containers:/opt/docker-container-logs:ro
environment:
- AXIOM_HOSTNAME=axiom
To start the Fluent Bit container using the Docker Compose configuration you’ve set up, execute the docker-compose up -d
command.
Was this page helpful?