​
Fluent Bit

Fluent Bit is an open-source Log Processor and Forwarder that allows you to collect any data like metrics and logs from different sources, enrich them with filters, and send them to multiple destinations like Axiom.

​
Installation

Visit the Fluent Bit download page to install Fluent Bit on your system.

You’d need to specify the org-id header if you are using personal token, it’s best to use an API token to avoid the need to specify the org-id header.

Learn more about API and personal token

​
Configuration

Fluent Bit configuration file supports four types of sections:

  • Service: Defines global properties of your service using different keys available for a specific version.
  • Input: Defines the input plugin and base configuration of your file.
  • Filter: Defines the input plugin and configure the pattern tags for your configuration.
  • Output: Specify a destination that certain records should follow after a Tag match.

All sections are configured in your .conf file.

​
Example

The example below shows fluent Bit configuration that sends data to Axiom:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
    # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

​
Fluent Bit filters

Fluent Bit provides several filter plugins that can be used to modify the logs. These filters can be added to the configuration file in the [FILTER] section.

Here’s how you can do it:

​
AWS ECS filter

For AWS ECS, you can use the grep filter which enriches logs with Amazon ECS metadata:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[FILTER]
    Name  grep
    Match *
    Regex ecs_task_arn .*app1.*

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
    # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

​
Kubernetes Filter

The kubernetes filter enriches logs with Kubernetes metadata:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[FILTER]
    Name             kubernetes
    Match            *
    Kube_URL         https://kubernetes.default.svc:443
    Merge_Log        On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
    # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

​
WASM Filter

Fluent Bit allows the usage of WebAssembly (WASM) based filters.

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[FILTER]
    Name         wasm
    Match        *
    Path         /path/to/wasm/filter.wasm
    public_token xxxxxxxxxxx

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
     # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

​
Send logs from Docker Compose with Fluent Bit

This section outlines how to configure Fluent Bit with Docker Compose to forward logs to Axiom. It includes setting up fluent-bit.conf for log processing and docker-compose.yaml for deploying Fluent Bit as a container. The setup captures logs from various system metrics, logs, and forwards them to Axiom.

​
Create Fluent Bit configuration file (fluent-bit.conf)

Replace $DATASET with your Axiom dataset name and $API_TOKEN with your Axiom API token.

[SERVICE]
    Flush        1
    Daemon       off
    Log_Level    debug

[INPUT]
    Name        cpu
    Tag         system.cpu
    Interval_Sec 5

[INPUT]
    Name        mem
    Tag         system.mem
    Interval_Sec 5

[INPUT]
    Name forward
    Listen 0.0.0.0
    port 24224

[INPUT]
    Name          netif
    Tag           netif
    Interval_Sec  1
    Interval_NSec 0
    Interface     eth0

[INPUT]
    Name          disk
    Tag           disk
    Interval_Sec  1
    Interval_NSec 0

[FILTER]
    Name         record_modifier
    Match        *
    Record       hostname ${HOSTNAME}

[OUTPUT]
    Name        http
    Match       *
    Host        api.axiom.co
    Port        443
    URI         /v1/datasets/$DATASET_NAME/ingest
    Header      Authorization Bearer $API_TOKEN
    Compress    gzip
    Format      json
    JSON_Date_Key _time
    JSON_Date_Format iso8601
    TLS         On

​
Create Docker Compose file (docker-compose.yaml)

Ensure the volumes section correctly maps the fluent-bit.conf file to /fluent-bit/etc/fluent-bit.conf inside the container with read-only access.

version: '3'

services:
  fluentbit:
    image: fluent/fluent-bit:latest
    container_name: fluent-bit
    user: root # Required for accessing host log files
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - /var/lib/docker/containers:/opt/docker-container-logs:ro
    environment:
      - AXIOM_HOSTNAME=axiom

To start the Fluent Bit container using the Docker Compose configuration you’ve set up, execute the docker-compose up -d command.

Was this page helpful?

Elastic Bulk APIFluentd