Send Fluent Bit logs to Axiom

Fluent Bit

Fluent Bit is an open-source Log Processor and Forwarder that allows you to collect any data like metrics and logs from different sources, enrich them with filters, and send them to multiple destinations like Axiom.

Installation

Visit the Fluent Bit download page to install Fluent Bit on your system.

You'd need to specify the org-id header if you are using Personal Token, it's best to use an API Token to avoid the need to specify the org-id header.

Learn more about API and Personal Token

Configuration

Fluent Bit configuration file supports four types of sections:

Service: Defines global properties of your service using different keys available for a specific version.
Input: Defines the input plugin and base configuration of your file.
Filter: Defines the input plugin and configure the pattern tags for your configuration.
Output: Specify a destination that certain records should follow after a Tag match.

All sections will be configured in your .conf file.

Example

The example below shows fluent Bit configuration that sends data to Axiom:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
    # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

Fluent Bit Filters

Fluent Bit provides several filter plugins that can be used to modify the logs. These filters can be added to the configuration file in the [FILTER] section.

Here's how you can do it:

AWS ECS Filter

For AWS ECS, you can use the grep filter which enriches logs with Amazon ECS metadata:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[FILTER]
    Name  grep
    Match *
    Regex ecs_task_arn .*app1.*

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
    # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

Kubernetes Filter

The kubernetes filter enriches logs with Kubernetes metadata:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[FILTER]
    Name             kubernetes
    Match            *
    Kube_URL         https://kubernetes.default.svc:443
    Merge_Log        On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
    # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On

Wasm Filter

Fluent Bit allows the usage of WebAssembly (Wasm) based filters.

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name cpu
    Tag  cpu

[FILTER]
    Name         wasm
    Match        *
    Path         /path/to/wasm/filter.wasm
    public_token xxxxxxxxxxx

[OUTPUT]
    Name            http
    Match           *
    Host            api.axiom.co
    Port            443
    URI             /v1/datasets/$DATASET_NAME/ingest
     # Authorization Bearer should be an API token
    Header Authorization Bearer xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    compress gzip
    format json
    json_date_key _time
    json_date_format iso8601
    tls On