> ## Documentation Index
> Fetch the complete documentation index at: https://axiom.co/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Send data from Fluentd to Axiom

> This page explains how to collect, aggregate, analyze, and route log files from multiple Fluentd sources into Axiom

Fluentd is an open-source log collector that allows you to collect, aggregate, process, analyze, and route log files.

With Fluentd, you can collect logs from multiple sources and ship it instantly into Axiom

## Prerequisites

* [Create an Axiom account](https://app.axiom.co/register).
* [Create a dataset in Axiom](/reference/datasets#create-dataset) where you send your data.
* [Create an API token in Axiom](/reference/tokens) with permissions to ingest data to the dataset you have created.

## Installation

Visit the [Fluentd download page](https://www.fluentd.org/download) to install Fluentd on your system.

## Configuration

Fluentd lifecycle consist of five different components which are:

* Setup: Configure your `fluent.conf` file.
* Inputs: Define your input listeners.
* Filters: Create a rule to allow or disallow an event.
* Matches: Send output to Axiom when input data match and pair specific data from your data input within your configuration.
* Labels: Groups filters and simplifies tag handling.

When setting up Fluentd, the configuration file `.conf` is used to connect its components.

## Configuring Fluentd using the HTTP output plugin

The example below shows a Fluentd configuration that sends data to Axiom using the [HTTP output plugin](https://docs.fluentd.org/output/http):

```xml theme={null}
<source>
  @type forward
  port 24224
</source>

<match *.**>
  @type http

  endpoint https://AXIOM_DOMAIN/v1/ingest/DATASET_NAME
  # Authorization Bearer should be an ingest token
  headers {"Authorization": "Bearer API_TOKEN"}
  json_array false
  open_timeout 3

  <format>
    @type json
  </format>

  <buffer>
    flush_interval 5s
  </buffer>
</match>
```

<Info>
  Replace `AXIOM_DOMAIN` with the base domain of your edge deployment. For more information, see [Edge deployments](/reference/edge-deployments).

  Replace `API_TOKEN` with the Axiom API token you have generated. For added security, store the API token in an environment variable.

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>

## Configuring Fluentd using the OpenSearch output plugin

The example below shows a Fluentd configuration that sends data to Axiom using the [OpenSearch plugin](https://docs.fluentd.org/output/opensearch):

```xml theme={null}
<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/*.log
  tag td.logs
</source>

<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'AXIOM_DOMAIN'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
</match>
```

<Info>
  Replace `AXIOM_DOMAIN` with the base domain of your edge deployment. For more information, see [Edge deployments](/reference/edge-deployments).

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>

## Configure buffer interval with filter patterns

The example below shows a Fluentd configuration to hold logs in memory with specific flush intervals, size limits, and how to exclude specific logs based on patterns.

```xml theme={null}
# Collect common system logs
<source>
  @type tail
  @id system_logs
  <parse>
    @type none
  </parse>
  path /var/log/*.log
  pos_file /var/log/fluentd/system.log.pos
  read_from_head true
  tag system.logs
</source>

# Collect Apache2 logs (if they’re located in /var/log/apache2/)
<source>
  @type tail
  @id apache_logs
  <parse>
    @type apache2
  </parse>
  path /var/log/apache2/*.log
  pos_file /var/log/fluentd/apache2.log.pos
  read_from_head true
  tag apache.logs
</source>

# Filter to exclude certain patterns (optional)
<filter **>
  @type grep
  <exclude>
    key message
    pattern /exclude_this_pattern/
  </exclude>
</filter>

# Send logs to Axiom
<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'AXIOM_DOMAIN'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"

  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>
```

<Info>
  Replace `AXIOM_DOMAIN` with the base domain of your edge deployment. For more information, see [Edge deployments](/reference/edge-deployments).

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>

## Collect and send PHP logs to Axiom

The example below shows a Fluentd configuration that sends PHP data to Axiom.

```xml theme={null}
# Collect PHP logs
<source>
  @type tail
  @id php_logs
  <parse>
    @type multiline
    format_firstline /^\[\d+-\w+-\d+ \d+:\d+:\d+\]/
    format1 /^\[(?<time>\d+-\w+-\d+ \d+:\d+:\d+)\] (?<message>.*)/
  </parse>
  path /var/log/php*.log
  pos_file /var/log/fluentd/php.log.pos
  read_from_head true
  tag php.logs
</source>

# Send PHP logs to Axiom
<match php.logs>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'AXIOM_DOMAIN'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'php-logs'}"

  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>
```

<Info>
  Replace `AXIOM_DOMAIN` with the base domain of your edge deployment. For more information, see [Edge deployments](/reference/edge-deployments).

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>

## Collect and send Scala logs to Axiom

The example below shows a Fluentd configuration that sends Scala data to Axiom

```xml theme={null}
# Collect Scala logs
<source>
  @type tail
  @id scala_logs
  <parse>
    @type multiline
    format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}/
    format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<thread>.*)\] (?<level>\w+) (?<class>[\w\.$]+) - (?<message>.*)/
  </parse>
  path /var/log/scala-app.log
  pos_file /var/log/fluentd/scala.log.pos
  read_from_head true
  tag scala.logs
</source>

# Send Scala logs using HTTP plugin to Axiom
<match scala.logs>
  @type http
  endpoint "#{ENV['FLUENT_HTTP_ENDPOINT'] || 'https://AXIOM_DOMAIN/v1/ingest/DATASET_NAME'}"
  headers {"Authorization": "Bearer #{ENV['FLUENT_HTTP_TOKEN'] || '<your-token>'}"}
  <format>
    @type json
  </format>

  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>
```

<Info>
  Replace `AXIOM_DOMAIN` with the base domain of your edge deployment. For more information, see [Edge deployments](/reference/edge-deployments).

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>

## Send virtual machine logs to Axiom using the HTTP output plugin

The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the `apache` source type.

```xml theme={null}
<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/**/*.log
  pos_file /var/log/fluentd/fluentd.log.pos
  tag vm.logs
  read_from_head true
</source>

<filter vm.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    service "vm_service"
  </record>
</filter>

<match vm.logs>
  @type http
  @id out_http_axiom
  @log_level info
  endpoint "#{ENV['AXIOM_URL'] || 'https://api.axiom.co'}"
  path "/v1/ingest/DATASET_NAME"
  ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
  <headers>
    Authorization "Bearer API_TOKEN"
    Content-Type "application/json"
  </headers>
  <format>
    @type json
  </format>
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 5MB
    retry_forever true
  </buffer>
</match>
```

<Info>
  Replace `API_TOKEN` with the Axiom API token you have generated. For added security, store the API token in an environment variable.

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>

The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the `nginx` source type.

```xml theme={null}
<source>
  @type tail
  @id input_tail
  <parse>
    @type nginx
  </parse>
  path /var/log/nginx/access.log, /var/log/nginx/error.log
  pos_file /var/log/fluentd/nginx.log.pos
  tag nginx.logs
  read_from_head true
</source>

<filter nginx.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    service "nginx"
  </record>
</filter>

<match nginx.logs>
  @type http
  @id out_http_axiom
  @log_level info
  endpoint "#{ENV['AXIOM_URL'] || 'https://api.axiom.co'}"
  path "/v1/ingest/DATASET_NAME"
  ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
  <headers>
    Authorization "Bearer API_TOKEN"
    Content-Type "application/json"
  </headers>
  <format>
    @type json
  </format>
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 5MB
    retry_forever true
  </buffer>
</match>
```

<Info>
  Replace `API_TOKEN` with the Axiom API token you have generated. For added security, store the API token in an environment variable.

  Replace `DATASET_NAME` with the name of the Axiom dataset where you send your data.
</Info>