Fluentd is an open-source log collector that allows you to collect, aggregate, process, analyze, and route log files.

With Fluentd, you can collect logs from multiple sources and ship it instantly into Axiom

​
Prerequisites

​
Installation

Visit the Fluentd download page to install Fluentd on your system.

​
Configuration

Fluentd lifecycle consist of five different components which are:

  • Setup: Configure your fluent.conf file.
  • Inputs: Define your input listeners.
  • Filters: Create a rule to allow or disallow an event.
  • Matches: Send output to Axiom when input data match and pair specific data from your data input within your configuration.
  • Labels: Groups filters and simplifies tag handling.

When setting up Fluentd, the configuration file .conf is used to connect its components.

​
Configuring Fluentd using the HTTP output plugin

The example below shows a Fluentd configuration that sends data to Axiom using the HTTP output plugin:

<source>
  @type forward
  port 24224
</source>

<match *.**>
  @type http

  endpoint https://api.axiom.co/v1/datasets/DATASET_NAME/ingest
  # Authorization Bearer should be an ingest token
  headers {"Authorization": "Bearer API_TOKEN"}
  json_array false
  open_timeout 3

  <format>
    @type json
  </format>

  <buffer>
    flush_interval 5s
  </buffer>
</match>
  • Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
  • Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​
Configuring Fluentd using the OpenSearch output plugin

The example below shows a Fluentd configuration that sends data to Axiom using the OpenSearch plugin:

<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/*.log
  tag td.logs
</source>

<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'api.axiom.co'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
</match>

Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​
Configure buffer interval with filter patterns

The example below shows a Fluentd configuration to hold logs in memory with specific flush intervals, size limits, and how to exclude specific logs based on patterns.

# Collect common system logs
<source>
  @type tail
  @id system_logs
  <parse>
    @type none
  </parse>
  path /var/log/*.log
  pos_file /var/log/fluentd/system.log.pos
  read_from_head true
  tag system.logs
</source>

# Collect Apache2 logs (if they’re located in /var/log/apache2/)
<source>
  @type tail
  @id apache_logs
  <parse>
    @type apache2
  </parse>
  path /var/log/apache2/*.log
  pos_file /var/log/fluentd/apache2.log.pos
  read_from_head true
  tag apache.logs
</source>

# Filter to exclude certain patterns (optional)
<filter **>
  @type grep
  <exclude>
    key message
    pattern /exclude_this_pattern/
  </exclude>
</filter>

# Send logs to Axiom
<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'api.axiom.co'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"

  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​
Collect and send PHP logs to Axiom

The example below shows a Fluentd configuration that sends PHP data to Axiom.

# Collect PHP logs
<source>
  @type tail
  @id php_logs
  <parse>
    @type multiline
    format_firstline /^\[\d+-\w+-\d+ \d+:\d+:\d+\]/
    format1 /^\[(?<time>\d+-\w+-\d+ \d+:\d+:\d+)\] (?<message>.*)/
  </parse>
  path /var/log/php*.log
  pos_file /var/log/fluentd/php.log.pos
  read_from_head true
  tag php.logs
</source>

# Send PHP logs to Axiom
<match php.logs>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'api.axiom.co'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'php-logs'}"

  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​
Collect and send Scala logs to Axiom

The example below shows a Fluentd configuration that sends Scala data to Axiom

# Collect Scala logs
<source>
  @type tail
  @id scala_logs
  <parse>
    @type multiline
    format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}/
    format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<thread>.*)\] (?<level>\w+) (?<class>[\w\.$]+) - (?<message>.*)/
  </parse>
  path /var/log/scala-app.log
  pos_file /var/log/fluentd/scala.log.pos
  read_from_head true
  tag scala.logs
</source>

# Send Scala logs using HTTP plugin to Axiom
<match scala.logs>
  @type http
  endpoint "#{ENV['FLUENT_HTTP_ENDPOINT'] || 'https://api.axiom.co/v1/datasets/DATASET_NAME/ingest'}"
  headers {"Authorization": "Bearer #{ENV['FLUENT_HTTP_TOKEN'] || '<your-token>'}"}
  <format>
    @type json
  </format>

  <buffer>
    @type memory
    flush_mode interval
    flush_interval 10s
    chunk_limit_size 5M
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​
Send virtual machine logs to Axiom using the HTTP output plugin

The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the apache source type.

<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/**/*.log
  pos_file /var/log/fluentd/fluentd.log.pos
  tag vm.logs
  read_from_head true
</source>

<filter vm.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    service "vm_service"
  </record>
</filter>

<match vm.logs>
  @type http
  @id out_http_axiom
  @log_level info
  endpoint "#{ENV['AXIOM_URL'] || 'https://api.axiom.co'}"
  path "/v1/datasets/DATASET_NAME/ingest"
  ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
  <headers>
    Authorization "Bearer API_TOKEN"
    Content-Type "application/json"
  </headers>
  <format>
    @type json
  </format>
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 5MB
    retry_forever true
  </buffer>
</match>
  • Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
  • Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the nginx source type.

<source>
  @type tail
  @id input_tail
  <parse>
    @type nginx
  </parse>
  path /var/log/nginx/access.log, /var/log/nginx/error.log
  pos_file /var/log/fluentd/nginx.log.pos
  tag nginx.logs
  read_from_head true
</source>

<filter nginx.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    service "nginx"
  </record>
</filter>

<match nginx.logs>
  @type http
  @id out_http_axiom
  @log_level info
  endpoint "#{ENV['AXIOM_URL'] || 'https://api.axiom.co'}"
  path "/v1/datasets/DATASET_NAME/ingest"
  ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
  <headers>
    Authorization "Bearer API_TOKEN"
    Content-Type "application/json"
  </headers>
  <format>
    @type json
  </format>
  <buffer>
    @type memory
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 5MB
    retry_forever true
  </buffer>
</match>
  • Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
  • Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.