Send FluentD logs to Axiom

FluentD

FluentD is an open-source log collector that allows you to collect, aggregate, process, analyze, and route log files.

With FluentD, you can collect logs from multiple sources and ship it instantly into Axiom


Installation

Visit the FluentD download page to install FluentD on your system.


You'd need to specify the org-id header if you are using Personal Token, it's best to use an API Token to avoid the need to specify the org-id header.

Learn more about API and Personal Token


Configuration

FluentD lifecycle consist of five different components which are:

  • Setup: Configure your fluent.conf file.
  • Inputs: Define your input listeners.
  • Filters: Create a rule to allow or disallow an event.
  • Matches: Send output to Axiom when input data match and pair specific data from your data input within your configuration.
  • Labels: Groups filters and simplifies tag handling.

When setting up fluentD, the configuration file .conf is used to connect its components.


Configuring FluentD using the HTTP Output Plugin

The example below shows a FluentD configuration that sends data to Axiom using the http output plugin:

<source>
  @type forward
  port 24224
</source>

<match *.**>
  @type http

  endpoint https://api.axiom.co/v1/datasets/$DATASET_NAME/ingest
  # Authorization Bearer should be an ingest token
  headers {"Authorization": "Bearer <your-token>"}
  json_array false
  open_timeout 3

  <format>
    @type json
  </format>

  <buffer>
    flush_interval 5s
  </buffer>
</match>

Configuring FluentD using the Opensearch Output Plugin

The example below shows a FluentD configuration that sends data to Axiom using the opensearch plugin:

<source>
  @type tail
  @id input_tail
  <parse>
    @type apache2
  </parse>
  path /var/log/*.log
  tag td.logs
</source>

<match **>
  @type opensearch
  @id out_os
  @log_level info
  include_tag_key true
  include_timestamp true
  host "#{ENV['FLUENT_OPENSEARCH_HOST']  || 'cloud.axiom.co'}"
  port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
  path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/api/v1/datasets/$DATASET_NAME/elastic'}"
  scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
  ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
  ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
  password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
  index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
</match>

Was this page helpful?