Send data from Elastic Beats to Axiom

Elastic Beats serves as a lightweight platform for data shippers that transfer information from the source to Axiom and other tools based on the configuration. Before shipping data, it collects metrics and logs from different sources, which later are deployed to your Axiom deployments.

There are different Elastic Beats you could use to ship logs. Axiom’s documentation provides a detailed step by step procedure on how to use each Beats.

​

Prerequisites

• Create an Axiom account.
• Create a dataset in Axiom where you send your data.
• Create an API token in Axiom with permissions to update the dataset you have created.

​

Filebeat

Filebeat is a lightweight shipper for logs. It helps you centralize logs and files, and can read files from your system.

Filebeats is useful for workloads, system, app log files, and data logs you would like to ingest to Axiom in some way.

In the logging case, it helps centralize logs and files in a structured pattern by reading from your various apps, services, workloads, and VMs, then shipping to your Axiom deployments.

​

Installation

Visit the Filebeat OSS download page to install Filebeat. For more information, check out Filebeat’s official documentation

When downloading Filebeats, install the OSS version being that the non-oss version doesn’t work with Axiom.

​

Configuration

Axiom lets you ingest data with the ElasticSearch bulk ingest API.

In order for Filebeat to work, disable index lifecycle management (ILM). To do so, add setup.ilm.enabled: false to the filebeat.yml configuration file.

setup.ilm.enabled: false
filebeat.inputs:
  - type: log
    # Specify the path of the system log files to be sent to Axiom deployment.
    paths:
      - $PATH_TO_LOG_FILE
output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​

Metricbeat

Metricbeat is a lightweight shipper for metrics.

Metricbeat is installed on your systems and services and used for monitoring their performance, as well as different remote packages/utilities running on them.

​

Installation

Visit the MetricBeat OSS download page to install Metricbeat. For more information, check out Metricbeat’s official documentation

​

Configuration

setup.ilm.enabled: false
metricbeat.config.modules:
  path:
    -$PATH_TO_LOG_FILE
metricbeat.modules:
- module: system
  metricsets:
    - filesystem
    - cpu
    - load
    - fsstat
    - memory
    - network
output.elasticsearch:
  hosts: ["https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic"]
  # Specify Axiom API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​

Send AWS RDS metric set to Axiom

The RDS metric set enables you to monitor your AWS RDS service. RDS metric set fetches a set of metrics from Amazon RDS and Amazon Aurora DB. With Amazon RDS, users can monitor network throughput, I/O for read, write, and/or metadata operations, client connections, and burst credit balances for their DB instances and send the data to Axiom.

setup.ilm.enabled: false
metricbeat.config.modules:
  path:
    -$PATH_TO_LOG_FILE
metricbeat.modules:
- module: aws
  period: 60s
  metricsets:
    - rds
  access_key_id: '<access_key_id>'
  secret_access_key: '<secret_access_key>'
  session_token: '<session_token>'
  # Add other AWS configurations if needed
output.elasticsearch:
  hosts: ["https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic"]
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​

Winlogbeat

Winlogbeat is an open-source Windows specific event-log shipper that’s installed as a Windows service. It can be used to collect and send event logs to Axiom.

Winlogbeat reads from one or more event logs using Windows APIs, filters the events based on user-configured criteria, then sends the event data to the configured outputs.

You can Capture:

• app events
• hardware events
• security events
• system events

​

Installation

Visit the Winlogbeat download page to install Winlogbeat. For more information, check out Winlogbeat’s official documentation

• Extract the contents of the zip file into C:\Program Files.
• Rename the winlogbeat-$version directory to Winlogbeat
• Open a PowerShell prompt as an Administrator and run

PS C:\Users\Administrator> cd C:\Program Files\Winlogbeat

PS C:\Program Files\Winlogbeat> .\install-service-winlogbeat.ps1

​

Configuration

Configuration for Winlogbeat Service is found in the winlogbeat.yml file in C:\Program Files\Winlogbeat.

Edit the winlogbeat.yml configuration file found in C:\Program Files\Winlogbeat to send data to Axiom.

The winlogbeat.yml file contains the configuration on which windows events and service it should monitor and the time required.

winlogbeat.event_logs:
  - name: Application
  - name: System
  - name: Security
logging.to_files: true
logging.files:
  path: C:\ProgramData\Winlogbeat\Logs
logging.level: info
output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  # token should be an API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​

Validate configuration

# Check if your configuration is correct

PS C:\Program Files\Winlogbeat> .\winlogbeat.exe test config -c .\winlogbeat.yml -e

​

Start Winlogbeat

PS C:\Program Files\Winlogbeat> Start-Service winlogbeat

You can view the status of your service and control it from the Services management console in Windows.

To launch the management console, run this command:

PS C:\Program Files\Winlogbeat> services.msc

​

Stop Winlogbeat

PS C:\Program Files\Winlogbeat> Stop-Service winlogbeat

​

Ignore older Winlogbeat configuration

The ignore_older option in the Winlogbeat configuration is used to ignore older events.

Winlogbeat reads from the Windows event log system. When it starts up, it starts reading from a specific point in the event log. By default, Winlogbeat starts reading new events created after Winlogbeat started.

However, you might want Winlogbeat to read some older events as well. For instance, if you restart Winlogbeat, you might want it to continue where it left off, rather than skipping all the events that were created while it wasn’t running. In this case, you can use the ignore_older option to specify how old events Winlogbeat should read. The ignore_older option takes a duration as a value. Any events that are older than this duration are ignored. The duration is a string of a number followed by a unit. Units can be one of ms (milliseconds), s (seconds), m (minutes), h (hours) or d (days).

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  protocol: "https"
  ssl.verification_mode: "full"
  # token should be an API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

You can start Winlogbeat from the command line by running .\winlogbeat.exe -c winlogbeat.yml in the Winlogbeat installation directory.

​

Add verification modes and processors

Verification mode refers to the SSL/TLS verification performed when Winlogbeat connects to your output destination, for instance, a Logstash instance, ElasticSearch instance or an Axiom instance. You can add your verification modes, additional processors data, and multiple windows event logs to you configurations and send the logs to Axiom. The configuration is specified in thewinlogbeat.event_logs configuration option.

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System

output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  # token should be an API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true
  ssl.verification_mode: "certificate"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: info
logging.to_files: true
logging.files:
  path: C:/ProgramData/winlogbeat/Logs
  name: winlogbeat
  keepfiles: 7
  permissions: 0600

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

You can start Winlogbeat from the command line by running .\winlogbeat.exe -c winlogbeat.yml in the Winlogbeat installation directory.

For more information on Winlogbeat event logs, visit the Winlogbeat documentation.

​

Heartbeat

Heartbeat is a lightweight shipper for uptime monitoring.

It monitors your services and sends response time to Axiom. It lets you periodically check the status of your services and determine whether they’re available.

Heartbeat is useful when you need to verify that you’re meeting your service level agreements for service uptime.

Heartbeat currently supports monitors for checking hosts via:

• ICMP (v4 and v6) echo requests: Use the icmp monitor when you simply want to check whether a service is available. This monitor requires root access.
• TCP: Use the TCP monitor to connect via TCP. You can optionally configure this monitor to verify the endpoint by sending and/or receiving a custom payload.
• HTTP: Use the HTTP monitor to connect via HTTP. You can optionally configure this monitor to verify that the service returns the expected response, such as a specific status code, response header, or content.

​

Installation

Visit the Heartbeat download page to install Heartbeat on your system.

​

Configuration

Heartbeat provides monitors to check the status of hosts at set intervals. Heartbeat currently provides monitors for ICMP, TCP, and HTTP.

You configure each monitor individually. In heartbeat.yml, specify the list of monitors that you want to enable. Each item in the list begins with a dash (-).

The example below configures Heartbeat to use three monitors: an ICMP monitor, a TCP monitor, and an HTTP monitor deployed instantly to Axiom.

# Disable index lifecycle management (ILM)
setup.ilm.enabled: false
heartbeat.monitors:
  - type: icmp
    schedule: '*/5 * * * * * *'
    hosts: ['myhost']
    id: my-icmp-service
    name: My ICMP Service
  - type: tcp
    schedule: '@every 5s'
    hosts: ['myhost:12345']
    mode: any
    id: my-tcp-service
  - type: http
    schedule: '@every 5s'
    urls: ['http://example.net']
    service.name: apm-service-name
    id: my-http-service
    name: My HTTP Service
output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  # token should be an API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​

Auditbeat

Auditbeat is a lightweight shipper that ships events in real time to Axiom for further analysis. It Collects your Linux audit framework data and monitor the integrity of your files. It’s also used to evaluate the activities of users and processes on your system.

You can also use Auditbeat to detect changes to critical files, like binaries and configuration files, and identify potential security policy violations.

​

Installation

Visit the Auditbeat download page to install Auditbeat on your system.

​

Configuration

Auditbeat uses modules to collect audit information:

• Auditd
• File integrity
• System

By default, Auditbeat uses a configuration that’s tailored to the operating system where Auditbeat is running.

To use a different configuration, change the module settings in auditbeat.yml.

The example below configures Auditbeat to use the file_integrity module configured to generate events whenever a file in one of the specified paths changes on disk. The events contains the file metadata and hashes, and it’s deployed instantly to Axiom.

# Disable index lifecycle management (ILM)
setup.ilm.enabled: false
auditbeat.modules:
  - module: file_integrity
    paths:
      - /usr/bin
      - /sbin
      - /usr/sbin
      - /etc
      - /bin
      - /usr/local/sbin
output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  # token should be an API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

​

Packetbeat

Packetbeat is a real-time network packet analyzer that you can integrate with Axiom to provide an app monitoring and performance analytics system between the servers of your network.

With Axiom you can use Packetbeat to capture the network traffic between your app servers, decode the app layer protocols (HTTP, MySQL, Redis, PGSQL, Thrift, MongoDB, and so on), and correlate the requests with the responses.

Packetbeat sniffs the traffic between your servers, and parses the app-level protocols on the fly directly into Axiom.

Currently, Packetbeat supports the following protocols:

• ICMP (v4 and v6)
• DHCP (v4)
• DNS
• HTTP
• AMQP 0.9.1
• Cassandra
• MySQL
• PostgreSQL
• Redis
• Thrift-RPC
• MongoDB
• MemCache
• NFS
• TLS
• SIP/SDP (beta)

​

Installation

Visit the Packetbeat download page to install Packetbeat on your system.

​

Configuration

In packetbeat.yml, configure the network devices and protocols to capture traffic from.

To see a list of available devices for packetbeat.yml configuration , run:

Packetbeat supports these sniffer types:

• pcap

• af_packet

In the protocols section, configure the ports where Packetbeat can find each protocol. If you use any non-standard ports, add them here. Otherwise, use the default values:

# Disable index lifecycle management (ILM)
setup.ilm.enabled: false
packetbeat.interfaces.auto_promisc_mode: true
packetbeat.flows:
  timeout: 30s
  period: 10s
protocols:
  dns:
    ports: [53]
    include_authorities: true
    include_additionals: true
  http:
    ports: [80, 8080, 8081, 5000, 8002]
  memcache:
    ports: [11211]
  mysql:
    ports: [3306]
  pgsql:
    ports: [5432]
  redis:
    ports: [6379]
  thrift:
    ports: [9090]
  mongodb:
    ports: [27017]
output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  # api_key should be your API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

For more information on configuring Packetbeats, visit the documentation.

​

Journalbeat

Journalbeat is a lightweight shipper for forwarding and centralizing log data from systemd journals to a log management tool like Axiom.

Journalbeat monitors the journal locations that you specify, collects log events, and eventually forwards the logs to Axiom.

​

Installation

Visit the Journalbeat download page to install Journalbeat on your system.

​

Configuration

Before running Journalbeat, specify the location of the systemd journal files and configure how you want the files to be read.

The example below configures Journalbeat to use the path of your systemd journal files. Each path can be a directory path (to collect events from all journals in a directory), or a path configured to deploy logs instantly to Axiom.

# Disable index lifecycle management (ILM)
setup.ilm.enabled: false
journalbeat.inputs:
- paths:
  - "/dev/log"
  - "/var/log/messages/my-journal-file.journal"
  seek: head
journalbeat.inputs:
- paths: []
  include_matches:
    - "CONTAINER_TAG=redis"
    - "_COMM=redis"
    - "container.image.tag=redis"
    - "process.name=redis"
output.elasticsearch:
  hosts: ['https://api.axiom.co:443/v1/datasets/DATASET_NAME/elastic']
  # token should be an API token
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

For more information on configuring Journalbeat, visit the documentation.