input
sources, filter data for the specified configuration, and eventually store it.
Logstash sits between your data and where you want to keep it.
Prerequisites
- Create an Axiom account.
- Create a dataset in Axiom where you send your data.
- Create an API token in Axiom with permissions to update the dataset you have created.
- Visit the Logstash download page to install Logstash on your system.
Configuration
To configure thelogstash.conf
file, define the source, set the rules to format your data, and set Axiom as the destination where the data is sent.
The Logstash configuration works with OpenSearch, so you can use the OpenSearch syntax to define the source and destination.
The Logstash Pipeline has three stages:
- Input stage generates the event & Ingest Data of all volumes, Sizes, forms, and Sources
- Filter stage modifies the event as you specify in the filter component
- Output stage shifts and sends the event into Axiom.
OpenSearch output
For installation instructions for the plugin, check out the OpenSearch documentation Inlogstash.conf
, configure your Logstash pipeline to collect and send data logs to Axiom.
The example below shows Logstash configuration that sends data to Axiom:
Replace
AXIOM_DOMAIN
with api.axiom.co
if your organization uses the US region, and with api.eu.axiom.co
if your organization uses the EU region. For more information, see Regions.Replace API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable.Replace DATASET_NAME
with the name of the Axiom dataset where you send your data.Combining filters with conditionals on Logstash events
Logstash provides an extensive array of filters that allow you to enhance, manipulate, and transform your data. These filters can be used to perform tasks such as extracting, removing, and adding new fields and changing the content of fields. Some valuable filters include the following.Grok filter plugin
The Grok filter plugin allows you to parse the unstructured log data into something structured and queryable, and eventually send the structured logs to Axiom. It matches the unstructured data to patterns and maps the data to specified fields. Here’s an example of how to use the Grok plugin:Replace
AXIOM_DOMAIN
with api.axiom.co
if your organization uses the US region, and with api.eu.axiom.co
if your organization uses the EU region. For more information, see Regions.Replace API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable.Replace DATASET_NAME
with the name of the Axiom dataset where you send your data.COMBINEDAPACHELOG
.
Mutate filter plugin
The Mutate filter plugin allows you to perform general transformations on fields. For example, rename, convert, strip, and modify fields in event data. Here’s an example of using the Mutate plugin:Replace
AXIOM_DOMAIN
with api.axiom.co
if your organization uses the US region, and with api.eu.axiom.co
if your organization uses the EU region. For more information, see Regions.Replace API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable.Replace DATASET_NAME
with the name of the Axiom dataset where you send your data.hostname
to host
, converts the response
field value to an integer, changes the method
field to uppercase, and removes the request
and httpversion
fields.
Drop filter plugin
The Drop filter plugin allows you to drop certain events based on specified conditions. This helps you to filter out unnecessary data. Here’s an example of using the Drop plugin:Replace
AXIOM_DOMAIN
with api.axiom.co
if your organization uses the US region, and with api.eu.axiom.co
if your organization uses the EU region. For more information, see Regions.Replace API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable.Replace DATASET_NAME
with the name of the Axiom dataset where you send your data.syslog
with severity debug
.
Clone filter plugin
The Clone filter plugin creates a copy of an event and stores it in a new event. The event continues along the pipeline until it ends or is dropped. Here’s an example of using the Clone plugin:Replace
AXIOM_DOMAIN
with api.axiom.co
if your organization uses the US region, and with api.eu.axiom.co
if your organization uses the EU region. For more information, see Regions.Replace API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable.Replace DATASET_NAME
with the name of the Axiom dataset where you send your data.cloned_event
that’s a clone of the original event.
GeoIP filter plugin
The GeoIP filter plugin adds information about the geographical location of IP addresses. This data includes the latitude, longitude, continent, country, and so on. Here’s an example of using the GeoIP plugin:Replace
AXIOM_DOMAIN
with api.axiom.co
if your organization uses the US region, and with api.eu.axiom.co
if your organization uses the EU region. For more information, see Regions.Replace API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable.Replace DATASET_NAME
with the name of the Axiom dataset where you send your data.ip
field. Note that you may need to specify the path to the GeoIP database file in the plugin configuration, depending on your setup.