Send data from Amazon CloudWatch to Axiom

This page explains how to send data from Amazon CloudWatch to Axiom.

Axiom CloudWatch Forwarder is a set of easy-to-use AWS CloudFormation stacks designed to forward logs from Amazon CloudWatch to Axiom. It includes a Lambda function to handle the forwarding and stacks to create Amazon CloudWatch log group subscription filters for both existing and future log groups.

Axiom CloudWatch Forwarder includes templates for the following CloudFormation stacks:

Forwarder creates a Lambda function that forwards logs from Amazon CloudWatch to Axiom.
Subscriber runs once to create subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.
Listener creates a Lambda function that listens for new log groups and creates subscription filters for them on Forwarder. This way, you don’t have to create subscription filters manually for new log groups.
Unsubscriber runs once to remove subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.

To determine the best method to send data from different AWS services, see Send data from AWS to Axiom.

Prerequisites

Create an Axiom account.
Create a dataset in Axiom where you send your data.
Create an API token in Axiom with permissions to ingest data to the dataset you have created.

Create an account on AWS Cloud.

Installation

To install the Axiom CloudWatch Forwarder, choose one of the following:

Cloudformation stacks
Terraform module

Install with Cloudformation stacks

Launch the Forwarder stack template on AWS. Copy the Forwarder Lambda ARN because it’s referenced in the Subscriber stack.
Launch the Subscriber stack template on AWS.
Launch the Listener stack template on AWS.

Install with Terraform module

Create a new Forwarder module in your Terraform file in the following way:

HCL

module "forwarder" {
  source           = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/forwarder"
  axiom_dataset    = "DATASET_NAME"
  axiom_token      = "API_TOKEN"
  prefix           = "axiom-cloudwatch-forwarder"
}

Alternatively, create a dataset with the Axiom Terraform provider.

Create a new Subscriber module in your Terraform file in the following way:

HCL

module "subscriber" {
  source               = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/subscriber"
  prefix               = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}

Create a new Listener module in your Terraform file in the following way:

HCL

module "listener" {
  source               = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/listener"
  prefix               = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}

In your terminal, go to the folder of your main Terraform file, and then run terraform init.

Run terraform plan to check the changes, and then run terraform apply.

Filter Amazon CloudWatch log groups

The Subscriber and Unsubscriber stacks allow you to filter the log groups by a combination of names, prefix, and regular expression filters. If no filters are specified, the stacks subscribe to or unsubscribe from all log groups. You can also whitelist a specific set of log groups using filters in the CloudFormation stack parameters. The log group names, prefix, and regular expression filters included are additive, meaning the union of all provided inputs is matched.

Example

For example, you have the following list of log groups:

text

/aws/lambda/function-foo
/aws/lambda/function-bar
/aws/eks/cluster/cluster-1
/aws/rds/instance-baz

To subscribe to the Lambda log groups exclusively, use a prefix filter with the value of /aws/lambda.
To subscribe to EKS and RDS log groups, use a list of names with the value of /aws/eks/cluster/cluster-1,/aws/rds/instance-baz.
To subscribe to the EKS log group and all Lambda log groups, use a combination of prefix and names list.
To use the regular expression filter, write a regular expression to match the log group names. For example, \/aws\/lambda\/.* matches all Lambda log groups.
To subscribe to all log groups, leave the filters empty.

Listener architecture

The optional Listener stack does the following:

Creates an Amazon S3 bucket for AWS CloudTrail.
Creates a trail to capture the creation of new log groups.
Creates an event rule to pass those creation events to an Amazon EventBridge event bus.
Sends an event via EventBridge to a Lambda function when a new log group is created.
Creates a subscription filter for each new log group.

Remove subscription filters

To remove subscription filters for one or more log groups, launch the Unsubscriber stack template on AWS.

The log group filtering works the same way as the Subscriber stack. You can filter the log groups by a combination of names, prefix, and regular expression filters.

Alternatively, to turn off log forwarding to Axiom, create a new Unsubscriber module in your Terraform file in the following way:

HCL

module "unsubscriber" {
  source           = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/unsubscriber"
  prefix           = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}