Send data from Amazon CloudWatch to Axiom
This page explains how to send data from Amazon CloudWatch to Axiom.
Axiom CloudWatch Forwarder is a set of easy-to-use AWS CloudFormation stacks designed to forward logs from Amazon CloudWatch to Axiom. It includes a Lambda function to handle the forwarding and stacks to create Amazon CloudWatch log group subscription filters for both existing and future log groups.
Axiom CloudWatch Forwarder includes templates for the following CloudFormation stacks:
- Forwarder creates a Lambda function that forwards logs from Amazon CloudWatch to Axiom.
- Subscriber runs once to create subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.
- Listener creates a Lambda function that listens for new log groups and creates subscription filters for them on Forwarder. This way, you don’t have to create subscription filters manually for new log groups.
- Unsubscriber runs once to remove subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.
To determine the best method to send data from different AWS services, see Send data from AWS to Axiom.
The Axiom CloudWatch Forwarder is an open-source project and welcomes your contributions. For more information, see the GitHub repository.
Prerequisites
- Create an Axiom account.
- Create a dataset in Axiom where you send your data.
- Create an API token in Axiom with permissions to update the dataset you have created.
Installation
To install the Axiom CloudWatch Forwarder, choose one of the following:
Install with Cloudformation stacks
- Launch the Forwarder stack template on AWS. Copy the Forwarder Lambda ARN because it’s referenced in the Subscriber stack.
- Launch the Subscriber stack template on AWS.
- Launch the Listener stack template on AWS.
Install with Terraform module
Create a new Forwarder module in your Terraform file in the following way:
module "forwarder" {
source = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/forwarder"
axiom_dataset = "DATASET_NAME"
axiom_token = "API_TOKEN"
prefix = "axiom-cloudwatch-forwarder"
}
- Replace
API_TOKEN
with the Axiom API token you have generated. For added security, store the API token in an environment variable. - Replace
DATASET_NAME
with the name of the Axiom dataset where you want to send data.
Alternatively, create a dataset with the Axiom Terraform provider.
Create a new Subscriber module in your Terraform file in the following way:
module "subscriber" {
source = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/subscriber"
prefix = "axiom-cloudwatch-forwarder"
forwarder_lambda_arn = module.forwarder.lambda_arn
log_groups_prefix = "/aws/lambda/"
}
Create a new Listener module in your Terraform file in the following way:
module "listener" {
source = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/listener"
prefix = "axiom-cloudwatch-forwarder"
forwarder_lambda_arn = module.forwarder.lambda_arn
log_groups_prefix = "/aws/lambda/"
}
In your terminal, go to the folder of your main Terraform file, and then run terraform init
.
Run terraform plan
to check the changes, and then run terraform apply
.
Filter Amazon CloudWatch log groups
The Subscriber and Unsubscriber stacks allow you to filter the log groups by a combination of names, prefix, and regular expression filters. If no filters are specified, the stacks subscribe to or unsubscribe from all log groups. You can also whitelist a specific set of log groups using filters in the CloudFormation stack parameters. The log group names, prefix, and regular expression filters included are additive, meaning the union of all provided inputs is matched.
Example
For example, you have the following list of log groups:
/aws/lambda/function-foo
/aws/lambda/function-bar
/aws/eks/cluster/cluster-1
/aws/rds/instance-baz
- To subscribe to the Lambda log groups exclusively, use a prefix filter with the value of
/aws/lambda
. - To subscribe to EKS and RDS log groups, use a list of names with the value of
/aws/eks/cluster/cluster-1,/aws/rds/instance-baz
. - To subscribe to the EKS log group and all Lambda log groups, use a combination of prefix and names list.
- To use the regular expression filter, write a regular expression to match the log group names. For example,
\/aws\/lambda\/.*
matches all Lambda log groups. - To subscribe to all log groups, leave the filters empty.
Listener architecture
The optional Listener stack does the following:
- Creates an Amazon S3 bucket for AWS CloudTrail.
- Creates a trail to capture the creation of new log groups.
- Creates an event rule to pass those creation events to an Amazon EventBridge event bus.
- Sends an event via EventBridge to a Lambda function when a new log group is created.
- Creates a subscription filter for each new log group.
Remove subscription filters
To remove subscription filters for one or more log groups, launch the Unsubscriber stack template on AWS.
The log group filtering works the same way as the Subscriber stack. You can filter the log groups by a combination of names, prefix, and regular expression filters.
Alternatively, to turn off log forwarding to Axiom, create a new Unsubscriber module in your Terraform file in the following way:
module "unsubscriber" {
source = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/unsubscriber"
prefix = "axiom-cloudwatch-forwarder"
forwarder_lambda_arn = module.forwarder.lambda_arn
log_groups_prefix = "/aws/lambda/"
}
Was this page helpful?