When you’ve outgrown CloudWatch, turn to Axiom

It’s normal for teams new to AWS to start with Amazon CloudWatch Logs for log management. But as your application scales, the volume of logs and event data sent to CloudWatch grows non-linearly. You’ll soon find yourself paying a lot for data you may not use as often as you could.

└ Price comparison for 5 TB of log data per day.

That’s because not only is CloudWatch expensive, it also has limitations that make it hard or unusable for many use cases where you could apply all that data.

• Limited query operators, no query piping: CloudWatch has only a limited set of operators, with only a few beyond basic math and comparisons. You can’t pipe one query’s output into the input of another, either — a capability that Splunk, Elasticsearch and others have had for 15 years. For example, Cloudwatch insights can parse JSON, but it requires you to know the structure in advance and specify it in the parse command. APL parse_json is more flexible, it detects the structure/schema automatically.

Cloudwatch — Parsing nested JSON will also be similar to this, and way more complex:

parse @message "'fieldsA': '*', 'fieldsB': ['*']" as fld, array

APL — Parsing JSON is simple:

['Dataset']
| extend parsed = parse_json(payload)
| project field1 = parsed.field1, field2 = parsed.field2

APL — Nested JSON is also straightforward:

['Dataset']
| extend parsed = parse_json(EventData)
| extend fA = parsed.fieldsB[0].fA

• Sampling by default: The X-ray console uses sampling for traces, with a default sampling rate of 1 request per second and 5% of additional requests. In a high-traffic scenario, you might miss important traces.

• Performance limitations: CloudWatch has limits on the ingest rate and total size of log streams. These can lead to data loss, or at least to delayed processing when ingesting and slow query performance across a fragmented datastore.

• Poor integration: CloudWatch doesn’t integrate well with apps you may plan on using: Kubernetes, Docker, Vercel, Netlify, Tailscale, and log shippers Logstash, Fluentd and Fluent Bit. This doesn’t mean you can’t connect them, but it’s more challenging than it should be.

• Limited customization & alerting: CloudWatch dashboards offer basic experience and interactivity, but lack advanced features like drill-down capabilities or Axiom’s filter bar that contains all your filters, which can be interdependent It is difficult, for example, to configure global filters that combine traces and logs in a single pane.

Axiom avoids all of the above

Axiom’s architecture is both efficient – which keeps our pricing low, too — and flexible:

• Axiom engineers focus relentlessly on performance. optimized every step of ingest, storage, and query to use as few bytes and as little compute time as possible, delivering large queries at low cost.
• The APL query language has many more advanced operators. APL can pipe queries. That said, engineers accustomed to CloudWatch find APL familiar and easy to start using.
• Axiom supports distributed tracing, and easily puts traces and logs on the same pane of its dashboards, which were designed to be easy to spin up.
• Integration with everything from Kubernetes to Fluentd has been worked out to make connecting them to Axiom easy.
• Stop sampling! Axiom is optimized to let you ingest, store, and query 100% of your events affordably.
• Once you’ve paid to ingest events, there are no extra fees. You can reuse, reroute, or keep them in hot storage forever if you want.

Can you migrate from CloudWatch to Axiom?

Yes! To get started, understand that Amazon CloudWatch Logs captures three types of logs, categorized by which service or app created them. All of them can be sent to Axiom to be ingested, then stored and queried as much as you want at no additional cost.

└ AWS vended logs must be routed through CloudWatch to Axiom, but you save retention fees in CloudWatch. Service logs and custom logs can be routed directly to Axiom.

• Service logs: These are the logs published by more than 30 AWS services including Amazon API Gateway, AWS Lambda, and AWS CloudTrail.
  • These can bypass CloudWatch and go straight into Axiom via open-source collectors like Fluent Bit, Fluentd and Vector.

• Custom logs: These are logs published by your own applications, on-premise resources, and by other clouds.
  • These can also go straight into Axiom through open-source collectors such as those listed above.
• Vended logs: These are the logs published by specific AWS services like Amazon VPC and Amazon Route 53. These are hardwired to route into CloudWatch. - We provide a handy CloudFormation template to transfer logs from CloudWatch to Axiom. You can then minimize their retention period in CloudWatch to reduce costs.

Download our brief to learn more

Our CloudWatch solution brief explains everything you want to know about slashing costs while improving outcomes with Axiom. If you have any questions, email sales@axiom.co.