Axiom lets Hapn query a petabyte of AWS Lambda events to manage 4 billion messages

Takeaways:

About Hapn

Hapn's GPS platform helps small, medium, and enterprise businesses — plus individual customers — track assets that range from tractors to pets to older people living with dementia. With full visibility of every connected asset, they can keep track of remote teams, protect assets against theft, and glean powerful insights to improve performance. Hapn has 16 years of experience and more than 300,000 customers.

The Challenge: Three terabytes a day

Hapn began a decade ago as an online storefront of the popular Spytec store in Manhattan. They pivoted in 2015 to build a GPS-based system for tracking customers’ assets. Hapn’s engineers have built an entirely serverless architecture that employs a mix of databases including DynamoDB, PlanetScale, and RDS. Their all-Lambda infrastructure publishes three terabytes of event data daily.

Co-founder and CTO Andres Hernandez says these logs are the heart of Hapn’s offering. “We focus on devices you could put on a cargo container or a vehicle — or as a parent, in our kids backpacks. We also focus on rental construction tools that you need to track.”

Unlike social media or other mass consumer services, Hapn engineers can’t work with only samples. Some trackers are built to last five years in the field on one battery, so they only transmit one payload of data each day.

Sampling is not an option

Hapn can’t afford to sample or filter events to cut their cloud costs. They need every specific event for every specific device for every specific customer to investigate claims of devices not reporting in, to determine if drivers have been speeding, or to find events across all devices on an account.

“We have a service that is called Listener. It ingests all the device’s messages,” Andres explains. “That’s important because there’s a communication sequence back and forth between the devices and the listener. Sometimes the manufacturer asks, ‘Hey, could you send me the conversation for this period?’ We have to go to logs, and it cannot be sampled.”

Not surprisingly, the cost of keeping all these events on hand was staggering. AWS CloudWatch outweighed the rest of Hapn’s cloud bill combined, even though it wouldn’t let them query across their eleven AWS accounts — they needed to login to each one. “The first thing we said was, ‘OK, let’s log less.’ That was a disaster.”

Charged twice for the same data

Seeking an affordable way to keep 100% of their data, Hapn’s engineers looked beyond Amazon to vendors that advertised their products at appealing prices.

The reality proved different, Andres recalls: “They said, ‘Ingest what you want, and later if you don’t still have it there, you can rehydrate it. They used that word very conveniently: rehydrate, not ingest. What you realize later when you pay the bill is that the rehydration triggered an ingestion of the same data again — at the same price.

“We found ourselves limited to logging only errors,” he says, “which still cost more than we could afford, and led to frequent re-hydration of logs. This process was not only expensive, as we incurred charges every time logs were re-indexed, but it was also time-consuming, drastically affecting our engineering team’s efficiency.”

Seeking an affordable way to keep more events on hand, Andres tried using AWS Athena to handle some of the logs instead. He says he found that “Athena is not cheap, either. I was paying both a vendor and Athena to look at logs. It was nonsense.”

Cloud cost optimizer recommends Axiom

Andres called AWS cost management experts Duckbill Group. “Corey Quinn there recommended Axiom.” he says. “He was like, ‘Hey, I have been using Axiom for years. If what you want is to see your logs, go there.’

“So I hung up, went to Axiom’s site and filled out the contact form. Meanwhile I wanted to test it, so I created an account, paid the 25 bucks, started playing with it, and the rest is history.” Andres says Axiom engineers were quick to help him online even before he connected with their enterprise sales team to talk about scaling much, much bigger.

The Solution: Affordable 100% logging

Axiom’s starts-low, stays-low pricing makes it possible for Hapn’s engineers to capture and keep 100% of events from their entire serverless microservice architecture for a year for less than they had paid only to log errors with the previous vendor, and without the need to archive or rehydrate them. They needn’t perform data gymnastics to reduce ingest and storage costs, waste hours rehydrating logs from cold storage, or risk sampling out critical events from a customer’s device.

Axiom’s hyper-optimized architecture keeps 100% of their data available and performative. And Axiom’s free Lambda extension, developed with Hapn’s real-time involvement during their evaluation, listens for Lambda events much like an old syslog listener.

When it’s time to query their petabyte of events, Axiom’s APL language lets them zero in on Lambda timestamps and execution IDs to see all events involved in a transaction. They can follow the breadcrumbs of an interaction across every single piece of infrastructure. In addition, virtual fields let engineers extract individual fields from unstructured record fields at query time to filter and summarize.

Hapn also uses Axiom’s full capture of events to monitor for alert thresholds, create single-pane dashboards, and compute metrics based on aggregated log events to send at greatly reduced volume (and cost) to CloudWatch for now.

With all their events at hand, Hapn is much more responsive to customers. “Our operations involve numerous battery-powered devices that periodically broadcast positional and sensor data before returning to a low-power state,” Andres explaims. “These broadcasts can trigger alerts if certain conditions set by our customers are met. These alerts are crucial, as they can indicate situations like asset theft or usage outside designated zones. In our system, tracking the state of each device heavily relies on the business logs generated by our data pipeline. On the rare occasions when alerts fail to trigger, our support team uses these logs to investigate and understand what happened.

“This is where Axiom’s comprehensive data indexing and analysis capabilities come into play. By leveraging 100% of our data and utilizing Axiom’s powerful query language (APL), our team can swiftly pinpoint the issue, enabling us to assist our customers in recovering valuable assets. This level of detailed analysis and rapid response would not be feasible without the full data access and efficient processing that Axiom provides.”

Hapn still routes metrics data to CloudWatch for now, but Andres expects that as Axiom continuously improves its support for metrics it will meet Hapn’s needs entirely in the not-too-distant future.

“It was my mandate this year to optimize the infrastructure,” he says. “Axiom was a key piece of it. My finance team loves predictability. This is the year we can commit to Axiom.”

Interested to learn more about Axiom?

Sign up for free or contact us at sales@axiom.co to talk with one of the team about our enterprise plans.