Featured
Ray Jacobson
VP of Engineering, Audius
“Blockchains themselves won’t give you the ability to construct complicated queries directly across them.”
Ray Jacobson
VP of Engineering
Takeaways
- Audius is a distributed marketplace based on the Solana blockchain, which has no built-in aggregation, search or filtering for events from individual nodes. They had to craft their own solution, for which Axiom is a “simple and straightforward” centerpiece.
- With 5 million monthly users and 400,000 music plays daily, Audius needs to capture at least 10 terabytes of log data per month to monitor their high-volume streaming network.
- Their decentralized network is hosted by 70+ node operators who range from individuals to large institutions. As a result, they suffered accidental expensive overruns on logging volume. Axiom provides a cost-safe way to collect all the data they need.
About Audius
Audius is a decentralized music streaming platform that operates on a blockchain-based protocol. It’s designed to empower artists and fans by allowing direct music publishing and monetization without intermediaries. The platform is community-owned and artist-controlled, offering an open-source environment where developers can build applications on top of it. Audius leverages multiple blockchains for different aspects of the service, including Ethereum for smart contracts and Solana for high-throughput transactions. It’s a unique ecosystem where artists, fans, and developers can interact in a transparent and secure way, with the native AUDIO token used for transactions within the network.
“There was one month where we were dropping logs, yet log volume cost us half of our infrastructure bill.”
Ray Jacobson
VP of Engineering
The Challenge: Maintain the reliability of a decentralized, blockchain-based network
Audius, founded in 2018, is a sincere attempt to create an ecosystem and marketplace for music artists, listeners, and fans that doesn’t need them to play by the rules set by major labels or streaming supergiants. VP of Engineering Roy Jacobson says that from an engineering perspective, “We write a bunch of software, and push that software out open source on the Internet for people to participate in serving the infrastructure for a music streaming platform. The product itself that we build looks like Spotify or Soundcloud. It's just a music streaming platform where people upload, sell and manage music catalogs.”
Node operators — there are more than 70 at the moment — sign up by buying a stake in the company’s AUDIO cryptocurrency. They can earn revenue in AUDIO, or be penalized for bad behavior.
In order to keep the their network reliable, Audius needed to create a way to collect and query logs coming from all those nodes. “Blockchains themselves won’t give you the ability to construct complicated queries directly across them, because they’re pretty much like a transactional log themselves,” Ray says. Audius engineers needed to build aggregators that collected data from each node into a central datastore.
The volume of logs generated by node operators turned out to spike wildly, as individual node admins either naively or accidentally flooded Audius’ aggregators with gigabytes of verbose data. As a result, their monthly bill from Loggly far outstripped what they’d budgeted. “There was one month where we were dropping logs,” Ray says, “yet log volume cost us half of our infrastructure bill. That was the straw that broke the camel’s back.”
The Solution: simple, straightforward, and no surprise bills
Ray says Axiom’s simple, predictable pricing made it a must for his team evaluate. But they quickly found other upsides, such as Axiom’s APL query language that supports schema-on-read and piped queries. With Loggly, he says, “Their advanced tools are pretty weak, so we would just do fuzzy string matches.”
They found Axiom easy to set up, too, and to extend to new uses. “It’s simple and straightforward. We have a staging test environment that we hooked up within a day. I looked at the stream view and it was working. Cool.”
Audius uses Axiom to investigate customer issues, which can involve purchases that didn’t go through properly. Not only do engineers need to gather the events, they also need to monitor that their event-aggregating apparatus is working correctly — it’s no use if they don’t have the events from the customer’s transaction because of a bug in the aggregator code. If things aren’t being indexed, they need to figure out why.
“That kind of thing is really important for us to understand,” Ray says. “So we have a lot of incremental logs that show progress along these indexing jobs. We also have logs for HTTP response codes and whether things are actually succeeding, users are streaming stuff, and so on.”
“If you have a lot of log volume, I would say go to Axiom.”
Ray Jacobson
VP of Engineering
Ray doesn’t need to be prompted to recommend Axiom to others with high-volume logging needs. “If you're pushing ten terabytes a month — structured or unstructured — and want to have good, organized logging, it’s nice,” he says. “And generally speaking, you guys just have the most price-effective product right now. So if you have a lot of log volume, I would say go to Axiom.”
Interested to learn more about Axiom?
Sign up for free or contact us at sales@axiom.co to talk with one of the team about our enterprise plans.
