Let me start by saying that Gorilla, the influential algorithm published by Facebook in 2015, is an excellent one for time series caching and storage. It's brilliant in its simplicity and effectiveness and has become the de facto standard for it. Fundamentally it is based on the observation that timestamps are monotonically increasing and most time series values in real-world scenarios do not constantly experience large swings.
But can we do better? The short answer is: it depends. Improving on its elegance is hard, but we can shift the tradeoffs toward something that suits a large, multi-tenant deployment rather than an in-house one.
Gorilla's tradeoffs
And here is where it's worth talking about Gorilla's dark sides, it's built by Facebook for Facebook and they made the tradeoffs that make it work for their use case. The two big ones are that it's float-only and append-only in strict time-order data. If you monitor your infrastructure both those decisions are perfectly valid, you know what you lose and what you gain. Let's quickly talk about those.
In-order data sounds trivial but it's surprisingly not when you do not have tight control over the entire pipeline. We were surprised just how much data arrives out of order. We are monitoring this statistic, and throughout our customer base there is a constant stream of thousands of data points that do not arrive in order every second, spiking to millions of data points when there are infrastructure changes happening at customers.
Just dropping that data is kind of a no-go for a SaaS, fine if you do it yourself but for us it's "someone loses data" so it needs to be treated with more care. Now, sure you could build a layer on top of a Gorilla store that handles out-of-order writes, an uncompressed buffer for recent data, a reconciliation store for extra data points, but then you've just lost both the elegance and the performance characteristics of Gorilla — the solution now is more complex, uses more memory, more CPU and possibly requires more storage.
The second one is integers. Not all measurements are floating-point numbers, in fact most aren't. You don't have half bytes, or a third of an HTTP request. Gorilla made the decision to only use floats — again that's fine if you make that decision for yourself it's a tradeoff of simplification over efficiency and precision. OTel, the standard we follow, defines data points as integers, so if we want to follow the specification, so do we. Again you could build a layer on top of Gorilla that deals with this, represents ints as floats for the space that floats safely cover and throw away a few bits for what doesn't or implement a Gorilla-ish compression for integers alongside. But both again come at a cost.
What we built
So what did we build? The out-of-order data is a crucial part, it's the biggest hurdle for Gorilla. It's also the one that can have the biggest impact. So while Gorilla compresses data as it comes in we have two separate buffers, a compressed and an uncompressed one. As data arrives it lands in the uncompressed buffer, handling out-of-order data there is fairly easy as re-shuffling it is virtually free (sorting a few hundred numbers doesn't really register).
Then once that buffer fills up it gets compressed, and here is where it gets interesting. Instead of a Gorilla-style streaming compression we can compress a batch of numbers. This yields significantly higher compression ratios especially when shapes are not a perfect fit for the XOR compression Gorilla uses. The result is that while the uncompressed buffer is much larger than the direct Gorilla compressed data, the compressed section is significantly smaller in most cases.
This characteristic has an interesting effect, when a cache is fresh Gorilla wins on memory hands down. However, as time progresses this two-tiered approach starts to overtake Gorilla — the larger the cache grows the more efficient the algorithm becomes. And that's the kind of amortization you want to see. You take a hit on memory when there isn't much pressure but save with the load and pressure increasing.
Time as a bitmap
The second and perhaps most radically different part is how we treat time. Gorilla stores delta-of-delta encoded timestamps. It comes with the same streaming advantage and when your intervals are perfect that really compresses well. However it doesn't play nicely with the compressed / uncompressed region approach we use as storing the timestamps would be between 32–64 bits extra per data point, that adds up quickly. The second observation we have is again that user data and Facebook data differ, we rarely see perfect periodicity in the data we receive.
The choice we made is using a bitmap, where every index is an offset from the start time. This is a wash memory-wise at 60s intervals compared to timestamps, but realistically the average data density we see is more around 15-second intervals throughout all the datasets we have, so in the uncompressed region a timestamp takes roughly 15 bits.
Why not use a u16 and a base offset? Fair question, and for ingest alone that might be a good choice. The advantage of a bitmap comes at query time. Queries always aggregate over a time range, and with a bitmap that becomes two AND operations and range/64 POPCNT operations, followed by aggregation over a known number of elements that are all colocated. Compare that to interleaved time/value pairs and a comparison of an unknown number of timestamps.
The extra cost goes away with the compressed section. When we compress a batch, we cut off the bitmap and zstd-compress the slice covering that batch. With the shape of data it compresses extremely well.
When it comes to datatypes, the algorithm we use for compression pcodec works with both integers and floats, so we can natively support both with an auto-promotion to floats for mixed series (we've yet to observe this happening; usually a series stays in its datatype).
The cost
But it's not a no-brainer: it's neither a hands-down improvement nor a regression, but an incremental improvement that adapts to a broader use case. Our observation so far is that depending on data shape memory usage is between 30% to 200% of what Gorilla uses with the majority lying in the 70% to 100%. It's an incremental improvement on memory not something that completely blows the Gorilla approach out of the water. However, it's worth keeping in mind that it is an incremental improvement while resolving the out-of-order and type problem that Gorilla has.
That said, it comes at a cost: CPU efficiency. Gorilla is a monster in that regard. An optimized Gorilla implementation (not a naive one) is about 15x more CPU efficient than our approach. That's a tradeoff, but it's a tradeoff worth it (to us). Gorilla uses nearly no CPU, so 15x of nearly nothing is still nearly nothing, just 15x more nothing than before. Our observation has been that this workload is heavily memory-bound, so trading some CPU for increased capabilities and sometimes even some gains in memory is a tradeoff we take without thinking twice.
The hidden surprise
We talked about CPU and memory but those are only two of the three parts of time series data. Storage is the last part of the puzzle. If you read between the lines when we talked about memory you might have expected this, the approach really shines here. We get the benefit of the high compression without the cost of an uncompressed section.
In addition we don't need to balance batch size or limit ourselves by memory as to how many points we can put together when compacting that data. Once persisted, we can rewrite it into its optimal shape. The result? Storage at 10% to 80% of Gorilla's size on our data, with most cases falling between 30% and 50%.
Takeaway
The approach we chose largely moves tradeoffs but it is a net win for our use case. We take a massive hit at CPU efficiency on the cache, mostly a tie on memory and a massive win on storage. We achieve that while adding the capability to cover the OTel data space (integers and floats) and be able to deal with outliers both of which to us outweigh the hit in CPU for a memory-bound service.
All of these measurements are highly workload-dependent — the numbers below reflect what we observe across our datasets, not guarantees for any single workload:
| Dimension | Gorilla | Axiom |
|---|---|---|
| Write model | In-order / append-only | Out-of-order tolerant |
| Value types | Floats only | Integers and floats (auto-promotes mixed series) |
| Cache CPU | Negligible | Negligible (higher, but not the bottleneck) |
| Cache memory | Good | Slight edge (30–200%, mostly 70–100%); improves with cache size |
| Persisted storage | Good | Excellent (10–80% of Gorilla's size, mostly 30–50%) |
| Complexity | Elegant streaming path | Two-tier buffers + batch compression |
With that all said, we won't stop here. As we continue to observe more data shapes, we'll keep refining the algorithm. Our next goal is to reduce memory usage further, so stay tuned for the next post on the topic!
