P99 CONF 2026: Five Database Designs That Trade Storage Costs for Speed

Oct 30, 2026 762 views
Abstract 3D grid of stacked metallic blocks glowing red and purple, evoking layered database storage

Across five sessions at the virtual P99 CONF, held October 21–22, 2026, engineers made the same point from different directions: database performance comes from choosing which resource to spend. Their systems trade memory, storage, CPU time, network work and implementation effort against the needs of particular workloads.

The talks were among more than 60 sessions on high-performance, low-latency applications. Together, they offer a practical tour of the read, write and storage costs often grouped as the RUM trade-off. Lowering one kind of amplification can raise another, so the useful question isn't whether a design is faster in the abstract. It's which bottleneck matters for the system being built.

Object storage changes the LSM calculation

Almog Gavra, co-founder of Responsive, now part of Databricks, began with a storage structure familiar to database engineers: the log-structured merge tree. An LSM sits between two extremes. A log accepts writes quickly but can make reads search through more data; a sorted array makes reads direct but is more costly to update. “I think about LSM trees as a mechanism for tuning between these two extremes,” Gavra said.

That tuning has a new setting when immutable files live in object storage. LSMs were designed around local disks, but their write pattern also suits services such as Amazon S3: the system writes new files rather than modifying existing ones. Object storage brings lower storage costs, durability and availability, while charging for individual GET and PUT requests and imposing slower access for cold data.

Gavra illustrated the latency difference with a local NVMe cache miss of about 150 microseconds, compared with 20–30 milliseconds for S3 Standard. That gap makes caching central to keeping object storage away from latency-sensitive reads. The LSM batches writes into immutable files; a cache can then serve frequently accessed data without a remote request.

Gavra's SlateDB packages that approach as an open-source, object-store-native LSM library. It offers a key-value interface over S3, GCS and R2, giving developers a route to build data systems on object storage without implementing an LSM themselves. The design's appeal depends on whether lower storage costs and simpler durability fit the workload's tolerance for remote reads.

ScyllaDB spends memory to shorten reads

ScyllaDB's Michael Litvak, a senior software engineer, and Felipe Cardeneti Mendes, a technical director, described a different response to read amplification. Its opt-in Logstor engine targets read-heavy key-value workloads and runs alongside ScyllaDB's existing LSM engine. The session's premise is that some tables benefit from a storage path built specifically for point lookups, without forcing every workload onto it.

Logstor keeps an in-memory index for each key, including a hash and the record's location on flash. A lookup can return a cached value or locate the record with one disk I/O. That avoids the sequence of checks through SSTables, Bloom filters and indexed structures that an LSM read may require. Both engine types can coexist in a cluster and use ScyllaDB's existing drivers and consistency options.

In a preliminary, read-dominant test, the speakers reported 1.1 million reads per second, with p99.9 latency below one millisecond and p99.99 latency around 1.5 milliseconds. The test used 2 KB values; earlier LSM examples in the talk used 1 KB values, so the figures don't provide a direct comparison. In a separate workload with 80% reads and 20% writes, they reported similar throughput and sub-millisecond p99.9 latency.

The cost is a larger RAM index and more write work. Even a small update writes the full new value to a log, leaving the previous record as dead space until segment compaction reclaims it. The team sees Logstor as a fit for bounded key sets, frequent overwrites and point lookups where the memory cost makes sense. Mendes summarized the choice as “optimized for time at the sacrifice of space.”

Valkey raises its single-node ceiling

Madelyn Olson, a principal software engineer at AWS, focused on a bottleneck beyond storage. Some Amazon ElastiCache customers had seen sudden p50 latency spikes, and her team's testing traced one limit to Valkey's main execution thread. In an unpipelined GET benchmark with about three million keys and 650 clients, latency climbed sharply near 150,000 requests per second.

Valkey already had I/O threads, but they were idle while the main thread handled commands. The team used lockless queues to let those threads continue network I/O and command parsing while keeping command execution on the main thread. In the same workload, Olson reported a ceiling of about 1.2 million requests per second, roughly eight times the earlier threshold. That's a workload-specific result, not a general throughput promise.

The talk also covered horizontal scaling. Valkey 9.0 introduced atomic slot migration, which lets a source shard retain ownership while streaming a snapshot and subsequent changes to a target. Ownership moves only after the target catches up. Olson said a roughly 40 GB migration across three to four shards ran about 10 times faster without load and about five times faster under a heavy GET workload. If migration fails before the handoff, the staged transfer can be aborted without losing end-user data.

Olson's operational advice was to use horizontal scaling for baseline capacity and vertical scaling for brief bursts. I/O threads can be added or removed as demand changes, while command execution remains single-threaded for now. “Efficiency trumps headline numbers,” she said—a useful reminder that adding concurrency is valuable when it removes a measured limit.

Vector search shifts from bandwidth to arithmetic

Nathan VanBenschoten, chief architect at turbopuffer, described a search index covering 100 billion vectors and more than 200 TiB of dense vector data. The company reported 200-millisecond p99 latency at more than 1,000 queries per second, with 92% recall in early results. Object storage holds durable data, while local SSD caches keep the high-query-rate workload close to compute.

The index combines hierarchical clustering with RaBitQ binary quantization. The quantized shadow index stores one bit per dimension, narrowing the candidate set before the system reranks a smaller group using full-precision vectors. That cuts the amount of vector data that must move through the search path.

Before quantization, the main constraint was bandwidth: getting vectors from storage into memory. With less data to move, the system became more CPU-bound because the work involved more arithmetic per byte read. “Often when you push down one bottleneck, a new one emerges,” VanBenschoten said. The next target may therefore be more efficient vector instructions, rather than another storage change.

To keep an index of this size on SSDs, turbopuffer distributes it across machines. VanBenschoten said its separated compute-and-storage architecture favors fewer, larger shards. A replacement machine can reconnect to object storage and rebuild its local cache instead of receiving a full shard from another node. Queries fan out across shards, with a coordinator combining their top results.

TigerBeetle builds its own object-storage client

Georg Kreuzmayr, a software engineer at TigerBeetle, connected object storage to recovery time. TigerBeetle is a database for financial transactions, and his “diagonal scaling” concept keeps hot data on local NVMe while extending the storage hierarchy to shared object storage for colder data. A failed machine can then recover data already held in object storage without reconstructing all of it from another machine.

The scale of recovery matters. Kreuzmayr estimated that transferring 128 TB over a 100 Gbps network takes about three hours at line rate. When recovery takes longer than the interval between failures, a cluster can enter a failure spiral. Object storage can change that recovery path, but the client still has to meet TigerBeetle's processing constraints.

TigerBeetle's transaction processing runs on a single core, which made parts of the AWS C++ SDK a poor fit. Kreuzmayr cited its blocking getaddrinfo call: on the transaction core, DNS lookup could stall transaction work. SDK thread pools, runtime allocation and dependencies also conflicted with the project's design goals.

The team built a multi-cloud client for S3, Azure and GCP, with simulated I/O and injected faults for testing. Written in Zig, it includes custom DNS, TLS and HTTP components, uses io_uring and zero-copy sends, and allocates working memory at startup. Kreuzmayr put the implementation at under 40,000 lines of code, with no dependencies beyond its toolchain. Its design follows TigerStyle's three principles: safety, performance and developer experience.

The performance figures came from specific tests. For S3 PutObject, using 8 MB requests over plain HTTP without checksums on an AWS c7gn.metal-24xlarge instance, Kreuzmayr reported uploads about 20 times faster than official SDKs and five times faster than a research implementation. For GetObject, he reported about three times the SDK throughput and roughly 25% more than the research implementation, attributing the smaller gain to the cloud TCP receive path. The conference described the wider comparison as 4–10 times faster than official GCP and AWS SDKs.

Building a client in-house required more upfront work, Kreuzmayr said, but gave the team code it could test and tune around its constraints. “Code stops being technical debt and ends up being an asset.”

Performance follows the bottleneck

These sessions cover different layers, from LSM file layout and RAM indexes to thread scheduling, vector search and cloud networking. Each design improves a particular path by spending something elsewhere: memory for shorter reads, CPU work for less data movement, or engineering effort for tighter control over a client.

That makes measurement the thread connecting the talks. The reported gains came with workload details, test conditions and costs that shape where each approach fits. As systems grow, teams will keep finding new constraints after the current one moves. P99 CONF recordings are available on demand after registration at p99conf.io; the conference is a ScyllaDB community.

Source: Doug Sillars · thenewstack.io

Comments

Sign in to comment.
No comments yet. Be the first to comment.

Related Articles

What five P99 CONF talks reveal about database trade-offs