LSM Trees Transform Heavy Database Writes into Sequential Storage

Computer databases handle incoming data through two very different approaches to persistent storage media. Relational engines like PostgreSQL, MySQL, MariaDB, and Oracle write updates into balanced B-trees on disk, but heavy ingestion workloads face steep latency walls from disk head movements and page rewrites. Distributed storage engines solve this bottleneck with lsm trees, which buffer incoming updates in memory and flush large sorted files to storage in sequential streams [15]. This architecture trades immediate disk organization for raw ingestion throughput [16].

The Random Disk Write Problem

Storage systems struggle when millions of small writes land scattered over disconnected disk blocks. In a standard B-tree index, modifying a single row needs finding the target page on disk, reading that block into memory, updating the record bytes, and writing the entire block back to permanent storage. Ben Stopford observed that sequential access on magnetic disks and solid-state drives remains roughly three orders of magnitude faster than random input and output operations. When writes scatter over distant addresses, drive controllers spend most of their time seeking sectors rather than moving application payload bytes. Disk heads crawl on seeks. In an ACM report on storage efficiency, Stopford showed that raw sequential disk writes easily reach 200 to 300 megabytes per second [16].

When Patrick O’Neil, Edward Cheng, Dieter Gawlick, and Elizabeth O’Neil published their foundational paper in Acta Informatica in 1996, relational engines like PostgreSQL and Oracle forced disk actuators to jump over disconnected sectors, throttling overall write throughput [13]. Rather than updating pages in place, their design appends incoming writes sequentially to an append-only log [15].

Simple journal logs work well for Apache Kafka message pipelines that read records by sequential offset, but general databases need rapid point lookups and ordered range scans. To bridge this divide, storage systems combine memory buffers with multi-tiered disk runs. Martin Kleppmann described this trade-off in Designing Data-Intensive Applications, explaining how deferred background merging preserves high ingestion rates without sacrificing read access [12]. The architecture delivers high write throughput while maintaining fast logarithmic search times [17].

How MemTables Buffer Writes

A MemTable buffers incoming database records in memory as a sorted data structure so write operations finish without touching disk storage. When an application sends an update, the storage engine places the key-value pair into this memory buffer [12]. In their 2018 survey on arXiv, Chen Luo and Michael J. Carey described how in-memory structures organize updates; readers should note that their survey is an un-reviewed pre-print rather than a peer-reviewed journal article [1]. Writes complete in nanoseconds [11].

Many distributed storage engines, including RocksDB and Apache Cassandra, build their mutable MemTable using a concurrent skip list. Skip lists rely on probabilistic multi-level forward pointers, providing average lookup and insertion speeds of O(log n) without needing complex rebalancing steps. Jidin Dinesh observed that skip lists work well with bump-pointer arena allocators, grouping adjacent allocations into eight-megabyte memory blocks that improve CPU cache locality [9]. Because skip lists rely on probabilistic multi-level forward pointers rather than balanced tree rebalancing rotations, concurrent worker threads in RocksDB can insert incoming key-value pairs at the same time using atomic compare-and-swap operations without acquiring global mutex locks [15]. Alex Petrov explained in Database Internals that skip lists provide predictable P99 latencies under heavy concurrent ingestion [13]. This design prevents thread contention under heavy ingest workloads [9].

To protect durability during power failures, the engine appends mutations to a write-ahead log on disk before updating the MemTable [11]. When the buffer reaches its configured memory limit (typically 64 megabytes in RocksDB), it becomes immutable and queues for a background disk flush [15].

SSTables on Disk

An SSTable, short for Sorted String Table, stores immutable key-value pairs sorted in lexicographical order on disk. When an immutable MemTable flushes to permanent storage, a background thread writes the entire memory buffer sequentially into a new SSTable file [11]. Because the data in memory was already sorted, this conversion avoids random seeking. Data files stay immutable [14].

The layout of an SSTable divides stored entries into fixed-size data blocks that typically measure four kilobytes in RocksDB. Consecutive keys inside each data block share common prefixes, allowing RocksDB to use prefix compression to trim duplicate characters by 20 to 40 percent [15]. An index block at the end of the file records the highest key of each data block with its exact byte offset. In lsm trees, writers never overwrite existing files, eliminating concurrency bugs and simplifying operating system page caching. Imre Tomosvari showed in TMSVR Dev Blog experiments that keeping SSTables immutable enables safe concurrent readers without acquiring locks on data files [12].

Deletions and updates follow the same append-only pattern as initial insertions. When an application deletes a record, the database writes a special tombstone marker [15]. Reads encounter the newer value or tombstone first and stop searching. Background compaction workers clean up redundant bytes later [14].

Why Compaction Keeps LSM Trees Fast

Compaction keeps LSM trees fast by merging multiple overlapping on-disk files into consolidated runs, removing duplicate keys and discarding expired tombstone markers. Over time, frequent MemTable flushes create dozens of disconnected SSTables on storage media [14]. If the storage engine left these files untouched, every read query would have to examine every file on disk, driving read latency up. Queries slow on multiple files [9].

A background compactor merges several files using a multi-way merge sort, producing clean, non-overlapping tables that restore fast search paths. Garbage collection of deleted keys depends on this background merge mechanism. A tombstone marker must remain visible through disk levels until the compaction job reaches the oldest level containing older copies of that key. If an engine drops a tombstone prematurely, an older version of the key residing in a deeper disk level will resurrect, corrupting database consistency [15]. Martinuke0 observed on production systems that high delete volumes can trigger compaction storms where disk input and output bandwidth saturates as workers rewrite large files. Backpressure pipelines must throttle ingest to prevent saturation [11].

Compaction introduces a trade-off known as write amplification, which measures the ratio of total bytes written to disk relative to the bytes sent by the client. When an engine merges files repeatedly, a single payload byte may be rewritten five to thirty times over its lifetime [14]. Zhang and Du proposed NVLSM in ACM Transactions on Storage, showing that accumulative compaction on byte-addressable persistent memory cuts write amplification [2].

Bloom Filters for LSM Trees

When an application issues a point lookup query, the storage engine inspects the mutable MemTable first, checks queued immutable MemTables in RAM, and scans Level 0 SSTables on disk, skipping files whose Bloom filters verify that the requested key does not exist [15]. If the key is missing from memory, read operations in lsm trees navigate through disk files to find the newest valid version [9]. Because searching disk blocks causes expensive read amplification, engines need a fast filter [14].

A Bloom filter is a compact probabilistic data structure stored with each SSTable that tells the engine if a key is definitely absent. If the filter returns false, the database skips that SSTable. GeeksforGeeks documented that a tuned Bloom filter achieves a 99.9 percent accuracy rate [8].

Configuring Bloom filters balances main memory consumption against read speed. In RocksDB, allocating ten bits per key produces a false-positive rate near one percent [15]. For a database holding one billion keys, the corresponding Bloom filters consume roughly 1.25 gigabytes of memory per SSTable set. TheCodeForge warned that having one hundred unpartitioned SSTables could inflate filter memory usage to 125 gigabytes, prompting engines to use partitioned filters that load small blocks on demand [14]. Why waste gigabytes of cache memory on inactive index files when partitioned filters solve the problem? Francesco Tomaselli showed in a Java LSM implementation that variable-byte integer encoding and delta-encoded offsets compress filter and index blocks efficiently [10].

Database Engines Using LSM Trees

Leading distributed NoSQL databases rely on LSM trees to process write-intensive workloads across large server clusters without storage contention. Apache Cassandra, originally developed at Facebook and maintained by the Apache Software Foundation, utilizes this architecture to ingest massive streams of sensor telemetry and messaging data [11]. Google Bigtable and LevelDB pioneered implementation patterns, inspiring Facebook engineers to fork LevelDB into RocksDB in 2012 [13]. RocksDB now powers storage layers inside CockroachDB, TiKV, and Meta internal database infrastructure [15].

Other production engines adopt hybrid storage designs to benefit from sequential writes. ScyllaDB re-implemented Cassandra storage model in C++ to achieve high single-node throughput, while SQLite offers an LSM extension for write-heavy embedded applications [13]. MongoDB added WiredTiger as its default storage engine, incorporating an optional LSM storage architecture with its B-tree engine [16]. Distributed time-series platforms like InfluxDB use an LSM-derived Time Structured Merge index to manage time-stamped metric points. Can analytics pipelines handle millions of metrics per second without sequential disk batching? [13]

Researchers continue to expand the architecture into new runtime environments and hardware platforms. Paraskar, Dhas, and Mane introduced KiwiDB in 2026, creating an asynchronous Python 3.12 storage engine that uses subprocess workers to bypass the CPython Global Interpreter Lock during compaction [7]. In Procedia Technology, Riyaz and Varghese showed an optimized variant called SLSM for Apache HBase on commodity hardware [5].

Leveled Versus Tiered Compaction

Storage engines choose between size-tiered compaction and leveled compaction depending on whether the workload prioritizes write throughput or read latency. In size-tiered compaction, which serves as the default policy in Apache Cassandra, the engine groups SSTables of similar sizes into tiers. When a tier accumulates a threshold number of files (often four tables), the background compactor merges them into a single larger file. This strategy keeps write amplification low, between two and three times the original write volume, but read amplification increases because multiple overlapping files must be searched. It works best for append-only time-series data. Martinuke0 noted that Cassandra deployments under LinkedIn and Apple production workloads adjust tombstone thresholds to prevent unbounded tier growth [11].

Leveled compaction organizes disk storage into distinct levels labeled Level 0 through Level N, where each level is ten times larger than the previous one. While Level 0 files may have overlapping key ranges because they come from MemTable flushes, Level 1 and deeper levels enforce non-overlapping key ranges across all files. When Level 1 grows beyond its size threshold (typically 256 megabytes), the engine picks a file and merges it with overlapping files in Level 2. Level 1 holds 256 megabytes [15].

The cost of leveled compaction is higher write amplification, which often reaches ten to thirty times the incoming payload volume as data moves down the hierarchy [14]. In IEEE Access, Lee, Choi, and Park introduced TLSM, utilizing non-volatile memory to cut LevelDB write latency by 34 percent [3]. Teams tune their lsm trees carefully to balance read latency against SSD endurance [11].

Operational Pitfalls in Production

Production teams operating write-heavy clusters face operational incidents when background compaction falls behind incoming traffic [14]. When writes arrive faster than background threads can merge SSTables, the number of files in Level 0 multiplies, causing read latency to jump by an order of magnitude. If the MemTable pool exhausts its allocated memory budget, the engine initiates a write stall, artificially throttling client requests or throwing timeout exceptions until compaction catches up. Write stalls protect cluster memory [11].

Another common failure mode involves uncontrolled space amplification caused by accumulated tombstones and duplicate records. Jaranilla, Zhao, and Choi reported in Electronics that separating large values from small keys with BlobDB cuts compaction overhead, lifting range query throughput by 1.8 times over RocksDB on NVMe solid-state drives while reducing storage hardware costs by 49.5 percent [6]. When applications issue bursts of deletes, running manual compaction routines or scheduling tombstone-aware compaction windows prevents disks from running out of capacity. Storage nodes risk capacity exhaustion during heavy deletion cycles [11].

Engineers must configure write buffers and thread pools to match hardware constraints rather than relying on default settings. Heidari, Ahmadi, and Zhang introduced DobLIX in Proceedings of the VLDB Endowment, using learned index models to tune data access paths dynamically [4]. In real-world clusters consuming high-throughput streams from Kafka, pausing consumer loops prevents lsm trees from destabilizing under ingest spikes [11].

1. ^PREPRINT Luo, C., & Carey, M. J. (2018). LSM-based Storage Techniques: A Survey. arXiv. [Article Link]
2. ^{ACADEMIC JOURNAL} Zhang, B., & Du, D. H. C. (2021). NVLSM: A Persistent Memory Key-Value Store Using Log-Structured Merge Tree with Accumulative Compaction. ACM Transactions on Storage, 17(3), 1–26. [Article Link]
3. ^{ACADEMIC JOURNAL} Lee, J., Choi, W. G., Kim, D., Sung, H., & Park, S. (2020). TLSM: Tiered Log-Structured Merge-Tree Utilizing Non-Volatile Memory. IEEE Access, 8, 100948–100962. [Article Link]
4. ^{ACADEMIC JOURNAL} Heidari, A., Ahmadi, A., & Zhang, W. (2025). DobLIX: A Dual-Objective Learned Index for Log-Structured Merge Trees. Proceedings of the VLDB Endowment, 18(11), 3965–3978. [Article Link]
5. ^{ACADEMIC JOURNAL} Riyaz, P., & Varghese, S. M. (2016). SLSM – A Scalable Log Structured Merge Tree with Bloom Filters for Low Latency Analytics. Procedia Technology, 24, 1491–1498. [Article Link]
6. ^{ACADEMIC JOURNAL} Jaranilla, C., Zhao, G., Choi, G., Park, S., & Choi, J. (2025). A Framework for Integrating Log-Structured Merge-Trees and Key–Value Separation in Tiered Storage. Electronics, 14(3), 564. [Article Link]
7. ^{ACADEMIC JOURNAL} Paraskar, C., Dhas, S., Gajbhiye, S., Gopnarayan, S., & Mane, M. H. (2026). KiwiDB: An Async-Native Log-Structured Merge Tree Key-Value Store. Zenodo (CERN European Organization for Nuclear Research). [Article Link]
8. ^WEBSITE GeeksforGeeks. (2023). Introduction to Log Structured Merge (LSM) Tree. [Article Link]
9. ^WEBSITE Dinesh, J. (2025). Log-Structured Merge Trees. jidin.org. [Article Link]
10. ^WEBSITE Tomaselli, F. (2023). Log-Structured Merge Tree. [Article Link]
11. ^WEBSITE Martinuke0. (2026). Mastering Log-Structured Merge Trees: Optimizing Write-Intensive Distributed Databases for High-Throughput Production Environments. Martinuke0’S Blog. [Article Link]
12. ^WEBSITE Tomosvari, I. (2023). Building a Database I – Log-Structured Merge Trees. TMSVR – Dev Blog. [Article Link]
13. ^WEBSITE System Design Mastery. (2026). The Log-Structured Merge-Tree (LSM-Tree). [Article Link]
14. ^WEBSITE TheCodeForge. (2026). LSM Trees and SSTables: How They Work and Why They Dominate. [Article Link]
15. ^WEBSITE JZ. (2026). System Design – How LSM Trees Work (The Engine Behind RocksDB, LevelDB, and Cassandra). JZLeetCode. [Article Link]
16. ^WEBSITE Stopford, B. (2015). Log Structured Merge Trees. [Article Link]
17. ^WEBSITE Wikipedia contributors. (2026). Log-Structured Merge-Tree. Wikipedia, The Free Encyclopedia. [Article Link]