Introduction:

Speedb is a modern key-value store that leverages advanced techniques to deliver exceptional performance. It is built upon RocksDB, a popular open-source storage engine, but incorporates additional optimizations to maximize performance and efficiency.
In this blog post, we will explore the problems Speedb solves and how, benefits and performance comparisons of Speedb and RocksDB to understand why it is becoming a popular choice for performance-critical applications.

What Speedb is?

Embedded key value storage engine
Open source
It has an enterprise version
Drop-in replacement for rocksDB (fully compatible)
Written in C++, API exposed in Java and C as well

What Speedb is NOT?

Stand alone application
Database Management System
Not written in Go/Rust (but wrappers are available)

Which problems Speedb solves?

Performance Hiccups - One of the pain points of RocksDB users is the hiccups that actually affect performance of the production applications. Speedb improved the mechanism of delayed writes when the write rate reaches a certain threshold. The delay is now done moderately, taking into account the limit set by the user and this changed dramatically the application behavior and not more hiccups observed while using it.

Unexpected memory usage: In order to run Speedb properly, the memory allocation should be well defined by the user. One of the parameters is the write buffer size. Which means, how much data can the application hold in memory before it flushes it to the disk. This of course affects the application behaviors. If you define a much lower size than the applican’s write rate, you can get into a situation where the storage engine can’t handle all of the writes and it will slow down until zero, which means stalls. In case your application is more sensitive to performance than you are limited in memory, you can observe high memory usage for the dirty data.

Speedb eliminates this tradeoff when using the new write buffer manager - your application will perform well, with no hiccups and without exceeding the memory limit you have defined.

The graph below compares RocksDB write buffer manager and Speedb write buffer manager with 32 column families and 4 memtables of 256MB max each. The write buffer manager is set to 4GB.
With rocksdb the actual memory consumption when the allow stalls is enabled reached 11GB and many stalls occurred, leading to unstable performance, while with Speedb it was consumed 4GB only and not stalls at all.
The graph represents a test of heavy write workload (95%) and shows the stalls reached while using RocksDB’s write buffer manager vs Speedb write buffer manager.

Read performance: RocksDB is designed for write intensive workload. But read performance should not be neglected. With the static pinning feature you can enjoy the benefits of pinning filter and index blocks to the cache without risking out of memory condition. The pinning has a major advantage over LRU cache and users barely use it because of the risk of being out of memory. Speedb forcing a cap of the pinned data and allows you to enjoy the performance without risking the application.

Usability: RocksDB is a very smart but also very complex storage engine. You sometimes need to make some configuration changes when you realize the current configuration is not optimal.

1. Live configuration changes: Speedb improves the usability of this complex data engine: when you want to make changes to the storage engine without any downtime, you can easily do it using the live configuration changes mechanism. This is like a backdoor that allows changing immutable options on the fly without affecting your application’s availability.

2. Tuning function:
with Speedb’s tuning function you can enable Speedb features and tune basic parameters for optimized performance for single and multi Speedb instances, in case you are running in a multi-db environment. The tuning function also sets the write buffer manager size for optimized dirty data management of all your instances.

3. Log parser: This is a very useful python tool to analyze the storage engine logs. It gives a high level insight on your database structure and performance profile and also provides deep understanding of internal processes such as compaction in a clear and readable format.

Enables 100% parallel read/write workload:
The Sorted hash memtable, is a new memtable type that improves read and write performance from O (logn) to O(1) without compromising on seek performance, supports parallel reads and writes since the data structure has changed from skip list to a combination to hash table and array of vectors.

Improved write flow: Speedb introduced major performance improvements and 100% parallel write support by changing the existing write flow: Instead of a global DB mutex, a read/write lock is used, and now data is written to the WAL (write ahead log) and memtables simultaneously.

Write amplification (Available on Speedb Enterprise):
Speedb Enterprise's revolutionary compaction method provides several benefits over traditional LSM trees. It significantly reduces write amplification, from 24 to 4, eliminates processing latency and throughput issues, and reduces CPU utilization and memory consumption, making it a cost-effective and efficient solution for enterprise use.

No Shards - Performance at Scale: (Available on Speedb Enterprise)
Speedb enterprise introduces a multidimensional compassion method that allows keeping extremely high performance with much larger datasets without the need to shard your database. This of course simplifies the day to day management of your environment and you can get more with less.

Speedb is also managing the biggest community for Speedb and RocksDB users.
Join us to learn more: Speedb Discord