ScyllaDB: The NoSQL Database Beating Cassandra at Its Own Game
NoSQL databases have taken the technology world by storm over the past decade. When you need to store and query huge volumes of data with demanding performance requirements, relational databases often struggle to keep up. NoSQL databases provide a more scalable, flexible alternative by relaxing data consistency in favor of availability and partition tolerance, based on the famous CAP theorem.
Apache Cassandra has long been one of the most popular and proven NoSQL databases. Originally developed at Facebook and inspired by Amazon‘s Dynamo paper, Cassandra is a wide column store database that provides high write throughput and built-in data replication across multiple commodity servers. Some of Cassandra‘s notable users include Apple, Netflix, eBay, and thousands of other companies that need to manage operational data at massive scale.
However, a newer NoSQL database called ScyllaDB has emerged that takes on Cassandra head-to-head, claiming to be a drop-in replacement with significantly better performance. How can ScyllaDB achieve an order of magnitude lower latency and higher throughput than the battle-tested Cassandra? Let‘s take a closer look at the design and architecture of ScyllaDB to find out.
Implementing Cassandra in C++
The key to ScyllaDB‘s performance advantage is that it‘s written in modern C++, while Cassandra is implemented in Java. ScyllaDB contains an independent reimplementation of the Cassandra server, including replication, the CQL query language, and data partitioning. Because C++ compiles to low-level machine code, it avoids much of the overhead of Java and the JVM, leading to significantly reduced CPU usage and memory footprint.
However, just doing a straight port of Cassandra to C++ is not enough to achieve top performance. The ScyllaDB team has rethought many of Cassandra‘s internal architecture choices and come up with novel optimizations that improve throughput, latency, and reliability. Let‘s examine some of the most important design decisions.
Shared-Nothing Architecture
Like Cassandra, ScyllaDB utilizes a shared-nothing architecture where each node is independent and self-sufficient. However, ScyllaDB takes this concept to the extreme. It uses a framework called Seastar to implement a shared-nothing model all the way down to the CPU core.
With Seastar, each CPU core gets its own memory, TCP/IP stack, and even a cooperative userspace scheduler. Requests are sharded onto individual cores, so the whole application has only one thread per core with no shared memory between them. If a request on one core needs data that resides on another core, it‘s sent over via an asynchronous message passing system, rather than sharing any state.
ScyllaDB‘s shared-nothing Seastar architecture avoids shared state and context switches between cores. (Source: ScyllaDB)
This design avoids the need for any locking, mutexes, or atomic operations on shared data structures. As a result, ScyllaDB eliminates many of the context switches, memory barriers, and cache invalidations that slow down Cassandra‘s Java-based architecture.
Multiple studies have shown that a traditional thread-per-request design like Cassandra‘s can spend up to 30% of total execution time just on locking and context switching between threads. In a highly concurrent NoSQL workload with sub-millisecond latencies, that overhead quickly adds up.
By contrast, a shared-nothing design with message passing like ScyllaDB‘s has very little communication overhead. One study found synchronization costs as low as 200 nanoseconds between cores when using asynchronous futures in Seastar. Avoiding shared state also improves cache locality, since each core only needs to load its own data into L1/L2 cache.
The DPDK userspace network stack also yields huge efficiency gains, as we‘ll see next.
Kernel Bypass Networking
Another major bottleneck in Cassandra is the networking stack. Like most databases, Cassandra relies on the standard OS kernel TCP/IP stack to communicate with clients and other nodes in the cluster. The kernel provides a robust, general-purpose network stack, but it comes with a lot of overhead from system calls, buffer copying between kernel and userspace, and thread scheduling.
ScyllaDB takes a radically different approach by completely bypassing the kernel network stack. It uses the Data Plane Development Kit (DPDK) to implement the entire network stack in userspace. DPDK is a set of libraries and drivers that allow applications to process network packets directly in userspace, skipping the kernel entirely.
With DPDK, ScyllaDB can send and receive network packets with minimal latency, copying data directly to and from the NIC ring buffers. Studies have shown this technique can process a packet in as little as 80 CPU cycles. ScyllaDB‘s userspace TCP/IP stack is built on top of DPDK to provide compatibility with Cassandra clients and tools, while benefiting from kernel bypass for inter-node communication.
This optimization is especially important for deployments with high-bandwidth network interfaces like 10 GbE or higher. Kernel bypass allows ScyllaDB to saturate multi-gigabit links with millions of small packets per second for user queries and cluster coordination.
Skipping the Page Cache
Most databases, including Cassandra, make heavy use of the filesystem page cache to speed up reads. The page cache keeps recently accessed data from disk in memory, avoiding slow I/O operations. While caching is crucial for good performance, it doesn‘t always fit the I/O patterns of NoSQL workloads.
In particular, Cassandra‘s SSTables are not a good match for the page cache. The data in an SSTable is already compressed and must be deserialized to be used, so caching the on-disk bytes wastes a lot of RAM. Worse, when an SSTable is compacted, all of its cached pages get invalidated and purged from RAM even if the actual data has not changed.
ScyllaDB solves this by allocating its own DRAM outside the page cache to hold a row cache of deserialized, ready-to-use data. The row cache is not invalidated by SSTable compactions, so ScyllaDB can continue serving read requests from RAM even as maintenance operations occur in the background. This allows ScyllaDB to make much more effective use of large memory machines for caching.
Another advantage of the row cache is that it only needs to hold the subset of "hot" data that‘s actively being queried, rather than wasting space on caching large amounts of cold data on persistent storage. ScyllaDB automatically manages the row cache using an LRU eviction algorithm, sizing it to fit the working set in RAM.
Impressive Performance Results
How do all of these low-level optimizations translate into real-world performance? Extensive benchmarks have shown that ScyllaDB lives up to its claims of being much faster than Cassandra:
| Workload | Measurement | ScyllaDB | Cassandra |
|---|---|---|---|
| Reads/sec | 99% latency | 0.65 ms | 19.61 ms |
| Reads/sec | Max throughput | 4.6 M/s | 0.22 M/s |
| Updates/sec | 99% latency | 0.62 ms | 18.66 ms |
| Updates/sec | Max throughput | 0.66 M/s | 0.024 M/s |
Performance comparison of ScyllaDB and Cassandra on AWS i3.4xlarge instances. (Source: ScyllaDB)
In the Yahoo! Cloud Serving Benchmark (YCSB), ScyllaDB achieves over 1 million operations per second with a 99th percentile latency of less than 1 millisecond on a 3-node commodity server cluster. To reach that same performance level, Cassandra would require 30 nodes, resulting in 10 times the hardware and operational costs.
ScyllaDB‘s own benchmarks show even more impressive results, with P99 read latencies up to 8x lower than Cassandra‘s and maximum throughput up to 5x higher. While vendor benchmarks should always be taken with a grain of salt, the overall picture from multiple independent tests is clear: ScyllaDB delivers substantially better performance than Cassandra across a variety of workloads and deployment scenarios.
These performance gains have convinced many former Cassandra users to switch to ScyllaDB in production:
- GE Healthcare migrated its Predix Industrial IoT platform from Cassandra to ScyllaDB, improving P99 latency by 10x and reducing infrastructure costs by 4.8x.
- Grab adopted ScyllaDB for its real-time data pipeline, replacing 50+ Cassandra nodes with 3 ScyllaDB nodes while increasing throughput 5x.
- Lookout moved its threat intelligence workload from Cassandra to ScyllaDB, reducing read and write latency by 50% and cluster size by 75%.
"We‘re now doing about 1 million transactions per second with single digit millisecond P99 response times on a 3 node ScyllaDB cluster, and no one on my team has lost any sleep worrying about it."
– Dan Wertheim, Software Engineer at GE Transportation
Full Cassandra Compatibility
ScyllaDB‘s biggest advantage may be what it doesn‘t change: the API and operational interface. ScyllaDB is a drop-in replacement for Cassandra that supports the same CQL queries, data model, drivers, tools, and file formats. Migrating an existing Cassandra application to ScyllaDB is generally as simple as pointing it to the new cluster.
ScyllaDB is even compatible with the Cassandra query tracing format and many open source tools in the ecosystem like the DataStax Bulk Loader and Apache Spark™. This allows teams to adopt ScyllaDB incrementally and leverage existing Cassandra expertise and integrations.
Of course, no database technology is without drawbacks. ScyllaDB‘s potential disadvantages compared to Cassandra include:
- Smaller and younger open source community
- Fewer managed service options (ScyllaDB offers one; many more for Cassandra)
- Some obscure Cassandra features like range deletes and static columns not supported
- Single-datacenter deployments only (so far; multi-DC coming soon)
For most users, though, the raw performance of ScyllaDB will outweigh these limitations. And the ScyllaDB team has an aggressive roadmap to reach parity with Cassandra and expand into new use cases like graph, search, and streaming analytics.
The Road Ahead for ScyllaDB
What‘s next for this up-and-coming NoSQL database? The ScyllaDB team is hard at work on new features that will continue to expand its performance lead and make it viable for even more projects:
- Lightweight Transactions (LWT): Adds linearizable consistency using Paxos consensus, a good fit for financial and retail apps
- Incremental compaction strategy: Reduces write amplification and improves client tail latency
- Change Data Capture (CDC): Streams changes from the database to external sinks like Apache Kafka or Amazon S3
- Alternator: DynamoDB-compatible API to ease migration from Amazon‘s NoSQL cloud database
The ScyllaDB team is also investing in improved Kubernetes integration, tiered storage for cost-effective cold data storage, and machine learning powered automation for easier operations. These efforts will make ScyllaDB more cloud native, cost effective, and user friendly over time.
Looking further out, there are exciting possibilities to combine ScyllaDB with other scalable data infrastructure. Imagine a unified API to query ScyllaDB tables alongside full-text search in Elasticsearch, real-time stream processing in Apache Flink, and cloud object storage in MinIO. The shared-nothing, DPDK-accelerated Seastar framework could potentially be extended to these other data-intensive systems.
Conclusion
ScyllaDB has shown that a NoSQL database can have the best of both worlds: the features and ecosystem of Cassandra with the blistering speed and efficiency of C++. By rethinking many of Cassandra‘s foundational architecture choices and applying modern techniques like shared-nothing, kernel bypass, and a row cache, ScyllaDB shatters performance barriers without compromising on scalability or availability.
Companies looking to get the most out of their infrastructure budget and developers seeking a highly performant yet operationally simple database should give ScyllaDB serious consideration as a Cassandra replacement. With proven results from early adopters and an ambitious roadmap, ScyllaDB is poised to take on Cassandra and other NoSQL databases head-on in the coming years.
To learn more about ScyllaDB and get started, check out these resources: