The trajectory of Redis from a 2009 prototype to a 2026 data platform

Salvatore Sanfilippo started Redis in 2009 to solve scaling problems at his Italian startup, LLOOGG. He designed the system to handle real-time web log analysis because MySQL handled high-frequency data operations too slowly. The initial version 0.1 contained only 2,000 lines of code and lacked features like sets, sorted sets, or fork() based persistence. Sanfilippo implemented the first data type, the list, and translated his Tcl prototype into C. He later moved the project to GitHub and saw early adoption from companies like GitHub and Instagram.

The architecture relies on in-memory storage to eliminate disk I/O bottlenecks for reads. Sanfilippo designed the system to use a single-threaded event loop to avoid context switching and locking overhead. This design ensures that commands execute sequentially, which maintains data consistency. While Redis supports multi-threaded I/O in recent versions, the command processing remains single-threaded. This architecture allows Redis to handle millions of operations per second with sub-millisecond latency.

The project received corporate sponsorship from VMware and later Pivotal starting in 2010 and 2013. Garantia Data founded in 2011 eventually became Redis Labs to build commercial products like Redis Enterprise. Redis Enterprise uses a shared-nothing architecture to scale linearly by adding shards and nodes. In 2018, Redis Enterprise set a performance record by delivering 200 million ops/sec with under 1 millisecond latency using 40 AWS instances. This benchmark achieved a 2.6X performance improvement compared to a 50 million ops/sec test from 15 months prior that used 26 AWS instances.

Licensing shifts and the Valkey fork

The licensing history of Redis includes several changes that divided the community. Sanfilippo originally released Redis under the BSD license to allow for competing branches. In 2018, Redis Ltd. changed the license for Redis Modules to the Apache License with a Commons Clause to restrict cloud providers from reselling these modules without a commercial agreement. In March 2024, Redis Ltd. switched the core software license from BSD-3-Clause to a dual license under the Redis Source Available License (RSAL) and the Server Side Public License (SSPL).

This change prompted the Linux Foundation to host Valkey, a fork of the last BSD-licensed version of Redis. AWS, Google Cloud, Oracle, Ericsson, and Snap back the Valkey project. Madelyn Olson, a longtime Redis maintainer, started the Valkey project at AWS after the license change. Many maintainers joined the Valkey community because they refused to contribute to the new restrictive licenses. While Redis remains the dominant choice for enterprise deployments, Valkey serves as the default on AWS ElastiCache and GCP Memorystore.

The company management at Redis Labs argues that the license change fixes a broken open source model. They claim cloud companies use monopoly power to adopt successful projects without contributing back. CEO Ofer Bengal stated that the company decided to raise a $60 million Series E round to accelerate growth in a competitive environment. This funding round included investors like Goldman Sachs Private Capital Investing, Bain Capital Ventures, Viola Ventures, and Dell Technologies Capital. The company aims to use the capital to push product development and prepare for a potential IPO.

Scaling to 10 billion keys

Handling 10 billion keys requires specific architectural discipline and memory management. A single Redis instance can hold hundreds of millions of keys, but 10 billion keys necessitate a cluster. Users must use Redis Cluster to partition the keyspace across 16,384 hash slots. Each node in the cluster owns a subset of these slots, which allows the system to distribute both the dataset and the request load across multiple machines.

Memory overhead is a major constraint for large datasets. Every key consumes memory for the key string itself, the dictionary entry, and expiry metadata. If a user stores a string key and a small value that takes 60 to 80 bytes, 10 billion keys will require 600 to 800 GB of RAM just for the data, excluding the overhead for the hash table and metadata. You should keep key names short to save space. Saving 25 bytes per key across 10 billion keys reduces memory requirements by 250 GB.

Efficiency improves when users group small values into Hashes instead of using individual string keys. Redis automatically encodes small integers as integers using OBJ_ENCODING_INT to save space. For high-performance workloads, developers use Redis Cluster with multiple primary nodes and replicas. A primary node handles writes and propagates data to replicas to provide redundancy and scale read throughput. If you need to scale a read-heavy workload, adding replicas is more effective than adding more primary nodes. Does the increased complexity of managing a large-scale cluster outweigh the performance gains for your specific application?