Manticore Search vs Elasticsearch performance and feature comparison

Manticore Search 27.1.5 outperforms tuned Elasticsearch 9.4.3 by 3.85x for 100M+ comments. In tests with 10M Nginx logs, Manticore was 22 times faster than Elasticsearch for analytical queries. This speed advantage helps users who need timely and relevant data during high traffic. For a dataset of 1.7 billion documents, Manticore was 4 times faster. Data ingestion speeds also differ. On a 32-core server with SSD, Manticore ingested a dataset in 1 hour and 8 minutes, while Elasticsearch required 28 hours and 33 minutes. This means Manticore is 25 times faster for data ingestion tasks.

Manticore Search 27.1.5 is 340x faster than MySQL 9.7.1 for 1.1M Hacker News comments. It is also 3.85x faster than tuned Elasticsearch 9.4.3 for 100M+ comments and 5.03x faster for selected DevOps queries on 10M Nginx logs. In those Nginx tests, Manticore ingested data in 5m 46s versus 10m 15s for Elasticsearch while using 3.98 GB of RAM compared to 36.98 GB of RAM.

Elasticsearch relies on the Java Virtual Machine and the Lucene library. Users must manage JVM heap size and deal with garbage collection pauses. Manticore uses C++ to operate closer to the hardware. An empty Manticore instance uses around 40MB RSS. I would skip Elasticsearch if your project requires low resource consumption and predictable memory behavior. As you already know, managing JVM memory is a headache.

Storage and Query Logic

Manticore Search includes row-wise and columnar storage. Row-wise storage provides fast performance for small to large datasets. Columnar storage, which uses the Manticore Columnar Library, handles datasets that exceed RAM capacity. Manticore also has a docstore that does not require RAM but only allows fetching original values without sorting or grouping.

Manticore uses SQL as the native syntax. Elasticsearch has very limited SQL support and cannot perform basic actions like SELECT id, INSERT, UPDATE, or DELETE. Manticore understands the MySQL protocol, so you can use MySQL clients for administration. You can also use Manticore with the MySQL/MariaDB FEDERATED engine or via ProxySQL.

Manticore provides vector search through the HNSW algorithm. This implementation supports cosine similarity, inner product, and squared L2 distance. You can integrate this with SQL or JSON APIs for semantic search and recommendation systems. Manticore also allows you to power conversational search. You can use SQL CALL CHAT or the HTTP JSON /search endpoint to ask questions with LLM providers. This process uses KNN search to retrieve relevant documents and builds context from them.

Migration and Scaling

Teams moving from Sphinx to Manticore find the transition easy because Manticore is a fork of Sphinx. One user replaced a 2016 release of Sphinx with Manticore 4.2 and saw that their existing PHP application worked perfectly without any changes to the code or the queries. However, upgrading from version 2.x to 3.x requires the index_converter tool to handle the redesigned table storage. This tool can convert tables one at a time or all at once using the –all flag. In the Sphinx era, users relied on .sps, .spa, .spd, and .spp files. Manticore 3.x uses different extensions like .spb for var-length attributes, .sphi for secondary index histograms, and .spds for document storage.

Manticore supports two usage modes: imperative and declarative. The imperative mode allows you to manage everything online using CREATE TABLE or DROP TABLE commands, whereas the declarative mode uses configuration files for mappings to help with CI/CD and ETL processes. Manticore’s sharded tables distribute reads and writes across physical shards on one node or across a replication cluster.

Manticore provides full-text capabilities with over 20 operators and more than 20 ranking factors. It includes text stemming, lemmatization, synonyms, and wordforms. It also handles complex languages like East Asian and Slavic languages. Manticore provides built-in authentication and fine-grained authorization. You can protect SQL, HTTP, HTTPS, and replication-related access with user credentials or bearer tokens. This authentication is opt-in, so you can introduce it when your deployment is ready.

Manticore provides official clients for PHP, Python, JavaScript, TypeScript, Java, Elixir, Go, and Rust, and it integrates with Apache Superset, Grafana, Fluentbit, and Vector.dev. Unlike Elasticsearch, Manticore also supports Kafka integration. Elasticsearch is the standard for log management because of its tools like Logstash and Kibana. Manticore can interface with these tools, but its support for Kibana is in a beta stage. Does Manticore’s focus on performance outweigh the ecosystem depth of the ELK stack?