rqlite replaces complex distributed storage requirements

Replicated replaced PostgreSQL with rqlite to simplify the operation of its Application Manager. This change removed the requirement for distributed storage, such as Rook or Longhorn, when running in a multi-node Kubernetes installer cluster. Instead, the Application Manager uses rqlite to store version information. Because rqlite replicates data itself, the team avoids using complex infrastructure to replicate data to nodes in a cluster. This switch enables the use of lighter weight storage like OpenEBS Local PV. rqlite provides fault tolerance and high availability for critical data using the Raft consensus algorithm and SQLite. It provides relational data modeling, a feature missing in tools like etcd or Consul. rqlite is available for Linux, OSX, and Microsoft Windows. The system includes a Discovery Service to allow nodes to automatically connect and form a cluster. A single node automatically becomes the leader after installation. The rqlite command-line interface supports functions similar to SQLite dot-commands. Recent improvements to rqlite include a fix for an important bug related to Named Parameters, which allows NULL to be used as a parameter. The software also upgraded to SQLite 3.39.4. The latest configuration for Kubernetes includes both a Headless Service for rqlite autoclustering and a standard service so clients can communicate easily with the cluster. You know how managing multi-node Kubernetes clusters often requires complex distributed storage to ensure availability.

Infrastructure fragmentation and performance bottlenecks

Infrastructure fragmentation remains a significant hurdle for DevOps adoption. A Quali survey found that 54% of IT organizations have no access to self-service infrastructure. This lack of access forces more than half of respondents to use a ticket-based approach to infrastructure delivery, which impacts productivity. The survey also identified 14% of respondents saying company culture is a top barrier, 13% saying testing automation is a barrier, and 12% saying legacy systems are a major barrier. Other difficulties include limited IT skills and managing multiple environments. According to an Intel Data Center Manager survey of 204 IT managers and engineers, 51% of companies use infrastructure-as-a-service, while 65% of system engineers use platform-as-a-service. Professionals spend an average of 10.1 hours per week tuning infrastructure and nine hours on remediation analysis. Managing these environments is difficult because DevOps engineers use fragmented toolsets across different projects. 70% of companies aspire to deploy to hybrid clouds, yet only 23% of respondents run applications on a hybrid cloud platform. The survey also noted that 65% of respondents run fewer than 24 applications in hybrid environments, while only 8% run more than 75. Jeff Klaus of Intel DCM says determining the monetary means to use IaaS is a challenge for companies. rqlite performance depends on disk performance and network latency. Because the Raft log is the authoritative store for all data and is written to disk by each node, an in-memory database can be fully recreated on start-up if the entire cluster loses power. Write throughput for rqlite stays between 10 and 200 operations per second per node. Frequent snapshots reduce node start-up time but increase CPU and I/O load.

Optimizing relational data for edge deployments

Teams can increase rqlite write throughput by two orders of magnitude through the bulk API or by using transactions. Using a memory-backed filesystem can lead to a 100x improvement in performance by storing the Raft log and SQLite database in memory. This option risks total data loss if the entire cluster loses power. Users can also use the SQLite VACUUM command to defragment the database and shrink it. Performing a VACUUM may temporarily double the disk usage. To improve query performance, rqlite automatically performs a PRAGMA optimize once a day. If memory usage spikes due to large requests, such as a request larger than 25MB on a 1GB allocation, administrators can reduce the GOGC value. This adjustment increases the frequency of garbage collection cycles but increases CPU load. Users can also adjust the GOMEMLIMIT setting to manage memory. For those who can tolerate a small risk of data loss during a node crash, Queued Writes can provide orders of magnitude in write performance improvement. Only deterministic SQL statements are safe to use because rqlite performs statement-based replication. For instance, using time-based functions can result in different SQLite databases under each node. The system allows for the adjustment of snapshotting through the -raft-snap, -raft-snap-wal-size, and -raft-snap-int flags. Can rqlite manage the growing demand for relational data in edge environments as effectively as traditional distributed databases? Administrators should use the bulk API to maximize throughput when deploying to high-volume pipelines.