Modernizing Write-Heavy Scenarios in Distributed Systems: LSM Trees and Scylla DB

Abstract

As modern applications become increasingly dataintensive, there is a growing demand for database systems that can efficiently manage high write-throughput workloads without compromising on performance or scalability. This research examines the internal mechanisms of ScyllaDB, a highperformance NoSQL database that utilises Log-Structured Merge (LSM) trees to address the challenges of write-intensive environments. ScyllaDB optimises data ingestion and persistence by utilising in-memory MemTables, commit logs for durability, and immutable SSTables for disk storage, which are periodically merged through compaction processes to enhance read efficiency and reduce disk usage. The study examines the architectural components of ScyllaDB, focusing on the roles of SS Tables and compaction strategies—Size-Tiered, Levelled, and Time-Window Compaction—in managing storage and retrieval. Experimental deployment within an educational institutional setting demonstrates ScyllaDB’s ability to maintain low-latency writes and scalable data distribution across multiple nodes, highlighting the effectiveness of schema design, partitioning, and resource management. Furthermore, the paper introduces a novel cluster auto-tuning algorithm for ScyllaDB that dynamically adjusts system parameters (e.g., memory allocation, compaction rate, and thread pools) based on real-time performance metrics. A mathematical model is proposed to optimise system latency and throughput under workload variations, utilising reinforcement learning-inspired logic and resource-constrained optimisation. The findings validate ScyllaDB’s effectiveness in high-volume scenarios such as real-time analytics, IoT data management, and time-series databases, positioning it as a robust alternative to traditional relational systems and legacy NoSQL platforms. The proposed auto-tuning framework holds promise for future selfadaptive database clusters in dynamic environments. This research provides practical insights into deploying and optimising write-intensive database systems for modern distributed computing needs.