Why I Wrote a Database Engine in Rust as a Student

The Problem I Was Actually Solving

Why Rust and Not C, Go, or Python

The Architecture: Bitcask Meets AES-256

1. All writes are appended to an immutable data file. You never modify existing data, you only append.
2. An in-memory index (called a KeyDir in Bitcask, similar in FerrumDB) maps every key to the exact offset and size of its value on disk.
3. To read any key, you do exactly one disk seek to the offset in the KeyDir. This gives you O(1) reads regardless of database size.

terminal
COPY
pub struct KeyDirEntry {
    pub offset: u64,     // byte position in data file
    pub size: usize,     // byte length of the value
    pub timestamp: u64,  // unix timestamp for versioning
}

pub struct FerrumStore {
    keydir: HashMap<String, KeyDirEntry>,
    data_file: AsyncFile,
    crypto_engine: AesGcmEngine,
}

terminal
COPY
impl FerrumStore {
    pub fn get(&self, key: &str) -> Result<Vec<u8>> {
        // Step 1: O(1) lookup in the in-memory index
        let entry = self.keydir.get(key)
            .ok_or(Error::NotFound)?;

        // Step 2: Single positional read from disk
        let mut encrypted_data = vec![0u8; entry.size];
        self.data_file.read_at(&mut encrypted_data, entry.offset)?;

        // Step 3: AES-256-GCM authenticated decryption
        self.crypto_engine.decrypt(&encrypted_data)
            .map_err(|_| Error::CryptoFailure)
    }
}

The Encryption Layer: Why AES-256-GCM and Not AES-256-CBC

aes-gcm

terminal
COPY
use aes_gcm::{Aes256Gcm, Key, Nonce};
use aes_gcm::aead::{Aead, NewAead};

pub struct AesGcmEngine {
    cipher: Aes256Gcm,
}

impl AesGcmEngine {
    pub fn encrypt(&self, data: &[u8]) -> Result<Vec<u8>> {
        let nonce = Nonce::from_slice(&self.generate_nonce());
        let ciphertext = self.cipher.encrypt(nonce, data)
            .map_err(|_| Error::EncryptionFailed)?;
        
        // Prepend nonce to ciphertext for storage
        let mut result = nonce.to_vec();
        result.extend_from_slice(&ciphertext);
        Ok(result)
    }

    pub fn decrypt(&self, data: &[u8]) -> Result<Vec<u8>> {
        let (nonce_bytes, ciphertext) = data.split_at(12);
        let nonce = Nonce::from_slice(nonce_bytes);
        
        self.cipher.decrypt(nonce, ciphertext)
            .map_err(|_| Error::DecryptionFailed)
    }
}

The Hardest Parts: What No Tutorial Tells You

1. Cross-Language Memory Safety

2. Async Disk I/O With Tokio

fsync

3. The Compaction Problem

What I Would Do Differently

RwLock

proptest

The Broader Lesson