Hello, Liven — A Database for Data That Moves

Why Another Database?

Databases today force you to choose: batch or stream? Historical or real-time? Key-value or vector? Liven was built to erase those lines.

One query language, two modes:

• Historical queries against stored data
• Real-time subscriptions on the same pipeline — just add .listen()

Three deployment models:

• Embedded library (~1.5 MB) — runs inside your Rust process
• Network server — TCP, thousands of concurrent clients
• Interactive TUI shell — for ad-hoc queries and monitoring

The Architecture

Liven's architecture is built around a simple idea: the same pipeline that processes streaming data should also query historical data.

• Writes are appended to segment files. A background flusher batches them for throughput without sacrificing durability.
• Reads go through a lock-free in-memory index. Point lookups resolve in microseconds.
• Subscriptions broadcast every write to all listeners. The server evaluates pipeline filters before delivery.

Storage & Data Model

Records are stored in a safe, append-only format that survives crashes. An in-memory index provides instant lookups by key. The engine supports text, numbers, booleans, nested objects, binary data, and vector embeddings — all within named collections called streams.

Deleted data is automatically compacted in the background to reclaim disk space, and the engine tunes its memory usage based on available RAM.

Query Capabilities

• Insert, update, upsert (insert-or-replace), and delete individual records
• Batch insert and update for throughput
• Filter by field values with equality, comparison, range, substring, and membership checks
• Sort, limit, paginate (offset and cursor-based)
• Count, group by fields with aggregations, and deduplicate
• Project only the fields you need

Streaming & Real-Time

Subscribe to live updates and get notified when new records match your filter. Tail a stream to see records as they arrive. Time-windowed aggregations let you analyze rolling data in motion.

Vector Search

Store and search quantized vector embeddings — like those produced by AI models — directly alongside your other data. Cosine similarity search with a configurable threshold means no separate vector database is needed.

Cross-Stream Joins

• Enrich: Left-join records with data from another stream
• Correlate: Find related records across streams within a time window
• Chain: Follow key relationships across multiple streams hop by hop
• Sequence: Detect ordered event patterns within a time window — for example, detect when a login is followed by a high-value purchase within five minutes

Deployment

• Embedded — runs inside your application as a Rust library (~1.5 MB)
• Network server — standalone TCP server that many clients can connect to simultaneously
• TUI shell — interactive terminal for exploration and debugging

SDKs

Client libraries are available for Rust (native), TypeScript, Python, and Go — all using the same native wire protocol with no HTTP overhead.

Security

API key authentication with configurable roles (admin, read-only, write, write-delete). Keys are hashed and stored securely. Optional mutual TLS for zero-trust network architectures.

Operations

• CLI commands: start, stop, status, list streams, tail, import, export, and key management
• TOML configuration with sensible defaults
• Automatic memory detection — tunes itself to available system RAM
• Cross-platform: macOS, Linux, Windows
• Import and export data as JSONL or CSV

What's Next

This is just the beginning. Upcoming work includes multi-node replication for high availability and expanded SDK support for more languages.

Stay tuned, and check out the documentation to get started.