Query Guide

Filtering Data

Use the filter() stage to narrow down records by field conditions.

Comparison Operators

Filter records by comparing field values against constants:

from("orders") | filter(amount > 100)
from("orders") | filter(amount >= 100)
from("orders") | filter(amount < 50)
from("orders") | filter(amount <= 50)
from("orders") | filter(status == "completed")
from("orders") | filter(status != "cancelled")

String Matching

Liven supports prefix, substring, and suffix matching on string fields:

// Prefix match
from("users") | filter(key startsWith "admin_")
// Substring match (anywhere in the string)
from("logs") | filter(message contains "error")
// Suffix match
from("files") | filter(name endsWith ".log")

Set Membership

Check if a field matches any value in a list:

from("users") | filter(key in ["user_1", "user_2", "user_3"])

Range Checks

Check if a numeric field falls within an inclusive range:

from("orders") | filter(amount between [100, 500])

Logical Combinations

Combine conditions with and, or, and not:

// AND — both conditions must be true
  from("orders") | filter(amount > 100 and status == "completed")
  // OR — either condition can be true
  from("orders") | filter(status == "error" or status == "warn")
  // NOT — invert any condition
  from("users") | filter(not status == "inactive")
  // Grouped with parentheses
  from("events") | filter(not (type == "info" or type == "debug") and severity > 3)

Filtering by Timestamp

Timestamp fields support range queries. Liven uses a per-stream timestamp index for efficient lookups — no full segment scan needed:

// Records after a specific time
  from("events") | filter(timestamp > 1700000000000)
  // Records within a time window
  from("sensors") | filter(timestamp between [1700000000000, 1700086400000])

Timestamps are in milliseconds since Unix epoch. The between operator is inclusive on both bounds.

Performance Tip

Order your filters from most to least restrictive. The most selective filter should come first to reduce the record set before later stages process it:

// More efficient — narrows data first
  from("orders") | filter(amount > 1000) | filter(status == "pending")
  // Less efficient — broad filter first
  from("orders") | filter(status == "pending") | filter(amount > 1000)