Downloads
- Spec (Markdown)
- Sample SFLOG block (text)
- Sample dictionary (JSON)
- sf128log (Linux amd64)
- sf128log (macOS arm64)
CLI (source): build with go build -o bin/sf128log ./cmd/sf128log from the repository.
Live Demo
Try the Streaming Demo: raw logs (top), live SF128‑LOG conversion (bottom), and live analytics in the middle.
Concept
Emit a single Header (shared context) and many tiny Delta rows (per‑event) — then deterministically reconstruct full 128‑bit Snowflakes from Header.ts_base + delta, node, geo, and type.
- Header: magic/version, type, ts_base (MJD×10⁶), geo, node.
- Delta: +NNNNN microsecond offset or sequence.
- Unique, sortable IDs with excellent compression.
Interactive Demo
Why It Compresses
- Stable header repeats once per block; deltas are short integers.
- Binary packing (varints) shrinks deltas to ~2–3 bytes each.
- Deterministic reconstruction preserves global ID semantics.
Structured SF128 Payload Example
The 20‑bit Entropy field can embed compact semantics for reconstruction. Example packing:
payload20 = (user8 << 12) | (daemon4 << 8) | (action4 << 4) | object4.
External Dictionary Reconstruction
Given Header(dict_id, ts_base, node) and an SF128 with a 20‑bit payload, we can reconstitute the natural log with a shared dictionary.