OBSESC
Get in touch
Docs: The map and the territory
Documentation / Concepts

The map and the territory

How OBSESC pairs a fast, rebuildable navigation tier with a complete, open fidelity tier, and why.

OBSESC stores every log event you send it. It also keeps a compact picture of those events so you can ask questions without reading them all back. We call the picture the map (the navigation tier) and the complete record the territory (the fidelity tier). This page explains how the two relate. The rest of the Concepts section builds on it.

Two tiers, one history

Both tiers describe the same events. They are built for different jobs.

Navigation tier (the map)Fidelity tier (the territory)
What it holdsCompact summaries that make exploring history fastEvery accepted event, complete: timestamp, service, original body bytes and every attribute
Where it livesEBS volumes attached to your OBSESC node, inside your accountYour S3 bucket, inside your account
FormatOBSESC’s own proprietary formatApache Parquet files with Apache Iceberg table metadata
RoleAn accelerator for the consoleThe source of truth
If lostRebuilt from the fidelity tierNot recoverable from the map. The map holds summaries, not your events
Readable without OBSESCNoYes, with any engine that reads Parquet or Iceberg

The map answers aggregate questions

Questions like “how many errors did checkout log last March?”, “what was p99 latency by endpoint over the last year?” or “which log patterns are new this week?” are answered from the map, quickly, however long the time range. In the console these are the Explore views, What changed?, Seen this before?, What tends to follow? and unusual behaviour.

Counts are exact. Some figures, such as percentiles and distinct counts, are close estimates. See Navigation tier.

The territory answers “show me the events”

When you need the events themselves, OBSESC reads the fidelity tier. This happens when you search, run SQL over raw_events, pull the context around a hit, or drill down from something unusual. OBSESC tells you how much a search will read before it starts. This is deliberate search.

The fidelity tier lives in your bucket as open columnar files. You can query the same data with your own engines. OBSESC doesn’t have to be running. See Fidelity tier.

The rules that tie them together

  • The territory is authoritative. The map is derived from your stored events, never the other way round.
  • The map can be redrawn. In the default deployment, if a navigation volume is lost or wiped, OBSESC rebuilds the map from the data you still retain in your bucket. See Rebuilding the navigation tier.
  • SQL reads your events. SQL over raw_events reads your stored events in S3, not the map.
  • Both tiers stay in your account. The navigation tier is on EBS volumes you own. The fidelity tier is in an S3 bucket you own, encrypted with a KMS key you own.

One thing to watch: map lag

The map is built slightly behind the territory. Under a burst, the map lags instead of slowing ingest down.

Aggregate views in the console read the map. A period the map hasn’t caught up with yet shows no data, not an error. When you need the most recent events, use SQL over raw_events. OBSESC’s built-in alerts tell you when the map is falling behind, so lag isn’t mistaken for silence. See Navigation tier.

What you keep if you leave

If you stop running OBSESC, every event you sent stays in your bucket as Parquet with Iceberg metadata, readable by Athena, Trino, Spark, DuckDB or any other engine that speaks those formats. You lose the map, which only OBSESC reads.

Next steps