Navigation tier
What the navigation tier does for you, where it lives, how to keep it fresh and how to restore it.
The navigation tier is OBSESC’s map of your logs. OBSESC keeps compact summaries on EBS that make exploring history fast. The original events stay in S3 in open formats. See The map and the territory for how it relates to the fidelity tier.
What the map lets you do
In the console, the map powers:
- Explore. Count and break down events by service, attribute and time, over any range, and read percentiles such as p50 and p99 for numeric attributes.
- Log patterns. OBSESC groups similar messages into patterns, so you can see which are new, growing or gone.
- What changed? Compare two periods. See What changed.
- Seen this before? Find past periods that looked like the one you’re investigating. See Seen this before?
- What tends to follow? See what happened next during an incident. See What tends to follow.
- Unusual behaviour. OBSESC highlights unusual behaviour and links it to the events behind it.
OBSESC also offers optional custody verification, which lets you verify the records OBSESC keeps in the navigation tier. It does not cover your raw events in S3. It is off by default. Turn it on by setting the stack’s EvaluationFeatures parameter to Enabled. On an existing stack, contact us before you change it. See Audit and verification.
Accuracy
Counts are exact. Percentiles and distinct counts are close estimates.
Freshness and lag
Ingest writes to S3 at full speed and the map catches up shortly afterwards. Under a burst the map lags instead of slowing ingest down.
Aggregate views in the console read the map. If the map hasn’t caught up with a period yet, those views show no data for it. They do not show an error.
OBSESC’s built-in alerts tell you when the map is falling behind, and deliver to your Alertmanager. See Monitoring. Two things matter when the map is behind:
- You can still read the events with SQL over
raw_events. See Deliberate search. - Retention deletes raw data by age whether or not the map has caught up with it. Data deleted before the map catches up never appears in the map, so act on lag alerts well inside your retention window.
Where it lives
The CloudFormation stack attaches three encrypted gp3 volumes to the node:
| Mount | Contents | Parameter | Default size |
|---|---|---|---|
/var/lib/obsesc/wal | Write-ahead log (not part of the navigation tier) | WalVolumeSizeGiB | 100 GiB |
/var/lib/obsesc/summary | The navigation tier | SummaryVolumeSizeGiB | 500 GiB |
/var/lib/obsesc/anomaly | Records of unusual behaviour | AnomalyVolumeSizeGiB | 100 GiB |
The files on these volumes are in a proprietary format that only OBSESC reads. Do not edit, move or delete them by hand.
By default (EnableDataVolumeSnapshots=true) the stack snapshots the data volumes daily and keeps seven snapshots.
Retention
Your retention settings apply to the navigation tier as well as your raw events, measured by event time. Per-service retention policies apply to the navigation tier too. See Storage and formats.
Configuration
The CloudFormation template configures the navigation tier for you. There is nothing you need to set. Contact us if you think you need to change it.
Rebuilding
The fidelity tier is the source of truth, so the navigation tier can be restored in two ways:
- From a snapshot. Restore the newest snapshot of the summary volume. OBSESC then catches up on everything stored since the snapshot.
- From scratch. On an empty summary volume, OBSESC rebuilds the map from the raw data you still retain.
Rebuilding reads your own S3 data from inside your account. See Rebuilding the navigation tier and Backups and recovery for procedures.