What happens next

Once your shipper is delivering, OBSESC makes each event durable, stores it in full, and makes it explorable. This page explains that path and what you should review in your first week.

The life of an event

shipper ──> OBSESC node ──> write-ahead log (EBS) ──> acknowledged
                 │
                 ├──> raw Parquet on S3   (fidelity tier)
                 └──> navigation tier     (EBS)

1. Accepted and made durable

Before an event is written, the node checks it:

• Timestamps. Events stamped more than 7 days in the future are clamped to that limit by default. Old events are accepted, because buffered shippers and backfills legitimately deliver them.
• Size limits. A single event over 8 MiB, or with more than 4,096 attributes, rejects its whole batch. OBSESC never truncates an event silently. See Limits.
• Retries. Delivery is at-least-once. OBSESC recognises many shipper retries of a request it has recently accepted and drops the repeat, but an occasional duplicate is still possible.
• Backpressure. When the node is saturated it answers 503 with Retry-After, rather than buffering without limit. Your shipper retries.

The event is then written to the write-ahead log (WAL) on its own EBS volume. Your shipper gets its success response only after this write. If the node restarts, an acknowledged event isn’t lost.

2. Stored in full (the fidelity tier)

Events are then written as Apache Parquet files to your S3 bucket, under the raw/ prefix, organised by date and hour and encrypted with your KMS key. Table metadata is kept under catalog/ in the same bucket.

This is the complete, original record. It’s in an open format, so you can read it with your own tools, such as Athena, DuckDB or Spark, without going through OBSESC. Within OBSESC, you query it with SQL over the raw_events table, or open it from the console. See Data flow and the SQL reference.

3. Made explorable (the navigation tier)

OBSESC also keeps a compact view of your history on EBS, and highlights unusual behaviour with links to the evidence. That’s what makes exploring months of data fast. Your shipper’s success response doesn’t wait for it.

There’s a short lag. The most recent events may not appear yet in the console’s history views. That’s expected, not an error. Once events are stored in your bucket, you can query them with SQL even before they appear in those views. On a heavily loaded node the lag can grow, and OBSESC’s built-in alerts will tell you when it does. See Navigation tier.

Your first-week checklist

Retention

By default, OBSESC keeps everything and deletes nothing. Time-based retention is opt-in, through the node configuration:

storage:
  retention_window_days: 400

When this is set, the node regularly removes data older than the window from your S3 bucket and from the navigation tier together. Legal holds take priority over retention.

If your raw bucket uses S3 lifecycle transitions, declare them too:

storage:
  lifecycle:
    standard_ia_after_days: 30
    glacier_ir_after_days: 180

Standard-IA has a 30-day minimum storage charge, and Glacier Instant Retrieval has a 90-day one. Deleting an object before its minimum has elapsed still incurs the charge for the remainder. When the retention window would cause this, the node warns you. It can only name the storage class involved if you’ve declared your lifecycle rules. If the stack created the bucket (CreateRawBucket=true), this block is written for you.

Contact us for the supported way to apply configuration changes to a running deployment. See Configuration reference and Compliance.

If you may need to erase specific records, contact us to enable erasure before you need it.

Alert delivery

OBSESC ships alert rules that deliver to your Alertmanager. They cover problems such as a growing history lag, write-ahead log problems, stalled ingest and failed writes to S3. Unless you set AlertmanagerEndpoint on the stack, those alerts aren’t delivered to anyone. Check the stack’s AlertDelivery output:

• DELIVERING to host:port: alerts reach your Alertmanager.
• NOT DELIVERING …: alerts aren’t leaving the node. Update the stack with AlertmanagerEndpoint set.

Route the built-in heartbeat alert, which fires continuously, to a dead-man’s-switch receiver so you’ll know if alert delivery itself breaks. See Monitoring.

Backups

With EnableDataVolumeSnapshots=true (the default), the data EBS volumes are snapshotted daily, and the last 7 snapshots are kept. Your raw events stay in S3 regardless. See Backups and recovery.

Security

If you launched without them, plan to add:

• AuthSecretArn for API and ingest tokens
• TLS certificates for the listeners
• the internal load balancer, with sign-in through your identity provider and a role for each user

See Deployment (AWS Marketplace) and Security model.

Keeping up with the deployment

• Health. GET /ready on port 18080 is the node’s readiness check. It returns 503, with a reason, when the node can’t safely accept or store data. See Monitoring.
• Capacity. Watch CPU, and act on OBSESC’s built-in alerts. To scale, move to a larger c6in instance. Contact us about multi-node deployments. See Scaling.
• Upgrades. New releases ship as new AMIs. Follow Upgrades so no recently accepted events are lost during the change.

Where to go from here

• Learn the model: The map and the territory
• Connect more sources: Supported sources
• Start asking questions: Explore and Search and filtering