Data flow

This page follows one event from your log shipper to the point where you can query it. Knowing the path helps you reason about acknowledgements, backpressure, freshness and what a restart can and cannot lose.

The path at a glance

your hosts ──► log shipper ──┬──► your existing platform (unchanged)
                             │
                             └──► OBSESC node
                                   1. receives it on a protocol port
                                   2. checks it (service, redaction, validation)
                                   3. writes it to the WAL volume on EBS  ──► ack to shipper
                                   4. stores it as Parquet in your S3 bucket, with Iceberg metadata
                                   5. makes it available in the console and to SQL

1. Your shipper fans out

OBSESC is added as an extra destination in the shipper you already run. Fluent Bit, Vector, the OpenTelemetry Collector, Filebeat, Logstash and others all support sending to more than one output. Your existing destination keeps receiving everything it did before. See Supported sources.

2. The node receives it

Each protocol has its own port on the node:

Protocol	Port	Open in the stack’s security group
OTLP/gRPC	4317	Yes
OTLP/HTTP (/v1/logs, protobuf only)	4318	Yes
Elasticsearch bulk (/_bulk)	9200	Yes
Splunk HEC	8088	No
Fluent Forward	24224	No
Vector native	9000	No

The stack’s security group allows the first three, plus the console and query port (18080), from AllowedIngestCidr. To use Splunk HEC, Fluent Forward or Vector native, add a rule for that port to the security group. Splunk is supported through HEC only, not the Splunk forwarder protocol. See Network configuration.

Pull and AWS-native sources (Firehose, Kinesis, SQS/S3, Kafka, Loki push, GELF) are off by default and enabled individually. Whichever protocol delivers an event, it is stored and queried the same way.

3. Checks before storage

Every event goes through the same checks before it is stored:

• Service. service is how events are grouped in the console and in queries. Shipper placeholders for “no service” (unknown, default, an empty name, unknown_service:*) are all mapped to one default service. You can also have OBSESC take the service from an attribute you choose, such as a Kubernetes namespace label.
• Canonical dimensions. host, env, namespace and tenant are resolved from each protocol’s own conventions (for example host.name or k8s.namespace.name) and added as attributes when absent. The originals are kept as shipped.
• Redaction. Redaction rules you configure replace a whole attribute value, or regex matches in the body and attribute values. Redaction happens before anything is written, so the original bytes never reach any tier.
• Validation. Timestamps too far in the future, or optionally too old, are moved to the boundary (the default) or cause the batch to be refused. An event that is too large or has too many attributes causes its whole batch to be rejected. Events are never silently truncated. See Limits.
• Multiline (optional). For line-oriented paths with no shipper-side assembler, OBSESC can fold stack-trace continuation lines into their first line. It is off by default. When it is on, a crash can lose a partly assembled group of lines that was already acknowledged. Do multiline assembly in the shipper where you can.
• Duplicates. Events that carry a shipper-supplied ID (the Elasticsearch bulk _id, for example) are deduplicated when a shipper resends them shortly afterwards. Events without an ID are stored at-least-once, so a retried batch is stored again.

4. The write-ahead log and the acknowledgement

Accepted events are appended to a write-ahead log on the node’s WAL volume (/var/lib/obsesc/wal). The shipper is acknowledged after that append. If the node restarts, acknowledged events that had not reached S3 yet are not lost.

When OBSESC can’t accept more, it refuses. It never drops silently:

• 400 for malformed input. Fix the payload, because retrying it will not help.
• 503 (or gRPC RESOURCE_EXHAUSTED / UNAVAILABLE) when the node is at capacity, draining, or too much data is waiting to reach S3. Well-behaved shippers buffer and retry with backoff.

A refused payload has not been accepted. It is stored once, when a retry succeeds.

5. Events are stored in S3

Shortly after acknowledging events, the node stores them as Parquet files in your bucket under raw/<yyyy>/<mm>/<dd>/<hh>/…, keyed by the events’ event time, and updates the Iceberg metadata for external engines. Once that is done, the events are durable in S3 and queryable with SQL. See Fidelity tier.

6. The map catches up

Once events are safely stored, OBSESC adds them to the navigation tier. Ingest never waits for the map. If the map falls behind, it lags, and ingest is not throttled. OBSESC’s built-in alerts tell you when it is falling behind. See Navigation tier.

7. Queries

You work with your data in the console, which the node serves on port 18080. You can put it behind the stack’s optional load balancer (EnableUiLoadBalancer) and turn on EnableUiPerUserRbac, so each user signs in through your identity provider with their own role. See Network configuration.

• Aggregate views (Explore, What changed?, Seen this before?, What tends to follow?, unusual behaviour) read the navigation tier.
• Search, SQL over raw_events, event context and drill-down read your events in S3. See Deliberate search.

Freshness, end to end

Stage	When an event reaches it
Acknowledged to your shipper	After the WAL append
Readable with SQL over raw_events	Once it is stored in S3, shortly after the acknowledgement
Visible in aggregate views	Once the navigation tier has caught up, shortly after it is stored in S3

Multi-node deployments

The stock CloudFormation template deploys a single node. Contact us about multi-node deployments.

Background work

• Retention. Runs regularly (hourly by default) and removes data older than your retention window, measured by event time. See Storage and formats.
• Compaction (optional, off by default). Tidies raw files into fewer, larger ones. See Fidelity tier.