Supported sources

OBSESC receives logs over the protocols your shippers already speak, so you add it as a second destination next to your current tool and change nothing else in your pipeline. This page lists every source the node accepts and how each one authenticates and acknowledges data.

How OBSESC fits into your pipeline

OBSESC is designed for dual-write. You keep your existing destination (Splunk, Datadog, Elastic, Loki and so on) and add an OBSESC output alongside it. Each shipper page in this section shows the extra output block to add, with your existing output left in place.

The node treats every record it receives as one event. Each event has four core parts:

• service: the name everything downstream is grouped by. Each protocol has its own rule for where this comes from (see the protocol pages).
• body: the log line, as the shipper sent it.
• timestamp: taken from the record when present, otherwise the time the node received it.
• attributes: every other field, flattened to dotted keys such as kubernetes.pod_name.

Push protocols (on by default)

These listeners start with every node.

Protocol	Wire format	Default port	Endpoint	Authentication
OTLP/HTTP	Protobuf over HTTP	4318	POST /v1/logs, POST /v1/traces	Authorization: Bearer <token>
OTLP/gRPC	gRPC	4317	LogsService/Export, TraceService/Export	authorization: Bearer <token> metadata
Elasticsearch bulk	NDJSON over HTTP	9200	POST /_bulk	Authorization: ApiKey <token> or Bearer <token>
Splunk HEC	JSON or raw text over HTTP	8088	POST /services/collector, /services/collector/event, /services/collector/raw	Authorization: Splunk <token>
Fluent Forward	msgpack over TCP	24224	Native Forward protocol (not HTTP)	Shared-key handshake
Vector native	gRPC	9000	vector.Vector/PushEvents	authorization: Bearer <token> metadata

The Data page in the console lists every listener configured on the node, with its port. Check it there instead of assuming the defaults.

Which protocol should I use?

• OpenTelemetry Collector or OTel SDKs: use OTLP. It’s the only protocol that also accepts traces.
• Fluent Bit or Fluentd: use Fluent Forward.
• Vector: use the native Vector sink. If you need token authentication, use Vector’s elasticsearch sink instead.
• Filebeat, Winlogbeat, Logstash, or anything else with an Elasticsearch output: use Elasticsearch bulk.
• Anything with a Splunk HEC output: use Splunk HEC. OBSESC accepts HEC only, not the Splunk forwarder-to-indexer protocol.

Optional listeners and AWS sources (off by default)

Each of these is enabled by its own ingest.<name> section in the node configuration. Listeners go through the same ingest checks as the push protocols, and all of them except GELF acknowledge only once data is durably stored. The consumers (Kinesis, SQS and Kafka) only move their read position forward once the data they’ve read is durably stored.

Source	Kind	Default port	Config section	Authentication
Amazon Data Firehose HTTP endpoint (for example, CloudWatch Logs subscription to Firehose to OBSESC)	Listener	8555	ingest.firehose	X-Amz-Firehose-Access-Key must match ingest.firehose.access_key (fails closed)
Loki push API (Promtail, Grafana Alloy, Docker loki driver, Fluent Bit or Vector loki outputs)	Listener	3100	ingest.loki	Authorization: Bearer <token>
GELF over UDP and TCP (Docker gelf driver, log4j2 and logback appenders, NXLog)	Listener	12201	ingest.gelf	None. GELF has no authentication, so restrict network access
Kinesis Data Streams	Consumer	n/a	ingest.kinesis	Node instance role
S3 event notifications through SQS	Consumer	n/a	ingest.sqs	Node instance role
Kafka topic (Amazon MSK, Confluent, self-hosted)	Consumer	n/a	ingest.kafka	TLS plus SASL PLAIN or SCRAM

Setup for each one is in Configuration examples.

Sources that go through a shipper

The node has no listener for these. A shipper on the host reads them and forwards them over one of the protocols above:

• Syslog and rsyslog: there is no native syslog listener. rsyslog sends through its Elasticsearch output module to /_bulk.
• journald: use Vector’s journald source with the native Vector sink, Fluent Bit’s systemd input, or Promtail/Alloy with the Loki listener.
• Windows Event Log: use Vector’s windows_event_log source, or Winlogbeat with the Elasticsearch bulk listener.
• Cribl Stream: use Cribl’s Elasticsearch or Splunk HEC destination.

Authentication at a glance

Tokens are set in the security section of the node configuration, or in the Secrets Manager secret the node reads at startup:

Setting	Covers	Behaviour when empty
security.ingest_tokens	OTLP/HTTP, OTLP/gRPC, Elasticsearch bulk, Vector native, Loki	Ingest is unauthenticated and the node logs a startup warning
security.hec_tokens	Splunk HEC	Fails closed: every HEC request is rejected
security.fluent_shared_key	Fluent Forward	Handshake disabled; connections go straight to event frames

Token matching is exact. See Configuration examples for both ways to supply them.

Delivery guarantees

Every listener that acknowledges (all of them except GELF) does so only after the batch is durably stored. When the node is busy it answers 503 with Retry-After (gRPC: RESOURCE_EXHAUSTED). Well-behaved shippers retry, and nothing from a rejected batch is stored. A malformed or over-limit batch is rejected whole rather than partly stored.

Retries could create duplicates, so the node removes a retried event when it can identify it reliably:

Source	Duplicates removed when
Elasticsearch bulk	The action line carries an explicit _id
Vector native	The event carries Vector’s source_event_id (the vector sink sets this)
OTLP logs	The record carries its own timestamp
OTLP traces	The span carries valid trace and span IDs
Splunk HEC (JSON endpoints)	The event carries an explicit time
Fluent Forward	The shipper requests acknowledgements (Fluent Bit Require_ack_response On)
HEC raw, ES without _id, Loki, Firehose, Kinesis, SQS, Kafka	Never. These are at-least-once, so a retried batch can be stored twice
GELF	Never. GELF has no acknowledgement, so delivery is at-most-once: a dropped UDP datagram, or TCP frames not yet durable when the node stops, are lost

Duplicate removal applies to retries that arrive soon after the original. A retry that arrives much later, or after the node restarts, may be stored twice. OBSESC never removes an event based on its content alone: two identical log lines are always both kept.

Before you connect a shipper

1. Open the port. The stock CloudFormation stack’s security group opens only three ingest ports, 4317, 4318 and 9200, from AllowedIngestCidr. It also opens 18080, the console and query port, from the same range. To use HEC (8088), Fluent Forward (24224), Vector (9000) or any optional listener, add an inbound rule for that port. See Network configuration.
2. Decide on TLS. Listeners use plain HTTP/TCP unless TLS is configured, either through the stack’s TLS certificate parameters or security.tls on the node. When it does, they serve TLS only on the same ports. See Encryption.
3. Assemble multiline events in the shipper. A Java stack trace sent as 20 lines becomes 20 events. Every example in this section turns on the shipper’s multiline support. See Configuration examples.
4. Set a service name. Events with no service all land in one bucket, unknown by default (ingest.default_service).