Elasticsearch bulk

OBSESC accepts the Elasticsearch _bulk API. Anything that already writes to Elasticsearch can send to OBSESC by changing the host. This page describes exactly what the endpoint accepts and how documents become events.

Endpoint

Setting	Value
Default port	9200 (ingest.es_bulk_port), open to AllowedIngestCidr in the stock stack
Ingest route	POST /_bulk
Client preflight	GET / (reports an Elasticsearch 8.x version) and GET /_license (reports an active basic licence)
Response header	X-elastic-product: Elasticsearch on every response
Body format	NDJSON: an action line, then a document line
Compression	Content-Encoding: gzip accepted
Maximum request	100 MiB on the wire. Gzip bodies are also limited in how far they can decompress
Authentication	Authorization: ApiKey <token> or Authorization: Bearer <token>, checked against security.ingest_tokens

Only /_bulk is served. Index-scoped paths such as /<index>/_bulk and the search, index-management and template APIs aren’t available. Configure your client to skip template and ILM setup.

Quick test

cat > /tmp/bulk.ndjson <<'EOF'
{"index":{"_index":"logs"}}
{"@timestamp":"2026-01-01T00:00:00.000Z","service":"checkout","message":"order placed id=42","host.name":"web-1"}
EOF

curl -sS -X POST "http://obsesc.your-domain.internal:9200/_bulk" \
  -H "Content-Type: application/x-ndjson" \
  -H "Authorization: ApiKey $OBSESC_INGEST_TOKEN" \
  --data-binary @/tmp/bulk.ndjson

Leave out the Authorization header if the node doesn’t enforce security.ingest_tokens.

Supported actions

Action	Behaviour
index, create	The document line becomes one event
update	The doc (or upsert) object becomes one event. A scripted update with neither is acknowledged but not stored
delete	Acknowledged but ignored. A bulk delete doesn’t remove anything from OBSESC

The response follows Elasticsearch’s bulk format and reports each item as created.

How documents are mapped

Document field	OBSESC field
service, else service.name, else the action’s _index	service
message, else log	body. If neither is present, the whole document line is stored as the body
@timestamp, else timestamp, else ts	timestamp
Every other field	Attributes, flattened to dotted keys: {"http":{"status":500}} becomes http.status=500, and arrays use [0], [1]

Timestamps

A timestamp can be an ISO 8601 string with any UTC offset (2026-01-01T10:00:00.123+10:00) or a numeric epoch in seconds, milliseconds, microseconds or nanoseconds. A document with no recognisable timestamp is stamped with the time the node received it.

Choose the service carefully

The service is how OBSESC groups everything downstream. Many clients write to generated index names such as filebeat-8.x or logs-generic-default. If you rely on the index fallback, all your applications collapse into one service. Set a service field on every document instead. The shipper pages show how:

• Filebeat
• Logstash

Dimensions

Standard fields such as host.name, host.hostname, hostname, environment and kubernetes.namespace are read into the built-in host, env and namespace dimensions. The original fields are kept as attributes too.

Authentication

One allowlist, security.ingest_tokens, covers this listener along with OTLP, Vector native and Loki. Both ApiKey and Bearer are accepted, and the value after the scheme must match an entry exactly.

Elastic clients encode their API key before sending it. Filebeat’s and Logstash’s api_key: "id:key" goes on the wire as ApiKey base64(id:key). The node compares that base64 string, so add the base64 form to ingest_tokens.

Basic authentication isn’t supported. This affects clients that can only send a username and password, such as rsyslog’s omelasticsearch and Fluent Bit’s es output. For those, either:

• limit network access to port 9200 and run without ingest tokens for that path, or
• put a shipper that can send an ApiKey or Bearer header in front of them.

Responses and retries

Response	Meaning
200	Batch durable. Items reported as created
400	Malformed NDJSON, an action line without its document line, an unknown action, a bad gzip body, or a batch that broke an ingest limit. Nothing from the batch is stored
413	Request too large, or a gzip body that decompresses too far
503 + Retry-After	Backpressure or draining. Retry the same batch

A batch is accepted or rejected as a whole, so a batch is never partly stored.

Avoiding duplicates with _id

If the action line carries an explicit _id, the node uses it to recognise a retry. A resent document with the same _id is dropped rather than stored twice, provided the retry arrives soon after the original (see Supported sources):

{"create":{"_index":"logs","_id":"7f3c9a2e-request-42"}}
{"service":"checkout","message":"order placed id=42","@timestamp":"2026-01-01T00:00:00Z"}

Documents without _id are at-least-once. Use an _id that’s unique per event, never a hash of the event’s content. A content hash would make two genuinely identical lines look like one event.

Clients that use this endpoint

• Filebeat and Winlogbeat: output.elasticsearch
• Logstash: the elasticsearch output
• Fluent Bit: the es output with Suppress_Type_Name On
• Vector: the elasticsearch sink
• rsyslog: omelasticsearch (see Configuration examples)
• Cribl Stream: the Elasticsearch destination with Bulk API URL http://obsesc.your-domain.internal:9200/_bulk