Filebeat

Filebeat has no generic HTTP output, but OBSESC speaks the Elasticsearch bulk protocol. You use Filebeat’s standard output.elasticsearch and point it at the node. The same approach works for Winlogbeat and the other Beats.

Endpoint

Setting	Value
Protocol	Elasticsearch _bulk (NDJSON over HTTP)
Default port	9200 (ingest.es_bulk_port)
Compression	gzip accepted (Filebeat’s default)
Authentication	Authorization: ApiKey <token> or Bearer <token>, checked against security.ingest_tokens

The node answers the version and licence checks Filebeat makes on startup (GET / and GET /_license) as an Elasticsearch 8.x cluster with a basic licence. Filebeat then ships normally. Port 9200 is open to AllowedIngestCidr in the stock CloudFormation stack.

Configuration

filebeat.inputs:
  - type: filestream
    id: app-logs
    paths: [/var/log/app/*.log]
    # Assemble multiline events here. Without this, a Java stack trace
    # arrives as one event per line.
    parsers:
      - multiline:
          type: pattern
          pattern: '^\d{4}-\d{2}-\d{2}'
          negate: true
          match: after
    fields:
      service: checkout
    fields_under_root: true

output.elasticsearch:
  hosts: ["http://obsesc.your-domain.internal:9200"]
  allow_older_versions: true
  # When the node enforces security.ingest_tokens:
  # api_key: "id:api_key"

# OBSESC doesn't use Elasticsearch index templates or ILM.
setup.template.enabled: false
setup.ilm.enabled: false

• allow_older_versions: true lets a newer Filebeat connect to the node, which reports an 8.x version.
• setup.template.enabled and setup.ilm.enabled stop Filebeat from trying to install templates and lifecycle policies. The node doesn’t serve those APIs.

Set a service field

OBSESC reads the service from the document’s service field, then service.name, and finally falls back to the index name on the bulk action line. By default Filebeat writes to a generated index such as filebeat-<version>, so without a service field every event would land in one service with that name.

Set service on each input, either with fields and fields_under_root: true as above, or with a processor:

processors:
  - add_fields:
      target: ""
      fields:
        service: checkout

Running alongside your existing output

Filebeat allows only one output per instance. To dual-write, choose one of these:

• Run a second Filebeat with this configuration, reading the same files. Give it its own registry path (path.data) so the two instances track their positions separately.
• Send Filebeat to a pipeline that fans out, such as Logstash, Kafka or Cribl, and add OBSESC there. See Logstash and Configuration examples.

Don’t replace your existing output with OBSESC unless you mean to stop sending to it.

Authentication

When security.ingest_tokens is set, uncomment api_key. Filebeat sends Authorization: ApiKey base64(id:api_key). The node compares that base64 string with its allowlist, so the value you add to security.ingest_tokens must be the base64 encoding of id:api_key, not the raw key:

printf '%s' 'id:api_key' | base64

Basic authentication (username / password) isn’t accepted.

How documents are mapped

Document field	OBSESC field
service, else service.name, else the index name	service
message, else log	body. If neither is present, the whole document is stored as the body
@timestamp, else timestamp, else ts	timestamp: ISO 8601 or epoch seconds, milliseconds, microseconds or nanoseconds
Every other field	Attributes, flattened to dotted keys (for example host.hostname, log.file.path)

Standard ECS fields such as host.name, host.hostname and kubernetes.namespace are read into the built-in host and namespace dimensions.

Delivery and retries

• The node replies with an Elasticsearch-compatible bulk response only after the batch is durable. Every item is reported as created.
• Under backpressure, the node answers 503 with Retry-After, and Filebeat retries with backoff.
• Filebeat doesn’t set a document _id, so delivery is at-least-once. A retried batch can be stored twice. See Elasticsearch bulk to avoid duplicates on retry.

Winlogbeat

Winlogbeat uses the same output. Copy the channel into service so that Application, System and Security become separate services:

winlogbeat.event_logs:
  - name: Application
  - name: System
  - name: Security

processors:
  - copy_fields:
      fields:
        - { from: winlog.channel, to: service }
      fail_on_error: false
      ignore_missing: true

output.elasticsearch:
  hosts: ["http://obsesc.your-domain.internal:9200"]
  index: "winlog"
  # api_key: "id:api_key"

setup.template.enabled: false
setup.ilm.enabled: false