OBSESC
Get in touch
Docs: Overview
Documentation / Get started

Overview

What OBSESC is, how it fits alongside your existing observability stack, and what it deploys in your AWS account.

OBSESC is a log retention and analysis system that runs inside your own AWS account. You point your existing log shippers at it as well as your current observability platform, and it keeps a complete, queryable copy of everything for as long as you choose.

A parallel destination, not a replacement

OBSESC doesn’t ask you to migrate. Your shippers (Fluent Bit, Vector, the OpenTelemetry Collector, Filebeat, Logstash and others) already support sending the same stream to more than one place. You add OBSESC as a second output, and nothing about your current pipeline changes.

                        ┌──> your current platform (short retention, live ops)
your log shipper ───────┤
                        └──> OBSESC (complete history, in your AWS account)

This means you can:

  • run OBSESC in parallel for as long as you need to trust it
  • keep your current platform for live alerting and on-call work
  • decide later, with evidence, whether to shorten retention elsewhere
  • stop at any time, because removing an output stanza undoes the change

OBSESC isn’t a log shipper, an APM or tracing product, or a hosted service. It doesn’t collect from hosts directly. It accepts what your shippers already send.

Where your data lives

Everything OBSESC stores stays in resources you own:

WhatWhereFormat
Complete events (the fidelity tier)Your S3 bucket, under your KMS keyOpen Apache Parquet files, readable by Athena, DuckDB, Spark, Trino and similar tools
The data that makes exploring history fast (the navigation tier)Encrypted EBS volumes attached to the OBSESC nodeInternal to OBSESC
Recently accepted events not yet written to S3A dedicated encrypted EBS volume on the node (the write-ahead log, or WAL)Internal to OBSESC

We have no role in your account and no access to your logs, queries or results. If you stop using OBSESC, every raw event is still in your bucket in an open format.

For more, see The map and the territory and Storage and formats.

What gets deployed

OBSESC ships as an AWS Marketplace AMI with a CloudFormation template. A default deployment creates:

  • One EC2 instance from the c6in family (network-optimised, x86_64), running ingest, query and the console on one host.
  • Three encrypted EBS gp3 volumes for the node’s data.
  • A security group that admits shipper and console traffic from a CIDR range you choose.
  • An IAM role for the instance. It grants access to OBSESC’s prefixes in your raw bucket and to your KMS key, read access to the optional secrets and parameters you name, Session Manager access, AWS Marketplace metering, and permission to manage EC2 instances and volumes tagged to this stack.
  • A daily EBS snapshot policy for the data volumes (on by default).

The S3 bucket is either one you already own (the default) or one the stack creates with a Standard → Standard-IA → Glacier Instant Retrieval lifecycle. You supply the VPC, the subnet and the customer-managed KMS key.

Optional modules add an internal Application Load Balancer for the console, a private DNS name, TLS, and single sign-on through your identity provider. See Deployment (AWS Marketplace).

The stock template deploys a single node. Contact us about multi-node deployments.

How data gets in

The node speaks the protocols your shippers already use. The ones the default security group opens are:

ProtocolPortTypical shippers
OTLP/gRPC4317OpenTelemetry Collector, OTel SDKs
OTLP/HTTP (protobuf only)4318OpenTelemetry Collector
Elasticsearch _bulk9200Filebeat, Logstash, rsyslog (omelasticsearch), Vector’s elasticsearch sink

The node also listens for Fluent Forward (24224), Splunk HTTP Event Collector (HEC, 8088) and Vector’s native protocol (9000). To use those, you add a rule to the security group yourself. See Supported sources for the full list.

OTLP traces are accepted too. Each span is stored as an ordinary event, unsampled.

How you use it

Once events are flowing, the console lets you:

  • Explore any time range and see how each service behaved.
  • Spot unusual behaviour and open the raw events behind it.
  • Search across services and drill down to the exact events you need.
  • Ask What changed?, Seen this before? and What tends to follow? while you investigate.

You can also query the raw events yourself with SQL, through OBSESC or with your own tools over the Parquet files in your bucket.

The console is served by the node on port 18080, the same port as the query API. You can put it behind an internal load balancer with sign-in through your identity provider.

Next steps