OBSESC
Get in touch
Docs: Scaling
Documentation / Operations

Scaling

Scale an OBSESC deployment with a larger instance and bigger volumes, and find out about multi-node deployments.

The standard OBSESC stack runs a single node. You scale it with a bigger instance and bigger volumes. For a deployment that spreads the work across several nodes, contact us about multi-node deployments.

When to scale

OBSESC’s built-in alerts will tell you when:

  • the node has been applying backpressure for a sustained period because it can’t keep up with ingest
  • the navigation tier is falling behind ingest for hours. Treat this seriously: data that passes your retention period before the navigation tier catches up never appears in the console

Also keep an eye on volume usage on the summary volume (/var/lib/obsesc/summary).

Backpressure because data isn’t reaching S3 is usually not a capacity problem. It’s normally a permissions or bucket issue (see Troubleshooting). Fix the cause before you add capacity.

Instance type

The stack’s InstanceType parameter accepts the network-optimised c6in family, from c6in.2xlarge (the default) to c6in.24xlarge. To resize:

  1. Stop your producers, and give the node a few minutes to commit what it has accepted to S3.
  2. Snapshot the data volumes (see Backups and recovery).
  3. Update the stack with the new InstanceType, keeping every other parameter at its current value. The same pattern is shown in Upgrades.
  4. Check /ready returns 200, then re-enable producers.

If the navigation tier still falls behind after resizing, contact us.

Volume sizes

The node has three encrypted gp3 data volumes, set by stack parameters:

ParameterMountDefaultHolds
WalVolumeSizeGiB/var/lib/obsesc/wal100 GiBWrite-ahead log: acknowledged events not yet committed to S3
SummaryVolumeSizeGiB/var/lib/obsesc/summary500 GiBNavigation tier: grows with your retention period
AnomalyVolumeSizeGiB/var/lib/obsesc/anomaly100 GiBAnomaly records

The volumes are XFS. To grow one, raise the parameter in a stack update, or modify the volume directly in EC2. Then extend the filesystem on the node:

sudo xfs_growfs /var/lib/obsesc/summary

Your raw events live in your S3 bucket and need no sizing.

Multi-node deployments

The standard stack doesn’t run a cluster. Contact us about multi-node deployments, or if a single service outgrows one node.