Overview

ComplyBeacon is an open-source observability toolkit that collects, normalizes, and enriches compliance evidence by extending the OpenTelemetry (OTel) standard.

It bridges the gap between raw policy scanner output and modern logging pipelines, providing a unified, enriched, and auditable data stream for security and compliance analysis.

WARNING: All components are under initial development and are not ready for production use.

Architecture#

ComplyBeacon is composed of four components that form an evidence enrichment pipeline:

1. Proofwatch#

An instrumentation library that accepts raw compliance evidence from policy scanners (OPA, Gatekeeper, Conforma, Sentinel), normalizes it into OCSF/Gemara format, and emits it as an OpenTelemetry log stream with accompanying metrics.

2. Beacon#

A custom OpenTelemetry Collector distribution that hosts the pipeline. It receives log records via OTLP, webhook, or file input, batches and transforms them, then routes them through enrichment and on to exporters.

3. Truthbeam#

A custom OpenTelemetry Collector processor that enriches log records by calling the Compass service to attach compliance framework mappings, risk scores, and remediation guidance.

4. Compass#

A policy-to-control lookup service that maps evidence attributes to compliance frameworks, baselines, and risk data. Compass is an external service that must be provided separately for TruthBeam enrichment to function.

Data Flow#

Prerequisites#

• Go 1.25+
• Task (taskfile.dev)
• Podman
• Git

## Install Task
brew install go-task/tap/go-task  # macOS
## Or
go install github.com/go-task/task/v3/cmd/task@latest

Quick Start#

1. Build And Deploy#

The demo stack runs Beacon, Loki, and Grafana via compose:

## Build The Collector Image (resolves Deps, Syncs Versions, Builds Container)
task build

## Start All Services
task infra:deploy

## Stop Everything
task infra:undeploy

2. Test The Pipeline#

## Send Sample Compliance Evidence
curl -X POST http://localhost:8088/eventsource/receiver \
  -H "Content-Type: application/json" \
  -d @hack/sampledata/evidence.json

## View Enriched Logs In Grafana At Http://localhost:3000

3. Evidence Storage (s3)#

The demo stack includes rustfs, an S3-compatible object store, so evidence archival works out-of-the-box with no configuration.

After task infra:deploy, the following are available:

Evidence files are written to the complybeacon-evidence bucket and can be browsed in the rustfs console.

Using Real Aws S3 Instead#

To export evidence to a real AWS S3 bucket, override the defaults by exporting environment variables before starting the stack:

Production deployments should use a dedicated collector configuration with disable_ssl: false and s3_force_path_style: false. See Sync_Evidence2Hyperproof for the full AWS integration guide.

Development#

Task Commands#

task                       # List all available tasks
task build                 # Build the collector container image
task test                  # Run tests with coverage
task test-race             # Run tests with race detection
task lint                  # Run golangci-lint
task check                 # Run all quality gates (lint + test)
task deps                  # Tidy, verify, and download dependencies
task workspace             # Set up Go workspace
task clean                 # Remove build artifacts and test output

Code Generation#

task codegen:api-codegen               # Regenerate OpenAPI client (truthbeam)
task codegen:weaver-codegen            # Regenerate attribute constants from model/
task codegen:weaver-docsgen            # Regenerate attribute docs from model/
task codegen:weaver-check              # Validate semantic convention model
task codegen:weaver-semantic-check     # Validate logs against semantic conventions

Version Management#

The project prevents version drift between the Go modules and the beacon-distro collector distribution. Version checks run automatically with task test and block CI if versions diverge.

task version:check-otel-versions       # Check alignment
task version:sync-otel-versions        # Sync beacon-distro with truthbeam
task version:check-go-version          # Verify Go version compatibility
task version:check-go-mod-consistency  # Check OTel dependency consistency

Container Image#

The beacon collector image is built from beacon-distro/Containerfile.collector:

## Build With Default Tag (complybeacon/collector:latest)
task build

## Build With Custom Image Name And Tag
task build IMAGE=ghcr.io/complytime/complybeacon TAG=v1.0.0

## Run Standalone (without Compose)
podman run --rm \
  -v ./hack/demo/demo-config.yaml:/etc/otel-collector.yaml:Z \
  -p 4317:4317 -p 8088:8088 \
  complybeacon/collector:latest \
  --config=/etc/otel-collector.yaml

Grafana Dashboard: To configure Loki as default datasource, see hack/demo/terraform/README.md.

Additional Resources#

• Design & Architecture: docs/DESIGN.md
• Development Guide: docs/DEVELOPMENT.md
• Image Publishing: docs/publish_image/publish_image.md
• OpenTelemetry: opentelemetry.io