Overview

🤖 LLM WARNING 🤖

This project was written with LLM (AI) assistance.

🤖 LLM WARNING 🤖

ComplyBeacon is an open-source observability toolkit that collects, normalizes, and enriches compliance evidence by extending the OpenTelemetry (OTel) standard.

It bridges the gap between raw policy scanner output and modern logging pipelines, providing a unified, enriched, and auditable data stream for security and compliance analysis.

WARNING: All components are under initial development and are not ready for production use.

Architecture#

ComplyBeacon is composed of four components that form an evidence enrichment pipeline:

1. Proofwatch#

An instrumentation library that accepts raw compliance evidence from policy scanners (OPA, Gatekeeper, Conforma, Sentinel), normalizes it into OCSF/Gemara format, and emits it as an OpenTelemetry log stream with accompanying metrics.

2. Beacon#

A custom OpenTelemetry Collector distribution that hosts the pipeline. It receives log records via OTLP, webhook, or file input, batches and transforms them, then routes them through enrichment and on to exporters.

3. Truthbeam#

A custom OpenTelemetry Collector processor that enriches log records by calling the Compass service to attach compliance framework mappings, risk scores, and remediation guidance.

4. Compass#

A policy-to-control lookup service that maps evidence attributes to compliance frameworks, baselines, and risk data. Compass is an external service that must be provided separately for TruthBeam enrichment to function.

Data Flow#

Prerequisites#

• Go 1.26+
• Task ( taskfile.dev)
• Podman
• Git

## Install Task
brew install go-task/tap/go-task  # macOS
## Or
go install github.com/go-task/task/v3/cmd/task@latest

Quick Start#

1. Build And Deploy#

The demo stack runs Beacon, Loki, and Grafana via compose:

## Build The Collector Image (resolves Deps, Syncs Versions, Builds Container)
task build

## Start All Services
task infra:deploy

## Stop Everything
task infra:undeploy

2. Test The Pipeline#

## Send Sample Compliance Evidence
curl -X POST http://localhost:8088/eventsource/receiver \
  -H "Content-Type: application/json" \
  -d @tests/integration/fixtures/evidence-fail.json

## View Enriched Logs In Grafana At Http://localhost:3000

3. Evidence Storage (s3)#

The demo stack includes rustfs, an S3-compatible object store, so evidence archival works out-of-the-box with no configuration.

After task infra:deploy, the following are available:

Evidence files are written to the complybeacon-evidence bucket and can be browsed in the rustfs console.

Using Real Aws S3 Instead#

To export evidence to a real AWS S3 bucket, override the defaults by exporting environment variables before starting the stack:

Production deployments should use a dedicated collector configuration with disable_ssl: false and s3_force_path_style: false. See Sync_Evidence2Hyperproof for the full AWS integration guide.

Development#

Task Commands#

task                       # List all available tasks
task build                 # Build the collector container image
task test                  # Run tests with coverage
task test-race             # Run tests with race detection
task lint                  # Run golangci-lint
task check                 # Run all quality gates (lint + test)
task test:integration      # Run end-to-end integration tests (Ginkgo)
task deps                  # Tidy, verify, and download dependencies
task workspace             # Set up Go workspace
task clean                 # Remove build artifacts and test output

Code Generation#

task codegen:api-codegen               # Regenerate OpenAPI client (truthbeam)
task codegen:weaver-codegen            # Regenerate attribute constants from model/
task codegen:weaver-docsgen            # Regenerate attribute docs from model/
task codegen:weaver-check              # Validate semantic convention model
task codegen:weaver-semantic-check     # Validate logs against semantic conventions

Version Management#

The project prevents version drift between the Go modules and the beacon-distro collector distribution. Version checks run automatically with task test and block CI if versions diverge.

task version:check-otel-versions       # Check alignment
task version:sync-otel-versions        # Sync beacon-distro with truthbeam
task version:check-go-version          # Verify Go version compatibility
task version:check-go-mod-consistency  # Check OTel dependency consistency

Container Image#

The beacon collector image is built from beacon-distro/Containerfile.collector.

Local builds:

## Build With Default Tag (complybeacon/collector:latest)
task build

## Build With Custom Image Name And Tag
task build IMAGE=ghcr.io/complytime/complybeacon TAG=v1.0.0

## Run Standalone (without Compose)
podman run --rm \
  -v ./configs/collector-enrichment.yaml:/etc/otel-collector.yaml:Z \
  -p 4317:4317 -p 8088:8088 \
  complybeacon/collector:latest \
  --config=/etc/otel-collector.yaml

CI-built images:

Images are automatically published to GHCR:

• Main branch: Merged changes → sha-<commit> (immutable)
• PRs (org members): Open/update PR → dev-pr<number> (mutable) + sha-<commit>
• External PRs: No images built (security)

Verify published images (using skopeo):

## Install Skopeo
brew install skopeo  # macOS
dnf install skopeo   # Fedora/RHEL/CentOS

## Authenticate (required For Private Repos)
## Create A Token At Https://github.com/settings/tokens With 'read:packages' Scope
skopeo login ghcr.io
## Username: Your-github-username
## Password: Paste Your Token (ghp_...)

## List All Tags
skopeo list-tags docker://ghcr.io/complytime/complybeacon-beacon-distro

## Inspect Your Pr Image (replace 123 With Your Pr Number)
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

## Get Just The Digest
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123 --format "{{.Digest}}"

## Pull And Use (also Requires Authentication For Private Images)
podman login ghcr.io  # If not already logged in
podman pull ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

Troubleshooting:

• “authentication required” error: Login with skopeo login ghcr.io (token from github.com/settings/tokens)
• PR didn’t build an image: Check you’re a complytime org member, changed relevant files, and view “Publish Images to GHCR” in Actions tab
• More details: See docs/DEVELOPMENT.md

See docs/publish_image/publish_image.md for the complete publishing pipeline.

Grafana Dashboard: To configure Loki as default datasource, see deploy/terraform/README.md.

Additional Resources#

• Design & Architecture: docs/DESIGN.md
• Development Guide: docs/DEVELOPMENT.md
• Image Publishing: docs/publish_image/publish_image.md
• OpenTelemetry: opentelemetry.io