Development

This guide provides comprehensive instructions for setting up, building, and testing the ComplyBeacon project. It complements the DESIGN.md document by focusing on the practical aspects of development.

ComplyBeacon Development Guide

Prerequisites#

Required Software#

Go 1.26+: The project uses Go 1.26.3
Podman: For containerized development and deployment (Docker is not supported)
Task: For build automation ( installation guide)
Git: For version control
openssl: Cryptography toolkit

Development Environment Setup#

1. Clone The Repository#

git clone https://github.com/complytime/complybeacon.git
cd complybeacon

2. Install Task (if Needed)#

The project uses Task for build automation. Install it if you don’t have it:

## Macos
brew install go-task/tap/go-task

## Linux
sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin

## Or Using Go
go install github.com/go-task/task/v3/cmd/task@latest

## Verify Installation
task --version

3. Initialize Go Workspace#

The project uses Go workspaces to manage multiple modules:

task workspace

This creates a go.work file that includes all project modules:

./proofwatch
./truthbeam

4. Install Dependencies#

Dependencies are managed per module. Install them for all modules:

task deps

This automatically:

Syncs the Go workspace
Runs go mod tidy for each module
Verifies and downloads dependencies

5. Verify Installation#

## Run Tests To Verify Everything Works
task test

## Run All Quality Gates (lint + Test)
task check

Dependency Management#

Regular Dependency Updates#

Update all Go packages to their latest versions:

task dev:deps:update

This command automatically:

Updates all non-OTel Go packages to the latest versions
Excludes OTel packages (they are updated manually - see below)
Runs go mod tidy for all modules
Syncs the Go workspace
Verifies all modules

Opentelemetry Collector Versions#

Why OTel has two version series:

OpenTelemetry Collector publishes two version series in each release:

Stable API (v1.x) — backward compatible core interfaces (component, pdata, consumer)
Experimental API (v0.x) — may break between releases, helpers & new features (componenttest, processorhelper, config/*)

Each release publishes both versions together:

v0.151.0 release (April 2025): v1.57.0 (stable) + v0.151.0 (experimental)
v0.152.0 release (May 2025): v1.58.0 (stable) + v0.152.0 (experimental)

How we handle it:

OTel versions are constrained by contrib package availability — we pin to versions where all contrib packages exist
task dev:deps:update excludes OTel packages — they are updated manually after verifying contrib compatibility
task version:sync propagates pinned OTel versions to all modules, Containerfiles, and CI configs

Current version: v1.58.0 (stable) + v0.152.0 (experimental)

Why the constraint? Contrib packages (used in beacon-distro/manifest.yaml) release 1-2 versions behind the main collector packages. Blindly upgrading to the latest OTel version causes build failures when contrib packages don’t exist yet.

How To Upgrade Opentelemetry Collector#

Important: OTel packages are not upgraded automatically by task dev:deps:update. Follow this process:

Step 1: Check latest contrib release

Visit the contrib releases page and note the latest version (e.g., v0.152.0).

Step 2: Identify required OTel version

Check a contrib package’s go.mod to see what OTel version it requires:

go mod download -json github.com/open-telemetry/opentelemetry-collector-contrib/connector/signaltometricsconnector@v0.152.0 | \
  jq -r '.GoMod' | xargs cat | grep 'go.opentelemetry.io/collector/component'

This will show something like go.opentelemetry.io/collector/component v1.58.0 — that’s your target stable version.

Step 3: Update truthbeam

cd truthbeam
go get go.opentelemetry.io/collector/component@v1.58.0 \
      go.opentelemetry.io/collector/consumer@v1.58.0 \
      go.opentelemetry.io/collector/pdata@v1.58.0 \
      go.opentelemetry.io/collector/processor@v1.58.0 \
      go.opentelemetry.io/collector/component/componenttest@v0.152.0 \
      go.opentelemetry.io/collector/config/confighttp@v0.152.0 \
      go.opentelemetry.io/collector/processor/processorhelper@v0.152.0 \
      go.opentelemetry.io/collector/processor/processortest@v0.152.0
go mod tidy
cd ..

Step 4: Propagate across the project

task version:sync

This syncs the versions to all workspace modules, Containerfiles, and CI configs.

Step 5: Verify

task test
task integration:test

Troubleshooting Version Conflicts#

Error: unknown revision v0.XXX.0

The OTel version doesn’t exist yet. Check the releases page for the latest available version.

Error: dependency conflicts after go get -u

You’ve mixed stable/experimental versions from different releases. Reset to the pinned versions:

cd truthbeam
## Downgrade To Current Pinned Versions
go get go.opentelemetry.io/collector/component@v1.57.0 \
       go.opentelemetry.io/collector/component/componenttest@v0.151.0
go mod tidy
cd ..
task version:sync

Project Structure#

complybeacon/
├── compose.yaml                # Container orchestration configuration
├── Taskfile.yml                # Build automation
├── .taskfiles/                 # Task modules and helper scripts
├── docs/                       # Documentation
│   ├── DESIGN.md              # Architecture and design documentation
│   ├── DEVELOPMENT.md         # This file
│   └── attributes/            # Attribute documentation
├── model/                      # OpenTelemetry semantic conventions
│   ├── attributes.yaml        # Attribute definitions
│   └── entities.yaml          # Entity definitions
├── proofwatch/                 # ProofWatch instrumentation library
│   ├── attributes.go          # Attribute definitions
│   ├── evidence.go            # Evidence types
│   └── proofwatch.go          # Main library
├── truthbeam/                  # TruthBeam processor module
│   ├── internal/              # Internal packages
│   ├── config.go              # Configuration
│   └── processor.go           # Main processor logic
├── beacon-distro/              # OpenTelemetry Collector distribution
│   ├── config.yaml            # Collector configuration
│   └── Containerfile.collector # Container definition
├── configs/                    # Deployment configs (collector, Loki)
│   ├── collector-base.yaml    # Base layer: OCSF transform + Loki
│   ├── collector-storage.yaml # Storage layer: adds S3 export
│   ├── collector-enrichment.yaml # Enrichment layer: adds TruthBeam
│   └── loki.yaml              # Loki configuration
├── certs/                      # TLS certificate generation
├── deploy/                     # Deployment infrastructure (Terraform)
├── tests/                      # Test infrastructure
│   └── integration/           # E2E Ginkgo tests, mock Compass, fixtures
└── bin/                        # Built binaries (created by task infra:deploy)

Testing#

Running Tests#

## Run All Tests (includes Version Checks And Coverage)
task test

## Run Tests With Race Detection
task test-race

## Generate Coverage Reports
task dev:coverage-report

## Run Tests For Specific Module
cd proofwatch && go test -v ./...
cd truthbeam && go test -v ./...

Integration Testing#

The project includes automated integration tests using Ginkgo that validate the evidence pipeline at three deployment layers:

Layer	Profile	What it tests
Base	(default)	OCSF transform + Loki export
Storage	`storage`	S3 evidence export + partitioning
Enrichment	`enrichment`	TruthBeam enrichment via mock Compass

Prerequisites:

Podman and podman-compose
Go 1.26+ (Ginkgo CLI is managed via tool directive in root go.mod)

Run all layers:

task integration:test

Run a single layer:

task integration:test-profile PROFILE=base
task integration:test-profile PROFILE=storage
task integration:test-profile PROFILE=enrichment

Each run builds the collector image, starts the appropriate services, runs the matching Ginkgo test suite (filtered by label), and tears down. Certificates are generated automatically if missing. Test output is written to .test-output/integration/.

For details on test cases, fixtures, and mock Compass configuration, see tests/integration/README.md.

Component Development#

1. Proofwatch Development#

ProofWatch is an instrumentation library for emitting compliance evidence.

Key Files:

proofwatch/proofwatch.go - Main library interface
proofwatch/evidence.go - Evidence type definition
proofwatch/attributes.go - OpenTelemetry attributes

Development Workflow:

cd proofwatch

## Run Tests
go test -v ./...

## Run Linting (from Root)
cd ..
task lint

## Format Code
go fmt ./...

2. Compass Development#

Compass is an external enrichment service that TruthBeam connects to for compliance lookups. It must be provided separately and is not included in the demo stack.

3. Truthbeam Development#

TruthBeam is an OpenTelemetry Collector processor for enriching logs.

Key Files:

truthbeam/processor.go - Main processor logic
truthbeam/config.go - Configuration structures
truthbeam/factory.go - Processor factory

Development Workflow:

cd truthbeam

## Run Tests
go test -v ./...

## Test With Collector (requires Beacon-distro)
cd ../beacon-distro
## Modify Config To Use Local Truthbeam
## Run Collector With Local Processor

Local development config

If you want locally test the TruthBeam, remember to change the manifest.yaml

Add replace directive at the end of manifest.yaml, to make sure collector use your truthbeam code. Default collector will use - gomod: github.com/complytime/complybeacon/truthbeam main

For example:

replaces:
  - github.com/complytime/complybeacon/truthbeam => github.com/AlexXuan233/complybeacon/truthbeam 52e4a76ea0f72a7049e73e7a5d67d988116a3892

replaces:
  - github.com/complytime/complybeacon/truthbeam => github.com/AlexXuan233/complybeacon/truthbeam main

4. Beacon Distro Development#

The Beacon distribution is a custom OpenTelemetry Collector.

Key Files:

beacon-distro/config.yaml - Collector configuration
beacon-distro/Containerfile.collector - Container definition
beacon-distro/manifest.yaml - Collector builder configuration

Development Workflow:

Local builds (quick iteration):

## Build The Collector Image Locally
podman build -f beacon-distro/Containerfile.collector -t complybeacon-collector beacon-distro/

## Or Force Rebuild Without Cache
podman build --no-cache -f beacon-distro/Containerfile.collector -t complybeacon-collector beacon-distro/

## Run Locally For Quick Testing
podman run --rm complybeacon-collector --config /etc/otelcol-beacon/config.yaml

## Full Stack Deployment For Integration Testing
task infra:deploy

CI builds (automated image publishing):

When you modify Containerfiles or source code and open a PR, the CI automatically builds and publishes dev images (if you’re an org member):

## 1. Make Changes To Beacon-distro, Proofwatch, Or Truthbeam
vim beacon-distro/Containerfile.collector

## 2. Commit And Push To Your Branch
git add .
git commit -s -m "feat(beacon-distro): update base image to UBI10"
git push origin your-branch

## 3. Open A Pr To Main
## The Workflow Will Automatically:
## - Verify You're An Org Member
## - Build The Image
## - Scan For Vulnerabilities
## - Sign The Image
## - Run Integration Tests
## - Publish To Ghcr.io/complytime/complybeacon-beacon-distro:dev-pr<number>

## 4. Verify Your Image Was Published
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

## 5. Use The Dev Image In Testing
podman pull ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123
## Or In Compose.yaml:
## Image: Ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

When images are built:

✅ Push to main branch (production, tagged sha-<commit>)
✅ PRs from org members (dev, tagged dev-pr<number> + sha-<commit>)
❌ PRs from external contributors (no image built for security)

See docs/publish_image/publish_image.md for complete details on the image publishing pipeline.

Debugging And Troubleshooting#

Debugging Tools#

## View All Container Logs
podman-compose -f compose.yaml logs -f

## View Specific Service Logs
podman-compose -f compose.yaml ps            # List running services
podman-compose -f compose.yaml logs -f collector

## Check Container Status
podman images | grep complybeacon            # List built images
podman inspect complybeacon-collector        # Inspect image details

Verifying Published Images#

When you open a PR or merge to main, the CI pipeline automatically builds and publishes container images to GitHub Container Registry (GHCR). Use skopeo to verify your images without pulling them.

Install Skopeo#

## Macos
brew install skopeo

## Fedora/rhel/centos
dnf install skopeo

## Ubuntu/debian
apt-get install skopeo

Quick Checks#

## List All Available Tags
skopeo list-tags docker://ghcr.io/complytime/complybeacon-beacon-distro

## Check If Your Pr Image Exists (replace 123 With Your Pr Number)
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

## Check If A Main Branch Image Exists (replace Abc123 With Commit Sha)
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:sha-abc123

## Get Just The Digest
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123 \
  --format "{{.Digest}}"

## Get Image Creation Timestamp
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123 \
  --format "{{.Created}}"

Authentication#

If the repository is private, authenticate first:

## Create A Github Personal Access Token With 'read:packages' Scope At:
## Https://github.com/settings/tokens

## Then Login
skopeo login ghcr.io
## Username: Your-github-username
## Password: Paste Your Token (ghp_...)

## Or Use Environment Variable
echo $GITHUB_TOKEN | skopeo login ghcr.io -u your-github-username --password-stdin

Image Tagging Strategy#

Build Type	Trigger	Tags	Notes
Production	Merge to `main`	`sha-<commit>`	Immutable, builds after CI passes
Dev	PR from org member	`dev-pr<number>`, `sha-<commit>`	`dev-pr` is mutable (updates on push)
External PR	PR from non-member	None	No images built (security)

Common Scenarios#

Verify your PR image was published:

## Find Your Pr Number (visible In Pr Title, E.g., #123)
## Check The Actions Tab For The "publish Images To Ghcr" Workflow

## List All Tags To Confirm
skopeo list-tags docker://ghcr.io/complytime/complybeacon-beacon-distro | grep "dev-pr123"

## Inspect The Image
skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

Use dev image in local testing:

## Pull The Image
podman pull ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

## Or Reference It Directly In Compose.yaml
## Edit Compose.yaml:
## Services:
## Collector:
## Image: Ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123

Compare dev and production images:

## Both Should Have The Same Digest If Built From The Same Commit
DEV_DIGEST=$(skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:dev-pr123 --format "{{.Digest}}")
SHA_DIGEST=$(skopeo inspect docker://ghcr.io/complytime/complybeacon-beacon-distro:sha-abc123 --format "{{.Digest}}")

echo "Dev digest:  $DEV_DIGEST"
echo "SHA digest:  $SHA_DIGEST"

if [ "$DEV_DIGEST" = "$SHA_DIGEST" ]; then
  echo "✅ Images are identical"
else
  echo "❌ Images differ (expected if commits are different)"
fi

Troubleshooting image builds:

If your PR doesn’t produce an image:

Check org membership: Only complytime org members’ PRs build images

# Verify you're listed as a member
curl -s https://api.github.com/orgs/complytime/members | jq -r '.[].login' | grep your-username

Check if files changed: Image builds only trigger when:
- Any Containerfile* changes
- Source code in beacon-distro/, proofwatch/, truthbeam/ changes
- The workflow file (.github/workflows/ci_publish_ghcr.yml) changes
Check workflow run: Go to Actions → Publish Images to GHCR and check for errors
Check CI status: The workflow waits for CI to complete on PRs to main

For complete details on the image publishing pipeline, see docs/publish_image/publish_image.md. For quick skopeo examples, see the Container Image section in the README.

Code Generation#

The project uses several code generation tools:

1. Opentelemetry Semantic Conventions#

Generate documentation and Go code from semantic convention models:

## Generate Documentation
task codegen:weaver-docsgen

## Generate Go Code
task codegen:weaver-codegen

## Validate Models
task codegen:weaver-check

## Validate Logs Against Semantic Conventions
task codegen:weaver-semantic-check

2. Manual Code Generation#

If you modify the semantic conventions:

## Update Semantic Conventions
vim model/attributes.yaml
vim model/entities.yaml

## Regenerate All Code (api + Weaver)
task codegen:api-codegen
task codegen:weaver-codegen

Deployment And Demo#

Local Development Demo#

The demo environment orchestrates multiple containers (Grafana, Loki, Beacon Collector, Compass).

Generate self-signed certificate

Since compass and truthbeam enable TLS by default, first generate self-signed certificates for testing/development:

task infra:generate-self-signed-cert

Start the full stack:

## Interactive Mode (shows Logs In Terminal)
task deploy

## Or Background/detached Mode
podman-compose -f compose.yaml up -d

This automatically:

Syncs OTel versions from truthbeam to beacon-distro
Builds the beacon collector image
Starts all services (Grafana, Loki, Collector)

Test the pipeline:

curl -X POST http://localhost:8088/eventsource/receiver \
  -H "Content-Type: application/json" \
  -d @tests/integration/fixtures/evidence-fail.json

View results:

Grafana: http://localhost:3000
View logs: podman-compose -f compose.yaml logs -f

Stop the stack:

task infra:undeploy

Additional Resources#

For questions or support, please open an issue in the GitHub repository.

Edit this page on GitHub

Design

Policy

Docs

ComplyTime

Title here

Development

Prerequisites#

Required Software#

Development Environment Setup#

1. Clone The Repository#

2. Install Task (if Needed)#

3. Initialize Go Workspace#

4. Install Dependencies#

5. Verify Installation#

Dependency Management#

Regular Dependency Updates#

Opentelemetry Collector Versions#

How To Upgrade Opentelemetry Collector#

Troubleshooting Version Conflicts#

Project Structure#

Testing#

Running Tests#

Integration Testing#

Component Development#

1. Proofwatch Development#

2. Compass Development#

3. Truthbeam Development#

4. Beacon Distro Development#

Debugging And Troubleshooting#

Debugging Tools#

Verifying Published Images#

Install Skopeo#

Quick Checks#

Authentication#

Image Tagging Strategy#

Common Scenarios#

Code Generation#

1. Opentelemetry Semantic Conventions#

2. Manual Code Generation#

Deployment And Demo#

Local Development Demo#

Additional Resources#