Development

This guide provides comprehensive instructions for setting up, building, and testing the ComplyBeacon project. It complements the DESIGN.md document by focusing on the practical aspects of development.

• ComplyBeacon Development Guide
  • Prerequisites
    • Required Software
  • Development Environment Setup
    • 1. Clone the Repository
    • 2. Install podman-compose (if needed)
    • 3. Initialize Go Workspace
    • 4. Install Dependencies
    • 5. Verify Installation
  • Project Structure
  • Testing
    • Running Tests
    • Integration Testing
  • Component Development
    • 1. ProofWatch Development
    • 2. Compass Development
    • 3. TruthBeam Development
    • 4. Beacon Distro Development
  • Debugging and Troubleshooting
    • Debugging Tools
  • Code Generation
    • 1. OpenTelemetry Semantic Conventions
    • 2. Manual Code Generation
  • Deployment and Demo
    • Local Development Demo
  • Additional Resources

Prerequisites#

Required Software#

• Go 1.24+: The project uses Go 1.24.0 with toolchain 1.24.5
• Podman: For containerized development and deployment
• podman-compose: For orchestrating multi-container development environments
• Make: For build automation
• Git: For version control
• openssl Cryptography toolkit

Development Environment Setup#

1. Clone The Repository#

git clone https://github.com/complytime/complybeacon.git
cd complybeacon

2. Install Podman-compose (if Needed)#

The project uses podman-compose for container orchestration. Install it if you don’t have it:

## Install Podman-compose
pip install podman-compose

## Alternatively For Fedora:
dnf install podman-compose

## Verify Installation
podman-compose --version

3. Initialize Go Workspace#

The project uses Go workspaces to manage multiple modules:

make workspace

This creates a go.work file that includes all project modules:

• ./proofwatch
• ./truthbeam

4. Install Dependencies#

Dependencies are managed per module. Install them for all modules:

## Install Dependencies For All Modules
for module in proofwatch truthbeam; do
    cd $module && go mod download && cd ..
done

5. Verify Installation#

## Run Tests To Verify Everything Works
make test

## Build All Binaries
make build

Project Structure#

complybeacon/
├── compose.yaml                # podman-compose configuration for demo environment
├── Makefile                    # Build automation
├── docs/                       # Documentation
│   ├── DESIGN.md              # Architecture and design documentation
│   ├── DEVELOPMENT.md         # This file
│   └── attributes/            # Attribute documentation
├── model/                      # OpenTelemetry semantic conventions
│   ├── attributes.yaml        # Attribute definitions
│   └── entities.yaml          # Entity definitions
├── proofwatch/                 # ProofWatch instrumentation library
│   ├── attributes.go          # Attribute definitions
│   ├── evidence.go            # Evidence types
│   └── proofwatch.go          # Main library
├── truthbeam/                  # TruthBeam processor module
│   ├── internal/              # Internal packages
│   ├── config.go              # Configuration
│   └── processor.go           # Main processor logic
├── beacon-distro/              # OpenTelemetry Collector distribution
│   ├── config.yaml            # Collector configuration
│   └── Containerfile.collector # Container definition
├── hack/                       # Development utilities
│   ├── demo/                  # Demo configurations
│   ├── sampledata/            # Sample data for testing
│   └── self-signed-cert/      # self signed cert, testing/development purpose
└── bin/                        # Built binaries (created by make build)

Testing#

Running Tests#

## Run All Tests
make test

## Run Tests For Specific Module
cd proofwatch && go test -v ./...
cd truthbeam && go test -v ./...

Integration Testing#

The project includes integration tests using the demo environment:

## Start The Demo Environment
make deploy

## Test The Pipeline
curl -X POST http://localhost:8088/eventsource/receiver \
  -H "Content-Type: application/json" \
  -d @hack/sampledata/evidence.json

## Check Logs In Grafana At Http://localhost:3000
## Check Compass API At Http://localhost:8081/v1/enrich

Component Development#

1. Proofwatch Development#

ProofWatch is an instrumentation library for emitting compliance evidence.

Key Files:

• proofwatch/proofwatch.go - Main library interface
• proofwatch/evidence.go - Evidence type definition
• proofwatch/attributes.go - OpenTelemetry attributes

Development Workflow:

cd proofwatch

## Run Tests
go test -v ./...

## Check For Linting Issues
go vet ./...

## Format Code
go fmt ./...

2. Compass Development#

Compass is maintained as a separate project. See gemara-content-service for its source code, API specification, and contribution guidelines.

In the ComplyBeacon demo stack, Compass runs as a pre-built container image (ghcr.io/complytime/gemara-content-service:latest) defined in compose.yaml.

3. Truthbeam Development#

TruthBeam is an OpenTelemetry Collector processor for enriching logs.

Key Files:

• truthbeam/processor.go - Main processor logic
• truthbeam/config.go - Configuration structures
• truthbeam/factory.go - Processor factory

Development Workflow:

cd truthbeam

## Run Tests
go test -v ./...

## Test With Collector (requires Beacon-distro)
cd ../beacon-distro
## Modify Config To Use Local Truthbeam
## Run Collector With Local Processor

Local development config

If you want locally test the TruthBeam, remember to change the manifest.yaml

Add replace directive at the end of manifest.yaml, to make sure collector use your truthbeam code. Default collector will use - gomod: github.com/complytime/complybeacon/truthbeam main

For example:

replaces:
  - github.com/complytime/complybeacon/truthbeam => github.com/AlexXuan233/complybeacon/truthbeam 52e4a76ea0f72a7049e73e7a5d67d988116a3892

or

replaces:
  - github.com/complytime/complybeacon/truthbeam => github.com/AlexXuan233/complybeacon/truthbeam main

4. Beacon Distro Development#

The Beacon distribution is a custom OpenTelemetry Collector.

Key Files:

• beacon-distro/config.yaml - Collector configuration
• beacon-distro/Containerfile.collector - Container definition

Development Workflow:

cd beacon-distro

## Build The Collector Image
podman build -f Containerfile.collector -t complybeacon-beacon-distro:latest .

## Test With Local Configuration
podman run --rm -p 4317:4317 -p 8088:8088 \
  -v $(pwd)/config.yaml:/etc/otel-collector.yaml:Z \
  complybeacon-beacon-distro:latest

Debugging And Troubleshooting#

Debugging Tools#

## View Container Logs
podman-compose logs -f compass
podman-compose logs -f collector

Code Generation#

The project uses several code generation tools:

1. Opentelemetry Semantic Conventions#

Generate documentation and Go code from semantic convention models:

## Generate Documentation
make weaver-docsgen

## Generate Go Code
make weaver-codegen

## Validate Models
make weaver-check

2. Manual Code Generation#

If you modify the semantic conventions:

## Update Semantic Conventions
vim model/attributes.yaml
vim model/entities.yaml

## Regenerate Semantic Convention Code
make weaver-codegen

Deployment And Demo#

Local Development Demo#

The demo environment uses podman-compose to orchestrate multiple containers. Ensure you have podman-compose installed before proceeding.

1. Generate self-signed certificate

Since compass and truthbeam enabled TLS by default, first we need to generate self-signed certificate for testing/development

make generate-self-signed-cert

2. Start the full stack:

make deploy

3. Test the pipeline:

curl -X POST http://localhost:8088/eventsource/receiver \
  -H "Content-Type: application/json" \
  -d @hack/sampledata/evidence.json

4. View results:

• Grafana: http://localhost:3000

5. Stop the stack:

make undeploy

Additional Resources#

• OpenTelemetry Documentation
• Go Documentation
• Podman Documentation
• Project Design Document
• Attribute Documentation
• Containers Guide

For questions or support, please open an issue in the GitHub repository.