Background Jobs, Testing & Production Operations

Reliability and operational visibility are essential for financial infrastructure. This guide covers how asynchronous background processing, testing suites, real-time diagnostics, and GitOps deployments function across Roja.

---

Asynq Background Task Processing

Roja processes all asynchronous, deferred, and scheduled tasks using Asynq backed by Redis 8.

┌───────────────────────┐
│       API SERVER      │
│  (Business Operation) │
└───────────┬───────────┘
            │ 1. Atomic DB Write
            ▼
┌───────────────────────┐
│     OUTBOX EVENTS     │ (Postgres Table `outbox_events`)
└───────────┬───────────┘
            │ 2. Leases & Publishes (At-least-once)
            ▼
┌───────────────────────┐
│       REDIS 8         │
│  (15 Weighted Queues) │
└───────────┬───────────┘
            │ 3. Distributes Tasks by Queue Weight
            ▼
┌───────────────────────┐
│     ASYNQ WORKER      │ (backend/cmd/asynq-worker)
│  • Payment Settlement │
│  • Webhook Dispatch   │
│  • Push Notifications │
└───────────────────────┘

Worker Daemons & Binaries

Binary / Command	Source Path	Responsibility
asynq-worker	backend/cmd/asynq-worker	Primary consumer daemon executing async task handlers across all queues.
asynq-scheduler	backend/cmd/asynq-scheduler	Emits periodic cron tasks: daily direct debit sweeps, hourly ledger drift reapers, score recalibrations.
asynq-monitor	backend/cmd/asynq-monitor	Exposes Prometheus metrics and HTTP queue statistics for monitoring.
asynq-ui / asynq-ui-web	Taskfile commands	Terminal CLI or password-protected web dashboard for inspecting queue health and dead letters.

Queue Priority Taxonomy & Concurrency Weights

Asynq uses weighted-random scheduling without strict priority starvation. Concurrency slots are distributed across 15 dedicated queues configured in backend/cmd/asynq-worker/main.go and backend/internal/jobs/constants.go:

Priority Tier / Queue	Default Weight	Environment Override	Purpose
score	5	ASYNQ_SCORE_PRIORITY	High-frequency credit score recalculations and tier transitions.
superlender	3	ASYNQ_SUPERLENDER_PRIORITY	Super lender qualification evaluations and grace period warnings.
domain_events	3	ASYNQ_DOMAIN_EVENTS_PRIORITY	Outbox event delivery: funding, settlement, loan activation, money notifications.
disbursement	2	ASYNQ_DISBURSEMENT_PRIORITY	Money-moving disbursement scans, retry loops, and expiry checks.
insurance	2	ASYNQ_INSURANCE_PRIORITY	Policy purchases (Curacel, MyCover) and policy reconciliation.
notifications	1	ASYNQ_NOTIFICATION_PRIORITY	Push notifications, SMS OTP delivery, and email reminders.
kyc	1	ASYNQ_KYC_PRIORITY	Third-party identity verification, document OCR, and selfie liveness evaluation.
collections	1	ASYNQ_COLLECTIONS_PRIORITY	Arrears tracking, restructuring tasks, and dunning progression.
ledger	1	ASYNQ_LEDGER_PRIORITY	Periodic ledger mirror audit scans and drift reapers.
wallet	1	ASYNQ_WALLET_PRIORITY	Drain-only queue for wallet initialization retries.
markov	1	ASYNQ_MARKOV_PRIORITY	Batch Markov transition matrix calculations and risk modeling.
payout_observability	1	ASYNQ_PAYOUT_OBSERVABILITY_PRIORITY	1-minute refresh of payout triage and balance materialized views.
closure	1	ASYNQ_CLOSURE_PRIORITY	Account deletion finalization and data anonymization.
support	1	ASYNQ_SUPPORT_PRIORITY	Customer support ticket sync and CRM integrations.
messaging	1	ASYNQ_MESSAGING_PRIORITY	Standalone messaging webhooks (WhatsApp/Telegram/SMS).

ℹ️

Platform liquidity bot agents run on a dedicated queue (agent) processed by the separate backend/cmd/agent-worker daemon, ensuring agent operations scale independently and cannot starve customer money movements.

---

Transactional Outbox Pattern

To prevent the "Dual-Write Problem" (where a database write succeeds but publishing a message to Redis fails), Roja implements the Transactional Outbox Pattern (backend/internal/events).

1. Transactional Event Insertion

When a business service updates state (e.g. MarkTicketFunded), it writes a domain event row to outbox_events (56_outbox_events.sql) within the exact same database transaction:

events.Write(ctx, tx, "ticket.funded", payload)

2. Lease & Dispatch

The background outbox dispatcher periodically leases batches of pending events (ClaimOutboxEvents), enqueues them into Redis via Asynq, and stamps published_at.

3. Event History Archive

During claim, a Common Table Expression (CTE) automatically copies the event into event_history (59_event_history.sql).

4. Consumer Deduplication

Consumers record idempotent execution in processed_events. If a network glitch re-delivers an event, the consumer detects the receipt and skips execution.

Deliberate Table Separation: Why 3 Tables?

• outbox_events (Queue): A transient queue designed to drain to zero. Retention workers hard-delete rows where published_at < now() - 7 days to keep the pending index fast.
• event_history (Permanent Log): An immutable, append-only log of every domain event ever produced on the platform. It is never reaped.
• processed_events (Idempotency Receipts): Tracks consumer-side execution receipts. It has no retention and outlives outbox events.

---

Runtime Diagnostics & Observability

Roja embeds production-grade profiling and runtime tracing hooks into every long-running backend service.

Go Trace Flight Recorder

Every backend daemon continuously retains a bounded in-memory Go runtime execution trace (10 seconds / 10 MiB buffer). Services write their real OS process ID to /tmp/roja-traces/<component>.pid.

To capture an instant trace snapshot of a live process without restarting:

# Capture trace snapshot of the running API server
kill -USR2 "$(cat /tmp/roja-traces/api-server.pid)"

# Inspect the generated trace in your browser
go tool trace /tmp/roja-traces/api-server-*.trace

Dedicated pprof Profilers

Dedicated pprof HTTP listeners run on dedicated loopback ports:

• API Server: http://localhost:6060
• Asynq Worker: http://localhost:6061
• Asynq Scheduler: http://localhost:6062
• Agent Worker: http://localhost:6063
• Asynq Monitor: http://localhost:6064

# Analyze CPU profile for 30 seconds
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Inspect active heap allocations
go tool pprof http://localhost:6060/debug/pprof/heap

ℹ️

In Kubernetes environments, Grafana Alloy continuously scrapes these pprof endpoints using internal security tokens and streams profiles into Grafana Pyroscope for real-time flame graph visualization.

---

Testing Strategy

Roja enforces a multi-layered testing pyramid:

       ▲
      / \     Maestro E2E (mobile/e2e)
     /───\    Storybook Render Tests (Vitest + composeStories)
    /─────\   Postgres Integration Harness (pgharness)
   /───────\  Go Unit Tests (task backend:test)
  ───────────

1. Go Unit Tests

Unit tests use standard Go testing idioms with mocks for external banking and KYC APIs:

task backend:test    # Runs go test ./... in backend/

2. Database Integration Tests (pgharness)

Real database integration tests run against isolated PostgreSQL instances:

task backend:test-integration    # Docker-backed integration test suite

The integration packages validated by CI are listed in backend/scripts/integration-packages.txt.

3. Storybook Component Tests

Renders every UI component story across apps/web and apps/staff in JSDOM:

bun --filter @roja/storybook-web test

4. Load & Capacity Benchmarks (tools/loadtest)

Roja uses k6 to validate API latency and throughput ceilings:

task loadtest    # Runs k6 capacity baseline against local/staging API

⚠️

task loadtest explicitly refuses to run against production hostnames (api.getroja.com) unless ALLOW_PROD=true is set.

---

Deployment & GitOps Architecture

Roja separates deployment into containerized backend services and edge-deployed frontends.

Nix Flake Container Image Builder

Container images are compiled deterministically by Nix using flake.nix—there are no Dockerfiles in the repository:

• api-image: HTTP API server.
• jobs-image: Asynq worker and scheduler binaries.
• agents-image: Autonomous portfolio-manager bot agent.
• migrate-image: Atlas migration runner.
• asynqmon-image: Queue monitoring web dashboard.

Images are distroless, minimal, and shell-free, eliminating container shell vulnerabilities.

Kubernetes & Flux GitOps

The Go backend and background workers run on a Kubernetes cluster managed via Flux GitOps (getroja/service-registry):

• CI builds container images and pushes them to GitHub Container Registry (ghcr.io/getroja/roja-*).
• Image automation commits new image tags to the GitOps repository.
• Flux continuously reconciles the cluster state against Kustomize overlays:
  • apps/workers: Asynq workers, schedulers, monitors.
  • apps/agents: Autonomous liquidity agents.
• Databases: Development and PR environments run on in-cluster CloudNativePG PostgreSQL; Production runs on managed Neon with high-availability read replicas.

Cloudflare Workers (Frontends)

All web frontends run at the Cloudflare edge:

• apps/web & apps/staff: Built with OpenNext and deployed as Cloudflare Workers for both dev (web.roja.dev, staff.roja.dev) and prod (app.getroja.com, staff.getroja.com).
• apps/docs & Storybooks: Deployed as static asset Cloudflare Workers:
  • Docs Staging: https://docs.roja.dev
  • Docs Production: https://docs.getroja.com
  • PR Previews: https://docs-pr-<n>.roja.dev (automatically provisioned on pull requests).

---

Next Steps

🗺️Architecture Overview

Review the high-level system topology, monorepo layout, and tech stack.

💻Local Development

Set up your machine with Nix flakes and run native services locally.

⚖️Financial Engine & Ledger

Understand the double-entry audit mirror and financial safety invariants.

🧱Domain Models & Lifecycles

Explore the lifecycles of tickets, loans, KYC verification, and collections.