OTel BI App

OpenTelemetry-instrumented business intelligence platform with two MSSQL data warehouse sources (AdventureWorks, WorldWideImporters), a PostgreSQL write store, scheduled analytics jobs, an audit log, per-view CSV/PDF data exports, full XLSX+PDF report generation to a shared network path, and an OIDC-secured React frontend.

---

Architecture

Browser (React + TypeScript)
  │  OIDC Authorization Code + PKCE
  │  Bearer JWT on every API call
  ▼
api-gateway  :8000           ← public surface; validates JWT; mints internal tokens
  ├── /api/aw/**             → aw-service  :8001  (AdventureWorks DW, MSSQL, read-only)
  ├── /api/wwi/**            → wwi-service :8002  (WorldWideImporters DW, MSSQL, read-only)
  ├── /api/reports/generate  → aggregates both services, writes XLSX + PDF to shared mount
  ├── /api/*/export/**       → streams per-view XLSX or PDF directly to browser
  ├── /api/audit             → reads AuditLog from PostgreSQL
  ├── /api/jobs/**           → proxies job history from services
  └── /api/config            → runtime OIDC config (unauthenticated)

aw-service                   AdventureWorks analytics + APScheduler background jobs
wwi-service                  WorldWideImporters analytics + APScheduler background jobs

PostgreSQL                   Shared write store
  ├── audit_log              Append-only event trail
  ├── job_executions         Scheduled job run history
  ├── export_records         Per-view export metadata
  ├── aw_*                   AW persistence tables
  └── wwi_*                  WWI persistence tables

Grafana Alloy (OTLP/HTTP)    Traces + metrics receiver → Tempo / Prometheus

---

Features

Feature	Description
AW Analytics	Sales KPIs, 4-year history, linear-regression forecast, rep scores, product demand
WWI Analytics	Sales KPIs, reorder recommendations, supplier reliability scores, business events, what-if scenarios
Scheduled Jobs	APScheduler cron jobs in each service; per-job OTel root span; job_executions DB record
Audit Log	Append-only audit_log table; every analytics call, job, export, and report is logged
Data Export	Per-view download as XLSX or PDF; export_records table; OTel span per download
Full Reports	POST /api/reports/generate aggregates all views; writes .xlsx + .pdf to SMB/CSI mount
Runtime OIDC	Frontend fetches OIDC config from GET /api/config at boot — no build-time secrets baked into the image
Data Quality	Daily DQ checks on both warehouses; results surfaced in job history
OTel	Frontend W3C trace propagation; FastAPI, SQLAlchemy, HTTPX auto-instrumentation; manual spans on analytics

---

Repository Layout

.
├── .gitea/workflows/
│   └── docker-publish.yml   # Gitea Actions: build & push all 4 images (matrix)
├── backend/
│   ├── pyproject.toml
│   ├── uv.lock
│   ├── .env.example
│   ├── shared/
│   │   ├── core/
│   │   │   ├── audit.py     # SharedBase, AuditLog, JobExecution, ExportRecord, helpers
│   │   │   ├── config.py    # Pydantic settings (all services)
│   │   │   ├── export.py    # to_xlsx_bytes(), to_pdf_bytes()
│   │   │   ├── otel.py      # configure_otel(), instrument_fastapi/sqlalchemy/httpx
│   │   │   ├── reports.py   # save_report() → XLSX + PDF on filesystem
│   │   │   └── security.py  # JWT validation, internal token mint/verify
│   │   └── db/
│   │       ├── mssql.py     # create_aw_engine(), create_wwi_engine()
│   │       └── postgres.py  # create_postgres_engine(), create_session_factory()
│   └── services/
│       ├── api_gateway/
│       │   └── main.py
│       ├── aw_service/
│       │   ├── main.py
│       │   ├── analytics.py
│       │   ├── models.py
│       │   └── scheduler.py
│       └── wwi_service/
│           ├── main.py
│           ├── analytics.py
│           ├── models.py
│           └── scheduler.py
├── frontend/
│   ├── Dockerfile
│   ├── src/
│   │   ├── api/
│   │   │   └── config.ts    # fetchAppConfig() / getAppConfig()
│   │   ├── auth/
│   │   │   └── oidc.ts
│   │   └── main.tsx
│   └── ...
└── docker-compose.yml

---

Prerequisites

Tool	Version
Docker + Docker Compose	24+
Python	3.11+ (local dev only)
uv	latest (local dev only)
Node.js	20+ (local dev only)
SQL Server	AdventureWorks DW + WorldWideImporters DW accessible
PostgreSQL	15+

---

Installation

Docker Compose (recommended)

git clone <repo-url>
cd zavrsni-rad-otel-app

# Copy and fill in environment variables
cp backend/.env.example backend/.env
$EDITOR backend/.env

# Start everything
docker compose up -d

# View logs
docker compose logs -f api-gateway

The frontend is served at http://localhost:5173 (Vite dev server in the compose file) or the container port if using the production image.

Local Development

Backend:

cd backend

# Install uv if needed
pip install uv

# Create virtual environment and install all dependencies
uv sync

# Copy and edit env
cp .env.example .env
$EDITOR .env

# Run services in separate terminals
uvicorn services.api_gateway.main:app  --host 0.0.0.0 --port 8000 --reload
uvicorn services.aw_service.main:app   --host 0.0.0.0 --port 8001 --reload
uvicorn services.wwi_service.main:app  --host 0.0.0.0 --port 8002 --reload

Frontend:

cd frontend
npm install

# Only non-OIDC vars needed at build time
cp .env.example .env.local
# Set VITE_API_BASE_URL=http://localhost:8000
# OIDC config is fetched at runtime from GET /api/config

npm run dev

---

Configuration Reference

All backend services share shared/core/config.py (Pydantic BaseSettings). Values are read from environment variables or .env.

Database

Variable	Default	Description
AW_MSSQL_SERVER	—	AdventureWorks SQL Server host
AW_MSSQL_DATABASE	AdventureWorksDW2022	Database name
AW_MSSQL_USERNAME	—	SQL login (read-only recommended)
AW_MSSQL_PASSWORD	—	SQL password
WWI_MSSQL_SERVER	—	WorldWideImporters SQL Server host
WWI_MSSQL_DATABASE	WideWorldImportersDW	Database name
WWI_MSSQL_USERNAME	—	SQL login
WWI_MSSQL_PASSWORD	—	SQL password
MSSQL_TRUST_SERVER_CERTIFICATE	false	Set true for self-signed certs (dev only)
POSTGRES_URL	—	PostgreSQL connection URL
POSTGRES_SSLMODE	prefer	require in production

Security

Variable	Default	Description
REQUIRE_FRONTEND_AUTH	true	Enforce JWT validation on public endpoints
FRONTEND_JWT_ISSUER_URL	—	OIDC issuer (e.g. https://sso.example.com/realms/myapp)
FRONTEND_JWT_JWKS_URL	—	JWKS endpoint URL
FRONTEND_JWT_AUDIENCE	—	Expected aud claim
FRONTEND_REQUIRED_SCOPES	""	Space-separated required scopes
INTERNAL_SERVICE_SHARED_SECRET	—	Shared secret for internal JWT (≥32 bytes)
INTERNAL_SERVICE_ALLOWED_ISSUERS	api-gateway	Accepted internal token issuers

OIDC (served to frontend at runtime)

Variable	Default	Description
OIDC_ENABLED	false	Enable OIDC in the frontend
OIDC_AUTHORITY	""	OIDC provider base URL
FRONTEND_OIDC_CLIENT_ID	""	Frontend client ID
FRONTEND_OIDC_SCOPE	openid profile email	Requested scopes

These are returned by GET /api/config (unauthenticated) and consumed by the frontend at boot. They are never baked into the Docker image.

Analytics

Variable	Default	Description
DEFAULT_HISTORY_DAYS	365	Sales history look-back
FORECAST_HORIZON_DAYS	30	Forecast horizon
RANKING_DEFAULT_TOP_N	10	Default ranking list length
STORAGE_DEFAULT_LIMIT	100	Default record list page size

Reports / Exports

Variable	Default	Description
REPORT_OUTPUT_DIR	/tmp/otel-bi-reports	Directory where full reports are written; mount an SMB/CSI volume here in Kubernetes

Observability

Variable	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4318	OTLP/HTTP endpoint (Grafana Alloy)
OTEL_SERVICE_NAME	(per service)	Service name in traces
LOG_LEVEL	INFO	Python log level

Frontend (build-time only)

Variable	Description
VITE_API_BASE_URL	Gateway base URL seen by the browser
VITE_OTEL_COLLECTOR_ENDPOINT	OTLP/HTTP endpoint for frontend traces
VITE_OTEL_SERVICE_NAME	Frontend service name in traces
VITE_OTEL_SERVICE_NAMESPACE	Frontend service namespace

---

Scheduled Jobs

Jobs run automatically on startup. All jobs are recorded in job_executions and emit an OTel root span.

Job ID	Service	Schedule (UTC)	What it does
aw.daily.forecast	aw-service	02:00 daily	Recompute sales forecast
aw.daily.scores	aw-service	02:30 daily	Recompute rep + product demand scores
aw.daily.data_quality	aw-service	03:00 daily	Data quality checks on AW DW
wwi.hourly.reorder	wwi-service	:00 every hour	Refresh reorder recommendations
wwi.hourly.events	wwi-service	:30 every hour	Scan for stock level events
wwi.daily.supplier_scores	wwi-service	03:30 daily	Recompute supplier reliability scores
wwi.daily.data_quality	wwi-service	04:00 daily	Data quality checks on WWI DW

---

API Reference

Public (require valid Bearer JWT unless REQUIRE_FRONTEND_AUTH=false)

Method	Path	Description
GET	/api/config	Runtime OIDC config (unauthenticated)
GET	/api/health	Gateway health
GET	/api/aw/sales/kpis	AW sales KPIs
GET	/api/aw/sales/history	AW sales history (?days_back=)
GET	/api/aw/sales/forecast	AW sales forecast (?horizon_days=)
GET	/api/aw/reps/scores	AW rep scores (?top_n=)
GET	/api/aw/products/demand	AW product demand (?top_n=)
GET	/api/wwi/sales/kpis	WWI sales KPIs
GET	/api/wwi/stock/recommendations	WWI reorder recommendations
GET	/api/wwi/suppliers/scores	WWI supplier scores (?top_n=)
GET	/api/wwi/events	WWI business events (?limit=)
GET	/api/wwi/scenarios	List what-if scenarios
POST	/api/wwi/scenarios	Create what-if scenario
GET	/api/aw/records/forecasts	Stored AW forecasts
GET	/api/aw/records/rep-scores	Stored AW rep scores
GET	/api/aw/records/product-demand	Stored AW product demand
GET	/api/wwi/records/reorder-recommendations	Stored WWI reorder records
GET	/api/wwi/records/supplier-scores	Stored WWI supplier scores
GET	/api/audit	Audit log (?domain=aw|wwi&limit=)
GET	/api/jobs/aw	AW job history
GET	/api/jobs/wwi	WWI job history
POST	/api/reports/generate	Generate full XLSX + PDF report to shared mount

Per-view Exports (download as attachment)

Method	Path	Format
GET	/api/aw/export/sales-history	?format=xlsx|pdf
GET	/api/aw/export/sales-forecast	?format=xlsx|pdf
GET	/api/aw/export/rep-scores	?format=xlsx|pdf
GET	/api/aw/export/product-demand	?format=xlsx|pdf
GET	/api/wwi/export/stock-recommendations	?format=xlsx|pdf
GET	/api/wwi/export/supplier-scores	?format=xlsx|pdf
GET	/api/wwi/export/business-events	?format=xlsx|pdf

---

PostgreSQL Schema

Table	Purpose	Mutability
audit_log	Append-only event trail for every analytics call, job, export, report	Immutable rows
job_executions	One row per scheduled job run; updated on completion/failure	Mutable
export_records	Metadata for every per-view download	Immutable rows
aw_forecasts	Persisted AW sales forecast points	Append
aw_rep_scores	Persisted AW rep score snapshots	Append
aw_product_demand	Persisted AW product demand snapshots	Append
wwi_reorder_recommendations	Persisted WWI reorder recommendations	Append
wwi_supplier_scores	Persisted WWI supplier score snapshots	Append
wwi_whatif_scenarios	What-if scenario results	Append
wwi_business_events	Stock-level business events	Append

All tables are created automatically on service startup via metadata.create_all() (idempotent).

---

CI/CD (Gitea Actions)

.gitea/workflows/docker-publish.yml builds and pushes all four images in parallel using a matrix strategy.

Required repository variables (vars.):

Name	Example
REGISTRY	registry.example.com
IMAGE_PREFIX	myorg/otel-bi

Required repository secrets (secrets.):

Name	Description
REGISTRY_USERNAME	Registry login
REGISTRY_PASSWORD	Registry password / token

Images are tagged with sha-<short>, branch name, semver (on tags), and latest on pushes to master. Push is skipped on pull requests.

---

OTel Coverage

Frontend:

• W3C traceparent/tracestate propagation on all fetch calls
• @opentelemetry/instrumentation-document-load
• @opentelemetry/instrumentation-user-interaction
• @opentelemetry/instrumentation-fetch / XHR
• Manual spans around dashboard aggregation

Backend (all services):

• FastAPI auto-instrumentation (request span per endpoint)
• SQLAlchemy auto-instrumentation (MSSQL + PostgreSQL)
• HTTPX auto-instrumentation (service-to-service calls)
• Manual root spans per scheduled job
• Manual spans on analytics functions
• x-trace-id / x-span-id response headers on internal endpoints

---

Read-Only Guarantee (MSSQL)

• ApplicationIntent=ReadOnly on all MSSQL connection strings
• Query layer only accepts SELECT / WITH statements
• All write operations target PostgreSQL only
• Use a SQL Server login with SELECT grants only — no DDL or DML permissions needed

---

Verification Checklist

1. GET /api/config returns OIDC settings without authentication.
2. GET /api/health returns {"status": "ok"}.
3. After login, GET /api/aw/sales/kpis returns data.
4. GET /api/audit shows rows for the KPI call.
5. GET /api/jobs/aw shows scheduled job runs (populated after the next cron tick or after ~1 minute if jobs fired on startup).
6. GET /api/aw/export/sales-forecast?format=xlsx downloads an .xlsx file.
7. POST /api/reports/generate returns paths to .xlsx and .pdf files; confirm they appear in REPORT_OUTPUT_DIR.
8. In Grafana Tempo, verify a trace spans api-gateway → aw-service → PostgreSQL/MSSQL.