domagoj/zavrsni-rad-otel-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e41fd24325a817c2ed9122aa5ed7f9801cbd20d5
zavrsni-rad-otel-app/README.md
Domagoj Andrić 0031caf16c Push the rest
2026-05-11 10:58:46 +02:00

15 KiB
Raw Blame History

OTel BI App

OpenTelemetry-instrumented business intelligence platform with two MSSQL data warehouse sources (AdventureWorks, WorldWideImporters), a PostgreSQL write store, scheduled analytics jobs, an audit log, per-view CSV/PDF data exports, full XLSX+PDF report generation to a shared network path, and an OIDC-secured React frontend.

Architecture

Browser (React + TypeScript)
  │  OIDC Authorization Code + PKCE
  │  Bearer JWT on every API call
  ▼
api-gateway  :8000           ← public surface; validates JWT; mints internal tokens
  ├── /api/aw/**             → aw-service  :8001  (AdventureWorks DW, MSSQL, read-only)
  ├── /api/wwi/**            → wwi-service :8002  (WorldWideImporters DW, MSSQL, read-only)
  ├── /api/reports/generate  → aggregates both services, writes XLSX + PDF to shared mount
  ├── /api/*/export/**       → streams per-view XLSX or PDF directly to browser
  ├── /api/audit             → reads AuditLog from PostgreSQL
  ├── /api/jobs/**           → proxies job history from services
  └── /api/config            → runtime OIDC config (unauthenticated)

aw-service                   AdventureWorks analytics + APScheduler background jobs
wwi-service                  WorldWideImporters analytics + APScheduler background jobs

PostgreSQL                   Shared write store
  ├── audit_log              Append-only event trail
  ├── job_executions         Scheduled job run history
  ├── export_records         Per-view export metadata
  ├── aw_*                   AW persistence tables
  └── wwi_*                  WWI persistence tables

Grafana Alloy (OTLP/HTTP)    Traces + metrics receiver → Tempo / Prometheus

Features

Feature Description
AW Analytics Sales KPIs, 4-year history, linear-regression forecast, rep scores, product demand
WWI Analytics Sales KPIs, reorder recommendations, supplier reliability scores, business events, what-if scenarios
Scheduled Jobs APScheduler cron jobs in each service; per-job OTel root span; job_executions DB record
Audit Log Append-only audit_log table; every analytics call, job, export, and report is logged
Data Export Per-view download as XLSX or PDF; export_records table; OTel span per download
Full Reports POST /api/reports/generate aggregates all views; writes .xlsx + .pdf to SMB/CSI mount
Runtime OIDC Frontend fetches OIDC config from GET /api/config at boot — no build-time secrets baked into the image
Data Quality Daily DQ checks on both warehouses; results surfaced in job history
OTel Frontend W3C trace propagation; FastAPI, SQLAlchemy, HTTPX auto-instrumentation; manual spans on analytics

Repository Layout

.
├── .gitea/workflows/
│   └── docker-publish.yml   # Gitea Actions: build & push all 4 images (matrix)
├── backend/
│   ├── pyproject.toml
│   ├── uv.lock
│   ├── .env.example
│   ├── shared/
│   │   ├── core/
│   │   │   ├── audit.py     # SharedBase, AuditLog, JobExecution, ExportRecord, helpers
│   │   │   ├── config.py    # Pydantic settings (all services)
│   │   │   ├── export.py    # to_xlsx_bytes(), to_pdf_bytes()
│   │   │   ├── otel.py      # configure_otel(), instrument_fastapi/sqlalchemy/httpx
│   │   │   ├── reports.py   # save_report() → XLSX + PDF on filesystem
│   │   │   └── security.py  # JWT validation, internal token mint/verify
│   │   └── db/
│   │       ├── mssql.py     # create_aw_engine(), create_wwi_engine()
│   │       └── postgres.py  # create_postgres_engine(), create_session_factory()
│   └── services/
│       ├── api_gateway/
│       │   └── main.py
│       ├── aw_service/
│       │   ├── main.py
│       │   ├── analytics.py
│       │   ├── models.py
│       │   └── scheduler.py
│       └── wwi_service/
│           ├── main.py
│           ├── analytics.py
│           ├── models.py
│           └── scheduler.py
├── frontend/
│   ├── Dockerfile
│   ├── src/
│   │   ├── api/
│   │   │   └── config.ts    # fetchAppConfig() / getAppConfig()
│   │   ├── auth/
│   │   │   └── oidc.ts
│   │   └── main.tsx
│   └── ...
└── docker-compose.yml

Prerequisites

Tool Version
Docker + Docker Compose 24+
Python 3.11+ (local dev only)
uv latest (local dev only)
Node.js 20+ (local dev only)
SQL Server AdventureWorks DW + WorldWideImporters DW accessible
PostgreSQL 15+

Installation

Docker Compose (recommended)

git clone <repo-url>
cd zavrsni-rad-otel-app

# Copy and fill in environment variables
cp backend/.env.example backend/.env
$EDITOR backend/.env

# Start everything
docker compose up -d

# View logs
docker compose logs -f api-gateway

The frontend is served at http://localhost:5173 (Vite dev server in the compose file) or the container port if using the production image.

Local Development

Backend:

cd backend

# Install uv if needed
pip install uv

# Create virtual environment and install all dependencies
uv sync

# Copy and edit env
cp .env.example .env
$EDITOR .env

# Run services in separate terminals
uvicorn services.api_gateway.main:app  --host 0.0.0.0 --port 8000 --reload
uvicorn services.aw_service.main:app   --host 0.0.0.0 --port 8001 --reload
uvicorn services.wwi_service.main:app  --host 0.0.0.0 --port 8002 --reload

Frontend:

cd frontend
npm install

# Only non-OIDC vars needed at build time
cp .env.example .env.local
# Set VITE_API_BASE_URL=http://localhost:8000
# OIDC config is fetched at runtime from GET /api/config

npm run dev

Configuration Reference

All backend services share shared/core/config.py (Pydantic BaseSettings). Values are read from environment variables or .env.

Database

Variable Default Description
AW_MSSQL_SERVER — AdventureWorks SQL Server host
AW_MSSQL_DATABASE AdventureWorksDW2022 Database name
AW_MSSQL_USERNAME — SQL login (read-only recommended)
AW_MSSQL_PASSWORD — SQL password
WWI_MSSQL_SERVER — WorldWideImporters SQL Server host
WWI_MSSQL_DATABASE WideWorldImportersDW Database name
WWI_MSSQL_USERNAME — SQL login
WWI_MSSQL_PASSWORD — SQL password
MSSQL_TRUST_SERVER_CERTIFICATE false Set true for self-signed certs (dev only)
POSTGRES_URL — PostgreSQL connection URL
POSTGRES_SSLMODE prefer require in production

Security

Variable Default Description
REQUIRE_FRONTEND_AUTH true Enforce JWT validation on public endpoints
FRONTEND_JWT_ISSUER_URL — OIDC issuer (e.g. https://sso.example.com/realms/myapp)
FRONTEND_JWT_JWKS_URL — JWKS endpoint URL
FRONTEND_JWT_AUDIENCE — Expected aud claim
FRONTEND_REQUIRED_SCOPES "" Space-separated required scopes
INTERNAL_SERVICE_SHARED_SECRET — Shared secret for internal JWT (≥32 bytes)
INTERNAL_SERVICE_ALLOWED_ISSUERS api-gateway Accepted internal token issuers

OIDC (served to frontend at runtime)

Variable Default Description
OIDC_ENABLED false Enable OIDC in the frontend
OIDC_AUTHORITY "" OIDC provider base URL
FRONTEND_OIDC_CLIENT_ID "" Frontend client ID
FRONTEND_OIDC_SCOPE openid profile email Requested scopes

These are returned by GET /api/config (unauthenticated) and consumed by the frontend at boot. They are never baked into the Docker image.

Analytics

Variable Default Description
DEFAULT_HISTORY_DAYS 365 Sales history look-back
FORECAST_HORIZON_DAYS 30 Forecast horizon
RANKING_DEFAULT_TOP_N 10 Default ranking list length
STORAGE_DEFAULT_LIMIT 100 Default record list page size

Reports / Exports

Variable Default Description
REPORT_OUTPUT_DIR /tmp/otel-bi-reports Directory where full reports are written; mount an SMB/CSI volume here in Kubernetes

Observability

Variable Default Description
OTEL_EXPORTER_OTLP_ENDPOINT http://localhost:4318 OTLP/HTTP endpoint (Grafana Alloy)
OTEL_SERVICE_NAME (per service) Service name in traces
LOG_LEVEL INFO Python log level

Frontend (build-time only)

Variable Description
VITE_API_BASE_URL Gateway base URL seen by the browser
VITE_OTEL_COLLECTOR_ENDPOINT OTLP/HTTP endpoint for frontend traces
VITE_OTEL_SERVICE_NAME Frontend service name in traces
VITE_OTEL_SERVICE_NAMESPACE Frontend service namespace

Scheduled Jobs

Jobs run automatically on startup. All jobs are recorded in job_executions and emit an OTel root span.

Job ID Service Schedule (UTC) What it does
aw.daily.forecast aw-service 02:00 daily Recompute sales forecast
aw.daily.scores aw-service 02:30 daily Recompute rep + product demand scores
aw.daily.data_quality aw-service 03:00 daily Data quality checks on AW DW
wwi.hourly.reorder wwi-service :00 every hour Refresh reorder recommendations
wwi.hourly.events wwi-service :30 every hour Scan for stock level events
wwi.daily.supplier_scores wwi-service 03:30 daily Recompute supplier reliability scores
wwi.daily.data_quality wwi-service 04:00 daily Data quality checks on WWI DW

API Reference

Public (require valid Bearer JWT unless REQUIRE_FRONTEND_AUTH=false)

Method Path Description
GET /api/config Runtime OIDC config (unauthenticated)
GET /api/health Gateway health
GET /api/aw/sales/kpis AW sales KPIs
GET /api/aw/sales/history AW sales history (?days_back=)
GET /api/aw/sales/forecast AW sales forecast (?horizon_days=)
GET /api/aw/reps/scores AW rep scores (?top_n=)
GET /api/aw/products/demand AW product demand (?top_n=)
GET /api/wwi/sales/kpis WWI sales KPIs
GET /api/wwi/stock/recommendations WWI reorder recommendations
GET /api/wwi/suppliers/scores WWI supplier scores (?top_n=)
GET /api/wwi/events WWI business events (?limit=)
GET /api/wwi/scenarios List what-if scenarios
POST /api/wwi/scenarios Create what-if scenario
GET /api/aw/records/forecasts Stored AW forecasts
GET /api/aw/records/rep-scores Stored AW rep scores
GET /api/aw/records/product-demand Stored AW product demand
GET /api/wwi/records/reorder-recommendations Stored WWI reorder records
GET /api/wwi/records/supplier-scores Stored WWI supplier scores
GET /api/audit Audit log (?domain=aw|wwi&limit=)
GET /api/jobs/aw AW job history
GET /api/jobs/wwi WWI job history
POST /api/reports/generate Generate full XLSX + PDF report to shared mount

Per-view Exports (download as attachment)

Method Path Format
GET /api/aw/export/sales-history ?format=xlsx|pdf
GET /api/aw/export/sales-forecast ?format=xlsx|pdf
GET /api/aw/export/rep-scores ?format=xlsx|pdf
GET /api/aw/export/product-demand ?format=xlsx|pdf
GET /api/wwi/export/stock-recommendations ?format=xlsx|pdf
GET /api/wwi/export/supplier-scores ?format=xlsx|pdf
GET /api/wwi/export/business-events ?format=xlsx|pdf

PostgreSQL Schema

Table Purpose Mutability
audit_log Append-only event trail for every analytics call, job, export, report Immutable rows
job_executions One row per scheduled job run; updated on completion/failure Mutable
export_records Metadata for every per-view download Immutable rows
aw_forecasts Persisted AW sales forecast points Append
aw_rep_scores Persisted AW rep score snapshots Append
aw_product_demand Persisted AW product demand snapshots Append
wwi_reorder_recommendations Persisted WWI reorder recommendations Append
wwi_supplier_scores Persisted WWI supplier score snapshots Append
wwi_whatif_scenarios What-if scenario results Append
wwi_business_events Stock-level business events Append

All tables are created automatically on service startup via metadata.create_all() (idempotent).

CI/CD (Gitea Actions)

.gitea/workflows/docker-publish.yml builds and pushes all four images in parallel using a matrix strategy.

Required repository variables (vars.):

Name Example
REGISTRY registry.example.com
IMAGE_PREFIX myorg/otel-bi

Required repository secrets (secrets.):

Name Description
REGISTRY_USERNAME Registry login
REGISTRY_PASSWORD Registry password / token

Images are tagged with sha-<short>, branch name, semver (on tags), and latest on pushes to master. Push is skipped on pull requests.

OTel Coverage

Frontend:

  • W3C traceparent/tracestate propagation on all fetch calls
  • @opentelemetry/instrumentation-document-load
  • @opentelemetry/instrumentation-user-interaction
  • @opentelemetry/instrumentation-fetch / XHR
  • Manual spans around dashboard aggregation

Backend (all services):

  • FastAPI auto-instrumentation (request span per endpoint)
  • SQLAlchemy auto-instrumentation (MSSQL + PostgreSQL)
  • HTTPX auto-instrumentation (service-to-service calls)
  • Manual root spans per scheduled job
  • Manual spans on analytics functions
  • x-trace-id / x-span-id response headers on internal endpoints

Read-Only Guarantee (MSSQL)

  • ApplicationIntent=ReadOnly on all MSSQL connection strings
  • Query layer only accepts SELECT / WITH statements
  • All write operations target PostgreSQL only
  • Use a SQL Server login with SELECT grants only — no DDL or DML permissions needed

Verification Checklist

  1. GET /api/config returns OIDC settings without authentication.
  2. GET /api/health returns {"status": "ok"}.
  3. After login, GET /api/aw/sales/kpis returns data.
  4. GET /api/audit shows rows for the KPI call.
  5. GET /api/jobs/aw shows scheduled job runs (populated after the next cron tick or after ~1 minute if jobs fired on startup).
  6. GET /api/aw/export/sales-forecast?format=xlsx downloads an .xlsx file.
  7. POST /api/reports/generate returns paths to .xlsx and .pdf files; confirm they appear in REPORT_OUTPUT_DIR.
  8. In Grafana Tempo, verify a trace spans api-gateway → aw-service → PostgreSQL/MSSQL.
Reference in New Issue View Git Blame Copy Permalink