390 lines
15 KiB
Markdown
390 lines
15 KiB
Markdown
# OTel BI App
|
|
|
|
OpenTelemetry-instrumented business intelligence platform with two MSSQL data warehouse sources (AdventureWorks, WorldWideImporters), a PostgreSQL write store, scheduled analytics jobs, an audit log, per-view CSV/PDF data exports, full XLSX+PDF report generation to a shared network path, and an OIDC-secured React frontend.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Browser (React + TypeScript)
|
|
│ OIDC Authorization Code + PKCE
|
|
│ Bearer JWT on every API call
|
|
▼
|
|
api-gateway :8000 ← public surface; validates JWT; mints internal tokens
|
|
├── /api/aw/** → aw-service :8001 (AdventureWorks DW, MSSQL, read-only)
|
|
├── /api/wwi/** → wwi-service :8002 (WorldWideImporters DW, MSSQL, read-only)
|
|
├── /api/reports/generate → aggregates both services, writes XLSX + PDF to shared mount
|
|
├── /api/*/export/** → streams per-view XLSX or PDF directly to browser
|
|
├── /api/audit → reads AuditLog from PostgreSQL
|
|
├── /api/jobs/** → proxies job history from services
|
|
└── /api/config → runtime OIDC config (unauthenticated)
|
|
|
|
aw-service AdventureWorks analytics + APScheduler background jobs
|
|
wwi-service WorldWideImporters analytics + APScheduler background jobs
|
|
|
|
PostgreSQL Shared write store
|
|
├── audit_log Append-only event trail
|
|
├── job_executions Scheduled job run history
|
|
├── export_records Per-view export metadata
|
|
├── aw_* AW persistence tables
|
|
└── wwi_* WWI persistence tables
|
|
|
|
Grafana Alloy (OTLP/HTTP) Traces + metrics receiver → Tempo / Prometheus
|
|
```
|
|
|
|
---
|
|
|
|
## Features
|
|
|
|
| Feature | Description |
|
|
|---|---|
|
|
| **AW Analytics** | Sales KPIs, 4-year history, linear-regression forecast, rep scores, product demand |
|
|
| **WWI Analytics** | Sales KPIs, reorder recommendations, supplier reliability scores, business events, what-if scenarios |
|
|
| **Scheduled Jobs** | APScheduler cron jobs in each service; per-job OTel root span; `job_executions` DB record |
|
|
| **Audit Log** | Append-only `audit_log` table; every analytics call, job, export, and report is logged |
|
|
| **Data Export** | Per-view download as XLSX or PDF; `export_records` table; OTel span per download |
|
|
| **Full Reports** | `POST /api/reports/generate` aggregates all views; writes `.xlsx` + `.pdf` to SMB/CSI mount |
|
|
| **Runtime OIDC** | Frontend fetches OIDC config from `GET /api/config` at boot — no build-time secrets baked into the image |
|
|
| **Data Quality** | Daily DQ checks on both warehouses; results surfaced in job history |
|
|
| **OTel** | Frontend W3C trace propagation; FastAPI, SQLAlchemy, HTTPX auto-instrumentation; manual spans on analytics |
|
|
|
|
---
|
|
|
|
## Repository Layout
|
|
|
|
```
|
|
.
|
|
├── .gitea/workflows/
|
|
│ └── docker-publish.yml # Gitea Actions: build & push all 4 images (matrix)
|
|
├── backend/
|
|
│ ├── pyproject.toml
|
|
│ ├── uv.lock
|
|
│ ├── .env.example
|
|
│ ├── shared/
|
|
│ │ ├── core/
|
|
│ │ │ ├── audit.py # SharedBase, AuditLog, JobExecution, ExportRecord, helpers
|
|
│ │ │ ├── config.py # Pydantic settings (all services)
|
|
│ │ │ ├── export.py # to_xlsx_bytes(), to_pdf_bytes()
|
|
│ │ │ ├── otel.py # configure_otel(), instrument_fastapi/sqlalchemy/httpx
|
|
│ │ │ ├── reports.py # save_report() → XLSX + PDF on filesystem
|
|
│ │ │ └── security.py # JWT validation, internal token mint/verify
|
|
│ │ └── db/
|
|
│ │ ├── mssql.py # create_aw_engine(), create_wwi_engine()
|
|
│ │ └── postgres.py # create_postgres_engine(), create_session_factory()
|
|
│ └── services/
|
|
│ ├── api_gateway/
|
|
│ │ └── main.py
|
|
│ ├── aw_service/
|
|
│ │ ├── main.py
|
|
│ │ ├── analytics.py
|
|
│ │ ├── models.py
|
|
│ │ └── scheduler.py
|
|
│ └── wwi_service/
|
|
│ ├── main.py
|
|
│ ├── analytics.py
|
|
│ ├── models.py
|
|
│ └── scheduler.py
|
|
├── frontend/
|
|
│ ├── Dockerfile
|
|
│ ├── src/
|
|
│ │ ├── api/
|
|
│ │ │ └── config.ts # fetchAppConfig() / getAppConfig()
|
|
│ │ ├── auth/
|
|
│ │ │ └── oidc.ts
|
|
│ │ └── main.tsx
|
|
│ └── ...
|
|
└── docker-compose.yml
|
|
```
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
| Tool | Version |
|
|
|---|---|
|
|
| Docker + Docker Compose | 24+ |
|
|
| Python | 3.11+ (local dev only) |
|
|
| uv | latest (local dev only) |
|
|
| Node.js | 20+ (local dev only) |
|
|
| SQL Server | AdventureWorks DW + WorldWideImporters DW accessible |
|
|
| PostgreSQL | 15+ |
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
### Docker Compose (recommended)
|
|
|
|
```bash
|
|
git clone <repo-url>
|
|
cd zavrsni-rad-otel-app
|
|
|
|
# Copy and fill in environment variables
|
|
cp backend/.env.example backend/.env
|
|
$EDITOR backend/.env
|
|
|
|
# Start everything
|
|
docker compose up -d
|
|
|
|
# View logs
|
|
docker compose logs -f api-gateway
|
|
```
|
|
|
|
The frontend is served at `http://localhost:5173` (Vite dev server in the compose file) or the container port if using the production image.
|
|
|
|
### Local Development
|
|
|
|
**Backend:**
|
|
|
|
```bash
|
|
cd backend
|
|
|
|
# Install uv if needed
|
|
pip install uv
|
|
|
|
# Create virtual environment and install all dependencies
|
|
uv sync
|
|
|
|
# Copy and edit env
|
|
cp .env.example .env
|
|
$EDITOR .env
|
|
|
|
# Run services in separate terminals
|
|
uvicorn services.api_gateway.main:app --host 0.0.0.0 --port 8000 --reload
|
|
uvicorn services.aw_service.main:app --host 0.0.0.0 --port 8001 --reload
|
|
uvicorn services.wwi_service.main:app --host 0.0.0.0 --port 8002 --reload
|
|
```
|
|
|
|
**Frontend:**
|
|
|
|
```bash
|
|
cd frontend
|
|
npm install
|
|
|
|
# Only non-OIDC vars needed at build time
|
|
cp .env.example .env.local
|
|
# Set VITE_API_BASE_URL=http://localhost:8000
|
|
# OIDC config is fetched at runtime from GET /api/config
|
|
|
|
npm run dev
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration Reference
|
|
|
|
All backend services share `shared/core/config.py` (Pydantic `BaseSettings`). Values are read from environment variables or `.env`.
|
|
|
|
### Database
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `AW_MSSQL_SERVER` | — | AdventureWorks SQL Server host |
|
|
| `AW_MSSQL_DATABASE` | `AdventureWorksDW2022` | Database name |
|
|
| `AW_MSSQL_USERNAME` | — | SQL login (read-only recommended) |
|
|
| `AW_MSSQL_PASSWORD` | — | SQL password |
|
|
| `WWI_MSSQL_SERVER` | — | WorldWideImporters SQL Server host |
|
|
| `WWI_MSSQL_DATABASE` | `WideWorldImportersDW` | Database name |
|
|
| `WWI_MSSQL_USERNAME` | — | SQL login |
|
|
| `WWI_MSSQL_PASSWORD` | — | SQL password |
|
|
| `MSSQL_TRUST_SERVER_CERTIFICATE` | `false` | Set `true` for self-signed certs (dev only) |
|
|
| `POSTGRES_URL` | — | PostgreSQL connection URL |
|
|
| `POSTGRES_SSLMODE` | `prefer` | `require` in production |
|
|
|
|
### Security
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `REQUIRE_FRONTEND_AUTH` | `true` | Enforce JWT validation on public endpoints |
|
|
| `FRONTEND_JWT_ISSUER_URL` | — | OIDC issuer (e.g. `https://sso.example.com/realms/myapp`) |
|
|
| `FRONTEND_JWT_JWKS_URL` | — | JWKS endpoint URL |
|
|
| `FRONTEND_JWT_AUDIENCE` | — | Expected `aud` claim |
|
|
| `FRONTEND_REQUIRED_SCOPES` | `""` | Space-separated required scopes |
|
|
| `INTERNAL_SERVICE_SHARED_SECRET` | — | Shared secret for internal JWT (≥32 bytes) |
|
|
| `INTERNAL_SERVICE_ALLOWED_ISSUERS` | `api-gateway` | Accepted internal token issuers |
|
|
|
|
### OIDC (served to frontend at runtime)
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `OIDC_ENABLED` | `false` | Enable OIDC in the frontend |
|
|
| `OIDC_AUTHORITY` | `""` | OIDC provider base URL |
|
|
| `FRONTEND_OIDC_CLIENT_ID` | `""` | Frontend client ID |
|
|
| `FRONTEND_OIDC_SCOPE` | `openid profile email` | Requested scopes |
|
|
|
|
These are returned by `GET /api/config` (unauthenticated) and consumed by the frontend at boot. They are never baked into the Docker image.
|
|
|
|
### Analytics
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `DEFAULT_HISTORY_DAYS` | `365` | Sales history look-back |
|
|
| `FORECAST_HORIZON_DAYS` | `30` | Forecast horizon |
|
|
| `RANKING_DEFAULT_TOP_N` | `10` | Default ranking list length |
|
|
| `STORAGE_DEFAULT_LIMIT` | `100` | Default record list page size |
|
|
|
|
### Reports / Exports
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `REPORT_OUTPUT_DIR` | `/tmp/otel-bi-reports` | Directory where full reports are written; mount an SMB/CSI volume here in Kubernetes |
|
|
|
|
### Observability
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTLP/HTTP endpoint (Grafana Alloy) |
|
|
| `OTEL_SERVICE_NAME` | *(per service)* | Service name in traces |
|
|
| `LOG_LEVEL` | `INFO` | Python log level |
|
|
|
|
### Frontend (build-time only)
|
|
|
|
| Variable | Description |
|
|
|---|---|
|
|
| `VITE_API_BASE_URL` | Gateway base URL seen by the browser |
|
|
| `VITE_OTEL_COLLECTOR_ENDPOINT` | OTLP/HTTP endpoint for frontend traces |
|
|
| `VITE_OTEL_SERVICE_NAME` | Frontend service name in traces |
|
|
| `VITE_OTEL_SERVICE_NAMESPACE` | Frontend service namespace |
|
|
|
|
---
|
|
|
|
## Scheduled Jobs
|
|
|
|
Jobs run automatically on startup. All jobs are recorded in `job_executions` and emit an OTel root span.
|
|
|
|
| Job ID | Service | Schedule (UTC) | What it does |
|
|
|---|---|---|---|
|
|
| `aw.daily.forecast` | aw-service | 02:00 daily | Recompute sales forecast |
|
|
| `aw.daily.scores` | aw-service | 02:30 daily | Recompute rep + product demand scores |
|
|
| `aw.daily.data_quality` | aw-service | 03:00 daily | Data quality checks on AW DW |
|
|
| `wwi.hourly.reorder` | wwi-service | :00 every hour | Refresh reorder recommendations |
|
|
| `wwi.hourly.events` | wwi-service | :30 every hour | Scan for stock level events |
|
|
| `wwi.daily.supplier_scores` | wwi-service | 03:30 daily | Recompute supplier reliability scores |
|
|
| `wwi.daily.data_quality` | wwi-service | 04:00 daily | Data quality checks on WWI DW |
|
|
|
|
---
|
|
|
|
## API Reference
|
|
|
|
### Public (require valid Bearer JWT unless `REQUIRE_FRONTEND_AUTH=false`)
|
|
|
|
| Method | Path | Description |
|
|
|---|---|---|
|
|
| `GET` | `/api/config` | Runtime OIDC config (**unauthenticated**) |
|
|
| `GET` | `/api/health` | Gateway health |
|
|
| `GET` | `/api/aw/sales/kpis` | AW sales KPIs |
|
|
| `GET` | `/api/aw/sales/history` | AW sales history (`?days_back=`) |
|
|
| `GET` | `/api/aw/sales/forecast` | AW sales forecast (`?horizon_days=`) |
|
|
| `GET` | `/api/aw/reps/scores` | AW rep scores (`?top_n=`) |
|
|
| `GET` | `/api/aw/products/demand` | AW product demand (`?top_n=`) |
|
|
| `GET` | `/api/wwi/sales/kpis` | WWI sales KPIs |
|
|
| `GET` | `/api/wwi/stock/recommendations` | WWI reorder recommendations |
|
|
| `GET` | `/api/wwi/suppliers/scores` | WWI supplier scores (`?top_n=`) |
|
|
| `GET` | `/api/wwi/events` | WWI business events (`?limit=`) |
|
|
| `GET` | `/api/wwi/scenarios` | List what-if scenarios |
|
|
| `POST` | `/api/wwi/scenarios` | Create what-if scenario |
|
|
| `GET` | `/api/aw/records/forecasts` | Stored AW forecasts |
|
|
| `GET` | `/api/aw/records/rep-scores` | Stored AW rep scores |
|
|
| `GET` | `/api/aw/records/product-demand` | Stored AW product demand |
|
|
| `GET` | `/api/wwi/records/reorder-recommendations` | Stored WWI reorder records |
|
|
| `GET` | `/api/wwi/records/supplier-scores` | Stored WWI supplier scores |
|
|
| `GET` | `/api/audit` | Audit log (`?domain=aw\|wwi&limit=`) |
|
|
| `GET` | `/api/jobs/aw` | AW job history |
|
|
| `GET` | `/api/jobs/wwi` | WWI job history |
|
|
| `POST` | `/api/reports/generate` | Generate full XLSX + PDF report to shared mount |
|
|
|
|
### Per-view Exports (download as attachment)
|
|
|
|
| Method | Path | Format |
|
|
|---|---|---|
|
|
| `GET` | `/api/aw/export/sales-history` | `?format=xlsx\|pdf` |
|
|
| `GET` | `/api/aw/export/sales-forecast` | `?format=xlsx\|pdf` |
|
|
| `GET` | `/api/aw/export/rep-scores` | `?format=xlsx\|pdf` |
|
|
| `GET` | `/api/aw/export/product-demand` | `?format=xlsx\|pdf` |
|
|
| `GET` | `/api/wwi/export/stock-recommendations` | `?format=xlsx\|pdf` |
|
|
| `GET` | `/api/wwi/export/supplier-scores` | `?format=xlsx\|pdf` |
|
|
| `GET` | `/api/wwi/export/business-events` | `?format=xlsx\|pdf` |
|
|
|
|
---
|
|
|
|
## PostgreSQL Schema
|
|
|
|
| Table | Purpose | Mutability |
|
|
|---|---|---|
|
|
| `audit_log` | Append-only event trail for every analytics call, job, export, report | Immutable rows |
|
|
| `job_executions` | One row per scheduled job run; updated on completion/failure | Mutable |
|
|
| `export_records` | Metadata for every per-view download | Immutable rows |
|
|
| `aw_forecasts` | Persisted AW sales forecast points | Append |
|
|
| `aw_rep_scores` | Persisted AW rep score snapshots | Append |
|
|
| `aw_product_demand` | Persisted AW product demand snapshots | Append |
|
|
| `wwi_reorder_recommendations` | Persisted WWI reorder recommendations | Append |
|
|
| `wwi_supplier_scores` | Persisted WWI supplier score snapshots | Append |
|
|
| `wwi_whatif_scenarios` | What-if scenario results | Append |
|
|
| `wwi_business_events` | Stock-level business events | Append |
|
|
|
|
All tables are created automatically on service startup via `metadata.create_all()` (idempotent).
|
|
|
|
---
|
|
|
|
## CI/CD (Gitea Actions)
|
|
|
|
`.gitea/workflows/docker-publish.yml` builds and pushes all four images in parallel using a matrix strategy.
|
|
|
|
**Required repository variables (`vars.`):**
|
|
|
|
| Name | Example |
|
|
|---|---|
|
|
| `REGISTRY` | `registry.example.com` |
|
|
| `IMAGE_PREFIX` | `myorg/otel-bi` |
|
|
|
|
**Required repository secrets (`secrets.`):**
|
|
|
|
| Name | Description |
|
|
|---|---|
|
|
| `REGISTRY_USERNAME` | Registry login |
|
|
| `REGISTRY_PASSWORD` | Registry password / token |
|
|
|
|
Images are tagged with `sha-<short>`, branch name, semver (on tags), and `latest` on pushes to `master`. Push is skipped on pull requests.
|
|
|
|
---
|
|
|
|
## OTel Coverage
|
|
|
|
**Frontend:**
|
|
- W3C `traceparent`/`tracestate` propagation on all fetch calls
|
|
- `@opentelemetry/instrumentation-document-load`
|
|
- `@opentelemetry/instrumentation-user-interaction`
|
|
- `@opentelemetry/instrumentation-fetch` / XHR
|
|
- Manual spans around dashboard aggregation
|
|
|
|
**Backend (all services):**
|
|
- FastAPI auto-instrumentation (request span per endpoint)
|
|
- SQLAlchemy auto-instrumentation (MSSQL + PostgreSQL)
|
|
- HTTPX auto-instrumentation (service-to-service calls)
|
|
- Manual root spans per scheduled job
|
|
- Manual spans on analytics functions
|
|
- `x-trace-id` / `x-span-id` response headers on internal endpoints
|
|
|
|
---
|
|
|
|
## Read-Only Guarantee (MSSQL)
|
|
|
|
- `ApplicationIntent=ReadOnly` on all MSSQL connection strings
|
|
- Query layer only accepts `SELECT` / `WITH` statements
|
|
- All write operations target PostgreSQL only
|
|
- Use a SQL Server login with `SELECT` grants only — no DDL or DML permissions needed
|
|
|
|
---
|
|
|
|
## Verification Checklist
|
|
|
|
1. `GET /api/config` returns OIDC settings without authentication.
|
|
2. `GET /api/health` returns `{"status": "ok"}`.
|
|
3. After login, `GET /api/aw/sales/kpis` returns data.
|
|
4. `GET /api/audit` shows rows for the KPI call.
|
|
5. `GET /api/jobs/aw` shows scheduled job runs (populated after the next cron tick or after ~1 minute if jobs fired on startup).
|
|
6. `GET /api/aw/export/sales-forecast?format=xlsx` downloads an `.xlsx` file.
|
|
7. `POST /api/reports/generate` returns paths to `.xlsx` and `.pdf` files; confirm they appear in `REPORT_OUTPUT_DIR`.
|
|
8. In Grafana Tempo, verify a trace spans `api-gateway → aw-service → PostgreSQL/MSSQL`.
|