# SignalBench

An end-to-end, local predictive data engineering reference.

**Implemented scope:** CSV ingestion, strict data checks, a SQL data store, two forecasting baselines, chronological evaluation, seven-horizon analysis and hash-verifiable reports. Python standard library only, including sqlite3. No cloud resources, real customer data or trained ML service are used.

Created for this portfolio with AI assistance. All example observations are synthetic. This is not airline performance or a production deployment.

## Run it

Use Python 3.10 or newer with sqlite3 available. From this extracted directory:

```sh
python -m unittest -v test_signalbench.py
python pipeline.py --input data/workload.csv --output my-first-run
```

Choose a new output directory for every run. The pipeline refuses to overwrite an existing path. Example generated outputs are included under `example-output/`.

## Why it exists

A forecast can look convincing even when its data is wrong or its evaluation uses information from the future. SignalBench makes those decisions inspectable: check the source, keep a record, compare simple methods, and expose the error before using the outlook.

In plain English: check whether the prediction would have worked on days we already know, without letting it peek at those answers.

## Architecture

```mermaid
flowchart TD
  A[Daily workload CSV] --> B{Quality contract}
  B -->|Invalid| C[Stop before publishing output]
  B -->|Valid| D[Input and code fingerprint]
  D --> E[SQLite observations]
  E --> F[Seasonal repeat]
  E --> G[Seven-day mean]
  F --> H[Chronological evaluation]
  G --> H
  H --> I[Holdout and horizon metrics]
  I --> J[Outlook and report exports]
  J --> K[Completion manifest and hashes]
```

## What you can inspect

| File | Purpose |
| --- | --- |
| `data/workload.csv` | The same 56 synthetic observations used by the website |
| `pipeline.py` | Validation, run identity, transactional SQL inserts and exports |
| `forecast.py` | Transparent baseline mathematics and rolling-origin evaluation |
| `schema.sql` | Observations, runs, predictions and metrics with explicit keys |
| `test_signalbench.py` | Data failures, reproducibility, no-peeking, outputs and interruption handling |
| `example-output/report.json` | Full results and provenance, generated by the code |
| `example-output/backtest.csv` | 28 one-day predictions across two methods |
| `example-output/outlook.csv` | Seven future point estimates for each method |
| `example-output/horizon_metrics.csv` | Separate +1 to +7-day validation, eight origins each |
| `example-output/warehouse.sqlite3` | Queryable local warehouse with source and model results |
| `example-output/manifest.json` | Output file hashes and final completion marker |

## Input contract

Exactly two headers: `date,events`. Exactly 56 observations. Dates use canonical ISO `YYYY-MM-DD`, must be ordered, unique and consecutive. Counts must be non-negative integers within JavaScript's safe integer range. Empty, fractional, negative, scientific-notation and missing values are rejected. Maximum CSV size: two megabytes. This is an intentionally narrow, auditable reference contract, not a general telemetry connector.

## Evaluation design

- Days 1 to 28: starting history.
- Days 29 to 42: one-day error-band calibration.
- Days 43 to 56: separate rolling one-day holdout with 14 predictions.
- Days 57 to 63: future outlook using all observed history.

Both baselines use only observations before each forecast origin. Seasonal repeat uses the matching day one week earlier. The seven-day mean uses the previous seven values. Neither is a trained AI model.

The separate horizon study uses eight origins ending at observed days 42 to 49, with each origin producing seven predictions. All test targets are within days 43 to 56. Horizon-specific calibration uses origins 28 to 35 with targets in days 29 to 42. Errors are reported separately for each horizon. Eight origins is a small sample and overlapping windows are not independent trials.

The original outlook plot's band is a symmetric nearest-rank 90th-percentile absolute **one-day** calibration error, clipped at zero. It is an illustrative error envelope, not calibrated seven-day coverage. The horizon report contains separate horizon-specific empirical radii for inspection; the website does not pretend the original band guarantees 90% future coverage.

MAE measures average absolute error. RMSE penalizes large errors more. WAPE divides total absolute error by total absolute actual volume; it is undefined for an all-zero holdout. The weekly-pattern fixture favors a seasonal baseline by construction. Lower error here is not evidence that it wins on real workloads.

## SQL examples

Open the generated database with an SQLite client. These queries are read-only:

```sql
SELECT model, mae, rmse, wape, covered, evaluated
FROM metrics ORDER BY mae;

SELECT model, date, actual, predicted, ABS(actual - predicted) AS error
FROM predictions WHERE stage = 'holdout' ORDER BY date, model;

SELECT source_sha256, code_sha256, row_count FROM runs;
```

## Reliability choices

- Invalid input stops before an output directory is created.
- Run ids bind pipeline version, source bytes and implementation hashes.
- SQL writes use one transaction and parameterized inserts. Primary keys reject duplicate observations within a run.
- An existing output directory is refused. A second successful run to a new directory produces the same run id and report data for the same source and code.
- The completion manifest is written last. If export fails, a partial directory may remain without a manifest. Treat it as incomplete and run again into a new directory. There is no claim of a filesystem-wide atomic commit.
- Hashes detect differences; they are not signatures, authentication or tamper-proof storage.

## What this demonstrates

Data contracts, data modeling, SQL integrity, reproducible batch processing, evaluation without future leakage, model comparison and traceable artifacts. Review the implementation rather than interpreting the project name as a claim of enterprise scale.

## What remains outside this project

No live ingestion, scheduling service, cloud provisioning, online model serving, automatic capacity changes, long-term drift monitoring or measured business savings. A production extension would need more representative data, point-in-time revisions, forecasting at the real decision horizon, protected secrets, observability, cost control and independent evaluation.

## Design references

- [Python CSV module](https://docs.python.org/3/library/csv.html)
- [Python sqlite3 module](https://docs.python.org/3/library/sqlite3.html)
- [Forecasting: Principles and Practice, forecasting tools and time-series evaluation](https://otexts.com/fpp3/toolbox.html)

These references inform the approach. The executable implementation and example results are included here for independent inspection.