Menu

RISHIJAY S KOCHER / DATA ENGINEER

Engineered
for trust.

Cloud data platforms. Agentic AI.

I’m Rishijay. I design reliable data platforms that turn scattered information into trusted foundations for analytics, automation, and better decisions.

Explore engineering work Connect on LinkedIn (opens in a new tab)
01 / DATA TO DECISIONS

Conceptual architecture, not a live service. Files, events and databases enter a data contract gate. Failed checks route records to review. Accepted data passes through repeatable transformations into raw, checked and served storage layers. Analytics, AI and forecasting consume the governed outputs. Lineage, access rules and monitoring span the system.

FilesEventsDatabases
01 / ACCEPT OR REVIEWData contracts

Check the shape, quality and arrival of data.

Failed checks → review
02 / MAKE IT USEFULTransform + validate

Repeatable steps. Traceable results.

ReadProcessCheck
RAW Preserve inputsCHECKED Apply trust rulesSERVED Publish useful models
AnalyticsAI contextForecasts

Lineage / Access rules / Observability

Validate first. Process in parallel. Keep the data traceable.

Trust is built into the system.Architecture concept · not live telemetry

CORE STACK

01 / FEATURED ENGINEERING WORK

The architecture.
The implementation.

Source you can inspect. Decisions you can question. A clear line between what works today and what comes next.

A quick guide to the evidence.

Start with the implementation, then inspect the decisions. Working code and future designs are labelled separately so you can assess each on its own evidence.

SkyFlow REPOSITORY-BACKED / CLOUD DATAOPS

Batch and streaming paths, quality checks, operational logs and linked CI evidence. Cloud health is not inferred from offline tests.

SignalBench RUNNABLE LOCAL PROJECT / FORECAST EVALUATION

Download the code, reproduce the results and query the generated SQL database. All example data is synthetic.

TracePilot PLANNED SYSTEM / WORKING POLICY PROTOTYPE

Inspect agent boundaries, evidence contracts and approval rules. No autonomous backend is connected.

LoadCast PLANNED SYSTEM / WORKING QUEUE SIMULATOR

Test capacity assumptions and failure gates. The plotted demand curves are authored scenarios, not trained forecasts.

INDEPENDENT PROJECT / GOOGLE CLOUD

SkyFlow

An airline DataOps platform

Flight events arrive through batch files and streams. SkyFlow organizes them into queryable data and records what happened when the pipeline ran.

Explore repository (opens in a new tab)Inspect CI run (opens in a new tab)
47offline tests documented
10batch data-quality checks
IaCTerraform resource adoption

Python · SQL · BigQuery
Pub/Sub · Terraform · GitHub Actions

SKYFLOW / IMPLEMENTATION MAP

Every event has a path.

Two ingestion paths. One view of how the system behaves.

DOCUMENTED ARCHITECTURE
NOT LIVE TELEMETRY
01
Batch

Files to checked metrics.

Replace and rebuild. The batch loader replaces RAW; SQL rebuilds CLEAN/CURATED. Checks assess the batch outputs.

Separate ingestion paths

Streaming is not merged into batch CLEAN/CURATED.

02
Streaming

Events to queryable records.

Illustrative flow. No live service is contacted.

Delivery boundary. Retries can create duplicates. No application-level exactly-once guarantee or configured dead-letter workflow.

READ-ONLY EVIDENCE
Batch run logsStreaming health views
03
AI diagnostics

Evidence before advice.

Read-only assistance. Not an autonomous remediation agent. Optional AI explanation code is implemented; a successful live AI acceptance run remains pending.

SUPPORTING CONTROLS

Read the engineering decisions

The business question

Can operational teams trace a metric back to a known input and understand whether the pipeline completed? SkyFlow separates ingestion, transformation, checking, and observation.

The implementation boundary

Terraform adopts four datasets, a topic, and a subscription. Tables, views, and IAM are not independently managed in that step. Fresh-environment provisioning and automated deployment remain planned.

What the evidence establishes

The repository documents 47 offline cases. The linked CI run passed for commit b9f34cf. Offline checks mock cloud clients; they do not establish current cloud health or execute transformation SQL.

CI scope (opens in a new tab)Infrastructure decisions (opens in a new tab)Diagnostic workflow (opens in a new tab)

Independent portfolio implementation using synthetic data. Not affiliated with an airline and not a production airline deployment.

NEXT / ENGINEERING BLUEPRINTS

Think it through.
Then build it.

Two ambitious data engineering projects, designed from the failure cases outward. Explore the systems, test the decisions, and inspect the plan to build them.

Planned projects, not completed deployments. AI-assisted designs with working browser-local prototypes and synthetic inputs. Backend services and trained models are not implemented.

01 / AGENTIC DATA ENGINEERINGTracePilot Find the fault. Control the fix.02 / PREDICTIVE DATA OPERATIONSLoadCast See demand. Plan capacity.

01 / AGENTIC DATA ENGINEERING

TracePilot.

An AI investigator.
A human-controlled repair.

PLANNED / DESIGN BLUEPRINT

When a data pipeline fails, TracePilot would find the cause, show the evidence and suggest a fix. Tests check the fix. A person decides whether it can run.

Download the build kit
THE PROBLEM

A failed pipeline leaves an engineer searching across logs and data checks.

THE IDEA

An AI investigator gathers evidence and proposes a fix within clear limits.

WHAT YOU CAN TRY

A working control-policy prototype. The full agent and backend remain planned.

PythonLangGraphBigQueryPub/SubPostgreSQLTerraform

PROPOSED SYSTEM / DATA + CONTROL PLANES

Find the cause.
Keep control of the fix.

TracePilot proposed architectureA failed run triggers a bounded investigator. Read-only tools collect run, schema, quality and lineage evidence. The agent produces a proposal, not an action. Policy tests, exact-version human approval and a separate staging runner control execution. Missing evidence or a failed gate stops the process. EVIDENCE PLANEREASONING PLANEPROPOSAL REQUEST ONLY ALLOWLISTED READS Run historyFailure + timingData evidenceSchema + qualityContextLineage + runbook Tool brokerScope + redact + limit READ-ONLY ACCESS INVESTIGATORAsk → inspect → explainBOUNDED REASONING LOOPCheckpoint every completed step Repair proposalArtifact + evidence STOP / ESCALATEIf unsupported TRUST BOUNDARY: AI HAS NO EXECUTION CREDENTIAL Policy + testsExact artifact Human reviewBound + expiring Staging runnerExecute → verify CONTROL PLANECode decides eligibility.A person approves the change. CHECKPOINTS / SCOPED IAM / AUDIT / RECOVERY PLAN
01 / COLLECT THE FACTSRead-only evidence

Run history, schema, quality and lineage pass through a limited tool broker.

02 / INVESTIGATEAsk. Inspect. Explain.

A bounded agent loop checks a hypothesis and writes a repair proposal.

Not enough evidence? Stop and escalate.
TEST Check the exact draftREVIEW A person approves itRUN Separate staging-only service

The AI can propose a repair. It cannot grant itself permission.

Solid lines: evidence and artifact flowDashed lines: control, review and feedbackProposed architecture
Inspect the components and engineering boundaries

01 READ-ONLY CONTEXT

Incident intakeRun failure / quality alert

Pub/Sub event becomes a deduplicated incident. Its idempotency key prevents duplicate investigations from becoming duplicate actions.

Evidence ledgerSchema + lineage + run logs

Read-only adapters collect timestamped evidence references and content digests. Sensitive row data stays outside model prompts.

Tool brokerAllowlisted reads only

Typed arguments, dataset allowlists, query cost ceilings, redaction and timeouts are enforced independently of model output. No arbitrary shell or SQL.

Versioned runbooksRetrieved as untrusted context

Guidance is evidence, not authority. Embedded instructions cannot broaden permissions. Every cited runbook version is recorded.

Incident + evidence references feed the investigation. The broker handles all tool requests.
BOUNDED AGENT LOOP MAX 6 TOOL CALLS / EXAMPLE LIMIT
Form a hypothesisWhat could explain the failure?

A planner selects the next permitted evidence request. The graph checkpoints state after each completed node.

Inspect the evidenceRequest → broker → result

Returned data is validated and linked to the hypothesis. Missing or contradictory evidence is explicit, not converted into confidence.

Draft a repairChange + evidence + test plan

Output is a typed proposal, never an executed command. It binds the artifact, action and target, and describes the expected effect and recovery plan.

Need more evidence? Loop within budget. Unsupported or exhausted? Stop and escalate.
Proposed change crosses into a separate control plane. The AI has no execution credential.

03 CONTROLLED ACTION

Policy + testsFail closed on any unmet gate

Validate the schema, target, action, evidence freshness and artifact-bound tests. Run code against isolated synthetic fixtures, not production.

Human approvalExact artifact + expiry

A reviewer approves the precise artifact digest, target and action. Any revision invalidates approval. Authorization is rechecked on resume.

Scoped runnerStaging-only first release

A separate service executes one allowlisted operation with an idempotency record. It must reconcile uncertain outcomes before retrying.

Verify + auditPost-check / stop / recovery

Compare expected and actual results. Failed verification pauses the run. Code rollback is not data rollback; data repair needs a separate approved replay.

Durable checkpointsScoped IAMAppend-only audit eventsCI + Terraform
THE CONTROL RULEeligible = evidence ∧ budget ∧ scope ∧ tests ∧ approval

Every condition must pass. A model’s confidence is not permission.

INTERACTIVE WIREFRAME / SYNTHETIC INCIDENT

Watch the guardrails work.

No agent is running. This prototype evaluates deterministic control rules in your browser.

TRACEPILOT / INCIDENT WORKSPACELOCAL PROTOTYPE
01 / EVIDENCE TRAILREAD ONLY
  1. Run metadataSchema mismatch at the mapping stage.
  2. Schema comparisoncustomer_id renamed to customer_key.
  3. Lineage checkOne curated staging table depends on the mapping.
  4. Proposed repairRestore the expected alias. Preserve the raw source.
02 / PROPOSED ARTIFACT draft-v1customer_key AS customer_idIllustrative mapping only. No query is executed.
03 / CONTROL GATES
  • Evidence complete and freshPass
  • Read-only tool budgetPass
  • Approved action and targetPass
  • Tests match this draftPass
  • Current reviewer approvalBlock
Awaiting review

Technical gates pass. A reviewer must approve this exact draft and target.

Prototype version labels represent artifact bindings. The backend design requires canonical SHA-256 digests and server-side authorization.

Try approval, then revise the draft. The old approval and test result no longer authorize the new artifact.

BUILD ROADMAP / NOT YET IMPLEMENTED

Make the reasoning inspectable.

Get specifications + reference tests
  1. 01
    Define the incident contract

    Version events, deduplicate alerts, and build failure fixtures. Reject malformed or unauthorized incident sources.

  2. 02
    Build read-only evidence tools

    Run, schema, lineage and quality adapters. Add allowlists, redaction, timeouts and query-cost ceilings.

  3. 03
    Add a bounded agent graph

    Checkpoint hypotheses, tool results and budgets. Test crash recovery and malicious instructions inside retrieved context.

  4. 04
    Bind tests and approval

    Canonical artifact digest, authenticated reviewer, target binding and expiry. Fail closed when any binding changes.

  5. 05
    Execute one staging action

    Separate runner identity, idempotency ledger, post-check and uncertain-outcome reconciliation. No production action in the MVP.

  6. 06
    Evaluate before expanding

    Measure evidence accuracy, safe abstention, forbidden-action attempts, replay behavior and cost per investigation on a fixed incident set.

What this demonstrates

Stateful orchestration, data lineage, contract design, controlled AI tool use, replay safety and observable failure handling.

The deliberate trade-off

One orchestrator with typed tools is easier to audit than a swarm of loosely controlled agents. Start narrow, prove the boundaries, then add capability.

What would count as evidence

A reproducible run, cited inputs, artifact-specific tests, approval records and verified post-conditions. No production impact or accuracy is claimed today.

Design references: LangGraph persistence ↗ and review interrupts ↗. These support the proposed design, not an implemented backend.

02 / PREDICTIVE DATA OPERATIONS

LoadCast.

See the surge.
Before it becomes a queue.

PLANNED / DESIGN BLUEPRINT

LoadCast would estimate how much data is coming and test whether the system can keep up. It shows the likely queue, the workers needed and the assumptions behind the plan.

Download the build kit
THE PROBLEM

More data arrives than a pipeline can process. A queue starts to grow.

THE IDEA

Forecast arrivals, simulate the queue and explain the capacity needed.

WHAT YOU CAN TRY

A working queue simulator. Telemetry ingestion and trained models remain planned.

PySparkBigQueryCloud StoragePythonscikit-learnTerraform

PROPOSED SYSTEM / EXPAND A COMPONENT

Know what is coming.
Plan how to handle it.

LoadCast proposed architectureTelemetry is preserved, checked and converted to point-in-time features. Training and evaluation feed an approved model registry. Inference combines fresh features with the approved model. A queue simulator tests capacity and sends an explainable scenario to a human reviewer. Actual arrivals feed later evaluation. 01 / COLLECT + CHECK02 / MODEL WHAT WAS KNOWN03 / PLAN CAPACITY TelemetryArrivals + throughput Raw archiveReplay every input Quality layerPySpark / deduplicate Point-in-time featuresOnly information known then available_at ≤ forecast_origin Train + backtestRolling origins / 12 horizons Model registryBaseline + quantile candidates PROMOTE ONLY AFTER EVALUATION INFERENCEP10 / P50 / P90 Capacity simulatorQueue + workers + warm-up Operator decisionReview assumptions first RECOMMENDATION, NOT AUTOSCALING FEEDBACK: ACTUAL ARRIVALS / DRIFT / CALIBRATION Stale input? Block.
01 / ENGINEER THE INPUTTelemetry → trusted features

Preserve raw events, remove duplicates and keep missing values visible.

Use only data available at the forecast time.
02 / ESTIMATE DEMANDBacktest before forecasting

Compare a baseline with quantile models. Keep the model and data versions together.

FORECAST Fresh features + approved modelSIMULATE Demand, workers and queueREVIEW Explain the capacity choice

Actual arrivals feed evaluation. Stale data blocks the plan.

Solid lines: data and forecastsDashed lines: model control and feedbackProposed architecture
Inspect the components and engineering boundaries

01 ENGINEER THE INPUTS

Pipeline telemetryArrivals / backlog / throughput

Collect queue and run observations with event_time, observed_at and available_at. Do not infer missing measurements as zero demand.

Raw event archiveReplayable + versioned

Append events to Cloud Storage with source identifiers, schema versions and ingestion metadata. Preserve original observations for deterministic reprocessing.

PySpark quality layerDeduplicate / align / quarantine

Use logical event keys, late-arrival rules and five-minute windows. Invalid records are quarantined with reasons and reference ids.

Point-in-time featuresBigQuery snapshot ledger

At each forecast origin, use only data available at that time. Keep a history of late corrections instead of silently rewriting the evaluation past.

Versioned features feed training and inference. Future observations must not enter either feature snapshot.
OFFLINE / LEARN + TEST
Rolling-origin evaluation

Seasonal baseline versus quantile models. Evaluate the actual 60-minute horizon, not a convenient random split.

REGISTRY / RELEASE GATE
Promote only with evidence

Version training data, features, model and calibration. Check P10 ≤ P50 ≤ P90, coverage and segment-level errors.

Inference joins the approved model with a fresh feature snapshot. A failed freshness gate blocks the recommendation.

03 FORECAST → PLAN → REVIEW

Demand distribution12 future five-minute buckets

Serve versioned P10, P50 and P90 arrival forecasts. These describe uncertain demand, not a promise about the next hour.

Capacity simulatorQueue + warm-up + constraints

Test candidate worker pools against backlog, start-up delay and measured service rates. Pointwise P90 demand is a planning stress path, not a 90% queue guarantee.

Operator reviewRecommendation, not autoscaling

Explain assumptions, limits and a resource-use proxy. A person chooses whether to act. Log decisions and compare forecasts with later observations.

Actual arrivals return to evaluation. Drift, missing data or poor calibration pause model promotion.
Schema contractsData lineageFreshness gatesCI + Terraform
THE QUEUE MODELB(t+1) = max(0, B(t) + [λ(t) − w(t) × μ] × Δt)

Backlog = unfinished events + new arrivals − processing capacity. Units: events and minutes.

INTERACTIVE WIREFRAME / CAPACITY PLANNER

Move capacity.
See the consequence.

Synthetic demand assumptions, not a trained model. The queue calculations below run locally.

LOADCAST / NEXT 60 MINUTESPLANNING SANDBOX
P50 assumptionP90 assumptionService capacity
Events / minute Scale calculated in the prototype
+5 min+30 min+60 min
Pointwise P50-to-P90 planning envelope. Not a calibrated confidence band or a joint probability guarantee.
Peak queueNot calculatedevents
Worst drain timeNot calculatedminutes, if arrivals stop
Resource proxyNot calculatedworker-minutes, not cost
Enable JavaScript to explore the local queue model. The architecture and build kit remain available without it.
Inspect all 12 calculated buckets
Synthetic P90 stress-path calculations. All rates are events per minute.
MinuteP50P90CapacityBacklogDrain min
Enable JavaScript to calculate. The build kit includes the same reference model and tests.

This is a deterministic queue simulation. It does not claim forecast accuracy, cloud savings, end-to-end latency or a production service level.

BUILD ROADMAP / NOT YET IMPLEMENTED

Engineer the time axis first.

Get specifications + reference tests
  1. 01
    Generate replayable telemetry

    Simulate daily patterns, bursts, worker changes and outages. Version the generator and preserve seed and assumptions.

  2. 02
    Build the quality pipeline

    Validate timestamps and units, deduplicate event keys, quarantine errors, and distinguish zero demand from missing observations.

  3. 03
    Create point-in-time features

    Lagged arrivals, queue depth, throughput and calendar features. Join only observations available at each forecast origin.

  4. 04
    Backtest the full horizon

    Compare seasonal naïve and quantile gradient boosting on chronological folds. Score each of the 12 horizons independently.

  5. 05
    Integrate the capacity planner

    Use empirical service rates and start-up delays. Test infeasible demand and explicit bounds before considering any automation.

  6. 06
    Ship monitoring and evidence

    Track freshness, drift, pinball loss and interval coverage. Publish reproducible results and model cards before making performance claims.

What this demonstrates

Event-time processing, feature lineage, data quality, leakage prevention, forecasting evaluation and constraint-based capacity planning.

The deliberate trade-off

A simple baseline and transparent queue model come before a complex model. A better model matters only if it makes a better operational decision.

What would count as evidence

Held-out errors by horizon, calibration by load segment, baseline comparison, replay tests and honest results under a traffic shift.

EVALUATION, NOT A RESULTPinball(q, e) = max(q × e, (q − 1) × e)

e = actual − prediction. Evaluate q = 0.1, 0.5 and 0.9 on untouched future periods. No trained-model scores exist yet.

Design reference: scikit-learn time-series validation and quantile forecasting ↗. The supplied queue simulator does not train these models.

03 / PREDICTIVE DATA ENGINEERING

SignalBench.

A forecast is useful.
A forecast you can verify is better.

WORKING LOCAL PROJECT

A complete local workflow that checks daily workload data, compares two forecasting methods and produces reports you can reproduce. Every prediction has a visible method, input and error.

Download the project Inspect generated results
THE QUESTION

Can a simple forecast help us anticipate the next batch of data?

THE ENGINEERING

Validate inputs, store them in SQL, replay earlier days and measure mistakes.

THE PROOF

Runnable code, a test suite, generated reports and a queryable database.

SignalBench implemented local workflowA daily CSV passes strict validation before SQLite storage. Two baseline methods make predictions using only earlier observations. Chronological evaluation exports metrics and outlook files with a completion manifest. Invalid input stops before output is published. INPUTCONTRACTWAREHOUSETWO BASELINESEVIDENCE Workload CSV56 synthetic daysQuality gateDates / counts / gapsSQLiteKeys + run id Weekly repeatSame day last week7-day meanRecent average Evaluate + exportBacktests / outlook CSV + JSON + MANIFEST Invalid input → stop SOURCE HASH + CODE HASH → REPRODUCIBLE RUN ID
01 / CHECK THE INPUTDaily workload CSV

Reject missing days, duplicate dates and invalid counts before writing output.

02 / REPLAY THE PASTStore. Predict. Compare.

SQLite holds the observations. Each baseline predicts using only earlier days.

MEASURE One-day and seven-horizon errorsEXPORT Reports, tables and outlookTRACE Input, code and artifact hashes
Implemented local pipeline. No cloud or live-data connection.Python + SQL + reproducible evidence
56 OBSERVED DAYS / 7-DAY OUTLOOK
ObservedOutlookEmpirical error band
Daily event count. Synthetic data with a weekly pattern.
MAE events5.43
RMSE events5.53
WAPE error4.01%
Weekly repeat: measured on 14 rolling one-day holdout predictions. Not a claim of seven-day accuracy.
DATA QUALITY IS PART OF THE MODEL

TIME IS PART OF THE CONTRACT

Build historyDays 1–28
CalibrateDays 29–42
EvaluateDays 43–56
Look aheadDays 57–63
Each historical prediction uses only earlier observations. The future outlook is not the holdout. Small-sample error bands do not guarantee future coverage.

Data pipeline

Strict CSV contract, ordered daily observations, SQL keys and transactional inserts. Invalid input stops before creating output.

pipeline.py / schema.sql

Forecast evaluation

Two transparent baselines, a separate calibration period, one-day evaluation and a seven-horizon study.

forecast.py / horizon_metrics.csv

Reproducible evidence

Source and code fingerprints identify the run. A final manifest lists output hashes. Existing outputs are never overwritten.

report.json / warehouse.sqlite3 / manifest.json
Method, uncertainty & the numbers

Separate calibration from evaluation

Days 1–28 provide initial history. Days 29–42 calibrate the error band. Days 43–56 form a separate rolling one-day holdout. Each prediction uses only earlier observations.

State what the band means

The band uses the nearest-rank 90th percentile of 14 absolute calibration errors. It does not guarantee 90% future coverage. 14 of 14 holdout outcomes fall inside the band.

Keep the claims proportionate

These are transparent baselines, not trained AI models. The seven-day outlook and one-day evaluation are different horizons. Real use needs more history, drift checks, and evaluation at the intended horizon.

MAE = mean(|actual − estimate|)

RMSE = √mean((actual − estimate)²)

WAPE = Σ|actual − estimate| ÷ Σ|actual| × 100

Seven-day synthetic outlook
DayEstimateError band
57107.099.0 to 115.0
58119.0111.0 to 127.0
59131.0123.0 to 139.0
60126.0118.0 to 134.0
61148.0140.0 to 156.0
62182.0174.0 to 190.0
63151.0143.0 to 159.0
Separate 14-day holdout
DayActualEstimate
4310297.0
44114109.0
45126121.0
46121129.0
47143138.0
48177172.0
49146154.0
50107102.0
51119114.0
52131126.0
53126121.0
54148143.0
55182177.0
56151146.0
python pipeline.py --input data/workload.csv --output my-first-runRead the engineering guide

AI-assisted portfolio implementation with synthetic data, not airline performance or a deployed SkyFlow feature. Complete within its local batch-and-evaluation scope. No trained ML model, live service or production impact is claimed.

OPTIONAL / ENGINEERING PLAYGROUNDInspect the smaller decisions.Replay, failure recovery, and version-specific release controls.

Supporting references built for this portfolio with AI assistance. Browser-local logic and synthetic inputs. Not client deployments or live cloud services.

01 / RELIABLE INGESTIONWORKING REFERENCE · SYNTHETIC DATA

A pipeline that
can safely replay.

Orders arrive twice. Some contain mistakes. Some arrive late. A report should not quietly count all of them.

The implementation
Validate each record, recognize repeated event IDs, and separate invalid or late data before publishing.
The important decision
Stage a new state snapshot before committing it. A failed run leaves the last committed state untouched.
Download source & tests Architecture notes

IMPLEMENTED IN JAVASCRIPT · NO EXTERNAL SERVICES

BEHAVIOR YOU CAN CHECKBROWSER-LOCAL
01EventsPreserve the input
02ChecksValidate + deduplicate
03SnapshotStage, then commit
04ReportAccepted orders only
IF INVALIDQuarantineKeep the record and the reason.
IF LATEReconciliation queueKeep it for a separate review.
Failed before commit? Retry from the last unchanged snapshot.
3published orders
2quarantined
1late event
FIRST RUN

3 orders published. 1 repeat skipped. 2 invalid records quarantined. 1 late event kept for reconciliation.

Accepted total: 300.00 example currency units.

Architecture, trade-offs & verification

What is implemented

Input validation, stable-ID deduplication, a fixed sample watermark, separate rejected/late outputs, and copy-on-write state. The browser and downloadable checks use the same functions.

What the tests establish

The seven-record fixture yields three accepted orders. Replay adds no new records with handled IDs. Injecting a failure before commit does not mutate the prior state.

Where the boundary is

State is in memory. This is not a distributed exactly-once guarantee. A real deployment needs durable state, a transactional sink, a conflicting-ID policy, and an implemented late-data reconciliation path.

PRODUCTION MAPPING, NOT CONNECTED HERE

Kafka or Auto Loader → PySpark validation → Delta tables → BigQuery or BI

02 / CONTROLLED AUTOMATIONWORKING POLICY · NO LIVE AI MODEL

AI can propose.
Release is earned.

A useful draft is only the beginning. Failed tests, stale approval, or an unauthorized target must stop the change.

The implementation
A deterministic release policy checks test results, version-specific approval, the allowed target, and a retry limit.
The important decision
Approval belongs to an exact version. Change the draft and the previous approval no longer counts.
Download source & tests Architecture notes

AGENTIC AI PATTERN · DETERMINISTIC POLICY LOGIC

CHANGE THE CONDITIONSNO DEPLOYMENT
Proposed versionPolicy checksDecision
  • All data checks passBlock
  • Approval matches this versionPass
  • Target is allowedPass
  • Retry limit is respectedPass
DRAFT-V3Release blocked

A data check failed. The draft cannot proceed.

This example returns a policy decision. It does not generate code, run an AI model, or deploy a change.

Architecture, trade-offs & verification

Separate proposal from permission

An agent may propose a transformation. A separate policy decides whether that exact artifact is eligible to continue. The policy does not rely on the agent saying its own work is safe.

Bound the work

Restrict the execution target and cap retry attempts. The tests reject failed checks, missing or stale approval, unauthorized targets, and exceeded retry limits.

Enforce it in the right place

A browser is not a security boundary. Production enforcement needs a trusted service, authenticated approvers, immutable artifact identities, audit records, least-privilege execution, and rollback controls.

PRODUCTION EXTENSION, NOT CONNECTED HERE

AI-generated draft → CI checks → authenticated review → controlled deployment → monitoring

02 / ENGINEERING EXPERTISE

Built for useful
business decisions.

Reliable information, controlled automation, and planning that acknowledges uncertainty.

01 / DATA PLATFORMS

Reliable data.

Organize, validate, and serve information that teams can trace and use.

See SkyFlow 02 / AI-ASSISTED OPERATIONS

Controlled automation.

Let evidence guide the investigation. Keep permissions and decisions explicit.

Inspect the diagnostic workflow 03 / PREDICTIVE FOUNDATIONS

Better-informed planning.

Compare baselines, measure errors, and expose the limits of a forecast.

Explore the backtest
Explore the full toolkit
Data engineering
Python, SQL, PySpark
Distributed processing
Apache Spark, Structured Streaming
Cloud platforms
GCP, Databricks, Microsoft Fabric
Streaming
Kafka, Pub/Sub
Storage & architecture
Delta Lake, BigQuery, data modeling
Infrastructure & reliability
Terraform, CI/CD, testing, data quality, observability

03 / PROFESSIONAL EXPERIENCE

Experience built
through engineering.

My work centers on building and maintaining pipelines so data is accurate, timely, and accessible.

PROFESSIONAL ROLE

Data Engineer

Arbre Creations

Professional role listed on LinkedIn. Explore my work history there, or connect to discuss the systems and responsibilities behind it.

View professional history (opens in a new tab)Request résumé (opens in a new tab)GitHub (opens in a new tab)

04 / ENGINEERING PHILOSOPHY

Reliability. Governance. Observability.

01

Design for the retry.

Know what happens after a failure, not only when a run succeeds.

02

Make authority explicit.

A useful AI suggestion is not permission to change a system.

03

Leave a readable trail.

Record outcomes and limits so the next engineer can reason about the system.

05 / CREDENTIALS

Knowledge, verified.

Four certifications and one academy accreditation. Original badges. Direct issuer verification.

Issuer records last checked on 4 November 2026. Follow a verification link for current status. DP-750 and DP-700 share Microsoft’s official Associate badge.