Bhavya Upadhyay

Problem & constraint

Answer one deliberately modest question honestly: given everything knowable on the day a company files its 10-K, will next year's revenue be higher? On real, messy financial data, at zero ongoing cost.

Architecture

Challenges & solutions

• Real financial-data landmines, caught by tests

  XBRL tag migrations, silent API pagination truncation, REIT accounting quirks, and filer sign errors were each caught by an automated dbt data test. The suite caught four real extraction bugs.

• Margins above 1.0 — scoped out, not papered over

  Financials (~70 companies) produced computed margins above 1.0 because bank and insurer revenue isn't captured by the XBRL revenue concepts used. I excluded them and scoped every claim to the S&P 500 ex-Financials rather than ship an indefensible number.

• Hallucination risk, designed out

  Agents use code-supplied provenance, so the model cannot fabricate a citation, and every memo is independently scored by an LLM judge.

Results

0.726

Test ROC-AUC vs 0.500 baseline — framed as a ranked screen, not a classifier (83% positive base rate makes raw accuracy meaningless)

498 / 503

current S&P 500 constituents loaded — 7,863 XBRL rows, 4,758 parsed 10-Ks, 11,583 FRED observations, 6,234 labeled training rows

39 / 29 / 16 / 16

SHAP attribution % — fundamentals / macro regime / sector / filing language

~4.6–5 / 5

Agent memos scored by the LLM judge across 5 companies, 3 sectors · 124 tests passing, 93% coverage

Tech stack

Problem & constraint

Build a re-runnable, test-gated pipeline over messy government CSVs (BTS files ship ~110 columns and occasionally re-publish corrected rows) that never duplicates data and never lets bad data reach the marts.

Architecture

Challenges & solutions

• Idempotent loads

  Delete the (year, month) partition before COPY INTO, so re-runs and backfills never duplicate rows.

• Schema-drift tolerance

  MATCH_BY_COLUMN_NAME on COPY with a target table that declares only the 25 columns used, so the pipeline survives BTS adding or reordering columns.

• Test gate inside the DAG

  dbt build runs in the DAG — a failing test fails the run, so bad data never silently reaches the marts.

• Dedup for re-published rows

  Surrogate key plus row_number() dedup in staging, because BTS re-publishes corrected rows for the same flight leg.

• Modern auth, secrets at runtime

  RSA key-pair auth (Snowflake deprecated single-factor password auth in late 2025); the private key loads at runtime and is never stored in config.

• Kafka for decoupling, not buzz

  Kafka makes the poller and loader independent processes — decoupling, not a throughput claim bolted on.

Results

26 / 26

dbt tests passing, including two custom singular tests encoding real business rules

2

business-rule tests: no negative distance; and 15+ min late flights must reconcile BTS delay-cause buckets to arrival delay within 5 min

CI

sqlfluff lint + dbt build on an ephemeral schema, on every push

Tech stack