Stage 3 · Data platform case
Reconciliation platform
Balances held with several outside parties never quite matched the internal ledger. I designed and built the platform that compares them every business day and sends real differences to a person.
One business day, compared
Four steps, in this order. The day is unified, counts are compared, then amounts, and each account lands in one of four states. Everything here is sample data.
How it decides
One business day
- Outside parties cut their day at different times, so both sides are first moved onto the same business day
- Nothing is compared across days
Counts before amounts
- If the number of records differs, the amounts are not compared at all
- A count gap is the cheapest alarm there is
Four outcomes
- matched, acceptable, mismatched, carried
- Acceptable means inside a tolerance. Carried means it cannot be compared yet
Real differences go to a person
- A mismatch becomes an approval task with the evidence attached
- The platform never writes the difference away
The day about 40,000 rows became about 91
One upstream sub-account endpoint also returned the main account's balance. Deduplication ran after the sub-account rows were expanded, so distinct rows looked like repeats and were dropped. Switch the order and replay it.
- Fetch40,000
- Expand sub-accounts40,000
- Deduplicate91
- Load91
Each sub-account response also carried the main account’s balance. Deduplicating after the expand step treated distinct rows as repeats, and the load kept almost nothing.
- Fetch40,000
- Deduplicate40,000
- Expand sub-accounts40,000
- Load40,000
Deduplicating first works on the raw responses, so only the repeated main-account balance is removed. Every sub-account row survives to the load.
Fix 1 order
Deduplication moved ahead of the expand step, so it only sees raw responses and removes only the repeated main-account balance.
Fix 2 deploy
The job can be deployed only from the repository. A hand-edited copy cannot reach production.
What I took from it
- Order of steps is part of correctness. The same steps in a different order gave a different answer.
- Compare counts first. A row count that falls by a few hundred times is visible long before any amount is.
- Close the route that let an unreviewed copy run, not just the bug.
Count the rows before you trust the totals.
Next
Stage 3 · data platform · public
Data on EKS
One identity store and one policy store for every engine.