The Batch Job That Was Silently Corrupting $10M/Day

Context

I was the lead developer across multiple supply-chain applications on a team of eight at a major retailer. Our area owned nine applications: eight admin dashboards and one batch job. Logistics administrators used the dashboards to define product-routing changes across the network. Those changes were persisted to a work-in-progress synchronization database.

The batch job was the control point between that editable data and downstream execution. It read pending route changes, applied route-validation rules, recorded validation errors in the work-in-progress database, and wrote approved routes to the supply-chain synchronization database. That database was the source of truth for downstream systems.

flowchart LR
    D["8× Supply-Chain Logistics<br/>Admin Dashboards"] -->|route changes| WIP[("Supply-Chain<br/>Synchronization<br/>WIP Database")]
    B["Supply-Chain<br/>Validation Batch"] -->|reads & writes errors| WIP
    B -->|writes validated routes| SYNC[("Supply-Chain<br/>Synchronization Database<br/>(source of truth)")]

The Problem

The failure mode was silent. A route with an uncaught validation error did not throw an exception, fail the batch, or trigger an alert. It was written as approved data and propagated downstream.

For multi-stop routes, bad data could turn into trucks arriving at distribution centers with loads a facility did not expect. The operational response involved manual triage, congestion, and cascading delays across the network. The business exposure was estimated at $10M+/day. During onboarding, route validation was treated as a critical safeguard because the batch job was the last validation boundary before synchronization.

flowchart LR
    T["🚚 Multi-stop truck"] ==>|"validated ✔"| A["📦 Route stop A"]
    A ==>|"unvalidated ⚠"| S["📦 Route stop B"]
    S -.- N["Uncaught validation errors:<br/>$10M/day and congestion, confusion,<br/>and chaos at distribution centers"]
    style N fill:transparent,stroke-dasharray:4 4

How I Found It

Three months in, I was still running my own onboarding curriculum. I worked through every application our team owned, mapped the codebase structure, traced the data path through each system, and studied the synchronization schema until I understood the route lifecycle end to end. I then audited the system one application at a time, following each dashboard change from its write to the work-in-progress database through the batch job's validation and persistence steps.

When I reached the batch job, I traced the validation call in the service layer and found a scope defect: the function evaluated only the first stop in each route. Every stop after the first was never passed through validation. The job could therefore persist a multi-stop route as approved even when a later stop violated a rule. Those later-stop failures produced neither an error record nor a batch failure, which explained why the defect had remained invisible.

The Fix

I showed my manager the control-flow path that omitted later stops, wrote the ticket, implemented the fix, and updated the supporting tests that same day. The fix ensured that validation covered every stop in a route before the batch persisted it as approved data.

The application had been in production for more than three years. No alert or incident had identified the problem because the failure mode was not an exception; it was missing validation coverage.

Outcome

• Closed a $10M+/day exposure that had existed in production for more than three years
• Restored validation coverage across every stop in a multi-stop route before downstream synchronization
• Found the defect through an end-to-end data-flow and code-path audit, not through an alert
• Also on this team: modernized the frontend from AngularJS to Angular for performance and maintainability