The $1.4M/Year Bill Nobody Was Watching

Context

A 20-store RFID loss-prevention proof of concept at a major retailer used in-store RFID readers to track tagged products. Reader events flowed through GCP Pub/Sub into a real-time pipeline. I was the lead full-stack engineer for the integration of RFID tracking, video analytics, and the front-end applications.

Before implementation, I prepared a 17-page impact analysis on shrink at this retailer: the items most commonly stolen, theft patterns, repeat-offender behavior, and the technology's false-recognition constraints. Two findings shaped the design. First, theft was dominated by repeat groups stealing the same categories of items, so the useful output was evidence for case building rather than a stream of instant alerts. Second, the program had a potential $500–700M recovered-shrink opportunity if it could support that operating model.

The False-Flag Problem

The detection algorithm ran on premises, inside the readers, but its comparison logic did not match the hardware's sampling limit. Zebra RFID antennas, including four-dish units reading in four directions, can report about 200 labels at a time. A typical aisle held well over 1,000 tagged items.

The algorithm diffed the labels from one second against the previous second and treated a missing label as a loss event. Since the antenna could not observe the whole aisle in a single read, labels fell out of a one-second sample constantly. The system generated hundreds of false flags per second per aisle. Of every 10,000 events sent from stores to Pub/Sub, about 10 were valid.

Following the Money

No one asked me to examine the bill, but I wanted to quantify the cost of the false-positive stream. I built a GCP usage calculator in Excel that modeled event volume against Pub/Sub and pipeline pricing. It showed a ~$119K/month cost, or a ~$1.4M annual run rate for 20 stores before rollout. Nearly all of that spend was transporting noise.

The analysis also clarified the system's operational requirement. Loss prevention could not reliably interrupt a person leaving with an item; its value was in building cases from repeat incidents. Per-second real-time alerting added cloud spend without a corresponding operational response.

The Fix: Windowed Differentials

I redesigned the edge detection algorithm around time windows instead of per-second diffs:

1. Deduplicate reads into one-minute SKU sets. A minute gives the antenna about 60 chances to observe each label. Aggregating by SKU absorbs the 200-label sampling limit.
2. Compare consecutive minute sets. A SKU missing from the next minute's set has been absent from roughly 60 reads, which is meaningful evidence instead of a one-sample gap.
3. Confirm the absence for five minutes, then emit one batch event. The event retained the timestamp when each item first went missing, allowing investigators to retrieve the correct CCTV interval.

flowchart TB
    subgraph Before ["Before: per-second diff"]
        direction LR
        R1["RFID reads<br/>(200-label cap per antenna)"] --> D1["Diff vs previous<br/>second"] --> F1["Hundreds of false flags<br/>per second, per aisle"] --> P1["~10 valid per 10,000<br/>events to Pub/Sub"]
    end
    subgraph After ["After: windowed differential"]
        direction LR
        R2["Reads deduped into<br/>1-minute SKU sets"] --> D2["Minute-over-minute<br/>differential"] --> W["Missing 60 reads<br/>= likely gone"] --> C["Still missing at 5 min:<br/>one batch event,<br/>first-missing timestamp"]
    end
    Before ~~~ After

The edge filter reduced the event volume before it reached Pub/Sub. Detection quality also improved: fewer false flags and timestamps suitable for video review. The bill dropped about 99%, to ~$1.3K/month.

From Detection to Case-Building

I also built the loss-prevention dashboard. It presented likely-stolen items, the timestamps for when each product left its aisle and store, and links to the relevant CCTV footage. Camera systems logged the faces of people exiting at the time of a theft so repeat incidents could be correlated. A match across incidents marked an individual as high-risk pending human confirmation of the pattern.

My proposal extended the case-building workflow. Once human review confirmed a pattern and the individual was banned, re-entry could trigger an immediate alert to police. That created a path from documenting loss to interrupting repeat-offender activity, which was central to the $500–700M opportunity.

flowchart TB
    A["RFID event:<br/>item leaves aisle,<br/>then store"] --> B["Pull CCTV footage<br/>at exact timestamp"]
    B --> C["Log faces of exits<br/>at time of theft"]
    C --> D{"Face matches a<br/>prior incident?"}
    D -->|no| E["Retain incident log only"]
    D -->|yes| F["Flag individual<br/>as high-risk"]
    F --> G{"Pattern confirmed<br/>by human review?"}
    G -->|no| E
    G -->|yes| H["Store biometric record<br/>· ban from store"]
    H --> I["Re-entry detected"]
    I --> J["Immediate police alert<br/>before the person leaves"]
    style G stroke-width:3px

Outcome

• ~$1.4M/year saved by treating the system's economics as an engineering constraint
• Replaced a false-positive stream with fewer, more reliable detections and case-ready timestamps
• Moved the architecture from POC toward production with a sized roadmap from detection to case building