Computer Vision Models Die in Production. Here's Why.

A computer vision model that scores 94% mAP against a validation set and then falls apart within a month of deployment is one of the most common and most avoidable failure patterns in applied CV. It's rarely a modeling problem. By the time the model is training, most of the outcome has already been decided by decisions made about data collection, months earlier.

The validation set lied

The single biggest cause: the validation set was drawn from the same narrow conditions as the training set — same camera angle, same lighting rig, same time of day — so it measured how well the model memorized those conditions, not how well it generalizes. A defect-detection model trained entirely on daytime-shift footage will quietly degrade on the night shift, where the lighting, motion blur, and shadow patterns are different enough to shift the input distribution without anyone noticing until yield numbers move.

Domain shift is the default, not the exception

Anything mounted in the real world drifts: a camera gets bumped and its angle changes by three degrees, a new batch of raw material has a slightly different surface texture, a lens accumulates dust. Models trained without exposure to that variation treat it as out-of-distribution input, and confidence scores that looked reliable in testing become meaningless. Production CV systems need either enough real-world variation in training data to be robust to this drift, or a monitoring layer that flags when the input distribution has moved — ideally both.

• Collect training data across every shift, season, and camera position the model will actually see.
• Hold out a validation set from a different time window than training, not just a random split of the same window.
• Log confidence distributions in production and alert on drift, not just on outright failures.
• Re-train on a cadence tied to how fast the physical environment actually changes, not a fixed calendar schedule.

The annotation quality ceiling

The other quiet killer is annotation inconsistency. Two labelers who disagree on where a bounding box edge belongs, or on what counts as a borderline defect, inject noise that no amount of model architecture can train around. Before touching hyperparameters, it's worth measuring inter-annotator agreement on a sample — if two people can't agree on the label, the model has no chance of learning a stable decision boundary.

None of this is a reason to avoid computer vision in production — it's a reason to budget the data pipeline and monitoring as real engineering work, not a preprocessing step that happens once before the "real" work of training begins.