Facial recognition has been deployed in security systems, phones, and border checkpoints long enough now that we can stop treating it as emerging technology and start talking about what actually happens when these systems run. The process is more fragile than marketing materials suggest, and understanding where things break down matters if you’re working with or relying on these systems.
At its core, facial recognition is a pattern-matching problem. A camera captures an image, the system extracts data about facial geometry, and that data gets compared against a database or a reference image. This sounds straightforward until you start dealing with real lighting, real angles, real faces, and real databases at scale. The gap between a controlled demonstration and a system running in an airport or on a phone becomes obvious quickly.
Capture and Initial Processing
Everything starts with image capture, and this is where the first set of constraints appears. The camera needs enough resolution to pick up facial landmarks – the distance between eyes, the shape of the jawline, the position of the nose – but not so much that processing becomes computationally expensive. Most systems work with images between 100×100 and 600×600 pixels. Too small and you lose detail; too large and you’re wasting processing power on information that doesn’t improve matching accuracy.
Lighting is the first real obstacle. A face photographed in bright sunlight produces a completely different image than the same face under fluorescent office lighting or at night with infrared illumination. The system has to normalize for this. Modern systems apply histogram equalization or adaptive contrast stretching to even out brightness and shadow across the image. It helps, but it’s not perfect. I’ve watched systems struggle with faces that are half in shadow or backlit by a window. The normalization can actually make things worse in those cases, creating artifacts that confuse the matching algorithm.
Head angle matters significantly. A face photographed straight-on is easier to match than a profile or a three-quarter view. Most systems can handle mild angle variations – maybe 30 degrees in any direction – but beyond that, accuracy drops. This is why border control systems often ask people to look directly at the camera. It’s not just procedure; it’s a technical requirement. Some newer systems use 3D reconstruction to handle wider angles, but that requires either multiple cameras or depth sensors, which adds cost and complexity.
Feature Extraction
Once the image is captured and normalized, the system extracts facial features. This is where the actual intelligence lives. The algorithm identifies key landmarks: the corners of the eyes, the tip of the nose, the edges of the mouth, the outline of the face. These aren’t just pixel coordinates. The system calculates relationships between these points – distances, angles, ratios – and converts them into a numerical representation called a face embedding or face vector.
Modern systems use deep neural networks trained on millions of faces to learn which features matter for distinguishing one person from another. These networks don’t work the way older algorithms did, where engineers manually defined what features to measure. Instead, the network learns through exposure to training data what patterns reliably identify individuals. This is more powerful but also more opaque. You can’t always explain why the network considers two faces similar or different.
The embedding is typically a vector of 128 to 512 numbers, depending on the system. Think of it as a mathematical fingerprint of the face. Two embeddings from the same person, captured at different times and under different conditions, should be close to each other numerically. Two embeddings from different people should be far apart. The quality of the embedding determines how well the matching will work later.
Matching and Comparison
When a new face is captured, the system generates a new embedding and compares it against either a reference embedding (one-to-one matching, like unlocking a phone) or against a database of embeddings (one-to-many matching, like searching a watchlist). The comparison is a distance calculation – usually Euclidean distance or cosine similarity. The smaller the distance, the more likely the faces belong to the same person.
Here’s where thresholds become critical. The system doesn’t say “this is definitely the same person.” It says “the distance is 0.42.” Is that a match? That depends on the threshold you set. Set it too low and you get false negatives – the system fails to recognize someone who should match. Set it too high and you get false positives – the system matches two different people. This threshold is a tuning parameter, and different applications set it differently. A phone unlock might use a strict threshold (low false positive rate) because you’re only comparing against one reference image. A border control system might use a looser threshold to catch more potential matches, accepting more false alarms that human officers can then investigate.
The threshold decision reflects a policy choice, not a technical truth. There’s no objectively correct threshold. A 99% accuracy rate means nothing without knowing what the threshold was and what the cost of false positives versus false negatives actually is in that context.
Real-World Failure Modes
After working with these systems across different deployments, certain failure patterns appear consistently. Aging is one. A face embedding trained on people in their 20s and 30s often performs poorly on older faces. The feature distributions are different. Significant weight gain or loss changes facial geometry enough to degrade matching. Facial hair, glasses, and makeup can all reduce accuracy, though modern systems handle these better than earlier versions.
Demographic variation is a documented issue. Most facial recognition systems perform better on lighter skin tones than darker skin tones, particularly in matching accuracy. This isn’t accidental. The training data tends to be skewed toward lighter skin tones, and the network learns patterns that are more discriminative for those faces. It’s a data problem, not a fundamental limitation of the technology, but it’s a real problem that persists in deployed systems.
Database quality matters enormously. If you’re doing one-to-many matching against a large database, the quality of the reference embeddings in that database determines what’s possible. Embeddings generated from poor-quality photos or from photos taken years ago may not match well with current captures. I’ve seen systems where the reference database was generated from mugshots taken under terrible lighting, and the matching performance was correspondingly poor.
Temperature and infrared systems introduce their own quirks. Thermal cameras see heat distribution, not visible features. A face with a fever looks different than a normal face. Glasses reflect infrared differently than skin. These systems work, but they’re solving a different problem than visible-light systems, and the accuracy characteristics are distinct.
Spoofing – presenting a photo or video of someone’s face to fool the system – is possible with most systems if the attacker has high-quality material and the system isn’t specifically designed to detect it. Liveness detection (checking that the face belongs to a living person in front of the camera) helps, but it’s not foolproof. Methods include checking for eye movement, asking the person to perform an action, or using depth sensors. Each approach has tradeoffs between security and user friction.
Practical Deployment Considerations
When facial recognition systems are deployed at scale, operational reality quickly diverges from lab performance. A system that achieves 99% accuracy on a test set might achieve 85% accuracy in an airport because of lighting variation, crowd movement, and the sheer volume of faces being processed. The system has to work in real time, which means less processing per face, which means lower accuracy.
False positive rates become a serious issue with large databases. If you’re searching a database of 10 million faces and your false positive rate is 0.1%, you’ll get 10,000 false matches. A human operator then has to review these candidates, which is expensive and slow. This is why large-scale systems often use facial recognition as a first filter, narrowing down candidates for human review rather than as a final identification method.
System updates and retraining are ongoing. As the network learns from new data or as the training approach improves, the embeddings change. This means reference databases have to be regenerated periodically. A face that matched yesterday might not match today if the embedding algorithm changed. This creates operational complexity that isn’t always visible to users.
The technology works. It’s genuinely useful for certain applications. But it works within constraints that matter. It’s not magic, and it’s not infallible. It’s a statistical system with known failure modes, demographic biases, and operational limitations. Understanding those limitations is as important as understanding what the technology can do.
