Ask for a square thumbnail of a landscape photograph and something has to decide which part to keep. The default answer everywhere is the geometric centre, which is right often enough that the failures are surprising when they happen — and the failures are the ones users see.
Why the centre fails
A centre crop assumes the subject is in the middle. Photographs regularly disagree:
- Anything composed on the rule of thirds puts the subject a third of the way in, by construction. That is most photography that has had any thought applied.
- Group shots spread faces across the frame; the middle is a gap between two people.
- Product photography with copy space deliberately leaves empty area for text. Crop to the centre and you keep the empty half.
- Tall images cropped to wide lose the top and bottom, which on a portrait is where the head is.
The rate matters more than the examples. If one image in twenty crops badly and you have a catalogue of ten thousand, that is five hundred bad thumbnails, and nobody is going to find them by looking.
What saliency actually predicts
Saliency detection is a small neural network trained on where people's eyes go when shown an image. The output is not a list of objects — it does not know it is looking at a dog. It is a grayscale map the same shape as the image, bright where a viewer's attention is predicted to land and dark where it is not.
Given that map, choosing a crop is no longer a guess. Slide the target window across every valid position, sum the saliency inside it, and take the best one. It is a search over a one-dimensional range for a fixed aspect ratio — cheap arithmetic once you have the map.
The tie-breaker that stops it looking odd
Pure "maximum saliency wins" produces an unsettling failure of its own. On an image where the map is nearly flat — a texture, a landscape, an empty wall — the winning window is decided by noise, so two near-identical photographs can crop to completely different places.
The fix is a mild preference for the centre, applied as a weight rather than a rule: full strength in the middle, tapering towards the edges. When there is a clear subject the saliency overwhelms it and the crop goes where the subject is. When there is nothing to find, the weight breaks the tie and you get the centre crop you would have got anyway.
The size of that bias is a genuine tuning decision. Too little and flat images crop randomly; too much and it stops moving for real subjects near the edge of the frame. It is worth checking against your own library rather than accepting anyone's default, including ours.
Inference is the expensive part — cache the decision
Running a network is the most costly thing in a transformation pipeline, and it is the part that does not need repeating. The crop for a given image and target ratio does not change: the source image is the same and the map is deterministic.
So the thing worth caching is not only the rendered output but the crop rectangle. Compute it once and every future rendition at that aspect ratio — every width, every format, every quality — is an ordinary resize.
This also means the cost is per image, not per request, which is the difference between a feature you can afford and one you cannot.
Know what it is not
Three honest limits:
- It is a prediction, not a decision. The model has no idea what your picture is for. It will happily centre on the most visually striking thing in a product shot even when that is a reflection.
- Art direction still wins. For a hero image on a landing page, a human choosing the crop beats any model. Automation is for the ten thousand images nobody will ever open individually.
- Faces are a different tool. If what you need is "always keep the face", a face detector answers that question directly. Saliency often lands on faces because people look at faces, but it is not promising to.
Used for what it is good at — bulk thumbnails where the alternative is a fixed rule that is sometimes badly wrong — it turns a visible, unpredictable failure into a rare and mild one. That is the whole of the claim.