The brightness of a patch on the retinal image depends not just on the pigmentation of the surface in the world but on the intensity of the illumination falling on it: a gray patch could have arisen from a black surface illuminated by a bright light or from a white surface illuminated by a dim one.
It also depends on the brightness of adjacent regions.
