Index-only scans try returning data for a query directly from the index, without accessing the heap. However, in addition to the data itself, pg needs to check if a given row in the index is visible to the transaction’s MVCC snapshot; this information is only stored in the heap, not in the index.
As a workaround, pg uses a visibility map, which stores a bit for each page in the heap; the bit is set if all rows in the page are old enough to be visible to all current/future transactions. If the visibility map’s bit is set for a row, it is definitely visible. If it is not set, it might or might not be visible, so the heap has to be visited to figure it out.
A visibility map is 4 orders of magnitude smaller than the heap it describes, so this is a significant win regardless.