Listing Gallery Images

A listing detail page shows every photo the listing has. The MLS publishes each one in exactly two variants: the original, around 1.5 MB at up to 3000x2250, and a 5.6 KB thumbnail too small to use for anything. On prod listing 26-9577 that is 74 photos and 110 MB of image bytes for a single page view, slow enough that the smoke suite timed out with 260 aborted image requests.

HomeStar derives its own renditions and serves them from the photos host, so a visitor never loads an MLS original. This page covers where they live and what fills them. Grid tiles are a different derivative with different sizing — see the card images described under Data Management.

The two widths

The page shows a photo at two sizes and nothing between them, so there are two renditions and no more.

Width	Where it is used	Roughly
400	The strip under the hero and the grid view, always visible	~30 KB
1280	The photo that is open in the viewer, one at a time	~220 KB

Both are WebP, quality 78 (PHOTO_GALLERY_QUALITY), width-constrained with the height following the aspect ratio. Nothing is upscaled: a 900px original asked for at 1280 comes back at 900 rather than a soft enlargement.

What	Where
The files	`./data/photo-cache/g/{aa}/{bb}/{key}-{width}.webp` on the host, `/data/photo-cache/g/...` inside containers
Public URL	`{PHOTO_PUBLIC_URL}/g/{key}-{width}.webp` — flat, the host puts the directories back
Key	The SHA-256 of the MLS photo URL, the same key the card images use
Key → URL	`photo_cache_keys`, so a request carrying only a key can be resolved

The public URL has no directories in it and the storage does, because 160k files in one directory is legal and miserable to list, back up or rsync. The photos host rewrites /g/{key}-{width}.webp to the fanned-out path; only the Caddyfile and services/photo_cache.py know that layout.

Caching is a year, immutable, for the same reason the rest of the host is: the path is a hash of the source URL, so it cannot change content. When the MLS republishes a photo it publishes a new URL, which derives to a new key.

Warmed, or derived when asked for

Only the 400px width is derived ahead of time. Every listing page shows the whole strip immediately, so those have to be ready. The 1280 is one photo per visitor rather than all of them, and warming 160k of them would cost roughly 35 GB against 47 GB free on the host, so it is derived the first time somebody opens that photo and free for everyone after.

That is what the fallback in photo-host/Caddyfile is for. The host serves /g/* off disk when the file is there, and when it is not it proxies to the API at /api/photos/g/{key}-{width}.webp. The API looks the key up, fetches the MLS original once, writes both widths from that one fetch, and answers the request. Concurrent requests for the same key queue behind one lock, so a page opening ten photos at once does not pull ten copies of the same original.

A key that names no photo is a 404, as is a width other than 400 or 1280. Neither is cached.

The warm worker

The gallery-warm-worker compose service runs idx-api/scripts/gallery_warm_worker.py on the same image as the API. It walks the displayable feed listing by listing, records every photo’s key in photo_cache_keys so the on-demand path can resolve one, and derives the 400px rendition for each.

There is no “warmed” flag. The worker keeps a cursor, walks in listing_id order and wraps to the top when it reaches the end; a photo whose file is already on disk is skipped with one stat. A re-walk is therefore nearly free, and a listing that gains photos, or whose photos are republished at new URLs, is picked up on the next pass without anything having to notice.

# Default: 5 listings a batch, 30s between passes, 400px only
uv run scripts/gallery_warm_worker.py

# Pre-warm the opened-photo width for a subset, then stop
uv run scripts/gallery_warm_worker.py --widths 400,1280 --batch-size 50 --once

# Walk the feed once and exit (cron slot)
uv run scripts/gallery_warm_worker.py --drain

One line per batch:

batch: 5 listings, 174 photos, 40 derived, 134 cached, 0 failed, 174 indexed, 1.2 MB written, 31.4s

It stops cleanly on SIGTERM, finishing the photo in flight, and lives on the internal network only.

The byte budget

--max-bytes is the guard that keeps the cache from filling the volume Postgres lives on. It defaults to PHOTO_CACHE_MAX_BYTES (20 GB). When the photo cache directory passes it the worker stops deriving and logs why; the site keeps working, it just derives on demand instead of ahead of time.

The size is measured from disk at startup and at the end of each pass, with bytes written added in between — walking a 160k-file tree every batch would cost more than the derivations do.

The arithmetic behind the default, at the 160,016 photos prod holds today. The per-photo figures are extrapolated from the card derivative, which was measured across the CdA feed at ~28 KB for 400px at quality 75:

	Per photo	Whole feed
400px WebP	~30 KB	~4.6 GB
1280px WebP	~220 KB	~34 GB

The 400s fit comfortably in the 47 GB the host has free. Both widths together do not, which is the entire reason the 1280 is on demand.

Stale data on prod

./data/images/ on the prod host holds about 2.3 GB of photos from the retired KroetchMock feed. Nothing serves them and nothing references them: they are not the photo cache, which lives in ./data/photo-cache/. An operator can remove the directory to reclaim the space. It is left in place here rather than deleted by a deploy, because a deploy quietly removing 2.3 GB of anything is a worse habit than the 2.3 GB.