Listing Documents

An MLS feed lists each listing’s documents (seller disclosures, surveys, HOA paperwork) in the same Media array as its photos, under MediaCategory: "Document". HomeStar records every non-photo media item in the property_media table, fetches each document into shared storage so the site serves its own copy, and extracts the text so it can be processed later.

Whether a document is shown on a public page is a separate decision, gated per feed by the display policy. This page covers only storage.

---

Where documents live

What	Where
Row per document	property_media (category = 'Document'), written by the sync worker
The PDF itself	./data/documents/<listing id>/<hash>.pdf on the host, /data/documents inside containers
Public URL	{PHOTO_PUBLIC_URL}/documents/<listing id>/<hash>.pdf, served by the photos static host
Extracted text	property_media.text_content, capped at 200,000 characters with a trailing marker when truncated

The filename is the first 32 hex characters of the SHA-256 of the MLS URL. The static host caches every path as immutable for a year, which is only honest because a given path can never change content: when the MLS republishes a document it publishes a new URL, the sync worker resets the row’s fetch state, and the next fetch lands on a new file. Old files are not removed automatically.

Only PDFs are stored. A response that is neither declared application/pdf nor begins with %PDF is recorded as a failed fetch, with the content type in fetch_error, and nothing is written to disk.

---

The document worker

The document-worker compose service runs idx-api/scripts/document_worker.py on the same image as the API. Every 15 seconds when idle it selects up to 20 rows where category = 'Document', fetched_at IS NULL and fetch_attempts < 5, downloads each one (30 second timeout, redirects followed), extracts the page count and text, renders page one as a WebP thumbnail next to the file, writes everything atomically and fills the row:

Column	Meaning
local_path	Path relative to the document cache root, NULL until fetched
fetched_at, text_extracted_at	When the copy and its text were taken
content_type, byte_size, page_count	What was fetched
text_source, ocr_pages	How the text was read (pdf, ocr, image, none) and how many pages tesseract read
thumbnail_url, thumbnail_generated_at	Page one as a WebP; generated_at set with url NULL means the render failed and is not retried
fetch_attempts	Counts every try; a row stops being retried after five
fetch_error	The last failure, cleared on success

Text is read by whichever route the bytes call for. A PDF with a text layer is read with pypdfium2 (text_source = 'pdf'). A PDF whose native text is thinner than 50 characters a page or 200 in total is treated as a scan: its pages are rendered at 200 dpi and read with tesseract, at most 60 pages per document, and the OCR text wins when it is longer ('ocr'). Documents the feed serves as PNG, JPEG or WebP are stored with that extension and OCR’d directly ('image'). 'none' means nothing produced text. Spare capacity in each batch goes to rows already fetched that still need a thumbnail; --reprocess-text also re-reads rows whose text_source is NULL (fetched before OCR existed) from the file on disk, with no download.

The worker logs one line per batch (batch: 20 queued, 18 fetched, 0 reprocessed, 2 failed, 18 thumbnails, text pdf=7 ocr=9 image=2, 41.3s) and stops cleanly on SIGTERM, finishing the document in flight. It lives on the internal network only. --once runs a single batch and exits, --drain runs until the queue is empty; without either it polls forever.

Note

make promote-prod recreates document-worker together with the sync and vision workers. Like them it imports its code at start, so a deploy that skips the recreate leaves the old code running.

To see what is stuck:

SELECT listing_id, media_key, fetch_attempts, fetch_error
  FROM property_media
 WHERE category = 'Document' AND fetched_at IS NULL AND fetch_attempts >= 5;

Setting fetch_attempts back to 0 on a row puts it back in the queue.

---

Backfilling listings synced before this existed

The sync worker has always stored the full Media array in properties.raw_fields, so documents for listings synced before property_media existed are already on disk. The backfill reads them out rather than re-syncing the MLS:

# Count what would be written
make backfill-media DRY_RUN=1

# Write the rows; the document worker fetches the new ones on its next poll
make backfill-media

# The same command without the Makefile
docker compose --profile sync run --rm pyresync backfill-media
docker compose --profile sync run --rm pyresync backfill-media --dry-run --listing 26-5035

The backfill is idempotent. It uses the same upsert as the sync worker, keyed on (listing_id, media_key), so re-running it refreshes descriptions and leaves fetched documents alone unless the MLS URL changed.