The reprocess Worker

What it is for

Changing the embedding model invalidates every vector, which is what this exists to repair — reprocessing a corpus is the walkthrough, pithy vector starts it, and it runs as a durable job like every other in the kit.

You changed the embedding model. Every vector already in the index was produced by the old one, and a query embedded with the new model is asking a question in a language the corpus does not speak.

VECTOR_REPROCESS re-embeds the corpus. By default it selects the documents whose stamped model differs from the configured one; it can also be told to re-embed everything.

Keyset pagination, and this is the correctness argument for the whole thing

The default pass selects rows whose model differs from the configured one — and then sets that model on the rows it writes.

So the result set shrinks underneath the scan.

Reading id > cursor in id order cannot skip and cannot repeat, whatever the predicate does to the rows behind the cursor. That is the entire reason for the choice.

Each page is one durable step

Workflow steps are journalled, so an instance that dies at page 4,000 resumes at page 4,000 with the same cursor rather than re-embedding four million documents.

The step names are derived from a page counter, so a replay asks for the same steps in the same order — which is what makes the journal usable at all. A step name that varied between runs would make every resume a fresh start.

The one thing it does not pick up

A row written during the run whose id sorts behind the cursor is not seen.

That is the honest cost of keyset order, and it is not a problem in practice: a document written after the model changed is embedded with the new model by the write path anyway. The pass exists for the backlog, and the backlog does not grow behind it.

Everything is a seam

The step runner, the document store, the index and the AI binding all arrive as parameters. The Workflow entrypoint is a thin shell that supplies the real ones.

Which means the whole orchestration — including resuming mid-run — is tested with fakes and no network. Resume behavior is the part of a long-running job that is hardest to verify in production and easiest to get wrong, so it is the part most worth being able to test at all.

Running it

Upserts go out in batches bounded by the binding’s 1,000-vector ceiling, and each written row is stamped with the new model and an updatedAt from an injected clock.

Because the stamp is what the default selection reads, an interrupted run leaves the corpus in a coherent partial state: the documents already done have the new model and will not be selected again, and the rest are still waiting. Re-running picks up exactly what is left.

ESC