Add Vector

pithy add vector

No required prerequisites. Add auth if it is not already there — search results are scoped to a caller.

What lands in your repo

apps/<worker>/pithy.config.ts gains the registration, and the index schema is yours to write:

vector({
  indexes: {
    docs: {
      dimensions: 768,
      metadata: {
        ownerId: { type: "string", filterable: true },
        publishedAt: { type: "number", filterable: true },
      },
    },
  },
  defaultTopK: 10,
}),

apps/<worker>/wrangler.jsonc gains two bindings, in every environment stanza:

Two are not written here. The vectorize entry needs an index_name provisioning mints, and VECTOR_REPROCESS is a Workflow whose deployed name is per environment. Both arrive with provision.

One migration runs, creating the corpus tables.

Declaring filterable metadata is the decision that matters

This is the whole reason the capability exists, so it is worth spending a minute on before you write the schema.

Mark a field filterable and it gets a metadata index. Leave it unmarked and it is stored but cannot be filtered on — and a filter naming it is a compile error rather than a silent partial result.

Two hard limits, both Vectorize’s:

• Ten metadata indexes per index. Choose deliberately; you cannot have them all.
• A metadata index created after vectors were written covers none of them, and there is no backfill.

That second one is why this is a provisioning-time decision wearing a query-time costume. Adding a filterable field to a live index is not an edit — it is a rebuild.

Think about what your users will filter by before the first write: owner, date range, type, visibility. Those are the four that show up in almost every product.

Then provision

pithy vector provision --env dev

That creates the index, creates every metadata index the schema marks filterable, and waits for each one to go live before deploying the Worker that writes vectors — because a write landing before an index exists is permanently unfilterable.

Then it deploys the reprocess Worker and writes two things into your config: the environment’s vectorize and workflows bindings, and a VECTOR_PROVISIONED var recording exactly what it observed.

That var is the boot check

The Worker compares its declarations against that record at startup and refuses to serve on drift.

It is the only way somebody who edits the schema and deploys without re-provisioning finds out, because Vectorize answers such a filter with partial results and no error. Failing at startup with the field named is a much better afternoon than a search that quietly returns half of what it should.

The var is written last, and only on success, so it never claims more than provisioning got done.

When you have to reset

Adding a filterable field to an index that already holds vectors is the case. There is no backfill, so:

pithy vector reset --env staging --confirm-reset "yes, i really want to reset staging"

That deletes the index, rebuilds it through the ordinary provision path, and re-embeds the corpus from D1. Only what your D1 holds comes back — which is the argument for keeping the corpus there rather than treating the index as the store of record.

The phrase names its environment, so one typed for staging cannot be pasted into a command targeting production. dev is free.

Changing the embedding model

The model is pinned per index for writes and queries alike, because a query embedded with a different model than the vectors it searches produces results that look plausible and are wrong.

Changing it means re-embedding:

pithy vector reprocess --env staging --all

Without --all, only the documents whose model differs from config are re-embedded, which is usually what you want. --filter narrows further, as a JSON metadata object — parsed in your terminal, so a malformed filter fails there rather than inside a running Workflow.

Chunking stays yours

The package embeds, indexes and queries. How a long document is split into embeddable pieces depends on what your documents are, and a wrong default produces bad search that looks like a model problem.

Write the chunks into the corpus as separate documents with shared metadata, and the filter you declared is what groups them back together.

Check it worked

pithy doctor reports vector under the Worker’s health. After provisioning, an index carrying a metadata index the config no longer declares is reported rather than removed — it still costs one of your ten slots.