gbrain/docs/guides/multi-language-fts.md

Multi-language full-text search

GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery). The tokenizer language is configurable via the GBRAIN_FTS_LANGUAGE environment variable. Default: english.

How it works

Postgres text-search configurations control stemming and stop-word removal. GBRAIN_FTS_LANGUAGE is read by src/core/fts-language.ts and applied on both sides of the search:

• Query side — websearch_to_tsquery('<lang>', $query) in both engines (Postgres and PGLite).
• Write side — the update_page_search_vector and update_chunk_search_vector trigger functions that populate pages.search_vector and content_chunks.search_vector.

The value is validated against /^[a-z][a-z0-9_]*$/ before it is ever interpolated into SQL (tsvector functions don't accept parameterized config names). Invalid values fall back to english with a warning.

Built-in languages

Set the env var to any configuration your Postgres instance ships:

export GBRAIN_FTS_LANGUAGE=portuguese
export GBRAIN_FTS_LANGUAGE=spanish
export GBRAIN_FTS_LANGUAGE=german

List what's available:

SELECT cfgname FROM pg_ts_config;

PGLite (the embedded default engine) ships the same built-in snowball configurations as stock Postgres.

First install vs. changing language later

On first install (or upgrade), the configurable_fts_language schema migration reads GBRAIN_FTS_LANGUAGE and stamps the trigger functions with that language. After the migration has run, changing the env var alone does NOT retokenize existing rows — the migration shows as applied and is skipped. Use the explicit command:

export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run    # preview: language + row counts
gbrain reindex-search-vector --yes        # recreate triggers + backfill

The stamp survives later schema work: initSchema() — including the replay behind gbrain init --migrate-only on every upgrade — applies the schema template under the configured language, so it re-creates the trigger functions as they already are instead of reverting them to english.

The command recreates both trigger functions under the new language and backfills every existing pages and content_chunks row in batches, streaming progress to stderr. It is idempotent: re-running with the same language produces identical vectors. --json prints a machine-readable result envelope but still requires --yes (or an interactive confirm).

No cache purge is needed. The resolved language is part of the query-cache key, so rows written under the previous language are unreachable after the switch — searches read the retokenized index immediately instead of being served pre-switch results for up to search.cache.ttl_seconds. Switching back reaches the original rows rather than rebuilding them.

Recipe: accent-insensitive Portuguese (pt_br)

Brazilian Portuguese content often mixes accented and unaccented spellings ("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via the unaccent extension, then stems with the portuguese snowball dictionary:

CREATE EXTENSION IF NOT EXISTS unaccent;

CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);

ALTER TEXT SEARCH CONFIGURATION pt_br
  ALTER MAPPING FOR hword, hword_part, word
  WITH unaccent, portuguese_stem;

Then point GBrain at it:

export GBRAIN_FTS_LANGUAGE=pt_br
gbrain reindex-search-vector --yes

Note: custom configurations require a real Postgres instance (e.g. the Supabase engine). The config must exist BEFORE the migration or the reindex command runs, or Postgres will reject the trigger recreation with text search configuration "pt_br" does not exist.

Caveats

• One language per brain: the setting is global to the database, not per-source. Mixed-language brains should pick the dominant language (the vector-search arm is language-agnostic and covers the rest).
• Keep GBRAIN_FTS_LANGUAGE set consistently in every environment that writes to the brain (CLI shells, MCP server, cron jobs) — a writer without the env var tokenizes new rows in english until the next reindex.
• Interrupted reindex: the trigger flip commits before the backfill, so a reindex-search-vector run killed mid-way (crash, full disk, SIGKILL) leaves new writes in the new language and un-backfilled rows in the old one — keyword search then matches only part of the corpus. The command records an in-progress marker plus a per-batch checkpoint, gbrain doctor fails with fts_reindex_incomplete until the run completes, and re-running with the same GBRAIN_FTS_LANGUAGE resumes from the checkpoint rather than starting over. Budget minutes, not seconds, on brains with 100K+ chunks.
• CJK (Chinese / Japanese / Korean): none of the built-in snowball configurations can tokenize CJK text, so the FTS arm would return nothing for CJK queries. Both engines detect CJK queries and route them to a term-by-term ILIKE fallback with term-frequency ranking instead (shared SQL in src/core/search/cjk-keyword-sql.ts). The fallback is correct but not index-accelerated: it scans content_chunks.chunk_text, so latency grows with corpus size. Note the routing is query-driven: any query containing CJK characters takes the fallback today, even on a Postgres instance with a CJK-aware extension (pgroonga / zhparser) installed. Wiring a CJK-capable GBRAIN_FTS_LANGUAGE config past the fallback is a filed follow-up.