# Full Text Search

Search pasted documents with BM25 or TF-IDF ranking, stemming, stopwords, phrase and exclusion queries, and highlighted snippets.

## Run it

- **CLI:** `gizza tool full-text-search "Refund policy
Refunds take five days.
---
Shipping guide
Orders ship within two days." 'query=refund -shipping'`
- **Web:** https://gizza.ai/tools/full-text-search/
- **Agents:** machine-readable descriptor (parameters JSON Schema) at https://gizza.ai/tools/full-text-search/tool.json

## Inputs

- `corpus` — Documents _(field)_
- `query` — Query _(field)_
- `separator` — Document separator _(field)_
- `algorithm` — Ranking _(field)_
- `stemming` — Porter stemming _(field)_
- `stopwords` — Remove stop words _(field)_
- `match` — Term matching _(field)_
- `prefix` — Prefix search _(field)_
- `max_results` — Max results _(field)_
- `snippet_words` — Snippet words _(field)_
- `k1` — BM25 k1 _(field)_
- `b` — BM25 b _(field)_
- `title_boost` — Title boost _(field)_
- `output` — Output _(field)_

## Output

- Search results (text)

## Query parameters

Open the tool pre-filled and auto-run via URL:

- `corpus` — Documents
- `query` — Query
- `separator` — Document separator
- `algorithm` — Ranking
- `stemming` — Porter stemming
- `stopwords` — Remove stop words
- `match` — Term matching
- `prefix` — Prefix search
- `max_results` — Max results
- `snippet_words` — Snippet words
- `k1` — BM25 k1
- `b` — BM25 b
- `title_boost` — Title boost
- `output` — Output

Example: `https://gizza.ai/tools/full-text-search/?corpus=Refund%20policy%0ARefunds%20take%20five%20days.%0A---%0AShipping%20guide%0AOrders%20ship%20within%20two%20days.&query=refund%20-shipping&separator=dashes&algorithm=bm25&stemming=true&stopwords=true&match=any&prefix=true&max_results=10&snippet_words=30&k1=1.2&b=0.75&title_boost=2.0&output=text`

---

## About this tool

Full Text Search turns a pasted corpus into a one-shot local search engine. Split documents with `---` lines, blank lines, or form-feed characters, enter a query, and the tool ranks matching documents with BM25 or classic TF-IDF. Results include document numbers, scores, titles, matched terms, and keyword-in-context snippets with hits wrapped in `«…»`.

The query parser supports the search patterns people expect from document search: bare terms, `"quoted phrases"` that must appear adjacent, `-term` exclusions, AND/OR term matching, optional prefix search, English Porter stemming, and stop-word filtering. The first non-blank line of each document is treated as a title and can be boosted with `title_boost`.

This tool is stateless and local. It builds the index from the paste on each run; it does not store or incrementally update an index. For PDFs, DOCX files, or EPUBs, extract the text first with a document extraction tool, then paste the text here.

## Worked examples

Search three policy snippets with BM25:

```text
query: refund
corpus:
Refund policy
Refunds take five business days.
---
Shipping guide
Orders ship within two days.
---
Return labels
Print a return label before requesting a refund.
```

The title match and repeated term push the refund policy document above unrelated shipping content.

Use phrase search plus an exclusion:

```text
query: "refund policy" -shipping
```

Documents must contain the adjacent phrase `refund policy`, and any document containing `shipping` is removed before ranking.

Use prefix search for partial terms:

```text
query: moto oil
prefix: true
```

With prefix search enabled, `moto` can match words such as `motorcycle` while `oil` is scored normally.

## Limits and model fit

- English stemming uses a dependency-free Porter-style stemmer; it is not a multi-language Snowball pipeline.
- No typo tolerance is included. Use the separate fuzzy document search tool when edit-distance matching is the primary need.
- No persisted index is stored. For reusable static-site indexes, use an index-builder workflow instead.
- No semantic/vector search is performed; that would require an embedding model.
- Snippets highlight normalized term hits, not rich HTML fragments.

## FAQ

<details>
<summary>When should I use BM25 instead of TF-IDF?</summary>

BM25 is the recommended default for most document search because it saturates repeated terms and normalizes by document length. TF-IDF is useful when you want a simpler classic score for comparison or when repeated terms should keep contributing more directly.

</details>

<details>
<summary>How are documents separated?</summary>

Choose `dashes` for a line containing three or more hyphens, `blank-line` for one or more empty lines, or `form-feed` for page-separated text from extractors. The first non-blank line inside each separated chunk becomes that document's title.

</details>

<details>
<summary>Does stemming work for every language?</summary>

No. The built-in stemmer targets common English Porter cases, so `running`, `runs`, and `run` can meet. Non-English stemming and language-specific stop-word lists are deliberately out of scope for this pure single-shot tool.

</details>

<details>
<summary>Can this search PDFs or DOCX files directly?</summary>

No. This tool accepts text only. Extract text from binary documents first, then paste the result. Keeping extraction separate avoids duplicating PDF/DOCX parsers and keeps the ranked search model deterministic.

</details>

## Related tools

- [Fuzzy Text Search](https://gizza.ai/tools/fuzzy-doc-search/): Paste text and fuzzy-search it — find the best-matching lines even with typos, ranked by relevance with the matches highlighted. Runs in your browser, no upload.
- [Build a Search Index](https://gizza.ai/tools/search-index-builder/): Build an offline full-text search index JSON from documents. Pick fields, store result data, set boosts, and run privately in your browser.
- [Extract Action Items from Meeting Notes](https://gizza.ai/tools/action-item-extractor/): Extract action items, owners, and decisions from meeting notes or daily notes with deterministic rules. Markdown checklist or JSON, private in-browser.
- [Add Line Numbers](https://gizza.ai/tools/add-line-numbers/): Add line numbers to every line of text online, like nl or cat -n — custom start, step, separator, and alignment. Free and private, runs in your browser.
- [ANSI Log Renderer](https://gizza.ai/tools/ansi-log-renderer/): Paste ANSI-colored terminal output or CI logs and render them as HTML, or strip escape codes to plain text. Handles 16-color, 256-color, and truecolor SGR codes.