# Chunk Text for RAG Split long text into overlapping chunks by token, character, or word count for RAG and embedding pipelines. Word, sentence, or paragraph boundaries. Runs in your browser. ## Run it - **CLI:** `gizza tool chunk-text "Paste the document you want to split into chunks…"` - **Web:** https://gizza.ai/tools/chunk-text/ - **Agents:** machine-readable descriptor (parameters JSON Schema) at https://gizza.ai/tools/chunk-text/tool.json ## Inputs - `text` — Text _(field)_ - `chunk_size` — Chunk size _(field)_ - `overlap` — Overlap _(field)_ - `unit` — Measure in _(field)_ - `boundary` — Split on _(field)_ - `chars_per_token` — Chars per token (estimate) _(field)_ - `format` — Output format _(field)_ - `trim` — Trim whitespace from each chunk _(field)_ ## Output - Chunks (text) ## Query parameters Open the tool pre-filled and auto-run via URL: - `text` — Text - `chunk_size` — Chunk size - `overlap` — Overlap - `unit` — Measure in - `boundary` — Split on - `chars_per_token` — Chars per token (estimate) - `format` — Output format - `trim` — Trim whitespace from each chunk Example: `https://gizza.ai/tools/chunk-text/?text=Paste%20the%20document%20you%20want%20to%20split%20into%20chunks%E2%80%A6&chunk_size=500&overlap=50&unit=tokens&boundary=word&chars_per_token=4&format=json&trim=true` --- ## About this tool **Chunk Text** splits a long document into smaller, overlapping pieces so you can embed them for retrieval-augmented generation (RAG), semantic search, or any pipeline that feeds context to a language model. Each chunk is a contiguous slice of your original text, and every record carries its `start`/`end` character offsets so downstream code can cite the exact span. Choose how the size is measured — **tokens** (estimated from characters, since no real BPE tokenizer runs in the browser), **characters**, or **words** — and how much each chunk overlaps the one before it. Overlap keeps a sentence that straddles a boundary from being lost when it is retrieved. Pick where a chunk is allowed to end with **Split on**: `word` never cuts a word in half, `sentence` ends on `.`/`!`/`?`, `paragraph` ends on blank lines, and `character` is a hard cut at the exact size. Coarser boundaries fall back to finer ones when a single unit is bigger than the chunk size — the same recursive behaviour as LangChain's `RecursiveCharacterTextSplitter`. Output as a pretty **JSON** array (with `id`, `text`, `chars`, `tokens`, `start`, `end`), newline-delimited **JSONL** (one object per line, ready to stream into a vector store), or **plain** text with the chunks separated by a `---` divider. Everything runs locally in your browser — your document is never uploaded. ## FAQ
What chunk size and overlap should I use? A common starting point for RAG is **around 500 tokens per chunk with ~10% overlap** (so 50 tokens), which is why those are the defaults here. Smaller chunks (128–256 tokens) give more precise retrieval but fragment context; larger chunks (800–1000 tokens) keep more context but dilute the embedding. Tune to your embedding model's context window and your documents.
Are the token counts exact? No — token counts are an **estimate** based on the *chars per token* setting (default 4.0, a common English GPT approximation). A real byte-pair tokenizer does not run in the browser wasm sandbox, so the number is a fast heuristic, not a per-model BPE count. If you need exact tokens for a specific model, use a dedicated token-counter tool; for splitting documents into embed-sized pieces the estimate is usually close enough.
What does "overlap" actually do? Overlap makes each chunk repeat the last few units of the previous chunk. If an important sentence falls right on a chunk boundary, overlap ensures it appears *whole* in at least one chunk, so a retriever can still surface it. Overlap is measured in the same unit as the chunk size and must be **less than** the chunk size.
How is this different from a plain text splitter? A plain splitter cuts on a single delimiter. This tool produces **overlapping, size-bounded** chunks and never splits a word (or sentence/paragraph) unless a single unit is larger than the whole chunk — then it recursively falls back to a finer boundary. That sliding-window-with-overlap behaviour is what RAG and embedding pipelines expect.
What do the start and end fields mean? `start` and `end` are **character offsets** into your original text (0-based, end exclusive), so `text.slice(start, end)` in JavaScript — or `text[start:end]` in Python — reproduces the chunk. They let you highlight, cite, or link back to the exact source span a retrieved chunk came from.
## Related tools - [Split Text by Delimiter](https://gizza.ai/tools/split-text/): Split text on any delimiter into one item per line — comma, tab, custom, whitespace, or characters, with trim and drop-blanks options. Free, in-browser. - [Extract Action Items from Meeting Notes](https://gizza.ai/tools/action-item-extractor/): Extract action items, owners, and decisions from meeting notes or daily notes with deterministic rules. Markdown checklist or JSON, private in-browser. - [Add Line Numbers](https://gizza.ai/tools/add-line-numbers/): Add line numbers to every line of text online, like nl or cat -n — custom start, step, separator, and alignment. Free and private, runs in your browser. - [ANSI Log Renderer](https://gizza.ai/tools/ansi-log-renderer/): Paste ANSI-colored terminal output or CI logs and render them as HTML, or strip escape codes to plain text. Handles 16-color, 256-color, and truecolor SGR codes. - [Censor Text](https://gizza.ai/tools/censor-text/): Redact a list of words (or common profanity) in any text by masking them — case-insensitive, whole-word or substring. Runs in your browser, nothing is uploaded, free.