Chunk Text for RAG
Split a long document into overlapping chunks by token, character, or word count — the way LangChain's RecursiveCharacterTextSplitter does — for retrieval-augmented generation and embedding pipelines. Word, sentence, and paragraph boundaries with recursive fallback. Runs entirely in your browser, no upload.
About this tool
Chunk Text splits a long document into smaller, overlapping pieces so you can
embed them for retrieval-augmented generation (RAG), semantic search, or any
pipeline that feeds context to a language model. Each chunk is a contiguous slice
of your original text, and every record carries its start/end character
offsets so downstream code can cite the exact span.
Choose how the size is measured — tokens (estimated from characters, since no real BPE tokenizer runs in the browser), characters, or words — and how much each chunk overlaps the one before it. Overlap keeps a sentence that straddles a boundary from being lost when it is retrieved.
Pick where a chunk is allowed to end with Split on: word never cuts a word
in half, sentence ends on ./!/?, paragraph ends on blank lines, and
character is a hard cut at the exact size. Coarser boundaries fall back to finer
ones when a single unit is bigger than the chunk size — the same recursive
behaviour as LangChain's RecursiveCharacterTextSplitter.
Output as a pretty JSON array (with id, text, chars, tokens, start,
end), newline-delimited JSONL (one object per line, ready to stream into a
vector store), or plain text with the chunks separated by a --- divider.
Everything runs locally in your browser — your document is never uploaded.
FAQ
What chunk size and overlap should I use?
A common starting point for RAG is around 500 tokens per chunk with ~10% overlap (so 50 tokens), which is why those are the defaults here. Smaller chunks (128–256 tokens) give more precise retrieval but fragment context; larger chunks (800–1000 tokens) keep more context but dilute the embedding. Tune to your embedding model's context window and your documents.
Are the token counts exact?
No — token counts are an estimate based on the chars per token setting (default 4.0, a common English GPT approximation). A real byte-pair tokenizer does not run in the browser wasm sandbox, so the number is a fast heuristic, not a per-model BPE count. If you need exact tokens for a specific model, use a dedicated token-counter tool; for splitting documents into embed-sized pieces the estimate is usually close enough.
What does "overlap" actually do?
Overlap makes each chunk repeat the last few units of the previous chunk. If an important sentence falls right on a chunk boundary, overlap ensures it appears whole in at least one chunk, so a retriever can still surface it. Overlap is measured in the same unit as the chunk size and must be less than the chunk size.
How is this different from a plain text splitter?
A plain splitter cuts on a single delimiter. This tool produces overlapping, size-bounded chunks and never splits a word (or sentence/paragraph) unless a single unit is larger than the whole chunk — then it recursively falls back to a finer boundary. That sliding-window-with-overlap behaviour is what RAG and embedding pipelines expect.
What do the start and end fields mean?
start and end are character offsets into your original text (0-based, end
exclusive), so text.slice(start, end) in JavaScript — or text[start:end] in
Python — reproduces the chunk. They let you highlight, cite, or link back to the
exact source span a retrieved chunk came from.
Developer & Automation Access
Run it from the terminal
Same engine as this page, headless — via the gizza CLI:
gizza tool chunk-text "Paste the document you want to split into chunks…"New to the CLI? Get gizza →
Open it by URL
Pre-fill and auto-run this tool with query parameters — the names match the API/CLI:
https://gizza.ai/tools/chunk-text/?text=Paste%20the%20document%20you%20want%20to%20split%20into%20chunks%E2%80%A6&chunk_size=500&overlap=50&unit=tokens&boundary=word&chars_per_token=4&format=json&trim=trueMachine-readable descriptor: tool.json — title + parameters JSON Schema for agents.