# Text Corpus Cleaner Clean a line-oriented text corpus: normalize Unicode, trim/collapse whitespace, drop duplicate, short, or junk lines, and filter by language. Runs in your browser, free. ## Run it - **CLI:** `gizza tool text-corpus-cleaner "The Quick Brown Fox the quick brown fox ===== >>>>> ===== हैलो वर्ल्ड"` - **Web:** https://gizza.ai/tools/text-corpus-cleaner/ - **Agents:** machine-readable descriptor (parameters JSON Schema) at https://gizza.ai/tools/text-corpus-cleaner/tool.json ## Inputs - `input` — Text corpus (one record/sentence per line) _(field)_ - `unicode_form` — Unicode normalization _(field)_ - `whitespace` — Whitespace _(field)_ - `lowercase` — Lowercase every line _(field)_ - `dedupe` — Deduplicate lines (keep first) _(field)_ - `language` — Keep only this language _(field)_ - `min_chars` — Minimum characters per line (0 = keep all) _(field)_ - `min_words` — Minimum words per line (0 = keep all) _(field)_ - `max_symbol_ratio` — Max symbol ratio (0.0–1.0; 1.0 = keep all) _(field)_ - `collapse_blank` — Collapse blank-line runs & trim edges _(field)_ ## Output - Cleaned corpus (text) ## Query parameters Open the tool pre-filled and auto-run via URL: - `input` — Text corpus (one record/sentence per line) - `unicode_form` — Unicode normalization - `whitespace` — Whitespace - `lowercase` — Lowercase every line - `dedupe` — Deduplicate lines (keep first) - `language` — Keep only this language - `min_chars` — Minimum characters per line (0 = keep all) - `min_words` — Minimum words per line (0 = keep all) - `max_symbol_ratio` — Max symbol ratio (0.0–1.0; 1.0 = keep all) - `collapse_blank` — Collapse blank-line runs & trim edges Example: `https://gizza.ai/tools/text-corpus-cleaner/?input=The%20Quick%20Brown%20Fox%0Athe%20quick%20brown%20fox%0A%3D%3D%3D%3D%3D%20%3E%3E%3E%3E%3E%20%3D%3D%3D%3D%3D%0A%E0%A4%B9%E0%A5%88%E0%A4%B2%E0%A5%8B%20%E0%A4%B5%E0%A4%B0%E0%A5%8D%E0%A4%B2%E0%A5%8D%E0%A4%A1&unicode_form=nfc&whitespace=trim&lowercase=true&dedupe=exact&language=any&min_chars=0&min_words=0&max_symbol_ratio=1.0&collapse_blank=true` --- ## Clean a raw text corpus line by line Paste a line-oriented corpus — one record, sentence, or word per line — and this tool cleans every line in one deterministic pass, locally in your browser. **Nothing is uploaded.** The steps always run in the same fixed order: 1. **Split** on line endings (`\n`, `\r\n`, or `\r`). 2. **Transform** each line: Unicode normalization → whitespace handling → optional lowercase. 3. **Filter** out lines that are too short (**min characters** / **min words**), mostly punctuation (**max symbol ratio**), or not the **target language**. 4. **Deduplicate** non-blank lines, keeping the first occurrence. 5. **Blank-line handling**: optionally collapse runs of blank lines to a single blank and trim leading/trailing blanks (paragraph breaks are preserved). Language detection is trigram/script based (no model files, no network), so lines too short to identify are always kept rather than silently dropped. ### Worked example With **Whitespace → Collapse** and **Deduplicate → Normalized**, this input: ``` The Quick Brown Fox the quick brown fox The Quick Brown Fox ``` becomes: ``` The Quick Brown Fox ``` Collapsing squeezes the double space in line 1 to a single space. Normalized dedupe compares lines after folding case and interior spacing, so all three lines share the key `the quick brown fox` — only the **first occurrence is kept, verbatim after its transforms** (`The Quick Brown Fox`), and the two repeats are dropped. ### Limits & edge cases - **Line-oriented only.** Input is split on newlines and each line is cleaned independently — this tool does not reflow paragraphs or wrap text. - **Blank lines are structural.** They are never language- or symbol-filtered and never deduplicated, so paragraph breaks survive. A positive **min characters** or **min words** does drop blank lines (they have 0 of each). - **Symbol ratio** counts non-alphanumeric, non-space characters against visible characters; `1.0` keeps everything, `0.5` drops lines that are more than half punctuation. A line with no visible characters has a ratio of `0`. - **No trailing newline** is added — cleaned lines are joined with single `\n`s. - **Detection needs enough text.** Very short lines can't be language-detected and are kept even when a target language is set, so mixed-language word lists may keep short foreign words. - Everything runs in memory, so very large corpora are bounded by your browser's available memory. ### FAQ
What is the difference between exact and normalized deduplication? **Exact** drops a line only when it is byte-for-byte identical to an earlier line. **Normalized** folds case and collapses interior whitespace before comparing, so `Hello World` and `hello world` count as duplicates. Either way the **first** occurrence is kept exactly as it appears after the other transforms; only later repeats are removed. Blank lines are never treated as duplicates.
When should I use NFC versus NFKC normalization? Use **NFC** (the recommended default) for everyday text — it composes characters into their canonical form without changing what they mean. Use **NFKC** when you want to flatten *compatibility* variants: it turns ligatures like `fi` into `fi`, full-width `ＡＢＣ` into `ABC`, circled or styled letters into plain ones, and fractions into their spelled-out forms. Pick **None** to leave code points exactly as they are.
How does the language filter decide what to keep? Set a target language and each non-blank line is passed through trigram/script based detection (the `whatlang` approach — no downloaded model, no network call). A line is kept when its detected language matches the target. A line that is **too short or too ambiguous to detect is kept**, not dropped, so short valid records aren't lost. Leave the filter on **Any** to disable it entirely.
Why are some short or symbol-heavy lines still in the output? The length and symbol filters only apply when you set them. **Min characters** and **min words** default to `0` (keep everything), and **max symbol ratio** defaults to `1.0` (keep everything). Raise the minimums or lower the ratio to trim junk — for example `min words = 3` and `max symbol ratio = 0.5` removes one- or two-word fragments and lines that are mostly `=====` or `>>>>>` separators.
Is my text uploaded anywhere? No. The whole clean runs locally with WebAssembly; your corpus never leaves your browser, so it is safe to paste private or proprietary text.
## Related tools - [Remove Duplicate Lines](https://gizza.ai/tools/remove-duplicate-lines/): Remove duplicate lines from text in your browser — keep first or last occurrence, ignore case or whitespace, uniq-style consecutive mode. Free, private. - [Sort Lines](https://gizza.ai/tools/sort-lines/): Sort lines of text alphabetically, numerically, naturally or by length, remove duplicates and blank lines — free and private in your browser. - [Extract Action Items from Meeting Notes](https://gizza.ai/tools/action-item-extractor/): Extract action items, owners, and decisions from meeting notes or daily notes with deterministic rules. Markdown checklist or JSON, private in-browser. - [Add Line Numbers](https://gizza.ai/tools/add-line-numbers/): Add line numbers to every line of text online, like nl or cat -n — custom start, step, separator, and alignment. Free and private, runs in your browser. - [ANSI Log Renderer](https://gizza.ai/tools/ansi-log-renderer/): Paste ANSI-colored terminal output or CI logs and render them as HTML, or strip escape codes to plain text. Handles 16-color, 256-color, and truecolor SGR codes.