# Persian Tokenizer
Tokenize Persian text into words and sentences. ZWNJ half-space compounds stay whole, Persian punctuation and digits handled. Free, in your browser.
## Run it
- **CLI:** `gizza tool persian-tokenizer "ما کتاب میخوانیم. یادگیری خوب است."`
- **Web:** https://gizza.ai/tools/persian-tokenizer/
- **Agents:** machine-readable descriptor (parameters JSON Schema) at https://gizza.ai/tools/persian-tokenizer/tool.json
## Inputs
- `text` — Persian text _(field)_
- `mode` — Tokenize into _(field)_
- `format` — Output format _(field)_
- `punctuation` — Punctuation _(field)_
- `split_zwnj` — Split half-space (ZWNJ) compounds — میخوانیم → می + خوانیم _(field)_
- `normalize` — Normalize Arabic letters and strip harakat first _(field)_
- `keep_entities` — Keep URLs, emails, hashtags and numbers whole _(field)_
- `newlines` — Line breaks _(field)_
## Output
- Tokens (text)
## Query parameters
Open the tool pre-filled and auto-run via URL:
- `text` — Persian text
- `mode` — Tokenize into
- `format` — Output format
- `punctuation` — Punctuation
- `split_zwnj` — Split half-space (ZWNJ) compounds — میخوانیم → می + خوانیم
- `normalize` — Normalize Arabic letters and strip harakat first
- `keep_entities` — Keep URLs, emails, hashtags and numbers whole
- `newlines` — Line breaks
Example: `https://gizza.ai/tools/persian-tokenizer/?text=%D9%85%D8%A7%20%DA%A9%D8%AA%D8%A7%D8%A8%20%D9%85%DB%8C%E2%80%8C%D8%AE%D9%88%D8%A7%D9%86%DB%8C%D9%85.%20%DB%8C%D8%A7%D8%AF%DA%AF%DB%8C%D8%B1%DB%8C%20%D8%AE%D9%88%D8%A8%20%D8%A7%D8%B3%D8%AA.&mode=words&format=lines&punctuation=separate&split_zwnj=true&normalize=true&keep_entities=true&newlines=paragraph`
---
## About this tool
Persian text uses a few conventions that generic whitespace tokenizers miss. The most important is the zero-width non-joiner (ZWNJ, نیمفاصله): words such as `میخوانیم`, `نمیشود` and `کتابها` look like two visible pieces, but they are normally one word for word counts, search indexing and NLP preprocessing. This tokenizer keeps those compounds together by default and can split them when you need morpheme-like parts.
Paste Persian or Farsi text, choose whether you want words, sentences or both, and select the output format. The tokenizer is deterministic and rule-based — no model download, no training data, and no upload. It also recognizes Persian punctuation (`،` `؛` `؟` `«` `»`), Persian and Arabic-Indic digits, URLs, emails, mentions, hashtags, and date/number separators.
### Worked example
Input:
```text
ما کتاب میخوانیم. حال شما چطور است؟ قیمت ۱٬۲۵۰ تومان است.
```
With the default **Words** mode, **One per line** format and punctuation set to **Separate**, the output is:
```text
ما
کتاب
میخوانیم
.
حال
شما
چطور
است
؟
قیمت
۱٬۲۵۰
تومان
است
.
```
Turn on **Split half-space compounds** and `میخوانیم` becomes two tokens: `می` and `خوانیم`. Switch **Punctuation** to **Remove** when you want only lexical words and numbers.
### What is handled
- ZWNJ and ZWJ joiners, with optional splitting at ZWNJ.
- Persian and Arabic punctuation, including `؟` and `۔` as sentence endings.
- ASCII, Arabic-Indic and Persian digits; separator-bearing numbers such as `۱۳۹۶/۰۶/۱۱`, `۳٫۵`, `۱٬۰۰۰` and `1,250.75` stay whole by default.
- URLs, email addresses, `@mentions` and `#hashtags` stay one token when **Keep entities** is on.
- Optional normalization folds Arabic keyboard forms (`ي`, `ك`, `ى`, `ة`) to Persian forms and strips harakat/tatweel.
- Newline handling for paragraphs, wrapped prose, subtitles and one-item-per-line lists.
### Limits and edge cases
- Maximum input length is **200,000 Unicode characters**.
- This is a tokenizer, not a stemmer, lemmatizer, POS tagger or named-entity recognizer.
- Half-space correction is not automatic: if the input is missing ZWNJ characters, the tool will not infer where they should be inserted.
- JSON output is compact by design so it can be copied into scripts without cleanup.
- Punctuation **Attach** mode is a whitespace split; use **Separate** or **Remove** for NLP-style token lists.
## FAQ
Why does `میخوانیم` stay one token by default?
The character between `می` and `خوانیم` is ZWNJ (U+200C), not a normal space. In Persian writing it marks a half-space inside one written word. For word counts, search indexing and most preprocessing, keeping that compound as one token is the least surprising default. Turn on **Split half-space compounds** only when you specifically want the pieces.
Does this normalize Arabic keyboard characters?
Yes, when **Normalize** is on (the default). Arabic `ي` and `ك` are folded to Persian `ی` and `ک`, `ى` becomes `ی`, `ة` becomes `ه`, Arabic-Indic digits are converted to Persian digits, and harakat plus kashida are stripped. Turn normalization off if you need to preserve the exact original characters.
How are sentences split?
Sentence mode treats `.`, `!`, `?`, Persian `؟`, Arabic full stop `۔`, reversed question mark `⸮` and ellipsis as sentence terminators when they are followed by whitespace or the end of the text. Periods inside numbers, URLs and email addresses are not treated as sentence boundaries.
Can I use it for word counts?
Yes. Use **Words** mode, **Punctuation: Remove**, leave **Split half-space compounds** off, and keep **Normalize** on. That gives a practical word-token list for counting, search indexing and simple Persian text statistics.
Is this the same as Hazm or Parsivar?
No. Hazm, Parsivar and similar NLP libraries provide larger pipelines such as normalization, stemming, lemmatization, POS tagging or parsing. This tool intentionally stays smaller: a deterministic in-browser tokenizer with the controls needed for copy-paste text cleanup and lightweight preprocessing.
## Related tools
- [Extract Action Items from Meeting Notes](https://gizza.ai/tools/action-item-extractor/): Extract action items, owners, and decisions from meeting notes or daily notes with deterministic rules. Markdown checklist or JSON, private in-browser.
- [Add Line Numbers](https://gizza.ai/tools/add-line-numbers/): Add line numbers to every line of text online, like nl or cat -n — custom start, step, separator, and alignment. Free and private, runs in your browser.
- [ANSI Log Renderer](https://gizza.ai/tools/ansi-log-renderer/): Paste ANSI-colored terminal output or CI logs and render them as HTML, or strip escape codes to plain text. Handles 16-color, 256-color, and truecolor SGR codes.
- [API response diff](https://gizza.ai/tools/api-response-diff/): Compare two JSON API responses and see only the meaningful changes: ignore request ids, timestamps and UUIDs, match arrays by key or as sets, export a JSON Patch.
- [Rust AST Diff](https://gizza.ai/tools/ast-diff/): Compare two Rust source snippets structurally. The tool parses both files, canonicalizes their ASTs, and ignores formatting, whitespace, and comments.