Persian Tokenizer
Split Persian and Farsi text into words and sentences. ZWNJ half-space compounds like میخوانیم stay one word, Persian punctuation (، ؛ ؟ « ») and all three digit sets are handled, and URLs, emails, hashtags and dates stay whole. Runs entirely in your browser, no upload, no sign-up.
About this tool
Persian text uses a few conventions that generic whitespace tokenizers miss. The most important is the zero-width non-joiner (ZWNJ, نیمفاصله): words such as میخوانیم, نمیشود and کتابها look like two visible pieces, but they are normally one word for word counts, search indexing and NLP preprocessing. This tokenizer keeps those compounds together by default and can split them when you need morpheme-like parts.
Paste Persian or Farsi text, choose whether you want words, sentences or both, and select the output format. The tokenizer is deterministic and rule-based — no model download, no training data, and no upload. It also recognizes Persian punctuation (، ؛ ؟ « »), Persian and Arabic-Indic digits, URLs, emails, mentions, hashtags, and date/number separators.
Worked example
Input:
ما کتاب میخوانیم. حال شما چطور است؟ قیمت ۱٬۲۵۰ تومان است.
With the default Words mode, One per line format and punctuation set to Separate, the output is:
ما
کتاب
میخوانیم
.
حال
شما
چطور
است
؟
قیمت
۱٬۲۵۰
تومان
است
.
Turn on Split half-space compounds and میخوانیم becomes two tokens: می and خوانیم. Switch Punctuation to Remove when you want only lexical words and numbers.
What is handled
- ZWNJ and ZWJ joiners, with optional splitting at ZWNJ.
- Persian and Arabic punctuation, including
؟and۔as sentence endings. - ASCII, Arabic-Indic and Persian digits; separator-bearing numbers such as
۱۳۹۶/۰۶/۱۱,۳٫۵,۱٬۰۰۰and1,250.75stay whole by default. - URLs, email addresses,
@mentionsand#hashtagsstay one token when Keep entities is on. - Optional normalization folds Arabic keyboard forms (
ي,ك,ى,ة) to Persian forms and strips harakat/tatweel. - Newline handling for paragraphs, wrapped prose, subtitles and one-item-per-line lists.
Limits and edge cases
- Maximum input length is 200,000 Unicode characters.
- This is a tokenizer, not a stemmer, lemmatizer, POS tagger or named-entity recognizer.
- Half-space correction is not automatic: if the input is missing ZWNJ characters, the tool will not infer where they should be inserted.
- JSON output is compact by design so it can be copied into scripts without cleanup.
- Punctuation Attach mode is a whitespace split; use Separate or Remove for NLP-style token lists.
FAQ
Why does `میخوانیم` stay one token by default?
The character between می and خوانیم is ZWNJ (U+200C), not a normal space. In Persian writing it marks a half-space inside one written word. For word counts, search indexing and most preprocessing, keeping that compound as one token is the least surprising default. Turn on Split half-space compounds only when you specifically want the pieces.
Does this normalize Arabic keyboard characters?
Yes, when Normalize is on (the default). Arabic ي and ك are folded to Persian ی and ک, ى becomes ی, ة becomes ه, Arabic-Indic digits are converted to Persian digits, and harakat plus kashida are stripped. Turn normalization off if you need to preserve the exact original characters.
How are sentences split?
Sentence mode treats ., !, ?, Persian ؟, Arabic full stop ۔, reversed question mark ⸮ and ellipsis as sentence terminators when they are followed by whitespace or the end of the text. Periods inside numbers, URLs and email addresses are not treated as sentence boundaries.
Can I use it for word counts?
Yes. Use Words mode, Punctuation: Remove, leave Split half-space compounds off, and keep Normalize on. That gives a practical word-token list for counting, search indexing and simple Persian text statistics.
Is this the same as Hazm or Parsivar?
No. Hazm, Parsivar and similar NLP libraries provide larger pipelines such as normalization, stemming, lemmatization, POS tagging or parsing. This tool intentionally stays smaller: a deterministic in-browser tokenizer with the controls needed for copy-paste text cleanup and lightweight preprocessing.
Developer & Automation Access
Run it from the terminal
Same engine as this page, headless — via the gizza CLI:
gizza tool persian-tokenizer "ما کتاب میخوانیم. یادگیری خوب است."New to the CLI? Get gizza →
Open it by URL
Pre-fill and auto-run this tool with query parameters — the names match the API/CLI:
https://gizza.ai/tools/persian-tokenizer/?text=%D9%85%D8%A7%20%DA%A9%D8%AA%D8%A7%D8%A8%20%D9%85%DB%8C%E2%80%8C%D8%AE%D9%88%D8%A7%D9%86%DB%8C%D9%85.%20%DB%8C%D8%A7%D8%AF%DA%AF%DB%8C%D8%B1%DB%8C%20%D8%AE%D9%88%D8%A8%20%D8%A7%D8%B3%D8%AA.&mode=words&format=lines&punctuation=separate&split_zwnj=true&normalize=true&keep_entities=true&newlines=paragraphMachine-readable descriptor: tool.json — title + parameters JSON Schema for agents.