# MBOX Dedup Paste an mbox archive and remove duplicate messages by Message-ID. Keep first or last, handle ID-less messages, and download the cleaned mbox locally. ## Run it - **CLI:** `gizza tool mbox-dedup "From alice@example.com Mon Sep 03 10:00:00 2018 From: Alice Subject: Hello Message-ID: first copy From alice@example.com Mon Sep 03 11:00:00 2018 From: Alice Subject: Hello again Message-ID: duplicate copy"` - **Web:** https://gizza.ai/tools/mbox-dedup/ - **Agents:** machine-readable descriptor (parameters JSON Schema) at https://gizza.ai/tools/mbox-dedup/tool.json ## Inputs - `mbox` — MBOX text _(field)_ - `keep` — Keep duplicate occurrence _(field)_ - `ignore_case` — Compare Message-IDs case-insensitively _(field)_ - `no_message_id` — Messages without Message-ID _(field)_ ## Output - Deduplicated mbox (text) ## Query parameters Open the tool pre-filled and auto-run via URL: - `mbox` — MBOX text - `keep` — Keep duplicate occurrence - `ignore_case` — Compare Message-IDs case-insensitively - `no_message_id` — Messages without Message-ID Example: `https://gizza.ai/tools/mbox-dedup/?mbox=From%20alice%40example.com%20Mon%20Sep%2003%2010%3A00%3A00%202018%0AFrom%3A%20Alice%20%3Calice%40example.com%3E%0ASubject%3A%20Hello%0AMessage-ID%3A%20%3Ca1%40example.com%3E%0A%0Afirst%20copy%0A%0AFrom%20alice%40example.com%20Mon%20Sep%2003%2011%3A00%3A00%202018%0AFrom%3A%20Alice%20%3Calice%40example.com%3E%0ASubject%3A%20Hello%20again%0AMessage-ID%3A%20%3Ca1%40example.com%3E%0A%0Aduplicate%20copy&keep=first&ignore_case=true&no_message_id=keep` --- ## About this tool Paste an mbox mailbox export and this tool removes duplicate messages by their `Message-ID` header. It is useful after merging mail archives, importing the same mailing-list dump twice, or cleaning a support mailbox before sharing a smaller archive. The input is treated as classic mbox text: messages are separated by `From ` postmark lines at the start of a line. For each message, the header section is scanned for `Message-ID`. Values are normalized by trimming whitespace and one surrounding pair of angle brackets, so `` and ` abc@example.com ` compare the same. By default comparison is case-sensitive, which matches RFC 5322; turn on case-insensitive matching only when an exporter has changed the case of otherwise-identical IDs. You can keep the **first** or **last** copy of each duplicated ID. Surviving messages are emitted in their original order and preserved verbatim, including postmark lines, headers, body text, and blank-line separators. Messages with no `Message-ID` are kept by default because two drafts without IDs may still be different messages; switch that option to **drop** if you want to remove them. Everything runs locally in your browser through WebAssembly. The original mbox is never uploaded or modified; the output is a fresh de-duplicated mbox you can copy or download. ### Worked example Input: ```text From a@example.com Mon Sep 03 10:00:00 2018 Message-ID: first copy From b@example.com Mon Sep 03 11:00:00 2018 Message-ID: unique message From c@example.com Mon Sep 03 12:00:00 2018 Message-ID: duplicate copy ``` With **keep first**, the output keeps `first copy` and `unique message`, and removes `duplicate copy` because it has the same normalized `Message-ID` as the first message. ## FAQ
What counts as a duplicate? Two messages are duplicates when their `Message-ID` header matches after trimming whitespace and one surrounding `<...>` pair. The tool does not compare subject, sender, body text, attachments, or dates; that avoids fuzzy matches and keeps this tool focused on the standard mail identifier.
Should I keep the first or the last copy? Use **keep first** when the first archive is your trusted source and later imports are accidental repeats. Use **keep last** when later copies are more likely to contain the message version you want. Either way, the surviving messages stay in their original order in the output.
Why are messages without Message-ID kept by default? Drafts, malformed exports, and some generated messages may not have a `Message-ID`. Without an ID there is no safe Message-ID-based key, so the default is to keep every ID-less message instead of merging distinct mail by mistake. If you know ID-less messages are junk in your archive, choose **drop**.
Does the tool change line endings or rewrite messages? No. Surviving message blocks are concatenated verbatim from the input, including postmark lines, headers, bodies, blank lines, and existing CRLF or LF endings. Only whole duplicate message blocks are removed.
Is the mbox uploaded anywhere? No. The parser and de-duplicator run entirely in your browser as WebAssembly. Your mailbox text stays on your machine; the output is generated locally.
## Related tools - [Extract Action Items from Meeting Notes](https://gizza.ai/tools/action-item-extractor/): Extract action items, owners, and decisions from meeting notes or daily notes with deterministic rules. Markdown checklist or JSON, private in-browser. - [Amazon Order Analyzer](https://gizza.ai/tools/amazon-order-analyzer/): Paste an Amazon order-history CSV export to summarize total spend by month, top items, and category breakdowns. Browser-only, private, with Markdown or JSON output. - [Base Decoder](https://gizza.ai/tools/base-decoder/): Auto-detect and decode Base16, Base32, Base45, Base58, Base64, and Base85 text, including nested layers, locally in your browser. - [Before / after image slider generator](https://gizza.ai/tools/before-after-slider/): Turn two images into an interactive before/after comparison slider: one self-contained HTML file with inline CSS + JS, no libraries. - [Budget Planner](https://gizza.ai/tools/budget-planner/): Plan a monthly budget from take-home pay: 50/30/20 needs/wants/savings targets with custom splits, or a zero-based plan showing what's left to allocate.