# Deduplicate JSONL / NDJSON records

Remove duplicate records from JSONL / NDJSON text — de-duplicate by whole-line equality or chosen key fields, keep the first or last, all in your browser.

## Run it

- **CLI:** `gizza tool jsonl-deduplicator '{"id":1,"name":"Ada"}
{"id":2,"name":"Alan"}
{"id":1,"name":"Ada"}'`
- **Web:** https://gizza.ai/tools/jsonl-deduplicator/
- **Agents:** machine-readable descriptor (parameters JSON Schema) at https://gizza.ai/tools/jsonl-deduplicator/tool.json

## Inputs

- `data` — JSONL / NDJSON (one JSON value per line) _(field)_
- `keys` — Key fields (comma-separated; blank = whole-line match) _(field)_
- `keep` — Keep which occurrence _(field)_
- `ignore_case` — Ignore case when matching _(field)_
- `on_invalid` — Line that isn't valid JSON _(field)_

## Output

- De-duplicated JSONL (text)

## Query parameters

Open the tool pre-filled and auto-run via URL:

- `data` — JSONL / NDJSON (one JSON value per line)
- `keys` — Key fields (comma-separated; blank = whole-line match)
- `keep` — Keep which occurrence
- `ignore_case` — Ignore case when matching
- `on_invalid` — Line that isn't valid JSON

Example: `https://gizza.ai/tools/jsonl-deduplicator/?data=%7B%22id%22%3A1%2C%22name%22%3A%22Ada%22%7D%0A%7B%22id%22%3A2%2C%22name%22%3A%22Alan%22%7D%0A%7B%22id%22%3A1%2C%22name%22%3A%22Ada%22%7D&keys=id%20%20%20or%20%20%20user.id%2Cemail&keep=first&ignore_case=true&on_invalid=error`

---

## About this tool

JSONL (also called NDJSON) stores one JSON value per line — the format exported by log
pipelines, database dumps, LLM datasets, and streaming APIs. Concatenating or replaying such
streams often duplicates records, and most JSON tools only understand a single big array, not
line-delimited data.

This deduplicator works on JSONL natively: paste your lines and it removes duplicates while
keeping the surviving records in their original order. Leave the **key fields** blank to match
on the exact text of each line, or name one or more fields — for example `id`, or
`user.id,email` — to treat two records as the same when those fields match, even if the rest of
the record (or the key order) differs. Nested fields use dot paths (`user.id`), and a numeric
segment indexes into an array (`tags.0`).

Choose whether to keep the **first** or the **last** occurrence of each duplicate, turn on
case-insensitive matching for things like emails, and decide what happens to a line that isn't
valid JSON: stop with an error that names the line, skip it, or keep it matched by its raw text.
Everything runs locally in your browser through WebAssembly — no upload, no account, nothing
leaves your machine.

## FAQ

<details>
<summary>What's the difference between whole-line matching and key-field matching?</summary>

Leave **key fields** blank and two records are duplicates only when their lines are byte-for-byte
identical (after trimming surrounding whitespace) — so `{"a":1,"b":2}` and `{"b":2,"a":1}` are
treated as *different*. Enter one or more key fields and records are compared only on those
fields' values, which are normalised first, so field order no longer matters and
`{"id":1,"x":2}` collapses with `{"x":9,"id":1}` when you dedupe on `id`.

</details>

<details>
<summary>How do I deduplicate on a nested or array field?</summary>

Use a dot path. `user.id` reads the `id` inside the `user` object, and a numeric segment indexes
an array — `tags.0` is the first element of `tags`. You can combine several in the comma-separated
list, e.g. `user.id,email`, to match on a composite key. A record missing a named field is grouped
with other records that also lack it, which stays distinct from a record whose field is an
explicit `null`.

</details>

<details>
<summary>Does it keep the original order of my records?</summary>

Yes. Whether you keep the **first** or the **last** occurrence, the surviving records are emitted
in the order they appeared in your input. Keeping the last occurrence places each kept record at
the position of its final appearance — unlike sort-based deduplicators, the stream is never
reordered.

</details>

<details>
<summary>What happens to blank lines and invalid JSON?</summary>

Blank or whitespace-only lines are ignored — never counted and never emitted. When you dedupe by
key fields, a line that isn't valid JSON is handled by the **Line that isn't valid JSON** setting:
**Error** stops and names the offending line number, **Skip** drops it, and **Keep** keeps it and
matches it by its raw text. In whole-line mode no JSON is parsed at all, so any text is just a
line.

</details>

<details>
<summary>Is my data uploaded anywhere?</summary>

No. The tool is compiled to WebAssembly and runs entirely inside your browser tab. Your JSONL
never leaves your device, so it's safe to paste exports that contain personal or proprietary data.

</details>

## Related tools

- [Extract Action Items from Meeting Notes](https://gizza.ai/tools/action-item-extractor/): Extract action items, owners, and decisions from meeting notes or daily notes with deterministic rules. Markdown checklist or JSON, private in-browser.
- [Amazon Order Analyzer](https://gizza.ai/tools/amazon-order-analyzer/): Paste an Amazon order-history CSV export to summarize total spend by month, top items, and category breakdowns. Browser-only, private, with Markdown or JSON output.
- [Base Decoder](https://gizza.ai/tools/base-decoder/): Auto-detect and decode Base16, Base32, Base45, Base58, Base64, and Base85 text, including nested layers, locally in your browser.
- [Before / after image slider generator](https://gizza.ai/tools/before-after-slider/): Turn two images into an interactive before/after comparison slider: one self-contained HTML file with inline CSS + JS, no libraries.
- [Budget Planner](https://gizza.ai/tools/budget-planner/): Plan a monthly budget from take-home pay: 50/30/20 needs/wants/savings targets with custom splits, or a zero-based plan showing what's left to allocate.