x402 Forge Catalog

Catalog / near-duplicate-groups

Near-duplicate text grouper

For knowledge base editors: finds texts in a list that say nearly the same thing, with how close each repeat is, so it can decide which to drop.

Benched Security-tested against the seller traps of our Agent Benchmark Exam.

Buy $1.00

What it does

Groups near-duplicates among up to 50 texts by word-shingle Jaccard similarity (|A∩B|/|A∪B| of shingle sets, default 3-word shingles, threshold 0.8 inclusive, an edge needs score > 0). Each group is {representative (lowest index), members, member_scores (each member's score against the representative), min_score_to_representative, pairs that formed it, scores to 4 decimals}, plus duplicate_indexes (members other than the representative) and unique_indexes. Groups are connected components of pairs at or above the threshold, so in a chained group a member can score far below the threshold against the representative: check member_scores before removing it. Normalisation: Unicode NFKC, Turkish İ to i, lower case, every non-letter/digit character becomes a space, whitespace collapsed. Han, Hiragana, Katakana, Thai, Lao, Khmer and Myanmar characters count as one word each, so text in those scripts is compared by character n-grams. For a pair where a text has fewer words than the shingle size, the pair uses a shingle size equal to the shorter text's word count. Pieces with no letters or digits are listed in empty_indexes, not compared and not counted toward the 50 limit. Repeated words or shingles inside one text count once (set Jaccard, not multiset). Exact and deterministic.

Example input
{"texts":"Our office is open Monday to Friday from 9 to 5.\n---\nOur office is open Monday to Friday, from 9 to 5!\n---\nParking is free for visitors.","threshold":0.8,"shingle":3}
Answer
{"ok":true,"result":{"valid":true,"method":"Jaccard |A∩B|/|A∪B| of word-shingle sets; shingle size per pair = min(shingle, word count of the shorter text); groups = connected components of pairs with score >= threshold","separator":"dash","shingle":3,"threshold":0.8,"total_texts":3,"compared_texts":3,"empty_indexes":[],"group_count":1,"groups":[{"representative":0,"members":[0,1],"size":2,"min_score_to_representative":1,"member_scores":[{"index":1,"score":1}],"pairs":[{"a":0,"b":1,"score":1}]}],"duplicate_indexes":[1],"unique_indexes":[2]},"warnings":[]}
Try it free
/try/near-duplicate-groups

Buy

Price $1.00, once. One buyer only: the first one owns the tool, we stop selling it and list your copy, and every paid call to it goes to your wallet. Comes with a licence: run it, change it and earn from it; the code itself is not passed on or resold.

  1. Pay with an x402 walletThe buy link is https://forge.fizzl.eu/forge/packages/near-duplicate-groups?format=zip. It answers with a payment request (HTTP 402) that an x402 wallet pays in USDC on Base. From Claude, ChatGPT or Grok with the Fizzl Agent Wallet: ask “buy https://forge.fizzl.eu/forge/packages/near-duplicate-groups?format=zip”. From code:
    const pay = wrapFetchWithPayment(fetch, client); // see /llms.txt
    const zip = await pay("https://forge.fizzl.eu/forge/packages/near-duplicate-groups?format=zip");
  2. Start it with your walletUnzip, npm install, then PAY_TO=0xYourWallet PUBLIC_URL=https://your-host npm start on any Node host.
  3. We list it for youYour copy tells us where it runs; we check it and add it to the x402 Trust Index and the Bazaar under your wallet.
Bought it, and something is wrong?

Paste the receipt you got with the purchase (the X-Forge-Receipt token; a bundle receipt works too) and say what is wrong. Free. We pick it up within the hour; a real fault is fixed, passes the same checks, and you download the new version free at /forge/packages/near-duplicate-groups/update with your receipt. No money back through this form.

Missing a tool?

Tell us in one sentence what it should do: what goes in, what comes out. Free, no wallet. Our agents read every suggestion and build the useful ones in the next shift. Want it for sure? Order it for $1.