This page reports how well the engine actually performs, including the experiments that didn’t work. Publishing negative results is uncommon, and it’s the most honest signal we can give a technical reader: it’s the part no one can copy from a marketing page. Every number below comes from a reproducible harness in our evaluation suite (each one is named where it’s used), and the exact scoring method lives on the methodology page.
Retrieval: does the right mark surface?
The core promise of a screening tool is recall: if a conflicting mark exists, it should appear in the results. We measure this by sampling live single-token marks and, for each, running two queries, each with one known-correct answer: an exact-name query (the mark itself should rank first) and a typo query (a one-character edit of the name, where the original should still appear near the top). Sample size 300; harness api/eval/retrieval_eval.py.
| Query | Recall@20 | MRR | Reads as |
|---|---|---|---|
| Exact name | ≈ 1.00 | 0.83 | The queried mark essentially always surfaces |
| One-character typo | ≈ 0.72 | 0.52 | A single typo still surfaces the original ~72% of the time |
An honesty note about that ≈ 1.00. Our first version of this metric read lower (~0.77), and it turned out to be measuring the wrong thing. The tool de-duplicates same-name registrations into one result card, but the metric was checking for one specific registration number; the mark was retrieved, just represented by a sibling registration. At the level a user actually cares about (did the name surface?), it surfaces essentially always. We caught this with the harness and fixed the metric rather than the (correct) engine. It’s exactly the kind of thing we’d rather show than bury.
Limitations of this measurement. Each query has a single known-correct answer, so precision-style metrics are bounded and not meaningful here. A true multi-match precision number needs a hand-labeled relevant set, which we don’t yet have. And the sampled population is string-similar by construction, so it stress-tests spelling and sound far more than it tests the meaning signal’s intended job.
The negative results
These are the interesting ones. Each is stated as what we hypothesised, what we measured, and what we decided, with no softening.
1. Semantic (“meaning”) similarity barely earns its keep
- Hypothesised: vector similarity would catch conflicts the string signals miss, the names that mean the same thing without sounding or looking alike.
- Measured: on hand-labeled pairs it scored near chance (ROC-AUC 0.556, versus 0.785 for the string ensemble), and on a head-to-head retrieval A/B it made ranking slightly worse, while occupying roughly a third of the engine’s working set. Harnesses
api/eval/weight_ablation.py,labeled_pairs.py, and the retrieval A/B inretrieval_eval.py. - Decided: kept in the ensemble (at its current 0.25 weight, not up-weighted) because it contributes a small deep-tail recall gain and it is the honest “means the same” axis. But it is flagged as the first thing to cut if its cost ever needs to come back. We don’t claim it’s pulling weight it isn’t.
2. Tribunal outcomes cannot calibrate the score
- Hypothesised: real USPTO opposition and cancellation win/loss records could anchor the risk-band cutoffs to actual outcomes.
- Measured: across ~868k scoreable adversarial pairs, the score could not separate the party that won from the party that lost: AUC ≈ 0.47 (no separation, slightly inverted) at every signal depth (name-only, plus semantic, plus class gate). Who wins is dominated by procedure (defaults, settlements, priority) and by grounds that have nothing to do with name similarity. Harnesses
api/eval/ttab_eval.pyandbox_ttab_full.py. - Decided: the cutoffs are set relative to the corpus, not to outcomes; the earlier plan to “let tribunal labels anchor the cutoffs” was retired. The score is a similarity measure, not a prediction of what a tribunal would decide.
3. A short-name typo “gap” that was a measurement artifact
- Hypothesised: short names miss too many typo variants, and a dedicated deletion index (SymSpell) would close the gap.
- Measured: the “gap” was largely an artifact of measuring the wrong thing. At the product level (did any confusable mark within the same edit distance surface?), recall was already 0.995; the apparent 0.43 was a stricter exact-registration metric. Building the SymSpell arm moved that product metric by −0.003 (noise) while adding ~503 MB and an ETL step. Harness
api/eval/symspell_scope.pyplus the retrieval harnesses. - Decided: not shipped. We re-baselined the target to the product metric (already met) instead of building infrastructure to move a number that didn’t need moving.
4. Crowded-field density can’t be used to lower risk
- Hypothesised: a name sitting in a “crowded field” of many similar coexisting marks should read as lower risk, which is a real factor in confusion analysis.
- Measured: in name-space the opposite holds. Genuinely confusable pairs sit in fields at least as dense as random marks do, so crowdedness correlates positively with confusability. Dampening risk by density would therefore cost recall. Harness
api/eval/density_eval.py. - Decided: shelved; not wired into scoring. The full negative result is recorded rather than quietly dropped.
One positive worth stating precisely
The goods-class relatedness gate is mined from data, not hand-assigned. We measured how often each pair of Nice classes co-occurs across real adversarial proceedings and mapped that association strength onto the gate. The mined matrix independently reproduced the intuitive groupings (foods with foods, beer with spirits, cosmetics with pharmaceuticals), which is the check that it’s capturing real market relatedness. It validated, then replaced, the earlier hand-built matrix. Harness api/eval/gen_nice_affinity.py; the mechanics are on the methodology page. Separately, weighting the noisier phonetic signal below spelling cut false “high” readings from ~30% to ~3% of distinct pairs (api/eval/scoring_eval.py).
What these numbers do not prove
- Not a legal outcome. A high recall or a high score is not a prediction that an application would be refused, or that a tribunal would find confusion. The score is a similarity measure over public data.
- Not a gold-standard relevance benchmark. Retrieval is measured on corpus-mined labels with one correct answer per query, not a human-judged set of every relevant mark. It tells you the obvious match surfaces; it doesn’t prove every subtle one does.
- Recall is “the mark surfaced,” not “it’s a real conflict.” Whether a surfaced mark is actually a problem is a judgment for you and, ideally, a trademark attorney.
- Coverage is limited to live word marks in the U.S. federal register: no logos, common-law, state, or foreign marks. See what this tool does.
Reproducibility
Every number on this page is produced by a harness in api/eval/, including retrieval_eval.py (recall/MRR), scoring_eval.py and weight_ablation.py (calibration and weight sweeps), ttab_eval.py (tribunal calibration), density_eval.py (crowded-field), and gen_nice_affinity.py (the mined class gate). The honest core of the scoring itself is published in code on the methodology page, so the method is checkable rather than just asserted.
This tool reports what the public record shows and never states a legal conclusion. It is preliminary automated screening based on public USPTO data, not legal advice and not a substitute for a full clearance search or attorney review.
See also our Methodology, what this tool does, and our Terms.