How accurate is Unfret?
We test Unfret on 2,641 web pages whose answer we already know: shops, product pages, second-hand ads and tradespeople's websites, some honest and some not. Here is how often each verdict turned out right, how many bad pages it caught, how often it worried about a good shop, and what these numbers don't tell you.
Measured on November 18, 2026 with scorer v1-20261118-113117-C,
the retrained scorer unfret.net runs. It is the scorer this server runs. Both scorers' numbers are side by side
below, with every earlier version.
- 100% of product pages called Serious concerns really were unreliable
- 83% of unreliable product pages got Think twice or Serious concerns
- 16% of legitimate shops got one of those two warnings: a false alarm
Shops and product pages
For each verdict: how many test pages got it, how many of those were actually unreliable (a scam, or junk sold as something better), and what share of all the unreliable pages ended up with that verdict. A good Looks fine has few unreliable pages; a good Serious concerns has almost only unreliable ones.
| Verdict | Pages | Actually unreliable | Of all unreliable pages |
|---|---|---|---|
| Looks fine | 770 | 7.1% (55) | 6.7% |
| Worth a look | 166 | 49% (82) | 9.9% |
| Think twice | 699 | 83% (582) | 70% |
| Serious concerns | 107 | 100% (107) | 13% |
Put together: Unfret warned (Think twice or Serious concerns) about 83% of the unreliable pages, and 86% of its warnings were right. Counting Worth a look too, it flagged 93% of them.
Based on 1,742 pages that got a verdict: 826 unreliable and 916 legitimate. 339 of those 1,742 pages are synthetic. 20 of the 1,762 labelled pages couldn't be measured (3 from long-established businesses, 17 from published research datasets): we replay recorded answers from the language model, and those pages' answers aren't in the recording yet. They are left out rather than guessed.
The scorer learnt from part of these pages (the training share), so the figures above flatter it a little. On the pages held back for testing, which it never saw (505 pages): 79% of unreliable pages warned about, 86% of warnings right, 100% of Serious concerns right, and 11% of legitimate pages warned about. With the checks for ads and tradespeople switched off, product pages come out the same: 83% of unreliable pages warned about, 86% of warnings right.
False alarms on legitimate shops
How often a legitimate shop's product page got a warning it didn't deserve.
| Shops | Pages | Think twice or worse | Worth a look |
|---|---|---|---|
| All legitimate shops | 753 | 16% | 11% |
| European shops | 450 | 21% | 14% |
| US shops | 299 | 7.0% | 5.0% |
0% of legitimate shops got Serious concerns. Worth a look is not a warning: it means a few things are worth checking yourself, such as a missing return address, and many honest small shops get it. European shops are told off more often than US ones: the European legitimate shops in the test are mostly small shops from 2023, and EU law asks more of a shop's page (company details, withdrawal rights) than US practice does, so there is more to find missing.
The scorer unfret.net runs, and the open baseline
The code is open source, and so is one version of the scorer, the open baseline: a self-hosted Unfret runs it. unfret.net runs a version retrained on more labelled pages, kept private like the pages it learnt from. Both on a replay of every labelled product page:
| Measure | Retrained, runs on unfret.net (this server)v1-20261118-113117-C | Open baseline, self-hostedv1-20261109-222323-C |
|---|---|---|
| Unreliable pages warned about | 83% | 73% |
| Warnings right | 86% | 88% |
| Serious concerns right | 100% | 96% |
| Legitimate shops: Think twice or worse | 16% | 11% |
| Legitimate shops: Worth a look | 11% | 30% |
| Legitimate shops: Serious concerns | 0% | 1.6% |
| ROC AUC | 0.928 | 0.918 |
The numbers above this table are those of the scorer this server runs (or the baseline's, where that one's aren't published); the changes from one version to the next are in the changelog.
Earlier versions
The other scorer versions we published, newest first, each with its release date and the same numbers as the table above. Open one to see them.
Each version was measured on the labelled pages we had when it was released, and we keep adding pages, so numbers measured on different numbers of pages aren't strictly comparable.
This is the first retrained scorer, so there are no earlier versions to show yet. Each version it replaces will appear here, with its release date.
Second-hand ads
Ads on Marktplaats, 2dehands, Vinted, eBay, Craigslist and Facebook Marketplace. These verdicts come from rules written from public scam warnings, not yet from a trained model. Every ad here is synthetic: the platforms forbid collecting real ads, so we wrote them from patterns that consumer warnings document (a deposit first, "I'm abroad", WhatsApp only, a price too good to be true) and normal ones. Read these numbers as a check that the rules catch the documented tricks, not as real-world accuracy.
| Verdict | Pages | Actually unreliable | Of all unreliable pages |
|---|---|---|---|
| Looks fine | 4 | 0% (0) | 0% |
| Worth a look | 236 | 42% (98) | 67% |
| Think twice | 46 | 100% (46) | 31% |
| Serious concerns | 3 | 100% (3) | 2.0% |
Unfret says Looks fine about an ad only with concrete good signs (an account some years old, and pickup or the platform's own payment protection), never for lack of evidence, so most honest ads get Worth a look: 97% of the honest ads here. 100% of its warnings were right, and it warned about 33% of the scam ads; counting Worth a look, 100% of them.
Based on 289 pages that got a verdict: 147 unreliable and 142 legitimate. 289 of those 289 pages are synthetic. 289 of them were measured without the language model's reading of the text: its answers for these pages aren't in the recording we replay. The rule checks on everything else ran as usual, but in real use that reading can add a warning (scam-like wording, a push to pay outside the platform) and can hold back a Looks fine, so read these numbers as rough.
Tradespeople
Plumbers', locksmiths' and other trades' websites, and their profiles on platforms such as Werkspot and Thumbtack. Also rules, not yet a trained model. The websites are real: rogue trade sites named in decisions of the Dutch consumer authority, and long-established firms. The platform profiles, and a few websites, are synthetic.
| Verdict | Pages | Actually unreliable | Of all unreliable pages |
|---|---|---|---|
| Looks fine | 129 | 0% (0) | 0% |
| Worth a look | 45 | 33% (15) | 7.9% |
| Think twice | 120 | 100% (120) | 64% |
| Serious concerns | 54 | 100% (54) | 29% |
Unfret warned about 92% of the unreliable trade pages it could be measured on, 100% of its warnings were right, and it warned about 0% of the honest ones.
Based on 348 pages that got a verdict: 189 unreliable and 159 legitimate. 329 of those 348 pages are synthetic. 344 of them were measured without the language model's reading of the text: its answers for these pages aren't in the recording we replay. The rule checks on everything else ran as usual, but in real use that reading can add a warning (scam-like wording, a push to pay outside the platform) and can hold back a Looks fine, so read these numbers as rough. 242 of the 590 labelled pages couldn't be measured (86 from consumer-authority warning lists, 156 from long-established businesses): we replay recorded answers from the language model, and those pages' answers aren't in the recording yet. They are left out rather than guessed.
While most of the measured pages are synthetic, as now, these figures mostly show that the rules recognise the patterns those pages were built from. They say little yet about real tradespeople; we'll update them once the real websites are measured too.
How we measure
The test set
| Kind of page | Pages | Legitimate | Unreliable |
|---|---|---|---|
| Shops' own product pages | 1,415 | 758 | 657 |
| Product pages on a marketplace | 347 | 163 | 184 |
| Second-hand ads | 289 | 142 | 147 |
| Tradespeople's own websites | 311 | 186 | 125 |
| Tradespeople's profiles on a platform | 279 | 129 | 150 |
413 more pages were collected but left out because their label wasn't certain enough.
Where the labels come from
Labels come from outside sources, not from our own judgement:
- Consumer-authority warning lists (253 unreliable): shops on the Czech Trade Inspection Authority's list of risky web shops, and rogue trade websites named in decisions of the Dutch consumer authority (ACM), as archived while the warning applied.
- Published research datasets (327 legitimate, 499 unreliable): fraudulent-shop and phishing datasets from academic studies, with the legitimate shops one of them lists.
- Long-established businesses (605 legitimate): shops and trade firms with many years of history (from public sources such as Wikidata and OpenStreetMap, and a hand-picked list of EU retailers).
- Synthetic pages (446 legitimate, 511 unreliable): pages we generated: marketplace product pages built from public review data, and ads and trade pages built only from scam and normal patterns that public consumer warnings document.
A second, larger language model (not the one Unfret uses) then read every page with written instructions; 2,102 of the 2,641 labels are confirmed by it. Ads, tradespeople and the most recently added shops keep their label only when it confidently agreed. For the research datasets, whose labels came from people, a disagreement leaves the label as it was and puts the page up for review.
How it's split
Each page is assigned once, at random but repeatably, to training (70%: 1,866 pages) or testing (30%: 775), in the same proportion within each source and label. The scorer for shops and product pages learns only from the training pages. The numbers above cover both, as a replay of every labelled page; the test-only figures are given under the product table.
Replaying recorded pages
Each page was saved once (from the Internet Archive, from a dataset, fetched live, or generated), and the language model's answers about it were recorded. A measurement replays exactly that: the same pages through the same checks as a real check, with the recorded answers. So every run is repeatable and costs nothing. Lookups that would ask today's world about an old page are off: the age of the domain, the EU VAT register, reverse image search and linking shops that share an owner. In real use those add evidence; they aren't counted here.
What these numbers can't tell you
- Unfret sees the page, not the deal. It can't see whether a parcel arrives, what happens after you pay, a seller's messages, or anything behind a login it isn't shown. A clever scam with a careful page can look fine, and a real shop with a sloppy page can look worrying.
- Many pages are a few years old. Most real pages are archived copies, many from 2022 and 2023. Scams change; older pages measure older tricks.
- Europe and the US, unevenly. Where the test pages are from: shops' own product pages: 701 US, 695 Europe, 19 elsewhere; product pages on a marketplace: 342 US, 5 Europe; second-hand ads: 147 Europe, 86 Europe and US, 56 US; tradespeople's own websites: 216 Europe, 95 US; tradespeople's profiles on a platform: 166 Europe, 113 US. Shops from elsewhere are rare in the test.
- Synthetic pages. 957 of the 2,641 pages are generated: the marketplace product pages (built from a public dataset of real products and reviews), all ads, the tradespeople's platform profiles and some of their websites. They test whether Unfret notices documented signs, not whether it finds new ones.
- Labels can be wrong. A warning list can name a shop that later cleaned up, and an established business can still treat a customer badly.
- Signals, not verdicts on a business. Unfret tells you what stands out on a page and how careful to be. It doesn't say a business is a scam, and the decision stays yours.
Changes per scorer version
Shops and product pages get their verdict from a small trained scorer; its version is in every result (and the code and the open baseline are open source). Ads and tradespeople get theirs from rules. What changed, newest first:
- 0.13.0, November 18, 2026 (unfret.net: scorer
v1-20261118-113117-C, quick checksv1-quick-20261118-113123-C; open baselinev1-20261109-222323-Cunchanged) - The first retrained scorer unfret.net runs, kept private with the pages it learnt from; self-hosted servers keep the open baseline. In the replay of every labelled product page it warns about 83% of the unreliable pages (the baseline: 73%), and called no legitimate shop Serious concerns (the baseline: 1.6%). Fewer legitimate shops get Worth a look (11% instead of 30%), more get Think twice (16% instead of 11%). On the pages held back from its training: 79% of unreliable pages warned about (the baseline: 68%), 11% of legitimate pages (the baseline: 6.6%). On 180 product pages of well-known sellers, a deep check gave 6 Worth a look or worse and no Serious concerns (the baseline: 24), a quick check 5 of 179 (the baseline: 160). With both, a page they put at Worth a look without a concrete reason to show reads Looks fine.
- Serious concerns only when the scorer is at least 60% sure; below that the same page shows Think twice, still a warning. With the open baseline, the share of right Serious concerns went from 94% to 95% in the replay of every labelled product page, with 15 instead of 25 legitimate pages called that. Checks for second-hand ads and tradespeople come to everyone. This page.
- 0.12.0, November 16, 2026 (scorer unchanged)
- More established US shops and shops on a consumer-authority warning list in the test set, so that "a US shop" or "this source" no longer gives the answer away. A scorer retrained on it warned about more legitimate shops, so it was not shipped.
- 0.11.0 and 0.11.1, November 15–16, 2026 (scorer unchanged)
- Checks for second-hand ads and tradespeople, and seven new checks on product pages that show reasons without changing the score. Training now guards against learning the test set instead of shops; a retrain under those rules wasn't better, so it was not shipped.
- 0.7.0, November 10, 2026 (scorer
v1-20261109-222323-C) - Fewer false alarms: the scorer no longer assumes most pages are bad (the test set has more unreliable pages than the web does). On the pages it never saw, legitimate pages warned about went from 35 to 24 of 143; unreliable pages caught from 168 to 151 of 190. US shops are no longer marked down for lacking EU company details.
- 0.6.0, November 8, 2026 (scorer
v1-20261108-C) - The first trained scorer: a small, readable model over the checks' signals, so each reason's weight in the verdict is exact. Released as open source.
The numbers on this page, as data (totals only, no pages):
accuracy-history.json for every scorer version published, the one
unfret.net runs included, and accuracy.json for the open baseline. They are made with
make accuracy in the
source code; the test pages
themselves can't be published, as they are other people's web pages.