How accurate is Unfret?

We test Unfret on 2,641 web pages whose answer we already know: shops, product pages, second-hand ads and tradespeople's websites, some honest and some not. Here is how often each verdict turned out right, how many bad pages it caught, how often it worried about a good shop, and what these numbers don't tell you.

Measured on November 18, 2026 with scorer v1-20261118-113117-C, the retrained scorer unfret.net runs. It is the scorer this server runs. Both scorers' numbers are side by side below, with every earlier version.

Shops and product pages

For each verdict: how many test pages got it, how many of those were actually unreliable (a scam, or junk sold as something better), and what share of all the unreliable pages ended up with that verdict. A good Looks fine has few unreliable pages; a good Serious concerns has almost only unreliable ones.

VerdictPagesActually unreliableOf all unreliable pages
Looks fine7707.1% (55)6.7%
Worth a look16649% (82)9.9%
Think twice69983% (582)70%
Serious concerns107100% (107)13%

Put together: Unfret warned (Think twice or Serious concerns) about 83% of the unreliable pages, and 86% of its warnings were right. Counting Worth a look too, it flagged 93% of them.

Based on 1,742 pages that got a verdict: 826 unreliable and 916 legitimate. 339 of those 1,742 pages are synthetic. 20 of the 1,762 labelled pages couldn't be measured (3 from long-established businesses, 17 from published research datasets): we replay recorded answers from the language model, and those pages' answers aren't in the recording yet. They are left out rather than guessed.

The scorer learnt from part of these pages (the training share), so the figures above flatter it a little. On the pages held back for testing, which it never saw (505 pages): 79% of unreliable pages warned about, 86% of warnings right, 100% of Serious concerns right, and 11% of legitimate pages warned about. With the checks for ads and tradespeople switched off, product pages come out the same: 83% of unreliable pages warned about, 86% of warnings right.

False alarms on legitimate shops

How often a legitimate shop's product page got a warning it didn't deserve.

ShopsPagesThink twice or worseWorth a look
All legitimate shops75316%11%
European shops45021%14%
US shops2997.0%5.0%

0% of legitimate shops got Serious concerns. Worth a look is not a warning: it means a few things are worth checking yourself, such as a missing return address, and many honest small shops get it. European shops are told off more often than US ones: the European legitimate shops in the test are mostly small shops from 2023, and EU law asks more of a shop's page (company details, withdrawal rights) than US practice does, so there is more to find missing.

The scorer unfret.net runs, and the open baseline

The code is open source, and so is one version of the scorer, the open baseline: a self-hosted Unfret runs it. unfret.net runs a version retrained on more labelled pages, kept private like the pages it learnt from. Both on a replay of every labelled product page:

MeasureRetrained, runs on unfret.net (this server)v1-20261118-113117-CReleased November 18, 2026, measured on 2,641 labelled pagesOpen baseline, self-hostedv1-20261109-222323-CReleased November 10, 2026, measured on 2,641 labelled pages
Unreliable pages warned about83%73%
Warnings right86%88%
Serious concerns right100%96%
Legitimate shops: Think twice or worse16%11%
Legitimate shops: Worth a look11%30%
Legitimate shops: Serious concerns0%1.6%
ROC AUC0.9280.918

The numbers above this table are those of the scorer this server runs (or the baseline's, where that one's aren't published); the changes from one version to the next are in the changelog.

Earlier versions

The other scorer versions we published, newest first, each with its release date and the same numbers as the table above. Open one to see them.

Each version was measured on the labelled pages we had when it was released, and we keep adding pages, so numbers measured on different numbers of pages aren't strictly comparable.

This is the first retrained scorer, so there are no earlier versions to show yet. Each version it replaces will appear here, with its release date.

Second-hand ads

Ads on Marktplaats, 2dehands, Vinted, eBay, Craigslist and Facebook Marketplace. These verdicts come from rules written from public scam warnings, not yet from a trained model. Every ad here is synthetic: the platforms forbid collecting real ads, so we wrote them from patterns that consumer warnings document (a deposit first, "I'm abroad", WhatsApp only, a price too good to be true) and normal ones. Read these numbers as a check that the rules catch the documented tricks, not as real-world accuracy.

VerdictPagesActually unreliableOf all unreliable pages
Looks fine40% (0)0%
Worth a look23642% (98)67%
Think twice46100% (46)31%
Serious concerns3100% (3)2.0%

Unfret says Looks fine about an ad only with concrete good signs (an account some years old, and pickup or the platform's own payment protection), never for lack of evidence, so most honest ads get Worth a look: 97% of the honest ads here. 100% of its warnings were right, and it warned about 33% of the scam ads; counting Worth a look, 100% of them.

Based on 289 pages that got a verdict: 147 unreliable and 142 legitimate. 289 of those 289 pages are synthetic. 289 of them were measured without the language model's reading of the text: its answers for these pages aren't in the recording we replay. The rule checks on everything else ran as usual, but in real use that reading can add a warning (scam-like wording, a push to pay outside the platform) and can hold back a Looks fine, so read these numbers as rough.

Tradespeople

Plumbers', locksmiths' and other trades' websites, and their profiles on platforms such as Werkspot and Thumbtack. Also rules, not yet a trained model. The websites are real: rogue trade sites named in decisions of the Dutch consumer authority, and long-established firms. The platform profiles, and a few websites, are synthetic.

VerdictPagesActually unreliableOf all unreliable pages
Looks fine1290% (0)0%
Worth a look4533% (15)7.9%
Think twice120100% (120)64%
Serious concerns54100% (54)29%

Unfret warned about 92% of the unreliable trade pages it could be measured on, 100% of its warnings were right, and it warned about 0% of the honest ones.

Based on 348 pages that got a verdict: 189 unreliable and 159 legitimate. 329 of those 348 pages are synthetic. 344 of them were measured without the language model's reading of the text: its answers for these pages aren't in the recording we replay. The rule checks on everything else ran as usual, but in real use that reading can add a warning (scam-like wording, a push to pay outside the platform) and can hold back a Looks fine, so read these numbers as rough. 242 of the 590 labelled pages couldn't be measured (86 from consumer-authority warning lists, 156 from long-established businesses): we replay recorded answers from the language model, and those pages' answers aren't in the recording yet. They are left out rather than guessed.

While most of the measured pages are synthetic, as now, these figures mostly show that the rules recognise the patterns those pages were built from. They say little yet about real tradespeople; we'll update them once the real websites are measured too.

How we measure

The test set

Kind of pagePagesLegitimateUnreliable
Shops' own product pages1,415758657
Product pages on a marketplace347163184
Second-hand ads289142147
Tradespeople's own websites311186125
Tradespeople's profiles on a platform279129150

413 more pages were collected but left out because their label wasn't certain enough.

Where the labels come from

Labels come from outside sources, not from our own judgement:

A second, larger language model (not the one Unfret uses) then read every page with written instructions; 2,102 of the 2,641 labels are confirmed by it. Ads, tradespeople and the most recently added shops keep their label only when it confidently agreed. For the research datasets, whose labels came from people, a disagreement leaves the label as it was and puts the page up for review.

How it's split

Each page is assigned once, at random but repeatably, to training (70%: 1,866 pages) or testing (30%: 775), in the same proportion within each source and label. The scorer for shops and product pages learns only from the training pages. The numbers above cover both, as a replay of every labelled page; the test-only figures are given under the product table.

Replaying recorded pages

Each page was saved once (from the Internet Archive, from a dataset, fetched live, or generated), and the language model's answers about it were recorded. A measurement replays exactly that: the same pages through the same checks as a real check, with the recorded answers. So every run is repeatable and costs nothing. Lookups that would ask today's world about an old page are off: the age of the domain, the EU VAT register, reverse image search and linking shops that share an owner. In real use those add evidence; they aren't counted here.

What these numbers can't tell you

Changes per scorer version

Shops and product pages get their verdict from a small trained scorer; its version is in every result (and the code and the open baseline are open source). Ads and tradespeople get theirs from rules. What changed, newest first:

0.13.0, November 18, 2026 (unfret.net: scorer v1-20261118-113117-C, quick checks v1-quick-20261118-113123-C; open baseline v1-20261109-222323-C unchanged)
The first retrained scorer unfret.net runs, kept private with the pages it learnt from; self-hosted servers keep the open baseline. In the replay of every labelled product page it warns about 83% of the unreliable pages (the baseline: 73%), and called no legitimate shop Serious concerns (the baseline: 1.6%). Fewer legitimate shops get Worth a look (11% instead of 30%), more get Think twice (16% instead of 11%). On the pages held back from its training: 79% of unreliable pages warned about (the baseline: 68%), 11% of legitimate pages (the baseline: 6.6%). On 180 product pages of well-known sellers, a deep check gave 6 Worth a look or worse and no Serious concerns (the baseline: 24), a quick check 5 of 179 (the baseline: 160). With both, a page they put at Worth a look without a concrete reason to show reads Looks fine.
Serious concerns only when the scorer is at least 60% sure; below that the same page shows Think twice, still a warning. With the open baseline, the share of right Serious concerns went from 94% to 95% in the replay of every labelled product page, with 15 instead of 25 legitimate pages called that. Checks for second-hand ads and tradespeople come to everyone. This page.
0.12.0, November 16, 2026 (scorer unchanged)
More established US shops and shops on a consumer-authority warning list in the test set, so that "a US shop" or "this source" no longer gives the answer away. A scorer retrained on it warned about more legitimate shops, so it was not shipped.
0.11.0 and 0.11.1, November 15–16, 2026 (scorer unchanged)
Checks for second-hand ads and tradespeople, and seven new checks on product pages that show reasons without changing the score. Training now guards against learning the test set instead of shops; a retrain under those rules wasn't better, so it was not shipped.
0.7.0, November 10, 2026 (scorer v1-20261109-222323-C)
Fewer false alarms: the scorer no longer assumes most pages are bad (the test set has more unreliable pages than the web does). On the pages it never saw, legitimate pages warned about went from 35 to 24 of 143; unreliable pages caught from 168 to 151 of 190. US shops are no longer marked down for lacking EU company details.
0.6.0, November 8, 2026 (scorer v1-20261108-C)
The first trained scorer: a small, readable model over the checks' signals, so each reason's weight in the verdict is exact. Released as open source.

The numbers on this page, as data (totals only, no pages): accuracy-history.json for every scorer version published, the one unfret.net runs included, and accuracy.json for the open baseline. They are made with make accuracy in the source code; the test pages themselves can't be published, as they are other people's web pages.