Are AI domain appraisals accurate?
We tested an AI appraisal and a statistical model on 300 sales neither had seen. The ranking was weak and the misses were all on the names that mattered.
Not very, and the error is lopsided. We tested an AI appraisal on 300 real .com sales it had never seen. It ranked them barely better than "shorter name, higher price" (rank correlation 0.21). It was fair on ordinary names and badly wrong on the expensive ones.
We built NameFew's AI opinion, so we had every reason to want it to look good. We tested it anyway, on sales that happened after everything it had seen, and we're publishing the result as it came out. As far as we could find, nobody else has published a backtest of an automated domain appraiser against later sales. Appraisal tools mostly ask you to trust them. This is a small test, one model, one prompt, one run. It's still a measurement.
How we tested it
We drew 300 .com sales at random from July 1 to October 21, 2026. All of them sold after the site's data was frozen, so nothing in them could have leaked into the model's comparables. Wholesale auction venues were out. The mix is natural, not hand-picked: the median real price was $2,125.
For each name we ran the product itself, the same prompt and the same parser a visitor gets, with comparable sales from January to June only. Then we compared what it said with what the name sold for. Three things were scored: how well it ordered the names by price, how far off the price was, and whether its stated range held the real price.
We scored it against two yardsticks. One was our own statistical model, which only knows length and whether the name is made of dictionary words. The other was a joke predictor: shorter names are worth more, full stop.
The ordering is weak
| Predictor | Rank correlation | Typical price miss |
|---|---|---|
| AI opinion | 0.21 | x1.49 |
| Statistical model | 0.23 | x1.50 |
| "Shorter is higher" | 0.24 | n/a |
| AI and statistical, averaged | 0.25 | n/a |
Rank correlation says how well a predictor sorts names by price: 0 is a coin flip and 1 is perfect. With 300 names the uncertainty is about plus or minus 0.11, so these four numbers are the same number. The AI didn't beat the one-line rule. That's the headline, and it's a little funny: a language model that has read half the internet, matched by "count the letters."
The typical miss of x1.49 means the real price was usually within about one and a half times the AI's estimate, in either direction. If you've priced anything, you know that's not bad for the middle of the market. It's also not what a single dollar figure on a screen suggests.
The ranges are honest. That's the good news.
The AI gives a low, a middle and a high. Its 80% range contained the real price 85% of the time, so it was, if anything, slightly cautious. The middle estimate had no bias: the median real price was 0.97 of the AI's middle figure. The ranges are also wide. The median range was five times from low to high.
A range five times wide is the honest answer to "how much is this worth?" for most names. Anything that gives you a tight number is hiding the width.
Where it falls apart: the expensive names
This is the part the headline correlation hides. Split the 300 sales by what they really sold for:
In the two middle bands the AI is fine. Below $1,500 it's too high and above $3,000 it's too low, which is what any predictor that always hedges toward the middle will do. In the top band it's off by a factor of nearly four, and its range contained the real price only 26% of the time. For the cheapest three bands the range held the price 83% to 98% of the time.
The AI's middle estimate never went above $9,000 for any of the 300 names. Its median across all of them was $2,000. The five biggest misses sold for $24,000 to $40,000, and the AI had said $1,350 to $4,000. Three were numeric names, the kind of thing where the pattern is the value. Two looked like ordinary names: a local diner and a lab. Nothing in the name or the comparable sales gave a model any reason to expect those prices. Somebody wanted them. A buyer's reason isn't written in the letters.
Is a bigger test any better?
A little. The 300-name test was of the AI opinion. A separate, larger test looked at the statistical model after we added signals read by a small language model: what kind of name it is, its category, a brandability score, and so on, none of which ever saw a price. We trained that on 12,932 sales from January to June 2026 and tested it on 6,443 unseen sales from July to October. Rank correlation went from 0.26 to 0.31. Its ability to flag the top 7% of sales, those at $8,000 or more, was 0.78 on a scale where 0.5 is chance. That's real, and the gain is statistically solid, but 0.31 is still a modest number. We tried seven further labels, interactions and gradient-boosted trees after that. None of it helped. That's about the ceiling for a machine reading only the name.
A warning about our own first result
The first time we tested the AI, we got 0.5 to 0.6. That test used a sample deliberately spread across the whole price range, from cheap to very expensive, and it was far easier: sorting a $1,000 name from a $50,000 one is simple. The natural mix is harder, and it's the one a real owner faces. If you see an accuracy claim for a valuation tool, ask what the test set was: unseen sales or old ones, natural mix or stratified, how many names. An easy test can make a mediocre tool look sharp. We found out the embarrassing way.
What this means for you
- Use an AI appraisal for the ordinary middle. If your name looks like thousands of others, a range of roughly $700 to $3,500 around a $2,000 middle is the truth, and the tool will give you something close.
- Don't use one to decide a name isn't valuable. The names it misses are the ones that sell high. If you think yours might be one of them (a short name, a number pattern, something a specific business would need), the tool can't tell you. Ask a buyer, or read how to find real comparable sales.
- Don't trust a precise number. A tool that says "$3,847" is pretending. Use the range.
- Check the test set. Any accuracy figure should say what the model was tested on, and the test should be on sales it hadn't seen.
Where people might do better
Both machines are blind in the same place, the right tail, where the money is. That's where a person might help: someone who knows the diner is a real regional chain, or that four-digit numbers are a market, or that a particular word is hot in an industry. We think people will beat the models there. We haven't measured it, because the human side was built only now and nobody has scored a name yet. We simulated how the scoring would behave, and with only five judges the human middle figure barely beats the robot. The estimate improves as more people score the same name. When real data exists we'll publish it, whichever way it goes.
You can help with the experiment by playing the game. It's one minute: you rank real names, then see what they sold for and how the robot did.
Test sets, for the record. AI opinion: 300 random .com sales from July 1 to October 21, 2026, wholesale venues excluded, comparables from January to June only, model claude-sonnet-5-5, one run. Signals model: trained on 12,932 sales from January to June 2026, tested on 6,443 unseen sales from July to October, median real price $2,000, with bootstrap confidence intervals. We didn't measure any other vendor's tool.