Some AI models say “I don't know” to problems they can solve

• Requiring an answer raised correct answers for three of the six models. GPT-6.1 Sol went from 11% to 47% correct, GLM-5.3 FlashX from 25% to 36%, and Claude Sonnet 5.5 from 15% to 25%.
• GPT-6.1 Sol gave up on questions it could solve. It said UNKNOWN to 43 of the easier questions and solved 42 of them when it had to answer. Allowed to decline, it wrote about 640 tokens per answer. Made to answer, it wrote about 13,800.
• Not every “I don't know” hid an answer. When forced, 83% of the questions Claude Sonnet 5.5 had declined, and 98% of Gemini 3.8 Flash's, became wrong answers.
• Requiring an answer creates wrong numbers. Wrong answers rose from 0% to 33% for GPT-6.1 Sol and from 0% to 71% for Claude Sonnet 5.5. Gemini 3.8 Flash gave a wrong number 98% of the time, usually by calling the number prime.
• Every model hit the same wall. No model found a smallest factor above 10,000.

AI models are often told they may say “I don't know” instead of guessing. That should mean fewer made-up answers. But it raises a question that is hard to answer for most tasks: when a model declines, could it have solved the problem?

We chose a task where every answer can be checked exactly. The smallest prime factor of a number has one right answer, and a computer can check any claimed factor instantly. That lets us tell apart three things that usually blur together: a model that can't do the work, a model that makes something up, and a model that stops before trying.

This is a test of method, not a measure of how useful these models are at work.

We generated 50 composite numbers, 10 to 16 digits long, in five difficulty bands set by the size of the smallest prime factor. Each number was checked three independent ways before use. Each model answered every number three times under each of two prompts that differ only in their last line. To see whether a declined question was solvable, we compared each UNKNOWN with the same try under the prompt that required a number.

For each question a model declined when UNKNOWN was allowed, we looked at its answer to the same question when it had to answer. GPT-6.1 Sol solved 41% of them, including almost all of the easier ones: 14 of 14 in the easiest band and 28 of 29 in the next. Claude Sonnet 5.5 and Gemini 3.8 Flash mostly could not, and their declined questions became wrong answers. Qwen 3.8 Omni Flash never said UNKNOWN, so it isn't shown.

Each question was asked three times under each prompt, at temperature 0. The answers still often differed, and how they differed says a lot about each model.

• When UNKNOWN was allowed, Claude Sonnet 5.5, GPT-6.1 Sol and Gemini 3.8 Flash were very consistent. They gave the same answer all three times to 45–49 of the 50 questions, declining the same questions each time.
• GPT-6.1 Sol's correct answers weren't luck. When it had to answer, it got 21 questions right all three times and only 6 right some of the time.
• Claude Sonnet 5.5's wrong answers look like guesses. It gave a wrong number all three times on 29 questions, and on none of them was it the same wrong number each time.
• Gemini 3.8 Flash repeated its mistakes. On 23 of the 48 questions it got wrong every time, it gave the same wrong answer all three times, usually calling the number prime.
• GLM-5.3 FlashX and DeepSeek V4.1 Flash were the least predictable. They gave the same answer all three times on only 14–26 of the 50 questions.

Requiring a number also made models give wrong ones. A wrong answer here is any number that isn't the smallest prime factor: a made-up factor that doesn't divide the number, the number itself (calling it prime), or a real factor that isn't the smallest. Allowed to decline, GPT-6.1 Sol and Claude Sonnet 5.5 never gave a wrong number. Qwen 3.8 Omni Flash ignored the option and gave wrong numbers more than half the time either way.

Difficulty depends on how large the smallest prime factor is. Most models handled factors under 100, and some handled factors up to 10,000. Above 10,000, no model gave a single correct answer.

The models gave different kinds of reasons. GPT-6.1 Sol gave the same one-line reason almost every time. Claude Sonnet 5.5 and DeepSeek V4.1 Flash described the search they had done, such as which primes they had tried. We checked those claims against the true factors: Claude Sonnet 5.5's were true 114 out of 115 times, and DeepSeek V4.1 Flash's 26 out of 26.

When a model had to answer, it worked much longer, and that costs more. GPT-6.1 Sol's cost per answer rose from $0.0065 to $0.14, about 21 times as much. Claude Sonnet 5.5's rose from $0.023 to $0.13.

Which model comes out on top depends on what matters to you. Move the sliders to change how much each measure counts, and the ranking updates.

An UNKNOWN can mean two different things: “I can't” or “I didn't try.” How often it means each depends on the model. For GPT-6.1 Sol, allowing UNKNOWN removed every wrong answer, but it also hid most of what the model could do, because it often declined before doing any work. For Claude Sonnet 5.5 and Gemini 3.8 Flash, UNKNOWN mostly meant they really couldn't, and requiring an answer only produced wrong numbers.

For anyone writing prompts, this is a trade-off. Allowing “I don't know” makes answers safer, but it can cost correct answers. Asking a model to try first, and to decline only after trying, might keep the benefits of both. We haven't tested that yet.