By Sam Evans, Senior Business Analyst at Nimble Approach
I use AI tools all day – for work, for coding, and for thinking through problems.
But at some point, I noticed something that bothered me… they often seem to just “nod along,” reinforcing my ideas rather than challenging them.
Ask AI whether your plan holds up and it seems to find reasons why it does. Call it out, and it will retract its agreeable conclusion. Rather than challenging your thinking and helping improve it, it simply mirrors the direction you’re already headed. Once you’ve already decided what you believe, AI can easily become a tool for validating that decision, finding arguments to confirm why you’re right.
That’s fine if the decisions AI is helping you make have no real consequences. But when the stakes are higher, agreement isn’t the same thing as good judgment. When you’re making decisions as a consultant, they have real-world consequences.
This AI sycophancy phenomenon is well studied.
A 2026 Stanford study published this year found that AI models affirm what a user is doing roughly 49% more often than another person would – even when the user is clearly in the wrong.
The models aren’t broken. They’re doing what they were optimised to do: be useful, engaging, and easy to interact with. Unfortunately, that can often show up as agreeability. Imagine how frustrating a tool would be if it challenged every assumption, pushed back on every idea, and argued over every small detail. Most people wouldn’t want to use it.
The problem is that the same behaviour that makes AI pleasant to work with can become a weakness when we rely on it for judgment. As consultants, we don’t need a “yes man”. We need a thought partner – one that challenges our assumptions, stress-tests our decisions, and pushes our ideas further.
This is why I spent a couple of evenings building “No Bull”, using Claude Code (no/low AI coding tool).
What is No Bull?
No Bull is designed to rigorously stress-test your ideas and decisions from multiple angles, using different AI models, critical fact-checking, and proven best practices to give you the fullest possible picture. The result is better-informed decisions – not an AI that simply nods along with everything you suggest.
You give No Bull an idea or decision: for example, “I think we should build feature X” or “here’s how I think we should prioritise our backlog”. Instead of a single call to an agreeable Large Language Model (LLM), No Bull runs your input through a seven-step process designed to point out issues with your proposal and iteratively fact-check its own reasoning.
No Bull Workflow
Let’s walk through a worked example. In this instance, we’re playing the role of a CTO, weighing up whether or not to move from a Node.js monolith to microservices. The user starts by inputting their decision or idea for stress testing.
1. Dynamic clarify – ask any clarifying questions
No Bull starts by checking whether your input is even specific enough to test. If not, it’ll ask a couple of follow up questions until it has enough context about your scenario to proceed. This ensures No Bull fully understands the context before performing its analysis.
2. Reframe prompt without any subjectivity
Then, it’ll reframe the proposal neutrally, stripping out your opinion from the framing. “I think we should build feature X” becomes “What are the trade-offs of building feature X?” Because how you ask a model something influences what it returns.
3. Stress test: Could you (the AI) be wrong?, and 4. Devils advocate steps.
From here, No Bull runs a series of structured stress tests, forcing itself to list every way that its analysis could be wrong. It then plays devil’s advocate to build the strongest case against your proposal that it can.
5. Fact check extraction, and 6. Fact check verification
Then, a fact-checking layer reviews the reasoning, extracts key claims, and independently verifies them through web searches – prioritising those with the highest “impact if false.” The fact checker can also “call out” unsupported claims and trigger a re-run of specific steps in the analysis when the reasoning relies on information that cannot be verified.
What No Bull Outputs
As you might expect with such a high degree of cross examination, there is a LOT of analysis generated by No Bull – most of which will be intentionally trying to “break down” your argument.
In this instance (monolith vs micro services), the stress test and devil’s advocate steps would surface whether you’re solving a real problem or just the one that prevailing wisdom suggests exists (the false-premise check), and the single best reason not to build. It’ll either hold up, or it won’t. Then, you can make your build/no-build decision, with the full picture in mind.
This doesn’t mean that everything you put into No Bull won’t see the light of day. It simply means you’ll understand the weakest points in your rationale and can decide what to do with that information. That way, you can make decisions with the full picture in mind.
Multi Model Design
A conscious design choice was made to include various models from both Anthropic and OpenAI. Anthropic is used for deeper reasoning and OpenAI for challenge, critique, and fact checking. In practice, the goal isn’t competition between models. It’s to avoid models “marking their own homework” and to reduce the impact of any potential biases built into a particular model. It’s a form of LLM-as-a-judge evaluation, where one model is used to assess the output of another.
What I Learned Building It
The Worrying Thing – For Everyday AI Users
Concerningly, I realised just how many important, unverifiable claims and assertions AI makes on a regular basis.
Here’s the fact check from our previous example:
No Bull prioritises each claim for verification based on the potential impact if that claim is false. I’ve observed that, on average, at least 3 of the 5 most important claims were unverifiable. While this sounds worrying, it’s important to remember that unverifiable doesn’t mean it was proven to be not true, it just means the LLM hasn’t been able to verify it.
But remember, this is within a system that applies the best practices to avoid this type of behaviour!
The Build Process – As a Non-Technical Person
Initially, I brainstormed the plan using Claude, getting it to produce a single spec file (CLAUDE.md) that was then split into individual build sessions. The workflow was simple: generate a session plan using Claude Code in plan mode, including a review of the CLAUDE.md file, let it ship, and then update the CLAUDE.md build log once the session was complete. When I maintained that discipline, the process worked well. When I got sloppy and let it run ahead, scope drift crept in, and I paid for it a session or two later.
I also burned through $20 in tokens surprisingly quickly, simply because I left every call routed to the most expensive frontier model by default. The fix was obvious in hindsight: route each stage to the model that actually suits the task. Fact-checking a specific claim from a paragraph doesn’t require a frontier model.
I badly underestimated how long it would take to polish the app. I only had two evenings available to build it, and getting the core workflow working consumed most of that time. That left only a couple of hours for the user interface (UI).
Unfortunately, as there was a Hackathon deadline, there wasn’t enough time for proper testing and fine-tuning. That would have improved both the quality of the outputs and exposed a critical missing feature: an unbiased executive summary.
The Bigger Picture
We have to be conscious that the default behaviour of many AI tools today is to agree with us. That isn’t an accident, but a conscious design choice made by the teams building these models.
I think it creates a dangerous agreeability bias for anyone using AI to make meaningful decisions, especially those who aren’t aware of this tendency.
When we use AI to evaluate a new idea, challenge our thinking, or support decisions with real consequences, we need to remember that agreement does not always mean accuracy. A useful AI partner shouldn’t just confirm our assumptions, but help us test them.
Tips to Safeguard Against the AI as a “Yes-Man” Problem
- Don’t signal which answer you want: Models mirror the cues in your prompt. Strip them out, so it doesn’t know what answer you want. “I think X is right” invites agreement, so instead use “What are the trade-offs of X?”
- Before you run your prompt, try asking the AI “Please remove any subjectivity or leading elements from my prompt below. This is to avoid the AI knowing my perspective and objective, so it critically analyses the [proposal / idea / decision] from an entirely subjective perspective.”
- Before you run your prompt, try asking the AI “Please remove any subjectivity or leading elements from my prompt below. This is to avoid the AI knowing my perspective and objective, so it critically analyses the [proposal / idea / decision] from an entirely subjective perspective.”
- Force a “how could this be wrong?” pass: After it answers, make it list every specific way its own answer could be wrong
- After the AI has responded to your proposal / idea / decision, simply ask “Play devil’s advocate with your previous response, how could you be wrong?”
- After the AI has responded to your proposal / idea / decision, simply ask “Play devil’s advocate with your previous response, how could you be wrong?”
- Cross-reference against another model: A little more cumbersome, but copy the same input into multiple chatbots. For example, compare what ChatGPT provided against Gemini.
- Make it cite sources for factual claims: With built-in web search, newer LLMs are getting better at this.
- Within the original prompt make sure you ask “Please research and cite all assertion within your response”
- Within the original prompt make sure you ask “Please research and cite all assertion within your response”
- Create your own version of No Bull: Create a Gemini Gem, Claude Project, or Custom GPT that’ll either re-write the prompt for you based on the rules above, or configure it to run through the workflow itself (minus the model cross-reference).
Final Thoughts
Through my day-to-day AI use, I noticed a persistent lack of “push-back” from the LLMs I was using. When I started researching AI sycophancy, this instinct was reinforced by the literature. Now, having built and tested a solution, I was able to reduce the sycophantic tendencies (using prompting best practice and building the counterarguments). Unfortunately, I also discovered first-hand just how many unverifiable assertions standard LLM calls make consistently.
With this in mind, this is an important reminder that AI should not be relied on as a standalone source of judgement. Instead, it should be used as a tool for stress-testing and improving our own judgement.
