Don't let AI make your UI design decisions

TL;DR

AI is well suited to supporting design decisions, but not to making them for you .

The problem:

• AI only “reads” static mockups; it doesn't understand interaction, task flows, or consistency across screens [3].
• When asked about actual behavioral impact (trust, cognitive load, conversion), AI still has very limited understanding [4].
• In the UICrit evaluation using few-shot and visual prompting, raters preferred human critiques in 81% of comparisons against the study’s best LLM configuration.
• AI readily produces false positive (~24%) and tends to “guess” beyond the data it sees [6].
• Fluent prose combined with sycophancy can lead teams to trust too much without verification [7][8].

Benefits when used in the right role:

• AI is useful as copilot: checking surface issues (spacing, contrast, labeling), writing microcopy, expanding ideas, and supporting a first-pass review [1][9].
• AI is highly reliable only when the problem is narrow enough and the rubric is clear enough, as when Baymard achieved 95% accuracy with 15+ narrow systems instead of LLM reasoning [10].

How to apply it appropriately:

• Frame the problem around a specific scenario (persona + goal + flow), rather than asking a vague question [11].
• Send screens in sequence with a complete context package (business goal, constraints, design intent, prior research) [11].
• Write criteria in terms of appropriateness, rather than absolute rules; add your team's own principles [12][13].
• Require AI to return in a structured format (observation, severity, confidence, assumption) and challenge your brief as well [11][7].
• Treat AI as a first-pass reviewer; for major decisions, still verify through usability testing or real data [14].

AI is becoming familiar in designers' workflows: writing microcopy, suggesting layouts, checking UI issues, and even offering quick screen critiques. But there is an important boundary design teams should not overlook: AI is useful for supporting UI evaluation, but isn't yet reliable as the final judge of design quality.

Nielsen Norman Group describes AI as a UX assistant with roles such as content editor, research assistant, ideation partner, and design assistant - supporting roles, rather than a replacement for human design judgment [1].

AI helps you think faster. But thinking better, in many cases, is still a human responsibility.

The issue isn't using AI, but giving it the wrong role

What we should avoid today isn't “using AI in UI design,” but using AI as the primary evaluator or the final authority trong design review.

A meaningful design critique doesn't rate a screen as “beautiful or ugly”; it assesses whether the design achieves its goals. Nielsen Norman Group defines critique as analyzing a design and providing feedback on whether it meets its objectives [2].

To critique well, you need context:

• Who are the users? Are they new or experienced?
• What are they trying to accomplish?
• What is the business optimizing for?
• Which risks are being deliberately accepted?
• How does this flow affect actual behavior?

Without these layers of context, feedback can easily become comments that “sound reasonable” but remain shallow [2]. And that is precisely what today's AI often lacks when it sees only screenshots or mockups.

What does research say about AI's actual design review capabilities?

Before discussing how to use it, let's look at the evidence. Here is a summary of recent studies:

Let's examine each finding.

AI “reads representations” of UI; it doesn't actually “use” it

A CHI 2024 study from UC Berkeley found that a UI evaluation system relies on structured representations of the interface (JSON/XML) and heuristics to detect violations, then turns those violations into constructive advice. But context-window limits mean the tool can evaluate only static mockups, one screen at a time [3].

AI is reading traces of an experience, rather than truly understanding the experience over time.

AI is strong on surface signals, weak on behavioral impact

WiserUI-Bench was built on 300 real UI pairs from A/B tests, with winners validated by user behavior. The conclusion: current MLLMs still have limited understanding of the behavioral impact of UI/UX design [4].

AI can see a more prominent CTA, an airier layout, or clearer hierarchy. But inferring whether users will trust more, hesitate less, decide faster, hay make fewer mistakes remains difficult [4].

A “clean” screen isn't necessarily better than a “dense” one. A shorter flow isn't necessarily easier to use. A prettier interface isn't necessarily more trustworthy.

Beyond the pixels lie behavior, usage context, familiarity, motivation, and psychological barriers, and sometimes even business strategy.

Few-shot and visual prompting results in UICrit

UICrit (UIST 2024) contains 3,059 critiques for 983 mobile UIs. In the authors’ evaluation, few-shot combined with visual prompting improved the usefulness of LLM feedback by 55% over the zero-shot configuration. This result applies to the study setup, not every AI tool.

In the same study, human critiques were preferred in 81% of comparisons with the best LLM configuration. The experiment used examples in prompts, not model fine-tuning.

AI readily produces false positives and “guesses” what it cannot see

An INTERACT 2025 study compared GPT-4o with HCI experts:

• Only 21,2% of the issues found by experts were also detected by GPT-4o.
• 24,3% of the issues GPT-4o raised were classified as false positives [6].

GPT-4o tended to “guess” interaction problems it couldn't observe from screenshots alone, or make comments too general to verify [6]. This is a risk when teams ask AI broad questions such as “audit this screen for me”.

A “reasonable tone” can make teams trust too much

The greatest risk isn't that AI is wrong. It's that it is wrong in a very convincing way.

Anthropic describes sycophancy as a model's tendency to give answers that match the user's expectations rather than the truth; this behavior is widespread in RLHF models [7]. NN/g warns that users often struggle to check chatbot output for errors [8]. Microsoft Research surveyed 319 knowledge workers with 936 real-world examples and found that confidence in AI was associated with less engagement in critical thinking [8].

In a design review, an AI-written critique can be fluent enough to shut down debate, but not accurate enough to be given decision-making authority [7][8].

So how should AI be used in UI design?

The best approach today is to treat AI as copilot: helping expand the design space, check surface issues, write and edit microcopy, support first-pass reviews, and bring precedents and patterns within reach. NN/g emphasizes that narrow-scope are the most useful features of current design AI [1][9].

The broader the question, the more AI tends to fill gaps with plausible but unsupported reasoning [9].

An interesting exception: AI is reliable only when the problem is narrowly constrained

Baymard Institute reported that UX-Ray achieved 95% accuracy on standardized e-commerce heuristics [10]. But their approach is what matters:

• UX-Ray they do not use LLMs to perform UX reasoning.
• They do not use generative AI to produce UX analysis or improvement suggestions.
• Instead, they use a sequence of more than 15 narrow systems; generative AI is used only in a tightly limited role to classify a few UI patterns against a predefined set of answers [10].

This exception doesn't prove that “general-purpose AI is ready to judge design.” It reinforces the opposite: AI is highly reliable only when the problem is narrow enough, the rubric is clear enough, and its scope for judgment is tightly constrained.

A practical guide: Using AI to evaluate UI appropriately

Rather than throwing a screenshot at AI and asking “evaluate this,” a more sensible approach is to: have AI review a specific objective within a specific flow against clear criteria, then use it as a first-pass reviewer rather than the final evaluator [11].

Prepare the input for AI

1. Frame a scenario, not a generic question

Don't ask: “Evaluate this UI for me.”

Ask: “For persona A, doing task B, in flow C, what risks does this screen pose under the following heuristics?”

Microsoft UXR explicitly recommends providing a specific persona and goal, so AI can understand what matters in that workflow [11].

2. Send the complete screen sequence in flow order

If reviewing a flow, send the entire screenshot sequence in the correct order, rather than one isolated image. Microsoft UXR recommends that images be clear, in-sequence to reduce misinterpretation [11]. The CHI 2024 study also highlights a major limitation of current AI reviews: evaluating only static mockups, one screen at a time [3].

3. Include a complete context package

Alongside screenshots, provide at least:

• What the product is and who uses it
• Where users are in the flow
• The screen's objective and business goal
• Constraint (design system, tech, legal)
• The current design intent
• Insights from earlier research

Microsoft UXR recommends including details not visible on the screen such as prior research, stakeholder priorities, and terminology [11]. NN/g also emphasizes that AI design tools lack sufficient context about business goals, user needs, and the existing product when the prompt contains too little information [9].

Set evaluation criteria

4. Write criteria around appropriateness, not absolute rules

Instead of “the UI must have fewer elements,” ask: “Is the current element density appropriate for this persona and task?”

Instead of “it must be simple,” ask: “Are the hierarchy and scan path clear enough for users to understand and act?”

Instead of “optimize cognitive load,” ask: “Are there signs of unnecessary cognitive load, such as too many equally prominent choices, poor chunking, competing CTAs, or copy that is hard to scan?”

Heuristics are broad rules of thumb. NN/g emphasizes applying them in context; the same principle may be applied very differently in a complex app for experienced users than in an onboarding flow for newcomers [12].

5. Add principles specific to your team and product

NN/g makes a clear distinction between heuristics, design principles, patterns, and team charters [13]. Alongside general principles, add product-specific ones, for example:

• “Prioritize trust over speed”
• “Don't sacrifice data scannability for a minimal appearance”
• “Every primary CTA must be clear within the first 3 seconds”

Control and verify the output

6. Require structured output rather than impressions

Request a clear format for each comment:

1. The issue identified
2. Why it matters
3. The heuristic or principle affected
4. Visible evidence on the screen
5. Assumptions being inferred rather than confirmed
6. Priority (Low / Medium / High)
7. Confidence (Low / Medium / High)

Both Microsoft UXR and NN/g warn that AI can generate false positives, and heuristic reviews in general can produce many minor issues that dilute the focus without severity filtering [11][14].

7. Ask AI to challenge your brief

Add to the prompt: “Don't automatically agree with my assumptions; point out where my criteria may be biased toward minimalism or inappropriate for the context.”

Anthropic's research shows that sycophancy is common in RLHF models [7]. When you provide a belief such as “the UI must have fewer elements,” AI can easily be drawn into confirming it unless you actively ask for counterarguments.

8. Treat AI as a first-pass reviewer, not the final authority

NN/g says heuristic evaluation is most useful for finding problems early, especially with limited research budgets, but it cannot replace user research because UX is deeply contextual [14]. Microsoft UXR likewise concludes that today's AI should be used as assistant to jumpstart reviews, especially for less experienced people [11].

9. Major decisions still need verification with real people

If AI feedback leads to changes in UI density, action priorities, onboarding, trust, or how users make decisions, you should still check through usability testing, hallway testing, analytics, or A/B tests. NN/g is clear: heuristic evaluation only complement research, rather than replacing research [14].

A sample prompt framework for AI UI reviews

You are a first-pass UI reviewer, not the final decision-maker.

Before critiquing, do two things:
1. Summarize your understanding of the persona, goal, flow, and design intent.
2. List what you cannot know from screenshots alone.

Context:
- Product:
- Platform:
- Persona:
- User experience level:
- User goal:
- Flow / current step in the flow:
- Frequency of this task:
- Risk if the user makes a mistake:
- Business goal:
- Current design intent:
- Constraints / design system / technical constraints:
- Known research insights:
- Information not visible in screenshots but needed for the review:

Input:
- Screenshot / sequence of screenshots in order

Evaluate against these criteria:
1. Clarity of purpose and visual hierarchy
2. Appropriateness of information density for this user and task
3. Signs of unnecessary cognitive load
4. Scan path and competition for attention
5. Action priority and affordance
6. Copy / label clarity
7. Error prevention and recoverability
8. Consistency with the product's design principles

For each comment, return:
- Observation
- Why it matters
- Relevant heuristic / principle
- Evidence visible on screen
- Assumptions you are making
- Severity: Low / Medium / High
- Confidence: Low / Medium / High
- Recommendation

Finally, provide:
- 3 things working well
- The 5 highest-priority issues
- 3 things that need user testing to verify
- 3 ways my brief may be biased
- If “simplicity” conflicts with “enough information,” explain the trade-off instead of defaulting to minimalism.

Conclusion

AI is well worth using in UI design, but it is most valuable as a copilot, rather than the final approver. It helps you think faster, expand options, perform an initial review, and save time on repetitive tasks. But when the question concerns user behavior, trust, cognitive load, product trade-offs, or business impact, human judgment remains the most important decision-making layer [1][4][9].

Use AI to accelerate your thinking. Don't use it to avoid responsibility for design judgment.

References

[1] Nielsen Norman Group, AI as a UX Assistant. nngroup.com/articles/ai-roles-ux

[2] Nielsen Norman Group, Design Critiques: Encourage a Positive Culture to Improve Products. nngroup.com/articles/design-critiques

[3] Duan et al., Generating Automatic Feedback on UI Mockups with Large Language Models (CHI 2024, UC Berkeley). people.eecs.berkeley.edu

[4] Jeon et al., Do MLLMs Capture How Interfaces Guide User Behavior? / WiserUI-Bench. arxiv.org/html/2505.05026v4

[5] Duan et al., UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset (UIST 2024). people.eecs.berkeley.edu

[6] Guerino et al., Can GPT-4o Evaluate Usability Like Human Experts? (INTERACT 2025). arxiv.org/abs/2506.16345

[7] Anthropic, Towards Understanding Sycophancy in Language Models. anthropic.com/research/towards-understanding-sycophancy-in-language-models

[8] Nielsen Norman Group, AI Chatbots Discourage Error Checking; Microsoft Research, The Impact of Generative AI on Critical Thinking. nngroup.com/articles/ai-chatbots-discourage-error-checking

[9] Nielsen Norman Group, AI Design Tools Are Marginally Better: Status Update. nngroup.com/articles/ai-design-tools-update-2

[10] Baymard Institute, AI Heuristic UX Evaluations with a 95% Accuracy Rate. baymard.com/blog/ai-heuristic-evaluations

[11] Microsoft UXR / Jackie Ianni, Why AI Tools Are Not Ready to Replace Human Heuristic Evaluations, Yet. medium.com/uxr-microsoft

[12] Nielsen Norman Group, 10 Usability Heuristics Applied to Complex Applications. nngroup.com/articles/usability-heuristics-complex-applications

[13] Nielsen Norman Group, Design Guidance: Principles, Patterns, Heuristics, and Team Charters. nngroup.com/articles/design-guidance

[14] Nielsen Norman Group, How to Conduct a Heuristic Evaluation. nngroup.com/articles/how-to-conduct-a-heuristic-evaluation