MLLM as a UI Judge: Predicting Human Perception of User Interfaces

Summary: AI UX Research Report for Adobe Creative Cloud Apps

Summary: UX Research Report for Adobe
Customer Journey Analytics

Summary: UX Research Report
for Adobe Customer
Journey Analytics

PROJECT SUMMARY

PROJECT SUMMARY

COMPANY

Adobe Research

Adobe Research

ROLE

Lead Researcher / First Author

Lead Researcher / First Author

QUAL METHODS

Contextual Inq · RITE testing · Semi-structured interviews

Contextual Inq · RITE testing · Semi-structured interviews

QUANT METHODS

Likert scoring · Pairwise A/B · Avg Error · Model–human agreement · Data Viz

Likert scoring · Pairwise A/B · Avg Error · Model–human agreement · Data Viz

TIMELINE

Phase 1: 3 RITE rounds · Phase 2: benchmark study

Phase 1: 3 RITE rounds · Phase 2: benchmark study

STUDY SCALE

10 designers · 30 interfaces · 3 MLLMs · 9 UI factors

10 designers · 30 interfaces · 3 MLLMs · 9 UI factors

Research Methods

Research Methods

Qualitative and Quantitative

Qualitative and Quantitative

How do Designers use AI in Adobe Workflows for evaluation?

How do Designers use AI in Adobe Workflows for evaluation?

Research Methods Timeline

Research Methods Timeline

PHASE 1: Qualitative Research

PHASE 1: Qualitative Research

PHASE 2: Quantitative Research

PHASE 2: Quantitative Research

Research Methods timeline with Phase 2 emphasized: survey with humans, survey with AI, and compare results.

Qualitative Research

Qualitative Research

Contextual Inquiry Insights

Contextual Inquiry Insights

Designers treat AI feedback as a fast gut-check, not a verdict.

Designers treat AI feedback as a fast gut-check, not a verdict.

Usability Testing

Usability Testing

Prototype Testing:

Prototype Testing:

Deliverables

Deliverables

Persona: Understanding the Target User

Persona: Understanding the Target User

Customer Journey Map

Customer Journey Map

DESIGN RECOMMENDATIONS

Make feedback actionable

Make feedback actionable

Make feedback actionable

1. Keep feedback in the workflow

Keep critique inside the design tool so designers never have to leave their editing flow.

2. Anchor critique to the interface

Point to the exact interface region and pair each critique with a clear next action.

3. Adapt feedback across rounds

Start with full rationale, then shift to concise “what changed” summaries in later rounds.

Quantitative Research

Quantitative Research

Methods

Methods

Study Information (Human Ground Truth)

Study Information (Human Ground Truth)

Validating AI use for Design evaluation

Validating AI use for Design evaluation

Quantitative Results

Quantitative Results

How close were the MLLMs to Human Scores?

How close were the MLLMs to Human Scores?

MLLMs do increasingly well with A/B Testing

MLLMs do increasingly well with A/B Testing

Quantitative Study Takeaways

Quantitative Study Takeaways

Major Learnings and Takeaways

Major Learnings and Takeaways

Outcomes/Deliverables Summary

Outcomes/Deliverables Summary

Create a free website with Framer, the website builder loved by startups, designers and agencies.