Loading...
An empirical study of Traditional UI and Conversational AI using the Model Context Protocol (MCP), evaluating usability, cognitive workload, psychological experience, and user preference.
This stage of our study compares the traditional graphical interface and the MCP-based conversational system across usability, cognitive workload, psychological experience, and user preference for all completed participants (N=142) and a specialized subgroup with advanced technical proficiency (N=18).
Mean SUS scores were calculated and compared across systems and participant populations to evaluate system usability.
| System | All ParticipantsN = 142 | Advanced TechnicalN = 18 |
|---|---|---|
| Chat-based System | 62.7 | 74.0 |
| Traditional UI | 68.9 | 62.9 |
Key Finding
While the Traditional UI received higher overall usability ratings, technically advanced users preferred the Chat-Agent, which reached acceptable usability thresholds for this group. Experienced users appear better equipped to leverage intent-driven conversational workflows.
Cognitive workload was assessed using the Raw-TLX across six workload dimensions.
*Performance: higher value = better self-assessed task performance (not reverse-coded in this dimension view).
Key Finding
The Chat-Agent imposes a higher cognitive load on general users, notably increasing mental demand and frustration. In contrast, advanced users show significantly better adaptability, with workload levels paritying the traditional interface and even showing reduced frustration.
Psychological experience was evaluated using questionnaire items derived from Self-Determination Theory (SDT), measuring Autonomy, Competence, Performance Satisfaction, and System Satisfaction.
Key Finding
General participants reported higher autonomy and competence with the Traditional UI, while advanced users felt more empowered and efficient using the Chat-Agent. Despite these gains, overall satisfaction remained higher for the Traditional UI, highlighting a disconnect between interaction flexibility and outcome confidence.
User preferences were evaluated through comparative survey questions measuring perceived control, predictability, clarity of system actions, and trust.
Key Finding
Control and trust remain strongly associated with the Traditional UI for all users. However, we see a clear preference shift among advanced users, who favor the Chat-Agent for its superior clarity and predictability compared to traditional methods.
Participants were asked which interaction modality they would prefer depending on the task.
| Preference | All Participants | Advanced |
|---|---|---|
| Chat for simple tasks, UI for complex tasks | 39% | 67% |
| Traditional UI for most tasks | 39% | 17% |
| Chat-based system for most tasks | 17% | 17% |
| No clear preference | 5% | — |
Key Finding
Users across all skill levels overwhelmingly prefer a hybrid interaction model. While general users still lean heavily on traditional UIs, the data suggests conversational agents are most effective as specialized supplements rather than total replacements.