mirror of
https://github.com/soconnor0919/honors-thesis.git
synced 2026-05-08 07:08:55 -04:00
49 lines
3.8 KiB
TeX
49 lines
3.8 KiB
TeX
\chapter{Experimental Evaluation of HRIStudio}
|
|
\label{ch:evaluation}
|
|
|
|
This chapter describes the pilot validation conducted to assess whether HRIStudio meets its design goals in practice. The primary contribution of this thesis is the conceptual framework and reference implementation; the pilot assessment serves to validate that the approach is viable, not to conduct a definitive empirical evaluation.
|
|
|
|
\section{Assessment Goals}
|
|
|
|
The pilot validation addressed two feasibility questions:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Usability}: Can users with no programming experience design and execute Wizard-of-Oz experiments using the system?
|
|
\item \textbf{Technical validity}: Does the system maintain responsive robot control and comprehensive logging during live sessions?
|
|
\end{enumerate}
|
|
|
|
These questions assess whether the reference implementation successfully instantiates the proposed framework, providing evidence that the approach is sound.
|
|
|
|
\section{Pilot Design}
|
|
|
|
The assessment used a within-subjects design with participants from non-technical backgrounds (psychology, education, and related fields). Each participant completed two tasks:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Design task}: Create a simple experiment protocol using the visual design interface
|
|
\item \textbf{Execution task}: Conduct a trial session using the wizard interface, controlling a robot
|
|
\end{enumerate}
|
|
|
|
Task completion, time-on-task, and error rates were recorded. Participants provided feedback via a brief questionnaire.
|
|
|
|
\section{Procedure}
|
|
|
|
Participants attended a 30-minute orientation covering the HRIStudio interface. They then completed the design and execution tasks independently, with the researcher available for technical support. The researcher observed and recorded any usability issues or technical problems. After completing both tasks, participants completed the feedback questionnaire.
|
|
|
|
\section{Results}
|
|
|
|
All participants successfully completed both tasks without requiring assistance beyond initial orientation. Participants designed functional experiment protocols using only the visual interface, confirming that programming knowledge is not required (R2). During execution, the wizard interface guided participants through the protocol, and the robot responded appropriately to commands.
|
|
|
|
The system maintained responsive control throughout all sessions, with no perceptible delay between wizard input and robot action. Comprehensive event logs were generated automatically, capturing every action with millisecond-precision timestamps.
|
|
|
|
Participant feedback was generally positive regarding interface usability, with suggestions for improving the visual design of the protocol editor.
|
|
|
|
\section{Interpretation}
|
|
|
|
The pilot validation confirms that HRIStudio is usable by non-programmers and technically functional for live Wizard-of-Oz experiments. These results support the feasibility of the proposed approach: a web-based framework can enable domain experts to conduct HRI research without programming expertise.
|
|
|
|
This assessment is necessarily limited in scope. A more comprehensive evaluation would involve larger samples, direct comparison with alternative tools, and formal measurement of experimental validity. The focus here is on demonstrating feasibility rather than establishing generalizable findings about the framework's effectiveness.
|
|
|
|
\section{Chapter Summary}
|
|
|
|
This chapter described the pilot validation conducted with the HRIStudio reference implementation. Results indicate that the system is usable by non-programmers and capable of maintaining responsive robot control with comprehensive logging. These findings validate the technical approach while acknowledging that further empirical evaluation is needed to assess the framework's impact on research quality and accessibility. The following chapters conclude the thesis with results, discussion, and directions for future work.
|