Study Protocol Draft

# Study Information

**Principal Investigator:** @Sean O'Connor
**Thesis Adviser:** L. Felipe Perrone
**Robot Platform:** Aldebaran NAO6
**Date:** January 23, 2026

# **1. Experimental Overview**

**Objective:** To quantitatively and qualitatively compare two Wizard-of-Oz (WoZ) interfaces- **Choregraphe** (Industry Standard) and **HRIStudio** (Proposed Platform)- to determine their impact on:

1. **Disciplinary Accessibility:** Can non-technical domain experts successfully design a robot interaction?
2. **Scientific Reproducibility:** Does the tool minimize human error and data loss during experiment execution?

## **Experimental Design:**

- **Method:** Between-Subjects User Study.
- **Participants:** N=20 (10 Wizards, 10Subjects).
    - *Wizards:* Faculty or Graduate Students from non-Computer Science disciplines (e.g., Psychology, Education).
    - *Subjects:* Undergraduate students (confederates or recruited volunteers).
- **Conditions:**
    - **Group A (Control):** Choregraphe
    - **Group B (Experimental):** HRIStudio

# **2. Task Specification ("Paper Spec")**

*This is the document handed to the Wizard at the start of the Design Phase. It serves as the "Independent Variable" control—every participant attempts to build this exact scenario.*

## **Scenario: "The Geography Quiz Proctor"**

**Goal:** You must program the robot to act as a quiz proctor for a single student. The robot will ask a question, wait for an answer, and provide feedback based on whether the student is right or wrong.

### **Script & Logic Flow:**

1. **Start State:**
    - Robot is standing.
    - Robot LED eyes are Blue.
2. **Interaction Step 1: Introduction**
    - **Robot Speech:** "Hello. I am the Quiz Bot. Please sit down so we can begin.”
    - **Action:** Robot waves its right hand.
3. **Interaction Step 2: The Question**
    - **Robot Speech:** "Question One. What is the capital city of Pennsylvania?"
    - **Wizard Action:** Wait for the student to speak.
4. **Interaction Step 3: Branching Logic (The Wizard's Choice)**
    - *If Student says "Harrisburg" (Correct):*
        - **Robot Speech:** "That is correct. Well done."
        - **Action:** Robot nods head.
    - *If Student says anything else (Incorrect):*
        - **Robot Speech:** "I am sorry, that is incorrect. The answer is Harrisburg."
        - **Action:** Robot shakes head.
5. **Interaction Step 4: Conclusion**
    - **Robot Speech:** "Thank you for participating. Goodbye."
    - **System Action:** Save the log of the interaction.

# **3. Study Procedure (Timeline: 75 Minutes)**

## **Phase 1: Training (15 Minutes)**

- Participant (Wizard) is seated at the control computer.
- **Script:** "You have 15 minutes to learn this software. I will show you how to: 1) Make the robot speak, 2) Make the robot move, and 3) Create a 'Trigger' button."
- *Researcher demonstrates these three core functions only.*

## **Phase 2: Design Challenge (30 Minutes)**

- **Script:** "Here is the 'Paper Spec' for the Geography Quiz. Your task is to implement this interaction in the software. You have 30 minutes. If you get completely stuck, you may ask for a hint, but I will record it."
- **Data Collection:**
    - Start Timer.
    - Count **Help Requests**.
    - Stop Timer when they say "I'm finished" or when 30 mins elapses.

## **Phase 3: The Live Trial (15 Minutes)**

- A "Student Subject" enters the room.
- **Script:** "Please use the interface you just built to run the experiment with this student. Do your best to follow the script exactly."
- **Data Collection:**
    - Video record the robot and the student.
    - Researcher notes **Wizard Errors** (e.g., Wizard clicks 'Correct' when student was 'Wrong').

## **Phase 4: Data Hand-off & Survey (15 Minutes)**

- **Task:** Ask the Wizard to export the video/logs from the session.
- **Survey:** Administer the Post-Study Questionnaire.

# **4. Measurement Rubrics**

*These rubrics generate the quantitative data for the Results section.*

## **A. Design Fidelity Score (Max 10 Points)**

| Component | Criteria | Points |
| --- | --- | --- |
| Completeness | All 4 steps of the script are present in the interface. | 0-3 |
| Logic | Interface allows for *both* Correct and Incorrect paths (Branching). | 0-3 |
| Behaviors | Robot speech and gestures match the Paper Spec exactly. | 0-2 |
| Usability | The interface is labeled clearly (e.g., buttons named "Correct" vs "Button 1"). | 0-2 |
| Total |  | / 10 |

## **B. Execution Reliability Score (Max 10 Points)**

| Component | Criteria | Points |
| --- | --- | --- |
| Timing | Robot responded to the student within 3 seconds. | 0-3 |
| Accuracy | Wizard triggered the correct branch (Correct vs Incorrect). | 0-3 |
| Stability | No software crashes, freezes, or connection losses. | 0-2 |
| Fideilty | Robot did not improvise or say things outside the script. | 0-2 |
| Total |  | / 10 |

# **5. Post-Study Questionnaire**

## **Part 1: System Usability Scale (SUS)**

*Participants rate the following on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree).*

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

## **Part 2: Reproducibility & Confidence**

*Specific to the Thesis goals.*

1. **Data Confidence:** "If I had to analyze the data from this session tomorrow, I am confident I could find the logs and video." (1-5)
2. **Sharing:** "If I sent this project file to a colleague at another university, how confident are you that they could run it without my help?" (1-5)
3. **Role Fit:** "I felt that I could focus on the *student* rather than fighting the *software*." (1-5)

# **6. Required Equipment & Setup**

- **Room:** Quiet lab space with two desks (one for Wizard, one for Subject).
- **Hardware:**
    - 1x Softbank NAO6 Robot.
    - 1x Wizard Laptop (Pre-loaded with HRIStudio Server).
    - 1x Control Laptop (Pre-loaded with Choregraphe).
    - 1x External Camera (for recording the session).
- **Software:**
    - **Group A:** Choregraphe 2.8.
    - **Group B:** HRIStudio (v1.0 Release Candidate).