irb-application/context/study_draft.md

Study Protocol Draft

# Study Information

**Principal Investigator:** @Sean O'Connor
**Thesis Adviser:** L. Felipe Perrone
**Robot Platform:** Aldebaran NAO6
**Date:** January 23, 2026

# **1. Experimental Overview**

**Objective:** To quantitatively and qualitatively compare two Wizard-of-Oz (WoZ) interfaces- **Choregraphe** (Industry Standard) and **HRIStudio** (Proposed Platform)- to determine their impact on14: 1. **Disciplinary Accessibility:** Can domain experts without prior robotics experience successfully design a robot interaction?
2. **Scientific Reproducibility:** Does the tool minimize human error and data loss during experiment execution?

## **Experimental Design:**

- **Method:** Between-Subjects User Study.
- **Participants:** N=20 (10 Wizards, 10Subjects).
    - *Wizards:* Faculty from the Computer Science Department (with no prior robot experience).
    - *Subjects:* Undergraduate students (confederates or recruited volunteers).
- **Conditions:**
    - **Group A (Control):** Choregraphe
    - **Group B (Experimental):** HRIStudio

# **2. Task Specification ("Paper Spec")**

*This is the document handed to the Wizard at the start of the Design Phase. It serves as the "Independent Variable" control—every participant attempts to build this exact scenario.*

## **Scenario: "The Geography Quiz Proctor"**

## **Scenario: "The Interactive Storyteller"**

*(Adapted from [Mutlu et al., 2006](https://www.researchgate.net/publication/224060505_A_Storytelling_Robot_Modeling_and_Evaluation_of_Human-like_Gaze_Behavior))*

**Goal:** You must program the robot to tell a short interactive story to the participant. The robot must introduce the story, deliver the narrative with appropriate gestures, ask a comprehension question, and respond to the participant's answer.

**Script & Logic Flow:**

1. **Start State:**
    - Robot is standing.
    - Robot is looking at the participant.
2. **Interaction Step 1: The Hook**
    - **Robot Speech:** "Hello. I have a story to tell you about a space traveler. Are you ready?"
    - **Action:** Robot performs a 'Welcome' or 'Open Hand' gesture.
3. **Interaction Step 2: The Narrative**
    - **Robot Speech:** "The traveler flew to Mars. He found a red rock that glowed in the dark. He put it in his pocket."
    - **Action:** Robot looks away (to the side) while thinking/telling, then looks back at the user. *(This tests the Wizard's ability to sequence gaze/motion).*
4. **Interaction Step 3: The Comprehension Check (Branching)**
    - **Robot Speech:** "What color was the rock the traveler found?"
    - **Branch A (Correct - "Red"):**
        - **Robot Speech:** "Yes! It was a glowing red rock."
        - **Action:** Robot nods.
    - **Branch B (Incorrect - Anything else):**
        - **Robot Speech:** "Actually, it was red."
        - **Action:** Robot shakes head.
5. **Interaction Step 4: Conclusion**
    - **Robot Speech:** "The End. Thank you for listening."
    - **Action:** Robot sits down (or bows).

# **3. Study Procedure (Timeline: 75 Minutes)**

## **Phase 1: Training (15 Minutes)**

- Participant (Wizard) is seated at the control computer.
- **Script:** "You have 15 minutes to learn this software. I will show you how to: 1) Make the robot speak, 2) Make the robot move, and 3) Create a 'Trigger' button."
- *Researcher demonstrates these three core functions only.*

## **Phase 2: Design Challenge (30 Minutes)**

- **Script:** "Here is the 'Paper Spec' for the Geography Quiz. Your task is to implement this interaction in the software. You have 30 minutes. If you get completely stuck, you may ask for a hint, but I will record it."
- **Data Collection:**
    - Start Timer.
    - Count **Help Requests**.
    - Stop Timer when they say "I'm finished" or when 30 mins elapses.

## **Phase 3: The Live Trial (15 Minutes)**

- A "Student Subject" enters the room.
- **Script:** "Please use the interface you just built to run the experiment with this student. Do your best to follow the script exactly."
- **Data Collection:**
    - Video record the robot and the student.
    - Researcher notes **Wizard Errors** (e.g., Wizard clicks 'Correct' when student was 'Wrong').

## **Phase 4: Data Hand-off & Survey (15 Minutes)**

- **Task:** Ask the Wizard to export the video/logs from the session.
- **Survey:** Administer the Post-Study Questionnaire.

# **4. Measurement Rubrics**

*These rubrics generate the quantitative data for the Results section.*

## **A. Design Fidelity Score (Max 10 Points)**

| Component | Criteria | Points |
| --- | --- | --- |
| Completeness | All 4 steps of the script are present in the interface. | 0-3 |
| Logic | Interface allows for *both* Correct and Incorrect paths (Branching). | 0-3 |
| Behaviors | Robot speech and gestures match the Paper Spec exactly. | 0-2 |
| Usability | The interface is labeled clearly (e.g., buttons named "Correct" vs "Button 1"). | 0-2 |
| Total |  | / 10 |

## **B. Execution Reliability Score (Max 10 Points)**

| Component | Criteria | Points |
| --- | --- | --- |
| Timing | Robot responded to the student within 3 seconds. | 0-3 |
| Accuracy | Wizard triggered the correct branch (Correct vs Incorrect). | 0-3 |
| Stability | No software crashes, freezes, or connection losses. | 0-2 |
| Fideilty | Robot did not improvise or say things outside the script. | 0-2 |
| Total |  | / 10 |

# **5. Post-Study Questionnaire**

## **Part 1: System Usability Scale (SUS)**

*Participants rate the following on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree).*

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

## **Part 2: Reproducibility & Confidence**

*Specific to the Thesis goals.*

1. **Data Confidence:** "If I had to analyze the data from this session tomorrow, I am confident I could find the logs and video." (1-5)
2. **Sharing:** "If I sent this project file to a colleague at another university, how confident are you that they could run it without my help?" (1-5)
3. **Role Fit:** "I felt that I could focus on the *student* rather than fighting the *software*." (1-5)

# **6. Required Equipment & Setup**

- **Room:** Quiet lab space with two desks (one for Wizard, one for Subject).
- **Hardware:**
    - 1x Softbank NAO6 Robot.
    - 1x Wizard Laptop (Pre-loaded with HRIStudio Server).
    - 1x Control Laptop (Pre-loaded with Choregraphe).
    - 1x External Camera (for recording the session).
- **Software:**
    - **Group A:** Choregraphe 2.8.
    - **Group B:** HRIStudio (v1.0 Release Candidate).