hristudio/docs/roman-2025-talk.md

A Web-Based Wizard-of-Oz Platform for Collaborative and Reproducible Human-Robot Interaction Research

1) Introduction

• HRI needs rigorous methods for studying robot communication, collaboration, and coexistence with people.
• WoZ: a wizard remotely operates a robot to simulate autonomous behavior, enabling rapid prototyping and iterative refinement.
• Challenges with WoZ:
  • Wizard must execute scripted sequences consistently across participants.
  • Deviations and technical barriers reduce methodological rigor and reproducibility.
  • Many available tools require specialized technical expertise.
• Goal: a platform that lowers barriers to entry, supports rigorous, reproducible WoZ experiments, and provides integrated capabilities.

2) Assessment of the State-of-the-Art

• Technical infrastructure and architectures:
  • Polonius: ROS-based, finite-state machine scripting, integrated logging for real-time event recording; designed for non-programming collaborators.
  • OpenWoZ: runtime-configurable, multi-client, supports distributed operation and dynamic evaluator interventions (requires programming for behavior creation).
• Interface design and user experience:
  • NottReal: interface for voice UI studies; tabbed pre-scripted messages, customization slots, message queuing, comprehensive logging, familiar listening/processing feedback.
  • WoZ4U: GUI designed for non-programmers; specialized to Aldebaran Pepper (limited generalizability).
• Domain specialization vs. generalizability:
  • System longevity is often short (2–3 years for general-purpose tools).
  • Ozlab’s longevity due to: general-purpose design, curricular integration, flexible wizard UI that adapts to experiments.
• Standardization and methodological approaches:
  • Interaction Specification Language (ISL) and ADEs (Porfirio et al.): hierarchical modularity, formal representations, platform independence for reproducibility.
  • Riek: methodological transparency deficiencies in WoZ literature (insufficient reporting of protocols/training/constraints).
  • Steinfeld et al.: “Oz of Wizard” complements WoZ; structured permutations of real vs. simulated components; both approaches serve valid objectives.
  • Belhassein et al.: recurring HRI study challenges (limited participants, inadequate protocol reporting, weak replication); need for validated measures and comprehensive documentation.
  • Fraune et al.: practical guidance (pilot testing, ensuring intended perception of robot behaviors, managing novelty effects, cross-field collaboration).
• Remaining challenges:
  • Accessibility for interdisciplinary teams.
  • Methodological standardization and comprehensive data capture/sharing.
  • Balance of structure (for reproducibility) and flexibility (for diverse research questions).

3) Reproducibility Challenges in WoZ Studies

• Inconsistent wizard behavior across trials undermines reproducibility.
• Publications often omit critical procedural details, making replication difficult.
• Custom, ad-hoc setups are hard to recreate; unrecorded changes hinder transparency.
• HRIStudio’s reproducibility requirements (five areas):
  • Standardized terminology and structure.
  • Wizard behavior formalization (clear, consistent execution with controlled flexibility).
  • Comprehensive, time-synchronized data capture.
  • Experiment specification sharing (package and distribute complete designs).
  • Procedural documentation (automatic logging of parameters and methodological details).

4) The Design and Architecture of HRIStudio

• Guiding design principles:
  • Accessibility for researchers without deep robot programming expertise.
  • Abstraction to focus on experimental design over platform details.
  • Comprehensive data management (logs, audio, video, study materials).
  • Collaboration through multi-user accounts, role-based access control, and data sharing.
  • Embedded methodological guidance to encourage scientifically sound practices.
• Conceptual separation aligned to research needs:
  • User-facing tools for design, execution, and analysis; stewarded data and access control; and standardized interfaces to connect experiments with robots and sensors.
• Three-layer architecture [Screenshot Placeholder: Architecture Overview]:
  • User Interface Layer:
    • Experiment Designer (visual programming for specifying experiments).
    • Wizard Interface (real-time control for trials).
    • Playback & Analysis (data exploration and visualization).
  • Data Management Layer:
    • Structured storage of experiment definitions, metadata, and media.
    • Role-based access aligned with study responsibilities.
    • Collaboration with secure, compartmentalized access for teams.
  • Robot Integration Layer:
    • Translates standardized abstractions to robot behaviors through plugins.
    • Standardized plugin interfaces support diverse platforms without changing study designs.
    • Integrates with external systems (robot hardware, sensors, tools).
• Sustained reproducibility and sharing:
  • Study definitions and execution environments can be packaged and shared to support faithful reproduction by independent teams.

5) Experimental Workflow Support

• Directly addresses reproducibility requirements with standardized structures, wizard guidance, and comprehensive capture.

5.1 Hierarchical Structure for WoZ Studies

• Standard terminology and elements:
  • Study: top-level container with one or more experiments.
  • Experiment: parameterized protocol template composed of steps.
  • Trial: concrete, executable instance of an experiment for a specific participant; all trial data recorded.
  • Step: type-bound container (wizard or robot) comprising a sequence of actions.
  • Action: atomic task for wizard or robot (e.g., input gathering, speech, movement), parameterized per trial.
• [Screenshot Placeholder: Experiment Hierarchy Diagram].
• [Screenshot Placeholder: Study Details View]:
  • Overview of execution summaries, trials, participant info and documents (e.g., consent), members, metadata, and audit activity.

5.2 Collaboration and Knowledge Sharing

• Dashboard for project overview, collaborators, trial schedules, pending tasks.
• Role-based access control (pre-defined roles; flexible extensions):
  • Administrator: system configuration/management.
  • Researcher: create/configure studies and experiments.
  • Observer: read-only access and real-time monitoring.
  • Wizard: execute experiments.
• Packaging and dissemination of complete materials for replication and meta-analyses.

5.3 Visual Experiment Design (EDE)

• Visual programming canvas for sequencing steps and actions (drag-and-drop).
• Abstract robot actions translated by plugins into platform-specific commands.
• Contextual help and documentation in the interface.
• [Screenshot Placeholder: Experiment Designer].
• Inspiration: Choregraphe’s flow-based, no-code composition for steps/actions.

5.4 Wizard Interface and Experiment Execution

• Adaptable, experiment-specific wizard UI (avoids one-size-fits-all trap).
• Incremental instructions, “View More” for full script, video feed, timestamped event log, and “quick actions.”
• Observer view mirrors wizard interface without execution controls.
• Action execution process:
  1. Translate abstract action into robot-specific calls via plugin.
  2. Route calls through appropriate communication channels.
  3. Process robot feedback, log details, update experiment state.
• [Screenshot Placeholder: Wizard Interface].

5.5 Robot Platform Integration (Plugin Store)

• Two-tier abstraction/translation of actions:
  • High-level action components (movement, speech, sensors) with parameter schemas and validation rules.
  • Robot plugins implement concrete mappings appropriate to each platform.
• [Screenshot Placeholder: Plugin Store]:
  • Trust levels: Official, Verified, Community.
  • Source repositories for precise version tracking and reproducibility.

5.6 Comprehensive Data Capture and Analysis

• Timestamped logs of all executed actions and events.
• Robot sensor data (position, orientation, sensor readings).
• Audio/video recordings of interactions.
• Wizard decisions/interventions (including unplanned deviations).
• Observer notes and annotations.
• Structured storage for long-term preservation and analysis integration.
• Sensitive participant data encrypted at the database level.
• Playback for step-by-step trial review and annotation.

6) Conclusion and Future Directions

• HRIStudio supports rigorous, reproducible WoZ experimentation via:
  • Standardized hierarchy and terminology.
  • Visual designer for protocol specification.
  • Configurable wizard interface for consistent execution.
  • Plugin-based, robot-agnostic integration.
  • Comprehensive capture and structured storage of multimodal data.
• Future directions:
  • Interface-integrated documentation for installation and operation.
  • Enhanced execution and analysis (advanced guidance, dynamic adaptation, real-time feedback).
  • Playback for synchronized streams and expanded hardware integration.
  • Continued community engagement to refine integration with existing research infrastructures and workflows.
  • Preparation for an open beta release.