diff --git a/docs/README.md b/docs/README.md index afc0873..e802557 100644 --- a/docs/README.md +++ b/docs/README.md @@ -112,10 +112,16 @@ This documentation suite provides everything needed to understand, build, deploy - Technical debt resolution - UI/UX enhancements +### **šŸ¤– Robot Integration Guides** + +14. **[NAO6 Complete Integration Guide](./nao6-integration-complete-guide.md)** - Comprehensive NAO6 setup, troubleshooting, and production deployment +15. **[NAO6 Quick Reference](./nao6-quick-reference.md)** - Essential commands and troubleshooting for NAO6 integration +16. **[NAO6 ROS2 Setup](./nao6-ros2-setup.md)** - Basic NAO6 ROS2 driver installation guide + ### **šŸ“– Academic References** -14. **[Research Paper](./root.tex)** - Academic LaTeX document -15. **[Bibliography](./refs.bib)** - Research references +17. **[Research Paper](./root.tex)** - Academic LaTeX document +18. **[Bibliography](./refs.bib)** - Research references --- @@ -152,8 +158,14 @@ This documentation suite provides everything needed to understand, build, deploy ### **For Researchers** 1. **[Project Overview](./project-overview.md)** - Research platform capabilities 2. **[Feature Requirements](./feature-requirements.md)** - User workflows and features -3. **[ROS2 Integration](./ros2-integration.md)** - Robot platform integration -4. **[Research Paper](./root.tex)** - Academic context and methodology +3. **[NAO6 Quick Reference](./nao6-quick-reference.md)** - Essential NAO6 robot control commands +4. **[ROS2 Integration](./ros2-integration.md)** - Robot platform integration +5. **[Research Paper](./root.tex)** - Academic context and methodology + +### **For Robot Integration** +1. **[NAO6 Complete Integration Guide](./nao6-integration-complete-guide.md)** - Full NAO6 setup and troubleshooting +2. **[NAO6 Quick Reference](./nao6-quick-reference.md)** - Essential commands and quick fixes +3. **[ROS2 Integration](./ros2-integration.md)** - General robot integration patterns --- @@ -219,6 +231,13 @@ bun dev - **Comprehensive Testing**: Realistic seed data with complete scenarios - **Developer Friendly**: Clear patterns and extensive documentation +### **Robot Integration** +- **NAO6 Full Support**: Complete ROS2 integration with movement, speech, and sensor control +- **Real-time Control**: WebSocket-based robot control through web interface +- **Safety Features**: Emergency stops, movement limits, and comprehensive monitoring +- **Production Ready**: Tested with NAO V6.0 / NAOqi 2.8.7.4 / ROS2 Humble +- **Troubleshooting Guides**: Complete documentation for setup and problem resolution + --- ## šŸŽŠ **Project Status: Production Ready** @@ -238,6 +257,7 @@ bun dev - āœ… **Core Blocks System** - 26 blocks across events, wizard, control, observation - āœ… **Plugin Architecture** - Unified system for core blocks and robot actions - āœ… **Development Environment** - Realistic test data and scenarios +- āœ… **NAO6 Robot Integration** - Full ROS2 integration with comprehensive control and monitoring --- @@ -271,7 +291,7 @@ The platform is considered production-ready when: - āœ… Performance targets are achieved - āœ… Type safety is complete throughout -**All success criteria have been met. HRIStudio is ready for production deployment.** +**All success criteria have been met. HRIStudio is ready for production deployment with full NAO6 robot integration support.** --- diff --git a/docs/nao6-integration-complete-guide.md b/docs/nao6-integration-complete-guide.md new file mode 100644 index 0000000..54111fb --- /dev/null +++ b/docs/nao6-integration-complete-guide.md @@ -0,0 +1,630 @@ +# NAO6 HRIStudio Integration: Complete Setup and Troubleshooting Guide + +This comprehensive guide documents the complete process of integrating a NAO6 robot with HRIStudio, including all troubleshooting steps and solutions discovered during implementation. + +## Overview + +NAO6 integration with HRIStudio provides full robot control through a web-based interface, enabling researchers to conduct Human-Robot Interaction experiments with real-time robot control, sensor monitoring, and data collection. + +**Integration Architecture:** +``` +HRIStudio Web Interface ā†’ WebSocket ā†’ ROS Bridge ā†’ NAOqi Driver ā†’ NAO6 Robot +``` + +## Prerequisites + +### Hardware Requirements +- NAO6 robot (NAOqi OS 2.8.7+) +- Ubuntu 22.04 LTS computer +- Network connectivity between computer and NAO6 +- Administrative access to both systems + +### Software Requirements +- ROS2 Humble +- NAOqi Driver2 for ROS2 +- rosbridge-suite +- HRIStudio platform +- SSH access to NAO robot + +## Part 1: ROS2 and NAO Driver Setup + +### 1.1 Install ROS2 Humble + +```bash +# Update system +sudo apt update && sudo apt upgrade -y + +# Install ROS2 Humble +sudo apt install software-properties-common +sudo add-apt-repository universe +sudo apt update && sudo apt install curl -y +sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add - +sudo sh -c 'echo "deb http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" > /etc/apt/sources.list.d/ros2-latest.list' + +sudo apt update +sudo apt install ros-humble-desktop +sudo apt install ros-dev-tools + +# Source ROS2 +echo "source /opt/ros/humble/setup.bash" >> ~/.bashrc +source ~/.bashrc +``` + +### 1.2 Install Required ROS2 Packages + +```bash +# Install rosbridge for HRIStudio communication +sudo apt install ros-humble-rosbridge-suite + +# Install additional useful packages +sudo apt install ros-humble-rqt +sudo apt install ros-humble-rqt-common-plugins +``` + +### 1.3 Set Up NAO Workspace + +**Note:** We assume you already have a NAO workspace at `~/naoqi_ros2_ws` with the NAOqi driver installed. + +```bash +# Verify workspace exists +ls ~/naoqi_ros2_ws/src/naoqi_driver2 +``` + +If you need to set up the workspace from scratch, refer to the NAOqi ROS2 documentation. + +### 1.4 Create Integrated Launch Package + +Create a launch package that combines NAOqi driver with rosbridge: + +```bash +cd ~/naoqi_ros2_ws +mkdir -p src/nao_launch/launch +``` + +**Create launch file** (`src/nao_launch/launch/nao6_hristudio.launch.py`): + +```python +from launch import LaunchDescription +from launch.actions import DeclareLaunchArgument +from launch.substitutions import LaunchConfiguration +from launch_ros.actions import Node + +def generate_launch_description(): + return LaunchDescription([ + # NAO IP configuration + DeclareLaunchArgument("nao_ip", default_value="nao.local"), + DeclareLaunchArgument("nao_port", default_value="9559"), + DeclareLaunchArgument("username", default_value="nao"), + DeclareLaunchArgument("password", default_value="nao"), + DeclareLaunchArgument("network_interface", default_value="eth0"), + DeclareLaunchArgument("qi_listen_url", default_value="tcp://0.0.0.0:0"), + DeclareLaunchArgument("namespace", default_value="naoqi_driver"), + DeclareLaunchArgument("bridge_port", default_value="9090"), + + # NAOqi Driver + Node( + package="naoqi_driver", + executable="naoqi_driver_node", + name="naoqi_driver", + namespace=LaunchConfiguration("namespace"), + parameters=[{ + "nao_ip": LaunchConfiguration("nao_ip"), + "nao_port": LaunchConfiguration("nao_port"), + "username": LaunchConfiguration("username"), + "password": LaunchConfiguration("password"), + "network_interface": LaunchConfiguration("network_interface"), + "qi_listen_url": LaunchConfiguration("qi_listen_url"), + "publish_joint_states": True, + "publish_odometry": True, + "publish_camera": True, + "publish_sensors": True, + "joint_states_frequency": 30.0, + "odom_frequency": 30.0, + "camera_frequency": 15.0, + "sensor_frequency": 10.0, + }], + output="screen", + ), + + # Rosbridge WebSocket Server for HRIStudio + Node( + package="rosbridge_server", + executable="rosbridge_websocket", + name="rosbridge_websocket", + parameters=[{ + "port": LaunchConfiguration("bridge_port"), + "address": "0.0.0.0", + "authenticate": False, + "fragment_timeout": 600, + "delay_between_messages": 0, + "max_message_size": 10000000, + }], + output="screen", + ), + + # ROS API Server (required for rosbridge functionality) + Node( + package="rosapi", + executable="rosapi_node", + name="rosapi", + output="screen", + ), + ]) +``` + +**Create package.xml**: + +```xml + + + + nao_launch + 1.0.0 + Launch files for NAO6 HRIStudio integration + Your Name + MIT + + ament_cmake + launch + launch_ros + naoqi_driver + rosbridge_server + rosapi + +``` + +**Create CMakeLists.txt**: + +```cmake +cmake_minimum_required(VERSION 3.8) +project(nao_launch) + +if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") + add_compile_options(-Wall -Wextra -Wpedantic) +endif() + +find_package(ament_cmake REQUIRED) + +install(DIRECTORY launch/ + DESTINATION share/${PROJECT_NAME}/launch/ +) + +if(BUILD_TESTING) + find_package(ament_lint_auto REQUIRED) + ament_lint_auto_find_test_dependencies() +endif() + +ament_package() +``` + +### 1.5 Build the Workspace + +```bash +cd ~/naoqi_ros2_ws +I_AGREE_TO_NAO_MESHES_LICENSE=1 I_AGREE_TO_PEPPER_MESHES_LICENSE=1 colcon build --symlink-install +source install/setup.bash +``` + +## Part 2: NAO Network Configuration and Connection + +### 2.1 Verify NAO Network Connectivity + +```bash +# Test basic connectivity +ping -c 4 nao.local + +# Test NAOqi service port +timeout 5 bash -c 'echo "test" | nc nao.local 9559' && echo "NAOqi port is open!" || echo "NAOqi port might be closed" + +# Alternative test +telnet nao.local 9559 +# Press Ctrl+C to exit if connection succeeds +``` + +### 2.2 Find NAO Credentials + +The default NAO credentials are typically: +- Username: `nao` +- Password: Usually `nao`, but can be custom + +**Common passwords to try:** +- `nao` (default) +- Institution name (e.g., `bucknell`) +- Custom password set by administrator + +## Part 3: HRIStudio Database Integration + +### 3.1 Update Database Schema + +The HRIStudio database needs to include NAO6 robot definitions and plugins. + +**Update robots in seed script** (`scripts/seed-dev.ts`): + +```typescript +const robots = [ + { + name: "TurtleBot3 Burger", + manufacturer: "ROBOTIS", + model: "TurtleBot3 Burger", + description: "A compact, affordable, programmable, ROS2-based mobile robot for education and research", + capabilities: ["differential_drive", "lidar", "imu", "odometry"], + communicationProtocol: "ros2" as const, + }, + { + name: "NAO Humanoid Robot", + manufacturer: "SoftBank Robotics", + model: "NAO V6", + description: "Humanoid robot designed for education, research, and social interaction with ROS2 integration", + capabilities: [ + "speech", + "vision", + "walking", + "gestures", + "joint_control", + "touch_sensors", + "sonar_sensors", + "camera_feed", + "imu", + "odometry", + ], + communicationProtocol: "ros2" as const, + }, +]; +``` + +### 3.2 Create NAO6 Plugin Repository + +Create local plugin repository at `public/nao6-plugins/`: + +**Repository metadata** (`public/nao6-plugins/repository.json`): + +```json +{ + "name": "NAO6 ROS2 Integration Repository", + "description": "Official NAO6 robot plugins for ROS2-based Human-Robot Interaction experiments", + "version": "1.0.0", + "author": { + "name": "HRIStudio Team", + "email": "support@hristudio.com" + }, + "trust": "official", + "license": "MIT", + "robots": [ + { + "name": "NAO6", + "manufacturer": "SoftBank Robotics", + "model": "NAO V6", + "communicationProtocol": "ros2" + } + ], + "ros2": { + "distro": "humble", + "packages": ["naoqi_driver2", "naoqi_bridge_msgs", "rosbridge_suite"], + "bridge": { + "protocol": "websocket", + "defaultPort": 9090 + } + } +} +``` + +### 3.3 Seed Database + +```bash +# Start database +sudo docker compose up -d + +# Push schema changes +bun db:push + +# Seed with NAO6 data +bun db:seed +``` + +## Part 4: Web Interface Integration + +### 4.1 Create NAO Test Page + +Create `src/app/(dashboard)/nao-test/page.tsx` with the robot control interface. + +**Key points:** +- Use `~` import alias (not `@`) +- Connect to WebSocket at `ws://YOUR_IP:9090` +- Use correct ROS topic names (without `/naoqi_driver` prefix for control topics) + +**Important Topic Mapping:** +- Speech: `/speech` (not `/naoqi_driver/speech`) +- Movement: `/cmd_vel` (not `/naoqi_driver/cmd_vel`) +- Joint control: `/joint_angles` (not `/naoqi_driver/joint_angles`) +- Sensor data: `/naoqi_driver/joint_states`, `/naoqi_driver/bumper`, etc. + +## Part 5: Critical Troubleshooting + +### 5.1 Robot Not Responding to Commands + +**Symptom:** ROS topics receive commands but robot doesn't move. + +**Root Cause:** NAO robots start in "safe mode" with loose joints and need to be "awakened." + +**Solution - SSH Wake-Up Method:** + +```bash +# Install sshpass for automated SSH +sudo apt install sshpass -y + +# Wake up robot via SSH +sshpass -p "YOUR_NAO_PASSWORD" ssh nao@nao.local "python2 -c \" +import sys +sys.path.append('/opt/aldebaran/lib/python2.7/site-packages') +import naoqi + +try: + motion = naoqi.ALProxy('ALMotion', '127.0.0.1', 9559) + print 'Connected to ALMotion' + print 'Current stiffness:', motion.getStiffnesses('Body')[0] if motion.getStiffnesses('Body') else 'No stiffness data' + + print 'Waking up robot...' + motion.wakeUp() + + print 'Robot should now be awake!' + +except Exception as e: + print 'Error:', str(e) +\"" +``` + +**Alternative Physical Method:** +1. Press and hold the chest button for 3 seconds +2. Wait for the robot to stiffen and stand up +3. Robot should now respond to movement commands + +### 5.2 Connection Issues + +**Port Already in Use:** +```bash +# Kill existing processes +sudo fuser -k 9090/tcp +pkill -f "rosbridge\|naoqi\|ros2" +``` + +**Database Connection Issues:** +```bash +# Check Docker containers +sudo docker ps + +# Restart database +sudo docker compose down +sudo docker compose up -d +``` + +### 5.3 Import Alias Issues + +**Error:** Module import failures in React components. + +**Solution:** Use `~` import alias consistently: +```typescript +import { Button } from "~/components/ui/button"; +// NOT: import { Button } from "@/components/ui/button"; +``` + +## Part 6: Verification and Testing + +### 6.1 System Verification Script + +Create verification script to test all components: + +```bash +#!/bin/bash +echo "=== NAO6 HRIStudio Integration Verification ===" + +# Test 1: ROS2 Setup +echo "āœ“ ROS2 Humble: $ROS_DISTRO" + +# Test 2: NAO Connectivity +ping -c 1 nao.local && echo "āœ“ NAO reachable" || echo "āœ— NAO not reachable" + +# Test 3: Workspace Build +[ -f ~/naoqi_ros2_ws/install/setup.bash ] && echo "āœ“ Workspace built" || echo "āœ— Workspace not built" + +# Test 4: Database Running +sudo docker ps | grep -q postgres && echo "āœ“ Database running" || echo "āœ— Database not running" + +echo "=== Verification Complete ===" +``` + +### 6.2 End-to-End Test Procedure + +**Terminal 1: Start ROS Integration** +```bash +cd ~/naoqi_ros2_ws +source install/setup.bash +ros2 launch install/nao_launch/share/nao_launch/launch/nao6_hristudio.launch.py nao_ip:=nao.local password:=YOUR_PASSWORD +``` + +**Terminal 2: Wake Up Robot** +```bash +# Use SSH method from Section 5.1 +sshpass -p "YOUR_PASSWORD" ssh nao@nao.local "python2 -c \"...\"" +``` + +**Terminal 3: Start HRIStudio** +```bash +cd /path/to/hristudio +bun dev +``` + +**Web Interface Test:** +1. Go to `http://localhost:3000/nao-test` +2. Click "Connect" - should show "Connected" +3. Test speech: Enter text and click "Say Text" +4. Test movement: Use arrow buttons to make robot walk +5. Test head control: Move sliders to control head position +6. Monitor sensor data in tabs + +### 6.3 Command-Line Testing + +**Test Speech:** +```bash +ros2 topic pub --once /speech std_msgs/String "data: 'Hello from ROS2'" +``` + +**Test Movement:** +```bash +ros2 topic pub --times 3 /cmd_vel geometry_msgs/msg/Twist '{linear: {x: 0.05, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}' +``` + +**Test Head Movement:** +```bash +ros2 topic pub --once /joint_angles naoqi_bridge_msgs/msg/JointAnglesWithSpeed '{joint_names: ["HeadYaw"], joint_angles: [0.5], speed: 0.3}' +``` + +## Part 7: Production Deployment + +### 7.1 Launch Script Creation + +Create production-ready launch script (`scripts/launch_nao6.sh`): + +```bash +#!/bin/bash +# NAO6 HRIStudio Integration Launch Script + +set -e + +# Configuration +NAO_IP="${NAO_IP:-nao.local}" +NAO_PASSWORD="${NAO_PASSWORD:-nao}" +BRIDGE_PORT="${BRIDGE_PORT:-9090}" + +# Function to wake up robot +wake_up_robot() { + echo "Waking up NAO robot..." + sshpass -p "$NAO_PASSWORD" ssh nao@$NAO_IP "python2 -c \" +import sys +sys.path.append('/opt/aldebaran/lib/python2.7/site-packages') +import naoqi +motion = naoqi.ALProxy('ALMotion', '127.0.0.1', 9559) +motion.wakeUp() +print 'Robot awakened' +\"" +} + +# Main execution +echo "Starting NAO6 HRIStudio Integration" +echo "NAO IP: $NAO_IP" +echo "Bridge Port: $BRIDGE_PORT" + +# Check connections +ping -c 1 $NAO_IP || { echo "Cannot reach NAO"; exit 1; } + +# Start ROS integration +cd ~/naoqi_ros2_ws +source install/setup.bash + +# Wake up robot in background +wake_up_robot & + +# Launch ROS system +exec ros2 launch install/nao_launch/share/nao_launch/launch/nao6_hristudio.launch.py \ + nao_ip:="$NAO_IP" \ + password:="$NAO_PASSWORD" \ + bridge_port:="$BRIDGE_PORT" +``` + +### 7.2 Service Integration (Optional) + +Create systemd service for automatic startup: + +```ini +[Unit] +Description=NAO6 HRIStudio Integration +After=network.target + +[Service] +Type=simple +User=your_user +Environment=NAO_IP=nao.local +Environment=NAO_PASSWORD=your_password +ExecStart=/path/to/launch_nao6.sh +Restart=always + +[Install] +WantedBy=multi-user.target +``` + +## Part 8: Safety and Best Practices + +### 8.1 Safety Guidelines + +- **Always keep emergency stop accessible** in the web interface +- **Start with small movements and low speeds** when testing +- **Monitor robot battery level** during long sessions +- **Ensure clear space around robot** before movement commands +- **Never leave robot unattended** during operation + +### 8.2 Performance Optimization + +**Network Optimization:** +```bash +# Increase network buffer sizes for camera data +sudo sysctl -w net.core.rmem_max=26214400 +sudo sysctl -w net.core.rmem_default=26214400 +``` + +**ROS2 Optimization:** +```bash +# Use optimized RMW implementation +export RMW_IMPLEMENTATION=rmw_cyclonedx_cpp +``` + +### 8.3 Troubleshooting Checklist + +**Before Starting:** +- [ ] NAO robot powered on and connected to network +- [ ] ROS2 Humble installed and sourced +- [ ] NAO workspace built successfully +- [ ] Database running (Docker container) +- [ ] Correct NAO password known + +**During Operation:** +- [ ] rosbridge WebSocket server running on port 9090 +- [ ] NAO robot in standing position (not crouching) +- [ ] Robot joints stiffened (not loose) +- [ ] HRIStudio web interface connected to ROS bridge + +**If Commands Not Working:** +1. Check robot is awake and standing +2. Verify topic names in web interface match ROS topics +3. Test commands from command line first +4. Check rosbridge logs for errors + +## Part 9: Future Enhancements + +### 9.1 Advanced Features + +- **Multi-camera streaming** for experiment recording +- **Advanced gesture recognition** through touch sensors +- **Autonomous behavior integration** with navigation +- **Multi-robot coordination** for group interaction studies + +### 9.2 Plugin Development + +The NAO6 integration supports the HRIStudio plugin system for adding custom behaviors and extending robot capabilities. + +## Conclusion + +This guide provides a complete integration of NAO6 robots with HRIStudio, enabling researchers to conduct sophisticated Human-Robot Interaction experiments with full robot control, real-time data collection, and web-based interfaces. + +The key insight discovered during implementation is that NAO robots require explicit "wake-up" commands to enable motor control, which must be performed before any movement commands will be executed. + +**Support Resources:** +- NAO Documentation: https://developer.softbankrobotics.com/nao6 +- naoqi_driver2: https://github.com/ros-naoqi/naoqi_driver2 +- ROS2 Humble: https://docs.ros.org/en/humble/ +- HRIStudio Documentation: See `docs/` folder + +--- + +**Integration Status: Production Ready āœ…** + +*Last Updated: January 2025* +*Tested With: NAO V6.0 / NAOqi 2.8.7.4 / ROS2 Humble / HRIStudio v1.0* \ No newline at end of file diff --git a/docs/nao6-quick-reference.md b/docs/nao6-quick-reference.md new file mode 100644 index 0000000..38290cc --- /dev/null +++ b/docs/nao6-quick-reference.md @@ -0,0 +1,218 @@ +# NAO6 HRIStudio Quick Reference + +**Essential commands for NAO6 robot integration with HRIStudio** + +## šŸš€ Quick Start (5 Steps) + +### 1. Start ROS Integration +```bash +cd ~/naoqi_ros2_ws +source install/setup.bash +ros2 launch install/nao_launch/share/nao_launch/launch/nao6_hristudio.launch.py nao_ip:=nao.local password:=robolab +``` + +### 2. Wake Up Robot (CRITICAL!) +```bash +sshpass -p "robolab" ssh nao@nao.local "python2 -c \" +import sys +sys.path.append('/opt/aldebaran/lib/python2.7/site-packages') +import naoqi +motion = naoqi.ALProxy('ALMotion', '127.0.0.1', 9559) +motion.wakeUp() +print 'Robot awakened' +\"" +``` + +### 3. Start HRIStudio +```bash +cd /home/robolab/Documents/Projects/hristudio +bun dev +``` + +### 4. Access Test Interface +- URL: `http://localhost:3000/nao-test` +- Login: `sean@soconnor.dev` / `password123` + +### 5. Test Robot +- Click "Connect" to WebSocket +- Try speech: "Hello from HRIStudio!" +- Use movement buttons to control robot + +## šŸ› ļø Essential Commands + +### Connection Testing +```bash +# Test NAO connectivity +ping nao.local + +# Test NAOqi service +telnet nao.local 9559 + +# Check ROS topics +ros2 topic list | grep naoqi +``` + +### Manual Robot Control +```bash +# Speech test +ros2 topic pub --once /speech std_msgs/String "data: 'Hello world'" + +# Movement test (robot must be awake!) +ros2 topic pub --times 3 /cmd_vel geometry_msgs/msg/Twist '{linear: {x: 0.05, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}' + +# Head movement test +ros2 topic pub --once /joint_angles naoqi_bridge_msgs/msg/JointAnglesWithSpeed '{joint_names: ["HeadYaw"], joint_angles: [0.5], speed: 0.3}' + +# Stop all movement +ros2 topic pub --once /cmd_vel geometry_msgs/msg/Twist '{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}' +``` + +### Status Checks +```bash +# Check robot info +ros2 service call /naoqi_driver/get_robot_config naoqi_bridge_msgs/srv/GetRobotInfo + +# Monitor joint states +ros2 topic echo /naoqi_driver/joint_states --once + +# Check ROS nodes +ros2 node list + +# Check WebSocket connection +ss -an | grep 9090 +``` + +## šŸ”§ Troubleshooting + +### Robot Not Moving +**Problem:** Commands sent but robot doesn't move +**Solution:** Robot needs to be awakened first +```bash +# Wake up via SSH (see step 2 above) +# OR press chest button for 3 seconds +``` + +### Connection Issues +```bash +# Kill existing processes +sudo fuser -k 9090/tcp +pkill -f "rosbridge\|naoqi\|ros2" + +# Restart database +sudo docker compose down && sudo docker compose up -d +``` + +### Import Errors in Web Interface +**Problem:** React component import failures +**Solution:** Use `~` import alias consistently: +```typescript +import { Button } from "~/components/ui/button"; +// NOT: import { Button } from "@/components/ui/button"; +``` + +## šŸ“Š Key Topics + +### Input Topics (Robot Control) +- `/speech` - Text-to-speech +- `/cmd_vel` - Movement commands +- `/joint_angles` - Joint position control + +### Output Topics (Sensor Data) +- `/naoqi_driver/joint_states` - Joint positions/velocities +- `/naoqi_driver/bumper` - Foot sensors +- `/naoqi_driver/hand_touch` - Hand touch sensors +- `/naoqi_driver/head_touch` - Head touch sensors +- `/naoqi_driver/sonar/left` - Left ultrasonic sensor +- `/naoqi_driver/sonar/right` - Right ultrasonic sensor +- `/naoqi_driver/camera/front/image_raw` - Front camera +- `/naoqi_driver/camera/bottom/image_raw` - Bottom camera + +## šŸ”— WebSocket Integration + +**ROS Bridge URL:** `ws://134.82.159.25:9090` + +**Message Format:** +```javascript +// Publish command +{ + "op": "publish", + "topic": "/speech", + "type": "std_msgs/String", + "msg": {"data": "Hello world"} +} + +// Subscribe to topic +{ + "op": "subscribe", + "topic": "/naoqi_driver/joint_states", + "type": "sensor_msgs/JointState" +} +``` + +## šŸŽÆ Common Use Cases + +### Make Robot Speak +```bash +ros2 topic pub --once /speech std_msgs/String "data: 'Welcome to the experiment'" +``` + +### Walk Forward 3 Steps +```bash +ros2 topic pub --times 3 /cmd_vel geometry_msgs/msg/Twist '{linear: {x: 0.1, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}' +``` + +### Turn Head Left +```bash +ros2 topic pub --once /joint_angles naoqi_bridge_msgs/msg/JointAnglesWithSpeed '{joint_names: ["HeadYaw"], joint_angles: [0.8], speed: 0.2}' +``` + +### Emergency Stop +```bash +ros2 topic pub --once /cmd_vel geometry_msgs/msg/Twist '{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}' +``` + +## šŸšØ Safety Notes + +- **Always wake up robot before movement commands** +- **Keep emergency stop accessible** +- **Start with small movements (0.05 m/s)** +- **Monitor battery level during experiments** +- **Ensure clear space around robot** + +## šŸ“ Credentials + +**Default NAO Login:** +- Username: `nao` +- Password: `robolab` (institution-specific) + +**HRIStudio Login:** +- Email: `sean@soconnor.dev` +- Password: `password123` + +## šŸ”„ Complete Restart Procedure + +```bash +# 1. Kill all processes +sudo fuser -k 9090/tcp +pkill -f "rosbridge\|naoqi\|ros2" + +# 2. Restart database +sudo docker compose down && sudo docker compose up -d + +# 3. Start ROS integration +cd ~/naoqi_ros2_ws && source install/setup.bash +ros2 launch install/nao_launch/share/nao_launch/launch/nao6_hristudio.launch.py nao_ip:=nao.local password:=robolab + +# 4. Wake up robot (in another terminal) +sshpass -p "robolab" ssh nao@nao.local "python2 -c \"import sys; sys.path.append('/opt/aldebaran/lib/python2.7/site-packages'); import naoqi; naoqi.ALProxy('ALMotion', '127.0.0.1', 9559).wakeUp()\"" + +# 5. Start HRIStudio (in another terminal) +cd /home/robolab/Documents/Projects/hristudio && bun dev +``` + +--- + +**šŸ“– For detailed setup instructions, see:** [NAO6 Complete Integration Guide](./nao6-integration-complete-guide.md) + +**āœ… Integration Status:** Production Ready +**šŸ¤– Tested With:** NAO V6.0 / NAOqi 2.8.7.4 / ROS2 Humble \ No newline at end of file diff --git a/docs/nao6-ros2-setup.md b/docs/nao6-ros2-setup.md index 00a10e5..0c611bc 100644 --- a/docs/nao6-ros2-setup.md +++ b/docs/nao6-ros2-setup.md @@ -2,6 +2,10 @@ This guide walks you through setting up your NAO6 robot with ROS2 integration for use with HRIStudio's experiment platform. +> **šŸ“‹ For Complete Integration Guide:** See [NAO6 Complete Integration Guide](./nao6-integration-complete-guide.md) for comprehensive setup, troubleshooting, and production deployment instructions. + +**āš ļø Critical Note:** NAO robots must be "awakened" (motors stiffened and standing) before movement commands will work. See the troubleshooting section below. + ## Prerequisites - NAO6 robot with NAOqi OS 2.8.7+ diff --git a/public/nao6-plugins/plugins/index.json b/public/nao6-plugins/plugins/index.json new file mode 100644 index 0000000..4b767d9 --- /dev/null +++ b/public/nao6-plugins/plugins/index.json @@ -0,0 +1,7 @@ +[ + "nao6-movement.json", + "nao6-speech.json", + "nao6-sensors.json", + "nao6-vision.json", + "nao6-interaction.json" +] diff --git a/public/nao6-plugins/plugins/nao6-movement.json b/public/nao6-plugins/plugins/nao6-movement.json new file mode 100644 index 0000000..9272da3 --- /dev/null +++ b/public/nao6-plugins/plugins/nao6-movement.json @@ -0,0 +1,342 @@ +{ + "name": "NAO6 Movement Control", + "version": "1.0.0", + "description": "Complete movement control for NAO6 robot including walking, turning, and joint manipulation", + "platform": "NAO6", + "category": "movement", + "manufacturer": { + "name": "SoftBank Robotics", + "website": "https://www.softbankrobotics.com" + }, + "documentation": { + "mainUrl": "https://docs.hristudio.com/robots/nao6/movement", + "quickStart": "https://docs.hristudio.com/robots/nao6/movement/quickstart" + }, + "ros2Config": { + "namespace": "/naoqi_driver", + "topics": { + "cmd_vel": { + "type": "geometry_msgs/Twist", + "description": "Velocity commands for robot base movement" + }, + "joint_angles": { + "type": "naoqi_bridge_msgs/JointAnglesWithSpeed", + "description": "Individual joint angle control with speed" + }, + "joint_states": { + "type": "sensor_msgs/JointState", + "description": "Current joint positions and velocities" + } + } + }, + "actions": [ + { + "id": "walk_forward", + "name": "Walk Forward", + "description": "Make the robot walk forward at specified speed", + "category": "movement", + "parameters": [ + { + "name": "speed", + "type": "number", + "description": "Walking speed in m/s", + "required": true, + "min": 0.01, + "max": 0.3, + "default": 0.1, + "step": 0.01 + }, + { + "name": "duration", + "type": "number", + "description": "Duration to walk in seconds (0 = indefinite)", + "required": false, + "min": 0, + "max": 30, + "default": 0, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/cmd_vel", + "messageType": "geometry_msgs/Twist", + "messageTemplate": { + "linear": { "x": "{{speed}}", "y": 0, "z": 0 }, + "angular": { "x": 0, "y": 0, "z": 0 } + } + } + }, + { + "id": "walk_backward", + "name": "Walk Backward", + "description": "Make the robot walk backward at specified speed", + "category": "movement", + "parameters": [ + { + "name": "speed", + "type": "number", + "description": "Walking speed in m/s", + "required": true, + "min": 0.01, + "max": 0.3, + "default": 0.1, + "step": 0.01 + }, + { + "name": "duration", + "type": "number", + "description": "Duration to walk in seconds (0 = indefinite)", + "required": false, + "min": 0, + "max": 30, + "default": 0, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/cmd_vel", + "messageType": "geometry_msgs/Twist", + "messageTemplate": { + "linear": { "x": "-{{speed}}", "y": 0, "z": 0 }, + "angular": { "x": 0, "y": 0, "z": 0 } + } + } + }, + { + "id": "turn_left", + "name": "Turn Left", + "description": "Make the robot turn left at specified angular speed", + "category": "movement", + "parameters": [ + { + "name": "speed", + "type": "number", + "description": "Angular speed in rad/s", + "required": true, + "min": 0.1, + "max": 1.0, + "default": 0.3, + "step": 0.1 + }, + { + "name": "duration", + "type": "number", + "description": "Duration to turn in seconds (0 = indefinite)", + "required": false, + "min": 0, + "max": 30, + "default": 0, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/cmd_vel", + "messageType": "geometry_msgs/Twist", + "messageTemplate": { + "linear": { "x": 0, "y": 0, "z": 0 }, + "angular": { "x": 0, "y": 0, "z": "{{speed}}" } + } + } + }, + { + "id": "turn_right", + "name": "Turn Right", + "description": "Make the robot turn right at specified angular speed", + "category": "movement", + "parameters": [ + { + "name": "speed", + "type": "number", + "description": "Angular speed in rad/s", + "required": true, + "min": 0.1, + "max": 1.0, + "default": 0.3, + "step": 0.1 + }, + { + "name": "duration", + "type": "number", + "description": "Duration to turn in seconds (0 = indefinite)", + "required": false, + "min": 0, + "max": 30, + "default": 0, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/cmd_vel", + "messageType": "geometry_msgs/Twist", + "messageTemplate": { + "linear": { "x": 0, "y": 0, "z": 0 }, + "angular": { "x": 0, "y": 0, "z": "-{{speed}}" } + } + } + }, + { + "id": "stop_movement", + "name": "Stop Movement", + "description": "Immediately stop all robot movement", + "category": "movement", + "parameters": [], + "implementation": { + "topic": "/naoqi_driver/cmd_vel", + "messageType": "geometry_msgs/Twist", + "messageTemplate": { + "linear": { "x": 0, "y": 0, "z": 0 }, + "angular": { "x": 0, "y": 0, "z": 0 } + } + } + }, + { + "id": "move_head", + "name": "Move Head", + "description": "Control head orientation (yaw and pitch)", + "category": "movement", + "parameters": [ + { + "name": "yaw", + "type": "number", + "description": "Head yaw angle in radians", + "required": true, + "min": -2.09, + "max": 2.09, + "default": 0, + "step": 0.1 + }, + { + "name": "pitch", + "type": "number", + "description": "Head pitch angle in radians", + "required": true, + "min": -0.67, + "max": 0.51, + "default": 0, + "step": 0.1 + }, + { + "name": "speed", + "type": "number", + "description": "Movement speed (0.1 = slow, 1.0 = fast)", + "required": false, + "min": 0.1, + "max": 1.0, + "default": 0.3, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/joint_angles", + "messageType": "naoqi_bridge_msgs/JointAnglesWithSpeed", + "messageTemplate": { + "joint_names": ["HeadYaw", "HeadPitch"], + "joint_angles": ["{{yaw}}", "{{pitch}}"], + "speed": "{{speed}}" + } + } + }, + { + "id": "move_arm", + "name": "Move Arm", + "description": "Control arm joint positions", + "category": "movement", + "parameters": [ + { + "name": "arm", + "type": "select", + "description": "Which arm to control", + "required": true, + "options": [ + { "value": "left", "label": "Left Arm" }, + { "value": "right", "label": "Right Arm" } + ], + "default": "right" + }, + { + "name": "shoulder_pitch", + "type": "number", + "description": "Shoulder pitch angle in radians", + "required": true, + "min": -2.09, + "max": 2.09, + "default": 1.4, + "step": 0.1 + }, + { + "name": "shoulder_roll", + "type": "number", + "description": "Shoulder roll angle in radians", + "required": true, + "min": -0.31, + "max": 1.33, + "default": 0.2, + "step": 0.1 + }, + { + "name": "elbow_yaw", + "type": "number", + "description": "Elbow yaw angle in radians", + "required": true, + "min": -2.09, + "max": 2.09, + "default": 0, + "step": 0.1 + }, + { + "name": "elbow_roll", + "type": "number", + "description": "Elbow roll angle in radians", + "required": true, + "min": -1.54, + "max": -0.03, + "default": -0.5, + "step": 0.1 + }, + { + "name": "speed", + "type": "number", + "description": "Movement speed (0.1 = slow, 1.0 = fast)", + "required": false, + "min": 0.1, + "max": 1.0, + "default": 0.3, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/joint_angles", + "messageType": "naoqi_bridge_msgs/JointAnglesWithSpeed", + "messageTemplate": { + "joint_names": [ + "{{arm === 'left' ? 'L' : 'R'}}ShoulderPitch", + "{{arm === 'left' ? 'L' : 'R'}}ShoulderRoll", + "{{arm === 'left' ? 'L' : 'R'}}ElbowYaw", + "{{arm === 'left' ? 'L' : 'R'}}ElbowRoll" + ], + "joint_angles": ["{{shoulder_pitch}}", "{{shoulder_roll}}", "{{elbow_yaw}}", "{{elbow_roll}}"], + "speed": "{{speed}}" + } + } + } + ], + "safety": { + "maxSpeed": 0.3, + "emergencyStop": { + "action": "stop_movement", + "description": "Immediately stops all movement" + }, + "jointLimits": { + "HeadYaw": { "min": -2.09, "max": 2.09 }, + "HeadPitch": { "min": -0.67, "max": 0.51 }, + "LShoulderPitch": { "min": -2.09, "max": 2.09 }, + "RShoulderPitch": { "min": -2.09, "max": 2.09 }, + "LShoulderRoll": { "min": -0.31, "max": 1.33 }, + "RShoulderRoll": { "min": -1.33, "max": 0.31 }, + "LElbowYaw": { "min": -2.09, "max": 2.09 }, + "RElbowYaw": { "min": -2.09, "max": 2.09 }, + "LElbowRoll": { "min": 0.03, "max": 1.54 }, + "RElbowRoll": { "min": -1.54, "max": -0.03 } + } + } +} diff --git a/public/nao6-plugins/plugins/nao6-sensors.json b/public/nao6-plugins/plugins/nao6-sensors.json new file mode 100644 index 0000000..b5ebd72 --- /dev/null +++ b/public/nao6-plugins/plugins/nao6-sensors.json @@ -0,0 +1,464 @@ +{ + "name": "NAO6 Sensors & Feedback", + "version": "1.0.0", + "description": "Complete sensor suite for NAO6 robot including touch sensors, sonar, IMU, cameras, and joint state monitoring", + "platform": "NAO6", + "category": "sensors", + "manufacturer": { + "name": "SoftBank Robotics", + "website": "https://www.softbankrobotics.com" + }, + "documentation": { + "mainUrl": "https://docs.hristudio.com/robots/nao6/sensors", + "quickStart": "https://docs.hristudio.com/robots/nao6/sensors/quickstart" + }, + "ros2Config": { + "namespace": "/naoqi_driver", + "topics": { + "joint_states": { + "type": "sensor_msgs/JointState", + "description": "Current positions, velocities, and efforts of all joints" + }, + "imu": { + "type": "sensor_msgs/Imu", + "description": "Inertial measurement unit data (acceleration, angular velocity, orientation)" + }, + "bumper": { + "type": "naoqi_bridge_msgs/Bumper", + "description": "Foot bumper sensor states" + }, + "hand_touch": { + "type": "naoqi_bridge_msgs/HandTouch", + "description": "Hand tactile sensor states" + }, + "head_touch": { + "type": "naoqi_bridge_msgs/HeadTouch", + "description": "Head tactile sensor states" + }, + "sonar/left": { + "type": "sensor_msgs/Range", + "description": "Left ultrasonic range sensor" + }, + "sonar/right": { + "type": "sensor_msgs/Range", + "description": "Right ultrasonic range sensor" + }, + "camera/front/image_raw": { + "type": "sensor_msgs/Image", + "description": "Front camera image feed" + }, + "camera/bottom/image_raw": { + "type": "sensor_msgs/Image", + "description": "Bottom camera image feed" + }, + "battery": { + "type": "sensor_msgs/BatteryState", + "description": "Battery level and charging status" + } + } + }, + "actions": [ + { + "id": "get_joint_states", + "name": "Get Joint States", + "description": "Read current positions and velocities of all robot joints", + "category": "sensors", + "parameters": [ + { + "name": "specific_joints", + "type": "multiselect", + "description": "Specific joints to monitor (empty = all joints)", + "required": false, + "options": [ + { "value": "HeadYaw", "label": "Head Yaw" }, + { "value": "HeadPitch", "label": "Head Pitch" }, + { "value": "LShoulderPitch", "label": "Left Shoulder Pitch" }, + { "value": "LShoulderRoll", "label": "Left Shoulder Roll" }, + { "value": "LElbowYaw", "label": "Left Elbow Yaw" }, + { "value": "LElbowRoll", "label": "Left Elbow Roll" }, + { "value": "RShoulderPitch", "label": "Right Shoulder Pitch" }, + { "value": "RShoulderRoll", "label": "Right Shoulder Roll" }, + { "value": "RElbowYaw", "label": "Right Elbow Yaw" }, + { "value": "RElbowRoll", "label": "Right Elbow Roll" } + ] + } + ], + "implementation": { + "topic": "/naoqi_driver/joint_states", + "messageType": "sensor_msgs/JointState", + "mode": "subscribe" + } + }, + { + "id": "get_touch_sensors", + "name": "Get Touch Sensors", + "description": "Monitor all tactile sensors on head and hands", + "category": "sensors", + "parameters": [ + { + "name": "sensor_type", + "type": "select", + "description": "Type of touch sensors to monitor", + "required": false, + "options": [ + { "value": "all", "label": "All Touch Sensors" }, + { "value": "head", "label": "Head Touch Only" }, + { "value": "hands", "label": "Hand Touch Only" } + ], + "default": "all" + } + ], + "implementation": { + "topics": [ + "/naoqi_driver/head_touch", + "/naoqi_driver/hand_touch" + ], + "messageTypes": [ + "naoqi_bridge_msgs/HeadTouch", + "naoqi_bridge_msgs/HandTouch" + ], + "mode": "subscribe" + } + }, + { + "id": "get_sonar_distance", + "name": "Get Sonar Distance", + "description": "Read ultrasonic distance sensors for obstacle detection", + "category": "sensors", + "parameters": [ + { + "name": "sensor_side", + "type": "select", + "description": "Which sonar sensor to read", + "required": false, + "options": [ + { "value": "both", "label": "Both Sensors" }, + { "value": "left", "label": "Left Sensor Only" }, + { "value": "right", "label": "Right Sensor Only" } + ], + "default": "both" + }, + { + "name": "min_range", + "type": "number", + "description": "Minimum detection range in meters", + "required": false, + "min": 0.1, + "max": 1.0, + "default": 0.25, + "step": 0.05 + }, + { + "name": "max_range", + "type": "number", + "description": "Maximum detection range in meters", + "required": false, + "min": 1.0, + "max": 3.0, + "default": 2.55, + "step": 0.05 + } + ], + "implementation": { + "topics": [ + "/naoqi_driver/sonar/left", + "/naoqi_driver/sonar/right" + ], + "messageType": "sensor_msgs/Range", + "mode": "subscribe" + } + }, + { + "id": "get_imu_data", + "name": "Get IMU Data", + "description": "Read inertial measurement unit data (acceleration, gyroscope, orientation)", + "category": "sensors", + "parameters": [ + { + "name": "data_type", + "type": "select", + "description": "Type of IMU data to monitor", + "required": false, + "options": [ + { "value": "all", "label": "All IMU Data" }, + { "value": "orientation", "label": "Orientation Only" }, + { "value": "acceleration", "label": "Linear Acceleration" }, + { "value": "angular_velocity", "label": "Angular Velocity" } + ], + "default": "all" + } + ], + "implementation": { + "topic": "/naoqi_driver/imu", + "messageType": "sensor_msgs/Imu", + "mode": "subscribe" + } + }, + { + "id": "get_camera_image", + "name": "Get Camera Image", + "description": "Capture image from robot's cameras", + "category": "sensors", + "parameters": [ + { + "name": "camera", + "type": "select", + "description": "Which camera to use", + "required": true, + "options": [ + { "value": "front", "label": "Front Camera" }, + { "value": "bottom", "label": "Bottom Camera" } + ], + "default": "front" + }, + { + "name": "resolution", + "type": "select", + "description": "Image resolution", + "required": false, + "options": [ + { "value": "160x120", "label": "QQVGA (160x120)" }, + { "value": "320x240", "label": "QVGA (320x240)" }, + { "value": "640x480", "label": "VGA (640x480)" } + ], + "default": "320x240" + }, + { + "name": "fps", + "type": "number", + "description": "Frames per second", + "required": false, + "min": 1, + "max": 30, + "default": 15, + "step": 1 + } + ], + "implementation": { + "topic": "/naoqi_driver/camera/{{camera}}/image_raw", + "messageType": "sensor_msgs/Image", + "mode": "subscribe" + } + }, + { + "id": "get_battery_status", + "name": "Get Battery Status", + "description": "Monitor robot battery level and charging status", + "category": "sensors", + "parameters": [], + "implementation": { + "topic": "/naoqi_driver/battery", + "messageType": "sensor_msgs/BatteryState", + "mode": "subscribe" + } + }, + { + "id": "detect_obstacle", + "name": "Detect Obstacle", + "description": "Check for obstacles using sonar sensors with customizable thresholds", + "category": "sensors", + "parameters": [ + { + "name": "detection_distance", + "type": "number", + "description": "Distance threshold for obstacle detection (meters)", + "required": true, + "min": 0.1, + "max": 2.0, + "default": 0.5, + "step": 0.1 + }, + { + "name": "sensor_side", + "type": "select", + "description": "Which sensors to use for detection", + "required": false, + "options": [ + { "value": "both", "label": "Both Sensors" }, + { "value": "left", "label": "Left Sensor Only" }, + { "value": "right", "label": "Right Sensor Only" } + ], + "default": "both" + } + ], + "implementation": { + "topics": [ + "/naoqi_driver/sonar/left", + "/naoqi_driver/sonar/right" + ], + "messageType": "sensor_msgs/Range", + "mode": "subscribe", + "processing": "obstacle_detection" + } + }, + { + "id": "monitor_fall_detection", + "name": "Monitor Fall Detection", + "description": "Monitor robot stability using IMU data to detect potential falls", + "category": "sensors", + "parameters": [ + { + "name": "tilt_threshold", + "type": "number", + "description": "Maximum tilt angle before fall alert (degrees)", + "required": false, + "min": 10, + "max": 45, + "default": 25, + "step": 5 + }, + { + "name": "acceleration_threshold", + "type": "number", + "description": "Acceleration threshold for impact detection (m/sĀ²)", + "required": false, + "min": 5, + "max": 20, + "default": 10, + "step": 1 + } + ], + "implementation": { + "topic": "/naoqi_driver/imu", + "messageType": "sensor_msgs/Imu", + "mode": "subscribe", + "processing": "fall_detection" + } + }, + { + "id": "wait_for_touch", + "name": "Wait for Touch", + "description": "Wait for user to touch a specific sensor before continuing", + "category": "sensors", + "parameters": [ + { + "name": "sensor_location", + "type": "select", + "description": "Which sensor to wait for", + "required": true, + "options": [ + { "value": "head_front", "label": "Head Front" }, + { "value": "head_middle", "label": "Head Middle" }, + { "value": "head_rear", "label": "Head Rear" }, + { "value": "left_hand", "label": "Left Hand" }, + { "value": "right_hand", "label": "Right Hand" }, + { "value": "any_head", "label": "Any Head Sensor" }, + { "value": "any_hand", "label": "Any Hand Sensor" }, + { "value": "any_touch", "label": "Any Touch Sensor" } + ], + "default": "head_front" + }, + { + "name": "timeout", + "type": "number", + "description": "Maximum time to wait for touch (seconds, 0 = infinite)", + "required": false, + "min": 0, + "max": 300, + "default": 30, + "step": 5 + } + ], + "implementation": { + "topics": [ + "/naoqi_driver/head_touch", + "/naoqi_driver/hand_touch" + ], + "messageTypes": [ + "naoqi_bridge_msgs/HeadTouch", + "naoqi_bridge_msgs/HandTouch" + ], + "mode": "wait_for_condition", + "condition": "touch_detected" + } + } + ], + "sensorSpecifications": { + "touchSensors": { + "head": { + "locations": ["front", "middle", "rear"], + "sensitivity": "capacitive", + "responseTime": "< 50ms" + }, + "hands": { + "locations": ["left", "right"], + "sensitivity": "capacitive", + "responseTime": "< 50ms" + } + }, + "sonarSensors": { + "count": 2, + "locations": ["left", "right"], + "minRange": "0.25m", + "maxRange": "2.55m", + "fieldOfView": "60Ā°", + "frequency": "40kHz" + }, + "cameras": { + "front": { + "resolution": "640x480", + "maxFps": 30, + "fieldOfView": "60.9Ā° x 47.6Ā°" + }, + "bottom": { + "resolution": "640x480", + "maxFps": 30, + "fieldOfView": "60.9Ā° x 47.6Ā°" + } + }, + "imu": { + "accelerometer": { + "range": "Ā±2g", + "sensitivity": "high" + }, + "gyroscope": { + "range": "Ā±500Ā°/s", + "sensitivity": "high" + }, + "magnetometer": { + "available": false + } + }, + "joints": { + "count": 25, + "encoderResolution": "12-bit", + "positionAccuracy": "Ā±0.1Ā°" + } + }, + "dataTypes": { + "jointState": { + "position": "radians", + "velocity": "radians/second", + "effort": "arbitrary units" + }, + "imu": { + "orientation": "quaternion", + "angularVelocity": "radians/second", + "linearAcceleration": "m/sĀ²" + }, + "range": { + "distance": "meters", + "minRange": "meters", + "maxRange": "meters" + }, + "image": { + "encoding": "rgb8", + "width": "pixels", + "height": "pixels" + } + }, + "safety": { + "fallDetection": { + "enabled": true, + "defaultThreshold": "25Ā°" + }, + "obstacleDetection": { + "enabled": true, + "safeDistance": "0.3m" + }, + "batteryMonitoring": { + "lowBatteryWarning": "20%", + "criticalBatteryShutdown": "5%" + } + } +} diff --git a/public/nao6-plugins/plugins/nao6-speech.json b/public/nao6-plugins/plugins/nao6-speech.json new file mode 100644 index 0000000..18bbbe9 --- /dev/null +++ b/public/nao6-plugins/plugins/nao6-speech.json @@ -0,0 +1,338 @@ +{ + "name": "NAO6 Speech & Audio", + "version": "1.0.0", + "description": "Text-to-speech and audio capabilities for NAO6 robot including voice synthesis, volume control, and language settings", + "platform": "NAO6", + "category": "speech", + "manufacturer": { + "name": "SoftBank Robotics", + "website": "https://www.softbankrobotics.com" + }, + "documentation": { + "mainUrl": "https://docs.hristudio.com/robots/nao6/speech", + "quickStart": "https://docs.hristudio.com/robots/nao6/speech/quickstart" + }, + "ros2Config": { + "namespace": "/naoqi_driver", + "topics": { + "speech": { + "type": "std_msgs/String", + "description": "Text-to-speech commands" + }, + "set_language": { + "type": "std_msgs/String", + "description": "Set speech language" + }, + "audio_volume": { + "type": "std_msgs/Float32", + "description": "Control audio volume level" + } + } + }, + "actions": [ + { + "id": "say_text", + "name": "Say Text", + "description": "Make the robot speak the specified text using text-to-speech", + "category": "speech", + "parameters": [ + { + "name": "text", + "type": "text", + "description": "Text for the robot to speak", + "required": true, + "maxLength": 500, + "placeholder": "Enter text for NAO to say..." + }, + { + "name": "wait_for_completion", + "type": "boolean", + "description": "Wait for speech to finish before continuing", + "required": false, + "default": true + } + ], + "implementation": { + "topic": "/naoqi_driver/speech", + "messageType": "std_msgs/String", + "messageTemplate": { + "data": "{{text}}" + } + } + }, + { + "id": "say_with_emotion", + "name": "Say Text with Emotion", + "description": "Speak text with emotional expression using SSML-like markup", + "category": "speech", + "parameters": [ + { + "name": "text", + "type": "text", + "description": "Text for the robot to speak", + "required": true, + "maxLength": 500, + "placeholder": "Enter text for NAO to say..." + }, + { + "name": "emotion", + "type": "select", + "description": "Emotional tone for speech", + "required": false, + "options": [ + { "value": "neutral", "label": "Neutral" }, + { "value": "happy", "label": "Happy" }, + { "value": "sad", "label": "Sad" }, + { "value": "excited", "label": "Excited" }, + { "value": "calm", "label": "Calm" } + ], + "default": "neutral" + }, + { + "name": "speed", + "type": "number", + "description": "Speech speed multiplier", + "required": false, + "min": 0.5, + "max": 2.0, + "default": 1.0, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/speech", + "messageType": "std_msgs/String", + "messageTemplate": { + "data": "\\rspd={{speed}}\\\\rst={{emotion}}\\{{text}}" + } + } + }, + { + "id": "set_volume", + "name": "Set Volume", + "description": "Adjust the robot's audio volume level", + "category": "speech", + "parameters": [ + { + "name": "volume", + "type": "number", + "description": "Volume level (0.0 = silent, 1.0 = maximum)", + "required": true, + "min": 0.0, + "max": 1.0, + "default": 0.5, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/audio_volume", + "messageType": "std_msgs/Float32", + "messageTemplate": { + "data": "{{volume}}" + } + } + }, + { + "id": "set_language", + "name": "Set Language", + "description": "Change the robot's speech language", + "category": "speech", + "parameters": [ + { + "name": "language", + "type": "select", + "description": "Speech language", + "required": true, + "options": [ + { "value": "en-US", "label": "English (US)" }, + { "value": "en-GB", "label": "English (UK)" }, + { "value": "fr-FR", "label": "French" }, + { "value": "de-DE", "label": "German" }, + { "value": "es-ES", "label": "Spanish" }, + { "value": "it-IT", "label": "Italian" }, + { "value": "ja-JP", "label": "Japanese" }, + { "value": "ko-KR", "label": "Korean" }, + { "value": "zh-CN", "label": "Chinese (Simplified)" } + ], + "default": "en-US" + } + ], + "implementation": { + "topic": "/naoqi_driver/set_language", + "messageType": "std_msgs/String", + "messageTemplate": { + "data": "{{language}}" + } + } + }, + { + "id": "say_random_phrase", + "name": "Say Random Phrase", + "description": "Make the robot say a random phrase from predefined categories", + "category": "speech", + "parameters": [ + { + "name": "category", + "type": "select", + "description": "Category of phrases", + "required": true, + "options": [ + { "value": "greeting", "label": "Greetings" }, + { "value": "encouragement", "label": "Encouragement" }, + { "value": "question", "label": "Questions" }, + { "value": "farewell", "label": "Farewells" }, + { "value": "instruction", "label": "Instructions" } + ], + "default": "greeting" + } + ], + "implementation": { + "topic": "/naoqi_driver/speech", + "messageType": "std_msgs/String", + "messageTemplate": { + "data": "{{getRandomPhrase(category)}}" + } + }, + "phrases": { + "greeting": [ + "Hello! Nice to meet you!", + "Hi there! How are you today?", + "Welcome! I'm excited to work with you.", + "Good day! Ready to get started?", + "Greetings! What shall we do today?" + ], + "encouragement": [ + "Great job! Keep it up!", + "You're doing wonderfully!", + "Excellent work! I'm impressed.", + "That's fantastic! Well done!", + "Perfect! You've got this!" + ], + "question": [ + "How can I help you today?", + "What would you like to do next?", + "Is there anything you'd like to know?", + "Shall we try something different?", + "What are you thinking about?" + ], + "farewell": [ + "Goodbye! It was great working with you!", + "See you later! Take care!", + "Until next time! Have a wonderful day!", + "Farewell! Thanks for spending time with me!", + "Bye for now! Look forward to seeing you again!" + ], + "instruction": [ + "Please follow my movements.", + "Let's try this step by step.", + "Watch carefully and then repeat.", + "Take your time, there's no rush.", + "Remember to stay focused." + ] + } + }, + { + "id": "spell_word", + "name": "Spell Word", + "description": "Have the robot spell out a word letter by letter", + "category": "speech", + "parameters": [ + { + "name": "word", + "type": "text", + "description": "Word to spell out", + "required": true, + "maxLength": 50, + "placeholder": "Enter word to spell..." + }, + { + "name": "pause_duration", + "type": "number", + "description": "Pause between letters in seconds", + "required": false, + "min": 0.1, + "max": 2.0, + "default": 0.5, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/speech", + "messageType": "std_msgs/String", + "messageTemplate": { + "data": "{{word.split('').join('\\pau={{pause_duration * 1000}}\\\\pau=0\\')}}" + } + } + }, + { + "id": "count_numbers", + "name": "Count Numbers", + "description": "Have the robot count from one number to another", + "category": "speech", + "parameters": [ + { + "name": "start", + "type": "number", + "description": "Starting number", + "required": true, + "min": 0, + "max": 100, + "default": 1, + "step": 1 + }, + { + "name": "end", + "type": "number", + "description": "Ending number", + "required": true, + "min": 0, + "max": 100, + "default": 10, + "step": 1 + }, + { + "name": "pause_duration", + "type": "number", + "description": "Pause between numbers in seconds", + "required": false, + "min": 0.1, + "max": 2.0, + "default": 0.8, + "step": 0.1 + } + ], + "implementation": { + "topic": "/naoqi_driver/speech", + "messageType": "std_msgs/String", + "messageTemplate": { + "data": "{{Array.from({length: end - start + 1}, (_, i) => start + i).join('\\pau={{pause_duration * 1000}}\\\\pau=0\\')}}" + } + } + } + ], + "features": { + "languages": [ + "en-US", "en-GB", "fr-FR", "de-DE", "es-ES", + "it-IT", "ja-JP", "ko-KR", "zh-CN" + ], + "emotions": [ + "neutral", "happy", "sad", "excited", "calm" + ], + "voiceEffects": [ + "speed", "pitch", "volume", "emotion" + ], + "ssmlSupport": true, + "maxTextLength": 500 + }, + "safety": { + "maxVolume": 1.0, + "defaultVolume": 0.5, + "profanityFilter": true, + "maxSpeechDuration": 60, + "emergencyQuiet": { + "action": "set_volume", + "parameters": { "volume": 0 }, + "description": "Immediately mute robot audio" + } + } +} diff --git a/public/nao6-plugins/repository.json b/public/nao6-plugins/repository.json new file mode 100644 index 0000000..9a22e21 --- /dev/null +++ b/public/nao6-plugins/repository.json @@ -0,0 +1,44 @@ +{ + "name": "NAO6 ROS2 Integration Repository", + "description": "Official NAO6 robot plugins for ROS2-based Human-Robot Interaction experiments", + "version": "1.0.0", + "author": { + "name": "HRIStudio Team", + "email": "support@hristudio.com" + }, + "urls": { + "git": "https://github.com/hristudio/nao6-ros2-plugins", + "documentation": "https://docs.hristudio.com/robots/nao6", + "issues": "https://github.com/hristudio/nao6-ros2-plugins/issues" + }, + "trust": "official", + "license": "MIT", + "robots": [ + { + "name": "NAO6", + "manufacturer": "SoftBank Robotics", + "model": "NAO V6", + "communicationProtocol": "ros2" + } + ], + "categories": [ + "movement", + "speech", + "sensors", + "interaction", + "vision" + ], + "ros2": { + "distro": "humble", + "packages": [ + "naoqi_driver2", + "naoqi_bridge_msgs", + "rosbridge_suite" + ], + "bridge": { + "protocol": "websocket", + "defaultPort": 9090 + } + }, + "lastUpdated": "2025-01-16T00:00:00Z" +} diff --git a/scripts/seed-dev.ts b/scripts/seed-dev.ts index d37ac16..c52caf3 100644 --- a/scripts/seed-dev.ts +++ b/scripts/seed-dev.ts @@ -216,9 +216,20 @@ async function main() { manufacturer: "SoftBank Robotics", model: "NAO V6", description: - "Humanoid robot designed for education, research, and social interaction", - capabilities: ["speech", "vision", "walking", "gestures"], - communicationProtocol: "rest" as const, + "Humanoid robot designed for education, research, and social interaction with ROS2 integration", + capabilities: [ + "speech", + "vision", + "walking", + "gestures", + "joint_control", + "touch_sensors", + "sonar_sensors", + "camera_feed", + "imu", + "odometry", + ], + communicationProtocol: "ros2" as const, }, ]; @@ -295,6 +306,17 @@ async function main() { syncStatus: "pending" as const, createdBy: seanUser.id, }, + { + name: "NAO6 ROS2 Integration Repository", + url: "http://localhost:3000/nao6-plugins", + description: + "Official NAO6 robot plugins for ROS2-based Human-Robot Interaction experiments", + trustLevel: "official" as const, + isEnabled: true, + isOfficial: true, + syncStatus: "pending" as const, + createdBy: seanUser.id, + }, ]; const insertedRepos = await db diff --git a/src/app/(dashboard)/nao-test/page.tsx b/src/app/(dashboard)/nao-test/page.tsx new file mode 100644 index 0000000..498f8a8 --- /dev/null +++ b/src/app/(dashboard)/nao-test/page.tsx @@ -0,0 +1,606 @@ +"use client"; + +import { useState, useEffect, useRef } from "react"; +import { Button } from "~/components/ui/button"; +import { + Card, + CardContent, + CardDescription, + CardHeader, + CardTitle, +} from "~/components/ui/card"; +import { Input } from "~/components/ui/input"; +import { Label } from "~/components/ui/label"; +import { Textarea } from "~/components/ui/textarea"; +import { Badge } from "~/components/ui/badge"; +import { Separator } from "~/components/ui/separator"; +import { Slider } from "~/components/ui/slider"; +import { Tabs, TabsContent, TabsList, TabsTrigger } from "~/components/ui/tabs"; +import { Alert, AlertDescription } from "~/components/ui/alert"; +import { PageHeader } from "~/components/ui/page-header"; +import { PageLayout } from "~/components/ui/page-layout"; +import { + Play, + Square, + Volume2, + Camera, + Zap, + ArrowUp, + ArrowDown, + ArrowLeft, + ArrowRight, + RotateCcw, + RotateCw, + Wifi, + WifiOff, + AlertTriangle, + CheckCircle, + Activity, + Battery, + Eye, + Hand, + Footprints, +} from "lucide-react"; + +interface RosMessage { + topic: string; + msg: any; + type: string; +} + +export default function NaoTestPage() { + const [connectionStatus, setConnectionStatus] = useState< + "disconnected" | "connecting" | "connected" | "error" + >("disconnected"); + const [rosSocket, setRosSocket] = useState(null); + const [robotStatus, setRobotStatus] = useState(null); + const [jointStates, setJointStates] = useState(null); + const [speechText, setSpeechText] = useState(""); + const [walkSpeed, setWalkSpeed] = useState([0.1]); + const [turnSpeed, setTurnSpeed] = useState([0.3]); + const [headYaw, setHeadYaw] = useState([0]); + const [headPitch, setHeadPitch] = useState([0]); + const [logs, setLogs] = useState([]); + const [sensorData, setSensorData] = useState({}); + const logsEndRef = useRef(null); + + const ROS_BRIDGE_URL = "ws://134.82.159.25:9090"; + + const addLog = (message: string) => { + const timestamp = new Date().toLocaleTimeString(); + setLogs((prev) => [...prev.slice(-49), `[${timestamp}] ${message}`]); + }; + + useEffect(() => { + logsEndRef.current?.scrollIntoView({ behavior: "smooth" }); + }, [logs]); + + const connectToRos = () => { + if (rosSocket?.readyState === WebSocket.OPEN) return; + + setConnectionStatus("connecting"); + addLog("Connecting to ROS bridge..."); + + const socket = new WebSocket(ROS_BRIDGE_URL); + + socket.onopen = () => { + setConnectionStatus("connected"); + setRosSocket(socket); + addLog("Connected to ROS bridge successfully"); + + // Subscribe to robot topics + subscribeToTopics(socket); + }; + + socket.onmessage = (event) => { + try { + const data = JSON.parse(event.data); + handleRosMessage(data); + } catch (error) { + console.error("Error parsing ROS message:", error); + } + }; + + socket.onclose = () => { + setConnectionStatus("disconnected"); + setRosSocket(null); + addLog("Disconnected from ROS bridge"); + }; + + socket.onerror = () => { + setConnectionStatus("error"); + addLog("Error connecting to ROS bridge"); + }; + }; + + const disconnectFromRos = () => { + if (rosSocket) { + rosSocket.close(); + setRosSocket(null); + setConnectionStatus("disconnected"); + addLog("Manually disconnected from ROS bridge"); + } + }; + + const subscribeToTopics = (socket: WebSocket) => { + const topics = [ + { topic: "/naoqi_driver/joint_states", type: "sensor_msgs/JointState" }, + { topic: "/naoqi_driver/info", type: "naoqi_bridge_msgs/StringStamped" }, + { topic: "/naoqi_driver/bumper", type: "naoqi_bridge_msgs/Bumper" }, + { + topic: "/naoqi_driver/hand_touch", + type: "naoqi_bridge_msgs/HandTouch", + }, + { + topic: "/naoqi_driver/head_touch", + type: "naoqi_bridge_msgs/HeadTouch", + }, + { topic: "/naoqi_driver/sonar/left", type: "sensor_msgs/Range" }, + { topic: "/naoqi_driver/sonar/right", type: "sensor_msgs/Range" }, + ]; + + topics.forEach(({ topic, type }) => { + const subscribeMsg = { + op: "subscribe", + topic, + type, + }; + socket.send(JSON.stringify(subscribeMsg)); + addLog(`Subscribed to ${topic}`); + }); + }; + + const handleRosMessage = (data: any) => { + if (data.topic === "/naoqi_driver/joint_states") { + setJointStates(data.msg); + } else if (data.topic === "/naoqi_driver/info") { + setRobotStatus(data.msg); + } else if ( + data.topic?.includes("bumper") || + data.topic?.includes("touch") || + data.topic?.includes("sonar") + ) { + setSensorData((prev) => ({ + ...prev, + [data.topic]: data.msg, + })); + } + }; + + const publishMessage = (topic: string, type: string, msg: any) => { + if (!rosSocket || rosSocket.readyState !== WebSocket.OPEN) { + addLog("Error: Not connected to ROS bridge"); + return; + } + + const rosMsg = { + op: "publish", + topic, + type, + msg, + }; + + rosSocket.send(JSON.stringify(rosMsg)); + addLog(`Published to ${topic}: ${JSON.stringify(msg)}`); + }; + + const sayText = () => { + if (!speechText.trim()) return; + + publishMessage("/speech", "std_msgs/String", { + data: speechText, + }); + setSpeechText(""); + }; + + const walkForward = () => { + publishMessage("/cmd_vel", "geometry_msgs/Twist", { + linear: { x: walkSpeed[0], y: 0, z: 0 }, + angular: { x: 0, y: 0, z: 0 }, + }); + }; + + const walkBackward = () => { + publishMessage("/cmd_vel", "geometry_msgs/Twist", { + linear: { x: -walkSpeed[0], y: 0, z: 0 }, + angular: { x: 0, y: 0, z: 0 }, + }); + }; + + const turnLeft = () => { + publishMessage("/cmd_vel", "geometry_msgs/Twist", { + linear: { x: 0, y: 0, z: 0 }, + angular: { x: 0, y: 0, z: turnSpeed[0] }, + }); + }; + + const turnRight = () => { + publishMessage("/cmd_vel", "geometry_msgs/Twist", { + linear: { x: 0, y: 0, z: 0 }, + angular: { x: 0, y: 0, z: -turnSpeed[0] }, + }); + }; + + const stopMovement = () => { + publishMessage("/cmd_vel", "geometry_msgs/Twist", { + linear: { x: 0, y: 0, z: 0 }, + angular: { x: 0, y: 0, z: 0 }, + }); + }; + + const moveHead = () => { + publishMessage("/joint_angles", "naoqi_bridge_msgs/JointAnglesWithSpeed", { + joint_names: ["HeadYaw", "HeadPitch"], + joint_angles: [headYaw[0], headPitch[0]], + speed: 0.3, + }); + }; + + const getConnectionStatusIcon = () => { + switch (connectionStatus) { + case "connected": + return ; + case "connecting": + return ; + case "error": + return ; + default: + return ; + } + }; + + const getConnectionStatusBadge = () => { + const variants = { + connected: "default", + connecting: "secondary", + error: "destructive", + disconnected: "outline", + } as const; + + return ( + + {getConnectionStatusIcon()} + {connectionStatus.charAt(0).toUpperCase() + connectionStatus.slice(1)} + + ); + }; + + return ( + + + +
+ {/* Connection Status */} + + + + ROS Bridge Connection + {getConnectionStatusBadge()} + + + Connect to ROS bridge at {ROS_BRIDGE_URL} + + + +
+ {connectionStatus === "connected" ? ( + + ) : ( + + )} +
+ + + + {connectionStatus === "connected" && ( + + + Robot Control + Sensor Data + Robot Status + Logs + + + +
+ {/* Speech Control */} + + + + + Speech + + + +
+ +