nao6 ros2 integration updated

2025-12-11 22:54:45 -05:00 · 2025-11-13 10:58:45 -05:00
parent 70882b9dbb
commit 86b5ed80c4
276 changed files with 4288 additions and 1552 deletions
--- a/HANDOFF-NAO6-INTEGRATION.md
+++ b/HANDOFF-NAO6-INTEGRATION.md
@@ -0,0 +1,344 @@
+# NAO6 Integration Handoff Document
+
+**Date**: 2024-11-12  
+**Status**: ✅ Production Ready - Action Execution Pending  
+**Session Duration**: ~3 hours  
+
+---
+
+## 🎯 What's Ready
+
+### ✅ Completed
+1. **Live Robot Connection** - NAO6 @ nao.local fully connected via ROS2
+2. **Plugin System** - NAO6 ROS2 Integration plugin (v2.1.0) with 10 actions
+3. **Database Integration** - Plugin installed, experiments seeded with NAO6 actions
+4. **Web Test Interface** - `/nao-test` page working with live robot control
+5. **Documentation** - 1,877 lines of comprehensive technical docs
+6. **Repository Cleanup** - Consolidated into `robot-plugins` git repo (pushed to GitHub)
+
+### 🚧 Next Step: Action Execution
+**Current Gap**: Experiment designer → WebSocket → ROS2 → NAO flow not implemented
+
+The plugin is loaded, actions are in the database, but clicking "Execute" in the wizard interface doesn't send commands to the robot yet.
+
+---
+
+## 🚀 Quick Start (For Next Agent)
+
+### Terminal 1: Start NAO6 Integration
+```bash
+cd ~/Documents/Projects/nao6-hristudio-integration
+./start-nao6.sh
+```
+**Expect**: Color-coded logs showing NAO Driver, ROS Bridge, ROS API running
+
+### Terminal 2: Start HRIStudio
+```bash
+cd ~/Documents/Projects/hristudio
+bun dev
+```
+**Access**: http://localhost:3000
+
+### Verify Setup
+1. **Test Page**: http://localhost:3000/nao-test
+   - Click "Connect" → Should turn green
+   - Click "Speak" → NAO should talk
+   - Movement buttons → NAO should move
+
+2. **Experiment Designer**: http://localhost:3000/experiments/[id]/designer
+   - Check "Basic Interaction Protocol 1"
+   - Should see NAO6 actions in action library
+   - Drag actions to experiment canvas
+
+3. **Database Check**:
+   ```bash
+   bun db:seed  # Should complete without errors
+   ```
+
+---
+
+## 📁 Key File Locations
+
+### NAO6 Integration Repository
+```
+~/Documents/Projects/nao6-hristudio-integration/
+├── start-nao6.sh              # START HERE - runs everything
+├── nao6-plugin.json           # Plugin definition (10 actions)
+├── SESSION-SUMMARY.md         # Complete session details
+└── docs/                      # Technical references
+    ├── NAO6-ROS2-TOPICS.md           (26 topics documented)
+    ├── HRISTUDIO-ACTION-MAPPING.md   (Action specs + TypeScript types)
+    └── INTEGRATION-SUMMARY.md        (Quick reference)
+```
+
+### HRIStudio Project
+```
+~/Documents/Projects/hristudio/
+├── robot-plugins/             # Git submodule @ github.com/soconnor0919/robot-plugins
+│   └── plugins/
+│       └── nao6-ros2.json    # MAIN PLUGIN FILE (v2.1.0)
+├── src/app/(dashboard)/nao-test/
+│   └── page.tsx              # Working test interface
+├── src/components/experiments/designer/
+│   └── ActionRegistry.ts     # Loads plugin actions
+└── scripts/seed-dev.ts       # Seeds NAO6 plugin into DB
+```
+
+---
+
+## 🔧 Current System State
+
+### Database
+- **2 repositories**: Core + Robot Plugins
+- **4 plugins**: Core System, TurtleBot3 Burger, TurtleBot3 Waffle, **NAO6 ROS2 Integration**
+- **NAO6 installed in**: "Basic Interaction Protocol 1" study
+- **Experiment actions**: Step 1 has "NAO Speak Text", Step 3 has "NAO Move Head"
+
+### ROS2 System
+- **26 topics** available when `start-nao6.sh` is running
+- **Key topics**: `/speech`, `/cmd_vel`, `/joint_angles`, `/joint_states`, `/bumper`, etc.
+- **WebSocket**: ws://localhost:9090 (rosbridge_websocket)
+
+### Robot
+- **IP**: nao.local (134.82.159.168)
+- **Credentials**: nao / robolab
+- **Status**: Awake and responsive (test with ping)
+
+---
+
+## 🎯 Implementation Needed
+
+### 1. Action Execution Flow
+**Where to implement**: 
+- `src/components/trials/WizardInterface.tsx` or similar
+- Connect "Execute Action" button → WebSocket → ROS2
+
+**What it should do**:
+```typescript
+// When wizard clicks "Execute Action" on a NAO6 action
+function executeNAO6Action(action: Action) {
+  // 1. Get action parameters from database
+  const { type, parameters } = action;
+  
+  // 2. Connect to WebSocket (if not connected)
+  const ws = new WebSocket('ws://localhost:9090');
+  
+  // 3. Map action type to ROS2 topic
+  const topicMapping = {
+    'nao6_speak': '/speech',
+    'nao6_move_forward': '/cmd_vel',
+    'nao6_move_head': '/joint_angles',
+    // ... etc
+  };
+  
+  // 4. Create ROS message
+  const rosMessage = {
+    op: 'publish',
+    topic: topicMapping[type],
+    msg: formatMessageForROS(type, parameters)
+  };
+  
+  // 5. Send to robot
+  ws.send(JSON.stringify(rosMessage));
+  
+  // 6. Log to trial_events
+  logTrialEvent({
+    trial_id: currentTrialId,
+    event_type: 'action_executed',
+    event_data: { action, timestamp: Date.now() }
+  });
+}
+```
+
+### 2. Message Formatting
+**Reference**: See `nao6-hristudio-integration/docs/HRISTUDIO-ACTION-MAPPING.md`
+
+**Examples**:
+```typescript
+function formatMessageForROS(actionType: string, params: any) {
+  switch(actionType) {
+    case 'nao6_speak':
+      return { data: params.text };
+      
+    case 'nao6_move_forward':
+      return {
+        linear: { x: params.speed, y: 0, z: 0 },
+        angular: { x: 0, y: 0, z: 0 }
+      };
+      
+    case 'nao6_move_head':
+      return {
+        joint_names: ['HeadYaw', 'HeadPitch'],
+        joint_angles: [params.yaw, params.pitch],
+        speed: params.speed,
+        relative: 0
+      };
+  }
+}
+```
+
+### 3. WebSocket Connection Management
+**Suggested approach**:
+- Create `useROSBridge()` hook in `src/hooks/`
+- Manage connection state, auto-reconnect
+- Provide `publish()`, `subscribe()`, `callService()` methods
+
+---
+
+## 🧪 Testing Checklist
+
+Before marking as complete:
+- [ ] Can execute "Speak Text" action from wizard interface → NAO speaks
+- [ ] Can execute "Move Forward" action → NAO walks
+- [ ] Can execute "Move Head" action → NAO moves head
+- [ ] Actions are logged to `trial_events` table
+- [ ] Connection errors are handled gracefully
+- [ ] Emergency stop works from wizard interface
+- [ ] Multiple actions in sequence work
+- [ ] Sensor monitoring displays in wizard interface
+
+---
+
+## 📚 Reference Documentation
+
+### Primary Sources
+1. **Working Example**: `src/app/(dashboard)/nao-test/page.tsx`
+   - Lines 67-100: WebSocket connection setup
+   - Lines 200-350: Action execution examples
+   - This is WORKING code - use it as template!
+
+2. **Action Specifications**: `nao6-hristudio-integration/docs/HRISTUDIO-ACTION-MAPPING.md`
+   - Lines 1-100: Each action with parameters
+   - TypeScript types already defined
+   - WebSocket message formats included
+
+3. **ROS2 Topics**: `nao6-hristudio-integration/docs/NAO6-ROS2-TOPICS.md`
+   - Complete message type definitions
+   - Examples for each topic
+
+### TypeScript Types
+Already defined in action mapping doc:
+```typescript
+interface SpeakTextAction {
+  action: 'nao6_speak';
+  parameters: {
+    text: string;
+    volume?: number;
+  };
+}
+
+interface MoveForwardAction {
+  action: 'nao6_move_forward';
+  parameters: {
+    speed: number;
+    duration: number;
+  };
+}
+```
+
+---
+
+## 🔍 Where to Look
+
+### To understand plugin loading:
+- `src/components/experiments/designer/ActionRegistry.ts`
+- `src/components/experiments/designer/panels/ActionLibraryPanel.tsx`
+
+### To see working WebSocket code:
+- `src/app/(dashboard)/nao-test/page.tsx` (fully functional!)
+
+### To find action execution trigger:
+- Search for: `executeAction`, `onActionExecute`, `runAction`
+- Likely in: `src/components/trials/` or `src/components/experiments/`
+
+---
+
+## 🚨 Important Notes
+
+### DO NOT
+- ❌ Modify `robot-plugins/` without committing/pushing (it's a git repo)
+- ❌ Change plugin structure without updating seed script
+- ❌ Remove `start-nao6.sh` - it's the main entry point
+- ❌ Hard-code WebSocket URLs - use config/env vars
+
+### DO
+- ✅ Use existing `/nao-test` page code as reference
+- ✅ Test with live robot frequently
+- ✅ Log all actions to `trial_events` table
+- ✅ Handle connection errors gracefully
+- ✅ Add loading states for action execution
+
+### Known Working
+- ✅ WebSocket connection (`/nao-test` proves it works)
+- ✅ ROS2 topics (26 topics verified)
+- ✅ Plugin loading (shows in action library)
+- ✅ Database integration (seed script works)
+
+### Known NOT Working
+- ❌ Action execution from experiment designer/wizard interface
+- ❌ Sensor data display in wizard interface (topics exist, just not displayed)
+- ❌ Camera streaming to browser
+
+---
+
+## 🤝 Session Handoff Summary
+
+### What We Did
+1. Connected to live NAO6 robot at nao.local
+2. Documented all 26 ROS2 topics with complete specifications
+3. Created clean plugin with 10 actions
+4. Integrated plugin into HRIStudio database
+5. Built working test interface proving WebSocket → ROS2 → NAO works
+6. Cleaned up repositories (removed duplicates, committed to git)
+7. Updated experiments to use NAO6 actions
+8. Fixed APT repository issues
+9. Created comprehensive documentation (1,877 lines)
+
+### What's Left
+**ONE THING**: Connect the experiment designer's "Execute Action" button to the WebSocket/ROS2 system.
+
+The hard part is done. The `/nao-test` page is a fully working example of exactly what you need to do - just integrate that pattern into the wizard interface.
+
+---
+
+## 🎓 Key Insights
+
+1. **We're NOT writing a WebSocket server** - using ROS2's official `rosbridge_websocket`
+2. **The test page works perfectly** - copy its pattern
+3. **All topics are documented** - no guesswork needed
+4. **Plugin is in database** - just needs execution hookup
+5. **Robot is live and responsive** - test frequently!
+
+---
+
+## ⚡ Quick Commands
+
+```bash
+# Start everything
+cd ~/Documents/Projects/nao6-hristudio-integration && ./start-nao6.sh
+cd ~/Documents/Projects/hristudio && bun dev
+
+# Test robot
+curl -X POST http://localhost:9090 -d '{"op":"publish","topic":"/speech","msg":{"data":"Test"}}'
+
+# Reset robot
+sshpass -p robolab ssh nao@nao.local \
+  "python2 -c 'import sys; sys.path.append(\"/opt/aldebaran/lib/python2.7/site-packages\"); import naoqi; p=naoqi.ALProxy(\"ALRobotPosture\",\"127.0.0.1\",9559); p.goToPosture(\"StandInit\", 0.5)'"
+
+# Reseed database
+cd ~/Documents/Projects/hristudio && bun db:seed
+```
+
+---
+
+**Next Agent**: Start by reviewing `/nao-test/page.tsx` - it's the Rosetta Stone for this integration. Everything you need is already working there!
+
+**Estimated Time**: 2-4 hours to implement action execution  
+**Difficulty**: Medium (pattern exists, just needs integration)  
+**Priority**: High (this is the final piece)
+
+---
+
+**Status**: 🟢 Ready for implementation  
+**Blocker**: None - all prerequisites met  
+**Dependencies**: Robot must be running (`start-nao6.sh`)