← Back to the Log Index · Previous day: 2026-10-09 · Next day: (not yet written)
Summary of the day
- Continuing the hardware stage from 2026-10-09 > 7. Next session: testing every joint of both arms, one at a time.
- All 14 arm joints tested on the real robot (0.3 rad, 5 s, safe directions) with a new safety-checked script: every one moved the expected way and matched RViz.
- A demo script was written. Its first real run moved the wrong arm, because the launch had been typed without the CAN arguments. Fixed for good with a launch wrapper (
openarm_launch.sh real) plus an identification wiggle, confirmed on the robot.- Shared scripts folder
Prandium/robot_scripts=/root/scripts(new containeropenarm, imageopenarm-jazzy:rt2).- Guide V3.1 written: everything learned on the real robot since V2.2/V3. See 8. Documentation: Guide V3.1.
- Ctrl+C on the real launch crashes
ros2_control_nodebefore the motors are switched off (the pending shutdown test). Fixed with a safe-stop script (openarm_stop.sh): both arms reportedOpenArm V10 deactivated, clean shutdown.
Contents of this log
- 1. Session start
- 2. Joint-test helper script
- 3. Joint tests
- 4. Demo script
- 5. Incident: right-arm commands moved the left arm
- 6. Shutdown test (H5d): the controller manager crashes on Ctrl+C
- 7. Shared scripts folder
- 8. Documentation: Guide V3.1
1. Session start
- Start-up checks (Manuel): both CAN buses up (
can0re-configured after the night), everything else as expected. - Last night’s shutdown test (H5d) was not recorded. To be watched at the end of today’s session, with the arms at zero.
Launch (both arms powered first), log /root/h7_launch.log. Start-up check passed for both arms, all values real and inside 0.5 rad:
| Joint | Right (first) | Left |
|---|---|---|
| 1 | −0.009 | −0.047 |
| 2 | +0.005 | −0.011 |
| 3 | −0.009 | −0.013 |
| 4 | −0.007 | −0.025 |
| 5 | −0.054 | +0.135 |
| 6 | +0.015 | −0.259 (≈15°, hung further off than before; still well inside the check) |
| 7 | +0.072 | +0.109 |
Both Reached zero position ~10.4 s after their start.
Overruns with both arms answering: loops of 1.34–1.58 ms against the 1.333 ms budget (750 Hz); read 0.53–0.81 ms, write 0.53–0.70 ms, update 0.10–0.16 ms. Much better than on 2026-10-09 with the virtual right arm (2.0–2.6 ms), but still just over budget. Almost all the time is CAN I/O (read + write), not computation. The PCAN adapter is plugged into a USB-C docking station, which adds USB latency and jitter. To try: the adapter directly in a laptop USB port, and compare. Harmless at today’s test speeds (a late cycle = the next target arrives ~1 ms later).
2. Joint-test helper script
Prandium/robot_scripts/openarm_jmove.py (written by Claude, reviewed and run by Manuel inside the container). Usage: python3 /root/openarm_jmove.py <left|right> <joint 1-7> <target_rad> [seconds].
What it does, and why:
- Moves one joint; the other six keep their current positions, so a forgotten earlier move is never silently undone or redone, and there’s no 7-value array to mistype.
- Refuses targets outside that arm’s real limits (per-arm table from 2026-10-09 > 5.9 H6: first commanded moves, and the joint direction map, with a 0.05 rad margin), steps larger than 0.35 rad from the current position, average speeds above 0.1 rad/s, and durations under 5 s.
- Prints the current and goal positions and asks y/N before sending.
- Ctrl+C during the move cancels the goal (the arm holds where it is).
- Prints the final position and the error afterwards.
Sequence version: Prandium/robot_scripts/openarm_jsweep.py <left|right|both> [amplitude_rad] [seconds] (requested by Manuel). Tests joints 1–7 one after another: each goes from where it is to the test position and back, 5 s each way with a 1 s hold, default 0.3 rad (17°), the other joints held. Same limit, step (≤0.35 rad) and speed (≤0.1 rad/s) checks; prints the whole plan with the expected direction for every joint and asks y/N once; stops if a joint misses its target by more than 0.1 rad; Ctrl+C cancels the current motion and stops.
Test directions, chosen so nothing swings toward the body (from the direction map):
| Joint | Left | Right | Why |
|---|---|---|---|
| 1 | −0.3 | +0.3 | forward on both arms |
| 2 | −0.3 | +0.3 | outward on both (only 10° of room inward) |
| 3, 4, 5, 7 | +0.3 | +0.3 | no collision risk from the hanging pose (joint4 can only go +) |
| 6 | +0.3 | −0.3 | outward on both, so the hand doesn’t swing in toward the base (Manuel’s requirement) |
Both scripts: rclpy.init(signal_handler_options=SignalHandlerOptions.NO), because by default rclpy shuts its context down on Ctrl+C, which would make the goal cancel fail exactly when it’s needed.
3. Joint tests
python3 /root/openarm_jsweep.py left, then right (0.3 rad, 5 s each way, directions as in section 2). Reported by Manuel: everything ran correctly: every joint of both arms reached its target and came back, and moved in the expected direction, matching RViz. (Script output not pasted, so per-joint errors weren’t recorded.)
This completes the motor-direction check of Guide §12.3 for all 14 arm joints: every motor’s direction agrees with the model, which confirms the direction map on the real robot.
4. Demo script
There’s no ready-made demo in openarm_ros2 (the MoveIt demo is interactive planning, not a show), so Claude wrote Prandium/robot_scripts/openarm_demo.py [time_scale]. It runs both arms and both grippers through a ~55 s choreography, starting and ending at the hanging zero pose:
| Time (s) | Stage |
|---|---|
| 0 → 6 | both forearms come up in front (“ready”: joint4 = 1.0) |
| 6 → 14 | grippers open, close, open, close |
| 14 → 25 | forearms twist (joint5 ±0.6, mirrored), then back to centre |
| 25 → 43 | right hand up (joint1 +0.4 forward, joint4 1.6), two waves (joint3 ±0.35) |
| 43 → 55 | back to “ready”, then down to zero |
Safety built in: every waypoint is checked against the per-arm limits and every segment against an average speed of 0.25 rad/s (the plan’s fastest segment is 0.24 rad/s, the forearm twist); the arms must start within 0.1 rad of zero; the plan is printed and confirmed y/N; time_scale can only make it slower (≥ 1); Ctrl+C cancels all four goals. Waypoints have zero velocity and acceleration, so the controller makes smooth quintic moves that stop gently at each pose.
Poses checked with forward kinematics on the expanded v1.0 model (Claude, laptop scratch folder):
| Pose | Left hand (x, y, z) m | Right hand (x, y, z) m | Hands apart |
|---|---|---|---|
| zero (hanging) | 0, +0.153, 0.159 | 0, −0.153, 0.159 | 0.31 m |
| ready | 0.268, +0.153, 0.306 | 0.268, −0.153, 0.306 | 0.31 m |
| right hand up | 0.268, +0.153, 0.306 | 0.375, −0.153, 0.628 | 0.46 m |
| wave, inner end | same | 0.358, −0.044, 0.620 | 0.38 m |
x = forward, y = robot’s left, z = up from the robot’s base frame. Hands always in front of the torso, never lower than when hanging. Joint5 twists don’t move the hand position (they rotate the gripper).
Joint4's limit is exactly 0
With a 0.05 rad safety margin, the hanging pose (joint4 = 0) itself looked “outside the limits”. Found by running the demo’s own checks offline; both
openarm_demo.pyandopenarm_jmove.pynow allow exactly 0 (the home pose). (openarm_jsweep.pynever targets 0, so it was unaffected.)
Plan: run it in simulation first and watch RViz, then on the robot with time_scale 1.5 (≈83 s), weight on the base.
5. Incident: right-arm commands moved the left arm
issue/can issue/ros status/resolved pitfall
What happened (Manuel): the demo ran fine in simulation, where the wave was on the right arm. On the real robot, the wave motion was executed by the physical left arm. Manuel cut the motor power with the switch, then switched it back on (while the ROS launch was presumably still running).
Why it matters: the arms are mirror images, so a right-arm target on the left arm means a different motion (joint1 +0.4 = forward on the right arm, backward on the left). The per-arm limit checks in the scripts are applied to the arm the command is meant for, so they don’t protect the physical arm that actually receives it (e.g. right joint2 outward = left joint2 into the body).
Most likely cause (to confirm): the real launch was started without left_can_interface:=can0 right_can_interface:=can1, or with the guide’s §12.2 example, which uses the ROS defaults (right = can0, left = can1). On zeus the cabling is the reverse (left = can0, Guide §22). This morning’s joint tests (launched with the correct arguments) moved the correct arms. Other possibility: the PCAN adapter was moved to another USB port and the channels enumerated differently.
Root cause (confirmed by Manuel): the real launch was started without left_can_interface:=can0 right_can_interface:=can1, so ROS used its defaults (right = can0, left = can1), the reverse of zeus’s cabling. The robot is fine; the power switch stopped it. The start-up check can’t catch this, since both arms hang at zero either way.
Fix: the mapping is no longer typed by hand. Prandium/robot_scripts/openarm_launch.sh real|sim (run inside the container):
real: launches withleft_can_interface:=can0 right_can_interface:=can1 can_fd:=true use_fake_hardware:=false, refuses if a launch is already running or ifcan0/can1aren’t up, reminds that both arms must be powered and hanging first, logs to/root/launch_real_<date>.logwithtee -i, and prints the identification step:python3 /root/openarm_jmove.py left 7 0.1must move the robot’s own left wrist (the robot’s own left = on your left when standing behind it).sim: mock hardware.
Rule: real-robot launches only through
openarm_launch.sh real, followed by the identification wiggleAnd “left/right” always means the robot’s own left/right.
Status: resolved by the wrapper (to be used from the next launch on).
6. Shutdown test (H5d): the controller manager crashes on Ctrl+C
issue/ros status/resolved pitfall
The pending shutdown test happened when Manuel stopped the morning’s real launch (Ctrl+C once, arms at zero). Output:
- Controller manager:
Shutdown request received→ deactivates and shuts down all 5 controllers →'deactivate' hardware 'openarm_right_hardware_interface'→OpenArmHW: Deactivating OpenArm V10... - at the same moment,
ros2_control_nodecrashed:Segmentation fault (Address not mapped to object [(nil)])injoint_trajectory_controller::JointTrajectoryController::update, called fromControllerManager::update. The real-time loop was still calling a controller that had just been shut down (a shutdown race).process has died … exit code -11. OpenArm V10 deactivatednever appeared, and the left hardware was never deactivated.
Consequence: on this setup, Ctrl+C does not reliably switch the motors off. The left arm (and possibly the right) stays enabled, holding its last target, until the motor power is cut. Harmless at the hanging pose, but not in a raised pose: if the arms are later depowered there, they drop.
Safe stop procedure until this is fixed (to verify):
- Bring the arms to zero (hanging).
- Switch the hardware off explicitly, while ROS is still running:
ros2 control set_hardware_component_state openarm_left_hardware_interface inactive(and…_right_…). The driver’son_deactivatedisables the motors. - Only then Ctrl+C the launch, and switch the motor power off.
Implemented as Prandium/robot_scripts/openarm_stop.sh (asks for confirmation that the arms hang at zero; deactivates the 4 arm/gripper controllers, which should also stop the race behind the crash; sets both hardware components inactive; then pkill -INT on the launch). First test in sim (bash /root/openarm_launch.sh sim, then bash /root/openarm_stop.sh):
- Step 1:
Deactivated controllers: [ left_joint_trajectory_controller right_joint_trajectory_controller left_gripper_controller right_gripper_controller ]. - Step 2: both hardware components
Successful 'deactivate'→inactive. - Shutdown: all processes
finished cleanly, no segfault. - One flaw:
pkill -INTreached only theros2 launchprocess, so its children waited 5 s for a SIGTERM escalation. Fixed: the script now signals the launch’s whole process group (kill -INT -- -<pgid>), like Ctrl+C. Harmless messages: the switch “strictness level” warning (best effort is what’s wanted here) andpal_statistics … context cannot be slept withat shutdown.
Also fixed before the test: both wrappers used set -u, which breaks sourcing /opt/ros/jazzy/setup.bash (AMENT_TRACE_SETUP_FILES: unbound variable); now relaxed around the source lines.
Real-robot test (Manuel): bash /root/openarm_launch.sh real → identification python3 /root/openarm_jmove.py left 7 0.1: the robot’s left wrist moved ✅ (mapping confirmed) → back to 0 → bash /root/openarm_stop.sh with both arms hanging:
- controllers deactivated;
OpenArmHW: Deactivating OpenArm V10...→OpenArm V10 deactivated, for the left and then the right arm (motors disabled while ROS was still running);- shutdown immediate (the group SIGINT reached every process), all processes
finished cleanly, no segfault.
Status: resolved. Safe stop = openarm_stop.sh, never a plain Ctrl+C on the real launch.
7. Shared scripts folder
Why: a container’s files are isolated from the laptop by default; until now every script went in with docker cp, and logs came out the same way. Manuel asked for a shared folder for the choreography and his own future scripts.
Folder: Prandium/robot_scripts/ on the laptop = /root/scripts/ in the container (a bind mount, -v). The five OpenArm scripts were moved there from Prandium/Scripts/ (paths inside them updated to /root/scripts/…), plus a README.md with the real-robot routine. Commands earlier in this log that use /root/openarm_*.py refer to the old docker cp copies.
Rule: write and edit scripts on the laptop, run them in the container. Files created inside the container are owned by root on the laptop.
Like the real-time flags, a bind mount can only be set at docker run, so the container was recreated (Manuel):
| Step | Command |
|---|---|
| Stop | docker stop openarm |
| Snapshot | docker commit openarm openarm-jazzy:rt2 |
| Keep the old one | docker rename openarm openarm-noshare |
| New container | same docker run as on 2026-10-09 + -v "$HOME/Documents/vault/Prandium/robot_scripts:/root/scripts", image openarm-jazzy:rt2 |
| Tidy | rm /root/openarm_*.py /root/openarm_*.sh (old docker cp copies) |
All checks passed: /root/scripts lists the scripts, real-time limits 99 / unlimited, git diff --stat = the same 3 files, GPU visible, and a file created on the laptop appeared in the container immediately.
Shortcut in the landing folder (Manuel’s request): shells start in /ws (the image’s default working directory, empty, left over from an earlier setup), so a symbolic link ln -s /root/scripts /ws/scripts makes the shared folder reachable as scripts/ from where you land. Same folder, no container change needed; it lives in the container’s filesystem (would need redoing only if the container is recreated from an older image).
Containers now
openarm(imageopenarm-jazzy:rt2, real-time + shared folder): the only one to use. Backups, never run alongside:openarm-noshare(this morning),openarm-deprecated(from 2026-10-08).
Demo speed: Manuel asked about time_scale 0.8. The script refuses < 1 by design, and at 0.8 the fastest segment would average 0.30 rad/s (peak ≈ 0.56 rad/s), above MAX_SPEED = 0.25. Agreed approach: real runs at 1.5, then 1.0; 0.8 only after that, by raising MAX_SPEED to 0.31 and the minimum scale to 0.8 (base unclamped: faster = higher accelerations = more tipping force).
8. Documentation: Guide V3.1
Base version: a V3 already existed (OpenArm_ROS2_and_teleop_Guide_V3.md, 2026-10-09 19:08, written outside these test sessions). It is V2.2 plus Part III (§30–§43, running a trained policy with lerobot-rollout) and three LeRobot corrections; its ROS part (Part I) was identical to V2.2. So V3.1 = V3 + the changes below, and Manuel chose the name V3.1. V3 itself is unchanged (versioning rule, 2026-10-09 > 1.2 Versioning convention (agreed today)). The navigation box of every version (V1, V2, V2.1, V2.2, V3) now links to V3.1 as the current one; that’s a one-line, formatting-only change in each (allowed in place).
What changed in V3.1, by section (a “What changed in V3.1” banner at the top lists the same; V3’s banner is kept below it as a note):
| Section | Change | Source |
|---|---|---|
| Header | New box “Local changes on zeus”: splines + patches 0001/0002, how to check them (git diff --stat), what undoes them | 10-09 §5.3, §5.5 |
| §3 Joint names | Left/right rows show the ROS default bus and zeus’s (left can0, right can1) | §22 |
| §3 Joint limits | Table replaced by the per-arm limits as actually loaded (from the expanded URDF); explains why joint_limits.yaml is wrong for joint1 (left) and joint2 (both); warning that nothing enforces limits | 10-09 §5.9 |
| §3 (new) | Direction map: which way ”+” moves each joint, per arm; robot frame; hand positions at zero; confirmed on the robot | 10-09 §5.9, 3. Joint tests |
| §4 Hardware | Bus defaults annotated with zeus’s mapping; on_activate row: upstream behaviour (incl. the kick) vs patched | 10-09 §5.5 |
| §4 Consequences | New: joints stop short (friction vs kp·error, with numbers); torque-on only at start-up → power before launch | 10-09 §5.8, patch 0002 |
| §4 (new) “Start-up on zeus (patched driver)“ | Before/after table of the patches; expected log lines; refusal messages; ~21 s start-up, spawner retries, controllers at ~25–30 s; both arms must be powered (no vcan trick); apply/undo | 10-09 §5.5–5.8, 1. Session start |
| §5 Container | Rewritten for openarm / openarm-jazzy:rt2: the actual docker run; table of containers and images (openarm, openarm-noshare, openarm-deprecated) and “never two at once”; .bashrc already sources ROS; flags table incl. RT flags and the shared folder | 10-09 §5.6, 7. Shared scripts folder |
| §5 (new) | “Why real-time scheduling, and why Docker doesn’t allow it by default”; “Shared scripts folder” (incl. /ws/scripts, docker cp for other files); “Changing container flags later” (snapshot procedure) | same |
§6 (new) “On zeus: use openarm_launch.sh” | What the wrapper does and why (wrong-arm incident); launch-argument table annotated with zeus’s buses | 5. Incident: right-arm commands moved the left arm |
| §7 | Sim via the wrapper; Ctrl+C fine in sim | — |
| §8.1 | Rule added: check per-arm limits and the direction map | — |
| §8.7 (new) “Helper scripts on zeus” | All five scripts: usage, checks; jsweep directions; demo choreography and its limits; why the scripts handle Ctrl+C themselves | 2. Joint-test helper script, 4. Demo script |
| §10.1, §10.3 | zeus note: one PEAK PCAN-USB Pro FD, stable names, reversed cabling, USB-C dock; udev naming not needed | 10-09 §5.4 |
| §10.4 | zeus commands with expected output, mtu 72, bus silent with motors off; tip: why bit rates must be set by hand | 10-09 §5.4 |
| §10.5 | Correction: motors aren’t “disabled” during monitor; table of what discover/monitor/diagnose/set_zero really do; DM4310 decoding | 10-09 §5.5 and H2/H3 |
| §11 | Option B: set_zero = all 8 motors; new Option C (single motor, --no-arm --id N, the right joint5 example); new “Checking the zero (every session)” with thresholds | 10-09 H3/H3b |
| §12.1 | Checklist rewritten (power switch, base weighted, nothing else on the buses, power before launch, wrapper + identification, local changes, RT, stop plan) | all |
| §12.2 | Rewritten: launch via the wrapper, what happens with log lines and timing, checks, identification wiggle, raw commands with zeus arguments, “one arm only” | 10-09 §5.8, 5. Incident: right-arm commands moved the left arm |
| §12.3 | Rewritten around jmove / jsweep / demo; rules of thumb updated (splines, per-arm limits, direction map, stopping short) | 3. Joint tests |
| §12.4 | Rewritten: openarm_stop.sh as the normal stop; why not Ctrl+C (the segfault); other stops; pitfalls (power while running, repeated Ctrl+C, tee -i) | 6. Shutdown test (H5d): the controller manager crashes on Ctrl+C |
| §12.5 | RT done on zeus; measured overrun timings; docking-station hypothesis; next steps (direct USB port, 500 Hz) | 1. Session start |
| §13 Safety notes | Items 1, 2, 7 updated (patched start-up; Ctrl+C may leave torque on; zeus defaults are the swapped mapping); new items 10–12 (limits file, power after launch, CLI enables motors) | — |
| §14 Cheat-sheet | tee -i; zeus scripts; git diff --stat for the local changes | — |
| §15 Troubleshooting | 12 new rows: arms missing for ~25 s, spawner retries, no reply, rad from zero, all-zero start values, other arm moves, Ctrl+C segfault, tee without -i, overruns, ros2: command not found, set -u in scripts | both days |
| §16 Known issues | Upstream return_to_zero kick/2 s/no checks; torque-on only at activation; Ctrl+C segfault; joint_limits.yaml vs URDF; CLI behaviour | both days |
| §22 | ROS bullet: the wrapper passes the arguments; the 2026-10-10 incident; identify after every launch | 5. Incident: right-arm commands moved the left arm |
| §25 | ROS → LeRobot uses openarm_stop.sh; LeRobot → ROS uses the wrapper, 10 s start-up, identification | — |
| §29 Quick reference | ROS lines replaced by the zeus routine (CAN setup, wrapper, identification, jsweep, demo, stop) | — |
Unchanged: §1–§2, §9, Part II apart from §22/§25/§29, and all of Part III. Checks after writing: no broken internal links in V3.1; every link into the logs points to an existing heading; V1–V3 differ from before only in the navigation line.