23. Step-by-step teleoperation

Part II: Teleoperation and data collection · §23 of 43

Everything below runs on the host. No ROS launch may be running on the follower buses (§25). The base must be clamped or screwed to the table (OpenArm safety guide). On zeus the whole robot tipped over once when an arm snapped.

23.1 Setup and calibration: setup_openarm_lerobot.sh

bash ~/Documents/openarm_lerobot/setup_openarm_lerobot.sh                       # normal run (re-runnable)
bash ~/Documents/openarm_lerobot/setup_openarm_lerobot.sh --recalibrate-minis   # force a Mini recalibration

It activates the lerobot env itself and stops at the first problem:

1. Environment: dialout group active, both Minis present, no teleop/record/ROS control running.
2. CAN: brings can0/can1 up in CAN-FD (1 / 5 Mbit/s) if needed (sudo), then requires 16/16 follower motors to answer.
3. Follower placeholders: creates my_bimanual_follower_left/right.json if missing. It never re-zeroes motors (§21).
4. Minis: check_mini_calibration.py; if the files don’t match the Minis (or you ask), a guided lerobot-calibrate --teleop.type=bi_openarm_mini … (left Mini first, then right), then fix_mini_gripper_range.py, then the check again.
5. Sanity table: one no-torque compare_mini_follower.py --once.

23.2 Calibrating the Minis correctly

LeRobot’s Mini calibration sets each servo’s zero to wherever the Mini is when you press Enter (half-turn homing). A Mini zeroed in the wrong pose makes the follower go to the wrong pose. This is the most likely cause of the zeus snap incident.

At the zero-pose prompt, for each Mini:

• arm hanging straight down, gripper closed;
• wrist joints 5 and 7 visually matching the follower’s hanging wrist;
• joint 5 in the middle of its travel, not against a stop. On zeus one Mini was once zeroed 90° off, against its stop;
• hold it still, then press Enter.

The gripper “close / open” prompts of lerobot-calibrate are unreliable: a buffered Enter makes it record min == max (Invalid calibration for motor 'gripper': min and max are equal). fix_mini_gripper_range.py re-measures the range from a live readout. Open only as far as your finger opens it in use, not to the servo’s end, because the left Mini’s gripper wraps past 0/4095 there. The setup script runs it automatically.

Afterwards, at every teleop start, LeRobot asks “ENTER to use existing calibration … or type ‘c’”: press ENTER.

Verify before teleop (no torque; everything hanging, grippers closed → all values ≈ 0):

python ~/Documents/openarm_lerobot/compare_mini_follower.py

Then move one Mini joint and the same follower joint by hand in the same direction: both numbers must change with the same sign and by roughly the same amount. Hanging, the followers read within a few degrees of 0 (their motor zero is fine). A Mini reading far from 0 while hanging means that Mini needs recalibrating.

23.3 Teleop: safe_teleop.py (use this, not lerobot-teleoperate)

python ~/Documents/openarm_lerobot/safe_teleop.py --arms right --max-vel 30   # first sessions: one arm, slow
python ~/Documents/openarm_lerobot/safe_teleop.py                             # both arms

Why not stock lerobot-teleoperate: on connect it enables the followers at full stiffness (kp 240) and the first command sends them straight to the Mini’s reading, with no check that it’s plausible. A motor that doesn’t reply is silently treated as being at 0°. max_relative_target only limits each step relative to the current position (≈ 150°/s at 30 fps), so a wrong target is still reached, just slightly slower. It also makes low-stiffness joints “stick” (the wrist stalls when its steady-state error exceeds the clamp).

What safe_teleop.py does instead:

Guard	Behaviour
Pre-flight	Refuses to start unless the Mini files match the Minis and all 16 follower motors answer.
Alignment gate	Follower torque stays OFF and a live table shows Mini vs follower per joint. Torque only comes on after every joint has been within 10° (gripper 15°) for 1 s. You move the Mini to the arm, never the reverse.
Soft start	Stiffness ramps 0 → 100 % over 2 s.
Speed limit	The command changes by at most --max-vel deg/s per joint (default 60, gripper ×2). Limited on the command, so joints don’t stick.
Hold on fault	Joint > 25° from its command for 0.5 s, a motor not answering, or a Mini reading jumping > 45° in one frame (persisting): that arm holds its current position (doesn’t go limp) and resumes once you re-align the Mini.
Exit	Ctrl+C → arms hold. Support/lower them, then Enter → torque off. A second Ctrl+C = torque off immediately (arms fall).

Options: --arms left|right|both, --fps 30, --align-tol 10, --ramp-s 2, --max-vel 60, --track-err 25, --follower-id my_bimanual_follower, --mini-id my_mini.

The control logic was checked in a simulation with fake arms (gate stays closed when misaligned, kp starts at 0, ≤ 2° per cycle at 60°/s, blocked joint and bogus 90° Mini jump both → HOLD), not yet on the real robot. Do the first run with one arm, low --max-vel, someone at the power switch.

23.4 Stock lerobot-teleoperate (for reference, not recommended)

If you do use it, use the zeus mapping, keep max_relative_target, disable Rerun (it blocked the loop for 12.8 s once, causing a freeze and then a catch-up jerk), and align the Minis with compare_mini_follower.py first:

lerobot-teleoperate \
  --robot.type=bi_openarm_follower \
  --robot.left_arm_config.port=can0  --robot.left_arm_config.side=left  --robot.left_arm_config.max_relative_target=5.0 \
  --robot.right_arm_config.port=can1 --robot.right_arm_config.side=right --robot.right_arm_config.max_relative_target=5.0 \
  --robot.id=my_bimanual_follower \
  --teleop.type=bi_openarm_mini \
  --teleop.left_arm_config.port=/dev/serial/by-id/usb-1a86_USB_Single_Serial_5876043720-if00  --teleop.left_arm_config.side=left \
  --teleop.right_arm_config.port=/dev/serial/by-id/usb-1a86_USB_Single_Serial_5876043592-if00 --teleop.right_arm_config.side=right \
  --teleop.id=my_mini \
  --fps=30 --display_data=false

Stop it with Ctrl+C (torque off, so the arms fall; lower the Minis first), never Ctrl+Z. Ctrl+Z only suspends it: it keeps the buses open and resumes commanding on fg. If it happens, kill -9 <pid>.

23.5 Cameras

lerobot-find-cameras opencv          # lists indices/paths and saves test frames to outputs/
lerobot-find-cameras realsense       # needs: pip install -e ".[intelrealsense]"

Add cameras to the robot (they become part of the observation and are recorded as videos):

--robot.cameras='{top: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30},
                  left_wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}}'

For bimanual setups, --robot.cameras is opened by the left arm but keys stay unprefixed. Per-arm cameras can go in --robot.left_arm_config.cameras / --robot.right_arm_config.cameras. Prefer /dev/v4l/by-id/… paths over indices.

23.6 Python API

See ~/Documents/openarm_lerobot/safe_teleop.py for a complete, guarded example built on LeRobot’s classes (OpenArmMini, OpenArmFollower, robot.send_action(..., custom_kp=...)). The key points if you write your own loop:

• OpenArmFollower.connect() enables torque immediately; to control the start, use robot.bus.connect(handshake=False) (the handshake sends ENABLE) and call robot.bus.enable_torque() yourself after checking alignment.
• bus.sync_read_all_states() returns the last known state for motors that didn’t reply (initially 0).
• OpenArmFollowerConfig(use_velocity_and_torque=True) adds joint_i.vel (deg/s) and joint_i.torque (Nm) to observations.

Previous22. Bus and port layoutNext24. Recording, training and running policies