Android · BLE · Modbus TCP · Pose estimation

Jog a robot arm from your phone. Or just wave at it.

Jog The Robot is a Kotlin / Jetpack Compose app that moves a 5-axis igus robolink-style arm. Hold a button to jog an axis, or stand in front of the camera and let an on-device YOLO pose model turn your arm positions into jog commands.

Watch it move See the hardware View the code

Animated Jog screen: selecting a joint and jogging it with Left and Right

What this project is

A complete phone-to-robot control chain: the Android app, the ESP32 firmware that bridges Bluetooth to the PLC, and the PLC logic that drives the arm.

Jog

Pick an axis (A1 to A5), then hold Left or Right. The command repeats every 100 ms while you hold and a stop is sent the moment you let go.

Gesture Jog

The front camera feeds YOLO26n-pose (ONNX Runtime, CPU or NNAPI). Seventeen body keypoints become axis selection and jog direction, no touch needed.

Safe by default

A 300 ms deadman on the ESP32, a heartbeat watchdog to the PLC, and jog buttons that stay disabled unless the PLC is online and ready.

How it works

Every jog travels through four hops. Status flows back the other way, so the app always shows what the PLC is actually doing.

Android app Kotlin · Compose · Hilt

BLE · Nordic UART → ← status lines

ESP32-S3-ETH BLE server + Modbus server

Modbus TCP · Ethernet → ← state + heartbeat

B&R X20CP0484 PLC state machine

Drives →

igus robot 5 axes

Connect

The app scans for 15 s and flags devices advertising the Nordic UART service. Pick JogTheRobot; it is saved for next time. On connect the app requests an MTU of 185 and subscribes to notifications.

Power on and home

The status card shows the PLC state (Power off, Powered not homed, Ready…). Flip the Power switch (P1), then tap Home (H). Jogging unlocks once the robot is Ready.

Jog

Tap a joint on the arm drawing or an axis button, then hold Left or Right. The app sends A2:R every 100 ms, the ESP32 puts axis and direction into Modbus input registers, and the PLC moves the arm.

Release to stop

Letting go sends S. If the phone goes quiet for 300 ms or disconnects, the ESP32 stops the jog on its own.

Gesture Jog sequence

Each camera frame is letterboxed to 640×640 and run through the pose model. Gestures follow a fixed order: neutral, then an axis pose, then a turn pose to jog.

Pose	Command	Effect
Both arms hanging straight down	`NEUTRAL`	Arms the selector and clears the axis
Arms crossed above the head (X)	`STOP`	Select A1
T-pose, both arms horizontal	`START`	Select A2
Both hands up in a V above the head	`FORWARD`	Select A3
Both arms down at ~45°, hands apart	`BACKWARD`	Select A4
Left arm horizontal, right arm down	`TURN_LEFT`	Jog selected axis Left
Right arm horizontal, left arm down	`TURN_RIGHT`	Jog selected axis Right
No person in frame	`NO_PERSON`	Clear the axis and stop

While a turn pose is held the jog repeats every 100 ms. If no frame shows it for 400 ms, the loop ends and sends S, so inference needs to stay above about 3 FPS. On a Nothing Phone (A015) with NNAPI FP32 it runs at about 6 FPS.

Gesture Jog screen detecting TURN_LEFT, robot state Jogging — **`TURN_LEFT`** Left arm out, right arm down: jogs the selected axis Left.

Gesture Jog screen detecting TURN_RIGHT, robot state Jogging — **`TURN_RIGHT`** Right arm out, left arm down: jogs the selected axis Right.

Gesture Jog screen detecting STOP with arms crossed above the head — **`STOP` (arms crossed)** Arms crossed above the head: selects axis A1.

The app

A tour of the app on a Nothing Phone (A015), from connecting to jogging. The Jog and Axis selected screens were taken with the PLC offline; Homing, Ready and Jogging come from a live session with the robot.

Home screen showing connection state — **Home** Connection state and a shortcut to connect to a saved or nearby device.

Navigation drawer with Home, Jog, Gesture Jog, saved devices and Disconnect — **Navigation drawer** Home, Jog, Gesture Jog, saved devices and Disconnect.

Jog screen with the PLC offline — **Jog, PLC offline** Status card with Power, Stop and Home, the arm drawing, axis buttons and hold-to-jog. Buttons stay disabled while the PLC is offline.

Jog screen with axis A2 selected — **Axis selected** Tap a joint to select it. The joint lights up with its current angle.

Jog screen while the robot is homing — **Homing** After Power on, Home (H) drives every axis to its reference. Stop is active while homing.

Jog screen with the robot powered and Ready — **Ready** Powered and homed. Jogging unlocks.

Jog screen while jogging axis A1 — **Jogging** Holding Right on A1 (Base). The app sends A1:R every 100 ms.

Gesture Jog screen with pose skeleton, detecting TURN_LEFT — **Gesture Jog** Camera preview with the pose skeleton, detected command, robot state and FPS counters. More gestures under How it works.

See the robot move

Phone screen on the left, robot on the right. First touch Jog, then Gesture Jog.

Jog The Jog screen and the arm side by side.

Gesture Jog The phone reads body poses from the front camera (left) and the arm moves to match (right): select an axis with a pose, then jog it Left or Right with one arm out. 1 min 43 s.

The hardware

A B&R X20 PLC drives a 5-axis igus arm. The ESP32-S3-ETH sits between the phone and the PLC, talking BLE on one side and Modbus TCP over Ethernet on the other. Tap a photo to open it full size.

PLC and bridge

B&R X20 PLC rack with I/O modules and wiring — **B&R X20 PLC** The X20 rack with its I/O modules, wired into the robot cabinet.

Close-up of the X20CP0484 CPU module — **X20CP0484 CPU** Close-up of the CPU that runs the jog state machine and polls the ESP32.

ESP32-S3-ETH board connected with an Ethernet cable — **ESP32-S3-ETH** The BLE-to-Modbus bridge, connected to the PLC network by Ethernet.

The igus 5-axis arm

igus 5-axis robot arm, side view — **Side view** The arm on its bench.

igus 5-axis robot arm from the left — **Angle 2** From the left, base and shoulder in view.

igus 5-axis robot arm, another angle from the left — **Angle 3** Elbow and wrist folded toward the bench.

igus 5-axis robot arm from the front — **Angle 4** From the front, facing the windows.

More demos

Short clips of both control modes. Videos start muted; use the player controls to unmute.

Gesture Jog in front of the robot The operator holds arm poses in front of the phone, and the arm jogs to follow.

Jog screen recording A full screen recording of the Jog screen while driving the arm.

Under the hood

Plain ASCII lines over BLE, ending in \n. Both sides buffer until a newline because BLE packets can split or join lines.

Phone → ESP32

Command	Meaning
`A1:L` … `A5:L`	Jog axis negative, every 100 ms while held
`A1:R` … `A5:R`	Jog axis positive, every 100 ms while held
`S`	Stop jogging
`P1` / `P0`	Robot power on / off
`H`	Home the robot
`X`	Stop a running program, move or homing

ESP32 → phone

Line	Meaning
`ST:20,1`	Robot state code and PLC online flag, sent every second and on change
`OK A2:R`	Acknowledges a new jog direction
`OK S`	Acknowledges a stop

BLE service

Role	UUID
Service	`6E400001-B5A3-F393-E0A9-E50E24DCCA9E`
RX (write)	`6E400002-B5A3-F393-E0A9-E50E24DCCA9E`
TX (notify)	`6E400003-B5A3-F393-E0A9-E50E24DCCA9E`

Robot states

0 Power off 5 Powering on 9 Not homed 10 Homing 20 Ready 30 Program 40 Jogging 50 Moving 99 Error

Jogging is allowed only in Ready and Jogging, and only while the PLC is online.

Built with

Kotlin 2.2
Jetpack Compose + Material 3
Hilt
CameraX
ONNX Runtime 1.18
YOLO26n-pose
Android BLE GATT
ESP32-S3 + W5500
modbus-esp8266
B&R Structured Text