home/projects
Hands-On Planning · Universitat de Girona

Exploration with a Deep RL Motion Planner

🧭 frontier-based exploration, RRT* global planning and a SAC local planner on a real TurtleBot 2

code slides video

This is my project within the Hands-On Planning course at Universitat de Girona, done together with Elchin Aslanli. We built a ROS 2 global-local planner that lets a robot explore an unknown environment on its own. Some key points:

  • Occupancy-grid mapping from LiDAR, with frontier-based exploration to pick the next goal.
  • RRT* path planning over an inflated map to produce collision-free waypoints.
  • A Soft Actor-Critic (SAC) local planner, trained in MuJoCo, that turns waypoints and LiDAR into velocity commands.
  • Deployed end to end on a real TurtleBot 2 in a maze built in the lab.
Real-robot exploration
TurtleBot 2 exploring the lab maze (left) while the occupancy map, frontiers and RRT* path build up in RViz (right)

System Overview

Conceptual design
The global layer chooses where to go; the learned local layer decides how to drive there
  1. LiDAR scans are fused into an occupancy grid map.
  2. Frontier detection finds free cells bordering unknown space, and frontier selection picks the best safe goal.
  3. RRT* plans a collision-free path to that goal and outputs a list of waypoints.
  4. The SAC local planner takes the waypoints, LiDAR, odometry and IMU, and outputs the control commands $(v, \omega)$.

Each exploration cycle groups the reachable frontier cells into clusters and scores them by information gain − distance + cluster size. The chosen goal is shifted inward toward free space and checked for clearance before it is sent to the planner. Goals that time out or stop making progress are temporarily blacklisted, so the robot does not get stuck chasing an unreachable frontier.

DRL Local Planner

SAC network architecture
A 1D CNN encodes the stacked LiDAR scans; its features are fused with the robot state for the SAC actor and critic

The LiDAR scan and a validity mask are passed through three Conv1d layers (32 → 64 → 128 channels) and max-pooled into a 64-d feature, which is concatenated with the robot-state vector. The policy outputs continuous linear and angular velocities. The reward combines ten terms:

$$ R_{total} = 5r_1 + 3r_2 + 0.5r_3 + 0.3r_4 + r_5 + 0.2r_6 + 0.02r_7 + 1.5r_8 + r_9 + r_{10} $$

covering forward motion, progress toward the next waypoint, heading alignment, velocity smoothness, collision and obstacle-proximity penalties, path following, a turn-or-drive term for large heading errors, a waypoint bonus, and a final goal / timeout reward.

MuJoCo training environment
MuJoCo training environment with randomized obstacles and RRT*-style waypoint paths
Training curves
Over ~250k steps the goal-reached rate converges to ~1.0 and the collision rate drops close to 0

Real-World Experiment

Real-world maze
The maze built at Universitat de Girona for the real-robot test

The full stack runs on the real robot: it selects useful exploration goals and generates safe motion commands in real-world conditions. As future work, we would like a CI/CD-style learning loop in which the robot keeps learning from every real-world run, not only from simulation.

My part of the work was training the RL model, the RRT* planner, and testing the model in MuJoCo. Elchin worked on frontier-based exploration, pose-controller tests, and integrating the RL model on the real robot.

← back to projects