Leveraging Scene Embeddings
for Gradient-Based Motion Planning in Latent Space

2023 IEEE International Conference on Robotics and Automation (ICRA)

Best Poster Award at 2023 UK Manipulation workshop

Jun Yamada*

Oxford A2I

Chia-Man Hung*

Oxford A2I, DRS

Jack Collins

Oxford A2I

Ioannis Havoutis

Oxford DRS

Ingmar Posner

Oxford A2I

(*Equal contribution)


Motion planning framed as optimisation in structured latent spaces has recently emerged as competitive with traditional methods in terms of planning success while significantly outperforming them in terms of computational speed. However, the real-world applicability of recent work in this domain remains limited by the need to express obstacle information directly in state-space, involving simple geometric primitives. In this work we address this challenge by leveraging learned scene embeddings together with a generative model of the robot manipulator to drive the optimisation process. In addition we introduce an approach for efficient collision checking which directly regularises the optimisation undertaken for planning. Using simulated as well as real-world experiments, we demonstrate that our approach, AMP-LS, is able to successfully plan in novel, complex scenes while outperforming competitive traditional baselines in terms of computation speed by an order of magnitude. We show that the resulting system is fast enough to enable closed-loop planning in real-world dynamic scenes.


We introduce a method remarkably extending Latent Space Path Planning (LSPP) and name it Activation Maximisation Planning in Latent Space (AMP-LS). Similar to LSPP, AMP-LS leverages a variational autoencoder (VAE) to learn a structured latent space to generate kinematically feasible joint trajectories. While a collision predictor in the prior work relies on state-based obstacle representations, our collision predictor leverages scene embeddings obtained from SceneCollisionNet to readily achieve zero-shot transfer to unseen environments. Further, we present an approach for explicit collision checking to directly regularise the optimisation to plan collision-free trajectories.


Experimental results

Open-loop planning for reaching static targets in simulation

Traj16 AMP-LS

AMP-LS (ours)

Traj16 RRTConnect

RRTConnect — Collision

Traj16 LazyPRM*



CHOMP — Fails to plan

Traj23 AMP-LS

AMP-LS (ours)

Traj23 RRTConnect


Traj23 LazyPRM*

LazyPRM* — Collision


CHOMP — Fails to plan

Traj36 AMP-LS

AMP-LS (ours)

Traj36 RRTConnect


Traj36 LazyPRM*

LazyPRM* — Collision


CHOMP — Fails to plan

Traj70 AMP-LS

AMP-LS (ours)

Traj70 RRTConnect


Traj70 LazyPRM*

LazyPRM* — Collision


CHOMP — Fails to plan

Open-loop planning in a complex scene in real


Closed-loop planning for moving obstacles and target in simulation

Sim conveyor 03
Sim conveyor 09

Closed-loop planning for moving obstacles and target in real

Real conveyor track

Track the QR code

Real conveyor same pos

Maintain the same pose


  title={Leveraging scene embeddings for gradient-based motion Planning in latent space},
  author={Yamada, Jun and Hung, Chia-Man and Collins, Jack and Havoutis, Ioannis and Posner, Ingmar},
  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},


This work was supported by a UKRI/EPSRC Programme Grant [EP/V000748/1], we would also like to thank the University of Oxford for providing Advanced Research Computing (ARC) facility in carrying out this work (http://dx.doi.org/10.5281/zenodo.22558).