Oliver Groth, Chia-Man Hung, Andrea Vedaldi, Ingmar Posner
Visuomotor control (VMC) is an effective means of achieving basic manipulation tasks such as pushing or pick-and-place from raw images. Conditioning VMC on desired goal states is a promising way of achieving versatile skill primitives. However, common conditioning schemes either rely on task-specific fine tuning - e.g. using one-shot imitation learning (IL) - or on sampling approaches using a forward model of scene dynamics i.e. model-predictive control (MPC), leaving deployability and planning horizon severely limited. In this paper we propose a conditioning scheme which avoids these pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion and the distance to a given target observation. In contrast to related works, this enables our approach to efficiently perform complex manipulation tasks from raw image observations without predefined control primitives or test time demonstrations. We report significant improvements in task success over representative MPC and IL baselines. We also demonstrate our model's generalisation capabilities in challenging, unseen tasks featuring visual noise, cluttered scenes and unseen object geometries.
Chia-Man Hung, Shaohong Zhong, Walter Goodwin, Oiwi Parker Jones, Martin Engelcke, Ioannis Havoutis, Ingmar Posner
We present a novel approach to path planning for robotic manipulators, in which paths are produced via iterative optimisation in the latent space of a generative model of robot poses. Constraints are incorporated through the use of constraint satisfaction classifiers operating on the same space. Optimisation leverages gradients through our learned models that provide a simple way to combine goal reaching objectives with constraint satisfaction, even in the presence of otherwise non-differentiable constraints. Our models are trained in a task-agnostic manner on randomly sampled robot poses. In baseline comparisons against a number of widely used planners, we achieve commensurate performance in terms of task success, planning time and path length, performing successful path planning with obstacle avoidance on a real 7-DoF robot arm.
Chia-Man Hung, Li Sun, Yizhe Wu, Ioannis Havoutis, Ingmar Posner
End-to-end visuomotor control is emerging as a compelling solution for robot manipulation tasks. However, imitation learning-based visuomotor control approaches tend to suffer from a common limitation, lacking the ability to recover from an out-of-distribution state caused by compounding errors. In this paper, instead of using tactile feedback or explicitly detecting the failure through vision, we investigate using the uncertainty of a policy neural network. We propose a novel uncertainty-based approach to detect and recover from failure cases. Our hypothesis is that policy uncertainties can implicitly indicate the potential failures in the visuomotor control task and that robot states with minimum uncertainty are more likely to lead to task success. To recover from high uncertainty cases, the robot monitors its uncertainty along a trajectory and explores possible actions in the state-action space to bring itself to a more certain state. Our experiments verify this hypothesis and show a significant improvement on task success rate: 12% in pushing, 15% in pick-and-reach and 22% in pick-and-place.
Robert Jarolim, Martin Sanner, Chia-Man Hung, Emma Stevenson, Hala Lamdouar, Josh Veitch-Michaelis, Ioanna Bouri, Anna Malanushenko, Elena Provornikova, Vít Růžička, Carlos Urbina-Ortega
Sep 16, 2025·astro-ph.SR·PDF Coronagraphic observations enable direct monitoring of coronal mass ejections (CMEs) through scattered light from free electrons, but determining the 3D plasma distribution from 2D imaging data is challenging due to the optically-thin plasma and the complex image formation processes. We introduce SuNeRF-CME, a framework for 3D tomographic reconstructions of the heliosphere using multi-viewpoint coronagraphic observations. The method leverages Neural Radiance Fields (NeRFs) to estimate the electron density in the heliosphere through a ray-tracing approach, while accounting for the underlying Thomson scattering of image formation. The model is optimized by iteratively fitting the time-dependent observational data. In addition, we apply physical constraints in terms of continuity, propagation direction, and speed of the heliospheric plasma to overcome limitations imposed by the sparse number of viewpoints. We utilize synthetic observations of a CME simulation to fully quantify the model's performance for different viewpoint configurations. The results demonstrate that our method can reliably estimate the CME parameters from only two viewpoints, with a mean velocity error of $3.01\pm1.94\%$ and propagation direction errors of $3.39\pm1.94^\circ$ in latitude and $1.76\pm0.79^\circ$ in longitude. We further show that our approach can achieve a full 3D reconstruction of the simulated CME from two viewpoints, where we correctly model the three-part structure, deformed CME front, and internal plasma variations. Additional viewpoints can be seamlessly integrated, directly enhancing the reconstruction of the plasma distribution in the heliosphere. This study underscores the value of physics-informed methods for reconstructing the heliospheric plasma distribution, paving the way for unraveling the dynamic 3D structure of CMEs and enabling advanced space weather monitoring.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, Shimon Whiteson
In the last few years, deep multi-agent reinforcement learning (RL) has become a highly active area of research. A particularly challenging class of problems in this area is partially observable, cooperative, multi-agent learning, in which teams of agents must learn to coordinate their behaviour while conditioning only on their private observations. This is an attractive research area since such problems are relevant to a large number of real-world systems and are also more amenable to evaluation than general-sum problems. Standardised environments such as the ALE and MuJoCo have allowed single-agent RL to move beyond toy domains, such as grid worlds. However, there is no comparable benchmark for cooperative multi-agent RL. As a result, most papers in this field use one-off toy problems, making it difficult to measure real progress. In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap. SMAC is based on the popular real-time strategy game StarCraft II and focuses on micromanagement challenges where each unit is controlled by an independent agent that must act based on local observations. We offer a diverse set of challenge maps and recommendations for best practices in benchmarking and evaluations. We also open-source a deep multi-agent RL learning framework including state-of-the-art algorithms. We believe that SMAC can provide a standard benchmark environment for years to come. Videos of our best agents for several SMAC scenarios are available at: https://youtu.be/VZ7zmQ_obZ0.
Jun Yamada, Chia-Man Hung, Jack Collins, Ioannis Havoutis, Ingmar Posner
Motion planning framed as optimisation in structured latent spaces has recently emerged as competitive with traditional methods in terms of planning success while significantly outperforming them in terms of computational speed. However, the real-world applicability of recent work in this domain remains limited by the need to express obstacle information directly in state-space, involving simple geometric primitives. In this work we address this challenge by leveraging learned scene embeddings together with a generative model of the robot manipulator to drive the optimisation process. In addition, we introduce an approach for efficient collision checking which directly regularises the optimisation undertaken for planning. Using simulated as well as real-world experiments, we demonstrate that our approach, AMP-LS, is able to successfully plan in novel, complex scenes while outperforming traditional planning baselines in terms of computation speed by an order of magnitude. We show that the resulting system is fast enough to enable closed-loop planning in real-world dynamic scenes.