Robotics

Vision-Language-Action Models (VLA) for robotics are trained by combining large language models with vision encoders and then fine-tuning them on various robot datasets; this allows generalization to new instructions, unseen objects, and distribution shifts. However, various real-world robot datasets mostly require human control, which makes scaling difficult. On the other hand, Internet video data offers…

Visual understanding is the abstracting of high-dimensional visual signals like images and videos. Many problems are involved in this process, ranging from depth prediction and vision-language correspondence to classification and object grounding, which include tasks defined along spatial and temporal axes and tasks defined along coarse to fine granularity, like object grounding. In light of…

Technological advancements in sensors, AI, and processing power have propelled robot navigation to new heights in the last several decades. To take robotics to the next level and make them a regular part of our lives, many studies suggest transferring the natural language space of ObjNav and VLN to the multimodal space so the robot…

In robotics, understanding the position and movement of a sensor suite within its environment is crucial. Traditional methods, called Simultaneous Localization and Mapping (SLAM), often face challenges with unsynchronized sensor data and require complex computations. These methods must estimate the position at discrete time intervals, making it difficult to handle data from various sensors that…

The field of deep reinforcement learning (DRL) is expanding the capabilities of robotic control. However, there has been a growing trend of increasing algorithm complexity. As a result, the latest algorithms need many implementation details to perform well on different levels, causing issues with reproducibility. Moreover, even state-of-the-art DRL models have simple problems, like the…

A major weakness of current robotic manipulation policies is their inability to generalize beyond their training data. While these policies, trained for specific skills or language instructions, can adapt to new conditions like different object positions or lighting, they often fail when faced with scene distractors or new objects, and need help to follow unseen…

Learning in simulation and applying the learned policy to the real world is a potential approach to enable generalist robots, and solve complex decision-making tasks. However, the challenge to this approach is to address simulation-to-reality (sim-to-real) gaps. Also, a huge amount of data is needed while learning to solve these tasks, and the load of…

The practical application of robotic technology in automatic assembly processes holds immense value. However, traditional robotic systems have struggled to adapt to the demands of production environments characterized by high-mix, low-volume manufacturing. Robotic learning presents a potential solution to this challenge by enabling robots to acquire assembly skills through demonstration rather than scripted trajectories, thus…

The exploration of artificial intelligence within dynamic 3D environments has emerged as a critical area of research, aiming to bridge the gap between static AI applications and their real-world usability. Researchers at Google DeepMind have pioneered this realm, developing sophisticated agents capable of interpreting and acting on complex instructions within various simulated settings. This new…

Reinforcement learning has exhibited notable empirical success in approximating solutions to the Hamilton-Jacobi-Bellman (HJB) equation, consequently generating highly dynamic controllers. However, the inability to bind the suboptimality of resulting controllers or the approximation quality of the true cost-to-go function due to finite sampling and function approximators has limited the broader application of such methods.  Consequently,…