EN ·
🌏 中文

arXiv Highlights: Latest Research Trends in Vision-Language Models and Embodied AI (May 2025)

arXiv Highlights: Latest Research Trends in Vision-Language Models and Embodied AI (May 2025)

Beichen Wen et al. present 3D Scene Generation: A Survey, a comprehensive review of 3D scene generation technologies. The paper systematically categorizes state-of-the-art methods into four paradigms: procedural generation, neural-based 3D generation, image-based generation, and video-based generation. It analyzes the technical foundations, strengths, and limitations of these methods, while discussing future potential in high-fidelity, physics-aware, and interactive generation.

Wenqi Wang et al. introduce SITE: towards Spatial Intelligence Thorough Evaluation, a benchmark designed for assessing spatial intelligence. Utilizing multiple-choice visual question answering, the dataset evaluates large-scale vision-language models across diverse modalities (images, videos) and spatial factors (e.g., spatial visualization, orientation). Experimental results indicate that current models still trail human experts in basic spatial orientation, and spatial reasoning ability correlates positively with performance in embodied AI tasks.

Zhaohan Feng et al. review the landscape of Multi-agent Embodied AI: Advances and Future Directions. The authors note that while single-agent systems have made significant progress in closed environments, complex real-world scenarios necessitate multi-agent collaboration and real-time learning. The paper analyzes current limitations and proposes future development paths for multi-agent embodied AI in dynamic, open environments.

Ranjan Sapkota et al. provide a comprehensive survey in Vision-Language-Action Models: Concepts, Progress, Applications and Challenges. The article summarizes progress in integrating perception, natural language understanding, and embodied action, exploring applications in humanoid robotics and autonomous driving. It also proposes solutions to challenges such as real-time control, multimodal action representation, and system scalability.

Liam Boyle et al. present RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning, a method for achieving embodied intelligence in low-parameter LLMs. Experiments show that small-scale LLMs, through closed-loop interaction with the environment, can outperform larger models in tasks like autonomous driving, demonstrating the feasibility of deploying compact LLMs in robotics.

Huangyue Yu et al. introduce MetaScenes: Towards Automated Replica Creation for Real-world 3D Scenes, a large-scale, simulatable 3D scene dataset derived from real-world scans, featuring 15,366 objects across 831 fine-grained categories. The study also presents Scan2Sim, a multimodal alignment model that automates high-quality asset replacement, reducing reliance on artist-driven design. This work was published at CVPR 2025.

Irene Wang et al. propose Carbon Aware Transformers Through Joint Model-Hardware Optimization, a framework named CATransformers. By jointly optimizing model and hardware architectures, it reduces the total carbon footprint of machine learning systems. Applied to multimodal CLIP models, the resulting CarbonCLIP family achieves a 17% reduction in carbon emissions while maintaining accuracy and latency.

Roberto Bigazzi explores the agent creation process in Autonomous Embodied Agents: When Robotics Meets Deep Learning Reasoning. The research covers the full lifecycle from concept to implementation and deployment, providing a critical reference for embodied AI research via large-scale training in simulated environments.

Wayne Wu et al. present Towards Autonomous Micromobility through Scalable Urban Simulation, a high-performance robotics learning platform called URBAN-SIM. It enhances the diversity, realism, and efficiency of robot learning through hierarchical city generation, interactive dynamics, and asynchronous scenario sampling. This work was published at CVPR 2025.

Lang Feng et al. propose Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning, an online fine-tuning method for VLM agents named CoSo. By dynamically assessing the causal impact of individual tokens on post-processing actions via counterfactual reasoning, CoSo significantly improves exploration efficiency. This work was published at ICML 2025.

Ruochen Jiao et al. present Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systems, a backdoor attack framework (BALD) targeting LLM-based embodied decision-making. The study explores three mechanisms: word injection, scene manipulation, and knowledge injection. Experiments demonstrate high efficiency and stealth in tasks such as autonomous driving and home robotics. This work was accepted at ICLR 2025.

Jiwen Yu et al. provide a survey on A Survey of Interactive Generative Video, defining it as technology that combines generative capabilities with interactive functionality, suitable for gaming, embodied AI, and autonomous driving. The study proposes an ideal IGV system framework comprising five modules: generation, control, memory, dynamics, and intelligence.

Zhuoqi Zeng et al. introduce TinyMA-IEI-PPO: Exploration Incentive-Driven Multi-Agent DRL with Self-Adaptive Pruning for Vehicular Embodied AI Agent Twins Migration. This framework utilizes a Multi-Leader Multi-Follower (MLMF) Stackelberg game incentive mechanism and a lightweight MADRL algorithm to optimize Vehicular Embodied AI Agent Twins (VEAAT) migration.

Seonghee Lee et al. present IRL Dittos: Embodied Multimodal AI Agent Interactions in Open Spaces, an AI-driven embodied agent designed to represent remote colleagues in shared office spaces, facilitating real-time social interaction.

Yibin Yan et al. propose Learning Streaming Video Representation via Multitask Training, introducing StreamFormer, a backbone network for efficient streaming video processing. By integrating causal temporal attention into pre-trained visual Transformers, it unifies diverse spatiotemporal video understanding tasks.

Run Luo et al. introduce VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning. This framework builds a vision concept model without expensive concept-level annotations, significantly reducing computational costs (e.g., 85% reduction in FLOPs for LLaVA-1.5-7B).

Rajeev Gupta et al. propose Personalized Artificial General Intelligence (AGI) via Neuroscience-Inspired Continuous Learning Systems, an architecture integrating brain-like learning mechanisms to support continuous learning and personalization on resource-constrained edge devices.

Yiren Xu et al. explore the influence of AI on Balancing Creativity and Automation: The Influence of AI on Modern Film Production and Dissemination. The study suggests positioning AI as an “embodied tool” rather than an autonomous partner to preserve human authorship.

Li Jin et al. study Embodied World Models Emerge from Navigational Task in Open-Ended Environments, demonstrating that continuous sensorimotor interaction is sufficient to generate compact embodied world models.

Zishen Wan et al. perform Generative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability. The study reveals challenges such as planning/communication latency and memory inconsistency, proposing optimization strategies. Published at IEEE ISPASS 2025.

Tianliang Yao et al. present Advancing Embodied Intelligence in Robotic-Assisted Endovascular Procedures: A Systematic Review of AI Solutions, exploring how data-driven methods and machine learning enhance intelligent perception and real-time control in surgery.

Pei Lin et al. propose PP-Tac: Paper Picking Using Tactile Feedback in Dexterous Robotic Hands. This system uses high-resolution omnidirectional tactile sensors for real-time slip detection and friction control, achieving an 87.5% success rate. Accepted at RSS 2025.

Yun Li et al. introduce STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?, a benchmark evaluating MLLMs on tasks like distance estimation and motion analysis in robotic and autonomous applications.

Haotian Xu et al. propose GeoNav: Empowering MLLMs with Explicit Geospatial Reasoning Abilities for Language-Goal Aerial Navigation, featuring dynamic construction of global cognitive maps and local scene graphs.

Haoming Li et al. present PLANET: A Collection of Benchmarks for Evaluating LLMs’ Planning Capabilities, covering domains from robotic environments to daily task automation.

Steeven Janny et al. analyze Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, exploring emergence of reasoning capabilities in end-to-end trained robots. Published at CVPR 2025.

Jirui Yang et al. propose Concept Enhancement Engineering: A Lightweight and Efficient Robust Defense Against Jailbreak Attacks in Embodied AI, a framework that uses representation engineering to guide internal activations and harden safety.

Jiaxin Lu et al. present HUMOTO: A 4D Dataset of Mocap Human Object Interactions, a high-precision dataset featuring 736 sequences of human-object interactions for animation and robotics.

Haiyong Yu et al. introduce Efficient Task-specific Conditional Diffusion Policies: Shortcut Model Acceleration and SO(3) Optimization, achieving nearly 5x speedup in diffusion inference. Accepted at CVPR 2025 Workshop on 2nd MEIS.

Key Research Directions

  1. 3D Scene Generation & Simulation

    • Focus: Generating high-quality 3D environments via procedural or neural methods for embodied AI.
    • Key papers: 3D Scene Generation: A Survey, MetaScenes.

  2. Multi-agent Embodied AI

    • Focus: Collaboration and real-time learning in open, dynamic environments.
    • Key paper: Multi-agent Embodied AI: Advances and Future Directions.

  3. Vision-Language-Action (VLA) Models

    • Focus: Integrating perception, language, and motor control for robotics.
    • Key paper: Vision-Language-Action Models: Concepts, Progress, Applications and Challenges.

  4. Spatial Intelligence & Navigation

    • Focus: Enhancing spatial reasoning, positioning, and geospatial navigation.
    • Key papers: SITE, GeoNav.

  5. Security & Defense

    • Focus: Protecting embodied systems against backdoor and jailbreak attacks.
    • Key papers: Can We Trust Embodied Agents?, Concept Enhancement Engineering.

  6. Generative AI in Embodied Systems

    • Focus: Addressing performance, latency, and scalability in embedded systems.
    • Key paper: Generative AI in Embodied Systems.

  7. Robotic-Assisted Surgery

    • Focus: Data-driven perception and real-time control in medical robotics.
    • Key paper: Advancing Embodied Intelligence in Robotic-Assisted Endovascular Procedures.