Abstract
Developing generalizable robotic policies that balance inference efficiency, manipulation accuracy, and robustness remains a formidable challenge. Existing vision-language-action models demand prohibitive data scales, while keyframe-based approaches struggle to reconcile the expressivity of generative models with the latency of iterative sampling. To address this trilemma, we present Flow2Act, a unified framework that integrates agglomerative perception with a deterministic one-step generative policy. Unlike prior methods relying on separate semantic encoders or iterative diffusion processes, our approach introduces three key innovations. First, we employ an agglomerative multi-teacher visual backbone that distills complementary strengths from diverse foundation models, capturing semantics, spatial structure, and segmentation to yield robust representations without task-specific pretraining. Second, we propose a conditional MeanFlow policy that parameterizes the interval-averaged velocity field. This formulation enables genuine single-step action generation, eliminating the discretization errors and computational overhead inherent in ODE-based flow matching. Third, we devise a curriculum region-aware mechanism via a Spatial-Grounded State Space Model, which progressively shifts attention from global flow stability to fine-grained contact precision. We evaluate Flow2Act on challenging simulation benchmarks and real-world robotic tasks, demonstrating significant gains in policy performance, robustness to environmental perturbations, and cross-task real-world applicability.
Overview of Flow2Act
Architectural comparison of perceptual encoders and policy networks. Left: 2D-only perception lacks critical 3D reasoning for manipulation, while 3D-enhanced methods (e.g., RGB-D, point cloud and voxel) are constrained to global scene understanding; our region-aware schedule enables fine-grained, adaptive spatial perception. Right: Discrete action policies lack precision for manipulation, while diffusion and flow-based methods both learn invertible transformations to map prior Gaussian noise samples into the target action distribution. Diffusion-based approaches require iterative denoising steps to gradually refine noise into valid actions, whereas our MeanFlow framework directly learns a single-step mapping from noise to continuous actions. This unified policy network achieves higher computational efficiency and improved manipulation accuracy.
Overview of the Flow2Act framework. The architecture unifies a multi-view perception module, a Spatial-Grounded State Space Duality, and a one-step policy head. Processing language instruction l and multiview RGB-D observations, the framework derives dense semantic encodings via an agglomerative vision foundation model alongside spatial tokens from a point encoder using curriculum region aware learning with a dynamic radius schedule. Stacked parallel Mamba blocks fuse these multimodal tokens to capture spatiotemporal dependencies. Subsequently, the one-step policy employs average velocity flow matching to synthesize precise robot actions from initial noise.
Results across Benchmarks
Evaluation results of Multi-Task on RLBench.
Evaluation results of High-precision on RLBench.
Evaluation results of Few-Shot on RLBench.
Evaluation results on COLOSSEUM.
Evaluation results of ablation.
Results on RLBench
Real World Deployment
Failure Cases
We analyze all 95 failed trials in the real-world Generalization Setting. Three primary failure causes are identified: severe occlusion during local refinement (43/95, 45.3%), depth sensor artifact or missing point cloud (28/95, 29.5%), and kinematic singularity or planner-level reachability issue (24/95, 25.3%).
Grasp Black Ring (Failed)
Novel View Generalization (Failed)
Stack Blocks Novel View (Failed)
Stack 3 Blocks (Failed)
Manipulation Failure
Grab Banana Novel View (Failed)
Real-World Failure Demo
BibTeX
@article{wangflow2act2025,
title={Integrating Agglomerative Perception with One-step Action Generation for Robotic Manipulation},
author={Sen Wang, Le Wang, Hongcheng Huo, Sanping Zhou, Kun Xia, Gang Hua},
year={2025},
}