From Pixels to Actions - Towards Native Vision-Language-Action Policies
기존 Vision-Language-Action 모델의 시각 인코더 기반 압축 방식이 조작 작업에 필요한 미세한 시각 정보를 손실시킨다는 문제를 지적합니다. 이를 해결하기 위해 시각 인코더 없이 패치 컨볼루션을 통해 언어와 시각 토큰이 백본에서 함께 진화하는 NEO-VLA 프레임워크를 제안합니다. 실험 결과, 이러한 Native 아키텍처가 고정된 시각 엔드포인트를 사용하는 기존 방식보다 제어된 비교에서 더 우수한 성능을 보임을 확인했습니다.
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additio