-
Training Software Engineering Agents and Verifiers with SWE-Gym
Jiayi Pan*, Xingyao Wang*, Graham Neubig, Navdeep Jaitly, Ji Heng, Alane Suhr^, Yizhe Zhang^
Preprint 2024.
We present SWE-Gym, the first publicly-available environment for training real-world software engineering agents. We use it to train strong LM agents that achieve state-of-the-art open results on SWE-Bench, with early, promising scaling characteristics as we increase training and inference-time compute.
-
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Xingyao Wang et al., OpenHands Community.
Preprint 2024 / ✨ 38K+ Github Stars.
We introduce OpenHands, a platform for developing AI agents that interact with the digital world. OpenHands is a community project from over 180 contributors and 30K+ stars.
-
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Hao Bai*, Yifei Zhou*, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar.
NeurIPS 2024 / 🔍 Covered in State of AI Annual Report 2024.
We develop reinforcement learning techniques to post-train device-control language agents. Our 2B VLM, when post-trained with an autonomous evaluator, improves its success rate from 17% to 67% on Android device-control tasks.
-
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Yuexiang Zhai, Hao Bai*, Zipeng Lin*, Jiayi Pan*, Shengbang Tong*, Yifei Zhou*, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, Sergey Levine.
NeurIPS 2024.
We provide infrastructure and environment for training VLMs with RL on decision-making tasks. We show RL training enables our 7B model to outperform GPT-4V on these tasks. Additionally, we show the intriguing effectiveness of CoT reasoning for performance improvement
-
Autonomous Evaluation and Refinement of Digital Agents
Jiayi Pan, Yichi Zhang, Nickolas Tomlin, Yifei Zhou, Sergey Levine, Alane Suhr.
COLM 2024 / ⭐️ MAR Workshop @ CVPR 2024 Best Paper.
We design model-based evaluators to both evaluate and autonomously improve the performance of digital agents. We show that these open-ended evaluators can significantly improve agents' performance, through either fine-tuning or inference-time guidance, without any extra supervision.
-
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, Aviral Kumar.
ICML 2024.
We present ArCHer, a new framework of multi-turn RL algorithms for training LM agents. It preserves the flexibility of mainstream single-turn LM RL methods like PPO, while effectively handling multiple turns, long horizons, and delayed rewards.
-
Inversion-Free Image Editing with Natural Language
Sihan Xu*, Yidong Huang*, Jiayi Pan, Ziqiao Ma, Joyce Chai.
CVPR 2024.
We present an inversion-free editing (InfEdit) method which enables consistent natural language guided image editing. InfEdit excels in complex editing tasks and is ~10X faster than prior methods.
-
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?
Yichi Zhang, Jiayi Pan, Yuchen Zhou, Rui Pan, Joyce Chai.
EMNLP 2023 / 🔍 Covered in Scientific American.
Do Vision-Language Models, an emergent human-computer interface, experience visual illusions similarly to humans, or do they accurately depict reality? We created GVIL dataset to study this. Among other findings, we discover that larger models align more closely with human perception.
-
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models
Ziqiao Ma*, Jiayi Pan*, Joyce Chai.
⭐️ ACL 2023 Outstanding Paper.
We study grounding and bootstrapping in open-world language learning through Grounded Open Vocabulary Acquisition. Our visually-grounded language model, OctoBERT, excels in learning grounded words quickly and robustly.
-
SEAGULL: An Embodied Agent for Instruction Following through Situated Dialog
Team SEAGULL at UMich, Perception Lead.
🏆 1st Place in the inaugural Alexa Prize SimBot Challenge 2023.
SEAGULL is an interactive embodied agent that completes complex tasks through natural language dialog in the Arena simulation
environment. The agent is designed to be efficient, user-friendly, and continuously improving.
-
Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification
Jiayi Pan, Glen Chou, Dmitry Berenson.
ICRA 2023.
We present a learning-based approach to translate from natural language commands to LTL specifications with only a handful of labeled data. It enables few-shot learning of LTL translators while achieving state-of-the-art performance.
-
DANLI: Deliberative Agent for Following Natural Language Instructions
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Peter Yu, Yuwei Bao, Joyce Chai.
EMNLP 2022, Oral.
We introduce DANLI, a neural-symbolic agent that proactively reasons and plans according to its past experiences. DANLI achieves a 70% improvement on the challenging TEACh benchmark while improving transparency in its behaviors.
