Publications

2024

EXTRACT: Efficient Policy Learning by Extracting Transferrable Robot Skills from Offline Data
Jesse Zhang, Minho Heo, Zuxin Liu, Erdem Biyik, Joseph J. Lim, Yao Liu, Rasool Fakoor
CoRL 2024 [PDF] [Project]
DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset
DROID Dataset Team Collaboration (including Karl Pertsch, Youngwoon Lee, Sungjae Park, Minho Heo, Joseph J. Lim)
RSS 2024 [PDF] [Project]
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Open X-Embodiment Collaboration (including Joseph J. Lim, Karl Pertsch, Minho Heo, Youngwoon Lee)
ICRA 2024 (Best Conference Paper Award) [PDF] [Project] [Dataset]
SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling
Jesse Zhang, Karl Pertsch, Jiahui Zhang, Joseph J. Lim
ICRA 2024 [PDF] [Project]

2023

Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance
Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, Joseph J Lim
CoRL 2023 (Oral) [PDF] [Project] [Code]
FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation
Minho Heo*, Youngwoon Lee*, Doohyun Lee, Joseph J. Lim
RSS 2023 (Best System Paper Award) (IJRR 2024) [PDF] [Project] [Code] [Talk]
PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
Shivin Dass*, Karl Pertsch*, Hejia Zhang, Youngwoon Lee, Joseph J. Lim, Stefanos Nikolaidis
RSS 2023 [PDF] [Project]
Hierarchical Neural Program Synthesis
Linghan Zhong, Ryan Lindeborg, Jesse Zhang, Joseph J. Lim, Shao-Hua Sun
Preprint 2023 [PDF] [Project]

2022

Cross-Domain Transfer via Semantic Skill Imitation
Karl Pertsch, Ruta Desai, Vikash Kumar, Franziska Meier, Joseph J. Lim, Dhruv Batra, Akshara Rai
CoRL 2022 [PDF] [Project] [Code]
Skill-based Model-based Reinforcement Learning
Lucy Xiaoyang Shi, Joseph J. Lim, Youngwoon Lee
CoRL 2022 [PDF] [Project] [Code]
Skill-based Meta-Reinforcement Learning
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, Joseph J. Lim
ICLR 2022 [PDF] [Project] [Code]
Task-Induced Representation Learning
Jun Yamada, Karl Pertsch, Anisha Gunjal, Joseph J. Lim
ICLR 2022 [PDF] [Project] [Code]
Know Your Action Set: Learning Action Relations for Reinforcement Learning
Ayush Jain*, Norio Kosaka*, Kyung-Min Kim, Joseph J. Lim
ICLR 2022 [PDF] [Project] [Code]

2021

Learning to Synthesize Programs as Interpretable and Generalizable Policies
Dweep Trivedi*, Jesse Zhang*, Shao-Hua Sun, Joseph J. Lim
NeurIPS 2021 [PDF] [Project] [Code]
Generalizable Imitation Learning from Observation via Inferring Goal Proximity
Youngwoon Lee*, Andrew Szot*, Shao-Hua Sun, Joseph J. Lim
NeurIPS 2021 [PDF] [Project] [Code]
Demonstration-Guided Reinforcement Learning with Learned Skills
Karl Pertsch, Youngwoon Lee, Yue Wu, Joseph J. Lim
CoRL 2021 [PDF] [Project] [Code]
Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
Youngwoon Lee, Joseph J. Lim, Anima Anandkumar, Yuke Zhu
CoRL 2021 [PDF] [Project]
Distilling Motion-Planner Augmented Policies into Visual Control Policies for Robot Manipulation
I-Chun Arthur Liu*, Shagun Uppal*, Gaurav S. Sukhatme, Joseph J. Lim, Peter Englert, Youngwoon Lee
CoRL 2021 [PDF] [Project] [Code]
Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding
Grace Zhang, Linghan Zhong, Youngwoon Lee, Joseph J. Lim
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
Youngwoon Lee, Edward S. Hu, Joseph J. Lim
ICRA 2021 [PDF] [Project] [Code]
Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning
Taehyeong Kim, Injune Hwang, Hyundo Lee, Hyunseo Kim, Won-Seok Choi, Joseph J. Lim, Byoung-Tak Zhang
ICML 2021 [PDF]

2020

Accelerating Reinforcement Learning with Learned Skill Priors
Karl Pertsch, Youngwoon Lee, Joseph J. Lim
CoRL 2020 (Plenary Talk, Best Paper Presentation Award) [PDF] [Project] [Code] [Talk]
Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
Jun Yamada*, Youngwoon Lee*, Gautam Salhotra, Karl Pertsch, Max Pflueger, Gaurav S. Sukhatme, Joseph J. Lim, Peter Englert
CoRL 2020 [PDF] [Project] [Code]
Generalization to New Actions in Reinforcement Learning
Ayush Jain*, Andrew Szot*, Joseph J. Lim
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning
Karl Pertsch*, Oleh Rybkin*, Jingyun Yang, Shenghao Zhou, Kosta Derpanis, Joseph J. Lim, Kostas Daniilidis, Andrew Jaegle (* - indicates equal contribution)
L4DC 2020 [PDF] [Project]
Program Guided Agent
Shao-Hua Sun, Te-Lin Wu, Joseph J. Lim
ICLR 2020 (Spotlight) [PDF] [Project]
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification
Youngwoon Lee, Jingyun Yang, Joseph J. Lim
ICLR 2020 [PDF] [Project] [Code]

2019

To Follow or not to Follow: Selective Imitation Learning from Observations
Youngwoon Lee, Edward S. Hu, Zhengyu Yang, Joseph J. Lim
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
Risto Vuorioi*, Shao-Hua Sun*, Hexiang Hu, Joseph J. Lim (* - indicates equal contribution)
NeurIPS 2019 (Spotlight) [PDF] [Project] [Code] [Spotlight]
Composing Complex Skills by Learning Transition Policies
Youngwoon Lee*, Shao-Hua Sun*, Sriram Somasundaram, Edward Hu, Joseph J. Lim (* - indicates equal contribution)
ICLR 2019 [PDF] [Project] [Code]
Scaling Simulation-to-Real Transfer by Learning Composable Robot Skills
Ryan Julian*, Eric Heiden*, Zhanpeng He, Hejia Zhang, Stefan Schaal, Joseph J. Lim, Gaurav Sukhatme, Karol Hausman (* - indicates equal contribution)
ISER 2018 (IJRR 2020) [PDF]

2018

Multi-view to Novel view: Synthesizing Novel Views with Self-Learned Confidence
Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, Joseph J. Lim
ECCV 2018 [PDF] [Project] [Code]
Neural Program Synthesis from Diverse Demonstration Videos
Shao-Hua Sun*, Hyeonwoo Noh*, Sriram Somasundaram, Joseph J. Lim (* - indicates equal contribution)
ICML 2018 [PDF] [Project] [Code]
Demo2Vec: Reasoning Object Affordances From Online Videos
Kuan Fang*, Te-Lin Wu*, Daniel Yang, Silvio Savarese, Joseph J. Lim (* - indicates equal contribution)
CVPR 2018 [PDF] [Project]
Learning and Using the Arrow of Time
Donglai Wei, Joseph J. Lim, Andrew Zisserman, William Freeman
CVPR 2018 [PDF]
Universal Agent for Disentangling Environments and Tasks
Jiayuan Mao, Honghua Dong, Joseph J. Lim
ICLR 2018 [PDF]

2017

Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, Joseph J. Lim
NIPS 2017 [PDF]
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
De-An Huang, Joseph J. Lim, Li Fei-Fei, Juan Carlos Niebles
CVPR 2017 [PDF]
Knowledge Acquisition for Visual Question Answering via Iterative Querying
Yuke Zhu, Joseph J. Lim, Li Fei-Fei
CVPR 2017 [PDF]
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi
ICRA 2017 [PDF] [arXiv]

2016

High-Fidelity Facial and Speech Animation for VR HMDs
Kyle Olszewski, Joseph J. Lim, Shunsuke Saito, Hao Li
SIGGRAPH Asia 2016 [PDF]
Single Image 3D Interpreter Network
Jiajun Wu*, Tianfan Xue*, Joseph J. Lim, Yuandong Tian, Joshua B. Tenenbaum, Antonio Torralba, William T. Freeman (* - indicates equal contribution)
ECCV 2016 [PDF] [arXiv]
Physics 101: Learning Physical Object Properties from Unlabeled Videos
Jiajun Wu, Joseph J. Lim, Hongyi Zhang, Joshua B. Tenenbaum, William T. Freeman
BMVC 2016 [PDF]

Before 2016

Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning
Jiajun Wu*, Ilker Yildirim*, Joseph J. Lim, William T. Freeman, Joshua B. Tenenbaum. (* - indicates equal contribution)
NIPS 2015 [PDF]
Discovering States and Transformations in Image Collections
Phillip Isola*, Joseph J. Lim*, Edward H. Adelson. (* - indicates equal contribution)
CVPR 2015 [PDF]
FPM: Fine pose Parts-based Model with 3D CAD models
Joseph J. Lim, Aditya Khosla, Antonio Torralba.
ECCV 2014 [PDF]
Looking Beyond the Visible Scene
Aditya Khosla*, Byoungkwon An*, Joseph J. Lim*, Antonio Torralba. (* - indicates equal contribution)
ECCV 2014 [PDF]
Parsing IKEA Objects: Fine Pose Estimation
Joseph J. Lim, Hamed Pirsiavash, Antonio Torralba.
ICCV 2013 [PDF]
Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection
Joseph J. Lim, C. Lawrence Zitnick, and Piotr Dollar.
CVPR 2013 [PDF]
Transfer Learning by Borrowing Examples for Multiclass Object Detection
Joseph J. Lim, Ruslan Salakhutdinov, and Antonio Torralba
NIPS 2011 [PDF]
Experimental Evaluation of Support Vector Machine-based and Correlation-based Approaches to Automatic Particle Selection
Pablo Arbelaez, Bong-Gyoon Han, Dieter Typke, Joseph J. Lim, Robert M. Glaeser, Jitendra Malik.
Journal of Structural Biology 2011. doi:10.1016/j.jsb.2011.05.017 [PDF]
Exploiting Hierarchical Context on a Large Database of Object Categories
Myung Jin Choi, Joseph J. Lim, Antonio Torralba, and Alan S. Willsky.
CVPR 2010 [PDF]
Context by Region Ancestry
Joseph J. Lim, Pablo Arbelaez, Chunhui Gu, Jitendra Malik.
ICCV 2009 [PDF]
Recognition using Regions
Chunhui Gu, Joseph J. Lim, Pablo Arbelaez, Jitendra Malik.
CVPR 2009 [PDF]
A Distributed Message Passing Algorithm for Sensor Localization
Max Welling and Joseph J. Lim.
ICANN 2007 [PDF]