Tsu-Jui (Ray) Fu

I am a Ph.D. candidate at UCSB CS, advised by William Wang. My research lies in vision+language and text-guided visual editing. I am also interested in language grounding and information extraction.


2019 - Now
Research Assistant @ UCSB NLP
Advisor: William Yang Wang

Summer 2022
Research Intern @ Meta AI
Advisor: Licheng Yu and Sean Bell

Summer 2021
Research Intern @ Microsoft Azure AI
Advisor: Linjie Li, Zhe Gan, and Lijuan Wang

Summer 2020
Research Intern @ Microsoft Research
Advisor: Yale Song and Daniel McDuff

Summer 2019
Research Intern @ Preferred Networks
Advisor: Yuta Tsuboi and Jason Naradowsky

2018 - 2019
Research Assistant @ Academia Sinica CKIP
Advisor: Wei-Yun Ma

Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yang Fu, Jong-Chyi Su, William Yang Wang, and Sean Bell
Paper / Code

An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Tsu-Jui Fu*, Linjie Li*, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu

ULN: Towards Underspecified Vision-and-Language Navigation
Weixi Feng, Tsu-Jui Fu, Yujie Lu, and William Yang Wang
EMNLP'22 (Long)
Paper / Code

CPL: Counterfactual Prompt Learning for Vision and Language Models
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang
EMNLP'22 (Long)
Paper / Code

Language-Driven Artistic Style Transfer
Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang
Paper / Project / Slide / Video / Code

M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformer
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang
Paper / Slide / Video / Dataset

DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song
Paper / Project / Slide / Video / Code

VIOLET: End-to-End Video-Language Transformers with Masked Visual-token Modeling
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu
Paper / Video (zh) / Code

H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extraction
Tsu-Jui Fu*, Jhih-Wei Chen*, Chen-Kang Lee, and Wei-Yun Ma
ACL'21 (Findings)
Paper / Slide / Video / Code

Semi-Supervised Policy Initialization for Playing Games with Language Hints
Tsu-Jui Fu and William Yang Wang
NAACL'21 (Short)
Paper / Slide / Video / Code

L2C: Describing Visual Differences Needs Semantic Understanding of Individuals
An Yan, Xin Eric Wang, Tsu-Jui Fu, and William Yang Wang
EACL'21 (Short)

Multimodal Style Transfer Learning for Outdoor Vision-and-Language Navigation
Wanrong Zhu, Xin Eric Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, and William Yang Wang
EACL'21 (Long)
Paper / Code

SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang
EMNLP'20 (Oral)
Paper / Slide / Code

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampler
Tsu-Jui Fu, Xin Eric Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, and William Yang Wang
ECCV'20 (Spotlight)
Paper / Slide / Video / Model

Why Attention? Analyzing and Remedying BiLSTM Deficiency in Modeling Cross-Context for NER
Peng-Hsuan Li, Tsu-Jui Fu, and Wei-Yun Ma
AAAI'20 (Oral)
Paper / Code

Learning from Observation-Only Demonstration for Task-Oriented Language Grounding via Self-Examination
Tsu-Jui Fu, Yuta Tsuboi, Sosuke Kobayashi, and Yuta Kikuchi
NeurIPSW'19 (ViGIL workshop)

A Distributed Scheme for Accelerating Semantic Video Segmentation on An Embedded Cluster
Tsu-Jui Fu*, Hsuan-Kung Yang*, Kuan-Wei Ho, Po-Han Chiang, and Chun-Yi Lee
ICCD'19 (Oral)
Paper / Video

Adversarial Active Exploration for Inverse Dynamics Model Learning
Zhang-Wei Hong, Tsu-Jui Fu, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun-Yi Lee
CoRL'19 (Oral)

GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma
ACL'19 (Long)
Paper / Slide / Code

Attentive and Adversarial Learning for Video Summarization
Tsu-Jui Fu, Shao-Heng Tai, and Hwann-Tzong Chen
WACV'19 (Oral)
Paper / Video / Code

Region-Semantics Preserving Image Synthesis
Kang-Jun Liu, Tsu-Jui Fu, and Shan-Hung Wu
Paper / Video / Code

Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee
Paper / Video

Speed Reading: Learning to Read ForBackward via Shuttle
Tsu-Jui Fu and Wei-Yun Ma
EMNLP'18 (Long)
Paper / Code

Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information
Hsuan-Kung Yang, An-Chieh Cheng*, Kuan-Wei Ho*, Tsu-Jui Fu, and Chun-Yi Lee
ECCVW'18 (PIC workshop)

Dynamic Video Segmentation Network
Yu-Syuan Xu, Tsu-Jui Fu*, Hsuan-Kung Yang*, and Chun-Yi Lee
Paper / Video / Code

Template from Jon Barron