Youngtaek Oh

I am a Ph.D. student advised by Prof. In So Kweon and Prof. Junmo Kim at KAIST, and was a research intern at LG AI Reseach, working with Dr. Pyunghwan Ahn and Dr. Jinhyung Kim.

My research centers on developing data-efficient and robust recognition systems that address limitations and biases in image and video data. I am currently focusing on enhancing vision-language models to improve compositional reasoning and multi-modal understanding for more robust and comprehensive recognition systems.

Publications

Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality

Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, Junmo Kim

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

Oral presentation

Absract Project arXiv Code

In this paper, we propose a new method to enhance compositional understanding in pre-trained vision and language models (VLMs) without sacrificing performance in zero-shot multi-modal tasks.

Traditional fine-tuning approaches often improve compositional reasoning at the cost of degrading multi-modal capabilities, primarily due to the use of global hard negative (HN) loss, which contrasts global representations of images and texts. This global HN loss pushes HN texts that are highly similar to the original ones, damaging the model’s multi-modal representations.

To overcome this limitation, we propose Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard negative loss and selective calibrated regularization. These innovations provide fine-grained negative supervision while preserving the model’s representational integrity.

Our extensive evaluations across diverse benchmarks for both compositionality and multi-modal tasks show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities.
Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

Youngtaek Oh, Pyunghwan Ahn, Jinhyung Kim, Gwangmo Song, Soonyoung Lee, In So Kweon, Junmo Kim

CVPR Workshop on ‘What is Next in Multimodal Foundation Models?’ (MMFM), 2024

Absract BIBTEX arXiv Code

Vision and language models (VLMs) such as CLIP have showcased remarkable zero-shot recognition abilities yet face challenges in visio-linguistic compositionality, particularly in linguistic comprehension and fine-grained image-text alignment.

This paper explores the intricate relationship between compositionality and recognition – two pivotal aspects of VLM capability. We conduct a comprehensive evaluation of existing VLMs, covering both pre-training approaches aimed at recognition and the fine-tuning methods designed to improve compositionality. Our evaluation employs 12 benchmarks for compositionality, along with 21 zero-shot classification and two retrieval benchmarks for recognition.

In our analysis from 274 CLIP model checkpoints, we reveal patterns and trade-offs that emerge between compositional understanding and recognition accuracy. Ultimately, this necessitates strategic efforts towards developing models that improve both capabilities, as well as the meticulous formulation of benchmarks for compositionality.
@article{oh2024exploring, title = {Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition}, author = {Oh, Youngtaek and Ahn, Pyunghwan and Kim, Jinhyung and Song, Gwangmo and Lee, Soonyoung and Kweon, In So and Kim, Junmo}, journal = {arXiv preprint arXiv:2406.09388}, year = {2024}, url = {https://arxiv.org/abs/2406.09388} }

Empirical study on using Adapters for debiased Visual Question Answering

Jae Won Cho, Dawit Mureja Argaw, Youngtaek Oh, Dong-Jin Kim, In So Kweon

Computer Vision and Image Understanding (CVIU), 2023

BIBTEX HTML

@article{cho2023empirical,
  title = {Empirical study on using Adapters for debiased Visual Question Answering},
  author = {Cho, Jae Won and Argaw, Dawit Mureja and Oh, Youngtaek and Kim, Dong-Jin and Kweon, In So},
  journal = {Computer Vision and Image Understanding},
  pages = {103842},
  year = {2023},
  publisher = {Elsevier}
}

NICE 2023 Zero-shot Image Captioning Challenge

Taehoon Kim, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Mark Marsden, Alessandra Sala, Seung Hwan Kim, Honglak Lee, Kyounghoon Bae, and 33 more authors

preprint, 2023

BIBTEX arXiv Code Workshop

@article{kim2023nice,
  title = {NICE 2023 Zero-shot Image Captioning Challenge},
  author = {Kim, Taehoon and Ahn, Pyunghwan and Kim, Sangyun and Lee, Sihaeng and Marsden, Mark and Sala, Alessandra and Kim, Seung Hwan and Lee, Honglak and Bae, Kyounghoon and Han, Bohyung and Lee, Kyoung Mu and Wu, Xiangyu and Gao, Yi and Zhang, Hailiang and Yang, Yang and Guo, Weili and Lu, Jianfeng and Oh, Youngtaek and Cho, Jae Won and Kim, Dong-Jin and Kweon, In So and Kim, Junmo and Kang, Wooyoung and Jhoo, Won Young and Roh, Byungseok and Mun, Jonghwan and Oh, Solgil and Ak, Kenan Emir and Lee, Gwang-Gook and Xu, Yan and Shen, Mingwei and Hwang, Kyomin and Shin, Wonsik and Lee, Kamin and Park, Wonhark and Lee, Dongkwan and Kwak, Nojun and Wang, Yujin and Wang, Yimu and Gu, Tiancheng and Lv, Xingchang and Sun, Mingmao},
  year = {2023},
  journal = {arXiv preprint arXiv:2309.01961}
}

Self-Sufficient Framework for Continuous Sign Language Recognition

Youngjoon Jang, Youngtaek Oh, Jae Won Cho, Myungchul Kim, Dong-Jin Kim, In So Kweon, Joon Son Chung

International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023

Oral presentation, Top 3% recognition of all accepted papers

BIBTEX Project PDF

@inproceedings{jang2023self,
  title = {Self-Sufficient Framework for Continuous Sign Language Recognition},
  author = {Jang, Youngjoon and Oh, Youngtaek and Cho, Jae Won and Kim, Myungchul and Kim, Dong-Jin and Kweon, In So and Chung, Joon Son},
  booktitle = {International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
  year = {2023}
}

Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition

Youngjoon Jang, Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, Joon Son Chung, In So Kweon

British Machine Vision Conference (BMVC), 2022

BIBTEX Project PDF Code

@inproceedings{jang2022signing,
  title = {Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition},
  author = {Jang, Youngjoon and Oh, Youngtaek and Cho, Jae Won and Kim, Dong-Jin and Chung, Joon Son and Kweon, In So},
  booktitle = {British Machine Vision Conference (BMVC)},
  year = {2022}
}

DASO: Distribution-Aware Semantics-Oriented Pseudo-Label for Imbalanced Semi-Supervised Learning

Youngtaek Oh, Dong-Jin Kim, In So Kweon

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

BIBTEX Project arXiv PDF Code

@inproceedings{oh2022daso,
  title = {DASO: Distribution-Aware Semantics-Oriented Pseudo-Label for Imbalanced Semi-Supervised Learning},
  author = {Oh, Youngtaek and Kim, Dong-Jin and Kweon, In So},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages = {9786--9796},
  year = {2022}
}

KSL-Guide: A Large-scale Korean Sign Language Dataset Including Interrogative Sentences for Guiding the Deaf and Hard-of-Hearing

Soomin Ham, Kibaek Park, Youngjoon Jang, Youngtaek Oh, Seokmin Yun, Sukwon Yoon, Chang Jo Kim, Han-Mu Park, In So Kweon

International Conference on Automatic Face and Gesture Recognition (FG), 2021

BIBTEX HTML Code Website

@inproceedings{ham2021ksl,
  title = {KSL-Guide: A Large-scale Korean Sign Language Dataset Including Interrogative Sentences for Guiding the Deaf and Hard-of-Hearing},
  author = {Ham, Soomin and Park, Kibaek and Jang, Youngjoon and Oh, Youngtaek and Yun, Seokmin and Yoon, Sukwon and Kim, Chang Jo and Park, Han-Mu and Kweon, In So},
  booktitle = {International Conference on Automatic Face and Gesture Recognition (FG)},
  pages = {1--8},
  year = {2021},
  organization = {IEEE}
}

SideGuide: A Large-scale Sidewalk Dataset for Guiding Impaired People

{Kibaek Park, Youngtaek Oh, Soomin Ham, Kyungdon Joo}*, Hyokyoung Kim, Hyoyoung Kum, In So Kweon (*: equally contributed)

International Conference on Intelligent Robots and Systems (IROS), 2020

BIBTEX HTML PDF Code Website

@inproceedings{park2020sideguide,
  title = {SideGuide: A Large-scale Sidewalk Dataset for Guiding Impaired People},
  author = {Park, Kibaek and Oh, Youngtaek and Ham, Soomin and Joo, Kyungdon and Kim, Hyokyoung and Kum, Hyoyoung and Kweon, In So},
  booktitle = {International Conference on Intelligent Robots and Systems (IROS)},
  pages = {10022--10029},
  year = {2020},
  organization = {IEEE}
}

Honors and Awards

Jun 2023	Top 3% Recognition Certificates, ICASSP 2023 [Certificate]
May 2023	2nd place in NICE Challenge ($5000), NICE Workshop at CVPR 2023 [Press] [Code]
Oct 2022	Outstanding reviewer award, ECCV 2022.