Youngtaek Oh

Publications

1. 

   Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality

   Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, Junmo Kim

   Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

   Oral presentation

   Absract Project arXiv Code

   In this paper, we propose a new method to enhance compositional understanding in pre-trained vision and language models (VLMs) without sacrificing performance in zero-shot multi-modal tasks.

   Traditional fine-tuning approaches often improve compositional reasoning at the cost of degrading multi-modal capabilities, primarily due to the use of global hard negative (HN) loss, which contrasts global representations of images and texts. This global HN loss pushes HN texts that are highly similar to the original ones, damaging the model’s multi-modal representations.

   To overcome this limitation, we propose Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard negative loss and selective calibrated regularization. These innovations provide fine-grained negative supervision while preserving the model’s representational integrity.

   Our extensive evaluations across diverse benchmarks for both compositionality and multi-modal tasks show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities.

2. 

   Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

   Youngtaek Oh, Pyunghwan Ahn, Jinhyung Kim, Gwangmo Song, Soonyoung Lee, In So Kweon, Junmo Kim

   CVPR Workshop on ‘What is Next in Multimodal Foundation Models?’ (MMFM), 2024

   Absract BIBTEX arXiv Code

   Vision and language models (VLMs) such as CLIP have showcased remarkable zero-shot recognition abilities yet face challenges in visio-linguistic compositionality, particularly in linguistic comprehension and fine-grained image-text alignment.

   This paper explores the intricate relationship between compositionality and recognition – two pivotal aspects of VLM capability. We conduct a comprehensive evaluation of existing VLMs, covering both pre-training approaches aimed at recognition and the fine-tuning methods designed to improve compositionality. Our evaluation employs 12 benchmarks for compositionality, along with 21 zero-shot classification and two retrieval benchmarks for recognition.

   In our analysis from 274 CLIP model checkpoints, we reveal patterns and trade-offs that emerge between compositional understanding and recognition accuracy. Ultimately, this necessitates strategic efforts towards developing models that improve both capabilities, as well as the meticulous formulation of benchmarks for compositionality.

   @article{oh2024exploring,
     title = {Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition},
     author = {Oh, Youngtaek and Ahn, Pyunghwan and Kim, Jinhyung and Song, Gwangmo and Lee, Soonyoung and Kweon, In So and Kim, Junmo},
     journal = {arXiv preprint arXiv:2406.09388},
     year = {2024}
   }

3. 

   Empirical study on using Adapters for debiased Visual Question Answering

   Jae Won Cho, Dawit Mureja Argaw, Youngtaek Oh, Dong-Jin Kim, In So Kweon

   Computer Vision and Image Understanding (CVIU), 2023

   BIBTEX HTML

   @article{cho2023empirical,
     title = {Empirical study on using Adapters for debiased Visual Question Answering},
     author = {Cho, Jae Won and Argaw, Dawit Mureja and Oh, Youngtaek and Kim, Dong-Jin and Kweon, In So},
     journal = {Computer Vision and Image Understanding},
     pages = {103842},
     year = {2023},
     publisher = {Elsevier}
   }

4. 

   NICE: CVPR 2023 challenge on zero-shot image captioning

   Taehoon Kim, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Mark Marsden, Alessandra Sala, Seung Hwan Kim, Honglak Lee, Kyounghoon Bae, and 33 more authors

   preprint, 2024

   2nd place in the Image Captioning Challenge at the NICE Workshop, CVPR 2023

   BIBTEX arXiv Code

   @inproceedings{kim2023nice,
     title = {NICE: CVPR 2023 challenge on zero-shot image captioning},
     author = {Kim, Taehoon and Ahn, Pyunghwan and Kim, Sangyun and Lee, Sihaeng and Marsden, Mark and Sala, Alessandra and Kim, Seung Hwan and Lee, Honglak and Bae, Kyounghoon and Han, Bohyung and Lee, Kyoung Mu and Wu, Xiangyu and Gao, Yi and Zhang, Hailiang and Yang, Yang and Guo, Weili and Lu, Jianfeng and Oh, Youngtaek and Cho, Jae Won and Kim, Dong-Jin and Kweon, In So and Kim, Junmo and Kang, Wooyoung and Jhoo, Won Young and Roh, Byungseok and Mun, Jonghwan and Oh, Solgil and Ak, Kenan Emir and Lee, Gwang-Gook and Xu, Yan and Shen, Mingwei and Hwang, Kyomin and Shin, Wonsik and Lee, Kamin and Park, Wonhark and Lee, Dongkwan and Kwak, Nojun and Wang, Yujin and Wang, Yimu and Gu, Tiancheng and Lv, Xingchang and Sun, Mingmao},
     year = {2024},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     pages = {7356--7365}
   }

5. 

   Self-Sufficient Framework for Continuous Sign Language Recognition

   Youngjoon Jang, Youngtaek Oh, Jae Won Cho, Myungchul Kim, Dong-Jin Kim, In So Kweon, Joon Son Chung

   International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023

   Oral presentation, Top 3% recognition of all accepted papers

   BIBTEX Project PDF

   @inproceedings{jang2023self,
     title = {Self-Sufficient Framework for Continuous Sign Language Recognition},
     author = {Jang, Youngjoon and Oh, Youngtaek and Cho, Jae Won and Kim, Myungchul and Kim, Dong-Jin and Kweon, In So and Chung, Joon Son},
     booktitle = {International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
     year = {2023}
   }

6. 

   Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition

   Youngjoon Jang, Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, Joon Son Chung, In So Kweon

   British Machine Vision Conference (BMVC), 2022

   BIBTEX Project PDF Code

   @inproceedings{jang2022signing,
     title = {Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition},
     author = {Jang, Youngjoon and Oh, Youngtaek and Cho, Jae Won and Kim, Dong-Jin and Chung, Joon Son and Kweon, In So},
     booktitle = {British Machine Vision Conference (BMVC)},
     year = {2022}
   }

7. 

   DASO: Distribution-Aware Semantics-Oriented Pseudo-Label for Imbalanced Semi-Supervised Learning

   Youngtaek Oh, Dong-Jin Kim, In So Kweon

   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

   BIBTEX Project arXiv PDF Code

   @inproceedings{oh2022daso,
     title = {DASO: Distribution-Aware Semantics-Oriented Pseudo-Label for Imbalanced Semi-Supervised Learning},
     author = {Oh, Youngtaek and Kim, Dong-Jin and Kweon, In So},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     pages = {9786--9796},
     year = {2022}
   }

8. 

   KSL-Guide: A Large-scale Korean Sign Language Dataset Including Interrogative Sentences for Guiding the Deaf and Hard-of-Hearing

   Soomin Ham, Kibaek Park, Youngjoon Jang, Youngtaek Oh, Seokmin Yun, Sukwon Yoon, Chang Jo Kim, Han-Mu Park, In So Kweon

   International Conference on Automatic Face and Gesture Recognition (FG), 2021

   BIBTEX HTML Code Website

   @inproceedings{ham2021ksl,
     title = {KSL-Guide: A Large-scale Korean Sign Language Dataset Including Interrogative Sentences for Guiding the Deaf and Hard-of-Hearing},
     author = {Ham, Soomin and Park, Kibaek and Jang, Youngjoon and Oh, Youngtaek and Yun, Seokmin and Yoon, Sukwon and Kim, Chang Jo and Park, Han-Mu and Kweon, In So},
     booktitle = {International Conference on Automatic Face and Gesture Recognition (FG)},
     pages = {1--8},
     year = {2021},
     organization = {IEEE}
   }

9. 

   SideGuide: A Large-scale Sidewalk Dataset for Guiding Impaired People

   {Kibaek Park, Youngtaek Oh, Soomin Ham, Kyungdon Joo}*, Hyokyoung Kim, Hyoyoung Kum, In So Kweon (*: equally contributed)

   International Conference on Intelligent Robots and Systems (IROS), 2020

   BIBTEX HTML PDF Code Website

   @inproceedings{park2020sideguide,
     title = {SideGuide: A Large-scale Sidewalk Dataset for Guiding Impaired People},
     author = {Park, Kibaek and Oh, Youngtaek and Ham, Soomin and Joo, Kyungdon and Kim, Hyokyoung and Kum, Hyoyoung and Kweon, In So},
     booktitle = {International Conference on Intelligent Robots and Systems (IROS)},
     pages = {10022--10029},
     year = {2020},
     organization = {IEEE}
   }