Heeseung Kim

Email  /  CV  /  LinkedIn  /  Google Scholar  /  GitHub

I am a Ph.D. candidate at the Data Science & AI Lab (DSAIL) at Seoul National University (SNU). My research primarily focuses on speech synthesis (text-to-speech, voice conversion) and speech large language and dialog models (speech LLMs), and I have also researched generative models on other domains (vision, NLP, ...). I received my B.S. in Electrical and Computer Engineering from Seoul National University.

I have a broad interest in generative models, and I am currently particularly focused on multimodal large language models in speech and audio. Specifically, I am interested in speech large language models (speech LLMs) and spoken dialog models. Additionally, both now and in the past, I have conducted research with a focus on diffusion models. My previous research primarily centered around speech synthesis, where I worked on tasks such as text-to-speech and voice conversion, with a focus on keywords like personalization and data efficiency. Below are my representative works.

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, Kang Min Yoo Neural Information Processing Systems (NeurIPS), 2024 arXiv / code / demo USDM is a paralinguistic-aware spoken dialog model built through supervised fine-tuning (SFT) on spoken dialog data, on top of a cross-modal pretrained model trained using a speech-text interleaving technique.

UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data Heeseung Kim, Sungwon Kim, Jiheum Yeom, Sungroh Yoon INTERSPEECH, Oral Presentation, 2023 arXiv / code / demo UnitSpeech is a speaker adaptation model that enables personalized text-to-speech and any-to-any voice conversion with only 5 to 10 seconds of untranscribed speech.

Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance Heeseung Kim*, Sungwon Kim*, Sungroh Yoon International Conference on Machine Learning (ICML), 2022 arXiv / demo Guided-TTS is a method for building a TTS model using long-form untranscribed speech data of the target speaker.

VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based Personalized Text-to-Speech Heeseung Kim, Sang-gil Lee, Jiheum Yeom, Che Hyun Lee, Sungwon Kim, Sungroh Yoon INTERSPEECH, 2024 project page / arXiv VoiceTailor is a one-shot speaker-adaptive text-to-speech model, which proposes combining low-rank adapters to perform speaker adaptation in a parameter-efficient manner.

Edit-A-Video: Single Video Editing with Object-Aware Consistency Chaehun Shin*, Heeseung Kim*, Che Hyun Lee, Sang-gil Lee, Sungroh Yoon Asian Conference on Machine Learning (ACML), Oral, Best Paper Award, 2023 project page / arXiv Edit-A-Video is a diffusion-based one-shot video editing model that solves a background inconsistency problem using a new sparse-causal mask blending method.

During my time at DSAIL, I collaborated with NAVER Cloud on developing spoken language and spoken dialog models based on pre-trained large language models (LLMs). In this project, we extended a pre-trained LLM into a spoken language model by leveraging a large-scale speech-text paired dataset and further expanded it into a spoken dialog model through supervised fine-tuning. I successfully built a USDM, an English end-to-end spoken dialog model that incorporates paralinguistic features and contributed to the construction of Naver’s end-to-end Korean spoken dialog model, SpeechX.

Project Overview

• 2022.03 ~ 2023.07: Built a speaker-adaptive TTS model for more natural voice generation. Incorporated speech input into a GPT-2–scale model by attaching an encoder for ASR (Automatic Speech Recognition) and SER (Speech Emotion Recognition), aiming to enrich speech understanding capabilities.
• 2023.08 ~ 2024.05: Developed an English spoken dialog system with an end-to-end pipeline and paralinguistic awareness. This research led to a publication at NeurIPS 2024, while simultaneously laying the groundwork for Naver’s proprietary Speech LLM.
• 2024.06 ~ 2025.01: Focused on developing a text-to-speech model for synthetic spoken dialog generation.
• 2025.02 ~ 2025.05 (Ongoing): Beyond supervised fine-tuning, exploring RLHF (e.g., DPO, GRPO) methods to capture more natural and affective nuances in spoken dialog. Currently investigating strategies for reward/preference data collection, either from human annotators or synthetic approaches, to enhance voice interaction quality.

Related Articles & Blog Posts

• Naver-Seoul National University Release Speech Model, Achieves Natural Speech Like GPT-4o - Naver and Seoul National University researchers develop a large-scale spoken language model, achieving natural spoken dialog generation and publishing their work at NeurIPS 2024.
• AI that can talk and understand human emotions - Spoken dialog model that can engage in spoken interactions while understanding and responding to human emotions, highlighting advancements in emotional recognition and natural conversation.

Below is a list of my publications.