Heeseung Kim

Email  /  CV  /  LinkedIn  /  Google Scholar  /  GitHub

I am currently a Senior Research Engineer at Qualcomm AI Research Korea, where I work on developing more human-like, real-time voice agents. I received my Ph.D. as a candidate at the Data Science & AI Lab (DSAIL) at Seoul National University (SNU). My research there primarily focused on speech synthesis (text-to-speech, voice conversion) and speech large language and dialog models (speech LLMs), and I also explored generative models in other domains (vision, NLP, ...). I received my B.S. in Electrical and Computer Engineering from Seoul National University.

I have a broad interest in generative models, and I am currently particularly focused on multimodal large language models in speech and audio. Specifically, I am interested in speech large language models (speech LLMs) and spoken dialog models. Additionally, both now and in the past, I have conducted research with a focus on diffusion models. My previous research primarily centered around speech synthesis, where I worked on tasks such as text-to-speech and voice conversion, with a focus on keywords like personalization and data efficiency. Below are my representative works.

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, Kang Min Yoo Neural Information Processing Systems (NeurIPS), 2024 arXiv / code / demo USDM is a paralinguistic-aware spoken dialog model built through supervised fine-tuning (SFT) on spoken dialog data, on top of a cross-modal pretrained model trained using a speech-text interleaving technique.

Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models Heeseung Kim*, Che Hyun Lee*, Sangkwon Park, Jiheum Yeom, Nohil Park, Sangwon Yu, Sungroh Yoon ACL Findings, 2025 arXiv / demo / dataset We propose ContextDialog, a benchmark to evaluate a model’s ability to utilize past information, and observe and analyze that open-source multi-turn voice interaction models often fail to recall past information and, even in RAG scenarios, remain highly error-prone, resulting in inadequate responses.

UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data Heeseung Kim, Sungwon Kim, Jiheum Yeom, Sungroh Yoon INTERSPEECH, Oral Presentation, 2023 arXiv / code / demo UnitSpeech is a speaker adaptation model that enables personalized text-to-speech and any-to-any voice conversion with only 5 to 10 seconds of untranscribed speech.

Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance Heeseung Kim*, Sungwon Kim*, Sungroh Yoon International Conference on Machine Learning (ICML), 2022 arXiv / demo Guided-TTS is a method for building a TTS model using long-form untranscribed speech data of the target speaker.

VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based Personalized Text-to-Speech Heeseung Kim, Sang-gil Lee, Jiheum Yeom, Che Hyun Lee, Sungwon Kim, Sungroh Yoon INTERSPEECH, 2024 project page / arXiv VoiceTailor is a one-shot speaker-adaptive text-to-speech model, which proposes combining low-rank adapters to perform speaker adaptation in a parameter-efficient manner.

During my time at DSAIL, I collaborated with NAVER Cloud on developing spoken language and spoken dialog models based on pre-trained large language models (LLMs). In this project, we extended a pre-trained LLM into a spoken language model by leveraging a large-scale speech-text paired dataset and further expanded it into a spoken dialog model through supervised fine-tuning. I successfully built a USDM, an English end-to-end spoken dialog model that incorporates paralinguistic features and contributed to the construction of Naver’s end-to-end Korean spoken dialog model, SpeechX.

Project Overview

• 2022.03 ~ 2023.07: Built a speaker-adaptive TTS model for more natural voice generation. Incorporated speech input into a GPT-2–scale model by attaching an encoder for ASR (Automatic Speech Recognition) and SER (Speech Emotion Recognition), aiming to enrich speech understanding capabilities.
• 2023.08 ~ 2024.05: Developed an English spoken dialog system with an end-to-end pipeline and paralinguistic awareness. This research led to a publication at NeurIPS 2024, while simultaneously laying the groundwork for Naver’s proprietary Speech LLM.
• 2024.06 ~ 2025.01: Focused on developing a text-to-speech model for synthetic spoken dialog generation.

Related Articles & Blog Posts

• Naver-Seoul National University Release Speech Model, Achieves Natural Speech Like GPT-4o - Naver and Seoul National University researchers develop a large-scale spoken language model, achieving natural spoken dialog generation and publishing their work at NeurIPS 2024.
• AI that can talk and understand human emotions - Spoken dialog model that can engage in spoken interactions while understanding and responding to human emotions, highlighting advancements in emotional recognition and natural conversation.

Below is a list of my publications.