Heeseung Kim

Email  /  CV  /  LinkedIn  /  Google Scholar  /  GitHub

I am a Ph.D. candidate at the Data Science & AI Lab (DSAIL) at Seoul National University (SNU). My research primarily focuses on speech synthesis (text-to-speech, voice conversion) and spoken language and dialog models, and I have also researched generative models on other domains (vision, NLP, ...). I received my B.S. in Electrical and Computer Engineering from Seoul National University.

I have a broad interest in generative models, and I am currently particularly focused on multimodal large language models in speech and audio. Specifically, I am interested in large spoken language models (LSLM) and spoken dialog models. Additionally, both now and in the past, I have conducted research with a focus on diffusion models. My previous research primarily centered around speech synthesis, where I worked on tasks such as text-to-speech and voice conversion, with a focus on keywords like personalization and data efficiency. Representative papers are highlighted.

USDM is a paralinguistic-aware spoken dialog model built through supervised fine-tuning (SFT) on spoken dialog data, on top of a cross-modal pretrained model trained using a speech-text interleaving technique.

UnitSpeech is a speaker adaptation model that enables personalized text-to-speech and any-to-any voice conversion with only 5 to 10 seconds of untranscribed speech.

Guided-TTS is a method for building a TTS model using long-form untranscribed speech data of the target speaker.

Diptych Prompting is a novel zero-shot subject-driven text-to-image generation that reinterprets as an inpainting task with precise subject alignment by leveraging the emergent property of diptych generation in large-scale text-to-image models.

Style-friendly sampler shifts the diffusion fine-tuning toward higher noise levels, enabling FLUX and SD3.5 to effectively learn new, unique artistic styles and expand the scope of style-driven generation.

NanoVoice is a method that efficiently learns personalized adapters for each speaker simultaneously when given multiple speakers' voices.

VoiceGuider is a personalized text-to-speech model that proposes a guidance method during inference, enabling robust speaker adaptation even for out-of-domain speakers.

VoiceTailor is a one-shot speaker-adaptive text-to-speech model, which proposes combining low-rank adapters to perform speaker adaptation in a parameter-efficient manner.

HyperCLOVA X is a series of large language models (LLMs) specifically designed to accommodate the Korean language and culture, while also excelling in English, mathematics, and coding tasks.

Edit-A-Video is a diffusion-based one-shot video editing model that solves a background inconsistency problem using a new sparse-causal mask blending method.

The proposed high-performance strain sensors, optimally positioned using deep learning analysis, enable accurate detection of directional facial muscle movement for silent speech recognition.

Guided-TTS 2 is a model that enables personalized text-to-speech using only 10 seconds of untranscribed speech.

AGG addresses the degeneration problem in neural language models by gating the specific part of the gradient for rare token embeddings.

PriorGrad presents an efficient method for constructing a data-dependent non-standard Gaussian prior for training and sampling from diffusion models applied to speech synthesis.

SLOGAN introduces Stein latent optimization and a novel unsupervised conditional contrastive loss for GANs with clustered latent spaces, enabling better conditional generation of imbalanced real-world data.