Transcribe and Diarize Audio Recordings

Transcribes one or more audio recordings using Whisper and performs speaker diarization (who speaks when) using NeMo, NVIDIA's open-source speech toolkit. Runs in a subprocess to avoid OpenMP conflicts on macOS. Requires the talk conda environment installed via talkrpp_install().

Usage

talkTranscribeDiarise(
  audio,
  output_dir = NULL,
  model_name = "medium.en",
  language = NULL,
  device = if (Sys.info()[["sysname"]] == "Darwin") "cpu" else "cuda",
  stemming = TRUE,
  suppress_numerals = FALSE,
  batch_size = 8,
  num_speakers = 2,
  oracle_num_speakers = TRUE,
  vad_model = "vad_multilingual_marblenet",
  speaker_model = "titanet_large",
  onset = 0.8,
  offset = 0.5,
  pad_offset = -0.05,
  domain_type = "telephonic",
  output_formats = "csv",
  remove_stutters = FALSE,
  stutter_threshold = 0.8,
  condaenv = getOption("talkrpp_condaenv", "talkrpp_condaenv"),
  verbose = FALSE
)

talkTextDiarise(
  audio,
  output_dir = NULL,
  model_name = "medium.en",
  language = NULL,
  device = if (Sys.info()[["sysname"]] == "Darwin") "cpu" else "cuda",
  stemming = TRUE,
  suppress_numerals = FALSE,
  batch_size = 8,
  num_speakers = 2,
  oracle_num_speakers = TRUE,
  vad_model = "vad_multilingual_marblenet",
  speaker_model = "titanet_large",
  onset = 0.8,
  offset = 0.5,
  pad_offset = -0.05,
  domain_type = "telephonic",
  output_formats = "csv",
  remove_stutters = FALSE,
  stutter_threshold = 0.8,
  condaenv = getOption("talkrpp_condaenv", "talkrpp_condaenv"),
  verbose = FALSE
)

Arguments

audio

(string or character vector) Path to a single audio file (e.g., .wav) or a vector of file paths. Several files are processed separately, returning a named list of transcript tibbles (names are the file names).

output_dir

(string) Optional directory where transcript files (csv/txt/srt, see output_formats) are saved. Default NULL: no files are written – the transcript is only returned as a tibble.

model_name

(string) Whisper model name (e.g., "medium.en"). Options: "tiny", "base", "small", "medium", "large".

language

(string) Language code (e.g., "en", "sv"). If NULL, Whisper auto-detects the language.

device

(string) Device to run inference on: "cuda" (NVIDIA GPU), "cpu", or "mps" (Apple Silicon GPU). Defaults to "cpu" on macOS and "cuda" elsewhere. "cuda" is fast and produces correct results. "mps" needs care: Whisper's decoder on MPS can silently drop large parts of the transcript (about half, on some files) while appearing to work; the pipeline therefore automatically runs the transcription step on the CPU when device = "mps" and uses the GPU only for the diarisation step. See Details.

stemming

(logical) If TRUE (default), first isolate the vocals with Demucs source separation, so that background music and noise do not end up in the transcript or confuse speaker assignment. Adds processing time; if separation fails, the original audio is used.

suppress_numerals

(logical) If TRUE, numbers are transcribed as words ("twenty" rather than "20"). Spoken words align better with the audio, which improves word-level timestamps. Default is FALSE.

batch_size

(integer) Number of audio segments processed at once. Default is 8. Larger values are faster but use more memory.

num_speakers

(integer) Expected number of speakers for NeMo clustering.

oracle_num_speakers

(logical) If TRUE, force the diarisation to use exactly num_speakers speakers; if FALSE, the number of speakers is estimated.

vad_model

(string) NeMo model for Voice Activity Detection (VAD) – the step that detects which parts of the audio contain speech at all, before any transcription or speaker assignment. Default is "vad_multilingual_marblenet".

speaker_model

(string) NeMo model that computes speaker embeddings (numeric representations of voice characteristics used to tell speakers apart). Default is "titanet_large".

onset

(numeric) VAD threshold (0-1) for detecting the start of a speech segment: higher values require stronger evidence before opening a segment (fewer false starts, but soft speech onsets may be missed). Default is 0.8.

offset

(numeric) VAD threshold (0-1) for detecting the end of a speech segment: lower values keep segments open longer (less clipping of trailing speech, but more trailing noise). Default is 0.5.

pad_offset

(numeric) Seconds added to the end of each detected speech segment; negative values trim the segment end. Default is -0.05.

domain_type

(string) Selects a NeMo diarisation configuration tuned to the recording situation: "telephonic" (close-microphone, two-party recordings such as phone calls or one-on-one interviews; the default), "meeting" (several speakers around a room microphone), or "general". Choosing the type that matches your recordings improves speech detection and speaker assignment.

output_formats

(character vector) Output formats to write. Any of "csv", "txt", "srt". Default is "csv".

remove_stutters

(logical) If TRUE, post-process the transcript to remove immediately repeated, near-duplicate words. Such repetitions come both from genuine speaker disfluencies (stuttering, false starts) and from a known Whisper artifact where the model repeats words. Keep FALSE (the default) when disfluencies themselves are of interest to your research. Applied to the CSV output.

stutter_threshold

(numeric) Similarity (0-1) above which adjacent words are treated as repetitions when remove_stutters = TRUE; higher values remove only near-identical repeats. Default is 0.8.

condaenv

(string) Name of the conda environment with the talk stack installed. Defaults to the environment saved by talkrpp_initialize(save_profile = TRUE) (the "talkrpp_condaenv" option), falling back to "talkrpp_condaenv" – so a shared environment (e.g. "text_talk") only needs to be set once via talkrpp_initialize().

verbose

(logical) If FALSE (default), the technical output from the Python backend (model-loading logs, progress bars, warnings) is hidden and only short status messages are shown. Set TRUE to stream the full backend output, e.g. when debugging.

Value

For a single audio file, a tibble with one row per speaker turn and columns speaker, start_timestamp, end_timestamp and message. On failure an error is raised (with a hint about the likely fix). The settings used are saved as a comment (retrieve with comment()). For several audio files, a named list of such tibbles.

Details

Transcript files (diarized CSVs, SRTs, or TXT) are written only when output_dir is provided (the saved paths are then shown in a message); by default the transcript is only returned as a tibble.

Devices and result correctness. "cpu" always produces correct results and is the safe default on macOS. "cuda" (NVIDIA GPUs on Linux/Windows) is substantially faster and produces correct results. "mps" (Apple Silicon GPU) can run but is treated with caution: Whisper's decoder on MPS has a known bug where it skips ahead and silently drops large parts of the audio – the transcript looks plausible but can be missing about half its content. To protect against this, when device = "mps" the pipeline automatically runs the transcription step on the CPU and uses the GPU only for the diarisation step.

On macOS, NeMo's OpenMP library conflicts with R's when loaded in the same process. This function automatically runs diarization in a subprocess via callr::r() to avoid the crash.

See also

talkEmbed, talkText, talkrpp_initialize

Examples

if (FALSE) { # \dontrun{
wav_path <- system.file("extdata", "test_diarise.wav", package = "talk")
talk::talkTranscribeDiarise(audio = wav_path, num_speakers = 2)
} # }