Audio Deepfake Detection

Voice cloning technology has advanced rapidly, enabling convincing synthetic speech from just seconds of audio. Detecting audio deepfakes requires specialized techniques including spectrogram analysis, voice biometric verification, and multi-modal consistency checks.

Voice Cloning Techniques

Modern voice cloning systems that detection must counter:

• Text-to-Speech (TTS): Models like VALL-E, Bark, and XTTS clone a voice from a few seconds of reference audio, generating speech from any text input.
• Voice Conversion (VC): Transform one speaker's voice to sound like another in real-time (e.g., RVC, So-VITS-SVC).
• Speech-to-Speech: Real-time voice transformation during live calls — the most dangerous for social engineering attacks.

Spectrogram-Based Detection

Audio spectrograms reveal artifacts invisible in the time domain:

import librosa
import numpy as np
import torch

def extract_audio_features(audio_path, sr=16000):
    """Extract features for deepfake detection."""
    y, sr = librosa.load(audio_path, sr=sr)

    # Mel spectrogram - primary feature
    mel_spec = librosa.feature.melspectrogram(
        y=y, sr=sr, n_mels=128, n_fft=2048, hop_length=512
    )
    mel_db = librosa.power_to_db(mel_spec, ref=np.max)

    # MFCC - captures vocal tract characteristics
    mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=40)

    # Linear frequency cepstral coefficients (LFCC)
    # More effective than MFCC for spoofing detection
    spec = np.abs(librosa.stft(y, n_fft=2048))
    lfcc = librosa.feature.mfcc(S=librosa.power_to_db(spec**2), n_mfcc=40)

    return {
        "mel_spectrogram": mel_db,
        "mfcc": mfccs,
        "lfcc": lfcc
    }

ASVspoof Challenge

The Automatic Speaker Verification Spoofing challenge is the primary benchmark for audio deepfake detection:

• ASVspoof 2019: Logical access (TTS/VC attacks) and physical access (replay attacks)
• ASVspoof 2021: Added compressed audio and telephony channel conditions
• ASVspoof 5 (2024): Focuses on robustness to codec compression and real-world conditions
• Evaluation metric: Equal Error Rate (EER) and tandem Detection Cost Function (t-DCF)

FakeAVCeleb Dataset

A multimodal deepfake dataset combining both audio and visual manipulation:

• Contains both face swap and voice cloning manipulations, separately and combined
• Enables research on audio-visual consistency for detection
• If audio says "happy" but visual expression shows "neutral," the content may be manipulated
• Multi-modal detectors achieve higher accuracy than single-modality approaches

Multi-Modal Detection

Combining audio and visual analysis provides the strongest detection:

• Lip-audio sync: Check if lip movements match the audio waveform timing and phonemes
• Emotion consistency: Verify that vocal emotion matches facial expression
• Speaker identity: Cross-reference voice biometrics with face recognition results
• Environmental consistency: Check if audio acoustics match the visual environment (room reverb, background noise)