← All posts

Building a Deep Learning-Based Voice Activity Detection Model from Scratch

Aug 9, 20264 min readVoice Activity DetectionVADDeep LearningSpeech ProcessingAudio ProcessingMachine LearningArtificial IntelligenceSpeech AIPyTorchOpen SourceAudio AI
Building a Deep Learning-Based Voice Activity Detection Model from Scratch

Building a Deep Learning-Based Voice Activity Detection Model from Scratch

In my previous post, I explained what Voice Activity Detection (VAD) is and why it matters.

This time, I wanted to go one step further. Instead of using an existing VAD, I built my own deep learning-based VAD from scratch. The goal was simple: build a small model that can detect speech in noisy audio, work in real time, and be easy to deploy.

Why Deep Learning?

I had already built three traditional VADs based on:

  • Energy

  • Zero Crossing Rate

  • Spectral Features

They are fast and surprisingly useful, especially in clean environments. But they also have obvious limitations. A whisper can have very little energy. Music can have a strong spectral structure. Background noise can sometimes be louder than the speaker. At some point, adding another threshold does not solve the real problem. So instead of manually defining what speech should look like, I have decided to let a neural network learn it from data (publicaly available).

The Biggest Challenge Wasn't the Model

The first question was not:

"Which neural network should I use?"

It was:

"Where do I get good labels?"

A VAD needs speech/non-speech labels at very small time intervals. Manually labeling thousands of hours of audio was not realistic. So I used a different approach.

Label the clean audio first, then add the noise.

I used clean speech recordings to create initial labels with an energy-based detector. Then I mixed them with traffic, fans, cafes, reverberation, and other noises.

The audio became difficult. The labels stayed correct. That gave me difficult training examples without manually labeling every noisy recording.

Creating Realistic Training Data

Each training example is a 4-second synthetic scene.

I randomly combine:

  • Speech segments

  • Silence

  • Background noise

  • Room impulse responses

  • Different SNR levels

  • Low-pass filtering

  • Clipping

The SNR ranges from -10 dB to 25 dB. Also some scenes contain no speech at all. This is important because a VAD that detects speech in an empty room is extremely annoying in real applications.

The Model

The model works on 64-bin log-mel spectrograms, rather than raw audio.

The main configuration is: ParameterValue

Sample Rate : 16 kHz

Window : 30 ms

Hop : 10 ms

Mel Bins : 64

FFT : 480

I also use center=False so the feature extractor does not look into the future. That is important for real-time inference. The architecture is a causal MarbleNet-style CNN + GRU:

Log-Mel → Separable Convolutions → Residual Blocks → GRU → Speech Probability

I deliberately kept it small because I wanted something that could run efficiently on a CPU, not just on a GPU.

Training

The complete training pipeline runs on a single Kaggle P100 GPU.

The dataset contains approximately:

  • 35 hours of speech

  • 20 hours of noise

  • 3,000 room impulse responses

Training takes roughly 2 to 3 hours for 20 epochs. I also use boundary weighting and frequency masking to make training more robust.

How Well Does It Work?

On the held-out validation set:

ROC-AUC

0.969

F1 Score

0.903

False Alarm Rate

7.9%

Miss Rate

9.9%

ROC-AUC

0.969

EER

9.1%

The F1 score remains 0.835 even at -5 dB SNR, where the noise is louder than the speech.

Making It Work in Real Time

Training a model is one thing. Making it work on a live microphone is another. The model is causal, so it does not need future audio. For streaming inference, I maintain two kinds of state:

Convolution context: previous mel frames required by the convolution layers.

GRU state: the recurrent memory from previous frames.

The important part is not sending the old frames through the GRU again. The GRU has already processed them. I also verified that streaming inference matches offline inference, with the difference kept below 1e-4.

From PyTorch to Production

The model is trained in PyTorch, but the production backend runs it using ONNX Runtime.

The final pipeline is: Audio → NumPy Frontend → ONNX Runtime → Speech Probability

I also export the exact frontend parameters used during training so the production system does not accidentally calculate different mel features. Before exporting the final model, I verify:

assert front_err < 1e-3
assert graph_err < 1e-3
assert e2e < 1e-3

This checks the NumPy frontend, ONNX model, and streaming pipeline against the original PyTorch implementation.

Source Code

The complete project is open source.

GitHub: https://github.com/Hemang-patel-9/vad-from-scratch

The trained model is also available to download, and the VAD can be tested through the live web application.

Final Thoughts

The biggest thing I learned from this project is that the model itself was only one part of the problem.

The harder questions were:

How do you create reliable labels?

How do you generate realistic training data?

How do you evaluate the model honestly?

How do you make offline and streaming inference behave the same?

How do you make sure the production model is actually the model you evaluated?

The neural network is relatively small. The engineering around it is where most of the interesting work happened. And that is probably the biggest lesson I took from building my own VAD:

Building the model is only one part of building the system.

ShareX / TwitterLinkedIn