Skip to content

feat: add senselab VAD, eGeMAPS, and embedding processors (#566)#1169

Open
Chelsy-AI wants to merge 2 commits into
sunlabuiuc:masterfrom
Chelsy-AI:feat/senselab-processor
Open

feat: add senselab VAD, eGeMAPS, and embedding processors (#566)#1169
Chelsy-AI wants to merge 2 commits into
sunlabuiuc:masterfrom
Chelsy-AI:feat/senselab-processor

Conversation

@Chelsy-AI

Copy link
Copy Markdown

Closes #566

Summary

Adds three FeatureProcessor subclasses wrapping senselab for voice/speech analysis in pyhealth/processors/senselab_processor.py:

  • SenselabVADProcessor — removes non-speech segments, returns speech-only waveform tensor
  • SenselabEGeMAPSProcessor — extracts 88 hand-crafted eGeMAPS acoustic features
  • SenselabEmbeddingProcessor — extracts deep neural embeddings via WavLM or ECAPA-TDNN

All three follow the existing AudioProcessor pattern: lazy imports, ImportError with install hint, process() takes a file path, returns a torch.Tensor. Processors are registered in the processor registry and exported from __init__.py.

Tests are in tests/processors/test_senselab_processor.py using pytest.importorskip("senselab") so they skip gracefully in environments where senselab is unavailable.

Note on dev environment

senselab>=1.3.0 requires torch>=2.8.0 with CUDA libraries (~3GB), which conflicts with available disk space on GitHub Codespaces alongside PyHealth's own dependencies. Local macOS testing is blocked by llvmlite 0.47 requiring LLVM 20 while Homebrew ships LLVM 22. Happy to test in a larger environment if maintainers can advise on the recommended dev setup for audio dependencies.

@jhnwu3

jhnwu3 commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator

Hey, do you mind giving more context for what this is for?

@Chelsy-AI

Copy link
Copy Markdown
Author

Hey, do you mind giving more context for what this is for?

Hi @jhnwu3 — happy to clarify!

This PR implements issue #566, which proposed integrating senselab into PyHealth as a set of audio processor wrappers. Senselab is an open-source package for standardized voice and speech analysis in health and clinical research contexts.

The three processors I added each wrap a different senselab capability:

SenselabVADProcessor — runs Voice Activity Detection to strip silence and noise, returning only the speech segments as a waveform tensor. Useful as a preprocessing step before feature extraction.

SenselabEGeMAPSProcessor — extracts the eGeMAPS feature set (88 hand-crafted acoustic features) widely used in clinical voice research for depression detection, Parkinson's monitoring, and cognitive assessment.

SenselabEmbeddingProcessor — extracts deep neural speaker embeddings using WavLM or ECAPA-TDNN, which capture paralinguistic features useful for voice biomarker tasks.

The motivation from the original issue was to extend PyHealth's existing sensor/EHR pipeline to support end-to-end voice biomarker workflows — so clinicians could run speech-based assessments (e.g. depression screening from voice) using the same processor interface as the rest of PyHealth.

All three follow the existing AudioProcessor pattern in the codebase — lazy imports, optional dependency, process() takes a file path and returns a torch.Tensor. Let me know if you'd like me to adjust the scope, API design, or add a usage example to the docstrings.

@jhnwu3

jhnwu3 commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Ahh, that's super cool. Can you follow the documentation requirements to add:

docs/.rst files for each processor
An example in examples/
And some test cases for these. (It can be as simple as just showing that they initialize and process).

Tbh, I think I'll need to update the website later to include easier to read documentation on contribution styling, etc.

@Chelsy-AI

Copy link
Copy Markdown
Author

Ahh, that's super cool. Can you follow the documentation requirements to add:

docs/.rst files for each processor An example in examples/ And some test cases for these. (It can be as simple as just showing that they initialize and process).

Tbh, I think I'll need to update the website later to include easier to read documentation on contribution styling, etc.

Done — added individual .rst files for each processor under docs/api/processors/, updated the toctree and Available Processors list in docs/api/processors.rst, and added examples/senselab_processor_tutorial.py showing initialization and processing for all three. Tests were already included in the original commit. Let me know if anything needs adjusting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate senselab for speech analysis in health

2 participants