GStreamer Python ML — Pure Python ML Framework for GStreamer

Engines

Choose Your ML Engine

Same GStreamer elements, same pipelines. Pick the engine that matches your hardware via the engine-name property.

🔥

PyTorch

Default engine with full GPU and CPU support. TorchVision model zoo, HuggingFace Transformers, and Ultralytics YOLO — all battle-tested.

⚡

ONNX Runtime

CPU and GPU inference via ONNX Runtime. Set engine-name=onnx and point to any .onnx model. Automatic YOLO anchor-free NMS decoding built in.

🔷

OpenVINO

Intel-optimized inference on CPUs, iGPUs, and dGPUs. Set engine-name=openvino and point to an OpenVINO IR .xml model file.

📡

LiteRT (TFLite)

Lightweight inference for edge and mobile. Set engine-name=tflite and point to a .tflite model. NHWC input by default — no format flags needed.

📊

TensorFlow

SavedModel format for TensorFlow ecosystem compatibility. Set engine-name=tensorflow and point to a TensorFlow SavedModel directory.

🚀

Apache TVM

Compiler-optimized inference via Apache TVM. Set engine-name=tvm and point to a compiled TVM .so module.

💫

tinygrad

Minimal-dependency inference with tinygrad. Set engine-name=tinygrad for lightweight GPU/CPU execution with automatic kernel optimization.

🍏

Apple MLX

Apple Silicon native inference via MLX. Set engine-name=mlx for unified memory GPU/CPU execution on M-series chips. Supports mlx-lm text generation.

📱

ExecuTorch

Meta’s ExecuTorch runtime for on-device inference. Set engine-name=executorch and point to a .pte model.

🤖

llama.cpp

GGUF model inference via llama-cpp-python. Set engine-name=llamacpp for efficient quantized LLM inference on CPU or GPU.

🕯

Candle

HuggingFace Candle Rust-based inference via Python bindings. Set engine-name=candle for minimal-overhead safetensors model execution.

🔬

JAX / Flax

Google JAX with XLA compilation. Set engine-name=jax for JIT-compiled inference with automatic hardware acceleration on GPU, TPU, or CPU.

🔴

AMD MiGraphX

AMD’s graph inference engine for ROCm GPUs. Set engine-name=migraphx and point to an .onnx model. Optimized compilation with fp16 quantization support.

⚙

IREE

Compiler-based ML runtime (AMD/Google). Set engine-name=iree for optimized inference on ROCm, Vulkan, CUDA, or CPU via ahead-of-time compilation.

🔮

NCNN (Vulkan)

Lightweight inference with Vulkan GPU support. Set engine-name=ncnn for GPU inference on AMD, NVIDIA, or Intel — no ROCm or CUDA required.

Elements

Native GStreamer Elements

Drop-in GStreamer elements for every ML task. Use them in any pyml-launch pipeline or Python app.

👁 pyml_objectdetector

Object Detection

All TorchVision detection models plus ONNX, OpenVINO, TFLite, and TensorFlow engines. Faster R-CNN, SSD, and more.

Vision

🎯 pyml_yolo

YOLO Detection + Tracking

YOLO11 object detection with built-in multi-object tracking. Real-time bounding boxes with track IDs.

Vision

🎭 pyml_maskrcnn

Instance Segmentation

Mask R-CNN pixel-level instance segmentation with per-object masks and bounding boxes.

Vision

🤸 pyml_yolo_pose

Pose Estimation

COCO 17-keypoint skeleton detection with YOLO pose models. Optional in-element skeleton visualization.

Vision

🌋 pyml_depth

Monocular Depth

DepthAnything V2 monocular depth estimation. Multiple colormaps (inferno, jet, viridis, plasma, magma) and frame-stride support.

Vision

📚 pyml_clip

Zero-Shot Classification

CLIP and SigLIP zero-shot classification with user-defined labels. No fixed label set — set labels at pipeline launch.

Vision

📈 pyml_classifier

Image Classification

All TorchVision classifiers including ResNet, EfficientNet, and more. GPU or CPU inference.

Vision

💬 pyml_caption_qwen

Video Captioning (Qwen VL)

Qwen2.5-VL vision-language model for video captioning and visual understanding. Supports caption history, LLM summarization, and AWQ quantization.

Vision

🎙 pyml_whispertranscribe

Speech Transcription

OpenAI Whisper-based transcription with language detection, translation to English, and initial prompt support.

Audio

📡 pyml_whisperlive

Live Transcription + LLM

Combined live transcription with LLM integration. Transcribe, translate, and generate speech in one pipeline.

Audio

🔊 pyml_vad

Voice Activity Detection

Silero VAD with configurable threshold. Standalone metadata mode or gating mode to mute silent audio.

Audio

🗣 pyml_coquitts

Text-to-Speech (Coqui)

Coqui TTS engine for high-quality speech synthesis from transcribed or translated text.

Audio

🔈 pyml_whisperspeechtts

Text-to-Speech (WhisperSpeech)

WhisperSpeech-based TTS for natural-sounding speech generation from text input.

Audio

🎵 pyml_demucs

Audio Separation

Demucs source separation for isolating vocals from background audio. Clean input for downstream transcription.

Audio

🌐 pyml_mariantranslate

Translation

Helsinki-NLP Marian machine translation. Hundreds of language pairs from HuggingFace model hub.

Language

🧠 pyml_llm

Large Language Models

Run HuggingFace LLMs (Phi-2, Qwen3, etc.) as GStreamer elements. Text-in, text-out with system prompts.

Language

🌐 pyml_llm_remote

Remote LLM (Ollama)

Send text to a remote LLM via HTTP. Works with Ollama, OpenAI-compatible APIs. No local GPU required.

Language

🎨 pyml_stablediffusion

Text-to-Image

Stable Diffusion image generation from text prompts. Outputs PNG frames as GStreamer buffers.

Language

📦 pyml_overlay

Bounding Box Overlay

Render detection metadata as bounding boxes, labels, and tracking IDs directly on video frames.

Utility

📨 pyml_kafkasink

Kafka Metadata Sink

Serialize model inference metadata to Apache Kafka with JSON schema validation.

Utility

🔀 pyml_streammux

Stream Multiplexer

Multiplex multiple video streams into a single batched stream for efficient multi-stream inference.

Utility

🔁 pyml_streamdemux

Stream Demultiplexer

Split a batched stream back into individual streams for per-stream display or processing.

Utility

🔭 pyml_inference

Generic Inference

Run any model through any engine and inspect raw output. Useful for testing and debugging new models.

Utility

📷 pyml_video_transform

Video Transform

Pre/post-processing transforms for video frames within ML pipelines.

Utility

🎶 pyml_sepformer

Speech Separation

SpeechBrain SepFormer model for separating overlapping speakers. Clean single-speaker audio from mixed recordings.

Audio

⛏ pyml_sam

Segment Anything

Meta SAM2 zero-shot segmentation. Point, box, or auto prompts. Outputs per-object masks as GstAnalytics metadata.

Vision

📜 pyml_ocr

Optical Character Recognition

EasyOCR and TrOCR text detection and recognition. Outputs bounding boxes and recognized text in GstAnalytics metadata.

Vision

👤 pyml_face

Face Detection & Recognition

RetinaFace detection with ArcFace embedding-based recognition. Configurable face gallery for identification.

Vision

🌀 pyml_optical_flow

Optical Flow

RAFT dense optical flow estimation. Outputs per-pixel motion vectors as color-coded visualization or raw flow fields.

Vision

🔍 pyml_superres

Super-Resolution

Real-ESRGAN image upscaling (2×/4×). Tile-based processing for arbitrary input sizes with configurable overlap.

Vision

🎬 pyml_action

Action Recognition

SlowFast and X3D temporal action recognition over sliding windows. Classifies activities across configurable frame clips.

Vision

🚨 pyml_anomaly

Anomaly Detection

PatchCore visual anomaly detection for manufacturing QA. Generates anomaly score heatmaps with configurable thresholds.

Vision

🎧 pyml_clap

Audio Classification (CLAP)

LAION CLAP zero-shot audio classification. User-defined text labels matched against audio — no fixed label set.

Audio

👀 pyml_vlm

Vision-Language Model

Generic VLM element supporting LLaVA, InternVL, and more. Visual question answering with configurable prompts.

Vision

💾 pyml_embedding

Embedding Extractor

Extract dense vector embeddings from video frames using CLIP, DINOv2, or any vision backbone. Outputs to buffer metadata or file.

Vision

🔎 pyml_tracker

Multi-Object Tracker

Standalone SORT/ByteTrack tracker. Reads upstream detection metadata and assigns persistent track IDs — works with any detector.

Utility

🚨 pyml_alert

ML Alert

Rule-based alerting triggered by upstream detections. Webhook, MQTT, and metadata output with per-rule cooldowns.

Utility

Advanced Features

Built for Real Pipelines

Production features that make GStreamer Python ML more than a demo — it’s a framework.

🎯 Multi-Object Tracking

Built-in tracking with pyml_yolo. Set track=True and get persistent track IDs across frames with no extra elements.

📊 Multi-Engine Support

Switch engines with a single property: engine-name=onnx, engine-name=openvino, engine-name=tflite, engine-name=tensorflow. Same pipeline, different hardware.

🔀 Multi-Stream Batching

Use pyml_streammux and pyml_streamdemux to batch multiple video streams through a single model for GPU-efficient multi-camera inference.

🔊 VAD Gating

Voice Activity Detection with gate=true silences non-speech audio before transcription, reducing Whisper latency and false positives.

💬 Qwen VL + LLM Pipelines

Chain pyml_caption_qwen (Qwen2.5-VL-3B-Instruct-AWQ) with coalescehistory and pyml_llm (Qwen3-0.6B) for real-time video narration with caption history summarization.

🎵 Audio Separation

Demucs vocal isolation (pyml_demucs) cleans audio before transcription for dramatically better accuracy in noisy environments.

🌐 Transcribe → Translate → Speak

Chain pyml_whispertranscribe → pyml_mariantranslate → pyml_coquitts for end-to-end speech-to-speech translation in one pipeline.

📨 Kafka Integration

Stream inference metadata to Kafka topics with pyml_kafkasink. JSON schema validation, configurable broker and topic, Docker-ready setup.

⚡ Frame Stride

Reduce compute with frame-stride=N on heavy elements like depth estimation. Process every Nth frame and pass others through unchanged.

📊 Batch Processing

Set batch-size=4 on detection elements to batch multiple frames into a single GPU call for higher throughput.

🔥 CUDA Acceleration

Set device=cuda or device=cuda:0 on any element. Automatic GPU memory management and model caching.

🔬 ONNX Post-Processing

Automatic YOLO anchor-free NMS decoding with post-process=anchor_free. No manual post-processing code required.

Example Pipelines

Copy, Paste, Run

Real pipelines you can run right now. Every example works with pyml-launch.

Vision

YOLO + Tracking

YOLO11 detection with multi-object tracking and overlay.

python pyml-launch.py filesrc location=video.mp4 \
  ! decodebin ! videoconvertscale \
  ! video/x-raw,width=640,height=480 \
  ! pyml_yolo model-name=yolo11m \
    device=cuda:0 track=True \
  ! pyml_overlay ! videoconvert \
  ! autovideosink

Vision

Torch Compile

Object detection with torch.compile for optimized steady-state throughput.

python pyml-launch.py filesrc location=video.mp4 \
  ! decodebin ! videoconvert ! videoscale \
  ! video/x-raw,width=640,height=480 \
  ! pyml_objectdetector \
    model-name=fasterrcnn_resnet50_fpn \
    device=cuda compile=True \
  ! pyml_overlay ! videoconvert \
  ! autovideosink

Vision

Pose Estimation

YOLO pose with skeleton visualization on frame.

python pyml-launch.py filesrc location=video.mp4 \
  ! decodebin ! videoconvert \
  ! videoscale ! video/x-raw,width=640,height=480 \
  ! pyml_yolo_pose model-name=yolo11n-pose \
    device=cuda \
  ! videoconvert ! autovideosink

Vision

Depth Estimation

DepthAnything V2 monocular depth with colormap.

python pyml-launch.py filesrc location=video.mp4 \
  ! decodebin ! videoconvert \
  ! videoscale ! video/x-raw,width=640,height=480 \
  ! pyml_depth device=cuda \
    model-name=depth-anything/\
    Depth-Anything-V2-Small-hf \
  ! videoconvert ! autovideosink

Vision

Zero-Shot CLIP

Classify video frames with custom text labels.

python pyml-launch.py filesrc location=video.mp4 \
  ! decodebin ! videoconvert \
  ! videoscale ! video/x-raw,width=640,height=480 \
  ! pyml_clip device=cuda \
    labels="person, car, dog, cat" \
    top-k=3 \
  ! pyml_overlay ! videoconvert \
  ! autovideosink

Audio

Transcribe + Translate

Korean audio to English text transcription.

python pyml-launch.py filesrc location=audio.wav \
  ! decodebin ! audioconvert \
  ! pyml_whispertranscribe \
    device=cuda language=ko \
    translate=yes \
  ! fakesink

Audio

VAD + Transcription

Voice activity gating before Whisper for lower latency.

python pyml-launch.py filesrc location=audio.wav \
  ! decodebin ! audioconvert \
  ! audioresample \
  ! audio/x-raw,format=S16LE,\
    rate=16000,channels=1 \
  ! pyml_vad threshold=0.6 gate=true \
  ! pyml_whispertranscribe device=cuda \
  ! fakesink

Language

LLM Text Generation

Run a HuggingFace LLM as a GStreamer element.

python pyml-launch.py filesrc location=prompt.txt \
  ! pyml_llm device=cuda \
    model-name="microsoft/phi-2" \
  ! fakesink

Language

Stable Diffusion

Text prompt to PNG image generation.

python pyml-launch.py filesrc location=prompt.txt \
  ! pyml_stablediffusion device=cuda \
  ! pngenc \
  ! filesink location=output.png

Multi-Stream

Mux/Demux Pipeline

Batch two video streams through one model.

python pyml-launch.py \
  filesrc location=cam1.mp4 ! decodebin \
  ! videoconvertscale \
  ! video/x-raw,width=640,height=480 \
  ! pyml_streammux name=mux \
  filesrc location=cam2.mp4 ! decodebin \
  ! videoconvertscale \
  ! video/x-raw,width=640,height=480 \
  ! mux. \
  mux. ! pyml_yolo device=cuda \
  ! pyml_streamdemux name=demux \
  demux. ! autovideosink \
  demux. ! autovideosink

Quick Start

Up and Running in Minutes

Pick your platform and start building ML pipelines.

                
                
                
                
            

# Ubuntu 24.04 (Python 3.12) or Ubuntu 26.04+ (Python 3.14)
# Install system packages
sudo apt update && sudo apt -y upgrade
sudo apt install -y python3-pip python3-venv \
    gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad python3-gst-1.0 \
    gstreamer1.0-python3-plugin-loader libcairo2 libcairo2-dev git

# Clone and set up
git clone https://github.com/collabora/gst-python-ml.git
cd gst-python-ml

# Create venv (must use system Python to match GStreamer)
python3 -m venv --system-site-packages .venv
source .venv/bin/activate

# Install PyTorch with CUDA support
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

# Install gst-python-ml
pip install -e .

# Optional: flash-attention (must match python/torch/cuda versions)
# pip install ./flash_attn-*.whl

# Set plugin path and run
export GST_PLUGIN_PATH=$PWD/plugins:$GST_PLUGIN_PATH
python pyml-launch.py filesrc location=data/people.mp4 ! decodebin \
  ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 \
  ! pyml_yolo model-name=yolo11m device=cuda ! pyml_overlay \
  ! videoconvert ! autovideosink
                

# Fedora 42+ (ships Python 3.14)
# Install RPM Fusion (for NVIDIA)
sudo dnf install https://download1.rpmfusion.org/free/fedora/\
  rpmfusion-free-release-42.noarch.rpm \
  https://download1.rpmfusion.org/nonfree/fedora/\
  rpmfusion-nonfree-release-42.noarch.rpm
sudo dnf install akmod-nvidia xorg-x11-drv-nvidia-cuda -y

# Install system packages
sudo dnf install -y python3-pip python3-devel \
    cairo cairo-devel cairo-gobject-devel pkgconfig git \
    gstreamer1-plugins-base gstreamer1-plugins-good \
    gstreamer1-plugins-bad-free python3-gstreamer1

# Clone and set up
git clone https://github.com/collabora/gst-python-ml.git
cd gst-python-ml

# Create venv (must use system Python to match GStreamer)
python3 -m venv --system-site-packages .venv
source .venv/bin/activate

# Install PyTorch with CUDA support
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

# Install gst-python-ml
pip install -e .

# Optional: flash-attention (must match python/torch/cuda versions)
# pip install ./flash_attn-*.whl

# Set plugin path and run
export GST_PLUGIN_PATH=$PWD/plugins:$GST_PLUGIN_PATH
                

# Build Docker container (Fedora 42 — recommended, has Python 3.14)
docker build -f ./Dockerfile_fedora42 -t fedora42:latest .

# Or Ubuntu 26.04 (Python 3.14)
docker build -f ./Dockerfile_ubuntu26 -t ubuntu26:latest .

# Or Ubuntu 24.04 (Python 3.12)
docker build -f ./Dockerfile_ubuntu24 -t ubuntu24:latest .

# Run with GPU
docker run -v ~/src/gst-python-ml/:/root/gst-python-ml \
  -it --rm --gpus all --name fedora42 fedora42:latest /bin/bash

# Run without GPU
docker run -v ~/src/gst-python-ml/:/root/gst-python-ml \
  -it --rm --name fedora42 fedora42:latest /bin/bash

# Inside container: set up venv and install
python3 -m venv --system-site-packages .venv
source .venv/bin/activate
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip install -e .
                

# 1. Install GStreamer MSVC x86_64 (runtime + development)
#    https://gstreamer.freedesktop.org/download/#windows

# 2. Set environment variables
[Environment]::SetEnvironmentVariable("PATH",
  "C:\gstreamer\1.0\msvc_x86_64\bin;" + $env:PATH, "User")
[Environment]::SetEnvironmentVariable("GST_PLUGIN_PATH",
  "D:\gst-python-ml\plugins", "User")

# 3. Install Python 3.12+ and PyGObject
pip install gstreamer-python

# 4. Install CUDA PyTorch (optional)
pip install torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/cu128
                

Extensibility

Build Your Own Plugins

Create custom GStreamer elements that inherit from gst-python-ml base classes. Full access to the engine abstraction, model loading, and pipeline integration.

my_plugins/python/my_detector.py

from base_objectdetector import BaseObjectDetector
from gi.repository import GObject, Gst

class MyDetector(BaseObjectDetector):
    __gstmetadata__ = (
        "My Custom Detector", "Video/Filter",
        "A custom object detector", "Your Name",
    )

GObject.type_register(MyDetector)
__gstelementfactory__ = ("my_detector", Gst.Rank.NONE, MyDetector)

Environment Setup

# Framework first (provides base classes on sys.path), custom second
export GST_PLUGIN_PATH=$HOME/src/gst-python-ml/plugins:$HOME/my_plugins:$GST_PLUGIN_PATH
export PYTHONPATH=$HOME/my_plugins/python:$PYTHONPATH

# Verify your element
gst-inspect-1.0 my_detector

Base Classes

Available for Inheritance

BaseObjectDetector · BaseTransform · BaseClassifier · BaseCaption · BaseLLM · BaseTranscribe · BaseTranslate · BaseTTS · BaseSeparate

Inherited Features

Zero Boilerplate

Engine selection · model loading · device management · buffer handling · metadata propagation · all GObject properties

Pure Python ML forGStreamer

15

39

3

Choose Your ML Engine

PyTorch

ONNX Runtime

OpenVINO

LiteRT (TFLite)

TensorFlow

Apache TVM

tinygrad

Apple MLX

ExecuTorch

llama.cpp

Candle

JAX / Flax

AMD MiGraphX

IREE

NCNN (Vulkan)

Native GStreamer Elements

Object Detection

YOLO Detection + Tracking

Instance Segmentation

Pose Estimation

Monocular Depth

Zero-Shot Classification

Image Classification

Video Captioning (Qwen VL)

Speech Transcription

Live Transcription + LLM

Voice Activity Detection

Text-to-Speech (Coqui)

Text-to-Speech (WhisperSpeech)

Audio Separation

Translation

Large Language Models

Remote LLM (Ollama)

Text-to-Image

Bounding Box Overlay

Kafka Metadata Sink

Stream Multiplexer

Stream Demultiplexer

Generic Inference

Video Transform

Speech Separation

Segment Anything

Optical Character Recognition

Face Detection & Recognition

Optical Flow

Super-Resolution

Action Recognition

Anomaly Detection

Audio Classification (CLAP)

Vision-Language Model

Embedding Extractor

Multi-Object Tracker

ML Alert

Built for Real Pipelines

🎯 Multi-Object Tracking

📊 Multi-Engine Support

🔀 Multi-Stream Batching

🔊 VAD Gating

💬 Qwen VL + LLM Pipelines

🎵 Audio Separation

🌐 Transcribe → Translate → Speak

📨 Kafka Integration

⚡ Frame Stride

📊 Batch Processing

🔥 CUDA Acceleration

🔬 ONNX Post-Processing

From Pixels to Understanding

Object Detection

Pose Estimation

Depth Estimation

Zero-Shot Classification

Speech & Transcription

LLMs & Generation

Copy, Paste, Run

YOLO + Tracking

Pure Python ML for
GStreamer