ModelPipe runs a wide variety of ML models in GStreamer and Glass2Glass multimedia pipelines.
ML Engines
GStreamer Elements
Platforms
Same GStreamer elements, same pipelines. Pick the engine that matches your hardware via the engine-name property.
Default engine with full GPU and CPU support. TorchVision model zoo, HuggingFace Transformers, and Ultralytics YOLO — all battle-tested.
CPU and GPU inference via ONNX Runtime. Set engine-name=onnx and point to any .onnx model. Automatic YOLO anchor-free NMS decoding built in.
Intel-optimized inference on CPUs, iGPUs, and dGPUs. Set engine-name=openvino and point to an OpenVINO IR .xml model file.
NVIDIA TensorRT through ONNX Runtime. Set engine-name=onnx and device=tensorrt-fp16. The engine is built once and cached.
Lightweight inference for edge and mobile. Set engine-name=tflite and point to a .tflite model. NHWC input by default — no format flags needed.
SavedModel format for TensorFlow ecosystem compatibility. Set engine-name=tensorflow and point to a TensorFlow SavedModel directory.
Compiler-optimized inference via Apache TVM. Set engine-name=tvm and point to a compiled TVM .so module.
Minimal-dependency inference with tinygrad. Set engine-name=tinygrad for lightweight GPU/CPU execution with automatic kernel optimization.
Apple Silicon native inference via MLX. Set engine-name=mlx for unified memory GPU/CPU execution on M-series chips. Supports mlx-lm text generation.
Meta’s ExecuTorch runtime for on-device inference. Set engine-name=executorch and point to a .pte model.
GGUF model inference via llama-cpp-python. Set engine-name=llamacpp for efficient quantized LLM inference on CPU or GPU.
Google JAX with XLA compilation. Set engine-name=jax for JIT-compiled inference with automatic hardware acceleration on GPU, TPU, or CPU.
AMD’s graph inference engine for ROCm GPUs. Set engine-name=migraphx and point to an .onnx model. Optimized compilation with fp16 quantization support.
Compiler-based ML runtime (AMD/Google). Set engine-name=iree for optimized inference on ROCm, Vulkan, CUDA, or CPU via ahead-of-time compilation.
Lightweight inference with Vulkan GPU support. Set engine-name=ncnn for GPU inference on AMD, NVIDIA, or Intel — no ROCm or CUDA required.
Drop-in GStreamer elements for every ML task. Use them in any mp-launch pipeline or Python app.
mp_objectdetector
All TorchVision detection models plus ONNX, OpenVINO, TFLite, and TensorFlow engines. Faster R-CNN, SSD, and more.
Visionmp_yolo
YOLO11 object detection with built-in multi-object tracking. Real-time bounding boxes with track IDs.
Visionmp_maskrcnn
Mask R-CNN pixel-level instance segmentation with per-object masks and bounding boxes.
Visionmp_yolo_pose
COCO 17-keypoint skeleton detection with YOLO pose models. Optional in-element skeleton visualization.
Visionmp_depth
DepthAnything V2 monocular depth estimation. Multiple colormaps (inferno, jet, viridis, plasma, magma) and frame-stride support.
Visionmp_clip
CLIP and SigLIP zero-shot classification with user-defined labels. No fixed label set — set labels at pipeline launch.
Visionmp_classifier
All TorchVision classifiers including ResNet, EfficientNet, and more. GPU or CPU inference.
Visionmp_caption_qwen
Qwen2.5-VL vision-language model for video captioning and visual understanding. Supports caption history, LLM summarization, and AWQ quantization.
Visionmp_whispertranscribe
OpenAI Whisper-based transcription with language detection, translation to English, and initial prompt support.
Audiomp_whisperlive
Combined live transcription with LLM integration. Transcribe, translate, and generate speech in one pipeline.
Audiomp_vad
Silero VAD with configurable threshold. Standalone metadata mode or gating mode to mute silent audio.
Audiomp_whisperspeechtts
WhisperSpeech-based TTS for natural-sounding speech generation from text input.
Audiomp_demucs
Demucs source separation for isolating vocals from background audio. Clean input for downstream transcription.
Audiomp_mariantranslate
Marian machine translation. model-name is the Hugging Face model.
Languagemp_llm
Run HuggingFace LLMs (Phi-2, Qwen3, etc.) as GStreamer elements. Text-in, text-out with system prompts.
Languagemp_llm_remote
Send text to a remote LLM via HTTP. Works with Ollama, OpenAI-compatible APIs. No local GPU required.
Languagemp_stablediffusion
Stable Diffusion image generation from text prompts. Outputs PNG frames as GStreamer buffers.
Languagemp_overlay
Render detection metadata as bounding boxes, labels, and tracking IDs directly on video frames.
Utilitymp_kafkasink
Serialize model inference metadata to Apache Kafka with JSON schema validation.
Utilitymp_streammux
Multiplex multiple video streams into a single batched stream for efficient multi-stream inference.
Utilitymp_streamdemux
Split a batched stream back into individual streams for per-stream display or processing.
Utilitymp_inference
Run any model through any engine and inspect raw output. Useful for testing and debugging new models.
Utilitymp_video_transform
Pre/post-processing transforms for video frames within ML pipelines.
Utilitymp_sepformer
SpeechBrain SepFormer model for separating overlapping speakers. Clean single-speaker audio from mixed recordings.
Audiomp_sam
Meta SAM2 zero-shot segmentation. Point, box, or auto prompts. Outputs per-object masks as GstAnalytics metadata.
Visionmp_ocr
EasyOCR and TrOCR text detection and recognition. Outputs bounding boxes and recognized text in GstAnalytics metadata.
Visionmp_face
RetinaFace detection with ArcFace embedding-based recognition. Configurable face gallery for identification.
Visionmp_optical_flow
RAFT dense optical flow estimation. Outputs per-pixel motion vectors as color-coded visualization or raw flow fields.
Visionmp_superres
Real-ESRGAN image upscaling (2×/4×). Tile-based processing for arbitrary input sizes with configurable overlap.
Visionmp_action
SlowFast and X3D temporal action recognition over sliding windows. Classifies activities across configurable frame clips.
Visionmp_anomaly
PatchCore visual anomaly detection for manufacturing QA. Generates anomaly score heatmaps with configurable thresholds.
Visionmp_clap
LAION CLAP zero-shot audio classification. User-defined text labels matched against audio — no fixed label set.
Audiomp_vlm
Generic VLM element supporting LLaVA, InternVL, and more. Visual question answering with configurable prompts.
Visionmp_embedding
Extract dense vector embeddings from video frames using CLIP, DINOv2, or any vision backbone. Outputs to buffer metadata or file.
Visionmp_tracker
Standalone SORT/ByteTrack tracker. Reads upstream detection metadata and assigns persistent track IDs — works with any detector.
Utilitymp_alert
Rule-based alerting triggered by upstream detections. Webhook, MQTT, and metadata output with per-rule cooldowns.
Utilitymp_alertrecorder
Writes a video clip around each alert: a few seconds before, and until the alerts stop. Any encoder and muxer GStreamer has.
Utilitymp_zeroshotdetector
Type the classes: labels="forklift, hard hat". OWLv2 or Grounding DINO boxes flow into alerts, tracking and overlay with no training.
mp_metareplay
Reattaches a recorded JSON lines file to the frames, so tracker, alert and overlay run with no model and no GPU.
Utilitymp_embeddingsink
Stores CLIP frame embeddings in a local index. The MCP server's search_video finds moments by text.
mp_digest
Collects a time window of captions or transcripts into one text buffer, so an LLM summarises a minute rather than a line.
Utilitymp_metasink
One JSON line per frame: detections, alerts, depth and other blobs, transcripts. To a file, stdout, or the bus for the MCP server.
Utilitymp-mcp is a Model Context Protocol server over stdio. It runs a GStreamer pipeline in-process, so an agent in Claude Code, Claude Desktop or any MCP client can start a pipeline, read what it sees, and tune it while it runs.
start_pipeline, pipeline_status, stop_pipeline, latest_metadata, set_property, get_property, list_elements, inspect, search_video.
End the pipeline with mp_metasink. Every record it posts, detections, alerts, transcripts, reaches the agent through latest_metadata.
uv sync --extra mcp claude mcp add modelpipe -- \ /path/to/modelpipe/.venv/bin/mp-mcp
Production features that make ModelPipe more than a demo — it’s a framework.
Built-in tracking with mp_yolo. Set track=True and get persistent track IDs across frames with no extra elements.
Switch engines with a single property: engine-name=onnx, engine-name=openvino, engine-name=tflite, engine-name=tensorflow. Same pipeline, different hardware.
Use mp_streammux and mp_streamdemux to batch multiple video streams through a single model for GPU-efficient multi-camera inference.
Voice Activity Detection with gate=true silences non-speech audio before transcription, reducing Whisper latency and false positives.
Chain mp_caption_qwen (Qwen2.5-VL-3B-Instruct-AWQ) with coalescehistory and mp_llm (Qwen3-0.6B) for real-time video narration with caption history summarization.
Demucs vocal isolation (mp_demucs) cleans audio before transcription for dramatically better accuracy in noisy environments.
Chain mp_whispertranscribe → mp_mariantranslate → mp_whisperspeechtts for end-to-end speech-to-speech translation in one pipeline.
mp-mcp is an MCP server that runs a pipeline in-process. An LLM agent starts it, reads detections and transcripts from mp_metasink, and changes element properties while it runs.
only-on=alert on any video element runs it only on the frames a cheap detector flagged. YOLO decides, the VLM answers.
mp_embeddingsink indexes CLIP embeddings, and an agent asks search_video("a red van at the gate") over MCP to get timestamps back.
mp_alert tags the frame, mp_alertrecorder writes the seconds around it to a file. No external recorder, no polling.
Stream inference metadata to Kafka topics with mp_kafkasink. JSON schema validation, configurable broker and topic, Docker-ready setup.
Reduce compute with frame-stride=N on heavy elements like depth estimation. Process every Nth frame and pass others through unchanged.
Set batch-size=4 on detection elements to batch multiple frames into a single GPU call for higher throughput.
Set device=cuda or device=cuda:0 on any element. Automatic GPU memory management and model caching.
On the g2g backend, each Python element runs on its own OS thread. On free-threaded CPython 3.14t with the GIL off, those threads run Python at the same time. Experimental, tested with YOLO and mp_tracker on CUDA. Setup in FREE_THREADING.md.
Automatic YOLO anchor-free NMS decoding with post-process=anchor_free. No manual post-processing code required.
ModelPipe spans vision, audio, and language. The elements compose in GStreamer and Glass2Glass pipelines.
Faster R-CNN, SSD, YOLO11 with tracking. Five inference engines. GPU batch processing.
See elements →COCO 17-keypoint human pose with YOLO. Skeleton visualization or metadata-only mode.
See elements →DepthAnything V2 monocular depth. Side-by-side with original, multiple colormaps, frame stride.
See elements →CLIP and SigLIP for open-vocabulary classification. Custom labels, top-k, confidence thresholds.
See elements →Whisper transcription, live transcription, VAD gating, audio separation, and multi-language support.
See elements →Text generation with LLMs, image generation with Stable Diffusion, text-to-speech with multiple engines.
See elements →Real pipelines you can run right now. Every example works with mp-launch.
YOLO detections raise an alert, the recorder writes a clip, the sink logs every frame as JSON.
python mp-launch.py filesrc location=video.mp4 \
! decodebin ! videoconvert ! videoscale \
! video/x-raw,width=640,height=480 \
! mp_yolo model-name=yolo11m device=cuda \
! mp_alert rules='{"class":"person","min_score":0.7}' \
! mp_alertrecorder location=alert-%s.webm \
! mp_metasink location=people.jsonl
YOLO11 detection with multi-object tracking and overlay.
python mp-launch.py filesrc location=video.mp4 \
! decodebin ! videoconvertscale \
! video/x-raw,width=640,height=480 \
! mp_yolo model-name=yolo11m \
device=cuda:0 track=True \
! mp_overlay ! videoconvert \
! autovideosink
Object detection with torch.compile for optimized steady-state throughput.
python mp-launch.py filesrc location=video.mp4 \
! decodebin ! videoconvert ! videoscale \
! video/x-raw,width=640,height=480 \
! mp_objectdetector \
model-name=fasterrcnn_resnet50_fpn \
device=cuda compile=True \
! mp_overlay ! videoconvert \
! autovideosink
YOLO pose with skeleton visualization on frame.
python mp-launch.py filesrc location=video.mp4 \
! decodebin ! videoconvert \
! videoscale ! video/x-raw,width=640,height=480 \
! mp_yolo_pose model-name=yolo11n-pose \
device=cuda \
! videoconvert ! autovideosink
DepthAnything V2 monocular depth with colormap.
python mp-launch.py filesrc location=video.mp4 \
! decodebin ! videoconvert \
! videoscale ! video/x-raw,width=640,height=480 \
! mp_depth device=cuda \
model-name=depth-anything/\
Depth-Anything-V2-Small-hf \
! videoconvert ! autovideosink
Classify video frames with custom text labels.
python mp-launch.py filesrc location=video.mp4 \
! decodebin ! videoconvert \
! videoscale ! video/x-raw,width=640,height=480 \
! mp_clip device=cuda \
labels="person, car, dog, cat" \
top-k=3 \
! mp_overlay ! videoconvert \
! autovideosink
Korean audio to English text transcription.
python mp-launch.py filesrc location=audio.wav \
! decodebin ! audioconvert \
! mp_whispertranscribe \
device=cuda language=ko \
translate=yes \
! fakesink
Voice activity gating before Whisper for lower latency.
python mp-launch.py filesrc location=audio.wav \
! decodebin ! audioconvert \
! audioresample \
! audio/x-raw,format=S16LE,\
rate=16000,channels=1 \
! mp_vad threshold=0.6 gate=true \
! mp_whispertranscribe device=cuda \
! fakesink
Run a HuggingFace LLM as a GStreamer element.
python mp-launch.py filesrc location=prompt.txt \
! mp_llm device=cuda \
model-name="microsoft/phi-2" \
! fakesink
Text prompt to PNG image generation.
python mp-launch.py filesrc location=prompt.txt \ ! mp_stablediffusion device=cuda \ ! pngenc \ ! filesink location=output.png
Batch two video streams through one model.
python mp-launch.py \ filesrc location=cam1.mp4 ! decodebin \ ! videoconvertscale \ ! video/x-raw,width=640,height=480 \ ! mp_streammux name=mux \ filesrc location=cam2.mp4 ! decodebin \ ! videoconvertscale \ ! video/x-raw,width=640,height=480 \ ! mux. \ mux. ! mp_yolo device=cuda \ ! mp_streamdemux name=demux \ demux. ! autovideosink \ demux. ! autovideosink
Pick your platform and start building ML pipelines.
# Ubuntu 24.04 (Python 3.12) or Ubuntu 26.04+ (Python 3.14) # Install system packages sudo apt update && sudo apt -y upgrade sudo apt install -y python3-pip python3-venv \ gstreamer1.0-plugins-base gstreamer1.0-plugins-good \ gstreamer1.0-plugins-bad python3-gst-1.0 \ gstreamer1.0-python3-plugin-loader libcairo2 libcairo2-dev git # Clone and set up git clone https://github.com/collabora/modelpipe.git cd modelpipe # Create venv (must use system Python to match GStreamer) python3 -m venv --system-site-packages .venv source .venv/bin/activate # Install PyTorch with CUDA support pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128 # Install modelpipe pip install -e . # Optional: flash-attention (must match python/torch/cuda versions) # pip install ./flash_attn-*.whl # Set plugin path and run export GST_PLUGIN_PATH=$PWD/plugins:$GST_PLUGIN_PATH python mp-launch.py filesrc location=data/people.mp4 ! decodebin \ ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 \ ! mp_yolo model-name=yolo11m device=cuda ! mp_overlay \ ! videoconvert ! autovideosink
# Fedora 42+ (ships Python 3.14) # Install RPM Fusion (for NVIDIA) sudo dnf install https://download1.rpmfusion.org/free/fedora/\ rpmfusion-free-release-42.noarch.rpm \ https://download1.rpmfusion.org/nonfree/fedora/\ rpmfusion-nonfree-release-42.noarch.rpm sudo dnf install akmod-nvidia xorg-x11-drv-nvidia-cuda -y # Install system packages sudo dnf install -y python3-pip python3-devel \ cairo cairo-devel cairo-gobject-devel pkgconfig git \ gstreamer1-plugins-base gstreamer1-plugins-good \ gstreamer1-plugins-bad-free python3-gstreamer1 # Clone and set up git clone https://github.com/collabora/modelpipe.git cd modelpipe # Create venv (must use system Python to match GStreamer) python3 -m venv --system-site-packages .venv source .venv/bin/activate # Install PyTorch with CUDA support pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128 # Install modelpipe pip install -e . # Optional: flash-attention (must match python/torch/cuda versions) # pip install ./flash_attn-*.whl # Set plugin path and run export GST_PLUGIN_PATH=$PWD/plugins:$GST_PLUGIN_PATH
# Build Docker container (Fedora 42 — recommended, has Python 3.14) docker build -f ./Dockerfile_fedora42 -t fedora42:latest . # Or Ubuntu 26.04 (Python 3.14) docker build -f ./Dockerfile_ubuntu26 -t ubuntu26:latest . # Or Ubuntu 24.04 (Python 3.12) docker build -f ./Dockerfile_ubuntu24 -t ubuntu24:latest . # Run with GPU docker run -v ~/src/modelpipe/:/root/modelpipe \ -it --rm --gpus all --name fedora42 fedora42:latest /bin/bash # Run without GPU docker run -v ~/src/modelpipe/:/root/modelpipe \ -it --rm --name fedora42 fedora42:latest /bin/bash # Inside container: set up venv and install python3 -m venv --system-site-packages .venv source .venv/bin/activate pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128 pip install -e .
# PowerShell, with Python 3.10 or later installed # Clone and set up git clone https://github.com/collabora/modelpipe.git cd modelpipe python -m venv .venv .venv\Scripts\Activate.ps1 # Install GStreamer and its Python bindings from the GStreamer wheels pip install "gstreamer-meta[cli]" # Install modelpipe pip install -e . # The wheels set GST_PLUGIN_PATH_1_0, which hides GST_PLUGIN_PATH # The plugin loader embeds the base interpreter, so it needs PYTHONPATH too $venv = "$PWD\.venv" $env:GST_PLUGIN_PATH_1_0 = "$PWD\plugins" $env:PYTHONPATH = "$venv\Lib\site-packages;$venv\Lib\site-packages\gstreamer_python\Lib\site-packages;$PWD\plugins\python" # Check the elements load gst-inspect-1.0 mp_objectdetector # Optional: CUDA PyTorch, after the CUDA Toolkit matching your driver pip install torch torchvision torchaudio \ --index-url https://download.pytorch.org/whl/cu128
# Set up modelpipe as in the Ubuntu or Fedora tab and keep its venv active # Build g2g-launch-py from a glass2glass checkout next to modelpipe (needs Rust) cd .. git clone https://github.com/Glass2GlassHQ/glass2glass.git cd glass2glass PYO3_PYTHON=$(which python) cargo build --release -p g2g-python \ --features ml --bin g2g-launch-py cd ../modelpipe # Run any pipeline under glass2glass export MP_BACKEND=g2g python mp-launch.py filesrc location=data/people.mp4 ! decodebin \ ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 \ ! mp_yolo model-name=yolo11m device=cuda ! videoconvert ! mp_overlay \ ! videoconvert ! autovideosink
Create custom elements that inherit from ModelPipe base classes and run on either backend. Use the same inference engines and model loading code.
from base_objectdetector import BaseObjectDetector from gi.repository import GObject, Gst class MyDetector(BaseObjectDetector): __gstmetadata__ = ( "My Custom Detector", "Video/Filter", "A custom object detector", "Your Name", ) GObject.type_register(MyDetector) __gstelementfactory__ = ("my_detector", Gst.Rank.NONE, MyDetector)
# Framework first (provides base classes on sys.path), custom second export GST_PLUGIN_PATH=$HOME/src/modelpipe/plugins:$HOME/my_plugins:$GST_PLUGIN_PATH export PYTHONPATH=$HOME/my_plugins/python:$PYTHONPATH # Verify your element gst-inspect-1.0 my_detector
BaseObjectDetector · BaseTransform · BaseClassifier · BaseCaption · BaseLLM · BaseTranscribe · BaseTranslate · BaseTTS · BaseSeparate
Engine selection · model loading · device management · buffer handling · metadata propagation · all GObject properties
ModelPipe runs on Linux, Windows, and macOS with GStreamer, and on systems supported by Glass2Glass.
Native host installation with distribution GStreamer packages. Full GPU support with NVIDIA CUDA. Tested on Ubuntu 24.04, Ubuntu 26.04, and Fedora 42.
Pre-built Dockerfiles for Ubuntu 24.04, Ubuntu 26.04, and Fedora 42. NVIDIA Container Toolkit for GPU pass-through. CPU-only mode available.
GStreamer and its Python bindings from the gstreamer-meta wheels, with Python 3.10 or later. CUDA GPU support via PyTorch.
GStreamer and PyGObject from Homebrew. On Apple Silicon, the MLX engine and the ONNX Runtime CoreML provider with device=coreml.
See ModelPipe run real-time YOLO object tracking on a live soccer match.
ModelPipe is free, open source (MPL-2.0), and ready for production. Build ML-powered video and audio pipelines today.
Supported by Collabora · Open to contributions from everyone