AI Detection

Meridian VMS includes a hardware-accelerated object detection engine that analyzes camera frames in real time and generates detection events. The system automatically detects available hardware and uses the optimal inference backend.

Hardware Auto-Detection

On startup, the detection engine probes for available acceleration hardware and selects the best backend automatically:

Priority	Backend	Hardware	Performance
1	CUDA	NVIDIA GPU (Tesla, RTX, GTX)	Highest throughput, recommended
2	Hailo	Hailo-8/8L AI accelerator	Low power, good for edge deployments
3	Coral	Google Coral TPU (USB/PCIe)	Budget-friendly acceleration
4	CPU	Any x86_64 CPU	Slowest, suitable for small camera counts

The selected backend is logged at startup:

[Detection] Auto-detected backend: CUDA (NVIDIA Tesla T4, 16GB VRAM)
[Detection] Model loaded: object_detection_v3.onnx
[Detection] Ready for inference

To override auto-detection, set the preferred backend in the recording server’s detection configuration.

Shared Memory Pipeline

The detection system uses a shared memory (SHM) pipeline, eliminating data copies between capture and inference processes for maximum efficiency.

Pipeline Architecture

Camera --RTSP--> MediaMTX --RTSP--> Capture Process
                                        |
                                  [Shared Memory Buffer]
                                        |
                                   Inference Process --Events--> Management API
                                        |
                                   [GPU / Accelerator]

1. Capture processes decode RTSP frames and write raw pixels directly to shared memory buffers.
2. Inference processes read frames from shared memory, run the detection model, and produce events.
3. Shared memory avoids serialization and IPC overhead — frames are never copied between processes.

Configuration

The shared memory allocation is automatically sized based on the number of cameras and available system memory.

Multi-GPU Support

For large deployments, the detection engine distributes work across multiple GPUs:

GPU Assignment

Mode	Description
Auto	Cameras are distributed evenly across available GPUs
Manual	Assign specific cameras to specific GPUs in detection settings
Dedicated live	Reserve one GPU exclusively for live detection overlay

GPU assignment is configured through the recording server’s detection settings in the web interface.

Scaling

Cameras	Recommended GPUs	GPU Model
1-50	1x	NVIDIA Tesla T4 (16 GB)
50-150	2x	NVIDIA Tesla T4
150-300	3-4x	NVIDIA Tesla T4 or A10
300+	4x+	NVIDIA A10 or A100

Pooled RTSP Capture

Opening persistent RTSP connections to all cameras simultaneously consumes excessive memory. With 160 cameras and all connections open, memory usage peaked at over 42 GB.

Meridian VMS uses a pooled capture approach: the detection worker maintains a configurable pool of concurrent RTSP captures and rotates through cameras in round-robin batches.

The default pool size is 40 concurrent captures, which can be adjusted in the detection configuration.

With 160 cameras and a pool size of 40, the detector cycles through 4 batches. Memory usage drops from 42+ GB to approximately 3 GB.

Effective per-camera FPS

Detection FPS per camera depends on total camera count and pool size. With 160 cameras, a pool size of 40, and 2 FPS cycle rate, each camera is sampled at approximately 0.5 FPS — sufficient for surveillance event detection.

Detection Pipeline

The full detection pipeline for each frame:

1. Frame capture — Capture process grabs a frame from the RTSP stream relay.
2. SHM write — Frame is written to the shared memory ring buffer.
3. Batch inference — Inference process reads frames and runs the detection model. Multiple frames are batched for GPU efficiency.
4. Cross-class NMS — Non-maximum suppression is applied across classes to prevent duplicate detections where bounding boxes overlap significantly (e.g., a “person” and “pedestrian” detection on the same object).
5. Zone filtering — Detections are filtered against configured detection zones. Only detections whose center point falls within an enabled zone are kept.
6. Per-zone class filtering — Each zone’s class filter is checked. Detections of classes not in the zone’s filter list are discarded.
7. Confidence thresholds — Detections below the configured confidence threshold are dropped.
8. Event coalescing — Repeated detections of the same class on the same camera are coalesced into event windows, reducing noise.
9. Event submission — Events with annotated thumbnails (bounding boxes and labels) are batched and submitted to the management API.
10. Heatmap accumulation — All detections (including lower-confidence ones) are accumulated into per-camera, per-class heatmap grids and flushed every 5 minutes.

Detection Classes

The default model supports these surveillance-relevant classes:

Class	Description
person	People
car	Cars and sedans
truck	Trucks and large vehicles
bus	Buses
motorcycle	Motorcycles
bicycle	Bicycles
dog	Dogs
cat	Cats
bird	Birds

Additional classes from the training dataset are available but disabled by default. Class selection is configured per server in Settings > AI Detection.

Per-Camera Detection Toggle

Detection can be enabled or disabled per camera. When disabled, the capture pool skips that camera, freeing capacity for others.

Tip

Disabling detection on cameras that do not need it improves detection throughput on remaining cameras by reducing the rotation pool size.

Confidence Thresholds

Configure minimum confidence per class to reduce false positives:

Setting	Default	Description
Global threshold	0.45	Applied to all classes unless overridden
Per-class threshold	—	Override for specific classes (e.g., 0.6 for bird to reduce false positives)

Detections below the threshold are still accumulated for heatmaps but are not recorded as events.

Scheduled Training

The detection model can be fine-tuned on flagged false detections. See Custom Training for details on:

• Flagging false positives and false negatives from the Events page
• Scheduling training jobs during low-activity hours
• Model versioning and A/B testing
• Automatic rollback if the new model underperforms

Process Architecture

The detection worker runs as a separate process, fully isolated from the recorder daemon:

• Process isolation — independent process with its own memory space
• Memory isolation — crashes do not affect recording
• Independent restart — detection can restart without interrupting recordings
• Health monitoring — the recorder daemon monitors the detection process and restarts it on failure

Requirements

Component	Minimum	Recommended
GPU	NVIDIA with 4 GB VRAM	NVIDIA Tesla T4 (16 GB) or better
CUDA	11.8+	12.x
Driver	525+	Latest stable
Hailo	Hailo-8	Hailo-8L
Coral	USB Accelerator	PCIe Accelerator

Note

AI detection is optional. Recording servers without acceleration hardware function normally for recording and live streaming — only detection features are unavailable. CPU inference is available as a fallback but is not recommended for more than 10 cameras.

Next Steps

• Configuration — Enable detection and configure model, FPS, confidence, and class selection
• Detection Zones — Draw polygon zones to restrict where detections are triggered
• Custom Training — Fine-tune the detection model using flagged false detections
• Live Detection — Real-time detection overlay for monitoring and testing