Introducing SentrySearch: Semantic Video Search with Gemini Embedding 2

What Is SentrySearch?

SentrySearch transforms raw video files into searchable embeddings, allowing developers, data scientists, and video content managers to locate specific moments in massive video collections without manual scrubbing. By splitting videos into overlapping chunks, encoding each chunk directly as video with Gemini Embedding 2, and storing the vectors in a local Chroma DB integration, the tool bridges the gap between textual intent and visual content.

The project, announced on the UBOS news portal, targets three core audiences:

• Developers building AI‑enhanced video platforms.
• Data scientists needing fast, cost‑effective video indexing for research.
• Content managers who must retrieve specific scenes from large archives (e.g., dash‑cam footage, lecture recordings, marketing assets).

Key Features and Benefits

• Semantic Video Search

  Search by natural language (e.g., “red truck running a stop sign”) and receive the most relevant video clip.

• Tesla Metadata Overlay

  For Tesla dash‑cam recordings, SentrySearch can burn speed, GPS, and timestamp data directly onto the trimmed clip.

• Cost‑Optimized Indexing

  Built‑in still‑frame skipping and chunk‑size tuning keep Gemini API costs low (≈ $2.80 per hour of footage).

• Zero‑Transcription Required

  Unlike traditional video search that relies on speech‑to‑text, Gemini Embedding 2 embeds raw pixel data, eliminating transcription errors.

• Local Vector Store

  All embeddings are stored in a lightweight Chroma DB integration, enabling fast offline queries.

• Cross‑Platform Compatibility

  Works with any MP4 footage, not limited to Tesla Sentry Mode files.

Technical Architecture

SentrySearch is built on a modern Python stack (≥ 3.10) and leverages three core technologies:

1. Gemini Embedding 2 – Google’s multimodal model that projects raw video frames into a 768‑dimensional vector space shared with text embeddings.
2. FFmpeg (or bundled imageio‑ffmpeg) – Handles chunking, down‑scaling, and final clip trimming.
3. ChromaDB – An open‑source vector database that stores embeddings locally for instant similarity search.

Video Chunking Strategy

By default, SentrySearch creates 30‑second chunks with a 5‑second overlap. Overlap ensures that events spanning chunk boundaries are still discoverable. Users can adjust --chunk-duration and --overlap to balance cost vs. granularity.

Tesla Metadata Overlay Pipeline

When the --overlay flag is used, the tool extracts telemetry from Tesla’s proprietary Sentry Mode metadata (speed, GPS, timestamp). It then renders a HUD‑style overlay using ElevenLabs AI voice integration for optional audio narration (optional, not required for basic search).

Embedding Cost Model

Installation and Quick‑Start Guide

Prerequisites: Python 3.10+, Git, and either a system‑wide FFmpeg or the bundled imageio‑ffmpeg package.

Step‑by‑Step Setup

git clone https://github.com/ssrajadh/sentrysearch.git
cd sentrysearch
python -m venv venv && source venv/bin/activate
pip install -e .

Configure Gemini API Key (free tier available):

sentrysearch init
# Prompt: Enter your Gemini API key (get one at https://aistudio.google.com/apikey)

Indexing Your Video Library

Run the index command pointing to the folder that contains MP4 files:

sentrysearch index /path/to/your/video/footage

Typical output:

Indexing file 1/12: dashcam_2024-01-15.mp4 [chunk 1/4]
Indexing file 1/12: dashcam_2024-01-15.mp4 [chunk 2/4]
...
Indexed 48 new chunks from 12 files.

Performing a Semantic Search

A single‑line query returns the best matching clip:

sentrysearch search "red truck running a stop sign"

Sample result:

#1 [0.87] dashcam_2024-01-15.mp4 @ 02:15‑02:45
Saved clip: ./match_dashcam_2024-01-15_02m15s-02m45s.mp4

Adding Tesla Overlay (Optional)

sentrysearch search "car cutting me off" --overlay

The resulting clip includes speed, GPS coordinates, and timestamp rendered as a HUD overlay.

Cost and Performance Considerations

While the Gemini Embedding 2 API is priced per frame, SentrySearch’s default preprocessing dramatically reduces the number of frames sent for embedding. Below is a quick cost calculator for typical workloads:

• 1 hour of 30‑fps footage → 3,600 frames → $2.84 (baseline).
• Enabling --skip-still on idle dash‑cam footage can cut API calls by up to 70 %.
• Increasing --chunk-duration to 60 seconds halves the number of embeddings, reducing cost proportionally.

Performance‑wise, indexing 1 hour of video on a mid‑range laptop (Intel i5, 8 GB RAM) takes ~5 minutes. Search queries are sub‑second because they only involve a text embedding and a nearest‑neighbor lookup in ChromaDB.

How SentrySearch Stacks Up Against Competitors

For teams that need a cost‑effective, developer‑friendly solution, SentrySearch offers a unique blend of semantic precision and extensibility that most commercial platforms cannot match.

Get Started with SentrySearch Today

Ready to embed semantic video search into your product? Explore the full documentation on the UBOS platform overview and see how SentrySearch can complement other AI services such as the OpenAI ChatGPT integration.

If you’re a startup looking for rapid prototyping, check out the UBOS for startups page for special credits and sandbox environments.

Enterprises can leverage the Enterprise AI platform by UBOS to scale SentrySearch across thousands of video streams while maintaining governance and security.

Need a ready‑made UI? The Web app editor on UBOS lets you drag‑and‑drop the SentrySearch API into a low‑code dashboard in minutes.

For workflow automation, pair SentrySearch with the Workflow automation studio to trigger alerts when specific objects appear in new footage.

Pricing is transparent—review the UBOS pricing plans to understand compute and storage costs for large‑scale deployments.

Want inspiration? Browse the UBOS portfolio examples for real‑world use cases ranging from security monitoring to sports analytics.

Finally, explore the UBOS templates for quick start—including the AI Video Generator and AI Chatbot template—to accelerate your development cycle.

Dive into the source code, contribute, or raise issues on the official repository: SentrySearch GitHub.