Local Video Summarization Pipeline Using SmolVLM2-2.2B

Local Video Summarization Pipeline: Processing Frames with SmolVLM2-2.2B

Introduction

Most video understanding tools fall into one of two camps. The first camp requires a cloud API: your footage is uploaded, processed on someone else’s servers, and billed per minute of video. The second camp runs locally but demands the kind of GPU cluster most developers do not have — 70B+ models that need multiple A100s and take minutes per clip. Neither option works for a developer who wants to process a day’s worth of meeting recordings, a lecture series, or security footage on a workstation they already own.

SmolVLM2-2.2B-Instruct, released by Hugging Face on February 20, 2025, changes the calculation. It runs on 5.2 GB of GPU RAM, an RTX 3060, a MacBook Pro M2, and the free Google Colab T4 tier. On Video-MME, the standard long-form video understanding benchmark, it outperforms every existing 2B-scale model. That combination — consumer hardware paired with results that actually hold up — is what this article is built around.

The project we will build: a local pipeline that takes any video file, extracts frames at configurable intervals, analyzes them in batches with SmolVLM2-2.2B, and outputs a structured JSON summary — including per-frame scene descriptions, key moments with timestamps, action items, and a final narrative. The same pipeline handles meeting recordings, lectures, and surveillance footage without changing a line of code.

How SmolVLM2-2.2B Works

The reason SmolVLM2-2.2B can run on an RTX 3060 while outperforming larger models on video tasks is a design decision about how images are tokenized.

Most vision-language models tokenize images at high density. Qwen2-VL, for example, uses up to 16,000 tokens to represent a single image. Feeding 50 frames to such a model at that density would consume 800,000 tokens — far beyond any consumer GPU’s context budget. SmolVLM2 uses a pixel shuffle strategy that compresses each 384×384 image patch to 81 tokens. Fifty frames become approximately 4,050 image tokens, manageable in a single inference call. That compression is why SmolVLM2’s prefill throughput runs 3.3 to 4.5 times faster and generation throughput runs 7.5 to 16 times faster than Qwen2-VL-2B — not as a marketing claim but as a direct consequence of the token budget difference.

The model comes in three sizes. The 256M and 500M variants are designed for mobile and edge devices; the 256M can run on a phone. The 2.2B is the right choice for this pipeline. It is the only size with strong enough video benchmark scores to produce reliable multi-scene summaries: Video-MME of 52.1, MLVU of 55.2, and MVBench of 46.27, against the 500M’s 42.2, 47.3, and 39.73, respectively.

The video understanding approach is also worth understanding before you write any code. SmolVLM2 does not have a native video encoder; it treats video as a sequence of images. The official reference pipeline extracts up to 50 evenly sampled frames per video, bypasses internal frame resizing, and passes them as a multi-image sequence inside a single chat message. That approach scored 27.14% on CinePile, positioning it between InternVL2 (2B) and Video-LLaVA (7B) on cinematic video understanding — a strong result given the model’s size and that video was not the only thing it was trained for.

Setting Up the Environment

Hardware requirements:

Feature	Minimum	Recommended
GPU VRAM	6 GB (RTX 3060)	12–16 GB (RTX 4080)
Apple Silicon	M2 8 GB (MPS path)	M2 Pro / M3 16 GB
System RAM	16 GB	32 GB
Disk	10 GB free	20 GB+ SSD
Colab	T4 (free tier)	A100 (Colab Pro)

Python packages:

# Python 3.10+ required
python --version

python -m venv smolvlm2-env
source smolvlm2-env/bin/activate       # macOS / Linux
smolvlm2-env\Scripts\activate          # Windows

# Install from the stable SmolVLM-2 branch -- required for SmolVLM2 support
pip install git+https://github.com/huggingface/transformers@v4.49.0-SmolVLM-2

# Core dependencies
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install \
  opencv-python \
  Pillow \
  numpy \
  num2words \
  accelerate

# Flash Attention 2 for CUDA -- significantly faster on NVIDIA GPUs
# Skip this on Apple Silicon and CPU -- it is CUDA-only
pip install flash-attn --no-build-isolation

# decord -- required for SmolVLM2's native video input path (used in Section 5)
pip install decord

Note: The num2words package is a non-obvious dependency. SmolVLM2’s processor uses it to convert numeric digits to word representations (e.g. 3 → “three”) for consistency with natural language training patterns, as explained in this walkthrough. Omitting it causes a silent import error when the processor loads.

Device check (run this before loading the model):

# device_check.py
# Run: python device_check.py

import torch

def detect_device():
    if torch.cuda.is_available():
        name  = torch.cuda.get_device_name(0)
        vram  = torch.cuda.get_device_properties(0).total_memory / 1e9
        print(f"CUDA: {name} ({vram:.1f} GB VRAM)")
        return "cuda", torch.bfloat16, "flash_attention_2"
    elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
        print("Apple Silicon MPS detected")
        return "mps", torch.float16, "eager"
    else:
        print("CPU fallback (slow -- consider Colab T4)")
        return "cpu", torch.float32, "eager"

if __name__ == "__main__":
    device, dtype, attn = detect_device()
    print(f"Device: {device} | dtype: {dtype} | attn: {attn}")

Run it with:

python device_check.py

Building the Foundation of the Pipeline

Before SmolVLM2 sees anything, you need frames. The frame extractor converts a video file into a list of PIL (Python Imaging Library) images with timestamps attached — one pair per extracted frame.

Two modes matter for different use cases. Uniform sampling distributes frames evenly across the full video duration, guaranteeing coverage of everything regardless of content. This is the right choice for meetings and lectures where you cannot afford to miss a section. Keyframe sampling extracts frames only where the visual content changes significantly — such as scene cuts, a new slide, or a new speaker — which reduces the frame count and focuses attention on distinct moments. This is better for surveillance and highlight extraction.

# frame_extractor.py
# Prerequisites: pip install opencv-python Pillow numpy
# Usage: from frame_extractor import FrameExtractor

import cv2
import numpy as np
from PIL import Image


class FrameExtractor:
    """
    Extracts video frames as PIL Images for SmolVLM2 inference.
    Each extracted frame is paired with its timestamp in seconds.

    SmolVLM2 uses ~81 visual tokens per image. At 50 frames that is
    roughly 4,050 image tokens -- the practical upper limit before VRAM
    pressure affects generation quality on consumer GPUs.
    """

    MAX_FRAMES = 50

    def __init__(self, max_frames: int = MAX_FRAMES):
        """
        Args:
            max_frames: Hard cap on extracted frames. Default 50 matches
                        the SmolVLM2 reference pipeline's tested upper limit.
        """
        self.max_frames = max_frames

    def uniform_sample(self, video_path: str) -> list[tuple[float, Image.Image]]:
        """
        Extract evenly spaced frames across the full video duration.
        Best for: meeting recordings, lectures, tutorials, course content.

        Returns:
            List of (timestamp_seconds, PIL_Image) in chronological order.
        """
        cap = cv2.VideoCapture(video_path)
        if not cap.isOpened():
            raise IOError(f"Cannot open video: {video_path}")

        total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        fps          = cap.get(cv2.CAP_PROP_FPS) or 30.0
        n_extract    = min(self.max_frames, total_frames)

        # Build frame indices spread evenly from first to last frame
        indices = np.linspace(0, total_frames - 1, n_extract, dtype=int)
        results = []

        for idx in indices:
            cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
            ret, frame = cap.read()
            if not ret:
                continue
            timestamp = round(idx / fps, 2)
            rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            results.append((timestamp, Image.fromarray(rgb)))

        cap.release()
        return results

    def keyframe_sample(
        self, video_path: str, diff_threshold: float = 30.0
    ) -> list[tuple[float, Image.Image]]:
        """
        Extract frames where visual content changes significantly.
        Best for: surveillance, event detection, highlight extraction.

        Uses mean absolute pixel difference between consecutive grayscale frames
        as the change signal. When the diff exceeds diff_threshold, a new
        keyframe is recorded.

        Args:
            diff_threshold: Mean pixel difference to treat as a scene change.
                            30.0 works for most commercial content.
                            Lower = more sensitive, higher = fewer frames.

        Returns:
            List of (timestamp_seconds, PIL_Image) in chronological order,
            capped at self.max_frames.
        """
        cap = cv2.VideoCapture(video_path)
        if not cap.isOpened():
            raise IOError(f"Cannot open video: {video_path}")

        fps       = cap.get(cv2.CAP_PROP_FPS) or 30.0
        results   = []
        prev_gray = None
        idx       = 0

        while len(results) < self.max_frames:
            ret, frame = cap.read()
            if not ret:
                break

            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            if prev_gray is None:
                # Always capture the first frame as a baseline
                rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                results.append((round(idx / fps, 2), Image.fromarray(rgb)))
            else:
                diff = np.mean(np.abs(gray.astype(float) - prev_gray.astype(float)))
                if diff > diff_threshold:
                    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                    results.append((round(idx / fps, 2), Image.fromarray(rgb)))

            prev_gray = gray
            idx += 1

        cap.release()
        return results

With uniform and keyframe extraction in place, the pipeline has everything it needs to feed frames to SmolVLM2-2.2B. The uniform mode ensures full coverage for structured content like meetings and lectures, while keyframe mode keeps the frame count lean for content-dense or event-driven footage. Both modes respect the 50-frame ceiling that keeps inference within consumer GPU memory limits, making the pipeline practical to run entirely on hardware you already own.