Related Pages

Related topics: System Architecture, Clip System

Overview

What is OpenVideo?

OpenVideo is a powerful, browser-based video composition and rendering library that enables developers to create, edit, and export video content directly in the web environment. Built on modern web APIs, OpenVideo provides a high-performance alternative to traditional desktop video editing software while maintaining the accessibility and portability of web applications.

The library leverages hardware-accelerated video processing through the WebCodecs API, combined with PixiJS for robust 2D/3D rendering, making it suitable for both client-side video editing interfaces and server-side rendering pipelines.

Core Architecture

OpenVideo follows a modular architecture with three primary layers:

graph TD
    A[User Application] --> B[Studio Layer]
    B --> C[Compositor Layer]
    C --> D[Rendering Engine<br/>PixiJS + WebCodecs]
    
    B --> E[Clip Objects]
    E --> E1[Video]
    E --> E2[Audio]
    E --> E3[Text]
    E --> E4[Image]
    E --> E5[Caption]
    
    C --> F[Compositor]
    F --> F1[Video Export]
    F --> F2[Playback]
    F --> F3[Seeking]

Core Components

Technology Stack

OpenVideo is built on three foundational technologies that work together to provide high-performance video processing:

graph LR
    A[WebCodecs API] --> D[Video Processing]
    B[PixiJS] --> E[2D/3D Rendering]
    C[wrapbox] --> F[MP4 Muxing]
    
    D --> G[OpenVideo Core]
    E --> G
    F --> G

Sources: README.md:15-17

Key Features

Multi-Media Clip Support

OpenVideo supports a comprehensive set of clip types for video composition:

• Video Clip - Load and manipulate video files with frame-accurate control
• Audio Clip - Audio tracks with volume control and synchronization
• Text Clip - Rich text rendering with fonts, styles, and animations
• Image Clip - Static image support for backgrounds and overlays
• Caption Clip - Subtitle and caption rendering with styling options

Visual Transitions

The library includes numerous built-in transitions for professional video effects:

Sources: packages/openvideo/src/transition/transition.ts:1-100

Animation Presets

Clips can be animated using preset animations:

• Fade Effects - fadeIn, fadeOut
• Movement - slideIn, slideOut, pulse
• Transformations - blurIn, blurOut, scaleIn, scaleOut
• Rotation - rotateIn, rotateOut

Sources: packages/openvideo/src/animation/presets.ts:1-80

Package Structure

The OpenVideo repository is organized as a monorepo with multiple packages:

openvideo/
├── packages/
│   ├── openvideo/          # Core library
│   │   ├── src/
│   │   │   ├── clips/      # Clip implementations
│   │   │   ├── transition/ # Transition effects
│   │   │   ├── animation/  # Animation presets
│   │   │   ├── studio.ts   # Main studio class
│   │   │   └── compositor.ts # Rendering engine
│   │   └── dist/           # Built output
│   │
│   ├── node/               # Node.js server-side renderer
│   │   ├── src/
│   │   │   ├── renderer.ts # Main Renderer class
│   │   │   ├── types.ts    # TypeScript definitions
│   │   │   └── template.html # Browser rendering template
│   │   └── dist/           # Built output
│   │
│   └── web/                # Web UI components
│
├── examples/                # Example applications
└── README.md

Core Library (`packages/openvideo`)

The main library providing all core functionality:

import { Studio, Video, Text, Image } from 'openvideo';

// Initialize Studio
const studio = new Studio({
  canvas: document.getElementById('preview-canvas') as HTMLCanvasElement,
  spacing: 20
});

// Add clips
const video = await Video.fromUrl('https://example.com/video.mp4');
await studio.addClip(video);

// Start preview
studio.play();

Sources: README.md:45-60

Node.js Renderer (`packages/node`)

For server-side video rendering without a browser UI:

import { Renderer } from '@combo/node';

const renderer = new Renderer({
  json: videoConfig,
  outputPath: './output.mp4',
});

renderer.on('progress', ({ phase, progress, message }) => {
  console.log(`[${phase}] ${Math.round(progress * 100)}% - ${message}`);
});

await renderer.render();

Sources: packages/node/README.md:100-120

Rendering Workflow

Client-Side Rendering

sequenceDiagram
    participant User
    participant Studio
    participant Compositor
    participant WebCodecs
    
    User->>Studio: Create project with clips
    User->>Studio: Configure timeline
    User->>Studio: Start playback/preview
    Studio->>Compositor: Render frame
    Compositor->>PixiJS: Draw scene
    Compositor->>WebCodecs: Encode frame
    WebCodecs-->>Compositor: Encoded video data
    Compositor-->>User: Display/Preview

Server-Side Rendering (Node.js)

graph TD
    A[Create Renderer Instance] --> B[Load JSON Config]
    B --> C[Launch Headless Browser]
    C --> D[Load HTML Template]
    D --> E[Initialize Compositor]
    E --> F[Render Frames]
    F --> G[Encode Video via WebCodecs]
    G --> H[Save to File System]
    H --> I[Return Output Path]

The Node.js renderer uses Playwright to launch a headless browser, inject the JSON configuration, and use the HTML template to render the video using the same Compositor and WebCodecs engine.

Sources: packages/node/README.md:1-50

Studio API Reference

Initialization

const studio = new Studio({
  canvas: HTMLCanvasElement,  // Required: Target canvas
  width?: number,             // Default: 1280
  height?: number,            // Default: 720
  spacing?: number,           // Default: 20
});

Clip Management

// Add a clip
await studio.addClip(clip: BaseClip): Promise<void>;

// Remove a clip by ID
await studio.removeClip(clipId: string): Promise<void>;

// Access clips
studio.clips; // Readonly array of clips

Sources: packages/openvideo/src/studio.spec.ts:1-50

Compositor Configuration

const compositorOpts = {
  width: 1280,              // Output width
  height: 720,              // Output height
  fps: 30,                  // Frames per second
  bgColor: '#000000',       // Background color
  videoCodec?: 'avc1.64001f', // Specific codec
  bitrate?: 5000000,         // Video bitrate
  audio?: true,              // Include audio track
  metaDataTags?: {}          // Metadata for output
};

Use Cases

OpenVideo is suitable for various applications:

Requirements

Browser Environment

• Modern browser with WebCodecs support (Chrome 94+, Edge 94+)
• WebGL 2.0 support for PixiJS rendering
• ES2020+ JavaScript support

Node.js Environment

• Node.js >= 18
• Playwright for headless browser rendering
• openvideo package installed

License

OpenVideo is released under the MIT License.

Sources: README.md:70-72

Related Pages

Related topics: Overview

Installation and Setup

Overview

This page documents the complete installation and setup process for the openvideo project. openvideo is a video rendering and processing library built on WebCodecs, PixiJS, and wrapbox technologies. The project supports multiple deployment scenarios including browser-based video rendering, Node.js server-side rendering, and integration with frontend frameworks.

System Requirements

Runtime Requirements

Browser Environment Requirements

The openvideo library relies on the WebCodecs API for hardware-accelerated video processing. This requires:

• A Chromium-based browser (Chrome, Edge, Brave, Arc)
• Secure context (HTTPS or localhost)
• Access to the VideoEncoder, VideoDecoder, and VideoFrame APIs

Sources: packages/openvideo/package.json:1-30

Project Structure

The openvideo repository follows a monorepo structure using pnpm workspaces:

openvideo/
├── packages/
│   ├── openvideo/           # Core rendering library
│   │   ├── src/
│   │   │   ├── clips/       # Video, Audio, Text, Image clips
│   │   │   ├── animation/   # Animation presets
│   │   │   ├── transition/  # Transition effects
│   │   │   └── mp4-utils/   # MP4 muxing utilities
│   │   ├── dist/            # Built output
│   │   └── package.json
│   └── node/                # Node.js renderer package
│       ├── src/
│       │   ├── renderer.ts  # Server-side renderer
│       │   ├── types.ts     # Type definitions
│       │   └── template.html
│       └── package.json
└── examples/                # Example applications

Installation Steps

1. Install pnpm

If you don't have pnpm installed, install it globally:

npm install -g pnpm

2. Install Dependencies

Clone the repository and install all dependencies:

git clone https://github.com/openvideodev/openvideo.git
cd openvideo
pnpm install

3. Build the Core Package

Build the main openvideo package before using it:

pnpm --filter openvideo build

This generates the distribution files in packages/openvideo/dist/:

• index.es.js - ES Module build
• index.umd.js - UMD build for CDN usage
• index.d.ts - TypeScript definitions

Sources: packages/openvideo/package.json:23-25

4. Build Node.js Renderer (Optional)

For server-side rendering capabilities:

pnpm --filter @combo/node build

Sources: packages/node/README.md:1-100

Browser Usage

Script Tag (UMD)

Include openvideo directly via CDN:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Video Render</title>
</head>
<body>
    <script src="https://unpkg.com/openvideo/dist/index.umd.js"></script>
    <script>
        const { Studio, Compositor } = window.openvideo;
        // Your code here
    </script>
</body>
</html>

ES Module with Import Map

For modern browser applications using ES modules:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Video Render</title>
    <script type="importmap">
        {
            "imports": {
                "openvideo": "/node_modules/openvideo/dist/index.es.js"
            }
        }
    </script>
</head>
<body>
    <script type="module">
        import { Compositor } from 'openvideo';
        // Your code here
    </script>
</body>
</html>

Sources: packages/node/src/template.html:1-25

Node.js Server-Side Rendering

The @combo/node package enables server-side video rendering using Playwright and Chromium.

Renderer Configuration

import { Renderer } from '@combo/node';

const renderer = new Renderer({
  json: videoConfig,        // Video configuration JSON
  outputPath: './output.mp4', // Output file path
  browserOptions: {
    headless: true,          // Default: true
    timeout: 300000          // Default: 5 minutes
  }
});

Renderer Options

Rendering Events

The Renderer extends EventEmitter and emits the following events:

// Progress tracking
renderer.on('progress', ({ phase, progress, message }) => {
  console.log(`[${phase}] ${Math.round(progress * 100)}% - ${message}`);
});

// Error handling
renderer.on('error', (error) => {
  console.error('Render failed:', error);
});

// Completion
renderer.on('complete', (path) => {
  console.log(`Video saved to: ${path}`);
});

// Start rendering
await renderer.render();

Sources: packages/node/README.md:60-120

Build Workflow

graph TD
    A[pnpm install] --> B[Build openvideo Core]
    B --> C{Use Case}
    C -->|Browser App| D[Import openvideo ES Module]
    C -->|Server Rendering| E[Build @combo/node]
    C -->|Development| F[Run Examples]
    
    D --> G[Create Compositor Instance]
    E --> H[Create Renderer Instance]
    
    G --> I[Configure width, height, fps]
    H --> J[Configure JSON config]
    
    I --> K[Add clips to Studio]
    J --> K
    
    K --> L[Render video]
    L --> M[Output MP4]
    
    F --> N[Start dev server]
    N --> O[Browse examples]

Development

Running Examples

Start the development server for the examples application:

cd examples
pnpm dev

This serves the examples at http://localhost:5173 (Vite default).

Sources: examples/package.json

Testing

Run unit tests:

pnpm --filter openvideo test

Run browser-specific tests:

pnpm --filter openvideo test:browser

Sources: packages/openvideo/package.json:23-25

Development Workflow

1. Make source changes in packages/openvideo/src/

``bash pnpm --filter openvideo build ``

1. Rebuild the package:

``bash node dist/sample.js ``

1. Test changes:

``bash pnpm --filter @combo/node build ``

1. For Node.js renderer changes:

Troubleshooting

WebCodecs Not Available

If you encounter VideoEncoder is not defined:

• Ensure you're running in a Chromium-based browser
• Verify you're on a secure origin (HTTPS or localhost)
• Check that WebCodecs API is enabled in browser settings

Module Resolution Errors

If ES module imports fail:

# Ensure openvideo is built first
pnpm --filter openvideo build

Playwright Installation Issues

For Node.js renderer Chromium installation failures:

npx playwright install chromium

Timeout Errors

For large video renders, increase the timeout:

const renderer = new Renderer({
  json: videoConfig,
  outputPath: './output.mp4',
  browserOptions: {
    timeout: 600000  // 10 minutes
  }
});

Package Exports

The main openvideo package exports the following from packages/openvideo/package.json:

// ES Module import
import { Studio, Compositor, Video, Text, Image } from 'openvideo';

// UMD/CommonJS require
const { Studio } = require('openvideo');

Key Components for Setup

Sources: packages/openvideo/render.html:1-40

Related Pages

Related topics: Overview, Studio and State Management, Clip System

System Architecture

Overview

The openvideo system is a browser-based video editing and rendering engine built on modern web APIs. It provides a comprehensive architecture for composing, editing, and exporting video content directly in the browser environment. The system leverages WebCodecs for hardware-accelerated video processing, PixiJS for 2D/3D rendering, and internal utilities for MP4 box manipulation and muxing.

Sources: README.md:1-50

High-Level Architecture

The system follows a layered architecture with distinct responsibilities at each level:

graph TD
    A[User Application] --> B[Studio]
    B --> C[Compositor]
    C --> D[PixiJS Rendering Engine]
    C --> E[WebCodecs API]
    E --> F[MP4 Muxing]
    B --> G[Clips Layer]
    G --> H[Video Clip]
    G --> I[Audio Clip]
    G --> J[Text Clip]
    G --> K[Image Clip]
    G --> L[Caption Clip]
    G --> M[Shape Clip]

Sources: packages/openvideo/src/index.ts:1-100

Core Components

Studio

The Studio class serves as the central orchestrator for the entire video editing system. It manages:

• Project state: Tracks, clips, and timeline configuration
• Preview playback: Real-time video preview with seeking capabilities
• Export coordination: Manages the rendering pipeline for final output

graph LR
    A[Studio] --> B[addClip]
    A --> C[removeClip]
    A --> D[play]
    A --> E[pause]
    A --> F[seek]
    A --> G[export]

Sources: packages/openvideo/src/studio.ts:1-50

Compositor

The Compositor is the rendering engine responsible for:

• Playback control: Managing frame-accurate video playback
• Seeking: Navigating to specific timestamps in the timeline
• Final export: Encoding and muxing the video output using WebCodecs

const compositorOpts = {
  width: settings.width || 1280,
  height: settings.height || 720,
  fps: settings.fps || 30,
  bgColor: settings.bgColor || '#000000',
  videoCodec?: settings.videoCodec,
  bitrate?: settings.bitrate,
  audio?: boolean,
  metaDataTags?: metaDataTags,
};

Sources: packages/openvideo/render.html:30-50

Clips Layer

The clip system provides specialized objects for different media types. All clips inherit from a base sprite class that handles common properties and animations.

Clip Architecture

Base Sprite

All clips inherit from BaseSprite, which provides a unified transform and animation system:

graph TD
    A[BaseSprite] --> B[renderTransform]
    A --> C[animations]
    B --> D[x, y, width, height]
    B --> E[scale, scaleX, scaleY]
    B --> F[angle, opacity]
    B --> G[blur, brightness]
    B --> H[mirror, motionBlur]

The render transform accumulates values from multiple animations:

const defaultRenderTransform = {
  x: 0,
  y: 0,
  width: 0,
  height: 0,
  scale: 1,
  scaleX: 1,
  scaleY: 1,
  opacity: 1,
  angle: 0,
  blur: 0,
  brightness: 1,
  mirror: 0,
  motionBlur: 0,
};

Sources: packages/openvideo/src/sprite/base-sprite.ts:1-100

Video Clip

The Video class handles video file loading and playback:

sequenceDiagram
    participant User
    participant Video
    participant ResourceManager
    participant PixiJS
    
    User->>Video: fromUrl(url)
    Video->>ResourceManager: getReadableStream(url)
    ResourceManager-->>Video: stream
    Video->>Video: write to localFile
    Video->>PixiJS: init texture
    Video-->>User: Video instance

Key features include:

• URL-based loading via Video.fromUrl()
• OPFS (Origin Private File System) for local storage
• Configurable audio with volume control
• Position and size properties (left, top, width, height)

Sources: packages/openvideo/src/clips/video-clip.ts:1-100

Text Clip

Text clips support extensive styling options:

Sources: packages/openvideo/src/clips/text-clip.ts:1-100

Caption Clip

Caption clips support word-level animations with the following JSON structure:

interface CaptionColorsJSON {
  appeared?: string;
  appearing?: string;
  disappearing?: string;
}

interface CaptionDataJSON {
  words?: string[];
  colors?: CaptionColorsJSON;
  style?: TextStyleOptions;
}

Sources: packages/openvideo/src/clips/caption-clip.ts:1-100

Animation System

Animation Presets

The animation system provides predefined motion patterns:

Preset Configuration

case "slideOut": {
  const direction = normalized?.direction || "left";
  const distance = normalized?.distance || 300;
  return {
    "0%": {
      x: xPositionInit ?? 0,
      y: yPositionInit ?? 0,
      opacity: opacityInit ?? 1,
      mirror: defaultMirror,
    },
    "100%": {
      x: direction === "left" ? -distance : direction === "right" ? distance : 0,
      y: direction === "top" ? -distance : direction === "bottom" ? distance : 0,
      opacity: opacityEnd ?? 0,
      mirror: defaultMirror,
    },
  };
}

Sources: packages/openvideo/src/animation/presets.ts:1-100

Animation Transform Properties

Animations can modify the following sprite properties:

• Position: x, y
• Size: width, height
• Rotation: angle
• Scale: scale, scaleX, scaleY
• Appearance: opacity, blur, brightness
• Effects: mirror, motionBlur

Transition System

The transition system provides visual effects for clip transitions:

graph TD
    A[Transition Lookup] --> B[Check by name]
    A --> C[Check by lowercase name]
    A --> D[Check by label]
    A --> E[Variant generation]
    
    B --> F{Found?}
    C --> F
    D --> F
    E --> F
    
    F -->|Yes| G[Apply GLSL Fragment]
    F -->|No| H[Error: Transition not found]

Supported Transitions

Sources: packages/openvideo/src/transition/transition.ts:1-200

Rendering Pipeline

WebCodecs Integration

The compositor uses WebCodecs for hardware-accelerated video encoding:

graph LR
    A[Timeline Frames] --> B[PixiJS Render]
    B --> C[VideoFrame]
    C --> D[VideoEncoder]
    D --> E[EncodedVideoChunk]
    E --> F[MP4 Muxer]
    F --> G[Output File]

MP4 Muxing

The mp4-utils module handles sample normalization and muxing:

samples.forEach((s) => {
  let normalizedDTS: number;
  let normalizedCTS: number;
  
  // Normalize to start from 0
  normalizedDTS = s.dts - firstVDTS!;
  normalizedCTS = s.cts - (firstVCTS ?? 0);
  
  outfile.addSample(trackId, new Uint8Array(s.data), {
    duration: s.duration,
    dts: normalizedDTS + offsetDTS,
    cts: normalizedCTS + offsetCTS,
    is_sync: s.is_sync,
  });
});

Sources: packages/openvideo/src/mp4-utils/index.ts:1-100

Node Package Architecture

The @combo/node package provides server-side rendering capabilities:

graph TD
    A[Renderer] --> B[Playwright]
    A --> C[Local HTTP Server]
    B --> D[Browser Instance]
    C --> D
    D --> E[template.html]
    E --> F[Compositor]
    F --> G[WebCodecs]
    G --> H[MP4 Output]

Renderer Class

interface RendererOptions {
  json: VideoConfigJSON;
  outputPath: string;
  browserOptions?: {
    headless?: boolean;
    timeout?: number;
  };
}

#### Events

Sources: packages/node/README.md:1-100

HTML Template

The node package uses a template HTML file with import maps:

<script type="importmap">
    {
        "imports": {
            "openvideo": "/node_modules/openvideo/dist/index.es.js"
        }
    }
</script>

Sources: packages/node/src/template.html:1-30

Technology Stack

Sources: README.md:40-50

Data Flow

sequenceDiagram
    participant App
    participant Studio
    participant Compositor
    participant PixiJS
    participant WebCodecs
    participant File
    
    App->>Studio: addClip(video)
    App->>Studio: play()
    Studio->>Compositor: initPixiApp()
    Compositor->>PixiJS: Create Application
    
    loop Playback
        Studio->>Compositor: render frame
        Compositor->>PixiJS: render scene
        PixiJS-->>Compositor: frame buffer
    end
    
    App->>Studio: export()
    Studio->>Compositor: start encoding
    loop Encoding
        Compositor->>PixiJS: render frame
        PixiJS-->>Compositor: VideoFrame
        Compositor->>WebCodecs: encode
        WebCodecs-->>Compositor: EncodedChunk
        Compositor->>File: mux samples
    end
    Compositor-->>App: output path

Package Structure

packages/
├── openvideo/
│   └── src/
│       ├── studio.ts           # Main orchestrator
│       ├── compositor.ts       # Rendering engine
│       ├── clips/              # Media clip implementations
│       │   ├── index.ts
│       │   ├── video-clip.ts
│       │   ├── text-clip.ts
│       │   ├── caption-clip.ts
│       │   └── ...
│       ├── sprite/             # Base sprite classes
│       ├── animation/          # Animation presets
│       ├── transition/         # Transition effects
│       └── mp4-utils/          # MP4 muxing utilities
└── node/
    └── src/
        ├── renderer.ts         # Node.js renderer
        ├── template.html      # Browser template
        └── types.ts            # TypeScript types

Export and Serialization

The system supports JSON serialization for project portability:

interface VideoConfigJSON {
  settings: {
    width: number;
    height: number;
    fps: number;
    bgColor: string;
    videoCodec?: string;
    bitrate?: number;
    audio?: boolean;
    metaDataTags?: Record<string, string>;
  };
  tracks: TrackJSON[];
}

Sources: packages/openvideo/render.html:25-45

Related Pages

Related topics: System Architecture, Clip System, Animation System

Studio and State Management

Overview

The Studio module is the central orchestration layer in OpenVideo, responsible for managing project state, timeline operations, playback control, and user interactions. It provides a unified API for video editing workflows by coordinating between the Compositor (rendering engine), Timeline Model (data structure), Transport (playback control), Selection Manager (UI interactions), and History Manager (undo/redo).

Studio abstracts the complexity of WebCodecs and PixiJS rendering, exposing high-level primitives like Clips, Tracks, and Effects that map directly to user-facing editing concepts.

Architecture Overview

graph TD
    A[Studio] --> B[Compositor]
    A --> C[Timeline Model]
    A --> D[Transport]
    A --> E[Selection Manager]
    A --> F[History Manager]
    A --> G[Resource Manager]
    
    C --> H[Tracks]
    C --> I[Clips]
    H --> I
    I --> J[VideoClip]
    I --> K[AudioClip]
    I --> L[TextClip]
    I --> M[ImageClip]
    I --> N[CaptionClip]
    
    D --> B
    E --> A
    F --> C
    G --> I

Core Components

Studio Class

The Studio class serves as the main entry point and state container for video projects. It maintains references to all subsystem managers and provides the public API for editing operations.

Key Responsibilities:

• Project lifecycle management (create, load, save, clear)
• Coordination between timeline, compositor, and transport
• Event emission for UI updates
• Texture and resource cleanup

Key Methods:

Sources: packages/openvideo/src/studio.ts:1-100

Timeline Model

The Timeline Model manages the hierarchical data structure of a video project:

graph TD
    Timeline --> Tracks
    Tracks --> Clip
    Clip --> Properties
    Properties --> Transform
    Properties --> Effect
    Properties --> Animation

Data Structure:

Sources: packages/openvideo/src/studio/timeline-model.ts

Transport System

The Transport module handles playback control and timeline navigation.

Playback States

stateDiagram-v2
    [*] --> Stopped
    Stopped --> Playing : play()
    Playing --> Paused : pause()
    Paused --> Playing : play()
    Playing --> Stopped : stop()
    Paused --> Stopped : stop()
    Stopped --> Seeking : seek()
    Playing --> Seeking : seek()
    Seeking --> Playing : play()
    Seeking --> Paused : pause()

Transport API

interface Transport {
  play(): Promise<void>;
  pause(): void;
  stop(): Promise<void>;
  seek(time: number): Promise<void>;
  frameNext(): Promise<void>;
  framePrev(): Promise<void>;
}

Key Behaviors:

• play(): Starts playback from current position, returns Promise
• pause(): Immediately pauses playback
• stop(): Stops and resets playhead to timeline start
• seek(time): Moves playhead to specified time in microseconds

Sources: packages/openvideo/src/studio/transport.ts

Selection Management

The Selection Manager handles interactive clip selection and transformation in the editor canvas.

Selection Features

• Single Selection: Click to select a single clip
• Multi-Selection: Shift+click to add to selection
• Double-Click: Opens text editing for Text/Caption clips
• Transform Handles: Resize and reposition selected clips

Selection Events

Pointer Interaction Flow

sequenceDiagram
    participant User
    participant Canvas
    participant SelectionManager
    participant Studio
    
    User->>Canvas: pointerdown
    Canvas->>SelectionManager: handlePointerDown
    SelectionManager->>SelectionManager: Check for double-click
    Alt Double-click on Text/Caption
        SelectionManager->>Studio: emit('clip:dblclick')
    End
    
    User->>Canvas: pointerup
    Canvas->>SelectionManager: handlePointerUp
    SelectionManager->>SelectionManager: Update selection
    SelectionManager->>Studio: emit('clip:select')

Sources: packages/openvideo/src/studio/selection-manager.ts:1-80

History Management (Undo/Redo)

The History Manager maintains a stack of state snapshots to enable undo/redo operations.

State Snapshot Structure

interface HistoryState {
  timestamp: number;
  action: string;
  state: SerializedTimeline;
}

Operations

History Stack Behavior

graph LR
    A[Current] -->|Undo| B[State N-1]
    B -->|Undo| C[State N-2]
    A -->|Redo| D[State N+1]
    C -.->|Push| A

Sources: packages/openvideo/src/studio/history-manager.ts

Resource Management

The Resource Manager handles loading, caching, and lifecycle of media resources (video files, images, audio).

Resource Loading

// Loading a video from URL
const video = await Video.fromUrl('https://example.com/video.mp4');
await studio.addClip(video);

// Loading with position options
const videoWithOpts = await Video.fromUrl('https://example.com/video.mp4', {
  x: 100,
  y: 200,
  width: 640,
  height: 480
});

Resource Lifecycle

Sources: packages/openvideo/src/studio/resource-manager.ts

Clip Types

OpenVideo supports multiple clip types that extend a common base:

Clip Properties

interface IClip {
  id: string;
  type: 'Video' | 'Audio' | 'Text' | 'Image' | 'Caption';
  left: number;
  top: number;
  width: number;
  height: number;
  startTime: number;
  endTime: number;
  opacity: number;
  volume?: number;
}

Sources: packages/openvideo/src/clips/video-clip.ts:1-50

Event System

Studio extends EventEmitter to provide reactive updates:

// Event listeners
studio.on('progress', ({ phase, progress }) => {
  console.log(`[${phase}] ${Math.round(progress * 100)}%`);
});

studio.on('clip:select', ({ clip }) => {
  console.log('Selected:', clip.id);
});

studio.on('reset', () => {
  console.log('Studio cleared');
});

Event Reference

Studio Lifecycle

flowchart TD
    A[Create Studio] --> B[Initialize PixiJS]
    B --> C[Load Project/JSON]
    C --> D[User Editing]
    D --> E{Operation}
    E -->|Add Clip| F[Update Timeline]
    E -->|Select| G[Update Selection]
    E -->|Transform| H[Update Transform]
    E -->|Play| I[Transport Control]
    F --> D
    G --> D
    H --> D
    I --> J[Compositor Render]
    J --> K{Loop}
    K -->|Continue| I
    K -->|Stop| L[Export/Save]
    L --> M[Cleanup Resources]

Initialization

const studio = new Studio({
  width: 1280,
  height: 720,
  fps: 30,
});

// Studio is ready for use
await studio.ready;

Cleanup

// Clear all resources
await studio.clear();

// Releases:
// - All clips from timeline
// - PixiJS textures and graphics
// - Transition renderers
// - Selection state

Sources: packages/openvideo/src/studio.ts:80-150

JSON Serialization

Studio projects can be serialized to JSON for persistence and sharing:

// Export project
const json = studio.toJSON();

// Import project
const restored = Studio.fromJSON(json);

Serialized Format

interface SerializedStudio {
  version: string;
  settings: {
    width: number;
    height: number;
    fps: number;
  };
  timeline: {
    tracks: SerializedTrack[];
    duration: number;
  };
  resources: ResourceReference[];
}

Usage Example

import { Studio, Video, Text } from 'openvideo';

// Initialize studio
const studio = new Studio({
  width: 1920,
  height: 1080,
  fps: 30,
});

// Add video clip
const video = await Video.fromUrl('https://example.com/intro.mp4');
await studio.addClip(video);

// Add text overlay
const title = new Text({
  text: 'Hello World',
  fontSize: 72,
  color: 0xFFFFFF,
});
title.left = 100;
title.top = 500;
await studio.addClip(title);

// Preview
studio.play();

// Export
const output = await studio.compositor.render();

Sources: README.md

Related Pages

Related topics: System Architecture, Studio and State Management, Animation System

Clip System

Overview

The Clip System is the foundational data model and rendering layer within OpenVideo, responsible for representing, managing, and rendering media assets on a timeline. Clips encapsulate all media types (video, audio, image, text, caption) and provide a unified interface for timeline operations including playback, seeking, splitting, cloning, and serialization.

Sources: packages/openvideo/src/clips/iclip.ts:1-80

Related Pages

Related topics: Clip System, Effects System

Animation System

Overview

The Animation System in OpenVideo is a comprehensive framework that provides time-based property interpolation for sprites, clips, and UI elements within the video compositor. The system supports keyframe-based animations, easing functions, preset animations, and integration with external animation libraries like GSAP.

Animations in OpenVideo work by interpolating properties over time using easing functions. They can be applied to any visual element (sprites, clips, captions) and are processed during the render cycle to update the renderTransform properties.

Sources: packages/openvideo/src/sprite/base-sprite.ts:1-50

Architecture

graph TD
    A[Animation Request] --> B{Animation Type}
    B -->|Preset| C[Animation Presets]
    B -->|Keyframe| D[KeyframeAnimation]
    B -->|GSAP| E[GSAP Animation]
    C --> F[Factory Pattern]
    D --> G[linearTimeFn]
    E --> H[GSAP Core]
    G --> I[Easing Functions]
    I --> J[Interpolated Values]
    H --> J
    F --> J
    J --> K[RenderTransform Update]

Core Components

Sources: packages/openvideo/src/animation/types.ts

Easing Functions

Easing functions control the rate of change during an animation, providing smooth acceleration and deceleration. The system includes a comprehensive set of built-in easing functions.

Supported Easings

Sources: packages/openvideo/src/animation/easings.ts:1-30

Easing Interface

type EasingFunction = (t: number) => number;

All easing functions accept a normalized time parameter t (0 to 1) and return the interpolated progress value.

Animation Presets

Preset animations are pre-configured animation factories that can be applied to elements without defining custom keyframes. They use a factory pattern that accepts options and optional parameters.

Available Presets

#### Entry Animations

#### Exit Animations

#### Loop Animations

Sources: packages/openvideo/src/animation/presets.ts

Preset Configuration

Each preset accepts:

interface AnimationFactory {
  (opts: IAnimationOpts, params?: Record<string, any>): KeyframeAnimation;
}

Keyframe Animation

The KeyframeAnimation class is the core animation implementation that handles time-based property interpolation.

Animation Flow

sequenceDiagram
    participant Render as Render Loop
    participant Anim as KeyframeAnimation
    participant Easing as Easing Functions
    participant Target as Sprite/Clip
    
    Render->>Anim: getTransform(currentTime)
    Note over Anim: Calculate progress from time
    Anim->>Easing: apply easing(progress)
    Easing-->>Anim: eased progress
    Anim->>Anim: interpolate properties
    Anim-->>Target: Partial<TAnimateProps>

Sources: packages/openvideo/src/animation/keyframe-animation.ts

Animated Properties

The animation system can interpolate the following properties:

Sources: packages/openvideo/src/sprite/base-sprite.ts:80-100

Linear Time Function

The linearTimeFn performs the actual interpolation between keyframes:

export function linearTimeFn(
  time: number,
  keyFrame: TAnimationKeyFrame,
  opts: Required<IAnimationOpts>
): Partial<TAnimateProps>

The function:

1. Calculates normalized progress from elapsed time
2. Finds surrounding keyframes
3. Interpolates each property using linear interpolation
4. Returns partial props object with interpolated values

Sources: packages/openvideo/src/sprite/base-sprite.ts:200-250

Animation Integration with Sprites

Sprites process animations during their update cycle by iterating through all animations and accumulating transform values.

Sprite Animation Processing

// 1. Process new modular animations
for (const anim of this.animations) {
  const transform = anim.getTransform(time);
  if (transform.x !== undefined) this.renderTransform.x! += transform.x;
  if (transform.y !== undefined) this.renderTransform.y! += transform.y;
  if (transform.scale !== undefined)
    this.renderTransform.scale! *= transform.scale;
  // ... other properties
}

Transform values are accumulated rather than replaced, allowing multiple animations to affect the same property.

Sources: packages/openvideo/src/sprite/base-sprite.ts:20-45

Clip Animation Support

Clips support animation through both legacy animation field and the new modular animations array format.

Clip Serialization

// Extract new modular animations
const animations = this.animations.map((a) => {
  if ('toJSON' in a && typeof (a as any).toJSON === 'function') {
    return (a as any).toJSON();
  }
  return {
    type: a.type,
    opts: a.options,
    params: a.params || {},
  };
});

Clips serialize animations to JSON for configuration export and reloading.

Sources: packages/openvideo/src/clips/base-clip.ts

Caption Word Animations

The CaptionClip system implements specialized per-word animations with dynamic and keyword-based application modes.

Word Animation Application Modes

Animation Dynamics

if (wordAnimation.mode === 'dynamic' && isActive) {
  const progress = (currentTimeMs - word.from) / duration;
  animationFactor = Math.sin(progress * Math.PI);
} else {
  animationFactor = 1;
}

Sources: packages/openvideo/src/clips/caption-clip.ts

GSAP Integration

The system provides GSAP animation integration through the GSAPAnimation class, allowing users to leverage GSAP's extensive animation capabilities.

Sources: packages/openvideo/src/animation/gsap-animation.ts

Animation Registry

The registry system allows dynamic registration and lookup of custom animations, enabling extensibility.

// From registry.ts - conceptual usage
const registry = getAnimationRegistry();
registry.register('myCustomAnimation', customAnimation);
const animation = registry.get('myCustomAnimation');

Sources: packages/openvideo/src/animation/registry.ts

Usage Examples

Basic Preset Animation

import { blurIn, easeOutQuad } from '@openvideo/animation';

const animation = blurIn({
  easing: 'easeOutQuad',
  duration: 1000,
  delay: 500,
  iterCount: 1,
});

Custom Keyframe Animation

import { KeyframeAnimation } from '@openvideo/animation';

const customAnim = new KeyframeAnimation({
  '0%': { opacity: 0, y: 100 },
  '100%': { opacity: 1, y: 0 },
}, {
  easing: 'easeOutCubic',
  duration: 800,
});

Best Practices

1. Use Presets for Common Animations: Presets are optimized and provide consistent behavior across the application.

1. Avoid Animating Expensive Properties: Minimize animations on width, height, and blur as they require significant processing.

1. Reuse Easing Functions: Instead of custom easing, use the built-in easings which are well-tested and performant.

1. Consider Animation Composition: Multiple simple animations often produce better results than one complex animation.

1. Set Appropriate Iter Counts: Use iterCount: -1 only for looping background animations to avoid performance issues.

Related Pages

Related topics: System Architecture, Studio and State Management

Compositor and Rendering

Overview

The Compositor is the central rendering engine in the openvideo library. It orchestrates video playback, real-time composition, and final video export using modern browser APIs including WebCodecs for hardware-accelerated encoding and PixiJS for 2D rendering. The system transforms a timeline-based project configuration into a final rendered video file.

The Compositor serves as the bridge between high-level clip management (handled by Studio) and low-level rendering operations (handled by sprite renderers). It manages the rendering loop, coordinates frame composition, and produces the final encoded output.

Architecture

graph TD
    A[Studio] --> B[Compositor]
    B --> C[PixiJS Application]
    C --> D[Sprite Renderers]
    D --> E[Base Sprite]
    D --> F[Pixi Sprite Renderer]
    B --> G[WebCodecs Encoder]
    G --> H[MP4 Muxer]
    H --> I[Output File]
    
    J[Video Clips] --> D
    K[Text Clips] --> D
    L[Image Clips] --> D
    M[Audio Clips] --> D

Core Components

Compositor

The Compositor class initializes the rendering environment and manages the export pipeline. It accepts configuration options that define the output format.

#### Constructor Options

Sources: packages/node/src/template.html:30-48

The compositor initialization follows a specific lifecycle:

graph LR
    A[Create Compositor] --> B[initPixiApp]
    B --> C[Configure Encoder]
    C --> D[Setup Event Handlers]
    D --> E[Ready for Rendering]

PixiJS Integration

The Compositor uses PixiJS as its rendering backend. The initPixiApp() method initializes the PixiJS application with the configured canvas dimensions.

#### Canvas Initialization

The sprite renderer validates texture creation to prevent runtime errors:

// Validate texture was created successfully
// Use Texture.source instead of baseTexture (PixiJS v8.0.0+)
if (!this.texture || !this.texture.source) {
  console.error("PixiSpriteRenderer: Failed to create valid texture");
  return;
}

Sources: packages/openvideo/src/sprite/pixi-sprite-renderer.ts:35-42

Canvas dimensions are converted to integers to prevent WebCodecs encoding errors:

// Update canvas size using integers to prevent "Value is not of type unsigned long" errors
const intWidth = Math.floor(width);
const intHeight = Math.floor(height);

Sources: packages/openvideo/src/sprite/pixi-sprite-renderer.ts:23-25

Sprite Rendering System

Base Sprite

The BaseSprite class provides the foundation for all renderable elements. It maintains a renderTransform object that accumulates all transformation properties:

private renderTransform: {
  x: number;
  y: number;
  width: number;
  height: number;
  scale: number;
  scaleX: number;
  scaleY: number;
  opacity: number;
  angle: number;
  blur: number;
  brightness: number;
  mirror: number;
  motionBlur: number;
};

Sources: packages/openvideo/src/sprite/base-sprite.ts:29-44

Animation System

Animations are processed by iterating through the animations array and accumulating transforms at the current time:

for (const anim of this.animations) {
  const transform = anim.getTransform(time);
  if (transform.x !== undefined) this.renderTransform.x! += transform.x;
  if (transform.y !== undefined) this.renderTransform.y! += transform.y;
  if (transform.width !== undefined) this.renderTransform.width! += transform.width;
  if (transform.scale !== undefined) this.renderTransform.scale! *= transform.scale;
  if (transform.opacity !== undefined) this.renderTransform.opacity! *= transform.opacity;
  if (transform.mirror !== undefined)
    this.renderTransform.mirror = Math.max(this.renderTransform.mirror || 0, transform.mirror);
}

Sources: packages/openvideo/src/sprite/base-sprite.ts:46-65

Transform operations follow specific accumulation rules:

• Additive: x, y, width, height, angle, blur, motionBlur
• Multiplicative: scale, scaleX, scaleY, opacity, brightness
• Maximum: mirror (takes the highest value among all animations)

Animation Presets

The library includes built-in animation presets defined in presets.ts:

Sources: packages/openvideo/src/animation/presets.ts

Video Export Pipeline

WebCodecs Encoding

The compositor leverages the WebCodecs API for hardware-accelerated video encoding. The encoder produces raw video frames which are then multiplexed into an MP4 container.

MP4 Muxing

The MP4 utility module handles container creation and muxing:

graph TD
    A[Raw Video Frames] --> B[Encoded Video Samples]
    B --> C[MP4Box Muxer]
    C --> D[moov Atom]
    C --> E[mdat Atom]
    D --> F[Final MP4 File]
    E --> F

The muxer writes samples to the MP4 container and manages:

• Track configuration (video/audio)
• Sample timing and duration
• Chunk layout
• Metadata injection

Progress Reporting

The Compositor emits progress events during the export process:

compositor.on('OutputProgress', (progress: number) => {
  // Report encoding progress
});

Sources: packages/openvideo/render.html:45

Event System

The Compositor extends EventEmitter to provide a pub/sub mechanism for render lifecycle events:

stateDiagram-v2
    [*] --> initializing
    initializing --> loading
    loading --> rendering
    rendering --> saving
    saving --> complete
    complete --> [*]
    
    rendering --> error: On Error
    loading --> error: On Error

Event Types

Node.js Renderer

The @combo/node package provides a headless rendering solution using Playwright:

graph LR
    A[JSON Config] --> B[Node Renderer]
    B --> C[Local HTTP Server]
    C --> D[Browser Page]
    D --> E[Compositor]
    E --> F[MP4 Output]

Renderer Options

Texture Management

The PixiSpriteRenderer implements a texture caching strategy:

1. Canvas is created with integer dimensions matching output resolution
2. Video/image frames are drawn to canvas
3. PixiJS Texture is created/updated from canvas
4. Sprite displays the texture
5. Previous textures are destroyed to free memory

if (needsResize || isFirstFrame) {
  this.canvas.width = intWidth;
  this.canvas.height = intHeight;

  if (this.texture != null) {
    this.texture.destroy(true);
    this.texture = null;
  }

  this.texture = Texture.from(this.canvas as any);
}

Sources: packages/openvideo/src/sprite/pixi-sprite-renderer.ts:27-38

Transition Effects

The rendering system supports numerous transition effects including:

• GridFlip: Grid-based transition with flip animation
• Circle: Circular wipe transition
• Directional: Directional slide transition
• UndulatingBurnOut: Wave-like fade effect
• SquaresWire: Wireframe square expansion
• RotateScaleFade: Combined rotation, scaling, and fade
• Pixelate: Pixelation dissolve
• CrossZoom: Simultaneous zoom and crossfade
• PolkaDotsCurtain: Dotted curtain reveal

Transitions are resolved by name matching (case-insensitive) against multiple formats:

const transitionGridFlip =
  transition.name === 'GridFlip' ||
  name.toLowerCase() === 'gridflip' ||
  transition.label === 'gridflip';

Rendering Configuration

HTML Template Configuration

When using the HTML template for rendering, configuration is injected via window.RENDER_CONFIG:

const jsonConfig = window.RENDER_CONFIG;

if (!jsonConfig) {
  throw new Error('No render configuration provided');
}

const settings = jsonConfig.settings || {};
const compositorOpts = {
  width: settings.width || 1280,
  height: settings.height || 720,
  fps: settings.fps || 30,
  bgColor: settings.bgColor || '#000000',
};

Memory Management

The sprite renderer properly cleans up resources:

• Textures are destroyed with destroy(true) to free GPU memory
• Canvas dimensions are validated before allocation
• Frame textures are only recreated when size changes

See Also

• Studio and Timeline - Higher-level project management
• Clips and Media - Media element handling
• Animation System - Animation creation and playback
• Node.js Rendering - Server-side rendering guide

Related Pages

Related topics: Transitions System, Animation System

Effects System

Overview

The Effects System in OpenVideo is a modular, GPU-accelerated pipeline for applying visual transformations to clips during playback and rendering. Built on top of PixiJS and leveraging the WebCodecs API, the system provides both built-in effects (blur, brightness, color adjustment, chromakey) and a shader-based GLSL extension framework for custom visual effects. The architecture separates effect definition from effect application, allowing effects to be applied at the clip level with configurable timing, duration, and intensity.

Effects are defined as standalone modules in packages/openvideo/src/effect/ and can be attached to any clip through the timeline system. The system supports real-time preview during editing and high-quality final rendering through WebCodecs encoding.

Architecture

System Components

The Effects System consists of several interconnected layers:

graph TD
    A[Clip Instance] --> B[Effect Manager]
    B --> C[Built-in Effects]
    B --> D[Custom GLSL Effects]
    C --> E[Blur Effect]
    C --> F[Color Adjustment]
    C --> G[Chromakey Effect]
    D --> H[GLSL Fragment Shaders]
    D --> I[GLSL Vertex Shaders]
    B --> J[Compositor Rendering]
    J --> K[PixiJS Renderer]
    K --> L[WebCodecs Encoder]

Directory Structure

packages/openvideo/src/
├── effect/
│   ├── effect.ts           # Core Effect class and base implementation
│   ├── types.ts            # TypeScript type definitions for effects
│   ├── vertex.ts           # Vertex shader utilities
│   ├── glsl/
│   │   ├── gl-effect.ts    # WebGL effect renderer
│   │   └── custom-glsl.ts  # Custom shader registration system
│   └── interface/
│       └── index.ts        # Effect interface contracts
├── utils/
│   ├── chromakey.ts        # Green screen/chromakey utilities
│   └── color-adjustment.ts # Color correction utilities

Core Effect Interface

IEffect Interface

All effects must implement the IEffect interface defined in packages/openvideo/src/effect/interface/index.ts:

Sources: packages/openvideo/src/effect/interface/index.ts

EffectOptions Type

The EffectOptions type in packages/openvideo/src/effect/types.ts provides the base configuration structure:

interface EffectOptions {
  id?: string;
  key?: string;
  startTime?: number;
  duration?: number;
  targets?: number[];
  enabled?: boolean;
  [key: string]: any;  // Effect-specific parameters
}

Sources: packages/openvideo/src/effect/types.ts

Built-in Effects

Blur Effect

The blur effect applies Gaussian blur to the clip using PixiJS's built-in blur filter. Configuration includes blur radius and quality settings.

graph LR
    A[Input Texture] --> B[BlurFilter]
    B --> C[Radius Parameter]
    C --> D[Quality Iterations]
    D --> E[Output Texture]

Color Adjustment Effect

Color adjustments are applied through the ColorAdjustment type defined in packages/openvideo/src/utils/color-adjustment.ts. This effect modifies the visual characteristics of the source content:

Sources: packages/openvideo/src/utils/color-adjustment.ts

Chromakey Effect

The chromakey utility in packages/openvideo/src/utils/chromakey.ts implements green screen (chroma key) removal for video composition. This effect:

• Detects pixels matching a target color within a tolerance range
• Replaces matching pixels with transparency
• Supports spill suppression to remove color artifacts

graph TD
    A[Source Frame] --> B[Color Matching]
    B --> C{Tolerance Check}
    C -->|Match| D[Set Alpha = 0]
    C -->|No Match| E[Keep Pixel]
    D --> F[Spill Suppression]
    E --> F
    F --> G[Output Frame]

GLSL Shader Effects

Custom GLSL System

The custom shader system allows developers to register their own fragment and vertex shaders. The system is built around two main components:

1. GL Effect Renderer (packages/openvideo/src/effect/glsl/gl-effect.ts) - Handles WebGL shader compilation and execution
2. Custom GLSL Registry (packages/openvideo/src/effect/glsl/custom-glsl.ts) - Manages shader registration and retrieval

Shader Structure

Each custom shader effect follows a standard structure:

graph TD
    A[CustomShader Object] --> B[Fragment Shader]
    A --> C[Vertex Shader]
    A --> D[Uniforms]
    A --> E[Sampler Inputs]
    B --> F[GLSL Main Function]
    D --> G[uniform float time]
    D --> H[uniform vec2 resolution]
    D --> I[Effect Parameters]

Vertex Shader Utilities

The vertex.ts file provides common vertex shader utilities and transformations for effects that require custom vertex manipulation:

Sources: packages/openvideo/src/effect/vertex.ts

Effect Application in Clips

Effect Integration with Clips

Effects are attached to clips through the effects array in the clip's configuration. Each effect entry references an effect by key and timing:

graph TD
    A[Clip JSON Config] --> B[effects Array]
    B --> C[Effect Entry 1]
    B --> D[Effect Entry 2]
    C --> E{id: string}
    C --> F{key: string}
    C --> G{startTime: number}
    C --> H{duration: number}

The base clip JSON structure from packages/openvideo/src/json-serialization.ts shows the effect definition format:

effects?: Array<{
  id: string;
  key: string;
  startTime: number;
  duration: number;
  targets?: number[];
}>;

Sources: packages/openvideo/src/json-serialization.ts

Effect Timing and Targeting

Effects support precise temporal control:

Sprite-Level Effect Properties

The base sprite system in packages/openvideo/src/sprite/base-sprite.ts defines render-time effect properties that are applied during the render transform calculation:

These properties are combined from multiple animation sources at render time:

graph TD
    A[Base Transform] --> C[Combined Transform]
    B[Animation Transform] --> C
    E[Effect Properties] --> C
    C --> D[Final Render]

Sources: packages/openvideo/src/sprite/base-sprite.ts

Effect Processing Pipeline

Render Pipeline Integration

The effects system integrates with the overall render pipeline as follows:

sequenceDiagram
    participant Timeline
    participant Clip
    participant EffectManager
    participant Compositor
    participant WebCodecs
    
    Timeline->>Clip: currentTime update
    Clip->>EffectManager: getActiveEffects(time)
    EffectManager->>EffectManager: filter by startTime/duration
    EffectManager-->>Compositor: activeEffects[]
    Compositor->>Compositor: applyEffects(frame)
    Compositor->>WebCodecs: encodeFrame()

Effect Order of Application

Effects are applied in a deterministic order:

1. Color Adjustments - Brightness, contrast, saturation
2. Blur Effects - Applied as PixiJS filters
3. Custom GLSL Shaders - User-defined effects
4. Chromakey - Applied last for proper compositing

Effect Configuration API

Creating an Effect

import { Effect } from 'openvideo';

// Built-in effect
const blurEffect = new Effect({
  key: 'blur',
  startTime: 0,
  duration: 5000000,  // 5 seconds in microseconds
  intensity: 10
});

// Custom GLSL effect
const customEffect = new Effect({
  key: 'my-custom-shader',
  startTime: 1000000,
  duration: 3000000,
  uniforms: {
    color: [1.0, 0.5, 0.0],
    strength: 0.8
  }
});

Attaching Effects to Clips

const videoClip = await Video.fromUrl('video.mp4');
videoClip.effects = [
  {
    id: 'effect-1',
    key: 'colorAdjustment',
    startTime: 0,
    duration: clip.duration,
    brightness: 1.2,
    saturation: 0.8
  }
];

Custom Shader Registration

Registering Custom Shaders

Developers can register custom GLSL shaders through the custom shader registry:

import { registerCustomShader } from 'openvideo';

registerCustomShader({
  name: 'my-custom-effect',
  fragmentShader: `
    uniform float strength;
    uniform vec3 color;
    
    void main() {
      vec4 texture = texture2D(uSampler, vTextureCoord);
      vec3 adjusted = mix(texture.rgb, color, strength);
      gl_FragColor = vec4(adjusted, texture.a);
    }
  `,
  uniforms: [
    { name: 'strength', type: 'float', value: 0.5 },
    { name: 'color', type: 'vec3', value: [1, 1, 1] }
  ]
});

Sources: packages/openvideo/src/effect/glsl/custom-glsl.ts

Performance Considerations

GPU vs CPU Effects

Optimization Strategies

1. Batch similar effects - Group effects that use the same shader
2. Use appropriate precision - Prefer lowp or mediump where possible
3. Limit effect count - Excessive effect stacking impacts performance
4. Consider resolution scaling - Lower resolution during preview, full resolution for export

Related Components

Related Pages

Related topics: Effects System, Animation System

Transitions System

The Transitions System is a core module within the openvideo library that enables visual transitions between video clips using WebGL fragment shaders. It provides over 30 built-in transition effects including wipes, fades, geometric transformations, and artistic effects, all implemented through GLSL shaders with customizable uniforms.

Architecture Overview

The transitions system follows a layered architecture that separates shader definitions, uniform management, and transition lookup logic.

graph TD
    A[Transition Request] --> B[Name Resolution]
    B --> C[Variant Matching]
    C --> D{Gound?}
    D -->|Yes| E[Select GLSL Fragment]
    D -->|No| F[Variant Generation]
    F --> G{Lowercase match?}
    G -->|Yes| E
    G -->|No| H[Error/Default]
    E --> I[Uniform Assembly]
    I --> J[WebGL Render]
    
    K[Basic Uniforms] --> I
    L[Transition-Specific Uniforms] --> I

Core Components

Sources: packages/openvideo/src/transition/transition.ts:1-50

Transition Types

The system includes the following built-in transitions, categorized by their visual effect:

Wipe & Reveal Transitions

Geometric Transitions

Fade & Blend Transitions

Distortion Transitions

Creative Transitions

Sources: packages/openvideo/src/transition/transition.ts:60-150

Name Resolution & Matching

The transition system supports flexible name matching to accommodate various naming conventions used by developers.

Matching Algorithm

graph LR
    A[Input Name] --> B[Direct Match]
    A --> C[Lowercase Match]
    A --> D[Capitalized Match]
    A --> E[Snake_case Match]
    A --> F[No Separator Match]
    
    B --> G{Match Found?}
    C --> G
    D --> G
    E --> G
    F --> G
    
    G -->|Yes| H[Return Transition]
    G -->|No| I[Error with Available Names]

Variant Generation Strategy

The system attempts multiple name transformations when looking up transitions:

1. Exact match - transition.name === input
2. Lowercase match - name.toLowerCase() === input.toLowerCase()
3. Capitalized match - name.charAt(0).toUpperCase() + name.slice(1).toLowerCase()
4. Snake_case conversion - name.replace(/([A-Z])/g, '_$1').toLowerCase()
5. No separator - name.replace(/_/g, '').toLowerCase()

Sources: packages/openvideo/src/transition/transition.ts:180-220

Uniform System

Transitions use uniforms to pass parameters to GLSL shaders. The uniform system handles type conversion and default values.

Basic Uniforms

All transitions receive these fundamental uniforms:

interface BasicUniforms {
  tex: sampler2D;      // Source texture
  tex2: sampler2D;     // Target texture
  progress: number;    // Transition progress (0-1)
  time: number;        // Current time
  resolution: vec2;    // Canvas resolution
}

Uniform Type Conversion

The system automatically converts WebGPU-style uniform types to WebGL-compatible formats:

Sources: packages/openvideo/src/transition/transition.ts:100-115

Transition-Specific Uniforms

Each transition type defines its own uniforms that extend the basic set:

// Example: GridFlip uniforms
const GRIDFLIP_UNIFORMS = {
  gridSize: { type: 'f32', value: 10.0 },
  axis: { type: 'i32', value: 0 },
};

// Example: Circle uniforms
const CIRCLEOPEN_UNIFORMS = {
  radius: { type: 'f32', value: 1.5 },
  feather: { type: 'f32', value: 0.05 },
};

Sources: packages/openvideo/src/transition/uniforms.ts

GLSL Shader Structure

Vertex Shader

The vertex shader renders a full-screen quad for transition effects:

attribute vec2 a_position;
attribute vec2 a_texCoord;
varying vec2 v_texCoord;

void main() {
    gl_Position = vec4(a_position, 0.0, 1.0);
    v_texCoord = a_texCoord;
}

Fragment Shader Pattern

Each transition implements a fragment shader with the following signature:

uniform sampler2D tex;     // From texture
uniform sampler2D tex2;    // To texture
uniform float progress;    // 0.0 to 1.0
uniform vec2 resolution;   // Viewport size

// Transition-specific uniforms...

varying vec2 v_texCoord;

vec4 getFromColor(vec2 uv) {
    return texture2D(tex, uv);
}

vec4 getToColor(vec2 uv) {
    return texture2D(tex2, uv);
}

void main() {
    // Transition-specific logic
    vec4 from = getFromColor(uv);
    vec4 to = getToColor(uv);
    gl_FragColor = mix(from, to, progress);
}

Sources: packages/openvideo/src/transition/vertex.ts Sources: packages/openvideo/src/transition/fragment.ts

Rendering Pipeline

graph TD
    A[Timeline Position] --> B{Transition Zone?}
    B -->|No| C[Single Clip Render]
    B -->|Yes| D[Load From/To Textures]
    D --> E[Calculate Progress]
    E --> F[Set Uniform Values]
    F --> G[Compile GLSL if needed]
    G --> H[Render Transition Shader]
    H --> I[Output Composite Frame]
    
    J[Basic Uniforms] --> F
    K[Transition Uniforms] --> F
    L[Custom Uniforms] --> F

Rendering Steps

1. Texture Binding - Source and destination textures are bound to texture units
2. Progress Calculation - Transition progress (0-1) is calculated from timeline position
3. Uniform Assembly - Basic uniforms are merged with transition-specific uniforms
4. Shader Execution - WebGL renders the full-screen quad with the transition shader
5. Output - The composited frame is written to the canvas

Sources: packages/openvideo/src/transition/glsl/gl-transition.ts

Custom Transitions

Creating Custom GLSL

Developers can create custom transitions by implementing the fragment shader pattern:

import { GLTransition } from './types';

const CUSTOM_FRAGMENT = `
uniform float customParam;
varying vec2 v_texCoord;

vec4 getFromColor(vec2 uv) {
    return texture2D(tex, uv);
}

vec4 getToColor(vec2 uv) {
    return texture2D(tex2, uv);
}

// Custom transition logic
void main() {
    vec2 uv = v_texCoord;
    vec4 from = getFromColor(uv);
    vec4 to = getToColor(uv);
    
    float ratio = 1.0 - customParam;
    float pr = smoothstep(0.0, ratio, progress);
    
    gl_FragColor = mix(from, to, pr);
}
`;

const customTransition: GLTransition = {
  name: 'custom',
  fragment: CUSTOM_FRAGMENT,
  uniforms: {
    customParam: { type: 'f32', value: 0.5 },
  },
};

Sources: packages/openvideo/src/transition/glsl/custom-glsl.ts

Registering Custom Transitions

Custom transitions can be registered into the local transition registry for flexible lookup:

import { registerTransition, getAllTransitions } from './transition';

// Register custom transition
registerTransition('myCustom', customTransition);

// Retrieve all transitions including custom
const all = getAllTransitions();

Type Definitions

GLTransition Interface

interface GLTransition {
  name: string;           // Unique identifier
  label?: string;         // Display name
  fragment: string;       // GLSL fragment shader source
  vertex?: string;        // Custom vertex shader (optional)
  uniforms: {
    [key: string]: UniformDefinition;
  };
}

interface UniformDefinition {
  type: 'f32' | 'i32' | 'vec2<f32>' | 'vec4<f32>';
  value: number | number[];
}

Sources: packages/openvideo/src/transition/types.ts

Performance Considerations

Shader Compilation

• GLSL shaders are compiled once and cached for reuse
• Subsequent uses of the same transition use the cached program

Texture Sampling

• Transitions use bilinear filtering for smooth interpolation
• Mipmaps are generated for efficient downsampling during blur effects

Uniform Updates

• Progress uniform is updated every frame during transitions
• Type conversion is performed once when the transition starts, not per-frame

Error Handling

When a transition cannot be found, the system provides helpful error messages:

// Error message format
`Transition '${name}' not found. Available: ${availableNames}...`

The system attempts to suggest similar transitions by checking the first 5 available transitions and 3 local definitions.

Sources: packages/openvideo/src/transition/transition.ts:230-250

See Also

• Studio System - High-level composition API that uses transitions
• Clip System - Video clip handling with transition support
• Animation Presets - Keyframe animation system

Related Pages

Related topics: Compositor and Rendering, Clip System

Utilities and Helpers

Overview

The openvideo library provides a comprehensive suite of utility modules under packages/openvideo/src/utils/ that support core functionality across the video rendering pipeline. These utilities handle resource management, media processing, formatting, and color manipulation—enabling the higher-level components like Studio, Compositor, and various clip types to operate efficiently.

Architecture Overview

graph TD
    subgraph "Utility Layer"
        UTILS_INDEX[utils/index.ts<br/>Module Exports]
        ASSET_MGR[asset-manager.ts<br/>Resource Management]
        AUDIO_UTIL[audio.ts<br/>Audio Processing]
        SRT[SRT Parser<br/>Subtitle Handling]
        FONTS[fonts.ts<br/>Font Management]
        VIDEO_UTIL[video.ts<br/>Video Utilities]
        COLOR[color.ts<br/>Color Operations]
        COMMON[common.ts<br/>Common Helpers]
    end
    
    subgraph "Consumer Components"
        STUDIO[Studio]
        COMPOSITOR[Compositor]
        VIDEO_CLIP[VideoClip]
        TEXT_CLIP[TextClip]
        CAPTION_CLIP[CaptionClip]
    end
    
    UTILS_INDEX --> ASSET_MGR
    UTILS_INDEX --> AUDIO_UTIL
    UTILS_INDEX --> SRT
    UTILS_INDEX --> FONTS
    UTILS_INDEX --> VIDEO_UTIL
    UTILS_INDEX --> COLOR
    UTILS_INDEX --> COMMON
    
    ASSET_MGR --> VIDEO_CLIP
    ASSET_MGR --> STUDIO
    FONTS --> TEXT_CLIP
    AUDIO_UTIL --> COMPOSITOR
    COLOR --> TEXT_CLIP
    COLOR --> CAPTION_CLIP
    VIDEO_UTIL --> VIDEO_CLIP

Module Descriptions

Asset Manager (`asset-manager.ts`)

The asset manager provides centralized resource handling capabilities for loading and managing media assets. It abstracts the complexities of OPFS (Origin Private File System) operations and provides consistent interfaces for accessing streams and files.

Key Responsibilities:

• Stream management for remote and local resources
• OPFS file read/write operations
• Resource lifecycle management

Source: packages/openvideo/src/utils/asset-manager.ts:1-50

Audio Utilities (`audio.ts`)

Audio utilities provide processing functions for audio tracks within video compositions. These utilities handle audio decoding, volume control, and track synchronization.

Key Functions:

Source: packages/openvideo/src/utils/audio.ts:1-40

SRT Parser (`srt-parser.ts`)

The SRT parser handles SubRip subtitle file parsing and conversion. It transforms SRT format subtitles into structured caption data usable by the CaptionClip component.

Parsing Flow:

graph LR
    SRT_FILE[SRT File] --> PARSER[SRT Parser]
    PARSER --> CAPTION_DATA[Caption Data Structure]
    CAPTION_DATA --> CAPTION_CLIP[CaptionClip]

Source: packages/openvideo/src/utils/srt-parser.ts:1-60

Font Management (`fonts.ts`)

Font utilities manage font loading, caching, and fallback strategies for text rendering in TextClip and CaptionClip components.

Font Loading Strategy:

Source: packages/openvideo/src/utils/fonts.ts:1-45

Video Utilities (`video.ts`)

Video utilities provide helper functions for video frame processing, thumbnail generation, and metadata extraction. These utilities support the VideoClip component's frame-seeking and thumbnail capabilities.

Source: packages/openvideo/src/utils/video.ts:1-55

Color Utilities (`color.ts`)

Color utilities handle color format conversion, validation, and transformation for consistent styling across text, captions, and visual effects.

Supported Color Formats:

Usage in Clips:

The TextClip and CaptionClip components utilize color utilities for:

• fill property for text color
• stroke color configuration
• dropShadow color and alpha values

Source: packages/openvideo/src/utils/color.ts:1-35

Common Utilities (`common.ts`)

Common utilities provide shared helper functions used throughout the library, including type guards, validation helpers, and general-purpose transformations.

Source: packages/openvideo/src/utils/common.ts:1-30

Integration with Core Components

VideoClip Integration

The VideoClip class relies on utility modules for resource loading and video processing:

// From packages/openvideo/src/clips/video-clip.ts:42-55
const stream = await ResourceManager.getReadableStream(url);
const clip = new Video(stream, {}, url);
await clip.ready;

TextClip Integration

Text styling leverages color utilities for consistent rendering:

// From packages/openvideo/src/clips/text-clip.ts:95-110
if (this.originalOpts.fill !== undefined)
  style.color = this.originalOpts.fill as any;
if (this.originalOpts.stroke) {
  style.stroke = {
    color: this.originalOpts.stroke.color as any,
    width: this.originalOpts.stroke.width,
  };
}

CaptionClip Integration

Caption styling combines color, font, and formatting utilities:

// From packages/openvideo/src/clips/caption-clip.ts:120-135
if (opts.dropShadow) {
  style.shadow = {
    color: (opts.dropShadow.color ?? '#000000') as string,
    alpha: opts.dropShadow.alpha ?? 0.5,
    blur: opts.dropShadow.blur ?? 4,
    distance: opts.dropShadow.distance ?? 0,
    angle: opts.dropShadow.angle ?? 0,
  };
}

Module Export Pattern

All utilities are exported through the central utils/index.ts entry point:

// packages/openvideo/src/utils/index.ts
export * from './asset-manager';
export * from './audio';
export * from './srt-parser';
export * from './fonts';
export * from './video';
export * from './color';
export * from './common';

This pattern allows consumers to import specific utilities as needed:

import { ResourceManager, ColorUtils } from 'openvideo';

Utility Functions Reference

Resource Management

Color Operations

Font Operations

Summary

The utility modules in openvideo form a foundational layer that abstracts common operations away from business logic. By centralizing resource management, media processing, color handling, and font operations, the library maintains consistency and reduces duplication across components like VideoClip, TextClip, and CaptionClip. These utilities are designed to be composable and are exported through a single entry point for convenient consumption.

Doramagic Pitfall Log

Doramagic extracted 11 source-linked risk signals. Review them before installing or handing real data to the project.

1. Installation risk: Modular architecture: separate core logic from PixiJS renderer to enable React Native / headless use

• Severity: medium
• Finding: Installation risk is backed by a source signal: Modular architecture: separate core logic from PixiJS renderer to enable React Native / headless use. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/openvideodev/openvideo/issues/78

2. Configuration risk: Image and Audio clips lost during loadFromJSON/exportToJSON round-trip

• Severity: medium
• Finding: Configuration risk is backed by a source signal: Image and Audio clips lost during loadFromJSON/exportToJSON round-trip. Treat it as a review item until the current version is checked.
• User impact: Users may get misleading failures or incomplete behavior unless configuration is checked carefully.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/openvideodev/openvideo/issues/60

3. Configuration risk: [Bug] Transition effect "display" value update not reflected in Player after clip repositioning

• Severity: medium
• Finding: Configuration risk is backed by a source signal: [Bug] Transition effect "display" value update not reflected in Player after clip repositioning. Treat it as a review item until the current version is checked.
• User impact: Users may get misleading failures or incomplete behavior unless configuration is checked carefully.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/openvideodev/openvideo/issues/52

4. Configuration risk: display property not initialized for Image clips

• Severity: medium
• Finding: Configuration risk is backed by a source signal: display property not initialized for Image clips. Treat it as a review item until the current version is checked.
• User impact: Users may get misleading failures or incomplete behavior unless configuration is checked carefully.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/openvideodev/openvideo/issues/18

5. Configuration risk: meta property is read-only

• Severity: medium
• Finding: Configuration risk is backed by a source signal: meta property is read-only. Treat it as a review item until the current version is checked.
• User impact: Users may get misleading failures or incomplete behavior unless configuration is checked carefully.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/openvideodev/openvideo/issues/19

6. Capability assumption: README/documentation is current enough for a first validation pass.

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: The project should not be treated as fully validated until this signal is reviewed.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: capability.assumptions | github_repo:1125747446 | https://github.com/openvideodev/openvideo | README/documentation is current enough for a first validation pass.

7. Maintenance risk: Maintainer activity is unknown

• Severity: medium
• Finding: Maintenance risk is backed by a source signal: Maintainer activity is unknown. Treat it as a review item until the current version is checked.
• User impact: Users cannot judge support quality until recent activity, releases, and issue response are checked.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: evidence.maintainer_signals | github_repo:1125747446 | https://github.com/openvideodev/openvideo | last_activity_observed missing

8. Security or permission risk: no_demo

• Severity: medium
• Finding: no_demo
• User impact: The project may affect permissions, credentials, data exposure, or host boundaries.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: downstream_validation.risk_items | github_repo:1125747446 | https://github.com/openvideodev/openvideo | no_demo; severity=medium

9. Security or permission risk: no_demo

• Severity: medium
• Finding: no_demo
• User impact: The project may affect permissions, credentials, data exposure, or host boundaries.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: risks.scoring_risks | github_repo:1125747446 | https://github.com/openvideodev/openvideo | no_demo; severity=medium

10. Maintenance risk: issue_or_pr_quality=unknown

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: Users cannot judge support quality until recent activity, releases, and issue response are checked.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: evidence.maintainer_signals | github_repo:1125747446 | https://github.com/openvideodev/openvideo | issue_or_pr_quality=unknown

11. Maintenance risk: release_recency=unknown

• Severity: low
• Finding: release_recency=unknown。
• User impact: Users cannot judge support quality until recent activity, releases, and issue response are checked.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: evidence.maintainer_signals | github_repo:1125747446 | https://github.com/openvideodev/openvideo | release_recency=unknown

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using openvideo with real data or production workflows.

• display property not initialized for Image clips - github / github_issue
• [[Bug] Transition effect "display" value update not reflected in Player a](https://github.com/openvideodev/openvideo/issues/52) - github / github_issue
• Image and Audio clips lost during loadFromJSON/exportToJSON round-trip - github / github_issue
• Modular architecture: separate core logic from PixiJS renderer to enable - github / github_issue
• meta property is read-only - github / github_issue
• README/documentation is current enough for a first validation pass. - GitHub / issue