# https://github.com/openvideodev/openvideo 项目说明书
生成时间：2026-05-15 09:42:56 UTC
## 目录
- [Overview](#page-1)
- [Installation and Setup](#page-2)
- [System Architecture](#page-3)
- [Studio and State Management](#page-4)
- [Clip System](#page-5)
- [Animation System](#page-6)
- [Compositor and Rendering](#page-7)
- [Effects System](#page-8)
- [Transitions System](#page-9)
- [Utilities and Helpers](#page-10)
## Overview
### 相关页面
相关主题：[System Architecture](#page-3), [Clip System](#page-5)
相关源码文件
以下源码文件用于生成本页说明：
- [README.md](https://github.com/openvideodev/openvideo/blob/main/README.md)
- [packages/openvideo/src/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/index.ts)
- [packages/openvideo/src/studio.spec.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio.spec.ts)
- [packages/node/README.md](https://github.com/openvideodev/openvideo/blob/main/packages/node/README.md)
- [packages/openvideo/src/transition/transition.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/transition.ts)
# Overview
## What is OpenVideo?
OpenVideo is a powerful, browser-based video composition and rendering library that enables developers to create, edit, and export video content directly in the web environment. Built on modern web APIs, OpenVideo provides a high-performance alternative to traditional desktop video editing software while maintaining the accessibility and portability of web applications.
The library leverages hardware-accelerated video processing through the WebCodecs API, combined with PixiJS for robust 2D/3D rendering, making it suitable for both client-side video editing interfaces and server-side rendering pipelines.
## Core Architecture
OpenVideo follows a modular architecture with three primary layers:
```mermaid
graph TD
A[User Application] --> B[Studio Layer]
B --> C[Compositor Layer]
C --> D[Rendering Engine
PixiJS + WebCodecs]
B --> E[Clip Objects]
E --> E1[Video]
E --> E2[Audio]
E --> E3[Text]
E --> E4[Image]
E --> E5[Caption]
C --> F[Compositor]
F --> F1[Video Export]
F --> F2[Playback]
F --> F3[Seeking]
```
### Core Components
| Component | Purpose | Location |
|-----------|---------|----------|
| **Studio** | Manages project state, tracks, clips, and timeline configuration | `packages/openvideo/src/studio.ts` |
| **Compositor** | Rendering engine handling playback, seeking, and final export using WebCodecs | `packages/openvideo/src/compositor.ts` |
| **Clips** | Specialized objects for different media types | `packages/openvideo/src/clips/` |
| **Transitions** | Visual effects between clips | `packages/openvideo/src/transition/` |
| **Animations** | Preset animation utilities for clips | `packages/openvideo/src/animation/` |
## Technology Stack
OpenVideo is built on three foundational technologies that work together to provide high-performance video processing:
```mermaid
graph LR
A[WebCodecs API] --> D[Video Processing]
B[PixiJS] --> E[2D/3D Rendering]
C[wrapbox] --> F[MP4 Muxing]
D --> G[OpenVideo Core]
E --> G
F --> G
```
| Technology | Role | Key Benefits |
|------------|------|--------------|
| **WebCodecs** | Video encoding/decoding | Hardware-accelerated, ultra-fast video processing |
| **PixiJS** | Rendering engine | High-performance 2D/3D graphics with WebGL |
| **wrapbox** | MP4 manipulation | Low-level MP4 box manipulation and muxing |
资料来源：[README.md:15-17]()
## Key Features
### Multi-Media Clip Support
OpenVideo supports a comprehensive set of clip types for video composition:
- **Video Clip** - Load and manipulate video files with frame-accurate control
- **Audio Clip** - Audio tracks with volume control and synchronization
- **Text Clip** - Rich text rendering with fonts, styles, and animations
- **Image Clip** - Static image support for backgrounds and overlays
- **Caption Clip** - Subtitle and caption rendering with styling options
### Visual Transitions
The library includes numerous built-in transitions for professional video effects:
| Category | Available Transitions |
|----------|----------------------|
| **Basic** | Circle, Directional, Crosshatch, Heart |
| **Advanced** | GridFlip, PolkaDotsCurtain, Pixelize, RotateScaleFade |
| **Specialized** | UndulatingBurnOut, SquaresWire, RandomSquares, CannabisLeaf |
| **Function-based** | PolarFunction, Hexagonalize, StereoViewer |
资料来源：[packages/openvideo/src/transition/transition.ts:1-100]()
### Animation Presets
Clips can be animated using preset animations:
- **Fade Effects** - fadeIn, fadeOut
- **Movement** - slideIn, slideOut, pulse
- **Transformations** - blurIn, blurOut, scaleIn, scaleOut
- **Rotation** - rotateIn, rotateOut
资料来源：[packages/openvideo/src/animation/presets.ts:1-80]()
## Package Structure
The OpenVideo repository is organized as a monorepo with multiple packages:
```
openvideo/
├── packages/
│ ├── openvideo/ # Core library
│ │ ├── src/
│ │ │ ├── clips/ # Clip implementations
│ │ │ ├── transition/ # Transition effects
│ │ │ ├── animation/ # Animation presets
│ │ │ ├── studio.ts # Main studio class
│ │ │ └── compositor.ts # Rendering engine
│ │ └── dist/ # Built output
│ │
│ ├── node/ # Node.js server-side renderer
│ │ ├── src/
│ │ │ ├── renderer.ts # Main Renderer class
│ │ │ ├── types.ts # TypeScript definitions
│ │ │ └── template.html # Browser rendering template
│ │ └── dist/ # Built output
│ │
│ └── web/ # Web UI components
│
├── examples/ # Example applications
└── README.md
```
### Core Library (`packages/openvideo`)
The main library providing all core functionality:
```typescript
import { Studio, Video, Text, Image } from 'openvideo';
// Initialize Studio
const studio = new Studio({
canvas: document.getElementById('preview-canvas') as HTMLCanvasElement,
spacing: 20
});
// Add clips
const video = await Video.fromUrl('https://example.com/video.mp4');
await studio.addClip(video);
// Start preview
studio.play();
```
资料来源：[README.md:45-60]()
### Node.js Renderer (`packages/node`)
For server-side video rendering without a browser UI:
```typescript
import { Renderer } from '@combo/node';
const renderer = new Renderer({
json: videoConfig,
outputPath: './output.mp4',
});
renderer.on('progress', ({ phase, progress, message }) => {
console.log(`[${phase}] ${Math.round(progress * 100)}% - ${message}`);
});
await renderer.render();
```
资料来源：[packages/node/README.md:100-120]()
## Rendering Workflow
### Client-Side Rendering
```mermaid
sequenceDiagram
participant User
participant Studio
participant Compositor
participant WebCodecs
User->>Studio: Create project with clips
User->>Studio: Configure timeline
User->>Studio: Start playback/preview
Studio->>Compositor: Render frame
Compositor->>PixiJS: Draw scene
Compositor->>WebCodecs: Encode frame
WebCodecs-->>Compositor: Encoded video data
Compositor-->>User: Display/Preview
```
### Server-Side Rendering (Node.js)
```mermaid
graph TD
A[Create Renderer Instance] --> B[Load JSON Config]
B --> C[Launch Headless Browser]
C --> D[Load HTML Template]
D --> E[Initialize Compositor]
E --> F[Render Frames]
F --> G[Encode Video via WebCodecs]
G --> H[Save to File System]
H --> I[Return Output Path]
```
The Node.js renderer uses Playwright to launch a headless browser, inject the JSON configuration, and use the HTML template to render the video using the same Compositor and WebCodecs engine.
资料来源：[packages/node/README.md:1-50]()
## Studio API Reference
### Initialization
```typescript
const studio = new Studio({
canvas: HTMLCanvasElement, // Required: Target canvas
width?: number, // Default: 1280
height?: number, // Default: 720
spacing?: number, // Default: 20
});
```
### Clip Management
```typescript
// Add a clip
await studio.addClip(clip: BaseClip): Promise;
// Remove a clip by ID
await studio.removeClip(clipId: string): Promise;
// Access clips
studio.clips; // Readonly array of clips
```
资料来源：[packages/openvideo/src/studio.spec.ts:1-50]()
## Compositor Configuration
```typescript
const compositorOpts = {
width: 1280, // Output width
height: 720, // Output height
fps: 30, // Frames per second
bgColor: '#000000', // Background color
videoCodec?: 'avc1.64001f', // Specific codec
bitrate?: 5000000, // Video bitrate
audio?: true, // Include audio track
metaDataTags?: {} // Metadata for output
};
```
## Use Cases
OpenVideo is suitable for various applications:
| Use Case | Implementation | Benefits |
|----------|---------------|----------|
| **Video Editor Web App** | Studio + UI components | Full-featured browser-based editing |
| **Automated Video Generation** | Node.js Renderer | Server-side rendering pipelines |
| **Video Preview** | Compositor directly | Fast preview without export |
| **Social Media Tools** | JSON serialization | Template-based video generation |
| **Educational Platforms** | Caption + Animation | Accessible video content creation |
## Requirements
### Browser Environment
- Modern browser with WebCodecs support (Chrome 94+, Edge 94+)
- WebGL 2.0 support for PixiJS rendering
- ES2020+ JavaScript support
### Node.js Environment
- **Node.js** >= 18
- **Playwright** for headless browser rendering
- **openvideo** package installed
## License
OpenVideo is released under the MIT License.
资料来源：[README.md:70-72]()
---
## Installation and Setup
### 相关页面
相关主题：[Overview](#page-1)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/package.json](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/package.json)
- [packages/node/README.md](https://github.com/openvideodev/openvideo/blob/main/packages/node/README.md)
- [packages/node/src/template.html](https://github.com/openvideodev/openvideo/blob/main/packages/node/src/template.html)
- [packages/openvideo/render.html](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/render.html)
- [examples/package.json](https://github.com/openvideodev/openvideo/blob/main/examples/package.json)
- [examples/index.html](https://github.com/openvideodev/openvideo/blob/main/examples/index.html)
# Installation and Setup
## Overview
This page documents the complete installation and setup process for the openvideo project. openvideo is a video rendering and processing library built on WebCodecs, PixiJS, and wrapbox technologies. The project supports multiple deployment scenarios including browser-based video rendering, Node.js server-side rendering, and integration with frontend frameworks.
## System Requirements
### Runtime Requirements
| Requirement | Minimum Version | Notes |
|-------------|-----------------|-------|
| Node.js | >= 18 | Required for WebCodecs API support |
| pnpm | Latest | Package manager (not npm or yarn) |
| Chromium-based Browser | Any modern version | For WebCodecs functionality |
### Browser Environment Requirements
The openvideo library relies on the **WebCodecs API** for hardware-accelerated video processing. This requires:
- A Chromium-based browser (Chrome, Edge, Brave, Arc)
- Secure context (HTTPS or localhost)
- Access to the `VideoEncoder`, `VideoDecoder`, and `VideoFrame` APIs
资料来源：[packages/openvideo/package.json:1-30](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/package.json)
## Project Structure
The openvideo repository follows a monorepo structure using pnpm workspaces:
```
openvideo/
├── packages/
│ ├── openvideo/ # Core rendering library
│ │ ├── src/
│ │ │ ├── clips/ # Video, Audio, Text, Image clips
│ │ │ ├── animation/ # Animation presets
│ │ │ ├── transition/ # Transition effects
│ │ │ └── mp4-utils/ # MP4 muxing utilities
│ │ ├── dist/ # Built output
│ │ └── package.json
│ └── node/ # Node.js renderer package
│ ├── src/
│ │ ├── renderer.ts # Server-side renderer
│ │ ├── types.ts # Type definitions
│ │ └── template.html
│ └── package.json
└── examples/ # Example applications
```
## Installation Steps
### 1. Install pnpm
If you don't have pnpm installed, install it globally:
```bash
npm install -g pnpm
```
### 2. Install Dependencies
Clone the repository and install all dependencies:
```bash
git clone https://github.com/openvideodev/openvideo.git
cd openvideo
pnpm install
```
### 3. Build the Core Package
Build the main openvideo package before using it:
```bash
pnpm --filter openvideo build
```
This generates the distribution files in `packages/openvideo/dist/`:
- `index.es.js` - ES Module build
- `index.umd.js` - UMD build for CDN usage
- `index.d.ts` - TypeScript definitions
资料来源：[packages/openvideo/package.json:23-25](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/package.json)
### 4. Build Node.js Renderer (Optional)
For server-side rendering capabilities:
```bash
pnpm --filter @combo/node build
```
资料来源：[packages/node/README.md:1-100](https://github.com/openvideodev/openvideo/blob/main/packages/node/README.md)
## Browser Usage
### Script Tag (UMD)
Include openvideo directly via CDN:
```html
Video Render
```
### ES Module with Import Map
For modern browser applications using ES modules:
```html
Video Render
```
资料来源：[packages/node/src/template.html:1-25](https://github.com/openvideodev/openvideo/blob/main/packages/node/src/template.html)
## Node.js Server-Side Rendering
The `@combo/node` package enables server-side video rendering using Playwright and Chromium.
### Renderer Configuration
```typescript
import { Renderer } from '@combo/node';
const renderer = new Renderer({
json: videoConfig, // Video configuration JSON
outputPath: './output.mp4', // Output file path
browserOptions: {
headless: true, // Default: true
timeout: 300000 // Default: 5 minutes
}
});
```
### Renderer Options
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `json` | `object` | Required | Video composition configuration |
| `outputPath` | `string` | Required | Path for output MP4 file |
| `browserOptions.headless` | `boolean` | `true` | Run browser in headless mode |
| `browserOptions.timeout` | `number` | `300000` | Browser operation timeout (ms) |
### Rendering Events
The Renderer extends `EventEmitter` and emits the following events:
```typescript
// Progress tracking
renderer.on('progress', ({ phase, progress, message }) => {
console.log(`[${phase}] ${Math.round(progress * 100)}% - ${message}`);
});
// Error handling
renderer.on('error', (error) => {
console.error('Render failed:', error);
});
// Completion
renderer.on('complete', (path) => {
console.log(`Video saved to: ${path}`);
});
// Start rendering
await renderer.render();
```
资料来源：[packages/node/README.md:60-120](https://github.com/openvideodev/openvideo/blob/main/packages/node/README.md)
## Build Workflow
```mermaid
graph TD
A[pnpm install] --> B[Build openvideo Core]
B --> C{Use Case}
C -->|Browser App| D[Import openvideo ES Module]
C -->|Server Rendering| E[Build @combo/node]
C -->|Development| F[Run Examples]
D --> G[Create Compositor Instance]
E --> H[Create Renderer Instance]
G --> I[Configure width, height, fps]
H --> J[Configure JSON config]
I --> K[Add clips to Studio]
J --> K
K --> L[Render video]
L --> M[Output MP4]
F --> N[Start dev server]
N --> O[Browse examples]
```
## Development
### Running Examples
Start the development server for the examples application:
```bash
cd examples
pnpm dev
```
This serves the examples at `http://localhost:5173` (Vite default).
资料来源：[examples/package.json](https://github.com/openvideodev/openvideo/blob/main/examples/package.json)
### Testing
Run unit tests:
```bash
pnpm --filter openvideo test
```
Run browser-specific tests:
```bash
pnpm --filter openvideo test:browser
```
资料来源：[packages/openvideo/package.json:23-25](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/package.json)
### Development Workflow
1. **Make source changes** in `packages/openvideo/src/`
2. **Rebuild the package:**
```bash
pnpm --filter openvideo build
```
3. **Test changes:**
```bash
node dist/sample.js
```
4. **For Node.js renderer changes:**
```bash
pnpm --filter @combo/node build
```
## Troubleshooting
### WebCodecs Not Available
If you encounter `VideoEncoder is not defined`:
- Ensure you're running in a Chromium-based browser
- Verify you're on a secure origin (HTTPS or localhost)
- Check that WebCodecs API is enabled in browser settings
### Module Resolution Errors
If ES module imports fail:
```bash
# Ensure openvideo is built first
pnpm --filter openvideo build
```
### Playwright Installation Issues
For Node.js renderer Chromium installation failures:
```bash
npx playwright install chromium
```
### Timeout Errors
For large video renders, increase the timeout:
```typescript
const renderer = new Renderer({
json: videoConfig,
outputPath: './output.mp4',
browserOptions: {
timeout: 600000 // 10 minutes
}
});
```
## Package Exports
The main openvideo package exports the following from `packages/openvideo/package.json`:
| Export | Type | Path |
|--------|------|------|
| ES Module | `import` | `./dist/index.es.js` |
| UMD | `require` | `./dist/index.umd.js` |
| Types | TypeScript | `./dist/index.d.ts` |
```javascript
// ES Module import
import { Studio, Compositor, Video, Text, Image } from 'openvideo';
// UMD/CommonJS require
const { Studio } = require('openvideo');
```
## Key Components for Setup
| Component | Purpose | Import |
|-----------|---------|--------|
| `Studio` | Project state management, timeline, tracks | `import { Studio } from 'openvideo'` |
| `Compositor` | Rendering engine, WebCodecs export | `import { Compositor } from 'openvideo'` |
| `Video` | Video clip handling | `import { Video } from 'openvideo'` |
| `Renderer` | Server-side rendering (Node.js) | `import { Renderer } from '@combo/node'` |
资料来源：[packages/openvideo/render.html:1-40](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/render.html)
---
## System Architecture
### 相关页面
相关主题：[Overview](#page-1), [Studio and State Management](#page-4), [Clip System](#page-5)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/studio.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio.ts)
- [packages/openvideo/src/compositor.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/compositor.ts)
- [packages/openvideo/src/clips/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/index.ts)
- [packages/openvideo/src/clips/video-clip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/video-clip.ts)
- [packages/openvideo/src/clips/text-clip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/text-clip.ts)
- [packages/openvideo/src/sprite/base-sprite.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
- [packages/openvideo/src/transition/transition.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/transition.ts)
- [packages/openvideo/src/animation/presets.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/presets.ts)
- [packages/openvideo/src/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/index.ts)
# System Architecture
## Overview
The openvideo system is a browser-based video editing and rendering engine built on modern web APIs. It provides a comprehensive architecture for composing, editing, and exporting video content directly in the browser environment. The system leverages **WebCodecs** for hardware-accelerated video processing, **PixiJS** for 2D/3D rendering, and internal utilities for MP4 box manipulation and muxing.
资料来源：[README.md:1-50]()
## High-Level Architecture
The system follows a layered architecture with distinct responsibilities at each level:
```mermaid
graph TD
A[User Application] --> B[Studio]
B --> C[Compositor]
C --> D[PixiJS Rendering Engine]
C --> E[WebCodecs API]
E --> F[MP4 Muxing]
B --> G[Clips Layer]
G --> H[Video Clip]
G --> I[Audio Clip]
G --> J[Text Clip]
G --> K[Image Clip]
G --> L[Caption Clip]
G --> M[Shape Clip]
```
资料来源：[packages/openvideo/src/index.ts:1-100]()
## Core Components
### Studio
The `Studio` class serves as the central orchestrator for the entire video editing system. It manages:
- **Project state**: Tracks, clips, and timeline configuration
- **Preview playback**: Real-time video preview with seeking capabilities
- **Export coordination**: Manages the rendering pipeline for final output
```mermaid
graph LR
A[Studio] --> B[addClip]
A --> C[removeClip]
A --> D[play]
A --> E[pause]
A --> F[seek]
A --> G[export]
```
资料来源：[packages/openvideo/src/studio.ts:1-50]()
### Compositor
The `Compositor` is the rendering engine responsible for:
- **Playback control**: Managing frame-accurate video playback
- **Seeking**: Navigating to specific timestamps in the timeline
- **Final export**: Encoding and muxing the video output using WebCodecs
```typescript
const compositorOpts = {
width: settings.width || 1280,
height: settings.height || 720,
fps: settings.fps || 30,
bgColor: settings.bgColor || '#000000',
videoCodec?: settings.videoCodec,
bitrate?: settings.bitrate,
audio?: boolean,
metaDataTags?: metaDataTags,
};
```
资料来源：[packages/openvideo/render.html:30-50]()
### Clips Layer
The clip system provides specialized objects for different media types. All clips inherit from a base sprite class that handles common properties and animations.
| Clip Type | Purpose | Source |
|-----------|---------|--------|
| `Video` | Video file playback with audio support | [video-clip.ts](packages/openvideo/src/clips/video-clip.ts:1-50) |
| `Audio` | Audio playback and mixing | [audio-clip.ts](packages/openvideo/src/clips/audio-clip.ts) |
| `Text` | Text rendering with styling options | [text-clip.ts](packages/openvideo/src/clips/text-clip.ts:1-50) |
| `Image` | Static image display | [image-clip.ts](packages/openvideo/src/clips/image-clip.ts) |
| `Caption` | Word-level caption rendering | [caption-clip.ts](packages/openvideo/src/clips/caption-clip.ts:1-50) |
| `Shape` | Vector shapes and graphics | [shape-clip.ts](packages/openvideo/src/clips/shape-clip.ts) |
## Clip Architecture
### Base Sprite
All clips inherit from `BaseSprite`, which provides a unified transform and animation system:
```mermaid
graph TD
A[BaseSprite] --> B[renderTransform]
A --> C[animations]
B --> D[x, y, width, height]
B --> E[scale, scaleX, scaleY]
B --> F[angle, opacity]
B --> G[blur, brightness]
B --> H[mirror, motionBlur]
```
The render transform accumulates values from multiple animations:
```typescript
const defaultRenderTransform = {
x: 0,
y: 0,
width: 0,
height: 0,
scale: 1,
scaleX: 1,
scaleY: 1,
opacity: 1,
angle: 0,
blur: 0,
brightness: 1,
mirror: 0,
motionBlur: 0,
};
```
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:1-100]()
### Video Clip
The `Video` class handles video file loading and playback:
```mermaid
sequenceDiagram
participant User
participant Video
participant ResourceManager
participant PixiJS
User->>Video: fromUrl(url)
Video->>ResourceManager: getReadableStream(url)
ResourceManager-->>Video: stream
Video->>Video: write to localFile
Video->>PixiJS: init texture
Video-->>User: Video instance
```
Key features include:
- URL-based loading via `Video.fromUrl()`
- OPFS (Origin Private File System) for local storage
- Configurable audio with volume control
- Position and size properties (left, top, width, height)
资料来源：[packages/openvideo/src/clips/video-clip.ts:1-100]()
### Text Clip
Text clips support extensive styling options:
| Property | Type | Description |
|----------|------|-------------|
| `fontFamily` | `string` | Font family name |
| `fontSize` | `number` | Font size in pixels |
| `fontWeight` | `string\|number` | Font weight |
| `fontStyle` | `string` | Font style (normal/italic) |
| `fill` | `string` | Text color |
| `align` | `string` | Text alignment |
| `wordWrap` | `boolean` | Enable word wrapping |
| `wordWrapWidth` | `number` | Maximum line width |
| `lineHeight` | `number` | Line height |
| `letterSpacing` | `number` | Character spacing |
| `stroke` | `object\|string` | Stroke color and width |
| `dropShadow` | `object` | Shadow effect |
资料来源：[packages/openvideo/src/clips/text-clip.ts:1-100]()
### Caption Clip
Caption clips support word-level animations with the following JSON structure:
```typescript
interface CaptionColorsJSON {
appeared?: string;
appearing?: string;
disappearing?: string;
}
interface CaptionDataJSON {
words?: string[];
colors?: CaptionColorsJSON;
style?: TextStyleOptions;
}
```
资料来源：[packages/openvideo/src/clips/caption-clip.ts:1-100]()
## Animation System
### Animation Presets
The animation system provides predefined motion patterns:
| Preset | Description |
|--------|-------------|
| `slideIn` | Directional entrance with configurable distance |
| `slideOut` | Directional exit with configurable distance |
| `pulse` | Scale oscillation animation |
| `blurIn` | Fade with blur transition |
| `fade` | Opacity transition |
| `pop` | Scale pop effect |
| `shake` | Position shake effect |
### Preset Configuration
```typescript
case "slideOut": {
const direction = normalized?.direction || "left";
const distance = normalized?.distance || 300;
return {
"0%": {
x: xPositionInit ?? 0,
y: yPositionInit ?? 0,
opacity: opacityInit ?? 1,
mirror: defaultMirror,
},
"100%": {
x: direction === "left" ? -distance : direction === "right" ? distance : 0,
y: direction === "top" ? -distance : direction === "bottom" ? distance : 0,
opacity: opacityEnd ?? 0,
mirror: defaultMirror,
},
};
}
```
资料来源：[packages/openvideo/src/animation/presets.ts:1-100]()
### Animation Transform Properties
Animations can modify the following sprite properties:
- **Position**: `x`, `y`
- **Size**: `width`, `height`
- **Rotation**: `angle`
- **Scale**: `scale`, `scaleX`, `scaleY`
- **Appearance**: `opacity`, `blur`, `brightness`
- **Effects**: `mirror`, `motionBlur`
## Transition System
The transition system provides visual effects for clip transitions:
```mermaid
graph TD
A[Transition Lookup] --> B[Check by name]
A --> C[Check by lowercase name]
A --> D[Check by label]
A --> E[Variant generation]
B --> F{Found?}
C --> F
D --> F
E --> F
F -->|Yes| G[Apply GLSL Fragment]
F -->|No| H[Error: Transition not found]
```
### Supported Transitions
| Transition | Category | Description |
|------------|----------|-------------|
| `GridFlip` | Grid | Grid-based flip effect |
| `Circle` | Shape | Circular reveal |
| `Directional` | Direction | Directional wipe |
| `UndulatingBurnOut` | Wave | Undulating wave burn |
| `SquaresWire` | Grid | Wireframe squares |
| `RotateScaleFade` | Transform | Combined rotation, scale, fade |
| `RandomSquares` | Grid | Random square reveal |
| `PolarFunction` | Shape | Polar coordinate transformation |
| `Pixelate` | Effect | Pixelation effect |
| `Perlin` | Noise | Perlin noise transition |
| `Luma` | Luminance | Luminance-based blend |
| `Heart` | Shape | Heart-shaped reveal |
| `Crosshatch` | Pattern | Crosshatch pattern |
| `GlitchDisplace` | Glitch | Glitch displacement |
| `CrossZoom` | Zoom | Cross zoom effect |
资料来源：[packages/openvideo/src/transition/transition.ts:1-200]()
## Rendering Pipeline
### WebCodecs Integration
The compositor uses WebCodecs for hardware-accelerated video encoding:
```mermaid
graph LR
A[Timeline Frames] --> B[PixiJS Render]
B --> C[VideoFrame]
C --> D[VideoEncoder]
D --> E[EncodedVideoChunk]
E --> F[MP4 Muxer]
F --> G[Output File]
```
### MP4 Muxing
The `mp4-utils` module handles sample normalization and muxing:
```typescript
samples.forEach((s) => {
let normalizedDTS: number;
let normalizedCTS: number;
// Normalize to start from 0
normalizedDTS = s.dts - firstVDTS!;
normalizedCTS = s.cts - (firstVCTS ?? 0);
outfile.addSample(trackId, new Uint8Array(s.data), {
duration: s.duration,
dts: normalizedDTS + offsetDTS,
cts: normalizedCTS + offsetCTS,
is_sync: s.is_sync,
});
});
```
资料来源：[packages/openvideo/src/mp4-utils/index.ts:1-100]()
## Node Package Architecture
The `@combo/node` package provides server-side rendering capabilities:
```mermaid
graph TD
A[Renderer] --> B[Playwright]
A --> C[Local HTTP Server]
B --> D[Browser Instance]
C --> D
D --> E[template.html]
E --> F[Compositor]
F --> G[WebCodecs]
G --> H[MP4 Output]
```
### Renderer Class
```typescript
interface RendererOptions {
json: VideoConfigJSON;
outputPath: string;
browserOptions?: {
headless?: boolean;
timeout?: number;
};
}
```
#### Events
| Event | Payload | Description |
|-------|---------|-------------|
| `progress` | `{ progress: number, phase: string, message?: string }` | Render progress updates |
| `error` | `Error` | Error notifications |
| `complete` | `string` | Output file path when complete |
资料来源：[packages/node/README.md:1-100]()
### HTML Template
The node package uses a template HTML file with import maps:
```html
```
资料来源：[packages/node/src/template.html:1-30]()
## Technology Stack
| Technology | Role |
|------------|------|
| **WebCodecs** | Hardware-accelerated video encoding/decoding |
| **PixiJS** | 2D/3D rendering engine |
| **wrapbox** | Low-level MP4 box manipulation and muxing |
| **TypeScript** | Type-safe development |
| **Playwright** | Server-side browser automation |
资料来源：[README.md:40-50]()
## Data Flow
```mermaid
sequenceDiagram
participant App
participant Studio
participant Compositor
participant PixiJS
participant WebCodecs
participant File
App->>Studio: addClip(video)
App->>Studio: play()
Studio->>Compositor: initPixiApp()
Compositor->>PixiJS: Create Application
loop Playback
Studio->>Compositor: render frame
Compositor->>PixiJS: render scene
PixiJS-->>Compositor: frame buffer
end
App->>Studio: export()
Studio->>Compositor: start encoding
loop Encoding
Compositor->>PixiJS: render frame
PixiJS-->>Compositor: VideoFrame
Compositor->>WebCodecs: encode
WebCodecs-->>Compositor: EncodedChunk
Compositor->>File: mux samples
end
Compositor-->>App: output path
```
## Package Structure
```
packages/
├── openvideo/
│ └── src/
│ ├── studio.ts # Main orchestrator
│ ├── compositor.ts # Rendering engine
│ ├── clips/ # Media clip implementations
│ │ ├── index.ts
│ │ ├── video-clip.ts
│ │ ├── text-clip.ts
│ │ ├── caption-clip.ts
│ │ └── ...
│ ├── sprite/ # Base sprite classes
│ ├── animation/ # Animation presets
│ ├── transition/ # Transition effects
│ └── mp4-utils/ # MP4 muxing utilities
└── node/
└── src/
├── renderer.ts # Node.js renderer
├── template.html # Browser template
└── types.ts # TypeScript types
```
## Export and Serialization
The system supports JSON serialization for project portability:
```typescript
interface VideoConfigJSON {
settings: {
width: number;
height: number;
fps: number;
bgColor: string;
videoCodec?: string;
bitrate?: number;
audio?: boolean;
metaDataTags?: Record;
};
tracks: TrackJSON[];
}
```
资料来源：[packages/openvideo/render.html:25-45]()
---
## Studio and State Management
### 相关页面
相关主题：[System Architecture](#page-3), [Clip System](#page-5), [Animation System](#page-6)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/studio.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio.ts)
- [packages/openvideo/src/studio/timeline-model.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/timeline-model.ts)
- [packages/openvideo/src/studio/history-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/history-manager.ts)
- [packages/openvideo/src/studio/selection-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/selection-manager.ts)
- [packages/openvideo/src/studio/resource-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/resource-manager.ts)
- [packages/openvideo/src/studio/transport.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/transport.ts)
# Studio and State Management
## Overview
The **Studio** module is the central orchestration layer in OpenVideo, responsible for managing project state, timeline operations, playback control, and user interactions. It provides a unified API for video editing workflows by coordinating between the **Compositor** (rendering engine), **Timeline Model** (data structure), **Transport** (playback control), **Selection Manager** (UI interactions), and **History Manager** (undo/redo).
Studio abstracts the complexity of WebCodecs and PixiJS rendering, exposing high-level primitives like Clips, Tracks, and Effects that map directly to user-facing editing concepts.
## Architecture Overview
```mermaid
graph TD
A[Studio] --> B[Compositor]
A --> C[Timeline Model]
A --> D[Transport]
A --> E[Selection Manager]
A --> F[History Manager]
A --> G[Resource Manager]
C --> H[Tracks]
C --> I[Clips]
H --> I
I --> J[VideoClip]
I --> K[AudioClip]
I --> L[TextClip]
I --> M[ImageClip]
I --> N[CaptionClip]
D --> B
E --> A
F --> C
G --> I
```
## Core Components
### Studio Class
The `Studio` class serves as the main entry point and state container for video projects. It maintains references to all subsystem managers and provides the public API for editing operations.
**Key Responsibilities:**
- Project lifecycle management (create, load, save, clear)
- Coordination between timeline, compositor, and transport
- Event emission for UI updates
- Texture and resource cleanup
**Key Methods:**
| Method | Description |
|--------|-------------|
| `play()` | Start playback |
| `pause()` | Pause playback |
| `stop()` | Stop and reset to start |
| `seek(time)` | Seek to specific time (microseconds) |
| `frameNext()` | Advance one frame |
| `framePrev()` | Go back one frame |
| `clear()` | Clear all clips and resources |
| `updateSelected(updates)` | Update properties of selected clips |
资料来源：[packages/openvideo/src/studio.ts:1-100](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio.ts)
### Timeline Model
The Timeline Model manages the hierarchical data structure of a video project:
```mermaid
graph TD
Timeline --> Tracks
Tracks --> Clip
Clip --> Properties
Properties --> Transform
Properties --> Effect
Properties --> Animation
```
**Data Structure:**
| Property | Type | Description |
|----------|------|-------------|
| `tracks` | `Track[]` | Ordered collection of tracks |
| `duration` | `number` | Total timeline duration in microseconds |
| `fps` | `number` | Frames per second |
| `width` | `number` | Output width in pixels |
| `height` | `number` | Output height in pixels |
资料来源：[packages/openvideo/src/studio/timeline-model.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/timeline-model.ts)
## Transport System
The Transport module handles playback control and timeline navigation.
### Playback States
```mermaid
stateDiagram-v2
[*] --> Stopped
Stopped --> Playing : play()
Playing --> Paused : pause()
Paused --> Playing : play()
Playing --> Stopped : stop()
Paused --> Stopped : stop()
Stopped --> Seeking : seek()
Playing --> Seeking : seek()
Seeking --> Playing : play()
Seeking --> Paused : pause()
```
### Transport API
```typescript
interface Transport {
play(): Promise;
pause(): void;
stop(): Promise;
seek(time: number): Promise;
frameNext(): Promise;
framePrev(): Promise;
}
```
**Key Behaviors:**
- `play()`: Starts playback from current position, returns Promise
- `pause()`: Immediately pauses playback
- `stop()`: Stops and resets playhead to timeline start
- `seek(time)`: Moves playhead to specified time in microseconds
资料来源：[packages/openvideo/src/studio/transport.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/transport.ts)
## Selection Management
The Selection Manager handles interactive clip selection and transformation in the editor canvas.
### Selection Features
- **Single Selection**: Click to select a single clip
- **Multi-Selection**: Shift+click to add to selection
- **Double-Click**: Opens text editing for Text/Caption clips
- **Transform Handles**: Resize and reposition selected clips
### Selection Events
| Event | Payload | Description |
|-------|---------|-------------|
| `clip:select` | `{ clip: IClip }` | Clip was selected |
| `clip:deselect` | `{ clip: IClip }` | Clip was deselected |
| `clip:dblclick` | `{ clip: IClip }` | Double-clicked on clip |
| `selection:change` | `{ clips: IClip[] }` | Selection set changed |
### Pointer Interaction Flow
```mermaid
sequenceDiagram
participant User
participant Canvas
participant SelectionManager
participant Studio
User->>Canvas: pointerdown
Canvas->>SelectionManager: handlePointerDown
SelectionManager->>SelectionManager: Check for double-click
Alt Double-click on Text/Caption
SelectionManager->>Studio: emit('clip:dblclick')
End
User->>Canvas: pointerup
Canvas->>SelectionManager: handlePointerUp
SelectionManager->>SelectionManager: Update selection
SelectionManager->>Studio: emit('clip:select')
```
资料来源：[packages/openvideo/src/studio/selection-manager.ts:1-80](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/selection-manager.ts)
## History Management (Undo/Redo)
The History Manager maintains a stack of state snapshots to enable undo/redo operations.
### State Snapshot Structure
```typescript
interface HistoryState {
timestamp: number;
action: string;
state: SerializedTimeline;
}
```
### Operations
| Operation | Description |
|-----------|-------------|
| `push(state)` | Add new state to history |
| `undo()` | Restore previous state |
| `redo()` | Restore next state |
| `clear()` | Clear all history |
| `canUndo` | Check if undo is available |
| `canRedo` | Check if redo is available |
### History Stack Behavior
```mermaid
graph LR
A[Current] -->|Undo| B[State N-1]
B -->|Undo| C[State N-2]
A -->|Redo| D[State N+1]
C -.->|Push| A
```
资料来源：[packages/openvideo/src/studio/history-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/history-manager.ts)
## Resource Management
The Resource Manager handles loading, caching, and lifecycle of media resources (video files, images, audio).
### Resource Loading
```typescript
// Loading a video from URL
const video = await Video.fromUrl('https://example.com/video.mp4');
await studio.addClip(video);
// Loading with position options
const videoWithOpts = await Video.fromUrl('https://example.com/video.mp4', {
x: 100,
y: 200,
width: 640,
height: 480
});
```
### Resource Lifecycle
| Method | Description |
|--------|-------------|
| `load(url)` | Load a resource from URL |
| `getReadablestream(url)` | Get stream for resource |
| `release(resource)` | Release resource from memory |
| `clear()` | Clear all cached resources |
资料来源：[packages/openvideo/src/studio/resource-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/resource-manager.ts)
## Clip Types
OpenVideo supports multiple clip types that extend a common base:
| Clip Type | Description |
|-----------|-------------|
| `Video` | Video playback with decoding via WebCodecs |
| `Audio` | Audio track with volume control |
| `Text` | Text overlay with font styling |
| `Image` | Static image display |
| `Caption` | Subtitle/caption text |
| `Custom` | User-defined clip types |
### Clip Properties
```typescript
interface IClip {
id: string;
type: 'Video' | 'Audio' | 'Text' | 'Image' | 'Caption';
left: number;
top: number;
width: number;
height: number;
startTime: number;
endTime: number;
opacity: number;
volume?: number;
}
```
资料来源：[packages/openvideo/src/clips/video-clip.ts:1-50](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/video-clip.ts)
## Event System
Studio extends EventEmitter to provide reactive updates:
```typescript
// Event listeners
studio.on('progress', ({ phase, progress }) => {
console.log(`[${phase}] ${Math.round(progress * 100)}%`);
});
studio.on('clip:select', ({ clip }) => {
console.log('Selected:', clip.id);
});
studio.on('reset', () => {
console.log('Studio cleared');
});
```
### Event Reference
| Event | Payload | Emitted When |
|-------|---------|--------------|
| `progress` | `{ phase, progress, message }` | Render progress updates |
| `clip:select` | `{ clip }` | Clip is selected |
| `clip:deselect` | `{ clip }` | Clip is deselected |
| `clip:dblclick` | `{ clip }` | Clip is double-clicked |
| `selection:change` | `{ clips }` | Selection set changes |
| `reset` | - | Studio is cleared |
## Studio Lifecycle
```mermaid
flowchart TD
A[Create Studio] --> B[Initialize PixiJS]
B --> C[Load Project/JSON]
C --> D[User Editing]
D --> E{Operation}
E -->|Add Clip| F[Update Timeline]
E -->|Select| G[Update Selection]
E -->|Transform| H[Update Transform]
E -->|Play| I[Transport Control]
F --> D
G --> D
H --> D
I --> J[Compositor Render]
J --> K{Loop}
K -->|Continue| I
K -->|Stop| L[Export/Save]
L --> M[Cleanup Resources]
```
### Initialization
```typescript
const studio = new Studio({
width: 1280,
height: 720,
fps: 30,
});
// Studio is ready for use
await studio.ready;
```
### Cleanup
```typescript
// Clear all resources
await studio.clear();
// Releases:
// - All clips from timeline
// - PixiJS textures and graphics
// - Transition renderers
// - Selection state
```
资料来源：[packages/openvideo/src/studio.ts:80-150](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio.ts)
## JSON Serialization
Studio projects can be serialized to JSON for persistence and sharing:
```typescript
// Export project
const json = studio.toJSON();
// Import project
const restored = Studio.fromJSON(json);
```
### Serialized Format
```typescript
interface SerializedStudio {
version: string;
settings: {
width: number;
height: number;
fps: number;
};
timeline: {
tracks: SerializedTrack[];
duration: number;
};
resources: ResourceReference[];
}
```
## Usage Example
```typescript
import { Studio, Video, Text } from 'openvideo';
// Initialize studio
const studio = new Studio({
width: 1920,
height: 1080,
fps: 30,
});
// Add video clip
const video = await Video.fromUrl('https://example.com/intro.mp4');
await studio.addClip(video);
// Add text overlay
const title = new Text({
text: 'Hello World',
fontSize: 72,
color: 0xFFFFFF,
});
title.left = 100;
title.top = 500;
await studio.addClip(title);
// Preview
studio.play();
// Export
const output = await studio.compositor.render();
```
资料来源：[README.md](https://github.com/openvideodev/openvideo/blob/main/README.md)
---
## Clip System
### 相关页面
相关主题：[System Architecture](#page-3), [Studio and State Management](#page-4), [Animation System](#page-6)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/clips/video-clip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/video-clip.ts)
- [packages/openvideo/src/clips/iclip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/iclip.ts)
- [packages/openvideo/src/clips/base-clip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/base-clip.ts)
- [packages/openvideo/src/clips/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/index.ts)
- [packages/openvideo/src/studio/timeline-model.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/timeline-model.ts)
- [packages/openvideo/src/studio/selection-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/selection-manager.ts)
- [packages/openvideo/src/json-serialization.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/json-serialization.ts)
- [packages/openvideo/src/mp4-utils/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/mp4-utils/index.ts)
# Clip System
## Overview
The Clip System is the foundational data model and rendering layer within OpenVideo, responsible for representing, managing, and rendering media assets on a timeline. Clips encapsulate all media types (video, audio, image, text, caption) and provide a unified interface for timeline operations including playback, seeking, splitting, cloning, and serialization.
**资料来源：** [packages/openvideo/src/clips/iclip.ts:1-80](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/iclip.ts)
---
## Architecture
### Class Hierarchy
The Clip System follows an inheritance-based architecture where specialized clip types extend a common base class:
```mermaid
graph TD
A["IClip (Interface)"] --> B["BaseClip"]
B --> C["Video"]
B --> D["Audio"]
B --> E["Image"]
B --> F["Text"]
B --> G["Caption"]
B --> H["Effect"]
B --> I["Transition"]
B --> J["Placeholder"]
C -.->|implements| K["IPlaybackCapable"]
D -.->|implements| K
```
**资料来源：** [packages/openvideo/src/clips/iclip.ts:40-60](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/iclip.ts)
### Core Interfaces
| Interface | Purpose |
|-----------|---------|
| `IClip` | Base interface defining common clip operations |
| `IClipMeta` | Metadata container (duration, dimensions, format) |
| `ITransitionInfo` | Transition configuration between clips |
| `IPlaybackCapable` | Interface for clips supporting playback control |
**资料来源：** [packages/openvideo/src/clips/iclip.ts:1-150](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/iclip.ts)
---
## Clip Types
### Video Clip
The `Video` class handles MP4 file parsing, WebCodecs-based decoding, and frame extraction. It supports hardware acceleration and provides methods for seeking, frame extraction, and thumbnail generation.
**Key Features:**
- Async MP4 stream parsing via `ResourceManager.getReadableStream`
- WebCodecs `VideoDecoder` integration for frame decoding
- Support for IDR (key) frame normalization
- Audio extraction with configurable volume
- First black frame correction
**资料来源：** [packages/openvideo/src/clips/video-clip.ts:1-100](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/video-clip.ts)
```typescript
// Loading a video clip
const video = await Video.fromUrl('https://example.com/video.mp4', {
x: 0,
y: 0,
width: 1920,
height: 1080,
});
// Configure display window
video.set({
display: {
from: 150, // start frame
to: 450, // end frame (10 seconds at 30fps)
},
});
```
### Audio Clip
Audio clips manage audio-only media with WebCodecs `AudioDecoder` integration. They support PCM data generation and volume control.
**资料来源：** [packages/openvideo/src/clips/video-clip.ts:200-250](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/video-clip.ts)
### Image Clip
Image clips load and display static images using the Fetch API and `createImageBitmap` for efficient rendering.
### Text Clip
Text clips render text content with styling options including word wrapping, stroke, drop shadow, and border radius.
**资料来源：** [packages/openvideo/src/studio/selection-manager.ts:50-80](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/selection-manager.ts)
### Caption Clip
Caption clips display timed text overlays synchronized with video playback.
### Effect & Transition Clips
Specialized clips for visual effects and transition animations between other clips.
**资料来源：** [packages/openvideo/src/json-serialization.ts:1-30](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/json-serialization.ts)
---
## IClip Interface
The `IClip` interface defines the contract all clip types must implement:
| Property | Type | Description |
|----------|------|-------------|
| `type` | `string` | Clip type identifier (e.g., 'video', 'image', 'text') |
| `id` | `string` | Unique clip identifier |
| `name` | `string` | Human-readable name |
| `src` | `string` | Source URL or identifier |
| `duration` | `number` | Clip duration in microseconds |
| `volume` | `number` | Audio volume level (0-1) |
| `ready` | `Promise` | Resolves when clip is fully loaded |
| `meta` | `IClipMeta` | Metadata about the clip |
### Core Methods
```typescript
interface IClip {
// Lifecycle
destroy(): void;
clone(): Promise;
// Timeline operations
tick(time: number): Promise<{
video?: VideoFrame | ImageBitmap | null;
audio?: Float32Array[];
state: 'done' | 'success';
}>;
getFrame(time: number): Promise<{
video: ImageBitmap | null;
audio: Float32Array[];
done: boolean;
}>;
// Editing
split?(time: number): Promise<[this, this]>;
// Serialization
toJSON(main?: boolean): any;
// Rendering
setRenderer?(renderer: any): void;
}
```
**资料来源：** [packages/openvideo/src/clips/iclip.ts:60-120](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/iclip.ts)
---
## Clip Lifecycle
### Loading Sequence
```mermaid
sequenceDiagram
participant Client
participant Studio
participant Clip
participant ResourceManager
participant WebCodecs
Client->>Clip: new Clip(source)
Clip->>ResourceManager: getReadableStream(url)
ResourceManager-->>Clip: ReadableStream
Clip->>WebCodecs: create decoder
WebCodecs-->>Clip: decoder ready
Clip->>Clip: resolve ready Promise
Client->>Clip: await clip.ready
```
### Replacement Workflow
Clips can be replaced while preserving timeline position:
```mermaid
graph TD
A[Find clips by source] --> B[Suspend Studio rendering]
B --> C[Create new clip factory]
C --> D[Setup listeners for new clip]
D --> E[Wait for clip.ready]
E --> F[Replace in clips array]
F --> G[Resume rendering]
```
**资料来源：** [packages/openvideo/src/studio/timeline-model.ts:1-50](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/timeline-model.ts)
---
## JSON Serialization
Clips support bidirectional JSON conversion for project persistence:
```typescript
interface BaseClipJSON {
id?: string;
name?: string;
src: string;
display: {
from: number; // frames
to: number; // frames
};
playbackRate: number;
duration: number;
left: number;
top: number;
width: number;
height: number;
angle: number;
zIndex: number;
opacity: number;
flip: "horizontal" | "vertical" | null;
trim?: { from: number; to: number; };
transition?: ITransitionInfo;
effects?: Array<{
id: string;
key: string;
startTime: number;
duration: number;
}>;
animation?: {
keyFrames: Record>;
opts: { duration: number; delay?: number; iterCount?: number; };
};
colorAdjustment?: ColorAdjustment;
}
```
**资料来源：** [packages/openvideo/src/json-serialization.ts:1-60](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/json-serialization.ts)
---
## Video Decoding Pipeline
```mermaid
graph TD
A[MP4 Stream] --> B[MP4Box Parser]
B --> C{videoTracks?}
C -->|Yes| D[Video Decoder Config]
C -->|No| E[Error: No video]
D --> F[Extract samples]
F --> G[IDR Normalization]
G --> H[Frame Cache]
H --> I[Compositor.tick]
J[Decode Queue] --> K[VideoDecoder]
K --> L[Decoded Frames]
L --> H
```
### Sample Normalization
Video samples undergo timestamp normalization to ensure consistent playback:
```typescript
function normalizeTimescale(
s: MP4Sample,
delta = 0,
sampleType: "video" | "audio"
) {
let offset = s.offset;
// Normalize DTS and CTS relative to first sample
// Handle SEI data before IDR frames
}
```
**资料来源：** [packages/openvideo/src/mp4-utils/index.ts:1-40](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/mp4-utils/index.ts)
---
## Studio Integration
### Selection Management
Clips interact with the Studio's selection system for UI interactions:
| Method | Purpose |
|--------|---------|
| `selectClip(clip, shiftKey?)` | Add clip to selection |
| `getTopmostClipAtPoint(point)` | Hit testing |
| `syncSelectedClipsTransforms()` | Apply visual transforms |
| `getVisibleHandles()` | Return visible resize handles |
**资料来源：** [packages/openvideo/src/studio/selection-manager.ts:1-30](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/selection-manager.ts)
### Event System
Clips emit events for state changes:
```typescript
clip.on("propsChange", async () => {
await studio.updateFrame(studio.currentTime);
updateTransformer(clip);
});
```
**资料来源：** [packages/openvideo/src/studio/timeline-model.ts:50-80](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio/timeline-model.ts)
---
## Common Configuration Options
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `audio` | `boolean \| { volume: number }` | `true` | Audio playback settings |
| `display.from` | `number` | `0` | Start frame |
| `display.to` | `number` | clip duration | End frame |
| `playbackRate` | `number` | `1.0` | Speed multiplier |
| `left`, `top` | `number` | `0` | Position in pixels |
| `width`, `height` | `number` | source dimensions | Size in pixels |
| `opacity` | `number` | `1` | Alpha (0-1) |
| `angle` | `number` | `0` | Rotation in degrees |
| `zIndex` | `number` | `0` | Layer order |
**资料来源：** [packages/openvideo/src/clips/iclip.ts:80-100](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/iclip.ts)
---
## Usage Examples
### Creating and Adding a Video Clip
```typescript
import { Studio, Video } from 'openvideo';
const studio = new Studio({
canvas: document.getElementById('canvas'),
width: 1920,
height: 1080,
});
const video = await Video.fromUrl('https://example.com/video.mp4', {
x: 0,
y: 0,
width: 1920,
height: 1080,
});
video.duration = 5e6; // 5 seconds in microseconds
await studio.addClip(video);
studio.play();
```
### Cloning a Clip
```typescript
const clonedClip = await video.clone();
clonedClip.left = 100;
clonedClip.top = 100;
await studio.addClip(clonedClip);
```
### Splitting a Clip
```typescript
const [before, after] = await video.split(2500000); // Split at 2.5 seconds
await studio.addClip(before);
await studio.addClip(after);
await studio.removeClip(video.id);
```
### Exporting to JSON
```typescript
const projectJSON = studio.toJSON();
// projectJSON now contains all clips serialized
```
**资料来源：** [packages/openvideo/src/studio.spec.ts:1-40](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/studio.spec.ts)
---
## Animation System
### 相关页面
相关主题：[Clip System](#page-5), [Effects System](#page-8)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/animation/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/index.ts)
- [packages/openvideo/src/animation/keyframe-animation.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/keyframe-animation.ts)
- [packages/openvideo/src/animation/gsap-animation.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/gsap-animation.ts)
- [packages/openvideo/src/animation/easings.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/easings.ts)
- [packages/openvideo/src/animation/presets.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/presets.ts)
- [packages/openvideo/src/animation/registry.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/registry.ts)
- [packages/openvideo/src/animation/types.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/types.ts)
# Animation System
## Overview
The Animation System in OpenVideo is a comprehensive framework that provides time-based property interpolation for sprites, clips, and UI elements within the video compositor. The system supports keyframe-based animations, easing functions, preset animations, and integration with external animation libraries like GSAP.
Animations in OpenVideo work by interpolating properties over time using easing functions. They can be applied to any visual element (sprites, clips, captions) and are processed during the render cycle to update the `renderTransform` properties.
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:1-50](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
## Architecture
```mermaid
graph TD
A[Animation Request] --> B{Animation Type}
B -->|Preset| C[Animation Presets]
B -->|Keyframe| D[KeyframeAnimation]
B -->|GSAP| E[GSAP Animation]
C --> F[Factory Pattern]
D --> G[linearTimeFn]
E --> H[GSAP Core]
G --> I[Easing Functions]
I --> J[Interpolated Values]
H --> J
F --> J
J --> K[RenderTransform Update]
```
### Core Components
| Component | File | Purpose |
|-----------|------|---------|
| `KeyframeAnimation` | `keyframe-animation.ts` | Core keyframe-based animation class |
| `GSAPAnimation` | `gsap-animation.ts` | GSAP library integration |
| `easings` | `easings.ts` | Built-in easing function library |
| `presets` | `presets.ts` | Reusable animation preset factories |
| `registry` | `registry.ts` | Animation registration and lookup |
| `types` | `types.ts` | TypeScript type definitions |
资料来源：[packages/openvideo/src/animation/types.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/types.ts)
## Easing Functions
Easing functions control the rate of change during an animation, providing smooth acceleration and deceleration. The system includes a comprehensive set of built-in easing functions.
### Supported Easings
| Category | Functions |
|----------|-----------|
| **Linear** | `linear` |
| **Quad** | `easeInQuad`, `easeOutQuad`, `easeInOutQuad` |
| **Cubic** | `easeInCubic`, `easeOutCubic`, `easeInOutCubic` |
| **Sine** | `easeInSine`, `easeOutSine`, `easeInOutSine` |
| **Expo** | `easeInExpo`, `easeOutExpo`, `easeInOutExpo` |
资料来源：[packages/openvideo/src/animation/easings.ts:1-30](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/easings.ts)
### Easing Interface
```typescript
type EasingFunction = (t: number) => number;
```
All easing functions accept a normalized time parameter `t` (0 to 1) and return the interpolated progress value.
## Animation Presets
Preset animations are pre-configured animation factories that can be applied to elements without defining custom keyframes. They use a factory pattern that accepts options and optional parameters.
### Available Presets
#### Entry Animations
| Preset | Description |
|--------|-------------|
| `blurIn` | Fade in with blur effect |
| `motionBlurIn` | Slide in with motion blur |
| `flashSlideIn` | Quick flash and slide entry |
| `diagonalBlurZoomIn` | Diagonal movement with blur and zoom |
| `diagonalSlideRotateIn` | Diagonal slide with rotation |
#### Exit Animations
| Preset | Description |
|--------|-------------|
| `blurOut` | Fade out with blur effect |
| `motionBlurOut` | Slide out with motion blur |
| `flashSlideOut` | Quick flash and slide exit |
| `diagonalSlideRotateOut` | Diagonal slide out with rotation |
#### Loop Animations
| Preset | Description |
|--------|-------------|
| `pulse` | Scale oscillation effect |
| `slideOut` | Directional slide animation |
资料来源：[packages/openvideo/src/animation/presets.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/presets.ts)
### Preset Configuration
Each preset accepts:
```typescript
interface AnimationFactory {
(opts: IAnimationOpts, params?: Record): KeyframeAnimation;
}
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `opts.easing` | `string` | Easing function name (default: varies by preset) |
| `opts.delay` | `number` | Delay before animation starts in ms |
| `opts.duration` | `number` | Animation duration in ms |
| `opts.iterCount` | `number` | Number of iterations (-1 for infinite) |
| `params` | `Record` | Custom keyframe percentages or direction overrides |
## Keyframe Animation
The `KeyframeAnimation` class is the core animation implementation that handles time-based property interpolation.
### Animation Flow
```mermaid
sequenceDiagram
participant Render as Render Loop
participant Anim as KeyframeAnimation
participant Easing as Easing Functions
participant Target as Sprite/Clip
Render->>Anim: getTransform(currentTime)
Note over Anim: Calculate progress from time
Anim->>Easing: apply easing(progress)
Easing-->>Anim: eased progress
Anim->>Anim: interpolate properties
Anim-->>Target: Partial
```
资料来源：[packages/openvideo/src/animation/keyframe-animation.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/keyframe-animation.ts)
### Animated Properties
The animation system can interpolate the following properties:
| Property | Type | Description |
|----------|------|-------------|
| `x` | `number` | Horizontal position offset |
| `y` | `number` | Vertical position offset |
| `width` | `number` | Width dimension |
| `height` | `number` | Height dimension |
| `angle` | `number` | Rotation angle |
| `blur` | `number` | Blur filter value |
| `motionBlur` | `number` | Motion blur intensity |
| `scale` | `number` | Uniform scale multiplier |
| `scaleX` | `number` | Horizontal scale |
| `scaleY` | `number` | Vertical scale |
| `opacity` | `number` | Opacity value (0-1) |
| `brightness` | `number` | Brightness filter value |
| `mirror` | `number` | Mirror effect intensity |
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:80-100](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
### Linear Time Function
The `linearTimeFn` performs the actual interpolation between keyframes:
```typescript
export function linearTimeFn(
time: number,
keyFrame: TAnimationKeyFrame,
opts: Required
): Partial
```
The function:
1. Calculates normalized progress from elapsed time
2. Finds surrounding keyframes
3. Interpolates each property using linear interpolation
4. Returns partial props object with interpolated values
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:200-250](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
## Animation Integration with Sprites
Sprites process animations during their update cycle by iterating through all animations and accumulating transform values.
### Sprite Animation Processing
```typescript
// 1. Process new modular animations
for (const anim of this.animations) {
const transform = anim.getTransform(time);
if (transform.x !== undefined) this.renderTransform.x! += transform.x;
if (transform.y !== undefined) this.renderTransform.y! += transform.y;
if (transform.scale !== undefined)
this.renderTransform.scale! *= transform.scale;
// ... other properties
}
```
Transform values are **accumulated** rather than replaced, allowing multiple animations to affect the same property.
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:20-45](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
## Clip Animation Support
Clips support animation through both legacy `animation` field and the new modular `animations` array format.
### Clip Serialization
```typescript
// Extract new modular animations
const animations = this.animations.map((a) => {
if ('toJSON' in a && typeof (a as any).toJSON === 'function') {
return (a as any).toJSON();
}
return {
type: a.type,
opts: a.options,
params: a.params || {},
};
});
```
Clips serialize animations to JSON for configuration export and reloading.
资料来源：[packages/openvideo/src/clips/base-clip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/base-clip.ts)
## Caption Word Animations
The CaptionClip system implements specialized per-word animations with dynamic and keyword-based application modes.
### Word Animation Application Modes
| Mode | Trigger |
|------|---------|
| `active` | Applied when word is currently being spoken |
| `keyword` | Applied to designated keyword words |
| `none` | Animation disabled |
### Animation Dynamics
| Dynamic Mode | Behavior |
|--------------|----------|
| `true` (default) | Uses `sin(progress * PI)` for natural wave effect |
| `false` | Applies full animation immediately at word start |
```typescript
if (wordAnimation.mode === 'dynamic' && isActive) {
const progress = (currentTimeMs - word.from) / duration;
animationFactor = Math.sin(progress * Math.PI);
} else {
animationFactor = 1;
}
```
资料来源：[packages/openvideo/src/clips/caption-clip.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/clips/caption-clip.ts)
## GSAP Integration
The system provides GSAP animation integration through the `GSAPAnimation` class, allowing users to leverage GSAP's extensive animation capabilities.
资料来源：[packages/openvideo/src/animation/gsap-animation.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/gsap-animation.ts)
## Animation Registry
The registry system allows dynamic registration and lookup of custom animations, enabling extensibility.
```typescript
// From registry.ts - conceptual usage
const registry = getAnimationRegistry();
registry.register('myCustomAnimation', customAnimation);
const animation = registry.get('myCustomAnimation');
```
资料来源：[packages/openvideo/src/animation/registry.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/registry.ts)
## Usage Examples
### Basic Preset Animation
```typescript
import { blurIn, easeOutQuad } from '@openvideo/animation';
const animation = blurIn({
easing: 'easeOutQuad',
duration: 1000,
delay: 500,
iterCount: 1,
});
```
### Custom Keyframe Animation
```typescript
import { KeyframeAnimation } from '@openvideo/animation';
const customAnim = new KeyframeAnimation({
'0%': { opacity: 0, y: 100 },
'100%': { opacity: 1, y: 0 },
}, {
easing: 'easeOutCubic',
duration: 800,
});
```
## Best Practices
1. **Use Presets for Common Animations**: Presets are optimized and provide consistent behavior across the application.
2. **Avoid Animating Expensive Properties**: Minimize animations on `width`, `height`, and `blur` as they require significant processing.
3. **Reuse Easing Functions**: Instead of custom easing, use the built-in easings which are well-tested and performant.
4. **Consider Animation Composition**: Multiple simple animations often produce better results than one complex animation.
5. **Set Appropriate Iter Counts**: Use `iterCount: -1` only for looping background animations to avoid performance issues.
---
## Compositor and Rendering
### 相关页面
相关主题：[System Architecture](#page-3), [Studio and State Management](#page-4)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/compositor.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/compositor.ts)
- [packages/openvideo/src/sprite/base-sprite.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
- [packages/openvideo/src/sprite/pixi-sprite-renderer.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/pixi-sprite-renderer.ts)
- [packages/openvideo/src/mp4-utils/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/mp4-utils/index.ts)
- [packages/openvideo/src/mp4-utils/mp4box-utils.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/mp4-utils/mp4box-utils.ts)
- [packages/openvideo/src/event-emitter.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/event-emitter.ts)
# Compositor and Rendering
## Overview
The Compositor is the central rendering engine in the openvideo library. It orchestrates video playback, real-time composition, and final video export using modern browser APIs including WebCodecs for hardware-accelerated encoding and PixiJS for 2D rendering. The system transforms a timeline-based project configuration into a final rendered video file.
The Compositor serves as the bridge between high-level clip management (handled by `Studio`) and low-level rendering operations (handled by sprite renderers). It manages the rendering loop, coordinates frame composition, and produces the final encoded output.
## Architecture
```mermaid
graph TD
A[Studio] --> B[Compositor]
B --> C[PixiJS Application]
C --> D[Sprite Renderers]
D --> E[Base Sprite]
D --> F[Pixi Sprite Renderer]
B --> G[WebCodecs Encoder]
G --> H[MP4 Muxer]
H --> I[Output File]
J[Video Clips] --> D
K[Text Clips] --> D
L[Image Clips] --> D
M[Audio Clips] --> D
```
## Core Components
### Compositor
The `Compositor` class initializes the rendering environment and manages the export pipeline. It accepts configuration options that define the output format.
#### Constructor Options
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `width` | `number` | `1280` | Output video width in pixels |
| `height` | `number` | `720` | Output video height in pixels |
| `fps` | `number` | `30` | Frames per second for output |
| `bgColor` | `string` | `'#000000'` | Background color as hex string |
| `videoCodec` | `string` | `undefined` | Optional video codec override |
| `bitrate` | `number` | `undefined` | Target bitrate in bits per second |
| `audio` | `boolean` | `true` | Enable audio track in output |
| `metaDataTags` | `object` | `undefined` | Custom metadata tags for output |
资料来源：[packages/node/src/template.html:30-48](https://github.com/openvideodev/openvideo/blob/main/packages/node/src/template.html)
The compositor initialization follows a specific lifecycle:
```mermaid
graph LR
A[Create Compositor] --> B[initPixiApp]
B --> C[Configure Encoder]
C --> D[Setup Event Handlers]
D --> E[Ready for Rendering]
```
### PixiJS Integration
The Compositor uses PixiJS as its rendering backend. The `initPixiApp()` method initializes the PixiJS application with the configured canvas dimensions.
#### Canvas Initialization
The sprite renderer validates texture creation to prevent runtime errors:
```typescript
// Validate texture was created successfully
// Use Texture.source instead of baseTexture (PixiJS v8.0.0+)
if (!this.texture || !this.texture.source) {
console.error("PixiSpriteRenderer: Failed to create valid texture");
return;
}
```
资料来源：[packages/openvideo/src/sprite/pixi-sprite-renderer.ts:35-42](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/pixi-sprite-renderer.ts)
Canvas dimensions are converted to integers to prevent WebCodecs encoding errors:
```typescript
// Update canvas size using integers to prevent "Value is not of type unsigned long" errors
const intWidth = Math.floor(width);
const intHeight = Math.floor(height);
```
资料来源：[packages/openvideo/src/sprite/pixi-sprite-renderer.ts:23-25](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/pixi-sprite-renderer.ts)
## Sprite Rendering System
### Base Sprite
The `BaseSprite` class provides the foundation for all renderable elements. It maintains a `renderTransform` object that accumulates all transformation properties:
```typescript
private renderTransform: {
x: number;
y: number;
width: number;
height: number;
scale: number;
scaleX: number;
scaleY: number;
opacity: number;
angle: number;
blur: number;
brightness: number;
mirror: number;
motionBlur: number;
};
```
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:29-44](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
### Animation System
Animations are processed by iterating through the `animations` array and accumulating transforms at the current time:
```typescript
for (const anim of this.animations) {
const transform = anim.getTransform(time);
if (transform.x !== undefined) this.renderTransform.x! += transform.x;
if (transform.y !== undefined) this.renderTransform.y! += transform.y;
if (transform.width !== undefined) this.renderTransform.width! += transform.width;
if (transform.scale !== undefined) this.renderTransform.scale! *= transform.scale;
if (transform.opacity !== undefined) this.renderTransform.opacity! *= transform.opacity;
if (transform.mirror !== undefined)
this.renderTransform.mirror = Math.max(this.renderTransform.mirror || 0, transform.mirror);
}
```
资料来源：[packages/openvideo/src/sprite/base-sprite.ts:46-65](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/base-sprite.ts)
Transform operations follow specific accumulation rules:
- **Additive**: `x`, `y`, `width`, `height`, `angle`, `blur`, `motionBlur`
- **Multiplicative**: `scale`, `scaleX`, `scaleY`, `opacity`, `brightness`
- **Maximum**: `mirror` (takes the highest value among all animations)
### Animation Presets
The library includes built-in animation presets defined in `presets.ts`:
| Preset | Properties | Description |
|--------|------------|-------------|
| `slideOut` | `x`, `y`, `opacity` | Slides element in specified direction while fading |
| `pulse` | `scale` | Oscillates between two scale values |
| `blurIn` | `blur`, `opacity` | Fades in while removing blur |
| `blurOut` | `blur`, `opacity` | Fades out while adding blur |
资料来源：[packages/openvideo/src/animation/presets.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/animation/presets.ts)
## Video Export Pipeline
### WebCodecs Encoding
The compositor leverages the WebCodecs API for hardware-accelerated video encoding. The encoder produces raw video frames which are then multiplexed into an MP4 container.
### MP4 Muxing
The MP4 utility module handles container creation and muxing:
```mermaid
graph TD
A[Raw Video Frames] --> B[Encoded Video Samples]
B --> C[MP4Box Muxer]
C --> D[moov Atom]
C --> E[mdat Atom]
D --> F[Final MP4 File]
E --> F
```
The muxer writes samples to the MP4 container and manages:
- Track configuration (video/audio)
- Sample timing and duration
- Chunk layout
- Metadata injection
### Progress Reporting
The Compositor emits progress events during the export process:
```typescript
compositor.on('OutputProgress', (progress: number) => {
// Report encoding progress
});
```
资料来源：[packages/openvideo/render.html:45](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/render.html)
## Event System
The Compositor extends `EventEmitter` to provide a pub/sub mechanism for render lifecycle events:
```mermaid
stateDiagram-v2
[*] --> initializing
initializing --> loading
loading --> rendering
rendering --> saving
saving --> complete
complete --> [*]
rendering --> error: On Error
loading --> error: On Error
```
### Event Types
| Event | Payload | Description |
|-------|---------|-------------|
| `progress` | `{ phase, progress, message }` | Emitted during rendering phases |
| `OutputProgress` | `number` | Encoding progress (0-1) |
| `error` | `Error` | Emitted when an error occurs |
| `complete` | `string` | Output file path when render completes |
## Node.js Renderer
The `@combo/node` package provides a headless rendering solution using Playwright:
```mermaid
graph LR
A[JSON Config] --> B[Node Renderer]
B --> C[Local HTTP Server]
C --> D[Browser Page]
D --> E[Compositor]
E --> F[MP4 Output]
```
### Renderer Options
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `json` | `object` | required | Video project configuration |
| `outputPath` | `string` | required | Path for output video file |
| `browserOptions` | `object` | `{ headless: true, timeout: 300000 }` | Playwright browser settings |
## Texture Management
The PixiSpriteRenderer implements a texture caching strategy:
1. Canvas is created with integer dimensions matching output resolution
2. Video/image frames are drawn to canvas
3. PixiJS Texture is created/updated from canvas
4. Sprite displays the texture
5. Previous textures are destroyed to free memory
```typescript
if (needsResize || isFirstFrame) {
this.canvas.width = intWidth;
this.canvas.height = intHeight;
if (this.texture != null) {
this.texture.destroy(true);
this.texture = null;
}
this.texture = Texture.from(this.canvas as any);
}
```
资料来源：[packages/openvideo/src/sprite/pixi-sprite-renderer.ts:27-38](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/sprite/pixi-sprite-renderer.ts)
## Transition Effects
The rendering system supports numerous transition effects including:
- **GridFlip**: Grid-based transition with flip animation
- **Circle**: Circular wipe transition
- **Directional**: Directional slide transition
- **UndulatingBurnOut**: Wave-like fade effect
- **SquaresWire**: Wireframe square expansion
- **RotateScaleFade**: Combined rotation, scaling, and fade
- **Pixelate**: Pixelation dissolve
- **CrossZoom**: Simultaneous zoom and crossfade
- **PolkaDotsCurtain**: Dotted curtain reveal
Transitions are resolved by name matching (case-insensitive) against multiple formats:
```typescript
const transitionGridFlip =
transition.name === 'GridFlip' ||
name.toLowerCase() === 'gridflip' ||
transition.label === 'gridflip';
```
## Rendering Configuration
### HTML Template Configuration
When using the HTML template for rendering, configuration is injected via `window.RENDER_CONFIG`:
```typescript
const jsonConfig = window.RENDER_CONFIG;
if (!jsonConfig) {
throw new Error('No render configuration provided');
}
const settings = jsonConfig.settings || {};
const compositorOpts = {
width: settings.width || 1280,
height: settings.height || 720,
fps: settings.fps || 30,
bgColor: settings.bgColor || '#000000',
};
```
## Memory Management
The sprite renderer properly cleans up resources:
- Textures are destroyed with `destroy(true)` to free GPU memory
- Canvas dimensions are validated before allocation
- Frame textures are only recreated when size changes
## See Also
- [Studio and Timeline](studio.md) - Higher-level project management
- [Clips and Media](clips.md) - Media element handling
- [Animation System](animation.md) - Animation creation and playback
- [Node.js Rendering](node-rendering.md) - Server-side rendering guide
---
## Effects System
### 相关页面
相关主题：[Transitions System](#page-9), [Animation System](#page-6)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/effect/effect.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/effect/effect.ts)
- [packages/openvideo/src/effect/glsl/gl-effect.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/effect/glsl/gl-effect.ts)
- [packages/openvideo/src/effect/glsl/custom-glsl.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/effect/glsl/custom-glsl.ts)
- [packages/openvideo/src/effect/types.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/effect/types.ts)
- [packages/openvideo/src/effect/vertex.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/effect/vertex.ts)
- [packages/openvideo/src/effect/interface/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/effect/interface/index.ts)
- [packages/openvideo/src/utils/chromakey.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/chromakey.ts)
- [packages/openvideo/src/utils/color-adjustment.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/color-adjustment.ts)
# Effects System
## Overview
The Effects System in OpenVideo is a modular, GPU-accelerated pipeline for applying visual transformations to clips during playback and rendering. Built on top of PixiJS and leveraging the WebCodecs API, the system provides both built-in effects (blur, brightness, color adjustment, chromakey) and a shader-based GLSL extension framework for custom visual effects. The architecture separates effect definition from effect application, allowing effects to be applied at the clip level with configurable timing, duration, and intensity.
Effects are defined as standalone modules in `packages/openvideo/src/effect/` and can be attached to any clip through the timeline system. The system supports real-time preview during editing and high-quality final rendering through WebCodecs encoding.
## Architecture
### System Components
The Effects System consists of several interconnected layers:
```mermaid
graph TD
A[Clip Instance] --> B[Effect Manager]
B --> C[Built-in Effects]
B --> D[Custom GLSL Effects]
C --> E[Blur Effect]
C --> F[Color Adjustment]
C --> G[Chromakey Effect]
D --> H[GLSL Fragment Shaders]
D --> I[GLSL Vertex Shaders]
B --> J[Compositor Rendering]
J --> K[PixiJS Renderer]
K --> L[WebCodecs Encoder]
```
### Directory Structure
```
packages/openvideo/src/
├── effect/
│ ├── effect.ts # Core Effect class and base implementation
│ ├── types.ts # TypeScript type definitions for effects
│ ├── vertex.ts # Vertex shader utilities
│ ├── glsl/
│ │ ├── gl-effect.ts # WebGL effect renderer
│ │ └── custom-glsl.ts # Custom shader registration system
│ └── interface/
│ └── index.ts # Effect interface contracts
├── utils/
│ ├── chromakey.ts # Green screen/chromakey utilities
│ └── color-adjustment.ts # Color correction utilities
```
## Core Effect Interface
### IEffect Interface
All effects must implement the `IEffect` interface defined in `packages/openvideo/src/effect/interface/index.ts`:
| Property | Type | Description |
|----------|------|-------------|
| `id` | `string` | Unique identifier for the effect instance |
| `key` | `string` | Effect type key (e.g., "blur", "brightness") |
| `startTime` | `number` | Start time in microseconds |
| `duration` | `number` | Effect duration in microseconds |
| `targets` | `number[]` | Optional array of target indices for multi-target effects |
| `enabled` | `boolean` | Whether the effect is currently active |
资料来源：[packages/openvideo/src/effect/interface/index.ts]()
### EffectOptions Type
The `EffectOptions` type in `packages/openvideo/src/effect/types.ts` provides the base configuration structure:
```typescript
interface EffectOptions {
id?: string;
key?: string;
startTime?: number;
duration?: number;
targets?: number[];
enabled?: boolean;
[key: string]: any; // Effect-specific parameters
}
```
资料来源：[packages/openvideo/src/effect/types.ts]()
## Built-in Effects
### Blur Effect
The blur effect applies Gaussian blur to the clip using PixiJS's built-in blur filter. Configuration includes blur radius and quality settings.
```mermaid
graph LR
A[Input Texture] --> B[BlurFilter]
B --> C[Radius Parameter]
C --> D[Quality Iterations]
D --> E[Output Texture]
```
### Color Adjustment Effect
Color adjustments are applied through the `ColorAdjustment` type defined in `packages/openvideo/src/utils/color-adjustment.ts`. This effect modifies the visual characteristics of the source content:
| Parameter | Type | Range | Description |
|-----------|------|-------|-------------|
| `brightness` | `number` | 0.0 - 2.0 | Brightness multiplier (1.0 = normal) |
| `contrast` | `number` | 0.0 - 2.0 | Contrast adjustment (1.0 = normal) |
| `saturation` | `number` | 0.0 - 2.0 | Color saturation (0.0 = grayscale, 1.0 = normal) |
| `hue` | `number` | -180 to 180 | Hue rotation in degrees |
| `exposure` | `number` | -1.0 to 1.0 | Exposure adjustment |
资料来源：[packages/openvideo/src/utils/color-adjustment.ts]()
### Chromakey Effect
The chromakey utility in `packages/openvideo/src/utils/chromakey.ts` implements green screen (chroma key) removal for video composition. This effect:
- Detects pixels matching a target color within a tolerance range
- Replaces matching pixels with transparency
- Supports spill suppression to remove color artifacts
```mermaid
graph TD
A[Source Frame] --> B[Color Matching]
B --> C{Tolerance Check}
C -->|Match| D[Set Alpha = 0]
C -->|No Match| E[Keep Pixel]
D --> F[Spill Suppression]
E --> F
F --> G[Output Frame]
```
## GLSL Shader Effects
### Custom GLSL System
The custom shader system allows developers to register their own fragment and vertex shaders. The system is built around two main components:
1. **GL Effect Renderer** (`packages/openvideo/src/effect/glsl/gl-effect.ts`) - Handles WebGL shader compilation and execution
2. **Custom GLSL Registry** (`packages/openvideo/src/effect/glsl/custom-glsl.ts`) - Manages shader registration and retrieval
### Shader Structure
Each custom shader effect follows a standard structure:
```mermaid
graph TD
A[CustomShader Object] --> B[Fragment Shader]
A --> C[Vertex Shader]
A --> D[Uniforms]
A --> E[Sampler Inputs]
B --> F[GLSL Main Function]
D --> G[uniform float time]
D --> H[uniform vec2 resolution]
D --> I[Effect Parameters]
```
### Vertex Shader Utilities
The `vertex.ts` file provides common vertex shader utilities and transformations for effects that require custom vertex manipulation:
| Function | Purpose |
|----------|---------|
| `getStandardVertexShader()` | Returns the default passthrough vertex shader |
| `getUVTransform()` | Generates UV coordinate transformations |
| `applyTransform()` | Applies transformation matrix to vertices |
资料来源：[packages/openvideo/src/effect/vertex.ts]()
## Effect Application in Clips
### Effect Integration with Clips
Effects are attached to clips through the `effects` array in the clip's configuration. Each effect entry references an effect by key and timing:
```mermaid
graph TD
A[Clip JSON Config] --> B[effects Array]
B --> C[Effect Entry 1]
B --> D[Effect Entry 2]
C --> E{id: string}
C --> F{key: string}
C --> G{startTime: number}
C --> H{duration: number}
```
The base clip JSON structure from `packages/openvideo/src/json-serialization.ts` shows the effect definition format:
```typescript
effects?: Array<{
id: string;
key: string;
startTime: number;
duration: number;
targets?: number[];
}>;
```
资料来源：[packages/openvideo/src/json-serialization.ts]()
### Effect Timing and Targeting
Effects support precise temporal control:
| Property | Description |
|----------|-------------|
| `startTime` | Microseconds from clip start when effect begins |
| `duration` | Length of effect application in microseconds |
| `targets` | Optional array of sub-element indices (for multi-layer clips) |
## Sprite-Level Effect Properties
The base sprite system in `packages/openvideo/src/sprite/base-sprite.ts` defines render-time effect properties that are applied during the render transform calculation:
| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `blur` | `number` | 0 | Blur intensity applied during rendering |
| `brightness` | `number` | 1 | Brightness multiplier |
| `mirror` | `number` | 0 | Mirror effect intensity (0-1) |
| `motionBlur` | `number` | 0 | Motion blur amount for moving elements |
These properties are combined from multiple animation sources at render time:
```mermaid
graph TD
A[Base Transform] --> C[Combined Transform]
B[Animation Transform] --> C
E[Effect Properties] --> C
C --> D[Final Render]
```
资料来源：[packages/openvideo/src/sprite/base-sprite.ts]()
## Effect Processing Pipeline
### Render Pipeline Integration
The effects system integrates with the overall render pipeline as follows:
```mermaid
sequenceDiagram
participant Timeline
participant Clip
participant EffectManager
participant Compositor
participant WebCodecs
Timeline->>Clip: currentTime update
Clip->>EffectManager: getActiveEffects(time)
EffectManager->>EffectManager: filter by startTime/duration
EffectManager-->>Compositor: activeEffects[]
Compositor->>Compositor: applyEffects(frame)
Compositor->>WebCodecs: encodeFrame()
```
### Effect Order of Application
Effects are applied in a deterministic order:
1. **Color Adjustments** - Brightness, contrast, saturation
2. **Blur Effects** - Applied as PixiJS filters
3. **Custom GLSL Shaders** - User-defined effects
4. **Chromakey** - Applied last for proper compositing
## Effect Configuration API
### Creating an Effect
```typescript
import { Effect } from 'openvideo';
// Built-in effect
const blurEffect = new Effect({
key: 'blur',
startTime: 0,
duration: 5000000, // 5 seconds in microseconds
intensity: 10
});
// Custom GLSL effect
const customEffect = new Effect({
key: 'my-custom-shader',
startTime: 1000000,
duration: 3000000,
uniforms: {
color: [1.0, 0.5, 0.0],
strength: 0.8
}
});
```
### Attaching Effects to Clips
```typescript
const videoClip = await Video.fromUrl('video.mp4');
videoClip.effects = [
{
id: 'effect-1',
key: 'colorAdjustment',
startTime: 0,
duration: clip.duration,
brightness: 1.2,
saturation: 0.8
}
];
```
## Custom Shader Registration
### Registering Custom Shaders
Developers can register custom GLSL shaders through the custom shader registry:
```typescript
import { registerCustomShader } from 'openvideo';
registerCustomShader({
name: 'my-custom-effect',
fragmentShader: `
uniform float strength;
uniform vec3 color;
void main() {
vec4 texture = texture2D(uSampler, vTextureCoord);
vec3 adjusted = mix(texture.rgb, color, strength);
gl_FragColor = vec4(adjusted, texture.a);
}
`,
uniforms: [
{ name: 'strength', type: 'float', value: 0.5 },
{ name: 'color', type: 'vec3', value: [1, 1, 1] }
]
});
```
资料来源：[packages/openvideo/src/effect/glsl/custom-glsl.ts]()
## Performance Considerations
### GPU vs CPU Effects
| Effect Type | Processing Location | Performance Impact |
|-------------|---------------------|-------------------|
| Built-in Filters | GPU (PixiJS) | Low |
| Color Adjustments | GPU (Shader) | Low |
| Chromakey | GPU (Shader) | Medium |
| Custom GLSL | GPU (User Shader) | Varies |
| Complex Blur | CPU/GPU | High |
### Optimization Strategies
1. **Batch similar effects** - Group effects that use the same shader
2. **Use appropriate precision** - Prefer `lowp` or `mediump` where possible
3. **Limit effect count** - Excessive effect stacking impacts performance
4. **Consider resolution scaling** - Lower resolution during preview, full resolution for export
## Related Components
| Component | File Path | Relationship |
|-----------|-----------|--------------|
| BaseSprite | `packages/openvideo/src/sprite/base-sprite.ts` | Applies effect properties during render |
| Compositor | `packages/openvideo/src/compositor.ts` | Executes effects during frame rendering |
| JsonSerialization | `packages/openvideo/src/json-serialization.ts` | Serializes effect configurations |
| CaptionClip | `packages/openvideo/src/clips/caption-clip.ts` | Uses color adjustment effects |
| ColorAdjustment | `packages/openvideo/src/utils/color-adjustment.ts` | Utility for color corrections |
| Chromakey | `packages/openvideo/src/utils/chromakey.ts` | Green screen removal utility |
---
## Transitions System
### 相关页面
相关主题：[Effects System](#page-8), [Animation System](#page-6)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/transition/transition.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/transition.ts)
- [packages/openvideo/src/transition/glsl/gl-transition.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/glsl/gl-transition.ts)
- [packages/openvideo/src/transition/glsl/custom-glsl.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/glsl/custom-glsl.ts)
- [packages/openvideo/src/transition/fragment.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/fragment.ts)
- [packages/openvideo/src/transition/uniforms.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/uniforms.ts)
- [packages/openvideo/src/transition/types.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/types.ts)
- [packages/openvideo/src/transition/vertex.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/transition/vertex.ts)
# Transitions System
The Transitions System is a core module within the openvideo library that enables visual transitions between video clips using WebGL fragment shaders. It provides over 30 built-in transition effects including wipes, fades, geometric transformations, and artistic effects, all implemented through GLSL shaders with customizable uniforms.
## Architecture Overview
The transitions system follows a layered architecture that separates shader definitions, uniform management, and transition lookup logic.
```mermaid
graph TD
A[Transition Request] --> B[Name Resolution]
B --> C[Variant Matching]
C --> D{Gound?}
D -->|Yes| E[Select GLSL Fragment]
D -->|No| F[Variant Generation]
F --> G{Lowercase match?}
G -->|Yes| E
G -->|No| H[Error/Default]
E --> I[Uniform Assembly]
I --> J[WebGL Render]
K[Basic Uniforms] --> I
L[Transition-Specific Uniforms] --> I
```
### Core Components
| Component | File | Purpose |
|-----------|------|---------|
| Main Controller | `transition.ts` | Transition lookup, name resolution, GLSL selection |
| GLSL Base | `gl-transition.ts` | WebGL transition shader base class |
| Custom GLSL | `custom-glsl.ts` | Custom shader compilation and execution |
| Fragment Shaders | `fragment.ts` | GLSL shader source code for each transition |
| Uniforms | `uniforms.ts` | Uniform type definitions and defaults |
| Types | `types.ts` | TypeScript interfaces for transitions |
| Vertex Shader | `vertex.ts` | Vertex shader for quad rendering |
资料来源：[packages/openvideo/src/transition/transition.ts:1-50]()
## Transition Types
The system includes the following built-in transitions, categorized by their visual effect:
### Wipe & Reveal Transitions
| Transition Name | Description | Key Uniforms |
|-----------------|-------------|--------------|
| `directional` | Directional wipe effect | direction, smoothness |
| `directionalwipe` / `directional_wipe` | Wipe with direction control | angle, feather |
| `directionalwarp` / `directional_warp` | Warped directional wipe | strength, angle |
| `crosshatch` | Crosshatch pattern reveal | density, smoothness |
### Geometric Transitions
| Transition Name | Description | Key Uniforms |
|-----------------|-------------|--------------|
| `circle` / `circleopen` | Circle expand/reveal | radius, feather |
| `squareswire` / `squaresWire` | Square wireframe reveal | size, smoothness |
| `hexagonalize` | Hexagonal mosaic transition | scale, angle |
| `bowtiehorizontal` / `BowTieHorizontal` | Bow tie shape reveal | size, angle |
### Fade & Blend Transitions
| Transition Name | Description | Key Uniforms |
|-----------------|-------------|--------------|
| `crosszoom` / `CrossZoom` | Cross-zoom effect with blur | zoom, rotation |
| `rotate_scale_fade` / `rotatescalefade` | Combined rotation, scale, fade | angle, scale |
| `luma` | Luminance-based dissolve | threshold, smoothness |
| `luminance_melt` / `luminancemelt` | Melt effect based on brightness | direction, smoothness |
| `perlin` | Perlin noise-based transition | scale, seed |
### Distortion Transitions
| Transition Name | Description | Key Uniforms |
|-----------------|-------------|--------------|
| `glitchdisplace` / `GlitchDisplace` | Glitch-style displacement | intensity, seed |
| `undulatingburnout` / `UndulatingBurnOut` | Undulating wave burnout | amplitude, frequency |
| `displacement` | Generic displacement map | scale, map |
| `polar_function` / `polar_function` | Polar coordinate transform | radius, angle |
### Creative Transitions
| Transition Name | Description | Key Uniforms |
|-----------------|-------------|--------------|
| `gridflip` / `GridFlip` | Grid-based flip reveal | gridSize, axis |
| `randomSquares` / `RandomSquares` | Random square reveal | size, smoothness |
| `pixelize` / `Pixelize` | Pixelation effect | pixelSize |
| `heart` | Heart-shaped reveal | scale, smoothness |
| `cannabisleaf` / `cannabis_leaf` | Cannabis leaf pattern | scale, rotation |
| `polkadotscurtain` / `PolkaDotsCurtain` | Polka dots curtain | dotSize, spacing |
| `stereoviewer` / `StereoViewer` | Stereo viewer effect | depth, offset |
| `crazyparametricfun` / `CrazyParametricFun` | Parametric function effect | function, iterations |
资料来源：[packages/openvideo/src/transition/transition.ts:60-150]()
## Name Resolution & Matching
The transition system supports flexible name matching to accommodate various naming conventions used by developers.
### Matching Algorithm
```mermaid
graph LR
A[Input Name] --> B[Direct Match]
A --> C[Lowercase Match]
A --> D[Capitalized Match]
A --> E[Snake_case Match]
A --> F[No Separator Match]
B --> G{Match Found?}
C --> G
D --> G
E --> G
F --> G
G -->|Yes| H[Return Transition]
G -->|No| I[Error with Available Names]
```
### Variant Generation Strategy
The system attempts multiple name transformations when looking up transitions:
1. **Exact match** - `transition.name === input`
2. **Lowercase match** - `name.toLowerCase() === input.toLowerCase()`
3. **Capitalized match** - `name.charAt(0).toUpperCase() + name.slice(1).toLowerCase()`
4. **Snake_case conversion** - `name.replace(/([A-Z])/g, '_$1').toLowerCase()`
5. **No separator** - `name.replace(/_/g, '').toLowerCase()`
资料来源：[packages/openvideo/src/transition/transition.ts:180-220]()
## Uniform System
Transitions use uniforms to pass parameters to GLSL shaders. The uniform system handles type conversion and default values.
### Basic Uniforms
All transitions receive these fundamental uniforms:
```typescript
interface BasicUniforms {
tex: sampler2D; // Source texture
tex2: sampler2D; // Target texture
progress: number; // Transition progress (0-1)
time: number; // Current time
resolution: vec2; // Canvas resolution
}
```
### Uniform Type Conversion
The system automatically converts WebGPU-style uniform types to WebGL-compatible formats:
| WebGPU Type | WebGL Type | Conversion |
|-------------|------------|------------|
| `int` | `i32` | `Math.trunc(value)` |
| `ivec2` | `vec2` | Direct conversion |
资料来源：[packages/openvideo/src/transition/transition.ts:100-115]()
### Transition-Specific Uniforms
Each transition type defines its own uniforms that extend the basic set:
```typescript
// Example: GridFlip uniforms
const GRIDFLIP_UNIFORMS = {
gridSize: { type: 'f32', value: 10.0 },
axis: { type: 'i32', value: 0 },
};
// Example: Circle uniforms
const CIRCLEOPEN_UNIFORMS = {
radius: { type: 'f32', value: 1.5 },
feather: { type: 'f32', value: 0.05 },
};
```
资料来源：[packages/openvideo/src/transition/uniforms.ts]()
## GLSL Shader Structure
### Vertex Shader
The vertex shader renders a full-screen quad for transition effects:
```glsl
attribute vec2 a_position;
attribute vec2 a_texCoord;
varying vec2 v_texCoord;
void main() {
gl_Position = vec4(a_position, 0.0, 1.0);
v_texCoord = a_texCoord;
}
```
### Fragment Shader Pattern
Each transition implements a fragment shader with the following signature:
```glsl
uniform sampler2D tex; // From texture
uniform sampler2D tex2; // To texture
uniform float progress; // 0.0 to 1.0
uniform vec2 resolution; // Viewport size
// Transition-specific uniforms...
varying vec2 v_texCoord;
vec4 getFromColor(vec2 uv) {
return texture2D(tex, uv);
}
vec4 getToColor(vec2 uv) {
return texture2D(tex2, uv);
}
void main() {
// Transition-specific logic
vec4 from = getFromColor(uv);
vec4 to = getToColor(uv);
gl_FragColor = mix(from, to, progress);
}
```
资料来源：[packages/openvideo/src/transition/vertex.ts]()
资料来源：[packages/openvideo/src/transition/fragment.ts]()
## Rendering Pipeline
```mermaid
graph TD
A[Timeline Position] --> B{Transition Zone?}
B -->|No| C[Single Clip Render]
B -->|Yes| D[Load From/To Textures]
D --> E[Calculate Progress]
E --> F[Set Uniform Values]
F --> G[Compile GLSL if needed]
G --> H[Render Transition Shader]
H --> I[Output Composite Frame]
J[Basic Uniforms] --> F
K[Transition Uniforms] --> F
L[Custom Uniforms] --> F
```
### Rendering Steps
1. **Texture Binding** - Source and destination textures are bound to texture units
2. **Progress Calculation** - Transition progress (0-1) is calculated from timeline position
3. **Uniform Assembly** - Basic uniforms are merged with transition-specific uniforms
4. **Shader Execution** - WebGL renders the full-screen quad with the transition shader
5. **Output** - The composited frame is written to the canvas
资料来源：[packages/openvideo/src/transition/glsl/gl-transition.ts]()
## Custom Transitions
### Creating Custom GLSL
Developers can create custom transitions by implementing the fragment shader pattern:
```typescript
import { GLTransition } from './types';
const CUSTOM_FRAGMENT = `
uniform float customParam;
varying vec2 v_texCoord;
vec4 getFromColor(vec2 uv) {
return texture2D(tex, uv);
}
vec4 getToColor(vec2 uv) {
return texture2D(tex2, uv);
}
// Custom transition logic
void main() {
vec2 uv = v_texCoord;
vec4 from = getFromColor(uv);
vec4 to = getToColor(uv);
float ratio = 1.0 - customParam;
float pr = smoothstep(0.0, ratio, progress);
gl_FragColor = mix(from, to, pr);
}
`;
const customTransition: GLTransition = {
name: 'custom',
fragment: CUSTOM_FRAGMENT,
uniforms: {
customParam: { type: 'f32', value: 0.5 },
},
};
```
资料来源：[packages/openvideo/src/transition/glsl/custom-glsl.ts]()
### Registering Custom Transitions
Custom transitions can be registered into the local transition registry for flexible lookup:
```typescript
import { registerTransition, getAllTransitions } from './transition';
// Register custom transition
registerTransition('myCustom', customTransition);
// Retrieve all transitions including custom
const all = getAllTransitions();
```
## Type Definitions
### GLTransition Interface
```typescript
interface GLTransition {
name: string; // Unique identifier
label?: string; // Display name
fragment: string; // GLSL fragment shader source
vertex?: string; // Custom vertex shader (optional)
uniforms: {
[key: string]: UniformDefinition;
};
}
interface UniformDefinition {
type: 'f32' | 'i32' | 'vec2' | 'vec4';
value: number | number[];
}
```
资料来源：[packages/openvideo/src/transition/types.ts]()
## Performance Considerations
### Shader Compilation
- GLSL shaders are compiled once and cached for reuse
- Subsequent uses of the same transition use the cached program
### Texture Sampling
- Transitions use bilinear filtering for smooth interpolation
- Mipmaps are generated for efficient downsampling during blur effects
### Uniform Updates
- Progress uniform is updated every frame during transitions
- Type conversion is performed once when the transition starts, not per-frame
## Error Handling
When a transition cannot be found, the system provides helpful error messages:
```typescript
// Error message format
`Transition '${name}' not found. Available: ${availableNames}...`
```
The system attempts to suggest similar transitions by checking the first 5 available transitions and 3 local definitions.
资料来源：[packages/openvideo/src/transition/transition.ts:230-250]()
## See Also
- [Studio System](../studio) - High-level composition API that uses transitions
- [Clip System](../clips) - Video clip handling with transition support
- [Animation Presets](../animation/presets) - Keyframe animation system
---
## Utilities and Helpers
### 相关页面
相关主题：[Compositor and Rendering](#page-7), [Clip System](#page-5)
相关源码文件
以下源码文件用于生成本页说明：
- [packages/openvideo/src/utils/index.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/index.ts)
- [packages/openvideo/src/utils/asset-manager.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/asset-manager.ts)
- [packages/openvideo/src/utils/audio.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/audio.ts)
- [packages/openvideo/src/utils/srt-parser.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/srt-parser.ts)
- [packages/openvideo/src/utils/fonts.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/fonts.ts)
- [packages/openvideo/src/utils/video.ts](https://github.com/openvideovdev/openvideo/blob/main/packages/openvideo/src/utils/video.ts)
- [packages/openvideo/src/utils/color.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/color.ts)
- [packages/openvideo/src/utils/common.ts](https://github.com/openvideodev/openvideo/blob/main/packages/openvideo/src/utils/common.ts)
# Utilities and Helpers
## Overview
The `openvideo` library provides a comprehensive suite of utility modules under `packages/openvideo/src/utils/` that support core functionality across the video rendering pipeline. These utilities handle resource management, media processing, formatting, and color manipulation—enabling the higher-level components like `Studio`, `Compositor`, and various clip types to operate efficiently.
## Architecture Overview
```mermaid
graph TD
subgraph "Utility Layer"
UTILS_INDEX[utils/index.ts
Module Exports]
ASSET_MGR[asset-manager.ts
Resource Management]
AUDIO_UTIL[audio.ts
Audio Processing]
SRT[SRT Parser
Subtitle Handling]
FONTS[fonts.ts
Font Management]
VIDEO_UTIL[video.ts
Video Utilities]
COLOR[color.ts
Color Operations]
COMMON[common.ts
Common Helpers]
end
subgraph "Consumer Components"
STUDIO[Studio]
COMPOSITOR[Compositor]
VIDEO_CLIP[VideoClip]
TEXT_CLIP[TextClip]
CAPTION_CLIP[CaptionClip]
end
UTILS_INDEX --> ASSET_MGR
UTILS_INDEX --> AUDIO_UTIL
UTILS_INDEX --> SRT
UTILS_INDEX --> FONTS
UTILS_INDEX --> VIDEO_UTIL
UTILS_INDEX --> COLOR
UTILS_INDEX --> COMMON
ASSET_MGR --> VIDEO_CLIP
ASSET_MGR --> STUDIO
FONTS --> TEXT_CLIP
AUDIO_UTIL --> COMPOSITOR
COLOR --> TEXT_CLIP
COLOR --> CAPTION_CLIP
VIDEO_UTIL --> VIDEO_CLIP
```
## Module Descriptions
### Asset Manager (`asset-manager.ts`)
The asset manager provides centralized resource handling capabilities for loading and managing media assets. It abstracts the complexities of OPFS (Origin Private File System) operations and provides consistent interfaces for accessing streams and files.
**Key Responsibilities:**
- Stream management for remote and local resources
- OPFS file read/write operations
- Resource lifecycle management
**Source:** [packages/openvideo/src/utils/asset-manager.ts:1-50]()
### Audio Utilities (`audio.ts`)
Audio utilities provide processing functions for audio tracks within video compositions. These utilities handle audio decoding, volume control, and track synchronization.
**Key Functions:**
| Function | Purpose |
|----------|---------|
| `processAudioTrack()` | Process raw audio samples for integration |
| `adjustVolume()` | Apply volume modifications to audio streams |
| `syncAudioTracks()` | Synchronize multiple audio tracks |
**Source:** [packages/openvideo/src/utils/audio.ts:1-40]()
### SRT Parser (`srt-parser.ts`)
The SRT parser handles SubRip subtitle file parsing and conversion. It transforms SRT format subtitles into structured caption data usable by the `CaptionClip` component.
**Parsing Flow:**
```mermaid
graph LR
SRT_FILE[SRT File] --> PARSER[SRT Parser]
PARSER --> CAPTION_DATA[Caption Data Structure]
CAPTION_DATA --> CAPTION_CLIP[CaptionClip]
```
**Source:** [packages/openvideo/src/utils/srt-parser.ts:1-60]()
### Font Management (`fonts.ts`)
Font utilities manage font loading, caching, and fallback strategies for text rendering in `TextClip` and `CaptionClip` components.
**Font Loading Strategy:**
| Step | Operation |
|------|-----------|
| 1 | Check font cache for existing entry |
| 2 | If missing, fetch font from URL |
| 3 | Parse font data using FontFace API |
| 4 | Add to document fonts |
| 5 | Cache for subsequent use |
**Source:** [packages/openvideo/src/utils/fonts.ts:1-45]()
### Video Utilities (`video.ts`)
Video utilities provide helper functions for video frame processing, thumbnail generation, and metadata extraction. These utilities support the `VideoClip` component's frame-seeking and thumbnail capabilities.
**Source:** [packages/openvideo/src/utils/video.ts:1-55]()
### Color Utilities (`color.ts`)
Color utilities handle color format conversion, validation, and transformation for consistent styling across text, captions, and visual effects.
**Supported Color Formats:**
| Format | Example |
|--------|---------|
| Hex | `#FF5733` |
| RGB | `rgb(255, 87, 51)` |
| RGBA | `rgba(255, 87, 51, 0.5)` |
| Named | `red`, `blue`, `transparent` |
**Usage in Clips:**
The `TextClip` and `CaptionClip` components utilize color utilities for:
- `fill` property for text color
- `stroke` color configuration
- `dropShadow` color and alpha values
**Source:** [packages/openvideo/src/utils/color.ts:1-35]()
### Common Utilities (`common.ts`)
Common utilities provide shared helper functions used throughout the library, including type guards, validation helpers, and general-purpose transformations.
**Source:** [packages/openvideo/src/utils/common.ts:1-30]()
## Integration with Core Components
### VideoClip Integration
The `VideoClip` class relies on utility modules for resource loading and video processing:
```typescript
// From packages/openvideo/src/clips/video-clip.ts:42-55
const stream = await ResourceManager.getReadableStream(url);
const clip = new Video(stream, {}, url);
await clip.ready;
```
### TextClip Integration
Text styling leverages color utilities for consistent rendering:
```typescript
// From packages/openvideo/src/clips/text-clip.ts:95-110
if (this.originalOpts.fill !== undefined)
style.color = this.originalOpts.fill as any;
if (this.originalOpts.stroke) {
style.stroke = {
color: this.originalOpts.stroke.color as any,
width: this.originalOpts.stroke.width,
};
}
```
### CaptionClip Integration
Caption styling combines color, font, and formatting utilities:
```typescript
// From packages/openvideo/src/clips/caption-clip.ts:120-135
if (opts.dropShadow) {
style.shadow = {
color: (opts.dropShadow.color ?? '#000000') as string,
alpha: opts.dropShadow.alpha ?? 0.5,
blur: opts.dropShadow.blur ?? 4,
distance: opts.dropShadow.distance ?? 0,
angle: opts.dropShadow.angle ?? 0,
};
}
```
## Module Export Pattern
All utilities are exported through the central `utils/index.ts` entry point:
```typescript
// packages/openvideo/src/utils/index.ts
export * from './asset-manager';
export * from './audio';
export * from './srt-parser';
export * from './fonts';
export * from './video';
export * from './color';
export * from './common';
```
This pattern allows consumers to import specific utilities as needed:
```typescript
import { ResourceManager, ColorUtils } from 'openvideo';
```
## Utility Functions Reference
### Resource Management
| Function | Parameters | Returns | Description |
|----------|------------|---------|-------------|
| `getReadableStream()` | `url: string` | `Promise` | Fetches resource as stream |
| `write()` | `file: OPFSToolFile, stream: ReadableStream` | `Promise` | Writes stream to OPFS file |
### Color Operations
| Function | Parameters | Returns | Description |
|----------|------------|---------|-------------|
| `parseColor()` | `color: string` | `ColorObject` | Parse color string to object |
| `toHex()` | `color: ColorObject` | `string` | Convert to hex format |
### Font Operations
| Function | Parameters | Returns | Description |
|----------|------------|---------|-------------|
| `loadFont()` | `name: string, url: string` | `Promise` | Load and register font |
| `getFont()` | `name: string` | `FontFace \| null` | Retrieve cached font |
## Summary
The utility modules in openvideo form a foundational layer that abstracts common operations away from business logic. By centralizing resource management, media processing, color handling, and font operations, the library maintains consistency and reduces duplication across components like `VideoClip`, `TextClip`, and `CaptionClip`. These utilities are designed to be composable and are exported through a single entry point for convenient consumption.
---
---
## Doramagic 踩坑日志
项目：openvideodev/openvideo
摘要：发现 11 个潜在踩坑项，其中 0 个为 high/blocking；最高优先级：安装坑 - 来源证据：Modular architecture: separate core logic from PixiJS renderer to enable React Native / headless use。
## 1. 安装坑 · 来源证据：Modular architecture: separate core logic from PixiJS renderer to enable React Native / headless use
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Modular architecture: separate core logic from PixiJS renderer to enable React Native / headless use
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_28cbc35c1a0544478da4fb17013e349a | https://github.com/openvideodev/openvideo/issues/78 | 来源类型 github_issue 暴露的待验证使用条件。
## 2. 配置坑 · 来源证据：Image and Audio clips lost during loadFromJSON/exportToJSON round-trip
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：Image and Audio clips lost during loadFromJSON/exportToJSON round-trip
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_e2e8dc4e56504c3c909c7cf990b5a336 | https://github.com/openvideodev/openvideo/issues/60 | 来源类型 github_issue 暴露的待验证使用条件。
## 3. 配置坑 · 来源证据：[Bug] Transition effect "display" value update not reflected in Player after clip repositioning
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：[Bug] Transition effect "display" value update not reflected in Player after clip repositioning
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_f5e332be3e884e0eb6bb27bb45d7524a | https://github.com/openvideodev/openvideo/issues/52 | 来源类型 github_issue 暴露的待验证使用条件。
## 4. 配置坑 · 来源证据：display property not initialized for Image clips
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：display property not initialized for Image clips
- 对用户的影响：可能阻塞安装或首次运行。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_45024018965c442dbaaf3bb5441b75f2 | https://github.com/openvideodev/openvideo/issues/18 | 来源类型 github_issue 暴露的待验证使用条件。
## 5. 配置坑 · 来源证据：meta property is read-only
- 严重度：medium
- 证据强度：source_linked
- 发现：GitHub 社区证据显示该项目存在一个配置相关的待验证问题：meta property is read-only
- 对用户的影响：可能增加新用户试用和生产接入成本。
- 建议检查：来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- 防护动作：不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- 证据：community_evidence:github | cevd_2798efa2236a4fe1a82930d7a1b1cc80 | https://github.com/openvideodev/openvideo/issues/19 | 来源类型 github_issue 暴露的待验证使用条件。
## 6. 能力坑 · 能力判断依赖假设
- 严重度：medium
- 证据强度：source_linked
- 发现：README/documentation is current enough for a first validation pass.
- 对用户的影响：假设不成立时，用户拿不到承诺的能力。
- 建议检查：将假设转成下游验证清单。
- 防护动作：假设必须转成验证项；没有验证结果前不能写成事实。
- 证据：capability.assumptions | github_repo:1125747446 | https://github.com/openvideodev/openvideo | README/documentation is current enough for a first validation pass.
## 7. 维护坑 · 维护活跃度未知
- 严重度：medium
- 证据强度：source_linked
- 发现：未记录 last_activity_observed。
- 对用户的影响：新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。
- 建议检查：补 GitHub 最近 commit、release、issue/PR 响应信号。
- 防护动作：维护活跃度未知时，推荐强度不能标为高信任。
- 证据：evidence.maintainer_signals | github_repo:1125747446 | https://github.com/openvideodev/openvideo | last_activity_observed missing
## 8. 安全/权限坑 · 下游验证发现风险项
- 严重度：medium
- 证据强度：source_linked
- 发现：no_demo
- 对用户的影响：下游已经要求复核，不能在页面中弱化。
- 建议检查：进入安全/权限治理复核队列。
- 防护动作：下游风险存在时必须保持 review/recommendation 降级。
- 证据：downstream_validation.risk_items | github_repo:1125747446 | https://github.com/openvideodev/openvideo | no_demo; severity=medium
## 9. 安全/权限坑 · 存在评分风险
- 严重度：medium
- 证据强度：source_linked
- 发现：no_demo
- 对用户的影响：风险会影响是否适合普通用户安装。
- 建议检查：把风险写入边界卡，并确认是否需要人工复核。
- 防护动作：评分风险必须进入边界卡，不能只作为内部分数。
- 证据：risks.scoring_risks | github_repo:1125747446 | https://github.com/openvideodev/openvideo | no_demo; severity=medium
## 10. 维护坑 · issue/PR 响应质量未知
- 严重度：low
- 证据强度：source_linked
- 发现：issue_or_pr_quality=unknown。
- 对用户的影响：用户无法判断遇到问题后是否有人维护。
- 建议检查：抽样最近 issue/PR，判断是否长期无人处理。
- 防护动作：issue/PR 响应未知时，必须提示维护风险。
- 证据：evidence.maintainer_signals | github_repo:1125747446 | https://github.com/openvideodev/openvideo | issue_or_pr_quality=unknown
## 11. 维护坑 · 发布节奏不明确
- 严重度：low
- 证据强度：source_linked
- 发现：release_recency=unknown。
- 对用户的影响：安装命令和文档可能落后于代码，用户踩坑概率升高。
- 建议检查：确认最近 release/tag 和 README 安装命令是否一致。
- 防护动作：发布节奏未知或过期时，安装说明必须标注可能漂移。
- 证据：evidence.maintainer_signals | github_repo:1125747446 | https://github.com/openvideodev/openvideo | release_recency=unknown