第十章:Canvas 与 WebGL 视频处理
当
<video>元素只负责解码和渲染,Canvas 和 WebGL 就成了视频帧的"后处理工厂"。从最简单的截图下载、实时水印,到逐像素操作的绿幕抠图,再到 GPU 着色器驱动的实时特效,以及 WebCodecs 带来的帧级精确控制——这一章覆盖 Web 端视频处理的完整技术栈,并在每个层次上给出可落地的工程实现。
第一部分:Canvas 基础视频处理
drawImage 的核心机制
ctx.drawImage(video, x, y) 是一切视频帧处理的起点。它将 <video> 元素当前解码的帧直接绘制到 Canvas 上,整个操作在 GPU 完成,不经过 JS 堆内存。但有一个关键约束:只有视频处于播放或暂停状态(即已解码出至少一帧)时,drawImage 才会绘制出有效内容——在 readyState < 2 时调用,Canvas 上什么都不会出现。
跨域视频需要在 <video> 上设置 crossorigin="anonymous",且服务端必须返回正确的 CORS 响应头,否则 drawImage 后调用 canvas.toDataURL() 或 getImageData() 会抛出 SecurityError。
视频截图与下载
截图是最基础的 Canvas 视频处理场景。核心流程是:暂停视频(或在特定时刻)→ drawImage 绘制当前帧 → toBlob 导出 → 通过 <a> 标签触发下载:
class VideoScreenshot {
constructor(video) {
this.video = video;
this.canvas = document.createElement('canvas');
this.ctx = this.canvas.getContext('2d');
}
// 截取当前帧
capture(options = {}) {
const { format = 'image/jpeg', quality = 0.92 } = options;
const video = this.video;
// 匹配视频原始分辨率
this.canvas.width = video.videoWidth;
this.canvas.height = video.videoHeight;
this.ctx.drawImage(video, 0, 0);
return new Promise((resolve) => {
this.canvas.toBlob(resolve, format, quality);
});
}
// 截图并下载
async download(filename = `screenshot-`) {
const blob = await this.capture();
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;
a.click();
// 延迟释放 ObjectURL
setTimeout(() => URL.revokeObjectURL(url), 1000);
}
// 截取指定时间点的帧(需要 seek)
async captureAt(timeSeconds, options = {}) {
return new Promise((resolve, reject) => {
const video = this.video;
const wasPlaying = !video.paused;
// 保存当前时间
const savedTime = video.currentTime;
const onSeeked = async () => {
video.removeEventListener('seeked', onSeeked);
const blob = await this.capture(options);
// 恢复原始状态
video.currentTime = savedTime;
if (wasPlaying) video.play().catch(() => {});
resolve(blob);
};
video.addEventListener('seeked', onSeeked, { once: true });
video.currentTime = timeSeconds;
});
}
}
// 使用
const screenshot = new VideoScreenshot(videoElement);
document.getElementById('capture-btn').addEventListener('click', () => {
screenshot.download();
});实时水印叠加
水印有两种实现策略:Canvas 叠加渲染(将视频帧和水印同时绘制到 Canvas,输出合成图像)和 CSS 覆盖(用绝对定位的 HTML 元素覆盖在 <video> 上)。CSS 方案实现简单但无法嵌入到导出的视频帧中;Canvas 方案可以将水印真正"烧录"到每一帧。
class VideoWatermark {
constructor(video, canvas, config = {}) {
this.video = video;
this.canvas = canvas;
this.ctx = canvas.getContext('2d');
this.config = {
text: '© 版权所有',
font: '16px Arial',
color: 'rgba(255,255,255,0.6)',
position: 'bottom-right', // top-left | top-right | bottom-left | bottom-right | center
padding: 16,
logo: null, // HTMLImageElement,可选
...config
};
this._rafId = null;
}
start() {
this.canvas.width = this.video.videoWidth || 1280;
this.canvas.height = this.video.videoHeight || 720;
const render = () => {
if (this.video.paused || this.video.ended) {
this._rafId = null;
return;
}
this._drawFrame();
this._rafId = requestAnimationFrame(render);
};
this.video.addEventListener('play', () => {
if (!this._rafId) render();
});
// 如果视频已在播放
if (!this.video.paused) render();
}
_drawFrame() {
const { canvas, ctx, video, config } = this;
// 1. 绘制视频帧
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
// 2. 叠加文字水印
ctx.font = config.font;
ctx.fillStyle = config.color;
ctx.textBaseline = 'bottom';
const metrics = ctx.measureText(config.text);
const textW = metrics.width;
const textH = parseInt(config.font); // 近似行高
const pad = config.padding;
let x, y;
switch (config.position) {
case 'top-left':
x = pad; y = textH + pad; break;
case 'top-right':
x = canvas.width - textW - pad; y = textH + pad; break;
case 'bottom-left':
x = pad; y = canvas.height - pad; break;
case 'bottom-right':
x = canvas.width - textW - pad; y = canvas.height - pad; break;
case 'center':
x = (canvas.width - textW) / 2; y = (canvas.height + textH) / 2; break;
default:
x = pad; y = canvas.height - pad;
}
ctx.fillText(config.text, x, y);
// 3. 叠加 Logo(可选)
if (config.logo) {
const logoH = 32;
const logoW = config.logo.width * (logoH / config.logo.height);
ctx.globalAlpha = 0.7;
ctx.drawImage(config.logo, x - logoW - 8, y - logoH, logoW, logoH);
ctx.globalAlpha = 1;
}
}
stop() {
if (this._rafId) {
cancelAnimationFrame(this._rafId);
this._rafId = null;
}
}
}Canvas 滤镜效果
通过 getImageData 获取像素数据,逐像素修改后写回,可以实现任意自定义滤镜。但这是纯 CPU 操作,对高分辨率视频性能较差,适合静态截图处理;实时视频滤镜应使用 WebGL(见第三部分)。
class CanvasFilter {
constructor(canvas) {
this.ctx = canvas.getContext('2d', { willReadFrequently: true });
// willReadFrequently: true 告知浏览器会频繁读取像素,优化内部缓存策略
}
// 灰度滤镜
grayscale(imageData) {
const data = imageData.data;
for (let i = 0; i < data.length; i += 4) {
const avg = (data[i] * 0.299 + data[i+1] * 0.587 + data[i+2] * 0.114);
data[i] = data[i+1] = data[i+2] = avg;
}
return imageData;
}
// 亮度/对比度调整
brightnessContrast(imageData, brightness = 0, contrast = 0) {
const data = imageData.data;
const factor = (259 * (contrast + 255)) / (255 * (259 - contrast));
for (let i = 0; i < data.length; i += 4) {
data[i] = clamp(factor * (data[i] - 128) + 128 + brightness);
data[i+1] = clamp(factor * (data[i+1] - 128) + 128 + brightness);
data[i+2] = clamp(factor * (data[i+2] - 128) + 128 + brightness);
}
return imageData;
}
// 模糊(简单盒式模糊,生产环境建议用 CSS filter 或 WebGL)
applyFilter(video, filterFn) {
const canvas = this.ctx.canvas;
this.ctx.drawImage(video, 0, 0);
const imageData = this.ctx.getImageData(0, 0, canvas.width, canvas.height);
const filtered = filterFn(imageData);
this.ctx.putImageData(filtered, 0, 0);
}
}
function clamp(v) { return Math.max(0, Math.min(255, v)); }更高效的方案是直接使用 CSS filter 属性作用于 Canvas 上下文——Chrome 支持在 drawImage 之前设置 ctx.filter,从而让 GPU 完成滤镜运算:
// CSS filter 直接作用于 Canvas 绘制(无需 getImageData)
ctx.filter = 'grayscale(100%)';
ctx.drawImage(video, 0, 0);
ctx.filter = 'none';
// 多个滤镜叠加
ctx.filter = 'brightness(1.2) contrast(1.1) saturate(0.8)';
ctx.drawImage(video, 0, 0);
ctx.filter = 'none';第二部分:WebGL 实时视频特效
为什么需要 WebGL
Canvas 2D 的 getImageData + 逐像素处理在 1080P 视频(约 200 万像素 × 4 通道 = 8MB/帧)上,即便是简单的灰度转换也要遍历 800 万次循环,在主线程上执行耗时可能超过 16ms(一帧预算),导致掉帧。WebGL 的片段着色器(Fragment Shader)在 GPU 上并行处理所有像素,同样的操作在 GPU 上只需 1ms 以内。
WebGL 视频渲染管线
将 <video> 作为 WebGL 纹理(Texture)的核心步骤:
class WebGLVideoProcessor {
constructor(canvas) {
this.canvas = canvas;
this.gl = canvas.getContext('webgl2') || canvas.getContext('webgl');
if (!this.gl) throw new Error('WebGL not supported');
this._initShaders();
this._initBuffers();
this._initTexture();
}
_initShaders(fragSource) {
const gl = this.gl;
// 顶点着色器:将视频映射到全屏矩形
const vertSrc = `
attribute vec2 a_position;
attribute vec2 a_texCoord;
varying vec2 v_texCoord;
void main() {
gl_Position = vec4(a_position, 0.0, 1.0);
v_texCoord = a_texCoord;
}
`;
// 默认片段着色器:直接输出纹理颜色(无特效)
const defaultFragSrc = `
precision mediump float;
uniform sampler2D u_texture;
varying vec2 v_texCoord;
void main() {
gl_FragColor = texture2D(u_texture, v_texCoord);
}
`;
this.program = this._createProgram(vertSrc, fragSource || defaultFragSrc);
gl.useProgram(this.program);
}
_initBuffers() {
const gl = this.gl;
// 全屏矩形的顶点坐标(NDC 空间)和纹理坐标
const vertices = new Float32Array([
// x y u v
-1.0, -1.0, 0.0, 1.0, // 左下
1.0, -1.0, 1.0, 1.0, // 右下
-1.0, 1.0, 0.0, 0.0, // 左上
1.0, 1.0, 1.0, 0.0, // 右上
]);
const buf = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);
const stride = 4 * 4; // 4个float,每个4字节
const posLoc = gl.getAttribLocation(this.program, 'a_position');
const texLoc = gl.getAttribLocation(this.program, 'a_texCoord');
gl.enableVertexAttribArray(posLoc);
gl.vertexAttribPointer(posLoc, 2, gl.FLOAT, false, stride, 0);
gl.enableVertexAttribArray(texLoc);
gl.vertexAttribPointer(texLoc, 2, gl.FLOAT, false, stride, 2 * 4);
}
_initTexture() {
const gl = this.gl;
this.texture = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, this.texture);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
}
// 渲染一帧
render(videoOrCanvas) {
const gl = this.gl;
// 将视频当前帧上传为 GPU 纹理
gl.bindTexture(gl.TEXTURE_2D, this.texture);
gl.texImage2D(
gl.TEXTURE_2D, 0, gl.RGBA,
gl.RGBA, gl.UNSIGNED_BYTE,
videoOrCanvas
);
gl.viewport(0, 0, this.canvas.width, this.canvas.height);
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
}
// 更新 uniform 变量(向着色器传参数)
setUniform1f(name, value) {
const loc = this.gl.getUniformLocation(this.program, name);
this.gl.uniform1f(loc, value);
}
setUniform3f(name, r, g, b) {
const loc = this.gl.getUniformLocation(this.program, name);
this.gl.uniform3f(loc, r, g, b);
}
_createProgram(vertSrc, fragSrc) {
const gl = this.gl;
const vert = this._compileShader(gl.VERTEX_SHADER, vertSrc);
const frag = this._compileShader(gl.FRAGMENT_SHADER, fragSrc);
const prog = gl.createProgram();
gl.attachShader(prog, vert);
gl.attachShader(prog, frag);
gl.linkProgram(prog);
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) {
throw new Error('Shader link error: ' + gl.getProgramInfoLog(prog));
}
return prog;
}
_compileShader(type, src) {
const gl = this.gl;
const shader = gl.createShader(type);
gl.shaderSource(shader, src);
gl.compileShader(shader);
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
throw new Error('Shader compile error: ' + gl.getShaderInfoLog(shader));
}
return shader;
}
}绿幕抠图(Chroma Key)着色器
绿幕抠图的核心原理是在 YUV 色彩空间中计算像素与目标绿色的色度距离,距离小于阈值的像素设为透明。在 YUV 空间而非 RGB 空间做比较,是因为 YUV 将亮度(Y)与色度(UV)分离,对光照变化更鲁棒。fenghen.me
/* chroma_key.frag — 绿幕抠图片段着色器 */
precision mediump float;
uniform sampler2D u_texture;
uniform vec3 u_keyColor; // 目标绿色(RGB 归一化,如 vec3(0.0, 1.0, 0.0))
uniform float u_similarity; // 色度相似度阈值(推荐 0.3~0.5)
uniform float u_smoothness; // 边缘平滑度(推荐 0.05~0.15)
uniform float u_spill; // 绿色溢出抑制(推荐 0.05~0.1)
varying vec2 v_texCoord;
/* RGB 转 UV 色度分量 */
vec2 RGBtoUV(vec3 rgb) {
return vec2(
rgb.r * -0.169 + rgb.g * -0.331 + rgb.b * 0.500 + 0.5,
rgb.r * 0.500 + rgb.g * -0.419 + rgb.b * -0.081 + 0.5
);
}
void main() {
vec4 rgba = texture2D(u_texture, v_texCoord);
/* 计算当前像素与绿幕颜色的色度距离 */
vec2 chromaVec = RGBtoUV(rgba.rgb) - RGBtoUV(u_keyColor);
float chromaDist = sqrt(dot(chromaVec, chromaVec));
/* baseMask < 0:绿幕像素;> 0:前景像素 */
float baseMask = chromaDist - u_similarity;
/* 用平滑度参数将 baseMask 映射到 alpha 通道 */
float fullMask = pow(clamp(baseMask / u_smoothness, 0.0, 1.0), 1.5);
rgba.a = fullMask;
/* 绿色溢出抑制:边缘像素的绿色分量替换为灰度值 */
float spillVal = pow(clamp(baseMask / u_spill, 0.0, 1.0), 1.5);
float desat = clamp(rgba.r * 0.2126 + rgba.g * 0.7152 + rgba.b * 0.0722, 0.0, 1.0);
rgba.rgb = mix(vec3(desat), rgba.rgb, spillVal);
gl_FragColor = rgba;
}// 初始化绿幕处理器
const processor = new WebGLVideoProcessor(canvas);
processor._initShaders(CHROMA_KEY_FRAG_SRC);
// 设置绿幕颜色和参数
processor.setUniform3f('u_keyColor', 0.0, 0.8, 0.2); // 目标绿色
processor.setUniform1f('u_similarity', 0.35);
processor.setUniform1f('u_smoothness', 0.08);
processor.setUniform1f('u_spill', 0.08);
// 实时渲染循环
function renderLoop() {
processor.render(videoElement);
requestAnimationFrame(renderLoop);
}
videoElement.addEventListener('play', renderLoop);常用特效着色器集合
/* ── 灰度滤镜 ── */
void main() {
vec4 color = texture2D(u_texture, v_texCoord);
float gray = dot(color.rgb, vec3(0.299, 0.587, 0.114));
gl_FragColor = vec4(gray, gray, gray, color.a);
}
/* ── 复古/棕褐色调(Sepia) ── */
void main() {
vec4 c = texture2D(u_texture, v_texCoord);
gl_FragColor = vec4(
dot(c.rgb, vec3(0.393, 0.769, 0.189)),
dot(c.rgb, vec3(0.349, 0.686, 0.168)),
dot(c.rgb, vec3(0.272, 0.534, 0.131)),
c.a
);
}
/* ── 色相旋转(Hue Rotate) ── */
uniform float u_angle; // 旋转角度(弧度)
vec3 rgb2hsv(vec3 c) { /* ... HSV 转换 ... */ }
vec3 hsv2rgb(vec3 c) { /* ... HSV 转换 ... */ }
void main() {
vec4 color = texture2D(u_texture, v_texCoord);
vec3 hsv = rgb2hsv(color.rgb);
hsv.x = mod(hsv.x + u_angle / (2.0 * 3.14159), 1.0);
gl_FragColor = vec4(hsv2rgb(hsv), color.a);
}
/* ── 像素化(Pixelate) ── */
uniform float u_pixelSize; // 像素块大小(如 8.0)
uniform vec2 u_resolution;
void main() {
vec2 uv = v_texCoord;
vec2 blockSize = u_pixelSize / u_resolution;
vec2 snapped = floor(uv / blockSize) * blockSize + blockSize * 0.5;
gl_FragColor = texture2D(u_texture, snapped);
}第三部分:OffscreenCanvas 与 Worker 卸载
主线程卸载的必要性
WebGL 渲染循环在主线程上运行时,如果 JS 主线程因其他任务繁忙(如 React 渲染、数据处理),渲染帧就会延迟,导致视频特效出现卡顿。OffscreenCanvas + Web Worker 可以将整个渲染管线迁移到独立线程,与主线程完全解耦。cloud.tencent.com
// ── 主线程代码 ──
const canvas = document.getElementById('output-canvas');
// 将 canvas 控制权转移给 Worker(转移后主线程无法再操作该 canvas)
const offscreen = canvas.transferControlToOffscreen();
const worker = new Worker('./video-worker.js');
worker.postMessage(
{ type: 'init', canvas: offscreen },
[offscreen] // 第二个参数:可转移对象列表(零拷贝转移)
);
// 将 video 帧数据发送给 Worker
// 注意:video 元素本身不可转移,需要用 VideoFrame(WebCodecs)或 ImageBitmap
function sendFrame() {
if (!video.paused) {
// createImageBitmap 可在主线程创建可转移的位图
createImageBitmap(video).then(bitmap => {
worker.postMessage({ type: 'frame', bitmap }, [bitmap]);
});
}
requestAnimationFrame(sendFrame);
}
video.addEventListener('play', sendFrame);// ── video-worker.js ──
let processor = null;
self.addEventListener('message', ({ data }) => {
switch (data.type) {
case 'init':
// OffscreenCanvas 在 Worker 中直接使用 WebGL
processor = new WebGLVideoProcessor(data.canvas);
processor._initShaders(CHROMA_KEY_FRAG_SRC);
break;
case 'frame':
// 接收主线程传来的 ImageBitmap,在 Worker 中渲染
if (processor) {
processor.render(data.bitmap);
data.bitmap.close(); // 及时释放
}
break;
case 'setParam':
processor?.setUniform1f(data.name, data.value);
break;
}
});第四部分:WebCodecs 帧级处理
WebCodecs 的定位
WebCodecs 是 W3C 标准化的底层编解码 API,Chrome 94+ 支持。它直接暴露浏览器内置的硬件加速编解码器,绕过了 <video> 元素的高层封装,让开发者能够以帧为单位精确控制解码过程。与 WASM 软解方案相比,WebCodecs 利用设备硬件,性能高出数倍,功耗也更低。developer.chrome.com zhuanlan.zhihu.com
WebCodecs 的核心类有四个:VideoDecoder(视频解码)、VideoEncoder(视频编码)、VideoFrame(原始视频帧)和 ImageDecoder(图像解码,支持 GIF 逐帧解码)。VideoFrame 实现了 CanvasImageSource 接口,可以直接传给 ctx.drawImage() 和 gl.texImage2D()。
VideoDecoder:精确截帧
继续完成第十章剩余部分。
VideoDecoder:精确截帧(续)
// 3. 按时间间隔送入编码数据块
const interval = 1 / fps;
let lastSent = startTime - interval;
for (const chunk of chunks) {
const timeSec = chunk.timestamp / 1_000_000;
// 只送入目标时间范围内的帧
if (timeSec < startTime || timeSec > end) continue;
// 按 fps 间隔过滤(只送关键帧附近的块)
if (timeSec - lastSent < interval && chunk.type !== 'key') continue;
this.decoder.decode(new EncodedVideoChunk({
type: chunk.type, // 'key' | 'delta'
timestamp: chunk.timestamp, // 微秒
data: chunk.data,
}));
lastSent = timeSec;
}
// 刷新解码器,确保所有帧都输出
this.decoder.flush().then(() => {
if (results.length === 0) resolve(results);
});
});
}
// 检测 WebCodecs 支持
static isSupported() {
return 'VideoDecoder' in window && self.isSecureContext;
}
destroy() {
this.decoder?.close();
}
}视频转 GIF
WebCodecs + gif.js 是目前纯前端实现视频转 GIF 最高效的方案。WebCodecs 负责快速解码视频帧(利用硬件加速),gif.js 负责将 Canvas 帧序列编码为 GIF 格式。juejin.cn zhuanlan.zhihu.com
class VideoToGIF {
constructor(options = {}) {
this.options = {
fps: 10, // GIF 帧率
width: 480, // 输出宽度
quality: 10, // gif.js 质量(1-30,越小越好)
workers: 4, // gif.js 并行 Worker 数量
...options
};
}
async convert(videoFile, startTime = 0, duration = 5) {
// 检测支持
if (!VideoToGIF.isSupported()) {
throw new Error('WebCodecs not supported, falling back to canvas method');
}
// 1. 用 WebCodecs 提取帧序列
const extractor = new WebCodecsFrameExtractor();
const frames = await extractor.extractFrames(videoFile, {
fps: this.options.fps,
startTime,
endTime: startTime + duration,
});
extractor.destroy();
// 2. 用 gif.js 编码为 GIF
return this._encodeGIF(frames);
}
_encodeGIF(frames) {
return new Promise((resolve, reject) => {
// gif.js 需要通过 CDN 或本地引入
const gif = new GIF({
workers: this.options.workers,
quality: this.options.quality,
width: this.options.width,
workerScript: '/gif.worker.js',
});
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
const delay = Math.round(1000 / this.options.fps);
// 将每帧 dataURL 绘制到 canvas 后添加到 GIF
const loadFrame = (dataURL) => new Promise(res => {
const img = new Image();
img.onload = () => {
canvas.width = this.options.width;
canvas.height = Math.round(img.height * (this.options.width / img.width));
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
gif.addFrame(canvas, { copy: true, delay });
res();
};
img.src = dataURL;
});
// 串行加载所有帧(保证顺序)
frames.reduce(
(chain, frame) => chain.then(() => loadFrame(frame.dataURL)),
Promise.resolve()
).then(() => {
gif.on('finished', blob => resolve(blob));
gif.on('error', reject);
gif.render();
});
});
}
static isSupported() {
return WebCodecsFrameExtractor.isSupported();
}
}
// 使用示例
const converter = new VideoToGIF({ fps: 12, width: 480 });
const gifBlob = await converter.convert(videoFile, 10, 5); // 从第10秒截取5秒
// 下载 GIF
const url = URL.createObjectURL(gifBlob);
const a = document.createElement('a');
a.href = url;
a.download = 'output.gif';
a.click();
URL.revokeObjectURL(url);VideoEncoder:前端视频录制与处理
VideoEncoder 允许在浏览器中将帧序列编码为 H.264 或 VP8/VP9 等格式的视频流,结合 MediaRecorder 或 MP4 封装库可以实现完整的前端视频录制和导出:
class CanvasVideoRecorder {
constructor(canvas, options = {}) {
this.canvas = canvas;
this.options = {
codec: 'avc1.42E01E', // H.264 Baseline
width: canvas.width,
height: canvas.height,
bitrate: 2_000_000, // 2 Mbps
framerate: 30,
...options
};
this._chunks = [];
this._encoder = null;
this._frameNum = 0;
this._startTime = 0;
}
async start() {
// 检查编码器支持
const { supported } = await VideoEncoder.isConfigSupported({
codec: this.options.codec,
width: this.options.width,
height: this.options.height,
bitrate: this.options.bitrate,
framerate: this.options.framerate,
});
if (!supported) {
// 降级到 VP8
this.options.codec = 'vp8';
}
this._encoder = new VideoEncoder({
output: (chunk, meta) => {
// 收集编码后的数据块
const buffer = new Uint8Array(chunk.byteLength);
chunk.copyTo(buffer);
this._chunks.push({
data: buffer,
timestamp: chunk.timestamp,
type: chunk.type,
// 解码器配置(第一个关键帧附带)
decoderConfig: meta?.decoderConfig,
});
},
error: (e) => console.error('Encoder error:', e),
});
this._encoder.configure({
codec: this.options.codec,
width: this.options.width,
height: this.options.height,
bitrate: this.options.bitrate,
framerate: this.options.framerate,
});
this._frameNum = 0;
this._startTime = performance.now();
this._recording = true;
this._scheduleFrame();
}
_scheduleFrame() {
if (!this._recording) return;
requestAnimationFrame(() => {
this._captureFrame();
this._scheduleFrame();
});
}
_captureFrame() {
if (this._encoder.encodeQueueSize > 5) {
// 编码器积压,丢弃当前帧
return;
}
const timestamp = Math.round(
(performance.now() - this._startTime) * 1000 // 毫秒转微秒
);
// 从 Canvas 创建 VideoFrame
const frame = new VideoFrame(this.canvas, { timestamp });
// 每 30 帧强制一个关键帧
const keyFrame = this._frameNum % 30 === 0;
this._encoder.encode(frame, { keyFrame });
// 必须手动 close,否则内存泄漏
frame.close();
this._frameNum++;
}
async stop() {
this._recording = false;
// 等待所有帧编码完成
await this._encoder.flush();
this._encoder.close();
return this._chunks;
}
}第五部分:requestVideoFrameCallback 与 WebCodecs 协同
精准帧处理管线
requestVideoFrameCallback(rVFC)和 VideoFrame 可以组成一条精准的帧处理管线:rVFC 在每个新视频帧到达合成器时触发,提供帧的精确元数据;将帧绘制到 Canvas 后,可以创建 VideoFrame 对象交给 VideoEncoder 编码,实现实时的"视频处理 → 重新编码"工作流:
class RealtimeVideoProcessor {
constructor(inputVideo, outputCanvas) {
this.video = inputVideo;
this.canvas = outputCanvas;
this.ctx = outputCanvas.getContext('2d');
this.gl = null; // WebGL 上下文(用于特效处理)
this._rvfcId = null;
// 处理管线:函数数组,每帧依次执行
this._pipeline = [];
}
// 添加处理步骤
addStep(fn) {
this._pipeline.push(fn);
return this;
}
start() {
const process = (now, metadata) => {
const {
mediaTime, // 帧在视频时间轴上的位置(秒)
presentedFrames, // 已渲染总帧数
width, height, // 帧的实际尺寸
} = metadata;
// 同步 canvas 尺寸
if (this.canvas.width !== width || this.canvas.height !== height) {
this.canvas.width = width;
this.canvas.height = height;
}
// 执行处理管线
this._pipeline.forEach(step => step(this.video, this.canvas, metadata));
// 继续下一帧
if (!this.video.paused && !this.video.ended) {
this._rvfcId = this.video.requestVideoFrameCallback(process);
}
};
this._rvfcId = this.video.requestVideoFrameCallback(process);
}
stop() {
if (this._rvfcId) {
this.video.cancelVideoFrameCallback(this._rvfcId);
this._rvfcId = null;
}
}
}
// 组装处理管线
const processor = new RealtimeVideoProcessor(videoEl, outputCanvas);
processor
// 步骤1:将视频帧绘制到 Canvas
.addStep((video, canvas) => {
canvas.getContext('2d').drawImage(video, 0, 0);
})
// 步骤2:叠加时间戳水印
.addStep((video, canvas, meta) => {
const ctx = canvas.getContext('2d');
ctx.fillStyle = 'rgba(255,255,255,0.8)';
ctx.font = '14px monospace';
ctx.fillText(
``,
10, 24
);
})
// 步骤3:自定义分析(如帧差检测、运动检测)
.addStep((video, canvas, meta) => {
// 可以在这里做 AI 推理、人脸检测等
runInference(canvas);
});
processor.start();第六部分:完整工具函数库
将本章所有功能整合为一个统一的工具模块,方便在项目中直接引用:
const VideoCanvas = {
// ── 环境检测 ──────────────────────────────────────────────
support: {
get webgl() { const c = document.createElement('canvas'); return !!(c.getContext('webgl2') || c.getContext('webgl')); },
get webcodecs() { return 'VideoDecoder' in window && self.isSecureContext; },
get offscreen() { return 'OffscreenCanvas' in window; },
get rvfc() { return 'requestVideoFrameCallback' in HTMLVideoElement.prototype; },
},
// ── 截图 ──────────────────────────────────────────────────
async screenshot(video, options = {}) {
const { format = 'image/jpeg', quality = 0.92 } = options;
const canvas = Object.assign(document.createElement('canvas'), {
width: video.videoWidth,
height: video.videoHeight,
});
canvas.getContext('2d').drawImage(video, 0, 0);
return new Promise(r => canvas.toBlob(r, format, quality));
},
downloadBlob(blob, filename) {
const url = URL.createObjectURL(blob);
Object.assign(document.createElement('a'), {
href: url, download: filename
}).click();
setTimeout(() => URL.revokeObjectURL(url), 1000);
},
// ── Canvas 滤镜 ───────────────────────────────────────────
filters: {
grayscale: 'grayscale(100%)',
sepia: 'sepia(100%)',
invert: 'invert(100%)',
blur: (px = 4) => `blur()`,
brightness: (v = 1.2) => `brightness()`,
contrast: (v = 1.2) => `contrast()`,
saturate: (v = 1.5) => `saturate()`,
},
drawWithFilter(video, canvas, filter) {
const ctx = canvas.getContext('2d');
ctx.filter = filter;
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
ctx.filter = 'none';
},
// ── 像素操作 ──────────────────────────────────────────────
getPixelData(video) {
const canvas = Object.assign(document.createElement('canvas'), {
width: video.videoWidth,
height: video.videoHeight,
});
const ctx = canvas.getContext('2d', { willReadFrequently: true });
ctx.drawImage(video, 0, 0);
return ctx.getImageData(0, 0, canvas.width, canvas.height);
},
// ── 视频转 Blob(MediaRecorder 方案,兼容性好) ────────────
recordCanvas(canvas, options = {}) {
const {
mimeType = 'video/webm;codecs=vp9',
bitrate = 2_500_000,
duration = 10_000,
} = options;
const stream = canvas.captureStream(30);
const recorder = new MediaRecorder(stream, {
mimeType, videoBitsPerSecond: bitrate
});
const chunks = [];
recorder.ondataavailable = e => chunks.push(e.data);
return new Promise((resolve) => {
recorder.onstop = () => resolve(new Blob(chunks, { type: mimeType }));
recorder.start();
setTimeout(() => recorder.stop(), duration);
});
},
// ── WebCodecs 支持检测 ────────────────────────────────────
async checkCodecSupport(codec = 'avc1.42E01E') {
if (!this.support.webcodecs) return false;
const { supported } = await VideoDecoder.isConfigSupported({
codec,
codedWidth: 1280,
codedHeight: 720,
});
return supported;
},
};
// 使用示例
const blob = await VideoCanvas.screenshot(videoEl);
VideoCanvas.downloadBlob(blob, 'frame.jpg');
VideoCanvas.drawWithFilter(videoEl, canvas, VideoCanvas.filters.sepia);第七部分:技术选型决策树
不同的视频处理需求对应不同的技术方案,选型时需要综合考虑性能、兼容性和实现复杂度:
静态截图:直接用 ctx.drawImage(video) + canvas.toBlob(),零依赖,所有浏览器支持。
实时简单滤镜(灰度、亮度、对比度):优先用 ctx.filter 属性,GPU 加速且无需 WebGL;如需更精细控制才考虑 WebGL。
实时复杂特效(绿幕抠图、色相旋转、自定义算法):必须用 WebGL 片段着色器,否则无法达到实时帧率。如果特效计算量大,配合 OffscreenCanvas + Worker 将渲染从主线程卸载。
精确帧提取/批量截帧:WebCodecs VideoDecoder 是最快方案,比 video.currentTime seek 方案快 5~10 倍。需要 Chrome 94+ / Safari 16.4+,需降级处理。
前端视频录制:MediaRecorder + canvas.captureStream() 兼容性最好(Chrome/Firefox/Safari 均支持);需要精确控制编码参数时用 VideoEncoder,但仅限 Chrome 94+。
视频转 GIF:WebCodecs 解码 + gif.js 编码,纯前端实现,无需服务端。对不支持 WebCodecs 的浏览器可降级为 video.currentTime + Canvas 逐帧截图方案。
至此,第十章《Canvas 与 WebGL 视频处理》完整收尾。本章从 Canvas 2D 的基础视频截图、水印叠加、像素滤镜出发,深入 WebGL 着色器管线实现实时绿幕抠图和各类 GPU 特效,再到 OffscreenCanvas + Worker 的主线程卸载架构,最后覆盖 WebCodecs API 的帧级解码(精确截帧)、视频转 GIF 和 VideoEncoder 前端录制,并以一套统一的工具函数库和技术选型决策树收尾。
第十一章将进入 DRM 与内容保护专题,涵盖 EME(Encrypted Media Extensions)API、Widevine/FairPlay/PlayReady 三大 DRM 方案的接入流程、License 请求与响应处理,以及 DASH + CENC 加密流的完整工程实践。需要继续请告诉我。