第十章:Canvas 与 WebGL 视频处理

<video> 元素只负责解码和渲染,Canvas 和 WebGL 就成了视频帧的"后处理工厂"。从最简单的截图下载、实时水印,到逐像素操作的绿幕抠图,再到 GPU 着色器驱动的实时特效,以及 WebCodecs 带来的帧级精确控制——这一章覆盖 Web 端视频处理的完整技术栈,并在每个层次上给出可落地的工程实现。


第一部分:Canvas 基础视频处理

drawImage 的核心机制

ctx.drawImage(video, x, y) 是一切视频帧处理的起点。它将 <video> 元素当前解码的帧直接绘制到 Canvas 上,整个操作在 GPU 完成,不经过 JS 堆内存。但有一个关键约束:只有视频处于播放或暂停状态(即已解码出至少一帧)时,drawImage 才会绘制出有效内容——在 readyState < 2 时调用,Canvas 上什么都不会出现。

跨域视频需要在 <video> 上设置 crossorigin="anonymous",且服务端必须返回正确的 CORS 响应头,否则 drawImage 后调用 canvas.toDataURL()getImageData() 会抛出 SecurityError


视频截图与下载

截图是最基础的 Canvas 视频处理场景。核心流程是:暂停视频(或在特定时刻)→ drawImage 绘制当前帧 → toBlob 导出 → 通过 <a> 标签触发下载:

class VideoScreenshot {
  constructor(video) {
    this.video  = video;
    this.canvas = document.createElement('canvas');
    this.ctx    = this.canvas.getContext('2d');
  }
 
  // 截取当前帧
  capture(options = {}) {
    const { format = 'image/jpeg', quality = 0.92 } = options;
    const video = this.video;
 
    // 匹配视频原始分辨率
    this.canvas.width  = video.videoWidth;
    this.canvas.height = video.videoHeight;
 
    this.ctx.drawImage(video, 0, 0);
 
    return new Promise((resolve) => {
      this.canvas.toBlob(resolve, format, quality);
    });
  }
 
  // 截图并下载
  async download(filename = `screenshot-`) {
    const blob = await this.capture();
    const url  = URL.createObjectURL(blob);
    const a    = document.createElement('a');
 
    a.href     = url;
    a.download = filename;
    a.click();
 
    // 延迟释放 ObjectURL
    setTimeout(() => URL.revokeObjectURL(url), 1000);
  }
 
  // 截取指定时间点的帧(需要 seek)
  async captureAt(timeSeconds, options = {}) {
    return new Promise((resolve, reject) => {
      const video = this.video;
      const wasPlaying = !video.paused;
 
      // 保存当前时间
      const savedTime = video.currentTime;
 
      const onSeeked = async () => {
        video.removeEventListener('seeked', onSeeked);
 
        const blob = await this.capture(options);
 
        // 恢复原始状态
        video.currentTime = savedTime;
        if (wasPlaying) video.play().catch(() => {});
 
        resolve(blob);
      };
 
      video.addEventListener('seeked', onSeeked, { once: true });
      video.currentTime = timeSeconds;
    });
  }
}
 
// 使用
const screenshot = new VideoScreenshot(videoElement);
document.getElementById('capture-btn').addEventListener('click', () => {
  screenshot.download();
});

实时水印叠加

水印有两种实现策略:Canvas 叠加渲染(将视频帧和水印同时绘制到 Canvas,输出合成图像)和 CSS 覆盖(用绝对定位的 HTML 元素覆盖在 <video> 上)。CSS 方案实现简单但无法嵌入到导出的视频帧中;Canvas 方案可以将水印真正"烧录"到每一帧。

class VideoWatermark {
  constructor(video, canvas, config = {}) {
    this.video  = video;
    this.canvas = canvas;
    this.ctx    = canvas.getContext('2d');
    this.config = {
      text:       '© 版权所有',
      font:       '16px Arial',
      color:      'rgba(255,255,255,0.6)',
      position:   'bottom-right', // top-left | top-right | bottom-left | bottom-right | center
      padding:    16,
      logo:       null,           // HTMLImageElement,可选
      ...config
    };
    this._rafId = null;
  }
 
  start() {
    this.canvas.width  = this.video.videoWidth  || 1280;
    this.canvas.height = this.video.videoHeight || 720;
 
    const render = () => {
      if (this.video.paused || this.video.ended) {
        this._rafId = null;
        return;
      }
      this._drawFrame();
      this._rafId = requestAnimationFrame(render);
    };
 
    this.video.addEventListener('play', () => {
      if (!this._rafId) render();
    });
 
    // 如果视频已在播放
    if (!this.video.paused) render();
  }
 
  _drawFrame() {
    const { canvas, ctx, video, config } = this;
 
    // 1. 绘制视频帧
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
 
    // 2. 叠加文字水印
    ctx.font      = config.font;
    ctx.fillStyle = config.color;
    ctx.textBaseline = 'bottom';
 
    const metrics  = ctx.measureText(config.text);
    const textW    = metrics.width;
    const textH    = parseInt(config.font);  // 近似行高
    const pad      = config.padding;
 
    let x, y;
    switch (config.position) {
      case 'top-left':
        x = pad; y = textH + pad; break;
      case 'top-right':
        x = canvas.width - textW - pad; y = textH + pad; break;
      case 'bottom-left':
        x = pad; y = canvas.height - pad; break;
      case 'bottom-right':
        x = canvas.width - textW - pad; y = canvas.height - pad; break;
      case 'center':
        x = (canvas.width - textW) / 2; y = (canvas.height + textH) / 2; break;
      default:
        x = pad; y = canvas.height - pad;
    }
 
    ctx.fillText(config.text, x, y);
 
    // 3. 叠加 Logo(可选)
    if (config.logo) {
      const logoH = 32;
      const logoW = config.logo.width * (logoH / config.logo.height);
      ctx.globalAlpha = 0.7;
      ctx.drawImage(config.logo, x - logoW - 8, y - logoH, logoW, logoH);
      ctx.globalAlpha = 1;
    }
  }
 
  stop() {
    if (this._rafId) {
      cancelAnimationFrame(this._rafId);
      this._rafId = null;
    }
  }
}

Canvas 滤镜效果

通过 getImageData 获取像素数据,逐像素修改后写回,可以实现任意自定义滤镜。但这是纯 CPU 操作,对高分辨率视频性能较差,适合静态截图处理;实时视频滤镜应使用 WebGL(见第三部分)。

class CanvasFilter {
  constructor(canvas) {
    this.ctx = canvas.getContext('2d', { willReadFrequently: true });
    // willReadFrequently: true 告知浏览器会频繁读取像素,优化内部缓存策略
  }
 
  // 灰度滤镜
  grayscale(imageData) {
    const data = imageData.data;
    for (let i = 0; i < data.length; i += 4) {
      const avg = (data[i] * 0.299 + data[i+1] * 0.587 + data[i+2] * 0.114);
      data[i] = data[i+1] = data[i+2] = avg;
    }
    return imageData;
  }
 
  // 亮度/对比度调整
  brightnessContrast(imageData, brightness = 0, contrast = 0) {
    const data   = imageData.data;
    const factor = (259 * (contrast + 255)) / (255 * (259 - contrast));
 
    for (let i = 0; i < data.length; i += 4) {
      data[i]   = clamp(factor * (data[i]   - 128) + 128 + brightness);
      data[i+1] = clamp(factor * (data[i+1] - 128) + 128 + brightness);
      data[i+2] = clamp(factor * (data[i+2] - 128) + 128 + brightness);
    }
    return imageData;
  }
 
  // 模糊(简单盒式模糊,生产环境建议用 CSS filter 或 WebGL)
  applyFilter(video, filterFn) {
    const canvas = this.ctx.canvas;
    this.ctx.drawImage(video, 0, 0);
    const imageData = this.ctx.getImageData(0, 0, canvas.width, canvas.height);
    const filtered  = filterFn(imageData);
    this.ctx.putImageData(filtered, 0, 0);
  }
}
 
function clamp(v) { return Math.max(0, Math.min(255, v)); }

更高效的方案是直接使用 CSS filter 属性作用于 Canvas 上下文——Chrome 支持在 drawImage 之前设置 ctx.filter,从而让 GPU 完成滤镜运算:

// CSS filter 直接作用于 Canvas 绘制(无需 getImageData)
ctx.filter = 'grayscale(100%)';
ctx.drawImage(video, 0, 0);
ctx.filter = 'none';
 
// 多个滤镜叠加
ctx.filter = 'brightness(1.2) contrast(1.1) saturate(0.8)';
ctx.drawImage(video, 0, 0);
ctx.filter = 'none';

第二部分:WebGL 实时视频特效

为什么需要 WebGL

Canvas 2D 的 getImageData + 逐像素处理在 1080P 视频(约 200 万像素 × 4 通道 = 8MB/帧)上,即便是简单的灰度转换也要遍历 800 万次循环,在主线程上执行耗时可能超过 16ms(一帧预算),导致掉帧。WebGL 的片段着色器(Fragment Shader)在 GPU 上并行处理所有像素,同样的操作在 GPU 上只需 1ms 以内。


WebGL 视频渲染管线

&lt;video&gt; 作为 WebGL 纹理(Texture)的核心步骤:

class WebGLVideoProcessor {
  constructor(canvas) {
    this.canvas = canvas;
    this.gl     = canvas.getContext('webgl2') || canvas.getContext('webgl');
    if (!this.gl) throw new Error('WebGL not supported');
 
    this._initShaders();
    this._initBuffers();
    this._initTexture();
  }
 
  _initShaders(fragSource) {
    const gl = this.gl;
 
    // 顶点着色器:将视频映射到全屏矩形
    const vertSrc = `
      attribute vec2 a_position;
      attribute vec2 a_texCoord;
      varying   vec2 v_texCoord;
      void main() {
        gl_Position = vec4(a_position, 0.0, 1.0);
        v_texCoord  = a_texCoord;
      }
    `;
 
    // 默认片段着色器:直接输出纹理颜色(无特效)
    const defaultFragSrc = `
      precision mediump float;
      uniform sampler2D u_texture;
      varying vec2      v_texCoord;
      void main() {
        gl_FragColor = texture2D(u_texture, v_texCoord);
      }
    `;
 
    this.program = this._createProgram(vertSrc, fragSource || defaultFragSrc);
    gl.useProgram(this.program);
  }
 
  _initBuffers() {
    const gl = this.gl;
 
    // 全屏矩形的顶点坐标(NDC 空间)和纹理坐标
    const vertices = new Float32Array([
    //  x      y      u     v
      -1.0,  -1.0,   0.0,  1.0,   // 左下
       1.0,  -1.0,   1.0,  1.0,   // 右下
      -1.0,   1.0,   0.0,  0.0,   // 左上
       1.0,   1.0,   1.0,  0.0,   // 右上
    ]);
 
    const buf = gl.createBuffer();
    gl.bindBuffer(gl.ARRAY_BUFFER, buf);
    gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);
 
    const stride = 4 * 4; // 4个float,每个4字节
    const posLoc = gl.getAttribLocation(this.program, 'a_position');
    const texLoc = gl.getAttribLocation(this.program, 'a_texCoord');
 
    gl.enableVertexAttribArray(posLoc);
    gl.vertexAttribPointer(posLoc, 2, gl.FLOAT, false, stride, 0);
 
    gl.enableVertexAttribArray(texLoc);
    gl.vertexAttribPointer(texLoc, 2, gl.FLOAT, false, stride, 2 * 4);
  }
 
  _initTexture() {
    const gl      = this.gl;
    this.texture  = gl.createTexture();
 
    gl.bindTexture(gl.TEXTURE_2D, this.texture);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S,     gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T,     gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
  }
 
  // 渲染一帧
  render(videoOrCanvas) {
    const gl = this.gl;
 
    // 将视频当前帧上传为 GPU 纹理
    gl.bindTexture(gl.TEXTURE_2D, this.texture);
    gl.texImage2D(
      gl.TEXTURE_2D, 0, gl.RGBA,
      gl.RGBA, gl.UNSIGNED_BYTE,
      videoOrCanvas
    );
 
    gl.viewport(0, 0, this.canvas.width, this.canvas.height);
    gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
  }
 
  // 更新 uniform 变量(向着色器传参数)
  setUniform1f(name, value) {
    const loc = this.gl.getUniformLocation(this.program, name);
    this.gl.uniform1f(loc, value);
  }
 
  setUniform3f(name, r, g, b) {
    const loc = this.gl.getUniformLocation(this.program, name);
    this.gl.uniform3f(loc, r, g, b);
  }
 
  _createProgram(vertSrc, fragSrc) {
    const gl   = this.gl;
    const vert = this._compileShader(gl.VERTEX_SHADER,   vertSrc);
    const frag = this._compileShader(gl.FRAGMENT_SHADER, fragSrc);
    const prog = gl.createProgram();
 
    gl.attachShader(prog, vert);
    gl.attachShader(prog, frag);
    gl.linkProgram(prog);
 
    if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) {
      throw new Error('Shader link error: ' + gl.getProgramInfoLog(prog));
    }
    return prog;
  }
 
  _compileShader(type, src) {
    const gl     = this.gl;
    const shader = gl.createShader(type);
    gl.shaderSource(shader, src);
    gl.compileShader(shader);
 
    if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
      throw new Error('Shader compile error: ' + gl.getShaderInfoLog(shader));
    }
    return shader;
  }
}

绿幕抠图(Chroma Key)着色器

绿幕抠图的核心原理是在 YUV 色彩空间中计算像素与目标绿色的色度距离,距离小于阈值的像素设为透明。在 YUV 空间而非 RGB 空间做比较,是因为 YUV 将亮度(Y)与色度(UV)分离,对光照变化更鲁棒。fenghen.me

/* chroma_key.frag — 绿幕抠图片段着色器 */
precision mediump float;
 
uniform sampler2D u_texture;
uniform vec3      u_keyColor;    // 目标绿色(RGB 归一化,如 vec3(0.0, 1.0, 0.0))
uniform float     u_similarity;  // 色度相似度阈值(推荐 0.3~0.5)
uniform float     u_smoothness;  // 边缘平滑度(推荐 0.05~0.15)
uniform float     u_spill;       // 绿色溢出抑制(推荐 0.05~0.1)
 
varying vec2 v_texCoord;
 
/* RGB 转 UV 色度分量 */
vec2 RGBtoUV(vec3 rgb) {
  return vec2(
    rgb.r * -0.169 + rgb.g * -0.331 + rgb.b *  0.500 + 0.5,
    rgb.r *  0.500 + rgb.g * -0.419 + rgb.b * -0.081 + 0.5
  );
}
 
void main() {
  vec4  rgba       = texture2D(u_texture, v_texCoord);
 
  /* 计算当前像素与绿幕颜色的色度距离 */
  vec2  chromaVec  = RGBtoUV(rgba.rgb) - RGBtoUV(u_keyColor);
  float chromaDist = sqrt(dot(chromaVec, chromaVec));
 
  /* baseMask < 0:绿幕像素;> 0:前景像素 */
  float baseMask   = chromaDist - u_similarity;
 
  /* 用平滑度参数将 baseMask 映射到 alpha 通道 */
  float fullMask   = pow(clamp(baseMask / u_smoothness, 0.0, 1.0), 1.5);
  rgba.a           = fullMask;
 
  /* 绿色溢出抑制:边缘像素的绿色分量替换为灰度值 */
  float spillVal   = pow(clamp(baseMask / u_spill, 0.0, 1.0), 1.5);
  float desat      = clamp(rgba.r * 0.2126 + rgba.g * 0.7152 + rgba.b * 0.0722, 0.0, 1.0);
  rgba.rgb         = mix(vec3(desat), rgba.rgb, spillVal);
 
  gl_FragColor = rgba;
}
// 初始化绿幕处理器
const processor = new WebGLVideoProcessor(canvas);
processor._initShaders(CHROMA_KEY_FRAG_SRC);
 
// 设置绿幕颜色和参数
processor.setUniform3f('u_keyColor',   0.0, 0.8, 0.2); // 目标绿色
processor.setUniform1f('u_similarity', 0.35);
processor.setUniform1f('u_smoothness', 0.08);
processor.setUniform1f('u_spill',      0.08);
 
// 实时渲染循环
function renderLoop() {
  processor.render(videoElement);
  requestAnimationFrame(renderLoop);
}
videoElement.addEventListener('play', renderLoop);

常用特效着色器集合

/* ── 灰度滤镜 ── */
void main() {
  vec4  color = texture2D(u_texture, v_texCoord);
  float gray  = dot(color.rgb, vec3(0.299, 0.587, 0.114));
  gl_FragColor = vec4(gray, gray, gray, color.a);
}
 
/* ── 复古/棕褐色调(Sepia) ── */
void main() {
  vec4 c = texture2D(u_texture, v_texCoord);
  gl_FragColor = vec4(
    dot(c.rgb, vec3(0.393, 0.769, 0.189)),
    dot(c.rgb, vec3(0.349, 0.686, 0.168)),
    dot(c.rgb, vec3(0.272, 0.534, 0.131)),
    c.a
  );
}
 
/* ── 色相旋转(Hue Rotate) ── */
uniform float u_angle; // 旋转角度(弧度)
vec3 rgb2hsv(vec3 c) { /* ... HSV 转换 ... */ }
vec3 hsv2rgb(vec3 c) { /* ... HSV 转换 ... */ }
 
void main() {
  vec4 color  = texture2D(u_texture, v_texCoord);
  vec3 hsv    = rgb2hsv(color.rgb);
  hsv.x       = mod(hsv.x + u_angle / (2.0 * 3.14159), 1.0);
  gl_FragColor = vec4(hsv2rgb(hsv), color.a);
}
 
/* ── 像素化(Pixelate) ── */
uniform float u_pixelSize; // 像素块大小(如 8.0)
uniform vec2  u_resolution;
 
void main() {
  vec2 uv       = v_texCoord;
  vec2 blockSize = u_pixelSize / u_resolution;
  vec2 snapped  = floor(uv / blockSize) * blockSize + blockSize * 0.5;
  gl_FragColor  = texture2D(u_texture, snapped);
}

第三部分:OffscreenCanvas 与 Worker 卸载

主线程卸载的必要性

WebGL 渲染循环在主线程上运行时,如果 JS 主线程因其他任务繁忙(如 React 渲染、数据处理),渲染帧就会延迟,导致视频特效出现卡顿。OffscreenCanvas + Web Worker 可以将整个渲染管线迁移到独立线程,与主线程完全解耦。cloud.tencent.com

// ── 主线程代码 ──
const canvas = document.getElementById('output-canvas');
 
// 将 canvas 控制权转移给 Worker(转移后主线程无法再操作该 canvas)
const offscreen = canvas.transferControlToOffscreen();
const worker    = new Worker('./video-worker.js');
 
worker.postMessage(
  { type: 'init', canvas: offscreen },
  [offscreen]  // 第二个参数:可转移对象列表(零拷贝转移)
);
 
// 将 video 帧数据发送给 Worker
// 注意:video 元素本身不可转移,需要用 VideoFrame(WebCodecs)或 ImageBitmap
function sendFrame() {
  if (!video.paused) {
    // createImageBitmap 可在主线程创建可转移的位图
    createImageBitmap(video).then(bitmap => {
      worker.postMessage({ type: 'frame', bitmap }, [bitmap]);
    });
  }
  requestAnimationFrame(sendFrame);
}
 
video.addEventListener('play', sendFrame);
// ── video-worker.js ──
let processor = null;
 
self.addEventListener('message', ({ data }) => {
  switch (data.type) {
    case 'init':
      // OffscreenCanvas 在 Worker 中直接使用 WebGL
      processor = new WebGLVideoProcessor(data.canvas);
      processor._initShaders(CHROMA_KEY_FRAG_SRC);
      break;
 
    case 'frame':
      // 接收主线程传来的 ImageBitmap,在 Worker 中渲染
      if (processor) {
        processor.render(data.bitmap);
        data.bitmap.close(); // 及时释放
      }
      break;
 
    case 'setParam':
      processor?.setUniform1f(data.name, data.value);
      break;
  }
});

第四部分:WebCodecs 帧级处理

WebCodecs 的定位

WebCodecs 是 W3C 标准化的底层编解码 API,Chrome 94+ 支持。它直接暴露浏览器内置的硬件加速编解码器,绕过了 &lt;video&gt; 元素的高层封装,让开发者能够以帧为单位精确控制解码过程。与 WASM 软解方案相比,WebCodecs 利用设备硬件,性能高出数倍,功耗也更低。developer.chrome.com zhuanlan.zhihu.com

WebCodecs 的核心类有四个:VideoDecoder(视频解码)、VideoEncoder(视频编码)、VideoFrame(原始视频帧)和 ImageDecoder(图像解码,支持 GIF 逐帧解码)。VideoFrame 实现了 CanvasImageSource 接口,可以直接传给 ctx.drawImage()gl.texImage2D()


VideoDecoder:精确截帧

继续完成第十章剩余部分。


VideoDecoder:精确截帧(续)

      // 3. 按时间间隔送入编码数据块
      const interval = 1 / fps;
      let   lastSent = startTime - interval;
 
      for (const chunk of chunks) {
        const timeSec = chunk.timestamp / 1_000_000;
 
        // 只送入目标时间范围内的帧
        if (timeSec < startTime || timeSec > end) continue;
 
        // 按 fps 间隔过滤(只送关键帧附近的块)
        if (timeSec - lastSent < interval && chunk.type !== 'key') continue;
 
        this.decoder.decode(new EncodedVideoChunk({
          type:      chunk.type,      // 'key' | 'delta'
          timestamp: chunk.timestamp, // 微秒
          data:      chunk.data,
        }));
 
        lastSent = timeSec;
      }
 
      // 刷新解码器,确保所有帧都输出
      this.decoder.flush().then(() => {
        if (results.length === 0) resolve(results);
      });
    });
  }
 
  // 检测 WebCodecs 支持
  static isSupported() {
    return 'VideoDecoder' in window && self.isSecureContext;
  }
 
  destroy() {
    this.decoder?.close();
  }
}

视频转 GIF

WebCodecs + gif.js 是目前纯前端实现视频转 GIF 最高效的方案。WebCodecs 负责快速解码视频帧(利用硬件加速),gif.js 负责将 Canvas 帧序列编码为 GIF 格式。juejin.cn zhuanlan.zhihu.com

class VideoToGIF {
  constructor(options = {}) {
    this.options = {
      fps:      10,    // GIF 帧率
      width:    480,   // 输出宽度
      quality:  10,    // gif.js 质量(1-30,越小越好)
      workers:  4,     // gif.js 并行 Worker 数量
      ...options
    };
  }
 
  async convert(videoFile, startTime = 0, duration = 5) {
    // 检测支持
    if (!VideoToGIF.isSupported()) {
      throw new Error('WebCodecs not supported, falling back to canvas method');
    }
 
    // 1. 用 WebCodecs 提取帧序列
    const extractor = new WebCodecsFrameExtractor();
    const frames    = await extractor.extractFrames(videoFile, {
      fps:       this.options.fps,
      startTime,
      endTime:   startTime + duration,
    });
    extractor.destroy();
 
    // 2. 用 gif.js 编码为 GIF
    return this._encodeGIF(frames);
  }
 
  _encodeGIF(frames) {
    return new Promise((resolve, reject) => {
      // gif.js 需要通过 CDN 或本地引入
      const gif = new GIF({
        workers:   this.options.workers,
        quality:   this.options.quality,
        width:     this.options.width,
        workerScript: '/gif.worker.js',
      });
 
      const canvas = document.createElement('canvas');
      const ctx    = canvas.getContext('2d');
      const delay  = Math.round(1000 / this.options.fps);
 
      // 将每帧 dataURL 绘制到 canvas 后添加到 GIF
      const loadFrame = (dataURL) => new Promise(res => {
        const img  = new Image();
        img.onload = () => {
          canvas.width  = this.options.width;
          canvas.height = Math.round(img.height * (this.options.width / img.width));
          ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
          gif.addFrame(canvas, { copy: true, delay });
          res();
        };
        img.src = dataURL;
      });
 
      // 串行加载所有帧(保证顺序)
      frames.reduce(
        (chain, frame) => chain.then(() => loadFrame(frame.dataURL)),
        Promise.resolve()
      ).then(() => {
        gif.on('finished', blob => resolve(blob));
        gif.on('error',    reject);
        gif.render();
      });
    });
  }
 
  static isSupported() {
    return WebCodecsFrameExtractor.isSupported();
  }
}
 
// 使用示例
const converter = new VideoToGIF({ fps: 12, width: 480 });
const gifBlob   = await converter.convert(videoFile, 10, 5); // 从第10秒截取5秒
 
// 下载 GIF
const url = URL.createObjectURL(gifBlob);
const a   = document.createElement('a');
a.href    = url;
a.download = 'output.gif';
a.click();
URL.revokeObjectURL(url);

VideoEncoder:前端视频录制与处理

VideoEncoder 允许在浏览器中将帧序列编码为 H.264 或 VP8/VP9 等格式的视频流,结合 MediaRecorder 或 MP4 封装库可以实现完整的前端视频录制和导出:

class CanvasVideoRecorder {
  constructor(canvas, options = {}) {
    this.canvas  = canvas;
    this.options = {
      codec:     'avc1.42E01E', // H.264 Baseline
      width:     canvas.width,
      height:    canvas.height,
      bitrate:   2_000_000,     // 2 Mbps
      framerate: 30,
      ...options
    };
 
    this._chunks   = [];
    this._encoder  = null;
    this._frameNum = 0;
    this._startTime = 0;
  }
 
  async start() {
    // 检查编码器支持
    const { supported } = await VideoEncoder.isConfigSupported({
      codec:     this.options.codec,
      width:     this.options.width,
      height:    this.options.height,
      bitrate:   this.options.bitrate,
      framerate: this.options.framerate,
    });
 
    if (!supported) {
      // 降级到 VP8
      this.options.codec = 'vp8';
    }
 
    this._encoder = new VideoEncoder({
      output: (chunk, meta) => {
        // 收集编码后的数据块
        const buffer = new Uint8Array(chunk.byteLength);
        chunk.copyTo(buffer);
        this._chunks.push({
          data:      buffer,
          timestamp: chunk.timestamp,
          type:      chunk.type,
          // 解码器配置(第一个关键帧附带)
          decoderConfig: meta?.decoderConfig,
        });
      },
      error: (e) => console.error('Encoder error:', e),
    });
 
    this._encoder.configure({
      codec:     this.options.codec,
      width:     this.options.width,
      height:    this.options.height,
      bitrate:   this.options.bitrate,
      framerate: this.options.framerate,
    });
 
    this._frameNum  = 0;
    this._startTime = performance.now();
    this._recording = true;
    this._scheduleFrame();
  }
 
  _scheduleFrame() {
    if (!this._recording) return;
 
    requestAnimationFrame(() => {
      this._captureFrame();
      this._scheduleFrame();
    });
  }
 
  _captureFrame() {
    if (this._encoder.encodeQueueSize > 5) {
      // 编码器积压,丢弃当前帧
      return;
    }
 
    const timestamp = Math.round(
      (performance.now() - this._startTime) * 1000  // 毫秒转微秒
    );
 
    // 从 Canvas 创建 VideoFrame
    const frame = new VideoFrame(this.canvas, { timestamp });
 
    // 每 30 帧强制一个关键帧
    const keyFrame = this._frameNum % 30 === 0;
    this._encoder.encode(frame, { keyFrame });
 
    // 必须手动 close,否则内存泄漏
    frame.close();
    this._frameNum++;
  }
 
  async stop() {
    this._recording = false;
 
    // 等待所有帧编码完成
    await this._encoder.flush();
    this._encoder.close();
 
    return this._chunks;
  }
}

第五部分:requestVideoFrameCallback 与 WebCodecs 协同

精准帧处理管线

requestVideoFrameCallback(rVFC)和 VideoFrame 可以组成一条精准的帧处理管线:rVFC 在每个新视频帧到达合成器时触发,提供帧的精确元数据;将帧绘制到 Canvas 后,可以创建 VideoFrame 对象交给 VideoEncoder 编码,实现实时的"视频处理 → 重新编码"工作流:

class RealtimeVideoProcessor {
  constructor(inputVideo, outputCanvas) {
    this.video   = inputVideo;
    this.canvas  = outputCanvas;
    this.ctx     = outputCanvas.getContext('2d');
    this.gl      = null;   // WebGL 上下文(用于特效处理)
    this._rvfcId = null;
 
    // 处理管线:函数数组,每帧依次执行
    this._pipeline = [];
  }
 
  // 添加处理步骤
  addStep(fn) {
    this._pipeline.push(fn);
    return this;
  }
 
  start() {
    const process = (now, metadata) => {
      const {
        mediaTime,        // 帧在视频时间轴上的位置(秒)
        presentedFrames,  // 已渲染总帧数
        width, height,    // 帧的实际尺寸
      } = metadata;
 
      // 同步 canvas 尺寸
      if (this.canvas.width !== width || this.canvas.height !== height) {
        this.canvas.width  = width;
        this.canvas.height = height;
      }
 
      // 执行处理管线
      this._pipeline.forEach(step => step(this.video, this.canvas, metadata));
 
      // 继续下一帧
      if (!this.video.paused && !this.video.ended) {
        this._rvfcId = this.video.requestVideoFrameCallback(process);
      }
    };
 
    this._rvfcId = this.video.requestVideoFrameCallback(process);
  }
 
  stop() {
    if (this._rvfcId) {
      this.video.cancelVideoFrameCallback(this._rvfcId);
      this._rvfcId = null;
    }
  }
}
 
// 组装处理管线
const processor = new RealtimeVideoProcessor(videoEl, outputCanvas);
 
processor
  // 步骤1:将视频帧绘制到 Canvas
  .addStep((video, canvas) => {
    canvas.getContext('2d').drawImage(video, 0, 0);
  })
  // 步骤2:叠加时间戳水印
  .addStep((video, canvas, meta) => {
    const ctx = canvas.getContext('2d');
    ctx.fillStyle = 'rgba(255,255,255,0.8)';
    ctx.font      = '14px monospace';
    ctx.fillText(
      ``,
      10, 24
    );
  })
  // 步骤3:自定义分析(如帧差检测、运动检测)
  .addStep((video, canvas, meta) => {
    // 可以在这里做 AI 推理、人脸检测等
    runInference(canvas);
  });
 
processor.start();

第六部分:完整工具函数库

将本章所有功能整合为一个统一的工具模块,方便在项目中直接引用:

const VideoCanvas = {
  // ── 环境检测 ──────────────────────────────────────────────
 
  support: {
    get webgl()      { const c = document.createElement('canvas'); return !!(c.getContext('webgl2') || c.getContext('webgl')); },
    get webcodecs()  { return 'VideoDecoder' in window && self.isSecureContext; },
    get offscreen()  { return 'OffscreenCanvas' in window; },
    get rvfc()       { return 'requestVideoFrameCallback' in HTMLVideoElement.prototype; },
  },
 
  // ── 截图 ──────────────────────────────────────────────────
 
  async screenshot(video, options = {}) {
    const { format = 'image/jpeg', quality = 0.92 } = options;
    const canvas = Object.assign(document.createElement('canvas'), {
      width:  video.videoWidth,
      height: video.videoHeight,
    });
    canvas.getContext('2d').drawImage(video, 0, 0);
    return new Promise(r => canvas.toBlob(r, format, quality));
  },
 
  downloadBlob(blob, filename) {
    const url = URL.createObjectURL(blob);
    Object.assign(document.createElement('a'), {
      href: url, download: filename
    }).click();
    setTimeout(() => URL.revokeObjectURL(url), 1000);
  },
 
  // ── Canvas 滤镜 ───────────────────────────────────────────
 
  filters: {
    grayscale:   'grayscale(100%)',
    sepia:       'sepia(100%)',
    invert:      'invert(100%)',
    blur:        (px = 4) => `blur()`,
    brightness:  (v = 1.2) => `brightness()`,
    contrast:    (v = 1.2) => `contrast()`,
    saturate:    (v = 1.5) => `saturate()`,
  },
 
  drawWithFilter(video, canvas, filter) {
    const ctx  = canvas.getContext('2d');
    ctx.filter = filter;
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    ctx.filter = 'none';
  },
 
  // ── 像素操作 ──────────────────────────────────────────────
 
  getPixelData(video) {
    const canvas = Object.assign(document.createElement('canvas'), {
      width:  video.videoWidth,
      height: video.videoHeight,
    });
    const ctx = canvas.getContext('2d', { willReadFrequently: true });
    ctx.drawImage(video, 0, 0);
    return ctx.getImageData(0, 0, canvas.width, canvas.height);
  },
 
  // ── 视频转 Blob(MediaRecorder 方案,兼容性好) ────────────
 
  recordCanvas(canvas, options = {}) {
    const {
      mimeType = 'video/webm;codecs=vp9',
      bitrate  = 2_500_000,
      duration = 10_000,
    } = options;
 
    const stream   = canvas.captureStream(30);
    const recorder = new MediaRecorder(stream, {
      mimeType, videoBitsPerSecond: bitrate
    });
    const chunks   = [];
 
    recorder.ondataavailable = e => chunks.push(e.data);
 
    return new Promise((resolve) => {
      recorder.onstop = () => resolve(new Blob(chunks, { type: mimeType }));
      recorder.start();
      setTimeout(() => recorder.stop(), duration);
    });
  },
 
  // ── WebCodecs 支持检测 ────────────────────────────────────
 
  async checkCodecSupport(codec = 'avc1.42E01E') {
    if (!this.support.webcodecs) return false;
    const { supported } = await VideoDecoder.isConfigSupported({
      codec,
      codedWidth:  1280,
      codedHeight: 720,
    });
    return supported;
  },
};
 
// 使用示例
const blob = await VideoCanvas.screenshot(videoEl);
VideoCanvas.downloadBlob(blob, 'frame.jpg');
 
VideoCanvas.drawWithFilter(videoEl, canvas, VideoCanvas.filters.sepia);

第七部分:技术选型决策树

不同的视频处理需求对应不同的技术方案,选型时需要综合考虑性能、兼容性和实现复杂度:

静态截图:直接用 ctx.drawImage(video) + canvas.toBlob(),零依赖,所有浏览器支持。

实时简单滤镜(灰度、亮度、对比度):优先用 ctx.filter 属性,GPU 加速且无需 WebGL;如需更精细控制才考虑 WebGL。

实时复杂特效(绿幕抠图、色相旋转、自定义算法):必须用 WebGL 片段着色器,否则无法达到实时帧率。如果特效计算量大,配合 OffscreenCanvas + Worker 将渲染从主线程卸载。

精确帧提取/批量截帧:WebCodecs VideoDecoder 是最快方案,比 video.currentTime seek 方案快 5~10 倍。需要 Chrome 94+ / Safari 16.4+,需降级处理。

前端视频录制MediaRecorder + canvas.captureStream() 兼容性最好(Chrome/Firefox/Safari 均支持);需要精确控制编码参数时用 VideoEncoder,但仅限 Chrome 94+。

视频转 GIF:WebCodecs 解码 + gif.js 编码,纯前端实现,无需服务端。对不支持 WebCodecs 的浏览器可降级为 video.currentTime + Canvas 逐帧截图方案。


至此,第十章《Canvas 与 WebGL 视频处理》完整收尾。本章从 Canvas 2D 的基础视频截图、水印叠加、像素滤镜出发,深入 WebGL 着色器管线实现实时绿幕抠图和各类 GPU 特效,再到 OffscreenCanvas + Worker 的主线程卸载架构,最后覆盖 WebCodecs API 的帧级解码(精确截帧)、视频转 GIF 和 VideoEncoder 前端录制,并以一套统一的工具函数库和技术选型决策树收尾。

第十一章将进入 DRM 与内容保护专题,涵盖 EME(Encrypted Media Extensions)API、Widevine/FairPlay/PlayReady 三大 DRM 方案的接入流程、License 请求与响应处理,以及 DASH + CENC 加密流的完整工程实践。需要继续请告诉我。