Commit b2f8ebcd27 for ffmpeg

commit b2f8ebcd2733ce595a99956050a9322457d9d2d4
Author: Jake Fineman <jake@wave.online>
Date:   Fri Aug 7 17:57:03 2026 -0400

    avfilter/dnn: reject channel-mismatched model output in dnn_processing

    ff_proc_from_dnn_to_frame()'s AV_PIX_FMT_RGB24/BGR24 case assumes the
    model's output has exactly 3 channels, but never checks it.

    The first sws_scale() in that case reads from output->data with a
    hardcoded source stride of frame->width * 3 * src_datatype_size over
    frame->height rows, i.e. frame->width * 3 * frame->height *
    src_datatype_size bytes. For a model whose output has fewer channels
    than that -- e.g. the single-channel alpha matte an RGB-in
    background-removal model produces -- the real tensor only holds
    frame->width * frame->height * channels * src_datatype_size bytes, so
    the call reads roughly 3x past the end of it. This affects both
    DL_NCHW and DL_NHWC output, since it happens before the layout-specific
    planar-to-packed branch.

    For DL_NCHW there is a second, independent overflow on the write side:
    middle_data is allocated plane_size * output->dims[1] bytes, but
    linesize[0] is set to frame->width * 3 unconditionally, so the same
    sws_scale() also writes past that allocation. This reproduces as a
    SIGABRT (glibc "free(): invalid pointer", or a malloc.c assertion,
    depending on allocator state) inside dnn_processing, after inference
    has already completed successfully in the backend.

    The input side is guarded -- check_modelinput_inlink() in
    vf_dnn_processing.c -- but the output side is not. dnn_processing only
    supports channel-preserving models for these pixel formats by design,
    so reject a mismatched channel count with a clear error instead of
    reading and writing out of bounds. The channel dimension is resolved
    via dnn_get_channel_idx_by_layout() so the check covers both layouts.

    This does not add support for rendering a 1- or 2-channel model output
    through this filter; that would need a different pixel-format target
    and is left as future work.

    Signed-off-by: Jake Fineman <jake@wave.online>

diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 6b79336766..99d4b8bd32 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -50,6 +50,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx)
     int plane_size = frame->width * frame->height * sizeof(uint8_t);
     enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE;
     int src_datatype_size = get_datatype_size(output->dt);
+    int channel_idx;

     int bytewidth = av_image_get_linesize(frame->format, frame->width, 0);
     if (bytewidth < 0) {
@@ -83,6 +84,17 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx)
     switch (frame->format) {
     case AV_PIX_FMT_RGB24:
     case AV_PIX_FMT_BGR24:
+        channel_idx = dnn_get_channel_idx_by_layout(output->layout);
+        if (output->dims[channel_idx] != 3) {
+            av_log(log_ctx, AV_LOG_ERROR,
+                   "dnn_processing to a %s frame requires a 3-channel model "
+                   "output, got %d channels; channel-reducing/expanding "
+                   "models (e.g. single-channel matte output) are not "
+                   "supported by this filter\n",
+                   av_get_pix_fmt_name(frame->format), output->dims[channel_idx]);
+            ret = AVERROR(ENOSYS);
+            goto err;
+        }
         sws_ctx = sws_getContext(frame->width * 3,
                                  frame->height,
                                  src_fmt,