Commit 3bf26b81 for tesseract

commit 3bf26b81457248e69a101d4538ee33f1d0b49af8
Author: Stefan Weil <sw@weilnetz.de>
Date:   Sun Nov 15 08:19:06 2026 +0100

    serialis: remove the size limit for strings and byte vectors (#4703)

    TFile::DeSerialize rejected any string or vector with more than
    50000000 elements. A full training checkpoint embeds further trainer
    dumps as byte vectors, so a network whose trainer dump is larger than
    that limit wrote checkpoints which could not be read back: lstmeval
    failed with "Deserialize failed" and lstmtraining --continue_from with
    "Failed to read continue from".

    This happened in a training with two convolutional layers and the
    LSTM layers Lfys64 Lbx128 Lfx512, where a trainer dump had a size of
    61785585 bytes.

    For strings and byte vectors the limit is not needed to protect
    against bad data: the declared size must be backed by the bytes which
    remain in the stream, and each element needs exactly one byte, so the
    allocation cannot be larger than the input.

    The limit for vectors of other types is kept because their elements
    can need more memory than their serialized form.

    Fixes: 82727cc11c34 ("Fix memory-safety issues in .traineddata deserialization")
    Assisted-by: Claude Fable 5.2 (Anthropic)

    Signed-off-by: Stefan Weil <sw@weilnetz.de>

diff --git a/src/ccutil/serialis.cpp b/src/ccutil/serialis.cpp
index cd542262..615fde91 100644
--- a/src/ccutil/serialis.cpp
+++ b/src/ccutil/serialis.cpp
@@ -99,13 +99,13 @@ bool TFile::DeSerialize(std::string &data) {
   uint32_t size;
   if (!DeSerialize(&size)) {
     return false;
-  } else if (size > 50000000) {
-    // Arbitrarily limit the size to protect against bad data.
-    return false;
   } else if (size > 0) {
     // The declared size must be backed by the bytes that actually remain in
     // the stream, so a crafted file cannot declare a huge string while
-    // carrying only a few trailing bytes (denial of service).
+    // carrying only a few trailing bytes (denial of service). As each element
+    // needs exactly one byte, this also limits the allocation to the size of
+    // the input, so there is no need for an additional arbitrary limit.
+    // Training checkpoints embed trainer dumps of more than 50 MB.
     if (static_cast<uint64_t>(size) > RemainingBytes()) {
       return false;
     }
@@ -126,13 +126,13 @@ bool TFile::DeSerialize(std::vector<char> &data) {
   uint32_t size;
   if (!DeSerialize(&size)) {
     return false;
-  } else if (size > 50000000) {
-    // Arbitrarily limit the size to protect against bad data.
-    return false;
   } else if (size > 0) {
     // The declared size must be backed by the bytes that actually remain in
     // the stream, so a crafted file cannot declare a huge vector while
-    // carrying only a few trailing bytes (denial of service).
+    // carrying only a few trailing bytes (denial of service). As each element
+    // needs exactly one byte, this also limits the allocation to the size of
+    // the input, so there is no need for an additional arbitrary limit.
+    // Training checkpoints embed trainer dumps of more than 50 MB.
     if (static_cast<uint64_t>(size) > RemainingBytes()) {
       return false;
     }