Commit 828fdf282 for llama.cpp
commit 828fdf282e195300c2965bd9511807e24ed53bdb
Author: wendadawen <130649302+wendadawen@users.noreply.github.com>
Date: Tue Sep 22 21:04:52 2026 +0800
spec : support DFlash for HunyuanOCR (#28890)
* model : add DFlash layer-input taps for HunyuanVL
DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:
GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")
Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.
The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.
Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft conversion against HunYuan targets
Converting a DFlash draft with a HunYuan target failed in two ways.
1. DFlashModel.set_vocab() reuses the target class' vocab handling by
calling it unbound with the draft instance, but HunYuanModel.set_vocab()
called self._fix_special_tokens(), a method that only exists on
HunYuanModel, so the conversion always aborted with
AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'
Make the vocab helpers module-level functions taking the model
explicitly, so they do not depend on the instance being a HunYuanModel.
They have no other callers, so the two id lookups are folded into
_fix_special_tokens().
2. The delegated call runs with self.dir_model pointed at the target but
keeps the draft's self.hparams, so config lookups inside the target's
vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
config instead of the target's. That aborts on targets with
pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
writes special token ids that disagree with the target.
Add _vocab_hparams(): it returns the target's config (with text_config
merged to the root, as TextModel does) when the model is a draft
converted with --target-model-dir, and the model's own hparams
otherwise, so a normal conversion is unaffected.
Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft vocab against HunYuan targets
Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.
* convert : fix HunYuan special token ids for DFlash drafts
* convert : use load_hparams for HunYuan special token ids
diff --git a/conversion/hunyuan.py b/conversion/hunyuan.py
index ee1a10654..58dee2ec8 100644
--- a/conversion/hunyuan.py
+++ b/conversion/hunyuan.py
@@ -159,32 +159,14 @@ class HunYuanMoEModel(TextModel):
class HunYuanModel(TextModel):
model_arch = gguf.MODEL_ARCH.HUNYUAN_DENSE
- def _get_eod_token_id(self) -> int | None:
- """Get the actual end-of-generation token from config (eod_token_id)."""
- return self.hparams.get("eod_token_id")
-
- def _get_eot_token_id(self) -> int | None:
- """Get the end-of-turn token from generation_config.json.
- This is the first entry in eos_token_id when it's a list."""
- gen_cfg_path = self.dir_model / "generation_config.json"
- if gen_cfg_path.is_file():
- with open(gen_cfg_path, encoding="utf-8") as f:
- gen_cfg = json.load(f)
- eos = gen_cfg.get("eos_token_id")
- if isinstance(eos, list) and len(eos) >= 2:
- return eos[0]
- return None
-
- def _fix_special_tokens(self):
- """Fix EOS/EOT tokens that are incorrect in upstream configs."""
- eod_id = self._get_eod_token_id()
- if eod_id is not None:
- self.gguf_writer.add_eos_token_id(eod_id)
- eot_id = self._get_eot_token_id()
- if eot_id is not None:
- self.gguf_writer.add_eot_token_id(eot_id)
-
def set_vocab(self):
+ # Also called by draft models (e.g. DFlash), with dir_model pointing at
+ # the target model.
+ config = ModelBase.load_hparams(self.dir_model, self.is_mistral_format)
+ config = {**config, **config.get("text_config", {})}
+ self.hparams["pad_token_id"] = config.get("pad_token_id")
+ self.hparams["eod_token_id"] = config.get("eod_token_id")
+
if (self.dir_model / "tokenizer.json").is_file():
tokens, toktypes, tokpre = self.get_vocab_base()
self.gguf_writer.add_tokenizer_model("gpt2")
@@ -199,7 +181,6 @@ class HunYuanModel(TextModel):
token_types = ('bos', 'eos', 'unk', 'sep', 'cls', 'mask')
special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=True, special_token_types=token_types)
special_vocab.add_to_gguf(self.gguf_writer)
- self._fix_special_tokens()
else:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(self.dir_model, trust_remote_code=True)
@@ -251,7 +232,18 @@ class HunYuanModel(TextModel):
# FIX for BOS token: Overwrite incorrect id read from config.json
if self.hparams['hidden_size'] == 4096:
self.gguf_writer.add_bos_token_id(127958) # only for 7b dense, fix <|bos|> token
- self._fix_special_tokens()
+
+ # Fix EOS/EOT tokens that are incorrect in upstream configs.
+ eod_id = self.hparams.get("eod_token_id")
+ if eod_id is not None:
+ self.gguf_writer.add_eos_token_id(eod_id)
+
+ gen_cfg = self.dir_model / "generation_config.json"
+ if gen_cfg.is_file():
+ with open(gen_cfg, encoding="utf-8") as f:
+ eos = json.load(f).get("eos_token_id")
+ if isinstance(eos, list) and len(eos) >= 2:
+ self.gguf_writer.add_eot_token_id(eos[0])
def set_gguf_parameters(self):
# Some HunYuanVL variants set num_experts=1 (not real MoE);
diff --git a/src/models/hunyuan-vl.cpp b/src/models/hunyuan-vl.cpp
index da9bb74de..18b6eaf8c 100644
--- a/src/models/hunyuan-vl.cpp
+++ b/src/models/hunyuan-vl.cpp
@@ -83,6 +83,8 @@ llama_model_hunyuan_vl::graph::graph(const llama_model & model, const llm_graph_
ggml_tensor * inp_out_ids = build_inp_out_ids();
for (int il = 0; il < n_layer; ++il) {
+ res->t_layer_inp[il] = inpL;
+
ggml_tensor * inpSA = inpL;
// norm