Commit 10b8b7553 for llama.cpp
commit 10b8b7553494f3638da7668cf17cac089229122a
Author: Pascal <admin@serveurperso.com>
Date: Sat Oct 17 15:07:55 2026 +0200
tests: give the random model generator a per-tensor filler table (#30674)
* tests: give the random model generator a per-tensor filler table
Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about
six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a
threshold of 1.0e-05. The sliding window is what exposes it: a restored state
keeps the window alone, so the attention of an SWA layer reduces over a
different cell layout and the continuation is not bit exact. Every SWA model
carries that, at 1e-08. Muse Spark carried it at 1e-05.
The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate
holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one
raw weight, while the generator draws every tensor on its own. The pair then
scaled the stream and the branch independently, the stream lost an arbitrary
factor at each layer, and the model became an amplifier. The weights the test
ran on were not a Muse Spark.
Per-tensor rules lived inside the write path, where a flag picked the values of
a tensor inside each of the two branches that store them, so a new rule meant a
new flag threaded through both. The file asks to be rid of that in a TODO of its
own. Give each rule a filler, pick one from a table keyed on a substring of the
tensor name, and keep a single write path behind it. Mamba moves into the table
and Muse Spark joins it: the gate filler draws the raw weight once per block and
resolves alpha and beta the way the conversion does.
Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to
2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the
threshold. The other 131 models come out identical to the bit, since no other
architecture carries these tensors. ctest is green on a CPU build and on a CUDA
build.
* tests: draw every tensor filler from fill_normal
The fillers repeated the generator and the distribution each time. The
normal draw now takes its key and the other fillers transform its
values, so the generator lives in one place. The generated models are
identical to the bit.
diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp
index 143c1335e..94a22ff2f 100644
--- a/tests/test-llama-archs.cpp
+++ b/tests/test-llama-archs.cpp
@@ -53,33 +53,66 @@ struct tensor_data_params {
float stdev;
};
+// fills the values of one tensor, the name carries whatever the property needs
+typedef void (*tensor_filler)(std::vector<float> & dst, const char * name, const tensor_data_params & params);
+
+// a normal draw keyed on a name, so that tensors which share a draw ask for the same key
+static void fill_normal(std::vector<float> & dst, const char * key, const tensor_data_params & params) {
+ std::mt19937 gen(params.seed ^ std::hash<std::string>{}(key));
+ std::normal_distribution<float> dis(0.0f, params.stdev);
+ for (float & v : dst) {
+ v = dis(gen);
+ }
+}
+
+// Mamba A must be negative (state decay)
+static void fill_ssm_a(std::vector<float> & dst, const char * name, const tensor_data_params & params) {
+ fill_normal(dst, name, params);
+ for (float & v : dst) {
+ v = -fabsf(v);
+ }
+}
+
+// a norm preserving residual gate holds alpha^2 + beta^2 == 1, and the conversion resolves both from one raw weight
+// the alpha and the beta of a block share that weight, so the key drops the suffix and keeps the block
+static void fill_resid_gate(std::vector<float> & dst, const char * name, const tensor_data_params & params) {
+ const char * suffix = strstr(name, "_resid_");
+ const bool is_beta = strstr(suffix, "_resid_beta") != nullptr;
+ fill_normal(dst, std::string(name, suffix - name).c_str(), params);
+ for (float & v : dst) {
+ const float beta = 1.0f / (1.0f + expf(-v));
+ v = is_beta ? beta : sqrtf(1.0f - beta*beta);
+ }
+}
+
+// a tensor whose values hold a property of their own, matched on a substring of its name, first match wins
+static const struct { const char * match; tensor_filler fill; } TENSOR_FILLERS[] = {
+ { "ssm_a", fill_ssm_a },
+ { "_resid_", fill_resid_gate },
+};
+
static void set_tensor_data(struct ggml_tensor * tensor, void * userdata) {
const tensor_data_params & params = *(const tensor_data_params *) userdata;
- size_t seed = params.seed;
- std::hash<std::string> hasher;
- seed ^= hasher(tensor->name);
- std::mt19937 gen(seed);
- std::normal_distribution<float> dis(0.0f, params.stdev);
- // TODO: refactor per-tensor initialization logic in a cleaner way
+ tensor_filler fill = fill_normal;
+ for (const auto & filler : TENSOR_FILLERS) {
+ if (strstr(tensor->name, filler.match) != nullptr) {
+ fill = filler.fill;
+ break;
+ }
+ }
+
+ std::vector<float> tmp(ggml_nelements(tensor));
+ fill(tmp, tensor->name, params);
- // note: Mamba A must be negative (state decay)
- const bool is_ssm_a = strstr(tensor->name, "ssm_a") != nullptr;
- const int64_t ne = ggml_nelements(tensor);
if (tensor->type == GGML_TYPE_F32) {
- std::vector<float> tmp(ne);
- for (int64_t i = 0; i < ne; i++) {
- float val = dis(gen);
- tmp[i] = is_ssm_a ? -fabsf(val) : val;
- }
ggml_backend_tensor_set(tensor, tmp.data(), 0, ggml_nbytes(tensor));
} else if (tensor->type == GGML_TYPE_F16) {
- std::vector<ggml_fp16_t> tmp(ne);
- for (int64_t i = 0; i < ne; i++) {
- float val = dis(gen);
- tmp[i] = ggml_fp32_to_fp16(is_ssm_a ? -fabsf(val) : val);
+ std::vector<ggml_fp16_t> tmp16(tmp.size());
+ for (size_t i = 0; i < tmp.size(); i++) {
+ tmp16[i] = ggml_fp32_to_fp16(tmp[i]);
}
- ggml_backend_tensor_set(tensor, tmp.data(), 0, ggml_nbytes(tensor));
+ ggml_backend_tensor_set(tensor, tmp16.data(), 0, ggml_nbytes(tensor));
} else {
GGML_ABORT("fatal error");
}