Gemma 4 Model download issues

[AlpineVibrations]

tried the new gemma4. with

cargo run -r --example gemma4 --features metal -- --prompt "write a business model for magic spells" --model-id google/gemma-4-E2B-it

i tried it and it wont downlaod the model. so i used the hf-hub tool and downloaded the model and it still errors on trying to download both smaller model-id.

Error: request error: https://huggingface.co/google/gemma-4-E2B-it/resolve/main/model.safetensors.index.json: status code 404

[airpods69]

Is it resolved now? Did you try?

[KarstenB]

I did try it, but the file https://huggingface.co/google/gemma-4-E2B-it/resolve/main/model.safetensors.index.json really does not exist. Anyhow, when I download the safetensor and run it with --weight-files gemma4e4b-it.safetensors I get:

Running on CPU, to run on GPU, build this example with `--features cuda`
Error: cannot find tensor model.embed_tokens.weight

[AlpineVibrations]

looks like they fixed the download issue but not the embed_tokens issue

     Running `target/release/examples/gemma4 --prompt 'tell me a story about a man and his donkey in a magical meadow'`
avx: true, neon: false, simd128: false, f16c: true
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 605.050989ms
Error: cannot find tensor model.embed_tokens.weight

[AlpineVibrations]

@EricLBuehler not sure if you got it work when working on this #3457

[PCfVW]

The error cannot find tensor model.embed_tokens.weight is a naming mismatch. Gemma 4 is multimodal (vision + audio + language), so all tensors are namespaced under their modality. The embedding tensor candle is looking for actually exists as:

model.language_model.embed_tokens.weight BF16 [262144, 1536] 768.00 MiB

Here are all the "embed" tensors in the model:

$ hf-fm inspect google/gemma-4-E2B-it model.safetensors --filter embed

Tensor Dtype Shape Size Params
model.embed_audio.embedding_projection.weight BF16 [1536, 1536] 4.50 MiB 2.4M
model.embed_vision.embedding_projection.weight BF16 [768, 768] 2.25 MiB 1.2M
model.language_model.embed_tokens.weight BF16 [262144, 1536] 768.00 MiB 402.7M
model.language_model.embed_tokens_per_layer.weight BF16 [262144, 8960] 4.38 GiB 2.35B
model.vision_tower.patch_embedder.input_proj.weight BF16 [768, 768] 1.12 MiB 589.8K
model.vision_tower.patch_embedder.position_embedding_table BF16 [2, 10240, 768] 30.00 MiB 15.7M

6/2011 tensors, 2.77B/5.12B params (filter: "embed")

The candle gemma4 example likely needs to prepend model.language_model. when loading tensor names, similar to how other multimodal models (PaliGemma, LLaVA) handle the prefix.

PS: hf-fm inspect reads safetensors headers via HTTP Range requests, i.e. no weight data downloaded; if needed: install with cargo install hf-fetch-model --features cli.

[AlpineVibrations]

good catch.

I tried it

The candle gemma4 example likely needs to prepend model.language_model. when loading tensor names, similar to how other multimodal models (PaliGemma, LLaVA) handle the prefix.

and it finds the layer but now we get error

Error: shape mismatch for model.language_model.layers.15.mlp.gate_proj.weight, expected: [6144, 1536], got: [12288, 1536]

[PCfVW]

@AlpineVibrations Following up: gate_proj and up_proj are separate tensors (not merged), both [12288, 1536] (vision tower tensors trimmed):

$ hf-fm inspect google/gemma-4-E2B-it model.safetensors --filter "layers.15.mlp"

Tensor Dtype Shape Size
model.language_model.layers.15.mlp.down_proj.weight BF16 [1536, 12288] 36.00 MiB
model.language_model.layers.15.mlp.gate_proj.weight BF16 [12288, 1536] 36.00 MiB
model.language_model.layers.15.mlp.up_proj.weight BF16 [12288, 1536] 36.00 MiB

So the intermediate_size is 12288, not 6144. The candle gemma4 example likely reads the wrong value from config.json or has it hardcoded.

[PCfVW]

Following up: it's not hardcoded; but candle's gemma4 implementation is missing two config fields entirely.

Root cause (verified against transformers' modeling_gemma4.py)

The reference Python implementation does:

first_kv_shared_layer_idx = config.num_hidden_layers - config.num_kv_shared_layers
is_kv_shared_layer = layer_idx >= first_kv_shared_layer_idx > 0
use_double_wide_mlp = config.use_double_wide_mlp and is_kv_shared_layer
self.intermediate_size = config.intermediate_size * (2 if use_double_wide_mlp else 1)

For gemma-4-E2B-it:

• num_hidden_layers = 35
• num_kv_shared_layers = 20
• first_kv_shared_layer_idx = 35 - 20 = 15
• Layers 15..34 (last 20) use 2 × intermediate_size = 12288
• Layers 0..14 (first 15) use intermediate_size = 6144

The boundary at index 15 exactly explains your error: layer 15 is the first KV-shared (double-wide) layer, which is why you hit the shape mismatch there.

What candle is missing

candle's Gemma4TextConfig is missing two fields:

• num_kv_shared_layers: usize
• use_double_wide_mlp: bool

And DecoderLayer::new() in gemma4/text.rs passes a fixed cfg.intermediate_size to every MLP (no per-layer branching), so every layer gets 6144, and the weight file's 12288 for the last 20 layers fails to load.

Proposed fix

// config.rs — in Gemma4TextConfig
#[serde(default)]
pub num_kv_shared_layers: usize,
#[serde(default)]
pub use_double_wide_mlp: bool,

// text.rs — in DecoderLayer::new(), replace the MLP instantiation
let first_kv_shared_layer_idx = cfg.num_hidden_layers.saturating_sub(cfg.num_kv_shared_layers);
let is_kv_shared = first_kv_shared_layer_idx > 0 && layer_idx >= first_kv_shared_layer_idx;
let effective_intermediate = if cfg.use_double_wide_mlp && is_kv_shared {
    cfg.intermediate_size * 2
} else {
    cfg.intermediate_size
};
let mlp = MLP::new(
    cfg.hidden_size,
    effective_intermediate,
    cfg.hidden_activation,
    false,
    vb.pp("mlp"),
)?;

This is a candle model-code bug in candle-transformers/src/models/gemma4/, not an example bug; fixing it should unblock every Gemma 4 variant (E2B, E4B, etc.) not just the specific example...

[AlpineVibrations]

@PCfVW I tried that and it's still the same error as I reported above. I only added what you specified and it didn't seam to fix it.

[PCfVW]

Quick follow-up.

I cloned candle fresh, ran the example end-to-end against a local cache of google/gemma-4-E2B-it, and inspected the safetensors layout with hf-fm. The picture is clearer now and bigger than my earlier comments suggested (sorry!).

Reproduction (matches the OP)

$ git clone https://github.com/huggingface/candle.git
$ cd candle
$ cargo run --example gemma4 -- --cpu --prompt "test" --model-id google/gemma-4-E2B-it
...
retrieved the files in 176ms
Error: cannot find tensor model.embed_tokens.weight

Same error @AlpineVibrations reported. The example's text-only path calls TextModel::new(&config, vb), and text.rs:532 does
let vb_m = vb.pp("model");, looking for model.embed_tokens.weight.

In this multimodal repo the actual tensor lives at model.language_model.embed_tokens.weight. Adding vb.pp("language_model")
in the text-only call site fixes this immediate error, but it's only the first of three.

What hf-fm inspect reveals

gemma-4-E2B-it is architecturally closer to gemma-3n than to gemma-3; every text-decoder layer carries AltUp/Laurel per-layer-input weights that candle's gemma4/text.rs doesn't load and doesn't consume in the forward pass:

$ hf-fm inspect google/gemma-4-E2B-it model.safetensors --filter "language_model.layers.0"

Tensor Dtype Shape Size Params
model.language_model.layers.0.input_layernorm.weight BF16 [1536] 3.0 KiB 1.5K
model.language_model.layers.0.layer_scalar BF16 [1] 2 B 1 ← MISSING
model.language_model.layers.0.mlp.down_proj.weight BF16 [1536, 6144] 18.00 MiB 9.4M
model.language_model.layers.0.mlp.gate_proj.weight BF16 [6144, 1536] 18.00 MiB 9.4M
model.language_model.layers.0.mlp.up_proj.weight BF16 [6144, 1536] 18.00 MiB 9.4M
model.language_model.layers.0.per_layer_input_gate.weight BF16 [256, 1536] 768.0 KiB 393.2K ← MISSING
model.language_model.layers.0.per_layer_projection.weight BF16 [1536, 256] 768.0 KiB 393.2K ← MISSING
model.language_model.layers.0.post_attention_layernorm.weight BF16 [1536] 3.0 KiB 1.5K
model.language_model.layers.0.post_feedforward_layernorm.weight BF16 [1536] 3.0 KiB 1.5K
model.language_model.layers.0.post_per_layer_input_norm.weight BF16 [1536] 3.0 KiB 1.5K ← MISSING
model.language_model.layers.0.pre_feedforward_layernorm.weight BF16 [1536] 3.0 KiB 1.5K
model.language_model.layers.0.self_attn.k_norm.weight BF16 [256] 512 B 256
model.language_model.layers.0.self_attn.k_proj.weight BF16 [256, 1536] 768.0 KiB 393.2K
model.language_model.layers.0.self_attn.o_proj.weight BF16 [1536, 2048] 6.00 MiB 3.1M
model.language_model.layers.0.self_attn.q_norm.weight BF16 [256] 512 B 256
model.language_model.layers.0.self_attn.q_proj.weight BF16 [2048, 1536] 6.00 MiB 3.1M
model.language_model.layers.0.self_attn.v_proj.weight BF16 [256, 1536] 768.0 KiB 393.2K
17/2011 tensors, 36.2M/5.12B params (filter: "language_model.layers.0")

Four missing per-layer tensors per decoder layer (×35 layers), and one missing global table:

$ hf-fm inspect google/gemma-4-E2B-it model.safetensors --filter "embed_tokens_per_layer"

Tensor Dtype Shape Size Params
model.language_model.embed_tokens_per_layer.weight BF16 [262144, 8960] 4.38 GiB 2.35B
1/2011 tensor, 2.35B/5.12B params (filter: "embed_tokens_per_layer")

embed_tokens_per_layer is a [vocab_size, num_layers × hidden_size_per_layer_input] = [262144, 35 × 256] = [262144, 8960] lookup, ~2.35B parameters, providing a separate per-layer 256-dim embedding for each token. The "Effective 2B" naming convention is the giveaway, same trick as gemma-3n-E2B: 5.12B total params, ~2B "effective" at inference because the per-layer embedding lookup is sparse-access. The HF transformers reference for the forward flow is in the Gemma4Text* modules of modeling_gemma4_text.py (the Gemma4TextScaledWordEmbedding plus the per-layer-input branch inside Gemma4DecoderLayer).

Three bugs, ordered

1. Tensor name prefix (text-only path, text.rs:532): vb.pp("language_model") missing. Reproduces immediately above.
2. Per-layer MLP doubling: Gemma4TextConfig is missing num_kv_shared_layers and use_double_wide_mlp; layers 15..34 (last 20 of 35) need 2 × intermediate_size = 12288. Details in [my earlier comment]. The [12288, 1536] shape on layer 15 is exactly what AlpineVibrations hit after the prefix fix.
3. AltUp/Laurel forward pass: the substantive lift. The four per-layer tensors plus embed_tokens_per_layer need to be loaded, and at each layer the residual must be projected into the per-layer space, gated against the per-layer embedding, normalized, scaled by layer_scalar, and mixed back. None of this is in gemma4/text.rs today. Even if (1)+(2) succeed, the forward pass produces incorrect logits without (3).

Suggested fixing order

Practical fixing order would be:

• (1) + (2) standalone: small change, lets gemma-4-E2B-it load cleanly, with a clear TODO: per-layer-input flow not implemented comment and a runtime check that bails out with a descriptive error rather than producing garbage. A few hours.
• (3): the real fix, with a logit-equality test against the Python reference on a fixed input. Real architectural work, ~a day or two of careful porting + verification.

Hoping this helps...