Qwen3.5 Support

[lucasjinreal]

Hi, Qwen3.5 now native multimodal, and extremly nice on text abilities. Very suitable for local Agents. would consider support

[alan890104]

Really need Qwen3.5 model!!!

[alan890104]

I tried implementing my own 4-bit quantized version to run locally, but it seems I misconfigured something, as the output is meaningless.

[lucasjinreal]

consider add a PR once you get it work..

[woodRock]

I've got it working, but it is very slow, likely needs a custom kernel for the Fused Parallel Scan.

See: #3396

[lucasjinreal]

Any updates?

[latentcollapse]

I just opened a PR to add Qwen3.5 support. Waiting to see what happens

[lucasjinreal]

there are plenty of qwen3.5 merge request support, but one of them really fully tested and speed match the torch speed?

[latentcollapse]

I fixed the VRAM usage and with an 8gb RTX 5060, I was getting 30ish tokens per second with a 4bit quant Qwen3.5-9B. Might not be as fast as PyTorch, but it's usable

[lucasjinreal]

i think it should be faster, if can not even match torch, it's not very deployable in most cases.

[latentcollapse]

That's the hope, considering how much better Rust is. I figure tomorrow or the day after they'll review my PR and if not, you can check my candle fork directly as the code to make it work is there

[lucasjinreal]

how can we push it further for speed, what is currently the most slowest part?

[latentcollapse]

I mean, for my hardware, it's running pretty well. I'm limited by my GPU, which is an 8gb RTX 5060. I was getting 30-ish tokens per second on a 4bit 9B model, which was roughly comparable to PyTorch from my other testing. I just don't have the greatest GPU, which is probably why the speeds aren't ludicrously fast. But it was a bunch of new things I had to add and test to even make Qwen3.5 work, but it works now

[AlpineVibrations]

how fast does it go on python on your machine?
we could use a benchmark section to compare PyTorch
thanks for working on it

[latentcollapse]

PyTorch comparison — RTX 5060 (8151 MiB), CUDA 12.8, bf16

Model | PyTorch 2.10 | Candle (this PR) | Delta -- | -- | -- | -- Qwen3.5-0.8B | 15.1 tok/s | 89.3 tok/s | +5.9x Qwen3.5-2B | 15.4 tok/s | 52.1 tok/s | +3.4x

Worth noting: PyTorch logs "The fast path is not available — falling back to torch implementation" for this architecture. Qwen3.5's GatedDeltaNet layers require flash-linear-attention and causal-conv1d for the optimized PyTorch path, neither of which are installed by default. The Candle numbers above require no additional CUDA kernel dependencies beyond the standard toolkit.

If someone has flash-linear-attention installed, happy to rerun for a complete picture. But for the vast majority of users, this is the relevant comparison.