Running a converted MoE model locally

I wanted to run a community Qwen distillation locally, with working tool calls and room for a reusable prompt cache. The workstation was an RTX 5070 Ti with 16 GB of VRAM, a Ryzen 9 9900X, and 64 GB of RAM.

The converted checkpoint is 24.19 GB. It runs with MoE expert weights offloaded to system RAM and a smaller expert cache on the GPU. The full checkpoint does not fit in VRAM.

The conversion

I started with Empero’s community Qwen3.8-35B-A3B-Distill model. I converted it, packaged the files, and tested it locally. Most of the memory savings come from storing the expert weights in NVFP4. Some attention weights use FP8, while other parts stay in BF16.

The output has 21 main weight shards and one MTP shard. I checked the downloaded payload against its SHA-256 manifest and verified 10,280 gate/up global-scale pairs. The conversion used temporary RunPod hardware; the tests below ran on my Windows workstation in FreeToken.

What the cache changed

I made a long test prompt—138,187 tokens—with a marker at the beginning, middle, and end. I sent it twice and asked for those markers. The model found all three both times.

First request107.02 s

Cached repeat2.57 s

The second request reused 138,176 tokens from the cache and finished in 2.57 seconds, down from 107.02. That’s about 41.6 times faster for this particular repeat. The saving came from reusing work the model had already done; I’d need different tests to measure its answer speed or how well it handles other long prompts.

What I still need to check

So far, I’ve only run these quick checks on my own setup. I still need to compare the answers with the original BF16 model and try the full context window, image input, and speculative decoding. In a short coding test, it also added Markdown fences after I asked for code only.

The cache has 347,136 token slots shared across requests. A single conversation is limited to 262,144 tokens, and I haven’t tested that full length yet.