Trying a 320B model with 16 GB of VRAM

I wanted to find out how far I could get with a large local coding model on the PC I already had: a Ryzen 9 9900X, an RTX 5070 Ti with 16 GB of VRAM, and 64 GiB of system RAM.

The model was GLM-5.3-Flash, a mixture-of-experts model with 320 billion parameters and about 18 billion active for each token. The full text model eventually generated correct answers on my machine. Getting there took memory planning, component tests, and fixes to the code running it.

The result still wasn’t something I wanted to use for coding. On the short coding test, it produced roughly 0.65–0.68 tokens per second. A fresh process took about 92 seconds to produce its first token, including setup.

Why the active parameter count wasn’t enough

In a mixture-of-experts model, the router selects a subset of expert networks for each token. That reduces the computation compared with using every expert. The other experts still have to be available when the router selects them later.

GLM’s sparse layers select eight experts from 288. A token passes through many of those layers, so “eight experts” doesn’t describe the total amount of work needed to generate it.

I needed to account for where the weights would live, how they would get to the processor using them, and what memory would remain for the rest of the runtime.

The memory estimate ruled out an easy fit

The early calculations looked at much lower precision. At an ideal two bits per parameter, the routed expert weights alone came to about 70.9 GiB. At 1.5 bits, they came to about 53.2 GiB.

Those figures excluded quantization overhead, the rest of the model, runtime buffers, cached state, Windows, and other applications. They also said nothing about whether the model would still give useful answers at that precision.

For the actual generation test, I used Unsloth’s existing UD-Q4_K_XL pack. Its experts use a mixture of Q4, Q5, and Q6 formats; most other matrices use Q8. I didn’t produce or validate a new 1.5-bit version.

The working arrangement used GPU memory for non-expert weights and CPU execution for experts mapped from files on the NVMe drive. Memory mapping lets the operating system bring file pages into RAM as needed. The model’s weights were never all resident in the 16 GB GPU.

There were software bugs to fix first

Before the full run, the project checked smaller pieces against reference calculations: routing, attention, intermediate values, and saved state. That made failures easier to isolate than staring at a bad final answer.

One router test exposed a normalization error when the scores were very small. The runtime’s denominator handling didn’t match the reference. Fixing it required testing those small values directly; ordinary inputs hadn’t exposed the same problem.

Full generation then uncovered two more bugs. A diagnostic setting forced F32 activation precision on quantized matrix operations that didn’t support that policy. Restricting it to F32 weight matrices fixed the assertion.

The second bug was in a diagnostic callback. After a successful check, it returned False. In that part of the runtime, False meant stop processing the current section of the computation graph. The check was accidentally preventing later computation from running.

Resetting memory didn’t fix the first request. Correcting the callback did. Those early bad outputs couldn’t tell me whether the model or its quantization was any good, because the runner hadn’t completed the calculation.

What the working run actually showed

The final checks used two deliberately small prompts: a greeting and a Python function that adds two arguments. Each was run twice with a reset between requests. Both reached the end-of-sequence token and repeated the same output tokens after reset.

The generated function passed six functional cases after its syntax tree was checked. That established a small working example, not general coding ability.

The reported decode rate counts completed decode calls, including the one predicting the end-of-sequence token. It excludes setup and prompt processing. These were tiny tests with uncontrolled filesystem cache state, not sustained or cold-cache benchmarks.

Memory pressure was substantial. The coding process reached a 48.67 GiB working set, including mapped file pages. System-wide available RAM dropped to about 264 MB. Sampled GPU use peaked near 11,071 MiB across the two tests, including other applications.

The spare VRAM didn’t establish an easy performance fix. This path used llama.cpp’s CPU-mapped experts, not Strata’s optimized Qwen expert scheduler. Physical disk reads and expert reuse hadn’t been profiled, so I couldn’t honestly attribute every stall to the SSD.

Why I stopped there

I wanted an interactive coding tool. This run showed that the full quantized text model could generate on the machine, but the waiting time was too much for that use. I parked the GLM work and later removed the roughly 200 GB weight pack. The code and test records were kept.

A follow-up with the supported Strata Coder model returned much faster numbers on short tests, around 67–75 generated tokens per second. Longer use in the coding harness was still disappointing, with some turns spending 14–17 seconds rereading a long prompt. Those were different models, prompts, and runtimes, so the rates aren’t a controlled comparison.

That changed what I wanted to test next. I need to measure the tasks I actually do, including prompt rereads, tool calls, and the time before an answer starts. A fast short response can miss much of the waiting involved in a coding session.

If I return to GLM, I’d start by measuring disk reads, page faults, and expert reuse on this working baseline. I’d want that evidence before deciding whether another round of memory or kernel work is worth it.