Testing local models for Unreal tools

I’m building Unreal Agent to see how much Unreal Engine work I can do with a local model. For these tests, I’m checking whether it picks the right tool and fills in the right arguments. I previously tried QLoRA training on an 8B model; this round uses a 27B model without any further training.

Before spending time on another training run, I wanted to see whether better tool descriptions would fix the mistakes. It used Qwen3.8-27B L-TI in LM Studio with reasoning disabled, a 31,232-token context, and 80% GPU offload.

A small ambiguity caused a lot of errors

In the test, each Unreal value has a short reference the model is supposed to use. The model often picked the right tool, then supplied the value itself where the tool expected that reference.

I rewrote the descriptions to explain which reference to use, what each argument meant, what happened if an optional argument was left out, and when to refuse a request. Then I ran the same tests with the same model.

What changed

These were practice tool calls checked against expected answers. Nothing ran in Unreal or changed a project file. Since I reused the tests while improving the descriptions, I still need to check the model on fresh cases.

What it still got wrong

Of the seven answers that didn’t match, three left out an optional field that already defaulted to false. The other four were more concerning: one request that should have been refused got a harmless inventory call instead, and three refusals gave the wrong reason.

I wouldn’t leave it editing a project on its own yet. Before any tool runs, the software needs to check the arguments and whether the requested change is allowed.

Next experiment

Next I want to give it better Unreal reference material and try a fresh set of tests. Once I know which mistakes remain, I can decide whether another training run is worth doing.