Throw Away Your Skills ?
Background
I’ve just finished up a project investigating clankers writing code for a novel hardware architecture.
I was sure prior to trying this that I would need to create tools to keep them on track and maintain their understanding of the constraints of the system. I was wrong about that. Given the fundamentals of the system, the clankers (Opus 5.5) went away and built this tooling themselves. And they built something that I probably wouldn’t have targeted, believing it too complex. I would have just attempted to simulate the system using code written for the host machine rather than the RISC-V target, whereas they actually emulated the system with QEMU and tested the RISC-V binaries directly. Running the actual code comes with nice benefits that I would have missed out on: Given that we were working within specific timing and memory constraints, being closer to the final code size and timing was a big plus.
Skillz
Given that the model did a good job on its own, what is the point of skills? At least in a per-project sense, why not just let the clankers find the relevant context for a task and come up with their own approaches1? The only thing that skills do is give a framework for tasks. It is a generation-led, stochastic thing. The LLM is outputting tool calls based on the input from the skill file, which may not even be scripts that are associated with the skill. If its output goes a certain way it may even ignore the skill, choosing to do its own thing.
There is a chance that skills just get in the way of the model doing its thing. We should be reviewing them often ensuring they make sense to complement the models we use.
And if using skills for repetitive, pipeline-esque tasks (1 -> 2 -> 3 -?> 4), those should just be scripted anyway to make them more deterministic. It is pretty easy now to use harness libraries to create these things, (ask your favourite LLM about it 😉). ↩︎