Evolution of LLMs use by a programmer

Thinking that 6 months ago I didn’t use LLMs at all for programming and seeing the use I make of it today, I thought it would be interesting to write a blog post about the evolution.

This post is based on my experience developing Asfaload, an opensource multi-signature sign-off solution developed in Rust and using minisign for handling cryptographic signatures.

Some important points to note are:

• I’m working on an open source project, and I’m ok sharing the code with LLMs.
• I am the solo developer of the project.
• I was starting the project using a programming language I didn’t know at the time. Finding solutions to the error messages returned by the Rust compiler was a good incentive to interact with LMs.
• I’ve always enjoyed designing and specifying the systems I was working on. I spent months specifying how Asfaload would work and this as paid off now I’m using LLMs extensively.

While these factors are important regarding my experience, you might follow a similar path in your journey discovering what LLMs can bring to programming.

First phase: prudent discovery

In the first phase I started to use LMs via a chat interface to ask questions about how to write some code or about error messages I got from te compiler. I was just starting Rust and it was very interesting and very helpful to progress much faster than I would if I hadn’t had access to these chat interfaces.

But then I started asking questions that that were increasingly complex. And that led me to the second phase of my use of LLMs for programming.

Second phase: getting familiar

As my interactions with the LLMs via the chat interface were really helpful, I asked questions that were getting complex and that became deeply linked with my code. Inevitably I started to paste code excerpts. Initially I thought I would write the tests myself, but it appeared that it was very helpful to let the LLMs generate the tests and to review them afterwards. So a lot of interactions with the chat but of the kind “Write me exhaustive tests for this function that I paste below”. This worked fine even if I reviewed all the code generated and I regularly add fundamental changes to apply. In the end it made me win time and if it didn’t make me faster, it freed me from the burden of unpleasant work.

But as the test generation worked fine, I went further and began to ask a question regarding code in the application. That led me to the third phase.

Third phase: sharing more

As my interactions with the LLMs were getting more complex, the code that was involved in the discussion was spread in multiple files. The copy paste work was becoming cumbersome. That’s when I started to use tools to generate the context to be sent to the LM. My tool of choice was AI code prep. This was working great. It allowed me to pass information to the LLMs coming from all over my project. However, for the code or the content returned by the LLM, it was again a lot of copy paste work. That’s why I tested coding agents about which I had heard a lot. And that got me in the fourth phase.

Fourth phase: giving write access

The great thing about agents is that they have both read and write access to your code. The bad thing is that they could have access to your whole system. That’s why I’m running the agents in a container. I know a container is not a perfect isolation layer, but it’s already much better than running it with access to all my files. If interested you can have a look at my Docker configuration.

Again, this was a step up from the previous phase. But interacting with an agent is tiring. You have to precisely tell it what it can do or what it cannot do, and even then you are not sure your instructions will be followed precisely. Now I understood why I saw more and more Git repositories having an agents.md file at their root. But it led me to asking the question: are there standardised instructions for agents for my technology stack? I found some, but I found even a better thing: skills frameworks.

Fifth phase: using a skills framework

There are multiple skills frameworks and those provide information to the agent about how it should go to work and how it can split the work between sub agents. I’ve settled on the superpowers framework which was originally developed for clouds, but I use it with opencode.

The way I’m currently working with LLMs using skills framework is the following:

• First give it instructions to devise a plan for the features to be developed I described.
• Then I review the plan and ask for refinements.
• When I’m happy with the the the the the the the the the the the plan I tell it to implement it.
• When it is done developing it I review all the code that was generated.

The last step can take quite some time, but I feel I’m still going faster. And reviewing the code generators is in my experience really needed. Here are some examples of surprises I’ve had:

• There was a function generated for the application that was fully tested but never used in the application code!
• There are duplicated tests. Those tests were presented as different, but in the end they were doing exactly the same.
• Even though I asked it explicitly to use defined constants, it still hardcoded some values.
• Multiple tests were simply an assert!(true) with a comment Needs to be implemented. Of course it was passing!

In addition to my own reviews, I also used the Gemini Code Reviewer available for GitHub pull requests. It rarely gets off track and often times gives really useful reviews.

Sixth phase: becoming a paying customer

Until recently, I was using z.ai’s GLM model freely: via the z.ai chat for early phases, and with Opencode Zen’s free access to the GLM model. But these free offerings, even though working very well, seem to be on the slow side. So I went to look for a provider that was affordable. I wanted to avoid a pay per use offering, as these seems to be much more expensive than monthly plan. I settled on Synthetic.new as they provide affordable plans to access open source LLMs, including z.ai’s GLM, which perfectly match my needs.

How did your use of LLMs evolve? Similar or different, don’t hesitate to share in the comments.