Using LLM-Assisted Coding to Write a Custom Template Function
In this series of articles I’ll explore LLM-assisted coding through a particular lens: my own experience as an average developer, working in and around an open source codebase, chronicling the ways in which LLM assistants do (or don’t!) make me better at various tasks. I’ve been coding for decades, more as a toolsmith than a production developer; tasks in my lane have typically been feature prototypes, automation scripts, data analysis, document processing. The first iteration of GitHub Copilot wasn’t a game-changer for me. But the conversational assistants — ChatGPT-4, Sourcegraph Cody, and GitHub Copilot Chat — have become essential. I’m writing this series to investigate some of the emerging benefits.
My day job places me mostly at the periphery of the Steampipe codebase. Sure, I can read the source code at the system’s core, but I often struggled to rapidly acquire mental models adequate for even small improvements. I’ve long dreamed of a form of just-in-time learning that would enable a 1x developer like me to rapidly orient in a complex and unfamiliar codebase. I don’t expect a 10x boost, but even 1.5x or 2x will change the game for me and millions like me.
Expeditions into the Core
I would rate my first excursion into the Steampipe core application as a qualified success. The goal was to create a new flavor of dashboard widget: a slider. Relevant components are a Go-based Steampipe process that cooperates with Postgres and launches an auxiliary Go-based dashboard server, a React/TSX-based dashboard viewer, and dashboard widgets written in an HCL/SQL dialect. These components talk to one another using TCP, HTTP, and websocket.
Asking LLMs to implement a new slider widget, given examples of existing widgets, was (for now) a step too far. I had to walk a winding path to arrive at a working first iteration. But I wouldn’t have wanted to walk that path without their help! I got there more slowly than I’d have liked, but more quickly than I otherwise would have. Now that I’ve blazed a trail, and marked it with transcripts that document various points along the way, I feel well-equipped to reorient and dig deeper the next time around. Throughout the process my conversations with the LLMs mattered as much as the working code they helped me write, and maybe more. We talk to the rubber duck to clarify our thinking. When the rubber duck talks back, it becomes a far more powerful tool for thought.
I chose a simpler route for my next expedition. Here’s the backstory: Steampipe can report the results of queries and benchmarks in HTML format, but doesn’t decorate elements with identifiers that would enable direct links identifiers into sections of the report. I thought a custom output template could meet the need, but found that the functions available in the template language didn’t include the ones needed to transform an arbitrary string into an HTML id attribute. So the question became: “How do I add a function to the template language?”
When the LLM Knows Your Code
You can go a long way just pasting prompts into ChatGPT-4, but you can’t yet prompt it with your entire codebase. GitHub Copilot can see your files, and may (who knows?) acquire additional context from its backend. With Cody, it’s explicit: If Sourcegraph has indexed your repo, Cody can enrich prompts with context provided by the backend’s code intelligence engine. Such enrichment portends an era in which LLMs don’t just help us write code, they also help us explain and document it.
My widget exploration didn’t yield a coherent and useful explanation on the first try, though I’ll revisit the experiment because things change quickly. Meanwhile, in the more limited scope of the template subsystem, I scored an immediate win. I began by asking Cody: “Where are the source files that implement templating for control exports?” We should probably include the result in our official documentation, since it’s a good explanation of how templating works in Steampipe.
LLMs being LLMs, filenames can sometimes be hallucinations. But Cody detects and warns when that happens, and in this case the names are all real. template_functions.go was exactly where I needed to dig.
Note that while Cody is most well-known as a VSCode (and now JetBrains) extension, you can use it directly on sourcegraph.com; where, because the company kindly agreed to index turbot/steampipe, it knows about the code and can use that knowledge to enhance its prompts. This points to a new advantage for open source projects: If your code is open, you can make enriched LLM assistance available to everyone — including the long tail of would-be contributors.
Writing a Custom Template Function
My first thought was to add a safeFragmentId function to the template language, in template_functions.go, so the author of an output template could use it like so:
|
1 |
<section class="control" id="{{ safeFragmentId .Description }}"> |
Here was the initial prompt:
I’m the author of an HTML output template that transforms results exported from a Steampipe control run. I would like to be able to link to sections within the HTML output. Currently, those sections are rendered without ids that can serve as link targets (fragment identifiers). The functions available in the template language, as defined by the sprig package, aren’t sufficient to transform an arbitrary string, like the title of a control, into a valid DOM id that can serve as a fragment identifier. How can we provide a safeFragmentId function to do that?
All three LLMs produced variants of the same kind of solution, involving a regex-based search-and-replace that matches non-alphanumeric characters and replaces spaces or hyphens. Then I asked each to write tests for its generated function. All the tests matched my intuition that a good fragment identifier ought to begin with a letter, exclude special characters, normalize to lowercase, replace spaces with hyphens, and not exceed some reasonable length. But hang on, what’s that intuition based on? A bit of old-fashioned research led me to discover that things changed with HTML5: you can now use almost any character. Just because you can, though, doesn’t mean you should. Most sources recommend a more conservative approach. Did the LLM-written tests reflect that implicit consensus? I suspect so, but I don’t know, and that’s just one of many fascinating meta-questions that arises when working with these tools.
The most aggressive tests were Cody’s, but they weren’t always correct. One, for example, apparently intended to enforce a length rule, but expected an output of the same length as the input. I considered all the suggestions and arrived at an adjusted short list of tests.
|
1 2 3 4 5 6 |
{"", "id"}, // handle empty string {"'Single' & \"Double\" quotes", "single-and-double-quotes"}, // quotes {"Now \\&@#$%^&! then", "now-then"}, // other excluded characters {"Mixed CASE", "mixed-case"}, // case insensitive {"Spaces and_underscores", "spaces-and-underscores"}, // spaces and underscores {"abcdefghijklmnopqrstuvwxyz 0123456789 abcdefghijklmnopqrstuvwxyz 0123456789 abcdefghijklmnopqrstuvwxyz 0123456789", "abcdefghijklmnopqrstuvwxyz-0123456789-abcdefghijklmnopqrstuvwxyz-0123456789-abcdefghijklmnopqrstuvwx"}, // truncate long lines |
Now the prompt became: “Please adjust the safeFragmentId function to pass these tests.” All three LLMs fared poorly on this task, even after repeated prompts in which I reported test failures and asked for retries. I suspect that kind of feedback loop will soon be automated, as LLMs are empowered to wield external tools. In that case, will they tend to converge on working solutions or get lost in the weeds? That’s another fascinating open question. Meanwhile, I settled for a hand-tuned version of the function that passes the hand-tuned tests. With that in place, I was able to run a steampipe check command, export results to HTML, and link to section headings by way of generated id attributes.
Rethinking the Solution
Although the LLMs often suggested problematic code, our conversations helped me think about what I even meant by a “good” identifier, and to evaluate tradeoffs between HTML 4’s stricter and HTML 5’s looser rules. But there was another factor I’d failed to consider. As it turned out, a maintainer of the repo preferred not to register a new function, but rather to generate identifiers using existing functions if possible. I’d overlooked that possibility because I’d found that functions to transform and truncate strings weren’t available in our template language. But I should have looked more closely at template_functions.go! There are only a few custom functions defined there, and they’re documented as “ones we couldn’t find in sprig”. However the sprig library defines a lot of functions, only a few of which were being mapped into Steampipe’s templating subsystem. If we include the unmapped functions lower, replace, regexReplaceAllLiteral, and trunc, and that would enable the templatea author to write code like this:
|
1 2 3 4 |
{{ $loweredAndHyphenated := .Description | lower | replace " " "-" }} {{ $nonAsciiRemoved := $loweredAndHyphenated | regexReplaceAllLiteral "[^a-zA-Z0-9]+" "" }} <section class="control" id="{{ if $nonAsciiRemoved }}{{ $nonAsciiRemoved | trunc 100 }}{{ else }}{{ $loweredAndHyphenated | trunc 100}}{{ end }}"> |
Not being familiar with Go’s text/template package, I failed to come up with that syntax on my own. But my assistants quickly guided me to a working solution. If we adopt it, we’ll only need to add four existing sprig functions to the allowed list of functions that Steampipe makes available to template authors.
Finally, I asked the LLMs to evaluate the tradeoffs between method 1 (custom function) and method 2 (existing functions) and make a recommendation.
Copilot: I would recommend Method 1 if you prioritize ease of use and customization, and Method 2 if you prioritize using existing functions and avoiding custom code. Ultimately, the choice between these methods depends on the specific needs and preferences of your project.
ChatGPT: I would recommend Method 1, i.e., providing a new custom function safeFragmentId. This method offers a more maintainable and user-friendly approach. It encapsulates the complexity, which is advantageous for both ease of use and maintainability. While the preference of the Steampipe maintainer is to avoid custom functions, it is important to weigh the long-term benefits of maintainability and simplicity.
Cody: I would recommend Method 1 — adding a custom safeFragmentId function. While it does require adding a custom function, the benefits to the template author and encapsulating the logic outweigh the downsides in my view. The function can be well-tested and maintained to handle edge cases. And if sprig releases a comparable slugify or safeID function in the future, Steampipe could deprecate their custom function in favor of that.
Discounting Copilot’s equivocal answer, the consensus favored method 1: a custom function. The maintainers, however, favor method 2, and it turns out there’s another way to manage its complexity: include the id-generating code in the standard HTML template so no customization is required. I’ve yet to see an LLM come up with an idea like that. But if our conversations with them can help spark such ideas, such conversations are well worth having.
