Scrolling TikTok over the holidays, and I was served a video by Thomas Schlossmacher, a San Francisco-based founder who makes high-quality content about new developments in AI and tech. In five-minute clip, he builds a chrome extension from scratch with the help of a couple AI products. He builds a tool which turns highlighted text into a Perplexity search, breaking it down step-by-step.
As someone who never focused on learning software engineering skills, I wondered whether I'd be able to get the same result. I decided I would make an Anthropic developer account and use the Claude API to make a design analysis chrome extension for critiquing and evaluating individual web pages.
This process required a very specific skillset and methodological approach. It certainly diverged from my standard double diamond product design protocols.
This project was simple enough that I was able to wireframe it in my head: just a couple screens and a handful of components. Because I don't know really how to code, I had to intuit how to get the LLM to do what I wanted even when I didn't know the technical terms for desired outcomes. If Claude got stuck in a feedback loop debugging something, I would switch to ChatGPT and triage between them, occasionally Googling things.
It might sound trite, but actually getting the Claude API to work and talking to it through the VS Code terminal was a special experience – it reminded me of the feeling of customizing MySpace HTML as a kid. Before beginning this project, I conceptually understood that it would be possible, but actually doing it was something else.
Because I was doing this for the first time, my primary goal was to get a lay of the land for this new AI-assisted PM/designer/engineer hybrid role that everybody is talking about.
Nonetheless, there is modest use value in what I built. Assuming that you want to generate an AI design analysis of a given page design, it fast-tracks the process of taking a screenshot, switching to the right tab, uploading the screenshot, typing the correct, detailed prompt, and then getting the results.
Copilot initially had me create the extension without a content.js file in the directory, so when I finally got the earliest iterations of the extension to work, they had one fatal flaw. Instead of analyzing the web page at hand, the extension analyzed itself, which was a bit surreal to witness.
Progress and confusion
I spent a few hours arguing with Copilot, trying to get it to exit its feedback loop. I would copy and paste bug after bug from Chrome dev tools, and it would try all kinds of fixes aimed at the wrong problem, and I didn't know better than to go along with it. Once I set up the content.js file and specified the object of the analysis Javascript function with more specificity, things got rolling again.
I wasn't really doing software engineering, because I didn't know how every piece of my code worked. I was more of a system manager, letting automated processes do their thing, making them talk to each other and sort things out. I directed these various tools and intervened when outcomes diverged from expectations, which happened quite often. I gradually built up know-how for guiding these tools toward intended outcomes efficiently.
It is challenging to define exactly what these LLMs struggled with the most. They often forgot important context, and attempted to double down on genuinely counter-productive changes that I explicitly rejected. They seem to have been trained for a very specific model of efficiency that consistently resulted in unfavorable product outcomes.
Most importantly, I found that beyond a few superficial details, they were not good at accounting for user needs and goals. They also weren't great at thinking visually while generating code, which is interesting because the extension's analysis capabilities are not bad. The output may not always be information-rich, but it is generally accurate.
Visual design
Once I got the extension working reliably, I started thinking about polishing the UI.
Working on branding for such a small project was kind of a fun exercise: I wanted to capture something of the strangeness of interacting with this anonymous, stochastic helper.
When I find the time, I will properly add animations and microinteractions. I'd also like to turn it into a Figma plugin.
