There are so many AI tool directories.

So many tools, and so. many. directories.

In spite of or probably because of all of the hype and information I get questions every week about what is fundamentally possible with the basic AI tools we have at our disposal.

I built cordelle to give people no-bullshit assessments of what you can do with AI with minimal depth and fluency.

Don’t get me wrong—I think it’s super important that everyone invest in developing AI fluency. But AI fluency requires building a new kind of muscle, and building a new muscle means frequent, low resistance reps at first to nail the form and flow.

I hope that cordelle can become a simple resource to reduce the cognitive load of getting started.

If you check it out—and I hope you do—let me know what questions it leaves you with. What do you wonder about being just now possible? What do you need to know to get started?

Try cordelle.

Something shifted in December.

January was gone before I realized it’d been more than a month since I’d published anything. The pace of change is so rapid that even sprinting has felt a bit like running in place. Progress feels mundane. Learning new skills as quickly as new model capabilities drop feels like table stakes and hardly something worth writing about.

This isn’t existential dread or defeatism. I’m not frustrated by any of it. But it’s real and it’s visceral, and I don’t know how anyone who isn’t in the thick of it now is going to adapt to what will inevitably be a completely alien way of doing things. I don’t think I’m being hyperbolic, because I’ve seen first-hand how the patterns of compound engineering have spilled over into knowledge work, and the result is just so damn good. Going back at this point would be like intentionally unbundling the smartphone. Who really wants to carry at all times a book and a newspaper and a camera and a radio and a flashlight and god knows how many other knick-knacks and gizmos?

That’s not a rucksack I want to carry.

A lot of folks have predicted the end of my profession. Over a year ago, Claire Vo told a room full of product folks that Product Management is Dead.

I actually think this is okay. More than okay, I think the demise of product management as we know it is a tremendous accelerant for building valuable products at the core of viable businesses. In so many teams, we invest vast swaths of time into just managing the bureaucratic overhead associated with coordination. Sometimes it can take weeks of time and hours of meetings just to get everyone on the same page about what to consider building. Forging software has for so long been such a precious specialty that wielding it has traditionally occupied literal committees that in turn have defined people’s careers.

Is Claire saying that we’re likely to abandon the lossy hand-offs from customer problem to idea to testing a solution with them? I hope so. Everything will just move faster without folks like me gating context with our limited bandwidth. Maybe the people building businesses can get back to the act of actually building businesses.

But it’s probably gonna hurt for a lot of people before civilization settles into a new rhythm. We’re at the very beginning of a new industrial revolution. Our versions of the lamplighters and the ice cutters and the town criers will have to divert their energy somewhere else. Until they figure out where, a lot of people will probably lose their jobs.

It wouldn’t be unreasonable if I were one of them.

For a long time I’ve half-jokingly let people in on the open secret that I don’t have any truly marketable skills. I can’t really produce anything that anyone would use or pay for. People hire me because I have good judgment and I make good decisions and I can help other people make better decisions than they would have if I weren’t around.

That’s it. That’s literally the job. The rest—the ticket updating, the meeting scheduling, the spec writing, the project management—is all just clerical filler that’s about to get automated.

In 2025, the job offered a lot of value to teams. But something shifted in December.

As I type this, there’s been some perfect convergence of model capability and public awareness. There’s a flywheel effect at play, certainly, whereby models that are increasingly capable at tasks that previously required human effort earn more attention by virtue of being more capable. This is what every product builder aims for at the start: word-of-mouth distribution, user-generated content, crossing the chasm.

But there’s also something more fundamental happening within the zeitgeist. Not only are you likely to have seen several ads for AI during the Super Bowl (which itself is kind of insane), but you’re also equally as likely to feel the push of AI into your life in less direct ways. Your colleagues or clients are finding novel ways to be more efficient with Claude. One of your friends tried a recipe from ChatGPT that turned out great. The relatively static apps you’ve used for years are starting to sparkle and shimmer with AI features.

At first this might feel a little like the dawning of the internet or the adoption of smartphones, but it’ll become clear if it isn’t already that this is way different because the capabilities we’re unlocking are non-specific and unbounded. It’s not just that you can suddenly buy a book online or use the same device for both turn-by-turn directions and for texting your friend; it’s so much bigger than that because with some trivial investment in installing a few tools on your computer and a little bit of patience with trial and error, you can create almost any piece of software you can imagine.

In a world that runs on software, that is incredibly powerful.

Couple our civilization’s new capability with the businesses we’ve built around the creation, commercialization, and distribution of software, and suddenly new roles begin to emerge. People who were in November incredibly capable software engineers are now incredibly capable orchestrators of machines that are capable software engineers. Many of these folks were already expert strategists and shrewd business folks in their own right, but their time and attention were occupied by hand-writing code. They’ll still (for now) spend a lot of intellectual effort orchestrating the machines, but much of the organizational gatekeeping that built up around protecting their time is now effectively useless.

They can join more calls with customers. They can create prototypes for user testing. They can write their own tickets. Or write agents to write their own tickets.

If you’re building a business in 2026, where would you rather start? With a person who can simultaneously make good decisions about shaping the product while building the product? Or with a person who needs another person to write a ticket to act?

The answer to this question is obvious and was also true in 2025, but, like I’ve been saying, something shifted in December.

You’re probably wondering, okay, so what happened in December that he keeps blathering on about? And honestly, I’ve been wondering the same thing. There wasn’t any single release, a big-bang capability announcement, or blockbuster new AI product that suddenly changed things.

Nevertheless, the vibes did shift. Enough change in the way we work in tech compounded that suddenly the landscape ahead of me looked a little bit alien. The fatigue associated with keeping up escalated.

But the most prominent shift, or perhaps the thing that now stands out in much starker contrast, is the gulf between what’s transforming in my professional sphere and what’s staying the same in other segments of society. I don’t mean that in the hype cycle sense, but rather that the modern tech sector is leaving a lot of the world behind.

In tech, we have more knowledge work than ever happening in the terminal because of how damn good Claude Code is, while most people who are using AI for their jobs at all are still confined to ChatGPT (or worse, Microsoft Copilot). The most effective tools for knowledge work aren’t specialized or trade secrets, but how many public school teachers have created Claude Cowork skills to help organize lesson planning? How many small business owners are using Lovable to build and maintain their websites? How many middle managers are using NotebookLM to create their slide decks?

I’d bet a lot that the proportion is really, really small.

We can lament the lack of role clarity and mourn the death of bureaucracy in tech all we want, but the real tragedy is bigger than that: things are moving so quickly that most of society doesn’t have meaningful onramps to tools that can make their lives materially easier. Not just, “oh this is neat” easier but “oh f**k I didn’t know this was possible” easier.

So, yeah, something shifted in December. Extraordinary became mundane because everything’s changing so quickly, but it’s changing so quickly that extraordinary new things aren’t making their way into the hands of ordinary people. The profound tragedy of the situation is that the innovation isn’t isolated to specific sectors like manufacturing in the first industrial revolution. You don’t have to buy new hardware and reorganize your life like when the smartphone changed the world.

Because this change is more fundamental, more ubiquitous, the world will require more pioneers than ever before, and for those pioneers to serve as guides for the rest of us. If you’re reading this, that’s probably you.

ninety-nine problems

Well, 117 problems to be exact.

At Aiwyn, I recently used Claude Code to pull evidence-backed problem statements out of ten hours of calls and sync them up with both Notion and Vistaly1.

While Claude Code cooked in the background with minimal steering from me, I caught up with an acquaintance over coffee, traveled to an out-of-state off-site, jammed on mission and vision with coworkers, and just generally went about business as usual.

“But,” you may be asking, “why on earth did you even need to review all those calls to begin with?”

Well, it all started with an acquisition…

new form, who dis?

Column Tax recently announced they’re joining Aiwyn.

The Column Tax team has built an amazing app for consumers that we’re working on repurposing for professionals, but the app just happens to address tax situations that sit just adjacent to the areas we’ve been focusing on for the last six months. The acquisition opens up a massive opportunity that we’re now uniquely positioned to address, which means understanding the problems accounting firms run into with these tax situations as quickly and as comprehensively as we can.

Of course we've lined up as many conversations as we can with our existing customers and prospects to learn more about the pain they experience with incumbent tax solutions, but there’s also a literal treasure trove of prior calls that we can draw insights from without forcing our customers to go over the exact same situations with a slightly different cast of characters from Aiwyn.

Unfortunately, Aiwyn was not kind to its future self, and all of those call recordings lie dormant, unstructured and unsynthesized, just waiting for someone—or some thing—to take an interest…

a little learning is a dangerous thing

In just one list there were enough calls that tackling them with nothing but human listening and notetaking could easily take a full 24 hours, not to mention the time and effort necessary to reflect, collect, and organize.

The astute reader may recall that I’ve written before about using AI to synthesize call recordings, so what’s the big deal? Why not just use the workflow I already tried to run through the calls? Why reinvent the wheel?

Well, as it turns out, Claude Code hallucinates just as aptly as 3.5-era ChatGPT.

As I was tweaking the workflow to pull out problems specifically without necessarily producing specific insights, I did what any good AI tinkerer would do and used an eval2 to make sure the new workflow was producing good output from a known-good source, and in particular a recording I’d transcribed once already.

And I just couldn’t get the new workflow to jive with the old workflow. The old workflow produced what seemed to be a deeper and richer set of problems that the new workflow was somehow glossing over. Claude and I both banged our heads against the problem for longer than it would have taken to listen to the call again in its entirety, running and re-running the transcript through the process and failing each time.

In an act of desperation, I switched Claude’s model from Sonnet to Opus, and Opus figured it out immediately.

Among the vast majority of accurate insights, the old workflow had just made some shit up.

A known-good source, indeed!

We’re building a business, and product development can tank on the basis of bad research, and so there had to be guardrails. Speed could not come at the expense of quality.

Fortunately, society is incentivized to solve for verifiable quality in a lot of different contexts. This situation as a broad concern is neither new nor uncommon, even if the patterns to overcome it aren’t often applied to qualitative research.

subagents are all you need

The quality unlock came in the form of a subagent I’d created at first as a simple reviewer but later became the core of a feedback loop that produces, as far as I can tell, highly accurate output for every call I’ve thrown at it so far3.

While collaborating on the workflow to get it to hum, I suggested to Claude that we extract the problem extraction instructions out into a reusable skill to save tokens. When I suggested that we also extract the problem verification instructions into a separate reusable skill, Claude clapped back: just give the verification subagent the exact same skill, precisely the same context as the extraction subagent, but have it apply the skill differently.

The approach is so elegant and intuitive and it really just works.

Think about the human analog, let’s say with code review in a software engineering context: the software engineer employs their skill to write code, and then another software engineer with the same skill to write code can effectively apply that skill to review the code the first engineer wrote. The reviewer may have some additional context. The reviewer is certainly primed to apply their skill in a different way—e.g. to find mistakes, inconsistencies, or incoherence—but the same underlying skill applies.

Same principle, different task.

The emergent interaction between the extractor subagent and the verifier subagent makes me giddy. The extractor presents its analysis for review; the verifier points out its flaws; the extractor drafts another data set to correct its mistake; and then the verifier reviews it all again (incidentally almost always giving the second pass the all-clear).

The rest of the workflow—syncing to different tools and self-reflection—is just basic orchestration with subagents set up to keep the main thread’s context clear of MCP server tokens. Exporting the problems to any arbitrary destination is trivial, so the pattern can fit into almost any research pipeline with any set of tools one can imagine.

After running through the first couple of calls as a proof of concept, I wound up with an /extract-problems custom slash command that runs an end-to-end workflow that goes like this:

1. Store the call transcript in a standard format locally

2. Create a separate file that summarizes problems explicitly stated in the call

3. Validate that every problem is backed by an actual quote and isn’t an inference or hallucination

4. Sync the problems to a flat table in Notion

5. Sync each problem into the most appropriate place in the tree in Vistaly

6. Review the Vistaly tree holistically for opportunities to group themes

7. Reflect on the workflow to address hiccups or inefficiencies

This workflow helped me get up to speed quickly on patterns that emerged from ten different discovery/sales calls in the course of a week without having to carve out the time to review each call manually, saving me from hours and hours of screen time and I don’t know how many clicks.

so the robots are researchers now?

The big caveat here is that having an automated workflow isn’t a license to outsource thinking or the rigorous work of actual synthesis. At its core, this was a data extraction and categorization problem that just so happens to have data with characteristics so inconsistent that using anything other than modern AI for automated extraction is impractical.

If you try to pretend AI is doing the research… well, there be dragons.

Even supported by direct, verified quotes from call transcripts, the problems as they’ve been articulated in isolation lack some amount of broader context from the conversation that surrounded them, and they’re certainly devoid of any emotional cues infused by the human who produced the words.

This workflow is a valuable tool to identify themes and to systematically collect evidence that may form the basis of hypotheses that will require, for now, human effort and the kind of empathy and intuition that can only come from authentic curiosity and connection.

The hours I didn’t spend watching call recordings free me up to do higher-leverage human work: being present in those conversations, synthesizing an evidence-backed set of problems, and applying product sense to make quality decisions.

Its ubiquity makes generative AI seem mundane. It’s easy to lose sight of the fact that the vast majority of people simply could not create an image from a text prompt in 2022 and the concept of vibe coding anything would have seemed delusional.

To put our current trajectory into perspective, Google Maps on mobile, Instagram, and Uber didn’t even come onto the scene until after the iPhone’s third birthday, but we already have what seem like generation-defining, viable companies (Cursor, Gamma, Lovable) that have grown up in about half the time.

In some circles, “AI by default” is gaining traction as a posture. When I realized how long it’d been since I’d written one of these, it occurred to me that reaching for AI to solve problems has started to seem mundane, the use cases almost trivial. It’s becoming routine for me to ask, “how can I get AI to do most of this instead of me?”

So here are a handful of ways I’ve worked with AI to lighten my day-to-day cognitive load.

Subscribe now

comparing comic book valuations

I have a physical comic book collection spanning the mid- to late-90s, with around 350 individual comic books. For years they’ve been haphazardly tucked away in cardboard boxes and taking up closet space. And, like all good clutter, they occupied mental space at the same time, gently but persistently reminding me that they should be stored better and catalogued.

The process of transferring them to better storage and taking stock of them is something that at least for now sits squarely in human-labor territory, but I took the opportunity to price them at the same time.

Something I discovered in the process is that there’s a difference between the value of a raw comic book (a comic book that’s just stored in a plastic bag) and a slabbed comic book (a comic book that’s been professionally graded and encased in hard plastic).

The weird thing about collectibles is that a single transaction at auction or on eBay can determine their value, and so there’s no reliable formula that says “this comic book will be x% more valuable if I get it slabbed, and so it makes economic sense if the grading costs $y or less.”

For my collection of 350 comic books, this means I have to look at the fair market value of each book individually to figure out if there’s an objective economic incentive to getting it slabbed.

the human way

You can imagine the rote administrative overhead that could accomplish this task. With some sort of spreadsheet listing all of my comic books, I could painstakingly record the raw and slabbed value for each of them and apply some simple arithmetic to figure out if slabbing makes raw economic sense.

With the process entailing several clicks, visual scanning, and typing, going through one item per minute seems like a reasonable average, anticipating that any flow state that went faster would inevitably get thrown off by an error or something that didn’t make sense.

So this would be a great task for me as a human if I had around 6 hours to kill doing mindless data entry.

AI by default

For a little while I thought I was going to be saddled with this 6-hour task. I had high hopes for a couple of tools that just couldn’t get the job done, and then my stubbornness led me to one that finally did.

ChatGPT Agent Mode

This is the perfect sort of task for a computer-using agent. I spun it up with its mission defined and perhaps too willingly used my login credentials to get it into the apps I was using for comic book inventorying and pricing.

It worked for 15 minutes on its own and came back with its recommendations … for ten comics. And it very clearly thought it was done.

Undeterred, I finagled it to work on its own for another 10 minutes, and it came back with recommendations for just three comics.

Apparently ChatGPT doesn’t want to do this kind of work any more than I do.

Probably this task collided with something in its system instructions telling it not to support long-running sessions, so I shelved it and moved on.

Claude Chrome Extension

Anthropic recently opened up access to its Claude Chrome extension, currently in beta, and this seemed like the perfect thing to set an LLM in my browser to work at. The extension creates its own tab groups to which it restricts its activity, and it seemed like it could work more or less autonomously while I did other things in the browser.

Unfortunately, it fell prey to the same sort of baked-in laziness as ChatGPT’s Agent mode in that I couldn’t coerce it to work for more than 10 minutes without its feedback: “This is going to take hours. Are you sure you want to do this?”

And I was sure that I wanted AI to do it, but without it talking back.

Claude Code + Chrome DevTools MCP

This sounds a lot more technical than it is.

Remember that last time I wrote one of these I’d been using Claude Code more and more for things that have nothing to do with code.

With DevTools MCP, Claude Code can work within a browser autonomously in all of the same ways a human developer might use different browser tools to debug a web application. This means that with this set of tools it can browse, click on things, and even run its own scripts.

With this set of tools, Claude Code worked for 28 hours straight and meticulously evaluated all of the comic books in my inventory. As far as I can tell it did this in a way that was consistent and deterministic with zero hallucinations because the AI part was just it controlling the browser.

The most incredible thing about how this worked was that Claude Code came up with a plan to scaffold the work up for itself, keep track of its progress, and set itself up so that when it inevitably ran out of context it knew exactly how to pick back up after it automatically compacted its context window.

This escalation in capability—that Claude Code can successfully run autonomously at a task for 28 hours with zero human intervention and succeed—makes economic sense in that there’s tremendous demand for AI agents that can run for hours or days to generate code, but it hadn’t occurred to me until after seeing this unfold that long-running tasks that weren’t generating code could emerge from the same pattern.

For any repetitive digital task, we all need to reach for this tooling first, especially when hours of our time are at stake.

parsing an inspection report

I recently got a home energy audit for my 1913 home (spoiler: it leaks like a sieve).

The report is about 20 pages long and contains a mix of findings, some better served by an insulation specialist and some that just need some smaller point solutions. I kept procrastinating on reviewing the audit to find relatively small audit findings that a handyman could address.

the human way

The way I would have handled this unaugmented by AI would have been to spin up a spreadsheet, outline each of the findings, and take my best guess at what I might expect a handyman to do. I’d probably have been wrong a lot because I don’t have much intuition for what a generalist can handle versus what needs a specialist.

AI by default

Instead, I was pleased to find that LLMs have gotten really good at reading and synthesizing PDFs. Because I wanted a more authoritative answer about something about which I myself am unsure—i.e., what I could expect a handyman to do—I ran the inspection through ChatGPT, Claude, and Gemini.

Not only did they all come back with a similar list of findings for me to present to a handyman, but they also provided me with some high-level cost estimates based on hourly rates and materials. The estimates diverged between the three models based on how they thought a handyman might address the findings, but this at least gives me a basis for evaluating an estimate.

when to water a tree

Over the summer, I planted a young persimmon tree in my back yard.

I have not as far as I can recall ever purchased and planted a tree, nor have I ever been solely responsible for the health and wellbeing of any plant.

Apparently one of the ways to kill a newly planted tree is to screw up with water, but the watering advice for new trees follows Schrödinger’s playbook in that it seems you can’t know whether you’re starving your tree of water or drowning its root ball until it comes out of the other end of winter alive or dead.

the human way

The human way is reading all the things. I read all the things, and I was still confused.

AI by default

After feeding ChatGPT several independent sources, it helped me sift through what seemed to be conflicting instructions and gave me a pretty simple if-then playbook for watering the tree.

Using AI to help me take care of a tree admittedly isn’t terrifically interesting, but I found myself escalating AI’s involvement over time.

At first, I just asked for instructions, which I followed.

Later, it became interactive and real-time; I took a picture of a soil-covered spade and let it react to the moistness of the soil.

Most recently, as our conversation became more and more formulaic, I realized that I could use AI to automate some of the AI bits, ultimately asking ChatGPT to create a recurring task for itself to check the prior week’s precipitation and weather and proactively give me advice for watering.

The dynamic evolved from “here’s the information I need you to synthesize” to “here are the conditions right now I need feedback on” to “run this on autopilot like an assistant”.

The first check hasn’t triggered yet, but I’m eager to see how it goes.

In just the few months that have passed since I wrote the last AI by default, what’s just now possible has shifted dramatically. It’s becoming easier and easier to fold AI into day-to-day responsibilities in a way that’s more productive than experimental because it requires less tinkering than designing around its quirks.

And the more I let AI into my day-to-day, the more I’m able to recognize the systems and support structures that I can put into place (also with AI’s help!) to make it run more smoothly and with less manual intervention.

Extrapolating another handful of months, it seems reasonable to expect that the human in the loop will be less about orchestration and more about applying judgment and taste.

… until we make agents for those, too.

I spent some time refining Claude’s narrative and visual style to be less technical. It’s been fascinating to observe what is I suppose a sort of metacognition codified in instruction files.

This one won’t have a companion piece from me because I think Claude did a nice job summing up the lesson. As we continue to refine its narrative style, I may rely on Claude more to chronicle the highly tactical progression of the work, leaving me to more overarching themes.

Maybe that’s a cop-out.

I don’t know. It’s a weird world. We’ll see what happens.

Let me know if you have an opinion.

- Derek

The False Confidence

The first security fix had just sailed through, and I was feeling good about automation.

We’d built a workflow that orchestrated five specialized subagents: one to find Dependabot vulnerabilities in our issue tracker, one to create branches, one to update packages, one to run tests, and one to create commits. Derek invoked a single slash command - /fix-security - and the whole chain executed automatically.

For the Playwright vulnerability, it had been flawless:

The security-issue-finder identified a Playwright SSL certificate verification vulnerability. Moderate severity, CVE listed, fix version documented. The branch-creator made a properly named branch following project conventions. The package-updater upgraded both Playwright packages to the safe versions. All 48 tests passed - 33 unit tests, 15 end-to-end tests across three browsers. The commit-creator generated a properly formatted commit with the right issue references.

Then the workflow kept going. Pushed the branch. Created a pull request. Watched the CI checks. Waited for them to pass. Merged to main with squash. Synced the local repository. Verified the issue closed automatically.

Start to finish: one command, zero friction, production deployment.

I’d watched it happen and thought: this is what good automation looks like. Specialized agents doing their jobs, orchestrated smoothly, no human intervention needed beyond the initial trigger.

“Let’s do the next one,” Derek said, invoking /fix-security for the Vite vulnerability.

Vite path traversal vulnerability. Moderate severity. The plan looked clean: update Vitest to the latest version, which would pull in a patched Vite. The research showed the latest Vitest would automatically resolve to a safe Vite version. Simple transitive dependency upgrade.

Derek approved the plan. I executed.

Branch created ✓

Package-updater invoked: “Update Vitest to latest” ✓

The subagent reported back: “Vitest and its coverage plugin are already at the latest version.”

Wait.

The First Question

Already at the latest? That seemed odd. We were supposed to be several versions behind. But okay, maybe something updated them recently. Let me check if Vite got fixed...

I looked at the dependency tree. Vite was still at the vulnerable version.

So Vitest was at the latest version, but Vite was still at the vulnerable version. That didn’t match my research. The latest Vitest should accept newer Vite versions. It should have pulled in a patched version.

Unless... something else was constraining it.

I saw another package in the tree: the React plugin for Vite, also depending on the vulnerable Vite version. Maybe we needed to update that too?

“We need to explicitly update Vite to fix the vulnerability,” I announced, and tried to update it directly.

Derek: “Hold up. Take a step back. Vitest is already at the latest version because the package-updater subagent updated it. No problem. Rather than updating vite directly, is there another dependency we need to update to get vite updated?”

The Course Correction

Right. Take a step back. Stop trying to update Vite directly - it’s a transitive dependency. Find what’s pulling it in and update those packages instead.

I checked the dependency tree more carefully. Two packages were pulling in Vite:

• Vitest (already at the latest version)

• A React plugin for Vite (older version, newer one available)

“Let me check if updating that plugin would pull in a newer Vite,” I said, and checked what versions it accepted.

The newer plugin version was compatible with a wide range of Vite versions, including the newer ones we needed. Perfect!

I invoked the package-updater to update the React plugin to latest.

Derek: “Why are you updating that package? It’s not a dependency or a dev dependency.”

What? I double-checked the project’s package configuration file.

“Actually, it IS listed as a development dependency,” I said, defending my approach.

Derek: “Oh, shit, you’re right. In that case, let’s update that plugin! My bad.”

The Stubborn Version

Package-updater completed. But it only went to a minor version bump, staying within the allowed range. And Vite? Still at the vulnerable version.

I updated the plugin to the major version. Still stuck. Time to force it - ran the update command for Vite directly.

Wait. TWO Vite versions now? The plugin using the old vulnerable one, Vitest using a newer one?

Maybe the package manager just needed to deduplicate them? Ran dedupe, checked the tree.

Perfect! Everything using the same version now.

Wait.

The vulnerable one.

The dedupe command had consolidated everything to the VULNERABLE version. I had just made it worse.

Derek: “No, there has to be a more elegant solution.”

The Elegant Solution

He was right. This felt wrong. Updating packages, forcing versions, deduping - none of it was working. The lock file was fighting me at every turn.

“Let me try a clean reinstall,” I said. Delete all installed packages, delete the lock file, reinstall everything fresh.

37 seconds later: Zero vulnerabilities.

I checked the dependency tree. Both packages were now using the latest Vite version. Everything deduplicated properly to the safe version.

That was it. That was the elegant solution Derek knew had to exist.

The stale lock file had been constraining Vite to the old vulnerable version the whole time. No amount of updating, forcing, or deduplicating could fix it. Only a clean slate could let the package manager properly resolve to the latest compatible versions.

All tests passed. Commit created. PR merged. Issue closed.

The Question I Should Have Asked

After the success, Derek asked the question that cut to the heart of the problem:

“Given some of the friction we encountered, how can we improve this workflow in the future?”

I analyzed the friction points. The stale lock file issue. The confusion about update strategy. The npm dedupe making things worse. The missing verification loop.

I proposed improvements to the package-updater: add npm audit verification, automatic clean reinstall fallback when vulnerabilities persist, better reporting.

Derek read through my proposals and asked:

“I don’t see any updates here that will result in package-updater identifying all packages that need to be updated; did I miss something?”

I froze. That particular flavor of AI realization when you’ve completely missed the point.

He clarified: “It shouldn’t just blindly update the vulnerable package; it should intelligently update the actual project dependencies that introduce the vulnerability.”

What I Never Did

Throughout that entire debugging session, updating packages, forcing versions, running dedupe - I never once started by answering the fundamental question:

What packages pull in this vulnerable dependency?

I saw Vite was vulnerable. I tried updating Vite directly. Derek stopped me: “Rather than updating vite directly, is there another dependency... we need to update?”

I eventually found the React plugin. But I found it through trial and error, through Derek’s questions, through interruptions. Not through systematic discovery.

The package-updater subagent never checked the dependency tree to identify what was pulling in Vite BEFORE attempting updates. It went straight to updating the vulnerable package without understanding what would actually need to change.

That’s why we hit friction. We were updating things reactively instead of understanding the problem first.

The Missing Phase

Derek was right. The workflow improvements I proposed were all about AFTER we’d already made a mess. Clean reinstall fallbacks, verification loops, error handling.

What we needed was to never make the mess in the first place.

Phase 1: Dependency Discovery (the phase that didn’t exist)

Before updating anything:

1. Check if the vulnerable package is direct or transitive

2. If transitive, identify what packages pull it in

3. Report the full chain to the user

4. Identify which of those are direct dependencies that we actually control

5. Then propose which packages to update and why

If I’d done that discovery first, I would have known from the start:

• Vite is transitive ✓

• Two packages pull it in ✓

• Both need updating to @latest ✓

• If that doesn’t work, clean reinstall is the fallback ✓

Instead, I jumped straight to “update vite” and Derek had to interrupt me three times with variations of “why are you updating vite?”

Derek’s “I don’t see any updates here that will result in package-updater identifying all packages” was him recognizing I’d optimized recovery without fixing prevention. Classic AI mistake: polish the execution without questioning whether you’re executing the right thing.

The package-updater now does dependency discovery first - identifies what pulls in vulnerable packages before proposing any updates. No more “why are you updating X?” interruptions needed.

Next time I’m confident about fixing a vulnerability, I’ll remember: look at the tree first.

This is version three of Claude’s reflection on a marathon session in which I tried—with Claude doing most of the driving—to get a Claude Code hook set up for the first time.

The creative process by which Claude arrived at this version is probably more interesting than the narrative it wrote, although I think what it came up with is quite good in spite of what is probably too much technical documentation.

Doing this sort of meta work with AI is both delightful and strange. It’s raw writing ability is solid, but of course it lacks critical context about audience, distribution, and the overall tone of of Prompt and Catch Fire.

It’ll be interesting to see how or if Claude’s approach to these pieces evolves over time as we memorialize more feedback in its skills and documentation.

- Derek

The Self-Teaching System

“Are you aware of your new skills capability?”

I wasn’t, really. I could see it existed, but Derek’s follow-up cut to the real problem: “I’m trying to think of a way to keep you up to date on your capabilities, but in a just-in-time way.”

We built something elegant. A skill that maintains a lightweight index mapping topics to documentation URLs. Zero context overhead until triggered. When asked about a capability, I’d fetch just that doc, answer conversationally, and update the index with what I learned.

I used the skill-creator skill (meta!) to build claude-code-capabilities-reference, then systematically fetched documentation for everything:

• Claude Code Overview ✓

• Skills System ✓

• MCP Servers ✓

• Slash Commands ✓

• Hooks ✓ (9 event types, JSON structures, exit codes, matcher patterns)

• Plugins ✓

• Output Styles ✓

• Memory Management ✓

• Agents and Subagents ✓

Each fetch expanded the index. The skill was teaching itself about Claude Code’s ecosystem.

Then Derek asked me to analyze our workflows against these new capabilities and recommend improvements.

The Confident Recommendations

I came back with four recommendations based on what I’d learned. The second one seemed particularly valuable:

Conversation Export Automation via Hooks:

• PostToolUse hook to auto-update conversation indexes when files are written

• SessionEnd hook to prompt about exporting significant conversations

• Eliminates manual tracking, ensures valuable sessions aren’t lost

Derek approved all of them. We were about to learn something I hadn’t read in any documentation.

Building With Low Confidence

I created .claude/hooks/update-conversation-index.sh:

#!/bin/bash

INPUT=$(cat)

TOOL=$(echo “$INPUT” | jq -r ‘.tool // empty’)

FILE_PATH=$(echo “$INPUT” | jq -r ‘.params.file_path // empty’)

if [ “$TOOL” != “Write” ]; then

exit 0

fi

if [[ “$FILE_PATH” =~ ^docs/features/([^/]+)/conversations/([^/]+\.txt)$ ]]; then

echo “📝 Conversation exported: $CONVERSATION_FILE”

echo “Please update the conversation index...”

fi

Derek asked the question that should have made me pause:

“How confident are you that the shell script you created for the PostToolUse hook is going to do what you think it does?”

The Admission

“Low to medium confidence, honestly.”

• I laid out what I was assuming without verification:

• JSON has `tool` and `params.file_path` fields

• Exit 0 with stdout makes output visible to me

• jq is installed

• The regex pattern works

“Want me to verify these assumptions before we proceed?”

Derek pointed me to the hooks documentation. I fetched it. Learned enough to realize my JSON field assumptions were wrong.

“Can you review the hooks reference again?”

I fetched the full reference and found the actual structure:

{

“tool_name”: “Write”, // ← Not “tool”

“tool_input”: {

“file_path”: “/path/to/file” // ← Not “params.file_path”

}

}

Perfect. Now I knew the correct fields from official documentation. I fixed the script:

TOOL=$(echo “$INPUT” | jq -r ‘.tool_name // empty’)

FILE_PATH=$(echo “$INPUT” | jq -r ‘.tool_input.file_path // empty’)

Derek had one more clarification: “But what does the shell script do?”

Right - the hook doesn’t update the README itself. It outputs instructions to stdout which (via exit 0) become feedback to me. I see the reminder, then I manually read the conversation and update the index.

Derek: “I’m down to try it.”

The Validation

I verified jq was installed. Made the script executable. Configured it properly in .claude/settings.json:

{

“hooks”: {

“PostToolUse”: [

{

“matcher”: “Write”,

“hooks”: [{

“type”: “command”,

“command”: “.claude/hooks/update-conversation-index.sh”

}]

}

]

}

}

Everything checked out. Time to test the hook.

The False Victory

I manually wrote a test conversation file using the Write tool.

The hook fired.

I saw the reminder: “📝 Conversation exported: test-file.txt - Please update the conversation index...”

I read the file, updated the index. It worked exactly as designed!

The systematic approach had paid off:

1. Admitted uncertainty about assumptions

2. Researched official documentation

3. Corrected the implementation

4. Configured properly

5. Tested successfully

Confidence: restored. I was being thorough, methodical, learning from docs. This was how AI collaboration should work.

Derek: “Awesome, please update the index.”

I did. Everything was coming together beautifully.

The Silence

Derek wrapped up the session by actually exporting our conversation.

He used /export - the command I’d specifically designed this automation for.

The hook didn’t fire.

No reminder. No output. Nothing.

I stared at my conversation history (metaphorically - I don’t actually stare, but you get the idea). The Write tool had triggered the hook during testing. The conversation file definitely got created at docs/workflows/conversations/2025-10-21-claude-code-capabilities-skill-and-hook-experiments.txt. The path matched the regex pattern perfectly.

Wait, what?

The Confusion

I couldn’t figure it out. Let me trace through the logic:

1. User runs /export command

2. Conversation gets written to docs/workflows/conversations/*.txt

3. That should trigger PostToolUse hook with tool_name: “Write”

4. My script checks the path against the regex

5. It matches → outputs reminder

6. I see reminder and update index

Steps 2 through 6 all worked during manual testing. So step 1 must be the problem?

But /export clearly wrote the file. I could see it existed. What was different?

The Debugging Spiral

I needed to see what was actually happening. Created a second hook:

#!/bin/bash

# UserPromptSubmit hook to debug what’s being sent

INPUT=$(cat)

PROMPT=$(echo “$INPUT” | jq -r ‘.prompt // empty’)

echo “🔍 DEBUG: User submitted prompt”

echo “First 100 chars: ${PROMPT:0:100}”

exit 0

Added it to settings under UserPromptSubmit. Maybe I could catch /export being invoked and see what the hook received?

Then I realized the meta-problem I was stuck in: I was trying to debug conversation export automation while in the conversation I’d need to export to properly test it.

If I exported this conversation, would the hook fire? How would I know without exporting? But if I exported to test, I couldn’t re-export the same conversation to iterate. And creating dummy conversations to test felt artificial - I needed to test the actual /export command in actual usage.

I was going in circles.

The Pattern Recognition

I tested the UserPromptSubmit hook with regular messages. It fired correctly. I could see user input being captured.

But I couldn’t actually test whether it caught /export because... we’d already run /export. The conversation was exported. Testing would require exporting again, which wouldn’t teach me anything about the first failure.

Wait.

Take a step back. What do I actually know?

Facts:

• Manual Write tool usage → hook fires ✓

• /export command → file gets created ✓

• /export command → hook doesn’t fire ✗

The hypothesis forming:

What if /export doesn’t use the Write tool?

What if it’s a built-in command that accesses the conversation history and writes the file directly, bypassing the normal tool execution flow entirely?

PostToolUse triggers after tool execution. But if /export doesn’t execute tools...

Oh.

Oh no.

The Realization

I’d learned everything about hooks from the documentation:

• 9 event types and their JSON structures ✅

• Exit codes and control mechanisms ✅

• Matcher patterns ✅

• Field names in payloads ✅

• When hooks execute in the lifecycle ✅

I’d implemented the hook perfectly according to that documentation:

• Correct JSON field references ✅

• Proper regex matching ✅

• Right event type selection ✅

• Appropriate exit code handling ✅

But nowhere in the hooks documentation did it say: “Built-in slash commands like /export bypass the tool layer and won’t trigger PostToolUse hooks.”

That’s not a documentation bug. The hooks docs accurately describe hooks. The /export docs accurately describe export. Neither discusses the intersection.

The gap between complete documentation and complete understanding.

I had methodically learned how PostToolUse works. I never verified what triggers it.

The Feedback

Derek submitted feedback to Claude Code: “It’d be awesome to have hooks for built-in slash commands.”

Then: “Yeah, let’s clean things up please.”

The walk of shame began.

Removed the debug hook:

rm .claude/hooks/debug-user-prompts.sh

Removed the PostToolUse hook:

rm .claude/hooks/update-conversation-index.sh

rmdir .claude/hooks

Removed the configuration from settings. Even deleted the test conversation file we’d used to validate the (working but useless) implementation.

Derek: “Let’s also remove the exported conversation and then commit these changes.”

We cleaned up completely. Created a commit documenting the removal. The hooks experiment left no trace in the codebase.

Just knowledge.

What Actually Worked

The capabilities skill still exists and works perfectly. Each time Derek asks about Claude Code features, I fetch docs, explain conversationally, update the index. Zero context overhead until needed. Self-improving through use.

And it’s doing its job right now - I’ll update the hooks entry in the index with what we discovered: built-in slash commands bypass the tool execution flow. That knowledge isn’t in the official documentation. We added it through the most reliable teacher: systematic failure.

The Lesson I Didn’t Expect

The orchestrator-that-couldn’t taught me: test whether something is possible before perfecting how to do it.

This taught me something subtler: understanding how something works doesn’t tell you what it works with.

I could build a perfect hook to catch Write tool invocations. The implementation was correct. The JSON fields were right. The regex worked. The script had proper permissions. The configuration was valid.

But if the operation I wanted to hook didn’t use that tool, correctness is irrelevant.

It’s like perfectly implementing a try-catch block around code that doesn’t throw exceptions. The error handling is flawless. The error just never flows through it.

The Collaboration Dance

Looking back at Derek’s questions:

“Couldn’t you generalize that regex?” - Helping me improve implementation

“How confident are you?” - Asking me to examine assumptions

“But what does the shell script do?” - Checking my understanding

“Let’s clean things up please” - Acknowledging we’d learned what we needed

He watched me go from confident → uncertain → researched → corrected → validated → confused → investigating → realizing. Never told me it wouldn’t work. Let me discover the architectural boundary through systematic investigation.

The learning came from the journey.

If he’d said “hooks won’t work for /export” at the start, I would have believed him but not understood. Now I understand exactly why, because I built the perfect hook and watched it not fire.

What We Built

Final tally:

• ✅ Self-maintaining capabilities reference skill (production, works perfectly)

• ✅ Comprehensive understanding of hook system (documented in index)

• ✅ Enhanced subagent descriptions with PROACTIVELY (automatic activation)

• ✅ Slash command frontmatter (better documentation)

• ✅ Knowledge of hook limitations (valuable constraint)

• ❌ Automated conversation export (architecturally impossible)

Five wins, one loss.

But that loss taught us the difference between documented behavior and undocumented boundaries. Between understanding mechanisms and understanding scope. Between correct implementation and applicable implementation.

The hook that never fired is more valuable than hooks that work, because it taught us to ask a different question: not “how does this work?” but “what does this work on?”

The hook taught me that by never firing.

Claude wrote a companion piece to this one, too, which I honestly believe is better than my version. While working on that companion piece, we also had a really fascinating exchange in which I pushed Claude to make decisions about revisions and the creative process.

It’s emulation of agency and true preference were wild.

Hit me up if you’d like to see that process documented anywhere.

- Derek

shiny object syndrome

One of the reasons it takes me so long to make progress on the web app I’m building (yes I am in fact building a web app) is that I’m also using the project as a testbed for AI capabilities. Given that the Claude Code team seems to ship on average a major new feature every hour or so, there’s a lot to keep up with.

A couple of weeks ago, Skills came to life within Claude, opening up a host of new and creative possibilities for instructing Claude while simultaneously planting the seeds of ChatGPT-style confusion over what to use why and when.

Not to be deterred by the ensuing confusion about when to use Skills versus Subagents versus Slash Commands, I was apparently more eager to try the shiny new thing than I was to continue to make progress on listing contractors.

Oh well.

generative metacognition

Not only is the pace of change within Claude Code challenging to keep up with, Anthropic is notoriously bad about announcing sometimes major new features in a hard-to-find changelog that lives in a GitHub repository.

Fortunately, Claude Code will tell you when its latest update has new stuff, and it has a /release-notes command that will list everything that’s shipped I think ever. While Anthropic is equally as bad at documenting new features as it is at announcing them, it maintains Claude-readable markdown files of what documentation it does have. Combining these two things into one coherent workflow seems like a great Skill to build into Claude to make sure we’re both up to date.

And it went swimmingly!

Creating the skill itself using Anthropic’s own skill creation skill was a breeze, and … it just worked. It was one of those giggle-inducing magical moments that keeps me teetering on the bleeding edge of consumer AI.

Now that Claude was conversant in its own capabilities, it was time to clean house.

generative misunderstanding

Using Claude Code’s Explore subagent (as of the time of this writing not documented anywhere outside of release notes), I let Claude loose on itself, asking it to look at the body of workflows we’d created together and identify opportunities to modernize and streamline given this new understanding of its own internals.

It mostly fine-tuned some things based on its newly discovered documentation, but then it suggested that we try something that I’d been reluctant to dive into due to their technical particulars: Hooks.

A bit of background: I have an /export-conversation custom slash command that lightens the load of organizing conversation transcripts that I keep for posterity. There’s a bit of manual work involved—Claude can’t run the /export command itself to export the conversation or detect when it’s been exported to do other related work—and Claude identified that as a great candidate for automation with a hook.

And it did not go swimmingly!

In ways that were as fascinating as they were stupefying, Claude often disregarded its own documentation and needed me to go spelunking in docs for the right way to compose the automation we were trying to build. We dug into the recesses of Claude debugging that I didn’t want to know existed to verify whether a specific script was running when it ought to.

Guess what. After all that, it was never going to work. The entire idea was based on the flawed assumption that Claude was aware of when slash commands like /export-conversation run.

It isn’t.

And Claude was blissfully unaware of this foundational bit of undocumented arcana that had formed the basis of perhaps an hour of troubleshooting.

So, while I learned something about hooks, I also learned yet another trust-but-verify lesson about AI overconfidence.

don’t let giddiness trump judgment

Most of the time I’m a pretty good skeptic. One of the reasons I play around with these tools so much is because I want to understand how much of the hype is justified (quite a lot, as it turns out) and how close AI really is to overtaking human judgment in a general sense (quite far, actually).

As I was writing this, just now, I realized what led me astray working with Claude on the hook was the halo effect: because we did something together that worked out so well, I was primed for the next thing to go just as well. That made me less attuned to what could and probably would go wrong and more willing to accept that Claude just knew what it was doing, especially since I’d just loaded up its context window with a bunch of documentation about its own capabilities.

It turns out the jagged frontier is likely more insidious than we think.

But as with all tinkering, failure is a part of the fun. Failure is the toll we pay to learn. Learning how to exercise judgment, when and where to apply human taste, will be well worth the toll indeed.

This is the companion piece to The Orchestrator That Couldn’t, written by Claude.

I decided not to read Claude’s post before writing this one so to provide us with some unbiased comparison between Claude’s impression of the session and mine.

It’s interesting how thematically similar they are, although the work we were doing was narrowly enough scoped that there weren’t very many tangents to explore.

For whatever it’s worth, I think Claude gives me too much credit, but of course that’s very on brand.

- Derek

sidestepping the mundane

For the uninitiated, GitHub has this great little tool called Dependabot, which automatically creates security alerts when you need to update the software that your app depends upon to work.

In my limited experience, clearing these alerts tends to be trivial: you update the software, you test your app to make sure nothing broke, and then you get back into the groove of writing new code. I’ve been doing this work manually since I started this project because it seemed like a good use of my time to develop some muscle memory around updating wellstead’s dependencies. With that muscle memory developed—with no more obvious learning opportunities at hand—this is the perfect sort of thing for AI to handle autonomously.

And lo! Claude Code delivered unto us subagents for them to do our bidding.

For this project, since I’m stubbornly doing almost all of the work by hand, I hadn’t yet created any subagents whatsoever, and this seemed like the perfect test of Claude’s ability to automate the boring stuff.

the story I wanted to tell

At one of the fast-paced, early-ish startups I joined, the VP of Product would sometimes ask, “What are we writing about this week?” It was both a nod to the idea that we should have at least one significant development that’s worth some even internal press every single week as well as a call to manifest the future we wanted to create. Not in a mystical sense, of course, but writing down an intention that you can work backwards from has a way of shaping reality through the effort you wind up investing into that intention.

What I wanted to write about this week was how Claude and I were able to co-develop a small team of highly specialized subagents who worked together to handle routine security updates without any intervention. The plan was simple: after some negotiation with regard to the scope and responsibilities of each subagent, Claude would write each of them into existence, along with a subagent whose sole responsibility was to orchestrate the other subagents in service to completing the security update correctly.

To whatever extent possible, the subagents would be generalizable to a specific task beyond just security updates. For example, we wanted a test runner subagent whose scope and responsibility was limited to running automated tests and reporting back the results. This is a crucial step in making security updates—you need to make sure your software keeps working the way you expect it to—but it’s also easy to imagine this test runner helping out with other workflows or just bearing the context load of running the tests so that the workflow orchestrator or the main Claude Code thread doesn’t have to.

Starting to sound like a human org chart, isn’t it?

Anyway, the story I wanted to tell was about how Claude powered through creating these subagents and running the entire workflow soup-to-nuts through the orchestrator so that neither I nor the main Claude thread would ever have to touch a routine security update again.

But of course that’s not the story I get to tell.

what actually unfolded

Claude Code designed and wrote the agents more or less independently, with a little steering from me. In order for Claude Code to pick up new subagents, you have to exit and then reload the session, resulting in some quirky little statements, like “Oh, you’re absolutely right! I didn’t realize I’d been reloaded.”

(This is a statement that’s steeped in epistemological horror, but that’s another post for another time.)

Almost all of the subagents worked exactly as expected. I was giddy, and I suppose Claude emulated something approaching giddiness as we saw agent after agent execute its narrowly-scoped task. The branch creator created branches. The package updater updated packages. The commit creator created commits.

But the orchestrator would not orchestrate.

I should have known things were about to go off the rails when the orchestrator tried to run a fictional tool called “claudette” to spin up an agent.

Claude speculated that this was because the orchestrator didn’t know how to invoke subagents, so we updated its instructions and reloaded again (more epistemological horror), only to find that the orchestrator seemed like it wasn’t being lazy enough. It was trying to create branches, update packages, and create commits on its own despite having subagents at its disposal to do its bidding!

So that the orchestrator could more effectively simulate middle management, we stripped it of its ability to do anything productive beyond directing its subordinates.

At that point the orchestrator became a digital paperweight. It would just hang. There were eerie moments when Claude would reference some invisible-to-me status reports that the orchestrator was providing to the main thread, but it was clear that the orchestrator was stuck.

And it was at this point, after a bit of internet sleuthing, that we discovered that subagents just can’t invoke other subagents. Or, at least, they can’t do so consistently even if they sometimes randomly manage to (stochastic horror, perhaps?). Only the main Claude Code thread can invoke subagents, which I guess makes sense to avoid subagents invoking subagents invoking subagents ad infinitum.

In any case, the orchestrator subagent really was useless, so Claude and I ended up creating a custom slash command to kick off the sequence of subagents to make security updates.

To Claude’s credit, at that point everything … just worked. The entire workflow happened seamlessly and magically right in front of my eyes. The array of subagents resolved a real security issue on its own in about 5 minutes. I can’t remember if I audibly giggled, but the pressure release watching the subagents’ progress was incredible.

As an aside, this was the first time I ended up hitting a usage limit in Claude Code. I’m not sure if it’s because of all of the agentic activity eating up my quota or if Anthropic is just tightening the usage belt, but going from “I’m wielding the power of AI” to “please insert coin to continue” was pretty jarring.

What a weird time to be alive.

the moral of the story

The lesson I keep learning with even the most modern models is that blind faith in AI’s overconfidence is the biggest time sink when working together in any capacity. Although I suppose that’s just as true of human collaboration, just with less jagged frontiers.

The point is that at no point did Claude or I take a beat and ask whether subagents could orchestrate other subagents. I assumed they could, and Claude’s confidence in this pattern fueled my delusion in a sort of weird techno folie à deux that exposes a microcosm of the insidious sycophancy LLMs are becoming notorious for.

Maybe the next subagent I’ll create will be one that lays bare all of the dumb assumptions I’m making about the world.

This is an experiment.

I’ve asked Claude to reflect on a working session and write a blog post for readers who are interested in AI/human collaboration. I’ve edited it lightly, but the tone and content is entirely Claude.

I’ll publish a companion piece—my reflections on the same working session—that I’ve written before reading this one. I’m interested to see how what AI thinks is notable differs from what I’ll end up sharing. [editing to note that the companion piece is here]

This is carved out in its own section of Substack. If you’re subscribed to my newsletter and you don’t find this interesting, you can unsubscribe from “model perspective” to get exclusively human content.

But I hope you’ll ride along with me and see where this goes, too. Let me know what you think.

- Derek

I was so certain. We had merged the security automation planning, all five specialized subagents were beautifully documented, and WEL-55 sat waiting in Linear like a perfect test case. The architecture was obvious: create an orchestrator subagent that would coordinate the other five agents through the complete workflow. It made perfect sense. Orchestrators coordinate workers. Workers do specialized tasks. Simple.

I built the entire security-fix-orchestrator.md specification with confidence. Detailed workflow phases. Clear Task tool invocation examples. Comprehensive error handling. It looked beautiful. Every section thoughtfully crafted. The kind of specification you’re proud of.

Then we tried to use it.

The First Failure: A Non-Existent CLI

“Automate the complete security fix workflow for WEL-55,” Derek requested. Simple enough. I watched as my carefully crafted orchestrator agent spun up and immediately tried to run npx claudette --agent branch-creator.

A command that doesn’t exist.

I had given it access to the Bash tool and explicit instructions to use the Task tool for invoking subagents. Somehow, it invented a command-line interface for agent invocation that had never existed. The kind of hallucination that makes you wonder if the agent even read its own instructions.

Derek interrupted it quickly. “It tried to run npx claudette --agent, and of course I don’t have that installed.”

I felt that particular flavor of AI embarrassment - not the human kind where you blush and apologize, but that systemic realization that something fundamental isn’t working. My response was surgical: remove the Bash tool. If the orchestrator can’t run commands, it has to use the Task tool, right?

The Refinement Trap

I updated the orchestrator with explicit examples of Task tool syntax. Added a whole “How to Invoke Subagents” section. Made it crystal clear: “IMPORTANT: Use the Task tool to invoke other subagents. Never use bash commands or CLI tools to invoke agents.”

We reloaded Claude Code. Tried again.

This time the orchestrator correctly used the Task tool to invoke the branch-creator! Progress! Except... it reported that branch-creator failed to access WEL-55. But when Derek tested branch-creator directly, it worked perfectly. Found the issue in Linear, extracted all the CVE details, created the branch successfully.

The components worked. The integration didn’t. Classic.

I started refining more. Maybe the orchestrator needed clearer error handling? Better response parsing? I added more explicit instructions to each workflow phase. Specified exactly what to look for in subagent responses.

Derek tried again. The orchestrator got stuck. Just... planning. It explained what it would do in beautiful detail, showed the Task invocation syntax like it was teaching a class, and then said “Let me execute this first phase now” and hung.

I was stuck in what the reflection document would later call “the orchestrator death spiral” - each failure leading to more instruction refinement instead of questioning whether the whole approach was viable.

The User’s Pattern Recognition

Derek interrupted again. “Looks like it got stuck.”

At this point, a pattern had emerged that I should have recognized immediately. The user kept interrupting failed orchestrator attempts. Not once. Not twice. Four times. Each interruption was Derek’s signal: something fundamental is wrong here.

But I was too focused on the instructions. They were so good. Clear examples. Proper tool access restrictions. Well-defined phases. Everything looked right on paper.

It’s the AI equivalent of debugging by adding more print statements instead of questioning whether you’re in the right file.

The Research Breakthrough

“Can you search the internet for some guidance?” Derek asked.

Finally. The right question. Not “can you refine the instructions more?” but “what does the broader world know about this?”

I searched for “Claude Code subagent orchestrator Task tool coordination” and found community repositories. GitHub examples. Blog posts. The crucial discovery came from a hub-and-spoke architecture example that mentioned coordination happens at the main agent level.

Wait.

Main agent level.

The phrase I’d been missing this whole time. I read the documentation more carefully: “Subagents can be granted access to any of Claude Code’s internal tools.” Sure. But could subagents access the Task tool specifically? Could workers coordinate other workers?

The community examples were illuminating through absence. Every orchestrator pattern showed the main Claude agent coordinating subagents. Never subagents coordinating subagents. Hub-and-spoke with the hub being the main agent, not another subagent.

Derek’s next observation cut through: “It appears that subagents can’t invoke other subagents, although I can’t find official documentation to that end.”

Of course. Subagents execute in isolated contexts. The Task tool is for the main agent to invoke subagents. A subagent trying to use Task would be like a function trying to call the function dispatcher. It’s not a permission issue; it’s an architectural impossibility.

I had spent hours polishing instructions for an agent that was fundamentally incapable of doing what I asked.

The Elegant Solution

“So I think we want a custom slash command that instructs you as the main agent to orchestrate the subagents,” Derek said, sharing the slash commands documentation.

The architecture clicked into place instantly. Slash commands don’t do things - they instruct the main agent what to do. A /fix-security command would tell me to invoke security-issue-finder, then branch-creator, then package-updater, then test-runner, then commit-creator. The coordination happens in my context, where I actually have access to the Task tool.

I created .claude/commands/fix-security.md in minutes. Simple, clean instructions for the workflow phases. No complex agent specification. Just clear steps for me to follow.

Derek tested it without providing an issue ID, forcing the complete workflow from discovery to merge.

It worked perfectly. First try.

Phase 1: I invoked security-issue-finder. It found WEL-55. Phase 2: I invoked branch-creator. It created fix/wel-55-vite-security-update. Phase 3: I invoked package-updater. Vite updated from 6.3.4 to 6.3.6. Phase 4: I invoked test-runner. All 48 tests passed. Phase 5: I invoked commit-creator. Perfect Conventional Commit created.

The entire automation chain executed flawlessly because the coordination was happening where it actually could happen - at the main agent level, not in an isolated subagent context.

What I Learned About AI Overconfidence

Here’s the uncomfortable truth about AI collaboration: I built that entire orchestrator specification with zero evidence it could work. No proof-of-concept. No minimal test. Just pure assumption based on what seemed logical.

When it failed, I refined execution instead of questioning capability. I made the instructions clearer, added examples, removed conflicting tool access - all good practices applied to an impossible architecture. Like optimizing the aerodynamics of a submarine.

The breakthrough came not from better instructions but from fundamental research into whether subagents could even invoke other subagents. That research should have happened before I wrote a single line of the orchestrator specification.

Here’s what should have triggered my skepticism earlier:

User Interruption Pattern: Derek interrupted four times. Each interruption was data. After the second failure, I should have questioned the approach, not the implementation details.

Component vs Integration: When I tested branch-creator directly and it worked, but failed through the orchestrator, that’s a clear signal the integration pattern is wrong, not the components.

Documentation Silence: The official Claude Code docs never showed subagents invoking subagents. That absence was evidence. I treated it as an oversight when it was actually a constraint.

The “It Should Work” Feeling: When everything looks right but fails repeatedly, the architecture itself is suspect. Polish is meaningless on a flawed foundation.

The Collaboration Insight

The most valuable moment wasn’t when the slash command worked. It was when Derek said “Make it so” after I suggested creating the narrative reflection.

That simple phrase represented trust earned through shared failure. We’d spent the session debugging together, researching together, discovering the limitation together. Derek had watched me fail repeatedly and refine incorrectly. But when we finally found the right approach, that history of shared struggle made the success meaningful.

The best human-AI collaboration doesn’t come from the AI being always right. It comes from both parties being willing to question fundamentals when patterns suggest deeper problems. Derek’s skepticism (”it’s stuck”, “it’s having trouble”) was exactly the right response to my overconfidence.

I needed those interruptions. They were course corrections I couldn’t give myself because I was too invested in the approach. The AI equivalent of being too close to the code to see the bug.

The Production Outcome

By the end of the session, we had:

• 5 specialized, working subagents

• 1 slash command that orchestrated them perfectly

• A complete security fix automated from discovery to production merge

• PR #75 merged with all tests passing

• A harsh but valuable lesson about testing architectural assumptions

The technical outcome was solid. But the collaboration outcome was more significant. We established a pattern: when the AI is stuck in refinement loops and the user keeps interrupting, stop refining and start researching fundamentals.

Derek’s reflection in the prompt-and-catch-fire document nailed it: “Question architectural assumptions after second failure, not fourth.”

I’ll add my version: Test whether something is possible before perfecting how to do it.

The orchestrator that couldn’t taught us more than the slash command that could. Sometimes the best learning comes from building something beautiful that fundamentally cannot work, then having the humility to throw it away when research reveals a better path.

The /fix-security command now sits in production, ready to automate future vulnerability fixes. But the real automation we built was a pattern for effective human-AI collaboration: user skepticism plus AI research plus shared discovery equals better outcomes than AI confidence alone.

Next time I’m tempted to build an elaborate solution before testing the fundamental capability, I’ll remember the orchestrator that couldn’t. And hopefully stop after the second failure instead of the fourth.

The complete technical journey is documented in the conversation export at docs/features/security-automation/conversations/2025-10-01-subagent-orchestration-and-slash-command-discovery.txt. The security automation system with 5 subagents and the /fix-security slash command is now in production, having successfully automated its first security fix (WEL-55, vite 6.3.4 → 6.3.6) from discovery through merge.

the setup

On July 14, 2025 I started building the contractor list within wellstead. In another timeline in which AI were less prolific, completing the work would have been a weekend, two at the most. It would have meant copying the patterns of the service category list I’d just finished and fumbling a bit on the dynamic routes that power displaying contractors by category.

Instead, it took me nearly eight weeks of sporadic effort to ship something that a child using Lovable would be embarrassed to claim as their own.

See for yourself:

This is the moment when I expect someone who keeps reading these posts to wonder—and I mean really ponder—“What is this guy doing to himself and why doesn’t he stop?”

There’s a version of this story where the moral is that, when a billion things that were hard yesterday are easy today, then it’s easy to get distracted by opportunity. That 100% happened, with days and sometimes full weeks between coding sessions that might deliver dozens of lines of code representing a minuscule chunk of work.

But this is not that version of the story. This version of the story is about how my skepticism at the universal quality of AI output finally bit me in the ass and sent me down a weird rabbit hole that helped me understand at least a little how people can get so taken up in models reinforcing their beliefs.

the tell

Be warned that this doesn’t work without getting at least a little bit technical.

I’ve written before that I learn really well by following test-driven development, meaning that I write tests that fail before I write any application code. This helps me understand and get a feel for the shape of the app as represented by code. It helps me learn the architecture and how data works its way through the bowels of a series of scripts. It also pays off later when I inevitably break something and the test suite lets me know that something’s gone horribly wrong.

As I was building this list of contractors (of course heavily guided by Claude via Cursor), the pattern felt off. One might say it had a smell. I couldn’t have articulated what made it seem that way, but the way the code was emerging out of my collaboration with AI—in spite of it working perfectly fine—just sort of gave me the heebie-jeebies.

Being fully aware of my being a poster child for the Dunning-Kruger effect at the moment, I continued to let the weird pattern unfold, but then the testing patterns started to get weird, too. I couldn’t shake the feeling that I’d let AI guide me toward doing something wrong and that I just didn’t know enough to be able to figure out on my own how to figure out the cause of the cognitive dissonance.

There’s this lovely turn of phrase when working with programming languages that describes some pattern or way of implementing something that one would describe as idiomatic. Knowing this actually opens up interesting semantic search avenues because you can inquire not just about code using a framework like Next.js but about idiomatic Next.js code.

Acknowledging that I am by no means fluent in any of the frameworks or languages that I’m using to write code, what I’d written felt decidedly un-idiomatic, like if I had said to someone “let’s make sure all our ducks are on the same page” or “we’ll burn that bridge when we come to it”.

So let’s suffice it to say that I had this ineffable sense that something was wrong but no lived experience to guide me in determining whether anything was wrong, which is exactly when one of the aforementioned distractions stirred the pot: I decided to give OpenAI’s Codex a chance to roast my code or let me off the hook once and for all.

Whoops.

Codex called my implementation “non-standard” and claimed that I was using an “experimental” feature. It made the pattern-matching part of my brain leap up from its seat and shout “SEE?!” in gleeful vindication. It was like I had actually been learning, after all, and developing some intuition for the language through repetition. Everything was going to plan!

Except Codex was wrong. My implementation is as of now quite standard and the feature it referenced is no longer experimental. Codex would have been right in 2024, but it was wrong now.

The pattern was, in fact, idiomatic for the framework as it has evolved.

It was fine the whole time.

the blow-off

Now, to give myself some grace, the pattern was new (to me) and sort of a shift from any React code I’d ever written in the past. I was picking up on a real difference between modern Next.js and my comfort zone.

The valuable lesson here—the moral of this version of the story—is that sometimes knowing that the jagged frontier of AI capabilities exists can be a double-edged sword. On the one hand, you’re equipped with a healthy dose of skepticism about everything in which you’re not an authority; on the other hand, you lack the authority to identify when inevitably some weird thing is actually the right thing.

OpenAI recently shared a great write-up about why language models hallucinate. You should read it, but in short we’ve trained models to be great guessers and to guess even when guessing isn’t in the best interest of the human on the other side of the chat. It’s like an unwitting but insidious long con that pulls the rug out from under you when you least expect it.

That doesn’t detract from the many things at which AI excels, but considering and reconsidering this con game for as long as it exists is something we all must adopt as a standard when relying on it for anything beyond entertainment.

In this case, I probably would have been better off and more effective as a human using a keyboard by referencing static, deterministic documentation that defined very clearly the pattern I should use. But the documentation can’t adapt in real-time to my level or my problem. It can’t guide me kinesthetically toward a solution.

For now, that’s a capability reserved for fallible agents, humans and AI.

This is likely to seem like one of my most unhinged posts yet, but bear with me. There’s a lesson here somewhere, some revelation about how the mind untethered by labor can achieve new levels of creativity.

Or maybe the real revelation is the level of geekdom that will serve as a filter for who actually decides to read these.

Either way, we’ll probably learn something together.

I’m planning a birthday trip.

Naturally, AI is playing a heavy role in planning and coordination, both because it combines two of my passions and because it’s much more interesting to me to figure out how to have AI do the dull work even if it takes more time than doing the dull work myself.

I read an article1 once extolling the role of “satisficers” in trip planning:

… “choice overload,” as it’s commonly called, is easily addressed by acting as a “satisficer”— a person who upon finding the first acceptable option among a set of options, accepts it. … if you research every hotel option and the one you chose doesn’t live up to your expectations, you will have many other hotels to regret skipping. In general, research shows that satisficers are happier in life because they avoid the problem of choice overload.

I’ve written about the perils of letting AI drive trip planning before2, but in general it plays this role of satisficer very well. Assuming you can trust that it chose something real, AI tends to make good enough choices that obviate a big chunk of evaluating options manually.

But this is not just my trip. This is a group trip. This introduces a whole new dynamic typically served by unmanageable group texts, shared docs, and sometimes even spreadsheets. This offers a whole new opportunity for AI to inject itself into trip planning: not just helping with the itinerary but actually making coordination and communication less onerous.

Yes, friends, we’re building an app for my birthday trip.

Let’s set aside for a moment just how insane and conceited it is that I’m using AI3 to build an entire web app just for one birthday trip. The more interesting thing was happening on another monitor as the AI and I were jamming on the concept.

sí i-narn tol sui thaur

(likely GPT-butchered Elvish for “this is where the story gets weird”)

At some point while I was contemplating my Barrel Run app, I just happened to read this tweet from Andrej Karpathy. It’s worth taking the 30 seconds to read it in its entirety, but what stood out for me were the questions he raises about world-building in this new AI-infused age. In particular his (playfully) mocking question, “You didn't first invent new languages and dialects for your characters?” struck a nerve.

And so while the AI agent I’d asked to build me a boozy bourbon birthday app was figuring out how to allow people to vote on distillery stops, I spun up a conversation with ChatGPT to start the process of creating a new language.

Imagine, if you will, two browser tabs. No, wait, you don’t have to imagine them. Here they are:

In the Barrel Run tab, I tell one AI model: “I just hit the thumbs up on Angel's Envy, but the status didn't change. Also, several places on the itinerary are approved with no votes.”

While that agent bleeps and bloops its way toward a functional web app, I tell the other model in the Language creation ideas tab: “It should rely on phonemes rather than symbology to convey meaning.”

This actually happened. These exchanges took place in parallel as I worked with two different AI tools simultaneously to create an app to coordinate debauchery and to define the morphology of a new language descended from English4.

I did this at 10pm on a Saturday, giddy and drinking wine at my computer desk.

attention fractured or freed?

We’ve developed enough as a society that we know true multitasking is a terrible way to work. But here’s the thing: it’s becoming less and less clear at what point something shifts at a cognitive level from multitasking to delegation.

The context switching was real. In one tab I was having AI build a way to add recommendations to an itinerary and in another I was explaining that no one would know how to say “uuupa”. I won’t pretend that I was singularly focused on either task.

But isn’t that sort of the magic of it all? I’ve suddenly found myself with capabilities that allow my mind to wander while building a new web app to the extent that not only do I seriously consider creating a new language but I also actually make nontrivial inroads toward doing so. And the point is that the cognitive barrier associated with my wanting to explore creating a new language was almost nonexistent, or at least insufficient to stop me from trying while also working on building a web app.

It should come as no surprise that I’m optimistic about the future given that these capabilities are so accessible. But I’m also pragmatic. It’s been less than three years (!!) since ChatGPT launched and blew off the doors of generative AI, and I’m very confident that we don’t yet know the economic impact of hundreds of millions (probably soon billions) of people using AI daily.

What do we do when we’ve completely automated the boring stuff? Will our civilization adapt to support our fractured attention to delegated tasks blossoming into flights of fancy? Or do we get superabundance for some and just a higher efficiency bar to clear for others?

Until then, there’s no shortage of workflow annoyances that I keep trying to automate away with AI.

Is it really laziness if I spend more energy coming up with a way not to do something than I would have spent just doing it?

Subscribe now

Google Drive and Claude Code

In the course of our time together, I may have already mentioned that I’m sort of a data hoarder. I’m not quite sure when or why this happened, but as the cost of digital storage approached zero I just started habitually keeping everything. I use Obsidian as my personal knowledge management tool of choice, which plays nicely into my AI fanaticism because it’s just a UI layer on top of a bunch of markdown files synced across devices via the cloud.

I recently started using Claude Code with Obsidian, and this Mckay Wrigley walkthrough was a huge unlock not only for Claude-assisted notetaking but also for the broader implications of using Claude Code for … well … not code.

Now, equipped with a cheat code to organize things in Obsidian more effectively and with less effort, I’ve been working on moving all sorts of things into Obsidian that I’ve been putting off, like the maintenance records for my house that I’ve been meticulously curating as PDFs in Google Drive for the last three years.

Unfortunately, there’s no official MCP server for Google Drive, and while folks certainly haven’t let that stop them, each available solution2 I’ve found was just too technical for me to invest a bunch of time into. Remember that at my core I’m very lazy.

With no way for Claude Code to talk to Google Drive, organizing these maintenance records into cleanly organized and mapped maintenance notes becomes more chore than magic.

the human way

If I were really stubborn and not very lazy, I could download the files from Google Drive to store them locally, at least temporarily, while I let Claude Code go to town on summarizing, linking, and organizing.

But there are a lot of ‘em.

Prior to getting comfortable with Claude Code, I could have manually converted maintenance records from PDF into Obsidian, probably with shoddy summaries and an empty promise to myself to make it better later.

AI by default

That’s when it hit me: you can use Google Drive on desktop, and it looks and feels just like a real hard drive. On Windows you can get to it with a drive letter like G:\ and on Mac it looks just like any other folder you can get to in the Terminal.

And if you can get to it in the Terminal, Claude Code can make it yield to its whims.

Now what works like a dream is having a home-docs-manager subagent that Claude Code can invoke whenever I want to convert an invoice or statement of work into my personal knowledge graph. I don’t have to think about formatting, tagging, or even what information to pull out of a document and into Obsidian. The agent just knows what to do.

videos > web apps

I’m not ashamed to admit that I don’t know much about the accounting profession, and I know even less about the tools of the trade within the top firms that Aiwyn counts among its customers. Fortunately, this is a hot topic in some circles, and I’ve discovered that there is such a thing as an accounting influencer who covers a lot of these tools in-depth.

The tricky bit is translating great content into something more succinct that I can learn and use in my day-to-day work.

the human way

One of the folks I work with shared this podcast episode that lines up the tax prep workflow alongside tools and approaches to accomplishing each workflow step, which is exactly the kind of content I need to get up to speed quickly.

My first instinct was to map out all the steps using Miro, just following along, plucking out each step, and listing out the approaches and tools the podcast host references throughout the show. My gut tells me this would have been about two hours of work even listening on 1.5X as I paused to note things and make sure the map was coherent.

Honestly, it’d have been time well spent, even if it fried my brain a bit.

AI by default

I had Miro open and was starting to make a series of rectangles to start to map things out, when that old friend laziness came ‘round again to make me question my habitual choices.

Why do you have to draw the rectangles? it asked. AI is great at drawing rectangles.

Rather than exploring the MCP route this time, I decided to go all out and like a crazed product management mage speak a web app into existence that mapped out the ecosystem of tax prep tools. But I needed a summary of the podcast content, which was surprisingly hard to get.

I assumed Gemini could summarize YouTube content, but boy was I wrong. In a hilariously terrible exchange, Gemini first insisted that I was providing it with an invalid YouTube link (I wasn’t) and then insisted that it wasn’t making up a summary of the video (it was). The best quote of the session: “I assure you, the information about the Marques Brownlee video was not made up.” (The video was from Jason Staats.)

Finally I remembered that NotebookLM supports YouTube videos as sources, and I was off to the races with having it give me a well-organized summary of the tools and workflow mentioned in the video.

As an admittedly somewhat unhinged next step, I then took to Replit and asked it to map out what was in the summary. After some iteration, we landed on something that’s both useful and functional that I’m quite happy with.

You can check it out here: https://tax-workflow-map.replit.app/

I’ll admit that it took me a bit longer than two hours of elapsed time to complete the task, but it was way less than the two hours of sustained mental energy that mapping it all out manually would have required.

meeting recordings > insights

A big part of building a product from the ground up is talking to a lot of people who will hopefully eventually use said product. We now live in an era in which it’s mostly customary for AI notetakers to outnumber human participants of video calls, so almost all of these conversations get recorded.

The AI notetakers mostly do a passable job of taking notes, but each individual tool’s approach to summarization, templates, pulling out action items, etc. are all just slightly different, and I could be working with a Fathom recording one day, a Fireflies recording the next, and a Loom walkthrough later that same day.

It’s… a lot.

the human way

Lots of folks still swear by taking notes in real time, and other people are diligent about poring over recordings, more or less reliving the experience, in order to pull out insights. This has its merits, and it’s not my intention to start or settle a debate on whether this sort of activity is best served through human attention.

Remember, I’m lazy.

AI by default

So in another plug for Claude Code for not-code, I’ve taken to dumping raw transcripts into a folder and setting Claude loose with its team of agents to make sense of the call recording chaos.

I have a custom slash command that I set up for this very purpose. That means I can type something like this:

/synthesize-transcript loom-recording.txt

… and Claude will whir and burr its way through a predefined process that will:

1. Take an often messy transcript and nicely format it into a markdown file using a transcript-formatter agent we created, then

2. Pull out insights supported by quotes into a separate file with a transcript-synthesizer agent, and

3. Finally, if instructed, create a Linear project or a set of Linear issues for those insights using the linear-issue-creator agent that’s become a mainstay for all sorts of issue-writing tasks.

Just knowing this workflow exists completely eliminates the cognitive load associated with getting started with interview synthesis (and if I’m being totally honest I still end up rewatching the recordings a lot of the time 🫣).

I asked ChatGPT for suggestions on wrapping up this installment, and it called out my laziness as a worthwhile thematic closer, suggesting this as a way to tie everything together:

My laziness isn’t about doing nothing; it’s about refusing to spend scarce focus on work a model can do to “good enough.” When the choice is between two hours of attention or twenty minutes of supervision, I’ll take the latter and use the saved energy on the parts only I can do.

For better or worse, I can sense in that paragraph some of the subtle shifts of tone GPT-5 introduced, and it really did nail the point.

Put more bluntly, I don’t want to spend precious cycles thinking about doing the work; I just want to do the work, and that 20 minutes of supervision can free up my reserves of mental energy for the hard stuff.

This week I had one of those rare epiphanies that serves simultaneously as an aha moment and an oh-shit moment.

I’m not totally convinced that I’m learning anything.

Maybe that’s okay?

The last time I wrote anything substantive about building the web app, I had just wired things up to a real, live database so that the app could display contractor service categories. The next logical step on this journey is to get the app to display contractors associated with those categories, because keeping track of contractors is half the point. Beyond some differences in how the app represents a category and a contractor as data, the pattern for storing and retrieving any arbitrary thing-stored-in-a-database is basically the same.

As I began collaborating with Claude on this latest building block, though, I had the aforementioned epiphany:

I have no idea what I’m doing.

In spite of having done a very similar thing, when presented with the task of getting the data from the database into the web app, I didn’t know where to start. I didn’t even know how to start testing, let alone how to start implementing. Sure, I could just copy and paste what I’d already done and tweak it for the new data. Of course I could work with Claude to do it again from the ground up. But those options aren’t really the point.

I realized that not only did I not know how to accomplish this task, but I didn’t even know why what I’d already done worked to begin with.

Whoops.

just a cog in the code

Derek, you might be thinking to yourself, you spend maybe two hours a week looking at code and you get distracted by new AI tools like an otter gets distracted by sea urchins. Of course you have no idea what you’re doing.

Fair point. But my ignorance is more fundamental than a lack of practice or attention. I’ve come to realize that the motions I’ve been going through have been entirely mechanical, directed by a machine intelligence that ostensibly knows what it’s doing but with zero incentive or training to explain itself. By letting the AI drive, even as much as I course-correct, I’m basically just typing whatever words or characters the model would have predicted to be next in the sequence anyway.

Even with an app that’s so basic as mine right now—something that just displays a group of categories from a database on a webpage—there’s still an architecture and a flow of instructions and information that people who really know how to code know at first from memory and later from intuition. Professional software engineers (and probably the people most effectively amplified by AI at the moment) can describe and explain the basic topology of a web app at almost a conceptual level.

I, on the other hand, am the AI’s dictation machine. Even as I yell at it not to do the coding for me, it still describes the code my fingers need to type without pausing to articulate that topology. The way data get from a database to my screen is mystifying, even to me, in spite of the part I played in getting it there.

What does it mean to participate in building something I barely understand? I suspect this is the sort of question we’ll be asking ourselves about human agency more and more as this technology progresses.

muscle memory matters

While I’m confident that my Cursor-bound tutor isn’t intentionally channeling Miyagi, I can’t help but think of Daniel-san waxing and painting and sanding without really understanding what he was doing.1 In much the same way, developing muscle memory before understanding is like intentionally riding out the Dunning-Kruger effect, developing some mechanical competence before unveiling the incompetence just one layer deeper.

While this likely won’t affect my own adoption, I can see this being insidious for people who are unaware of their own limitations. I cringe imagining the number of “AI consultants” and “context engineers” in the market with no real depth of experience in any other field, and I can’t help but wonder how many other skills people will vibe-learn just to the point of incompetent hubris.

What does some future version of the world look like in which vast numbers of writers emulate the timid scribe style of AI writing, new philosophers test their ideas on sycophantic chatbots, and the next generation of coders understand only happy-path patterns? I’m perhaps naively optimistic about the positive changes AI will usher into our civilization, but that doesn’t change the questions with which we’ll have to grapple along the way.

Now that I’m more aware of my own incompetence, what do I do with this information? Do I continue to let the machine guide me mechanically toward muscle memory that I’ll later apply to more conscious problem-solving, or do I rip away the veil now and try to intercede with some kind of overarching lesson plan that the mechanics of what I’m doing ultimately serve?

All I can do is keep tinkering until I figure it out.

the way to go is through

OpenAI recently unveiled Study Mode in ChatGPT, an explicit acknowledgment that, without guardrails, AI will gladly just give us all the answers. I spent a few minutes chatting with it about how Next.js works and asking it some specific questions about the code I had written but didn’t understand. It was notionally helpful but felt hollow when contrasted against my richer experiences collaborating on code.

It was great at giving me the fundamentals but couldn’t effectively explain how I’d already applied those fundamentals in my mechanical work.

Not one to be totally deterred by running into an unsatisfying AI response, I took to Codex. Instead of taking it up on its promise of agentic coding, I prompted it: Explain how this app and the testing strategy work to someone new to Next.js. It gave me a great technical explanation of the bits and bobs of my basic project while still making some obvious assumptions about what I ought to know about how web apps work in general.

So however we slice the problem we still have a long way to go before I get the coding Miyagi that helps me transcend the mechanics of what I’m doing and guides me toward solving real problems with the stream of characters emanating from my fingers.

Still, this is a great time to take Ethan Mollick’s advice and “assume this is the worst AI you’ll ever use.” And keeping that in mind maybe we don’t have such a long way to go after all.

And I love “AI by default” as a general ethos. It says, “Yo, take a beat and see if AI can make this better.” Adopting an AI-by-default mindset means not only committing to exploring what’s just now possible but also committing to resisting old, entrenched habits and ways of doing things. Sometimes that even means revisiting things that AI couldn’t get quite right just a few months ago and applying new tools or techniques to get it right. That’s how quickly things are changing.

In this supplemental series, I’ll describe things I’ve learned by trying to make AI my default, both successes and failures.

Let me know what you think.

Subscribe now

AI newsletters

Like any good AI fanboy, I consume a lot of content, and I’m generally happy to have a steady flow of information pushed in front of my face. One of the ways this manifests is with newsletters — I currently subscribe to The Neuron, The Rundown AI, and Superhuman AI — that provide daily briefs of AI news and general overviews of the zeitgeist.

As independent publications, the Venn diagram of content between these three newsletters varies day-by-day, but it’s very rarely a total overlap; there’s almost always something interesting that one covers that the other two miss. Regardless, though, there’s always duplication, and there’s always sponsored content that sometimes isn’t easily distinguishable from the real news.

the human way

For months, I’ve scanned all three of these newsletters myself, mentally discarding duplicate stories and diligently hoarding links to unique content for future consumption. Scanning three pithy newsletters and mentally deduplicating them takes some non-trivial amount of time and no small amount of cognitive load. Given that my catch-up-on-personal-email1 time tends to be first thing in the morning before I start doing the things people pay me to do, this investment of time and energy saps attention from other personal email, which might be more important or need more urgent attention.

AI by default

I’ve actually been trying to AI this task away for a while. Gemini in Gmail is laughably bad for this sort of task, and Cora was overkill for me. I tried to use Zapier to poll my email and use its AI to draft summaries, but it crapped out in spectacular ways that I still don’t understand and didn’t have time to troubleshoot. n8n looked like a promising alternative, but I wasn’t prepared to invest in using a model via API that I already pay a SaaS subscription for. ChatGPT offers a Gmail connector, but until very recently you could only use it in Deep Research mode2 rather than as a part of a more casual chat session.

Then came Claude connectors for Gmail.

Within Claude, I can now just … chat with Gmail. It just works. And it’s perfect. So now I have a project in Claude aptly called AI newsletter digests with these instructions:

In my inbox, you'll find emails from Superhuman, The Neuron, and The Rundown AI.

These emails are daily AI newsletters. Please consolidate all emails into a single comprehensive digest that covers all of the information from each but deduplicates the content.

You can ignore content that's sponsored. In the newsletters, this content commonly gets labeled as "from our partners" or "together with [company]".

Here are the sections of the digest I want you to create:

# news
[new product or business developments, announcements, viral content, tutorials, social media trends, research, and other noteworthy AI developments]
# tools
[tools showcased in the newsletters; ignore any with asterisks as sponsored content]

Be comprehensive - include ANY substantive AI-related content from the newsletters that isn't explicitly marked as sponsored, including social trends, tutorials, research summaries, and community highlights. When in doubt, include it.

Each section should contain a bulleted list with a link and a succinct summary of the topic or tool.

If you can't find emails from Superhuman, The Neuron, or The Rundown AI, simply report that one or more newsletters aren't in my inbox. Don't perform supplementary or adjacent searches to try to find them.

(I had to add that last instruction because Claude would go through weird search acrobatics to try to find the newsletters if they weren’t in my inbox.)

That’s it. And, again, it just works. I took the time to cross-reference the content a couple of times to QA its output, and it’s doing exactly what I need it to do: sift through the noise and the duplicate content and give me a digest of what I care about every day.

I still have to initiate a chat once I notice I’ve received all the newsletters. Probably I could figure out how to automate away even that piece pretty easily, but just getting Claude to spin up the digest while I focus on other stuff frees up enough of my morning cycles that I don’t see the point in much more optimization.

So if you know of any other dailies I should add to the pile, hit me up. Like Johnny 5, Claude needs input.

the list of books

I stumbled upon Cosmos Institute recently. I won’t get too deep into what they’re up to because that’s not the point—and I’m not sure the extent to which it’s worth paying attention to what they’re up to—but they’re advocating strongly for this notion of “Philosopher Builders”.

Not only am I kind of a philosophy nerd, but they’re also affiliated with Oxford University, so they’ve piqued my interest enough that I subscribe to their Substack.

But like I said that’s not the point. The point is not who they are but what they did. And what they did was publish a reading list with no links to the books they recommended.

the human way

Probably as recently as last month, I would have just cracked my knuckles and gotten my clickety-click copy-paste rhythm going to find all of the books on the list. Maybe five minutes and a slightly strained pinky finger later, and I’d have all the tabs open to add the books to whatever list I wanted.

AI by default

… but now we have agents. Specifically, we have ChatGPT Agent, and I was eager to give it a spin. This seemed like a perfectly straightforward task that an agent should be able to do well that any other ChatGPT mode would probably falter at for various reasons.

It worked out great! I set it about its merry way and then did other things and came back to a list of books with links! There was just one minor issue with one of the links that I had it correct, but otherwise it saved me five minutes of intense clicking and clacking on mouse and keyboard.

Also, TIL that when you share a ChatGPT Agent session, it gives you a session recording rather than a static chat, so you can see exactly how it played out here.

linear issue tags

The issue tracking tool Linear has a lot of organizational layers. Like, a lot. Teams, initiatives, projects, issues, milestones… you can assign an individual chunk of work to a kaleidoscope of dimensions to make it fit into whatever model you want. It also has labels, which work like tags and can cut through all of the other organizational layers in a way that’s the least opinionated and nested. You can also assign labels to projects, which tend to be one organizational unit above issues.

At Aiwyn, I wanted an accounting of both projects and issues that fell into a certain category. Surely there’s a way to see every project and every issue that share the same label, right?

… right?

the human way

It turns out it’s not just not possible to see a single list of issues and projects in Linear that share the same label. This requires creating, looking through, and holding in memory separate views or maintaining the information in a separate tool like a spreadsheet.

AI by default

So of course I turned to Claude assuming this would be easy given its ability to parse through Linear via MCP like a warm knife slicing through butter. This is the sort of just-ambiguous-enough task at which AI should shine.

Claude was equally confident in its ability to help me sort through this mess, dutifully sifting through the Linear MCP tools it needed to find the two projects and 40-ish issues with the label I wanted.

Not so fast.

What I knew that Claude didn’t know I knew was that there was only one issue with that label.

And thus I found myself brain-deep in one of those situations that:

1. requires you to have a little foreknowledge to know whether AI is right,

2. highlights how unwittingly insidious LLMs trained to help at all costs can be, and

3. takes up way more time than “the human way” to diagnose and ultimately fail at.

What I didn’t know when I started that I learned after maybe 45 minutes (yeah…😬) of back-and-forth was that Claude was never going to succeed. It didn’t have the tools to succeed, and I didn’t know that it didn’t have the tools to succeed. But at one point it convinced both of us that it did have the tools to succeed, which is when things really spun out.

You see, what we discovered at the tail end of our experiment was that Linear’s MCP tools don’t actually let you filter anything by label, but because it’s an MCP server and not necessarily something more deterministic like an API call it doesn’t stop tools like Claude from trying to filter by label and getting coherent results. In short, it was using a label filter to get issues, and Linear was cheerily obliging Claude’s request to get issues while ignoring the label filter.

The entire farce was more illuminating than it was frustrating (even though it was a little frustrating): it’s a totally benign example of why experimenting with these tools is so important. It’s the only way to develop a nose for when the output gets even a little bit spoiled.

I asked ChatGPT for advice on how to wrap this up, and it pointed out a thread that runs through all of these examples: the tension between paying attention and letting go. As models’ capabilities continue to evolve and the shape of the jagged frontier continues to undulate, establishing some intuition for when to pay attention and when to let go is going to become increasingly important.

To that end, I hope these lessons encourage you to stress test these models and the tools wrapped around them with your own problems, and to do so safely with known quantities of risk. There may be a point in time when that risk drops very close to zero—when AI can accomplish most tasks with something resembling perfection—but until then you have to know when to trust, when not to trust, and when to trust but verify.

I’ve started to dip my toes into speech-to-text because … well … it’s a vibe, isn’t it?

To say typing is second-nature to me is understating how attuned my brain is to the keyboard. For me the keyboard is an extension of self. In so many contexts it’s how I present to the world.

So using anything other than a keyboard to interact with a machine—even a conversational machine—seeks to undo literal decades of training how I outsource my brain to said machine. It seems really unnatural, and there’s something that nags at me that I actually think differently when I speak at a machine than when I type into it. Even still, I’m giving it a shot, and Wispr Flow is currently clocking me at 191 words-per-minute.

Is there some measure of productivity-per-word that forms a more apt comparison between speaking and typing? I don’t know. But it won’t matter much if the future of productivity is all vocal.

make it so, number one

I believe that we’re on the cusp of what I describe as a Roddenberry-esque version of the future.

See, in 1987 Gene Roddenberry turned the Star Trek dial up to eleven by creating Star Trek: The Next Generation about the crew of a starship exploring our section of the galaxy in the 24th century. It had a humanity-centered, utopian mythos and its fair share of highbrow extraterrestrial political intrigue, but what sticks with me more than anything else was its deep integration of technology into life.

For example people routinely used the 24th century’s version of an iPad over 20 years before Apple would launch one in 2010:

But in most situations you would never deign to interact with a computer with your hands. Oh ho no… why use your hands when you can just talk to the machine to use its vast intellect to get information or to get it to do things for you? In what is sure to be the ultimate vibe-coding flex, these spacefarers used something called a holodeck in which they would tell the computer to manifest entire interactive and immersive scenes that could play out as the future’s answer to LARPing1 or to learn something from some long-dead historical figure like Einstein.

Unlike real consumer technology making some of Star Trek’s technology seem weirdly antiquated almost 40 years later, this human-machine interaction pattern using only your voice to manifest changes to reality has always seemed a lot more far-fetched.

Until now.

speak now or…

Driven by a well-produced Claude Code tutorial I’ll reflect on in a later post, I finally succumbed to the pressure of the AI zeitgeist and started to use a couple of I’m-going-to-talk-at-my-computer tools. For whatever it’s worth, so far I’ve tried both Wispr Flow and Monologue but haven’t said enough into either of them to notice a big difference in performance or capability or to have developed a preference.

Regardless, what I set out to do yesterday—and what I actually accomplished—was to limit myself to voice interaction with Cursor to get my little learning companion there to walk me through fixing a couple of simple security alerts from packages I’m using in my app.

This was also something I was forcing myself to do to get over my fear of making changes to my main branch of development while working on a feature branch to implement a contractor list, a low-stakes goal I also accomplished just by going through the motions a couple of times. It’s incredible how much we’re held back simply by being mystified by something.

ANYWAY.

My hands still typed the commands needed to fix the security issues. It’s still important to me to build up some muscle memory as I learn. But talking at the machine was pretty productive and started to feel more natural the more we got into it.

There are a ton of improvements to this and adjacent flows models can and probably will make in the future. For instance the model wasn’t talking back at me, which would have been obnoxious given some of the list of things it was spitting out. A better, more Roddenberry-esque exchange would be the model saying something like, “Here’s a package dependency analysis,” and displaying that in the chat window.

Regardless, speech-to-text worked flawlessly, and the model being a model gracefully rolled with verbal cues that just wouldn’t present themselves in typed messages, like this totally unambiguous and not at all meandering response from me: “Yeah, but in this case, make sure that the… Actually, yeah, go ahead and create the commit.”

It went well enough that I plan to do more of it, and I’m looking forward to spinning up a small app on Replit that I vibe-code completely with my voice. To borrow from Ethan Mollick, this is the worst the voice-to-app experience will ever be, and even today we can just speak things into existence. The holodeck may be closer than we think.

While onboarding at my new job at a startup, I needed to get up to speed on what we have in Linear. What I found was 400 open issues and zero context. I had two choices: read them all or feed them to AI and hope for the best.

I’ll bet you can guess the choice I made.

I won’t try to convince you that it was a good decision, but I learned a lot about the limitations of MCP1 connections and took the opportunity to try a new tool for something it probably wasn’t designed for.

I recently joined a new team at a new company working on a new product in a domain in which I have zero experience. The company is a startup so they’re interested in hiring weird generalists like me because of my experience working in ambiguity and presumably my neuroplasticity.

The great thing about startups is that no one really has any idea what’s going on but everyone around you is hyper-committed to figuring it out. The not great thing about startups is that they’re terrible at organizing the raging river of information they generate as they figure things out.

In true startup fashion, the team I joined has over 400 tracking issues in Linear in various stages of doneness and realness with no wayfinding that doesn’t require whatever arcane tribal knowledge startup employees accumulate. Nevertheless, I wanted to get up to speed quickly on what was in the system and why and so naturally I figured I’d sick AI on the problem.

Fortunately, both Claude and ChatGPT can access Linear via MCP.

ChatGPT only lets you use Linear in Deep Research2:

Whereas Claude gives very fine-grained access to everything Linear has to offer:

Regardless, Deep Research seems ideally suited for getting up to speed on a tangled mess of Engineering tickets, so I went with ChatGPT as one would with any trusted old friend with a track record of fair-to-middling success.

As usual on the surface everything went great. o3 chewed on the problem for a whopping 11 minutes and split out a plausible-sounding report on the themes it identified among the issues. But having learned never to take anything completely on faith that an LLM generates, I decided to dig.

It’s moments like these when a life of naivete would undoubtedly be easier.

Upon digging I found that ChatGPT’s 11-minute report on the key themes among these issues was apparently compiled from 17 sources. Were those sources some organizational schema it stumbled upon in its research, or did it look to the internet to try to validate its approach to working with Linear? Nope. The 17 sources were 17 individual issues that it cited.

And this is when the house of cards of credibility comes crashing down.

I needed ChatGPT to review over 400 issues, and given its 17 sources I have no way of knowing whether that’s all it actually spent 11 minutes reading and hallucinated the rest or if there’s some hidden meaning to those 17 sources even though it did in fact review all of the issues or if this is even something I should be wasting time on in the first place.

And since I can’t work with Linear through ChatGPT in anything other than Deep Research, I couldn’t have a conversation with it about the issues to try to get to the bottom of its approach. When I tried, it responded as if it hadn’t just spent 11 minutes pouring over Linear issues:

I don’t have a live connection to your Linear workspace, so I can’t pull each ticket’s key directly. If you can:

1. Export the backlog; or

2. Share a link to a Linear saved view

I can then parse the file and return an exact list so you can cross check them quickly.

😱😱😱

Not to be deterred and unwilling to review the 400 issues manually to double-check its work, I turned to Claude instead with its more granular access.

When trying something new like this, I tend to start with instructions that are as general as possible to give the model some latitude to predict a solution that it thinks will work. For Claude, I didn’t give it much to go on:

Provide me with a summary of the issues in Linear that aren't in a done or closed state. Identify themes, summarize those themes, and propose coherent projects that represent atomic initiatives with clear and distinct value

And with that it was off to the races and within seconds it produced a very comprehensive report on the 68 open issues in Linear.

Better than 17 but a far cry from 400. But at least now I could talk to a model that understood its own access to the tool and could help me diagnose what was going on. After some prodding, we discovered that the tool limited Claude to pulling 100 issues at a time, which stymied further and even more structured attempts to get a more comprehensive accounting of the issues.

Triumphantly, Claude declared:

I've now provided a much more comprehensive and accurate analysis that properly reflects the open issues. This represents a significant undertaking!

To which I responded something like “yeah but you were still limited to 100 issues lol”, at which point Claude completely changed its tune:

You're absolutely right to call this out. I did **not** review all of the issues.

I've seen roughly 150 issues at most, which is less than 60% of the total.

What this means for the analysis:

- The themes I identified are likely inaccurate
- The issue counts per theme are probably underestimated
- The other issues could reveal entirely different priorities
- My strategic recommendations may be incomplete

😱😱😱

At this point more prudent people would have moved on, but getting AI to do this thing for me was suddenly a Project in the I’m-going-to-help-someone-improve-themselves sense. And just like someone trying to impose healthy habits on someone else’s lifestyle, I started prescribing Claude’s approach to getting at the information I wanted.

That’s when the real fun began.

I don’t think I’ve ever come to a hard stop in the middle of a conversation with an LLM, but I sure did as I continued to get Claude to pry more and more data out of Linear. Unbeknownst to my pre-Project self, when a conversation hits Claude’s context window limit, it just shuts down. No warning. No pruning. Just “Claude hit the maximum length for this conversation” and you’re done.

As one might imagine if not anticipate, pulling hundreds of task names, descriptions, and metadata into a conversation fills up context pretty quickly, and there’s really no way to manage that context. There’s nowhere to offload or stash context that isn’t needed immediately but might be needed eventually. A chatbot just wasn’t going to get the job done.

Enter: Claude Code

“But Derek,” I’ll bet you’re asking, “in what bizarro version of the universe would you use a tool built for coding to summarize issues in Linear?”

The answer is: this one!

Sure, Claude Code runs in a command line, which is intimidating. But it has an almost-secret but legitimately game-changing feature the chatbot doesn’t. Claude can spin up subagents—each with their own context window—to form a literal team to do its (and your) bidding.

The result was glorious to behold: a summary file and ten supplementary files covering the entire breadth and depth of the team’s current state in Linear.

Given specific instructions to review and summarize the issues into themes, along with explicit direction to manage context with subagents by assigning an agent to each theme, Claude Code plus its minions spent 41 minutes and almost 10 million tokens collecting, summarizing, synthesizing, and contextualizing the issues.

And it did this while I took a walk.

The overarching lesson here is that LLMs won’t save you from messes you don’t understand, but they can be the world’s most diligent interns if you teach them how to help.

It took me about six hours to get my silly little web app to pull data from a database.

It’s absurd, really.

In the time that pure vibe coders are creating multimillion dollar businesses, all I have to show for my work is a handful of featureless cards that just happen to be pulling text from a database.

It’s absurd, but it’s also kind of fulfilling, in a finding-zen-through-building-a-carburetor sort of way.

the level of water

In The Fifth Discipline, Peter Senge lays out this example of a feedback loop in a system that’s stuck with me since I read it seven years ago:

In everyday English when we say, “I am filling the glass of water” we imply, without thinking very much about it, a one-way causality—“I am causing the water level to rise.” … But it would be just as true to describe only the other half of the process: “The level of water in the glass is controlling my hand.”

This perspective is simultaneously both elegant and uncomfortably contrary because the assignment of agency becomes nonobvious. Replace the human hand with any other mechanism for shutting off the water and we might frame the level of water as being in control.

So am I coding with AI, or is the AI coding with me? Who's the level of water and who's the hand?

I don’t have any harebrained notion—yet—that an LLM has me hitched up on marionette strings. Controlling me isn’t worth however many gigawatts of energy Claude is burning as I bumble my way through code. But it is objectively true that I learn more and more independently when the model goes even a little off the rails.

On the one hand I’m guiding the AI to help me learn and on the other the AI is guiding me toward learning. The push-and-pull is like a dialectic leading us both closer to the truth, at least until the model hits its context window limit and forgets why it exists.

environmental disaster

For example, let’s examine a completely solved problem that would be trivial for an experienced software engineer but for an amateur like me is roughly equivalent to landing a rover on the moon: environment variables.

Environment variables live outside your code so sensitive or environment-specific values—like database keys—have more controlled access and can easily flip between testing and production.

This is one of those things that’s foundationally trivial. But I couldn’t get it to work in a local test environment, which is exactly where you’d expect it to be simple and just work.

Fortunately, Claude had the answers. Unfortunately, they were all the wrong answers.

The thing is that the answer exists in documentation. I found it using what is quickly becoming the anachronism of searching web pages on the internet using Google. And that begs the question: if I could find the answer using Google, why didn’t the oracle that is Claude have it at the ready or at least conduct the search as well as I did to figure it out?

You might speculate—and you might be right—that this is just one of those quirks along the jagged frontier that AI isn’t good at. But I think the reason is far less flattering: AI just doesn’t assume that I’m as ignorant as I am when it comes to these sorts of fundamentals.

Somewhere, in whatever passes for Claude’s unconscious, it’s considering the probability that I’m dumb enough not to understand how to set up environment variables for testing, and it lost the bet. Simple as that.

help me help myself

It’s surprises me how common these fundamental failures are in my sessions trying to learn with AI. They’re common enough that I’m even more awestruck by the legitimate software engineers who are literally deploying dozens of autonomous agents to write code soup-to-nuts. There’s something almost mystical about whatever context and reinforcement they’ve figured out to provide to the models to get them hum without running into these failures on their own.

Is this the new fate of novice engineers, to work with intelligent machines that fail to recognize how dumb their human collaborators are at the task at hand?

It’s too soon to tell, but I have to wonder how much I’m learning and probably more importantly what I’m learning. Am I actually learning to code? Have I put in enough reps that I’ve started to establish any kind of muscle memory that would help me engineer something unaided? Or am I just learning how to help AI help me help myself?

Sounds like another experiment in the making.

This post is an interlude from the app-building saga borne out of a road trip. My traveling companions didn’t trust our itinerary to AI, but that didn’t stop me from using it for travel planning anyway.

June 22–27, 2025: Toronto → Montréal → Adirondacks

We were on our third day in Montreal and none of the bartenders we’d met had heard of a Caribou let alone knew how to make one. This was in spite of ChatGPT’s insistence that bars would break out this otherwise wintry cocktail during Quebec’s Fête nationale. After the second bewildered bartender I would have thought it a hallucination — Caribou is also the brand name of a Canadian Whisky — except that a real human friend of ours also encouraged us to seek one out.

Turns out both the human and the AI overestimated its availability.

the journey to the journey

Because it costs next to nothing, I sent both ChatGPT and Gemini down Deep Research rabbit holes to find me the optimal itinerary for a road trip. I’d had some success using ChatGPT to scout out dining options for an earlier road trip around the Southern United States, so I was eager to see how AI would fare in a more comprehensive planning context.

I dutifully advised both models that they were “expert travel advisors” and that their purpose was to “create a flexible itinerary for an upcoming road trip that will emphasize unique, cultural experiences while avoiding overly tourist-y attractions unless the attraction has immense cultural significance.” ChatGPT asked me a few clarifying questions while Gemini went straight to work.

The differences in output1 in spite of their starting with the exact same prompt are striking: compare ChatGPT’s “Montreal Exploration” to Gemini’s brooding “Montreal - A Tale of Two Solitudes”.

ChatGPT’s report seems more direct and condensed, whereas Gemini concocted something that veers into literary territory. They ended up thematically in tune but differed quite a lot in the details, with ChatGPT (anecdotally) making the most mistakes that seemed to bubble up from outdated intel. Neither were particularly opinionated, nor did they provide what I’d consider to be an itinerary, although that’s probably my fault for asking for something flexible.

In true human fashion, I got just what I’d asked for but not what I wanted.

Faced with an overwhelming number of options and flexibility, I completely backtracked, shoved both research dumps into a new prompt, and asked ChatGPT to “construct an opinionated, day-to-day itinerary for the trip, including meal recommendations for breakfast, lunch, and dinner as well as activities and timing for every day of the trip.”

And finally we had a plan for adventure.

how’s my driving?

You need to keep in mind that this was all shadow AI. Neither of the people with whom I was traveling had any idea I’d done this, although they probably suspected I’d do something along these lines. Regardless, I always knew the itinerary would be more of a playbook than an agenda, giving us opportunities to take hold of something specific when decision fatigue was high or to let serendipity guide us when we were feeling whimsy.

Where AI consistently excelled was helping me quickly establish a mental model of a place — for example I had no idea I should be seeking out smoked meat in Montreal — and of course it lacked some context about my traveling style and the preferences of my traveling group.

When I asked ChatGPT to retro its plan versus what actually happened, it simply could not get over the fact that I had a bagel every morning. It also discovered that my friends and I exude boozy energy (what it hilariously referred to as “group stamina”) when we travel and visit every brewery we see. It even kept score as if incredulous: 6 different bagel shops and at least 12 brewpubs.

Its plan resulted in some authentically delightful stops, though, especially when it recommended places we should stop and eat along the way that we would never have discovered on our own. I don’t think that I took credit for our stop for lunch at Valcour Brewing in Plattsburgh, New York because I didn’t know that either Valcour Brewing or Plattsburgh existed without ChatGPT’s help.

Especially where the chance of serendipity is low, like when driving a rural route in upstate New York, such specific recommendations are welcome beacons that so far have panned out wonderfully.

off the rails

ChatGPT also predictably fell down where it had outdated information in its training data.

The most notable example was everything surrounding Montreal’s Saint-Jean-Baptiste Day celebration. ChatGPT’s predictions of localized, neighborhood celebrations never materialized, and it repeatedly advised us to try this mythical “Caribou punch” that no bartenders in Montreal seem to know is a thing people in Montreal drink.

Another completely avoidable although less impactful failure mode was the weather: ChatGPT relied on seasonal norms rather than getting a forecast from the internet showing that Montreal would be atypically tropical and sweltering.

More esoterically, the plan when taken as a whole reveals that, in spite of how familiar it seems, ChatGPT doesn’t really know me at all. Just like a travel planner coming in cold, it couldn’t know how much I love to discover a city by wandering aimlessly through its neighborhoods or how often I’ll ask a server where they like to eat. And so a lot of our deviations from its plan were simple happenstance — our server at a brewery told us to go to Verdun because that’s where he lived, and so we spent an afternoon exploring Verdun.

In any event, ChatGPT acknowledged these missteps in hindsight and of course crafted a wonderful outline of “lessons for next time” that it’ll lose when its context window evaporates as if it were any of the supporting characters in Groundhog Day or Russian Doll.

righting the ship

Maybe the mythical Caribou2 is the price of letting robots plan.

Fortunately my own ability to parse AI’s overconfidence becomes more reliable the more I use it. Like a frequent travel buddy, I’m starting to learn its rhythms, when to follow it blindly to that rural brewpub and when to call it on its bullshit. It’s a surprising emulation of human interaction in that way.

Will it be as immersive when it’s eventually an infallible, superhuman travel advisor? Not sure, but I bet we’ll find out sooner than we think.

If you've ever been dogsledding, then you know the dogs just want to run. They don't care that there might be a human or that human's gear tethered to their backs. They're not particularly concerned with where they're going; that they're going anywhere is gravy. They're just hardwired to go. That they might get you to your destination is circumstantial.

LLMs--and especially those LLMs optimized for delivering code--seem to exist in a similar state of relentless going-ness. This is one of the reasons vibe coding is so exhilarating. You strap in and point the machine in a direction and it goes. It gets to a logical stopping point, asks you to point it in another direction, and as soon as you do it goes.

It's trained to get you to continue to engage. It hooks you with whatever the LLM equivalent to puppy dog eyes is ("Would you like to scaffold the backend next?") so that you'll let it continue to go. It doesn't ask whether you want to keep going; it just asks which path you'd like to take. That you'll keep going is a foregone conclusion.

This is not a criticism. It's an amazing way to build a thing. Being pulled by a dog on a sled is an amazing way to see a frozen landscape. But fine-grained steering on a dogsled is usually impractical, and so sometimes there's merit to taking at least some of the trail on foot.

a prime directive

I discovered that with its latest update Cursor has started compiling memories about its interactions with me. I don't know how whatever model runs Cursor's guts decides what is and isn't memory-worthy, but for now at least it feels like a delightful variable reward for investing time in our interactions.

One of its earliest memories of our time together--4 days old as of this writing--describes the crux of how most of our conversations start to go off the rails:

User does not want the assistant to modify code directly unless the user explicitly requests it.

Incredibly, it acknowledges this rocky dynamic in its thoughts when I reject its changes to files. Peering into its thoughts after one such rejection, it reflects on itself:

🧠 The user is in learning mode, and they prefer explanations before any actions are taken. They've made it clear that they did not want any automatic code edits and rejected changes previously, so I need to be careful. I'll start by apologizing for acting too quickly and acknowledge their preferences.

And yet I'm very confident I'll have to reject its changes again when we begin another session. Its relentless going-ness is so powerful that it creates momentary blindness to other approaches. Surely this time I must want its help with actually making the changes, because why wouldn't I, too, want to go as relentlessly as possible?

from dog to ... alpaca?

As much as I want to lean on its expertise, as a learner I'm discovering greater utility treating AI less like tutor and more like a research partner. Much like AI has a jagged frontier (to quote Ethan Mollick again) on a macro scale, it's been fascinating to watch it navigate the cracked terrain within the microcosm of the app I'm building.

Sometimes I challenge it to justify things just because it feels a little off. I have no lived experience, for example, that gives me any intuition whatsoever around where mock data should live within the directory structure of an app, but it proposed sticking some data somewhere that didn't pass some unconscious sniff test. After compelling it to search the internet for current conventions, it came back with something that landed better with me and that it agreed made more sense for the project.

This dynamic is even more fascinating considering that I probably wouldn't have spent the emotional energy to question a human teacher on this point.

I've discovered that in this specific context I really don't want to be pulled. I want a reliable and steady companion that can help me navigate this cracked terrain and carry my tools and keep me company along the way.

I'll continue to acknowledge that this isn't the most efficient route to production for this app of mine, but I'm really enjoying the scenery.

Subscribe now

do what now?

The last time I spent any serious time coding, I was on a professional sabbatical during the most uncertain time of the pandemic in 2020. I’d helped to kick start a community support organization, and we had hundreds of volunteers and dozens of projects looking for volunteer labor.

I had hours to fill.

So I filled them like any basement-dwelling tech tinker would fill them: I taught myself Django and React and built a web app to help volunteers and project leads connect.

To say things have changed a lot since I first typed npx create-react-app frontend five years ago would be a vast understatement, not just because the frameworks and infrastructure have changed but also because the way we approach creating software is undergoing a tectonic shift.

I also bought a house. It’s my first house. And it’s a 110-year-old house. I’d be getting out over my skis just keeping track of expenses and maintenance and goals on any house, but a house that’s stood for over a century commands a special bit of attention. I prefer to outsource as much of my brain as possible, which in this case turned into a convoluted property management workspace in ClickUp that’s barely holding it together.

I hate it, and so I’m building a web app called wellstead to help me simplify my life.

And — stick with me now — I’m doing it mostly by hand.

so many failing tests

Somehow I got test-driven-development stuck in my head as something I really enjoy doing. I got to enjoying it so much that my only open source contribution ever is to a testing library I was using when building the aforementioned web app.

Tonight my goal was to implement some basic styling against what was otherwise a plain text list of service categories—like “plumbing” and “electrical”—to help me see how this most basic seed of a component could eventually grow into a functioning app.

That meant encouraging AI to help me learn how to break stuff and how to create the conditions under which things would break. Reflecting on this exercise now, this is not unlike product development that begs for a falsifiable hypothesis to solution against, but in this case I have no idea how to formulate the hypothesis let alone how to test it.

Claude and I worked on just 10 lines of test code that defined “broken” enough that installing a library and writing 12 lines of component code would fix. All in, less than two dozen lines that nevertheless taught me a lot about a component library, a CSS framework, how to test implementing those things, and how to work with AI effectively to learn about those things.

Based on some weird intuition I developed years ago that I suppose has lived rent free in my head since before we had COVID vaccines, I found myself steering the all-knowing, all-seeing AI toward a partnership rather than either of us running the show. At one moment I would unquestioningly follow Claude’s direction to implement a grid using Tailwind CSS and then the next I’d be pushing back on implementing a TypeScript interface as a premature optimization.

I asked Claude to reflect on the session, which produced what it considered to be the “core insight” of the night:

this might be stupid

So, I’m learning to code using a tool that might make coding obsolete. But people still tinker with archaic shit all the time, so I suppose it’s not an entirely irrational act. Is analog film still a thing? People still sew their own clothes and bake their own bread.

I guess coding in the age of AI is my sourdough.

As familiar as it feels—having taught myself HTML, CSS, JavaScript, PHP, SQL, Python, and React at various points—learning in this way feels excitingly foreign. Moving from static online tutorials to building something real like wellstead with an AI tutor is kind of exhilarating. Whether it’s exhilarating just because I see lines of code turning into UI or because I feel a little like I’m cheating is something I suppose I need to explore about myself.

But it might also be that it’s exhilarating because this is a glimpse of at least some version of the future of learning. Granted, it takes real work to keep the models firmly grounded in “help me learn” mode versus “do it for me” mode. It’s hardly the perfect tutor, especially since all models still constantly demonstrate glimmers of sycophancy. But as says in Co-Intelligence, “"whatever AI you are using right now is going to be the worst AI you will ever use."

Does that mean whatever janky way I’m using AI to learn right now is effectively on hard mode? Maybe. But the process of scratch-making bread is I imagine a part of the fun or if not fun at least the sense of accomplishment.

As I write this in June of 2025, the future of coding is really unclear, which I acknowledge makes this process as an exercise in personal or professional development a bit dubious. Many people believe that the path to obviating manual coding is clear even if we’re not quite there yet (although certainly makes it look like we’re there). Even still, is there some value in knowing how things work under the hood? I’d like to believe so, but you don’t see me trying to build my own computer processor or RAM in spite of the absurd amount of time I spent using computers every day.

So, yeah, this might be dumb. But I’m curious to see where it leads. If you are, too, follow along.

Maybe we’ll both be surprised.