Action Precedes Insight: More Lessons from AI Experimentation
This past week, I read Gary Klein’s Sources of Power, a book about how experts make decisions in high-pressure, fast-moving situations with significant uncertainty.
The main finding in his research was that professionals like firefighters, paramedics, and military officers don’t follow the economist’s ideal—generating a set of options, rigorously analyzing them, and then deciding on the “best” course of action. Instead, they’re more likely to cycle through a mental Rolodex of solutions and choose the first good enough solution. For novel situations, they conduct mental simulations of what’s likely to occur with each potential solution, leveraging their understanding of the fundamental principles of their fields.
The intuition-based decision-making process is necessary because the time pressure does not allow for rigorous analysis. And it’s still effective because true experts have real experiences and knowledge to draw upon (it wouldn’t work for novices).
I was thinking about those lessons while building with AI this week.
When High-Fidelity Becomes Low-Cost
When I was a product manager, my teams would always keep ideas in low-fidelity mode for as long as possible. That’s because it was inexpensive to sketch 20 paper prototypes (and redraw them after customer testing) relative to using designers’ and engineers’ time to build working prototypes, which are often harder to update. The cost and faster iteration of paper prototypes are more valuable than the degradation in feedback quality from customers experiencing a “fake” version of the product.
When building a web application this week, I realized this dynamic was flipped with AI support. (The tool dynamically interviews nonprofit leaders to help them decide how to approach strategic planning, trained on content from my book Strategic FUEL for Nonprofits.)
When I provided instructions to Claude about what I wanted, it would reply, paraphrasing, “Here are three options to consider. Here are the pros and cons of each and my recommendation.”
Any other time I’ve needed to write code, I would have surely decided on one option. However, I realized the best response to the AI tool was something like, “Create a mockup of each option, and I’ll decide after seeing it.” In other words, I could create a higher-fidelity experience at the low cost of a paper prototype and still iterate quickly.
Beyond the ability to move faster, AI fundamentally pushes back the moment of decision—and that’s crucial. Instead of committing to a single direction based on limited testing in the lab, these tools enable building multiple complete solutions and testing them in real-world scenarios. That enables higher-quality feedback on whether people really intend to buy and use the product or service.
You Don't Know What You're Building Until You Build It
When we try to make decisions, we often start with what we want to accomplish—that’s how we know Option A is better or worse than Option B. But once we try to implement the decision, we not only learn more about the options, but we might also realize the goal itself should be different.
For example, halfway through building the app, I thought, “Actually, I need to design this completely differently.” In some ways, that made the initial effort a “waste,” but the insight was only possible once enough work had been done to have an informed reaction. It’s the same dynamic when we start writing an email, only to realize in the middle that we wanted to communicate something different, or that we just need to call the person.
The experience helped me remember something from my previous product roles—that you don’t really know what you’re creating until you start to create it. It’s action that enables the initial failure and new insights about what you’re really trying to accomplish. As Klein writes in Sources of Power, these failures, “when properly analyzed, are sources of new understanding about the goal.”
The implication for those considering AI-driven solutions for their teams is to just start building tools—maybe even before you precisely know what you need or what would be strategically ideal. One could spend months analyzing options and mapping the perfect solutions, but the process of creation is likely to yield greater insights about what’s useful than simply thinking about it.