Patrick Scannell is a technology executive and author who spent twenty-five years building the internet, mobile, IoT, and AI, and the past decade researching how those technologies reshape human thought. Ahead of our September summit, DFC draws on a conversation with Pat, during which he raised a framework for understanding complex systems and catastrophic failure. Here’s what it means for design professionals.
At 7:05 PM on July 17, 1981, the fourth-floor walkway at the Kansas City Hyatt Regency gave way, crashing into the second-floor walkway below and collapsing into a crowd gathered for a tea dance. 114 people died.
The cause was a small, uncaught change. The original design called for a single steel rod running continuously from the ceiling through both walkways, so the fourth-floor connection only had to bear the weight of the fourth-floor walkway itself. The fabricator, Havens Steel, found that impractical to build and proposed a two-rod version instead, anchoring a second rod to the fourth-floor beam and hanging the second-floor walkway from it. That change doubled the load on a connection that was never reengineered to hold it. The structural engineering firm, G.C.E. International, approved the change without rerunning the calculations. Nobody checked. The building stood for a year, used daily, before the connection gave way.
Sociologist Charles Perrow studied disasters like this and found that catastrophe rarely comes from one dramatic cause. It tends to come from systems coupled tightly enough that an ordinary, survivable mistake has nowhere to be caught before it does damage. The Hyatt Regency isn’t a textbook case of what Perrow called a normal accident; his own examples usually involve several independent failures colliding by chance. But one part of his framework does hold: in a complex system, one uncaught mistake has nothing to slow it down as it dominoes into catastrophe.
Patrick Scannell raised Perrow’s work in a recent conversation with DFC, in the context of AI. In that conversation, Pat describes AI as a magic wand: point it at something, and it executes with more power and speed than was possible before. Ask it to draft construction documents, model a structural system, or generate code, and it does, immediately, at a scale no team could match using traditional methods.
But a wand doesn’t ask whether the spell is a good idea. It casts it. Cast well, that power builds things no team could have built before. Cast carelessly, it builds the mistake just as fast, and at the same scale.
But a wand doesn’t ask whether the spell is a good idea. It casts it.
That carelessness isn’t a new problem for the design professions. It’s one of the oldest ones. G.C.E. didn’t fail because the math was too hard. It failed because nobody thought to ask whether the new design still held. Negligence, overconfidence, and the assumption that things will be fine, these are old habits, not new risks introduced by any tool.
What AI does change is how far an uncaught mistake can travel before anyone notices. G.C.E.’s mistake lived in one set of drawings, on one project. An error embedded in an AI-generated model can appear across every project built on the same basis, before a single person has reason to look twice. And AI makes that harder to catch precisely because it removes the tells. A rushed hand-drawn detail looks rushed. AI-generated work looks finished whether or not it’s correct, which gives a person no visible reason to slow down and check. A person who never built the habit of double-checking their own work doesn’t gain one by adopting AI. If anything, they lose their best remaining reason to start.
It’s tempting to treat this as a problem for later, while today’s AI tools still struggle to produce a convincing rendering, let alone a structural model ready to submit for permit. But the tools are advancing faster than most people are rebuilding their habits. The risk was never really the wand. It’s who holds it, and whether they’ve built the habit of asking before it’s needed. That habit is worth building now, while there’s still time to build it deliberately instead of under pressure.
The risk was never really the wand. It’s who holds it.
Pat sees this moment as hopeful, not dire. “I don’t think we’re at a ‘problem’ at this point,” he says. “We’re at an inflection point that carries existential risk for the planet and everyone on it. I have no doubt about that. But I also believe we can use these technologies to leapfrog forward.”
How are you using the magic wand? And are you the one asking the questions it won’t ask for you?
Patrick Scannell will be in the room at the DFC Leadership Summit, September 16-17 in Cambridge, MA, continuing this conversation about human, technology, and the business of design. Join us.