A few weeks ago, I spoke with a GC who had just wrapped up an expensive AI pilot. Six months, multiple stakeholders, plenty of vendor support. On their NDA template, the tool was flawless. It flagged deviations, suggested fixes, even produced a neat dashboard of risk.
Then they tried it on a supplier contract drafted by the other side. And it broke.
Not because the AI wasn’t capable. But because it had no rules to work with.
I’ve seen this movie before. The demo dazzles, the pilot starts strong, and then reality sets in. Suddenly, the lawyers are fielding alerts they don’t trust, contracts aren’t moving faster, and the AI is quietly switched off.
This is why 95% of AI pilots fail. Not because the technology is weak. But because the implementation is.
Templates Are Not Playbooks
Lawyers love templates. They’re safe. They carry the weight of precedent. They look like certainty on the page.
But in contract review, templates collapse.
I’ve watched teams hand an AI their “gold standard” NDA and expect it to act like a playbook. The logic goes: if this is how we draft, surely this is how we should review. But templates don’t say what happens when reality deviates. They don’t capture the judgment calls, the fallback positions, the “acceptable enough” outcomes.
So the AI looks for exact matches. And when it doesn’t find them, it either escalates everything or misses the point entirely. On your own paper, it can limp along. But the moment you put a third-party draft in front of it, the pilot collapses.
That’s not an AI problem. That’s a lawyer problem. We’re giving the machine the wrong inputs.
Why Lawyers Resist Playbooks
Here’s the deeper issue: lawyers resist writing playbooks because it forces us to expose our judgment.
We like nuance. We like saying “it depends.” A playbook doesn’t let us hide behind that. It asks:
- Which protections are non-negotiable?
- Where will you flex, and how far?
- At what point does this deal need a human to intervene?
If you can’t write it down, maybe it isn’t a standard.
That’s uncomfortable. It feels like a loss of control. But in reality, it’s the opposite. Codifying your rules is how you keep control when the AI starts to do the heavy lifting.
Own Paper vs. Third-Party Paper
Here’s another reason pilots fail: most teams don’t distinguish between reviewing their own paper and reviewing someone else’s.
On your own paper, the AI can enforce with surgical precision. Clause 1.2 must read this way. No exceptions. The playbook acts like a red pen.
On third-party paper, that logic doesn’t hold. Clauses won’t match your templates. The language will be messy, negotiated, drafted by a lawyer you’ve never met. If you try to enforce exact matches, the AI will fall over.
The only sensible approach is evaluation. Liability cap within range? Governing law on the approved list? If yes, move on. If not, escalate.
Too many teams design playbooks as if every contract will behave like their own paper. That’s fantasy. The reality is that most of your work will be on third-party drafts. If your pilot ignores that, it’s already doomed.
Writing Rules AI Can Actually Use
Now let’s talk about the rules themselves.
I’ve read playbooks written like policy manuals. Paragraphs of soft language, exceptions, caveats, and “preferred positions.” That’s how lawyers talk. But it’s poison for AI.
Machines don’t understand “preferred.” They don’t know what to do with “should” or “where appropriate.” They need yes or no. Pass or fail. Escalate or accept.
That’s why I like the Identify → Check → Act framework.
- Identify what the AI should find.
- Check the condition against your rule.
- Act based on the outcome.
It’s blunt. And that’s the point.
If you can’t reduce a rule to something the AI can test in under a second, it’s not a rule. It’s a preference. And preferences don’t scale.
The Real Shift
This is bigger than contract review.
For years, lawyers have traded on judgment wrapped in ambiguity. “It depends” has been our shield and our currency. But AI can’t run on “it depends.” It needs rules.
And here’s the uncomfortable question: if you can’t articulate your rules, do you really have them? Or have you been relying on instinct all along?
Playbooks aren’t just about efficiency. They’re about control. If you don’t write the rules, someone else will. Vendors. Opposing counsel. Or worse, the AI itself.
The danger isn’t that AI makes mistakes. It’s that AI makes decisions on someone else’s terms because we never defined our own.
That’s why I believe lawyers need to see themselves as system designers. We’re not just reviewing contracts anymore. We’re designing how contract review gets done. And if we don’t take that role seriously, we risk becoming bystanders in our own profession.
Want to go deeper?
You can now watch the full replay of our webinar How to Build AI Playbooks for Contract Review. In it, we cover the ICA framework, own vs. third-party paper, and how fallbacks actually work in practice.
Tags: Contract Review, AI
