Making a legal AI show its sources, down to the article.
Lawyers wouldn't act on an AI answer they couldn't check, so a fast answer still cost them an hour of research. I designed the system that lets every claim in Malakah's answers open the exact article of Saudi law it came from: the citation pattern lawyers see, the content model that makes article-level citations possible, and the tool the team uses to measure whether retrieval finds the right article.
- Problem
- 10 of 12 lawyers said they would re-check any AI answer that didn't name its source. Unverifiable answers didn't save time; they moved it.
- Reframe
- Not “make the answer more accurate” but “make every claim cheap to check”. The thing to design was the path from a sentence to the law behind it.
- What I designed
- Four citation patterns, compared on how much checking each left to the lawyer; a legislation CMS that breaks laws into articles, annexes and amendments; an evaluation tool that scores retrieval with precision, recall, F1 and nDCG.
- Key decision
- Cite at the level of the article, not the law file, and open it beside the answer with the cited text highlighted and its version shown.
- Outcome
- The claim-to-source pattern carried into the shipped workspace. Malakah serves 1,000+ legal professionals; after launch one lawyer told us “It saves me 90% of my time.”
An answer you can't check is a research task, not an answer
Malakah answers legal questions in Arabic and English for Saudi legal teams. In 12 interviews with in-house counsel, law-firm associates and business approvers, the reason was simple: a lawyer is accountable for every piece of advice they give. If they can't see where a claim comes from, they look it up themselves.
Saudi law also changes. Articles are amended, extended and annexed, so “the law says” is only useful if you know which version said it.
The design goal: from any claim in an answer, reach the exact article it rests on, in its current version, in one click and without leaving the conversation.
A citation is only as good as the three layers under it
A source link in the chat is the visible part. It only works if the law is stored in pieces small enough to point at, and if retrieval reliably finds the right piece. So I treated it as one system with three layers, each with its own users.
Four ways to show a source, and what each costs the lawyer
I judged every option by one question: how much work is left for the lawyer between reading a claim and trusting it?
ANo sources
The answer reads well, but every article number is a claim the lawyer has to look up elsewhere. This is exactly the answer 10 of 12 said they would re-check.
BA list of law files
Now the sources exist, but at the wrong level. The panel tells you which law, not which article, and every row looks the same. The lawyer still has to open a file and hunt for the passage, and nothing ties a given sentence to a given file.
CA link on every claim, opening the exact article
This is the pattern that worked. The link sits on the claim itself, so the connection between sentence and source is never in doubt. The panel opens beside the answer, so reading the source doesn't mean losing the conversation, and the highlight shows the exact passage instead of a whole law.
DShow which version of the article
A correct article in an outdated version is still wrong advice. An icon beside each cited article opens its history: the date, the type of amendment, and the text before it changed.
| Option | What the lawyer gets | Work left to verify |
|---|---|---|
| A · No sources | Article numbers in prose | Find and read each article elsewhere |
| B · File list | The right laws, not the right passages | Open a file, search for the passage, match it to a sentence |
| C · Claim links | The exact article, highlighted, beside the answer | Read the highlighted text |
| D · + Version | The article and when it last changed | Confirm the version is current |
What shipped
The claim-level pattern carried into the workspace. In the shipped version the source opens as a third panel next to the chat and the contract, at the right page, so a lawyer can check a claim while drafting without switching tools.
The trade-off I accepted
Cite whole laws. Easy to build and always “correct”, but it pushes the real work, finding the passage, back onto the lawyer.
Cite articles. Far more work behind the scenes, because every law has to be stored article by article and retrieval has to find the right one. That cost is what the next two sections are about.
Making the law addressable
You can only link to an article if the system knows where each article starts and ends. Saudi legislation isn't a flat list: articles sit inside parts and chapters, and laws carry annexes, extensions and amendments. So I designed a CMS where the content team rebuilds each law as a tree before it's published.
- A tree that matches how lawyers cite. Chapter → Part → Article, with Annex, Extension and Amendment as their own block types, so a citation can point at any of them.
- Metadata that answers “how binding is this?” Each law records its legislative instrument, what issued it (for example a royal decree), when it takes effect and when it expires, so an answer never cites a law that is no longer in force.
- Source priority. Sources are tagged as definitional, authoritative or informative, so answers can rank and label them by how binding they are, and tell “the law says” apart from “a guideline suggests”.
- A translation check next to Publish. Each law shows whether its translation has been verified, because a citation to an unchecked translation is a citation to the wrong text.
Measuring whether the right article comes back
Article-level citations raise the stakes: a wrong article, shown with confidence, is worse than no citation. The AI team needed a routine way to check retrieval quality, not a one-off script. I designed an evaluation tool inside the admin view.
- Test sets as files. The team uploads a CSV of questions and the articles that should answer them, then runs it manually or on a schedule, so quality is checked after every content or model change.
- The right metrics side by side. Recall shows whether the right article was found at all; precision shows how much noise came with it; nDCG shows whether the right article ranks first, which is what decides the citation a lawyer sees.
- Runs as a history. Every run keeps its status, date and scores, and can be viewed, downloaded or deleted, so a regression can be spotted and traced to a change.
How I'd know it's working
The goal was trust you can measure, so the metrics follow the user's path from answer to verified answer. Retrieval scores are already tracked in the evaluation tool; the product metrics below are the set I'd put in place next.
“It saves me 90% of my time.”Lawyer using Malakah, feedback after launch
One lawyer's own estimate, not a measured average.
What I took from this
Trust is a path, not a badge. “Sources” as a feature didn't help until the path from a sentence to its article was one click long.
The visible pattern depends on invisible structure. The citation link was the easy part to draw. The CMS and the evaluation tool are what made it true.
Design for the people behind the AI too. The content team and the AI team were users of this system as much as the lawyers were.
Next, I'd test citation density. One link per point worked for short answers. For long answers I'd test grouping citations per section against keeping them inline, measured by time to verify.