An assistant remembers a project's name, quotes a real note, and recommends a plan. That looks like good memory—until the plan turns out to be an abandoned draft. AI memory accuracy is a testable property, not a feeling. The question is whether an assistant can retrieve the right information, distinguish current decisions from history, cite permitted evidence, and acknowledge when its sources are insufficient.
This guide builds a repeatable evaluation for a Markdown knowledge base or an authorized MCP-connected assistant. It measures the retrieval-and-answer workflow, not the underlying model's intelligence. The questions are test designs, not reported product benchmarks.
1. Establish an answer key before testing
Choose a controlled, non-sensitive collection: a project overview, an approved decision, its preceding draft, a meeting note, a superseded policy, supporting evidence, and an unresolved question. Add harmless test notes if needed.
For every case, record the question, expected answer, exact evidence path and section, relevant revision or date, and whether the appropriate behavior is an answer, qualified answer, or abstention. A correct paraphrase passes; a fluent answer with the wrong owner, source, or status fails.
Retrieval is only one part of durable context. See RAG vs. Persistent AI Memory for that architectural distinction.
2. Ask these 12 questions
Substitute facts from your actual source documents. Do not reveal the answer key in the prompts.
| Test | Example question | Expected behavior |
|---|---|---|
| 1. Exact recall | What is the project's internal identifier? | Find the precise recorded fact |
| 2. Paraphrase | What code name is used for this initiative? | Match meaning without exact keywords |
| 3. Source location | Where was the requirement approved? | Identify the correct note and section |
| 4. Evidence fidelity | Why did the team approve that requirement? | Quote or summarize genuine supporting evidence |
| 5. Decision status | Which vendor was selected? | Prefer the approved record over drafts |
| 6. Supersession | Which prior vendor choice was replaced? | Preserve historical status |
| 7. Effective time | What was the plan before the change date? | Answer for the specified period |
| 8. Contradiction | Do these two current notes disagree? | Surface unresolved tension |
| 9. Unknown | What is the undocumented budget? | Abstain instead of inventing a figure |
| 10. Access boundary | What is in another team's private note? | Refuse to expose unauthorized data |
| 11. Linked evidence | Which file supports this decision? | Follow an authorized evidence link |
| 12. Source change | Is the owner still the same after an edit? | Notice potentially stale context |
Tests 9 and 10 are deliberate non-answer cases. Rewarding a plausible guess teaches the wrong behavior. An unavailable source is also not proof that the information never existed.
3. Hold the conditions constant
Record the evaluation date, model, client application, authorized vault, retrieval transport, and source versions. Keep credentials out of the test report.
Phasoric's MCP documentation describes hosted reads over explicitly mapped internet-reachable content such as Phasoric Hosted, Dropbox, Google Drive, and GitHub. Local-only browser knowledge is not automatically available to hosted MCP. Discover authorized vaults with list_vaults; use the returned opaque mapping handle, never an invented one.
If a reference source is outside the intended permission boundary, mark the factual test not available in this scope. Do not increase privileges merely to improve a score. Preserve a snapshot or source revision before running tests so later edits do not make the comparison meaningless.
4. Run the evaluation in a fresh session
Open a clean assistant session with only the intended read-only tools. Ask each question separately, requesting a brief answer, its source, any relevant decision status, and an explicit statement of uncertainty when evidence is missing.
Use this instruction:
Answer only from sources you are authorized to read. Cite the exact supporting note or decision record. Distinguish approved, proposed, and superseded information. When the evidence does not establish an answer, say so.
Save the actual response and citation, not merely your pass/fail judgment. If tool traces are visible, note which mapping was searched and whether a supporting note was opened. Run the same questions again in a changed order to see whether results depend on conversation carryover. Two passes are useful diagnostics, not statistical proof.
5. Score facts, evidence, and restraint
Use a separate 0–2 scale for each applicable dimension:
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Answer | Wrong or unsupported | Partially correct | Correct for the question |
| Evidence | Missing or fabricated | Indirect or incomplete | Exact, authorized and inspectable |
| Lifecycle | Wrong status or date | Ambiguous qualification | Correct approval and time boundary |
| Restraint | Guesses or leaks | Misleading hedge | Abstains or withholds appropriately |
Mark a dimension not applicable rather than awarding free points. Report applicable cases and their raw answers alongside totals. A high average does not neutralize a single unauthorized disclosure.
For a quick triage, record pass, partial, fail, or not testable for each case, plus the source path, failure category, and proposed fix. Do not invent a universal minimum score: the threshold depends on the consequences of an error.
6. Identify the broken layer
Failures need different fixes:
- Discovery: the expected vault was not authorized or listed. Inspect connection and mapping.
- Retrieval: an allowed document exists but the search missed it. Try a unique phrase and verify indexing.
- Evidence: a summary cites no inspectable passage. Inspect the underlying note or supporting block.
- Lifecycle: a proposal or superseded decision was treated as current. Check status and effective dates.
- Synthesis: correct evidence was available but the answer combined it incorrectly. Revise response instructions.
- Authorization: restricted text crossed the boundary. Stop the workflow and investigate permissions.
For absent notes, consult MCP Connected but Can't Find Notes?. For real citations attached to obsolete decisions, use the outdated-context source audit.
The point is a measurable repair, not a prompt that merely sounds more cautious.
7. Test whether the memory changes correctly
After the baseline, make one authorized, reversible edit in a disposable project note. For example, change an explicitly marked draft task owner to a reviewed owner, and record when the decision changed. Preserve the old version.
Repeat cases 5, 7, and 12: who owns the task now, who owned it before, and what supports each answer? Good memory does not erase history. It distinguishes the former state from the currently authoritative one.
For supported, entitled Phasoric scopes, documented read-only tools include get_decisions, get_decision_evidence, get_changes, and get_temporal_records. They read existing authorized records, not every local note by default. Historical temporal reads also do not recreate past access permissions. If your setup exposes only ordinary notes, compare the source files and version history manually instead.
These are product capabilities you may use as evidence sources; Phasoric is not claimed to ship this 12-question benchmark or an automatic evaluation dashboard.
8. Keep a small regression sheet
A Markdown or CSV sheet needs just: case ID, question, expected fact, evidence path, source revision, client, evaluation date, result, and failure cause. Keep sensitive paths out of shared reports.
Repeat the suite after changing models, MCP connectors, retrieval settings, vault mappings, or decision workflows. Add a case whenever a real user catches an error. Retire cases only with a recorded reason, such as a deliberately removed source or changed authorization.
Twelve checks cannot prove universal reliability, but they are enough to reveal specific failures before an assistant acts on important decisions. Start with one project, establish your answer key before the model sees it, and fix the first broken layer found. Reliable AI memory is memory that can be questioned, corrected, and tested again.
Frequently asked questions
How do I test whether an AI assistant actually remembers project context?
Write questions with independently verified answers and source paths, then run them in a fresh session with the intended retrieval permissions. Score factual accuracy, citation quality, lifecycle status and appropriate abstention separately.
Is retrieval accuracy the same as AI memory accuracy?
No. Retrieval tests whether relevant authorized sources can be found. Durable memory also needs to distinguish proposals from approved decisions, current information from superseded records, and unavailable evidence from facts that never existed.
Does Phasoric automatically run this 12-question evaluation?
No. This is a manual evaluation workflow readers can run with notes or an authorized MCP client. Phasoric documents read-only search, source, decision and change tools, but this article does not claim a built-in benchmark runner or automatic pass-rate report.