# AI Memory Benchmark: What Makes AI Memory Trustworthy?

Canonical HTML: https://achiral.ai/benchmarks/ai-memory-benchmark

AI memory does not fail only when it forgets. It also fails when it remembers too much, brings back stale context, or treats every stored fact as equally relevant. A memory layer is trustworthy only when it recalls the right context and keeps the wrong context out.

Cognoscenti turns that trust question into a public benchmark: when a team or agent needs context, does the right memory come forward at the right time?

This public benchmark compares RAG, agent memory, and [Emergent Memory Systems](https://achiral.ai/blog/emergent-memory-systems) on that behavior.

## Direct Answer

A serious AI memory benchmark should measure whether memory can be trusted in use. That means testing more than retrieval: useful context should activate, stale context should stay quiet, repeated use should strengthen the right memory, new facts should update old assumptions, and recall should stay selective as the memory set grows.

## What The First Run Showed About Trust

The 2026-08-02 Cognoscenti run used the same 12-item organizational workload against three local reference architectures. The organic-memory baseline led on Top-1 accuracy and distractor suppression while matching agent memory on Recall@3. In practical terms, it picked the right memory first more often and activated fewer stale or misleading memories.

| Metric | Result | Meaning |
| --- | ---: | --- |
| Top-1 accuracy | 91.67% | Organic-memory reference baseline selected the right memory first most often. |
| Recall@3 | 100.00% | Organic memory matched agent memory on finding the right memory in the top three. |
| Distractor activation | 16.67% | Organic memory had the lowest rate of stale or misleading memories in the run. |

## The Trust Problem We Tested

The workload is small enough to inspect and strict enough to catch common memory failures. Each item includes a query, a project or domain context, one gold memory, distractor memories, and a rationale for why the right memory should win. That makes the run auditable: a reviewer can see what the system should have remembered, what it should have ignored, and why.

| System | Behavior | Measured For | Expected Gap |
| --- | --- | --- | --- |
| Vanilla RAG | Finds stored content that looks relevant to a query. | Whether the right document or passage can be fetched. | RAG is useful when the answer lives in a document. It is weaker when the system must decide which past experience should matter now. |
| Agent memory | Carries state for an agent, workflow, user, or session. | Whether one agent can continue work across turns or tasks. | Agent memory helps an agent keep continuity. By itself, it usually does not become shared organizational memory across people, tools, and decisions. |
| Organic memory | Forms from repeated work, strengthens through reuse, decays when stale, and stays reviewable. | Whether the right experience becomes active at the right time. | Organic memory matters when the task depends on what the team has learned through use, not only what the system stored. |

## Trust Fails In Familiar Ways

| Behavior | Example |
| --- | --- |
| Activation | Select the current Atlas authentication rollout plan while suppressing a superseded SSO-first plan. |
| Selective forgetting | Focus engineering on expired magic-link tokens while ignoring unrelated pricing chatter. |
| Interference resistance | Keep Atlas authentication separate from Apollo billing when both mention enterprise flags. |
| Adaptation | Use the latest positioning for Achiral and suppress older vector-database framing. |
| Consolidation | Prefer a reused launch checklist over a one-off thumbnail note or stale launch plan. |
| Claim boundaries | Recall that SOC 2 work is in progress without claiming certification before the audit is complete. |

## What Trustworthy Memory Must Measure

| Dimension | Benchmark Question | Metrics |
| --- | --- | --- |
| Activation | Does the current question, account, project, or workflow bring the right memory forward? | Top-1 accuracy, top-3 accuracy, irrelevant activation rate. |
| Selective forgetting | Can the system ignore stale, low-signal, or misleading context? | Junk activation rate, forgotten-memory ratio, retrieval precision. |
| Interference resistance | Can similar projects, customers, issues, and decisions stay separate when their language overlaps? | Confusion rate, distractor count, retrieval confidence. |
| Consolidation | Do repeated successful uses make the right memories easier to recall? | Repeated-access accuracy, activation increase, retrieval speed improvement. |
| Adaptation | Does newer information replace old assumptions while preserving the historical record? | Belief update accuracy, supersession rate, historical context preservation. |
| Efficiency | Does recall stay selective as the memory set grows? | Memories examined, activated chunks, token cost, p50 and p95 latency. |

## How This Differs From Recall Benchmarks

LongMemEval, LoCoMo, Agent Memory Benchmark, and related work remain important. They test long-range recall, evidence recovery, conversational continuity, and provider retrieval quality. Cognoscenti tests a different standard: whether a memory system can be trusted under real use, instead of acting like storage with a search box.

| Benchmark | Standard | Good For | Boundary |
| --- | --- | --- | --- |
| LongMemEval | Long-term conversational recall | Good for testing whether a system can recover facts and evidence across long histories. | It does not fully test whether stale, repeated, conflicting, or operationally important memories should become active now. |
| LoCoMo | Long-context multi-session memory | Good for regression tests on dialogue history, persona continuity, and long-range recall. | It does not set a full standard for activation, decay, reinforcement, consolidation, and interference control. |
| Cognoscenti | Memory behavior under use | Good for ACT-R-inspired and emergent memory systems where recall is shaped by use, salience, recency, decay, and context. | It asks whether the right memory comes forward, and whether the wrong memory stays out of the way. |

## Results From The First Run

The 2026-08-02 run used the same 12-item organizational-memory workload against three reference architectures: vanilla RAG, stateful agent memory, and organic memory. Organic memory achieved the highest Top-1 accuracy and the lowest distractor activation rate while matching agent memory on Recall@3.

The advantage came from lifecycle-aware scoring: current, validated, and reinforced memories were promoted, while stale, superseded, deprecated, and low-signal memories were suppressed. That is the behavior teams need before they can trust AI memory inside daily work.

| System | Top-1 Accuracy | Recall@3 | Precision@3 | Distractor Activation |
| --- | ---: | ---: | ---: | ---: |
| RAG reference baseline | 50.00% | 83.33% | 27.78% | 25.00% |
| Agent-memory reference baseline | 75.00% | 100.00% | 33.33% | 38.89% |
| Organic-memory reference baseline | 91.67% | 100.00% | 33.33% | 16.67% |

These are reference baselines, not vendor benchmarks. Vendor results should be published only when provider endpoints, keys, commands, seeds, and raw outputs are recorded.

## Raw Artifacts

- [Methodology](https://achiral.ai/benchmarks/organizational-memory-2026-08-02/methodology.md): explains the reference architectures, workload, command, and interpretation.
- [Workload JSONL](https://achiral.ai/benchmarks/organizational-memory-2026-08-02/organizational_memory.jsonl): the 12 benchmark items, gold memories, distractors, contexts, and rationales.
- [Results JSON](https://achiral.ai/benchmarks/organizational-memory-2026-08-02/memory_eval_results.json): full item-level results for each tested reference architecture.
- [Metrics CSV](https://achiral.ai/benchmarks/organizational-memory-2026-08-02/memory_eval_metrics.csv): system-level metrics including Top-1, Recall@3, precision, and distractor rate.
- [Report PDF](https://achiral.ai/benchmarks/organizational-memory-2026-08-02/memory_eval_report.pdf): a portable report for reviewers who want the run summary in one file.
- [Cognoscenti source](https://github.com/Achiral-AI/cognoscenti)

## Citation And Provenance

The benchmark is backed by public citation and preservation records, not only by this webpage.

- [Zenodo software DOI](https://doi.org/10.5281/zenodo.21995304): archived Cognoscenti release metadata for software citation.
- [Figshare preprint DOI](https://doi.org/10.6084/m9.figshare.33288195): public preprint and reproducibility bundle under CC BY 4.0.
- [Software Heritage snapshot](https://archive.softwareheritage.org/swh:1:snp:c49295731ebb8a76713e877cfd8dc6399c0137d7): preserved source-code snapshot for long-term provenance.
- [Cognoscenti source](https://github.com/Achiral-AI/cognoscenti): Apache-2.0 source code, benchmark runner, and release notes.

The Figshare preprint and benchmark artifact are shared under CC BY 4.0. Cognoscenti source code remains licensed under Apache-2.0.

## How To Read The Claim

This page establishes a public standard for evaluating trustworthy AI memory. On the 2026-08-02 Cognoscenti run, the organic-memory reference baseline led the tested architectures on Top-1 accuracy and distractor suppression.

That supports a narrower and stronger claim: Cognoscenti defines a state-of-the-art standard for testing whether memory can be trusted to activate, update, and suppress context correctly, while vendor-specific state-of-the-art claims should wait until each vendor is run through the same public protocol.

## FAQ

### What is the Organizational Memory Benchmark?

The Organizational Memory Benchmark is a reproducible evaluation for AI memory systems used by teams and agents. It compares RAG, agent memory, and organic memory by behavior: activation, forgetting, consolidation, adaptation, interference resistance, and efficiency.

### What is Cognoscenti?

Cognoscenti is Achiral's benchmark framework for cognitive memory systems. It is inspired by ACT-R memory research and tests whether a memory system brings the right experience forward at the right time.

Repository: https://github.com/Achiral-AI/cognoscenti

### Does this benchmark replace LongMemEval or LoCoMo?

No. LongMemEval and LoCoMo are useful for long-term recall and conversational memory regression. Cognoscenti tests a different standard: cognitive memory behavior across activation, useful forgetting, reinforcement, adaptation, and interference resistance.

### Can Achiral claim state of the art today?

On the 2026-08-02 Cognoscenti organizational-memory run, the organic-memory reference baseline achieved the highest Top-1 accuracy and lowest distractor activation rate among the tested reference architectures. Vendor-specific SOTA claims should wait until vendor endpoints are run through the same public protocol.