Top-1 accuracy
91.67%
Organic-memory reference baseline
Cognoscenti Benchmark
Memory Benchmark: What Makes AI Memory Trustworthy?
AI memory does not fail only when it forgets. It also fails when it remembers too much, brings back stale context, or treats every stored fact as equally relevant. A memory layer is trustworthy only when it recalls the right context and keeps the wrong context out.
Cognoscenti turns that trust question into a public benchmark: when a team or agent needs context, does the right memory come forward at the right time? It compares RAG, agent memory, and Emergent Memory Systems on that behavior.
Direct answer
A serious AI memory benchmark should measure whether memory can be trusted in use. That means testing more than retrieval: useful context should activate, stale context should stay quiet, repeated use should strengthen the right memory, new facts should update old assumptions, and recall should stay selective as the memory set grows.
What the first run showed about trust
The 2026-08-02 Cognoscenti run used the same 12-item organizational workload against three local reference architectures. The organic-memory baseline led on Top-1 accuracy and distractor suppression while matching agent memory on Recall@3. In practical terms, it picked the right memory first more often and activated fewer stale or misleading memories.
Recall@3
100.00%
Matched agent memory on finding the right memory in the top three
Distractor activation
16.67%
Lowest rate of stale or misleading memories in the run
The trust problem we tested
The workload is small enough to inspect and strict enough to catch common memory failures. Each item includes a query, a project or domain context, one gold memory, distractor memories, and a rationale for why the right memory should win. That makes the run auditable: a reviewer can see what the system should have remembered, what it should have ignored, and why.
| System | Behavior | Measured for | Expected gap |
|---|---|---|---|
| Vanilla RAG | Finds stored content that looks relevant to a query. | Whether the right document or passage can be fetched. | RAG is useful when the answer lives in a document. It is weaker when the system must decide which past experience should matter now. |
| Agent memory | Carries state for an agent, workflow, user, or session. | Whether one agent can continue work across turns or tasks. | Agent memory helps an agent keep continuity. By itself, it usually does not become shared organizational memory across people, tools, and decisions. |
| Organic memory | Forms from repeated work, strengthens through reuse, decays when stale, and stays reviewable. | Whether the right experience becomes active at the right time. | Organic memory matters when the task depends on what the team has learned through use, not only what the system stored. |
Trust fails in familiar ways
Activation
Select the current Atlas authentication rollout plan while suppressing a superseded SSO-first plan.
Selective forgetting
Focus engineering on expired magic-link tokens while ignoring unrelated pricing chatter.
Interference resistance
Keep Atlas authentication separate from Apollo billing when both mention enterprise flags.
Adaptation
Use the latest positioning for Achiral and suppress older vector-database framing.
Consolidation
Prefer a reused launch checklist over a one-off thumbnail note or stale launch plan.
Claim boundaries
Recall that SOC 2 work is in progress without claiming certification before the audit is complete.
What trustworthy memory must measure
Activation
Does the current question, account, project, or workflow bring the right memory forward?
Top-1 accuracy, top-3 accuracy, irrelevant activation rate.
Selective forgetting
Can the system ignore stale, low-signal, or misleading context?
Junk activation rate, forgotten-memory ratio, retrieval precision.
Interference resistance
Can similar projects, customers, issues, and decisions stay separate when their language overlaps?
Confusion rate, distractor count, retrieval confidence.
Consolidation
Do repeated successful uses make the right memories easier to recall?
Repeated-access accuracy, activation increase, retrieval speed improvement.
Adaptation
Does newer information replace old assumptions while preserving the historical record?
Belief update accuracy, supersession rate, historical context preservation.
Efficiency
Does recall stay selective as the memory set grows?
Memories examined, activated chunks, token cost, p50 and p95 latency.
How this differs from recall benchmarks
LongMemEval, LoCoMo, Agent Memory Benchmark, and related work remain important. They test long-range recall, evidence recovery, conversational continuity, and provider retrieval quality. Cognoscenti tests a different standard: whether a memory system can be trusted under real use, instead of acting like storage with a search box.
| Benchmark | Standard | Good for | Boundary |
|---|---|---|---|
| LongMemEval | Long-term conversational recall | Good for testing whether a system can recover facts and evidence across long histories. | It does not fully test whether stale, repeated, conflicting, or operationally important memories should become active now. |
| LoCoMo | Long-context multi-session memory | Good for regression tests on dialogue history, persona continuity, and long-range recall. | It does not set a full standard for activation, decay, reinforcement, consolidation, and interference control. |
| Cognoscenti | Memory behavior under use | Good for ACT-R-inspired and emergent memory systems where recall is shaped by use, salience, recency, decay, and context. | It asks whether the right memory comes forward, and whether the wrong memory stays out of the way. |
Results from the first run
The 2026-08-02 run used the same 12-item organizational-memory workload against three reference architectures: vanilla RAG, stateful agent memory, and organic memory. Organic memory achieved the highest Top-1 accuracy and the lowest distractor activation rate while matching agent memory on Recall@3. The advantage came from lifecycle-aware scoring: current, validated, and reinforced memories were promoted, while stale, superseded, deprecated, and low-signal memories were suppressed. That is the behavior teams need before they can trust AI memory inside daily work.
| System | Top-1 | Recall@3 | Precision@3 | Distractor rate |
|---|---|---|---|---|
| RAG reference baseline | 50.00% | 83.33% | 27.78% | 25.00% |
| Agent-memory reference baseline | 75.00% | 100.00% | 33.33% | 38.89% |
| Organic-memory reference baseline | 91.67% | 100.00% | 33.33% | 16.67% |
These are reference baselines, not vendor benchmarks. Vendor results should be published only when provider endpoints, keys, commands, seeds, and raw outputs are recorded.
Methodology
Explains the reference architectures, workload, command, and interpretation.
Workload JSONL
The 12 benchmark items, gold memories, distractors, contexts, and rationales.
Results JSON
Full item-level results for each tested reference architecture.
Metrics CSV
System-level metrics including Top-1, Recall@3, precision, and distractor rate.
Report PDF
A portable report for reviewers who want the run summary in one file.
Citation and provenance
The benchmark is backed by public citation and preservation records, not only by this webpage. The Figshare preprint and benchmark artifact are shared under CC BY 4.0; the Cognoscenti source code remains Apache-2.0.
Zenodo software DOI
Archived Cognoscenti release metadata for software citation.
Figshare preprint DOI
Public preprint and reproducibility bundle under CC BY 4.0.
Software Heritage snapshot
Preserved source-code snapshot for long-term provenance.
Cognoscenti source
Apache-2.0 source code, benchmark runner, and release notes.
How to read the claim
This page establishes a public standard for evaluating trustworthy AI memory. On the 2026-08-02 Cognoscenti run, the organic-memory reference baseline led the tested architectures on Top-1 accuracy and distractor suppression. That supports a narrower and stronger claim: Cognoscenti defines a state-of-the-art standard for testing whether memory can be trusted to activate, update, and suppress context correctly, while vendor-specific state-of-the-art claims should wait until each vendor is run through the same public protocol.
FAQ
What is the Organizational Memory Benchmark?
The Organizational Memory Benchmark is a reproducible evaluation for AI memory systems used by teams and agents. It compares RAG, agent memory, and organic memory by behavior: activation, forgetting, consolidation, adaptation, interference resistance, and efficiency.
What is Cognoscenti?
Cognoscenti is Achiral's benchmark framework for cognitive memory systems. It is inspired by ACT-R memory research and tests whether a memory system brings the right experience forward at the right time.
Does this benchmark replace LongMemEval or LoCoMo?
No. LongMemEval and LoCoMo are useful for long-term recall and conversational memory regression. Cognoscenti tests a different standard: cognitive memory behavior across activation, useful forgetting, reinforcement, adaptation, and interference resistance.
Can Achiral claim state of the art today?
On the 2026-08-02 Cognoscenti organizational-memory run, the organic-memory reference baseline achieved the highest Top-1 accuracy and lowest distractor activation rate among the tested reference architectures. Vendor-specific SOTA claims should wait until vendor endpoints are run through the same public protocol.
Try Achiral memory your team can trust.
Chiro turns daily work into shared organic memory, so your team and agents can recall the right context and avoid stale signal.