Benchmark
LongMemEval-S
A matched comparison of how the OAK.memory SDK and Mem0 OSS store conversations and answer questions about them.
MethodologyBoth systems used the same improved answer instructions.
Cost and context
Confirmed API cost
OAK -9% vs. Mem0Total for storing memories, search embeddings, and one answer per question. Excludes evaluation, hosting, and service fees. The totals are calculated from recorded model usage.
Answer input tokens
OAK +18.3% vs. Mem0Median input per answer, including retrieved memories, the question, and instructions.
Accuracy by question type
Correct answers out of all questions in each category. Category sizes vary.
| Question type | OAK.memory | Mem0 OSS |
|---|---|---|
| Across sessions | 12/19 | 9/19 |
| Temporal reasoning | 10/17 | 15/17 |
| User statements | 9/11 | 8/11 |
| Knowledge updates | 7/7 | 6/7 |
| Assistant statements | 3/4 | 4/4 |
| Preferences | 2/2 | 2/2 |
Methodology
This is a 60-question development sample drawn from LongMemEval-S. It measures the local SDKs with shared retrieval and grading conditions. The sample includes questions seen in previous experiments. Production latency and hosted products were outside this comparison.
Store
Each question started with an empty store in each system. Both replayed the same conversation history in pairs of messages, with up to ten preceding messages for context. OAK used actual SDK writes and links. Mem0 ran OSS 2.2.1.
Retrieve
Both used text-embedding-3-small and cosine retrieval from up to 200 candidates. The answer received at most 50 memories, ordered by date, within an 8,192-token prompt budget.
Answer and grade
GPT-6 Luna Flex handled memory writes, answers, and grading on both sides. The judge used the dataset reference answers and the same Mem0 public grading prompt and parser. Both used the same default question-date field for grading.
Read the shared answer instruction
Match each memory to the requested event, person, item, or time by its meaning. Different wording can describe the same fact. Combine records across dates or conversations when they support the connection, and distinguish separate events from duplicate reports. For lists, counts, or totals, collect every supported item before checking the result. If some information is uncertain, identify what is missing while keeping the facts and deductions that are supported.Coverage and unresolved spend
The report uses every available verdict. The table also shows scores after excluding questions where evidence was missing from either store.
- Extension subset
- OAK.memory 23/30 · Mem0 OSS 23/30
- Excluding the question with missing evidence
- OAK.memory 42/59 · Mem0 OSS 43/59
- Unconfirmed API reservations
- OAK.memory $0.0657 · Mem0 OSS $0.2771