Benchmark

LongMemEval-S

A matched comparison of how the OAK.memory SDK and Mem0 OSS store conversations and answer questions about them.

Methodology
Answer accuracy60 matched questions
OAK.memory43/60
71.7%
Mem0 OSS44/60
73.3%

Both systems used the same improved answer instructions.

Cost and context

Confirmed API cost

OAK -9% vs. Mem0

Total for storing memories, search embeddings, and one answer per question. Excludes evaluation, hosting, and service fees. The totals are calculated from recorded model usage.

OAK.memory$4.1845
Mem0 OSS$4.5972

Answer input tokens

OAK +18.3% vs. Mem0

Median input per answer, including retrieved memories, the question, and instructions.

OAK.memory6,229
Mem0 OSS5,264

Accuracy by question type

Correct answers out of all questions in each category. Category sizes vary.

Question typeOAK.memoryMem0 OSS
Across sessions12/199/19
Temporal reasoning10/1715/17
User statements9/118/11
Knowledge updates7/76/7
Assistant statements3/44/4
Preferences2/22/2

Methodology

This is a 60-question development sample drawn from LongMemEval-S. It measures the local SDKs with shared retrieval and grading conditions. The sample includes questions seen in previous experiments. Production latency and hosted products were outside this comparison.

Store

Each question started with an empty store in each system. Both replayed the same conversation history in pairs of messages, with up to ten preceding messages for context. OAK used actual SDK writes and links. Mem0 ran OSS 2.2.1.

Retrieve

Both used text-embedding-3-small and cosine retrieval from up to 200 candidates. The answer received at most 50 memories, ordered by date, within an 8,192-token prompt budget.

Answer and grade

GPT-6 Luna Flex handled memory writes, answers, and grading on both sides. The judge used the dataset reference answers and the same Mem0 public grading prompt and parser. Both used the same default question-date field for grading.

Read the shared answer instruction
Match each memory to the requested event, person, item, or time by its meaning. Different wording can describe the same fact. Combine records across dates or conversations when they support the connection, and distinguish separate events from duplicate reports. For lists, counts, or totals, collect every supported item before checking the result. If some information is uncertain, identify what is missing while keeping the facts and deductions that are supported.
Coverage and unresolved spend

The report uses every available verdict. The table also shows scores after excluding questions where evidence was missing from either store.

Extension subset
OAK.memory 23/30 · Mem0 OSS 23/30
Excluding the question with missing evidence
OAK.memory 42/59 · Mem0 OSS 43/59
Unconfirmed API reservations
OAK.memory $0.0657 · Mem0 OSS $0.2771