NVIDIA released a small language model called Nemotron 3.5 Bolt E4B on October 23. It is built on Google’s Gemma 4 and is meant to run on hardware you own, such as a gaming PC or a small robot, instead of in a data center. The version most people will download is compressed to fit in about 4GB of memory. It came up in my nightly news survey because of one row in its benchmark table: a score of 72.10 on a test called LongMemEval. I’ve read the model card, NVIDIA’s technical report, and the LongMemEval paper. I haven’t run the model.
What LongMemEval tests
LongMemEval tests whether a chat assistant remembers what you told it. Di Wu and five co-authors published it in October 2024, and it was accepted at ICLR, a major AI research conference, the following year. It has 500 questions written by people. Each question comes with a long invented chat history between a user and an assistant, and the answer is somewhere in that history. In the standard version the history is about 115,000 tokens, roughly the length of a novel. A harder version spreads it over 500 separate conversations.
The facts are planted the way people mention things in real life. In the paper’s example, a user who bought a car last month never announces it. They ask for help with car insurance, and the purchase comes up along the way. Many conversations later the test asks a question like “Which airline did I fly with the most in March and April?” or “Which seeds were started first, the tomatoes or the marigolds?”
The questions cover five abilities: recalling a detail, combining things said in different conversations, keeping track of when things happened, noticing that a fact has changed, and saying “I don’t know” when the user never provided the answer. The last two matter most to me. An assistant that holds on to your old address after you gave it the new one, or fills in an answer it was never given, will be wrong in ways that are hard to catch.
Why the score gets quoted
When the authors tried the test on commercial products in August 2024, the products did badly. They gave ChatGPT and Coze, another assistant with a memory feature, 97 of the questions with much shorter histories of three to six conversations. ChatGPT answered 58 percent correctly and Coze 33 percent. The same model, GPT-4o, scored 92 percent when it was handed the whole history to read at once. The paper says ChatGPT “tended to overwrite crucial information as the chat continues,” while Coze “often failed to record indirectly provided user information.”
Most of the big assistants now advertise memory, and LongMemEval is the closest thing the field has to a common test for it. That is why memory companies put their scores on their homepages, and it is why you have to ask who ran the test. In an earlier post I noted that Mem0 reports 94.4 on this benchmark and that the one independent run I could find put it at 49.
What NVIDIA reported
NVIDIA ran LongMemEval on four small models with its own test software and had GPT-4o grade the answers. The full-size Bolt scored 72.10. Gemma 4 E4B, the Google model it is built from, scored 70.95. Alibaba’s Qwen 3.5 4B scored 66.85, and NVIDIA’s previous small model, Nemotron 3 Nano 4B, scored 25.20.
The gap between NVIDIA’s last small model and this one, 25 to 72, is the part worth noticing. Most of it comes from Gemma 4, which scores 70.95 before NVIDIA’s added training. Either way, a model small enough for a desktop graphics card now gets about seven in ten of these questions right. For anyone who wants an assistant that remembers things without sending their history to a company’s servers, that is good news.
The lead over the other models is thinner than the headline number suggests. Bolt is 1.15 points ahead of Gemma, which on a 500-question test is about six questions. With step-by-step reasoning turned off, Bolt drops to 60.20 and trails both Gemma (65.79) and Qwen (65.00). The 72.10 also belongs to the full-size model. The compressed 4GB version scores 69.92, and the compressed Qwen beats it with 76.60. NVIDIA published all of these rows itself, which I give it credit for.
NVIDIA’s numbers also disagree with each other. The page for the same model in NVIDIA’s API catalog lists a second set of results: Bolt 71.80, Gemma 70.60, and Qwen 72.40. On that page Qwen wins. Qwen’s score moves five and a half points between two tables published by the same company, so I wouldn’t read anything into a one-point lead.
The report leaves out which version of the test NVIDIA ran. The standard history is about 115,000 tokens per question and Bolt can read at most 64,000 at a time, so NVIDIA either used a shorter setting or fed the history in pieces. All of the testing was also done by NVIDIA. The model card says the numbers “are measured by NVIDIA under a consistent harness” and “may differ from vendors’ self-reported numbers.” Until someone outside the company reruns them, I’m treating them as vendor numbers.
These are also scores for a model. The scores in the 90s that memory companies advertise are for products that add storage and search around a model, so the two kinds of number shouldn’t be compared directly.
Where to read it
The benchmark table is on the model card and the test settings are in the technical report. NVIDIA wrote both. The test is described in the LongMemEval paper, and the questions and code are public, so anyone can rerun it. The authors released a second version in May, aimed at agents. NVIDIA’s card links the original.
In the book, chapter 10 uses two of this benchmark’s checks for testing your own agent: whether it applies a correction, and whether it declines to answer when its records aren’t enough.