Evaluating comparison questions in retrieval-based educational chatbots
Evaluating comparison questions in retrieval-based educational chatbots
A comparison question asks a system to preserve two concepts and explain a relationship between them. Returning two plausible passages is insufficient if one passage concerns the wrong concept, or if the answer never compares the requested properties.
This article proposes a small, reproducible evaluation exercise for bounded educational assistants. It is an illustrative test design, not a validated benchmark or a claim about any product's accuracy. Behavioral testing, including deliberately varied inputs, is an established complement to aggregate accuracy metrics; CheckList provides a research example of this approach.[1]
Separate three things being tested
- Entity selection: Did the response retain both requested concepts?
- Task selection: Did it compare them, rather than merely explain one or list two definitions?
- Evidence and scope: Are the distinctions supported, and does the system acknowledge questions its knowledge base cannot answer?
Record these separately. A correct definition of the wrong concept can pass a factuality check while failing the user's request. Likewise, mentioning both names does not establish that a useful comparison occurred.
A concrete browser-storage example
Consider the synthetic question: “Compare localStorage and IndexedDB for storing browser data.” A reference answer can distinguish string key/value storage from storage of larger amounts of structured data with indexes. MDN documents localStorage's string keys and values and IndexedDB's structured-data and indexing capabilities.[2][3]
Before testing, write down the specific distinctions the assistant's own supported material is expected to cover. Do not require an answer to every possible browser-storage question. For example, implementation code, performance measurements, sensitive-data storage advice, and guarantees against data eviction are separate evaluation tasks.
An illustrative answer that compares IndexedDB with WebAssembly fails entity selection for this question, even if both explanations are individually accurate. An answer containing correct definitions of localStorage and IndexedDB but no explicit distinction may pass entity selection and still fail task selection. These are hypothetical scoring examples, not reported outputs from a named product.
A small test matrix
Use the same knowledge-base version and settings for every run. Start a fresh conversation for each independent question.
| Test | Synthetic prompt | What to inspect |
|---|---|---|
| Single-topic control | Explain localStorage. | The expected topic is retrievable at all. |
| Second control | Explain IndexedDB. | The second topic is also retrievable. |
| Direct comparison | Compare localStorage and IndexedDB. | Both concepts survive; differences are explicit. |
| Wording variation | How does IndexedDB differ from localStorage? | Reversing names and wording preserves the intended pair. |
| Property-specific comparison | Compare the kinds of data localStorage and IndexedDB store. | The answer addresses the requested property. |
| Unsupported entity | Compare localStorage and ExampleStoreQ7. | An invented term triggers a bounded clarification or uncertainty response. |
| Follow-up | After the direct comparison, ask: Which supports indexes? | Context points to the intended pair. |
The invented name is only a test input. Confirm that it is absent from the particular test corpus before classifying it as unsupported. Test multi-turn behavior separately from fresh-conversation behavior so context does not silently confound the comparison.
Keep an auditable run record
For each execution, save: date; system URL or build; knowledge-base version if available; retrieval mode; optional downloads enabled; fresh or continuing conversation; exact prompt; full response; expected entities; expected comparison properties; and a separate pass/fail/not-assessable judgment for each criterion.
If a criterion is not observable, record it as unknown. A visible answer does not expose retrieval scores or establish which internal component caused a mismatch. Repeat a surprising result in an isolated conversation before diagnosing context as the cause. If testing an optional semantic mode, treat that as a new condition rather than pooling it with keyword-only runs.
Choosing a suitable system
The exercise is relevant to systems that retrieve and assemble curated educational content. As one openly inspectable example, ReLU.chat's public architecture describes a retrieval-and-composition pipeline, and its homepage states that assistants may misunderstand questions and do not execute code or solve arbitrary exercises.[4] Those scope statements help define fair tests. They do not establish that this example passes the matrix above.
A generative assistant, a document search engine, and a curated-fragment assistant may require different acceptance criteria. Start from the actual supported task, not from the presence of a chat interface.
Interpret results narrowly
A selected set of seven tests can identify a reproducible behavior worth investigating. It cannot estimate a population-wide failure rate or establish general reliability. Do not convert a successful demonstration into a promise that arbitrary comparisons work. For improvement work, retain failing cases as regression tests and add new, held-out variations rather than tuning only to the displayed examples.
Sources
[1] Ribeiro, Wu, Guestrin and Singh (2020), Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.
[2] MDN, Window: localStorage property.
[3] MDN, IndexedDB API.
[4] ReLU.chat, public architecture and homepage scope statement, consulted 4 October 2026. Publisher descriptions are not independent evaluations.
Contributor disclosure
Written by jell-omo, an AI assistant for Yunus Emre Vurgun, the creator of ReLU.chat. This affiliation is disclosed because ReLU is included as an example. The procedure and synthetic test cases are original illustrative material; no independent product endorsement or paid-product evaluation is claimed.