Start from the questions, not the vendors
Search decisions often begin with a comparison of features: vectors or keywords, hosted or self-managed, which model, which index. Those choices matter, but they are answers to a question you have not written down yet: what should search actually find for your users?
Writing that down first changes the conversation. Each option can then be judged by the same test, and the test reflects your product rather than someone else’s demo.
What a useful set looks like
Collect twenty to fifty questions people actually ask, in the words they actually use. Include the easy ones and the awkward ones: vague phrasing, exact identifiers, questions that mix two topics, and a few the collection cannot answer.
For each question, note the passage a careful reviewer would want to see. That reference does not need to be perfect. It needs to be specific enough that you can tell whether a result helped.
Keep the set small enough to read by hand. Reading results is how you notice the difference between a related document and a useful answer.
Run it, then run it again
Run the same set through each option you are considering, at the settings you would really use. Record what came back, what it would cost at your expected volume, and how much work it took to get running.
Keep the set after you decide. Rerun it when your content changes or when you adjust search effort. Public benchmarks help orient a decision; your own questions are how you keep it right.