Not all synthetic data is created equal: What actually matters when evaluating a synthetic research partner

Synthetic data is one of the hottest topics in research and the pressure to have a point of view is enormous. Vendors are rushing digital twin panels to market, other teams are experimenting and “we’ll figure it out next year” is no longer an option. Sarah Kling (Amazon), a self-described synthetic data skeptic, and Aneesh Dhawan (Knit) will walk through the framework Kling's research team at Amazon is developing to evaluate synthetic research partners: how a feature was built, whether it was tested before launch and whether it actually works. You’ll see why her Amazon team is evaluating aggregated synthetic personas first, the ethics questions most of the market hasn’t stopped to ask. and what responsible experimentation looks like when the game hasn’t even started yet.

Key takeaways:

- Skepticism is a strategy, not a stall. The pressure to adopt synthetic data is real, but rushing to a digital twin panel is riskier than a stepped, pilot-driven approach. You can’t put the evaluation off any longer and you don’t have to leap either.
- Evaluate the partner before the promise. Look at the bones: the development process, beta testing and co-design, how outcomes were measured and whether the vendor behaves like a responsible SaaS partner. Then layer in the research question: Does the thing actually work for your use case?
- Know where your synthetic data comes from. Not all synthetic users are the same. Aggregated personas built from traceable, protected inputs are a different proposition from digital twins of real people, which raise unanswered questions around privacy, consent and the right to be deleted. Where you start on that continuum determines both the value and the risk.