Synthetic focus groups: their potential, their limits, and what comes next

The focus group industry is outdated and has priced out many companies that would benefit greatly from the decision-making information and market insights that focus groups provide. When a company better understands their specific market and the factors that influence decision making, they can move faster, more efficiently, and make better internal decisions.

Synthetic focus groups are an alternative to traditional market research. The advantages of market research and focus groups have been written about for decades; the primary goal of this analysis is to explain how synthetic focus groups work, their limitations, and how, if they are shown to produce accurate research, they allow businesses of all sizes to experience those same advantages for a fraction of the cost.

Synthetic focus groups - how they work

Synthetic focus groups are focus group applications powered by large language models (LLMs), designed to allow businesses to run their own custom focus groups. This concept is centered around what we refer to as “personas”.

These personas are synthetic profiles that have extremely detailed sets of demographic and psychographic details, personal stories, and personal information. Personas are either asked to provide feedback individually, in what is known as a “test”, or multiple personas are grouped to provide group feedback in a focus group environment. A persona may be asked to provide feedback on new product messaging as an example.

When a persona is asked to evaluate an idea or participate in a focus group, it provides feedback using the LLM of choice as the primary engine for analysis. However, the analysis is influenced by the detailed context document that includes the preferences, history, and opinions of that specific persona. This creates a unique feedback response.

If these focus groups can be made to produce accurate feedback, they provide several advantages over traditional focus groups. Primarily, the economic advantage of running these focus groups synthetically allows for increased flexibility in market research. You can run the same test multiple times, with multiple personas, with multiple iterations of the messaging, all for fractions of what a traditional market research campaign would cost.

Additionally, it allows the ability to run these groups at scale with hundreds of versions of the same ideal customer profile (ICP). Let’s say your primary market segment is mothers aged 28-31, with a very specific and detailed customer profile. Even within that ICP, there are thousands of variations that will affect decision making and opinions. Small details (such as childhood success, tragic incidents, family relationships, how well they sleep) are included in these personas, allowing for the ability to create large-scale focus groups that show a wide spectrum of potential opinions.

There are several additional advantages of synthetic focus groups and their use cases, but most center around these two concepts of economics and scale.

How do we measure the accuracy of personas and feedback?

The viability of these synthetic focus groups hinges on the accuracy of the feedback and output. How is this accuracy measured? The measure for accuracy should be defined as how close the results map to the actual preferences and objections of the ICP. To validate the accuracy, small independent real focus groups would need to be run in parallel, and the results compared.

If the focus groups cannot be made to produce accurate preferences and objections to a test, the value case no longer holds up, and the underlying system would need to be reworked until accuracy meets a set threshold.

An additional way to measure accuracy would be predictive usefulness. How accurate are the focus groups in producing feedback that actually predicts the market conditions? This would need to be measured through a correlation of decisions and outcomes based on the feedback provided by these focus groups. For example, the synthetic focus group raises a messaging concern that was not previously raised, and that concern is then validated by the audience post-launch.

If these synthetic focus groups can show the ability to raise objections and preferences that have real impacts and can raise insights not discovered in traditional market research, the value proposition increases substantially.

The limitations and drawbacks

As with any new technology, specifically those that rely on a backend engine that centers around an LLM, there are limitations and drawbacks. Some of these drawbacks may be fixed as the industry moves into the agentic infrastructure described later in this analysis.

First, the feedback that the personas deliver is still subject to the bias of the model. Even with extremely detailed personas, the underlying bias and communication of the model will still leak through.

Second, the ability for these personas to communicate in a more traditional focus group setting and provide the feedback collectively is lacking in the current infrastructure. This can currently be bridged by aggregating feedback for a group of personas, then summarizing the group consensus. The agentic infrastructure later described has potential to provide a better solution to this.

Additionally, the accuracy of the context assigned to a persona will greatly impact how closely the feedback produced would mirror an actual consumer in the market. Due to this, inaccurate personas can be a major drawback and produce inaccurate feedback. There are several ideas of how to source this data and ensure personas accurately reflect the current state of the ICPs that they are built to mirror.

The pursuit of better source data for more accurate personas

One of the more interesting debates in the synthetic focus group discussion is how to source the data to build the detailed context behind each persona. Traditionally, these personas were created using a single ICP that then was used to create multiple personas with generated context. The accuracy of the feedback relies on how well these personas match the profile of the actual ICP.

The next evolution in data sourcing will be companies that pay for ongoing limited surveys of real customers in their market - and use that data as the source for their personas. This will create a feedback loop where the personas are consistently self-updating based on real-time feedback from the market.

The next step - the agentic infrastructure for synthetic focus groups

The recent rise of autonomous agents and the infrastructure to operate them at scale raises the opportunity for an exciting and substantial next step for synthetic focus groups - an agentic infrastructure.

This agentic infrastructure would encompass a system where agents can be deployed, grouped into focus groups, and provide feedback on tests. Each agent would have a persona - a detailed set of contextual and personal information that would impact their evaluation and decision making.

This infrastructure allows a path to build a system where these agents can access news, data, and current events in order to continuously update their persona. My hypothesis is that this could lead to the most accurate mirror of the ICP. It also allows the potential for memory in these agents and groups, so they are aware of previous tests and can apply context from them. The goal is that these improvements provide more accurate feedback and more dynamic research.

This infrastructure opens the door for these agents to communicate with each other in a more traditional focus group setting. They can collaborate, challenge opinions, and provide back the transcript of the dialogue that took place. This solves one of the drawbacks of the current persona-based system.

Upcoming adoption

I anticipate in 2027 we will see this agentic synthetic focus group infrastructure adopted by a growing number of companies that have a yearly budget for market research. I believe these will be built internally for companies, so they can own their data and work off a token spend system financially rather than through a subscription-style service. Opportunity exists for companies that can build and manage this infrastructure for specific industries.