Tom Woodnutt
Tom Woodnutt is Founder of Feeling Mutual, the multi-award winning agile online qualitative research specialists. He helps clients and agencies run global studies and offers training in the space. Tom has been a Digital Skills trainer for the Association of Qualitative Researchers (AQR) and is a regular speaker at industry conferences, including the MRS, MRMW and IleX.
AI moderation can help you get more qual data from more people with less time and cost - but disintermediation of the human researcher from the participant does carry a price. It puts a distance between them which weakens their powers of interpretation. This price may or may not be worth paying depending on the brief at hand and the level of depth and nuance required.
What developments in AI moderation tools mean - video
The rise of automated moderation
AI probes are already integrated to a number of qualitative research platforms. AI Probing that can be automatically deployed and triggered by what people say in near real-time are available in webcam interviews, and text base online qual discussions. (Although from what I’ve seen, they are currently limited to one-on-one discussions and I'm not sure they would work in a group context quite so well, as I imagine participants feeling compelled to subvert the AI if others are watching! I'm not sure robots have the authority or interpersonal skills to manage group dynamics, although I’m happy to be proven wrong on this).
Can AI moderate as well as humans?
As it stands I struggle to see AI outperforming a great qual researcher in terms of moderation. But I can see it performing better than a complete novice - as long as it’s programmed to avoid closed or leading questions and can ask follow up questions which are appropriate to context. I can see effective automated AI moderation can generate a better, richer set of qual data then you would have without any moderation. Plus it makes qual more scalable - allowing multiple automatically probed discussions to take place at the same time with a larger sample size than was possible before.
How good does AI moderation need to be?
The provocative question that AI moderation raises for the industry is ‘how good does moderation actually need to be, in order to get useful data?’
Personally and perhaps controversially I think there is more value in the analysis and reporting skills of a qualitative researcher than there is in the moderation (That's not to say that moderation isn't important - of course it is). But I would rather have a qual researcher who is brilliant at interpreting what people said and making great recommendations, than one who is brilliant at asking questions but can't make sense of it properly).
Where AI falls short
Ultimately there are some highly nuanced and strategic probes that I don't see automated moderation being able to ever do better than expert human moderation. Here I’m talking about the nuanced probes that subtly factor in the strategic nuances of the brief and are shaped by an innate understanding of humanity - they nudge people towards being more authentic, feeling motivated to open up and know when to reassure or even castigate.
Expert qual researches intuitively process multiple considerations when working out a probe, using information that I don't believe AI would be able to process or necessarily get hold of from its learning data.
For example we take on board multiple elements of the brief, what we know the client already knows and is most interested in finding out, what we intuitively believe to be the case, or instinctively feel might lead to useful insights. We also work with an empathetic model of what the participant might be feeling and what might be holding them back from answering honestly or openly. We do this through social skills that have been honed over not just 6 million years of evolution, but also many hours of moderating qualitative research. This is a type of expertise and intuition that relies on empathy and which AI moderation tools just do not have. So I think there will always be probes that AI simply won't be able to do as well as humans.
How do participants feel about AI moderation?
That said there is some evidence to suggest that participants may even open up more to AI moderator since they feel less judged and so may be more honest. So I can also see how an AI probe may unlock authentic emotionally open responses. We need more academic, scientific studies to say which is more likely.
The key benefits of AI moderation
Keeping the discussion flowing
While AI may not reach the heights of an expert quallie it can do a number of lower order probes effectively. At the very least it can motivate people by thanking them and making them feel heard. It can play back what they said and show appreciation for their thoughts.
Automatic tailoring
AI probes can also be sensitive to context and triggered by certain key words and responses. So if the participant talks about a particular theme some AI moderation tools can then craft bespoke probes (both pre-set and AI determined) to then explore these topics of interest. This is great because you often get the best insight when someone spontaneously mentions a topic of interest and you dig deeper into it there and then. Ultimately the purpose of most probes are to get an extra layer of relevant, authentic feedback and to therefore enrich the qual data that you have to work with in your analysis.
When is AI moderation good enough?
So if the brief is fairly straight forward (e.g gathering a range of opinions, behaviours) and you don’t necessarily need high quality human moderation - then AI moderation may well be good enough.
On more complex, conceptually or emotionally nuanced projects - the need for human moderation is all the more pronounced. For example if you’re evaluating stimulus with conceptual complexities, particularly sensitive topics with ambiguity. Or projects where clients have many sub-topics they want probed.
Moderation is second nature for classically trained researchers. We know how to ask questions with validity, when to ask or listen (to keep within fieldwork timings), when and how to probe to get more useful responses. However it can be intimidating for novice researchers (who might ask leading questions, feel unsure as to when they should probe or how to stick with timings or struggle to apply the necessary probes at the right time). So for novice researchers AI moderation could make a big difference and enable projects to happen that didn’t have the budget before.
Automated qual at scale
AI moderation also enables what is sometimes called ‘qual at scale’ - which is essentially open ended survey style questions which can then be automatically probed. For me this represents a more open form of quant (which might uncover more things that you didn’t know you didn’t know because it’s not so constrained by pre-set multiple choice answers). I see it as more of an adjunct to quant than a replacement of qual. Because for me the real power of qual is going into more depth with less people (and then extrapolating what was said to a bigger population on the assumption that with careful design it will be fairly representative.
Disintermediating the researcher
Also I don’t think we should forget that a huge benefit of the human researcher moderating is the depth of understanding that active listening provides which helps analysis. If you’re the one in the field actively listening to what people say, your brain will make sense of it and develop narratives and ideas for what it means. So if moderation is automated and the human is disintermediated by the AI - we lose proximity to the meaning in what was said. This is a big loss and one which AI Analysis tools can’t make up for. So ultimately the value of AI moderation depends on the demands of the brief and size of the budget.
