Doctors tend to ignore qualitative research because it’s too hard to judge whether it’s any good or not. But it’s possible to identify low-quality qualitative research
After years of hunting through medical research papers for potential stories, I’ve become pretty adept at spotting the rotten eggs.
As I quickly scan each paper, I judge it against a set of strict criteria: Does this paper make sense? Can I follow the logic behind the method? Is the conclusion supported by the results? Are there any conflicts of interest? Is the sample size large enough? Is the study placebo-controlled, blinded and randomised? Is it prospective? Are the results statistically significant?
Even in the top-ranking medical journals, some quantitative studies will not pass the ‘Felicity test’ and will go straight to the reject pile.
But, like some doctors, I am completely lost (and somewhat suspicious) when confronted with a piece of qualitative research.
The findings might sound interesting, nuanced and important – but the methodology is alien and unfamiliar. How do I know if the study is rigorous? How have the researchers controlled for their own personal biases? Why have they only interviewed ten people? Is that a large enough sample size to say anything about anything?
Qualitative research is often misunderstood and overlooked, particularly in medical circles where systematic analyses and RCTs reign supreme.
But qualitative research doesn’t sit at the bottom of the hierarchy of evidence – and it’s not just a string of anecdotes (as it’s sometimes made out to be).
Qualitative research is its own discipline, with entirely different research questions, strengths, objectives, methods, and quality control strategies.
It captures what numbers and statistics cannot: the human experience.
The good, the bad and the ugly
I interviewed a range of qualitative researchers for this story, and the major take-away from our conversations was this: If a piece of qualitative research is presented like a piece of quantitative research, that should set off alarm bells right away.
The aim of qualitative research is to provide a deep understanding of another person’s perspective. It’s impossible to explore the complex lived experiences of hundreds of participants at one time, so any piece of qualitative research that attempts to do this is highly suspect.
The number of participants in a good qualitative study will vary from a handful to maybe 50-60, depending on the type of qualitative research being done.
Qualitative research based in grounded theory (where almost nothing is known about the topic in advance) would include more study participants than ethnography (the study of culture), action research (targeting changes in communities) or phenomenology (research focusing on the lived experience of individuals), says Dr Darshini Ayton (PhD), a health services researcher at Monash University who runs courses on qualitative research.
But, generally, qualitative researchers will keep adding participants until they reach “saturation”, which means no new themes are being raised, says Dr Gemma Sharp, a clinical psychologist and body image and cosmetic surgery researcher at Monash University.
“Usually, if you go for huge numbers in qualitative research, you risk getting a superficial understanding of what’s happened,” says Dr Maggie Kirkman, a psychologist and senior research fellow at Monash University.
“So, if somebody has had 10 or 20 participants, you would expect a lot of depth and detail, which will make far more sense and will be far more valuable than having 50 or 100 people. Because that amount of data just isn’t analysable. It’s far too complicated.”
Because the sample sizes are so small, it makes no sense to pretend that qualitative research can ever be representative of a broader population.
“Look, it’s not generalisable,” says Associate Professor Louise Stone, a GP and qualitative researcher at ANU. “I can’t say, ‘My six women are typical of everyone in the world’. That’s just silly. What I say is, ‘This is what I’ve learned from a deep understanding of these participants. And now everyone else has to think about whether that applies to them’.”
While the results of RCTs are expected to be reproducible, the expectations are a little different in qualitative research.
“Qualitative research takes as given that people’s lives are different and complicated and perhaps not replicable and sets out to find the most rigorous way possible to learn something about those lives,” says Dr Kirkman.
“Consistency, or dependability of the results, does not mean that the same result would necessarily be found in other contexts but that, given the same data, other researchers would find similar patterns.”
Another thing to watch out for is qualitative research that reports numbers or statistics in its results section. The whole point of qualitative research is that it collects and analyses qualitative data, not numerical data. Any qualitative researcher who tries to report prevalence statistics, p values or t-tests based on focus groups or in-depth interviews has selected the wrong methodological approach.
“Another form of bad ‘qualitative’ research is when researchers, perhaps in order to persuade high impact factor medical journals to take their manuscripts, report everything in percentages and end up with poor research that is true neither to qualitative nor quantitative methods,” says Dr Kirkman. “It fails the standards of both.”
However, it is possible to use quantitative and qualitative research simultaneously or sequentially. This is called a “mixed method”, and it’s very useful for planning survey questions or identifying knowledge gaps.
By way of example, Dr Sharp shared the results of her recently-published mixed method study where she started with an online survey of men who had undergone penile augmentation, and then interviewed a subset.
“The interviews added the flesh to the bones of the online survey and helped us understand what specifically improved in men’s lives after penile augmentation,” says Dr Sharp.
After the procedure, men no longer felt self-conscious wearing swimwear and weren’t so embarrassed getting changed in the men’s room, the study found. Interestingly, their relationship satisfaction did not improve as some men felt like the increased size of their penis raised their partners’ expectations and put pressure on them to perform.
“You wouldn’t be able to capture this kind of data in a survey necessarily,” says Dr Sharp.
Sometimes mixed research goes the other way; researchers start with qualitative research to find out what survey questions they need to ask, and then they roll out a large-scale questionnaire to capture the data.
Controlling for bias
RCTs are designed to remove bias and force the researchers to be as objective as possible. No matter how much a researcher wants a drug trial to succeed, if they follow all the steps for running a high-quality clinical trial with integrity, it’s almost impossible for them to personally influence the results.
Of course, there are sneaky ways to skew the results, such as not reporting primary outcomes prior to launching the clinical trial and fishing through data for a favourable p value at the end. But this statistical mischief is easy enough to spot if you look for it.
The same challenges are present in qualitative research, but this discipline uses completely different tools for controlling bias.
Whereas quantitative researchers will simply list their commercial COIs as a footnote on the paper, excellent qualitative research is hyper-aware and explicit about the personal biases of each of the researchers involved in the study.
Researchers will generally be up front about their personal beliefs and the theoretical lens through which they have chosen to conduct the study.
For example, Dr Kirkman describes her research team as “women and feminists” in her recently published paper on labiaplasty.
In analysing how cosmetic surgery websites pathologise normal female anatomy, the authors purposefully positioned themselves inside the framework of postfeminist discourse and neoliberal consumerism.
Ads were promoting labiaplasty as a form of “empowerment”, appealing to the postfeminist idea of woman as independent and confident, the authors reported.
“The websites’ representation of ‘choice’ was consistent with individualistic neoliberal consumerism, in which our improved bodies are markers of success and identity,” the authors said.
Qualitative researchers who are worth their salt won’t stop at a simple one-off disclosure of personal biases. They will start the project by mapping out their assumptions and continuously reflect on their own preconceptions at regular intervals throughout the study.
“The danger in qualitative research is always going to be bias – that the researcher has already decided what they are going to say at the beginning and then they cherry pick quotes to back up what they thought they were going to say in the first place – and that happens a lot,” says Professor Ilan Katz, who focuses on social policy research into homelessness, Indigenous Australians, veteran affairs and disability services at UNSW.
Transparency about where the researcher is coming from allows the reader to judge whether any bias has crept in, he says.
There are also techniques in qualitative research to overcome bias, such as intercoder reliability, where different researchers independently analyse the raw data and check to see if their interpretations are the same, he says.
Qualitative researchers usually work in teams to write interview questions and analyse the data. Having a second or third pair of eyes is essential for keeping biases in check, and to avoid asking participants leading questions.
In Associate Professor Stone’s long-running study looking at the experiences of female doctors who were sexually abused by their colleagues, she continuously consulted with a multi-disciplinary reference group to assess how her role as a GP could influence the study.
“Because I am part of the community that has caused the abuse, that’s tricky,” she says. “But, on the other hand, I know that I understand the environment and I have the interview skills to cope if someone becomes distressed.”
In her study, Associate Professor Stone interviewed six women in-depth several times, with an interval of 2-3 years between each interview.
The aim was to follow women through their journey over time and learn about the difficulties in reporting, the court process, how the assault affected their career, and how they eventually recovered from the trauma and returned to work.
Associate Professor Stone also sent the quotes and analysis back to each participant to get their input. “I wanted them to correct me if I was wrong,” she says.
It was a valuable exercise. In the study, a junior doctor described being sexually assaulted by a registrar standing behind her while she was suturing a patient. “My interpretation of what was happening in her head was that she had dissociated,” says Associate Professor Stone.
“But she said, ‘No. It was quite a deliberate decision.’ She hadn’t dissociated at all. She just made the choice on behalf of the patient to continue with the procedure because that was in the patient’s best interest.”
Associate Professor Stone and two other GPs with very different research backgrounds coded some of the data separately, and then compared analyses.
Expert qualitative researchers will take self-reflective notes throughout the entire interview process, says Dr Sharp. For example, if an interviewer was rushing through the questions, or simplifying language for participants for whom English was a second language, that would be noted down, she said. “When I write my notes, it’s not just about the participant,” she said. “It’s about how I asked the questions.”
Peer review
Doctors should be sceptical of any qualitative research that has not undergone peer review.
Good qualitative research will “stand up to close scrutiny”, says Dr Kirkman. It will include a sufficient pattern of quotations to ensure that the conclusions are justified.
To cover everything, qualitative research papers are necessarily lengthy; it’s not unusual to find papers that are 8,000 words or 24-pages long.
“Some journals try to restrict you to 3,000 words, which undermines our capacity to judge the quality of the research,” says Dr Kirkman.
Qualitative researchers working in health often prefer to get their research published in the top medical journals. But it’s clear that some reviewers haven’t got a clue about how to assess the quality of the work, says Dr Kirkman.
For instance, a colleague of Dr Kirkman’s received a rejection letter from a reviewer who said, ‘Even for a qualitative study, I would expect some form of confidence interval and paired t-tables analysis.’
“This comment reveals the reviewer’s inappropriate application to qualitative research of criteria relevant only to quantitative research,” she says.
What Professor Amanda Kenny, the co-editor in chief Australian Journal of Primary Health, looks for in an excellent piece of qualitative research is: a knowledge gap, a well-described methodology, a theoretical underpinning for the study, and a research question that suits a qualitative design.
“Qualitative research is incredibly diverse,” she says. “Traditionally, people tend to think about interviews or focus groups but there is a wealth of different methods that are emerging. The methods used in research should not make a difference to quality.”
Often details of the data analysis are missing from the first draft of qualitative research papers submitted for publication, says Professor Meredith Temple-Smith, a qualitative researcher at The University of Melbourne who works on sensitive health issues in marginalised populations.
“I should be able to understand enough of the research method from what is in the paper to conduct a similar study,” she says. “I especially want to understand the pool of people from which recruitment was sought, and how the participants came to be involved.”
As Dr Kirkman says, “It’s often not until you understand why people do certain things – why people are not taking their medication or doing their exercises – that you can find ways of encouraging them to do what your evidence tells you is going to be the most beneficial for them.”
