Mixed Methods: A short guide to applied mixed methods research

by Sam Ladner

Quantitative researchers focus on scale on causation but fail to provide coherence and participant focus. Qualitative research, by contrast, focuses on coherence and participant focus but lacks scale and causation.

When we use quant data, we put the sensemaking onus on the consumer of the research, and when we use qual data, we sacrifice scale and causation. This is a terrible trade-off.

The word empirical simply refers to direct observation, so quant data can be subjective or empirical, and qual data can be either subjective or empirical.

At their core, these two approaches have differing belief systems about how knowledge is created (epistemology) and even more fundamentally, about what is reality itself (ontology).

this is not simply about mixing methods, but about opposing views on reality. If that sounds heavy, it is. That’s why we fail to mix methods—because we are using fundamentally different assumptions about what is even real!

In contrast to objectivist quant researchers, qualitative researchers typically believe that our social reality is constructed, which means the human world is not “real” in an objective sense, but based on everyday interpretations humans make when they go about their business. Unlike the natural science model, the constructivist perspective seeks knowledge by focusing on the interpretations humans make. Constructivists explain the social world through this interpretivist approach.

The goal of constructivist ontology is to understand the process by which people understand their social reality.

Interpretive flexibility is why users hack or mod their tools in ways their designers never intended, and it is why usability testing alone does not reveal the full picture of how technology will get adopted.

Objectivist researchers may observe “technology use” and take it for granted that people want to complete tasks more quickly. But constructivists may begin to ask, “How do humans interpret this technology?”

Quantitative researchers focus more on scale and causation, and like to have replicability and precise measurement. This differs significantly from qualitative researchers, who concern themselves with describing richness of context, the nature of change, and having empathy for participants.

Qual researchers welcome changes in research design, even after research has begun, because it further demonstrates the empathic, participant-led mindset. Quant researchers, by contrast, spend a lot of time preparing exactly the right research design, and do not deviate from that plan when collecting data because it would introduce confounding variables to their experiments.

these two approaches have very different expectations about what constitutes “success.” Quantitative researchers expect their results to show the scale of a thing and the nature of its cause. They are disappointed if their results lack this numerical precision, but don’t mind if it fails to yield rich stories. Qualitative researchers, by contrast, are disappointed if their results do not yield a coherent explanation of exactly how and in what ways a thing happens, who plays what role, and what kinds of objects are recruited for or rejected from a given process. They expect to spend quality time with their participants and are comfortable with ambiguity. They want their data to yield great stories, with lots of color and details, and they want a complete, coherent understanding.

Most people working in companies today are unfamiliar with constructivist approaches, so they unfortunately ask for—and usually get—only objectivist-driven data. Yet they hunger for the deep insight of constructivist data. Stakeholders consider scale and causation the only acceptable outcome for any sort of research, but this is only because they are unfamiliar with qualitative concepts of validity.

If you are a qualitative researcher, you should double down on the strengths the constructivist approach provides. Help your stakeholders luxuriate in the people they are making things for.

Qualitative researchers can augment this richness by sketching out some kind of scale and causation. How often did a particular workaround happen? You do not need to predict this incidence in the population at large, but you can at least show how often it happened in your study.

1. Complementarity: deepen or enhance other data 2. Expansion: expanding the inquiry to ask different questions 3. Development: use one method to inform and improve the other 4. Triangulation: corroboration of earlier data 5. Initiation: resolving earlier contradictory findings

Merging data is perhaps the most challenging because the conflicting belief systems themselves are merged,

managing the research project is about creating a shared sense of purpose and continually communicating that purpose to the wider team.

Communicate more when research

While it’s completely acceptable to have beliefs instead of hypotheses, a deductive approach requires a crisp, falsifiable statement. Help your stakeholders sharpen their deductive questions into falsifiable statements, which can then be proven true or false.

It seems like deductive research is super specific, to the point that the result is either true or false

A deductive approach begins with a reason to believe a certain thing is going on, either through prior research or through theory, and you set out to test whether that is true.

By contrast, an inductive study will start with very open-ended questions like, “What is going on in our organization? How are employees feeling? What are their concerns, if any?”

Its difference starts from the question they want the answer for

In other words, inductive studies seek to understand what counts.

“The researcher begins with an area of study and allows the theory to emerge from the data”

Inductive creates theory which could be proved or disproved later by deductive method

Mixed methods research, then, is more than simply collecting qualitative data from interviews, or collecting multiple forms of qualitative evidence (e.g., observations and interviews) or multiple types of quantitative evidence (e.g., surveys and diagnostic tests). It involves the intentional collection of both quantitative and qualitative data and the combination of the strengths of each to answer research question (J. Cresswell et al., 2011, p. 5)

To use mix method, researchers have to plan how ech method can complement each other

All too often, stakeholders believe research projects will guarantee success of a product or service, instead of simply learning about a product or service.

Expectation management is important

Imagine having several dozen hypothetical findings printed on index cards and asking stakeholders to place each finding along a spectrum of “useful” to “not useful.”

Guide your stakeholders to a realistic expectation of what is possible to prove in an inductive approach, and what is possible to understand deeply in a deductive approach.

Could create an example pack from past research as an expectation management pack. We could break down previous insights to index cards and ask the stakeholders to map which is useful, which is not.

The Stupidity Paradox, sociologists Alvesson and Spicer examine the apparent widespread epidemic of stupidity in companies today. They write, “Functional stupidity is the inclination to reduce one’s scope of thinking and focus only on the narrow, technical aspects of the job. You do the job correctly, without reflecting on purpose or the wider context”

Lol that hit hard

Any meeting focusing solely on expectations challenges functional stupidity because you are asking people to think broadly.

This might be an international phenomenon

The key difference is that qualitative researchers structure the knowledge after data are collected. If you describe this process using the double diamond of design, quant research design is convergent, zeroing in on specific things, while qual is divergent, being more open and exploratory.

An inductive-dominant research design will have a constructivist perspective, so it will assume that participants are making sense of something. The overall goal of the study is to interpret that process. A deductive-dominant research design will have an objectivist perspective, so it will assume there is a given set of facts, and the study’s objective is to uncover those facts by means of proving or disproving a hypothesis.

Whatever dominance you choose, be clear about what this means for your results. In an inductive-dominant approach, you are focusing first on that luxurious understanding of your subject matter. You are stepping back and then diving into The Encounter. You may not have a great grasp on scale or causation, but you will have deep understanding. If you choose a deductive-dominant approach, you will focus first scale and causation of what causes what. You may come up short in your pursuit to luxuriate in the customer, and you might not have the deepest understanding of your context. But you will be able to count the incidence of a thing and test what affects what. In either case, prepare your stakeholders—and yourself—for what you will and will not have.

This is a good description of each method

If you are planning on doing this yourself, prepare to switch contexts and modes of thinking.

You need to switch mode when you switch mdethid

It’s always easier to disprove a single fact than it is to describe a complete system. But when you mix methods, you must do both.

Many non-researchers are familiar with the term “data analysis,” but few people are familiar with the concept of interpretation, or the process whereby the researcher explains the data.

Lol

In their well-known text on qualitative data analysis, Miles, Huberman, and Saldana (2014) outline a very straightforward three-step process: 1. Reduce the data 2. Visualize the data 3. Draw conclusions and verify with the data

Interesting know how

Qualitative, inductive analysis is not trying to reduce data to a summary using averages or frequency tables, but to reduce in another way: to describe the phenomenon in abstract, explanatory ways. This is difficult because it involves understanding–and confidently stating–the essential drivers of change in the given area you are studying. This is less like summarizing and more akin to solving riddles.

Its abstract. No doubt bout that. Just be confident in your findings. And probably study more about the know hows

Inductive analysis involves unriddling, sensemaking, looking at the big picture, or explaining. In this sense, qual data analysis is harder than quant because it necessarily involves interpretation. A researcher can simply state average income, or average height of their quant study sample, and get away without saying anything about what it means. But qual researchers cannot do that. They must interpret as a function of analysis, which goes beyond just listing observations.

Analysis, for Spiggle, is performing specific actions on the data, while interpretation is something deeper and more creative.

Deep and creative

Step 2 of Miles, Huberman, and Saldana’s model involves using various visualization techniques (e.g., diagramming, summary tables like conceptual matrices), and then using that visualization to start making assertions and drawing conclusions.

Viz might be a decent option to convey abstract messages

Induction goes beyond “this is what we empirically observed” and uses symbolic tools like metaphor to describe the deeper meaning of a given set of behaviors.

Metaphor

Deduction involves starting with a general theory or set of beliefs, and using that as a starting point to interpret a specific case. Induction is the opposite: it starts with a specific case (say, when a scientist puts a specimen into a mass spectrometer) and then makes a general statement about the nature of science, for example.

This is kinda close to ajs approach...?

The challenge most researchers have with induction is that it appears to be “bias.” Even using the word “creative” to describe the interpretation process may raise eyebrows as evidence of bias. When people say “bias,” they really mean the researcher will not accurately predict future outcomes or the results of particular changes in independent variables (for example, if a respondent’s income is higher, will they be more likely to purchase this product?).

You cannot explain a general phenomenon without breaking out of established concepts because you are act...

Unfortunately, developing new concepts will by definition defy reliability because it is breaking new ground.

So we need to take qual data less definitively. Qual should never be a definitive answer

When you take qual data and attempt to falsify prior beliefs, this is where the dreaded “How many people did you talk to?” question comes from. Stakeholders believe that large sample sizes determine validity (they don’t), and also believe, however unconsciously, that falsification is the goal of all research.

Lol

qual data do not support falsification and can never rise to that standard.

Just never. Need falsification? Use other method

Often in applied settings, you will see researchers gather their stakeholders to do “group synthesis sessions.” The goal of such sessions is for stakeholders to internalize, adopt and hopefully advocate for the major insights from the study. This is a laudable goal, to be sure, but it is not a great strategy to gather untrained people, for a finite amount of time, and perform surgery on precious participant data.

Mybe just dont

Remember, if you have a mixed methods project, your goal is to show coherence and participant focus on the one hand, and causation and scale on the other.

Remember