I have excerpted my latest JREF article below. It is about the questions that you should ask yourself when judging the credibility of a scientific study. These questions will start you on a path of navigating an often confusing sea of literature.
What is a Good Study? Questions You Can Ask
In becoming a science-based person, I can imagine a process that involves three tiers. First, you decide that you are going to get your information from reputable sources like scientific journals and then decide that any other claims that you find should have a similar backing. Second, pushing past the veneer of scientific legitimacy, you decide to look into the claims for yourself. This involves not only getting your information from sources based on scientific journal articles, for example, but also going through the study yourself to determine whether it is a “good” study. Lastly, after having navigated scientific sources for some time, you are able to evaluate claims base on methodologies and procedures that you would expect the offered evidence to have if it were indeed credible. Because most of us are not scientists and find it hard to invest in the education it would require to reside comfortably in the third tier, I will try to offer some help with the second.
If you ever would consider a career as a science writer or science journalist, there are a few basic techniques that you must master or at least become proficient at. Among them are learning statistics and how to interpret them, interviewing scientists to get the best information, and how to translate sometimes complex and technical scientific information into something that the lay audience can digest. Another fundamental skill that you must wield effectively is being able to confidently answer the question, “What is a good study?” To this end, what follows are some basic questions that you should ask yourself when trying to determine the validity of a scientific study. You would find these kinds of questions in any introductory level science-writing textbook, and they will become a valuable tool in your skeptical arsenal.
Keep in mind that when you are evaluating a study, the more of these questions that you can have answered, the better off you are. However, if you find yourself questioning every single procedure, method, and ethical choice in a study, this may be a red flag in itself. As a properly skeptical consumer of scientific information, a good place to start at is what is called the null hypothesis. That is to say, assume that a new medical treatment or physics experiment won’t work. Without being downright cynical, greet every claim with this assumption. Your new motto when faced with a claim in a study or elsewhere should be “show me.”
Is the study large enough to pass statistical muster?
Numbers are very important in this regard. For example, the number of patients that a study includes in a clinical trial says a lot about that trial’s “power,” or relative generalizeability (does the study include enough patients to distinguish between treatments?, etc.). Taking a more basic approach, if you were to read in a study that “the majority of US citizens now reject the theory of evolution,” you should find out how many people were in the study. The statistics turn out that if you have less than around 1,024 people for a nationwide study, the margin or error exponentially increases beyond three percent. In study that reports a 49/51 split, this could render the claim worthless.
The other side of this question is to determine if the findings of a study are statistically significant, meaning that there is only an acceptably small chance that the findings were due to random chance alone. The value that is typically used in scientific research is p=0.05. This “p-value” means that the probability that chance alone will produce the findings in a study is only 1 in 20. This is because we must assume the null hypothesis is true, and then assess the probability of some outcome given this assumption. (If this seems to low, it should be noted that many fields in science have much more rigorous standards. Physicists use p-values of p=0.001 to validate their findings. Still, even with the less rigorous standards, most scientific papers are made to be replicated, eliminating chance occurrences even further.) When evaluating a study, pay close attention to this value. As a general rule, any correlation that has a p-value of greater than 0.05 (p>0.05) should not be taken as evidence for anything.
You can find my other JREF articles here.