How to Read Research Critically
Anisfeld, Moshe. (1987). “A Course to Develop Competence in Critical Reading of Empirical Research in Psychology.” Teaching of Psychology. 14 (December, no. 4): 224-227.
Anisfeld’s 1st guideline for reading empirical articles critically: Read the abstract and then direct your attention to the Methods and Results sections to facilitate a view of the findings that is as independent as possible from the author’s view. After you have formed your own conclusions, you will be in a stronger position to evaluate the conclusions of the author and to see whether they are influenced by any particular biases.
Anisfeld’s 2nd guideline for reading empirical articles critically: Whenever possible, schematize on your own the details of the method and results. For instance, if the study compared two groups, you may want to write down, in table format, the variables on which they were equated. This will readily reveal any gaps in the matching of the groups. Moreover, organizing the results in table format will provide a clear, concise view, and will raise issues that may not have otherwise occurred to you. If the results are already presented in tabular format, it may still be valuable to recognize them in another manner that is meaningful to you. The important thing is not to absorb the article passively, but to process it actively. As an active reader, you will be more likely recognize the author’s biases and not be swept away by them.
Anisfeld’s 3rd guideline for reading empirical articles critically: Look carefully at the results in the tables and graphs. Are the results internally consistent? Do they show what the author says that they do? If a particular finding does not “feel right,” you may be able to recompute the relevant statistic from the data provided in the article.
Anisfeld’s 4th guideline for reading empirical articles critically: Determine the weight of the positive results in the context of the entire picture of the findings? One should have less confidence in the conclusions when only one variable out of several was significant. Examine not only whether a particular effect is significant, but also what its magnitude is. Are the conclusions based on strong or marginal results? Be skeptical about results that are significant at the .05 level in one-tailed tests.
Anisfeld’s 5th guideline for reading empirical articles critically: Watch out not only for biased for biased analyses of the data. For instance, are the data coded in a way that presupposes the hypothesis that the author set out to test?
Anisfeld’s 6th guideline for reading empirical articles critically: Distinguish clearly between theoretical constructs and operations used to measure them, and consider whether the operations used are valid for the constructs at hand. Authors often state the results in theoretical constructs. In order to facilitate an independent evaluation of the results, you must not accept this bias.
Anisfeld’s 7th guideline for reading empirical articles critically: Try to identify the author’s bias and look for possible distortions that are consonant with the bias. A useful approach to seeing the results in a new light is to try to interpret them from the perspective of a bias that differs from the author’s bias.
Anisfeld’s 8th guideline for reading empirical articles critically: The results reported in an article are the final product of several steps of transformation and reduction of the original information obtained from the subjects. To gain an understanding of the inner workings of the study and to discover potential problems, reverse the process and try to determine what information was obtained from the subjects and what kind of analyses it was subjected to. By engaging in this step backward process, a reader discovered that the means reported in one article reflected not only response quality, as claimed by the authors, but also response quality.
Anisfeld’s 9th guideline for reading empirical articles critically: Recognize the distinction between limitations of a study and outright inadequacies. If a study only tested one type of subjects (e.g. fifth graders), this does not make it invalid. It only limits its generalizability.
Anisfeld’s 10th guideline for reading empirical articles critically: Do not force criticism on an article. Acknowledge a good article when you see one. Recognize the practical limitations of empirical research. Avoid nit-picking and easy, routine types of criticism (e.g. “why did they test only X number of subjects?”). Remember that by itself, a method is neither good or bad; it can be evaluated only in relation to questions that it seeks to answer. The most valuable critical analysis relates the results and conclusions to the method. Such an analysis may suggest that the results could have been an artifact of some aspect of the method.
Anisfeld’s 11th guideline for reading empirical articles critically: Above all, don’t be overawed or intimidated by authorities, fancy teams, and statistics. Authors have the advantage of knowledge of their work. But you have the advantage of objectivity, psychological distance, and unbiased common sense. Do not give up these tremendous assets!