Evaluating Psychological Studies (Quantitative)

Travis DixonResearch Methodology

Introduction

An evaluation requires the explanation of strengths and limitations. Thus, in order to evaluate anything we need to first understand its purpose or what it’s trying to do. So before you can evaluate a study in psychology you must first know and understand what the study is trying to do!

Psychology is the scientific study of behaviour and mental processes. It’s aim, therefore, is to describe, explain and even predict human behaviour and mental processes. So a study (e.g. experiment) in psychology is conducted to demonstrate some factor that influences human behaviour and/or mental processes that will help our understanding, explanation and/or prediction.

Quantitative studies in psychology are designed to show the relationship between variables. That is to say, between two factors involved in human behaviour. An evaluation of a psychological study, therefore, should focus on the extent to which the study can really demonstrate the relationship it is trying to demonstrate.

Thus, before you can evaluate a psychological study you must first be able to apply it and explain its conclusions; you must first know what the relationship is that’s being demonstrated in the study before you can determine the strengths and limitations of the study in showing that claim.

So if you’ve come to this page in order to figure out how to evaluate a particular study, make sure you understand the study first! Once you can do that, then move on to the evaluation.

When evaluating quantitative psychological studies (lab experiments, natural experiments, etc.) there are some key areas to look at:

  • Applicability

  • Validity 

  • Reliability 

  • Ethicality 

Applicability

When you are writing about psychological studies you will be applying them to a particular problem or question. Your first step will be to explain how the study demonstrates a particular point.

Thus, your first step in evaluating the research is to think about the limitations in your own explanation and application. The concepts of validity and reliability of studies can help you make that evaluation.

However, before you begin throwing around terms like ecological validity and the like, just think for yourself about the own limitations you can see in the explanations.

Validity and Reliability

A studies validity refers to the extent to which the results can be used to show what the study is trying to demonstrate.

In order to examine a study’s validity, it might be worthwhile thinking about what type of study was conducted. The type of study (experimental, field, natural, etc.) will influence what the study was trying to show and how you critique it’s validity.

  • Are there alternate explanations of the results?
  • Does the research have ecological validity?
    • Was the environment natural and/or were the participants asked to do something that they would normally do, or was it out of the ordinary? (mundane reality)
  • Could the results from the sample be generalized (applied) to the target population, and to all people?
    • Was the sample size representative of the target population?
    • Is there a cultural or gender bias in the sample and/or study?
    • Would the results be the same if the research was conducted today?
  • Are there any other possible explanations for the results of the research?
  • Could the results be explained in any other way other than the conclusions of the research?
  • Was the design of the research suitable for the aims?
    • E.g. was selected or random allocation used for a purpose in an independent sample study?
  • Is there a possibility that demand characteristics could have been present an influenced the results in some way?
  • Could other factors have influenced the research, such as the social desirability effect, maturation, contamination, etc.?
  • Has the research been replicated?
    • If so, were there similar or different results? (test-retest reliability)
  • Did more than one researcher record and interpret the data (inter-rater reliability)?
  • If the study is correlational, is there bidirectional ambiguity?

Ethicality

I sometimes wonder if I made up this word “ethicality” to describe the extent to which a study is ethical in nature. It appears on dictionary.com, but I know how credible that source really is. Nevertheless, it makes sense (at least in my mind).

Ethical Considerations

  • Animals
        • Were the animals’ well taken care of or did they suffer any unnecessary harm?
        • If they did suffer, were they euthanized humanely?
        • Did the research have a clear scientific purpose?
        • Does the research have benefits for humans and/or other animals?
  • Humans
      • Did the researchers obtain parental consent, retrospective consent and/or informed consent?
        • In a covert observation, would retrospective consent be advisable?
      • Were they deceived in anyway?
        • If so, were they debriefed to reduce the possibilities of long term effects?
      • Were they given the right to withdraw?
      • Would there be any long term effects for participants in the study?
      • Were the results kept confidential and the names anonymous?