Being able to critically evaluate a study is a key skill for any budding psychologist. However, like anything, when you’re first learning how to do this it can be very difficult. In this post, we look at 3 simple steps you can take to evaluate any study.
Read More
- 7 exam tips for evaluating studies
- So you want to assess ecological validity
- So you want to assess population validity
Teaching Tip: You could use this as a class activity by having students choose one study and following these steps by themselves.
Step 1: Identify the Method
Read the study you want to evaluate carefully and try to identify the method that is used. How was the data gathered and analyzed? Was it a true experiment, a correlational study, a case study, a survey, an observation, etc.? Perhaps it was a twin study, a meta-analysis, or a longitudinal study. Before you can evaluate the study, you must know the methodology.
The IB has very strict definitions of “research method.” When evaluating a study, feel free to focus on any methodology used by the researchers.
Tricky Bits: Sometimes studies combine multiple methods. For example, it might have been a survey that also calculated correlations. Or a twin-study that was longitudinal in nature. This is fine. Identifying more than one method or technique that was used in the study is OK. In fact, it might make the next steps a little easier.
Step 2: Identify General Strengths + Limitations
After you’ve identified the method, it’s important that you can recall the general strengths and limitations of this method. The table below is a brief summary.
| Method | Strengths | Limitations |
| True Experiment | Can control variables to establish a cause-effect relationship | Generalizability to real-life settings could be an issue. |
| Quasi or Natural Experiment | Can study the effect of a naturally occuring IV that can’t be manipulated in the lab. | Difficult to control all extraneous variables. |
| Field Experiment | Can study the effects of an IV on a DV in a real-life setting. | Difficult to control all extraneous variables. |
| Correlational Study | Can test the strength of a relationship between two naturally occurring variables. | Correlation does not mean causation. |
| Observations | Can see how people act in real-life, naturalistic settings. | Depending on the type (covert v overt), ethical issues or the Hawthorne effect might occur. |
| Survey/Questionnaire | Can gather data relatively quickly and easily from a wide range of people. | Gathers self-report data, which may not be 100% reliable. |
| Case Study | Can use a range of methods to get an in-depth understanding of behaviour. | Limited sample sizes, so generalizability could be an issue. |
Exam Tip: For IB Psychology exams, I recommend becoming an expert on true experiments and correlational studies. This makes all questions regarding research methods and ethics more manageable. It will also help you develop your ability to evaluate these two methodologies.
99% of IB Psychology students only do the first two, so learning how to complete the third step will separate your answers from the rest.
Step 3: Apply to YOUR Study (+ -)
Examiners are not impressed by one sentence, generic evaluations.
Nearly every student in psychology only ever gets as far as these two steps. The classic example of a student evaluating an experiment goes like this: “But this was a laboratory experiment so it lacks ecological validity.” This is not critical thinking. This third step is essential for you to show your critical thinking. You need to take one of the strengths and/or a limitation and explain how it might apply to your study. This is difficult and takes time and careful consideration. The following table gives you some pointers for a few common methods you may be evaluating.
| Method | Tip for Evaluation |
| True Experiment | Explain WHY the results might not apply to a real-life situation or scenario. |
| Quasi Experiment . | Explain how a specific extraneous variable might affect the results. |
| Case study | Explain WHY the results might not apply to another group or population. |
| Correlational Study | Explain possible bidirectional ambiguity, or how a third factor might explain the relationship. |
| Survey/ Questionnaire | Explain WHY people may not be 100% honest or accurate in their responses. Look at the nature of the topic they’re being asked about or the types of questions asked. |
Example: Loftus and Palmer’s Car Crash Study (Read more)
This study was a true experiment because the researchers manipulated the independent variable (verbs in the question) in a controlled environment. One strength of this is the ability to control variables. In their second experiment, they had a control condition who didn’t watch any video. This helps to isolate the verb in the question as the single variable that’s increasing speed estimates and the memory of seeing broken glass. It can help control for demand characteristics because people may say “yes” to seeing broken glass because they may feel they’re supposed to.
However, it also means that we can question the extent to which this study may apply to real-life situations. To explain this properly you need to explain why it might not apply to a real-life situation by focusing on specific details (this video on our YouTube channel will help explain how to do this).
So a student’s evaluation might look like this:
One strength of this this experiment is they could control variables to establish a cause-effect relationship. In their second experiment, they had a control condition who didn’t watch any video. This helps to make sure the verb in the question is the only factor affecting speed estimates and the memory of seeing broken glass. It can help control for demand characteristics because people may say “yes” to seeing broken glass because they may feel they’re supposed to. Using all students also helps to control for participant variability, because people who are older may have more driving experience and be better at estimating speeds and so less likely to be influenced by a leading question.
However, it also means that we can question the generalizability of this study and the extent to which the results may apply to real-life situations. For example, could we expect the same effect in a court room or interrogation room involving a real life crime? In these situations there are massive consequences for false memory and misremembering key details, so people may focus more and try harder to remember. Perhaps even the stress could improve their memory of the event (because acute release of cortisol improves memory consolidation) so their memories will be more reliable.
Read More
You can see from the example above that a true evaluation of a study is difficult and takes very careful consideration. But if you can write a fully-developed explanation of the strengths and limitations of a study, you are on track to do very well in your exams.
Exam Tip: If you are asked in an essay question to specifically evaluate a study, you should include strengths and limitations. However, if you’re evaluating a study as one of your counter-arguments in a more general question, then just focusing on the limitations is OK.
Watch more here…
Question or comment? Leave it below.
Find more exam tips in our Revision Textbook designed for IB Psychology students.
Travis Dixon is an IB Psychology teacher, author, workshop leader, examiner and IA moderator.