By Les Willis aka Nutrition Consultant
Scientific articles are referenced everywhere. The message is: say a scientist said it and you must be right. The thing is, if you want to look a little more closely at what the scientist actually said there is not necessarily a lot of help available. Just to make it a little bit harder, if you were to start looking for guidance on ‘how to read a scientific paper’ you get information aimed at students doing academic assessments. However, if you just want to know what to look for and how to spot the good from the bad, really there is not a lot of help. The purpose of this article is to help the non specialist reader dive into the specialist literature and make sense of it.
If you are following up to see if the study actually backs up the claim being made, either by a supplement company or forum poster, then you can, more often than not, just read the title and abstract. If you are reading to learn more, you need the whole article. In either case you need to be reading critically and asking questions.
When assessing the article, it is very important to remember what it is the article is being used to back up; for example if the article is supposed to tell you that you really want to eat oats for breakfast and the article is Protein Metabolism and Endurance Exercise (Gibala 2007) then you are right to think ‘that really isn’t going to tell me much about oats or breakfast’. Similarly, if you are a bodybuilder trying to find out how much protein to have, this one is not going to help you much either. Which brings us to the first part of assessing an article:
Look at the title
It seems obvious to start with the title, because it is often assumed that the article and the point it is being used to back up actually go together; don’t assume! The simple fact is, there are a number of times that they don’t; it is easy to be impressed by Muscle glycogen resynthesis rate in humans after supplementation of drinks containing carbohydrates with low and high molecular masses (Aulin et al 2000), but that title doesn’t tell you what the results were; nothing could have happened at all. Other titles promise a lot like Protein co-ingestion stimulates muscle protein synthesis during resistance type exercise (Beelen et al 2008) but, once again, when you look closely, this one too fails to deliver a definitive statement.
The key to reading a title and the whole process is to think critically and ask questions of what you are reading. Let’s take that second title: If ‘Protein co-ingestion stimulates muscle protein synthesis during resistance type exercise’ was quoted to back up the statement that you should have whey and glucose in water after a workout, you could be fooled into thinking that it does indeed support that recommendation. After all the title has the word ‘protein’, and whey is a protein, it says ‘co-ingestion’ which means having with something else, so whey with glucose – looking good. It says ‘stimulates muscle protein synthesis’ which sounds a lot like muscle growth, and it talks about resistance exercise, which we know is what scientists call weights workouts – job done.
Well no, actually, this article does not support having whey in glucose after a workout at all. Look closely: ‘during’ means that the protein has to have been consumed before the end of the exercise and ‘resistance-type’ doesn’t tell you it was a proper weights workout. In fact, in the hands of a scientist hints that really it wasn’t a resistance exercise but something similar to one, so already this article title has fallen down. But, if you look further, ‘protein co-ingestion’ with what? The title does not tell you that the protein was taken in with carbohydrate. So you can see, even by just the title the paper does not support the statement.
In the world of scientific articles, the title is a really important part of the article. Often the title is all you have got; if you are lucky then you could have the abstract as well.
The abstract
An abstract is a short summary of the article, or more precisely, the abstract is supposed to be a summary of the article. Some abstracts are great; other abstracts are really good at not actually telling you anything at all. If you have the abstract you are off to a great start; if you haven’t then don’t give up, instead go to PubMed and search for the title there. Pubmed is a massive database of scientific articles. To be honest, if the article you are looking up is not in Pubmed then it is too old or too obscure to be worth any more effort.
Once you have the abstract it is time to have a closer look and pull it apart to see what is really going on. First question is:
Who is the study on?
This is a fundamental consideration; many studies are not on people: Orally administered Endoxifen is a new therapeutic agent for breast cancer (Ahmad et al 2010) looks like a breakthrough and something to excited about; in reality this is a study on rats. Which is why it can be important to look at the abstract in the first place. Rats are not little humans, we share similarities, but we also have rather big differences especially when it comes to metabolism. If the study is on rats then in reality we are a long way from practical human studies, in fact only a test-tube (or in vitro) study is further away.
If the study is on people, then it is important what people the study was done on; in sports science studies practicalities mean often ‘healthy volunteers’ are used. Unfortunately a ‘healthy volunteer’ is not what you need. Over time habitual training has an effect on the body, after all that’s why people do it.
The thing is when you first undertake training, the effect is largely the same whether it is endurance or resistance, it is only over time that the body develops changes that are specific to the training you do. For example, an endurance trained athlete has more mitochondria along with an altered lipid and carbohydrate metabolism, whereas your resistance trained athlete with have a decreased density of mitochondria, and an increased phosphocreatine and glycogen content in their muscle, coupled to a more efficient protein metabolism. And that is just the start of the different changes that occur with regular training.
Once you know who the study was on you need to see:
What the study did
This is important because many studies can have little to do with what athletes actually do when they train. We have seen already that using athletes is important. Similarly it can also be important to have study design that replicates what actually happens. A great example of this is glutamine studies. Under normal conditions the body gets all the glutamine it needs by making it from other amino acids, however, in exceptional circumstances, like severe burns or trauma, the body needs extra to help it cope. You would think that a hard training athlete would benefit from glutamine; thing is, we don’t know. And one of the biggest reasons we don’t know is, not one study has trained the subjects hard enough for extra glutamine to be needed. It is a bit like saying wet weather tyres don’t give extra grip when you have only used them in the dry!
At this point you can often stop; with a bit of background knowledge combined with looking things up, you can very quickly see whether or not the study had a chance of actually finding out what it said it was going to find out. If the study hasn’t got a hope, like our glutamine studies, then what you are reading isn’t going to back anything up – unless the point being made was that the study didn’t show anything.
Some studies do survive this far, the next thing to ask is:
What did they measure?
This is different to what was done to whom. This is asking, when our white coated friends had done their poking, prodding and torturing, what did they measure. For example, Recovery from a cycling time trial is enhanced with carbohydrate-protein supplementation vs. isoenergenic carbohydrate supplementation (Berardi et al 2008) took trained cyclists, put them on a cycling machine and had them exercise for an hour on a simulated time trial. So, we have the right subjects doing a test that simulates what they would really do. The scientists even made sure the subjects had the same breakfast; in fact the study design looks really good, because it could measure recovery from exercise.
The measure of recovery here that the subjects did the time trial again 6 hours after the first one having had either carbohydrates or carbohydrates and protein. What you have to ask is, is the performance of these cyclists a measure of recovery, or does this measure back up the original point? After all, if I just said ‘having carbohydrates and protein will you do more squat reps?’ then this study doesn’t help my case at all.
Titles and Abstracts – putting it together
When you read a title and abstract put the two together. Ask would doing what they did find out what they said they were trying to find out? If you are not convinced that this is going to happen from the title and abstract, then you are unlikely to be convinced by the whole article.
Try this out on: Effects of 28 days of resistance exercise and consuming a commercially available pre-workout supplement, NO-Shotgun®, on body composition, muscle strength and mass, markers of satellite cell activation, and clinical safety markers in males (Shelmadine et al 2009), and ask ‘would this supplement help an experienced women competitor pre-contest?’. The answer is: you can’t tell, the study is with untrained males.
It’s a review article
By this point you may have found out that the study wasn’t on anyone and the scientists didn’t measure anything because you have a review article. There are two types of review article: your normal review and your systematic review. Systematic reviews are easy to spot; often they are helpful and have ‘systematic review’ at the top. If they don’t, then you will find they have a methodology section that tells you how they found the articles that were reviewed. The aim of a systematic review is to minimise the bias and present an up-to-date authoritative statement of what is being reviewed. For example, the helpfully titled Systematic review of the effect of the co-ingestion of protein with carbohydrate post exercise on recovery in athletes (Willis 2010 unpublished): The title tells you exactly what the rest of the paper is going to be about. You will also notice it is quite focussed in what it talks about; again systematic reviews are more often narrow in focus so that they can be authoritative in their findings.
The other type of review is much more common and you have to be a lot more wary of it. Unlike the systematic review, a review is under no obligation to be unbiased and won’t tell you how the papers it reviews were found. It could be that the writer went to their filing cabinet and pulled out what was there.
Going a bit deeper
This is where things can get interesting, and where you can find out a lot without actually reading the paper.
First thing, hit Pubmed and find the article. This can take a couple of goes, but if you have the full title it should be easy to find (some times you have to shorten it a little). If it doesn’t show up by title, try searching the author and two of the key words; for example ‘Shelmadine’, ‘effects’, ‘exercise’, would find the above Shelmadine article. This can often do the trick for a particularly stubborn piece of work. If you can’t find it, ask the person who quoted it for a copy (more about this in a minute).
If you have found the article in a database, the next step is to get it. Some articles are either open access or free text. If you have an article that is free then there will be a link and you get a nice PDF file to read. However, many articles are not open access or free text; if this is the case, ask the person who gave you the reference for a copy (more about what to do if they can’t later).
When you search the paper you may find that the original is not in English; this should raise a suspicious eyebrow. Not because science is only done in English, but if the person referencing hasn’t told you it is not in English it is usually because they haven’t read the whole article.
When you ask for the original, you should be able to get it. Supplement companies should be more than happy for you to read the whole thing, and if it was posted, the person posting should be able to send you a copy of it. Of course, having read this article, some people may try and play smart and make excuses – still be suspicious that they have only read the abstract.
It is a sad fact that on web forums many people say something, go to Pubmed with a few key terms and then quote a reference from the first page of results in an attempt to look like they are really intelligent and you should believe them. One trick is to do a key word search yourself; for example, if they are talking about protein after exercise you would search ‘protein’ and ‘post exercise’. Often you need to limit to searching just titles or title and abstract. Whatever you do, if their reference comes up first hit and it’s not open access or free text – call their bluff and ask for it.
Why ask for papers I can’t get for free?
Firstly, because you’re interested, and secondly, because an abstract is always the edited highlights of the paper. When you read the full text you can really see what happened; often the abstract doesn’t tell you enough to be able to know what you want to know, or sometimes the abstract doesn’t tell you important details that really change what the paper actually said. For example, one paper on the effect of protein and carbohydrate post-exercise used wheat protein; who uses wheat protein post-workout in the real world? That nugget is not in the abstract. Another paper claims that it used resistance trained males in its abstract. Turns out that the resistance trained males in the study had not used any supplements, including a protein supplement, in the 12 months prior to the study. Do you know any consistent serious weight trainers like that?
If a person is backing themselves up via only reading the abstract, they are attempting to look like they know more than they do, which is never a good thing. If a company is doing this then look carefully for their horse, six-shooter, spurs, boots and ten gallon hat!
Reading the whole thing – some tips
Firstly, a common criticism of sports science papers is that the number of people in them is really small. In the world of medicine studies can be done with a cohort of thousands of people in multiple centres across the world, truly huge trials. In sports science if you get 30 you have done well, and if muscle biopsies are involved getting 10 can be an achievement. At the end of the day, that is how it is, and to criticise a study for low numbers of participants and say its results don’t count is like saying ‘I don’t listen to people from Texas and you are from Texas’, short sighted, a little ignorant and could quite possibly get you shot in the foot.
Secondly, reading the whole of the paper is like reading the title and abstract, only longer. You are asking critical questions. The best thing is don’t be intimidated by long words and complicated language; don’t worry if the results section reads like a foreign language; and finally, if you don’t think the study is any good you don’t have to believe it.
Studies in journals are not the final word, they are what one group of people found and thought about what they found. Sure the people doing the paper have qualifications and degrees, and some of these people are Stephen Fry super smart. However, none of those things means they got it right, thought of everything or that you accept what they say without question.
It is all about asking questions
For example, in my study I have taken untrained teenagers, made them do single leg extensions and looked at whether having them eat seaweed for breakfast made a difference to how many reps they could do. In my results I have found that by eating a kilo of seaweed for breakfast my teenagers could do an extra two reps and therefore I have concluded that supplementing the breakfast of elite cyclists with seaweed will help them cycle for longer.
Is this a good study? Is this a credible conclusion? Even if I am right, would anyone really eat a kilo of seaweed? I may have written lots about how important performance is, have a results section that proves it is a significant result using every test in my PC statistics software package, and I may even use some very long words, but what I did was daft and my conclusions are ‘poop’. And the same follows for any real studies you read; if it all adds up to nothing much, then you don’t have to listen to it. Ask questions and be open for answers, because you never know you could learn something too.
When reading a paper, be it the title and abstract or the whole thing, don’t be afraid of thinking and concluding that the study is ‘poop’, because some studies really are that bad. And don’t be afraid of asking for more information, especially if it is a forum post, make the poster explain the paper, ask them the question. If the person is really trying to be helpful, and not indulging in an online ‘I can wee higher than you contest’, then they will be able to do this for you.
I hope you feel more confident about taking on the more specialist literature; most of it looks more complicated than it really is. Sure there are papers like Distinct anabolic signalling responses to amino acids in C2C12 skeletal muscle cells (Atherton et al 2009), which truth be told, you really don’t need to be reading; and quite possibly mentioning it out loud could be considered a social no-no! But there are also papers like Cereal and non-fat milk support muscle recovery following exercise (Kammer et al 2009), which are very friendly and are certainly worth a little read.
There you go; get critical, get thinking and develop the Knowledge Behind your Power!
