Junkfood lowers children's IQ and other myths

Last week, more than 400 news stories in just two days reported that a study had found conclusive evidence that fast food makes children stupid and lowers their school tests scores. How many journalists do you think actually went to the original source and read the study?

None.

How can we be so sure?

Because there is no published study in a peer-reviewed journal. There was no ability for educational or health professionals, let alone a journalist, to examine the research and its methodology, data and interpretations.

The study turned out to have been an abstract and paper* presented at a table at the American Educational Research Association’s annual conference in San Diego, California, more than a month ago — on April 16th between 1:15 pm and 1:55 pm. The American Educational Research Association is a professional organization of educators; administrators; directors of research; people working with testing or evaluation in federal, state and local agencies; counselors; evaluators; graduate students; and behavioral scientists. The abstract submitted to the conference program had been presented by a student from Vanderbilt University’s Peabody College of Education and Human Development in Tennessee, the nation’s #1 rated graduate school for education by U.S. News.

Going to the AERA’s meeting program finds the abstract:

Abstract:

That children in the United States are experiencing an epidemic of overweight and obesity is largely accepted by both the popular media and the educational research community. What has not yet been shown by research, however, is a link between consumption of one of the suspected obesity culprits, fast food, and students’ academic performance in school. This paper reports the results of a preliminary regression analysis, using propensity-score matched ECLS-K data, demonstrating a negative relationship between 5th graders’ reported fast-food consumption patterns and their reading and math test scores. Possible policy implications and directions for further research are discussed.

Before we examine this abstract in more detail, why did this obscure student paper suddenly make news headlines more than a month later? Here we have an example of media and ‘studies’ being used for marketing to advance an ideology and agenda.

Plenty of people want us to fear that foods that are not processed from scratch by Mom at home contain unseen ingredients that somehow make the foods unhealthy, “junkfood” and dangerous for children. They count on us to not understand nutritional science, or biology or cooking or chemistry or statistics.

The source of this recent media blitz was an article published in the Times Educational Supplement on May 22nd. TES is an online social network and job search engine for teachers in the UK. It was written by Adi Bloom, a reporter who covers the arts for TES. Within hours, her report had found its way around the world and to the United States.

Ms Bloom’s article was widely repeated nearly verbatim in the UK press — where the campaigns of young celebrity chef Jamie Oliver and the government’s Change 4 Life have been actively trying to eradicate “unhealthy” foods from children’s diets to slim them down— in stories with headlines like: “Fast food diet makes children more stupid” and “Too much fast food 'harms children's test scores.” The scary thing was that the Press Association thought being written up in a social media publication made this a “published study,” and even the education editor of a national newspaper took the TES article as its source for reporting the research.

It’s like that old game of telephone, where a string of people repeat what they’ve heard, with the story becoming more inaccurate and sensationalized with each telling. Although today, news can spread via the internet faster than Mark Twain might ever have imagined.

A lie can travel halfway around the world while the truth is putting on its shoes. — Mark Twain (1835 - 1910)

As Ms Bloom reported:

Fast-food diet can result in slow-brain children

US study finds direct link between consumption of junk food and academic performance. Eating too much fast food can affect pupils’ intelligence, seriously undermining their academic ability, according to new research. Kerri Tobin, of Vanderbilt University in Tennessee, studied the impact of a fast-food diet on the schoolwork of more than 5,500 10 and 11-year-olds. She found that those who ate higher-than-average amounts of junk food scored significantly lower than their classmates in a range of academic tests…

Until now, however, no research has shown a conclusive connection between high-fat and sugary foods and low academic results. Inspired by Jamie Oliver’s campaign to expunge the Turkey Twizzler from school lunch menus, most British schools have removed unhealthy snacks from vending machines, tuck shops and dining halls. But Dr Tobin decided to test whether eating habits out of school also had a significant impact on pupils’ achievement. She therefore asked 5,500 primary pupils to record how many times a week they ate at fast-food restaurants such as McDonald’s or Wendy’s…

Dr Tobin found no correlation between pupils’ fast food consumption and their weight, or between their parents’ income and the amount of fast food they ate. But there was a direct correlation between how much junk food they ate and their scores in a series of literacy and numeracy tests.

Ms Bloom went on to report that students who said they ate fast food daily scored 16.07 points below average in reading and those who ate it three times a day dropped 19.34 points. Math scores were similarly lower by 14.82 points and 18.48 points, respectively. “Overall, higher-than-average consumption of fast food resulted in lower- than-average test scores: 12.79 points less for reading and 12.35 points for numeracy,” she reported. According to a fast food restaurant spokesperson, most customers visit their restaurants two to three times a month.

Tobin was quoted as speculating that perhaps “the propensity to eat fast food is correlated with unobserved characteristics, like parental involvement in homework, which would also affect test scores” and proposing other nonsensical explanations like “the types of food served at fast-food restaurants cause cognitive difficulties that result in lower test scores” or that “pupils eat fast food as a means of coping with low test scores, reversing the cause-and-effect pattern.”

Had any reporter or editor gone to the original source material and understood it, they would have instantly realized that none of the claims they were hearing were credible. Since no one has cared to in more than a month, let’s take a look.

ECLS-K database

This study wasn’t a study as most consumers think that term means. It wasn't an intervention trial and dietary analyses or academic testings were not performed on the children and the children then followed to see if those eating more of certain types of fast food ended up with lower academic scores than a control group. The fifth grade reports in the ECLS-K database were dredged and, using computer modeling, a correlation found between undefined “junkfood” and selective test scores.

The Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) is a national observational study database under the U.S. Department of Education’s National Center for Education Statistics. It includes five surveys of descriptive information: student assessments, parent interviews, self-administered questionnaires from principals and teachers, and abstracts of student records from a nationally representative sample of children from kindergarten to eighth grade.

The most recent information on the children in eighth grade (from 2007) was not used. Instead, the fifth grade data was used. As the ECLS-K Psychometric Report for the Fifth Grade states, “the fourth- and fifth-graders in the field test were different children, not longitudinal measurements of the same children.”

The dietary information available from the fifth graders came from a self-administered questionnaires, containing 19 questions asking the kids to remember how many times they had consumed a list of foods and beverages over the previous seven days. Few adults could probably accurately remember everything they’d eaten for a week, let alone elementary school age kids. But none of the children’s answers were even confirmed by their parents or guardians.

Interestingly, correlations between test scores and any of the other foods or beverages the children reported eating were not reported. It called to mind the example given by Eric Meyer, with the College of Media at the University of Illinois, Urbana-Champaign, cautioning us to beware of incomplete data and seeing only what we think makes sense:

My personal favorite was a habit we use to have years ago, when I was working in Milwaukee. Whenever it snowed heavily, we'd call the sheriff's office, which was responsible for patrolling the freeways, and ask how many fender-benders had been reported that day. Inevitably, we'd have a lede that said something like, "A fierce winter storm dumped 8 inches of snow on Milwaukee, snarled rush-hour traffic and caused 28 fender-benders on county freeways" — until one day I dared to ask the sheriff's department how many fender-benders were reported on clear, sunny days. The answer — 48 — made me wonder whether in the future we'd run stories saying, "A fierce winter snowstorm prevented 20 fender-benders on county freeways today."— Eric Meyer

The focus was on Question #19, which asked the children about fast food:

Notice that the children were given no information on how a portion or snack was defined, they were asked only how many times they had eaten something from a fast food restaurant. The children were likely easily misled by the multiple choices, but how many parents do you know who drive their children to a fast food restaurant three or four times a day? Nor was there any attempt made to determine the type of food or the amount of food eaten: a bite, a nibble, a single fry or a fast food salad, fruit cup or carton of milk.

With this dubious dietary information, she then reported correlations to math and reading scores from the fifth grade assessments on just over 5,500 children. Going to the actual Psychometric Report for the Fifth Grade, however, finds that reading and math tests had been done on 11,267 children during the 2003-4 school year. We have no explanation for why more than half of the data was not used and we know nothing about the children who were included or excluded. Like studies released at meetings, this study wasn’t published in a journal and available to the scientific community for any critical peer review.

As we know, data dredges can find just about any correlations a researcher sets out to find, along with plenty of meaningless and spurious correlations, depending on the data selected, the assumptions made in their regression computer modeling, and the confounding factors considered or ignored. Even then, correlations can never prove causation.

As the ECLS fifth grade findings report cautions:

Readers are cautioned not to draw causal inferences... It is important to note that many of the variables examined in this report are related to one another, and complex interactions and relationships have not been explored here. The variables examined here are also just a few of the variables that can be examined…

According to the fifth grade findings from the ECLS-K, issued by the U.S. Dept. of Education, a multitude of factors were seen associated with test scores, such as:

● health and learning disabilities of children are known to affect performance on standardized tests

● poverty status (61 percent of students in households below the poverty line scored in the lowest third of reading scores, compared with 25 percent of students in households at or above the poverty threshold)

● economic and food security (children living continually in poverty scored lower than those moving in and out of poverty)

● mother’s education level (children of mothers with at least a bachelor’s degrees scored higher)

● educational focus and learning opportunities at home, also seen in ethnic disparities (whites and Asian students scored higher than other minority or disadvantaged children)

● primary language at kindergarten (children from homes where English was the home language when they started school had higher reading scores than those where English was not spoken at home)

● type of school (children in private schools scored higher than those in public schools)

● school and home stability (children who transferred from private to public, or who changed schools frequently and lived in multiple different places between kindergarten and fifth grade scored lower than children from more stable situations)

● absenteeism, especially in kindergarten, was also found to affect school achievement, according to the ECLS-K data (kindergarteners who missed 10% or more of the school year scored lower than those who were able to attend more classes during the school year)

● computer access and type of computer learning opportunities (children in the ECLS-K class with access to computers at home and school were found to associated with higher social skills and academic performance, especially language development, than youngsters without computer access)

● and more.

Yet, while all of these variables were available in the database, we were provided no evidence that any of these factors were considered in Tobin’s computer regression model… or if they were used… when deriving the correlations.

In the end, the faulty methodology was made even more meaningless by the inaccessible findings. We don’t know the children’s mean test scores and standard deviations to make any credible comparisons. As the ECLS-K fifth grade report noted: The mean reading scores among all the children was 136.7, with a 24.3 standard deviation. The math mean score was 111.2 with a standard deviation of 22.4.

The reported small 12-point average differences in the reading and math scores among children supposedly eating fast food three times a day compared to those with average consumptions was less than the wide standard deviations among the ECLS-K tests. There was no tenable correlation demonstrated. Hearing only what we were supposed to conclude doesn’t mean that’s what the data actually supports.

Yet, reported correlations were turned into causations and then flipped into reverse and used to make school policy recommendations for interventions that had never been tested and had absolutely no evidence to support their safety or effectiveness. That is not, ideally, how education or health professionals decide the best care for children. Never the less, Tobin insisted, according to Ms Bloom, that “continued investment in school nutrition plans, and curricula designed to make pupils and parents aware of the academic consequences of their food choices, would be one positive step that schools could take.”

“If you eat that, it will make you stupid” is not a positive, helpful or credible nutritional message for young children or their parents.

© 2009 Sandy Szwarc

* Correction: It was not a poster, as originally noted per the conference program.