Some weeks ago I had an interesting email exchange with Dr. Brian D. Ray who responded to me graciously but critically concerning several comments I have made about his organization in some of my blog posts. Specifically, Dr. Ray objected to two things. First, he objected to my association of his organization with HSLDA, maintaining that the two organizations are independent of one another. Second, he objected to my assertions that NHERI’s research typically does not meet the standards of scientific methodology that are commonly accepted among social scientists and are thus of limited value. In the course of our email exchange he challenged me to substantiate my claims or else stop making them. I admitted to him then that my beliefs about NHERI were based upon general impressions I had picked up over the course of my research on homeschooling rather than on any systematic examination of NHERI and promised him that I would engage in a more rigorous study of NHERI and report what I found. Here is what I found.
In this post I will address the subject of whether or not Ray’s work is reliable scholarship. In a future post I will address the association with HSLDA. Warning: this post is VERY LONG. If you don’t want to read the details you can just skip to my conclusion at the bottom.
Ray has been very prolific over the years, and he tends to cite himself quite a bit in his own writings. What tends to happen with this cross-referencing is that the caveats he may have originally included in a piece of scholarship are lost, while the conclusion (always a positive one about homeschooling) remains. Furthermore, the original reports conducted by Ray that contained the caveats about design and sampling limitations were hardly ever published in peer reviewed journals but were released by NHERI itself and must be purchased from them. Needless to say, few journalists or other readers bother to check Ray’s original sources for his claims. Let me give an example from one of the few articles he has published in an out-of-house journal.
In the Fall of 2004 the Journal of College Admission released a special issue on homeschooling and college admission. The lead article was penned by Brian Ray and served as an introduction to the homeschooling movement to set the stage for the more specialized articles that followed. Let me quote here at length what Ray had to say about the academic achievement of homeschoolers:
Many studies have been completed during the past 20 years that examine the academic achievement of the homeeducated (Ray, 2004b). Dozens of researchers have executed these studies. Examples of these studies ranged from a multiyear study in Washington state, three nationwide studies across the United States, and a nationwide study in Canada (Ray, 1994, 1997, 2001c; Rudner, 1999; Wartes, 1991). In study after study, the homeschooled scored, on average, at the 65th to 80th percentile on standardized academic achievement tests in the United States and Canada, compared to the public school average of the 50th percentile.
Researchers, wondering if only certain families––in which the parents have a high educational attainment or family income––are able to homeschool such that their children score high on achievement tests, show that children in homeschool families with low income and in which the parents have little education are scoring, on average, above state-school averages (Ray, 2000, 2004b, ch. 4). In addition, research shows that the parents’ teacher-certification has little to no relationship with their children’s academic achievement, and that the degree of state control of homeschooling (i.e., regulations) has no relationship with academic achievement (Ray, 2004b).
Reading over this summary of research on academic achievement one is struck by how often Ray cites himself. “Dozens of researchers” may have executed these studies (I counted ten in the source he cited for this claim), but here Ray cites only himself, the infamous Rudner study commissioned by HSLDA, and a report by Jon Wartes that was self published and is no longer available from any library in the country. A second thing to note is that in Ray’s text here there is no hint that these studies he cites have any methodological or sampling limitations that ought to make us pause before we generalize from their results to the entire homeschooling movement. The bottom line is that Ray cites Ray to show that homeschoolers score well above public school averages on standardized tests regardless of parent income or educational level, and that teacher certification and degree of state control over homeschooling have no relationship to these results.
Let’s now look at Ray’s five references he cites here to see if the conclusions he draws from them in this article are merited. Here are the five references:
Brian D. Ray, A Nationwide Study of Home Education in Canada: Family Characteristics, Student Achievement, and Other Topics (Salem, OR: NHERI, 1994)
Brian D. Ray, Strengths of Their Own—Homeschoolers across America: Academic Achievement, Family Characteristics, and Longitudinal Traits (Salem, OR: NHERI, 1997)
Brian D. Ray, “Home Schooling: The Ameliorator of Negative Influences on Learning?” in Peabody Journal of Education 75 (2000): 71-106.
Brian D. Ray, “Homeschooling in Canada” in Education Canada 41 (2001): 28-31
Brian D. Ray, Worldwide Guide to Homeschooling (Nashville: Broadman and Holman, 2004).
I will take each in turn.
1. First is Ray’s Nationwide Study of Home Education in Canada. This self-published document is only available through NHERI, who will sell it to you for $12. WorldCat, the best source for finding documents in libraries, found it in only three libraries around the country. I was able to obtain one of these copies from Northwest Christian College in Eugene, OR.
In its introductory chapter, Ray’s Canada study surveys previous research on the academic achievement question, much of it his own. Nowhere in his summaries of these other studies, most of which found homeschoolers scoring in the 80th percentile or above, does Ray mention variables like family income or educational level that might skew results. He does mention a few studies that found little difference between homeschoolers and public schooled students.
For this Canada study Ray used the same basic survey he had used earlier for his first big study of U.S. homeschoolers (all HSLDA members and volunteers) published in 1990 as A Nationwide Study of Home Education. As in the U.S. survey, the Canadian respondents were all volunteers who were “aggressively” recruited by “one national organization,” who distributed it via direct mail (p.11). 2,800 surveys were mailed out and 808 came back, a 31% response rate.
On p. 14 Ray makes a very important qualification about this study, “This is not a causal-comparative study….background variables in this ex post facto study are not controlled in such a way as to make possible conclusions about the causes of academic achievement test scores being higher (or lower) than those of students in conventional schools, or conclusions about the causes of learning difficulties. This study is not designed to determine whether type of schooling (i.e., home versus public or conventional) causes differential academic achievement.” Ray also notes that since all the subjects were volunteers his results are likely not representative.
Given his methodology, it is not surprising that almost all of his subjects were white (94%), two parent (97%), and “born again” Christian (89% of fathers, 92% of mothers). Average family size was 3.5 children. As with Ray’s previous studies in the U.S. that used a similar methodology, the children he surveyed typically scored “at about the 80th percentile” on standardized tests (p.40).
2. Second is Ray’s Strengths of Their Own. This slef-published book-length study also can be purchased from NHERI’s website (for $10.) It is much more readily available through interlibrary loan, so I had no trouble securing a copy.
This study sets out to be another survey snapshot of homeschooling, but with the added component of a longitudinal study of adults who had been homeschooled. After a review of research that is very like what he offered in the Canada study, Ray describes his methodology for this project.
As with his previous studies, Ray sent out a survey very like the one he had used before. Unlike previous studies, however, he tried this time to get some volunteers outside of HSLDA’s orbit. He especially wanted more religious diversity. Nevertheless, the bulk of his volunteers were recruited by HSLDA, “One well-known national organization, in particular, played a prominent role in distributing the survey instrument throughout the country via direct mail, articles in home education publications, and telephone contacts.” (p. 18) Ray also tried very hard to get responses from families who had responded to the first big HSLDA study back in 1990. In addition to all of this Ray also sent his survey to more secular (“inclusive,” or in my book’s term “open communion”) homeschooling organizations.
Ray sent out 5,995 surveys altogether. He got back 1,657, or 28.8 percent. The caveats he made in his Canada study he repeats here, almost verbatim. He notes that he is not controlling for variables that would make possible comparison between homeschoolers and conventional schools. He notes that since all of his subjects are volunteers the results aren’t representative of homeschoolers generally.
Despite his efforts to expand the pool, Ray’s subjects remained overwhelmingly white (96%), two parent (98%), and some sort of “biblically-based Christian” (84%-90%). He did get some Unitarian, New Age, and other non-Christian responses. Average family size was 3.3 children.
Again, students in his sample tended to score in the 80th percentile and above on most tests, “significantly higher than the national average of the 50th percentile.” (p.55) In addition, 69% of homeschool graduates went on to some sort of postsecondary education and 31% into employment.
3. The third article, “Homeschooling: The Ameliorator” comes from the well-respected Peabody Journal of Education. It begins with a summary of the research (as of 2000) on homeschooling and academic achievement. In this section Ray cites himself eight times along with data from the Departments of Education in Alaska, Tennessee, and Oregon, data from HSLDA, and the Wartes studies mentioned in the Journal of College Admissions article.
Interestingly, Ray also gives more coverage in this piece (note that this article was published in a peer-reviewed journal) to studies that have found little to no difference in scores between homeschoolers and institutionally-schooled children. He concludes, modestly, “Overall, the research base to date (see also Ray, 1988, 1993, 1999) indicates that home school students perform at least as well as public school students…” (p.75) This is a much more modest conclusion than what he claims in his self-published work and in the 2004 Journal of College Admissions article.
Ray then moves on to a discussion of his methodology. This article is based on the same data set used in Strengths of Their Own, which means that all of his data is coming from volunteer homeschoolers recruited from “the lists of various national and statewide organizations.” He doesn’t say anything in this article about which national and statewide organizations he tapped into, but he does acknowledge that his method does not “necessarily result in a representative sample of home education families.” Neither, he claims, does it “necessarily introduce sampling bias.” Hmm. (p. 79)
As we have seen, Ray mailed out 5,995 questionnaires and received back 1,657, a response rate of 29%. Ray took the results of this survey, turned the scores on various standardized tests taken by children of his respondents to Z scores, and did a multiple regression analysis on them to discover whether scores were at all affected by (a) educational level of father, (b) educational level of mother, (c) teacher certification status of father, or (d) teacher certification status of mother, (e) family income, (f) amount of money spent on home education, (g) degree to which family was complying with state law, (h) gender of student, (i) number of years student was homeschooled, (j) extent to which family visits public libraries, (k) time spent in formal educational activities, (l) age of child upon starting formal education, (m) degree of regulation by the state, (n) who administered the test to the child, and (o) degree of computer use in child’s education. Note that all of these variables concern differences in scores between homeschoolers. This article does not compare homeschoolers and public schooled children.
Next comes a “limitations” section where Ray admits that his subjects were all volunteers and that therefore “one should keep in mind the limitations of representativeness and generalizability.” He describes the study as “descriptive and exploratory in nature.” (p.81)
Ray’s findings, as usual, are off the charts. Students scored on average in the 85th percentile or above on most subjects when compared to national averages. The only variables he studied that produced statistically significant differences in student scores between homeschoolers (at the very rigorous .01 level) were father’s and mother’s educational level, gender of students, frequency of library visits, and whether or not parents administered the tests. But even these variables (especially parental educational level and gender) explained relatively little of the variation, less in fact than has been reported in public schooled children. Why? Maybe, Ray speculates, because there wasn’t a lot of variability in his study to start with. Most of his subjects were very similar to one another on most variables.
This is the closest Ray comes in this study to addressing the obvious methodological flaw it contains if one wished to use it to compare homeschoolers with public schooled children. Ray, like Rudner did in the study Ray describes briefly at the end of this paper, is comparing a cohort of homeschoolers with the national average. He never tells us in this paper what percentage of the parents in his sample had a college degree, nor does he tell us what their income was, their race, or percentage with two parents in the home. [In the original study (Strengths of Their Own) Ray did divulge this information and found small relationships between student achievement and a few variables (especially parent educational level). In Strengths of Their Own, however, which he will use in subsequent publications as evidence that such variables as family income do not affect homeschooler achievement, he did not note as he does here that the reason the variable doesn’t explain differences is because there wasn’t much variation in the sample to begin with.]
It’s not at all fair to compare homeschoolers from families where, say, both parents have a college degree and the family enjoys an annual income of $40,000 or more with national averages that include impoverished children from single parent homes. Ray’s study of variables in this paper only applied to differences among homeschoolers, not to differences between them and public schooled children. This same criticism has been regularly leveled at the Rudner study as well. In the Peabody article itself this isn’t a problem because Ray is not arguing that homeschoolers outperform public schoolers here. He’s only trying to account for variability among homeschoolers. But in subsequent publications Ray will cite this article as evidence that homeschoolers outperform public schooled children.
4. Finally we come to Ray’s Worldwide Guide to Homeschooling. We need not spend much time on it for it offers no new information but is only a short summary of Ray’s previous work. In a four page section Ray surveys the research on academic achievement, citing the same sources here as he has before. What is most notable is the way Ray himself ends this survey. Though, as we have seen, Ray is at pains in his own studies to remind readers that his results don’t control for variables that would make it possible to compare homeschoolers to public schooled children, in this book Ray concludes, “While not all studies show home-educated students scoring above average, overall research clearly indicates that homeschooled students perform at least as well as, and usually better than, their conventionally schooled counterparts…” (p.56)
My Conclusion:
What are we to make of all of this? Ray’s original studies are good pieces of research on their own, even if their research design severely limits their generalizability. Ray acknowledges their limitations himself and counsels against generalizing from their findings. Why then does Ray himself use them over and over in subsequent writings to buttress the claim that homeschoolers outperform public schoolers on standardized tests? What he ought to be saying and what other journalists and pundits who use his studies ought to be saying is not that homeschoolers outperform public schoolers. They ought to be saying that some middle-class, white, two-parent, conservative Protestant homeschoolers who volunteered for a research study that was pitched to them as a great opportunity to show off homeschooler success to the public, score in the 80th percentile or above on standardized tests. That’s really all Ray’s studies show.
[…] 7, 2008 by Milton Gaither In my previous post I described how a series of email exchanges with Brian Ray motivated me to devote more systematic […]
[…] oft-cited studies of homeschoolers, such as those conducted by Brian Ray and HSLDA (which I review here and here), use methods of data collection that lead to an over-representation of conservative […]
[…] months ago on this blog I paid extensive attention to Ray’s previous studies (post 1 and post 2). What I said of his previous studies is true as well of this new one. It is simply […]
[…] from Ray’s data. If you missed it you can find my review of Ray’s oeuvre here (part 1) and here (part […]
[…] even intervals. In two previous posts I reviewed this large body of work, which you can read here and here. This new study tries very hard to overcome one of the most persistent deficiencies of […]