Stanford: The Gold Standard for Measuring School Performance

Post Reply
Site Admin
Posts: 217
Joined: Fri Jul 13, 2007 8:38 pm

Stanford: The Gold Standard for Measuring School Performance

Post by solutions »

---------------------------- Original Message -----------------------------
Subject: Stanford U as The Gold Standard for Measuring School Performance
From: Solutions
Date: Sat, October 2, 2021 1:41 pm PDT

Dear John,

This morning’s weekly e-mail called attention to 4 items, including the Suggested Answers to the First Short Quiz entitled “Thomas Sowell vs. Stanford University” and referenced in last week’s e-mail, which are available at viewtopic.php?f=658&t=2111&sid=18c94812 ... ade48d9d8a.

Since you appear to have requested us to re-read the Suggested Answers to the First Short Quiz, I did so and have some questions.

I take your point that Thomas Sowell’s “Charter Schools and THEIR ENEMIES” (emphasis added) implies that he would recognize Stanford University as THE CHIEF ENEMY of Charter Schools because of their 2009 study -

(1) Which was funded by such pro-charter groups as the Walton Family Foundation and the Michael and Susan Dell Foundation;

(2) Which analyzed data from 2,403 charter schools in fifteen states and the District of Columbia -- about half of all charters and 70 percent of all charter students in the nation at the time;

(3) Which found that 37% had learning gains that were significantly below those of local public schools, 46% had gains that were no different, and only 17% showed growth that was significantly better; and

(4) WHICH DID NOT TAKE INTO ACCOUNT THAT the relatively-poor performance of charter schools occurred despite their admitting only students of parents who were functional enough to apply and then expelling students who did not meet performance standards.

And I take your point that Thomas Sowell STUDIOUSLY IGNORED the Stanford U 2009 study AND HAD NO COMPREHENSIVE STUDIES OF HIS OWN to counter the Stanford U study that Charter Schools perform WORSE THAN PUBLIC SCHOOLS.


(1) Why do you appear to believe that the Stanford U methodology for evaluating performance is the Gold Standard?

(2) Has Stanford U published the results of any studies since 2009?

Your friend,


---------------------------- Original Message -----------------------------
Subject: Re: Stanford U as The Gold Standard for Measuring School Performance
Date: Sun, October 3, 2021 8:47 am MDT
To: Solutions

Dear Solutions,

Thank you very much for your e-mail.

Taking you questions in reverse order –


Stanford U’s CREDO (aka Center for Research on Educational Outcomes) has been releasing reports since 2009.

Their website ( currently lists 67 reports issued 2009-2021 for which details can be obtained. Virtually all of the reports deal with -

(1) Individual states; or

(2) Individual cities.

However, a few have intriguing titles such as a 2017 study entitled “Charter Management Organizations.”

AMAZINGLY (though perhaps not, since the news was bad and CREDO is financed by such pro-charter groups as the Walton Family Foundation and the Michael and Susan Dell Foundation), Stanford U’s CREDO has issued ONLY TWO REPORTS OF “NATIONAL CHARTER SCHOOL STUDIES” – 2009 and 2013.

The 2013 study comprises 104 pages available at

The 2013 report covered 26 states which (p. 2 of the report) “educate over 95% of the nation’s charter school students.”

After making zillions of comparisons, its overall conclusion showed that the comparatively-poor performance of charter schools had not improved significantly since 2009 as disclosed on p. 56 –

In math –

2013 – only 29% of charter schools had significantly-stronger growth than their respective public-school competitors and 31% had significantly-weaker growth (40% no significant difference).

2009 – only 17% of charter schools had significantly-stronger growth than their respective public-school competitors and 37% had significantly-weaker growth (46% no significant difference).

In reading –

2013 – only 25% of charter schools had significantly-stronger growth than their respective public-school competitors and 19% had significantly-weaker growth (56% no significant difference).

2009 – not disclosed.


The Suggested Answers to the Short Quiz Q&A B-3 said –

“On pp. 3-4, Sowell confesses that he would only focus on charter schools meeting three criteria including – (1) a charter school and public school ‘serving the same local population,’ (2) the students in each such pair of schools ‘are taught in the same building,’ and (3) the students in each such pair of schools ‘have one or more classes at the same grade level in the same building.’

“For anyone who knows anything about education, the second and third criteria regarding “the same building” are pure nonsense and represent nothing more than an UNADMITTED EXCUSE to avoid addressing the 2009 COMPREHENSIVE study of Stanford University.

“The application of Sowell’s three criteria result in his focusing on SOLELY five charter-school networks in NYC!!!”

And Q&A B-4 continued the attack by referring to “THE CLASSIC ‘CENSORED SAMPLE’ THAT HE ENGINEERED.”

So what would an intellectually-honest method look like???

The answer is contained in the 2013 Stanford U CREDO report ( which discloses its methodology (pp. 8-12).

*****Stanford U’s CREDO methodology*****

As in the 2009 study, this study employed the virtual control record (VCR) method of analysis developed by CREDO. The VCR approach creates a “virtual twin” for each charter student who is represented in the data. In theory, this virtual twin would differ from the charter student only in that the charter student attended a charter school. The VCR matching protocol has been assessed against other possible study designs and judged to be reliable and valuable by peer reviewers. Details of these assessments of the VCR method are presented in the Technical Appendix.
Using the VCR approach, a “virtual twin” was constructed for each charter student by drawing on the available records of traditional public school (TPS) students with identical traits and identical or very similar prior test scores who were enrolled in TPS that the charter students would have likely attended if they were not in their charter school. Factors included in the matching criteria were:

• Grade level
• Gender
• Race/Ethnicity
• Free or Reduced-Price Lunch Eligibility
• English Language Learner Status
• Special Education Status
• Prior test score on state achievement tests

Table 3: Percent of Charter School Students with Matches State Reading Math

State - Reading Percentage - Math Percentage

Pooled Average 86.2% 84.1%

Arizona 74.5% 74.3%
Arkansas 88.8% 81.7%
California 89.4% 83.4%
Colorado 88.9% 88.4%
District of Columbia 82.6% 80.0%
Florida 88.0% 88.1%
Georgia 93.0% 91.1%
Illinois 89.9% 90.2%
Indiana 85.8% 84.0%
Louisiana 86.6% 86.2%
Massachusetts 76.9% 80.8%
Michigan 83.0% 84.6%
Minnesota 78.8% 76.7%
Missouri 79.0% 79.1%
Nevada 76.3% 76.6%
New Jersey 73.2% 72.7%
New Mexico 75.9% 75.5%
New York 84.1% 81.9%
New York City 85.4% 84.2%
North Carolina 81.1% 74.9%
Ohio 77.8% 78.7%
Oregon 79.6% 80.6%
Pennsylvania 86.5% 86.3%
Rhode Island 77.5% 74.4%
Tennessee 96.0% 95.1%
Texas 89.5% 90.0%
Utah 91.0% 86.1%

VCRs were re-examined in every subsequent test period to ensure that the conditions of the match still applied – namely that the students included in the VCR record were still enrolled in TPS and had not left the state. In cases where the conditions were violated, the VCR was reconstructed to delete the disqualified student records. This process allowed CREDO to follow the matched pairs over as many years as possible while maintaining the integrity of the match.

There were some circumstances that resulted in a charter student not finding an acceptable match. Students were not matched in the first year for which they had test data, since a prior score was required for matching. This restriction removed the first tested grade in each state from the analysis as well as the first year for students who relocated to a new state. It is also possible students could have all of their initial matches invalidated due to changes among their TPS matches. In those cases, students were re-matched with a new VCR when possible. Additionally, the tight match restrictions of the VCR protocol occasionally limited the number of possible matches.

Fair Analysis of Impacts on Student Academic Progress

Most researchers agree that the best method of measuring school effectiveness is to look at their impact on student academic growth, independent of other possible influences. The technical term for this is “value-added. ”10 The central idea is that schools should be judged on their direct contribution to student academic progress. This necessarily takes into consideration the students’ starting scores on standardized tests as well as student characteristics that might influence academic performance. This approach forms the foundation of our study design.

In order to conduct a fair analysis, this study followed the approach of the 2009 study: we looked at the academic growth of individual students as reflected in their performance on state achievement tests in both reading and math. To assure accurate estimates of charter school impacts, we use statistical methods to control for differences in student demographics and eligibility for categorical program support such as free or reduced-price lunch eligibility and special education. In this way, we have created the analysis so that differences in the academic growth between the two groups are a function of which schools they attended.

While we went to great efforts in each state to match the charter students and their virtual twins, it is important to recognize that states differ in the location of charter schools and the students they serve. These differences mean that charter students are not likely to be representative of the state’s full complement of students. These differences are described in the Demographics chapter. Our statistical models included controls for these differences between states to take these differences into account when estimating the overall impact of charter school attendance.

*****End of Stanford U’s CREDO methodology*****

If you have any additional comments or questions, please advise.

Your friend,

John K.

Post Reply

Return to “Participant Comments – Revisiting The Issue Of Charter Schools: Stanford University vs. Stanford’s Hoover Institution – Oct 13”

Who is online

Users browsing this forum: No registered users and 1 guest