Gaming the system | Arkansas Blog

Friday, November 14, 2008

Gaming the system

Posted By on Fri, Nov 14, 2008 at 10:18 AM

Blog reader Jake da Snake, a professional educator, checks in today with commentary on advertising in today's Democrat-Gazette from the eStem charter school. The ad includes three graphs purporting to show sharp gains in scores on standardized tests by eStem students in just three months of enrollment there.

Perhaps so. I'll reiterate that stats we compiled show that eStem opened with a student body disproportionately composed (compared with the surrounding LR school district) of students already demonstrating successful academic performance. They are continuing to do so. Good for them and the school, if not unanticipated.

But that's not Jake's concern. His concern is with presentation of the numbers, in what he indicates is a statistically dishonest fashion.  Read on. You won't find a similar analysis of these numbers in the D-G, a leading advocate of eStem financially and editorially.

eStem 2008 to 2009 Annual Report


The NWEA test used at eStem is used by over 2,200 school districts in the testing of over 3 million students.  It is recognized as a reliable and valid measure of academic growth.  NEA and other educational organizations tout the fact that NWEA believes that testing should take place more than once a year in order to provide a more accurate picture of progress.

Problems with eStem advertisement:

The primary problem with eStem’s advertisement is the use of bad graphs which exacerbate errors of scale.  This is called a Zoom Graph.

The Zoom Graph  is a graph that does not start at (0,0) thus making small changes seem big. Useful in the study of trends, but often passed off as actual changes. Such graphs are very popular in financial circles (and educational circles, as well).

By reducing the range of starting and ending scale numbers on the graph, one can make the graph appear much steeper.  The actual percentages of improvement are as follows:  (1)  Elementary – 4.1%; (2) Middle – 1.2%; (3) High – 1.1%.  However, looking at the graphs, it appears as if one is climbing Mt. Everest as one goes up these few percentage points. 

The graphs are distortions. 

* Graphs should not provide a distorted picture of the values they portray.

* Distortion can be either deliberate or accidental.

* (Of course, it can be useful to know how to produce a graph which bends the truth.)

Ed Tufte of Yale University has defined a “lie factor” as a measure of the amount of distortion in a graph. 

* The lie factor is defined to be:

   Lie Factor = size of effect shown in graphic divided by the size of effect shown in data

* If the lie factor of a graph is greater than 1, the graph is exaggerating the size of the effect.

The Lie Factor for these graphs is several times greater than 1.

The secondary problem is minor but is one that goes against NWEA recommendations.  All graphs use a dotted line to represent “Expected Growth.”  NWEA argues that this is not proper terminology and that one should use “Typical Growth.” 

Finally, no information is given as to how these scores compare to other schools using the NWEA.

From the ArkTimes store


Comments (21)

Showing 1-21 of 21

Add a comment

Subscribe to this thread:
Showing 1-21 of 21

Add a comment

More by Max Brantley

  • ASU announces $29 million athletic project backed by private booster foundation

    The Arkansas State University Board of Trustees today approved a ground lease with the Red Wolves Foundation which will take the responsibility for building $29. million worth of improvements to the football stadium.
    • Nov 20, 2017
  • Monday's open line

    Here's the Monday open line and today's news roundup
    • Nov 20, 2017
  • Protest set on tax legislation

    A coalition of groups will demonstrate at 12:15 p.m. Tuesday in front of the state Human Services headquarters at Seventh and Main to show displeasure with the Senate tax legislation.
    • Nov 20, 2017
  • More »

Readers also liked…

  • IHOP coming down, but .....

    I always scan the Little Rock City Board for items of interest this week and this one caught my eye: A zoning measure required by a proposal to tear down the IHOP at Markham and University.
    • Apr 30, 2016
  • LR speakers blast state board for double standard

    A series of speakers, beginning with Sen. Joyce Elliott, denounced what they saw as a hidden agenda favoring charter schools at the state Department of Education and asked the state Board of Education for return of local control.
    • May 12, 2016
  • The long and winding road: No exception yet for 30 Crossing

    The Arkansas highway department's representative on the Metroplan board of directors told the board today that the department is requesting an exception to the planning agency's cap on six lanes for its 30 Crossing project to widen Interstate 30 from six to 10 (and more) lanes.
    • Jun 29, 2016

Most Shared

Most Viewed

Most Recent Comments

  • Re: Monday's open line

    • I personally find the whole "Personhood for fetuses" absolutely absurd. How stupid to insert it…

    • on November 20, 2017
  • Re: Vox pop: Mixed response to Capitol monument proposals

    • Here is a good news for those interested. There is away you can earn money…

    • on November 20, 2017
  • Re: Protest set on tax legislation

    • I am sorry about the debit to your grandchildren Razor, but if the GOP majority…

    • on November 20, 2017



© 2017 Arkansas Times | 201 East Markham, Suite 200, Little Rock, AR 72201
Powered by Foundation