Gaming the system | Arkansas Blog

Friday, November 14, 2008

Gaming the system

Posted By on Fri, Nov 14, 2008 at 10:18 AM

Blog reader Jake da Snake, a professional educator, checks in today with commentary on advertising in today's Democrat-Gazette from the eStem charter school. The ad includes three graphs purporting to show sharp gains in scores on standardized tests by eStem students in just three months of enrollment there.

Perhaps so. I'll reiterate that stats we compiled show that eStem opened with a student body disproportionately composed (compared with the surrounding LR school district) of students already demonstrating successful academic performance. They are continuing to do so. Good for them and the school, if not unanticipated.

But that's not Jake's concern. His concern is with presentation of the numbers, in what he indicates is a statistically dishonest fashion.  Read on. You won't find a similar analysis of these numbers in the D-G, a leading advocate of eStem financially and editorially.

eStem 2008 to 2009 Annual Report


The NWEA test used at eStem is used by over 2,200 school districts in the testing of over 3 million students.  It is recognized as a reliable and valid measure of academic growth.  NEA and other educational organizations tout the fact that NWEA believes that testing should take place more than once a year in order to provide a more accurate picture of progress.

Problems with eStem advertisement:

The primary problem with eStem’s advertisement is the use of bad graphs which exacerbate errors of scale.  This is called a Zoom Graph.

The Zoom Graph  is a graph that does not start at (0,0) thus making small changes seem big. Useful in the study of trends, but often passed off as actual changes. Such graphs are very popular in financial circles (and educational circles, as well).

By reducing the range of starting and ending scale numbers on the graph, one can make the graph appear much steeper.  The actual percentages of improvement are as follows:  (1)  Elementary – 4.1%; (2) Middle – 1.2%; (3) High – 1.1%.  However, looking at the graphs, it appears as if one is climbing Mt. Everest as one goes up these few percentage points. 

The graphs are distortions. 

* Graphs should not provide a distorted picture of the values they portray.

* Distortion can be either deliberate or accidental.

* (Of course, it can be useful to know how to produce a graph which bends the truth.)

Ed Tufte of Yale University has defined a “lie factor” as a measure of the amount of distortion in a graph. 

* The lie factor is defined to be:

   Lie Factor = size of effect shown in graphic divided by the size of effect shown in data

* If the lie factor of a graph is greater than 1, the graph is exaggerating the size of the effect.

The Lie Factor for these graphs is several times greater than 1.

The secondary problem is minor but is one that goes against NWEA recommendations.  All graphs use a dotted line to represent “Expected Growth.”  NWEA argues that this is not proper terminology and that one should use “Typical Growth.” 

Finally, no information is given as to how these scores compare to other schools using the NWEA.

From the ArkTimes store


Comments (21)

Showing 1-21 of 21

Add a comment

Subscribe to this thread:
Showing 1-21 of 21

Add a comment

More by Max Brantley

  • When Johnny Reb comes marching to Hot Springs

    They are assembling for and against white supremacist symbols in Hot Springs today. Photographs by Brian Chilson of the Arkansas Times.
    • Aug 19, 2017
  • Dixie's Defenders and the scrapheap of history

    And you thought the Civil War was over. Supercharged by a neo-Nazi march in Charlottesville and Donald Trump's signal of sympathy, the Rebel remnants of America and assorted white supremacists, bigots and garden variety nuts have taken up arms in defense of symbols of the Lost Cause.
    • Aug 19, 2017
  • Inside the Charlottesville march with the neo-Nazis

    Vice has compiled a powerful documentary inside the white supremacist march that turned violent in Charlottesville last weekend.
    • Aug 19, 2017
  • More »

Readers also liked…

  • Al Gore remembers Dale Bumpers

    Former Vice President Al Gore, a former U.S. Senate colleague of Dale Bumpers, sent a statement on Bumpers' death Friday:
    • Jan 3, 2016
  • Two plead in fraud of sheriff's office

    A former employee of the Pulaski County sheriff and a North Little Rock woman who sold goods to the sheriff's office have pleaded guilty to mail fraud in a scheme to steal from the sheriff's office, according to a news release from the U.S. attorney's office.
    • May 16, 2017
  • In Little Rock, Marco Rubio sells American exceptionalism

    This is Rubio's axiomatic answer to Donald Trump's insistence that he and he alone will Make America Great Again: America is the greatest, always has been.
    • Feb 22, 2016

Most Shared

  • Take yourself there: Mavis Staples coming to LR for Central High performance

    Gospel and R&B singer and civil rights activist Mavis Staples, who has been inspiring fans with gospel-inflected freedom songs like "I'll Take You There" and "March Up Freedom's Highway" and the poignant "Oh What a Feeling" will come to Little Rock for the commemoration of the 60th anniversary of the desegregation of Central High.
  • Klan's president

    Everything that Donald Trump does — make that everything that he says — is calculated to thrill his lustiest disciples. But he is discovering that what was brilliant for a politician is a miscalculation for a president, because it deepens the chasm between him and most Americans.
  • On Charlottesville

    Watching the Charlottesville spectacle from halfway across the country, I confess that my first instinct was to raillery. Vanilla ISIS, somebody called this mob of would-be Nazis. A parade of love-deprived nerds marching bravely out of their parents' basements carrying tiki torches from Home Depot.

Most Viewed

Most Recent Comments



© 2017 Arkansas Times | 201 East Markham, Suite 200, Little Rock, AR 72201
Powered by Foundation