Leveraging Lotteries for School Value-Added: Testing and Estimation

New Findings, School Assignment, School Reform, February 2017

Conventional value-added mod­els (VAMs) com­pare aver­age test scores across schools after regression-adjusting for stu­dents’ demo­graphic char­ac­ter­is­tics and pre­vi­ous scores. The result­ing VAM esti­mates are biased if the avail­able con­trol vari­ables fail to cap­ture all cross-school dif­fer­ences in stu­dent abil­ity. This paper intro­duces a new test for VAM bias that asks whether VAM esti­mates accu­rately pre­dict the achieve­ment con­se­quences of ran­dom assign­ment to spe­cific schools. Test results from admis­sions lot­ter­ies in Boston sug­gest con­ven­tional VAM esti­mates may be mis­lead­ing. This find­ing moti­vates the devel­op­ment of a hier­ar­chi­cal model describ­ing the joint dis­tri­b­u­tion of school value-added, VAM bias, and lot­tery com­pli­ance. We use this model to assess the sub­stan­tive impor­tance of bias in con­ven­tional VAM esti­mates and to con­struct hybrid value-added esti­mates that opti­mally com­bine ordi­nary least squares and instru­men­tal vari­ables  esti­mates of VAM para­me­ters. Simulations cal­i­brated to the Boston data show that, bias notwith­stand­ing, pol­icy deci­sions based on con­ven­tional VAMs are likely to gen­er­ate sub­stan­tial achieve­ment gains. Estimates incor­po­rat­ing lot­ter­ies are less biased, how­ever, and yield fur­ther gains.