Leveraging Lotteries for School Value-Added: Testing and Estimation

New Findings, School Assignment, School Reform, February 2017

Conventional val­ue-added mod­els (VAMs) com­pare aver­age test scores across schools after regres­sion-adjust­ing for stu­dents’ demo­graph­ic char­ac­ter­is­tics and pre­vi­ous scores. The result­ing VAM esti­mates are biased if the avail­able con­trol vari­ables fail to cap­ture all cross-school dif­fer­ences in stu­dent abil­i­ty. This paper intro­duces a new test for VAM bias that asks whether VAM esti­mates accu­rate­ly pre­dict the achieve­ment con­se­quences of ran­dom assign­ment to spe­cif­ic schools. Test results from admis­sions lot­ter­ies in Boston sug­gest con­ven­tion­al VAM esti­mates may be mis­lead­ing. This find­ing moti­vates the devel­op­ment of a hier­ar­chi­cal mod­el describ­ing the joint dis­tri­b­u­tion of school val­ue-added, VAM bias, and lot­tery com­pli­ance. We use this mod­el to assess the sub­stan­tive impor­tance of bias in con­ven­tion­al VAM esti­mates and to con­struct hybrid val­ue-added esti­mates that opti­mal­ly com­bine ordi­nary least squares and instru­men­tal vari­ables  esti­mates of VAM para­me­ters. Simulations cal­i­brat­ed to the Boston data show that, bias notwith­stand­ing, pol­i­cy deci­sions based on con­ven­tion­al VAMs are like­ly to gen­er­ate sub­stan­tial achieve­ment gains. Estimates incor­po­rat­ing lot­ter­ies are less biased, how­ev­er, and yield fur­ther gains.