DissertationHow Transparency and Reproducibility Can Increase Credibility in Policy AnalysisA Case Study of the Minimum Wage Policy EstimateFernando Hoces de la GuardiaThis document was submitted as a dissertation in May 2017 in partial fulfillment of the requirements of the doctoral degree in public policy analysis at the Pardee RAND Graduate School. The committee that supervised and approved the dissertation consisted of Kristine M. Brown (Chair), Susan M. Paddock, and Chapin White.PARDEE RAND GRADUATE SCHOOLAbstractThe analysis of public policies, even when performed by the best non-partisanagencies, often lacks credibility (Manski, 2013). This allows policy makers to cherry-pick between reports, or within a specific report, to select estimates that better matchtheir beliefs. For example, in 2014 the Congressional Budget Oce (CBO) produceda report on the eects of raising the minimum wage that was cited both by opponentsand supporters of the policy, with each side accepting as credible only partial elementsof the report. Lack of transparency and reproducibility (TR) in a policy report impliesthat its credibility relies on the reputation of the authors, and their organizations,instead of on a critical appraisal of the analysis.This dissertation translates to policy analysis solutions developed to address thelack of credibility in a dierent setting: the reproducibility crisis in science. I adapt theTransparency and Openness Promotion (TOP) guidelines (Nosek et al, 2015) to thepolicy analysis setting. The highest standards from the adapted guidelines involve theuse of two key tools: dynamic documents that combine all elements of an analysis in oneplace, and open source version control (git). I then implement these high standards ina case study of the CBO report mentioned above, and present the complete analysis inthe form of an open-source dynamic document. In addition to increasing the credibilityof the case study analysis, this methodology brings attention to several components ofthe policy analysis that have been traditionally overlooked in academic research, forexample the distribution of the losses used to pay for the increase in wages. Increasingour knowledge in these overlooked areas may prove most valuable to an evidence-basedpolicy debate.JJJ Contents1 Introduction 81.1 Definition Of The Problem . 81.2 Relevance . 101.3 Research Objectives . 121.4 Structure of the Dissertation . 132 Background 152.1 Transparency and Reproducibility in Research: Lessons For Policy Analysis . 162.1.1 The Reproducibility Crisis in Science and Why It Matters for Policy Analysis? 162.1.1.1 The crisis . 162.1.1.2 The response: best practices for Open Science . 182.2 A General Framework For The Role Of Policy Analysis in Connecting Research AndPolicy . 202.2.1 Manskis Critique And The Role of Low Transparency and Reproducibility .222.2.1.1 Policy makers can cherry-pick the facts . 23W2.2.1.2 Little knowledge exists regarding how research aects policy estimates 272.2.1.3 Hard to automate or systematically improve policy reports . 292.3 Description of the case study . 293 Methods 333.1 Adapt TOP Guidelines to Policy Analysis . 343.1.1 Three standards of TR in Policy Analysis . 363.1.1.1 Workflow . 363.1.1.2 Data . 383.1.1.3 Methods . 383.2 Apply Guidelines to the CBO Report and Reproduce it Using the Highest Standardsof TR . 403.2.1 A Dynamic Document for CBOs report . 403.2.2 Distributed Version Control (Git) . 413.2.3 Dierences Between Reproducing Research and Policy Analysis . 423.3 Sensitivity Analysis . 424 Results 454.1 Description of the Dynamic Document for the CBO Report . 454.2 Assessing TR in Original CBO Report and in the DD . 504.2.1 Standard #1: Workflow . 504.2.2 Standard #2: Data . 58WJ4.2.3 Standard #3: Methods . 595 Sensitivity Analysis 615.1 Sensitivity of a Few Parameters for all Policy Estimates . 625.2 Sensitivity of all Parameters on a Sample Welfare Function . 646 Conclusions and Extensions 716.1 Conclusions . 716.2 Extensions within the current case study (minimum wage) . 736.3 General extensions to TR in policy analysis . 746.4 Extensions to other possible policy analyses . 757 Appendic