Value Measuring MethodologyWhy isnt “traditional” business case analysis providingthe information OMB is looking for? Primarily focused on financial benefits (e.g., ROI) that impact the government onlyAssumption that what is good for governmentis good for citizensNo structure to force the development of quantifiable measuresNon-financial benefits are not directly factored into analysisAnalysis is viewed as a means to get funding, not a tool for on-going management & evaluationHow can traditional analysis be supplemented to better address the challenges of the e-Government environment?In July 2001, the Social Security Administration (SSA), in cooperation with the General Services Administration (GSA), took on the task of developing an effective methodology to assess the value of electronic services that would be: 1. Compliant with current federal regulations & OMB guidance2. Applicable across the federal government3. “Do-Able” A team of Booz Allen analysts and thought-leaders affiliated with Harvard Universitys Kennedy School of Government were contracted to support this effort.Critical Inputs & ResearchTraditional / Emerging ApproachesFederal Agency StaffLegislation and OMB CircularsState Government StaffGovernment Developed Training DocumentsPrivate SectorAcademic InstitutionsOMBThink TanksGAOdevelopment presentationresearch & analysisdiscussionThe approach used to develop VMM was built upon the foundation of a public / private partnershipThe output of this effort was the Value Measuring Methodology (VMM)First articulated in Building a Methodology for Measuring the Value of e-Services (1/02)Refined & tested through application to two cross-agency e-Government initiatives (e-Travel & e-Authentication)Release of the VMM How-To-Guide and VMM Highlights document by the Best Practices Committee of the CIO Council (10/02)VMM Roll-Out, held by the Council for Excellence in Government in cooperation with the CIO Councils Best Practices Committee, OMB, and GSA (4/03)VMM OverviewVMM IS A scalable and flexible approach for quantifying and analyzing value, risk, and cost and evaluating the relationships among them Helps to create a roadmap for on-going management and evaluationSupports the development of critical management plansIt is important to understand what VMM IS and ISNT VMM IS NOTOne Size Fits AllA Way to Avoid AnalysisOnly Useful for e-Government InitiativesCOMPLIANT WITHGPRACCAOMB A-11CONSISTENT WITH THE PHILOSOPHY OF THE PMAThe Essential FactorsWhat Could Make Costs Go Up or Performance Slip From Projected Levels?How MuchWill It Cost?What Benefits Will It Provide toDirect UsersSocietyGovernmentA Decision FrameworkV A L U EValue Factorspriorityvalue measure(s)metric, target, scalepriorityR I S KCOSTCustomizedCost ElementStructure1.0 2.03.0 Risk InventoryRisk Tolerance BoundaryDefine User Needs & PrioritiesQuantifiable Measures of Performance (Metrics, Targets)Foundation for Analysis & On-going Performance MeasurementEarly Consideration of RiskThe technically superior digital technology offers more bandwidth than analog technology because the signal is.ORCommunicating Value to Customers and StakeholdersUsing digital LMR will prevent the catastrophic communications malfunctions and inefficiencies that cost lives in the aftermath of 9/11 in NYC. Digital LMR will accomplish this by What will make an Appropriations Committee staff member or OMB care about an investment in digital Land Mobile Radio (LMR) equipment for public safety agencies across government? PMA ImperativesCaptures All Value Factors/BenefitsAnalytic RigorClarityCompletenessFocus On ResultsVMM Effective in Building “WINNING”OMB Exhibit 300sPart I Capital Asset Plan and Business Case (All Assets)Summary of SpendingProject Description and JustificationPerformance Goals and MeasuresProgram ManagementAlternatives Analysis Risk Inventory and AssessmentAcquisition StrategyProject and Funding PlanPart II Additional Business Case Criteria for Information Technology Enterprise Architecture Security and PrivacyGPEAOMB 300Fully satisfied by VMM outputsSupported by VMM output and processValueDefinitionL A Y E R 1 Project ValueDefinitionDefinitionProject ValueDefinition(Measures)DefinitionDirect User(Customer)Social(Non-Direct User)Government Financial Government Operational/ Foundational Strategic/ PoliticalProject ValueDefinition(Measures)Project ValueDefinition(Measures)Project ValueDefinition(Measures)Project ValueDefinition(Measures)Project ValueDefinition(Measures)L A Y E R 2 Identifying and Defining Value Concise, Illustrative NameRobust Reliable Service Brief DescriptionService with:Minimal or no disruptionsConsistent service regardless of normal fluctuations in demandHigh fault tolerance with built-in redundancyAdequate speed to meet business needsPerformance MetricsTargetScale (0-100)Frequency of service disruptionsNone0 disruptions = 1001 disruption = 954 disruptions = 60Length of service disruptions 10 minutes 10 min. = 901114 min. = 60Is an executable Continuity of Operations plan in place (with a backup NOC) sufficient to pass annual certification?YesNo = 0 / Yes = 100Latency 75 milliseconds 1243The way measures are articulated can directly impact the way they are perceived and understood. The definition must consist of four parts:Structured Approach to Identifying and Defining Value MeasuresBuilding A Direct User Measure What Do Users Want?E-Travel InitiativeTravelers / ManagersAnytime Access to DataReal-Time Data Availability Simplified/Automated Trip Planning Speed of Reimbursements Budget & Accounting Staff Standardized and Electronic Auditing Function / Simplified Application and Monitoring of Travel Entitlements Access to Reporting Information Businesses (Travel Management Companies)Timely & Accurate Receipt of Bill PaymentsAnalytical Hierarchy Process (AHP) tools are designed to help groups enhance the quality of their decisions. These tools: Bring structure to the decision-making process;Elicit ideas, feelings and the judgments of stakeholders;Represent those judgments as meaningful numbers;Synthesize the results; andAnalyze the sensitivity of those judgments to changes. Through the use of pair-wise comparisons, the relative importance of each of the criteria is calculatedAttention is focused on areas of disagreementPrioritizing Value Factors - The Analytic Hierarchy ProcessSummary of VMM Weighting & Scoring for Title XVI “Check Your Benefits”Agency DirectionProgram Mgt. DirectionSub-CriteriaValue Factors and Value Measures(% share of total weight)Maximum Value of the Value FactorsMaximum Measure ValueScoring (“Normalized”)Sub-Criteria Points (%)Value ScoreDirect User25%Scoring Specialists: OPB, OAS, OQA, OCOMM21.0%Expanded Access (50%)12.5%10 of 1012.5%User Time Saved (30%)7.5%6 of 104.5%Increased Satisfaction (20%)5.0% 8 of 104.0%Social15%Scoring Specialists: OCOMM, OQA, OSM13.0%Increase Public Confidence (33%)5.0%10 of 105.0%Access for “Hard to Reach” (33%)5.0% 8 of 104.0%Equity and Fairness (33%)5.0% 8 of 104.0%Government Financial10%Scoring Specialists: OB, DCS, OQA9.0%Effectiveness and Efficiency (50%)5.0% 8 of 104.0%Return on Investment (50%)5.0%10 of 105.0%Operational/Foundational30%Scoring Specialists: DCS, OES, OPB, OAS25.0%Supports Future eService Transactions (50%)15.0%10 of 1015.0%Supports Transformation (33%)10.0% 6 of 106.0%Supports Organizational Learning (17%)5.0% 8 of 104.0%Strategic / Political20%Scoring Specialists: OSM, OES18.0%Satisfies External Mandates / Requirements (50%)10.0%10 of 1010.0%Supports ASP (50%)10.0% 8 of 108.0%TOTAL100%100%86%86%Applying VMM to Title VXI “Check Your Benefits,” we determined the following scores for each of the Value Factors and their respective value measuresRiskRisk that is not identified cannot be mitigated.Risks that are not mitigated can cause a project to fail either in the pursuit of funding or, more dramatically, while the project is being implemented.IDENTIFYING RISKS:Consider “standard” IT project risks Identify project specific risks via input from technical & policy staff, representatives of partner agencies collected from:Working SessionsSurvey EffortsEXAMPLE OMB RISK CATEGORIES:Project Resources / FinancialTechnical / TechnologyBusiness / OperationalOrganizational & Change ManagementData / InformationSecurityStrategicPrivacyIdentifying and Defining RiskOrganizational Tolerance For Cost Risk (increased cost)Organizational Tolerance for Value Risk (slippage in performance)What is the decision processbehind the following:1.Buying a $1 lottery ticket for the chance to win $1 million. Odds are 1 in 1,000.2.Buying a $100 lottery ticket for the chance to win $1 million. Odds are 1 in 1,000.3.Buying a $100 lottery ticket for the chance to win $10 million. Odds are 1 in 1,000.Defining Risk ToleranceAs the estimated most likely value score increases, risk tolerance is likely to increase.VALUE TOLERANCE BOUNDARY0%5%10%15%20%25%30%35%0102030405060708090100VALUE RISKUnacceptableArea AcceptableAreaValue Risk Boundary 0%5%10%15%20%25%30%0102030405060708090100VALUE SCOREUnacceptableArea AcceptableArea0%5%10%15%20%25%30%0102030405060708090100Unacceptable AreaAcceptable AreaVALUE RISK BOUNDARY COST RISK0%5%10%15%20%25%30%35%$-$5$10$15$20$25$30$35$40$45$50COST ($M)Cost Risk Boundary AcceptableArea0%5%10%15%20%25%30%35%$-$5$10$15$20$25$30$35$40$45$50COST RISK BOUNDARY Acceptable AreaUnacceptable AreaCOST TOLERANCE BOUNDARYAs the estimate most likely cost increases, risk toleranceis likely to decrease.Value and Cost Risk Tolerance Boundaries communicate the upper limit of the range of risk an organization will accept in both areas. CostConsider Value and RiskIdentifying & Defining CostsDirect User Value TrainingMarketingAccess (e.g., kiosks)IncentivesSocial Value Communications Public awareness AdvertisingPublic RelationsGovernment Operational / Foundational Value Maintain legacy systems and processes during transitionsOn-going maintenance of paper process Ensure a complete, comprehensive cost estimate Alleviate the risk of missing costs or double-counting by developing a Cost Element StructureInvestments made on incomplete or inaccurate estimates are likely to run out of funding and, therefore, require justification for additional funding or a reduction of initiative scope Estimating and Comparing Value, Cost, & RiskIdentify viable alternatives that have the potential to deliver an optimum mix of both value and cost efficiencyAlternatives Must Address People, Process & Technology!trainingoutreachmanagementstaffingcommunicationsrecruitmentsocializationuser support508 requirementslanguage requirementsEA / FEABPRAcquisitionOutsourcing/in-sourcingconcept of operationsrisksecurityprogram managementfundingcollaborationcommunicationsevaluationlegislative req.policy req.EA / FEAhardwaresoftwareinterfacedata req.EA / FEAIdentifying and Defining Viable AlternativesProjects the results of maintaining current systems and processes while attempting to keep pace with changes over time.T I M Erising demandworkforceattritioncustomersatisfactionstatus quobase caseThe Base CaseAvoid Analysis Paralysis: Match Information to the Phase of DevelopmentData sources and detail depend upon the initiatives stage of developmentUse the best information available rather than looking for information that doesnt existUpdate this information as “better” information becomes availableALWAYS DOCUMENT DATA SOURCES & ASSUMPTIONSStage of DevelopmentData SourcesStrategic PlanningStrategic & Performance plansSubject Matter Expert InputNew and existing user surveysPrivate/public sector best practices, lessons learned and benchmarksEnterprise ArchitectureModeling & simulationVendor / market surveyBusiness Modeling & PilotsSubject Matter Expert inputData from analogous government initiativesNew & existing user surveys for each business linePrivate/public sector best practices, lessons learned & benchmarksRefinement of modeling & simulationImplementation & EvaluationActual data from phased implementationActual spending/cost dataUser group / stakeholder focus groups / surveysOther performance measurementCollecting DataEXAMPLEUSE RANGES TO INCREASE CONFIDENCE IN COST ESTIMATES!200150100# of Employees to beTrained/yearAnnual Cost per Employee TrainedHighMedLowProjected Range of Training Costs Inputs$1500$1200$1000InputsUsing RangesConduct Uncertainty and Sensitivity Analyses on Both Cost & Value EstimatesUncertainty Analysis Based on considerationsof requirement, cost estimating, and technical uncertaintyIncreases confidence in the estimate. Doesnt increase the precision of the estimateTool: Monte Carlo SimulationOutput: “Most Likely” or “Expected” Cost & ValueSensitivity AnalysisBased on the output of the Monte Carlo Simulation Sensitive variables have a significant impact on the overall estimateOutput: Identification of which variables have a significant impact on the overall estimate. Can be used to determine which variables merit additional researchUncertainty and Sensitivity AnalysisALTERNATIVE 1 - COST RISK ANALYSISRiskProbabilityCost ImpactedImpactCost OverrunsMed1.0 System Planning & DevelopmentLow2.0 System Acquisition & Imp.High3.0 System Maintenance & OperationsMedCost of Lost Info / DataHigh1.0 System Planning & DevelopmentMed2.0 System Acquisition & Imp.Med3.0 System Maintenance & OperationsLowALTERNATIVE 1 - VALUE RISK ANALYSISRiskProbabilityValue ImpactedImpactCost OverrunsMedTotal Cost Savings to InvestmentLowTotal Cost Avoidance to InvestmentLowCost of Lost Info/ DataHighTotal Cost Savings to InvestmentLowTotal Cost Avoidance to InvestmentLowHW/ SWFailure & ReplacementMedAccessibility of e-Gov services to UsersHighUser Trust In Internet TransactionsHighApplication Owner Confidence in Identity of UsersHighReduction of Identity FraudHighRegulatory ComplianceHighTotal Cost to Savings InvestmentHighTotal Cost Avoidance to InvestmentHighThe probability of a specific risk occurring remains constant through out the analysis of a specific alternative, regardless of where it impacts the value or cost of a particular alternativeThe impact of a single risk factor may differ in magnitude at each point where it interacts with cost and value Analyzing Cost Risk and Value RiskYou should be able to answer the following questionsWhat is the estimated cost of each alternative?What is the financial return on investment associated with the alternatives?What is the value score associated with the alternatives?What are the cost and value risks associated with this alternative? What effect do they have? (value and cost risk scores)How do the value, risk and cost of the alternatives compare?Does the cost risk and value risk associated with the alternatives fall within the range represented by the relevant risk tolerance boundaries?Pulling Together the InformationInvestment Cost To Value(Expected & Risk-Adjusted)0102030405060708090100$-$5$10$15$20$25$30$35$40C O S T ($M)V A L U EExpected Alt 1Risk Adjusted Alt 1Expected Alt 2Risk Adjusted Alt 2Expected Alt 3Risk Adjusted Alt 3Alt 1Alt 2Alt 3Based on This Information, Which Alternative Would You Choose?Comparing Value to CostThe risk associated with all of the value scores fall within the acceptable area. Alt. 2 bears the lowest value risk. COMPARING VALUE TO VALUE RISK14%13%80700%5%10%15%20%25%30%35%01020304050608090100V A L U EV A L U E R I S KAlt 1 UnacceptableAreaAcceptable AreaAlt 3 Alt 2 7%7070VALUE RISK BOUNDARY COMPARING COST TO COST RISK17%10%$30$1012%$250%5%10%15%20%25%30%35%$-$5$10$15$20$25$30$35$40C O S T ($M)C O S T R I S KAlt 2 Alt 1AcceptableAreaAlt 3 UnacceptableAreaAcceptable AreaCOST RISK BOUNDARY The only alternative that falls squarely within the Cost Risk Boundary is Alt. 2. Comparing Value to Value Risk,and Cost to Cost RiskThe VMM GuideThe VMM How-To Guide provides best practice analysis techniques, real examples and required resourcesVMM Step 1: Develop a Decision FrameworkV A L U EValue Factorspriorityvalue measure(s)metric, target, scalepriorityR I S KCOSTCustomizedCost ElementStructure1.0 2.03.0 Risk InventoryRisk Tolerance BoundaryTask 4:Begin DocumentationTask 2:Identify & Define the Risk StructureTask 3: Identify & Define the Cost StructureTask 1: Identify & Define the Value StructureDefine User Needs & PrioritiesQuantifiable Measures of Performance (Metrics, Targets)Foundation for Analysis & On-going Performance MeasurementEarly Consideration of RiskR I S Kprioritypriority3.0 2.0V A L U ECOST1.0 metric, target, scalevalue measure(s)Value FactorsCustomizedCost ElementStructureVMM Step 2: Alternatives Analysis (estimate value, cost, & risk)Viable AlternativesBase Case - What will happen if nothing changes?Match levelsof information to the phases of developmentS E N S I T I V I T Y U N C E R T A I N T Y U N C E R T A I N T Y Risk AnalysisS E N S I T I V I T Y Step 1Task 1: Identify & Define AlternativesTask 4: On-going Documentation Task 2:Estimate Value & CostTask 3: Conduct Risk AnalysisRisk InventoryRisk Tolerance BoundaryLow High Expected Low High Expected VMM Step 3: Pull Together the InformationGovernment Cost Savings/AvoidanceExpectedValue ScoreExpected CostRisk Adjusted ExpectedValue and CostRisk ScoresExpected ROIRiskAdjusted Expected ROIS E N S I T I V I T Y U N C E R T A I N T Y U N C E R T A I N T Y S E N S I T I V I T Y Step 1Low High Expected R I S KLow High Expected prioritypriority3.0 2.0V A L U ECOST1.0 metric, target, scalevalue measure(s)Value FactorsCustomizedCost ElementStructureRisk Analysis Step 2Task 5: Compare Value, Risk, & CostTask 1:Aggregate the Cost EstimateTask 2: Calculate the Return-on-InvestmentTask 3: Calculate the Value ScoreTask 4: Calculate the Risk ScoresRisk InventoryRisk Tolerance BoundaryVMM Step 4: Communicate and DocumentReportingConsensus BuildingInvestment Planning Management PlanningExpectedValue ScoreExpected CostRisk Adjusted ExpectedValue and CostRisk ScoresExpected ROIRiskAdjusted Expected ROIGovernment Cost Savings/AvoidanceS E N S I T I V I T Y U N C E R T A I N T Y U N C E R T A I N T Y Risk InventoryS E N S I T I V I T Y Step 1Low High Expected R I S KLow High Expected prioritypriority3.0 2.0V A L U ECOST1.0 metric, target, scalevalue measure(s)Value FactorsCustomizedCost ElementStructureRisk Analysis Step 2 Step 3Task 2:Prepare Budget Justification Documents Task 3:Satisfy Ad Hoc Reporting RequirementsTask 4:Use Lessons Learned to Improve ProcessesTask 1: Communicate Value to Customers and StakeholdersRisk InventoryRisk Tolerance BoundaryQ & AMeasures tangible and intangible benefitsAccounts for risk in cost and value calculationsIncreases reliability of ROI through simulationTested and proven in multiple E-Gov projectsFlexible and adaptableResults and outcome drivenAllows examination of the relationships among Value, Cost and RiskFeasible for portfolio management VMM establishes an even scale for quantifying and analyzing value, risk, and costV A L U EValue Factorspriorityvalue measure(s)metric, target, scalepriorityR I S KCOSTCustomizedCost ElementStructure1.0 2.03.0 Risk InventoryRisk Tolerance BoundaryBuilding a Methodology for Measuring the Value of e-Serviceshttp:/www.estrategy.gov/documents/measuring_finalreport.pdfVMM How-To-Guide and VMM Highlightshttp:/www.cio.gov/ best practices page:-http:/www.cio.gov/documents/ValueMeasuring_Methodology_HowToGuide_Oct_2002.pdf-http:/www.cio.gov/documents/ValueMeasuring_Highlights_Oct_2002.pdf