Issue Identification and Appraisal
In my W03 Blog, it was determined that a ‘Weighted Scoring Method’ was the preferred method evaluate a schedule. As a follow-up, this week’s problem statement is, “How to develop a scoring model based on the GAO’s Ten Best Practices.”
The scoring model that requires to be developed is quite complex as it needs to weight the criteria, and provide quantitative limits to assess if the result as a pass/fail. i.e. for example Best Practice 1 ‘capturing all activities’ what determines the pass criteria to say the schedule met the GAO requirements 100%, 90%…thoughts anybody? Initially, the problem to solve is the how to achieve the vertical scoring determined, and look at horizontal scoring thereafter. There are two suitable alternatives; i) a system that treats all the criteria as being equal, or ii) a system that has imposed rankings against the criteria.
Develop the Outcomes for each
Based on my early weeks of research on the topic there does not appear to be a ‘one rule suits all’ method. As mentioned above the system will eventually be complex in nature. The benefits for the two alternatives are outlined in table 1 below.
Table 1: Benefits of each Alternative
The GAO Schedule Assessment Guide provides several criteria sets;
- Individual checklists for the Ten Best Practices – these can be found on the following pages of the document; 25/26, 46/47, 61/62, 68/69, 73/74, 89, 96/97, 119/120, 132/133, 146/147.
- Appendix II: An auditor’s key questions (pages 151 to 165).
- Appendix VI: Standard Quantitive measures for Assessing Schedule Health (pages 183 to 188).
All are excellent criteria sets, however data sets range between 74 to 152 points depending on the set chosen. The GAO specifically state, “…they are not intended as a series of steps for developing the schedule”, “…questions are related to the general policies in place and procedures undertaken to create and maintain the schedule” and “No “pass-or-fail” thresholds or tripwires are associated with the measures…. Moreover, severity of the errors or anomalies takes precedence over quantity because any error can potentially affect the reliability of the entire schedule network”.
As the scoring model is ultimately going to be used to evaluate schedule quality whichever set of GAO criteria is going to be used will need to be reviewed in detail and acceptance criteria developed. However, for the purposes of the Blog a comparison of the results from all three data sets will be provided.
Analysis and Comparison of the Alternatives
Comparing the alternatives using the information from the benefits table provided the assessment of which method to be used. Please note that the lower two items (scoring flexibility / data manipulation potential) did not relate to A1 and scored zero (Refer to table 2 below) but results of A2 were brought closer to A1 due to this.
Table 2 Analysis results of Alternatives
The next step in the process is to input all three data sets to compare the results Best Practice Checklists (152 points), Appendix II (87 points) and Appendix VI (74 points). Table 3 below shows the ranges that can be determined for each Best Practice depending on the data used.
Table 3 Best Practice weighting ranges
The next step would be to determine the acceptance criteria for each Best Practice, however due to the large dispersion of ranges a further analysis will be required to develop one set of criteria to be able give credibility to the weightings.
Selection of Preferred Alternative
The preferred solution would be to implement a scoring system that weights all criteria equally.
Monitoring Post Evaluation Performance
The preferred alternative has been determined and the system will provide results. However due to the wide range of weightings determined using the three different data sets there is a need to re-assess the input criteria. Blog 5 will further the research, and develop the input criteria and the acceptance criteria to finalize the scoring method.
- GAO (United States Governance Accountability Office), 2015, GAO-16-89G Schedule Assessment Guide
- Guild of Project Controls. (2015, October). GUILD OF PROJECT CONTROLS COMPENDIUM and REFERENCE (CaR) | Project Controls – planning, scheduling, cost management and forensic analysis (Planning Planet). Retrieved June 14, 2017, from http://www.planningplanet.com/guild/gpccar/managing-change-the-owners-perspective
- Ishizaka, A., Nemery, P., & John Wiley & Sons. (2013). Multi-criteria decision analysis: Methods and software[Kindle].
- W03_SJP_Scoring model to analyze a Baseline Schedule vs GAO requirements – Achieving Guild of Project Controls / AACE Certification BLOG [Web log post]. (n.d.). Retrieved from https://js-pag-cert-2017.com/w03_sjp_scoring-model-analyze-baseline-schedule-vs-gao-requirements/