W04_SJP_Development of a Scoring Model for the GAO Ten Best Practices

Issue Identification and Appraisal

In my W03 Blog, it was determined that a ‘Weighted Scoring Method’ was the preferred method evaluate a schedule. As a follow-up, this week’s problem statement is, “How to develop a scoring model based on the GAO’s Ten Best Practices.”

Feasible Alternatives

The scoring model that requires to be developed is quite complex as it needs to weight the criteria, and provide quantitative limits to assess if the result as a pass/fail. i.e. for example Best Practice 1 ‘capturing all activities’ what determines the pass criteria to say the schedule met the GAO requirements 100%, 90%…thoughts anybody?  Initially, the problem to solve is the how to achieve the vertical scoring determined, and look at horizontal scoring thereafter. There are two suitable alternatives; i) a system that treats all the criteria as being equal, or ii) a system that has imposed rankings against the criteria.

Develop the Outcomes for each

Based on my early weeks of research on the topic there does not appear to be a ‘one rule suits all’ method. As mentioned above the system will eventually be complex in nature. The benefits for the two alternatives are outlined in table 1 below.

Table 1: Benefits of each Alternative

Selection Criteria

The GAO Schedule Assessment Guide provides several criteria sets;

  • Individual checklists for the Ten Best Practices – these can be found on the following pages of the document; 25/26, 46/47, 61/62, 68/69, 73/74, 89, 96/97, 119/120, 132/133, 146/147.
  • Appendix II: An auditor’s key questions (pages 151 to 165).
  • Appendix VI: Standard Quantitive measures for Assessing Schedule Health (pages 183 to 188).

All are excellent criteria sets, however data sets range between 74 to 152 points depending on the set chosen. The GAO specifically state, “…they are not intended as a series of steps for developing the schedule”, “…questions are related to the general policies in place and procedures undertaken to create and maintain the schedule” and “No “pass-or-fail” thresholds or tripwires are associated with the measures…. Moreover, severity of the errors or anomalies takes precedence over quantity because any error can potentially affect the reliability of the entire schedule network”.

As the scoring model is ultimately going to be used to evaluate schedule quality whichever set of GAO criteria is going to be used will need to be reviewed in detail and acceptance criteria developed. However, for the purposes of the Blog a comparison of the results from all three data sets will be provided.

Analysis and Comparison of the Alternatives

Comparing the alternatives using the information from the benefits table provided the assessment of which method to be used. Please note that the lower two items (scoring flexibility / data manipulation potential) did not relate to A1 and scored zero (Refer to table 2 below) but results of A2 were brought closer to A1 due to this.

 

Table 2 Analysis results of Alternatives

The next step in the process is to input all three data sets to compare the results Best Practice Checklists (152 points), Appendix II (87 points) and Appendix VI (74 points). Table 3 below shows the ranges that can be determined for each Best Practice depending on the data used.

Table 3 Best Practice weighting ranges

The next step would be to determine the acceptance criteria for each Best Practice, however due to the large dispersion of ranges a further analysis will be required to develop one set of criteria to be able give credibility to the weightings.

Selection of Preferred Alternative

The preferred solution would be to implement a scoring system that weights all criteria equally.

Monitoring Post Evaluation Performance

The preferred alternative has been determined and the system will provide results. However due to the wide range of weightings determined using the three different data sets there is a need to re-assess the input criteria. Blog 5 will further the research, and develop the input criteria and the acceptance criteria to finalize the scoring method.

References

  • GAO (United States Governance Accountability Office), 2015, GAO-16-89G Schedule Assessment Guide
  • Guild of Project Controls. (2015, October). GUILD OF PROJECT CONTROLS COMPENDIUM and REFERENCE (CaR) | Project Controls – planning, scheduling, cost management and forensic analysis (Planning Planet). Retrieved June 14, 2017, from http://www.planningplanet.com/guild/gpccar/managing-change-the-owners-perspective
  • Ishizaka, A., Nemery, P., & John Wiley & Sons. (2013). Multi-criteria decision analysis: Methods and software[Kindle].
  • W03_SJP_Scoring model to analyze a Baseline Schedule vs GAO requirements – Achieving Guild of Project Controls / AACE Certification BLOG [Web log post]. (n.d.). Retrieved from https://js-pag-cert-2017.com/w03_sjp_scoring-model-analyze-baseline-schedule-vs-gao-requirements/
 

2 Replies to “W04_SJP_Development of a Scoring Model for the GAO Ten Best Practices

  1. OUTSTANDING, Steve….!!! Wow, I am really impressed that you have mastered this approach so very well!!!

    PLUS you are doing exactly what I hoping you would do, which is to “work smart” by using your blog postings to help you develop your paper…. PLUS PLUS you are building in a “risk buffer” by getting a week or two ahead in your postings.

    Not much more I can say other than keep up the good work and I really hope that you can use your “leadership by example” to help Juan catch up….

    IF you wish, you may want to use your W5 blog to test drive your scoring model on a single Attribute? Pick “Comprehensive” and see what it looks like when you score Best Practices #1, #3 and #4? IF it works the way you hope it will, then you can expand it to the remaining 3 attributes and your first paper will be pretty much done….

    Bottom line- doing great and if you can keep up this pace for the next 5 months you are going to be in really great shape.

    BR,
    Dr. PDG, Boston MA

     
  2. PS Keep in mind that although the GAO stated that these are not intended as a step by step guide, for most normal instances, you can surely use it that way……

    When you write your paper, you may also want to cite the process maps shown in the GPCCaR Modules 7, 8 and 9? While those process maps are not cast in concrete, the step by step process is applicable to “most projects, most of the time” unless there is something new or unique that requires a different sequence…..

    BR,
    Dr. PDG, Boston MA

     

Leave a Reply

Your email address will not be published. Required fields are marked *