Reports and Publications

Audit of Pardons Application Decision System Renewal Implementation Project

Table of Contents

• Executive Summary
• Background
• Audit Objectives
• Audit Scope
• Audit Criteria
• Methodology
• Conclusions
• Observations and Recommendations
• Management Response
• Appendix A: PADS-R Audit Interviews List

Executive Summary

Centre for Public Management Inc. (CPM) was engaged by the National Parole Board (NPB) in November 2005 to perform an implementation audit of the Pardons Application Decision System Renewal (PADS-R) to assess whether the following techniques were applied in support of sound project management: project planning, project tracking and oversight, contract and sub-contract management, requirements management, integration management, configuration management, quality assurance, training and change management. The objective of the NPB in requesting the audit was to document “Lessons Learned” regarding the PADS-R implementation which could be carried forward to other development projects, most notably the CRS implementation scheduled to commence in FY 2006/07.

The audit objective was to assess the soundness of project management processes in place on the project, as project management touches all aspects of a project and can be the single most important factor in a project’s success.

Planning & Process

The project commenced in January 2004, while a project charter was drafted in May 2004. A project plan was first created in the summer of 2004, with the arrival of a full time project manager. No business case was prepared for the project. This lack of up front planning contributed to unrealistic timelines and budgets, resulted in the use of incorrect contracting vehicles for the acquisition of the development team (had the full extent of the resources required been known at the outset, a more flexible contracting approach could have been used) and contributed to the many change requests and shifting ‘go-live’ dates.

Governance

The project steering committee only met on two (2) occasions over the life of the project, creating a governance vacuum for the project. In order to fill the vacuum, the effective governance level of the project fell to the Director level, which was too close to the day- to-day operations of the project to be effective in this oversight role. This issue was compounded by confusion relating to the roles of the CIO and the Director of IM at NPB. This governance vacuum persisted until March 2005, when on notification of the deadline slippage of the project the Executive Director became closely involved in the project.

Change Management (People)

Although the PADS-R vision represented a major change in the way pardons were to be processed, very little was done to facilitate buy-in to this vision by the pardons staff prior to the design of the system. In addition, the pardons staff participating in requirements sessions were not aware that these sessions would be their only input into the process. This resulted in significant delays in the definition and approval of user requirements, and a large number of change requests and bug fixes being generated as the system got closer to its ‘go-live’ date.

Approach

The PADS-R project was performed in-house using contracted independent consultants for the development, with NPB taking on the role as integrator. This approach is well-suited to sophisticated software development shops, where there are significant in-house policies, procedures, methodologies and experience in the development of systems. This lack of pre-requisites should have been a clear indicator that NPB did not have the capacity to take on the project itself thus assuming all of the risk of development.

Impact

The net impact of these items was the delivery of the project – over time and over budget. If it had been properly planned at the outset, more realistic budgets and timelines would have been prepared. If a business case had been prepared using realistic cost figures and timelines, it would have prompted key decisions regarding the nature, vision and scope of the project to be formalized prior to commencement of the project. Ultimately, proper planning would have provided the appropriate information on which to make an informed decision and to determine the level of risk to the organization regarding NPB’s capacity to take on such a project as the integrator.

Conclusion

In our opinion the PADS-R project, although achieving a successful go-live on 12 December 2005 as a direct result of the skill and dedication of the project team (both in-house staff and consultants), did not possess sound project management processes over the life cycle of the project, and this was reflected in the overall inefficiency of the project as well as the fact that it was delivered late and over budget. Based upon our work we are able to state this conclusion with high assurance.

Recommendations

We recommend that NPB outsource future large project development work to another government department or an integrator. We further recommend that a business case, project charter and detailed project plan be prepared prior to the commencement of the work, and change management (people) not be neglected when the project involves changes to the way in which staff perform their day-to-day jobs.

Background

The Centre for Public Management Inc. (CPM) was engaged by the National Parole Board (NPB) in November 2005 to perform an implementation audit of the PADS-R to assess whether the following techniques were applied in support of sound project management: project planning, project tracking and oversight, contract and sub-contract management, requirement management, integration management, configuration management, quality assurance, training and change management.

The PADS-R project commenced in January 2004 in response to notification in October of 2003 that the software program on which their existing Pardons application was running would no longer be supported by the vendor as of December 2004. While the application was relatively stable, this deadline provided a sense of urgency in the PADS-R project which was reflected in the way the project was planned and executed. The PADS-R application went live on December 12, 2005 .

The audit was undertaken by experienced staff from the Centre for Public Management, and consisted of interviews and documentation review. Given the objective of the audit, recommendations are drafted with a forward-looking view, and intended to provide practical guidance on undertaking a system development project of the size and scope of PADS-R or larger.

Audit Objectives

The audit had the following objective:

• to focus on the soundness of project management processes to identify lessons learned to ensure that future projects will benefit from the experience gained in this implementation.

Project management processes touch every aspect of a project, and have a large impact on its eventual success. A focus on project management processes enabled the identification of the maximum number of “Lessons Learned” within a limited project budget.

Audit Scope

The scope of the audit consisted of the PADS-R project from inception (January 2004) to post implementation (January 6, 2006). While the audit objective and the scope are focussed on project management processes, any observations made during the course of the audit are raised in this report, regardless of whether they were included in the original scope.

Audit Criteria

We derived our audit criteria from a number of reasonable and credible standards of performance and control relevant to this engagement. The sources of criteria for this audit included the following:

• Project Management Body of Knowledge (PMBOK)– the Project Management Institute’s (PMI) standards document for project management knowledge and practices;
• The Information Systems Audit and Control Association’s (ISACA) Audit Program and Internal Control Questionnaire, which reflected the Control Objectives for Information and related Technology (COBiT) – the IT Governance Institute’s framework for IT practices.

The criteria for the audit objective were as follows:

• Sound project management processes increase the likelihood that the project will succeed by ensuring that an effective management control framework exists.
  • Project planning techniques are sound.
  • Project management processes are in place.
  • The governance structure is appropriate and followed.
  • Project monitoring, tracking, and control techniques are appropriate.
  • Change management (IT) procedures are adequate.

Methodology

The audit began in November 2005, and we carried out our examination work from 9 November 2005 to 6 January 2006. This work involved:

Conducting interviews;

Reviewing available documentation; and

Reviewing best practices against the practices of recognized organizations such as the Project Management Institute’s (PMI) PMBOK and the IT Governance Institute’s COBiT.

As part of the audit, we interviewed fifteen (15) people, including key executives, PADS-R project members, and other key members of the project. We also obtained relevant documentation to validate the information gathered during interviews. See Appendix A for a list of interviewees.

Conclusions

The PADS-R system went live on December 12, 2005 to positive feedback, which is a testament to the hard work by both the members of the project team (in-house and consultants) and the users involved in the project. In order to learn from this experience, it is important to delve below the surface, and look at the factors that influenced the go-live. Based on the results of our audit, it is our opinion that many of the project management processes in place over the life of the project were not effective, and other project management processes were not implemented until the project was well underway. These actions led to an implementation which required more time and more resources than originally planned. The observations which lead to this conclusion, as well as associated recommendations are presented in the following section.

Observations and Recommendations

Project Planning

Background

In the context of project planning, the audit team reviewed the following elements:

• Business Case
• Project Charter
• Project Schedule

These planning items form the cornerstone of the project. The business case requires that the costs and benefits of the project be clearly outlined. This in turn drives the project charter, which will map out the approach to meeting the benefits outlined in the business case and ensure that the scope is clearly defined and consistent. This clearly defined scope then drives the project schedule, which will map out, on a task by task basis, how the objectives of the project will be achieved.

Strong up front planning also drives the planning for project resources. Once the scope, nature and duration of the project are clearly defined, it is easier to determine the proper contracting vehicles which should be used to obtain the required skills for the required duration of the project.

Observations

Planning

With respect to project planning we observed the following:

• There was no business case for the project prepared prior to commencement.
• The project charter was drafted four months after the commencement of the project, and is missing key components including project scope and timelines.
• The project schedule was developed on the arrival of the project manager, again after the commencement of the project.
• The lack of a detailed project schedule up front, coupled with the fact that the project was performed in-house, without an integrator, resulted in no formal methodology being used at the outset of the project.

Impact

These observations had the following impacts:

• There was insufficient planning up-front to determine the cost, resource requirements and timeframe of the project, resulting in the project being completed later than originally planned and at a greater cost. This project was not as significant in its overall dollar amounts as many government IT development projects can be, however, the size of the project and its impact on the NPB organization was significant given that the overall departmental budget for NPB is quite modest.
• The procurement method used for bringing consultants on staff did not provide flexibility for extensions. With the proper planning up front, the true duration of the project would have been known, and a different procurement method would likely have been used. This resulted in an inflexible contracting vehicle which made it difficult to extend consultants for the duration of the project, and introduced administrative overhead and uncertainty surrounding extensions. In many cases extensions were only issued on the last day prior to contract expiry.
• The user requirements phase of the project was performed without adequate HR change management, resulting in delays due to a lack of consensus on the new PADS-R vision within the pardons group.
• Document management functionality was omitted from the scope of the system. A proper front end plan, with appropriate user involvement from a change management perspective, would have likely ensured that this functionality would have been included in the original scope and funding allocations.

Project Processes

Background

For project processes, the team reviewed the project documentation binder for evidence of scope management, risk management, issue management and financial tracking.

Observations

We observed process documentation regarding risk management as well as status reports. However, the lack of an effective project steering committee resulted in this information staying, for the most part, at the project level. The content of such reports was observed to be very operational and did not convey time slippages or fully reveal the risks to NPB as a whole.

The financial tracking maintained as part of the project management materials was not as sophisticated as one would expect for a project of this size. Poor planning and the lack of clear project scheduling resulted in additional funds being required for the project at key points throughout.

Planned vs. actual project performance was not tracked in a rigorous fashion, as the lack of project charter and schedule at the outset of the project resulted in ad hoc reporting in this area.

Impact

These observations had the following impacts:

• The lack of formalized project management processes coupled with the weaknesses noted in the project governance contributed to the project being over time and over budget.

Recommendations for the Future

If NPB decides to undertake a major development effort in the future, we recommend:

• An integrator or project manager be the first resource brought on the project.
• A business case for the project be prepared.
• A complete project charter be prepared.
• A project schedule be prepared using a comprehensive methodology to ensure an integrated approach to the project and reasonable time lines.
• Appropriate project management processes be implemented.
• For all but the smallest projects, an RFP approach should be used to acquire the project team (see “Project Approach” for more details on this point).

Project Governance

Background

Project governance identifies and defines the relationships between various individuals and groups involved in the project to ensure the project is implemented successfully. The governance structure establishes the decision-making authority of these various groups, and ensures accountability through performance measurement. Effective project governance can also provide leverage in gaining consensus on contentious issues which may impact a project’s timing or financing. In the case of PADS-R, the governance structure defined the level of senior management involvement as well as the day-to-day operations of the project.

Observations

We observed the following:

• The governance structure of the project was defined in a governance and accountability document.
• A key aspect of the governance structure was the project steering committee, which contained senior management representation.
• The project steering committee only met twice over the course of the project, creating a governance vacuum.
• With the governance vacuum, effective project governance was at the Director level for the majority of the project, until the Executive Director became involved in the spring of 2005.
• In addition, at the beginning of the project, interviews reported that there was a lack of clarity in roles between the Director of IT and the CIO.

Impact

These items had the following impacts on the project:

• Although project risks and issues had been reported in status reports, they were not raised with senior management until the slippage of the go-live date. At this point, heavy intervention from the Executive Director was required to ensure the project met revised dates. Even once these issues became evident, they were not raised in a project steering committee meeting.
• With the project manager as an independent consultant, the de-facto project manager from the NPB side was the Director of IM. With the governance vacuum that existed for the majority of the project, the Director of IM was also the most senior NPB representative providing governance on the project. This resulted in a circular loop which was only broken when the Executive Director became more involved.

Recommendations for the Future

A significant project of the size and scope of PADS-R should have an effective, functioning governance structure. The project steering committee must meet on a regular basis (monthly is recommended), and there must be a direct roll up of issues, risks and progress discussed at these meetings so changes can be made as appropriate. A project dashboard which captures key performance metrics is a good approach to ensure that key elements are communicated at the appropriate level of detail.

Project Approach

Background

There are a number of different ways in which a project such as PADS-R can be undertaken. An organization may decide to perform the project in-house when it has the depth of methodologies, processes, procedures, experience and capabilities to do so, supplementing any missing resources with contractors. The second approach is to perform the project using an integrator. The integrator brings with him the staff, the project management and the methodological support to perform the project, and is accountable for its success. The level of risk then becomes assumed by the integrator rather than the department should the project fail altogether or run into severe time/financial constraints.

While both approaches have pros and cons, the first approach is better suited to a sophisticated IT development shop, while the second would be used by an organization that does not have the appropriate experience or expertise.

Observations

We observed the following:

• Although NPB did not have prior systems development experience of the size or scope of the PADS-R project, it elected to do the project “In-house”. While almost the entire development team and the project manager were external consultants, they were procured as individuals to bolster NPB’s development capabilities. This resulted in a lack of methodological support for the effort.
• Our interviews revealed that there was a general feeling that the consultants were a stand alone group, not integrated with NPB. This was further reinforced by the fact that they were located in a separate physical location and did not have access to the NPB Intranet or email system. While this would be expected in a project performed by an integrator, it is unusual for an in-house developed project. Although there was no integrator with ultimate accountability for the success of the project, NPB staff seemed to be under the impression that this accountability did exist, and rested with the project manager, who was an external consultant.
• There was a general perception that the contracting vehicle that was used saved time at the beginning of the project, by eliminating the need for an RFP.

Impact

The potential impacts of the project approach taken by NPB were mitigated by the high quality consulting resources that were retained. The resources demonstrated a sense of commitment to the project that exceeded what one would expect from a group of independent consultants. However, there still were a number of areas where this approach affected the outcome:

• The lack of physical and technical integration between the project team and the remainder of the NPB staff created logistical challenges and inefficiencies.
• Logistical challenges included the lack of test/QA environments. This necessitated work arounds and proxy testing, which is an inefficient approach.
• The lack of a formal methodology and development policies and procedures required an ad-hoc approach to the project.
• The ad-hoc approach resulted in some areas being underserved (see the section “Change Management”), and others being planned and executed at the last minute.
• This approach resulted in the initial timelines and resource requirements to be underestimated, with no accountability other than NPB itself.
• While the contracting approach eliminated the need for an RFP, the work that is required for an RFP is the same work that should have been performed prior to embarking on the project. The RFP exercise forces a certain amount of discipline into the planning process, and the fact that the consultants were procured individually contributed to the lack of planning discipline in the project.

Recommendations for the Future

Given the size of NPB and its IT development needs in the future (with PADS-R completed, CRS represents the only development project likely to be undertaken in the foreseeable future), it is unlikely that a business case could be built to support the creation of an in-house systems development organization. For this reason, we recommend that NPB outsource the development of any future systems, either to another government department such as CSC or to an integrator. This will provide greater accountability for deliverables, promote the use of a standard methodology, reduce the risk to the organization itself and enforce a more rigorous governance structure.

Change Management (People)

Background

In the context of this observation, Change Management refers to the management of change in the organization, not the management of change requests on the project. Change management is an important element of any large IT project, as they normally result in changes to the way people go about performing their day-to-day tasks. At a high level the change management process involves assessing the organization’s ability to accept the change, determining the true impact on the organization of the change, and then designing appropriate tools to help it adopt the change. The vision for the PADS-R project represented a major change to the way pardons are processed, and this project would have been a prime candidate for change management.

Observations

We made the following observations:

• Despite the fact that the vision for PADS-R represented a major change for the organization, there were no explicit change management initiatives deployed to bring the users “on-side” with the changes. The user requirements sessions became change management sessions, which slowed down the requirements process for the development team.
• A training plan was in place for the users of the system.
• The pardons staff was not aware of the expectations coming out of the user requirements sessions. They believed that they were specifying high level requirements, when in reality the sessions were meant to capture detailed user requirements.
• Poor communication was identified as an issue between the development team and the remainder of the project team. This was exacerbated by the segregation of the development team from the rest of the organization, both physically and electronically.

Impact

The impact of these items on the project was most evident in the project slippage due to delays in sign off of user requirements. The training plan, while an important component of change management (people), is implemented after the system has been designed and as such is only one aspect of a change management plan. The impact of the process change that was proposed was not assessed prior to commencement of the project, and caused resistance which was felt throughout the requirements. The lack of expectations management in the definition of user requirements caused delays in the sign off, and a large number of program change requests to be generated to add functionality after the fact.

Recommendation for the Future

A project of the size of PADS-R should include change management methodologies integrated with the development methodology. The project team should include, at a minimum on a part-time basis, a change management resource to assist the organization in implementing the changes brought about by the implementation.

Management Response

The NPB Evaluation and Audit Committee accepts the conclusions and the recommendations of the audit report on PADS-R prepared by the Centre for Public Management Inc. However, the Committee considers that the context leading to the urgent development of PADS-R should be explained in more detail in the report as it clarifies the reasons for the approach taken by the Board with the project.

In the fall of 2003 (Sept/Oct) the National Parole Board was informed that the software Filenet behind PADS would not be supported past December 31st 2004 . Additionally, RCMP advised the NPB that the existing CPIC connection would no longer work after March 2005. Therefore, the PADS-R project had to be commenced immediately. Given the urgency of the situation, the governance documents, and plans, etc, were finalized after project work started. The urgency of the start-up meant actions such as an RFP were not possible and other contracting vehicles were required immediately. Additionally TBS funded PADS-R (i.e. 50% up front) without a detailed Business Case, as it agreed with the urgency of the situation.

Although the NPB did not take the usual steps in the development of PADS-R, the project is a success due to the skill and dedication of the NPB staff and consultants that were hired. PADS-R users are very pleased with the new application that is much more efficient and user-friendly than the previous one.

Any project is an opportunity for learning. This audit assisted the NPB in achieving the most from its experience with PADS-R. When planning large projects in the future, the NPB will be guided strongly by the recommendations of this audit report. It will, in particular, choose a management approach that reflects a careful assessment of the complexity of the project, of the extent of the change management challenge, and of our internal capacity to lead and pursue the project successfully.

Appendix A: PADS-R Audit Interviews List

Mike Marshall, Consultant

Peter Blackmore, Consultant

Yves Bellefeuille, Director C&P

Marc Seguin, Director IMSD

George Sieniecky, Manager IM

Sean Moylan, Senior Programmer Analyst

Greg Edwards, Manager CTSS

Terry Rempel-Mroz, Manager AMS

Katherine Galliger-Spicer, Senior Systems Analyst

Collette Galipeau, Pardons Manager

Mary Rounopolous, Policy Analyst

Jean Yves Mailloux, Consultant, former Project Manager

Pat Liston, Former CIO

Denis Stevens, Executive Director

Pierre Couturier, Director, Performance Measurement