IRS Should Address Modernization Performance Problems, TIGTA Says

In summary, the IRS is currently engaged in the BSM program to modernize its systems and associated processes. All of the BSM projects initiated to this point have experienced cost overages and schedule delays. In mid-2003, the IRS and the PRIME contractor¹ initiated four studies to help identify the root causes of the problems hindering the BSM effort and make recommendations to remedy the problems identified.

Key IRS executives and stakeholders reviewed the results of the four studies, quickly acknowledged BSM shortcomings, and developed actions to address the studies' recommendations and resolve longstanding BSM issues. Collectively, these actions became known as the BSM Challenges Plan.

However, we determined the studies' recommendations were not fully addressed by the BSM Challenges Plan; no measurement plan was created to determine if the actions taken resulted, or will result, in actual improvements in the BSM program; and many BSM Challenges Plan actions were closed before all significant activities were completed.

Upon presenting our preliminary results to the IRS, we were informed the Associate Chief Information Officer (ACIO), Modernization Management, met with and interviewed a number of individuals in the IRS business units, the Treasury Inspector General for Tax Administration (TIGTA), the Government Accountability Office (GAO), and the Office of Management and Budget. The ACIO also reviewed the four studies, as well as various reports prepared by the TIGTA and the GAO. As a result, the ACIO determined what he believed to be the key barriers to success in the BSM program and created a plan to address these barriers. This plan includes 7 high-level key focus areas and 16 detailed highest priority initiatives to deal with the root causes that are creating the barriers.

The ACIO, Modernization Management, and his staff also performed an analysis of the BSM Challenges Plan and the activities taken to close each of its 48 actions. We were provided with documentation showing several BSM Challenges Plan actions have been reopened and others have additional follow-up activities planned; however, IRS officials informed us these new and additional follow-up activities were not being tracked using the Item Tracking, Reporting, and Control (ITRAC)² database.

To assist the IRS in improving the BSM program, we recommended the Chief Information Officer (CIO) reevaluate the study recommendations that were not addressed by the BSM Challenges Plan to determine whether corrective actions should be taken, create an overall measurement plan to determine whether activities are leading to improvements in the BSM program, ensure stakeholders understand additional work is necessary to complete actions in the BSM Challenges Plan beyond what was originally reported, and place new activities in the ITRAC database for tracking and closure purposes.

Management's Response: The CIO agreed with our recommendations and stated there was a lot of work ahead to continue to mature the BSM program. However, the CIO pointed to several improvements and accomplishments reflecting improved program performance. For instance, the CIO stated cost and schedule performance had dramatically improved in 2004 and strides were being made in the hiring of outside experts and aligning BSM workload with IRS and PRIME contractor management capacity.

The CIO also responded to our specific recommendations. The new ACIO, BSM, has thoroughly reviewed the study recommendations and is actively working many of them. Justifications have been documented for those study recommendations not being pursued. In addition, the BSM program will be developing measures similar to those outlined in our report. Also, presentations have been made to the stakeholder community on the status of improvement plans, and the ITRAC database has been updated. Management's complete response to the draft report is included as Appendix IX.

Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

² The BSM Risk Management Plan provides that all modernization organizations use the ITRAC database to record and update the status of issues.

The Internal Revenue Service (IRS) is currently engaged in an effort, known as Business Systems Modernization (BSM), to modernize its systems and associated processes. To facilitate the success of its modernization efforts, the IRS hired the Computer Sciences Corporation as the PRIME contractor and integrator for the BSM program and created the BSM Office (BSMO) to guide and oversee the work of the PRIME contractor.

Since its inception, the BSM program has experienced management problems at both the program and project levels, as well as problems between the IRS and the PRIME contractor. All of the BSM projects initiated to this point have experienced cost overages and schedule delays.

For example, by May 2003, the Customer Account Data Engine (CADE)¹ project had missed its deployment date for the third time and was over 2 years behind schedule. By the summer of 2003, another high profile project, the Integrated Financial System (IFS),² was also experiencing significant cost increases and the project team was concerned about potential schedule delays.³

In mid-2003, the IRS and the PRIME contractor initiated the following four studies to help identify the root causes of the problems hindering the BSM effort and make recommendations on remedying the problems identified.

• CADE Study - The Software Engineering Institute (SEI)⁴ conducted a study of the CADE project.

• PRIME Contractor Internal Study - The Bain and Company conducted an internal study of the PRIME contractor.

• IRS Root Cause Study - The BSMO conducted an internal study of the BSM program.

• High-Level Assessment of the IRS Office of Procurement Study - The Acquisition Solutions, Incorporated, conducted a study of the IRS' acquisition program.⁵

The IRS selected the Deputy Commissioner, Tax Exempt and Government Entities (TE/GE) Division, to oversee the implementation of the actions, beginning on November 10, 2003. On May 1, 2004, the Deputy Commissioner, TE/GE Division, issued a close-out report detailing the activities taken to close 44 of the 48 actions plus planned activities and projected closing dates for the remaining 4 actions. Further oversight responsibilities were then assumed by the Associate Chief Information Officer (ACIO), Modernization Management.

This review was performed at the BSMO facilities in New Carrollton, Maryland, during the period May through September 2004. The audit was conducted in accordance with Government Auditing Standards. The scope of this audit did not include verifying the actions taken as part of the BSM Challenges P' acquisition program.⁵

Key IRS executives and stakeholders reviewed the results of the 4 studies in October 2003 and created 46 actions to address the study recommendations. These 46 actions collectively became known as the BSM Challenges Plan. In December 2003, the IRS Oversight Board issued a special report⁶ containing 9 recommendations, 2 of which were added to the 46 actions already identified. The 48 actions were each assigned an implementation deadline of 30, 90, or 180 days and were entered into the Item Tracking, Reporting, and Control (ITRAC)⁷ database in order for the IRS to monitor activity on each action.

The IRS selected the Deputy Commissioner, Tax Exempt and Government Entities (TE/GE) Division, to oversee the implementation of the actions, beginning on November 10, 2003. On May 1, 2004, the Deputy Commissioner, TE/GE Division, issued a close-out report detailing the activities taken to close 44 of the 48 actions plus planned activities and projected closing dates for the remaining 4 actions. Further oversight responsibilities were then assumed by the Associate Chief Information Officer (ACIO), Modernization Management.

This review was performed at the BSMO facilities in New Carrollton, Maryland, during the period May through September 2004. The audit was conducted in accordance with Government Auditing Standards. The scope of this audit did not include verifying the actions taken as part of the BSM Challenges Plan were completed or ensuring completed actions achieved the intended results. Detailed information on our audit objectives, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.

This audit was conducted while changes were being made at the BSM program level. We communicated the results of our fieldwork to the BSMO in August 2004. Any changes that have occurred since we concluded our analyses are not reflected in this report. As a result, this report may not reflect the most current status.

The Business Systems Modernization Challenges Plan Did Not Address All Study Recommendations

IRS executives and key stakeholders quickly acknowledged BSM shortcomings and developed actions, as part of the BSM Challenges Plan, to resolve longstanding BSM issues. However, we determined that the study recommendations were not fully addressed by the BSM Challenges Plan. In addition, some of the activities undertaken as part of the BSM Challenges Plan did not fully address the actions in the Plan.

Some study recommendations were not addressed and others were only partially addressed by the BSM Challenges Plan

The results of the four studies were presented to key IRS executives and stakeholders on October 9, 2003. We reviewed the 4 studies and identified 21 BSM-related recommendations. We matched the 48 actions in the BSM Challenges Plan to the 21 recommendations and determined 9 of the recommendations were adequately addressed by the BSM Challenges Plan, 9 were partially addressed, and 3 were not addressed at all, as shown in Chart 1. See Appendix IV for a list of these study recommendations.

 

Chart 1: Study Recommendations

 

Addressed by the BSM Challenges Plan

 

 

 

For example, the IRS root cause study recommended eliminating funding emergencies by timely requesting spend plan changes. We could find no mention of this action in either the BSM Challenges Plan or the ITRAC database. As another example, the PRIME contractor internal study recommended enforcing compliance with requirement definition and program management processes. Part of this recommendation included increasing formal Enterprise Life Cycle (ELC)⁸ training for the IRS and the PRIME contractor. While we could find documented activities for several parts of the overall recommendation, we could not find any documented activities requiring ELC training for either the IRS or the PRIME contractor.

When we presented our preliminary observations to IRS officials, we were informed the BSM Challenges Plan was done in a somewhat random manner, meaning key IRS executives and stakeholders did not take each study recommendation and generate an action item to address it. In addition, the ACIO, Modernization Management, later informed us he believed the IRS did not address all recommendations because certain recommendations were deemed lower priority and actions were already underway to address the intent of other study recommendations. Thus, some study recommendations did not get addressed as part of the BSM Challenges Plan. Without addressing or creating a plan for addressing all conditions within the studies, the IRS runs the risk of having past problems continue to plague the BSM program.

Additional activities are needed to fully address BSM Challenges Plan actions

We reviewed the activities taken, or scheduled, to close the 48 actions listed in the BSM Challenges Plan Close-Out Report and identified 3 instances where the completion activities did not fully address the action. The BSM Challenges Plan action title and the gaps we identified follow.

• Ensure projects strictly follow the ELC and appropriate alternatives within the ELC are selected. While we identified activities taken to ensure appropriate alternatives within the ELC are selected, we could not find any activities to ensure that projects strictly follow the ELC.

• Designate business architects for all projects to support business insight throughout the project life. Instead of designating business architects for each project, the IRS designated business requirements directors for each project. For example, we determined the CADE business requirements director had no systems architecture experience. During discussions on a preliminary version of this report, BSMO officials stated it was never their intention to elicit systems architecture assistance from the IRS business units. Therefore, the IRS decided to perform activities under this BSM Challenges Plan action that would better align with the IRS modernization model.

• Strengthen the development environment by expanding capacity, greater automating testing [sic], separating IF [infrastructure] development. This is one of the four actions that remained open when the BSM Challenges Plan Close- Out Report was issued. Our review of the activities scheduled to close this action identified no activities for automating testing or separating the infrastructure environment.

Recommendation

To ensure all of the conditions in the studies are addressed, the Chief Information Officer (CIO) should:

The Business Systems Modernization Challenges Plan Did Not Include Measures to Determine Whether Actions Taken Were Successful

Management best practices indicate measures should be developed to determine whether actions taken actually result in intended improvements. While the BSM Challenges Plan included reasonable actions, no measurement plan was created to determine whether the actions taken resulted, or will result, in actual improvements in the BSM program. In addition, many actions were closed before all significant activities were completed.

The BSM Challenges Plan contained no criteria for measuring the effect of actions

The BSM Challenges Plan included several management controls such as details of specific activities that needed to be completed, assignment of responsibility for completing specific activities, and planned completion dates. However, the BSM Challenges Plan Close-Out Report, generally, did not include plans for determining whether the implemented actions were, or would be, effective in meeting their designated goal.

Management expert W. Edward Deming gained widespread attention in the 1980's after being featured in a National Broadcasting Corporation documentary titled "If Japan Can, Why Can't We?" Deming expanded the idea that quality can be measured and developed what is known as the Deming Cycle of Plan-Do-Check-Act (PDCA). The PDCA cycle is a well-known model for continuous process improvement that includes, 1) Plan: Design or revise business processes to improve results, 2) Do: Implement the plan and measure its performance, 3) Check: Assess the measurements and report the results to decision makers, and 4) Act: Decide on and implement changes needed to improve the process.

 

Chart 2: Plan-Do-Check-Act Visual Representation

 

 

 

The IRS planned the actions that need to be taken for improvement; however, it did not make plans to measure the success of its efforts. If measures are not employed to determine the effectiveness of BSM Challenges Plan actions, the IRS will only be able to subjectively determine whether the BSM Challenges Plan activities were worthwhile. In addition, the IRS will have no insight into what activities were successful as it moves forward with future improvement efforts to address the root causes of BSM program challenges.

Our analysis of the BSM Challenges Plan found there were no overall criteria for determining whether taking the actions set forth in the Plan will lead to any improvement, and many activities in the Plan did not lend themselves easily to the use of quantitative measurement. In addition, most action items were mapped to multiple study recommendations, which made it difficult to understand what study conditions were being addressed. Without the condition to be corrected clearly in mind, it is not feasible to identify suitable measurement activities.

Based on the interviews we conducted with individuals associated with the formulation and implementation of the BSM Challenges Plan, we determined the IRS had not analyzed the BSM Challenges Plan actions to determine what metrics would be used to measure the effectiveness of the actions. We were advised that the success of BSM Challenges Plan actions could be determined by assessing whether future project cost and schedule estimates were met.

We concluded more detailed measures could also be used to determine which of the multiple challenges within the BSM program were being adequately addressed and which challenges still needed further work. To assist the IRS with identifying measures for future activities, we have created potential measures that could have been used to measure the short-term and long-term success of BSM Challenges Plan actions. See Appendix VII for a list of these suggested measurements.

Actions taken under the BSM Challenges Plan were closed prematurely

In addition to not providing measures to determine the overall success of BSM Challenges Plan actions, we determined several actions within the BSM Challenges Plan were closed before significant activities were completed. The BSM Challenges Plan Close-Out Report listed 48 actions, 44 of which were classified as 'closed' and 4 as 'open.'⁹ This status was reported to the Congress and the press. We determined 10 of the 'closed' actions still had significant work activities to be completed. Seven of these 10 actions concerned staffing issues (funding, hiring, locating individuals) and 3 concerned completing documentation. See Appendix VIII for the listing of these 10 actions.

Management Actions: Upon presenting our preliminary results to the ACIO, Modernization Management, we were informed he and his staff had also performed an analysis of the BSM Challenges Plan and the activities taken to close each of the 48 actions. We were provided with documentation showing, as of August 10, 2004, some BSM Challenges Plan actions were reopened and others had additional follow-up activities planned (see Chart 3). However, IRS officials informed us these new and additional follow-up activities were not being tracked using the ITRAC database.

During preliminary discussions of this report, we were also informed the IRS had taken actions after our fieldwork was complete to inform the OMB and the GAO of the remaining work that needed to be performed on the BSM Challenges Plan.

 

Chart 3: Comparison of BSM Challenges Plan Actions

 

on May 1, 2004, and August 10, 2004

 

 

 

We agree with the IRS' action to reopen and add new activities to ensure BSM Challenges Plan actions are truly complete. Based on this new analysis, 8 of the 10 actions we determined were closed prematurely had new or follow-up activities concerning the areas we identified as needing to be addressed before the actions could be closed. For the two remaining actions, we provided information to the ACIO, Modernization Management, for his consideration. See Appendix VIII for further information on these 10 actions.

Recommendations

To ensure the IRS can objectively determine whether process improvement actions are leading to improvement in the BSM program, stakeholders are updated on the BSM Challenges Plan status, and new BSM Challenges Plan activities are monitored and tracked to closure, the CIO should:

² The IFS is intended to address administrative financial management weaknesses.

³ The IFS project team eventually experienced a 1-year delay from the original deployment date.

⁴ The SEI is a Federally Funded Research and Development Center whose purpose is to help others make measured improvements in their software engineering capabilities.

⁵ While the main focus of the study was not on the BSM program, BSM acquisition weaknesses were addressed.

⁶Independent Analysis of IRS Business Systems Modernization (dated December 2003).

⁷ The BSM Risk Management Plan provides that all modernization organizations use the ITRAC database to record and update the status of issues.

⁸ BSM processes and procedures are documented within the ELC. See Appendix V for an overview of the ELC.

⁹ The 44 closed actions include 5 actions that were merged.

To accomplish these objectives, we:

Study Recommendations Not Addressed by the BSM Challenges Plan

² To facilitate success of its modernization efforts, the Internal Revenue Service hired the CSC as the PRIME contractor and integrator for the BSM program and created the BSM Office to guide and oversee the work of the PRIME contractor.

³ "Tiger Teams" are short duration teams that are intended to establish a forum for escalating issues, facilitate the rapid escalation of issues, and gain commitments from managers to address escalated issues quickly.

⁴ The CADE is the foundation for managing taxpayer accounts in the IRS' modernization plan.

⁵ The current processing environment refers to the information technology currently used to process tax information.

⁶ Defense Acquisition Workforce Improvement Act, Pub. L. No. 101-510.

The ELC framework is a flexible and adaptable structure within which one plans, executes, and integrates business change. The ELC process layer was created principally from the Computer Sciences Corporation's Catalyst® methodology.² It is intended to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision making and information sharing. Other components and extensions were added as needed to meet the specific needs of the IRS BSM program.

ELC Processes

A process is an ordered, interdependent set of activities established to accomplish a specific purpose. Processes help to define what work needs to be performed. The ELC methodology includes two major groups of processes:

Enterprise Life-Cycle Processes

 

 

 

Life-Cycle Processes

The life-cycle processes of the ELC are divided into six phases, as described below:

• Vision and Strategy - This phase establishes the overall direction and priorities for business change for the enterprise. It also identifies and prioritizes the business or system areas for further analysis.

• Architecture - This phase establishes the concept/vision, requirements, and design for a particular business area or target system. It also defines the releases for the business area or system.

• Development - This phase includes the analysis, design, acquisition, modification, construction, and testing of the components of a business solution. This phase also includes routine planned maintenance of applications.

• Integration - This phase includes the integration, testing, piloting, and acceptance of a release. In this phase, the integration team brings together individual work packages of solution components developed or acquired separately during the Development phase. Application and technical infrastructure components are tested to determine whether they interact properly. If appropriate, the team conducts a pilot to ensure all elements of the business solution work together.

• Deployment - This phase includes preparation of a release for deployment and actual deployment of the release to the deployment sites. During this phase, the deployment team puts the solution release into operation at target sites.

• Operations and Support - This phase addresses the ongoing operations and support of the system. It begins after the business processes and system(s) have been installed and have begun performing business functions. It encompasses all of the operations and support processes necessary to deliver the services associated with managing all or part of a computing environment.

• The Operations and Support phase includes the scheduled activities, such as planned maintenance, systems backup, and production output, as well as the nonscheduled activities, such as problem resolution and service request delivery, including emergency unplanned maintenance of applications. It also includes the support processes required to keep the system up and running at the contractually specified level.

Besides the life-cycle processes, the ELC also addresses the various management areas at the process level. The management areas include:

• IRS Governance and Investment Decision Management - This area is responsible for managing the overall direction of the IRS, determining where to invest, and managing the investments over time.

• Program Management and Project Management - This area is responsible for organizing, planning, directing, and controlling the activities within the program and its subordinate projects to achieve the objectives of the program and deliver the expected business results.

• Architectural Engineering/Development Coordination - This area is responsible for managing the technical aspects of coordination across projects and disciplines, such as managing interfaces, controlling architectural changes, ensuring architectural compliance, maintaining standards, and resolving issues.

• Management Support Processes - This area includes common management processes, such as quality management and configuration management that operate across multiple levels of management.

The ELC establishes a set of repeatable processes and a system of milestones, checkpoints, and reviews that reduce the risks of systems development, accelerate the delivery of business solutions, and ensure alignment with the overall business strategy. The ELC defines a series of milestones in the life-cycle processes. Milestones provide for "go/no-go" decision points in the project and are sometimes associated with funding approval to proceed. They occur at natural breaks in the process where there is new information regarding costs, benefits, and risks and where executive authority is necessary for next phase expenditures.

There are five milestones during the project life cycle:

• Milestone 1 - Business Vision and Case for Action. In the activities leading up to Milestone 1, executive leadership identifies the direction and priorities for IRS business change. These guide which business areas and systems development projects are funded for further analysis. The primary decision at Milestone 1 is to select BSM projects based on both the enterprise-level Vision and Strategy and the Enterprise Architecture.

• Milestone 2 - Business Systems Concept and Preliminary Business Case. The activities leading up to Milestone 2 establish the project concept, including requirements and design elements, as a solution for a specific business area or business system. A preliminary business case is also produced. The primary decision at Milestone 2 is to approve the solution/system concept and associated plans for a modernization initiative and to authorize funding for that solution.

• Milestone 3 - Business Systems Design and Baseline Business Case. In the activities leading up to Milestone 3, the major components of the business solution are analyzed and designed. A baseline business case is also produced. The primary decision at Milestone 3 is to accept the logical system design and associated plans and to authorize funding for development, test, and (if chosen) pilot of that solution.

• Milestone 4 - Business Systems Development and Enterprise Deployment Decision. In the activities leading up to Milestone 4, the business solution is built. The system is integrated with other business systems and tested, piloted (usually), and prepared for deployment. The primary decision at Milestone 4 is to authorize the release for enterprise-wide deployment and commit the necessary resources.

• Milestone 5 - Business Systems Deployment and Postdeployment Evaluation. In the activities leading up to Milestone 5, the business solution is fully deployed, including delivery of training on use and maintenance. The primary decision at Milestone 5 is to authorize the release of performance-based compensation based on actual, measured performance of the business system.

² The IRS has acquired a perpetual license to Catalyst® as part of the PRIME contract, subject to certain restrictions. The license includes rights to all enhancements made to Catalyst® by the Computer Sciences Corporation during the contract period.

² The IRS created the BSMO to guide and oversee the work of the PRIME contractor.

³ The CADE is the foundation for managing taxpayer accounts in the IRS' modernization plan.

⁴ No high-priority issues were identified for the Empowerment/Accountability key focus area.

     Table 1: Possible Measures That Could Have Been Used to

Table 2: Ten of the 'Closed' Actions That Still Had Significant

I am pleased your report recognizes that we created the BSM Challenges Plan to resolve longstanding Business Systems Modernization (BSM) issues and that we have a continuous improvement plan for the BSM program. We appreciate your advice on further Improvements to our Program.

There is a lot of work ahead of us to continue to mature the BSM Program, However, while some of the BSM Challenges Plan action Items are on-going and continuous, we have already taken significant steps to address and improve high-priority areas of the program. For example,

• We have dramatically improved our cost and scheduling performance. With the exception of one (integrated Financial System), we delivered all projects In 2004 on time (either early or within a few weeks of schedule) and within budget. Because of our Improvements over the past year, 2004 did not follow the pattern of cost overruns that had occurred from 1999 to 2003.

• The 2005 IRS budget to Congress reflects a portfolio reduction of 37% from the 2003 level, which more closely aligns the BSM workload with the IRS's and PRIME contractor's management capacity,

• A Business Unit leader Is assigned to each project, with responsibility for Wading the related BSM Governance Committee and sharing accountability for delivering the modernization project, as stated in their annual performance commitments.

• We have made great progress in hiring experienced executives and seasoned managers who have expertise in running large Information Technology (IT) programs and projects. A little over a year ago the mix of leadership at the top of the BSM program consisted of one outside expert and six internal IRS executives. That mix will soon be five outside, experienced experts and three Internal IRS executives. This mix is a much better balance of the technology talents and tax administration experience needed to successfully run BSM.

• We have targeted our hiring across the organization to ensure program-wide functional Integration of the business solutions; enhanced program management skills and experience in modernization projects; and, increased systems Integration skills In engineering, architecture and infrastructure. Our efforts continue to bring in critical engineering talent.

• The Commissioner meals monthly with the Chief Operating Officer of Computer Sciences Corporation to reinforce the accountability of the contractor to the IRS. An alternative contracting vehicle (i.e., not the PRIME contract) will be used for now enforcement projects. The BSM project contracts with the PRIME have been restructured, shifting an appropriate amount of financial risk to the contractor and tying costs to performance. These steps have resulted in improved contractor performance as demonstrated in the deliverables in 2004 and the general adherence to costs and schedules.

For the most part, we agree with your report's recommendations. We have already completed corrective actions on two of your four recommendations. We provide additional comments to each of the recommendations in the attachment.

We appreciate your continued support and the valuable assistance and guidance from your staff. If you have any questions please contact me at (202) 622-6800, or Richard Spires, Associate Chief Information Officer, Business Systems Modernization, at (202) 622-7458.

Attachment

TO ENSURE ALL OF THE CONDITIONS IN THE STUDIES ARE ADDRESSED, THE CHIEF INFORMATION OFFICER (CIO) SHOULD:

RECOMMENDATION #1: Re-evaluate the study recommendations that were not completely addressed by the BSM Challenges Plan and determine whether corrective actions should be taken as part of either the 7 KFAs and 16 HPIs or future action items. If it is determined that no actions are needed, the reasons should be documented for future reference.

CORRECTIVE ACTION: Agree with this recommendation and the corrective action has been completed. We have taken comprehensive actions to address the first part of your recommendation. The Associate Chief information Officer (ACIO) for BSM reviewed the study recommendations and determined which HPIs needed to be addressed as part of his continual improvement process for the program. We are actively working many of the studies' recommendations throughout the program, outside of the current 16 HPIs and 27 open issues. Upon thorough review, the now ACIO for IBSM determined other recommendations were not deemed highest priority issues for the program.

With regard to the second part of your recommendation; we will document justifications for all those study recommendations that the BSM Challenges Plan did not address, as shown in Appendix IV of your audit report, We will attach those justifications as a matter of record to the control copy of the May 2004. BSM Challenges Plan Close-out Report.

IMPLEMENTATION DATE: November 22, 2004

RESPONSIBLE OFFICIAL: Associate CIO for Business Systems Modernization, OS:CIO:B

CORRECTIVE ACTION MONITORING PLAN: N/A

TO ENSURE THE IRS CAN OBJECTIVELY DETERMINE WHETHER PROCESS IMPROVEMENT ACTIONS ARE LEADING TO IMPROVEMENT IN THE BSM PROGRAM, STAKEHOLDERS ARE UPDATED ON THE BSM CHALLENGES PLAN STATUS, AND NEW OSM CHALLENGES PLAN ACTIVITIES ARE MONITORED AND TRACKED TO CLOSURE, THE CIO SHOULD:

RECOMMENDATION #2: Create an overall measurement plan to determine whether the activities in the 7 KFAs are leading to improvements in the BSM program by developing specific measurement criteria for each of the HPIs (where possible).

CORRECTIVE ACTION: Agree with this recommendation. Some of the measures we are developing are along the lines of those outlined in your report, For example:

• We are working on the identification of key productivity and quality metrics across the life cycle based on industry standards, such as a quality and timeliness measure and the defect reports analysis measure.

• We will implement measures relating to contracted cost and schedule variance, contracted requirements stability and contracted requirements delivered will be implemented at the beginning of FY 2005.

• We implemented performance-based contracting (PBC) practices at the beginning of FY 2004 and we measured the use of PBC during FY 2004. We have implemented corrective actions for further improvement, such as revisions of the PBC and Monitoring Plan templates and guidance based on FY 2004 lesson & learned and issuance of a Directive specifically regarding Performance Monitoring Plans to document the contractor's performance.

IMPLEMENTATION DATE: September 30, 2005

RESPONSIBLE OFFICIAL: Associate CIO for Business Systems Modernization, OS:CIO:B

CORRECTIVE ACTION MONITORING PLAN: We enter accepted corrective actions Into the Item Tracking, Reporting and Control System (ITRAC). These corrective actions are monitored an a monthly basis until completion.

RECOMMENDATION #3: Ensure stakeholders understand that additional work is necessary to complete actions In the BSM Challenges Plan beyond what was originally reported.

CORRECTIVE ACTION: Agree with this recommendation and the corrective action has been completed. We have made presentations to executives at various levels of our stakeholder community, such as the office of Management and Budget, Government Accountability Office and the Office of the Inspector General for Tax Administration. We will continue With these presentations as we update our KFAs and HPIs and develop periodic improvement plans.

IMPLEMENTATION DATE: June 30,2004

RESPONSIBLE OFFICIAL: Associate CIO for Business Systems Modernizaton, OS:CIO:B

CORRECTIVE ACTION MONITORING PLAN: N/A

REC0MMENDATION #4: Place new activities in the ITRAC database for tracking and closure purposes based on the recent analysis of the BSM Challenges Plan status.

CORRECTIVE ACTION: Agree with this recommendation and the corrective action has been completed. We have assigned responsible managers and owners and placed new, ongoing and closed activities in a shared partition of ITRAC. New conventions, closure criteria and deadlines have been defined.

IMPLEMENTATION DATE: November 22, 2004

RESPONSIBLE OFFICIAL: Associate CIO for Business Systems Modernization, OS:CIO:B

CORRECTIVE ACTION MONITORING PLAN: See corrective action under Recommendation #2.