Review of FFIEC Call Report
DATE: March 21, 2003
TO: Steven O. App, Deputy to the Chairman and Chief Financial Officer; and John F. Bovenzi, Deputy to the Chairman and Chief Operating Officer
FROM: Russell A. Rau [Electronically produced version; original signed by Stephen M. Beard for Russell A. Rau], Assistant Inspector General for Audits
SUBJECT: Review of FFIEC Call Report Modernization Cost Benefit Analysis (Audit Report No. 03-018)
At the request of Corporation senior managers, we reviewed the cost benefit analysis (CBA) and assumptions supporting the draft request to the FDIC Board of Directors for funding the Federal Financial Institutions Examination Council (FFIEC) Call Report Processing Central Data Repository, dated January 23, 2003. (Note: The FFIEC includes representatives from the Board of Governors of the Federal Reserve System (FRB), National Credit Union Administration (NCUA), Office of the Comptroller of the Currency (OCC), Office of Thrift Supervision (OTS), and FDIC.) The senior managers requested that we provide them with our analysis prior to the scheduled February 11, 2003, FDIC Board of Directors meeting.
Our objective was to determine whether the cost information contained in the CBA was supported and the assumptions made were reasonable. We reviewed the methodology and supporting documentation used to prepare the CBA. The scope of our review was limited to determining whether the FDIC had adequately supported amounts presented in the business case and the underlying assumptions were reasonable. We did not evaluate the viability or benefits of the alternatives and options in the case. A detailed discussion of our scope and methodology is included as Appendix II. The purpose of this report is to provide you with the results of our review and the slides (Appendix I) we used to brief management on February 7, 2003. (Note: We amended the briefing slides to include technical corrections and clarifications.) We conducted our audit in accordance with generally accepted government auditing standards during the period January 27, 2003 through February 14, 2003.
FFIEC Effort to Modernize the Call Report Process. In 1998 the FFIECís Task Force on Reports initiated a Call Report modernization effort. An outgrowth of this initiative was the interagency Call Modernization project launched in 2001. The Call Modernization project is a collaborative effort of the FDIC, FRB, and OCC to improve the processes and systems used to collect, validate, store, and distribute Call Report information. (Note: Every national bank, state member bank, and insured state nonmember bank each calendar quarter is required to file Consolidated Reports of Condition and Income (Call Report).) Institutions regulated by the OTS and NCUA are not required to submit Call Reports to the FDIC. Accordingly, the OTS and the NCUA were not included in this project.
The FFIEC Call Modernization Steering Committee also established a Cost Working Group to collect and analyze the cost of the current Call Report processes across the FDIC, FRB, OCC, and the reporting institutions. The FDIC in turn has a subgroup called the Institution Data Management (IDM) project team that performed the study for the FDIC. The study identified current (year 2002) annual operating costs of $10.5 million for the three agencies. Of the $10.5 million, the FDIC incurs $7.6 million, the FRB $2.4 million, and the OCC $0.5 million.
In May 2002, the FFIEC authorized the FDIC to issue a request for proposal to have a contractor design, implement, and host a new facility for Call Report and other bank regulatory reporting data. Bidding closed in October 2002, and the Call Modernization Steering Committee completed its analyses of the proposals in January 2003. The Committee tentatively selected a proposal in the amount of $39 million, which included $14 million in system development costs and $25 million for system operation and maintenance over a 10-year period. Additional funding of $5 million was requested for future contingencies, for a total contract budget of $44 million.
Two Alternatives Considered. The IDM project team prepared a CBA, dated January 23, 2003. The CBA compared two alternatives on the basis of cost, benefit, risk, and sensitivity. The alternatives presented were as follows:
Alternative 1. The current Call Report process uses a 10-year old distributed approach to obtain, validate, and distribute Call Report data. Reporting institutions provide their Call Reports in electronic format to a private vendor, Electronic Data Systems, that consolidates and distributes the Call Reports to the FRB using the 25-year old Bulk Data Transfer format. The FRB parses the consolidated data based on the primary regulator and distributes FRB-supervised institutionsí data among the 12 Federal Reserve District Banks and sends FDIC- and OCC-supervised institutionsí data to the FDIC. The FDIC and the District Banks then edit the data using in-house methods and systems, resolving exceptions based on a combination of common and agency-specific criteria. Once the data are edited, files are exchanged among the regulators, and a consolidated file of all reports is compiled and distributed to internal and external customers.
Alternative 2. The Agencies developed a consensus vision for a new Call Report processing business model that incorporates open data standards, uses a common reporting language, and offers tools to enable banks to submit better reports. The goals of the proposed environment are to create: (1) a single CDR serving as the official source of all the information necessary for collecting, validating, and distributing bank Call Report information; (2) a process driven by uniform business rules leveraging existing standards; (3) a movement of regulators and reporters to a recognizable Internet standard; (4) a common, comprehensive set of FFIEC data quality assurance standards that will be defined and published; and (5) a requirement that only Call Reports that pass all math validation criteria and that provide text explanations for logical/qualitative validation criteria exceptions will be accepted.
Alternative Chosen. The Call Modernization Steering Committee of the FFIEC Task Force on Reports recommended alternative 2, with three actions necessary to implement a CDR contract. The three actions included:
RESULTS OF REVIEW
The methodology used in comparing the alternatives was generally consistent with FDIC and Office of Management and Budget (OMB) guidance for the preparation of cost benefit analyses. The IDM project team obtained and analyzed cost data from several FDIC divisions, analyzed the benefits and risks associated with the alternatives, and projected the impact that the cost benefit assumptions could have on the recommended alternative in the sensitivity analysis. However, some of the assumptions and rationale used to arrive at the amounts included in the cost analysis were not consistently supported or clearly explained. Specifically, formal management verification, indicating agreement with the costs in the case, was not received from all affected divisions. As a result, there is a risk that the return on investment may not be fully realized. In addition, accountability for the cost benefits may be difficult to establish because the assumptions made were not adequately documented. Nonetheless, adjustments based on our concerns would fall within the range of the CBA sensitivity analysis, which indicates a positive return on investment for the CDR alternative chosen.
The applicable criteria for our review were FDIC Circular 4310.1, Utilizing Cost Benefit Analysis Methodology for the Purchase or Development of Capital Assets, dated July 17, 1998, and OMB Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, both of which provide guidance for preparing CBAs. (Note: Although OMB Circular No. A-94 does not apply specifically to the FDIC, it is used as a prudent business practice in the area of government cost benefit analysis.) According to the circulars, CBAs should be performed to promote efficient resource allocation through well-informed decision-making. The CBA is included as part of the decision-making process when determining whether to purchase or develop a capital asset. The analysis should be explicit about underlying assumptions used to arrive at estimates of future benefits and costs. The analysis should include a statement of the rationale behind the assumptions and a review of their strengths and weaknesses. Key data and results should be reported to promote independent analysis and review. In addition, post implementation verification should be performed to determine whether anticipated benefits and costs have been realized and whether improvements need to be made in future estimates of benefits and costs.
The FDIC and OMB guidance recognize that there are inherent limitations in the estimates of costs and benefits because of imprecision in both the underlying data and the modeling assumptions. These limitations are generally addressed in the sensitivity analysis where the risks associated with the cost projections are addressed.
The Project Teamís Methodology for Preparing the Cost Analysis
In preparing the cost analysis, the IDM project team obtained cost data from several divisions in the FDIC as well as the FRB and the OCC. In March 2002, the IDM project team sent each FDIC division involved in the CBA a copy of the current operating cost estimates for review. According to members of the IDM project team, they did not receive any feedback from the divisions on their March 2002 letter. The IDM project team did, however, obtain a subsequent costing letter from the Division of Information Resources Management (DIRM) regarding the DIRM costs included in the cost analysis.
Using this original cost data and a subsequent costing letter from DIRM, the IDM project team prepared two cost alternatives. (Note: Each alternative had two options. The first option related to core (Call Report) processes only, and the second option was for value-added services including Uniform Bank Performance Report processing, referred to as total or full implementation. Our analysis relates only to the full implementation option because that was the business-case-recommended option.) Alternative 1, Enhancing Existing Agency Systems, was derived by using the current (year 2002) estimated operating costs of the reporting processes and adding an additional $500,000 for the migration of one of the systems to a new operating platform. Under Alternative 2, CDR, the cost of current operations, which would no longer be necessary with the CDR, are eliminated as CDR is phased into operations. Further, the cost of the CDR contract is added to the cost of operations. The cost estimates were projected over a 10-year period with a 3-percent annual inflation factor. Using these amounts, alternative 2, CDR, results in a $27 million dollar reduction in costs over the 10-year period. A discount factor of 5.1 percent was applied to the cost reductions to arrive at a return on investment for the CDR of $17.9 million (133 percent) over the 10-year period.
Our Analysis of the Methodology
During our review, we attempted to verify the cost data and assumptions used to derive the projections made in the cost analysis. Appendix II describes the procedures we performed to evaluate the CBA. However, in certain instances the data used or assumptions applied could not be traced to information included in the CBA or to additional documentation obtained from the IDM project team. While the DIRM costing letter was useful in validating the information included in the cost analysis and supporting the reasonableness of the assumptions made, the IDM project team did not obtain similar evaluations from the other divisions impacted by the CBA. Also, the IDM project team did not prepare in advance a methodology for ensuring consistent treatment of certain costs. If evaluations had been obtained and cost assumptions defined, several of the cost concerns discussed below may have been resolved.
Our Specific Concerns with the Cost Analysis
Benefit and Risk Analysis
As identified in the business case to the FDIC Board of Directors, the assumptions related to benefits and risks are highly dependent on the vendorís successful implementation of the CDR. Because the CDR involves the use of new technology, the risks are greater than those that would be expected with existing proven technology both in terms of the success of the project and the cost in implementing the project. As discussed in the case, the IDM project team believes the potential benefits of the CDR project and new technology outweigh the potential risks. Some benefits and risks were not quantified in dollar terms in the CBA because the project team did not believe it was appropriate to do so, given the uncertainty of their nature. The scope of our review did not include an evaluation of these non-monetary benefits and risks. However, we did note that the business case included a risk mitigation plan to address concerns related to implementation of the new technology.
A sensitivity analysis was prepared to determine how sensitive outcomes are to changes in assumptions. The CBA identified the cost benefit of shutting down current operations as CDR is implemented as the dominant assumption. The sensitivity analysis projected the impact that a reduction in the cost benefit would have on the return on investment over a 10-year period. In preparing the sensitivity analysis, the IDM project team projected cost benefit reductions of 10, 25, 35, and 47 percent (47 percent represented the "break-even" point). If the reduction in cost benefit exceeds 47 percent, then the investment costs would not be recouped in full. The sensitivity analysis was performed properly.
CONCLUSIONS AND SUGGESTIONS
The effect of the cost concerns discussed above would be a reduction in the realizable ROI from $17.9 million (133 percent ROI) to about $14.6 million (109 percent ROI). Consequently, even with the adjustments, alternative 2, CDR, still provides the best-cost alternative given the other data included in the cost analysis. The potential adjustments fall within the 10-percent reduction of the sensitivity analysis included in the CBA. Based on the limited nature of our review, we do not have a sufficient basis to make recommendations related to the cost benefit analysis for the CDR or future investments. However, we suggest that the FDIC consider:
We discussed the results of our review and presented the attached slides to officials from the IDM project team on February 7, 2003. The IDM project team generally agreed that the assumptions and rationale used in preparing the cost analysis could have been better supported and more clearly explained. In consideration of the comments received from management, we amended the slides to include technical corrections and clarifications. The CFO and COO agreed with our suggestions and provided joint written comments to a draft of this report. These comments are included as Appendix III.
In response to our suggestions, the Corporation will:
SLIDES PRESENTED TO CDR PROJECT TEAM
Call Report Modernization Cost Benefit Analysis
OIG Cost Analysis Concerns
OIG Benefit and Risk Analysis Comments
OIG Summary Conclusions
Objective, Scope, & Methodology
SCOPE AND METHODOLOGY
To accomplish our objective, we interviewed Headquarters Division of Insurance and Research (DIR) officials who were responsible for the preparation of the CBA. We also interviewed DIR, DIRM, and Division of Finance (DOF) officials who were involved in the accumulation of FDIC cost data included in the CBA. We reviewed key documents supporting the CBA, including spreadsheets prepared by the IDM project team, cost assumptions prepared by DIRM, personnel cost figures prepared by DOF, contract costs, and other documents related to specific cost items.
The scope of our audit was limited to determining whether the FDIC had adequately supported amounts presented in the business case and the underlying assumptions were reasonable. Our audit did not evaluate the viability or benefits of the alternatives and options in the case. We did not evaluate the internal control environment over the preparation of the business case. We will assess internal controls in future audits of the IDM project. In addition, our audit did not assess the FDICís compliance with applicable laws and regulations because we did not identify specific laws or regulations pertaining to the development of the business case.
Our audit included a determination of whether the FDIC had incorporated performance measures in the business case. We noted that the case describes its alignment with the FDIC strategic vision, mission, and business goals by promoting interagency collaboration, consolidating back office operations, leveraging new technologies for current and future operations, and promoting the FDIC as the source for banking data.
In addition, we relied on computer-processed data without performing tests of system general and application controls to confirm the reliability of the data. We verified that the computer-generated computations were accurate, but we did not review the source data obtained from the systems. However, nothing came to our attention as a result of specified audit procedures that caused us to doubt the reliability of the computer-processed data. Throughout the audit, the auditors were sensitive to the possibility of abuse or illegal acts.
We conducted our audit in accordance with generally accepted government auditing standards during the period January 27, 2003 through February 14, 2003.
March 17, 2003
MEMORANDUM TO: Russell A. Rau, Assistant Inspector General for Audits
FROM: Steven O. App [Electronically produced version; original signed by Steven O. App], Deputy to the Chairman and CFO; and John F. Bovenzi [Electronically produced version; original signed by John F. Bovenzi], Deputy to the Chairman and COO
SUBJECT: Review of FFIEC Call Modernization Cost Benefit Analysis (Assignment No. 2003-026)
At the request of the Capital Investment Review Committee, the Office of Inspector General (OIG) reviewed the cost benefit analysis (CBA) and the assumptions supporting the draft request to the FDIC Board of Directors for funding the Federal Financial Institutions Examination Council (FFIEC) Call Report Processing Central Data Repository (CDR), dated January 23, 2003. The objective was to determine whether the cost information contained in the CBA was supported and whether the assumptions were reasonable.
The OIG reviewed the methodology and supporting documentation used to prepare the CBA. Based upon the limited nature of the review, the OIG did not have a sufficient basis to make recommendations related to the cost benefit analysis for the CDR or future investments. However, the OIG provided suggestions for improving the FDIC cost benefit analysis directive (FDIC Circular 4310.1).
We concur with and appreciate the OIG suggestions. We are currently in the process of updating or replacing our cost benefit policy to include additional elements such as return on investment (ROI). As part of this effort we will:
Overall, we believe the OIG review presented a fair evaluation of the work performed by the CDR project team.
Cc: Vijay G. Deshpande
|Last Updated 04/04/2003|