Public Activity Fields and the Present Situation of Policy Evaluation

(For general administrative areas, also refer to the following page)

Field Present situation of policy evaluation
Relation <Situation in Japan>
  • Although some attempts have been made, including the "Three-Year Program for Promoting Deregulation" (Cabinet Decision, March 1998) (when regulations are newly established, results of the examination of necessity, effects, burdens, etc. are to be published immediately after the closing of the regular session of the Diet every year) and the "Public Comment Procedure for Formulating, Amending or Repealing a Regulation" (Cabinet Decision, March 1999), analytical frameworks are yet to be designed.
    <Situation in the UK, the US, Canada and Australia>
  • In introducing or reviewing a regulation, regulatory impact analysis (RIA: estimation of costs and benefits brought about by the regulation, comparison among possible alternatives) is conducted, the draft of regulation and accompanied by the regulatory impact analysis is published for public comments, and the regulation is finalized with reflecting the public comments. RIA guides that stipulate the above-mentioned regulation-planning process and the principles of the analysis.
  • As specific methods, quantitative methods including cost-benefit analysis or cost-effectiveness analysis are considered. Actually, however, many of the cases include only qualitative descriptions.
  • Public works <Situation in Japan>
  • Regarding public works including soil enrichment, riparian work, general toll roads, improvement of coastal fisheries, and construction and maintenance of the Shinkansen Line, and sewers, there are stipulations specifying the obligation to estimate effectiveness. There are guides applied to each public works area. Ministry of Transport and Ministry of Construction have drawn up their own guides which govern common issues such as selection of the method, procedure, publication of the result, and unification of basic numerical values (discount rate, etc.)
  • On December 5, 1997, the Prime Minister requested the conduct of cost-effectiveness analysis in decision making process for all public works projects, and the introduction of the "re-evaluation system". Under the re-evaluation system, projects that have not been started after the lapse of a certain period of time after the adoption or long-term projects are re-evaluated. Based on the result, actions are taken. Projects that are determined not to be appropriate are suspended or canceled.
  • At present, re-evaluation is not conducted following some fixed quantitative method, but conducted as qualitative check.
    <Situation in the UK, the US, and Germany>
  • Cost-benefit analysis is most commonly used for the analysis of public works projects internationally. In the UK and the US, there are guides concerning basic issues commonly applied to various areas of public works by all departments.
  • In Germany, cost-benefit analysis incorporates regional income differentials. (For low-income regions, benefits are inflated by multiplying by a certain coefficient). In addition, different kinds of projects in the transportation area, such as roads, railways, and waterways, are mutually compared in terms of costs and benefits at parity. These unique ideas are not seen in Japan.
  • Science and Technology policy <Situation in Japan>
  • Regarding R&D projects and research institutes under the umbrella of the Agency of Industrial Science and Technology, the Ministry of International Trade and Industry conducts "technology evaluation" utilizing external experts, mainly concerning technological aspects.
    <Situation in OECD countries>
  • MITI is developing a method to measure socio-economic impacts of finished projects, and is also planning to evaluate each program or evaluate R&D projects as a portfolio for each technological field.
    <Situation in OECD countries>
  • Evaluation of individual R&D projects seems to be basically equal to the MITI's technology evaluation. (project evaluation)
  • In Australia, Canada, the UK and the US, there are examples of program level evaluations (that exceed the level of individual project) that take into consideration the socio-economic impacts. (However, there seems to be no established methodology.)
  • Official Development Assistance (ODA) including economic cooperation <Situation in Japan>
  • Organizations which provide ODA conduct project evaluation mainly at the point of completion and the after the lapse of a certain period of time after the end of assistance.
    <Situation in foreign countries>
  • Some foreign countries and international organizations adopt the "project design matrix," which is a list of major factors including objectives, inputs and external conditions of projects, and utilize it for planning, implementation and evaluation. Performance measurement and qualitative aspects of projects are emphasized.
  • In the US, qualitative methods including interviews, direct observations and review of publications ("quick and low-cost methods") are developed and utilized, essentially giving up strict statistical methods.
  • In Germany, a unified evaluation system (for each project, elements such as objectives, planning, implementation, control and impacts are qualitatively evaluated, and the project as a whole is scored on a 0-5 scale) is applied to every project. This enables a mutual comparison of different projects.
  • Public services <Situation in Japan>
  • Among local governments (in Japan as well as in the US), performance measurement type evaluations are spreading. These evaluations establish performance indicators concerning the effectiveness of administrative activities and customer satisfaction and monitor the achievements.
  • There are views that the outcomes of a program can be greatly influenced by the socio-economic environment that cannot be controlled by the responsible public sector, and therefore, the linkage of outcome achievements and the resource allocation requires prudence. However, partly because of its simplicity, more and more local governments are adopting performance measurement methods.
    <Situation in the UK and the US>
  • In the UK and the US, "market testing" is also conducted. Services that have been traditionally conducted by the government, and are considered unsuitable for privatization, may be put up for tender. Governmental sections that have been traditionally in charge of such services and possible private providers compete by bid for the operation of the services. (drainage disposal, garbage disposal, road repairing, etc.)
  • General administrative areas ROAME system of the UK Department of Trade and Industry (DTI) <Situation of DTI's ROAME system>
  • The DTI's internal evaluation system under which appraisal, implementation of a program, evaluation, and feedback on program planning are carried out as a closed loop process. The system is diffusing among the UK departments as a de facto standard.
  • Program managers prepare a simple appraisal document called ROAME statement (describing rationale (R), objectives (O), appraisal criteria for projects (A), monitoring methods (M), timing of evaluation (E)). The document is a kind of contract between the budget holder and the program manager. Only programs whose ROAME statement is approved by the budget holder are implemented.
  • Although the system seems to be applicable to any policy areas, in reality the system may not be functioning sufficiently, except for promotion of science and technology in the private sector.
  • Program evaluation <Situation in Japan>
  • Systematic measures are not taken.
  • Textile policy evaluation conducted by MITI on the policy redesign in 1997-98 can be categorized as program evaluation. The case was qualitative in nature, but concerns the effectiveness of the textile industry promotion programs rather than individual projects.
    <Situation in the UK and the US>
  • Performance auditing of departments' programs conducted by the supreme audit office in the UK and the US (NAO, GAO) is considered to be a typical example of "program evaluation." In the UK, the US and Canada, each department carries out program evaluation internally.
  • Most of the GAO's performance auditing cases do not estimate cost-benefit ratio or performance indicators of programs, but only present conclusions and analyses that can be made objectively and logically from the examination. In addition, even in the cases pointing out problems of a program, casual analysis is rarely conducted.
  • GPRA of the US (Note)

    (Government Performance and Results Act of 1993)
    <Situation of GPRA of the US>
  • Performance measurement type evaluation. This is an attempt to clarify federal departments' missions and strategies, to combine them with specific performance objectives (both output and outcome indicators are set), to direct the departments' eye on the performance and customer satisfaction, and to reflect the results on the distribution of policy resources (budgets, manpower).
  • Phased implementation is adopted. Pilot projects were conducted by limited number of federal sections, then, the experiences were accumulated for the full-scale application.
  • There are views that the outcomes of a program can be greatly influenced by the socio-economic environment that cannot be controlled by the responsible department, and therefore, the linkage of outcome achievements and the resource allocation requires prudence. Regarding GPRA, specific plans have not yet been shown regarding how to reflect performance results on the resource allocation.
  • GPRA is applied to all the federal departments and the programs exhaustively and uniformly. The actual operation of system is to be carefully watched.
  • According to a Congress report published in June, 1998, the format and content of the first performance plans prepared by departments did not reach the initially expected level.

  • (Note) As a system similar to GPRA, there was the Planning-Programming-Budgeting-System (PPBS) which was conducted in the 1960s and was abolished in 1971. PPBS was intended to systematize all the government activities into a policy-program-project hierarchical structure of purpose and means, to conduct the cost-benefit analysis, and to draft a budget based on the analysis. PPBS and GPRA are alike in terms of exhaustiveness and uniformity. However, while PPBS was intended to draft a budget based on the ex ante evaluation, GPRA attempts to increase or decrease a budget based on the ex post results. In addition, GPRA uses output and customer satisfaction measures as well as outcome measures, which is a reflection of one of the lessons learned from PPBS's failure, that is difficulty of measuring outcomes.