Document Titled “Evaluation Forms for Blockchain-based Systems ver. 1.0” Released
Blockchain technologies are expected to be applied in a wide variety of fields, including IoT. To promote this technology from neutral assessment, the Ministry of Economy, Trade and Industry (METI) has released a document titled “Evaluation Forms for Blockchain-based Systems ver. 1.0.”
Blockchain technologies are new types of technology used for trading virtual currencies, e.g., Bitcoins. The important features of blockchain technology are that it is extremely difficult to falsify compared to conventional systems, and that inexpensive systems that cause no downtime in effect could be built. Due to these features, the technologies are expected to be applied in a wide variety of fields, including IoT.
However, no evaluation indices or criteria had been established to adequately assess the features of the technologies and to compare them with existing systems. This causes the public anxiety on the one hand and unreasonable hopes for the technology on the other, leading to a potential unwillingness to introduce the technologies. Blockchain-based systems have tradeoff unique to the nature of blockchain technologies, e.g., due to consensus-building by nodes. This is one of the factors that makes it difficult to evaluate blockchain technologies in the same way as the the conventional system-evaluation methods.
Bearing these challenges in mind, METI held interviews with people from both domestic and foreign companies and other experts in the field of blockchain technologies, held discussions based on the interviews and studied new evaluation forms that will allow users to make comparisons between conventional systems and blockchain-based systems and that will comprehensively cover a variety of systems, and organized tradeoffs between evaluation items. Following these efforts, METI compiled these discussion results into the Evaluation Forms for Blockchain-based Systems Report, the world’s first comprehensive evaluation method for this field.
Figure 1: Conceptual diagram comparing a conventional system and a blockchain-based system
2. Outline of the report
(1) Intended scope of evaluation forms in this report
In discussing the introduction of a blockchain-based system, METI aimed to formulate evaluation forms by which system vendors and other stakeholders would be able to compare systems in terms of features and functionality. To this end, METI examined blockchain-based systems as a whole, including related sub-systems, and blockchain platforms.
Consequently, the evaluation forms are expected to be utilized not only in the comparison between a conventional system and a blockchain-based system, but also in the comparison between different blockchain-based systems (see Figure 2).
Figure 2: Study of the scope of evaluation in discussing evaluation forms (conceptual diagram)
(2) Elements of the evaluation forms
METI organized the evaluation forms within three major categories: quality, maintenance/operation, and cost effectiveness.
These categories will contribute to assisting system vendors, the expected users of blockchain technologies, in using the evaluation forms in conjunction with existing system-evaluation methods.
In this context, concerning the sections closely related to blockchain technologies, the quality category includes excerpts from ISO/IEC25010 (system and software quality models) and the maintenance/operation category includes excerpts from Chapter 4 (Maintenance), System Reference Manual (2005), a compilation of evaluation criteria for conventional systems, published by the Information-Technology Promotion Agency, Japan (IPA). The cost effectiveness category is a collection of points which system vendors should recognize in deciding pricing for customers (see Figure 3).
METI also clarified key points that users should be aware of in evaluating various systems, based on a consideration of features of blockchain technologies in which various evaluation items and characteristics are in tradeoff relationship.
For details of the evaluation forms, see the Appendices.
Figure 3: Study for elements of the evaluation forms
(3) How to utilize the evaluation forms
The evaluation forms formulated by METI are expected to be utilized for the following purposes:
- Comparing a conventional system and a blockchain-based system to evaluate replacement options
- Evaluating the results of demonstration tests of systems using blockchain technologies
The evaluation forms are expected to contribute to popularizing common evaluation methods or criteria among a variety of evaluators or system vendors, thereby assisting such evaluators in comparing and referring to a variety of evaluation results.
- Summary of the Evaluation Forms for Blockchain-Based Systems ver. 1.0（PDF：1,268KB）
- Full text of the Survey Report on Establishing Evaluation Forms for Blockchain-Based Systems (in Japanese) （PDF：2,627KB）
March 29, 2017
Division in Charge
Information Economy Division, Commerce and Information Policy Bureau