This article presents an analysis of testing methods used in the domain of blockchain applications, with a focus on their impact on system security and reliability. The main objective of the research is to enhance the testing process of blockchain systems and minimize security risks. The study demonstrates that implementing an appropriate approach to testing blockchain applications can considerably reduce the resources required to resolve bugs and increase the level of system reliability. In conclusion, the paper finds that the selection of testing methods during the planning phase has a substantial effect on the security and reliability of the blockchain system application domain Blockchain testing, data mining, knowledge management, testing methods, test, testing, bug resolution, planning phase, security risks, smart-contract, system security, system reliability, COLINS-2023: 7th International Conference on Computational Linguistics and Intelligent Systems, April 20-21, 2023, Kharkiv, Ukraine ORCID: 0000-0002-7686-6439 (I. Kyrychenko); 0009-0003-5024-0256 (O. Shyshlo); 0000-0002- 9613-186X (N. Shanidze)
from healthcare to banking to retail. blockchain technology are no exception.
There are many systems that process and save customers’ personal information. Every client would
like to be sure that their personal data will be transmitted, processed, and stored confidentially, securely,
and credibly. This can be achieved using blockchain technology. To solve this problem, many systems
use certain encryption algorithms or technologies to ensure the integrity, reliability, and confidentiality
of transmitted or received data [
A Blockchain is a decentralized, distributed, and digital ledger used to document transactions
(blocks) on many computers or, in other words, nodes, so that the record cannot be changed
"retroactively" without changing all subsequent blocks and without the consensus of the network [
This technology can be used to protect, store, and manage data in a decentralized and cryptographic format. Currently, it fully corresponds to the needs of users regarding the security of their data.
However, can we trust systems that have not been properly tested by experts? The answer is definitely no.
Blockchain was originally developed for digital currencies such as bitcoin, but later its advantages led to its use to record not only financial transactions, but also virtually anything of significant value, Therefore, one of the most important stages of system development is testing. And systems using Unfortunately, there is a tendency that only systems directly related to payment transactions usually have a high level of test coverage. Although this technology is used in a wide variety of industries and areas.
An important step in the testing process of any system is the development of a testing strategy.
2023 Copyright for this paper by its authors.
The goal of the research is to determine the impact of the testing methods on security and reliability on the blockchain-based systems.
The research objectives of this study are to consider the key categories of issues that may arise in blockchain-based systems, to examine the testing methodologies for blockchain-based systems, to evaluate the effect of the chosen testing methodologies on the security and dependability of the system, to analyze the results of the tests, and to provide recommendations for the effective use of methodologies by analyzing the test results.
Advantages of blockchain over non-blockchain database. The general advantages and disadvantages
of blockchain technology were described in [
The advantages include the following characteristics:
• Immutability. Blockchain supports immutability, meaning it is impossible to erase or replace
recorded data [
Traceability. Blockchain creates an irreversible auditable trail that allows to easily track changes in
the network [
Also, this technology has its own disadvantages, which include the following:
• Speed and performance. Blockchain is significantly slower than a traditional database because
it performs more operations. First, it verifies signatures, which involves cryptographically signing
transactions [
Blockchain security. Blockchain technology and the mechanism of sharing and opening were studied in [11]. They also discussed the management of data based on blockchain technology to ensure that the data is not copied, not stored by a third party and can be safely traded. But their solution is to discuss blockchain technology for data trading without technical implementation content.
It was proposed a solution and framework for a data trading mode based on a smart contract using blockchain and machine learning [12]. This solution can be used to protect the rights and interests of the data owner. In addition, this article covers various mechanisms and algorithms for selling and downloading data, authenticating and authorizing the data owner/data buyer to download data off-chain, resolving disputes during data trading, automatically paying for the completion of trades, and setting fines for dishonest behavior. The data resource will be uploaded via a secure SSL session [13]. It has a high level of confidentiality and can prevent replay attacks and man-in-the-middle (MITM) attacks [14–15]. Therefore, security of this approach can be called reliable.
First, it is necessary to consider the main bugs that occur in blockchain-based systems.
Categorizing bugs can help to understand the weaknesses of blockchain systems. Therefore, knowing the most vulnerable parts of the system, more efforts can be made to resolve the dominant category of problems.
Most of the bugs are labeled as semantic. Its percentage is 67.23%. Programmers could easily introduce semantic bugs due to the lack of a thorough understanding of the system. Additionally, since semantic bugs are application specific, it is hard to automatically detect semantic bugs [16].
Environment and configuration bugs are the second most frequent by occurrence – 11.42%. This is because blockchain systems are often used by end users in various environments. These environments can run on different hardware and operating systems with different versions of installed libraries. Even if the same versions of libraries are installed, end users may have individual configurations. Errors in libraries, mismatched library versions, and incorrect configurations can lead to environment and configuration errors in blockchain systems.
Security bugs correspond to vulnerabilities that allow attackers to reduce the information security of the system for their own purposes. Even though the average number of bugs related to system security is only 1.90%, the cost of these bugs is extremely high.
As a rule, security bugs are common to all systems using blockchain technology. For example, those related to the buffer overflow vulnerability.
It is also worth noting that fixing security bugs takes the longest average time. It makes sense to implement and use automatic testing tools to detect bugs of this nature in blockchain systems.
In addition, application-specific security bugs, such as synchronization attacks, application-level denial of service, and lack of security checks, are also common in blockchain systems.
The likely cause of such bugs may be that little attention is paid to these vulnerabilities. It could also be that an ineffective approach to system testing was chosen at the planning stage of the testing process.
Traditional testing includes the following types: functional testing, non-functional testing, performance testing, security testing and integration testing.
Functional testing is one of the most basic and fundamental testing methods for any system. Adapting this method to systems based on blockchain technology, the following checks can be identified within this method [17]:
1. Checking the block and chain size. This check should focus on what transaction data should be selected, what encryption methods should be used to connect these blocks, and similar complex scenarios.
2. Checking the ability to transfer data. This test should cover both the successful transmission of data and data loss during transmission between blocks. Data loss should also be tested, as the underlying blockchain architecture is based on transactions and data security.
3. Checking the ability to add blocks. Blocks that are added to the chain should be carefully evaluated, because they cannot be changed after adding them to the chain.
4. Checking the smart contracts. Ensuring that the parties that are involved in transactions adhere to the rules of the smart contract will ensure the smooth functioning of the blockchain application [18].
These are the key points that make it possible to confirm the correct functionality of a blockchain application.
Although smart contract testing is part of functional testing, QA should pay special attention to it. Because it is one of the main components of the system that gives the customer transparency, confidence, and trust in the reliability of the blockchain-based system.
The next test method that should be included in the test strategy is penetration testing [19]. Cyberattacks and fraudulent hacking attacks are extremely difficult to perform in blockchain-based systems when smart contracts are working correctly [20]. However, any mistake in the complex mathematical and programmatic rules can be a good opportunity for unauthorized users with malicious intentions.
Blockchain penetration testing allows QA engineers to create and execute test cases that simulate the behavior of cyberattacks to see if the actions of any unauthorized user in the system will be blocked. This method involves testing that simulates real-world scenarios by attempting to log in through network services, wireless networks, social engineering networks, and blockchain applications.
Security testing is also one of the most important testing methods [21]. At this stage, QA should check whether the blockchain application is fully protected from attacks, including viruses and malware. Any outgoing transaction or data transfer cannot be stopped, so QA testers must identify potential threats through security testing.
Also, at this stage, you should make sure that the blockchain system prevents the processing of faulty or damaged data before it becomes a risk for the user.
In order to ensure the security and reliability of a blockchain-based system, it is imperative to give particular attention to the following methods during the testing phase: • functional testing; • smart contract testing; • security testing; • penetration testing.
It is important to determine the extent to which each of these methods contributes to the overall security and reliability of the system.
Each bug has its own weight, which depends on many factors. This includes: • Criticality of the bug from a technical point of view. • Criticality of the bug from the business logic point of view. • Time spent on the fix. • Complexity of the fix from a technical point of view. • Time spent on checking the fix. • Complexity of checking the fix. • Potential risks after the fix. • Potential risks if the bug was not found.
To assess each of the coefficients, it is proposed to use a scale from 1 to 5. Therefore, it makes sense to consider the dependence of the impact on system security if a bug is found and fixed at a certain stage of testing.
The testing process for each system includes the following steps: 1. Unit testing. 2. Integration testing. 3. System testing. 4. Acceptance testing.
As an experiment, the authors tested their own blockchain-based system. Below is a more detailed overview of the bugs found as a result of testing this system at each of the testing stages.
Unit testing. The following checks should be performed as part of this testing phase: • Block size checks as part of the functional testing. • Checks of the ability to add blocks as part of the functional testing. • Smart-contracts checks.
Consider the following bugs that were found because of the above checks: • Max block size is less than 1 MB. This bug was found during block size checks as part of the functional testing. • Block is added incorrectly. This bug was found during checks of the ability to add blocks as part of the functional testing. • Buyer data is downloaded off-chain. This bug was found during smart contracts checks.
Table 1 shows each of the parameters that will affect the weight of the bugs described above.
Potential risks if the bug was not found 3 2 2 1 2 2 2 5 3 3 3 2 2 2
Buyer data is downloaded off-chain 3 5 3 3 3 2 3 2 Integration testing. The following checks should be performed as part of this testing phase: • Chain size checks as part of the functional testing. • Checks of the ability to transfer data as part of the functional testing. • Smart contracts checks.
Consider the following bugs that were found because of the above checks: • The chain size is smaller than specified. Unable to add a new block. This bug was found during chain size checks as part of the functional testing. • Part of the data is lost during transfer. This bug was found during checks of the ability to transfer data as part of the functional testing. • Buyer data is downloaded off-chain. This bug was found during smart contracts checks.
Table 2 shows each of the parameters that will affect the weight of the bugs described above.
System testing. The following checks should be performed as part of this testing phase: • Checks of the ability to transfer data as part of the functional testing. • Smart contracts checks. • Checks of the system security as part of the security testing. • Checks of the system security as part of the penetration testing.
Consider the following bugs that were found because of the above checks: • Part of the data is lost during transfer. This bug was found during checks of the ability to transfer data as part of the functional testing. • Buyer data is downloaded off-chain. This bug was found during smart contracts checks. • There is a possibility to decrypt the data. This bug was found during checks of the system security as part of the security testing. • Unauthorized user in the system is not blocked. This bug was found during checks of the system security as part of the penetration testing.
Table 3 shows each of the parameters that will affect the weight of the bugs described above.
technical point of view
business logic point of view
technical point of view
4 4 4 3 4 4 5 4 4 4 3 4 4 4 4 2 3 4 3 5 4 4 4 4 4 4 4 4 3 3 4 3 5 4 4 4 3 4 4 Acceptance testing. The following checks should be performed as part of this testing phase: • Checks of the ability to transfer data as part of the functional testing. • Smart contracts checks. • Checks of the system security as part of the security testing. • Checks of the system security as part of the penetration testing.
Consider the following bugs that were found because of the above checks: • Part of the data is lost during transfer. This bug was found during checks of the ability to transfer data as part of the functional testing. • Buyer data is downloaded off-chain. This bug was found during smart contracts checks. • There is a possibility to decrypt the data. This bug was found during checks of the system security as part of the security testing. • Unauthorized user in the system is not blocked. This bug was found during checks of the system security as part of the penetration testing.
Table 4 shows each of the parameters that will affect the weight of the bugs described above. 5. Results 5
After analyzing the results of the experiment described in the previous section, the following conclusions can be drawn.
Correct testing of the system at the unit testing stage can significantly reduce the number and severity of risks, including those to user data security. If a critical error is not found at this stage, the cost of the error will increase depending on which of the subsequent testing stages it is found.
It is also worth noting that errors in blocks or block chains are critical in terms of potential risks. After all, according to blockchain technology, data blocks cannot be changed, and block chains are built based on previous blocks. Therefore, one mistake can lead to significant problems for the entire system, especially when it comes to data security.
As for integration testing, problems with the interaction of system components also have a significant impact on the security and reliability of the system. That is why integration testing plays a significant role in the testing process. The cost of a mistake at this stage will also be much lower than at one of the subsequent stages. However, the potential risks after a fix are much lower than the potential risks if the bug was not found at this stage.
System testing was performed after integration testing. After analyzing the test results at this stage, we can conclude that the cost of a mistake increases significantly.
As for security testing and penetration testing, they should be conducted during system testing to ensure that the system is stable and meets the specified data security requirements.
One of the last stages is acceptance testing. At this stage, the cost and weight of a bug is the highest. After all, the greatest risks, both after the fix and if a bug that directly affects the security and reliability of the system is not found during acceptance testing.
Table 5 represents the summary data with the average weight and price of a bug found at one of the testing stages, as well as for the case when a bug related to system security was found in production.
Summarizing the results at each stage of the experiment, we can conclude that special attention should be paid to unit and integration testing. After all, it is from this point on that the cost of an error begins to grow exponentially. However, system and acceptance testing also play a significant role in the security of a blockchain-based system.
Accordingly, each of the above-mentioned testing methods will significantly affect the reliability of both individual components and the system.
It is necessary to perform functional testing at each stage of testing. Particular attention should be paid to the testing of smart contracts. The number and severity of risks can be greatly decreased by properly evaluating the system during the unit and integration testing stage. Security and penetration testing should also be performed as part of the system testing.
The study's findings can be applied to the planning phases of testing, such as when developing a test strategy and test plan.
In the found articles, the authors described approaches and mechanisms to ensure the security and reliability of blockchain-based systems, but none of them described an approach to selecting testing methods to ensure the actual reliability and security of such systems.
Accordingly, each of the above-mentioned testing methods will significantly affect the reliability of both individual components and the system.
Kent Beck wrote in his book that most defects end up costing more than it would have cost to prevent them. Defects are expensive when they occur, both the direct costs of fixing the defects and the indirect costs because of damaged relationships, lost business, and lost development time [22].
In the articles reviewed, the authors described approaches and mechanisms for ensuring the security and reliability of blockchain-based systems. The testing process was discussed, but in a rather brief manner. Also, the reviewed articles did not investigate specific methods of testing blockchain-based systems. Accordingly, the impact of the selected methods in the process of testing the system on the actual security and reliability of such systems was not considered either.
In this article, the following testing methods were considered: 1. Functional testing 2. Testing of the smart contract as part of the functional testing 3. Security testing 4. Penetration testing 5. Unit testing 6. Integration testing 7. System testing 8. Acceptance testing
The impact of the quality of testing using the selected testing methods at different levels of the testing process was also considered.
Data security is one of the key requirements for blockchain-based systems. Therefore, our research was aimed at analyzing the impact of the testing process on the system's reliability [23].
Since the choice of testing methods takes place at the planning stage of the system testing process, it is necessary to clearly understand which approach should be chosen to test the system as efficiently as possible and make sure that it is safe and reliable as soon as possible [24].
The solution is designed to help you plan the testing process properly at the planning stage.
After all, the cost of a mistake can be too high, both tangibly (time and money spent on fixing the problem) and intangibly (opportunities for system development and customer confidence).
Today, blockchain technology is widely used to secure, store, and manage data due to its advantages, such as: 1. Immutability. Blockchain supports immutability, meaning it is impossible to erase or replace recorded data. Therefore, the blockchain prevents data tampering within the network. maintains immutability, meaning that it is impossible to erase or replace already recorded data. Therefore, blockchain prevents data falsification in the network. 2. Transparency. Blockchain is decentralized, which means that any participant in the network can verify their data. Therefore, the public can trust this network. 3. Censorship. Blockchain technology is free from censorship because it is not controlled by any side. That is why no authority (including governments) can interrupt the work of the network. 4. Traceability. Blockchain creates an irreversible auditable trail that allows to easily track changes in the network.
This technology also has its disadvantages. One of them is the high cost of developing and maintaining the system. The use of this technology requires significant material resources. Accordingly, there is a need to avoid wasting extra resources due to poorly conducted testing. Since the cost of any bug is quite high and tends to increase significantly depending on the stage of the system's life cycle at which it was found.
But in addition to material losses, there are also intangible losses, such as loss of customer confidence, loss of personal information transmitted and stored in the system by fraudsters.
That is why it is extremely important to make sure that the system is safe and reliable. The only way to ensure the reliability of any system is to conduct a quality testing process. That is why this article discusses the testing methods that have the greatest impact on the security and reliability of blockchainbased systems.
The obtained results of the study can be used at the planning stages of the testing process, namely, when creating a test strategy and test plan. 9. References