Study greatest practices for debugging and error dealing with in an enterprise-grade blockchain utility – IBM Developer

Blockchain is a shared, replicated immutable ledger for recording transactions, monitoring belongings, and constructing belief. An asset might be tangible (for instance, a home or a automobile) or intangible (for instance, mental property or patents). Blockchain is constructed on properties like consensus, provenance, immutability, finality.

In a conventional enterprise situation, a transaction that includes a number of organizations is recorded in a different way by every enterprise. If two organizations disagree on the state of a transaction, then a dispute happens, which might usually be expensive and time consuming to resolve. Blockchain introduces the next ideas:

• Multiparty transactions: Signed by everybody concerned within the transaction.
• Shared ledgers: The identical ledger is replicated in each group within the community and saved synchronized utilizing a course of referred to as consensus. Ledgers are immutable and closing; after a multiparty transaction is written to the ledger, it can’t be reversed.

To get an outline of blockchain in additional element, take a look at the Get began with blockchain studying path. This weblog put up focuses on completely different factors of failures in a typical blockchain-based utility, doable causes for failures, and beneficial debugging and error dealing with methods.

The significance of error dealing with

In cloud-based utility deployments that contain a number of integration factors, there’s all the time a chance of encountering transient failures. Planning for and dealing with these transient failures is vital to take care of a resilient structure.

Unhandled error situations can result in failures and system crashes, and so they usually expose the applying in a weak state. Good exception dealing with may help to anticipate errors or programs crashes upfront after which put in applicable code to get well from them. It won’t be doable to deal with all distinctive circumstances or sudden situations, however a nicely designed system ensures sleek exit with out inflicting any main points, inconsistencies, and safety vulnerabilities to the system.

Useful blockchain terminology

• Peer: Maintains ledger and state, commits transactions, and may endorse transactions by receiving a transaction proposal and responding by granting or denying endorsement (should maintain good contract to endorse).
• Ordering node: Orders and packages transactions into blocks after which communicates these blocks to committing friends.
• CA: Points digital certificates to member organizations and their customers.
• Channels: Channels present privateness between completely different ledgers. Ledgers exist within the scope of a channel.
• Good contract: Incorporates the enterprise logic that governs how knowledge is written to and skim from the ledger.
• Transaction: Any operation that modifies the ledger state is recorded as a transaction, corresponding to asset alternate or a switch.
• Ledger: A ledger is maintained by every peer and contains the blockchain and world state.
• Identification: The sources that you could entry in a community are decided by the id and is often represented by an X.509 certificates issued by the CA.
• Endorsement coverage: Describes the situations by which a transaction might be endorsed. A transaction can solely be thought-about legitimate if it has been endorsed in keeping with its coverage.
• Connection profile: Incorporates community data corresponding to node stage connection data, TLS certs, and extra.

The next picture represents the steps concerned in a daily blockchain transaction circulation:

1. Consumer utility submits a transaction proposal.
2. Endorsers E0, E1, and E2 every execute the proposed transaction. None of those executions replace the ledger. Every execution captures the set of Learn Write knowledge (referred to as an RW-set), which now flows within the cloth.
3. Utility receives responses. The RW-sets are signed by every endorser and likewise embrace every report model quantity.
4. Utility submits proposal responses as a transaction for ordering.
5. Orderer sends blocks to committing friends.
6. Committing friends validate transactions. Validated transactions are utilized to the world state and retained on the ledger.
7. Utility is notified when a block is dedicated to the ledger of a peer.

Potential errors in a blockchain community

Errors can come up at any level of the transaction and are brought on by the underlying community, enterprise logic, outdated knowledge, or community time-outs. You possibly can resolve errors that aren’t brought on by utility enter or incorrect utilization by including the correct error dealing with and retry mechanism within the utility layer to construct in utility resiliency.

Errors typically fall into considered one of two classes:

• Retryable: Errors from chain code or the community needs to be propagated again to the applying layer for error dealing with and a retry mechanism.
• Nonretryable: Errors which can be brought on by incorrect utilization needs to be dealt with resulting in a sleek exit of the code path.

Community errors

A Hyperledger Material node or Java shopper communicates with the Hyperledger Material community utilizing gRPC. The gRPC expertise handles shifting knowledge reliably between the Material community and the Material shopper utility. The appliance units the gRPC settings primarily based on the applying utilization.

Downside abstract

grpc request timeout whereas submitting the proposal

This timeout sometimes occurs at any stage of blockchain transaction circulation. Within the earlier picture, the day trip happens throughout Steps 1-4 the place the shopper communicates with the community as a consequence of community latency, unavailability of the chain code container, or poor peer well being.

Error message:

sendPeersProposal - Promise is rejected: Error: REQUEST_TIMEOUT
Peer{ id: 1 , identify: peer1.org1.instance.com, channelName: mychannel,
url:grpcs://192.168.1.1:7051, mspid: Org1MSP} failed due to
timeout(35000 milliseconds) expiration
java.util.concurrent.TimeoutException: null

Request timeout in the course of the execution of a proposal

This error happens when the time taken for executing the proposal exceeds the configured execution time default of 30 seconds. As a greatest follow, you need to restrict the computations or operations that may be carried out in a sensible contract, and if the applying logic doesn’t assist the identical, then you may improve the execution timeout to mitigate the problem.

This error might be thrown by the lifecycle system chaincode (LSCC) in Hyperledger Material as nicely in the course of the startup of the chaincode container. Nonetheless, the decision stays the identical on this case as nicely.

Error message:

"peer1.org.com:7051" failed: message=did not execute transaction 799eb959954a7f2f8f75dee735969a4ba374b4bc98b4bbacd2fc85fc57a860b9: error sending: timeout expired whereas executing transaction, stack=Error: did not execute transaction 799eb959954a7f2f8f75dee735969a4ba374b4bc98b4bbacd2fc85fc57a860b9: error sending: timeout expired whereas executing transaction

Really useful dealing with

1. Set CORE_CHAINCODE_EXECUTETIMEOUT =<60s or larger> within the Material configuration for dealing with the request timeout in the course of the execution of a proposal.

2. Set the next gRPC settings on the Material shopper finish to assist improve the gRPC timeout, which typically occurs due to community latency:

   "grpc.keepalive_time_ms": 120000,
   "grpc.http2.min_time_between_pings_ms": 120000,
   "grpc.keepalive_timeout_ms": 20000,
   

   Present extra

3. Verify the Material shopper day trip configuration and tune it primarily based on the applying processing logic and community suggestions.
4. Verify the peer well being and the well being of IO operations on the peer, and if the peer is just not wholesome, a restart would possibly resolve the problem.
5. Consumer-side retry dealing with: Timeout Exception needs to be categorised as a retryable error and dealt with by writing retry logic on the shopper facet. You should use a easy retry dealing with of this exception with exponential backoff to get well whether it is brought on by intermittent community points.

MVCC_READ_CONFLICT errors

Learn-write units are generated by the peer when a transaction is submitted to a peer. This learn/write set is then used when the transaction is dedicated to the ledger. It comprises the identify of the variables to be learn/written and their model once they have been learn.

Downside abstract

Each peer within the community (VSCC) validates the variety of signed proposals and the model of each learn key within the read-write units towards the world state upon receiving the blocks from the ordering service.

If, in the course of the time between set creation and committing, a unique transaction was dedicated and altered the model of the important thing within the peer’s present world state, then the unique transaction is rejected throughout committal as a result of the model when learn is just not the present model. This error is often seen throughout commit.

Error message:

Peerpeer1.org.com:7051 has rejected transaction "c91172484bad08eaae2595522a0a8c0a30891b4a90110e0a4fc490c0aacdb399" with code "MVCC_READ_CONFLICT" , validationCode=11

Really useful dealing with

1. To deal with this situation, on the design stage, you may create knowledge and transaction constructions that keep away from modifying the identical key concurrently. Check out the Hyperledger Material samples for an instance of how to do that.

2. The appliance must keep away from key collisions as a lot as doable and would possibly want to put in writing retry logic on the shopper facet. The retry logic ought to question the newest state of the important thing from the ledger and apply the required modifications on the newest state.

Consumer-side retry dealing with

If(validationCode == 11) {
    Object a = // question the ledger once more for the newest state.the place a being an object on the ledger
    //carry out the writes on  a
    a.setAction() }

In a typical blockchain transaction circulation, when functions are registered to be notified when a block is dedicated to the ledger of a peer, when the transaction fails, the notification sometimes has VALIDATION_CODE: 11, which signifies MVCC_READ_CONFLICT.

You can even contemplate VALIDATION_CODE: 12 that signifies PHANTOM_READ_CONFLICT below the identical class.

A pattern blockchain occasion:

 [eventTransactionId: f948056aa59d42810d3318dc1df7643152e220d3bb75924d5228a5e9efb95017, ,status: failure,eventName: xxxx,actionType: xxxx,errorCode: VALIDATION_CODE: 11,errorMessage:null,validationCode:11,channel: xxxx,blockNum: 24712079]

The blockchain occasion contains the transaction ID that was submitted and the small print indicating whether or not it was dedicated on the blockchain or not. The standing: failure signifies the block was not dedicated and validationCode signifies the rationale for failure. On this instance, VALIDATION_CODE:11 signifies there was MVCC_READ_CONFLICT. VALIDATION_CODE: 12 indicating PHANTOM_READ_CONFLICT additionally falls in the identical class.

Pattern retry code:

If ( blockChainNotificationObject.getValidationCode() == 11 || blockChainNotificationObject.getValidationCode == 12 ){
              1.Question Ledger to get the newest state of the Object
     2. if the state is already carried out/utilized
                 {//in all probability parallel duplicate invocation
                   2.1 Think about the occasion as success  and proceed the applying logic  }
          3.else{  apply the required modifications on the newest state and retry the Transaction }}

Peer lag errors

Community delays can usually result in the friends being out of sync the place one peer could be nonetheless catching up with the newest block. So, any queries made on the lagging peer would return an outdated state that may not be legitimate for the applying.

Downside abstract

P1 and P2 are two friends, and K1 is the brand new key added to the ledger. Due to community latency, K1 is just not but added to P2’s ledger. The appliance has queried for K1 on P2, which might obtain a null worth.

Really useful dealing with

It is vital that you just program the applying to establish and deal with inconsistent knowledge brought on by peer lag. Purposes ought to have a mechanism to hear to dam addition occasions and confirm that the block has been added efficiently. Within the case of inconsistent behaviour or null values returned whereas querying, then you may implement retry logic to establish the state replace. A retry with exponential backoff offers the time for friends to beat the community points/delay and sync up on the state.

Endorsement coverage failures

All transactions must be endorsed by the endorsing friends within the execution part (second step) of a blockchain transaction circulation. The endorsement of transactions can fail for a number of causes, corresponding to invalid endorser signatures or different technical causes. Many of the doable causes for endorsement coverage failures are due to misconfigurations or transient world state inconsistencies between the friends. The important thing-value retailer, which maintains the world state, is up to date by every peer independently within the validation part. Due to this fact, transient world state inconsistencies between the friends are doable. On the identical time, the endorsing friends use the world state to generate learn/write units within the execution part. Thus, the world state inconsistencies result in a learn/write set mismatch within the endorsement response inflicting an endorsement coverage failure of the transaction.

Really useful dealing with

• Within the case of endorsement coverage failures as a consequence of misconfigurations, the it’s worthwhile to validate the configurations and proper them.
• Purposes can have resiliency and retry logic in place and check out fetching endorsements from each peer in a corporation earlier than giving up.
• The appliance can change the endorsement coverage, primarily based on its logic. To alter the endorsement coverage, you may specify Channel/Utility/Endorsement in configtx.yaml to ANY endorsement. Material makes use of this configuration because the default endorsement coverage in all chaincode.
• Within the case of the world state inconsistencies, it’s endorsed to have shopper facet retry logic as mentioned within the MVCC_READ_CONFLICT errors part.

Chaincode errors

Chaincode to chaincode communication error

There are functions the place chaincode enterprise logic would require communication with different chaincode to implement a rule or logic.

Downside abstract

Chaincode to chaincode communication can fail due to a number of causes, corresponding to unavailability of explicit chaincode on a given channel or the dependent chaincode not being prepared to just accept requests. A few of the errors fall below nonretryable errors like INVALID CHAINOCDE NAME, INVALID CHAINOCDE VERION, INVALID ARGUMENTS, and CHAINCODE_UNAVAILABLE. You must deal with different chaincode to chaincode communication errors by having retry logic on the shopper facet.

Error message:

Error: INVOKE_CHAINCODE failed: transaction ID: f6ab6dbd747ddf25ebfe158eea5a9b0b7478d6a031b66a08a4f3c2f02fe2f7fe: can not retrieve package deal for chaincode take a look at/1.0, error open /var/hyperledger/manufacturing/chaincodes/take a look at.1.0: no such file or listing"

This message signifies that the chaincode container is unavailable. Nonetheless, you probably have ascertained that the chaincode is put in and instantiated on the friends and is barely unavailable due to the dependent container startup delay, then you may implement personalized dealing with on the utility resiliency layer to get well the transient failure.

Really useful dealing with

To deal with chaincode to chaincode communication errors, distinctive error codes might be propagated by the caller chaincode and the shopper utility might be programmed to establish and categorise these as retriable errors primarily based on the identical. You possibly can retry with exponential backoff for restoration.

No LedgerContext discovered error
A few of the errors should not thrown as a part of the validation part VSCC however happen in the course of the execution part. There might be a number of causes for these errors, and so they usually occur if any of the operations on the ledger are taking longer than the anticipated time.

Error message:

ERRO 09f [ddc81d1b] Did not deal with PUT_STATE. error: no ledger context runtime.goexit /choose/go/src/runtime/asm_amd64.s:1333 PUT_STATE failed: transaction ID: ddc81d1bcb69eecd6c6bbcf85ba16b2168486d4b232ef3c03fe5bbc7bb2adea1 github.com/hyperledger/cloth/core/chaincode. runtime.goexit

Really useful dealing with

These error situations might be dealt with in chaincode whereby a singular error code might be propagated by the chaincode and the shopper utility might be programmed to establish and categorise these as retriable errors primarily based on the identical. You possibly can retry with exponential backoff for restoration.

ValidationCode=17 and ValidationCode=18 errors

A few of the chaincode invocation can fail with error validationCode=17 or validationCode=18. validationCode=17 signifies EXPIRED_CHAINCODE and validationCode=18 signifies CHAINCODE_VERSION_CONFLICT.

Downside abstract

The transaction submitted on an older chaincode container that later expired due to the supply of a brand new container; in the course of the validation part (VSCC), the transaction will get rejected.

Error message:

Nov 22 16:32:52  peerxxxxxxx-r2yyyy  peer  2021-11-22 11:02:52.180 UTC [valimpl] preprocessProtoBlock -> WARN fdd59 Channel [XXXXX]: Block [22596691] Transaction index [3] TxId [2224af5907927c60fad8b39f82537620b89d438ee5c44229533a3779985b833f] marked as invalid by committer. Purpose code [EXPIRED_CHAINCODE]

Really useful dealing with

You possibly can program the shopper to establish and categorise these as retryable errors. You possibly can retry with exponential backoff for restoration.

Abstract

These are a few of the main varieties of errors that you just would possibly run into in any blockchain-based utility. By understanding these factors of failure forward of time, you may have the correct configurations, error dealing with, and restoration methods in place from the design part to deployment. That is essential to constructing a resilient structure.