Broken Pipe Error Between Peer and external Chaincode.

[virendar5Kmr]

Description

Hello,

When i try to invoke chaincode with below command. I am getting Broken Pipe error in Peer logs.

When the Peer try to connect with Chaincode, TLS handshake happens and then I get broken pipe error.

The Setup is deployed on OCP cluster (Kuberenetes v1.27). There is no connectivity issues as they are in same network.

I have also created Headless service for chaincode deployment.

ERROR:
2024-07-15 09:04:35.602 UTC [grpc] WarningDepth -> DEBU 38e [core] [Channel #134 SubChannel #135] grpc: addrConn.createTransport failed to connect to {
"Addr": "chaincode1-service:9000",
"ServerName": "chaincode1-service:9000",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 38f [core] [Channel #134 SubChannel #135] Subchannel Connectivity change to TRANSIENT_FAILURE
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 390 [core] pickfirstBalancer: UpdateSubConnState: 0xc00227c140, {TRANSIENT_FAILURE write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe}
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 391 [core] [Channel #134] Channel Connectivity change to TRANSIENT_FAILURE
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 392 [core] [Channel #134] Channel Connectivity change to SHUTDOWN
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 393 [core] [Channel #134 SubChannel #135] Subchannel Connectivity change to SHUTDOWN
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 394 [core] [Channel #134 SubChannel #135] Subchannel deleted
2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 395 [core] [Channel #134] Channel deleted
2024-07-15 09:04:35.602 UTC 38e6 DEBU [chaincode] Launch -> stopping due to error while launching: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe
failed to create new connection
github.com/hyperledger/fabric/internal/pkg/comm.ClientConfig.Dial
/internal/pkg/comm/config.go:151
github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).createConnection
/core/chaincode/extcc/extcc_handler.go:33
github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).Stream
/core/chaincode/extcc/extcc_handler.go:45
github.com/hyperledger/fabric/core/chaincode.(*RuntimeLauncher).Launch.func1
/core/chaincode/runtime_launcher.go:90
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
error creating grpc connection to chaincode1-service:9000
error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5
connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed
2024-07-15 09:04:35.602 UTC 38e7 DEBU [chaincode] Launch -> launch complete
2024-07-15 09:04:35.602 UTC 38e8 DEBU [chaincode] Deregister -> deregister handler: asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5
2024-07-15 09:04:35.602 UTC 38e9 INFO [endorser] callChaincode -> finished chaincode: asset-transfer-basic_9 duration: 5ms channel=mychannel1 txID=ffb61445
2024-07-15 09:04:35.602 UTC 38ea ERRO [endorser] simulateProposal -> failed to invoke chaincode asset-transfer-basic_9, error: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe
failed to create new connection
github.com/hyperledger/fabric/internal/pkg/comm.ClientConfig.Dial
/internal/pkg/comm/config.go:151
github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).createConnection
/core/chaincode/extcc/extcc_handler.go:33
github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).Stream
/core/chaincode/extcc/extcc_handler.go:45
github.com/hyperledger/fabric/core/chaincode.(*RuntimeLauncher).Launch.func1
/core/chaincode/runtime_launcher.go:90
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
error creating grpc connection to chaincode1-service:9000
error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5
connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed
could not launch chaincode asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Launch
/core/chaincode/chaincode_support.go:87
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke
/core/chaincode/chaincode_support.go:203
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
/core/chaincode/chaincode_support.go:161
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
/core/endorser/endorser.go:350
github.com/hyperledger/fabric/internal/pkg/gateway.(*EndorserServerAdapter).ProcessProposal
/internal/pkg/gateway/gateway.go:41
github.com/hyperledger/fabric/internal/pkg/gateway.(*Server).Evaluate.func1
/internal/pkg/gateway/evaluate.go:64
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
failed to execute transaction ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4
github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult
/core/chaincode/chaincode_support.go:167
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
/core/chaincode/chaincode_support.go:162
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
/core/endorser/endorser.go:350
github.com/hyperledger/fabric/internal/pkg/gateway.(*EndorserServerAdapter).ProcessProposal
/internal/pkg/gateway/gateway.go:41
github.com/hyperledger/fabric/internal/pkg/gateway.(*Server).Evaluate.func1
/internal/pkg/gateway/evaluate.go:64
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598 channel=mychannel1 txID=ffb61445
2024-07-15 09:04:35.602 UTC 38eb DEBU [lockbasedtxmgr] Done -> Done with transaction simulation / query execution [ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4]
2024-07-15 09:04:35.602 UTC 38ec WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=mychannel1 chaincode=asset-transfer-basic_9 error="error in simulation: failed to execute transaction ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4: could not launch chaincode asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed: error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: error creating grpc connection to chaincode1-service:9000: failed to create new connection: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe"
2024-07-15 09:04:35.602 UTC 38ed DEBU [gateway] func1 -> Evaluate call to endorser failed chaincode=asset-transfer-basic_9 channel=mychannel1 txID=ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4 endorserAddress=fabric-peer1-service:7061 endorserMspid=Org6MSP error="error in simulation: failed to execute transaction ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4: could not launch chaincode asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed: error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: error creating grpc connection to chaincode1-service:9000: failed to create new connection: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe"
2024-07-15 09:04:35.602 UTC 38ee INFO [comm.grpc.server] 1 -> unary call completed grpc.service=gateway.Gateway grpc.method=Evaluate grpc.request_deadline=2024-07-15T09:06:35.573Z grpc.peer_address=10.244.6.110:45798 error="rpc error: code = Aborted desc = failed to evaluate transaction, see attached details for more info" grpc.code=Aborted grpc.call_duration=7.12576ms
2024-07-15 09:04:35.613 UTC [grpc] InfoDepth -> DEBU 396 [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing

Steps to reproduce

No response

[TomYuHua]

the resason of " write: broken pipe“ is alawys two points:
1.The server's host has exceeded the maximum number of connections.
2.The client disconnects before receiving a response from the server.

[TomYuHua]

For the "Broken Pipe" error, you can take the following solutions:

Gracefully Handle Connection Closures:
When writing network applications, ensure that you can gracefully handle connection closures. For example, catch "Broken Pipe" exceptions when attempting to write data and appropriately clean up resources (such as closing sockets).
Add Retry Logic:
For "Broken Pipe" errors that may be caused by network issues, add retry logic to your application. This can help recover communication after brief network interruptions.
Adjust Timeout and Idle Connection Settings:
If the issue is due to timeouts or idle connection closures, try adjusting relevant system or application settings. For example, increase connection timeout times or disable idle connection closure features.
Check System Resources:
Ensure that system resources (such as the number of file descriptors) are sufficient and have not reached their limits. If resources are insufficient, you may need to increase resource limits or optimize the application's resource usage.
Log and Analyze Logs:
By logging and analyzing the logs of network applications, you can better understand the circumstances and causes of "Broken Pipe" errors. This helps identify and resolve underlying issues.
Use a More Stable Network:
If the issue is due to an unstable network, consider using a more stable network connection or adding network redundancy to improve the reliability of communications.