BFT Block Puller: test byzantine behavior of the orderer during delivery against the peer

[tock-ibm]

When running a BFT ordering service the peer is pulling blocks from the ordering service using the BFT Deliverer that protects against censorship attacks of the orderer that is asked to deliver blocks.

In this task we have to create an integration test environment that emulates the behavior of a malicious orderer. We that have to emulate malicious + faulty behavior against a peer pulling blocks.

Scenario 1 - censorship detection

• an orderer O1 that is asked to deliver blocks, but delays delivering a block n
• other orderers (e.g. O2,3,4) deliver block attestation n, it is picked up by the header receivers and the censorship monitor
• after a BFTCensorshipTimeout expires, the monitor declares a censorship alert on O1, disconnects from it, and chooses a new orderer from (e.g. O2,3,4) to ask blocks from.

Scenario 2 - censorship suspicion that resets

• an orderer O1 that is asked to deliver blocks delays delivering a block n
• other orderers (e.g. O2,3,4) deliver block attestation n, it is picked up by the header receivers and the censorship monitor
• just before BFTCensorshipTimeout expires, O1 delivers the block n. The monitor resets the timeout.

Scenario 3 - bad blocks and comm problems

• an orderer O1 that is asked to deliver blocks
• the orderer sends a bad block: bad signature | bad content
• the BFTDeliverer disconnects from O1, and chooses a new orderer from (e.g. O2,3,4) to ask blocks from.

Scenario 4 - bad attestations

• an orderer O1 that is asked to deliver blocks, works ok
• other orderers (e.g. O2,3,4) deliver block attestation n, but are causing problems: some disconnect, some bad blocks, some bad signatures, etc.
• errors in header receivers connected orderers (e.g. O2,3,4) delivering block attestation are logged but ignored, and the receivers are relaunched (with a backoff interval).

These scenarios are tested in the unit tests of the BFTDeliverer and the CensorshipMonitor, now we want to test some of these basic scenarios in the integration tests.

The challenge here is to emulate the behavior of a byzantine orderer.

• Start a network with 4 BFT orderers 2 orgs and 4 peers
• In the genesis block, the orderer endpoints (delivery service address as the peer sees it) is set to different addresses then what the orderers actually bind too.
• the peers cannot connect to orderers.
• feed the orderers with TXs to create soem blocks (~100)
• the TXs can be junk, that will cause them to be invalid when committed by the peers, but we don't care.
• stop one orderer and copy its ledger to a side location.
• write a new test service that mimics the orderer's side of the delivery service - "orderer facade".
• the facade access the copied ledger.
• the facade accepts the SeekInfo request and responds as the orderer does, delivering the blocks that were created earlier.
• the facade has a test API that allows the test to control the behavior - start/ stop/ pause/ deliver until /etc
• use these controls to emulate the scenarios above
• start a facade for each orderer O1-4

Stages:

• Create the ledger: start an ordering service, submit TXs to make blocks, shut down, copy the ledger
• Mock process: start N mock processes that load the ledger
• Test the mocks can access the ledger
• Delivery service server side: add the delivery gRPC service to the mock, delivers block correctly
• Connect peer to mock, consume (pull) blocks correctly
  Byzantine behavior:
• Scenario 1 - Add byzantine behavior to f mocks, check peer pulls blocks correctly
• Scenario 1 - Style improvements BFT: Block deliverer integration test style #4724
• Scenario 2
• Scenario 3
• Scenario 4

[tock-ibm]

@semil and @arkadiPiven

[tock-ibm]

The delivery service server handlers are here:
common/deliver/deliver.go

func (h *Handler) Handle(ctx context.Context, srv *Server) error

instatiated here:
orderer/common/server/server.go

[tock-ibm]

Continue from PR #4402

[arkadiPiven]

Added the test PR: #4542

[tock-ibm]

See also #4724