Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

orderer crashes down after receiving fuzzed channel tx. #286

Open
SecTechTool opened this issue Nov 7, 2022 · 3 comments
Open

orderer crashes down after receiving fuzzed channel tx. #286

SecTechTool opened this issue Nov 7, 2022 · 3 comments

Comments

@SecTechTool
Copy link

System information:
version: fabric 2.3
OS: ubuntu 20.04

Experiments setup:
Using first-network in https://github.com/SmartBFT-Go/fabric-samples. Repeatedly sending fuzzed channel tx with the same Channel name.

Expected behavior:
Orderers run as usual.

Actual behaviour:
orderers crash down, and can not be recovered.

Bug location:
https://github.com/SmartBFT-Go/fabric/blob/release-2.3-bft/orderer/common/multichannel/registrar.go

// CreateChain makes the Registrar create a consensus.Chain with the given name.
func (r *Registrar) CreateChain(chainName string) {
	lf, err := r.ledgerFactory.GetOrCreate(chainName)
	if err != nil {
		logger.Panicf("Failed obtaining ledger factory for %s: %v", chainName, err)
	}
	chain := r.GetChain(chainName)
	if chain != nil {
		logger.Infof("A chain of type %T for channel %s already exists. "+
			"Halting it.", chain.Chain, chainName)
		chain.Halt()
	}
	r.newChain(configTx(lf))
}

In line 529, should not panic directly.

log info:

2022-11-06 03:12:09.321 UTC [orderer.commmon.multichannel] createNewChain -> PANI 028 Error creating chain support: error creating consenter for channel: fuzztest: failed to restore persisted raft data: failed to create or read WAL: failed to open WAL: fileutil: file already locked
panic: Error creating chain support: error creating consenter for channel: fuzztest:  failed to create or read WAL: failed to open WAL: fileutil: file already locked

goroutine 118 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00002c000, 0x0, 0x0, 0x0)
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x545

@yacovm
Copy link

yacovm commented Nov 9, 2022

I remember that to create a channel you need to sign the transaction by enough administrators in the network.
I think you can restrict it to a threshold of a majority, so I am not sure this is possible in a production network.

In any case, can you check the same problem on the official Fabric with Raft?

@SecTechTool
Copy link
Author

Sorry for the delayed response.

In the oficial Fabric with Raft, there are a locking mechanism in function 'CreateChain' and a exsiting name check in function 'newChain' to prevent this problem.
The detailed code is in :
hyperledger@9da0410

func (r *Registrar) CreateChain(chainName string) {
	lf, err := r.ledgerFactory.GetOrCreate(chainName)
	if err != nil {
		logger.Panicf("Failed obtaining ledger factory for %s: %v", chainName, err)
	}
	chain := r.GetChain(chainName)
	if chain != nil {
		logger.Infof("A chain of type %T for channel %s already exists. "+
			"Halting it.", chain.Chain, chainName)
		r.lock.Lock()
		chain.Halt()
		delete(r.chains, chainName)
		r.lock.Unlock()
	}
	r.newChain(configTx(lf))
}
func (r *Registrar) newChain(configtx *cb.Envelope) {
	r.lock.Lock()
	defer r.lock.Unlock()

	channelName, err := channelNameFromConfigTx(configtx)
	if err != nil {
		logger.Warnf("Failed extracting channel name: %v", err)
		return
	}

	// fixes https://github.com/hyperledger/fabric/issues/2931
	if existingChain, exists := r.chains[channelName]; exists {
		if _, isRaftChain := existingChain.Chain.(*etcdraft.Chain); isRaftChain {
			logger.Infof("Channel %s already created, skipping its creation", channelName)
			return
		}
	}

	cs := r.createNewChain(configtx)
	cs.start()
	logger.Infof("Created and started new channel %s", cs.ChannelID())
}

Maybe similar mechnism should be added here?

@yacovm
Copy link

yacovm commented Nov 11, 2022

That's why I asked you to check in Fabric... I know this bug.

hyperledger#2934

Do you want to push a fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants