go modules for fabric #954

MHBauer · 2020-03-31T04:07:52Z

wip

Type of change

Improvement - Infrastructure/Build
~~check_deps.sh checking implemented with modules base~~ don't care
Tools are installed into $(go env GOPATH)/bin. That's problematic in cases where $GOPATH is really a path or when $GOPATH is not on the $PATH.
more?

Description

Would like to use some of fabric as a dependency for golang plugins.

Additional details

pkcs11 unit-tests always fail for me. On master and this branch.
idemix integration-tests always fail for me. On master and this
branch.

Completed work:

internal dependency on gopath Prepare for possible move to go modules #970
more gopath Simplify test assets and setup for go chaincode #1077
tools using modules build: tools as module dependencies (except dep) #1057
gossip race detector fails https://github.com/hyperledger/fabric/pull/1069/files
code using modules Move to Go modules #1064
document changes to developer experience Move to Go modules #1064

Related issues

https://jira.hyperledger.org/browse/FAB-10562

A tools.go based dependency management.
https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module

Makefile

scripts/check_deps.sh

go.mod

integration/nwo/components.go

scripts/check_deps.sh

gotools.mk

MHBauer · 2020-03-31T17:47:50Z

Option 2 worked locally.

2Qs:

Is there a way to speed up the build? I see some stuff in the azure config + the run script, but I'm not sure if I can use it locally.
Is there a way to focus on a specific integration test? It doesn't seem available in the script, so I've been editing the grep line locally.

MHBauer · 2020-03-31T18:53:23Z

added gossip flake to https://jira.hyperledger.org/browse/FAB-17451

/ci-run

github-actions · 2020-03-31T18:53:52Z

AZP build triggered!

sykesm

I understand it's a WIP but it should not be merged. There a number of hacks in here that have some unsavory side effects.

I've dusted off my previous work and I think (go figure) it's a better path forward.

https://github.com/sykesm/fabric/commits/modules-playground-2

integration/externalbuilders/golang/bin/build

scripts/check_deps.sh

integration/nwo/components.go

integration/discovery/discovery_test.go

Makefile

go.mod

MHBauer

Thanks for review @sykesm
What's left in https://github.com/sykesm/fabric/commits/modules-playground-2
I'll give it a run and see what shakes out.

integration/externalbuilders/golang/bin/build

go.mod

scripts/check_deps.sh

integration/nwo/components.go

integration/discovery/discovery_test.go

sykesm · 2020-03-31T20:56:59Z

What's left in https://github.com/sykesm/fabric/commits/modules-playground-2

There's no implementation in check_deps.sh
Tools are installed into $(go env GOPATH)/bin. That's problematic in cases where $GOPATH is really a path or when $GOPATH is not on the $PATH.
Miscellaneous doc updates
Maybe more ...

sykesm · 2020-04-01T15:07:58Z

@MHBauer - Some of the prep changes have been merged into master. Are you planning to continue this thread by implementing some of the remaining items I enumerated above?

Also, with respect to the vendor vs. non-vendor: You're assuming that the modules required for a fabric build will be cached. While that's true for most local development, it's not generally true. For example, our docker images are built within an alpine container that has no cache. That means each time we build a binary, we go off to get all of the packages we need for build. The time it takes can vary widely depending on where you are in the world and what the network looks like.

I'd also assert that vendored packages help when doing code-scans and reproducible builds when building with -mod=vendor as (almost) all of the source comes from our tree or the go standard library.

MHBauer

flake, but the dependencies look weird to me from the tools update.

MHBauer · 2020-04-06T19:53:35Z

go.mod

+	go.uber.org/atomic v1.4.0 // indirect
+	go.uber.org/multierr v1.2.0 // indirect
+	go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee // indirect
+	go.uber.org/zap v1.11.0


especially weird that this would go backwards.

Did you look at sykesm@1493171 ?

Part of https://github.com/sykesm/fabric/commits/modules-playground-4

ah, you did the tools first. I can slap the third commit for the verify on top of those two and give it a go.

I'm worried about what happens in a clean environment in terms of it changing go.mod when building the tools.

I had a CRLF git issue with
vendor/github.com/Knetic/govaluate/.gitignore
but otherwise got it cleanly building.

Looks to work in the build as well, besides some gossip flaknes: https://dev.azure.com/mbauer0254/fabric/_build/results?buildId=4&view=results

I'll push a modified rebase with everything, and you can check it out or modify it yourself on my fork, or push your own. Let me know what you want to do.

MHBauer · 2020-04-07T02:23:34Z

tools/go.mod

+	github.com/vektra/mockery v0.0.0-20181123154057-e78b021dcbb5
+	golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f
+	golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4
+)


oh I get it now, a separate module for the tools.

MHBauer · 2020-04-07T02:24:44Z

tools/tools.go

+	_ "github.com/vektra/mockery/cmd/mockery"
+	_ "golang.org/x/lint/golint"
+	_ "golang.org/x/tools/cmd/goimports"
+)


is this going to need to be vendored as well?

MHBauer · 2020-04-07T17:13:45Z

I can repro a bunch of other flakes, but not this one.

/ci-run

github-actions · 2020-04-07T17:14:24Z

AZP build triggered!

Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com> Change-Id: I6e0b4666e12aec65ed9e3f2f6b871cd942ca66d9

original has crlf Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com> Change-Id: I2212e14f574d9e2c3d870bd718aee50ebcf1fc85

Change-Id: Iad450b21a63156a73b4299fbc22745cd2fe96e1f Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>

sykesm · 2020-04-10T14:59:14Z

@MHBauer tools as module looks to have been merged into master in #1057. One more step forward.

MHBauer · 2020-04-15T21:14:55Z

Looks mostly superceded by #1064, but it doesn't look like we actually care about replicating check-deps.

MHBauer requested a review from a team as a code owner March 31, 2020 04:07

This comment has been minimized.

Sign in to view

MHBauer commented Mar 31, 2020

View reviewed changes

Makefile Outdated Show resolved Hide resolved

scripts/check_deps.sh Show resolved Hide resolved

go.mod Show resolved Hide resolved

integration/nwo/components.go Outdated Show resolved Hide resolved

scripts/check_deps.sh Show resolved Hide resolved

gotools.mk Show resolved Hide resolved

MHBauer changed the title ~~wip - checking server build~~ wip - go modules (please glance at to see if you have any ideas) Mar 31, 2020

This comment has been minimized.

Sign in to view

MHBauer mentioned this pull request Mar 31, 2020

don't wait to expire dead endpoints if exiting #964

Merged

MHBauer changed the title ~~wip - go modules (please glance at to see if you have any ideas)~~ go modules for fabric Mar 31, 2020

sykesm suggested changes Mar 31, 2020

View reviewed changes

MHBauer commented Mar 31, 2020

View reviewed changes

MHBauer commented Apr 6, 2020

View reviewed changes

MHBauer commented Apr 7, 2020

View reviewed changes

sykesm and others added 3 commits April 8, 2020 10:03

WIP - tools as modules (except dep)

724390c

Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com> Change-Id: I6e0b4666e12aec65ed9e3f2f6b871cd942ca66d9

WIP - switch to modules

56a5642

original has crlf Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com> Change-Id: I2212e14f574d9e2c3d870bd718aee50ebcf1fc85

check-deps replacement

a9f4ddc

Change-Id: Iad450b21a63156a73b4299fbc22745cd2fe96e1f Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>

MHBauer closed this Apr 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

go modules for fabric #954

go modules for fabric #954

MHBauer commented Mar 31, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

MHBauer commented Mar 31, 2020

MHBauer commented Mar 31, 2020

github-actions bot commented Mar 31, 2020

sykesm left a comment

MHBauer left a comment

sykesm commented Mar 31, 2020

sykesm commented Apr 1, 2020 •

edited

Loading

MHBauer left a comment

MHBauer Apr 6, 2020

sykesm Apr 6, 2020 •

edited

Loading

MHBauer Apr 7, 2020

MHBauer Apr 7, 2020

MHBauer Apr 7, 2020

MHBauer Apr 7, 2020

MHBauer commented Apr 7, 2020

github-actions bot commented Apr 7, 2020

sykesm commented Apr 10, 2020

MHBauer commented Apr 15, 2020

go modules for fabric #954

go modules for fabric #954

Conversation

MHBauer commented Mar 31, 2020 • edited Loading

Type of change

Description

Additional details

Related issues

This comment has been minimized.

This comment has been minimized.

MHBauer commented Mar 31, 2020

MHBauer commented Mar 31, 2020

github-actions bot commented Mar 31, 2020

sykesm left a comment

Choose a reason for hiding this comment

MHBauer left a comment

Choose a reason for hiding this comment

sykesm commented Mar 31, 2020

sykesm commented Apr 1, 2020 • edited Loading

MHBauer left a comment

Choose a reason for hiding this comment

MHBauer Apr 6, 2020

Choose a reason for hiding this comment

sykesm Apr 6, 2020 • edited Loading

Choose a reason for hiding this comment

MHBauer Apr 7, 2020

Choose a reason for hiding this comment

MHBauer Apr 7, 2020

Choose a reason for hiding this comment

MHBauer Apr 7, 2020

Choose a reason for hiding this comment

MHBauer Apr 7, 2020

Choose a reason for hiding this comment

MHBauer commented Apr 7, 2020

github-actions bot commented Apr 7, 2020

sykesm commented Apr 10, 2020

MHBauer commented Apr 15, 2020

MHBauer commented Mar 31, 2020 •

edited

Loading

sykesm commented Apr 1, 2020 •

edited

Loading

sykesm Apr 6, 2020 •

edited

Loading