Bundle a default k8s Chaincode Builder into the peer

[jkneubuh]

As of Fabric > 2.4.1 the peer image contains the default ccaas external chaincode builder to simplify networks relying on the Chaincode as a Service pattern. This builder allows the user to bypass the normal chaincode lifecycle events, launching an external process / URL to receive chaincode invocations from the peer to a known service URL.

The "as a service" deployment provides full flexibility to the administrators on how, where, and when the chaincode systems will be launched. In interactive development, such as in a local debugging context, the flexibility is invaluable. But in post-development workflows, however, the added flexibility becomes a real challenge for Fabric administration, as the service lifecycle is now intertwined with the (already) complicated chaincode lifecycle managed by the peer, channel, and consortium.

On the 5/11 Fabric Community Contributor call, @jt-nti presented a new technique for managing chaincode deployments to greatly simplify the overall process of managing chaincode in cloud native environments.

A New Course:

• Chaincode compilation is performed outside of Fabric. (e.g. local builds, CI pipelines, public repos, etc.)

• An external fabric-builder-k8s is responsible for receiving and responding to lifecycle events from the peer.

• fabric-builder-k8s is responsible for managing the lifecycle of chaincode pods running in Kubernetes.

Using this hybrid approach, chaincode developers can build / test / edit routines locally, publish to a container registry, relying on the natural chaincode lifecycle for installing smart contracts on a channel. In tight build/edit/test iterations, development can occur in a debugger using CCaaS bound to a port on the host system.

On the Horizon

• Compile-time feedback : Trap CC build/compilation errors at build, not at run/deployment time.

• No requirements for DIND, docker, or chaincode builds in the network : You build it - Fabric will run it.

• Instant chaincode : Chaincode launch times measured in seconds, not minutes

• Goodbye, Docker! No more DIND, root privilege escalation, mobyd, etc.

• Dude, I just want to write some chaincode...

Compass Bearing

While working with external builders is possible in Fabric, it's still a tremendous challenge to actually install external builders in cloud-native environments.

Address this by:

1. Shore up / battle harden / batten down the hatches / etc. ... the compass bearing set by fabric-builder-k8s

2. Add support for imagePullSecret and imagePullPolicy attributes to the cc package json / metadata.

3. Include the image:label style syntax to reference containers in the cc package json / metadata.

4. Identify a technique to extend mTLS by default (or possibility) in the cc package json / metadata.

5. Build and distribute a golang-based default fabric-builder-k8s, adjacent to ccaas_builder on the peer Docker image and core yaml.

6. Document the overall approach, including a section or guide on the public docs site.

Charts and Maps

• fabric-builder-k8s : functional prototype - works with kube test network and nano test network

• Kube test network chaincode.sh : "Externally launching k8s resources"

• Fabric Community Contributor Meeting - 5/11 : (Chaincode / k8s discussions start ~ 00:11:00)

• Debugging Smart Contracts with Hyperledger Fabric on Kubernetes

Ahoy!

[jkneubuh]

cc: @denyeart @lehors @jt-nti @mbwhite @SamYuan1990

[yacovm]

Identify a technique to extend mTLS by default (or possibility) in the cc package json / metadata.

Can you elaborate? Why is this needed?

[yacovm]

While working with external builders is possible in Fabric, it's still a tremendous challenge to actually install external builders in cloud-native environments.

Why? The peer only cares about where the chaincode is running. Just deploy the chaincode and take care the packets find their way to it, and that the peer can authenticate the chaincode's TLS certificate. What's the difficulty?

Add support for imagePullSecret and imagePullPolicy attributes to the cc package json / metadata.

Include the image:label style syntax to reference containers in the cc package json / metadata.

Why would the Fabric peer need to know details about where or how the chaincode is deployed?

[mbwhite]

@jkneubuh @jt-nti thoughts that spring to mind..

• For any organisation, the same package needs to be installed on all that org's peers. Therefore the ccaas-builder used the concept of go-templates to let each peer address a different running chaincode. This isn't needed here as this each peer will end up its own chaincode pod running, that talks back to the peer.
• Lifecycle of the chaincode... problem is that the fabric doesn't really have the concept of shutdown of a chaincode when it's replaced with a new version. There's the lifecycle of a version of the chaincode definition and the lifecycle of the actual running process. Fabric should have knowledge of the first, it should care about the later, but let tools that are designed for this purpose (k8s) deal with it.

[mbwhite]

@yacovm just to your point of 'why and what's the difficulty'. You are correct, in theory the peer really only needs to know the address of where to talk to the chaincode. How packets get from A <-> B is really not an issue and has been solved.

The peer though, in practice, really takes an active interest in the details of standing up chaincodes. The as-a-server approach helps to break this - but not entirely.. from talking to @jt-nti there are updates to the builder lifecycle that would help - and remove from the peer the knowledge of the chaincode deployment.

The question is though - what elements of a chaincode define it's identity? To one organizations peers, how is 'equality' defined? And between organizations how is a chaincode defined? PackageID is only between one organization... so on a channel I think the identification is comprised of a string name, string version, montonically increase sequence

[mbwhite]

On a pragmatic point @jkneubuh when adding the makefiles for the ccaas builders, the makefiles in the repo are very sophisticated, I'm afraid to the point that it's taken a long time for me to get even a an approximate working knowledge.

The first iteration was very wrong; the second is still in PR but hopefully the 'correct way' to fit in with the overall makefile.

The pipeline as for release of Fabric binaries is little 'odd' in that on the release build, what is built and tested from this repo is not what is published. Fabric-test rebuilds this repo and publishes it directly. Not ideal.

I do wonder if it's worth reviewing the pipelines; many projects produce flavours of the docker images.

[jt-nti]

Shore up / battle harden / batten down the hatches / etc. ... the compass bearing set by fabric-builder-k8s

There are some issues on that repo for hatch battening, and I'll be adding more, if anyone is interested in helping with that effort.

Add support for imagePullSecret and imagePullPolicy attributes to the cc package json / metadata.

I've created an issue to keep track of this one hyperledger-labs/fabric-builder-k8s#19

Include the image:label style syntax to reference containers in the cc package json / metadata.

This is handled with the image.json file in the k8s builder, which follows the established pattern for builders and launchers. (I don't think we should be extending matadata.json with builder specific data but yell if there's a better name than image.json!)

[yacovm]

but not entirely.. from talking to @jt-nti there are updates to the builder lifecycle that would help - and remove from the peer the knowledge of the chaincode deployment.

But why would the peer take part of the chaincode deployment in the chaincode-as-a-service model?? The chaincode needs to be deployed and managed externally by the organization.

The question is though - what elements of a chaincode define it's identity? To one organizations peers, how is 'equality' defined? And between organizations how is a chaincode defined? PackageID is only between one organization... so on a channel I think the identification is comprised of a string name, string version, montonically increase sequence

Across organizations, all we need is a way to mark which namespaces the transaction refers to. That's it.
We have the name to mark the namespace, and the version and sequence - is used to prevent pre-computation attacks where you endorsed a blind write in the past from an earlier version of the chaincode but you never sent the transaction.

[SamYuan1990]

hi guys,

In my point of view
Please allow me sync with your guys with background for this discussion to make sure we are on the same page.

1. For communication between peer and chaincode, the point is about chaincode service URL.
   assuming chaincode.somewhere.com it can be a service at same k8s cluster or some domain name which peer can access is enough.
   when a traffic happens, the routing is by ... network routing table.
2. For chaincode runtime.
   The chaincode runtime, or say the how chaincode.somewhere.com deal with traffic.
   it's some running code as a PID or some debugging mode in IDE.
3. Why peer need to build chaincode?
   Can we have a fabric chaincode builder project to build fabric chaincode from source to image.
   I don't have time to go through https://github.com/hyperledgendary/fabric-builder-k8s, but it seems as a k8s internal registry is out of scope here.
   even if we can away DIND in peer(it's cool) but we can't away from container technological, at least we will implmentat source to image.
   So that k8s can start the chaincode from source code to a running service.
4. Here comes k8s
   when we talk about chaincode as a service.
   which means, the chaincode run in a pod, expose service ... as a container. the image can be generated by any kind of conatiner technology, docker, podman, etc.
5. CCAAS Builders
   We already have a CCAAS Builders
6. Do we need bundle a default k8s Chaincode Builder?
   Well, if we consider to provide a default builder, it sounds good as user friendly.
   meanwhile user is able to use their own CCAAS Builders right?

Questions

1. What's the difference between current CCAAS Builders and https://github.com/hyperledgendary/fabric-builder-k8s?
   I am not sure if I get the point, but at user point of view, if either current CCAAS Builders or https://github.com/hyperledgendary/fabric-builder-k8s provide same feature/service to fabric user... they are the same, but different way of implementaion.

Comments

1. Bundle a default k8s Chaincode Builder into the peer
   sorry for finally back to the topic, as this title has a lot of pre conditions.

• k8s
• source to image
• image registry
• default CCAAS Builders
• peer has a default CCAAS Builders
  ...
  if all questions above are correct, then we should make it.

1. Are we able to swith our sight from another point of view?

• user need build image for chaincdoe from source.
  Here is where we hope peer decouple with DIND. Will CCAAS Builders cover this part?

• user need a builder to help user deploy chaincdoe as service.
  Sounds like deploy a normal service on k8s?

• user need make peer know where is the chaincode endpoint/uri.
  I am not sure current connection.json and metadata.json is too diffcult or not...
  sound like it just contains necessary information as sample below right?

{
  "address": "org1-cc-nftsample:9999",
  "dial_timeout": "10s",
  "tls_required": false
}

{
  "type": "external",
  "label": "nft_1.0"
}

Thanks and Regards
Sam

[jt-nti]

Hi @SamYuan1990

Why peer need to build chaincode?
Can we have a fabric chaincode builder project to build fabric chaincode from source to image.
I don't have time to go through https://github.com/hyperledgendary/fabric-builder-k8s, but it seems as a k8s internal registry is out of scope here.
even if we can away DIND in peer(it's cool) but we can't away from container technological, at least we will implmentat source to image.
So that k8s can start the chaincode from source code to a running service.

I would really like to get away from the peer being responsible for building chaincode. It's actually already possible (and recommended) to provide a prebuilt jar in a Java chaincode package. The prototype k8s builder runs a pre-built/pre-published Docker image, so that these steps can be handled in a traditional CI/CD pipeline where they should be. There's an example of that here -> https://github.com/hyperledgendary/conga-nft-contract

What's the difference between current CCAAS Builders and https://github.com/hyperledgendary/fabric-builder-k8s?

The difference is that the CCaaS builder just tells the peer where there is some chaincode running so that it can connect to it, and you are left to handle starting the chaincode process. (This is the reverse of the traditional approach where the chaincode connects to the peer.) On the other hand the new k8s builder allows the peer to manage the lifecycle of the chaincode seamlessly using kubernetes, similar to what happens when using the existing Go, Java and Node.js chaincode packages (although these don't use the builder framework).

The advantages of the CCaaS builder are that it does not specify how you run the chaincode- you can use kubernetes but you don't have to. It's also really nice when developing chaincode because you can start it, update it, debug it, etc. all without having to go round the chaincode lifecycle loop again. The disadvantage is that you have to manage running the chaincode process yourself, which is more complicated than it really needs to be, plus the chaincode package says nothing about the actual chaincode implementation, just where it's running.

The potential advantage of a new k8s builder is that deploying chaincode in a kubernetes environment becomes as straightforward as deploying traditional Go, Java, and Node.js packages, with the added advantage that the deploy is more reliable since the build step has been done once up front.

user need build image for chaincdoe from source.
Here is where we hope peer decouple with DIND. Will CCAAS Builders cover this part?

Neither the CCaaS builder or the k8s builder require DIND.

user need a builder to help user deploy chaincdoe as service.
Sounds like deploy a normal service on k8s?

There is already a builder to deploy chaincode as a service, which can be running on k8s, but does not have to.

[jkneubuh]

Hi @yacovm I understand the reluctance to polluting the Fabric core with routines tied to any particular runtime or container orchestration engine (e.g. K8s, Docker Compose, Swarm, Mesos, VMWare, or whatever the current tech trend of the day happens to be...)

Theoretically the external builders provide a solid mechanism for Fabric Administrators to configure a network with binaries specifically tailored to the target environment. In practice we have observed:

• There is no Fabric Administrator. (Or in the cases where one exists, they are typically overloaded with the high initialization burdens of Fabric, or engaged on a short-term contract basis.) We are seeking a solution that solves the "99% use case" by default, and provides extensibility in cases requiring customization by a dedicated admin / systems programming team.

• Chaincode-as-a-Service requires endpoints to be orchestrated in concert with the peer lifecycle events. A successful CC launch is more complicated than "just start the endpoint" - intertwining the CC process lifecycle with the peer CC lifecycle is, in practice, straightforward but error prone:

  • prepare a CC package / archive
  • install the package on a targeted peer
  • generate / infer the CC package ID
  • launch the CC service endpoint with ID in scope
  • approve the chaincode
  • commit the chaincode

• Most critically : Fabric is currently published as a collection of container images. In practice, this makes it quite challenging to distribute external builder binaries onto the file system of the peers . Before the peer can fork/exec a builder, the routine need to be copied onto the peer file system and registered in the core.yaml.

The simple task of copying the k8s external builder into the scope and file system of container-based peers is ... deceptively complicated. Other than pre-bundling default external builder into the peer image at Docker build time, is there another technique that would provide for a more modular, but convenient approach?

The goal here is to make it "just work" for the 99% use case, and provide escape paths for sophisticated deployments requiring advanced configuration. I.e.. a compromise.

[jkneubuh]

Hi @SamYuan1990 - Builder comparison "at a glance" :

• Default Fabric "DIND" Builder :

  • User installs CC package containing Source code
  • Peer compiles CC and builds a Docker image
  • Peer starts the CC process

• External "ccaas" Builder:

  • User installs CC package containing Service URL
  • Peer does NOT compile CC or prepare Docker Image
  • Peer does NOT launch a CC process - (User must start)

• External "k8s" builder:

  • User installs CC package containing Docker Image URL
  • Peer does NOT compile CC or prepare Docker Image
  • Peer starts the CC process / Kubernetes pod

[yacovm]

launch the CC service endpoint with ID in scope

Why does the chaincode shim need to know its package ID?

[yacovm]

Other than pre-bundling default external builder into the peer image at Docker build time, is there another technique that would provide for a more modular, but convenient approach?

So you want to bundle inside the peer's file system, something that can interact with the peer and ask it about lifecycle? Is that why you need the mutual TLS for? To connect to the peer and withdraw the information about lifecycle or something like that?

[SamYuan1990]

• User installs CC package containing Docker Image URL

May I know where the image will be stored?

• Peer does NOT compile CC or prepare Docker Image

I like this

On the other hand the new k8s builder allows the peer to manage the lifecycle of the chaincode seamlessly using kubernetes..

• Peer starts the CC process / Kubernetes pod

according to https://github.com/hyperledgendary/fabric-builder-k8s/blob/main/docs/TEST_NETWORK_K8S.md#kubernetes-permissions
what if user can't accept with this permission?

@jt-nti , @jkneubuh

[jt-nti]

May I know where the image will be stored?

The example is using the GitHub container registry but they could be published anywhere the peer can connect to, or even loaded in a local registry.

what if user can't accept with this permission?

The builder won't need all those permissions- probably just permission to create a pod and secret but not sure yet. The intention will be to require the minimum permissions necessary. The namespace is configurable as well which will hopefully help anyone who still has concerns.

[SamYuan1990]

Hi @SamYuan1990 - Builder comparison "at a glance" :

• Default Fabric "DIND" Builder :

  • User installs CC package containing Source code
  • Peer compiles CC and builds a Docker image
  • Peer starts the CC process

• External "ccaas" Builder:

  • User installs CC package containing Service URL
  • Peer does NOT compile CC or prepare Docker Image
  • Peer does NOT launch a CC process - (User must start)

• External "k8s" builder:

  • User installs CC package containing Docker Image URL
  • Peer does NOT compile CC or prepare Docker Image
  • Peer starts the CC process / Kubernetes pod

wait...
it sounds like a "tekton" job to prebuild chaincode from source to image and deployment on k8s by a template and triggered by Peer?
What if a chaincode operator listening source code repo and a pipeline which triggered by chaincode deployment?

• A CI job to prebuild chaincode from source to image when each commit
• A CD job to deploy image on k8s
  so that

• User installs CC package containing Service URL / Docker Image URL
• Peer does NOT compile CC or prepare Docker Image
• Peer does NOT launch a CC process

and the operator is make since to have some additional k8s permission and user is able to use it or not? and avoid permission issue?

And we can make "ccaas" Builder integrate with the operator, and make new build as default one?

[SamYuan1990]

May I know where the image will be stored?

The example is using the GitHub container registry but they could be published anywhere the peer can connect to, or even loaded in a local registry.

what if user can't accept with this permission?

The builder won't need all those permissions- probably just permission to create a pod and secret but not sure yet. The intention will be to require the minimum permissions necessary. The namespace is configurable as well which will hopefully help anyone who still has concerns.

Which means we extend fabric peer permission with "k8s" builder. And have to make it able to access the local registry with secrets, right?

[jkneubuh]

Hi @SamYuan1990 :

• RE: Tekton pipelines. This is a great example of how custom, external builders can be integrated with Fabric as part of an overall solution approach. Tekton is a viable solution for integrating Fabric CC deployments with customer-specific needs, and will work very well for integrating the chaincode docker build cycle into an overall solution. While Tekton can help automate the overall process, the fact that we need a fully-functional workflow engine just to deploy CCaaS is an indicator of a larger problem - it's still too complicated! (BTW I love the idea of integrating Tekton pipelines into CCaaS development workflows for test, validation, and production deployments -- external builders will enable this as a supplemental feature.)

• RE: image pull permissions. The k8s chaincode package must include a pointer to k8s imagePullSecret references, which must be declared when launching the pod/deployments in k8s.

• RE: k8s launch permissions: Yes. the Kubernetes service account running the peer must be extended an RBAC sufficient for starting pods/deployments/etc. in a target namespace. The approach uses the standard Kubernetes permissions system, a vast improvement over the "run Docker daemon as root" or DIND techniques.

[SamYuan1990]

Let me summarize myself, what if

At peer side

• Peer does NOT compile CC or prepare Docker Image
• Peer does NOT start the CC process / Kubernetes pod to avoid extend k8s permission or any risk with k8s secret access.
• Enhance Peer with or "ccaas" Builder, "DIND" Builder, "k8s" Builder and make it able to config by environment variable.

To provide user friendly, an k8s operator

• User installs CC package containing Docker Image URL/Service URL/Source code link(some Customer resource definition).
• Operator able to create k8s resources with necessary permission.
• Operator able to access github with necessary permission to checkout code.
• Operator response for chaincode lifecycle on peers with in Organization.
• As a sidecar mode interaction with peer, when peer pod been deployed, overwrite peer chaincode builder setting, as an optional approach?

[jkneubuh]

Hi all - I would like to encourage the continued discussions on this topic; In this light, GitHub is not serving well as means to organize our efforts... May I suggest that we transition the discussion aspects over to Discord #fabric-kubernetes for interactive threads, and retain this ticket to summarize the action items necessary for inclusion / distribution of the k8s-builder with Fabric?

[SamYuan1990]

Hi all - I would like to encourage the continued discussions on this topic; In this light, GitHub is not serving well as means to organize our efforts... May I suggest that we transition the discussion aspects over to Discord #fabric-kubernetes for interactive threads, and retain this ticket to summarize the action items necessary for inclusion / distribution of the k8s-builder with Fabric?

it's hard for me to access discord from Beijing... just let me know the key point on github, and I will response.

[yacovm]

Hi all - I would like to encourage the continued discussions on this topic; In this light, GitHub is not serving well as means to organize our efforts... May I suggest that we transition the discussion aspects over to Discord #fabric-kubernetes for interactive threads, and retain this ticket to summarize the action items necessary for inclusion / distribution of the k8s-builder with Fabric?

I was inquiring about whether this approach makes sense, and you're saying that we should dedicate this discussion for action items, under some assumption that what is described here has already been decided to be applied to Fabric core.

[jkneubuh]

@yacovm I am open to altering the design and/or mechanics. I am suggesting:

a) We migrate the interactive discussion towards a different context (e.g. GitHub Discussions, Discord, etc.) to iterate and converge on an approach for:

1. Developing a sustainable approach for running Chaincode Builders in Kube / cloud-native environments.

2. Packaging the approach into the default distribution of Fabric images (this ticket). The current approach involves preparing and bundling the fabric-k8s-builder binaries into the Peer Image. I am absolutely open to alternative designs - the key target is that it should "just work" by default, and provide extension hooks for environments requiring customization.

b) Retain this ticket to track work items related to #2 above.

@SamYuan1990 is a GitHub discussion more suitable for your participation in the conversation thread?

[SamYuan1990]

@jkneubuh , yes, github discussion is more suitable for me... as I am located in China and there is GFW Blocking my access to discord.

[yacovm]

launch the CC service endpoint with ID in scope

Why does the chaincode shim need to know its package ID?

Other than pre-bundling default external builder into the peer image at Docker build time, is there another technique that would provide for a more modular, but convenient approach?

So you want to bundle inside the peer's file system, something that can interact with the peer and ask it about lifecycle? Is that why you need the mutual TLS for? To connect to the peer and withdraw the information about lifecycle or something like that?

[jkneubuh]

All : thanks for the lively debate / discussion. Can we shift over to #3407 to focus on the general discussion of running external builders for k8s.

[jkneubuh]

Per #3407 - summary -- closing this ticket as there is not agreement to bundle a k8s builder into the peer container.