From 03db51f6e23dc1d9c6f2bfa911d00a16d014ed1c Mon Sep 17 00:00:00 2001 From: Josh Kneubuhl Date: Fri, 19 Nov 2021 14:10:03 -0500 Subject: [PATCH 1/2] Fix Issue #257 by including additional details on Kube Service routing Signed-off-by: Josh Kneubuhl --- docs/source/gateway.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/source/gateway.md b/docs/source/gateway.md index ccf1dcc8b9b..f336bdd7167 100644 --- a/docs/source/gateway.md +++ b/docs/source/gateway.md @@ -90,6 +90,19 @@ The Fabric Gateway `Evaluate` and `Endorse` methods make gRPC requests to peers Each Gateway SDK also provides a mechanism for setting timeouts for each gateway method when invoked from the client application. +## Gateway Service Routing with Kubernetes + +In typical Fabric deployments on Kubernetes, each peer node is exposed via a single Kubernetes `Service` instance and resolved using a Kube DNS. This technique is sufficient for the Fabric Gateway and application clients to resolve individual peers within a Fabric network. In cases requiring high-availability and/or client connection load-balancing, Kubernetes can be configured with an additional `Service` resource bound to multiple gateway peers. Using an HA topology, gateway clients may reference a set of peer nodes with a single DNS alias. + +While [Kubernetes Service routing](https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies) does not provide request-level load balancing at the gRPC message layer, it can be utilized to provide a basic level of HA, failover, and client connection load balancing. In the event of an outage in a backing peer/pod, Kubernetes will assign an active peer at the next connection attempt from the gateway client. + +By default Kubernetes will use `iptables` to bind client connections to a peer pod using random assignment. Pods backing the service may additionally define Readiness Probes to ensure that Gateway client connections are distributed across healthy peer nodes. In addition to random assignment, the Kube cluster may be configured with the [IPVS proxy mode](https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs) for balancing traffic across backend Pods. For additional information on IPVS routing, refer to the Kubernetes [IPVS-based Load Balancing Deep Dive](https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/) blog. + +An example load-balanced gateway service is available in the [Kubernetes Test Network](https://github.com/hyperledger/fabric-samples/blob/main/test-network-k8s/docs/HIGH_AVAILABILITY.md). In this example: +- Each organization defines an `orgN-peer-gateway` `Service`, bound to a set of peer `Deployments`. +- The TLS enrollments / certificates for each peer include the shared service / Subject Alternate Name alias. +- Client applications reference the Fabric Gateway using the `orgN-peer-gateway` service alias. + ## Listening for events The gateway provides a simplified API for client applications to receive [chaincode events](peer_event_services.html#how-to-register-for-events) in the client applications. Each SDK provides a mechanism to handle these events using its language-specific idiom. From 397c5cd71f85cfedaedeccd0759c22462125f3d3 Mon Sep 17 00:00:00 2001 From: Josh Kneubuhl Date: Thu, 16 Dec 2021 14:22:32 -0500 Subject: [PATCH 2/2] Update the Gateway HA / Failover to a Kube Considerations section Signed-off-by: Josh Kneubuhl --- docs/source/deploykube/kubeconsiderations.md | 0 docs/source/deployment_guide_overview.rst | 62 ++++++++++++++++---- docs/source/gateway.md | 13 ---- 3 files changed, 52 insertions(+), 23 deletions(-) create mode 100644 docs/source/deploykube/kubeconsiderations.md diff --git a/docs/source/deploykube/kubeconsiderations.md b/docs/source/deploykube/kubeconsiderations.md new file mode 100644 index 00000000000..e69de29bb2d diff --git a/docs/source/deployment_guide_overview.rst b/docs/source/deployment_guide_overview.rst index 907896f5c68..48c04cd8304 100644 --- a/docs/source/deployment_guide_overview.rst +++ b/docs/source/deployment_guide_overview.rst @@ -69,16 +69,6 @@ However and wherever you choose to deploy your components, you will need to make By deploying a proof of concept network and testing it under load, you will have a better sense of the resources you will require. -Managing your infrastructure -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The exact methods and tools you use to manage your backend will depend on the backend you choose. However, here are some considerations worth noting. - -* Using secret objects to securely store important configuration files in your cluster. For information about Kubernetes secrets, check out `Kubernetes secrets `_. You also have the option to use Hardware Security Modules (HSMs) or encrypted Persistent Volumes (PVs). Along similar lines, after deploying Fabric components, you will likely want to connect to a container on your own backend, for example using a private repo in a service like Docker Hub. In that case, you will need to code the login information in the form of a Kubernetes secret and include it in the YAML file when deploying components. -* Cluster considerations and node sizing. In step 2 above, we discussed a general outline for how to think about the sizings of nodes. Your use case, as well as a robust period of development, is the only way you will truly know how large your peers, ordering nodes, and CAs will need to be. -* How you choose to mount your volumes. It is a best practice to mount the volumes relevant to your nodes external to the place where your nodes are deployed. This will allow you to reference these volumes later on (for example, restarting a node or a container that has crashed) without having to redeploy or regenerate your crypto material. -* How you will monitor your resources. It is critical that you establish a strategy and method for monitoring the resources used by your individual nodes and the resources deployed to your cluster generally. As you join your peers to more channels, you will need likely need to increase its CPU and memory allocation. Similarly, you will need to make sure you have enough storage space for your state database and blockchain. - .. _dg-step-three-set-up-your-cas: Step three: Set up your CAs @@ -204,6 +194,58 @@ Note: This tutorial assumes that a system channel genesis block will not be used deployorderer/ordererchecklist deployorderer/ordererdeploy + +.. _dg-kubernetes-deployment-considerations: + +Kubernetes Considerations +------------------------- + +Kubernetes provides a robust orchestration platform for the deployment, management, and operations of Hyperledger Fabric networks. While Fabric is generally agnostic about the environment in which it has been deployed, employing Kubernetes as a container orchestration engine generally brings advancements to the stability, reliability, and portability of Fabric networks. + +While the cloud vendor, distribution, methods, and tools you utilize to manage a Fabric network may vary, here are some common considerations and techniques worth noting when planning a deployment to Kubernetes: + +Secret Management +~~~~~~~~~~~~~~~~~ + +For the management of secret, sensitive, or private Fabric configuration files, `Kubernetes Secrets `_ may be utilized to restrict access to restricted service accounts. In addition, Fabric configuration may be stored using Hardware Security Modules (HSMs) or encrypted Persistent Volumes (PVs) for an additional layer of security. + +While Fabric images are available at the public Docker Hub image registry, in air-gapped or private environments with a private image registry, a Kubernetes secret must be created and referenced either by the service account or within the pod's deployment descriptor. For additional information on the use of private container registries, see the `Kubernetes guide `_. + +Cluster Sizing +~~~~~~~~~~~~~~ + +In step 2 above, we discussed a general outline for how to think about the sizings of nodes. Your use case, as well as a robust period of development, is the only way you will truly know how large your peers, ordering nodes, and CAs will need to be. + + +Persistent Volumes and Volume Claims +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is a best practice to mount the volumes relevant to your nodes external to the place where your nodes are deployed. This will allow you to reference these volumes later on (for example, restarting a node or a container that has crashed) without having to redeploy or regenerate your crypto material. + + +Resource Utilization and Service Monitoring +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is critical that you establish a strategy and method for monitoring the resources used by your individual nodes and the resources deployed to your cluster generally. As you join your peers to more channels, you will need likely need to increase its CPU and memory allocation. Similarly, you will need to make sure you have enough storage space for your state database and blockchain. + + +Gateway Service Routing, Discovery, and Failover +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In common Fabric deployments on Kubernetes, Fabric nodes (peers, orderers, CAs, chaincode, etc.) are exposed via a single Kubernetes `Service` instance and resolved using a Kube DNS. When provisioning TLS certificates with the CA, ensure that the Subject Alternate Name (SAN) specified in the CA signing request includes the host name as it will be referenced within the cluster. For Fabric networks spanning multiple Kubernetes namespaces, make sure that the SANs reference the fully qualified domain name within Kube DNS. + +While this technique is sufficient to establish direct communication links between Fabric nodes, application clients connecting to peers via the Gateway SDK may utilize a "virtual" service spanning a set of peers to achieve a level of High-Availability and Failover. Using the HA service endpoint, gateway clients may reference a set of peer nodes using a single, common DNS alias. + +While `Kubernetes Service routing `_ does not provide request-level load balancing at the gRPC message layer, it still can be utilized to provide a basic level of HA, failover, and client connection load balancing. In the event of an outage in a backing peer/pod, Kubernetes will assign an active peer at the next connection attempt from the gateway client. + +By default Kubernetes will use `iptables` to bind client connections to a peer pod using random assignment. Pods backing the service may additionally define Readiness Probes to ensure that Gateway client connections are distributed across healthy peer nodes. In addition to random assignment, the Kube cluster may be configured with the `IPVS proxy mode `_ for balancing traffic across backend Pods. For additional information on IPVS routing, refer to the Kubernetes `IPVS-based Load Balancing Deep Dive `_ blog. + +An example load-balanced gateway service is available in the `Kubernetes Test Network `_. In this example: + +* Each organization defines an `orgN-peer-gateway` `Service`, bound to a set of peer `Deployments`. +* The TLS enrollments / certificates for each peer include SAN entries for both the peer service name (`orgN-peerM`) and virtual Gateway peer address (`orgN-peer-gateway`). +* Client applications reference the Fabric Gateway using the virtual `orgN-peer-gateway` service alias. + Next steps ---------- diff --git a/docs/source/gateway.md b/docs/source/gateway.md index f336bdd7167..ccf1dcc8b9b 100644 --- a/docs/source/gateway.md +++ b/docs/source/gateway.md @@ -90,19 +90,6 @@ The Fabric Gateway `Evaluate` and `Endorse` methods make gRPC requests to peers Each Gateway SDK also provides a mechanism for setting timeouts for each gateway method when invoked from the client application. -## Gateway Service Routing with Kubernetes - -In typical Fabric deployments on Kubernetes, each peer node is exposed via a single Kubernetes `Service` instance and resolved using a Kube DNS. This technique is sufficient for the Fabric Gateway and application clients to resolve individual peers within a Fabric network. In cases requiring high-availability and/or client connection load-balancing, Kubernetes can be configured with an additional `Service` resource bound to multiple gateway peers. Using an HA topology, gateway clients may reference a set of peer nodes with a single DNS alias. - -While [Kubernetes Service routing](https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies) does not provide request-level load balancing at the gRPC message layer, it can be utilized to provide a basic level of HA, failover, and client connection load balancing. In the event of an outage in a backing peer/pod, Kubernetes will assign an active peer at the next connection attempt from the gateway client. - -By default Kubernetes will use `iptables` to bind client connections to a peer pod using random assignment. Pods backing the service may additionally define Readiness Probes to ensure that Gateway client connections are distributed across healthy peer nodes. In addition to random assignment, the Kube cluster may be configured with the [IPVS proxy mode](https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs) for balancing traffic across backend Pods. For additional information on IPVS routing, refer to the Kubernetes [IPVS-based Load Balancing Deep Dive](https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/) blog. - -An example load-balanced gateway service is available in the [Kubernetes Test Network](https://github.com/hyperledger/fabric-samples/blob/main/test-network-k8s/docs/HIGH_AVAILABILITY.md). In this example: -- Each organization defines an `orgN-peer-gateway` `Service`, bound to a set of peer `Deployments`. -- The TLS enrollments / certificates for each peer include the shared service / Subject Alternate Name alias. -- Client applications reference the Fabric Gateway using the `orgN-peer-gateway` service alias. - ## Listening for events The gateway provides a simplified API for client applications to receive [chaincode events](peer_event_services.html#how-to-register-for-events) in the client applications. Each SDK provides a mechanism to handle these events using its language-specific idiom.