From b83e0f2f9b987a3d87d9e138255360512a5b2d5c Mon Sep 17 00:00:00 2001 From: Juan Girini Date: Fri, 15 Dec 2023 17:12:05 +0100 Subject: [PATCH 1/4] initial table of contents --- docs/sdk/src/reference_docs/cli.rs | 129 ++++++++++++++++++++++++++++- 1 file changed, 128 insertions(+), 1 deletion(-) diff --git a/docs/sdk/src/reference_docs/cli.rs b/docs/sdk/src/reference_docs/cli.rs index 9274e86b04ef..696f85a3a246 100644 --- a/docs/sdk/src/reference_docs/cli.rs +++ b/docs/sdk/src/reference_docs/cli.rs @@ -3,5 +3,132 @@ //! //! Notes: //! -//! - Command line arguments of a typical substrate based chain, and how to find and learn them. +//! - Command line arguments of a typical substrate based chain +//! - how to find and learn them. //! - How to extend them with your custom stuff. +//! +//! source https://docs.substrate.io/reference/command-line-tools/ +//! +//! ## Command-line tools +//! +//! | Command Entry Point | Description | +//! |---------------------|-------------| +//! | [`archive`](#archive) | Index and store all blocks, state, and transaction data for a Substrate-based chain in a relational SQL database. | +//! | `memory-profiler` | Collect information about memory allocation and the behavior of blockchain applications over time. | +//! | `node-template` | Start and manage a Substrate node preconfigured with a subset of commonly-used FRAME pallets. | +//! | `polkadot-launch` | Launch a local Polkadot test network. | +//! | `polkadot-apps` | Interact with Polkadot or a Substrate node using a browser. | +//! | `sidecar` | Use a REST service to interact with blockchain nodes built using FRAME. | +//! | `srtool` | Build WASM runtime in a deterministic way, allowing continuous integration pipelines and users to produce a strictly identical WASM runtime. | +//! | `subkey` | Generate and manage public and private key pairs for accounts. | +//! | `subxt` | Submit extrinsics to a Substrate node using RPC. | +//! | `try-runtime` | Query a snapshot of runtime storage to retrieve state. | +//! | `tx-wrapper` | Publish chain specific offline transaction generation libraries. | +//! +//! ### `archive` +//! +//! The `archive` program is used to index all blocks, state, and transaction data for a +//! Substrate-based chain and store the indexed data in a relational SQL database. The database +//! created by the `archive` program mirrors all data from a running Substrate blockchain. After you +//! archive the data, you can use database tools to query and retrieve information from the SQL +//! database about the blockchain state. For examples of queries you might want to run against a +//! Substrate archive database, see [Useful queries](https://github.com/paritytech/substrate-archive/wiki/Useful-Queries). +//! +//! #### Before you begin +//! +//! Before you use `archive` to create a database for a Substrate-based chain, you need to prepare +//! your environment with the required files: +//! +//! - You must have PostgreSQL installed on the computer where you are running a Substrate node. +//! +//! You can download PostgreSQL packages for different platforms from the PostgreSQL [Downloads](https://www.postgresql.org/download/) page. +//! +//! Depending on your platform, you might be able to install PostgreSQL using a local package +//! manager. For example, you can install a PostgreSQL package on a macOS computer by running +//! `brew install postgresql` in a Terminal. +//! +//! - You must have RabbitMQ or Docker Compose installed on the computer where you have PostgreSQL +//! installed. +//! +//! Depending on your platform, the instruction and system requirements for installing RabbitMQ or +//! Docker can vary. For information about using [RabbitMQ](https://www.rabbitmq.com/) or [Docker](https://www.docker.com/), +//! see the [Setup](https://github.com/paritytech/substrate-archive/wiki/1-Setup) +//! `substrate-archive` wiki page. +//! +//! - Your Substrate chain must use RocksDB as its backend database. +//! +//! #### Install and configure +//! +//! To install the `substrate-archive-cli` program: +//! +//! 1. Open a terminal shell on your computer. +//! +//! 2. Clone the `substrate-archive` repository by running the following command: +//! +//! ``` +//! git clone https://github.com/paritytech/substrate-archive.git +//! ``` +//! +//! 3. Change to the root directory of the `substrate-archive` repository by running the following +//! command: +//! +//! ``` +//! cd substrate-archive +//! ``` +//! +//! 4. Start the PostgreSQL database (`postgres`) and Postgre administrative process (`pgadmin`) on +//! the Substrate node. +//! +//! If you have Docker Compose, you can start the services automatically by running the +//! `docker-compose up -d` command. +//! +//! 5. Start your Substrate node, with `pruning` set to archive. For example: ``` +//! ./target/release/node-template --pruning=archive ``` +//! +//! 6. Look at the current DBs: `psql -U postgres -hlocalhost -p6432` +//! +//! 7. Run `DATABASE_URL=postgres://postgres:123@localhost:6432/local_chain_db sqlx` database create +//! in `substrate-archive/src` to create the database. +//! +//! 8. Set `CHAIN_DATA_DB=""`. +//! +//! 9. Set up your `archive.conf` file: +//! +//! - make sure to set your base bath to primary DB +//! - tell it where the rocksdb is. State using CHAIN_DATA_DB +//! - secondary DB is an optimization +//! - postgres url (set to var if in prod) +//! +//! 10. (Optional) setup up logging and debugging. +//! +//! 11. Run a node template. Make sure you run it in `--release --dev base-path=/tmp/dir +//! --pruning=archive` +//! +//! 12. Make a transaction with your node template. +//! +//! 13. Start up the `substrate-archive` node for your target chain: +//! `cargo run --release -- -c archive-conf.toml --chain=polkadot` +//! +//! 14. Open a web browser and log in to the Postgres administrative console. +//! +//! - Default URL: localhost:16543 +//! - Default user name: pgadmin4@pgadmin.org +//! - Default password: admin +//! +//! +//! 15. Look at the reference to start making your queries. +//! +//! +//! +//! +//! +//! +//! +//! +//! +//! ## Extension through Community Tools +//! The [`substrate-cli-tools`](https://github.com/StakeKat/substrate-cli-tools) repository provides +//! a set of high-level tools to connect and consume Substrate-based chains. These tools leverage +//! the `py-substrate-interface`` library and offer functionalities like monitoring events as they +//! happen, decoding balances, and more. The library provided can also be reused to build your own +//! commands, offering a higher level of abstraction for easier use​​. From 760d88a0ae0a6c811e8b5e7eec1cd84aee971155 Mon Sep 17 00:00:00 2001 From: Juan Girini Date: Mon, 18 Dec 2023 17:42:11 +0100 Subject: [PATCH 2/4] add cli ref docs --- docs/sdk/src/reference_docs/cli.rs | 239 +++++++++++++---------------- 1 file changed, 105 insertions(+), 134 deletions(-) diff --git a/docs/sdk/src/reference_docs/cli.rs b/docs/sdk/src/reference_docs/cli.rs index 696f85a3a246..04a11145bf70 100644 --- a/docs/sdk/src/reference_docs/cli.rs +++ b/docs/sdk/src/reference_docs/cli.rs @@ -1,134 +1,105 @@ -//! # Command Line Arguments -//! -//! -//! Notes: -//! -//! - Command line arguments of a typical substrate based chain -//! - how to find and learn them. -//! - How to extend them with your custom stuff. -//! -//! source https://docs.substrate.io/reference/command-line-tools/ -//! -//! ## Command-line tools -//! -//! | Command Entry Point | Description | -//! |---------------------|-------------| -//! | [`archive`](#archive) | Index and store all blocks, state, and transaction data for a Substrate-based chain in a relational SQL database. | -//! | `memory-profiler` | Collect information about memory allocation and the behavior of blockchain applications over time. | -//! | `node-template` | Start and manage a Substrate node preconfigured with a subset of commonly-used FRAME pallets. | -//! | `polkadot-launch` | Launch a local Polkadot test network. | -//! | `polkadot-apps` | Interact with Polkadot or a Substrate node using a browser. | -//! | `sidecar` | Use a REST service to interact with blockchain nodes built using FRAME. | -//! | `srtool` | Build WASM runtime in a deterministic way, allowing continuous integration pipelines and users to produce a strictly identical WASM runtime. | -//! | `subkey` | Generate and manage public and private key pairs for accounts. | -//! | `subxt` | Submit extrinsics to a Substrate node using RPC. | -//! | `try-runtime` | Query a snapshot of runtime storage to retrieve state. | -//! | `tx-wrapper` | Publish chain specific offline transaction generation libraries. | -//! -//! ### `archive` -//! -//! The `archive` program is used to index all blocks, state, and transaction data for a -//! Substrate-based chain and store the indexed data in a relational SQL database. The database -//! created by the `archive` program mirrors all data from a running Substrate blockchain. After you -//! archive the data, you can use database tools to query and retrieve information from the SQL -//! database about the blockchain state. For examples of queries you might want to run against a -//! Substrate archive database, see [Useful queries](https://github.com/paritytech/substrate-archive/wiki/Useful-Queries). -//! -//! #### Before you begin -//! -//! Before you use `archive` to create a database for a Substrate-based chain, you need to prepare -//! your environment with the required files: -//! -//! - You must have PostgreSQL installed on the computer where you are running a Substrate node. -//! -//! You can download PostgreSQL packages for different platforms from the PostgreSQL [Downloads](https://www.postgresql.org/download/) page. -//! -//! Depending on your platform, you might be able to install PostgreSQL using a local package -//! manager. For example, you can install a PostgreSQL package on a macOS computer by running -//! `brew install postgresql` in a Terminal. -//! -//! - You must have RabbitMQ or Docker Compose installed on the computer where you have PostgreSQL -//! installed. -//! -//! Depending on your platform, the instruction and system requirements for installing RabbitMQ or -//! Docker can vary. For information about using [RabbitMQ](https://www.rabbitmq.com/) or [Docker](https://www.docker.com/), -//! see the [Setup](https://github.com/paritytech/substrate-archive/wiki/1-Setup) -//! `substrate-archive` wiki page. -//! -//! - Your Substrate chain must use RocksDB as its backend database. -//! -//! #### Install and configure -//! -//! To install the `substrate-archive-cli` program: -//! -//! 1. Open a terminal shell on your computer. -//! -//! 2. Clone the `substrate-archive` repository by running the following command: -//! -//! ``` -//! git clone https://github.com/paritytech/substrate-archive.git -//! ``` -//! -//! 3. Change to the root directory of the `substrate-archive` repository by running the following -//! command: -//! -//! ``` -//! cd substrate-archive -//! ``` -//! -//! 4. Start the PostgreSQL database (`postgres`) and Postgre administrative process (`pgadmin`) on -//! the Substrate node. -//! -//! If you have Docker Compose, you can start the services automatically by running the -//! `docker-compose up -d` command. -//! -//! 5. Start your Substrate node, with `pruning` set to archive. For example: ``` -//! ./target/release/node-template --pruning=archive ``` -//! -//! 6. Look at the current DBs: `psql -U postgres -hlocalhost -p6432` -//! -//! 7. Run `DATABASE_URL=postgres://postgres:123@localhost:6432/local_chain_db sqlx` database create -//! in `substrate-archive/src` to create the database. -//! -//! 8. Set `CHAIN_DATA_DB=""`. -//! -//! 9. Set up your `archive.conf` file: -//! -//! - make sure to set your base bath to primary DB -//! - tell it where the rocksdb is. State using CHAIN_DATA_DB -//! - secondary DB is an optimization -//! - postgres url (set to var if in prod) -//! -//! 10. (Optional) setup up logging and debugging. -//! -//! 11. Run a node template. Make sure you run it in `--release --dev base-path=/tmp/dir -//! --pruning=archive` -//! -//! 12. Make a transaction with your node template. -//! -//! 13. Start up the `substrate-archive` node for your target chain: -//! `cargo run --release -- -c archive-conf.toml --chain=polkadot` -//! -//! 14. Open a web browser and log in to the Postgres administrative console. -//! -//! - Default URL: localhost:16543 -//! - Default user name: pgadmin4@pgadmin.org -//! - Default password: admin -//! -//! -//! 15. Look at the reference to start making your queries. -//! -//! -//! -//! -//! -//! -//! -//! -//! -//! ## Extension through Community Tools -//! The [`substrate-cli-tools`](https://github.com/StakeKat/substrate-cli-tools) repository provides -//! a set of high-level tools to connect and consume Substrate-based chains. These tools leverage -//! the `py-substrate-interface`` library and offer functionalities like monitoring events as they -//! happen, decoding balances, and more. The library provided can also be reused to build your own -//! commands, offering a higher level of abstraction for easier use​​. +//! # Substrate CLI +//! +//! Let's see some examples of typical CLI arguments used when setting up and running a +//! Substrate-based blockchain network. We use the [`substrate-node-template`](https://github.com/substrate-developer-hub/substrate-node-template) +//! for this purpose. +//! +//! #### Checking the available CLI arguments +//! ```bash +//! ./target/debug/node-template --help +//! ``` +//! - `--help`: Displays the available CLI arguments. +//! +//! #### Starting a Local Substrate Node in Development Mode +//! ```bash +//! ./target/release/node-template \ +//! --dev +//! ``` +//! - `--dev`: Runs the node in development mode, using a pre-defined development chain +//! specification. +//! This mode ensures a fresh state by deleting existing data on restart. Example: +//! +//! #### Generating Custom Chain Specification +//! ```bash +//! ./target/debug/node-template \ +//! build-spec \ +//! --disable-default-bootnode \ +//! --chain local \ +//! > customSpec.json +//! ``` +//! +//! - `build-spec`: A subcommand to generate a chain specification file. +//! - `--disable-default-bootnode`: Disables the default bootnodes in the node template. +//! - `--chain local`: Indicates the chain specification is for a local development chain. +//! - `> customSpec.json`: Redirects the output into a customSpec.json file. +//! +//! #### Converting Chain Specification to Raw Format +//! ```bash +//! ./target/debug/node-template build-spec \ +//! --chain=customSpec.json \ +//! --raw \ +//! --disable-default-bootnode \ +//! > customSpecRaw.json +//! ``` +//! +//! - `--chain=customSpec.json`: Uses the custom chain specification as input. +//! - `--disable-default-bootnode`: Disables the default bootnodes in the node template. +//! - `--raw`: Converts the chain specification into a raw format with encoded storage keys. +//! - `> customSpecRaw.json`: Outputs to customSpecRaw.json. +//! +//! #### Starting the First Node (Bootnode) in a Private Network +//! ```bash +//! ./target/debug/node-template \ +//! --base-path /tmp/node01 \ +//! --chain ./customSpecRaw.json \ +//! --port 30333 \ +//! --ws-port 9945 \ +//! --rpc-port 9933 \ +//! --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \ +//! --validator \ +//! --rpc-methods Unsafe \ +//! --name MyNode01 \ +//! --password-interactive +//! ``` +//! +//! - `--base-path`: Sets the directory for node data. +//! - `--chain`: Specifies the chain specification file. +//! - `--port`: TCP port for peer-to-peer communication. +//! - `--ws-port`: WebSocket port for RPC. +//! - `--rpc-port`: HTTP port for JSON-RPC. +//! - `--telemetry-url`: Endpoint for sending telemetry data. +//! - `--validator`: Indicates the node’s participation in block production. +//! - `--rpc-methods Unsafe`: Allows potentially unsafe RPC methods. +//! - `--name`: Sets a human-readable name for the node. +//! - `--password-interactive`: Interactive password input for key operations. +//! +//! #### Adding a Second Node to the Network +//! ```bash +//! ./target/release/node-template \ +//! --base-path /tmp/bob \ +//! --chain local \ +//! --bob \ +//! --port 30334 \ +//! --rpc-port 9946 \ +//! --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \ +//! --validator \ +//! --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp +//! ``` +//! +//! - `--base-path`: Sets the directory for node data. +//! - `--chain`: Specifies the chain specification file. +//! - `--bob`: Initializes the node with the session keys of the "Bob" account. +//! - `--port`: TCP port for peer-to-peer communication. +//! - `--rpc-port`: HTTP port for JSON-RPC. +//! - `--telemetry-url`: Endpoint for sending telemetry data. +//! - `--validator`: Indicates the node’s participation in block production. +//! - `--bootnodes`: Specifies the address of the first node (bootnode) for peer discovery. +//! +//! --- +//! +//! > If you are interested in learning how to extend the CLI with your custom arguments, you can +//! > check out the [Customize your Substrate chain CLI](https://www.youtube.com/watch?v=IVifko1fqjw) +//! > seminar. +//! > Please note that the seminar is based on an older version of Substrate, and [Clap](https://docs.rs/clap/latest/clap/) +//! > is now used instead of [StructOpt](https://docs.rs/structopt/latest/structopt/) for parsing +//! > CLI arguments. From ef21eabc809703dd680f384b70944d3277469913 Mon Sep 17 00:00:00 2001 From: Juan Girini Date: Mon, 18 Dec 2023 17:44:45 +0100 Subject: [PATCH 3/4] improve text --- docs/sdk/src/reference_docs/cli.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/sdk/src/reference_docs/cli.rs b/docs/sdk/src/reference_docs/cli.rs index 04a11145bf70..44572a52cd13 100644 --- a/docs/sdk/src/reference_docs/cli.rs +++ b/docs/sdk/src/reference_docs/cli.rs @@ -1,8 +1,8 @@ //! # Substrate CLI //! //! Let's see some examples of typical CLI arguments used when setting up and running a -//! Substrate-based blockchain network. We use the [`substrate-node-template`](https://github.com/substrate-developer-hub/substrate-node-template) -//! for this purpose. +//! Substrate-based blockchain. We use the [`substrate-node-template`](https://github.com/substrate-developer-hub/substrate-node-template) +//! on these examples. //! //! #### Checking the available CLI arguments //! ```bash @@ -17,7 +17,7 @@ //! ``` //! - `--dev`: Runs the node in development mode, using a pre-defined development chain //! specification. -//! This mode ensures a fresh state by deleting existing data on restart. Example: +//! This mode ensures a fresh state by deleting existing data on restart. //! //! #### Generating Custom Chain Specification //! ```bash From c8283a48c3f065aad41bdb290fadc83841afd64c Mon Sep 17 00:00:00 2001 From: Juan Girini Date: Tue, 19 Dec 2023 15:01:22 +0100 Subject: [PATCH 4/4] review updates --- docs/sdk/src/reference_docs/cli.rs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/sdk/src/reference_docs/cli.rs b/docs/sdk/src/reference_docs/cli.rs index 44572a52cd13..5779e0f8d049 100644 --- a/docs/sdk/src/reference_docs/cli.rs +++ b/docs/sdk/src/reference_docs/cli.rs @@ -47,7 +47,7 @@ //! - `--raw`: Converts the chain specification into a raw format with encoded storage keys. //! - `> customSpecRaw.json`: Outputs to customSpecRaw.json. //! -//! #### Starting the First Node (Bootnode) in a Private Network +//! #### Starting the First Node in a Private Network //! ```bash //! ./target/debug/node-template \ //! --base-path /tmp/node01 \ @@ -58,8 +58,7 @@ //! --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \ //! --validator \ //! --rpc-methods Unsafe \ -//! --name MyNode01 \ -//! --password-interactive +//! --name MyNode01 //! ``` //! //! - `--base-path`: Sets the directory for node data. @@ -71,7 +70,6 @@ //! - `--validator`: Indicates the node’s participation in block production. //! - `--rpc-methods Unsafe`: Allows potentially unsafe RPC methods. //! - `--name`: Sets a human-readable name for the node. -//! - `--password-interactive`: Interactive password input for key operations. //! //! #### Adding a Second Node to the Network //! ```bash @@ -93,7 +91,8 @@ //! - `--rpc-port`: HTTP port for JSON-RPC. //! - `--telemetry-url`: Endpoint for sending telemetry data. //! - `--validator`: Indicates the node’s participation in block production. -//! - `--bootnodes`: Specifies the address of the first node (bootnode) for peer discovery. +//! - `--bootnodes`: Specifies the address of the first node for peer discovery. Nodes should find +//! each other using mDNS. This command needs to be used if they don't find each other. //! //! --- //!