Merge branch 'master' into fix/virtual-dataset-tracking-datatype

RELEASING/README.md

@@ -67,6 +67,33 @@ need to be done at every release.
     svn update
 ```
 
+To minimize the risk of mixing up your local development environment, it's recommended to work on the
+release in a different directory than where the devenv is located. In this example, we'll clone
+the repo directly from the main `apache/superset` repo to a new directory `superset-release`:
+
+```bash
+cd <MY PROJECTS PATH>
+git clone git@github.com:apache/superset.git superset-release
+cd superset-release
+```
+
+We recommend setting up a virtual environment to isolate the python dependencies from your main
+setup:
+
+```bash
+virtualenv venv
+source venv/bin/activate
+```
+
+
+In addition, we recommend using the [`cherrytree`](https://pypi.org/project/cherrytree/) tool for
+automating cherry picking, as it will help speed up the release process. To install `cherrytree`
+and other dependencies that are required for the release process, run the following commands:
+
+```bash
+pip install -r RELEASING/requirements.txt
+```
+
 ## Setting up the release environment (do every time)
 
 As the vote process takes a minimum of 72h, sometimes stretching over several weeks
@@ -78,35 +105,39 @@ the wrong files/using wrong names. There's a script to help you set correctly al
 necessary environment variables. Change your current directory to `superset/RELEASING`
 and execute the `set_release_env.sh` script with the relevant parameters:
 
-Usage (BASH):
+
+Usage (MacOS/ZSH):
 ```bash
-. set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
+cd RELEASING
+source set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
 ```
 
-Usage (ZSH):
+Usage (BASH):
 ```bash
-source set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
+. set_release_env.sh <SUPERSET_RC_VERSION> <PGP_KEY_FULLNAME>
 ```
 
 Example:
 ```bash
-source set_release_env.sh 0.38.0rc1 myid@apache.org
+source set_release_env.sh 1.5.1rc1 myid@apache.org
 ```
 
-The script will output the exported variables. Here's example for 0.38.0rc1:
+The script will output the exported variables. Here's example for 1.5.1rc1:
 
 ```
+-------------------------------
 Set Release env variables
-SUPERSET_VERSION=0.38.0
+SUPERSET_VERSION=1.5.1
 SUPERSET_RC=1
-SUPERSET_GITHUB_BRANCH=0.38
-SUPERSET_PGP_FULLNAME=myid@apache.org
-SUPERSET_VERSION_RC=0.38.0rc1
-SUPERSET_RELEASE=apache-superset-0.38.0
-SUPERSET_RELEASE_RC=apache-superset-0.38.0rc1
-SUPERSET_RELEASE_TARBALL=apache-superset-0.38.0-source.tar.gz
-SUPERSET_RELEASE_RC_TARBALL=apache-superset-0.38.0rc1-source.tar.gz
-SUPERSET_TMP_ASF_SITE_PATH=/tmp/superset-site-0.38.0
+SUPERSET_GITHUB_BRANCH=1.5
+SUPERSET_PGP_FULLNAME=villebro@apache.org
+SUPERSET_VERSION_RC=1.5.1rc1
+SUPERSET_RELEASE=apache-superset-1.5.1
+SUPERSET_RELEASE_RC=apache-superset-1.5.1rc1
+SUPERSET_RELEASE_TARBALL=apache-superset-1.5.1-source.tar.gz
+SUPERSET_RELEASE_RC_TARBALL=apache-superset-1.5.1rc1-source.tar.gz
+SUPERSET_TMP_ASF_SITE_PATH=/tmp/incubator-superset-site-1.5.1
+-------------------------------
 ```
 
 ## Crafting a source release
@@ -116,23 +147,101 @@ a branch named with the release MAJOR.MINOR version (on this example 0.37).
 This new branch will hold all PATCH and release candidates
 that belong to the MAJOR.MINOR version.
 
+### Creating an initial minor release (e.g. 1.5.0)
+
 The MAJOR.MINOR branch is normally a "cut" from a specific point in time from the master branch.
-Then (if needed) apply all cherries that will make the PATCH.
+When creating the initial minor release (e.g. 1.5.0), create a new branch:
+```bash
+git checkout master
+git pull
+git checkout -b ${SUPERSET_GITHUB_BRANCH}
+git push origin $SUPERSET_GITHUB_BRANCH
+```
+
+Note that this initializes a new "release cut", and is NOT needed when creating a patch release
+(e.g. 1.5.1).
+
+### Creating a patch release (e.g. 1.5.1)
+
+When getting ready to bake a patch release, simply checkout the relevant branch:
 
 ```bash
-git checkout -b $SUPERSET_GITHUB_BRANCH
-git push upstream $SUPERSET_GITHUB_BRANCH
+git checkout master
+git pull
+git checkout ${SUPERSET_GITHUB_BRANCH}
 ```
 
+### Cherry picking
+
+It is customary to label PRs that have been introduced after the cut with the label
+`v<MAJOR>.<MINOR>`. For example, for any PRs that should be included in the 1.5 branch, the
+label `v1.5` should be added.
+
+To see how well the labelled PRs would apply to the current branch, run the following command:
+
+```bash
+cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} ${SUPERSET_GITHUB_BRANCH}
+```
+
+This requires the presence of an environment variable `GITHUB_TOKEN`. Alternatively,
+you can pass the token directly via the `--access-token` parameter (`-at` for short).
+
+#### Happy path: no conflicts
+
+This will show how many cherries will apply cleanly. If there are no conflicts, you can simply apply all cherries
+by adding the `--no-dry-run` flag (`-nd` for short):
+
+```bash
+cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} -nd ${SUPERSET_GITHUB_BRANCH}
+```
+
+
+#### Resolving conflicts
+
+If there are conflicts, you can issue the following command to apply all cherries up until the conflict automatically, and then
+break by adding the `-error-mode break` flag (`-e break` for short):
+
+```bash
+cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} -nd -e break ${SUPERSET_GITHUB_BRANCH}
+```
+
+After applying the cleanly merged cherries, `cherrytree` will specify the SHA of the conflicted cherry. To resolve the conflict,
+simply issue the following command:
+
+```bash
+git cherry-pick <SHA>
+```
+
+Then fix all conflicts, followed by
+
+```bash
+git add -u  # add all changes
+git cherry-pick --continue
+```
+
+After this, rerun all the above steps until all cherries have been picked, finally pushing all new commits to the release branch
+on the main repo:
+
+```bash
+git push
+```
+
+### Updating changelog
+
 Next, update the `CHANGELOG.md` with all the changes that are included in the release.
-Make sure the branch has been pushed to `upstream` to ensure the changelog generator
+Make sure the branch has been pushed to `origin` to ensure the changelog generator
 can pick up changes since the previous release.
-Change log script requires a github token and will try to use your env var GITHUB_TOKEN.
-you can also pass the token using the parameter `--access_token`.
+Similar to `cherrytree`, the change log script requires a github token, either as an env var
+(`GITHUB_TOKEN`) or as the parameter `--access_token`.
+
+#### Initial release (e.g. 1.5.0)
+
+When generating the changelog for an initial minor relese, you should compare with
+the previous release (in the example, the previous release branch is `1.4`, so remember to
+update it accordingly):
 
-Example:
 ```bash
-python changelog.py --previous_version 0.37 --current_version 0.38 changelog
+python changelog.py --previous_version 1.4 --current_version ${SUPERSET_GITHUB_BRANCH} changelog
 ```
 
 You can get a list of pull requests with labels started with blocking, risk, hold, revert and security by using the parameter `--risk`.
@@ -141,16 +250,29 @@ Example:
 python changelog.py --previous_version 0.37 --current_version 0.38 changelog --access_token {GITHUB_TOKEN} --risk
 ```
 
-The script will checkout both branches and compare all the PR's, copy the output and paste it on the `CHANGELOG.md`
+The script will checkout both branches, compare all the PRs, and output the lines that are needed to be added to the
+`CHANGELOG.md` file in the root of the repo. Remember to also make sure to update the branch id (with the above command
+`1.5` needs to be changed to `1.5.0`)
 
 Then, in `UPDATING.md`, a file that contains a list of notifications around
 deprecations and upgrading-related topics,
 make sure to move the content now under the `Next Version` section under a new
 section for the new release.
 
-Finally bump the version number on `superset-frontend/package.json` (replace with whichever version is being released excluding the RC version):
+#### Patch release (e.g. 1.5.1)
+
+To compare the forthcoming patch release with the latest release from the same branch, set
+`--previous_version` as the tag of the previous release (in this example `1.5.0`; remember to update accordingly)
+
+```bash
+python changelog.py --previous_version 1.5.0 --current_version ${SUPERSET_GITHUB_BRANCH} changelog
+```
+
+### Set version number
 
-```json
+Finally, bump the version number on `superset-frontend/package.json` (replace with whichever version is being released excluding the RC version):
+
+```
 "version": "0.38.0"
 ```
 
@@ -162,7 +284,7 @@ git add ...
 git commit ...
 # push new tag
 git tag ${SUPERSET_VERSION_RC}
-git push upstream ${SUPERSET_VERSION_RC}
+git push origin ${SUPERSET_VERSION_RC}
 ```
 
 ## Preparing the release candidate
@@ -180,7 +302,7 @@ the tag and create a signed source tarball from it:
 
 Note that `make_tarball.sh`:
 
-- By default assumes you have already executed an SVN checkout to `$HOME/svn/superset_dev`.
+- By default, the script assumes you have already executed an SVN checkout to `$HOME/svn/superset_dev`.
 This can be overridden by setting `SUPERSET_SVN_DEV_PATH` environment var to a different svn dev directory
 - Will refuse to craft a new release candidate if a release already exists on your local svn dev directory
 - Will check `package.json` version number and fails if it's not correctly set
@@ -289,7 +411,7 @@ git branch
 # Create the release tag
 git tag -f ${SUPERSET_VERSION}
 # push the tag to the remote
-git push upstream ${SUPERSET_VERSION}
+git push origin ${SUPERSET_VERSION}
 ```
 
 ### Update CHANGELOG and UPDATING on superset

RELEASING/changelog.py

@@ -164,6 +164,12 @@ def _is_risk_pull_request(self, labels: List[Any]) -> bool:
         return False
 
     def _get_changelog_version_head(self) -> str:
+        if not len(self._logs):
+            print(
+                f"No changes found between revisions. "
+                f"Make sure your branch is up to date."
+            )
+            sys.exit(1)
         return f"### {self._version} ({self._logs[0].time})"
 
     def _parse_change_log(

RELEASING/requirements.txt

@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+cherrytree
+jinja2

RESOURCES/INTHEWILD.md

@@ -120,9 +120,10 @@ Join our growing community!
 
 ### Education
 - [Brilliant.org](https://brilliant.org/)
+- [Sunbird](https://www.sunbird.org/) [@eksteporg]
+- [The GRAPH Network](https://thegraphnetwork.org/)[@fccoelho]
 - [Udemy](https://www.udemy.com/) [@sungjuly]
 - [VIPKID](https://www.vipkid.com.cn/) [@illpanda]
-- [Sunbird](https://www.sunbird.org/) [@eksteporg]
 
 ### Energy
 - [Airboxlab](https://foobot.io) [@antoine-galataud]
@@ -143,10 +144,10 @@ Join our growing community!
 - [Symmetrics](https://www.symmetrics.fyi)
 
 ### Others
- - [Dropbox](https://www.dropbox.com/) [@bkyryliuk]
- - [Grassroot](https://www.grassrootinstitute.org/)
- - [komoot](https://www.komoot.com/) [@christophlingg]
- - [Let's Roam](https://www.letsroam.com/)
- - [Twitter](https://twitter.com/)
- - [VLMedia](https://www.vlmedia.com.tr/) [@ibotheperfect]
- - [Yahoo!](https://yahoo.com/)
+- [Dropbox](https://www.dropbox.com/) [@bkyryliuk]
+- [Grassroot](https://www.grassrootinstitute.org/)
+- [komoot](https://www.komoot.com/) [@christophlingg]
+- [Let's Roam](https://www.letsroam.com/)
+- [Twitter](https://twitter.com/)
+- [VLMedia](https://www.vlmedia.com.tr/) [@ibotheperfect]
+- [Yahoo!](https://yahoo.com/)

docs/docs/databases/clickhouse.mdx

@@ -1,13 +1,13 @@
 ---
-title: Clickhouse
+title: ClickHouse
 hide_title: true
 sidebar_position: 15
 version: 1
 ---
 
-## Clickhouse
+## ClickHouse
 
-To use Clickhouse with Superset, you will need to add the following Python libraries:
+To use ClickHouse with Superset, you will need to add the following Python libraries:
 
 ```
 clickhouse-driver==0.2.0
@@ -21,7 +21,7 @@ clickhouse-driver>=0.2.0
 clickhouse-sqlalchemy>=0.1.6
 ```
 
-The recommended connector library for Clickhouse is
+The recommended connector library for ClickHouse is
 [sqlalchemy-clickhouse](https://github.com/cloudflare/sqlalchemy-clickhouse).
 
 The expected connection string is formatted as follows:

docs/docs/security.mdx

@@ -49,7 +49,7 @@ to all databases by default, both **Alpha** and **Gamma** users need to be given
 
 To allow logged-out users to access some Superset features, you can use the `PUBLIC_ROLE_LIKE` config setting and assign it to another role whose permissions you want passed to this role.
 
-For example, by setting `PUBLIC_ROLE_LIKE = Gamma` in your `superset_config.py` file, you grant
+For example, by setting `PUBLIC_ROLE_LIKE = "Gamma"` in your `superset_config.py` file, you grant
 public role the same set of permissions as for the **Gamma** role. This is useful if one
 wants to enable anonymous users to view dashboards. Explicit grant on specific datasets is
 still required, meaning that you need to edit the **Public** role and add the public data sources to the role manually.

docs/package.json

@@ -38,7 +38,7 @@
     "react-dom": "^17.0.1",
     "react-github-btn": "^1.2.0",
     "stream": "^0.0.2",
-    "swagger-ui-react": "^4.1.2",
+    "swagger-ui-react": "^4.1.3",
     "url-loader": "^4.1.1"
   },
   "devDependencies": {