chore(releasing): automated changelog generation (#617)

* convert existing changelog.md

* remove authors

* feat(releasing): automated changelog generation

* update readme

* update readme

* spelling

* simpler check

.github/dependabot.yml

@@ -7,6 +7,7 @@ updates:
       time: "04:00" # UTC
     labels:
       - "domain: deps"
+      - "no-changelog"
     commit-message:
       prefix: "chore(deps)"
     open-pull-requests-limit: 10

.github/workflows/changelog.yaml

@@ -0,0 +1,32 @@
+# Changelog
+#
+# Validates that a changelog entry was added.
+# Runs on PRs when:
+#   - opened/re-opened
+#   - new commits pushed
+#   - label is added or removed
+
+name: Changelog
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+
+jobs:
+  check-changelog:
+    env:
+      PR_HAS_LABEL: ${{ contains( github.event.pull_request.labels.*.name, 'no-changelog') }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - run: |
+          if [[ $PR_HAS_LABEL == 'true' ]] ; then
+            echo "'no-changelog' label detected."
+            exit 0
+          fi
+
+          # helper script needs to reference the master branch to compare against
+          git fetch origin main:refs/remotes/origin/main
+
+          ./scripts/check_changelog_fragments.sh

CHANGELOG.md

@@ -1,7 +1,8 @@
 # Changelog
 
-## unreleased
-- added the `alias_sources` parameter for `parse_groks` to read sources from files
+ This project uses [*towncrier*](https://towncrier.readthedocs.io/) for changelog generation.
+
+<!-- changelog start -->
 
 ## `0.9.1` (2023-12-21)
 

CONTRIBUTING.md

@@ -15,10 +15,10 @@ provide everything you need to get started.
 4. Make your changes.
 5. Add and/or update tests to cover your changes.
 6. Run `./scripts/checks.sh` to run tests and other checks.
-7. Add a one line summary of changes to the "unreleased" section of `CHANGELOG.md`.
-8. [Submit the branch as a pull request][urls.submit_pr] to the repo. A team member should
+7. [Submit the branch as a pull request][urls.submit_pr] to the repo. A team member should
    comment and/or review your pull request.
-
+8. Add a changelog fragment (requires the PR number) to describe your changes which will
+   be included in the release changelog. See the [README.md](changelog.d/README.md) for details.
 
 [urls.existing_issues]: https://github.com/vectordotdev/vrl/issues
 [urls.new_issue]: https://github.com/vectordotdev/vrl/issues/new

changelog.d/194.enhancement.md

@@ -0,0 +1 @@
+Added the `alias_sources` parameter for `parse_groks` to read sources from files.

changelog.d/README.md

@@ -0,0 +1,75 @@
+## Overview
+
+This directory contains changelog "fragments" that are collected during a release to
+generate the project's changelog.
+
+The conventions used for this changelog logic follow [towncrier](https://towncrier.readthedocs.io/en/stable/markdown.html).
+
+The changelog fragments are located in `changelog.d/`.
+
+## Process
+
+Fragments for un-released changes are placed in the root of this directory during PRs.
+
+During a release, `scripts/generate_release_changelog.sh` is run in order to automatically
+generate the changes to the CHANGELOG.md file. As part of the script execution, the
+changelog fragment files that are being released, are removed from the repo.
+
+### Pull Requests
+
+By default, PRs are required to add at least one entry to this directory.
+This is enforced during CI.
+
+To mark a PR as not requiring changelog notes, add the label 'no-changelog'.
+
+To run the same check that is run in CI to validate that your changelog fragments have
+the correct syntax, commit the fragment additions and then run `./scripts/check_changelog_fragments.sh`
+
+The format for fragments is: `<pr_number>.<fragment_type>.md`
+
+### Fragment conventions
+
+When fragments used to generate the updated changelog, the content of the fragment file is
+rendered as an item in a bulleted list under the "type" of fragment.
+
+The contents of the file must be valid markdown.
+
+Filename rules:
+
+- The first segment (pr_number) must match the GitHub PR the change is introduced in.
+- The type must be one of the valid types in [Fragment types](#types)
+- Only the two period delimiters can be used.
+- The file must be markdown.
+
+#### Fragment types {#types}
+
+- breaking: A change that is incompatible with prior versions which requires users to make adjustments.
+- security: A change that is has implications for security.
+- deprecation: A change that is introducing a deprecation.
+- feature: A change that is introducing a new feature.
+- enhancement: A change that is enhancing existing functionality in a user perceivable way.
+- fix: A change that is fixing a bug.
+
+#### Fragment contents
+
+When fragments are rendered in the changelog, each fragment becomes an item in a markdown list.
+For this reason, when creating the content in a fragment, the format must be renderable as a markdown list.
+
+As an example, separating content with markdown header syntax should be avoided, as that will render
+as a heading in the main changelog and not the list. Instead, separate content with newlines.
+
+### Breaking changes
+
+When using the type 'breaking' to add notes for a breaking change, these should be more verbose than
+other entries typically. It should include all details that would be relevant for the user to need
+to handle upgrading to the breaking change.
+
+## Example
+
+Here is an example of a changelog fragment that adds a breaking change explanation.
+
+    $ cat changelog.d/42_very_good_words.breaking.md
+    This change is so great. It's such a great change that this sentence
+    explaining the change has to span multiple lines of text.
+
+    It even necessitates a line break. It is a breaking change after all.

changelog.toml

@@ -0,0 +1,57 @@
+# Configuration for `towncrier` used during release process to auto-generate changelog.
+
+[tool.towncrier]
+directory = "changelog.d"
+filename = "CHANGELOG.md"
+start_string = "<!-- changelog start -->\n"
+underlines = ["", "", ""]
+title_format = "## [{version} ({project_date})]"
+issue_format = "https://github.com/vectordotdev/vrl/pull/{issue}"
+
+# The following configurations specify which fragment "types" are
+# allowed.
+#
+# If a change applies to more than one type, select the one it most
+# applies to. Or, if applicable, multiple changelog fragments can be
+# added for one PR. For example, if a PR includes a breaking change
+# around some feature, but also fixes a bug in the same part of the
+# code but is tangential to the breaking change, a separate
+# fragment can be added to call out the fix.
+
+# A change that is incompatible with prior versions which
+# requires users to make adjustments.
+[[tool.towncrier.type]]
+directory = "breaking"
+name = "Breaking Changes & Upgrade Guide"
+showcontent = true
+
+# A change that has implications for security.
+[[tool.towncrier.type]]
+directory = "security"
+name = "Security"
+showcontent = true
+
+# A change that is introducing a deprecation.
+[[tool.towncrier.type]]
+directory = "deprecation"
+name = "Deprecations"
+showcontent = true
+
+# A change that is introducing a new feature.
+[[tool.towncrier.type]]
+directory = "feature"
+name = "New Features"
+showcontent = true
+
+# A change that is enhancing existing functionality in a user
+# perceivable way.
+[[tool.towncrier.type]]
+directory = "enhancement"
+name = "Enhancements"
+showcontent = true
+
+# A change that is fixing a bug.
+[[tool.towncrier.type]]
+directory = "fix"
+name = "Fixes"
+showcontent = true

scripts/check_changelog_fragments.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+
+# This script is intended to run during CI, however it can be run locally by
+# committing changelog fragments before executing the script. If the script
+# finds an issue with your changelog fragment, you can un-stage the fragment
+# from being committed and fix the issue.
+
+CHANGELOG_DIR="changelog.d"
+
+# NOTE: If these are altered, update both the 'changelog.d/README.md' and
+#       'scripts/generate_release_chanbgelog.sh' accordingly.
+FRAGMENT_TYPES="breaking|security|deprecation|feature|enhancement|fix"
+
+if [ ! -d "${CHANGELOG_DIR}" ]; then
+  echo "No ./${CHANGELOG_DIR} found. This tool must be invoked from the root of the repo."
+  exit 1
+fi
+
+# diff-filter=A lists only added files
+FRAGMENTS=$(git diff --name-only --diff-filter=A origin/main ${CHANGELOG_DIR})
+
+if [ -z "$FRAGMENTS" ]; then
+  echo "No changelog fragments detected"
+  echo "If no changes  necessitate user-facing explanations, add the GH label 'no-changelog'"
+  echo "Otherwise, add changelog fragments to changelog.d/"
+  echo "For details, see 'changelog.d/README.md'"
+  exit 1
+fi
+
+# extract the basename from the file path
+FRAGMENTS=$(xargs -n1 basename <<< "${FRAGMENTS}")
+
+# validate the fragments
+while IFS= read -r fname; do
+
+  if [[ ${fname} == "README.md" ]]; then
+    continue
+  fi
+
+  echo "validating '${fname}'"
+
+  IFS="." read -r -a arr <<< "${fname}"
+
+  if [ "${#arr[@]}" -ne 3 ]; then
+    echo "invalid fragment filename: wrong number of period delimiters. expected '<pr_number>.<fragment_type>.md'. (${fname})"
+    exit 1
+  fi
+
+  if ! [[ "${arr[0]}" =~ ^[0-9]+$ ]]; then
+    echo "invalid fragment filename: first segment must be PR number. (${fname})"
+    exit 1
+  fi
+
+  if ! [[ "${arr[1]}" =~ ^(${FRAGMENT_TYPES})$ ]]; then
+    echo "invalid fragment filename: fragment type must be one of: (${FRAGMENT_TYPES}). (${fname})"
+    exit 1
+  fi
+
+  if [[ "${arr[2]}" != "md" ]]; then
+    echo "invalid fragment filename: extension must be markdown (.md): (${fname})"
+    exit 1
+  fi
+
+done <<< "$FRAGMENTS"
+
+echo "changelog additions are valid."

scripts/generate_release_changelog.sh

@@ -0,0 +1,120 @@
+#!/bin/bash
+
+set -o errexit
+
+README="README.md"
+CHANGELOG="CHANGELOG.md"
+CHANGELOG_DIR="changelog.d"
+CHANGELOG_CFG="changelog.toml"
+
+ask_continue() {
+  while true; do
+    local choice
+    read -r choice
+    case $choice in
+      y) break; ;;
+      n) exit 1; ;;
+      *) echo "Please enter y or n"; ;;
+    esac
+  done
+}
+
+cd $(dirname "$0")/..
+VRL_ROOT=$(pwd)
+
+VRL_VERSION=$(awk '/^version = "[0-9]+.[0-9]+.[0-9]+"/{print $3}' "${VRL_ROOT}"/Cargo.toml | tr -d '"')
+
+if ! [[ ${VRL_VERSION} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] ; then
+  echo "Error reading release version from Cargo.toml!"
+  exit 1
+else
+  echo
+  echo "[detected VRL release version: ${VRL_VERSION}]"
+fi
+
+if ! python3 -m pip show towncrier > /dev/null ; then
+  echo "towncrier installation missing. Please install with 'python3 -m pip install towncrier'"
+  exit 1
+fi
+
+LOCAL_CHANGELOG_DIR=${VRL_ROOT}/${CHANGELOG_DIR}
+
+if [ ! -d "${LOCAL_CHANGELOG_DIR}" ]; then
+  echo "No ${CHANGELOG_DIR} found in ${VRL_ROOT}!"
+  exit 1
+fi
+
+LOCAL_CHANGELOG_CFG=${VRL_ROOT}/${CHANGELOG_CFG}
+
+if [ ! -f "${LOCAL_CHANGELOG_CFG}" ]; then
+  echo "No ${CHANGELOG_CFG} found in ${VRL_ROOT}!"
+  exit 1
+fi
+
+################################################################################
+echo
+echo -n "[checking for changelog fragments..."
+
+HAVE_FRAGMENTS=false
+
+# changelog fragments that haven't been released are added at the root of the changelog dir.
+for f in "${LOCAL_CHANGELOG_DIR}"/*.md ; do
+  if [[ $(basename "$f") == "${README}" ]] ; then
+    continue
+  fi
+  HAVE_FRAGMENTS=true
+  echo " done.]"
+  break
+done
+
+if [ "${HAVE_FRAGMENTS}" = false ] ; then
+  echo " no changelog fragments were found! Exiting]"
+  exit 1
+fi
+
+################################################################################
+
+echo
+echo "[generating the changelog..."
+if python3 -m towncrier build \
+  --config "${LOCAL_CHANGELOG_CFG}" \
+  --dir "${VRL_ROOT}" \
+  --version "${VRL_VERSION}" \
+  --keep
+then
+  echo
+  echo " done]"
+else
+  echo
+  echo " failed!]"
+  exit 1
+fi
+
+################################################################################
+echo
+echo "[please review the ${CHANGELOG} for accuracy.]"
+
+echo
+echo "[continue? The next step will retire changelog fragments being released by removing them from the repo (y/n)?]"
+
+ask_continue
+
+################################################################################
+echo
+echo -n "[removing changelog fragments included in this release..."
+
+for f in "${LOCAL_CHANGELOG_DIR}"/*.md ; do
+  if [[ $(basename "$f") == "${README}" ]] ; then
+    continue
+  fi
+  if ! git rm -f "$f" ; then
+    echo "... failed to remove $f !]"
+    exit 1
+  fi
+done
+
+echo " done]"
+
+################################################################################
+echo
+echo "[please review the changes to local VRL repo checkout and create a PR.]"