Release Process

This document aims to outline the process that should be followed for cutting a new release of cert-manager. If you would like to know more about current releases and the timeline for future releases, take a look at the Supported Releases page.

Prerequisites

First, ensure that you have all the tools required to perform a cert-manager release:

1. Install the release-notes CLI:

   go install k8s.io/release/cmd/release-notes@v0.7.0
   

2. Install our cmrel CLI:

   go install github.com/cert-manager/release/cmd/cmrel@latest
   

3. Clone the cert-manager/release repo:

   # Don't clone it from inside the cert-manager repo folder.
   git clone https://github.com/cert-manager/release
   cd release
   

4. Install the gcloud CLI.

5. Login to gcloud:

   gcloud auth application-default login
   

6. Make sure gcloud points to the cert-manager-release project:

   gcloud config set project cert-manager-release
   export CLOUDSDK_CORE_PROJECT=cert-manager-release # this is used by cmrel
   

7. Get a GitHub access token here with no scope ticked. It is used only by the release-notes CLI to avoid API rate limiting since it will go through all the PRs one by one.

Minor releases

A minor release is a backwards-compatible ‘feature’ release. It can contain new features and bug fixes.

Release schedule

We aim to cut a new minor release once per month. The rough goals for each release are outlined as part of a GitHub milestone. We cut a release even if some of these goals are missed, in order to keep up release velocity.

Process for releasing a version

1. Make sure to note which type of release you are doing. That will be helpful in the next steps.

   Type of release	Example of git tag
   initial alpha release	v1.3.0-alpha.0
   subsequent alpha release	v1.3.0-alpha.1
   initial beta release	v1.3.0-beta.0
   subsequent beta release	v1.3.0-beta.1
   final release	v1.3.0
   patch release (or “point release”)	v1.3.1

2. (final release only) Make sure that a PR with the new upgrade document is ready to be merged on cert-manager/website. See for example, see upgrading-1.0-1.1.

3. Update the release branch:

   • (initial alpha, subsequent alpha and initial beta) The release branch should already exist (it was created at the end of the last final release). Update the release branch with the latest commits from the master branch, as follows:

     # Must be run from the cert-manager repo folder.
     git fetch --all
     git branch --force release-1.0 origin/release-1.0
     git checkout release-1.0
     git merge --ff-only origin/master # don't run for a point release!
     

   • (subsequent beta, patch release and final release): do nothing since things have been merged using /cherry-pick release-1.0.

     Note about the code freeze:

     The first beta starts a new “code freeze” period that lasts until the final release. Just before the code freeze, we fast-forward everything from master into the release branch.

     During the code freeze, we continue merging PRs into master as usual.

     We don’t fast-forward master into the release branch for the second (and subsequent) beta, and only /cherry-pick release-1.0 the fixes that should be part of the subsequent beta.

     We don’t fast-forward for patch releases and final releases; instead, we prepare these releases using the /cherry-pick release-1.0 command.

4. Push the new or updated release branch:

   1. Check that the origin remote is correct. To do that, run the following command and make sure it returns the upstream https://github.com/jetstack/cert-manager.git:

      # Must be run from the cert-manager repo folder.
      git remote -v | grep origin
      

   2. Push the release branch:

      # Must be run from the cert-manager repo folder.
      git push --set-upstream origin release-1.0
      

      GitHub permissions: git push will only work if you have the admin GitHub permission on the cert-manager repo to create or push to the branch, see prerequisites. If you do not have this permission, you will have to open a PR to merge master into the release branch), and wait for the PR checks to become green.

5. Generate and edit the release notes:

   1. Use the following two tables to understand how to fill in the four environment variables needed for the next step. These four environment variables are documented on the README for the Kubernetes release-notes tool.

      Variable	Description
      START_REV*	The git tag of the “previous”* release
      END_REV	Name of your release branch (inclusive)
      BRANCH	Name of your release branch
      RELEASE_VERSION	The git tag without the leading v

      Examples for each release type (e.g., initial alpha release):

      Variable	Example 1	Example 2	Example 2	Example 3	Example 4
      	initial alpha	subsequent alpha	beta release	final release	patch release
      	v1.3.0-alpha.0	v1.3.0-alpha.1	v1.3.0-beta.0	v1.3.0	v1.3.1
      START_REV*	v1.2.0	v1.3.0-alpha.0	v1.3.0-alpha.1	v1.2.0	v1.3.0
      END_REV	release-1.3	release-1.3	release-1.3	release-1.3	release-1.3
      BRANCH	release-1.3	release-1.3	release-1.3	release-1.3	release-1.3
      RELEASE_VERSION	1.3.0-alpha.0	1.3.0-alpha.1	1.3.0-beta.0	1.3.0	1.3.1

      *The git tag of the “previous” release (START_REV) depends on which type of release you count on doing. Look at the above examples to understand a bit more what those are.

      After finding out the value for each of the 4 environment variables, set the variables in your shell (for example, following the example 1):

      export START_REV="v1.2.0"
      export END_REV="release-1.3"
      export BRANCH="release-1.3"
      export RELEASE_VERSION="1.3.0-alpha.0"
      

   2. Generate release-notes.md at the root of your cert-manager repo folder with the following command:

      # Must be run from the cert-manger folder.
      export GITHUB_TOKEN=*your-token*
      go install k8s.io/release/cmd/release-notes@v0.7.0
      release-notes --debug --repo-path cert-manager \
        --org jetstack --repo cert-manager \
        --required-author "jetstack-bot" \
        --output release-notes.md
      

      The GitHub token **does not need any scope**. The token is required only to avoid rate-limits imposed on anonymous API users.

   3. Sanity check the notes:

      • Make sure you haven’t duplicated the final release note from the previous release (START_REV and END_REV are inclusive)
      • Make sure the notes contain details of all the features and bug fixes that you expect to be in the release.
      • Add additional blurb, notable items and characterize change log.

      You can see the commits that will go into this release by using the GitHub compare. For example, while releasing v1.0.0, you want to compare it with the latest pre-released version v1.0.0-beta.1:

      https://github.com/jetstack/cert-manager/compare/v1.0.0-beta.1...master

   4. (final release only) Check the release notes include all changes since the last final release.

6. Run cmrel stage:

   1. In this example we stage a release using the ‘release-1.0’ branch, setting the release version to v1.0.0:

      # Must be run from the "cert-manager/release" repo folder.
      cmrel stage --branch=release-1.0 --release-version=v1.0.0
      

      This step takes ~10 minutes. It will build all Docker images and create all the manifest files, sign Helm charts and upload everything to a storage bucket on Google Cloud. These artifacts will then be published and released in the next steps.

      🔰 Remember to keep open the terminal where you run cmrel stage. Its output will be used in the next step.

   2. While the build is running, send a first Slack message to #cert-manager-dev:

      Releasing 1.2.0-alpha.2 🧵

      🔰 Please have a quick look at the build log as it might contain some unredacted data that we forgot to redact. We try to make sure the sensitive data is properly redacted but sometimes we forget to update this.

   3. Send a second Slack message in reply to this first message with the Cloud Build job link that cmrel displayed in “View logs at”. For example, the message would look like:

      Follow the cmrel stage build: https://console.cloud.google.com/cloud-build/builds/7641734d-fc3c-42e7-9e4c-85bfc4d1d547?project=1021342095237

7. Run cmrel publish:

   1. Set the CMREL_RELEASE_NAME variable in your shell. The value for the CMREL_RELEASE_NAME variable is found in the output of the previous command, cmrel stage. Look for the line that contains the gs:// link:

      gs://cert-manager-release/stage/gcb/release/v1.3.0-alpha.1-c2c0fdd78131493707050ffa4a7454885d041b08
      #                                           <---------- CMREL_RELEASE_NAME ----------------------->
      

      Copy that part into a variable in your shell (no need to export it):

      CMREL_RELEASE_NAME=v1.3.0-alpha.0-77b045d159bd20ce0ec454cd79a5edce9187bdd9
      

   2. Do a cmrel publish dry-run to ensure that all the staged resources are valid. Run the following command:

      # Must be run from the "cert-manager/release" repo folder.
      cmrel publish --skip-signing --release-name "$CMREL_RELEASE_NAME"
      

      You can view the progress by clicking the Google Cloud Build URL in the output of this command.

   3. While the build is running, send a third Slack message in reply to the first message:

      Follow the `cmrel publish` dry-run build: https://console.cloud.google.com/cloud-build/builds16f6f875-0a23-4fff-b24d-3de0af207463?project=1021342095237

   4. Next publish the release artifacts for real. The following command will publish “for real” the artifacts to GitHub, Quay.io, to our ChartMuseum instance:

      # Must be run from the "cert-manager/release" repo folder.
      # Skip signing while quay.io doesn't support cosign signatures
      cmrel publish --nomock --skip-signing --release-name "$CMREL_RELEASE_NAME"
      

      ⏰ Upon completion there will be:

      1. A draft release of cert-manager on GitHub.
      2. A pull request containing the new Helm chart.

   5. While the build is running, send a fourth Slack message in reply to the first message:

      Follow the cmrel publish build: https://console.cloud.google.com/cloud-build/builds/b6fef12b-2e81-4486-9f1f-d00592351789?project=1021342095237

8. Publish the GitHub release:

   1. Visit the draft GitHub release and paste in the release notes that you generated earlier. You will need to manually edit the content to match the style of earlier releases. In particular, remember to remove package-related changes.

   2. (initial alpha, subsequent alpha and beta only) Tick the box “This is a pre-release”.

   3. Click “Publish” to make the GitHub release live. This will create a Git tag automatically.

9. Merge the pull request containing the Helm chart:

   The Helm charts for cert-manager are served using Cloudflare pages and the Helm chart files and metadata are stored in the Jetstack charts repository. The cmrel publish --nomock step (above) will have created a PR in this repository which you now have to review and merge, as follows:

   1. Visit the pull request
   2. Review the changes
   3. Fix any failing checks
   4. Merge the PR
   5. Check that the cert-manager Helm chart is visible on ArtifactHUB.

10. (final release only) Add the new final release to the supported-releases page.

11. Post a Slack message as an answer to the first message. Toggle the check box “Also send to #cert-manager-dev” so that the message is well visible. Also cross-post the message on #cert-manager.

    https://github.com/jetstack/cert-manager/releases/tag/v1.0.0 🎉

12. (final release only) Show the release to the world:

    1. Send an email to cert-manager-dev@googlegroups.com with the release label (examples).

    2. Send a tweet (example) and make sure @JetstackHQ retweets it.

13. Proceed to the post-release steps:

    1. (initial alpha only) Create a PR on cert-manager/release in order to re-enable the next periodic tests configured in:

       config/jobs/cert-manager/release-next/cert-manager-release-next-periodics.yaml
       

       Why? Because we disable the “next” periodic tests right after a final release since next periodics are only useful after we do the first alpha (e.g., in PR 606).

    2. (initial alpha only) Open a PR to cert-manager/website in order to:

       • Update the section “How we determine supported Kubernetes versions” on the supported-releases page. In the table, change the “next periodic” line with the correct links.

    3. (final release only) Create a PR on cert-manager/release in order to disable the next periodic tests configured in:

       config/jobs/cert-manager/release-next/cert-manager-release-next-periodics.yaml
       

       (just remove the file and commit)

       Why? Because that saves us compute time between a final release and the first alpha.

    4. (final release only) Open a PR to cert-manager/website in order to:

       • Update the section “Supported releases” in the supported-releases page.
       • Update the section “Supported releases” in the supported-releases page.
       • Update the section “How we determine supported Kubernetes versions” on the supported-releases page. In the table, set “n/a” for the line where “next periodic” is since these tests will be disabled until we do our first alpha.

    5. (final release only) Open a PR to jetstack/testing and change Prow’s config. To do this, take inspiration from Maartje’s PR example.

    6. (final release only) Push a new release branch to jetstack/cert-manager. If the final release is v1.0.0, then push the new branch release-1.1:

       # Must be run from the cert-manager repo folder.
       git checkout -b release-1.1 v1.0.0
       git push origin release-1.1
       

    7. (final release only) Open a PR to cert-manager/website with updates to the website configuration. To do this, take inspiration from Maartje’s PR example.

    8. Ensure that any installation commands in cert-manager/website install the latest version. This should be done after every release, including patch releases as we want to encourage users to always install the latest patch.