Document a known issue with MySQL when upgrading to GHES 3.9 (#39751)

Co-authored-by: Laura Coursen <lecoursen@github.com>
Co-authored-by: Jiaqi Liu <1573931+itsJiaqi@users.noreply.github.com>
Co-authored-by: Rachael Rose Renk <91027132+rachaelrenk@users.noreply.github.com>

content/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance.md

@@ -108,4 +108,78 @@ Finally, if you're willing to help {% data variables.product.company_short %}  u
 
 The data you submit helps {% data variables.product.company_short %} continue to provide a performant product, but {% data variables.product.company_short %} does not guarantee any additional mitigation steps or changes to the product as a result of the data you provide.
 
+## MySQL does not start after upgrade to {% data variables.product.prodname_ghe_server %} 3.9
+
+During an upgrade to {% data variables.product.prodname_ghe_server %} 3.9, if MySQL did not gracefully shut down during the shutdown of the {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 instance, MySQL will attempt to go through crash recovery when the {% data variables.product.prodname_ghe_server %} 3.9 instance starts up. Since {% data variables.product.prodname_ghe_server %} 3.7 and 3.8 uses MySQL 5.7 and {% data variables.product.prodname_ghe_server %} 3.9 has been upgraded to MySQL 8.0, MySQL will not be able to complete crash recovery.
+
+If you experience this problem, the following error will be in the mysql error log (`/var/log/mysql/mysql.err`):
+
+```shell copy
+[ERROR] [MY-012526] [InnoDB] Upgrade after a crash is not supported. This redo log was created with MySQL 5.7.40. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
+```
+
+### Avoiding this issue
+
+To avoid this issue, update the nomad timeout for MySQL before starting an upgrade to {% data variables.product.prodname_ghe_server %} 3.9
+
+1. Put your instance into maintenance mode:
+
+  ```shell copy
+  ghe-maintenance -s
+  ```
+1. Update consul template for nomad:
+
+  ```shell copy
+  sudo sed -i.bak '/kill_signal/i \      kill_timeout = "10m"' /etc/consul-templates/etc/nomad-jobs/mysql/mysql.hcl.ctmpl
+  ```
+1. Render consul template for nomad:
+
+  ```shell copy
+  sudo consul-template -once -template /etc/consul-templates/etc/nomad-jobs/mysql/mysql.hcl.ctmpl:/etc/nomad-jobs/mysql/mysql.hcl
+  ```
+1. Verify current `kill_timeout` setting:
+
+  ```shell copy
+  nomad job inspect mysql | grep KillTimeout
+  ```
+  
+  Expected response:
+  
+  ```shell copy
+  "KillTimeout": 5000000000
+  ```
+1. Stop MySQL:
+
+  ```shell copy
+  nomad job stop mysql
+  ```
+1. Run new MySQL job:
+
+  ```shell copy
+  nomad job run /etc/nomad-jobs/mysql/mysql.hcl
+  ```
+1. Verify kill_timeout has been updated:
+
+  ```shell copy
+  nomad job inspect mysql | grep KillTimeout
+  ```
+  Expected response:
+  
+  ```shell copy
+  "KillTimeout": 600000000000,
+  ```
+1. Take instance out of maintenance mode:
+
+  ```shell copy
+  ghe-maintenance -u
+  ```
+
+Now that the nomad timeout for MySQL has been updated you can upgrade your {% data variables.product.prodname_ghe_server %} instance to 3.9.
+
+### Mitigating a failed restart of MySQL
+
+If you're affected by this problem, restore your {% data variables.product.prodname_ghe_server %} instance to the state it was in prior to the upgrade attempt, and then follow the steps from the previous section.
+
+For more information about restoring from a failed upgrade, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/upgrading-github-enterprise-server#restoring-from-a-failed-upgrade)."
+
 {% endif %}

data/release-notes/enterprise-server/3-7/0.yml

@@ -322,6 +322,8 @@ sections:
     - 'The maximum number of self-hosted runners in a runner group is limited to 10,000. Previously, there was no limit. [Updated: 2023-05-24]'
 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -378,4 +380,4 @@ sections:
 
     # https://github.com/github/releases/issues/2042
     - |
-      "[Features](#3.7.0-features)" incorrectly indicated that users of the GitHub Advisory Database can see advisories for Elixir, Erlang's Hex package manager, and more. This feature is unavailable in GitHub Enterprise Server 3.7, and will be available in a future release. [Updated 2023-06-01]
+      "[Features](#3.7.0-features)" incorrectly indicated that users of the GitHub Advisory Database can see advisories for Elixir, Erlang's Hex package manager, and more. This feature is unavailable in GitHub Enterprise Server 3.7, and will be available in a future release. [Updated 2023-06-01]

data/release-notes/enterprise-server/3-7/1.yml

@@ -26,6 +26,8 @@ sections:
     - If a site administrator has not yet configured GitHub Actions for the instance, the UI for setting up code scanning will prompt the user to configure GitHub Actions.
     - To avoid failing domain verification due to the 63-character limit enforced by DNS providers for DNS records, the GitHub-generated `TXT` record to verify domain ownership is now limited to 63 characters.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.

data/release-notes/enterprise-server/3-7/10.yml

@@ -24,6 +24,8 @@ sections:
     - |
       People with administrative SSH access who generate a support bundle using the `ghe-support-bundle` or `ghe-cluster-support-bundle` utilities can specify the period of time to gather data with `-p` or `--period` without using spaces or quotes. For example, in addition to `'-p 5 days'` or `-p '4 days 10 hours'`,  `-p 5days` or `-p 4days10hours` are valid.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/11.yml

@@ -12,6 +12,8 @@ sections:
   changes:
     - People with administrative SSH access to an instance can configure the maximum memory usage in gigabytes for Redis using `ghe-config redis.max-memory-gb VALUE`. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/12.yml

@@ -13,6 +13,8 @@ sections:
   changes:
     - If a configuration runs fails due to Elasticsearch errors, `ghe-config-apply` displays a more actionable error message. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/13.yml

@@ -56,6 +56,8 @@ sections:
     - |
       The Management Console displays a warning about unexpected consequences that may result from modification of the instance's hostname after initial configuration. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-7/2.yml

@@ -26,6 +26,8 @@ sections:
     - A user's list of recently accessed repositories no longer includes deleted repositories.
     - '{% data reusables.release-notes.scim-custom-mappings-supported-change %}'
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.

data/release-notes/enterprise-server/3-7/3.yml

@@ -22,6 +22,8 @@ sections:
     - The performance of configuration runs started with `ghe-config-apply` has been improved.
     - When exporting account data, backing up a repository, or performing a migration, the link to a repository archive now expires after 1 hour. Previously the archive link expired after 5 minutes.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.

data/release-notes/enterprise-server/3-7/4.yml

@@ -4,6 +4,8 @@ sections:
     - |
       {% data reusables.release-notes.2023-01-git-vulnerabilities %}
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -18,4 +20,4 @@ sections:
     - '{% data reusables.release-notes.babeld-max-threads-performance-issue %}'
     - '{% data reusables.release-notes.git-push-known-issue %}'
     - '{% data reusables.release-notes.replication-commands-in-maintenance-mode-known-issue %}'
-    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'
+    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'

data/release-notes/enterprise-server/3-7/5.yml

@@ -12,6 +12,8 @@ sections:
     - The additional committers count for GitHub Advanced Security always showed 0.
     - In some cases, users were unable to convert existing issues to discussions. If an issue is stuck while being converted to a discussion, enterprise owners can review the "Known issues" section below for more information.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -26,4 +28,4 @@ sections:
     - '{% data reusables.release-notes.stuck-discussion-conversion-issue %}'
     - '{% data reusables.release-notes.git-push-known-issue %}'
     - '{% data reusables.release-notes.replication-commands-in-maintenance-mode-known-issue %}'
-    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'
+    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'

data/release-notes/enterprise-server/3-7/6.yml

@@ -15,6 +15,8 @@ sections:
   changes:
     - After the Dependency submission REST API receives a submission with one or more dependencies without a version, the dependency graph will now correctly report this fact.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -29,4 +31,4 @@ sections:
     - '{% data reusables.release-notes.stuck-discussion-conversion-issue %}'
     - '{% data reusables.release-notes.git-push-known-issue %}'
     - '{% data reusables.release-notes.replication-commands-in-maintenance-mode-known-issue %}'
-    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'
+    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'

data/release-notes/enterprise-server/3-7/7.yml

@@ -8,6 +8,8 @@ sections:
     - |
       In the rare case when primary shards for Elasticsearch were located on a replica node, the `ghe-repl-stop` command would fail with `ERROR: Running migrations`.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -22,4 +24,4 @@ sections:
     - '{% data reusables.release-notes.stuck-discussion-conversion-issue %}'
     - '{% data reusables.release-notes.git-push-known-issue %}'
     - '{% data reusables.release-notes.replication-commands-in-maintenance-mode-known-issue %}'
-    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'
+    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'

data/release-notes/enterprise-server/3-7/8.yml

@@ -41,6 +41,8 @@ sections:
     - To avoid intermittent issues with the success of Git operations on an instance with multiple nodes, GitHub Enterprise Server checks the status of the MySQL container before attempting a SQL query. The timeout duration has also been reduced. 
     - The default path for output from `ghe-saml-mapping-csv -d` is `/data/user/tmp` instead of `/tmp`. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-saml-mapping-csv)."
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/9.yml

@@ -18,6 +18,8 @@ sections:
   changes:
     - If a site administrator provides an invalid configuration for blob storage for GitHub Actions or GitHub Packages on an instance, the preflight checks page displays details and troubleshooting information.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-8/0.yml

@@ -436,6 +436,8 @@ sections:
       Before squash-merging a pull request, the web UI displays the email address of the commit's author. Previously, the commit author was only displayed when merging with a merge commit.
 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -480,4 +482,4 @@ sections:
           For integrators who wish to receive webhooks for Dependabot alerts activity, the `dependabot_alert` webhook replaces the `repository_vulnerability_alert` webhook. For more information, see "[Webhook events and payloads](/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#dependabot_alert)."
 
   errata:
-    - '{% data reusables.release-notes.github-actions-secrets-encryption-docs %}'
+    - '{% data reusables.release-notes.github-actions-secrets-encryption-docs %}'

data/release-notes/enterprise-server/3-8/1.yml

@@ -37,6 +37,8 @@ sections:
     - The default path for output from `ghe-saml-mapping-csv -d` is `/data/user/tmp` instead of `/tmp`. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-saml-mapping-csv)."
     - On an instance with a GitHub Advanced Security license, users who author custom patterns for secret scanning can provide expressions that must or must not match that are up to 2,000 characters. This limit is an increase from 1,000 characters.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-8/2.yml

@@ -26,6 +26,8 @@ sections:
       After a site administrator exports a migration archive using GitHub Enterprise Importer's `gh-migrator` utility, the link to the archive remains accessible for 48 hours instead of one hour.
     - On an instance with a GitHub Advanced Security license, users who author custom patterns for secret scanning can provide expressions that must or must not match that are up to 2,000 characters. This limit is an increase from 1,000 characters.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-8/3.yml

@@ -26,6 +26,8 @@ sections:
       People with administrative SSH access who generate a support bundle using the `ghe-support-bundle` or `ghe-cluster-support-bundle` utilities can specify the period of time to gather data with `-p` or `--period` without using spaces or quotes. For example, in addition to `'-p 5 days'` or `-p '4 days 10 hours'`,  `-p 5days` or `-p 4days10hours` are valid.
     - After a site administrator exports a migration archive using GitHub Enterprise Importers `gh-migrator` utility, the link to the archive remains accessible for 48 hours instead of one hour. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-8/4.yml

@@ -15,6 +15,8 @@ sections:
   changes:
     - People with administrative SSH access to an instance can configure the maximum memory usage in gigabytes for Redis using `ghe-config redis.max-memory-gb VALUE`. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-8/5.yml

@@ -17,6 +17,8 @@ sections:
   changes:
     - If a configuration runs fails due to Elasticsearch errors, `ghe-config-apply` displays a more actionable error message. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-8/6.yml

@@ -78,6 +78,8 @@ sections:
     - |
       The Management Console displays a warning about unexpected consequences that may result from modification of the instance's hostname after initial configuration. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-9/0.yml

@@ -376,7 +376,9 @@ sections:
 
   known_issues:
     - |
-      If you upgrade from {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 to 3.9, the database server on your instance will be upgraded from MySQL 5.7 to MySQL 8.0. I/O utilization will increase as a result, and in some cases this may affect your instance's performance. Do not upgrade to this RC in a production environment, and ensure that you take and verify a backup of the instance before upgrading to the GA release. For more information, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance)."
+      After an administrator upgrades from {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 to 3.9, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance)."
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-9/1.yml

@@ -92,6 +92,8 @@ sections:
     - |
       On an instance with multiple nodes, internal tooling to repair repositories now attempts to resolve problems within the entire repository network.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       The `mbind: Operation not permitted` error in the `/var/log/mysql/mysql.err` file can be ignored. MySQL 8 does not gracefully handle when the `CAP_SYS_NICE` capability isn't required, and outputs an error instead of a warning.
     - |

data/reusables/enterprise/upgrade-to-3-9-or-to-3-10-mysql-cannot-start-up.md

@@ -0,0 +1 @@
+After an administrator upgrades from {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 to 3.9, MySQL may not start back up. For more information, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance#mysql-does-not-start-after-upgrade-to-github-enterprise-server-39)."

data/reusables/release-notes/upgrade-mysql8-cannot-start-up.md

@@ -0,0 +1 @@
+{% data reusables.enterprise.upgrade-to-3-9-or-to-3-10-mysql-cannot-start-up %} [Updated: 2023-07-27]