docs/data/release-notes/enterprise-server/3-9/1.yml

date: '2023-07-18'
intro: |
  {% warning %}

  **Warning**: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "[Known issues](#3.9.1-known-issues)" section of these release notes.

  The known issues originally published on 2023-07-18 omitted a number of known issues that still existed. The `Known issues` section below was updated on 2023-08-08.
  {% endwarning %}
sections:
  security_fixes:
    - |
      **MEDIUM**: An attacker with write access to a repository could craft a pull request that would hide commits made in its source branch. This vulnerability was reported via the [GitHub Bug Bounty Program](https://bounty.github.com/) and has been assigned [CVE-2023-23764](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-23764). [Updated: 2023-07-26]
    - |
      An attacker with access to the password hash of the root site administrator user for the instance's Management Console could make requests to the password API endpoint from outside of the instance.
    - |
      Packages have been updated to the latest security versions.
    - |
      **LOW:**  An incorrect comparison vulnerability was identified in GitHub Enterprise Server that allowed commit smuggling by displaying an incorrect diff in a re-opened Pull Request. To exploit this vulnerability, an attacker would need write access to the repository. This vulnerability was reported via the [GitHub Bug Bounty Program](https://bounty.github.com/) and was assigned [CVE-2023-23765](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-23765).
  bugs:
    - |
      If MinIO was configured for external blob storage on an instance with GitHub Actions enabled and MinIO was configured for bucket replication, the instance's credential validation with MinIO would occasionally fail.
    - |
      Customers who use Azure Blob store as the remote blob provider to back GitHub Packages would have validation errors if the `EndpointSuffix` part of their Connection string was anything other than `core.windows.net`.  Now all valid `EndpointSuffix` are accepted.
    - |
      When a user viewed a Jupyter notebook, GitHub Enterprise Server returned a `500` error code if the instance was configured with a self-signed TLS certificate.
    - |
      After creation of a blob object from the web UI, pre-receive hook events were missing from the instance's audit log.
    - |
      On an instance with an outbound web proxy server configured, the proxy interfered with internal operations that used `nomad alloc exec`.
    - |
      On an instance in a cluster configuration, the `ghe-cluster-balance` behaved inconsistently when displaying status or managing jobs with more than one task group.
    - |
      `.topojson` files would not render correctly, but files that conformed to the TopoJSON spec that used a `.geojson` extension would render correctly.
    - |
      On an instance configured for LDAP authentication, if the LDAP server sent an empty string for the `sshPublicKey` attribute, LDAP user sync would fail.
    - |
      REST API endpoints for managing GitHub Enterprise Server are now functional. For more information, see "[Manage GitHub Enterprise Server](/rest/enterprise-admin/manage-ghes?apiVersion=2022-11-28)" in the REST API documentation.
    - |
      After creation of a new Management Console user, the Management Console did not display the button to copy the new users invitation.
    - |
      On an instance with Dependabot enabled, in some situations, Dependabot alerts were not updated when a user pushed to a repository.
    - |
      In some cases, pull requests with more than 25 rich-diff renderable files required that users toggle the diff type to correctly render the files over the 25-file limit.
    - |
      In rare circumstances, Git commits signed with SSH keys using the RSA algorithm would incorrectly indicate the signature was invalid.
    - |
      After a migration using GitHub Enterprise Importer, some repository autolink references were created with an incorrect format.
    - |
      In some cases on an instance without a GitHub Advanced Security license, Redis exceeded the maximum default memory allocation, causing `500` errors for the instance's users.
    - |
      On an instance with many organizations, the enterprise security overview page returned a `500` error.
    - |
      On an instance that was not configured to deliver email notifications using SMTP, background jobs to deliver email were enqueued unnecessarily.
    - |
      Users were unable to configure a SSH certificate authority for an organization.
    - |
      An erroneous "Blocked Copilot Repositories" link was visible in site admin pages for organizations.
    - |
      On an instance with GitHub Actions enabled and a GitHub Advanced Security license, repository-level runner scale sets were not accounted for when determining whether default setup for code scanning could be used.
    - |
      Events related to repository notifications did not appear in the audit log.
    - |
      On an instance with a GitHub Advanced Security license and secret scanning enabled, in some cases, a committer would not receive an email notification for a secret scanning alert where push protections were bypassed.
    - |
      On an instance with a GitHub Advanced Security license, if a user filtered by a custom pattern on an organizations "Code & security analysis" page using an invalid query, the entire GitHub Advanced Security disappeared and an error reading "Sorry, something went wrong loading GitHub Advanced Security settings" appeared.
    - |
      On an instance with a GitHub Advanced Security license, if a user browsed to the alerts page for secret scanning without signing in, the instance responded with a `500` error.
    - |
      On an instance with a GitHub Advanced Security license and secret scanning enabled, output from Git for a push blocked by push protection always included an `http://` link.
    - |
      On an instance with GitHub Actions enabled, links to `http(s)://HOSTNAME/features/actions` from the web UI returned a `500` error.
    - |
      If a user added a new item to a projects roadmap view, and the item was outside of the viewport, the view would crash and display "This project failed to load".
    - |
      The audit log reported the incorrect target repository for pre-receive hook failures.
    - |
      Users can add issues and pull requests from any organization to a project, and are no longer limited to the user or organization of the project.
    - |
      On an instance with GitHub Actions enabled and a GitHub Advanced Security license, enterprise-level runner scale sets with the `code-scanning` label were not sufficient to allow default setup for code scanning.
    - |
      On an instance in a high availability configuration, existing nodes with out-of-sync repositories prevented new nodes from replicating those repositories.
    - |
      On an instance with multiple nodes, `ERROR`-level "resolver failed" errors no longer appear in system logs when the instance is unable to resolve an offline fileserver. The messages are now `DEBUG`-level.
    - |
      On an instance with a GitHub Advanced Security license that was also configured for a timezone greater than UTC, the list of secret scanning alerts displayed a "Loading secrets failed" error if a user sorted secrets by date in descending order.
    - |
      Code Scanning workflow runs now only request the `code-scanning` label so that they can be used with runner scale sets.
  changes:
    - |
      On an instance in a cluster configuration, the `ghe-cluster-config-check` command-line utility will return an affirmative message when no warnings or errors are detected. The affirmative message is "Configuration validation complete. No errors found."
    - |
      During initialization of a cluster configuration, output from the `ghe-cluster-config-init` command-line utility is improved and simplified.
    - |
      The API endpoint for management of the GitHub Enterprise Server instance was unavailable prior to initial configuration of the instance.
    - |
      The Management Console displays a warning about unexpected consequences that may result from modification of the instance's hostname after initial configuration.
    - |
      On an instance with multiple nodes, internal tooling to repair repositories now attempts to resolve problems within the entire repository network.
    - |
      To supplement a disaster recovery plan for a GitHub Enterprise Server instance in a cluster configuration, an administrator can configure a replica of an entire cluster in a separate datacenter, allowing the cluster to fail over to redundant nodes. For more information, see "[AUTOTITLE](/admin/monitoring-managing-and-updating-your-instance/configuring-clustering/configuring-high-availability-replication-for-a-cluster)."
  known_issues:
    - |
      {% data reusables.release-notes.upgrade-to-3-9-or-to-3-10-io-utilization-increase %}
    - |
      {% data reusables.release-notes.upgrade-to-3-9-or-to-3-10-mysql-cannot-start-up %} [Updated: 2023-08-11]
    - |
      {% data reusables.release-notes.2023-10-git-push-made-but-not-registered %} [Updated: 2023-10-26]
    - |
      The Management Console may get stuck in a loop showing "Checking requirements..." when attempting to download an update. It is safe to proceed the update process manually via the command line.
    - |
      When enabling CodeQL via default setup [at scale](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-at-scale), some checks related to GitHub Actions are omitted, potentially preventing the process from completing.
    - |
      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %} [Updated: 2023-07-31]
    - |
      On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run `ghe-config-apply`.

      ```shell
      Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
      ```

      You can safely ignore this message.
    - |
      Custom firewall rules are removed during the upgrade process.

    - |
      On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.
    - |
      The `mbind: Operation not permitted` error in the `/var/log/mysql/mysql.err` file can be ignored. MySQL 8 does not gracefully handle when the `CAP_SYS_NICE` capability isn't required, and outputs an error instead of a warning.
    - |
      When using an outbound web proxy server, the `ghe-btop` command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".
    - |
      If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using `ghe-ssl-ca-certificate-install` are not respected, and connections to the server fail.
    - |
      When running `ghe-config-apply`, the process may stall with the message `Deployment is running pending automatic promotion`.
    - |
      During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.
    - |
      If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "[AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account)."
    - |
      {% data reusables.release-notes.mermaid-rendering-known-issue %} [Updated: 2023-08-18]
    - |
      {% data reusables.release-notes.migrations-missing-section-known-issue %} [Updated: 2023-08-18]
    - |
      {% data reusables.release-notes.2023-08-mssql-replication-known-issue %} [Updated: 2023-08-24]
    - |
      On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a `500` error. You can avoid the issue by trying one of the following workarounds.

      - Add a runner scale set to the enterprise runner group shared with the repositories.
      - Remove access to the enterprise runner group from the affected repositories or organizations.

      [Updated: 2023-09-05]
    - |
      {% data reusables.release-notes.2023-09-config-apply-timeout-hookshot-go-replicas %} [Updated: 2023-09-21]
    - |
      {% data reusables.release-notes.2023-09-ephemeral-self-hosted-runners-not-auto-upgrading %} [Updated: 2023-09-29]
    - |
      {% data reusables.release-notes.2023-10-resource-activity-queue-not-processed %} [Updated: 2023-10-10]
    - |
      {% data reusables.release-notes.2023-10-support-bundle-p-flag-not-working %} [Updated: 2023-10-13]
    - |
      {% data reusables.release-notes.scheduled-reminders-unintentional %} [Updated: 2023-10-17]
    - |
      {% data reusables.release-notes.2023-10-actions-upgrade-bug %} [Updated: 2023-12-04]
    - |
      {% data reusables.release-notes.2023-11-aws-system-time %} [Updated 2023-11-10]
    - |
      {% data reusables.release-notes.2023-12-backup-utils-exit-early-redis %} [Updated 2023-12-05]
    - |
      {% data reusables.release-notes.2023-12-client-ip-addresses-incorrect-in-audit-log %} [Updated 2023-12-13]
    - |
      {% data reusables.release-notes.2024-01-ha-proxy-out-of-memory %} [Updated 2024-01-23]
    - |
      {% data reusables.release-notes.2024-03-increased-log-volume-in-syslog %} [Updated: 2024-03-08]