Rancher agents can be hijacked by taking over the Rancher Server URL

Impact

A vulnerability has been identified within Rancher that can be exploited in narrow circumstances through a man-in-the-middle (MITM) attack. An attacker would need to have control of an expired domain or execute a DNS spoofing/hijacking attack against the domain to exploit this vulnerability. The targeted domain is the one used as the Rancher URL.

SUSE is unaware of any successful exploitation of this vulnerability, which has a high complexity bar.

Please consult the associated MITRE ATT&CK - Technique - Adversary-in-the-Middle for further information about this attack category.

Patches

A new setting, agent-tls-mode, was added, which allows users to specify if agents will use strict certificate verification when connecting to Rancher. The field can be set to strict (which requires the agent to verify the certificate using only the Certificate Authority in the cacerts setting) or system-store (which allows the agent to verify the certificate using any Certificate Authority in the operating system's trust store). This setting will default to strict on new installs of Rancher v2.8.6, v2.9.0, and newer versions. When upgrading from a prior version, the current value will be kept. If updating from older versions, the settings must be manually configured.

Important: For non-Windows nodes, this is fixed since v2.8.6 and v2.9.0. For Windows nodes, this is fully fixed starting with v2.8.8 and v2.9.2

Patched versions include releases v2.8.8 and v2.9.2.

For non-Windows nodes, the fix was released with v2.7.15. However, if you are running Rancher v2.7.x and have Windows nodes, you must follow the below workaround to address this issue on those nodes.

Workarounds

If you can't update, please follow the standard security practices including:

1. Properly control the expiration and ownership of the domain used as the Rancher URL (the server-url of the Rancher cluster).
2. Enabling DNSSEC as a way to protect against DNS spoofing or hijacking attacks.
3. Properly clean up and decommission unused clusters and downstream clusters, instead of leaving them behind. For example, downstream clusters which are alive while the main Rancher server is no longer available.

In some cases, Windows nodes added to RKE2 clusters may not be automatically updated with the desired agent-tls-mode. Windows clusters running at least the August patches (v1.27.16, v1.28.13, v1.29.8, v1.30.4) will be automatically updated. For Windows nodes running older versions of RKE2, this issue can be manually resolved by following these instructions.

If you are running Rancher v2.7.x Windows nodes will not automatically update, and you must follow the above instructions, with the following notes:

1. This needs to be done for all existing Windows nodes and any new nodes provisioned.
2. You must omit the DownloadWins flag, and must instead manually download the rancher-wins version 0.4.18, or greater, from its GitHub repository and place it in the required directories: a. c:\Windows b. c:\user\local\bin
3. You must restart the nodes after running the script, simply restarting rancher-wins or RKE2 will result in pod networking errors. The only scenario where you do not need to completely restart the node is if the cluster is running version v1.27.16 or higher.

Credits

This issue was found and reported by Jarkko Vesiluoma from Redtest Security.

For more information

If you have any questions or comments about this advisory:

• Reach out to the SUSE Rancher Security team for security related inquiries.
• Open an issue in the Rancher repository.
• Verify with our support matrix and product support lifecycle.