Ray’s Azure autoscaler worked with its pinned SDK, but newer versions broke the same code. We traced the failure, built a backward-compatible patch and used reviewer feedback to catch a second break our first tests missed.
Last verified 2026-10-09 21:41 UTC
We were investigating which downstream repositories needed a fix after an SDK release. Ray gave us a concrete example: its Azure autoscaler relied on deployment APIs that Azure had moved in azure-mgmt-resource 25.0.0.
Ray’s tests pinned SDK 24. That version still supported the old calls. Install a newer SDK, and the same code could fail while creating Azure resources. Updating the dependency version would leave the broken calls in place.
I took on the migration using Ziek for investigation, implementation support and validation. The constraint was to support the newer SDK without breaking existing SDK 24 installations.
Azure had moved deployment operations into azure-mgmt-resource-deployments. We added a helper that uses its dedicated client when the package is installed and falls back to the original resource client for SDK 24. If neither interface is available, the error names the package to install.
We also updated Ray’s Azure install paths and dependency locks. The patch became PR #66714, where Ray contributors could review the implementation.
The import fallback worked on SDK 24 and 25. A Ray reviewer tried SDK 26 and found another failure: query parameters such as api_version and filter had become keyword-only.
Our mocks accepted positional arguments, so they let invalid calls pass. The pinned SDK accepted them too. We changed seven call sites to name their arguments, using a call style that all three SDK versions accept.
Then we replaced the permissive mock with a small fake that enforces SDK 26’s signatures. Python now rejects an invalid positional call even when the test environment uses SDK 24.
Two small edits solved the compatibility problem and made the patch survive the reviewer’s SDK 26 checks.
from azure.mgmt.resource.resources import ResourceManagementClient
try:
from azure.mgmt.resource.deployments import DeploymentsMgmtClient
except ImportError:
DeploymentsMgmtClient = None
if DeploymentsMgmtClient is None:
return resource_clientexisting_vnets = list(
list_by_rg(
resource_group,
filter=(
f"substringof('{unique_id}', name) and "
"resourceType eq 'Microsoft.Network/virtualNetworks'"
),
)
)
subnet = get_by_id(resource_id=vnid, api_version=vnet_api_version)We reverted each of the seven argument fixes individually. Each revert caused at least one test to fail, including when running against SDK 24. With the fixes restored, our follow-up evaluation recorded 39 tests and 10 subtests passing in each environment below. These runs used real SDK packages with the repository’s Azure autoscaler code substituted into an installed Ray release.
| Configuration | Reported result |
|---|---|
| SDK 24.0.0 without the deployments package | 39 tests + 10 subtests passed |
| SDK 25.0.0 + deployments package 2.0.0 | 39 tests + 10 subtests passed |
| SDK 26.0.0 + deployments package 2.0.0 | 39 tests + 10 subtests passed |
Reverting each of the seven keyword-argument fixes caused at least one test to fail. No live Azure deployment or full Ray build was run. Two YAML test failures also occurred on unmodified master; four dependency-lock configurations could not be checked because the package index was unreachable. These results have not been independently rerun for this article.
The Ray work required tracing a release to the affected calls, preserving compatibility, testing the patch and responding to maintainers. Finding the repository was only the start of that work.
We want SDK teams to be able to bring a release to Ziek, see which downstream repositories need changes and prepare patches with enough evidence for maintainers to review. The Ray contribution is one example of that workflow. We are still testing how well it generalizes to other SDKs and repositories.
I reviewed the changes and remain responsible for the contribution. The PR discloses AI assistance and Ziek’s role. The code, reviewer feedback and current merge status are available in the public pull request.
Bring a release and we will look at which downstream repositories it affects and what, if anything, they should do.