feat(infra): AML infrastructure improvements (flexible compute clusters, outbound rules, entra id membership)

[fbeltrao]

Component

Infrastructure (Terraform/Azure)

Problem Statement

We would like to propose the following improvements (happy to contribute to them)

• If possible replace azapi_resource with azure_rm_machine_learning_workspace for Azure ML workspace provisioning
• Make outbound rules independent of the private endpoint, so we can still choose a private endpoint but enable outbound internet access for services like Nvidia, pipy, and Hugging Face. Alternative users can selected "only approved" but provide a list of pre-approved FDQNs.
• Allow flexible AML compute cluster creation by passing a map of clusters, with cluster names reflecting the SKU and spot/dedicated type.
• Support entra id membership to AML and storage accounts. I have observed people being onboarded into AML through group memberships.

Proposed Solution

See above

Alternatives Considered

No response

Additional Context

No response

[katriendg]

Thanks a lot for your proposal @fbeltrao. Tagging @akzaidi @agreaves-ms @WilliamBerryiii for visbility.