[Fix] Add Sufficient Network Privileges to the Databricks Default Cross Account Policy

[caldempsey]

Changes

Currently, the Databricks-provided Cross Account Policy IAM Role does not include all the necessary permissions to set up a workspace. Attempting to set up a workspace using this policy results in the following error (see Issue #4026):

MALFORMED_REQUEST: Failed credentials validation checks: Allocate Address

This makes it difficult for new engineers to onboard to Databricks without troubleshooting unexpected errors. This PR adds the missing network permissions to the Databricks Managed VPC policy types ("managed" and "customer"), ensuring that all required permissions are included for successful workspace deployment. These changes are not applied to the "restricted" policy type to avoid allowing Elastic IP allocations, which may not be desirable for some Databricks customers. See the bottom of the description for the full list.

Tests

This change has been tested locally and is running in our staging workspace using the same configuration. As this is a fix for 'managed' type Databricks deployment configurations, I have matched this with positive and negative unit tests to guard precise and expected roles. I have then added extra tests to confirm the expected policies across each branch, 'managed', 'customer', and 'restricted'. Feel free to remove these if overboard, as I recognise you could make a similar weaker assertion using 'len'.

• make test run locally
• Relevant acceptance tests are passing
• Relevant change in docs/ folder (if necessary)
• Covered with integration tests in internal/acceptance
• Using Go SDK (N/A)

The full list of permissions which align with the Databricks documentation, now included in the "managed" policy type, are:

[
  "ec2:AllocateAddress",
  "ec2:AssignPrivateIpAddresses",
  "ec2:AssociateDhcpOptions",
  "ec2:AssociateIamInstanceProfile",
  "ec2:AssociateRouteTable",
  "ec2:AttachInternetGateway",
  "ec2:AttachVolume",
  "ec2:AuthorizeSecurityGroupEgress",
  "ec2:AuthorizeSecurityGroupIngress",
  "ec2:CancelSpotInstanceRequests",
  "ec2:CreateDhcpOptions",
  "ec2:CreateFleet",
  "ec2:CreateInternetGateway",
  "ec2:CreateLaunchTemplate",
  "ec2:CreateLaunchTemplateVersion",
  "ec2:CreateNatGateway",
  "ec2:CreateRoute",
  "ec2:CreateRouteTable",
  "ec2:CreateSecurityGroup",
  "ec2:CreateSubnet",
  "ec2:CreateTags",
  "ec2:CreateVolume",
  "ec2:CreateVpc",
  "ec2:CreateVpcEndpoint",
  "ec2:DeleteDhcpOptions",
  "ec2:DeleteFleets",
  "ec2:DeleteInternetGateway",
  "ec2:DeleteLaunchTemplate",
  "ec2:DeleteLaunchTemplateVersions",
  "ec2:DeleteNatGateway",
  "ec2:DeleteRoute",
  "ec2:DeleteRouteTable",
  "ec2:DeleteSecurityGroup",
  "ec2:DeleteSubnet",
  "ec2:DeleteTags",
  "ec2:DeleteVolume",
  "ec2:DeleteVpc",
  "ec2:DeleteVpcEndpoints",
  "ec2:DescribeAvailabilityZones",
  "ec2:DescribeFleetHistory",
  "ec2:DescribeFleetInstances",
  "ec2:DescribeFleets",
  "ec2:DescribeIamInstanceProfileAssociations",
  "ec2:DescribeInstanceStatus",
  "ec2:DescribeInstances",
  "ec2:DescribeInternetGateways",
  "ec2:DescribeLaunchTemplates",
  "ec2:DescribeLaunchTemplateVersions",
  "ec2:DescribeNatGateways",
  "ec2:DescribeNetworkAcls",
  "ec2:DescribePrefixLists",
  "ec2:DescribeReservedInstancesOfferings",
  "ec2:DescribeRouteTables",
  "ec2:DescribeSecurityGroups",
  "ec2:DescribeSpotInstanceRequests",
  "ec2:DescribeSpotPriceHistory",
  "ec2:DescribeSubnets",
  "ec2:DescribeVolumes",
  "ec2:DescribeVpcAttribute",
  "ec2:DescribeVpcs",
  "ec2:DetachInternetGateway",
  "ec2:DisassociateIamInstanceProfile",
  "ec2:DisassociateRouteTable",
  "ec2:GetLaunchTemplateData",
  "ec2:GetSpotPlacementScores",
  "ec2:ModifyFleet",
  "ec2:ModifyLaunchTemplate",
  "ec2:ModifyVpcAttribute",
  "ec2:ReleaseAddress",
  "ec2:ReplaceIamInstanceProfileAssociation",
  "ec2:RequestSpotInstances",
  "ec2:RevokeSecurityGroupEgress",
  "ec2:RevokeSecurityGroupIngress",
  "ec2:RunInstances",
  "ec2:TerminateInstances"
]

Resolves #4026

[alexott]

looks good...

[caldempsey]

Thanks!