EfsCsiDriverAddOn: When using a namespace it fails with "Error from server (NotFound): error when creating "/tmp/manifest.yaml": namespaces "XXXX" not found\n'

[jesperalmstrom]

Describe the bug

When trying to add a namespace to EfsCsiDriverAddOn({namespace: nameSpace}) it fails with the following error:

from custom resource. Message returned: Error: b'Error from server (NotFound):
error when creating "/tmp/manifest.yaml": namespaces "XXXX" not found\n'

To try to fix this I added a Team with the namespace "XXXX"
const teamUepe = new blueprints.PlatformTeam({ name: nameSpace });
Then add the team before addons:

            .teams(teamUepe)
            .addOns(...addOns)

Still getting the same error.

Expected Behavior

EfsCsiDriverAddOn should be created after the namespace is created or add support for createNamespace: true

Current Behavior

Fails with error message as seen above

Reproduction Steps

        const teamUepe = new blueprints.PlatformTeam({ name: nameSpace });
...
       const addOns: Array<blueprints.ClusterAddOn> = [
            new blueprints.addons.SSMAgentAddOn,
            new blueprints.addons.ClusterAutoScalerAddOn,
            new blueprints.addons.EfsCsiDriverAddOn({ namespace: nameSpace, kmsKeys: [kmsEfsKey] }), 
            new blueprints.addons.AwsLoadBalancerControllerAddOn({
                namespace: nameSpace,
            }),
            new blueprints.addons.ExternalDnsAddOn({
                namespace: nameSpace,
                hostedZoneResources: [blueprints.GlobalResources.HostedZone]
            }),
            new blueprints.addons.CertManagerAddOn({ installCRDs: true }),
            new blueprints.addons.ExternalsSecretsAddOn(),
            new blueprints.addons.SecretsStoreAddOn(),
            new blueprints.addons.IngressNginxAddOn(ingressNginxProps),
        ];
const stack = blueprints.EksBlueprint.builder()
            .version(KubernetesVersion.V1_30)
            .account(this.account)
            .region(this.region)
            .clusterProvider(clusterProvider)
            .resourceProvider(blueprints.GlobalResources.Vpc, new blueprints.VpcProvider(undefined, { primaryCidr: envContext.vpcCidr }))
            .resourceProvider(blueprints.GlobalResources.HostedZone, new blueprints.ImportHostedZoneProvider(hostedZone.hostedZoneId, hostedZoneName))
            .resourceProvider(blueprints.GlobalResources.Certificate, new blueprints.CreateCertificateProvider('secure-ingress-cert', `*.${hostedZone.zoneName}`, blueprints.GlobalResources.HostedZone))
            .resourceProvider(kmsEfsKeyName, new blueprints.CreateKmsKeyProvider(kmsEfsKeyName))
            .resourceProvider('uepe-efs', new blueprints.CreateEfsFileSystemProvider({
                name: envContext.efsName,
                kmsKeyResourceName: kmsEfsKeyName,
                efsProps: {
                    encrypted: true,
                    lifecyclePolicy: efs.LifecyclePolicy.AFTER_7_DAYS,
                    removalPolicy: RemovalPolicy.DESTROY,
                    throughputMode: efs.ThroughputMode.BURSTING,
                    fileSystemPolicy: eksFileSystemPolicy,
                },
            }))
            .teams(teamUepe)
            .addOns(...addOns)
            .build(this, 'eks-blueprints');

Possible Solution

EfsCsiDriverAddOn should be created after the namespace is created in or add support for createNamespace: true

Additional Information/Context

No response

CDK CLI Version

2.156.0 ("aws-cdk-lib": "2.147.3")

EKS Blueprints Version

1.15.1

Node.js Version

v22.8.0

Environment details (OS name and version, etc.)

MacOs Sequoia 15.0

Other information

No response

[jesperalmstrom]

When digging in the Lambda logs I did find this create request that is related to the error:

{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:us-east-1:xxx:function:EksUepeStackuepeeks1FE415-ProviderframeworkonEvent-0BukR7bjwpgv",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:us-east-1:xxx:stack/EksUepeStackuepeeks1FE4154B/185aea00-764f-11ef-b109-0e04891abfad",
    "RequestId": "bd2952a9-043a-4155-8161-48e55b06ffb3",
    "LogicalResourceId": "uepeeksefscsicontrollersamanifestefscsicontrollersaServiceAccountResourceF3B501C0",
    "ResourceType": "Custom::AWSCDK-EKS-KubernetesResource",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:us-east-1:xxx:function:EksUepeStackuepeeks1FE415-ProviderframeworkonEvent-0BukR7bjwpgv",
        "PruneLabel": "aws.cdk.eks/prune-c8f7f1aa13d9beee65d83d39aea548228bf9faadef",
        "ClusterName": "uepe-eks",
        "Manifest": "[{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"name\":\"efs-csi-controller-sa\",\"namespace\":\"XXX\",\"labels\":{\"aws.cdk.eks/prune-c8f7f1aa13d9beee65d83d39aea548228bf9faadef\":\"\",\"app.kubernetes.io/name\":\"efs-csi-controller-sa\"},\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::xxx:role/EksUepeStackuepeeks1FE415-uepeeksefscsicontrollersa-1Lp0t66HvgLq\"}}}]",
        "RoleArn": "arn:aws:iam::xxx:role/EksUepeStackuepeeks1FE415-uepeeksCreationRole088278-GOyKG17AEzI0"
    }
}