TLDR;

if you are struggling with spawnSync bash ENOENT. at AssetStaging.bundle Error when bundling a Lambda function check out the bundling props of your CDK NodejsFunction and try forceDockerBundling: true option.

Read more for full context and explanation.

This week I faced a weird issue that caused the CI/CD pipeline on Gitlab to stop working and I want to share the solution here for future reference.

Context

The backend had a Go Backend running on AWS Fargate behind a Load Balancer.
Deployment happens on Gitlab and stack is written with CDK, therefore the deploy job was running with a docker:20-dind (Docker in Docker) image to bundle the binary, build the docker image, upload it to ECR and then use it within a Fargate Task.

Normally in previous project I always used sleavely/node-awscli to build my lambdas and run the CDK deployment but with that go backend and ECS + ECR I had to use DinD.

Then I added to the stack a Target group pointing to a Lambda written in Node ( following a strangler pattern approach to refactoring, that I described here).

I deployed from my Mac directly to an ephemeral environment and everything went fine.
Once I pushed my changes and Gitlab pipeline tried to deploy from the Gitlab runner though, I got a weird error:

Error(`Failed to bundle asset ${this.node.path}, bundle output is located at ${bundleErrorDir}: ${err}`); -error: Error: spawnSync bash ENOENT. at AssetStaging.bundle

Investigation

Everything looked fine though, files were there, artifacts were passed along one pipeline job to the other just fine, every dependency was installed properly.

A quick search lead to StackOverflow but the answers did not help. I was already using esbuild and installing it in the dependencies.

Some other searches lead me to this and then finally to this repo where I noticed a parameter of the lambda.NodejsFunction Construct that I overlooked:

forceDockerBundling? Type: boolean (optional, default: false)
Force bundling in a Docker container even if local bundling is possible.
This is useful if your function relies on node modules that should be installed (nodeModules) in a Lambda compatible environment.

Honestly I never thought about forcing Docker bundling, why should I do that if Esbuild is way more performant?

Anyway, by the end of the morning I had really already tried multiple things with no success, so why not? In the end CDK in the pipeline was using Docker In Docker and Esbuild was not working, so trying to force Docker bundling kinda made sense.

And you know what? It worked of course.

Solution

Since I did not want to give up my esbuild when bundling locally on my Mac, I looked for env variables to figure out if I am running on Gitlab and this is the final working code:

 const myLambda = new NodejsFunction(this, `my-lambda`, {
     functionName: `my-lambda`,
     entry: 'lambdas/my-lambda.ts'),
     timeout: Duration.seconds(15),
     handler: 'handler',
     runtime: Runtime.NODEJS_16_X,
     bundling: {
        forceDockerBundling: Boolean(process.env.CI), 
        // use esbuild when running locally otherwise rely on docker
        define: {},
        minify: true,
        externalModules: ['aws-sdk'],
     }
  })

`

Since this caused quite some frustration and headaches during deployment time, I hope it turns out to be useful for others.