Help me understand how pnpm works and why deps sometimes cannot be found

[mobob]

I've had a few number of issues with deps lately in my pnpm monorepo, so i'm trying to dive a bit deeper to understand whats going on, and taking the issue below to do so.

For background; i'm on the latest pnpm (9.12.1 and have tried older), using a monorepo. My workspace file is as follows (and the project i'm working on rn is backend:

packages:
  - 'pipeline'
  - 'frontend'
  - 'backend'
  - 'packages/*'

Today, i got this error while running a test:

 FAIL  <snip>  ● Test suite failed to run

    Cannot find module '@aws-sdk/core' from '../node_modules/.pnpm/@[email protected]_@[email protected]/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/index.js'

Ok - so where is @aws-sdk/core? My cwd is /backend, so i check backend/node_modules:

(base) ➜  backend git:(oct7-pdfs) ✗ ls node_modules/@aws-sdk     
client-dynamodb        client-s3              client-ses             client-sts             util-dynamodb
client-eventbridge     client-secrets-manager client-sso-oidc        lib-dynamodb

Hmm - this seems odd. From how i understand packages that are dependencies should always show up at the root, and be (soft) symbolically linked in.

Probably doesn't matter much - the above search within lib-dynamodb isn't in the backend dir anyways, its in the root node_modules. So lets take a look in there:

(base) ➜  backend git:(oct7-pdfs) ✗ ls ../node_modules/.pnpm/@[email protected]_@[email protected]/node_modules/@aws-sdk
client-dynamodb lib-dynamodb    util-dynamodb

Ok, sure enough, its not there either. Which perhaps isn't a huge surprise, because @aws-sdk/core is not listed as a dependency of lib-dynamodb - it is of client-dynamodb.

https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-dynamodb/package.json
https://github.com/aws/aws-sdk-js-v3/blob/534e028f7f8cb6f20fe9be0b61aba19e05bbd461/clients/client-dynamodb/package.json#L25C4-L25C26

So how then do all these nested upon nested deps get resolved? This is where my understanding breaks down.

For the record there are seemingly a zillion references to different versions of @aws-sdk/core in my lock file, so i suspect this and me recently adding a gargantuan lib (cough cough llamaindex) isn't helping me here.

And the kicker; the only workaround i currently have for this is to install @aws-sdk/core in the workspace root 😭. It feels like something is broken in my setup as all this should just work, so any tips to fill in my understanding or where to look next would be appreciated!

[xty]

Ok - so where is @aws-sdk/core? My cwd is /backend, so i check backend/node_modules:

(base) ➜  backend git:(oct7-pdfs) ✗ ls node_modules/@aws-sdk     
client-dynamodb        client-s3              client-ses             client-sts             util-dynamodb
client-eventbridge     client-secrets-manager client-sso-oidc        lib-dynamodb

Hmm - this seems odd. From how i understand packages that are dependencies should always show up at the root, and be (soft) symbolically linked in.

Why would you think otherwise? I mean, it's not immediately clear from your terminal output that node_modules/@aws-sdk is not a symlink. In fact, it probably is. To find out for yourself, while inside backend, run ls -ld node_modules/@aws-sdk instead.

Ok, sure enough, its not there either. Which perhaps isn't a huge surprise, because @aws-sdk/core is not listed as a dependency of lib-dynamodb - it is of client-dynamodb.

You have correctly identified the root cause of the issue here – @aws-sdk/lib-dynamodb relies on @aws-sdk/core to work but doesn't explicitly declare it as a dependency. This is known as a phantom dependency, an anit-pattern that is frowned upon and discouraged by pnpm.

https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-dynamodb/package.json

It appears that @aws-sdk/core has been added since then, specifically, in aws/aws-sdk-js-v3@e4f9d59. Interestingly, it was added about the same time you posted this question.

To sum up, the issue you were having has little to do with pnpm per se; rather, it is caused by @aws-sdk/lib-dynamodb not explicitly declaring @aws-sdk/core as a dependency. The folks at AWS probably knew it was bad which is why they've patched it in v3.670.0. Update your package and you should be good to go.

[mobob]

Thank you @xty !!! This explains a ton, appreciate the clarifications around the points too.

One more clarification if i may; the reason why it worked when i installed it manually at the root vs in the backend package dir is due to how pnpm resolves things. It would have needed to have been an explicit dependency of the thing that was usuing it for it to be found, ie, because it'll really be looking in a path like this:

../../../node_modules/.pnpm/@AWS-SDK[email protected]/node_modules/@aws-sdk/client-dynamodb

And that won't exist in the nested package dir... only if its installed at the root will it be found at the more basic

/node_modules/@AWS-SDK ...

Would love to know if i'm loosely correct here and if there is other gaps i'm missing.

[xty]

Thank you @xty !!! This explains a ton, appreciate the clarifications around the points too.

I'm glad it has helped.

One more clarification if i may; the reason why it worked when i installed it manually at the root vs in the backend package dir is due to how pnpm resolves things. It would have needed to have been an explicit dependency of the thing that was usuing it for it to be found, ie, because it'll really be looking in a path like this:

../../../node_modules/.pnpm/@AWS-SDK[email protected]/node_modules/@aws-sdk/client-dynamodb

Technically, all pnpm does is creating links of various kind; it has no part in the module resolution process, which is done by Node.js, as per this algorithm.

And that won't exist in the nested package dir... only if its installed at the root will it be found at the more basic

/node_modules/@AWS-SDK ...

Pay special attention to LOAD_NODE_MODULES(X, START) and NODE_MODULES_PATHS(START) whereby it keeps looking for module_modules upwards along the path of the requiring module ($workspace_root/node_modules/.pnpm/@[email protected]_@[email protected]/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/index.js in your case). I suspect this would eventually lead to $workspace_root/node_modules/@aws-sdk/core when @aws-sdk/core is installed at the workspace root.

Would love to know if i'm loosely correct here and if there is other gaps i'm missing.

As you can see, the whole module resolution algorithm might come across as bit intimidating, but don't despair. You're definitely on the right track.