Feature Request: Use Node.js mirror fallbacks for downloads

[TheJaredWilcurt]

I pitched this idea months ago, buried in a different issue. But this is, to me, the single most important feature missing from Volta.

Problem

When attempting to download/install a new Node version, sometimes the Node.js servers rate-limit the download, causing it to fail. This happens when many Volta users happen to run their GitHub Actions at the same time, or when on a corporate VPN.

Solution

When Volta detects a download/unpack/install failure, before giving up and throwing an error it should:

1. Log a message that it encountered an issue and will automatically retry from a mirror
2. Randomly select a mirror server (so no single mirror gets hammered if Node's servers actually went down)
   • https://mirrors.dotsrc.org/nodejs/release/
   • https://mirror.yandex.ru/mirrors/nodejs.org/dist/
   • I thought there would be more mirrors, but maybe I'm just bad at googling
   • Maybe host a mirror on volta.sh or use the GitHub releases API to host a mirror?
   • Reach out to a university known for hosting open source mirrors and ask them to add Node to the list?
3. If the mirror fails, try a second random mirror
   • Official Node.js server
   • Random Mirror 1
   • Random Mirror 2
4. If it still can't get Node/npm to download/install, then fail with an error message like it does today.
5. If the mirror download completes, do a hash check to make sure the download is safe
   • https://github.com/nodejs/node#verifying-binaries
6. If the hash check passes, continue to install like normal

The minor annoyance of CI failing or getting rate limited locally and having to wait 5 minutes to try again is the only real issue Volta has. If it fell back to a mirror and just took a little longer but still worked without additional human intervention for retries, then Volta would be perfect! ✨ 🏆

[chriskrycho]

I like this idea overall, but at a gut level, I do not think Volta should be in the position of deciding what mirrors are “trustworthy”, and needing to keep an eye on that over time—for the simple reason that even if it is a good idea, our current maintainership is too, uhh, “rate limited” ourselves, as it were, to be able to commit to that. Also, notice that even the hash check could well fail/be rate limited!

I think it should be possible to extend Volta’s hooks system to support this kind of fallback configuration, though. That would then allow individual teams to opt into their own choice of fallback strategy. In very broad strokes, we would probably need something which allows an ordered list of values for the hooks, rather than a single value. This could use a lightweight RFC, but I think most of the details should be pretty straightforward!

[TheJaredWilcurt]

That's the beauty of file hashes, you don't need to "trust" a mirror. You just have to trust the hash file that Node provides, then check the downloaded file against that. For rate-limiting, I think the solution is to cache the hash files yourself on the volta.sh server. Making an entire mirror would be ideal, but if file sizes or bandwidth are a limitation, just hosting the hash files should still be doable. Then you only have to trust Node and yourself.

[chriskrycho]

Sure, and we use hashes extensively for checking things within Volta for that reason. But we definitely don’t have the resources to build and maintain even that small a piece of infrastructure, unfortunately! Right now I have time to work on this an hour here and an hour there in between consulting projects, and I’m the most available of any of the maintainers. Creating server infrastructure we have to maintain even just for hashes, much less for a mirror—the costs on a real mirror would be non-trivial!—are not something any of us has time or energy for. It’s super easy and not a problem, right up until it is a problem, and people have infrastructure relying on it, and… well, seeing as no one is paying for any of this, I would prefer not to put any of us in that situation. 😉