From b191ba3fc7a579336bcfb9b7d56e01cfba7c8747 Mon Sep 17 00:00:00 2001 From: conico974 Date: Fri, 3 May 2024 16:28:22 +0200 Subject: [PATCH] OpenNext V3 (#402) * created basic config file * basic wrapper and converter implementation * Minimal response writable * build config * change response to transform to allow to use pipeline * fix streaming for v3 * compression support * better docker handler * add converter for apigw-v1 & cloudfront * overridable queue * overridable s3 cache * overridable tag cache * prebuild middleware * refactor routing and middleware * big refactoring moved files around so that it makes more sense deleted a bunch of useless files added todo to remind myself of what i still need to do * refactor: cleanup plugins added a deletes options in open-next plugin * make other lambdas overridable as well * externalMiddleware * improve plugins * fix proxy request and make it work with streaming * bugfix * fix host * refactor wrapper * generate basic dockerfile * Only build open-next config once * generate basic output file for IAC to use * basic splitting * bundled next server * fix external middleware cloudfront * fix image adapter rebase * couple of fix for node * package version * support for warmer with splitted fn * basic support for edge runtime There is some restriction: Only 1 route per function Support only app route and page No streaming * external middleware support rewrite between splitted servers * fix alias * update package.json * use AsyncLocalStorage to scope lastModified to a single request * merge upstream/main * Add basic validation * fix EISDIR issue with copying traced symlink * added override name to the output for better IAC support * rename BuildOptions remove some unused options properly handle minify * normalize locale path before passing to middleware * Copy necessary static files * fix issues with fallback and i18n in page router * Add a big warning for build on windows * fix for cloudflare workers * add wasm fils and assets * fix 14.1 cache * fix wasm import node * update version * merge upstream * make open-next.config.ts optional * Fix cannot write default config file b/c folder not created (#364) * Fix cannot write default config file b/c folder not created * Removed copyTracedFiles debug log * fix for monorepo * fix for output for dynamodb provider * fix dynamoProvider, skipTrailingSlash, weird ISR deduplication issue * little improvement to streaming in lambda * fix another monorepo error * e2e fixes for v3 rc * update version * Not use custom-resource converter for dynamodb seeding adapter (#365) * Not use custom-resource converter for dynamodb seeding adapter * fix e2e --------- Co-authored-by: Dorseuil Nicolas * fix fallback false for route without i18n * version package update * Squashed commit of the following: commit ff37de235e653e3aecdbb31168ab411add2d0330 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed Mar 6 15:37:07 2024 +0100 Version Packages (#378) Co-authored-by: github-actions[bot] commit 32353923a4a0e99217d04d5790feac539a4c0e15 Author: Iakhub Seitasanov Date: Wed Mar 6 17:29:46 2024 +0300 fix: prevent duplication of location header (#369) * fix: prevent duplication of location header * changeset * fix linting --------- Co-authored-by: conico974 commit af2d3ce484cf5bcd57d11d07174097457a9cd119 Author: Chung Wei Leong <15154097+chungweileong94@users.noreply.github.com> Date: Wed Mar 6 22:06:33 2024 +0800 Fix image optimization support for Next 14.1.1 (#377) * Move image optimization to plugin * Refactor image optimization code * Added image optimization plugin for 14.1.1 * Fix image optimization plugin * Add changeset * Revert default sharp version to 0.32.6 * e2e test for image optimization * change one of the test to use an external image --------- Co-authored-by: Dorseuil Nicolas commit 3deb2022d0bb506e4c4b66baeae248c7c2b153e5 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Feb 13 08:39:35 2024 -0800 Version Packages (#363) Co-authored-by: github-actions[bot] commit f9b90b6219b7eef71be69ee94185c9f5be8537db Author: khuezy Date: Tue Feb 13 08:35:10 2024 -0800 changeset/2.3.6 (#362) commit 40c2b36696001ba2482e952189d327d24d1800e8 Author: Patrick Ufer <46608534+patrickufer@users.noreply.github.com> Date: Tue Feb 13 09:23:40 2024 -0700 security fix: upgrade sharp version to 0.32.6 (#361) * upgrade sharp version commit 63fab055c6e0e1fe4591555cc0f91dcf3d4d739a Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Feb 2 00:14:11 2024 +0100 Version Packages (#359) Co-authored-by: github-actions[bot] commit c80f1be1d987d48daec28a2e65934ecfb5aa107c Author: conico974 Date: Fri Feb 2 00:00:56 2024 +0100 Fix trailing slash redirect to external domain (#358) * fix trailing slash redirect to external domain * changeset commit 186e28f83409e6f96b5e05d3439c36a545eb7e08 Author: Jaden VanEckhout Date: Thu Feb 1 16:49:14 2024 -0600 fix(open-next): correctly set cache control for html pages (#353) * fix(open-next): correctly set cache control for html pages * changeset --------- Co-authored-by: conico974 commit b9eefca37f1846ac05da5a9aa25a2d421cb77542 Author: Manuel Antunes <57446204+Manuel-Antunes@users.noreply.github.com> Date: Thu Feb 1 19:41:47 2024 -0300 Fix Cache Support for Next@14.1.0 (#356) * feat: add cache support for next@14.1.0 * fix: lint files * chore: apply the proposed changes * Fix typo * changeset --------- Co-authored-by: conico974 commit afd9605048cfdf1c0cd3f224da34a27c3f27128f Author: conico974 Date: Sat Jan 27 15:19:11 2024 +0100 update docs for V3 (#351) commit 46241fe9c43a3de2e5507acca7758ddf6d6aa6d1 Author: Abhishek Malik Date: Sat Jan 27 19:45:18 2024 +0530 Update bundle_size.mdx for excluding pdfjs-dist optional dependency docs (#346) * Update bundle_size.mdx for excluding pdfjs-dist optional dependency docs The current fix didn't work, but this updated fix did work for me. Hence proposing this as another solution. * Update docs/pages/common_issues/bundle_size.mdx Co-authored-by: khuezy --------- Co-authored-by: khuezy commit 9a6473a9cfefda0da9d7a810cb983eeb8752ea82 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Jan 5 16:56:42 2024 +0100 Version Packages (#345) Co-authored-by: github-actions[bot] commit bbf9b305679a332f05947feebdc3854bbd13ea4f Author: Lucas Vieira Date: Fri Jan 5 12:45:13 2024 -0300 fix(open-next): use dynamic import handler for monorepo entrypoint (#341) * fix(open-next): use dynamic import handler for monorepo entrypoint * changeset --------- Co-authored-by: Dorseuil Nicolas commit 83b08389ab80d72972ed32ec8e36c9ee5a3399db Author: santiperone Date: Fri Jan 5 12:38:12 2024 -0300 add suport for bun lockfile in monorepo (#337) * add suport for bun lockfile in monorepo * changeset --------- Co-authored-by: Dorseuil Nicolas commit e773e67ed9394e2f3e6962f1ed29be0078e084c4 Author: Jan Stevens Date: Fri Jan 5 16:31:27 2024 +0100 fix: try to match errors, fall back to just adding raw key / value pare (#336) * fix: try to match errors, fall back to just adding raw key / value pair instead * changeset * fix lint --------- Co-authored-by: Dorseuil Nicolas commit fd90b260f5fe2c02bb1ca1f818b9c5c5d9937a1b Author: Dylan Irion <61515823+dylanirion@users.noreply.github.com> Date: Fri Jan 5 17:22:28 2024 +0200 Changes encoding on cache.body from utf8 to base64 (#329) * changes encoding on cache.body from utf8 to base64 * retain utf8 for json content-type * opting for less greedy base64 * use isBinaryContentType * changeset --------- Co-authored-by: Dorseuil Nicolas commit eb089800225bcedcd47b0b10a6938692f9e6cb0a Author: sommeeeR <91796856+sommeeeer@users.noreply.github.com> Date: Fri Jan 5 16:02:47 2024 +0100 fix: make invalidateCFPaths function async in docs (#344) commit 83207d87d30b622bce299895bb2cf605ee0f39af Author: conico974 Date: Thu Dec 14 16:59:15 2023 +0100 updated docs for v3 (#334) commit 0e827ce4aefab5e337822cda0d43be218e919cb8 Author: conico974 Date: Fri Dec 8 17:57:51 2023 +0100 ci: update node e2e commit 36da8198c30b9b0dc7cfd7628dbc3c6dffc59961 Author: conico974 Date: Thu Dec 7 17:44:06 2023 +0100 Initial docs for V3 (#330) * docs for V3 * fix link * clearer routes in config * fix for next 12 * add support for basePath * allow customization of sharp runtime * updated edge converter to match behaviour of lambda * update version * fix monorepo * improved streaming https://github.com/aws/aws-lambda-nodejs-runtime-interface-client/issues/94#issuecomment-1992245429 * update version * fix open-next config build that depends on node * fix crypto middleware node 20 * Sync * fix resolve in image optimization also fix image opt not using streaming * add better error when edge runtime is used inside node * update version * fix null error on lambda hopefully * update version * fix 500 on aws-lambda wrapper * update version * fix duplex for request in node * fix & refactor middleware response headers * update version * Sync * update version * removed specific lamda streaming hack It's been fixed upstream * add geo in middleware * added helpers function for config file Better typing as well * fix for 14.2 * update version * fix redirect lambda streaming * fix e2e tests * test: improve reliability of test for revalidateTag * update version * review fix * fix cookies in streaming also fix an issue when both middleware and page try to set cookies OpenNextNodeResponse also implements ServerResponse * make all write to ddb chunked * changeset * fix e2e --------- Co-authored-by: Frank --- .changeset/rotten-trees-kiss.md | 38 + docs/pages/inner_workings/_meta.json | 2 +- docs/pages/v3/_meta.json | 7 + docs/pages/v3/config.mdx | 69 ++ docs/pages/v3/index.mdx | 67 ++ docs/pages/v3/override.mdx | 74 ++ docs/pages/v3/reference-implementation.mdx | 435 +++++++++ docs/pages/v3/requirements.mdx | 59 ++ examples/app-pages-router/open-next.config.ts | 12 + examples/app-router/app/ssr/layout.tsx | 2 + examples/app-router/open-next.config.ts | 11 + examples/app-router/package.json | 2 +- examples/pages-router/open-next.config.ts | 7 + examples/shared/components/Filler/index.tsx | 17 + examples/sst/stacks/AppPagesRouter.ts | 15 +- examples/sst/stacks/AppRouter.ts | 24 +- .../stacks/OpenNextReferenceImplementation.ts | 462 +++++++++ examples/sst/stacks/PagesRouter.ts | 15 +- packages/open-next/package.json | 6 +- packages/open-next/src/adapters/cache.ts | 392 ++------ .../open-next/src/adapters/config/index.ts | 24 +- .../open-next/src/adapters/config/util.ts | 13 +- .../open-next/src/adapters/dynamo-provider.ts | 98 +- .../open-next/src/adapters/edge-adapter.ts | 73 ++ .../open-next/src/adapters/event-mapper.ts | 394 -------- packages/open-next/src/adapters/http/index.ts | 3 - .../open-next/src/adapters/http/response.ts | 150 --- .../src/adapters/http/responseStreaming.ts | 268 ------ packages/open-next/src/adapters/http/util.ts | 50 - .../adapters/image-optimization-adapter.ts | 162 +++- packages/open-next/src/adapters/middleware.ts | 82 ++ .../adapters/plugins/13.5/serverHandler.ts | 25 - .../src/adapters/plugins/13.5/util.ts | 5 - .../image-optimization.replacement.ts | 2 +- .../image-optimization.ts | 2 +- .../src/adapters/plugins/lambdaHandler.ts | 139 --- .../plugins/routing/default.replacement.ts | 162 ---- .../src/adapters/plugins/routing/default.ts | 81 -- .../src/adapters/plugins/routing/util.ts | 238 ----- .../plugins/serverHandler.replacement.ts | 29 - .../src/adapters/plugins/serverHandler.ts | 17 - .../adapters/plugins/streaming.replacement.ts | 104 -- .../src/adapters/plugins/util.replacement.ts | 25 - .../plugins/without-routing/requestHandler.ts | 16 + packages/open-next/src/adapters/revalidate.ts | 25 +- .../open-next/src/adapters/routing/util.ts | 101 -- .../open-next/src/adapters/server-adapter.ts | 57 +- .../open-next/src/adapters/types/plugin.ts | 36 - packages/open-next/src/adapters/util.ts | 47 +- .../open-next/src/adapters/warmer-function.ts | 153 +-- packages/open-next/src/build.ts | 895 ++++++------------ .../open-next/src/build/bundleNextServer.ts | 104 ++ .../open-next/src/build/copyTracedFiles.ts | 269 ++++++ .../open-next/src/build/createServerBundle.ts | 306 ++++++ .../src/build/edge/createEdgeBundle.ts | 165 ++++ .../open-next/src/build/generateOutput.ts | 373 ++++++++ packages/open-next/src/build/helper.ts | 246 +++++ .../open-next/src/build/validateConfig.ts | 56 ++ .../open-next/src/cache/incremental/s3.ts | 78 ++ .../open-next/src/cache/incremental/types.ts | 50 + packages/open-next/src/cache/next-types.ts | 76 ++ .../src/{adapters => cache/tag}/constants.ts | 0 packages/open-next/src/cache/tag/dynamoDb.ts | 156 +++ packages/open-next/src/cache/tag/types.ts | 9 + .../open-next/src/converters/aws-apigw-v1.ts | 109 +++ .../open-next/src/converters/aws-apigw-v2.ts | 92 ++ .../src/converters/aws-cloudfront.ts | 220 +++++ packages/open-next/src/converters/dummy.ts | 24 + packages/open-next/src/converters/edge.ts | 86 ++ packages/open-next/src/converters/node.ts | 55 ++ .../src/converters/sqs-revalidate.ts | 25 + packages/open-next/src/converters/utils.ts | 11 + .../src/core/createGenericHandler.ts | 56 ++ .../open-next/src/core/createMainHandler.ts | 78 ++ .../open-next/src/core/edgeFunctionHandler.ts | 82 ++ packages/open-next/src/core/requestHandler.ts | 187 ++++ .../src/{adapters => core}/require-hooks.ts | 5 +- packages/open-next/src/core/resolve.ts | 58 ++ .../src/{adapters => core}/routing/matcher.ts | 127 ++- .../{adapters => core}/routing/middleware.ts | 167 ++-- packages/open-next/src/core/routing/util.ts | 480 ++++++++++ packages/open-next/src/core/routingHandler.ts | 109 +++ .../src/{adapters/plugins => core}/util.ts | 33 +- .../open-next/src/helpers/withCloudflare.ts | 118 +++ packages/open-next/src/helpers/withSST.ts | 65 ++ packages/open-next/src/http/index.ts | 4 + .../open-next/src/http/openNextResponse.ts | 285 ++++++ .../src/{adapters => }/http/request.ts | 2 +- packages/open-next/src/http/util.ts | 42 + packages/open-next/src/index.ts | 21 +- packages/open-next/src/logger.ts | 8 +- packages/open-next/src/plugins/edge.ts | 185 ++++ .../src/{plugin.ts => plugins/replacement.ts} | 40 +- packages/open-next/src/plugins/resolve.ts | 64 ++ packages/open-next/src/queue/sqs.ts | 28 + packages/open-next/src/queue/types.ts | 13 + .../src/{adapters => }/types/aws-lambda.ts | 0 .../src/{adapters => }/types/next-types.ts | 45 +- packages/open-next/src/types/open-next.ts | 339 +++++++ .../src/wrappers/aws-lambda-streaming.ts | 111 +++ packages/open-next/src/wrappers/aws-lambda.ts | 72 ++ packages/open-next/src/wrappers/cloudflare.ts | 30 + packages/open-next/src/wrappers/node.ts | 67 ++ packages/open-next/tsconfig.json | 7 +- .../tests/appRouter/revalidateTag.test.ts | 10 + .../tests-unit/tests/event-mapper.test.ts | 1 + pnpm-lock.yaml | 173 +++- 107 files changed, 7587 insertions(+), 3227 deletions(-) create mode 100644 .changeset/rotten-trees-kiss.md create mode 100644 docs/pages/v3/_meta.json create mode 100644 docs/pages/v3/config.mdx create mode 100644 docs/pages/v3/index.mdx create mode 100644 docs/pages/v3/override.mdx create mode 100644 docs/pages/v3/reference-implementation.mdx create mode 100644 docs/pages/v3/requirements.mdx create mode 100644 examples/app-pages-router/open-next.config.ts create mode 100644 examples/app-router/open-next.config.ts create mode 100644 examples/pages-router/open-next.config.ts create mode 100644 examples/shared/components/Filler/index.tsx create mode 100644 examples/sst/stacks/OpenNextReferenceImplementation.ts create mode 100644 packages/open-next/src/adapters/edge-adapter.ts delete mode 100644 packages/open-next/src/adapters/event-mapper.ts delete mode 100644 packages/open-next/src/adapters/http/index.ts delete mode 100644 packages/open-next/src/adapters/http/response.ts delete mode 100644 packages/open-next/src/adapters/http/responseStreaming.ts delete mode 100644 packages/open-next/src/adapters/http/util.ts create mode 100644 packages/open-next/src/adapters/middleware.ts delete mode 100644 packages/open-next/src/adapters/plugins/13.5/serverHandler.ts delete mode 100644 packages/open-next/src/adapters/plugins/13.5/util.ts rename packages/open-next/src/adapters/plugins/{ => image-optimization}/image-optimization.replacement.ts (96%) rename packages/open-next/src/adapters/plugins/{ => image-optimization}/image-optimization.ts (95%) delete mode 100644 packages/open-next/src/adapters/plugins/lambdaHandler.ts delete mode 100644 packages/open-next/src/adapters/plugins/routing/default.replacement.ts delete mode 100644 packages/open-next/src/adapters/plugins/routing/default.ts delete mode 100644 packages/open-next/src/adapters/plugins/routing/util.ts delete mode 100644 packages/open-next/src/adapters/plugins/serverHandler.replacement.ts delete mode 100644 packages/open-next/src/adapters/plugins/serverHandler.ts delete mode 100644 packages/open-next/src/adapters/plugins/streaming.replacement.ts delete mode 100644 packages/open-next/src/adapters/plugins/util.replacement.ts create mode 100644 packages/open-next/src/adapters/plugins/without-routing/requestHandler.ts delete mode 100644 packages/open-next/src/adapters/routing/util.ts delete mode 100644 packages/open-next/src/adapters/types/plugin.ts create mode 100644 packages/open-next/src/build/bundleNextServer.ts create mode 100644 packages/open-next/src/build/copyTracedFiles.ts create mode 100644 packages/open-next/src/build/createServerBundle.ts create mode 100644 packages/open-next/src/build/edge/createEdgeBundle.ts create mode 100644 packages/open-next/src/build/generateOutput.ts create mode 100644 packages/open-next/src/build/helper.ts create mode 100644 packages/open-next/src/build/validateConfig.ts create mode 100644 packages/open-next/src/cache/incremental/s3.ts create mode 100644 packages/open-next/src/cache/incremental/types.ts create mode 100644 packages/open-next/src/cache/next-types.ts rename packages/open-next/src/{adapters => cache/tag}/constants.ts (100%) create mode 100644 packages/open-next/src/cache/tag/dynamoDb.ts create mode 100644 packages/open-next/src/cache/tag/types.ts create mode 100644 packages/open-next/src/converters/aws-apigw-v1.ts create mode 100644 packages/open-next/src/converters/aws-apigw-v2.ts create mode 100644 packages/open-next/src/converters/aws-cloudfront.ts create mode 100644 packages/open-next/src/converters/dummy.ts create mode 100644 packages/open-next/src/converters/edge.ts create mode 100644 packages/open-next/src/converters/node.ts create mode 100644 packages/open-next/src/converters/sqs-revalidate.ts create mode 100644 packages/open-next/src/converters/utils.ts create mode 100644 packages/open-next/src/core/createGenericHandler.ts create mode 100644 packages/open-next/src/core/createMainHandler.ts create mode 100644 packages/open-next/src/core/edgeFunctionHandler.ts create mode 100644 packages/open-next/src/core/requestHandler.ts rename packages/open-next/src/{adapters => core}/require-hooks.ts (98%) create mode 100644 packages/open-next/src/core/resolve.ts rename packages/open-next/src/{adapters => core}/routing/matcher.ts (75%) rename packages/open-next/src/{adapters => core}/routing/middleware.ts (51%) create mode 100644 packages/open-next/src/core/routing/util.ts create mode 100644 packages/open-next/src/core/routingHandler.ts rename packages/open-next/src/{adapters/plugins => core}/util.ts (78%) create mode 100644 packages/open-next/src/helpers/withCloudflare.ts create mode 100644 packages/open-next/src/helpers/withSST.ts create mode 100644 packages/open-next/src/http/index.ts create mode 100644 packages/open-next/src/http/openNextResponse.ts rename packages/open-next/src/{adapters => }/http/request.ts (98%) create mode 100644 packages/open-next/src/http/util.ts create mode 100644 packages/open-next/src/plugins/edge.ts rename packages/open-next/src/{plugin.ts => plugins/replacement.ts} (62%) create mode 100644 packages/open-next/src/plugins/resolve.ts create mode 100644 packages/open-next/src/queue/sqs.ts create mode 100644 packages/open-next/src/queue/types.ts rename packages/open-next/src/{adapters => }/types/aws-lambda.ts (100%) rename packages/open-next/src/{adapters => }/types/next-types.ts (82%) create mode 100644 packages/open-next/src/types/open-next.ts create mode 100644 packages/open-next/src/wrappers/aws-lambda-streaming.ts create mode 100644 packages/open-next/src/wrappers/aws-lambda.ts create mode 100644 packages/open-next/src/wrappers/cloudflare.ts create mode 100644 packages/open-next/src/wrappers/node.ts diff --git a/.changeset/rotten-trees-kiss.md b/.changeset/rotten-trees-kiss.md new file mode 100644 index 00000000..ceb1770f --- /dev/null +++ b/.changeset/rotten-trees-kiss.md @@ -0,0 +1,38 @@ +--- +"open-next": major +--- + +OpenNext V3 + +This is the V3 of OpenNext. It includes some breaking changes and cannot be used as a drop-in replacement for V2. If your IAC is using OpenNext V2, you will need to update it to use V3. + +If you are using OpenNext V2, please refer to the [migration guide](https://open-next.js.org/migration#from-opennext-v2) to upgrade to V3. + +### New Features + +- Add support for function splitting +- Add support for external middleware +- Custom config file support : `open-next.config.ts` +- Support for other deployment targets than lambda (Node.js, Docker and partial support for Cloudflare Workers) +- Allow for customizing the outputs bundle : + - Wrapper + - Converter + - Incremental Cache (Fetch cache and HTML/JSON/RSC cache) + - Tag Cache + - Queue (Used to trigger ISR revalidation) + - Origin Resolver (Only for external middleware) + - Image Loader (Only for image optimization) + - Invoke function (For the warmer function) +- Create an `open-next.output.json` file for easier integration with IAC tools + +### Breaking Changes + +- Edge runtime don't work out of the box anymore. You need to deploy them on a separate function see [the config for more info](https://open-next.js.org/config) +- Output directory structure has changed to support function splitting +- Removed build arguments in favor of `open-next.config.ts` + +### Internal Changes + +- Use OpenNextNodeResponse instead of ServerResponse (It uses transform stream to properly handle the stream) +- Big refactor of the codebase to support function splitting +- Added new plugins to support the new features and make the codebase more modular diff --git a/docs/pages/inner_workings/_meta.json b/docs/pages/inner_workings/_meta.json index 509e9ab1..85ac35fa 100644 --- a/docs/pages/inner_workings/_meta.json +++ b/docs/pages/inner_workings/_meta.json @@ -2,4 +2,4 @@ "caching": "Caching (ISR/SSG)", "components": "Main Components", "architecture": "Default Architecture" -} \ No newline at end of file +} diff --git a/docs/pages/v3/_meta.json b/docs/pages/v3/_meta.json new file mode 100644 index 00000000..caefcec5 --- /dev/null +++ b/docs/pages/v3/_meta.json @@ -0,0 +1,7 @@ +{ + "index": "What's new", + "config": "Configuration file", + "reference-implementation": "Reference Construct", + "requirements": "Requirements", + "override": "Advanced - Create your own override" +} \ No newline at end of file diff --git a/docs/pages/v3/config.mdx b/docs/pages/v3/config.mdx new file mode 100644 index 00000000..56ee0ba5 --- /dev/null +++ b/docs/pages/v3/config.mdx @@ -0,0 +1,69 @@ +Here is a simple example of an `open-next.config.ts` file: +This file need to be at the same place as your `next.config.js` file + +`server` in here could refer to a lambda function, a docker container, a node server or whatever that can support running nodejs code. (Even cloudflare workers in the future) + +For more information about the options here, just look at the source file + +```ts +import type { OpenNextConfig } from 'open-next/types/open-next' +const config = { + default: { // This is the default server, similar to the server-function in open-next v2 + // You don't have to provide the below, by default it will generate an output + // for normal lambda as in open-next v2 + override: { + wrapper: "aws-lambda-streaming", // This is necessary to enable lambda streaming + // You can override any part that is a `LazyLoadedOverride` this way + queue: () => Promise.resolve({ + send: async (message) => { + //Your custom code here + } + }) + }, + }, + // Below we define the functions that we want to deploy in a different server + functions: { + ssr: { + routes: [ + "app/api/isr/route", "app/api/sse/route", "app/api/revalidateTag/route", // app dir Api routes + "app/route1/page", "app/route2/page", // app dir pages + "pages/route3" // page dir pages + ], // For app dir, you need to include route|page, no need to include layout or loading + patterns: ['api/*', 'route1', 'route2', 'route3'], // patterns needs to be in a cloudfront compatible format, this will be used to generate the output + override: { + wrapper: "aws-lambda-streaming", + }, + experimentalBundledNextServer: true // This enables the bundled next server which is faster and reduce the size of the server + }, + pageSsr: { + routes: ["pages/pageSsr"], // For page dir routes should be in the form `pages/${route}` without the extension, it should match the filesystem + patterns: [ 'pageSsr', "_next/data/BUILD_ID/pageSsr.json"], + override: { + wrapper: "node", + converter: "node", + // This is necessary to generate the dockerfile and for the implementation to know that it needs to deploy on docker + generateDockerfile: true, + }, + }, + edge: { + runtime: "edge", + routes: ["app/ssr/page"], + patterns: ["ssr"], + override: {} + } + }, + // By setting this, it will create another bundle for the middleware, + // and the middleware will be deployed in a separate server. + // If not set middleware will be bundled inside the servers + // It could be in lambda@edge, cloudflare workers, or anywhere else + // By default it uses lambda@edge + // This is not implemented in the reference construct implementation. + middleware: { + external: true + } + buildCommand: "echo 'hello world'", +} satisfies OpenNextConfig + +export default config; +export type Config = typeof config +``` \ No newline at end of file diff --git a/docs/pages/v3/index.mdx b/docs/pages/v3/index.mdx new file mode 100644 index 00000000..db0161e6 --- /dev/null +++ b/docs/pages/v3/index.mdx @@ -0,0 +1,67 @@ +import { Callout } from 'nextra/components' + + + +`open-next@3.0.0-rc.3` is here!!! Please report any issues you find on [discord](https://discord.com/channels/983865673656705025/1164872233223729152) or on the github [PR](https://github.com/sst/open-next/pull/327) + + This is a release candidate, it is mostly ready for production (You might still experience some quirks). We are looking for feedback on this release, so please try it out and let us know what you think. See [getting started](#get-started) to quickly test it. + + It also requires an updated version of the IAC tools that you use, see the sst PR [here](https://github.com/sst/sst/pull/3567) for more information. + + You could also use SST Ion which should support it out of the box pretty soon. See [here for more info](https://github.com/sst/ion) or in the [ion discord](https://discord.com/channels/983865673656705025/1177071497974648952). + + +## What's new in V3? + +- Rewritten server (We moved from the serverless-http `ServerResponse` into our own version which respect nodejs stream) +- A new `open-next.config.ts` file to configure your project +- Native support for aws lambda, aws lambda streaming, lambda@edge, node and docker +- In this `open-next.config.ts` you can now override a lot of things which allow for more customization: + - Wrapper component + - Converter component + - Incremental Cache + - Tag Cache + - Queue + - Custom warmer function + - Custom revalidation function + - Custom loader for image optimization + - Custom initialization function + +- Allow for splitting, you can now split your next app into multiple servers, which could each have their own configuration +- Allow to move the middleware/routing part in a separate lambda or cloudflare workers in front of your server functions +- An experimental bundled `NextServer` could be used which can reduce the size of your lambda by up to 24 MB +- Experimental support for the `edge` runtime of next with some limitations: + - Only app router for now + - Only 1 route per function + - Works fine in node, only for api route in cloudflare workers + - No support for `revalidateTag` or `revalidatePath` for now + +## Get started + +The easiest way to get started is to use the [reference implementation construct](/v3/reference-implementation). Copy this reference implementation into your project and then use it like that in your sst or cdk project: + +```ts +import { OpenNextCdkReferenceImplementation } from "path/to/reference-implementation" + +const site = new OpenNextCdkReferenceImplementation(stack, "site", { + openNextPath: ".open-next", +}) +``` + +You also need to create an `open-next.config.ts` file in your project root, you can find more info [here](/v3/config). + +A very simple example of this file could be: + +```ts +import type { OpenNextConfig } from 'open-next/types/open-next' +const config = { + default: { + + } +} +export default config; +``` + +Then you need to run `npx open-next@3.0.0-rc.3 build` to build your project before running the `sst deploy` or `cdk deploy` command to deploy your project. + +In V3 `open-next build` don't accept any arguments, all the args are passed in the `open-next.config.ts` file. \ No newline at end of file diff --git a/docs/pages/v3/override.mdx b/docs/pages/v3/override.mdx new file mode 100644 index 00000000..2cb1af3a --- /dev/null +++ b/docs/pages/v3/override.mdx @@ -0,0 +1,74 @@ +In this version of open-next, you could override a lot of the default behaviour. + +For some real example of how to override each behaviour: +- [Wrapper](https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/wrappers/aws-lambda.ts) +- [Converter](https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/converters/aws-apigw-v2.ts) +- [IncrementalCache](https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/cache/incremental/s3.ts) +- [TagCache]( + https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/cache/tag/dynamoDb.ts +) +- [Queue]( + https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/queue/sqs.ts +) + +This means it could allow people to write their own custom open-next. +For example you could create a custom `withGcp` plugin to allow to deploy open-next on GCP functions + +A boilerplate for such a plugin could look like this (This is not real code): + +```ts + +import { OpenNextConfig } from "open-next/types/open-next"; + +function withGcp(config: TrimmedDownConfig): OpenNextConfig { + return { + default: { + override: { + wrapper: async () => (await import("./gcp-wrapper")).default, + converter: async () => (await import("./gcp-converter")).default, + incrementalCache: async () => (await import("./gcp-incremental-cache")).default, + tagCache: async () => (await import("./gcp-tag-cache")).default, + queue: async () => (await import("./gcp-queue")).default, + }, + ...config.default, + }, + functions: { + // Same as default but for each splitted function + //... + } + warmer: { + override: { + wrapper: async () => (await import("./gcp-wrapper")).default, + converter: async () => (await import("./gcp-converter")).default, + }, + invokeFunction: async () => (await import("./gcp-invoke-function")).default, + }, + revalidate: { + override: { + wrapper: async () => (await import("./gcp-wrapper")).default, + converter: async () => (await import("./gcp-queue-converter")).default, + }, + }, + imageOptimization: { + override: { + wrapper: async () => (await import("./gcp-wrapper")).default, + converter: async () => (await import("./gcp-converter")).default, + }, + loader: async () => (await import("./gcp-object-loader")).default, + }, + } +} +``` + +Using this plugin would look like this inside `open-next.config.ts`: + +```ts +import { withGcp } from "./with-gcp"; +const config = withGcp({ + default: { + // ... + }, +}); + +export default config; +``` \ No newline at end of file diff --git a/docs/pages/v3/reference-implementation.mdx b/docs/pages/v3/reference-implementation.mdx new file mode 100644 index 00000000..02d213fb --- /dev/null +++ b/docs/pages/v3/reference-implementation.mdx @@ -0,0 +1,435 @@ +import { Callout } from 'nextra/components' + +In order to help testing the rc release, we created a simple reference implementation using aws-cdk. + +If you wish to use it, just copy the code for the construct below. If you use it inside sst, make sure to use the same version of aws-cdk as sst. + + + + This is a reference implementation, and it is not meant to be used in production. + + The goal is to help with the adoption of the rc release, and to gather feedback from the community. + It also serves as a good example of how to use the new features. + + There is some features that are not implemented like the warmer function, or everything related to lambda@edge(It requires inserting env variables which is out of scope of this implementation). + + +```ts +import { Construct } from "constructs"; +import { readFileSync } from "fs"; +import path from "path"; +import { BlockPublicAccess, Bucket } from "aws-cdk-lib/aws-s3"; +import { BucketDeployment, Source } from "aws-cdk-lib/aws-s3-deployment"; +import { + CustomResource, + Duration, + Fn, + RemovalPolicy, + Stack, +} from "aws-cdk-lib/core"; +import { + AllowedMethods, + BehaviorOptions, + CacheCookieBehavior, + CacheHeaderBehavior, + CachePolicy, + CacheQueryStringBehavior, + CachedMethods, + Distribution, + ICachePolicy, + ViewerProtocolPolicy, + FunctionEventType, + OriginRequestPolicy, + Function as CloudfrontFunction, + FunctionCode, +} from "aws-cdk-lib/aws-cloudfront"; +import { HttpOrigin, S3Origin } from "aws-cdk-lib/aws-cloudfront-origins"; +import { + Code, + Function as CdkFunction, + FunctionUrlAuthType, + InvokeMode, + Runtime, +} from "aws-cdk-lib/aws-lambda"; +import { + TableV2 as Table, + AttributeType, + Billing, +} from "aws-cdk-lib/aws-dynamodb"; +import { + Service, + Source as AppRunnerSource, + Memory, + HealthCheck, + Cpu, +} from "@aws-cdk/aws-apprunner-alpha"; +import { DockerImageAsset } from "aws-cdk-lib/aws-ecr-assets"; +import { Queue } from "aws-cdk-lib/aws-sqs"; +import { SqsEventSource } from "aws-cdk-lib/aws-lambda-event-sources"; +import { IGrantable } from "aws-cdk-lib/aws-iam"; +import { Provider } from "aws-cdk-lib/custom-resources"; +import { RetentionDays } from "aws-cdk-lib/aws-logs"; + +type BaseFunction = { + handler: string; + bundle: string; +}; + +type OpenNextFunctionOrigin = { + type: "function"; + streaming?: boolean; +} & BaseFunction; + +type OpenNextECSOrigin = { + type: "ecs"; + bundle: string; + dockerfile: string; +}; + +type OpenNextS3Origin = { + type: "s3"; + originPath: string; + copy: { + from: string; + to: string; + cached: boolean; + versionedSubDir?: string; + }[]; +}; + +type OpenNextOrigins = + | OpenNextFunctionOrigin + | OpenNextECSOrigin + | OpenNextS3Origin; + +interface OpenNextOutput { + edgeFunctions: { + [key: string]: BaseFunction; + }; + origins: { + s3: OpenNextS3Origin; + default: OpenNextFunctionOrigin | OpenNextECSOrigin; + imageOptimizer: OpenNextFunctionOrigin | OpenNextECSOrigin; + [key: string]: OpenNextOrigins; + }; + behaviors: { + pattern: string; + origin?: string; + edgeFunction?: string; + }[]; + additionalProps?: { + disableIncrementalCache?: boolean; + disableTagCache?: boolean; + initializationFunction?: BaseFunction; + warmer?: BaseFunction; + revalidationFunction?: BaseFunction; + }; +} + +interface OpenNextCdkReferenceImplementationProps { + openNextPath: string; +} + +export class OpenNextCdkReferenceImplementation extends Construct { + private openNextOutput: OpenNextOutput; + private bucket: Bucket; + private table: Table; + private queue: Queue; + + private staticCachePolicy: ICachePolicy; + private serverCachePolicy: CachePolicy; + + public distribution: Distribution; + + constructor( + scope: Construct, + id: string, + props: OpenNextCdkReferenceImplementationProps, + ) { + super(scope, id); + this.openNextOutput = JSON.parse( + readFileSync( + path.join(props.openNextPath, "open-next.output.json"), + "utf-8", + ), + ) as OpenNextOutput; + + this.bucket = new Bucket(this, "OpenNextBucket", { + publicReadAccess: false, + blockPublicAccess: BlockPublicAccess.BLOCK_ALL, + autoDeleteObjects: true, + removalPolicy: RemovalPolicy.DESTROY, + enforceSSL: true, + }); + this.table = this.createRevalidationTable(); + this.queue = this.createRevalidationQueue(); + + const origins = this.createOrigins(); + this.serverCachePolicy = this.createServerCachePolicy(); + this.staticCachePolicy = this.createStaticCachePolicy(); + this.distribution = this.createDistribution(origins); + } + + private createRevalidationTable() { + const table = new Table(this, "RevalidationTable", { + partitionKey: { name: "tag", type: AttributeType.STRING }, + sortKey: { name: "path", type: AttributeType.STRING }, + pointInTimeRecovery: true, + billing: Billing.onDemand(), + globalSecondaryIndexes: [ + { + indexName: "revalidate", + partitionKey: { name: "path", type: AttributeType.STRING }, + sortKey: { name: "revalidatedAt", type: AttributeType.NUMBER }, + }, + ], + removalPolicy: RemovalPolicy.DESTROY, + }); + + const initFn = this.openNextOutput.additionalProps?.initializationFunction; + + const insertFn = new CdkFunction(this, "RevalidationInsertFunction", { + description: "Next.js revalidation data insert", + handler: initFn?.handler ?? "index.handler", + // code: Code.fromAsset(initFn?.bundle ?? ""), + code: Code.fromAsset(".open-next/dynamodb-provider"), + runtime: Runtime.NODEJS_18_X, + timeout: Duration.minutes(15), + memorySize: 128, + environment: { + CACHE_DYNAMO_TABLE: table.tableName, + }, + }); + + const provider = new Provider(this, "RevalidationProvider", { + onEventHandler: insertFn, + logRetention: RetentionDays.ONE_DAY, + }); + + new CustomResource(this, "RevalidationResource", { + serviceToken: provider.serviceToken, + properties: { + version: Date.now().toString(), + }, + }); + + return table; + } + + private createOrigins() { + const { + s3: s3Origin, + default: defaultOrigin, + imageOptimizer: imageOrigin, + ...restOrigins + } = this.openNextOutput.origins; + const s3 = new S3Origin(this.bucket, { + originPath: s3Origin.originPath, + }); + for (const copy of s3Origin.copy) { + new BucketDeployment(this, `OpenNextBucketDeployment${copy.from}`, { + sources: [Source.asset(copy.from)], + destinationBucket: this.bucket, + destinationKeyPrefix: copy.to, + prune: false, + }); + } + const origins = { + s3: new S3Origin(this.bucket, { + originPath: s3Origin.originPath, + originAccessIdentity: undefined, + }), + default: + defaultOrigin.type === "function" + ? this.createFunctionOrigin("default", defaultOrigin) + : this.createAppRunnerOrigin("default", defaultOrigin), + imageOptimizer: + imageOrigin.type === "function" + ? this.createFunctionOrigin("imageOptimizer", imageOrigin) + : this.createAppRunnerOrigin("imageOptimizer", imageOrigin), + ...Object.entries(restOrigins).reduce( + (acc, [key, value]) => { + if (value.type === "function") { + acc[key] = this.createFunctionOrigin(key, value); + } else if (value.type === "ecs") { + acc[key] = this.createAppRunnerOrigin(key, value); + } + return acc; + }, + {} as Record, + ), + }; + return origins; + } + + private createRevalidationQueue() { + const queue = new Queue(this, "RevalidationQueue", { + fifo: true, + receiveMessageWaitTime: Duration.seconds(20), + }); + const consumer = new CdkFunction(this, "RevalidationFunction", { + description: "Next.js revalidator", + handler: "index.handler", + code: Code.fromAsset( + this.openNextOutput.additionalProps?.revalidationFunction?.bundle ?? "", + ), + runtime: Runtime.NODEJS_18_X, + timeout: Duration.seconds(30), + }); + consumer.addEventSource(new SqsEventSource(queue, { batchSize: 5 })); + return queue; + } + + private getEnvironment() { + return { + CACHE_BUCKET_NAME: this.bucket.bucketName, + CACHE_BUCKET_KEY_PREFIX: "_cache", + CACHE_BUCKET_REGION: Stack.of(this).region, + REVALIDATION_QUEUE_URL: this.queue.queueUrl, + REVALIDATION_QUEUE_REGION: Stack.of(this).region, + CACHE_DYNAMO_TABLE: this.table.tableName, + // Those 2 are used only for image optimizer + BUCKET_NAME: this.bucket.bucketName, + BUCKET_KEY_PREFIX: "_assets", + }; + } + + private grantPermissions(grantable: IGrantable) { + this.bucket.grantReadWrite(grantable); + this.table.grantReadWriteData(grantable); + this.queue.grantSendMessages(grantable); + } + + private createFunctionOrigin(key: string, origin: OpenNextFunctionOrigin) { + const environment = this.getEnvironment(); + const fn = new CdkFunction(this, `${key}Function`, { + runtime: Runtime.NODEJS_18_X, + handler: origin.handler, + code: Code.fromAsset(origin.bundle), + environment, + memorySize: 1024, + }); + const fnUrl = fn.addFunctionUrl({ + authType: FunctionUrlAuthType.NONE, + invokeMode: origin.streaming + ? InvokeMode.RESPONSE_STREAM + : InvokeMode.BUFFERED, + }); + this.grantPermissions(fn); + return new HttpOrigin(Fn.parseDomainName(fnUrl.url)); + } + + // We are using AppRunner because it is the easiest way to demonstrate the new feature. + // You can use any other container service like ECS, EKS, Fargate, etc. + private createAppRunnerOrigin( + key: string, + origin: OpenNextECSOrigin, + ): HttpOrigin { + const imageAsset = new DockerImageAsset(this, `${key}ImageAsset`, { + directory: origin.bundle, + // file: origin.dockerfile, + }); + const service = new Service(this, `${key}Service`, { + source: AppRunnerSource.fromAsset({ + asset: imageAsset, + + imageConfiguration: { + port: 3000, + environmentVariables: this.getEnvironment(), + }, + }), + serviceName: key, + autoDeploymentsEnabled: false, + cpu: Cpu.HALF_VCPU, + memory: Memory.ONE_GB, + healthCheck: HealthCheck.http({ + path: "/__health", + }), + }); + this.grantPermissions(service); + return new HttpOrigin(service.serviceUrl); + } + + private createDistribution(origins: Record) { + const cloudfrontFunction = new CloudfrontFunction(this, 'OpenNextCfFunction', { + code: FunctionCode.fromInline(` + function handler(event) { + var request = event.request; + request.headers["x-forwarded-host"] = request.headers.host; + return request; + } + `) + }) + const fnAssociations = [ + { + function: cloudfrontFunction , + eventType: FunctionEventType.VIEWER_REQUEST, + }, + ] + + const distribution = new Distribution(this, "OpenNextDistribution", { + defaultBehavior: { + origin: origins.default, + viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS, + allowedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS, + cachedMethods: CachedMethods.CACHE_GET_HEAD_OPTIONS, + cachePolicy: this.serverCachePolicy, + originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + functionAssociations: fnAssociations + }, + additionalBehaviors: this.openNextOutput.behaviors + .filter((b) => b.pattern !== "*") + .reduce( + (acc, behavior) => { + return { + ...acc, + [behavior.pattern]: { + origin: behavior.origin + ? origins[behavior.origin] + : origins.default, + viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS, + allowedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS, + cachedMethods: CachedMethods.CACHE_GET_HEAD_OPTIONS, + cachePolicy: + behavior.origin === "s3" + ? this.staticCachePolicy + : this.serverCachePolicy, + originRequestPolicy: + behavior.origin === "s3" + ? undefined + : OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + functionAssociations: fnAssociations + }, + }; + }, + {} as Record, + ), + }); + return distribution; + } + + private createServerCachePolicy() { + return new CachePolicy(this, "OpenNextServerCachePolicy", { + queryStringBehavior: CacheQueryStringBehavior.all(), + headerBehavior: CacheHeaderBehavior.allowList( + "accept", + "accept-encoding", + "rsc", + "next-router-prefetch", + "next-router-state-tree", + "next-url", + "x-prerender-revalidate", + ), + cookieBehavior: CacheCookieBehavior.none(), + defaultTtl: Duration.days(0), + maxTtl: Duration.days(365), + minTtl: Duration.days(0), + }); + } + + private createStaticCachePolicy() { + return CachePolicy.CACHING_OPTIMIZED; + } +} + +``` diff --git a/docs/pages/v3/requirements.mdx b/docs/pages/v3/requirements.mdx new file mode 100644 index 00000000..71279afa --- /dev/null +++ b/docs/pages/v3/requirements.mdx @@ -0,0 +1,59 @@ +There is a couple of requirements necessary for open-next V3 to work. +It will be divided by functionality. This is still WIP, feel free to open a PR if you think something is missing. + +## General +- For the node runtime, you need at least Node 18. +- For the edge runtime, you can use both Node 18+ or cloudflare workers with `node_compat` flag enabled (Cloudflare workers support is experimental) +- Open-next doesn't work well on Windows. We recommend using WSL2 or a Linux VM. + +## ISR/SSG +ISR/SSG has 2 types of cache, the Incremental Cache and the Tag Cache. To actually trigger the ISR revalidation, we use a Queue system. + +The tag cache is only used in app router. +### Incremental Cache +By default we use S3 as the incremental cache. You can override this in `open-next.config.ts`. For this to work you need to provide server functions with the following environment variables: +- CACHE_BUCKET_REGION +- CACHE_BUCKET_NAME +- CACHE_BUCKET_KEY_PREFIX + +### Tag Cache +By default we use DynamoDb as the tag cache. For this to work you need to provide server functions with the following environment variables: +- CACHE_DYNAMO_TABLE +- CACHE_BUCKET_REGION + +### Queue +By default we use SQS as the queue. fFr this to work you need to provide server functions with the following environment variables: +- REVALIDATION_QUEUE_REGION +- REVALIDATION_QUEUE_URL + +## External Middleware +If you decide to use external middleware, you need to provide the following environment variables: +- OPEN_NEXT_ORIGIN + +This env variable should contain a stringified version of this, with every key corresponding to the key used in functions inside `open-next.config.ts`: +```ts +// For cloudflare workers +// THIS IS TEMPORARY, WE WILL CHANGE THIS TO USE THE SAME FORMAT AS NODE +{ + default: "example.com", + ssr: "example2.com", + ssg: "example3.com" +} +// Or for node +{ + default: { + host: "example.com", + protocol: "https", + port: 443 // Optional + customHeaders: { + "x-custom-header": "value" + } // Optional, headers that you'd want to pass to the origin + }, + ... +} +``` + +## Image Optimization +For image optimization to work, you need to provide the following environment variables: +- BUCKET_NAME +- BUCKET_KEY_PREFIX \ No newline at end of file diff --git a/examples/app-pages-router/open-next.config.ts b/examples/app-pages-router/open-next.config.ts new file mode 100644 index 00000000..2a9b827b --- /dev/null +++ b/examples/app-pages-router/open-next.config.ts @@ -0,0 +1,12 @@ +const config = { + default: {}, + functions: { + api: { + routes: ["app/api/client/route", "app/api/host/route", "pages/api/hello"], + patterns: ["/api/*"], + }, + }, + buildCommand: "npx turbo build", +}; + +module.exports = config; diff --git a/examples/app-router/app/ssr/layout.tsx b/examples/app-router/app/ssr/layout.tsx index 3a8338f5..5ab0b64a 100644 --- a/examples/app-router/app/ssr/layout.tsx +++ b/examples/app-router/app/ssr/layout.tsx @@ -4,6 +4,8 @@ export default function Layout({ children }: PropsWithChildren) { return (

SSR

+ {/* 16 kb seems necessary here to prevent any buffering*/} + {/* */} {children}
); diff --git a/examples/app-router/open-next.config.ts b/examples/app-router/open-next.config.ts new file mode 100644 index 00000000..f5f7c73b --- /dev/null +++ b/examples/app-router/open-next.config.ts @@ -0,0 +1,11 @@ +const config = { + default: { + override: { + wrapper: "aws-lambda-streaming", + }, + }, + functions: {}, + buildCommand: "npx turbo build", +}; + +module.exports = config; diff --git a/examples/app-router/package.json b/examples/app-router/package.json index 8775b4ea..87ce8c57 100644 --- a/examples/app-router/package.json +++ b/examples/app-router/package.json @@ -13,7 +13,7 @@ "dependencies": { "@example/shared": "workspace:*", "@open-next/utils": "workspace:*", - "next": "^14.0.3", + "next": "^14.1.4", "open-next": "workspace:*", "react": "latest", "react-dom": "latest" diff --git a/examples/pages-router/open-next.config.ts b/examples/pages-router/open-next.config.ts new file mode 100644 index 00000000..54e09d87 --- /dev/null +++ b/examples/pages-router/open-next.config.ts @@ -0,0 +1,7 @@ +const config = { + default: {}, + functions: {}, + buildCommand: "npx turbo build", +}; + +module.exports = config; diff --git a/examples/shared/components/Filler/index.tsx b/examples/shared/components/Filler/index.tsx new file mode 100644 index 00000000..51ddf2e4 --- /dev/null +++ b/examples/shared/components/Filler/index.tsx @@ -0,0 +1,17 @@ +interface FillerProps { + // Size in kb of the filler + size: number; +} + +//This component is there to demonstrate how you could bypass streaming buffering in aws lambda. +//Hopefully, this will be fixed in the future and this component will be removed. +// https://github.com/aws/aws-lambda-nodejs-runtime-interface-client/issues/94 +export default function Filler({ size }: FillerProps) { + const str = "a".repeat(size * 1024); + const byteSize = new TextEncoder().encode(str).length; + return ( + + ); +} diff --git a/examples/sst/stacks/AppPagesRouter.ts b/examples/sst/stacks/AppPagesRouter.ts index 80b8710e..d1c8750d 100644 --- a/examples/sst/stacks/AppPagesRouter.ts +++ b/examples/sst/stacks/AppPagesRouter.ts @@ -1,15 +1,18 @@ -import { NextjsSite } from "sst/constructs"; +import { OpenNextCdkReferenceImplementation } from "./OpenNextReferenceImplementation"; // NOTE: App Pages Router doesn't do streaming export function AppPagesRouter({ stack }) { - const site = new NextjsSite(stack, "apppagesrouter", { + const site = new OpenNextCdkReferenceImplementation(stack, "apppagesrouter", { path: "../app-pages-router", - buildCommand: "npm run openbuild", - bind: [], - environment: {}, }); + // const site = new NextjsSite(stack, "apppagesrouter", { + // path: "../app-pages-router", + // buildCommand: "npm run openbuild", + // bind: [], + // environment: {}, + // }); stack.addOutputs({ - url: site.url, + url: `https://${site.distribution.domainName}`, }); } diff --git a/examples/sst/stacks/AppRouter.ts b/examples/sst/stacks/AppRouter.ts index f0e5950d..e9aa3aa6 100644 --- a/examples/sst/stacks/AppRouter.ts +++ b/examples/sst/stacks/AppRouter.ts @@ -1,18 +1,22 @@ -import { NextjsSite } from "sst/constructs"; +import { OpenNextCdkReferenceImplementation } from "./OpenNextReferenceImplementation"; export function AppRouter({ stack }) { - const site = new NextjsSite(stack, "approuter", { + // We should probably switch to ion once it's ready + const site = new OpenNextCdkReferenceImplementation(stack, "approuter", { path: "../app-router", - buildCommand: "npm run openbuild", - bind: [], - environment: {}, - timeout: "20 seconds", - experimental: { - streaming: true, - }, }); + // const site = new NextjsSite(stack, "approuter", { + // path: "../app-router", + // buildCommand: "npm run openbuild", + // bind: [], + // environment: {}, + // timeout: "20 seconds", + // experimental: { + // streaming: true, + // }, + // }); stack.addOutputs({ - url: site.url, + url: `https://${site.distribution.domainName}`, }); } diff --git a/examples/sst/stacks/OpenNextReferenceImplementation.ts b/examples/sst/stacks/OpenNextReferenceImplementation.ts new file mode 100644 index 00000000..d005929d --- /dev/null +++ b/examples/sst/stacks/OpenNextReferenceImplementation.ts @@ -0,0 +1,462 @@ +import { execSync } from "node:child_process"; + +import { + AllowedMethods, + BehaviorOptions, + CacheCookieBehavior, + CachedMethods, + CacheHeaderBehavior, + CachePolicy, + CacheQueryStringBehavior, + Distribution, + Function as CloudfrontFunction, + FunctionCode, + FunctionEventType, + ICachePolicy, + OriginRequestPolicy, + ViewerProtocolPolicy, +} from "aws-cdk-lib/aws-cloudfront"; +import { HttpOrigin, S3Origin } from "aws-cdk-lib/aws-cloudfront-origins"; +import { + AttributeType, + Billing, + TableV2 as Table, +} from "aws-cdk-lib/aws-dynamodb"; +import { IGrantable, Policy, PolicyStatement } from "aws-cdk-lib/aws-iam"; +import { + Architecture, + Code, + Function as CdkFunction, + FunctionUrlAuthType, + InvokeMode, + Runtime, +} from "aws-cdk-lib/aws-lambda"; +import { SqsEventSource } from "aws-cdk-lib/aws-lambda-event-sources"; +import { BlockPublicAccess, Bucket } from "aws-cdk-lib/aws-s3"; +import { BucketDeployment, Source } from "aws-cdk-lib/aws-s3-deployment"; +import { Queue } from "aws-cdk-lib/aws-sqs"; +import { + CustomResource, + Duration, + Fn, + RemovalPolicy, + Stack, +} from "aws-cdk-lib/core"; +import { + AwsCustomResource, + AwsCustomResourcePolicy, + PhysicalResourceId, +} from "aws-cdk-lib/custom-resources"; +import { Construct } from "constructs"; +import { readFileSync } from "fs"; +import path from "path"; +import { Stack as SSTStack } from "sst/constructs"; + +type BaseFunction = { + handler: string; + bundle: string; +}; + +type OpenNextFunctionOrigin = { + type: "function"; + streaming?: boolean; +} & BaseFunction; + +type OpenNextECSOrigin = { + type: "ecs"; + bundle: string; + dockerfile: string; +}; + +type OpenNextS3Origin = { + type: "s3"; + originPath: string; + copy: { + from: string; + to: string; + cached: boolean; + versionedSubDir?: string; + }[]; +}; + +type OpenNextOrigins = + | OpenNextFunctionOrigin + | OpenNextECSOrigin + | OpenNextS3Origin; + +interface OpenNextOutput { + edgeFunctions: { + [key: string]: BaseFunction; + }; + origins: { + s3: OpenNextS3Origin; + default: OpenNextFunctionOrigin | OpenNextECSOrigin; + imageOptimizer: OpenNextFunctionOrigin | OpenNextECSOrigin; + [key: string]: OpenNextOrigins; + }; + behaviors: { + pattern: string; + origin?: string; + edgeFunction?: string; + }[]; + additionalProps?: { + disableIncrementalCache?: boolean; + disableTagCache?: boolean; + initializationFunction?: BaseFunction; + warmer?: BaseFunction; + revalidationFunction?: BaseFunction; + }; +} + +interface OpenNextCdkReferenceImplementationProps { + path: string; +} + +export class OpenNextCdkReferenceImplementation extends Construct { + private openNextOutput: OpenNextOutput; + private openNextBasePath: string; + private bucket: Bucket; + private table: Table; + private queue: Queue; + + private staticCachePolicy: ICachePolicy; + private serverCachePolicy: CachePolicy; + + public distribution: Distribution; + + constructor( + scope: Construct, + id: string, + props: OpenNextCdkReferenceImplementationProps, + ) { + super(scope, id); + this.openNextBasePath = path.join(process.cwd(), props.path); + execSync("npm run openbuild", { + cwd: path.join(process.cwd(), props.path), + stdio: "inherit", + }); + + this.openNextOutput = JSON.parse( + readFileSync( + path.join(this.openNextBasePath, ".open-next/open-next.output.json"), + "utf-8", + ), + ) as OpenNextOutput; + + this.bucket = new Bucket(this, "OpenNextBucket", { + publicReadAccess: false, + blockPublicAccess: BlockPublicAccess.BLOCK_ALL, + autoDeleteObjects: true, + removalPolicy: RemovalPolicy.DESTROY, + enforceSSL: true, + }); + this.table = this.createRevalidationTable(); + this.queue = this.createRevalidationQueue(); + + const origins = this.createOrigins(); + this.serverCachePolicy = this.createServerCachePolicy(); + this.staticCachePolicy = this.createStaticCachePolicy(); + this.distribution = this.createDistribution(origins); + this.createInvalidation(); + } + + private createRevalidationTable() { + const table = new Table(this, "RevalidationTable", { + partitionKey: { name: "tag", type: AttributeType.STRING }, + sortKey: { name: "path", type: AttributeType.STRING }, + pointInTimeRecovery: true, + billing: Billing.onDemand(), + globalSecondaryIndexes: [ + { + indexName: "revalidate", + partitionKey: { name: "path", type: AttributeType.STRING }, + sortKey: { name: "revalidatedAt", type: AttributeType.NUMBER }, + }, + ], + removalPolicy: RemovalPolicy.DESTROY, + }); + + const initFn = this.openNextOutput.additionalProps?.initializationFunction; + if (initFn) { + const insertFn = new CdkFunction(this, "RevalidationInsertFunction", { + description: "Next.js revalidation data insert", + handler: initFn?.handler ?? "index.handler", + // code: Code.fromAsset(initFn?.bundle ?? ""), + code: Code.fromAsset( + path.join(this.openNextBasePath, ".open-next/dynamodb-provider"), + ), + runtime: Runtime.NODEJS_18_X, + timeout: Duration.minutes(15), + memorySize: 128, + environment: { + CACHE_DYNAMO_TABLE: table.tableName, + }, + initialPolicy: [ + new PolicyStatement({ + actions: [ + "dynamodb:BatchWriteItem", + "dynamodb:PutItem", + "dynamodb:UpdateItem", + ], + resources: [table.tableArn], + }), + ], + }); + + const customResource = new AwsCustomResource( + this, + "RevalidationInitResource", + { + onUpdate: { + service: "Lambda", + action: "invoke", + parameters: { + FunctionName: insertFn.functionName, + }, + physicalResourceId: PhysicalResourceId.of(Date.now().toString()), + }, + + policy: AwsCustomResourcePolicy.fromStatements([ + new PolicyStatement({ + actions: ["lambda:InvokeFunction"], + resources: [insertFn.functionArn], + }), + ]), + }, + ); + customResource.node.addDependency(insertFn); + } + + return table; + } + + private createOrigins() { + const { + s3: s3Origin, + default: defaultOrigin, + imageOptimizer: imageOrigin, + ...restOrigins + } = this.openNextOutput.origins; + for (const copy of s3Origin.copy) { + new BucketDeployment(this, `OpenNextBucketDeployment${copy.from}`, { + sources: [Source.asset(path.join(this.openNextBasePath, copy.from))], + destinationBucket: this.bucket, + destinationKeyPrefix: copy.to, + prune: false, + }); + } + const origins = { + s3: new S3Origin(this.bucket, { + originPath: s3Origin.originPath, + originAccessIdentity: undefined, + }), + default: + defaultOrigin.type === "function" + ? this.createFunctionOrigin("default", defaultOrigin) + : this.createAppRunnerOrigin("default", defaultOrigin), + imageOptimizer: + imageOrigin.type === "function" + ? this.createFunctionOrigin("imageOptimizer", imageOrigin) + : this.createAppRunnerOrigin("imageOptimizer", imageOrigin), + ...Object.entries(restOrigins).reduce( + (acc, [key, value]) => { + if (value.type === "function") { + acc[key] = this.createFunctionOrigin(key, value); + // eslint-disable-next-line sonarjs/elseif-without-else + } else if (value.type === "ecs") { + acc[key] = this.createAppRunnerOrigin(key, value); + } + return acc; + }, + {} as Record, + ), + }; + return origins; + } + + private createRevalidationQueue() { + const queue = new Queue(this, "RevalidationQueue", { + fifo: true, + receiveMessageWaitTime: Duration.seconds(20), + }); + const consumer = new CdkFunction(this, "RevalidationFunction", { + description: "Next.js revalidator", + handler: "index.handler", + code: Code.fromAsset( + path.join( + this.openNextBasePath, + this.openNextOutput.additionalProps?.revalidationFunction?.bundle ?? + "", + ), + ), + runtime: Runtime.NODEJS_18_X, + timeout: Duration.seconds(30), + }); + consumer.addEventSource(new SqsEventSource(queue, { batchSize: 5 })); + return queue; + } + + private getEnvironment() { + return { + CACHE_BUCKET_NAME: this.bucket.bucketName, + CACHE_BUCKET_KEY_PREFIX: "_cache", + CACHE_BUCKET_REGION: Stack.of(this).region, + REVALIDATION_QUEUE_URL: this.queue.queueUrl, + REVALIDATION_QUEUE_REGION: Stack.of(this).region, + CACHE_DYNAMO_TABLE: this.table.tableName, + // Those 2 are used only for image optimizer + BUCKET_NAME: this.bucket.bucketName, + BUCKET_KEY_PREFIX: "_assets", + }; + } + + private grantPermissions(grantable: IGrantable) { + this.bucket.grantReadWrite(grantable); + this.table.grantReadWriteData(grantable); + this.queue.grantSendMessages(grantable); + } + + private createFunctionOrigin(key: string, origin: OpenNextFunctionOrigin) { + const environment = this.getEnvironment(); + const fn = new CdkFunction(this, `${key}Function`, { + architecture: Architecture.ARM_64, + runtime: Runtime.NODEJS_18_X, + handler: origin.handler, + code: Code.fromAsset(path.join(this.openNextBasePath, origin.bundle)), + environment, + memorySize: 1024, + timeout: Duration.seconds(20), + }); + const fnUrl = fn.addFunctionUrl({ + authType: FunctionUrlAuthType.NONE, + invokeMode: origin.streaming + ? InvokeMode.RESPONSE_STREAM + : InvokeMode.BUFFERED, + }); + this.grantPermissions(fn); + return new HttpOrigin(Fn.parseDomainName(fnUrl.url)); + } + + // We are using AppRunner because it is the easiest way to demonstrate the new feature. + // You can use any other container service like ECS, EKS, Fargate, etc. + private createAppRunnerOrigin( + _key: string, + _origin: OpenNextECSOrigin, + ): HttpOrigin { + throw new Error("Not implemented"); + } + + private createDistribution(origins: Record) { + const cloudfrontFunction = new CloudfrontFunction( + this, + "OpenNextCfFunction", + { + code: FunctionCode.fromInline(` + function handler(event) { + var request = event.request; + request.headers["x-forwarded-host"] = request.headers.host; + return request; + } + `), + }, + ); + const fnAssociations = [ + { + function: cloudfrontFunction, + eventType: FunctionEventType.VIEWER_REQUEST, + }, + ]; + + const distribution = new Distribution(this, "OpenNextDistribution", { + defaultBehavior: { + origin: origins.default, + viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS, + allowedMethods: AllowedMethods.ALLOW_ALL, + cachedMethods: CachedMethods.CACHE_GET_HEAD_OPTIONS, + cachePolicy: this.serverCachePolicy, + originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + functionAssociations: fnAssociations, + }, + additionalBehaviors: this.openNextOutput.behaviors + .filter((b) => b.pattern !== "*") + .reduce( + (acc, behavior) => { + return { + ...acc, + [behavior.pattern]: { + origin: behavior.origin + ? origins[behavior.origin] + : origins.default, + viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS, + allowedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS, + cachedMethods: CachedMethods.CACHE_GET_HEAD_OPTIONS, + cachePolicy: + behavior.origin === "s3" + ? this.staticCachePolicy + : this.serverCachePolicy, + originRequestPolicy: + behavior.origin === "s3" + ? undefined + : OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + functionAssociations: fnAssociations, + }, + }; + }, + {} as Record, + ), + }); + return distribution; + } + + private createInvalidation() { + const stack = SSTStack.of(this) as SSTStack; + const policy = new Policy(this, "OpenNextInvalidationPolicy", { + statements: [ + new PolicyStatement({ + actions: [ + "cloudfront:CreateInvalidation", + "cloudfront:GetInvalidation", + ], + resources: [ + `arn:${stack.partition}:cloudfront::${stack.account}:distribution/${this.distribution.distributionId}`, + ], + }), + ], + }); + + stack.customResourceHandler.role?.attachInlinePolicy(policy); + const resource = new CustomResource(this, "OpenNextInvalidationResource", { + serviceToken: stack.customResourceHandler.functionArn, + resourceType: "Custom::CloudFrontInvalidator", + properties: { + version: Date.now().toString(16) + Math.random().toString(16).slice(2), + distributionId: this.distribution.distributionId, + paths: ["/*"], + wait: true, + }, + }); + resource.node.addDependency(policy); + } + + private createServerCachePolicy() { + return new CachePolicy(this, "OpenNextServerCachePolicy", { + queryStringBehavior: CacheQueryStringBehavior.all(), + headerBehavior: CacheHeaderBehavior.allowList( + "accept", + "rsc", + "next-router-prefetch", + "next-router-state-tree", + "next-url", + "x-prerender-revalidate", + ), + cookieBehavior: CacheCookieBehavior.none(), + defaultTtl: Duration.days(0), + maxTtl: Duration.days(365), + minTtl: Duration.days(0), + }); + } + + private createStaticCachePolicy() { + return CachePolicy.CACHING_OPTIMIZED; + } +} diff --git a/examples/sst/stacks/PagesRouter.ts b/examples/sst/stacks/PagesRouter.ts index e6ae5842..75ddd227 100644 --- a/examples/sst/stacks/PagesRouter.ts +++ b/examples/sst/stacks/PagesRouter.ts @@ -1,14 +1,17 @@ -import { NextjsSite } from "sst/constructs"; +import { OpenNextCdkReferenceImplementation } from "./OpenNextReferenceImplementation"; export function PagesRouter({ stack }) { - const site = new NextjsSite(stack, "pagesrouter", { + const site = new OpenNextCdkReferenceImplementation(stack, "pagesrouter", { path: "../pages-router", - buildCommand: "npm run openbuild", - bind: [], - environment: {}, }); + // const site = new NextjsSite(stack, "pagesrouter", { + // path: "../pages-router", + // buildCommand: "npm run openbuild", + // bind: [], + // environment: {}, + // }); stack.addOutputs({ - url: site.url, + url: `https://${site.distribution.domainName}`, }); } diff --git a/packages/open-next/package.json b/packages/open-next/package.json index af0a741a..c768493d 100644 --- a/packages/open-next/package.json +++ b/packages/open-next/package.json @@ -13,7 +13,7 @@ "homepage": "https://open-next.js.org", "main": "./dist/index.js", "scripts": { - "build": "tsc", + "build": "tsc && tsc-alias", "dev": "tsc -w" }, "exports": { @@ -38,17 +38,19 @@ "@aws-sdk/client-lambda": "^3.398.0", "@aws-sdk/client-s3": "^3.398.0", "@aws-sdk/client-sqs": "^3.398.0", + "@esbuild-plugins/node-resolve": "0.2.2", "@node-minify/core": "^8.0.6", "@node-minify/terser": "^8.0.6", "@tsconfig/node18": "^1.0.1", + "chalk": "^5.3.0", "esbuild": "0.19.2", - "@esbuild-plugins/node-resolve": "0.2.2", "path-to-regexp": "^6.2.1", "promise.series": "^0.2.0" }, "devDependencies": { "@types/aws-lambda": "^8.10.109", "@types/node": "^18.16.1", + "tsc-alias": "^1.8.8", "typescript": "^4.9.3" }, "bugs": { diff --git a/packages/open-next/src/adapters/cache.ts b/packages/open-next/src/adapters/cache.ts index 347a0662..5e6f4582 100644 --- a/packages/open-next/src/adapters/cache.ts +++ b/packages/open-next/src/adapters/cache.ts @@ -1,22 +1,7 @@ -import { - BatchWriteItemCommand, - DynamoDBClient, - QueryCommand, -} from "@aws-sdk/client-dynamodb"; -import { - DeleteObjectsCommand, - GetObjectCommand, - ListObjectsV2Command, - PutObjectCommand, - PutObjectCommandInput, - S3Client, -} from "@aws-sdk/client-s3"; -import path from "path"; - +import { IncrementalCache } from "../cache/incremental/types.js"; +import { TagCache } from "../cache/tag/types.js"; import { isBinaryContentType } from "./binary.js"; -import { MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT } from "./constants.js"; import { debug, error, warn } from "./logger.js"; -import { chunk } from "./util.js"; interface CachedFetchValue { kind: "FETCH"; @@ -96,36 +81,6 @@ interface CacheHandlerValue { value: IncrementalCacheValue | null; } -type Extension = "cache" | "fetch"; - -interface Meta { - status?: number; - headers?: Record; -} -type S3CachedFile = - | { - type: "redirect"; - props?: Object; - meta?: Meta; - } - | { - type: "page"; - html: string; - json: Object; - meta?: Meta; - } - | { - type: "app"; - html: string; - rsc: string; - meta?: Meta; - } - | { - type: "route"; - body: string; - meta?: Meta; - }; - /** Beginning single backslash is intentional, to look for the dot + the extension. Do not escape it again. */ const CACHE_EXTENSION_REGEX = /\.(cache|fetch)$/; @@ -133,32 +88,16 @@ export function hasCacheExtension(key: string) { return CACHE_EXTENSION_REGEX.test(key); } -// Expected environment variables -const { - CACHE_BUCKET_NAME, - CACHE_BUCKET_KEY_PREFIX, - CACHE_DYNAMO_TABLE, - NEXT_BUILD_ID, -} = process.env; - declare global { - var S3Client: S3Client; - var dynamoClient: DynamoDBClient; + var incrementalCache: IncrementalCache; + var tagCache: TagCache; var disableDynamoDBCache: boolean; var disableIncrementalCache: boolean; - var lastModified: number; + var lastModified: Record; } - +// We need to use globalThis client here as this class can be defined at load time in next 12 but client is not available at load time export default class S3Cache { - private client: S3Client; - private dynamoClient: DynamoDBClient; - private buildId: string; - - constructor(_ctx: CacheHandlerContext) { - this.client = globalThis.S3Client; - this.dynamoClient = globalThis.dynamoClient; - this.buildId = NEXT_BUILD_ID!; - } + constructor(_ctx: CacheHandlerContext) {} public async get( key: string, @@ -184,21 +123,25 @@ export default class S3Cache { async getFetchCache(key: string) { debug("get fetch cache", { key }); try { - const { Body, LastModified } = await this.getS3Object(key, "fetch"); - const lastModified = await this.getHasRevalidatedTags( + const { value, lastModified } = await globalThis.incrementalCache.get( key, - LastModified?.getTime(), + true, ); - if (lastModified === -1) { + // const { Body, LastModified } = await this.getS3Object(key, "fetch"); + const _lastModified = await globalThis.tagCache.getLastModified( + key, + lastModified, + ); + if (_lastModified === -1) { // If some tags are stale we need to force revalidation return null; } - if (Body === null) return null; + if (value === undefined) return null; return { - lastModified, - value: JSON.parse((await Body?.transformToString()) ?? "{}"), + lastModified: _lastModified, + value: value, } as CacheHandlerValue; } catch (e) { error("Failed to get fetch cache", e); @@ -208,23 +151,26 @@ export default class S3Cache { async getIncrementalCache(key: string): Promise { try { - const { Body, LastModified } = await this.getS3Object(key, "cache"); - const cacheData = JSON.parse( - (await Body?.transformToString()) ?? "{}", - ) as S3CachedFile; - const meta = cacheData.meta; - const lastModified = await this.getHasRevalidatedTags( + const { value: cacheData, lastModified } = + await globalThis.incrementalCache.get(key, false); + // const { Body, LastModified } = await this.getS3Object(key, "cache"); + // const cacheData = JSON.parse( + // (await Body?.transformToString()) ?? "{}", + // ) as S3CachedFile; + const meta = cacheData?.meta; + const _lastModified = await globalThis.tagCache.getLastModified( key, - LastModified?.getTime(), + lastModified, ); - if (lastModified === -1) { + if (_lastModified === -1) { // If some tags are stale we need to force revalidation return null; } - globalThis.lastModified = lastModified; - if (cacheData.type === "route") { + const requestId = globalThis.__als.getStore() ?? ""; + globalThis.lastModified[requestId] = _lastModified; + if (cacheData?.type === "route") { return { - lastModified: LastModified?.getTime(), + lastModified: _lastModified, value: { kind: "ROUTE", body: Buffer.from( @@ -237,9 +183,9 @@ export default class S3Cache { headers: meta?.headers, }, } as CacheHandlerValue; - } else if (cacheData.type === "page" || cacheData.type === "app") { + } else if (cacheData?.type === "page" || cacheData?.type === "app") { return { - lastModified: LastModified?.getTime(), + lastModified: _lastModified, value: { kind: "PAGE", html: cacheData.html, @@ -249,9 +195,9 @@ export default class S3Cache { headers: meta?.headers, }, } as CacheHandlerValue; - } else if (cacheData.type === "redirect") { + } else if (cacheData?.type === "redirect") { return { - lastModified: LastModified?.getTime(), + lastModified: _lastModified, value: { kind: "REDIRECT", props: cacheData.props, @@ -277,10 +223,9 @@ export default class S3Cache { } if (data?.kind === "ROUTE") { const { body, status, headers } = data; - this.putS3Object( + await globalThis.incrementalCache.set( key, - "cache", - JSON.stringify({ + { type: "route", body: body.toString( isBinaryContentType(String(headers["content-type"])) @@ -291,36 +236,46 @@ export default class S3Cache { status, headers, }, - } as S3CachedFile), + }, + false, ); } else if (data?.kind === "PAGE") { const { html, pageData } = data; const isAppPath = typeof pageData === "string"; - this.putS3Object( - key, - "cache", - JSON.stringify({ - type: isAppPath ? "app" : "page", - html, - rsc: isAppPath ? pageData : undefined, - json: isAppPath ? undefined : pageData, - meta: { status: data.status, headers: data.headers }, - } as S3CachedFile), - ); + if (isAppPath) { + globalThis.incrementalCache.set( + key, + { + type: "app", + html, + rsc: pageData, + }, + false, + ); + } else { + globalThis.incrementalCache.set( + key, + { + type: "page", + html, + json: pageData, + }, + false, + ); + } } else if (data?.kind === "FETCH") { - await this.putS3Object(key, "fetch", JSON.stringify(data)); + await globalThis.incrementalCache.set(key, data, true); } else if (data?.kind === "REDIRECT") { - // // delete potential page data if we're redirecting - await this.putS3Object( + await globalThis.incrementalCache.set( key, - "cache", - JSON.stringify({ + { type: "redirect", props: data.props, - } as S3CachedFile), + }, + false, ); } else if (data === null || data === undefined) { - await this.deleteS3Objects(key); + await globalThis.incrementalCache.delete(key); } // Write derivedTags to dynamodb // If we use an in house version of getDerivedTags in build we should use it here instead of next's one @@ -333,10 +288,10 @@ export default class S3Cache { debug("derivedTags", derivedTags); // Get all tags stored in dynamodb for the given key // If any of the derived tags are not stored in dynamodb for the given key, write them - const storedTags = await this.getTagsByPath(key); + const storedTags = await globalThis.tagCache.getByPath(key); const tagsToWrite = derivedTags.filter((tag) => !storedTags.includes(tag)); if (tagsToWrite.length > 0) { - await this.batchWriteDynamoItem( + await globalThis.tagCache.writeTags( tagsToWrite.map((tag) => ({ path: key, tag: tag, @@ -351,219 +306,14 @@ export default class S3Cache { } debug("revalidateTag", tag); // Find all keys with the given tag - const paths = await this.getByTag(tag); + const paths = await globalThis.tagCache.getByTag(tag); debug("Items", paths); // Update all keys with the given tag with revalidatedAt set to now - await this.batchWriteDynamoItem( + await globalThis.tagCache.writeTags( paths?.map((path) => ({ path: path, tag: tag, })) ?? [], ); } - - // DynamoDB handling - - private async getTagsByPath(path: string) { - try { - if (disableDynamoDBCache) return []; - const result = await this.dynamoClient.send( - new QueryCommand({ - TableName: CACHE_DYNAMO_TABLE, - IndexName: "revalidate", - KeyConditionExpression: "#key = :key", - ExpressionAttributeNames: { - "#key": "path", - }, - ExpressionAttributeValues: { - ":key": { S: this.buildDynamoKey(path) }, - }, - }), - ); - const tags = result.Items?.map((item) => item.tag.S ?? "") ?? []; - debug("tags for path", path, tags); - return tags; - } catch (e) { - error("Failed to get tags by path", e); - return []; - } - } - - //TODO: Figure out a better name for this function since it returns the lastModified - private async getHasRevalidatedTags(key: string, lastModified?: number) { - try { - if (disableDynamoDBCache) return lastModified ?? Date.now(); - const result = await this.dynamoClient.send( - new QueryCommand({ - TableName: CACHE_DYNAMO_TABLE, - IndexName: "revalidate", - KeyConditionExpression: - "#key = :key AND #revalidatedAt > :lastModified", - ExpressionAttributeNames: { - "#key": "path", - "#revalidatedAt": "revalidatedAt", - }, - ExpressionAttributeValues: { - ":key": { S: this.buildDynamoKey(key) }, - ":lastModified": { N: String(lastModified ?? 0) }, - }, - }), - ); - const revalidatedTags = result.Items ?? []; - debug("revalidatedTags", revalidatedTags); - // If we have revalidated tags we return -1 to force revalidation - return revalidatedTags.length > 0 ? -1 : lastModified ?? Date.now(); - } catch (e) { - error("Failed to get revalidated tags", e); - return lastModified ?? Date.now(); - } - } - - private async getByTag(tag: string) { - try { - if (disableDynamoDBCache) return []; - const { Items } = await this.dynamoClient.send( - new QueryCommand({ - TableName: CACHE_DYNAMO_TABLE, - KeyConditionExpression: "#tag = :tag", - ExpressionAttributeNames: { - "#tag": "tag", - }, - ExpressionAttributeValues: { - ":tag": { S: this.buildDynamoKey(tag) }, - }, - }), - ); - return ( - // We need to remove the buildId from the path - Items?.map( - ({ path: { S: key } }) => key?.replace(`${this.buildId}/`, "") ?? "", - ) ?? [] - ); - } catch (e) { - error("Failed to get by tag", e); - return []; - } - } - - private async batchWriteDynamoItem(req: { path: string; tag: string }[]) { - try { - if (disableDynamoDBCache) return; - await Promise.all( - chunk(req, MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT).map((Items) => { - return this.dynamoClient.send( - new BatchWriteItemCommand({ - RequestItems: { - [CACHE_DYNAMO_TABLE ?? ""]: Items.map((Item) => ({ - PutRequest: { - Item: { - ...this.buildDynamoObject(Item.path, Item.tag), - }, - }, - })), - }, - }), - ); - }), - ); - } catch (e) { - error("Failed to batch write dynamo item", e); - } - } - - private buildDynamoKey(key: string) { - // FIXME: We should probably use something else than path.join here - // this could transform some fetch cache key into a valid path - return path.posix.join(this.buildId, key); - } - - private buildDynamoObject(path: string, tags: string) { - return { - path: { S: this.buildDynamoKey(path) }, - tag: { S: this.buildDynamoKey(tags) }, - revalidatedAt: { N: `${Date.now()}` }, - }; - } - - // S3 handling - - private buildS3Key(key: string, extension: Extension) { - return path.posix.join( - CACHE_BUCKET_KEY_PREFIX ?? "", - extension === "fetch" ? "__fetch" : "", - this.buildId, - extension === "fetch" ? key : `${key}.${extension}`, - ); - } - - private buildS3KeyPrefix(key: string) { - return path.posix.join(CACHE_BUCKET_KEY_PREFIX ?? "", this.buildId, key); - } - - private async listS3Object(key: string) { - const { Contents } = await this.client.send( - new ListObjectsV2Command({ - Bucket: CACHE_BUCKET_NAME, - // add a point to the key so that it only matches the key and - // not other keys starting with the same string - Prefix: `${this.buildS3KeyPrefix(key)}.`, - }), - ); - return (Contents ?? []).map(({ Key }) => Key) as string[]; - } - - private async getS3Object(key: string, extension: Extension) { - try { - const result = await this.client.send( - new GetObjectCommand({ - Bucket: CACHE_BUCKET_NAME, - Key: this.buildS3Key(key, extension), - }), - ); - return result; - } catch (e) { - warn("This error can usually be ignored : ", e); - return { Body: null, LastModified: null }; - } - } - - private putS3Object( - key: string, - extension: Extension, - value: PutObjectCommandInput["Body"], - ) { - return this.client.send( - new PutObjectCommand({ - Bucket: CACHE_BUCKET_NAME, - Key: this.buildS3Key(key, extension), - Body: value, - }), - ); - } - - private async deleteS3Objects(key: string) { - try { - const s3Keys = (await this.listS3Object(key)).filter( - (key) => key && hasCacheExtension(key), - ); - - if (s3Keys.length === 0) { - warn( - `No s3 keys with a valid cache extension found for ${key}, see type CacheExtension in OpenNext for details`, - ); - return; - } - - await this.client.send( - new DeleteObjectsCommand({ - Bucket: CACHE_BUCKET_NAME, - Delete: { - Objects: s3Keys.map((Key) => ({ Key })), - }, - }), - ); - } catch (e) { - error("Failed to delete cache", e); - } - } } diff --git a/packages/open-next/src/adapters/config/index.ts b/packages/open-next/src/adapters/config/index.ts index c686074f..d968bb0a 100644 --- a/packages/open-next/src/adapters/config/index.ts +++ b/packages/open-next/src/adapters/config/index.ts @@ -7,8 +7,9 @@ import { loadConfig, loadConfigHeaders, loadHtmlPages, + loadMiddlewareManifest, loadPrerenderManifest, - loadPublicAssets, + // loadPublicAssets, loadRoutesManifest, } from "./util.js"; @@ -17,11 +18,16 @@ export const OPEN_NEXT_DIR = path.join(__dirname, ".open-next"); debug({ NEXT_DIR, OPEN_NEXT_DIR }); -export const NextConfig = loadConfig(NEXT_DIR); -export const BuildId = loadBuildId(NEXT_DIR); -export const HtmlPages = loadHtmlPages(NEXT_DIR); -export const PublicAssets = loadPublicAssets(OPEN_NEXT_DIR); -export const RoutesManifest = loadRoutesManifest(NEXT_DIR); -export const ConfigHeaders = loadConfigHeaders(NEXT_DIR); -export const PrerenderManifest = loadPrerenderManifest(NEXT_DIR); -export const AppPathsManifestKeys = loadAppPathsManifestKeys(NEXT_DIR); +//TODO: inject these values at build time +export const NextConfig = /* @__PURE__ */ loadConfig(NEXT_DIR); +export const BuildId = /* @__PURE__ */ loadBuildId(NEXT_DIR); +export const HtmlPages = /* @__PURE__ */ loadHtmlPages(NEXT_DIR); +// export const PublicAssets = loadPublicAssets(OPEN_NEXT_DIR); +export const RoutesManifest = /* @__PURE__ */ loadRoutesManifest(NEXT_DIR); +export const ConfigHeaders = /* @__PURE__ */ loadConfigHeaders(NEXT_DIR); +export const PrerenderManifest = + /* @__PURE__ */ loadPrerenderManifest(NEXT_DIR); +export const AppPathsManifestKeys = + /* @__PURE__ */ loadAppPathsManifestKeys(NEXT_DIR); +export const MiddlewareManifest = + /* @__PURE__ */ loadMiddlewareManifest(NEXT_DIR); diff --git a/packages/open-next/src/adapters/config/util.ts b/packages/open-next/src/adapters/config/util.ts index b0455ed5..aa11ee3b 100644 --- a/packages/open-next/src/adapters/config/util.ts +++ b/packages/open-next/src/adapters/config/util.ts @@ -1,12 +1,13 @@ import fs from "fs"; import path from "path"; - -import { PublicFiles } from "../../build"; import { + MiddlewareManifest, NextConfig, PrerenderManifest, RoutesManifest, -} from "../types/next-types"; +} from "types/next-types"; + +import { PublicFiles } from "../../build"; export function loadConfig(nextDir: string) { const filePath = path.join(nextDir, "required-server-files.json"); @@ -100,3 +101,9 @@ export function loadAppPathsManifestKeys(nextDir: string) { return cleanedKey === "" ? "/" : cleanedKey; }); } + +export function loadMiddlewareManifest(nextDir: string) { + const filePath = path.join(nextDir, "server", "middleware-manifest.json"); + const json = fs.readFileSync(filePath, "utf-8"); + return JSON.parse(json) as MiddlewareManifest; +} diff --git a/packages/open-next/src/adapters/dynamo-provider.ts b/packages/open-next/src/adapters/dynamo-provider.ts index d2e677c5..d4557254 100644 --- a/packages/open-next/src/adapters/dynamo-provider.ts +++ b/packages/open-next/src/adapters/dynamo-provider.ts @@ -1,20 +1,11 @@ -import { - BatchWriteItemCommand, - DynamoDBClient, -} from "@aws-sdk/client-dynamodb"; -import { CdkCustomResourceEvent, CdkCustomResourceResponse } from "aws-lambda"; import { readFileSync } from "fs"; -import { - getDynamoBatchWriteCommandConcurrency, - MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT, -} from "./constants.js"; -import { chunk } from "./util.js"; +import { createGenericHandler } from "../core/createGenericHandler.js"; +import { resolveTagCache } from "../core/resolve.js"; -const PHYSICAL_RESOURCE_ID = "dynamodb-cache"; - -const dynamoClient = new DynamoDBClient({}); +const PHYSICAL_RESOURCE_ID = "dynamodb-cache" as const; +//TODO: modify this, we should use the same format as the cache type DataType = { tag: { S: string; @@ -27,62 +18,61 @@ type DataType = { }; }; -export async function handler( - event: CdkCustomResourceEvent, -): Promise { - switch (event.RequestType) { - case "Create": - case "Update": - return insert(); - case "Delete": +export interface InitializationFunctionEvent { + type: "initializationFunction"; + requestType: "create" | "update" | "delete"; + resourceId: typeof PHYSICAL_RESOURCE_ID; +} + +const tagCache = await resolveTagCache( + globalThis.openNextConfig?.initializationFunction?.tagCache, +); + +export const handler = await createGenericHandler({ + handler: defaultHandler, + type: "initializationFunction", +}); + +async function defaultHandler( + event: InitializationFunctionEvent, +): Promise { + switch (event.requestType) { + case "delete": return remove(); + case "create": + case "update": + default: + return insert(event.requestType); } } -async function insert(): Promise { - const tableName = process.env.CACHE_DYNAMO_TABLE!; - +async function insert( + requestType: InitializationFunctionEvent["requestType"], +): Promise { const file = readFileSync(`dynamodb-cache.json`, "utf8"); const data: DataType[] = JSON.parse(file); - const dataChunks = chunk(data, MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT); - - const batchWriteParamsArray = dataChunks.map((chunk) => { - return { - RequestItems: { - [tableName]: chunk.map((item) => ({ - PutRequest: { - Item: item, - }, - })), - }, - }; - }); + const parsedData = data.map((item) => ({ + tag: item.tag.S, + path: item.path.S, + revalidatedAt: parseInt(item.revalidatedAt.N), + })); - const paramsChunks = chunk( - batchWriteParamsArray, - getDynamoBatchWriteCommandConcurrency(), - ); - - for (const paramsChunk of paramsChunks) { - await Promise.all( - paramsChunk.map((params) => - dynamoClient.send(new BatchWriteItemCommand(params)), - ), - ); - } + await tagCache.writeTags(parsedData); return { - PhysicalResourceId: PHYSICAL_RESOURCE_ID, - Data: {}, + type: "initializationFunction", + requestType, + resourceId: PHYSICAL_RESOURCE_ID, }; } -async function remove(): Promise { +async function remove(): Promise { // Do we want to actually delete anything here? return { - PhysicalResourceId: PHYSICAL_RESOURCE_ID, - Data: {}, + type: "initializationFunction", + requestType: "delete", + resourceId: PHYSICAL_RESOURCE_ID, }; } diff --git a/packages/open-next/src/adapters/edge-adapter.ts b/packages/open-next/src/adapters/edge-adapter.ts new file mode 100644 index 00000000..09470a8a --- /dev/null +++ b/packages/open-next/src/adapters/edge-adapter.ts @@ -0,0 +1,73 @@ +import { InternalEvent, InternalResult } from "types/open-next"; + +// We import it like that so that the edge plugin can replace it +import { NextConfig } from "../adapters/config"; +import { createGenericHandler } from "../core/createGenericHandler"; +import { + convertBodyToReadableStream, + convertToQueryString, +} from "../core/routing/util"; + +const defaultHandler = async ( + internalEvent: InternalEvent, +): Promise => { + // TODO: We need to handle splitted function here + // We should probably create an host resolver to redirect correctly + + const host = internalEvent.headers.host + ? `https://${internalEvent.headers.host}` + : "http://localhost:3000"; + const initialUrl = new URL(internalEvent.rawPath, host); + initialUrl.search = convertToQueryString(internalEvent.query); + const url = initialUrl.toString(); + + // @ts-expect-error - This is bundled + const handler = await import(`./middleware.mjs`); + + const response: Response = await handler.default({ + headers: internalEvent.headers, + method: internalEvent.method || "GET", + nextConfig: { + basePath: NextConfig.basePath, + i18n: NextConfig.i18n, + trailingSlash: NextConfig.trailingSlash, + }, + url, + body: convertBodyToReadableStream(internalEvent.method, internalEvent.body), + }); + + const arrayBuffer = await response.arrayBuffer(); + const buffer = Buffer.from(arrayBuffer); + + const responseHeaders: Record = {}; + response.headers.forEach((value, key) => { + if (key.toLowerCase() === "set-cookie") { + responseHeaders[key] = responseHeaders[key] + ? [...responseHeaders[key], value] + : [value]; + } else { + responseHeaders[key] = value; + } + }); + // console.log("responseHeaders", responseHeaders); + const body = buffer.toString(); + // console.log("body", body); + + return { + type: "core", + statusCode: response.status, + headers: responseHeaders, + body: body, + // Do we need to handle base64 encoded response? + isBase64Encoded: false, + }; +}; + +export const handler = await createGenericHandler({ + handler: defaultHandler, + type: "middleware", +}); + +export default { + fetch: handler, +}; diff --git a/packages/open-next/src/adapters/event-mapper.ts b/packages/open-next/src/adapters/event-mapper.ts deleted file mode 100644 index 63272bbf..00000000 --- a/packages/open-next/src/adapters/event-mapper.ts +++ /dev/null @@ -1,394 +0,0 @@ -import type { - APIGatewayProxyEvent, - APIGatewayProxyEventV2, - APIGatewayProxyResult, - APIGatewayProxyResultV2, - CloudFrontHeaders, - CloudFrontRequestEvent, - CloudFrontRequestResult, -} from "aws-lambda"; - -import { debug } from "./logger.js"; -import { convertToQuery } from "./routing/util.js"; -import { parseCookies } from "./util.js"; - -export type InternalEvent = { - readonly type: "v1" | "v2" | "cf"; - readonly method: string; - readonly rawPath: string; - readonly url: string; - readonly body: Buffer; - readonly headers: Record; - readonly query: Record; - readonly cookies: Record; - readonly remoteAddress: string; -}; - -export type InternalResult = { - readonly type: "v1" | "v2" | "cf"; - statusCode: number; - headers: Record; - body: string; - isBase64Encoded: boolean; -}; - -export function isAPIGatewayProxyEventV2( - event: any, -): event is APIGatewayProxyEventV2 { - return event.version === "2.0"; -} - -export function isAPIGatewayProxyEvent( - event: any, -): event is APIGatewayProxyEvent { - return event.version === undefined && !isCloudFrontRequestEvent(event); -} - -export function isCloudFrontRequestEvent( - event: any, -): event is CloudFrontRequestEvent { - return event.Records !== undefined; -} - -export function convertFrom( - event: APIGatewayProxyEventV2 | APIGatewayProxyEvent | CloudFrontRequestEvent, -): InternalEvent { - let internalEvent: InternalEvent; - if (isCloudFrontRequestEvent(event)) { - internalEvent = convertFromCloudFrontRequestEvent(event); - } else if (isAPIGatewayProxyEventV2(event)) { - internalEvent = convertFromAPIGatewayProxyEventV2(event); - } else if (isAPIGatewayProxyEvent(event)) { - internalEvent = convertFromAPIGatewayProxyEvent(event); - } else throw new Error("Unsupported event type"); - - return internalEvent; -} - -export function convertTo( - result: InternalResult, -): APIGatewayProxyResultV2 | APIGatewayProxyResult | CloudFrontRequestResult { - if (result.type === "v2") { - return convertToApiGatewayProxyResultV2(result); - } else if (result.type === "v1") { - return convertToApiGatewayProxyResult(result); - } else if (result.type === "cf") { - return convertToCloudFrontRequestResult(result); - } - throw new Error("Unsupported event type"); -} - -function removeUndefinedFromQuery( - query: Record, -) { - const newQuery: Record = {}; - for (const [key, value] of Object.entries(query)) { - if (value !== undefined) { - newQuery[key] = value; - } - } - return newQuery; -} -function convertFromAPIGatewayProxyEvent( - event: APIGatewayProxyEvent, -): InternalEvent { - const { path, body, httpMethod, requestContext, isBase64Encoded } = event; - return { - type: "v1", - method: httpMethod, - rawPath: path, - url: path + normalizeAPIGatewayProxyEventQueryParams(event), - body: Buffer.from(body ?? "", isBase64Encoded ? "base64" : "utf8"), - headers: normalizeAPIGatewayProxyEventHeaders(event), - remoteAddress: requestContext.identity.sourceIp, - query: removeUndefinedFromQuery( - event.multiValueQueryStringParameters ?? {}, - ), - cookies: - event.multiValueHeaders?.cookie?.reduce((acc, cur) => { - const [key, value] = cur.split("="); - return { ...acc, [key]: value }; - }, {}) ?? {}, - }; -} - -function convertFromAPIGatewayProxyEventV2( - event: APIGatewayProxyEventV2, -): InternalEvent { - const { rawPath, rawQueryString, requestContext } = event; - return { - type: "v2", - method: requestContext.http.method, - rawPath, - url: rawPath + (rawQueryString ? `?${rawQueryString}` : ""), - body: normalizeAPIGatewayProxyEventV2Body(event), - headers: normalizeAPIGatewayProxyEventV2Headers(event), - remoteAddress: requestContext.http.sourceIp, - query: removeUndefinedFromQuery(convertToQuery(rawQueryString)), - cookies: - event.cookies?.reduce((acc, cur) => { - const [key, value] = cur.split("="); - return { ...acc, [key]: value }; - }, {}) ?? {}, - }; -} - -function convertFromCloudFrontRequestEvent( - event: CloudFrontRequestEvent, -): InternalEvent { - const { method, uri, querystring, body, headers, clientIp } = - event.Records[0].cf.request; - return { - type: "cf", - method, - rawPath: uri, - url: uri + (querystring ? `?${querystring}` : ""), - body: Buffer.from( - body?.data ?? "", - body?.encoding === "base64" ? "base64" : "utf8", - ), - headers: normalizeCloudFrontRequestEventHeaders(headers), - remoteAddress: clientIp, - query: convertToQuery(querystring), - cookies: - headers.cookie?.reduce((acc, cur) => { - const { key, value } = cur; - return { ...acc, [key ?? ""]: value }; - }, {}) ?? {}, - }; -} - -function convertToApiGatewayProxyResult( - result: InternalResult, -): APIGatewayProxyResult { - const headers: Record = {}; - const multiValueHeaders: Record = {}; - Object.entries(result.headers).forEach(([key, value]) => { - if (Array.isArray(value)) { - multiValueHeaders[key] = value; - } else { - if (value === null) { - headers[key] = ""; - return; - } - headers[key] = value; - } - }); - - const response: APIGatewayProxyResult = { - statusCode: result.statusCode, - headers, - body: result.body, - isBase64Encoded: result.isBase64Encoded, - multiValueHeaders, - }; - debug(response); - return response; -} - -function convertToApiGatewayProxyResultV2( - result: InternalResult, -): APIGatewayProxyResultV2 { - const headers: Record = {}; - Object.entries(result.headers) - .filter(([key]) => key.toLowerCase() !== "set-cookie") - .forEach(([key, value]) => { - if (value === null) { - headers[key] = ""; - return; - } - headers[key] = Array.isArray(value) ? value.join(", ") : value.toString(); - }); - - const response: APIGatewayProxyResultV2 = { - statusCode: result.statusCode, - headers, - cookies: parseCookies(result.headers["set-cookie"]), - body: result.body, - isBase64Encoded: result.isBase64Encoded, - }; - debug(response); - return response; -} - -const CloudFrontBlacklistedHeaders = [ - // Disallowed headers, see: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-function-restrictions-all.html#function-restrictions-disallowed-headers - "connection", - "expect", - "keep-alive", - "proxy-authenticate", - "proxy-authorization", - "proxy-connection", - "trailer", - "upgrade", - "x-accel-buffering", - "x-accel-charset", - "x-accel-limit-rate", - "x-accel-redirect", - /x-amz-cf-(.*)/, - "x-amzn-auth", - "x-amzn-cf-billing", - "x-amzn-cf-id", - "x-amzn-cf-xff", - "x-amzn-errortype", - "x-amzn-fle-profile", - "x-amzn-header-count", - "x-amzn-header-order", - "x-amzn-lambda-integration-tag", - "x-amzn-requestid", - /x-edge-(.*)/, - "x-cache", - "x-forwarded-proto", - "x-real-ip", - - // Read-only headers, see: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-function-restrictions-all.html#function-restrictions-read-only-headers - "accept-encoding", - "content-length", - "if-modified-since", - "if-none-match", - "if-range", - "if-unmodified-since", - "transfer-encoding", - "via", -]; - -function convertToCloudFrontRequestResult( - result: InternalResult, -): CloudFrontRequestResult { - debug("result headers", result.headers); - - const headers: CloudFrontHeaders = {}; - Object.entries(result.headers) - .filter( - ([key]) => - !CloudFrontBlacklistedHeaders.some((header) => - typeof header === "string" - ? header === key.toLowerCase() - : header.test(key.toLowerCase()), - ), - ) - .forEach(([key, value]) => { - if (key === "set-cookie") { - const cookies = parseCookies(value); - if (cookies) { - headers[key] = cookies.map((cookie) => ({ - key, - value: cookie, - })); - } - return; - } - - headers[key] = [ - ...(headers[key] || []), - ...(Array.isArray(value) - ? value.map((v) => ({ key, value: v })) - : [{ key, value: value.toString() }]), - ]; - }); - - const response: CloudFrontRequestResult = { - status: result.statusCode.toString(), - statusDescription: "OK", - headers, - bodyEncoding: result.isBase64Encoded ? "base64" : "text", - body: result.body, - }; - debug(response); - return response; -} - -function normalizeAPIGatewayProxyEventV2Headers( - event: APIGatewayProxyEventV2, -): Record { - const { headers: rawHeaders, cookies } = event; - - const headers: Record = {}; - - if (Array.isArray(cookies)) { - headers["cookie"] = cookies.join("; "); - } - - for (const [key, value] of Object.entries(rawHeaders || {})) { - headers[key.toLowerCase()] = value!; - } - - return headers; -} - -function normalizeAPIGatewayProxyEventV2Body( - event: APIGatewayProxyEventV2, -): Buffer { - const { body, isBase64Encoded } = event; - if (Buffer.isBuffer(body)) { - return body; - } else if (typeof body === "string") { - return Buffer.from(body, isBase64Encoded ? "base64" : "utf8"); - } else if (typeof body === "object") { - return Buffer.from(JSON.stringify(body)); - } - return Buffer.from("", "utf8"); -} - -function normalizeAPIGatewayProxyEventQueryParams( - event: APIGatewayProxyEvent, -): string { - // Note that the same query string values are returned in both - // "multiValueQueryStringParameters" and "queryStringParameters". - // We only need to use one of them. - // For example: - // "?name=foo" appears in the event object as - // { - // ... - // queryStringParameters: { name: 'foo' }, - // multiValueQueryStringParameters: { name: [ 'foo' ] }, - // ... - // } - const params = new URLSearchParams(); - for (const [key, value] of Object.entries( - event.multiValueQueryStringParameters || {}, - )) { - if (value !== undefined) { - for (const v of value) { - params.append(key, v); - } - } - } - const value = params.toString(); - return value ? `?${value}` : ""; -} - -function normalizeAPIGatewayProxyEventHeaders( - event: APIGatewayProxyEvent, -): Record { - event.multiValueHeaders; - const headers: Record = {}; - - for (const [key, values] of Object.entries(event.multiValueHeaders || {})) { - if (values) { - headers[key.toLowerCase()] = values.join(","); - } - } - for (const [key, value] of Object.entries(event.headers || {})) { - if (value) { - headers[key.toLowerCase()] = value; - } - } - return headers; -} - -function normalizeCloudFrontRequestEventHeaders( - rawHeaders: CloudFrontHeaders, -): Record { - const headers: Record = {}; - - for (const [key, values] of Object.entries(rawHeaders)) { - for (const { value } of values) { - if (value) { - headers[key.toLowerCase()] = value; - } - } - } - - return headers; -} diff --git a/packages/open-next/src/adapters/http/index.ts b/packages/open-next/src/adapters/http/index.ts deleted file mode 100644 index 82e9e941..00000000 --- a/packages/open-next/src/adapters/http/index.ts +++ /dev/null @@ -1,3 +0,0 @@ -export * from "./request.js"; -export * from "./response.js"; -export * from "./responseStreaming.js"; diff --git a/packages/open-next/src/adapters/http/response.ts b/packages/open-next/src/adapters/http/response.ts deleted file mode 100644 index 9731f55d..00000000 --- a/packages/open-next/src/adapters/http/response.ts +++ /dev/null @@ -1,150 +0,0 @@ -// Copied and modified from serverless-http by Doug Moscrop -// https://github.com/dougmoscrop/serverless-http/blob/master/lib/response.js -// Licensed under the MIT License - -import http from "node:http"; -import { Socket } from "node:net"; - -import { - convertHeader, - getString, - headerEnd, - NO_OP, - parseHeaders, -} from "./util.js"; - -const BODY = Symbol(); -const HEADERS = Symbol(); - -function addData(stream: ServerlessResponse, data: Uint8Array | string) { - if ( - Buffer.isBuffer(data) || - ArrayBuffer.isView(data) || - typeof data === "string" - ) { - stream[BODY].push(Buffer.from(data)); - } else { - throw new Error(`response.addData() of unexpected type: ${typeof data}`); - } -} - -export interface ServerlessResponseProps { - method: string; - headers: Record; -} - -export class ServerlessResponse extends http.ServerResponse { - [BODY]: Buffer[]; - [HEADERS]: Record; - private _wroteHeader = false; - private _header = ""; - private _initialHeaders: Record = {}; - - constructor({ method, headers }: ServerlessResponseProps) { - super({ method, headers } as any); - - this[BODY] = []; - this[HEADERS] = parseHeaders(headers) || {}; - this._initialHeaders = this[HEADERS]; - - this.useChunkedEncodingByDefault = false; - this.chunkedEncoding = false; - this._header = ""; - - const socket: Partial & { _writableState: any } = { - _writableState: {}, - writable: true, - on: NO_OP, - removeListener: NO_OP, - destroy: NO_OP, - cork: NO_OP, - uncork: NO_OP, - write: ( - data: Uint8Array | string, - encoding?: string | null | (() => void), - cb?: () => void, - ) => { - if (typeof encoding === "function") { - cb = encoding; - encoding = null; - } - - if (this._header === "" || this._wroteHeader) { - addData(this, data); - } else { - const string = getString(data); - const index = string.indexOf(headerEnd); - - if (index !== -1) { - const remainder = string.slice(index + headerEnd.length); - - if (remainder) { - addData(this, remainder); - } - - this._wroteHeader = true; - } - } - - if (typeof cb === "function") { - cb(); - } - return true; - }, - }; - - this.assignSocket(socket as Socket); - - this.once("finish", () => { - this.emit("close"); - }); - } - - static body(res: ServerlessResponse) { - return Buffer.concat(res[BODY]); - } - - static headers(res: ServerlessResponse) { - const headers = - typeof res.getHeaders === "function" ? res.getHeaders() : res[HEADERS]; - - return { - ...parseHeaders(headers), - ...res[HEADERS], - ...res._initialHeaders, - }; - } - - get headers() { - return this[HEADERS]; - } - - setHeader(key: string, value: string | number | string[]): this { - if (this._wroteHeader) { - this[HEADERS][key] = convertHeader(value); - } else { - super.setHeader(key, value); - } - return this; - } - - writeHead( - statusCode: number, - reason?: string | any | any[], - obj?: any | any[], - ) { - const headers = typeof reason === "string" ? obj : reason; - - for (const name in headers) { - this.setHeader(name, headers[name]); - - if (!this._wroteHeader) { - // we only need to initiate super.headers once - // writeHead will add the other headers itself - break; - } - } - - return super.writeHead(statusCode, reason, obj); - } -} diff --git a/packages/open-next/src/adapters/http/responseStreaming.ts b/packages/open-next/src/adapters/http/responseStreaming.ts deleted file mode 100644 index 233b2758..00000000 --- a/packages/open-next/src/adapters/http/responseStreaming.ts +++ /dev/null @@ -1,268 +0,0 @@ -import http from "node:http"; -import { Socket } from "node:net"; -import zlib from "node:zlib"; - -import { debug, error } from "../logger.js"; -import type { ResponseStream } from "../types/aws-lambda.js"; -import { parseCookies } from "../util.js"; -import { convertHeader, getString, NO_OP, parseHeaders } from "./util.js"; - -const HEADERS = Symbol(); - -export interface StreamingServerResponseProps { - method?: string; - headers?: Record; - responseStream: ResponseStream; - fixHeaders: (headers: Record) => void; - onEnd: (headers: Record) => Promise; -} -export class StreamingServerResponse extends http.ServerResponse { - [HEADERS]: Record = {}; - responseStream: ResponseStream; - fixHeaders: (headers: Record) => void; - onEnd: (headers: Record) => Promise; - private _wroteHeader = false; - private _hasWritten = false; - private _initialHeaders: Record = {}; - private _cookies: string[] = []; - private _compressed = false; - - constructor({ - method, - headers, - responseStream, - fixHeaders, - onEnd, - }: StreamingServerResponseProps) { - super({ method } as any); - if (headers && headers["set-cookie"]) { - this._cookies = parseCookies(headers["set-cookie"]) as string[]; - delete headers["set-cookie"]; - } - this[HEADERS] = parseHeaders(headers) || {}; - this._initialHeaders = { ...this[HEADERS] }; - - this.fixHeaders = fixHeaders; - this.onEnd = onEnd; - this.responseStream = responseStream; - - this.useChunkedEncodingByDefault = false; - this.chunkedEncoding = false; - - this.responseStream.cork(); - - const socket: Partial & { _writableState: any } = { - _writableState: {}, - writable: true, - on: NO_OP, - removeListener: NO_OP, - destroy: NO_OP, - cork: NO_OP, - uncork: NO_OP, - write: ( - data: Uint8Array | string, - encoding?: string | null | (() => void), - cb?: () => void, - ) => { - if (typeof encoding === "function") { - cb = encoding; - encoding = undefined; - } - const d = getString(data); - const isSse = d.endsWith("\n\n"); - this.internalWrite(data, isSse, cb); - - return !this.responseStream.writableNeedDrain; - }, - }; - - this.assignSocket(socket as Socket); - - this.responseStream.on("close", this.cancel.bind(this)); - this.responseStream.on("error", this.cancel.bind(this)); - - this.on("close", this.cancel.bind(this)); - this.on("error", this.cancel.bind(this)); - this.once("finish", () => { - this.emit("close"); - }); - } - - get headers() { - return this[HEADERS]; - } - - setHeader(key: string, value: string | number | string[]): this { - key = key.toLowerCase(); - // There can be multiple set-cookie response headers - // They need to be returned as a special "cookies" array, eg: - // {statusCode: xxx, cookies: ['Cookie=Yum'], ...} - if (key === "set-cookie") { - this._cookies.push(convertHeader(value)); - } else { - this[HEADERS][key] = convertHeader(value); - } - return this; - } - - removeHeader(key: string): this { - key = key.toLowerCase(); - if (key === "set-cookie") { - this._cookies.length = 0; - } else { - delete this[HEADERS][key]; - } - return this; - } - - writeHead( - statusCode: number, - _statusMessage?: - | string - | http.OutgoingHttpHeaders - | http.OutgoingHttpHeader[], - _headers?: http.OutgoingHttpHeaders | http.OutgoingHttpHeader[], - ): this { - const headers = - typeof _statusMessage === "string" ? _headers : _statusMessage; - const statusMessage = - typeof _statusMessage === "string" ? _statusMessage : undefined; - if (this._wroteHeader) { - return this; - } - try { - debug("writeHead", statusCode, statusMessage, headers); - const parsedHeaders = parseHeaders(headers); - this[HEADERS] = { - ...this[HEADERS], - ...parsedHeaders, - }; - - this.fixHeaders(this[HEADERS]); - this[HEADERS] = { - ...this[HEADERS], - ...this._initialHeaders, - }; - - this._compressed = this[HEADERS]["accept-encoding"]?.includes("br"); - if (this._compressed) { - this[HEADERS]["content-encoding"] = "br"; - } - delete this[HEADERS]["accept-encoding"]; - - debug("writeHead", this[HEADERS]); - - this._wroteHeader = true; - // FIXME: This is extracted from the docker lambda node 18 runtime - // https://gist.github.com/conico974/13afd708af20711b97df439b910ceb53#file-index-mjs-L921-L932 - // We replace their write with ours which are inside a setImmediate - // This way it seems to work all the time - // I think we can't ship this code as it is, it could break at anytime if they decide to change the runtime and they already did it in the past - this.responseStream.setContentType( - "application/vnd.awslambda.http-integration-response", - ); - const prelude = JSON.stringify({ - statusCode: statusCode as number, - cookies: this._cookies, - headers: this[HEADERS], - }); - - // Try to flush the buffer to the client to invoke - // the streaming. This does not work 100% of the time. - setImmediate(() => { - this.responseStream.write("\n\n"); - this.responseStream.uncork(); - }); - setImmediate(() => { - this.responseStream.write(prelude); - }); - - setImmediate(() => { - this.responseStream.write(new Uint8Array(8)); - - // After headers are written, compress all writes - // using Brotli - if (this._compressed) { - const br = zlib.createBrotliCompress({ - flush: zlib.constants.BROTLI_OPERATION_FLUSH, - }); - br.setMaxListeners(100); - br.pipe(this.responseStream); - this.responseStream = br as unknown as ResponseStream; - } - }); - - debug("writeHead", this[HEADERS]); - } catch (e) { - this.responseStream.end(); - error(e); - } - - return this; - } - - end( - _chunk?: Uint8Array | string | (() => void), - _encoding?: BufferEncoding | (() => void), - _cb?: (() => void) | undefined, - ): this { - const chunk = typeof _chunk === "function" ? undefined : _chunk; - const cb = typeof _cb === "function" ? _cb : undefined; - - if (!this._wroteHeader) { - // When next directly returns with end, the writeHead is not called, - // so we need to call it here - this.writeHead(this.statusCode ?? 200); - } - - if (!this._hasWritten && !chunk) { - // We need to send data here if there is none, otherwise the stream will not end at all - this.internalWrite(new Uint8Array(8), false, cb); - } - - const _end = () => { - setImmediate(() => { - this.responseStream.end(_chunk, async () => { - if (this._compressed) { - (this.responseStream as unknown as zlib.BrotliCompress).flush( - zlib.constants.BROTLI_OPERATION_FINISH, - ); - } - await this.onEnd(this[HEADERS]); - cb?.(); - }); - }); - }; - - if (this.responseStream.writableNeedDrain) { - this.responseStream.once("drain", _end); - } else { - _end(); - } - return this; - } - - private internalWrite(chunk: any, isSse: boolean = false, cb?: () => void) { - this._hasWritten = true; - setImmediate(() => { - this.responseStream.write(chunk, cb); - - // SSE need to flush to send to client ASAP - if (isSse) { - setImmediate(() => { - this.responseStream.write("\n\n"); - this.responseStream.uncork(); - }); - } - }); - } - - cancel(error?: Error) { - this.responseStream.off("close", this.cancel.bind(this)); - this.responseStream.off("error", this.cancel.bind(this)); - - if (error) { - this.responseStream.destroy(error); - } - } -} diff --git a/packages/open-next/src/adapters/http/util.ts b/packages/open-next/src/adapters/http/util.ts deleted file mode 100644 index 132f6798..00000000 --- a/packages/open-next/src/adapters/http/util.ts +++ /dev/null @@ -1,50 +0,0 @@ -import http from "node:http"; - -export function getString(data: any) { - // Note: use `ArrayBuffer.isView()` to check for Uint8Array. Using - // `instanceof Uint8Array` returns false in some cases. For example, - // when the buffer is created in middleware and passed to NextServer. - if (Buffer.isBuffer(data)) { - return data.toString("utf8"); - } else if (ArrayBuffer.isView(data)) { - //@ts-ignore - return Buffer.from(data).toString("utf8"); - } else if (typeof data === "string") { - return data; - } else { - throw new Error(`response.getString() of unexpected type: ${typeof data}`); - } -} - -export const headerEnd = "\r\n\r\n"; - -export const NO_OP: (...args: any[]) => any = () => void 0; - -export const parseHeaders = ( - headers?: http.OutgoingHttpHeader[] | http.OutgoingHttpHeaders, -) => { - const result: Record = {}; - if (!headers) { - return result; - } - - for (const [key, value] of Object.entries(headers)) { - if (value === undefined) { - continue; - } else { - result[key] = convertHeader(value); - } - } - - return result; -}; - -export const convertHeader = (header: http.OutgoingHttpHeader) => { - if (typeof header === "string") { - return header; - } else if (Array.isArray(header)) { - return header.join(","); - } else { - return String(header); - } -}; diff --git a/packages/open-next/src/adapters/image-optimization-adapter.ts b/packages/open-next/src/adapters/image-optimization-adapter.ts index 1d4d3584..e4f5974e 100644 --- a/packages/open-next/src/adapters/image-optimization-adapter.ts +++ b/packages/open-next/src/adapters/image-optimization-adapter.ts @@ -1,17 +1,16 @@ import { createHash } from "node:crypto"; -import { IncomingMessage, ServerResponse } from "node:http"; +import { + IncomingMessage, + OutgoingHttpHeaders, + ServerResponse, +} from "node:http"; import https from "node:https"; import path from "node:path"; import { Writable } from "node:stream"; import { GetObjectCommand, S3Client } from "@aws-sdk/client-s3"; -import type { - APIGatewayProxyEvent, - APIGatewayProxyEventHeaders, - APIGatewayProxyEventQueryStringParameters, - APIGatewayProxyEventV2, - APIGatewayProxyResultV2, -} from "aws-lambda"; +import { loadBuildId, loadConfig } from "config/util.js"; +import { OpenNextNodeResponse, StreamCreator } from "http/openNextResponse.js"; // @ts-ignore import { defaultConfig } from "next/dist/server/config-shared"; import { @@ -20,17 +19,16 @@ import { } from "next/dist/server/image-optimizer"; // @ts-ignore import type { NextUrlWithParsedQuery } from "next/dist/server/request-meta"; +import { ImageLoader, InternalEvent, InternalResult } from "types/open-next.js"; -import { loadBuildId, loadConfig } from "./config/util.js"; +import { createGenericHandler } from "../core/createGenericHandler.js"; import { awsLogger, debug, error } from "./logger.js"; -import { optimizeImage } from "./plugins/image-optimization.js"; +import { optimizeImage } from "./plugins/image-optimization/image-optimization.js"; import { setNodeEnv } from "./util.js"; // Expected environment variables const { BUCKET_NAME, BUCKET_KEY_PREFIX } = process.env; -const s3Client = new S3Client({ logger: awsLogger }); - setNodeEnv(); const nextDir = path.join(__dirname, ".next"); const config = loadConfig(nextDir); @@ -52,16 +50,22 @@ debug("Init config", { // Handler // ///////////// -export async function handler( - event: APIGatewayProxyEventV2 | APIGatewayProxyEvent, -): Promise { +export const handler = await createGenericHandler({ + handler: defaultHandler, + type: "imageOptimization", +}); + +export async function defaultHandler( + event: InternalEvent, + streamCreator?: StreamCreator, +): Promise { // Images are handled via header and query param information. debug("handler event", event); - const { headers: rawHeaders, queryStringParameters: queryString } = event; + const { headers, query: queryString } = event; try { - const headers = normalizeHeaderKeysToLowercase(rawHeaders); - ensureBucketExists(); + // const headers = normalizeHeaderKeysToLowercase(rawHeaders); + const imageParams = validateImageParams( headers, queryString === null ? undefined : queryString, @@ -74,6 +78,10 @@ export async function handler( if (etag && headers["if-none-match"] === etag) { return { statusCode: 304, + headers: {}, + body: "", + isBase64Encoded: false, + type: "core", }; } const result = await optimizeImage( @@ -83,9 +91,9 @@ export async function handler( downloadHandler, ); - return buildSuccessResponse(result, etag); + return buildSuccessResponse(result, streamCreator, etag); } catch (e: any) { - return buildFailureResponse(e); + return buildFailureResponse(e, streamCreator); } } @@ -93,13 +101,13 @@ export async function handler( // Helper functions // ////////////////////// -function normalizeHeaderKeysToLowercase(headers: APIGatewayProxyEventHeaders) { - // Make header keys lowercase to ensure integrity - return Object.entries(headers).reduce( - (acc, [key, value]) => ({ ...acc, [key.toLowerCase()]: value }), - {} as APIGatewayProxyEventHeaders, - ); -} +// function normalizeHeaderKeysToLowercase(headers: APIGatewayProxyEventHeaders) { +// // Make header keys lowercase to ensure integrity +// return Object.entries(headers).reduce( +// (acc, [key, value]) => ({ ...acc, [key.toLowerCase()]: value }), +// {} as APIGatewayProxyEventHeaders, +// ); +// } function ensureBucketExists() { if (!BUCKET_NAME) { @@ -108,15 +116,15 @@ function ensureBucketExists() { } function validateImageParams( - headers: APIGatewayProxyEventHeaders, - queryString?: APIGatewayProxyEventQueryStringParameters, + headers: OutgoingHttpHeaders, + query?: InternalEvent["query"], ) { // Next.js checks if external image URL matches the // `images.remotePatterns` const imageParams = ImageOptimizerCache.validateParams( // @ts-ignore { headers }, - queryString, + query, nextConfig, false, ); @@ -144,17 +152,33 @@ function computeEtag(imageParams: { .digest("base64"); } -function buildSuccessResponse(result: any, etag?: string) { +function buildSuccessResponse( + result: any, + streamCreator?: StreamCreator, + etag?: string, +): InternalResult { const headers: Record = { Vary: "Accept", "Content-Type": result.contentType, "Cache-Control": `public,max-age=${result.maxAge},immutable`, }; + debug("result", result); if (etag) { headers["ETag"] = etag; } + if (streamCreator) { + const response = new OpenNextNodeResponse( + () => void 0, + async () => void 0, + streamCreator, + ); + response.writeHead(200, headers); + response.end(result.buffer); + } + return { + type: "core", statusCode: 200, body: result.buffer.toString("base64"), isBase64Encoded: true, @@ -162,9 +186,27 @@ function buildSuccessResponse(result: any, etag?: string) { }; } -function buildFailureResponse(e: any) { +function buildFailureResponse( + e: any, + streamCreator?: StreamCreator, +): InternalResult { debug(e); + if (streamCreator) { + const response = new OpenNextNodeResponse( + () => void 0, + async () => void 0, + streamCreator, + ); + response.writeHead(500, { + Vary: "Accept", + "Cache-Control": `public,max-age=60,immutable`, + "Content-Type": "application/json", + }); + response.end(e?.message || e?.toString() || e); + } return { + type: "core", + isBase64Encoded: false, statusCode: 500, headers: { Vary: "Accept", @@ -176,6 +218,37 @@ function buildFailureResponse(e: any) { }; } +const resolveLoader = () => { + const openNextParams = globalThis.openNextConfig.imageOptimization; + if (typeof openNextParams?.loader === "function") { + return openNextParams.loader(); + } else { + const s3Client = new S3Client({ logger: awsLogger }); + return Promise.resolve({ + name: "s3", + // @ts-ignore + load: async (key: string) => { + ensureBucketExists(); + const keyPrefix = BUCKET_KEY_PREFIX?.replace(/^\/|\/$/g, ""); + const response = await s3Client.send( + new GetObjectCommand({ + Bucket: BUCKET_NAME, + Key: keyPrefix + ? keyPrefix + "/" + key.replace(/^\//, "") + : key.replace(/^\//, ""), + }), + ); + return { + body: response.Body, + contentType: response.ContentType, + cacheControl: response.CacheControl, + }; + }, + }); + } +}; +const loader = await resolveLoader(); + async function downloadHandler( _req: IncomingMessage, res: ServerResponse, @@ -208,30 +281,23 @@ async function downloadHandler( else { // Download image from S3 // note: S3 expects keys without leading `/` - const keyPrefix = BUCKET_KEY_PREFIX?.replace(/^\/|\/$/g, ""); - const response = await s3Client.send( - new GetObjectCommand({ - Bucket: BUCKET_NAME, - Key: keyPrefix - ? keyPrefix + "/" + url.href.replace(/^\//, "") - : url.href.replace(/^\//, ""), - }), - ); - - if (!response.Body) { + + const response = await loader.load(url.href); + + if (!response.body) { throw new Error("Empty response body from the S3 request."); } // @ts-ignore - pipeRes(response.Body, res); + pipeRes(response.body, res); // Respect the bucket file's content-type and cache-control // imageOptimizer will use this to set the results.maxAge - if (response.ContentType) { - res.setHeader("Content-Type", response.ContentType); + if (response.contentType) { + res.setHeader("Content-Type", response.contentType); } - if (response.CacheControl) { - res.setHeader("Cache-Control", response.CacheControl); + if (response.cacheControl) { + res.setHeader("Cache-Control", response.cacheControl); } } } catch (e: any) { diff --git a/packages/open-next/src/adapters/middleware.ts b/packages/open-next/src/adapters/middleware.ts new file mode 100644 index 00000000..8f6ca05b --- /dev/null +++ b/packages/open-next/src/adapters/middleware.ts @@ -0,0 +1,82 @@ +import { InternalEvent, Origin, OriginResolver } from "types/open-next"; + +import { debug, error } from "../adapters/logger"; +import { createGenericHandler } from "../core/createGenericHandler"; +import routingHandler from "../core/routingHandler"; + +const resolveOriginResolver = () => { + const openNextParams = globalThis.openNextConfig.middleware; + if (typeof openNextParams?.originResolver === "function") { + return openNextParams.originResolver(); + } else { + return Promise.resolve({ + name: "env", + resolve: async (_path: string) => { + try { + const origin = JSON.parse( + process.env.OPEN_NEXT_ORIGIN ?? "{}", + ) as Record; + for (const [key, value] of Object.entries( + globalThis.openNextConfig.functions ?? {}, + ).filter(([key]) => key !== "default")) { + if ( + value.patterns.some((pattern) => { + // Convert cloudfront pattern to regex + return new RegExp( + // transform glob pattern to regex + "/" + + pattern + .replace(/\*\*/g, "(.*)") + .replace(/\*/g, "([^/]*)") + .replace(/\//g, "\\/") + .replace(/\?/g, "."), + ).test(_path); + }) + ) { + debug("Using origin", key, value.patterns); + return origin[key]; + } + } + if (origin["default"]) { + debug("Using default origin", origin["default"]); + return origin["default"]; + } + return false as const; + } catch (e) { + error("Error while resolving origin", e); + return false as const; + } + }, + }); + } +}; + +const defaultHandler = async (internalEvent: InternalEvent) => { + const originResolver = await resolveOriginResolver(); + const result = await routingHandler(internalEvent); + if ("internalEvent" in result) { + debug("Middleware intercepted event", internalEvent); + let origin: Origin | false = false; + if (!result.isExternalRewrite) { + origin = await originResolver.resolve(result.internalEvent.rawPath); + } + return { + type: "middleware", + internalEvent: result.internalEvent, + isExternalRewrite: result.isExternalRewrite, + origin, + }; + } else { + debug("Middleware response", result); + return result; + } +}; + +export const handler = await createGenericHandler({ + handler: defaultHandler, + type: "middleware", +}); + +export default { + fetch: handler, +}; diff --git a/packages/open-next/src/adapters/plugins/13.5/serverHandler.ts b/packages/open-next/src/adapters/plugins/13.5/serverHandler.ts deleted file mode 100644 index dc7f9864..00000000 --- a/packages/open-next/src/adapters/plugins/13.5/serverHandler.ts +++ /dev/null @@ -1,25 +0,0 @@ -/*eslint-disable simple-import-sort/imports */ -import type { Options, PluginHandler } from "../../types/next-types.js"; -import type { IncomingMessage } from "../../http/request.js"; -import type { ServerlessResponse } from "../../http/response.js"; -//#override imports -//@ts-ignore -import { requestHandler } from "./util.js"; -//@ts-ignore -import { proxyRequest } from "./routing/util.js"; -//#endOverride - -//#override handler -export const handler: PluginHandler = async ( - req: IncomingMessage, - res: ServerlessResponse, - options: Options, -) => { - if (options.isExternalRewrite) { - return proxyRequest(req, res); - } else { - // Next Server - return requestHandler(req, res); - } -}; -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/13.5/util.ts b/packages/open-next/src/adapters/plugins/13.5/util.ts deleted file mode 100644 index 4bf53b92..00000000 --- a/packages/open-next/src/adapters/plugins/13.5/util.ts +++ /dev/null @@ -1,5 +0,0 @@ -import { debug } from "../../logger.js"; - -//#override requireHooks -debug("No need to override require hooks with next 13.4.20+"); -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/image-optimization.replacement.ts b/packages/open-next/src/adapters/plugins/image-optimization/image-optimization.replacement.ts similarity index 96% rename from packages/open-next/src/adapters/plugins/image-optimization.replacement.ts rename to packages/open-next/src/adapters/plugins/image-optimization/image-optimization.replacement.ts index c041f370..4b948be4 100644 --- a/packages/open-next/src/adapters/plugins/image-optimization.replacement.ts +++ b/packages/open-next/src/adapters/plugins/image-optimization/image-optimization.replacement.ts @@ -13,7 +13,7 @@ import { //#endOverride import type { NextUrlWithParsedQuery } from "next/dist/server/request-meta"; -import { debug } from "../logger.js"; +import { debug } from "../../logger.js"; //#override optimizeImage export async function optimizeImage( diff --git a/packages/open-next/src/adapters/plugins/image-optimization.ts b/packages/open-next/src/adapters/plugins/image-optimization/image-optimization.ts similarity index 95% rename from packages/open-next/src/adapters/plugins/image-optimization.ts rename to packages/open-next/src/adapters/plugins/image-optimization/image-optimization.ts index 8cbedbd0..5a9fc75e 100644 --- a/packages/open-next/src/adapters/plugins/image-optimization.ts +++ b/packages/open-next/src/adapters/plugins/image-optimization/image-optimization.ts @@ -7,7 +7,7 @@ import { imageOptimizer } from "next/dist/server/image-optimizer"; //#endOverride import { NextUrlWithParsedQuery } from "next/dist/server/request-meta"; -import { debug } from "../logger.js"; +import { debug } from "../../logger.js"; //#override optimizeImage export async function optimizeImage( diff --git a/packages/open-next/src/adapters/plugins/lambdaHandler.ts b/packages/open-next/src/adapters/plugins/lambdaHandler.ts deleted file mode 100644 index 7d7a614e..00000000 --- a/packages/open-next/src/adapters/plugins/lambdaHandler.ts +++ /dev/null @@ -1,139 +0,0 @@ -import { - APIGatewayProxyEvent, - APIGatewayProxyEventV2, - CloudFrontRequestEvent, -} from "aws-lambda"; - -import { BuildId, PublicAssets } from "../config"; -import { convertFrom, convertTo, InternalEvent } from "../event-mapper"; -import { type IncomingMessage, ServerlessResponse } from "../http"; -import { debug, error } from "../logger"; -import { CreateResponse } from "../types/plugin"; -import { generateUniqueId } from "../util"; -import { WarmerEvent, WarmerResponse } from "../warmer-function"; -//#override imports -import { - postProcessResponse, - processInternalEvent, -} from "./routing/default.js"; -//#endOverride -import { handler as serverHandler } from "./serverHandler"; - -const serverId = `server-${generateUniqueId()}`; - -//#override lambdaHandler -export async function lambdaHandler( - event: - | APIGatewayProxyEventV2 - | CloudFrontRequestEvent - | APIGatewayProxyEvent - | WarmerEvent, -) { - debug("event", event); - // Handler warmer - if ("type" in event) { - return formatWarmerResponse(event); - } - - // Parse Lambda event and create Next.js request - const internalEvent = convertFrom(event); - - // WORKAROUND: Set `x-forwarded-host` header (AWS specific) — https://github.com/serverless-stack/open-next#workaround-set-x-forwarded-host-header-aws-specific - if (internalEvent.headers["x-forwarded-host"]) { - internalEvent.headers.host = internalEvent.headers["x-forwarded-host"]; - } - - // WORKAROUND: public/ static files served by the server function (AWS specific) — https://github.com/serverless-stack/open-next#workaround-public-static-files-served-by-the-server-function-aws-specific - // TODO: This is no longer required if each top-level file and folder in "/public" - // is handled by a separate cache behavior. Leaving here for backward compatibility. - // Remove this on next major release. - if (PublicAssets.files.includes(internalEvent.rawPath)) { - return internalEvent.type === "cf" - ? formatCloudFrontFailoverResponse(event as CloudFrontRequestEvent) - : formatAPIGatewayFailoverResponse(); - } - - const createServerResponse: CreateResponse = ( - method, - headers, - ) => new ServerlessResponse({ method, headers }); - - const preprocessResult = await processInternalEvent( - internalEvent, - createServerResponse, - ); - if ("type" in preprocessResult) { - return convertTo(preprocessResult); - } else { - const { - req, - res, - isExternalRewrite, - internalEvent: overwrittenInternalEvent, - } = preprocessResult; - - await processRequest(req, res, overwrittenInternalEvent, isExternalRewrite); - - const internalResult = await postProcessResponse({ - internalEvent: overwrittenInternalEvent, - req, - res, - isExternalRewrite, - }); - - return convertTo(internalResult); - } -} -//#endOverride - -async function processRequest( - req: IncomingMessage, - res: ServerlessResponse, - internalEvent: InternalEvent, - isExternalRewrite?: boolean, -) { - // @ts-ignore - // Next.js doesn't parse body if the property exists - // https://github.com/dougmoscrop/serverless-http/issues/227 - delete req.body; - - try { - // `serverHandler` is replaced at build time depending on user's - // nextjs version to patch Nextjs 13.4.x and future breaking changes. - await serverHandler(req, res, { - internalEvent, - buildId: BuildId, - isExternalRewrite, - }); - } catch (e: any) { - error("NextJS request failed.", e); - - res.setHeader("Content-Type", "application/json"); - res.end( - JSON.stringify( - { - message: "Server failed to respond.", - details: e, - }, - null, - 2, - ), - ); - } -} - -function formatAPIGatewayFailoverResponse() { - return { statusCode: 503 }; -} - -function formatCloudFrontFailoverResponse(event: CloudFrontRequestEvent) { - return event.Records[0].cf.request; -} - -function formatWarmerResponse(event: WarmerEvent) { - return new Promise((resolve) => { - setTimeout(() => { - resolve({ serverId } satisfies WarmerResponse); - }, event.delay); - }); -} diff --git a/packages/open-next/src/adapters/plugins/routing/default.replacement.ts b/packages/open-next/src/adapters/plugins/routing/default.replacement.ts deleted file mode 100644 index de2f20e6..00000000 --- a/packages/open-next/src/adapters/plugins/routing/default.replacement.ts +++ /dev/null @@ -1,162 +0,0 @@ -/* eslint-disable simple-import-sort/imports */ -import type { - PostProcessOptions, - ProcessInternalEvent, -} from "../../types/plugin"; -import type { InternalResult } from "../../event-mapper"; -//#override imports - -import { debug } from "../../logger"; -import { IncomingMessage } from "../../http/request"; -import { - addNextConfigHeaders, - fixDataPage, - handleFallbackFalse, - handleRedirects, - handleRewrites, -} from "../../routing/matcher"; -import { - addOpenNextHeader, - fixCacheHeaderForHtmlPages, - fixISRHeaders, - fixSWRCacheHeader, - revalidateIfRequired, -} from "./util"; -import { convertRes } from "../../routing/util"; -import { handleMiddleware } from "../../routing/middleware"; -import { ServerlessResponse } from "../../http"; -import { - BuildId, - ConfigHeaders, - PrerenderManifest, - RoutesManifest, -} from "../../config"; - -//#endOverride - -//#override processInternalEvent -export const processInternalEvent: ProcessInternalEvent = async ( - event, - createResponse, -) => { - const nextHeaders = addNextConfigHeaders(event, ConfigHeaders) ?? {}; - - let internalEvent = fixDataPage(event, BuildId); - // If we return InternalResult, it means that the build id is not correct - // We should return a 404 - if ("statusCode" in internalEvent) { - return internalEvent; - } - - internalEvent = handleFallbackFalse(internalEvent, PrerenderManifest); - - const redirect = handleRedirects(internalEvent, RoutesManifest.redirects); - if (redirect) { - return redirect; - } - - const middleware = await handleMiddleware(internalEvent); - let middlewareResponseHeaders: Record = {}; - if ("statusCode" in middleware) { - return middleware; - } else { - middlewareResponseHeaders = middleware.responseHeaders || {}; - internalEvent = middleware; - } - - let isExternalRewrite = middleware.externalRewrite ?? false; - if (!isExternalRewrite) { - // First rewrite to be applied - const beforeRewrites = handleRewrites( - internalEvent, - RoutesManifest.rewrites.beforeFiles, - ); - internalEvent = beforeRewrites.internalEvent; - isExternalRewrite = beforeRewrites.isExternalRewrite; - } - const isStaticRoute = RoutesManifest.routes.static.some((route) => - new RegExp(route.regex).test(event.rawPath), - ); - - if (!isStaticRoute && !isExternalRewrite) { - // Second rewrite to be applied - const afterRewrites = handleRewrites( - internalEvent, - RoutesManifest.rewrites.afterFiles, - ); - internalEvent = afterRewrites.internalEvent; - isExternalRewrite = afterRewrites.isExternalRewrite; - } - - const isDynamicRoute = RoutesManifest.routes.dynamic.some((route) => - new RegExp(route.regex).test(event.rawPath), - ); - if (!isDynamicRoute && !isStaticRoute && !isExternalRewrite) { - // Fallback rewrite to be applied - const fallbackRewrites = handleRewrites( - internalEvent, - RoutesManifest.rewrites.fallback, - ); - internalEvent = fallbackRewrites.internalEvent; - isExternalRewrite = fallbackRewrites.isExternalRewrite; - } - - const reqProps = { - method: internalEvent.method, - url: internalEvent.url, - //WORKAROUND: We pass this header to the serverless function to mimic a prefetch request which will not trigger revalidation since we handle revalidation differently - // There is 3 way we can handle revalidation: - // 1. We could just let the revalidation go as normal, but due to race condtions the revalidation will be unreliable - // 2. We could alter the lastModified time of our cache to make next believe that the cache is fresh, but this could cause issues with stale data since the cdn will cache the stale data as if it was fresh - // 3. OUR CHOICE: We could pass a purpose prefetch header to the serverless function to make next believe that the request is a prefetch request and not trigger revalidation (This could potentially break in the future if next changes the behavior of prefetch requests) - headers: { ...internalEvent.headers, purpose: "prefetch" }, - body: internalEvent.body, - remoteAddress: internalEvent.remoteAddress, - }; - debug("IncomingMessage constructor props", reqProps); - const req = new IncomingMessage(reqProps); - const res = createResponse(reqProps.method, { - ...nextHeaders, - ...middlewareResponseHeaders, - }); - - return { internalEvent: internalEvent, req, res, isExternalRewrite }; -}; -//#endOverride - -//#override postProcessResponse -export async function postProcessResponse({ - internalEvent, - req, - res, - isExternalRewrite, -}: PostProcessOptions): Promise { - const { statusCode, headers, isBase64Encoded, body } = convertRes( - res as ServerlessResponse, - ); - - debug("ServerResponse data", { statusCode, headers, isBase64Encoded, body }); - - if (!isExternalRewrite) { - fixCacheHeaderForHtmlPages(internalEvent.rawPath, headers); - fixSWRCacheHeader(headers); - addOpenNextHeader(headers); - fixISRHeaders(headers); - - await revalidateIfRequired( - internalEvent.headers.host, - internalEvent.rawPath, - headers, - req, - ); - } - - return { - type: internalEvent.type, - statusCode, - headers, - body, - isBase64Encoded, - }; -} -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/routing/default.ts b/packages/open-next/src/adapters/plugins/routing/default.ts deleted file mode 100644 index 5f2694da..00000000 --- a/packages/open-next/src/adapters/plugins/routing/default.ts +++ /dev/null @@ -1,81 +0,0 @@ -/* eslint-disable simple-import-sort/imports */ -import type { - CreateResponse, - PostProcessOptions, - ProcessInternalEventResult, -} from "../../types/plugin"; -import type { InternalEvent, InternalResult } from "../../event-mapper"; -//#override imports -import { debug } from "../../logger"; -import { IncomingMessage } from "../../http/request"; -import { - addOpenNextHeader, - fixCacheHeaderForHtmlPages, - fixISRHeaders, - fixSWRCacheHeader, - revalidateIfRequired, -} from "./util"; -import { convertRes } from "../../routing/util"; -import { ServerlessResponse } from "../../http"; -import { ServerResponse } from "http"; -//#endOverride - -//#override processInternalEvent -export async function processInternalEvent( - internalEvent: InternalEvent, - createResponse: CreateResponse, -): Promise> { - const reqProps = { - method: internalEvent.method, - url: internalEvent.url, - //WORKAROUND: We pass this header to the serverless function to mimic a prefetch request which will not trigger revalidation since we handle revalidation differently - // There is 3 way we can handle revalidation: - // 1. We could just let the revalidation go as normal, but due to race condtions the revalidation will be unreliable - // 2. We could alter the lastModified time of our cache to make next believe that the cache is fresh, but this could cause issues with stale data since the cdn will cache the stale data as if it was fresh - // 3. OUR CHOICE: We could pass a purpose prefetch header to the serverless function to make next believe that the request is a prefetch request and not trigger revalidation (This could potentially break in the future if next changes the behavior of prefetch requests) - headers: { ...internalEvent.headers, purpose: "prefetch" }, - body: internalEvent.body, - remoteAddress: internalEvent.remoteAddress, - }; - const req = new IncomingMessage(reqProps); - const res = createResponse(reqProps.method, {}); - return { internalEvent, req, res, isExternalRewrite: false }; -} -//#endOverride - -//#override postProcessResponse -export async function postProcessResponse({ - internalEvent, - req, - res, - isExternalRewrite, -}: PostProcessOptions): Promise { - const { statusCode, headers, isBase64Encoded, body } = convertRes( - res as ServerlessResponse, - ); - - debug("ServerResponse data", { statusCode, headers, isBase64Encoded, body }); - - if (!isExternalRewrite) { - fixCacheHeaderForHtmlPages(internalEvent.rawPath, headers); - fixSWRCacheHeader(headers); - addOpenNextHeader(headers); - fixISRHeaders(headers); - - await revalidateIfRequired( - internalEvent.headers.host, - internalEvent.rawPath, - headers, - req, - ); - } - - return { - type: internalEvent.type, - statusCode, - headers, - body, - isBase64Encoded, - }; -} -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/routing/util.ts b/packages/open-next/src/adapters/plugins/routing/util.ts deleted file mode 100644 index ebba7bda..00000000 --- a/packages/open-next/src/adapters/plugins/routing/util.ts +++ /dev/null @@ -1,238 +0,0 @@ -import { SendMessageCommand, SQSClient } from "@aws-sdk/client-sqs"; -import crypto from "crypto"; -import { ServerResponse } from "http"; - -import { BuildId, HtmlPages } from "../../config/index.js"; -import { IncomingMessage } from "../../http/request.js"; -import { ServerlessResponse } from "../../http/response.js"; -import { awsLogger, debug } from "../../logger.js"; - -declare global { - var openNextDebug: boolean; - var openNextVersion: string; - var lastModified: number; -} - -enum CommonHeaders { - CACHE_CONTROL = "cache-control", - NEXT_CACHE = "x-nextjs-cache", -} - -// Expected environment variables -const { REVALIDATION_QUEUE_REGION, REVALIDATION_QUEUE_URL } = process.env; - -const sqsClient = new SQSClient({ - region: REVALIDATION_QUEUE_REGION, - logger: awsLogger, -}); - -export async function proxyRequest( - req: IncomingMessage, - res: ServerlessResponse, -) { - const HttpProxy = require("next/dist/compiled/http-proxy") as any; - - const proxy = new HttpProxy({ - changeOrigin: true, - ignorePath: true, - xfwd: true, - }); - - await new Promise((resolve, reject) => { - proxy.on("proxyRes", (proxyRes: ServerResponse) => { - const body: Uint8Array[] = []; - proxyRes.on("data", function (chunk) { - body.push(chunk); - }); - proxyRes.on("end", function () { - const newBody = Buffer.concat(body).toString(); - debug(`Proxying response`, { - // @ts-ignore TODO: get correct type for the proxyRes - headers: proxyRes.getHeaders?.() || proxyRes.headers, - body: newBody, - }); - res.end(newBody); - resolve(); - }); - }); - - proxy.on("error", (err: any) => { - reject(err); - }); - - debug(`Proxying`, { url: req.url, headers: req.headers }); - - proxy.web(req, res, { - target: req.url, - headers: req.headers, - }); - }); -} - -export function fixCacheHeaderForHtmlPages( - rawPath: string, - headers: Record, -) { - // WORKAROUND: `NextServer` does not set cache headers for HTML pages — https://github.com/serverless-stack/open-next#workaround-nextserver-does-not-set-cache-headers-for-html-pages - if (HtmlPages.includes(rawPath)) { - headers[CommonHeaders.CACHE_CONTROL] = - "public, max-age=0, s-maxage=31536000, must-revalidate"; - } -} - -export function fixSWRCacheHeader( - headers: Record, -) { - // WORKAROUND: `NextServer` does not set correct SWR cache headers — https://github.com/serverless-stack/open-next#workaround-nextserver-does-not-set-correct-swr-cache-headers - let cacheControl = headers[CommonHeaders.CACHE_CONTROL]; - if (!cacheControl) return; - if (Array.isArray(cacheControl)) { - cacheControl = cacheControl.join(","); - } - headers[CommonHeaders.CACHE_CONTROL] = cacheControl.replace( - /\bstale-while-revalidate(?!=)/, - "stale-while-revalidate=2592000", // 30 days - ); -} - -export function addOpenNextHeader(headers: Record) { - headers["X-OpenNext"] = "1"; - if (globalThis.openNextDebug) { - headers["X-OpenNext-Version"] = globalThis.openNextVersion; - } -} - -export async function revalidateIfRequired( - host: string, - rawPath: string, - headers: Record, - req?: IncomingMessage, -) { - fixISRHeaders(headers); - - if (headers[CommonHeaders.NEXT_CACHE] === "STALE") { - // If the URL is rewritten, revalidation needs to be done on the rewritten URL. - // - Link to Next.js doc: https://nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration#on-demand-revalidation - // - Link to NextInternalRequestMeta: https://github.com/vercel/next.js/blob/57ab2818b93627e91c937a130fb56a36c41629c3/packages/next/src/server/request-meta.ts#L11 - // @ts-ignore - const internalMeta = req?.[Symbol.for("NextInternalRequestMeta")]; - - // When using Pages Router, two requests will be received: - // 1. one for the page: /foo - // 2. one for the json data: /_next/data/BUILD_ID/foo.json - // The rewritten url is correct for 1, but that for the second request - // does not include the "/_next/data/" prefix. Need to add it. - const revalidateUrl = internalMeta?._nextDidRewrite - ? rawPath.startsWith("/_next/data/") - ? `/_next/data/${BuildId}${internalMeta?._nextRewroteUrl}.json` - : internalMeta?._nextRewroteUrl - : rawPath; - - // We need to pass etag to the revalidation queue to try to bypass the default 5 min deduplication window. - // https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html - // If you need to have a revalidation happen more frequently than 5 minutes, - // your page will need to have a different etag to bypass the deduplication window. - // If data has the same etag during these 5 min dedup window, it will be deduplicated and not revalidated. - try { - const hash = (str: string) => - crypto.createHash("md5").update(str).digest("hex"); - - const lastModified = - globalThis.lastModified > 0 ? globalThis.lastModified : ""; - - await sqsClient.send( - new SendMessageCommand({ - QueueUrl: REVALIDATION_QUEUE_URL, - MessageDeduplicationId: hash(`${rawPath}-${lastModified}`), - MessageBody: JSON.stringify({ host, url: revalidateUrl }), - MessageGroupId: generateMessageGroupId(rawPath), - }), - ); - } catch (e) { - debug(`Failed to revalidate stale page ${rawPath}`); - debug(e); - } - } -} - -// Since we're using a FIFO queue, every messageGroupId is treated sequentially -// This could cause a backlog of messages in the queue if there is too much page to -// revalidate at once. To avoid this, we generate a random messageGroupId for each -// revalidation request. -// We can't just use a random string because we need to ensure that the same rawPath -// will always have the same messageGroupId. -// https://stackoverflow.com/questions/521295/seeding-the-random-number-generator-in-javascript#answer-47593316 -function generateMessageGroupId(rawPath: string) { - let a = cyrb128(rawPath); - // We use mulberry32 to generate a random int between 0 and MAX_REVALIDATE_CONCURRENCY - var t = (a += 0x6d2b79f5); - t = Math.imul(t ^ (t >>> 15), t | 1); - t ^= t + Math.imul(t ^ (t >>> 7), t | 61); - const randomFloat = ((t ^ (t >>> 14)) >>> 0) / 4294967296; - // This will generate a random int between 0 and MAX_REVALIDATE_CONCURRENCY - // This means that we could have 1000 revalidate request at the same time - const maxConcurrency = parseInt( - process.env.MAX_REVALIDATE_CONCURRENCY ?? "10", - ); - const randomInt = Math.floor(randomFloat * maxConcurrency); - return `revalidate-${randomInt}`; -} - -// Used to generate a hash int from a string -function cyrb128(str: string) { - let h1 = 1779033703, - h2 = 3144134277, - h3 = 1013904242, - h4 = 2773480762; - for (let i = 0, k; i < str.length; i++) { - k = str.charCodeAt(i); - h1 = h2 ^ Math.imul(h1 ^ k, 597399067); - h2 = h3 ^ Math.imul(h2 ^ k, 2869860233); - h3 = h4 ^ Math.imul(h3 ^ k, 951274213); - h4 = h1 ^ Math.imul(h4 ^ k, 2716044179); - } - h1 = Math.imul(h3 ^ (h1 >>> 18), 597399067); - h2 = Math.imul(h4 ^ (h2 >>> 22), 2869860233); - h3 = Math.imul(h1 ^ (h3 >>> 17), 951274213); - h4 = Math.imul(h2 ^ (h4 >>> 19), 2716044179); - (h1 ^= h2 ^ h3 ^ h4), (h2 ^= h1), (h3 ^= h1), (h4 ^= h1); - return h1 >>> 0; -} - -export function fixISRHeaders(headers: Record) { - if (headers[CommonHeaders.NEXT_CACHE] === "REVALIDATED") { - headers[CommonHeaders.CACHE_CONTROL] = - "private, no-cache, no-store, max-age=0, must-revalidate"; - return; - } - if ( - headers[CommonHeaders.NEXT_CACHE] === "HIT" && - globalThis.lastModified > 0 - ) { - // calculate age - const age = Math.round((Date.now() - globalThis.lastModified) / 1000); - // extract s-maxage from cache-control - const regex = /s-maxage=(\d+)/; - const match = headers[CommonHeaders.CACHE_CONTROL]?.match(regex); - const sMaxAge = match ? parseInt(match[1]) : undefined; - - // 31536000 is the default s-maxage value for SSG pages - if (sMaxAge && sMaxAge !== 31536000) { - const remainingTtl = Math.max(sMaxAge - age, 1); - headers[ - CommonHeaders.CACHE_CONTROL - ] = `s-maxage=${remainingTtl}, stale-while-revalidate=2592000`; - } - - // reset lastModified - globalThis.lastModified = 0; - } - if (headers[CommonHeaders.NEXT_CACHE] !== "STALE") return; - - // If the cache is stale, we revalidate in the background - // In order for CloudFront SWR to work, we set the stale-while-revalidate value to 2 seconds - // This will cause CloudFront to cache the stale data for a short period of time while we revalidate in the background - // Once the revalidation is complete, CloudFront will serve the fresh data - headers[CommonHeaders.CACHE_CONTROL] = - "s-maxage=2, stale-while-revalidate=2592000"; -} diff --git a/packages/open-next/src/adapters/plugins/serverHandler.replacement.ts b/packages/open-next/src/adapters/plugins/serverHandler.replacement.ts deleted file mode 100644 index 1b311b0c..00000000 --- a/packages/open-next/src/adapters/plugins/serverHandler.replacement.ts +++ /dev/null @@ -1,29 +0,0 @@ -/*eslint-disable simple-import-sort/imports */ -import type { Options, PluginHandler } from "../types/next-types.js"; -import type { IncomingMessage } from "../http/request.js"; -import type { ServerlessResponse } from "../http/response.js"; -//#override imports - -import { proxyRequest } from "./routing/util.js"; -import { requestHandler, setNextjsPrebundledReact } from "./util.js"; -//#endOverride - -//#override handler -export const handler: PluginHandler = async ( - req: IncomingMessage, - res: ServerlessResponse, - options: Options, -) => { - let { internalEvent } = options; - - const { rawPath } = internalEvent; - - if (options.isExternalRewrite) { - return proxyRequest(req, res); - } else { - setNextjsPrebundledReact(rawPath); - // Next Server - return requestHandler(req, res); - } -}; -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/serverHandler.ts b/packages/open-next/src/adapters/plugins/serverHandler.ts deleted file mode 100644 index 620e1e90..00000000 --- a/packages/open-next/src/adapters/plugins/serverHandler.ts +++ /dev/null @@ -1,17 +0,0 @@ -import type { IncomingMessage } from "../http/request.js"; -import { ServerlessResponse } from "../http/response.js"; -import type { Options, PluginHandler } from "../types/next-types.js"; -//#override imports -import { requestHandler, setNextjsPrebundledReact } from "./util.js"; -//#endOverride - -//#override handler -export const handler: PluginHandler = async ( - req: IncomingMessage, - res: ServerlessResponse, - options: Options, -) => { - setNextjsPrebundledReact(options.internalEvent.rawPath); - return requestHandler(req, res); -}; -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/streaming.replacement.ts b/packages/open-next/src/adapters/plugins/streaming.replacement.ts deleted file mode 100644 index 7a319024..00000000 --- a/packages/open-next/src/adapters/plugins/streaming.replacement.ts +++ /dev/null @@ -1,104 +0,0 @@ -/*eslint-disable simple-import-sort/imports */ -import type { - APIGatewayProxyEvent, - APIGatewayProxyEventV2, - CloudFrontRequestEvent, -} from "aws-lambda"; - -import { convertFrom } from "../event-mapper"; -import { debug } from "../logger"; -import type { ResponseStream } from "../types/aws-lambda"; -import type { WarmerEvent } from "../warmer-function"; -//#override imports -import { StreamingServerResponse } from "../http/responseStreaming"; -import { processInternalEvent } from "./routing/default.js"; -import { - addOpenNextHeader, - fixCacheHeaderForHtmlPages, - fixISRHeaders, - fixSWRCacheHeader, - revalidateIfRequired, -} from "./routing/util"; -import { CreateResponse } from "../types/plugin"; -//#endOverride - -//#override lambdaHandler -export const lambdaHandler = awslambda.streamifyResponse(async function ( - event: - | APIGatewayProxyEventV2 - | CloudFrontRequestEvent - | APIGatewayProxyEvent - | WarmerEvent, - responseStream: ResponseStream, -) { - debug("event", event); - - // Handler warmer - if ("type" in event) { - // @ts-ignore formatWarmerResponse defined in lambdaHandler - const result = await formatWarmerResponse(event); - responseStream.end(Buffer.from(JSON.stringify(result)), "utf-8"); - return; - } - // Parse Lambda event and create Next.js request - const internalEvent = convertFrom(event); - - // WORKAROUND: Set `x-forwarded-host` header (AWS specific) — https://github.com/serverless-stack/open-next#workaround-set-x-forwarded-host-header-aws-specific - if (internalEvent.headers["x-forwarded-host"]) { - internalEvent.headers.host = internalEvent.headers["x-forwarded-host"]; - } - - const createServerResponse: CreateResponse = ( - method: string, - headers: Record, - ) => { - // sets the accept-encoding for responseStreaming.ts to set "content-encoding" - headers["accept-encoding"] = internalEvent.headers["accept-encoding"]; - return new StreamingServerResponse({ - method, - headers, - responseStream, - // We need to fix the cache header before sending any response - fixHeaders: (headers) => { - fixCacheHeaderForHtmlPages(internalEvent.rawPath, headers); - fixSWRCacheHeader(headers); - addOpenNextHeader(headers); - fixISRHeaders(headers); - }, - // This run in the callback of the response stream end - onEnd: async (headers) => { - await revalidateIfRequired( - internalEvent.headers.host, - internalEvent.rawPath, - headers, - ); - }, - }); - }; - - const preprocessResult = await processInternalEvent( - internalEvent, - createServerResponse, - ); - if ("type" in preprocessResult) { - const headers = preprocessResult.headers; - const res = createServerResponse("GET", headers); - - setImmediate(() => { - res.writeHead(preprocessResult.statusCode, headers); - res.write(preprocessResult.body); - res.end(); - }); - } else { - const { - req, - res, - isExternalRewrite, - internalEvent: overwrittenInternalEvent, - } = preprocessResult; - - //@ts-expect-error - processRequest is already defined in serverHandler.ts - await processRequest(req, res, overwrittenInternalEvent, isExternalRewrite); - } -}); -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/util.replacement.ts b/packages/open-next/src/adapters/plugins/util.replacement.ts deleted file mode 100644 index 6d44b404..00000000 --- a/packages/open-next/src/adapters/plugins/util.replacement.ts +++ /dev/null @@ -1,25 +0,0 @@ -import { NextConfig } from "../config"; - -//#override requestHandler -// @ts-ignore -export const requestHandler = new NextServer.default({ - conf: { - ...NextConfig, - // Next.js compression should be disabled because of a bug in the bundled - // `compression` package — https://github.com/vercel/next.js/issues/11669 - compress: false, - // By default, Next.js uses local disk to store ISR cache. We will use - // our own cache handler to store the cache on S3. - experimental: { - ...NextConfig.experimental, - // This uses the request.headers.host as the URL - // https://github.com/vercel/next.js/blob/canary/packages/next/src/server/next-server.ts#L1749-L1754 - trustHostHeader: true, - incrementalCacheHandlerPath: `${process.env.LAMBDA_TASK_ROOT}/cache.cjs`, - }, - }, - customServer: false, - dev: false, - dir: __dirname, -}).getRequestHandler(); -//#endOverride diff --git a/packages/open-next/src/adapters/plugins/without-routing/requestHandler.ts b/packages/open-next/src/adapters/plugins/without-routing/requestHandler.ts new file mode 100644 index 00000000..5d2cbc51 --- /dev/null +++ b/packages/open-next/src/adapters/plugins/without-routing/requestHandler.ts @@ -0,0 +1,16 @@ +import { InternalEvent } from "types/open-next"; + +import { MiddlewareOutputEvent } from "../../../core/routingHandler"; +// This is available in requestHandler.ts +declare const internalEvent: InternalEvent; + +//#override withRouting +const preprocessResult: MiddlewareOutputEvent = { + internalEvent: internalEvent, + isExternalRewrite: false, + origin: false, +}; +//#endOverride + +// We need to export something otherwise when compiled in js it creates an empty export {} inside the override +export default {}; diff --git a/packages/open-next/src/adapters/revalidate.ts b/packages/open-next/src/adapters/revalidate.ts index 5014da2b..471d4b1a 100644 --- a/packages/open-next/src/adapters/revalidate.ts +++ b/packages/open-next/src/adapters/revalidate.ts @@ -3,8 +3,7 @@ import type { IncomingMessage } from "node:http"; import https from "node:https"; import path from "node:path"; -import type { SQSEvent } from "aws-lambda"; - +import { createGenericHandler } from "../core/createGenericHandler.js"; import { debug, error } from "./logger.js"; const prerenderManifest = loadPrerenderManifest(); @@ -17,9 +16,17 @@ interface PrerenderManifest { }; } -export const handler = async (event: SQSEvent) => { - for (const record of event.Records) { - const { host, url } = JSON.parse(record.body); +export interface RevalidateEvent { + type: "revalidate"; + records: { + host: string; + url: string; + }[]; +} + +const defaultHandler = async (event: RevalidateEvent) => { + for (const record of event.records) { + const { host, url } = record; debug(`Revalidating stale page`, { host, url }); // Make a HEAD request to the page to revalidate it. This will trigger @@ -49,8 +56,16 @@ export const handler = async (event: SQSEvent) => { req.end(); }); } + return { + type: "revalidate", + }; }; +export const handler = await createGenericHandler({ + handler: defaultHandler, + type: "revalidate", +}); + function loadPrerenderManifest() { const filePath = path.join("prerender-manifest.json"); const json = fs.readFileSync(filePath, "utf-8"); diff --git a/packages/open-next/src/adapters/routing/util.ts b/packages/open-next/src/adapters/routing/util.ts deleted file mode 100644 index ce3ff879..00000000 --- a/packages/open-next/src/adapters/routing/util.ts +++ /dev/null @@ -1,101 +0,0 @@ -import fs from "node:fs"; -import path from "node:path"; - -import { isBinaryContentType } from "../binary"; -import { ServerlessResponse } from "../http/response"; -import { MiddlewareManifest } from "../types/next-types"; - -export function isExternal(url?: string, host?: string) { - if (!url) return false; - const pattern = /^https?:\/\//; - if (host) { - return pattern.test(url) && !url.includes(host); - } - return pattern.test(url); -} - -export function getUrlParts(url: string, isExternal: boolean) { - // NOTE: when redirect to a URL that contains search query params, - // compile breaks b/c it does not allow for the '?' character - // We can't use encodeURIComponent because modal interception contains - // characters that can't be encoded - url = url.replaceAll("?", "%3F"); - if (!isExternal) { - return { - hostname: "", - pathname: url, - protocol: "", - }; - } - const { hostname, pathname, protocol } = new URL(url); - return { - hostname, - pathname, - protocol, - }; -} - -export function convertRes(res: ServerlessResponse) { - // Format Next.js response to Lambda response - const statusCode = res.statusCode || 200; - const headers = ServerlessResponse.headers(res); - const isBase64Encoded = isBinaryContentType( - Array.isArray(headers["content-type"]) - ? headers["content-type"][0] - : headers["content-type"], - ); - const encoding = isBase64Encoded ? "base64" : "utf8"; - const body = ServerlessResponse.body(res).toString(encoding); - return { - statusCode, - headers, - body, - isBase64Encoded, - }; -} - -/** - * Make sure that multi-value query parameters are transformed to - * ?key=value1&key=value2&... so that Next converts those parameters - * to an array when reading the query parameters - */ -export function convertToQueryString(query: Record) { - const urlQuery = new URLSearchParams(); - Object.entries(query).forEach(([key, value]) => { - if (Array.isArray(value)) { - value.forEach((entry) => urlQuery.append(key, entry)); - } else { - urlQuery.append(key, value); - } - }); - const queryString = urlQuery.toString(); - - return queryString ? `?${queryString}` : ""; -} - -/** - * Given a raw query string, returns a record with key value-array pairs - * similar to how multiValueQueryStringParameters are structured - */ -export function convertToQuery(querystring: string) { - const query = new URLSearchParams(querystring); - const queryObject: Record = {}; - - for (const key of query.keys()) { - queryObject[key] = query.getAll(key); - } - - return queryObject; -} - -export function getMiddlewareMatch(middlewareManifest: MiddlewareManifest) { - const rootMiddleware = middlewareManifest.middleware["/"]; - if (!rootMiddleware?.matchers) return []; - return rootMiddleware.matchers.map(({ regexp }) => new RegExp(regexp)); -} - -export function loadMiddlewareManifest(nextDir: string) { - const filePath = path.join(nextDir, "server", "middleware-manifest.json"); - const json = fs.readFileSync(filePath, "utf-8"); - return JSON.parse(json) as MiddlewareManifest; -} diff --git a/packages/open-next/src/adapters/server-adapter.ts b/packages/open-next/src/adapters/server-adapter.ts index 6c26cf54..3fa6df50 100644 --- a/packages/open-next/src/adapters/server-adapter.ts +++ b/packages/open-next/src/adapters/server-adapter.ts @@ -1,9 +1,8 @@ -import { DynamoDBClient, DynamoDBClientConfig } from "@aws-sdk/client-dynamodb"; -import { S3Client, S3ClientConfig } from "@aws-sdk/client-s3"; +// We load every config here so that they are only loaded once +// and during cold starts +import { BuildId } from "config/index.js"; -import { BuildId } from "./config/index.js"; -import { awsLogger } from "./logger.js"; -import { lambdaHandler } from "./plugins/lambdaHandler.js"; +import { createMainHandler } from "../core/createMainHandler.js"; import { setNodeEnv } from "./util.js"; // We load every config here so that they are only loaded once @@ -12,57 +11,11 @@ setNodeEnv(); setBuildIdEnv(); setNextjsServerWorkingDirectory(); -//////////////////////// -// AWS global clients // -//////////////////////// - -declare global { - var S3Client: S3Client; - var dynamoClient: DynamoDBClient; -} - -const CACHE_BUCKET_REGION = process.env.CACHE_BUCKET_REGION; - -function parseS3ClientConfigFromEnv(): S3ClientConfig { - return { - region: CACHE_BUCKET_REGION, - logger: awsLogger, - maxAttempts: parseNumberFromEnv(process.env.AWS_SDK_S3_MAX_ATTEMPTS), - }; -} - -function parseDynamoClientConfigFromEnv(): DynamoDBClientConfig { - return { - region: CACHE_BUCKET_REGION, - logger: awsLogger, - maxAttempts: parseNumberFromEnv(process.env.AWS_SDK_DYNAMODB_MAX_ATTEMPTS), - }; -} - -function parseNumberFromEnv(envValue: string | undefined): number | undefined { - if (typeof envValue !== "string") { - return envValue; - } - - const parsedValue = parseInt(envValue); - - return isNaN(parsedValue) ? undefined : parsedValue; -} - -// Cache clients using global variables -// Note: The clients are used in `cache.ts`. The incremental cache is recreated on -// every request and required on every request (And the require cache is also -// cleared). It was causing some file to stay open which after enough time -// would cause the function to crash with error "EMFILE too many open". It -// was also making the memory grow out of control. -globalThis.S3Client = new S3Client(parseS3ClientConfigFromEnv()); -globalThis.dynamoClient = new DynamoDBClient(parseDynamoClientConfigFromEnv()); - ///////////// // Handler // ///////////// -export const handler = lambdaHandler; +export const handler = await createMainHandler(); ////////////////////// // Helper functions // diff --git a/packages/open-next/src/adapters/types/plugin.ts b/packages/open-next/src/adapters/types/plugin.ts deleted file mode 100644 index 47c7ed3a..00000000 --- a/packages/open-next/src/adapters/types/plugin.ts +++ /dev/null @@ -1,36 +0,0 @@ -import type { ServerResponse } from "http"; - -import type { InternalEvent, InternalResult } from "../event-mapper"; -import type { IncomingMessage } from "../http/request"; - -export type ProcessInternalEventResult< - Response extends ServerResponse = ServerResponse, -> = - | { - internalEvent: InternalEvent; - req: IncomingMessage; - res: Response; - isExternalRewrite: boolean; - } - | InternalResult; - -export type ProcessInternalEvent< - Response extends ServerResponse = ServerResponse, -> = ( - internalEvent: InternalEvent, - createResponse: CreateResponse, -) => Promise>; - -export interface PostProcessOptions< - Response extends ServerResponse = ServerResponse, -> { - internalEvent: InternalEvent; - req: IncomingMessage; - res: Response; - isExternalRewrite?: boolean; -} - -export type CreateResponse = ( - method: string, - headers: Record, -) => Response; diff --git a/packages/open-next/src/adapters/util.ts b/packages/open-next/src/adapters/util.ts index aca13a2c..d35545e8 100644 --- a/packages/open-next/src/adapters/util.ts +++ b/packages/open-next/src/adapters/util.ts @@ -1,3 +1,5 @@ +//TODO: We should probably move all the utils to a separate location + export function setNodeEnv() { process.env.NODE_ENV = process.env.NODE_ENV ?? "production"; } @@ -6,39 +8,6 @@ export function generateUniqueId() { return Math.random().toString(36).slice(2, 8); } -export function escapeRegex(str: string) { - let path = str.replace(/\(\.\)/g, "_µ1_"); - - path = path.replace(/\(\.{2}\)/g, "_µ2_"); - - path = path.replace(/\(\.{3}\)/g, "_µ3_"); - - return path; -} - -export function unescapeRegex(str: string) { - let path = str.replace(/_µ1_/g, "(.)"); - - path = path.replace(/_µ2_/g, "(..)"); - - path = path.replace(/_µ3_/g, "(...)"); - - return path; -} - -// AWS cookies are in a single `set-cookie` string, delimited by a comma -export function parseCookies( - cookies?: string | string[], -): string[] | undefined { - if (!cookies) return; - - if (typeof cookies === "string") { - return cookies.split(/(? c.trim()); - } - - return cookies; -} - /** * Create an array of arrays of size `chunkSize` from `items` * @param items Array of T @@ -54,3 +23,15 @@ export function chunk(items: T[], chunkSize: number): T[][] { return chunked; } + +export function parseNumberFromEnv( + envValue: string | undefined, +): number | undefined { + if (typeof envValue !== "string") { + return envValue; + } + + const parsedValue = parseInt(envValue); + + return isNaN(parsedValue) ? undefined : parsedValue; +} diff --git a/packages/open-next/src/adapters/warmer-function.ts b/packages/open-next/src/adapters/warmer-function.ts index 4ee564fa..00addca4 100644 --- a/packages/open-next/src/adapters/warmer-function.ts +++ b/packages/open-next/src/adapters/warmer-function.ts @@ -1,13 +1,9 @@ -import { InvokeCommand, LambdaClient } from "@aws-sdk/client-lambda"; -import type { Context } from "aws-lambda"; +import { Warmer } from "types/open-next.js"; +import { createGenericHandler } from "../core/createGenericHandler.js"; import { debug, error } from "./logger.js"; import { generateUniqueId } from "./util.js"; -const lambda = new LambdaClient({}); -const FUNCTION_NAME = process.env.FUNCTION_NAME!; -const CONCURRENCY = parseInt(process.env.CONCURRENCY!); - export interface WarmerEvent { type: "warmer"; warmerId: string; @@ -17,60 +13,105 @@ export interface WarmerEvent { } export interface WarmerResponse { + type: "warmer"; serverId: string; } -export async function handler(_event: any, context: Context) { +const resolveWarmerInvoke = async () => { + const openNextParams = globalThis.openNextConfig.warmer!; + if (typeof openNextParams?.invokeFunction === "function") { + return await openNextParams.invokeFunction(); + } else { + return Promise.resolve({ + name: "aws-invoke", + invoke: async (warmerId: string) => { + const { InvokeCommand, LambdaClient } = await import( + "@aws-sdk/client-lambda" + ); + const lambda = new LambdaClient({}); + const warmParams = JSON.parse(process.env.WARM_PARAMS!) as { + concurrency: number; + function: string; + }[]; + + for (const warmParam of warmParams) { + const { concurrency: CONCURRENCY, function: FUNCTION_NAME } = + warmParam; + debug({ + event: "warmer invoked", + functionName: FUNCTION_NAME, + concurrency: CONCURRENCY, + warmerId, + }); + const ret = await Promise.all( + Array.from({ length: CONCURRENCY }, (_v, i) => i).map((i) => { + try { + return lambda.send( + new InvokeCommand({ + FunctionName: FUNCTION_NAME, + InvocationType: "RequestResponse", + Payload: Buffer.from( + JSON.stringify({ + type: "warmer", + warmerId, + index: i, + concurrency: CONCURRENCY, + delay: 75, + } satisfies WarmerEvent), + ), + }), + ); + } catch (e) { + error(`failed to warm up #${i}`, e); + // ignore error + } + }), + ); + + // Print status + + const warmedServerIds = ret + .map((r, i) => { + if (r?.StatusCode !== 200 || !r?.Payload) { + error(`failed to warm up #${i}:`, r?.Payload?.toString()); + return; + } + const payload = JSON.parse( + Buffer.from(r.Payload).toString(), + ) as WarmerResponse; + return { + statusCode: r.StatusCode, + payload, + type: "warmer" as const, + }; + }) + .filter((r): r is Exclude => !!r); + + debug({ + event: "warmer result", + sent: CONCURRENCY, + success: warmedServerIds.length, + uniqueServersWarmed: [...new Set(warmedServerIds)].length, + }); + } + }, + }); + } +}; + +export const handler = await createGenericHandler({ + handler: defaultHandler, + type: "warmer", +}); + +async function defaultHandler() { const warmerId = `warmer-${generateUniqueId()}`; - debug({ - event: "warmer invoked", - functionName: FUNCTION_NAME, - concurrency: CONCURRENCY, - warmerId, - }); - // Warm - const ret = await Promise.all( - Array.from({ length: CONCURRENCY }, (_v, i) => i).map((i) => { - try { - return lambda.send( - new InvokeCommand({ - FunctionName: FUNCTION_NAME, - InvocationType: "RequestResponse", - Payload: Buffer.from( - JSON.stringify({ - type: "warmer", - warmerId, - index: i, - concurrency: CONCURRENCY, - delay: 75, - } satisfies WarmerEvent), - ), - }), - ); - } catch (e) { - error(`failed to warm up #${i}`, e); - // ignore error - } - }), - ); + const invokeFn = await resolveWarmerInvoke(); + + await invokeFn.invoke(warmerId); - // Print status - const warmedServerIds: string[] = []; - ret.forEach((r, i) => { - if (r?.StatusCode !== 200 || !r?.Payload) { - error(`failed to warm up #${i}:`, r?.Payload?.toString()); - return; - } - const payload = JSON.parse( - Buffer.from(r.Payload).toString(), - ) as WarmerResponse; - warmedServerIds.push(payload.serverId); - }); - debug({ - event: "warmer result", - sent: CONCURRENCY, - success: warmedServerIds.length, - uniqueServersWarmed: [...new Set(warmedServerIds)].length, - }); + return { + type: "warmer", + }; } diff --git a/packages/open-next/src/build.ts b/packages/open-next/src/build.ts index b7eba24f..5aed56f1 100755 --- a/packages/open-next/src/build.ts +++ b/packages/open-next/src/build.ts @@ -1,96 +1,59 @@ import cp from "node:child_process"; -import fs from "node:fs"; +import fs, { readFileSync } from "node:fs"; import { createRequire as topLevelCreateRequire } from "node:module"; +import os from "node:os"; import path from "node:path"; import url from "node:url"; -import { - build as buildAsync, - BuildOptions as ESBuildOptions, - buildSync, -} from "esbuild"; +import { buildSync } from "esbuild"; +import { MiddlewareManifest } from "types/next-types.js"; import { isBinaryContentType } from "./adapters/binary.js"; +import { createServerBundle } from "./build/createServerBundle.js"; +import { buildEdgeBundle } from "./build/edge/createEdgeBundle.js"; +import { generateOutput } from "./build/generateOutput.js"; +import { + BuildOptions, + compareSemver, + copyOpenNextConfig, + esbuildAsync, + esbuildSync, + getBuildId, + getHtmlPages, + normalizeOptions, + removeFiles, + traverseFiles, +} from "./build/helper.js"; +import { validateConfig } from "./build/validateConfig.js"; import logger from "./logger.js"; -import { minifyAll } from "./minimize-js.js"; -import openNextPlugin from "./plugin.js"; - -interface DangerousOptions { - /** - * The dynamo db cache is used for revalidateTags and revalidatePath. - * @default false - */ - disableDynamoDBCache?: boolean; - /** - * The incremental cache is used for ISR and SSG. - * Disable this only if you use only SSR - * @default false - */ - disableIncrementalCache?: boolean; -} -interface BuildOptions { - /** - * Minify the server bundle. - * @default false - */ - minify?: boolean; - /** - * Print debug information. - * @default false - */ - debug?: boolean; - /** - * Enable streaming mode. - * @default false - */ - streaming?: boolean; - /** - * The command to build the Next.js app. - * @default `npm run build`, `yarn build`, or `pnpm build` based on the lock file found in the app's directory or any of its parent directories. - * @example - * ```ts - * build({ - * buildCommand: "pnpm custom:build", - * }); - * ``` - */ - /** - * Dangerous options. This break some functionnality but can be useful in some cases. - */ - dangerous?: DangerousOptions; - buildCommand?: string; - /** - * The path to the target folder of build output from the `buildCommand` option (the path which will contain the `.next` and `.open-next` folders). This path is relative from the current process.cwd(). - * @default "." - */ - buildOutputPath?: string; - /** - * The path to the root of the Next.js app's source code. This path is relative from the current process.cwd(). - * @default "." - */ - appPath?: string; - - /** - * The path to the package.json file. This path is relative from the current process.cwd(). - */ - packageJsonPath?: string; -} +import { openNextReplacementPlugin } from "./plugins/replacement.js"; +import { openNextResolvePlugin } from "./plugins/resolve.js"; +import { OpenNextConfig } from "./types/open-next.js"; const require = topLevelCreateRequire(import.meta.url); const __dirname = url.fileURLToPath(new URL(".", import.meta.url)); -let options: ReturnType; +let options: BuildOptions; +let config: OpenNextConfig; export type PublicFiles = { files: string[]; }; -export async function build(opts: BuildOptions = {}) { +export async function build(openNextConfigPath?: string) { + showWindowsWarning(); + + // Load open-next.config.ts + const tempDir = initTempDir(); + const configPath = compileOpenNextConfig(tempDir, openNextConfigPath); + config = (await import(configPath)).default as OpenNextConfig; + validateConfig(config); + const { root: monorepoRoot, packager } = findMonorepoRoot( - path.join(process.cwd(), opts.appPath || "."), + path.join(process.cwd(), config.appPath || "."), ); // Initialize options - options = normalizeOptions(opts, monorepoRoot); + options = normalizeOptions(config, monorepoRoot); logger.setLevel(options.debug ? "debug" : "info"); // Pre-build validation @@ -106,48 +69,76 @@ export async function build(opts: BuildOptions = {}) { // Generate deployable bundle printHeader("Generating bundle"); initOutputDir(); + + // Compile cache.ts + compileCache(); + + // Compile middleware + await createMiddleware(); + createStaticAssets(); - if (!options.dangerous?.disableIncrementalCache) { - createCacheAssets(monorepoRoot, options.dangerous?.disableDynamoDBCache); - } - await createServerBundle(monorepoRoot, options.streaming); - createRevalidationBundle(); - await createImageOptimizationBundle(); - createWarmerBundle(); - if (options.minify) { - await minifyServerBundle(); - } + await createCacheAssets(monorepoRoot); + + await createServerBundle(config, options); + await createRevalidationBundle(config); + await createImageOptimizationBundle(config); + await createWarmerBundle(config); + await generateOutput(options.appPath, options.appBuildOutputPath, config); + logger.info("OpenNext build complete."); } -function normalizeOptions(opts: BuildOptions, root: string) { - const appPath = path.join(process.cwd(), opts.appPath || "."); - const buildOutputPath = path.join(process.cwd(), opts.buildOutputPath || "."); - const outputDir = path.join(buildOutputPath, ".open-next"); - - let nextPackageJsonPath: string; - if (opts.packageJsonPath) { - const _pkgPath = path.join(process.cwd(), opts.packageJsonPath); - nextPackageJsonPath = _pkgPath.endsWith("package.json") - ? _pkgPath - : path.join(_pkgPath, "./package.json"); +function showWindowsWarning() { + if (os.platform() !== "win32") return; + + logger.warn("OpenNext is not fully compatible with Windows."); + logger.warn( + "For optimal performance, it is recommended to use Windows Subsystem for Linux (WSL).", + ); + logger.warn( + "While OpenNext may function on Windows, it could encounter unpredictable failures during runtime.", + ); +} + +function initTempDir() { + const dir = path.join(process.cwd(), ".open-next"); + const tempDir = path.join(dir, ".build"); + fs.rmSync(dir, { recursive: true, force: true }); + fs.mkdirSync(tempDir, { recursive: true }); + return tempDir; +} + +function compileOpenNextConfig(tempDir: string, openNextConfigPath?: string) { + const sourcePath = path.join( + process.cwd(), + openNextConfigPath ?? "open-next.config.ts", + ); + const outputPath = path.join(tempDir, "open-next.config.mjs"); + + //Check if open-next.config.ts exists + if (!fs.existsSync(sourcePath)) { + //Create a simple open-next.config.mjs file + logger.debug("Cannot find open-next.config.ts. Using default config."); + fs.writeFileSync( + outputPath, + [ + "var config = { default: { } };", + "var open_next_config_default = config;", + "export { open_next_config_default as default };", + ].join("\n"), + ); } else { - nextPackageJsonPath = findNextPackageJsonPath(appPath, root); + buildSync({ + entryPoints: [sourcePath], + outfile: outputPath, + bundle: true, + format: "esm", + target: ["node18"], + external: ["node:*"], + platform: "neutral", + }); } - return { - openNextVersion: getOpenNextVersion(), - nextVersion: getNextVersion(nextPackageJsonPath), - nextPackageJsonPath, - appPath, - appBuildOutputPath: buildOutputPath, - appPublicPath: path.join(appPath, "public"), - outputDir, - tempDir: path.join(outputDir, ".build"), - minify: opts.minify ?? Boolean(process.env.OPEN_NEXT_MINIFY) ?? false, - debug: opts.debug ?? Boolean(process.env.OPEN_NEXT_DEBUG) ?? false, - buildCommand: opts.buildCommand, - dangerous: opts.dangerous, - streaming: opts.streaming ?? false, - }; + + return outputPath; } function checkRunningInsideNextjsApp() { @@ -188,13 +179,6 @@ function findMonorepoRoot(appPath: string) { return { root: appPath, packager: "npm" as const }; } -function findNextPackageJsonPath(appPath: string, root: string) { - // This is needed for the case where the app is a single-version monorepo and the package.json is in the root of the monorepo - return fs.existsSync(path.join(appPath, "./package.json")) - ? path.join(appPath, "./package.json") - : path.join(root, "./package.json"); -} - function setStandaloneBuildMode(monorepoRoot: string) { // Equivalent to setting `target: "standalone"` in next.config.js process.env.NEXT_PRIVATE_STANDALONE = "true"; @@ -205,7 +189,7 @@ function setStandaloneBuildMode(monorepoRoot: string) { function buildNextjsApp(packager: "npm" | "yarn" | "pnpm" | "bun") { const { nextPackageJsonPath } = options; const command = - options.buildCommand ?? + config.buildCommand ?? (["bun", "npm"].includes(packager) ? `${packager} run build` : `${packager} build`); @@ -251,11 +235,16 @@ function printOpenNextVersion() { function initOutputDir() { const { outputDir, tempDir } = options; + const openNextConfig = readFileSync( + path.join(tempDir, "open-next.config.mjs"), + "utf8", + ); fs.rmSync(outputDir, { recursive: true, force: true }); fs.mkdirSync(tempDir, { recursive: true }); + fs.writeFileSync(path.join(tempDir, "open-next.config.mjs"), openNextConfig); } -function createWarmerBundle() { +async function createWarmerBundle(config: OpenNextConfig) { logger.info(`Bundling warmer function...`); const { outputDir } = options; @@ -264,35 +253,41 @@ function createWarmerBundle() { const outputPath = path.join(outputDir, "warmer-function"); fs.mkdirSync(outputPath, { recursive: true }); + // Copy open-next.config.mjs into the bundle + copyOpenNextConfig(options.tempDir, outputPath); + // Build Lambda code // note: bundle in OpenNext package b/c the adatper relys on the // "serverless-http" package which is not a dependency in user's // Next.js app. - esbuildSync({ - entryPoints: [path.join(__dirname, "adapters", "warmer-function.js")], - external: ["next"], - outfile: path.join(outputPath, "index.mjs"), - banner: { - js: [ - "import { createRequire as topLevelCreateRequire } from 'module';", - "const require = topLevelCreateRequire(import.meta.url);", - "import bannerUrl from 'url';", - "const __dirname = bannerUrl.fileURLToPath(new URL('.', import.meta.url));", - ].join(""), + await esbuildAsync( + { + entryPoints: [path.join(__dirname, "adapters", "warmer-function.js")], + external: ["next"], + outfile: path.join(outputPath, "index.mjs"), + plugins: [ + openNextResolvePlugin({ + overrides: { + converter: config.warmer?.override?.converter ?? "dummy", + wrapper: config.warmer?.override?.wrapper, + }, + fnName: "warmer", + }), + ], + banner: { + js: [ + "import { createRequire as topLevelCreateRequire } from 'module';", + "const require = topLevelCreateRequire(import.meta.url);", + "import bannerUrl from 'url';", + "const __dirname = bannerUrl.fileURLToPath(new URL('.', import.meta.url));", + ].join(""), + }, }, - }); -} - -async function minifyServerBundle() { - logger.info(`Minimizing server function...`); - const { outputDir } = options; - await minifyAll(path.join(outputDir, "server-function"), { - compress_json: true, - mangle: true, - }); + options, + ); } -function createRevalidationBundle() { +async function createRevalidationBundle(config: OpenNextConfig) { logger.info(`Bundling revalidation function...`); const { appBuildOutputPath, outputDir } = options; @@ -301,12 +296,28 @@ function createRevalidationBundle() { const outputPath = path.join(outputDir, "revalidation-function"); fs.mkdirSync(outputPath, { recursive: true }); + //Copy open-next.config.mjs into the bundle + copyOpenNextConfig(options.tempDir, outputPath); + // Build Lambda code - esbuildSync({ - external: ["next", "styled-jsx", "react"], - entryPoints: [path.join(__dirname, "adapters", "revalidate.js")], - outfile: path.join(outputPath, "index.mjs"), - }); + esbuildAsync( + { + external: ["next", "styled-jsx", "react"], + entryPoints: [path.join(__dirname, "adapters", "revalidate.js")], + outfile: path.join(outputPath, "index.mjs"), + plugins: [ + openNextResolvePlugin({ + fnName: "revalidate", + overrides: { + converter: + config.revalidate?.override?.converter ?? "sqs-revalidate", + wrapper: config.revalidate?.override?.wrapper, + }, + }), + ], + }, + options, + ); // Copy over .next/prerender-manifest.json file fs.copyFileSync( @@ -315,7 +326,7 @@ function createRevalidationBundle() { ); } -async function createImageOptimizationBundle() { +async function createImageOptimizationBundle(config: OpenNextConfig) { logger.info(`Bundling image optimization function...`); const { appPath, appBuildOutputPath, outputDir } = options; @@ -324,22 +335,30 @@ async function createImageOptimizationBundle() { const outputPath = path.join(outputDir, "image-optimization-function"); fs.mkdirSync(outputPath, { recursive: true }); - const plugins = - compareSemver(options.nextVersion, "14.1.1") >= 0 - ? [ - openNextPlugin({ - name: "opennext-14.1.1-image-optimization", - target: /plugins\/image-optimization\.js/g, - replacements: ["./image-optimization.replacement.js"], - }), - ] - : undefined; + // Copy open-next.config.mjs into the bundle + copyOpenNextConfig(options.tempDir, outputPath); - if (plugins && plugins.length > 0) { - logger.debug( - `Applying plugins:: [${plugins - .map(({ name }) => name) - .join(",")}] for Next version: ${options.nextVersion}`, + const plugins = [ + openNextResolvePlugin({ + fnName: "imageOptimization", + overrides: { + converter: config.imageOptimization?.override?.converter, + wrapper: config.imageOptimization?.override?.wrapper, + }, + }), + ]; + + if (compareSemver(options.nextVersion, "14.1.1") >= 0) { + plugins.push( + openNextReplacementPlugin({ + name: "opennext-14.1.1-image-optimization", + target: /plugins\/image-optimization\/image-optimization\.js/g, + replacements: [ + require.resolve( + "./adapters/plugins/image-optimization/image-optimization.replacement.js", + ), + ], + }), ); } @@ -347,33 +366,39 @@ async function createImageOptimizationBundle() { // note: bundle in OpenNext package b/c the adapter relies on the // "@aws-sdk/client-s3" package which is not a dependency in user's // Next.js app. - await esbuildAsync({ - entryPoints: [ - path.join(__dirname, "adapters", "image-optimization-adapter.js"), - ], - external: ["sharp", "next"], - outfile: path.join(outputPath, "index.mjs"), - plugins, - }); + await esbuildAsync( + { + entryPoints: [ + path.join(__dirname, "adapters", "image-optimization-adapter.js"), + ], + external: ["sharp", "next"], + outfile: path.join(outputPath, "index.mjs"), + plugins, + }, + options, + ); // Build Lambda code (2nd pass) // note: bundle in user's Next.js app again b/c the adapter relies on the // "next" package. And the "next" package from user's app should // be used. - esbuildSync({ - entryPoints: [path.join(outputPath, "index.mjs")], - external: ["sharp"], - allowOverwrite: true, - outfile: path.join(outputPath, "index.mjs"), - banner: { - js: [ - "import { createRequire as topLevelCreateRequire } from 'module';", - "const require = topLevelCreateRequire(import.meta.url);", - "import bannerUrl from 'url';", - "const __dirname = bannerUrl.fileURLToPath(new URL('.', import.meta.url));", - ].join("\n"), + esbuildSync( + { + entryPoints: [path.join(outputPath, "index.mjs")], + external: ["sharp"], + allowOverwrite: true, + outfile: path.join(outputPath, "index.mjs"), + banner: { + js: [ + "import { createRequire as topLevelCreateRequire } from 'module';", + "const require = topLevelCreateRequire(import.meta.url);", + "import bannerUrl from 'url';", + "const __dirname = bannerUrl.fileURLToPath(new URL('.', import.meta.url));", + ].join("\n"), + }, }, - }); + options, + ); // Copy over .next/required-server-files.json file and BUILD_ID fs.mkdirSync(path.join(outputPath, ".next")); @@ -393,10 +418,14 @@ async function createImageOptimizationBundle() { const nodeOutputPath = path.resolve(outputPath); const sharpVersion = process.env.SHARP_VERSION ?? "0.32.6"; + const arch = config.imageOptimization?.arch ?? "arm64"; + const nodeVersion = config.imageOptimization?.nodeVersion ?? "18"; + //check if we are running in Windows environment then set env variables accordingly. try { cp.execSync( - `npm install --arch=arm64 --platform=linux --target=18 --libc=glibc --prefix="${nodeOutputPath}" sharp@${sharpVersion}`, + // We might want to change the arch args to cpu args, it seems to be the documented way + `npm install --arch=${arch} --platform=linux --target=${nodeVersion} --libc=glibc --prefix="${nodeOutputPath}" sharp@${sharpVersion}`, { stdio: "pipe", cwd: appPath, @@ -452,7 +481,9 @@ function createStaticAssets() { } } -function createCacheAssets(monorepoRoot: string, disableDynamoDBCache = false) { +async function createCacheAssets(monorepoRoot: string) { + if (config.dangerous?.disableIncrementalCache) return; + logger.info(`Bundling cache assets...`); const { appBuildOutputPath, outputDir } = options; @@ -546,7 +577,7 @@ function createCacheAssets(monorepoRoot: string, disableDynamoDBCache = false) { fs.writeFileSync(cacheFilePath, JSON.stringify(cacheFileContent)); }); - if (!disableDynamoDBCache) { + if (!config.dangerous?.disableTagCache) { // Generate dynamodb data // We need to traverse the cache to find every .meta file const metaFiles: { @@ -618,12 +649,28 @@ function createCacheAssets(monorepoRoot: string, disableDynamoDBCache = false) { if (metaFiles.length > 0) { const providerPath = path.join(outputDir, "dynamodb-provider"); - esbuildSync({ - external: ["@aws-sdk/client-dynamodb"], - entryPoints: [path.join(__dirname, "adapters", "dynamo-provider.js")], - outfile: path.join(providerPath, "index.mjs"), - target: ["node18"], - }); + await esbuildAsync( + { + external: ["@aws-sdk/client-dynamodb"], + entryPoints: [path.join(__dirname, "adapters", "dynamo-provider.js")], + outfile: path.join(providerPath, "index.mjs"), + target: ["node18"], + plugins: [ + openNextResolvePlugin({ + fnName: "initializationFunction", + overrides: { + converter: + config.initializationFunction?.override?.converter ?? "dummy", + wrapper: config.initializationFunction?.override?.wrapper, + }, + }), + ], + }, + options, + ); + + //Copy open-next.config.mjs into the bundle + copyOpenNextConfig(options.tempDir, providerPath); // TODO: check if metafiles doesn't contain duplicates fs.writeFileSync( @@ -641,420 +688,78 @@ function createCacheAssets(monorepoRoot: string, disableDynamoDBCache = false) { /* Server Helper Functions */ /***************************/ -async function createServerBundle(monorepoRoot: string, streaming = false) { - logger.info(`Bundling server function...`); - - const { appPath, appBuildOutputPath, outputDir } = options; - - // Create output folder - const outputPath = path.join(outputDir, "server-function"); - fs.mkdirSync(outputPath, { recursive: true }); - - // Resolve path to the Next.js app if inside the monorepo - // note: if user's app is inside a monorepo, standalone mode places - // `node_modules` inside `.next/standalone`, and others inside - // `.next/standalone/package/path` (ie. `.next`, `server.js`). - // We need to output the handler file inside the package path. - const isMonorepo = monorepoRoot !== appPath; - const packagePath = path.relative(monorepoRoot, appBuildOutputPath); - - // Copy over standalone output files - // note: if user uses pnpm as the package manager, node_modules contain - // symlinks. We don't want to resolve the symlinks when copying. - fs.cpSync(path.join(appBuildOutputPath, ".next/standalone"), outputPath, { - recursive: true, - verbatimSymlinks: true, - }); - - // Standalone output already has a Node server "server.js", remove it. - // It will be replaced with the Lambda handler. - fs.rmSync(path.join(outputPath, packagePath, "server.js"), { force: true }); - - // Build Lambda code - // note: bundle in OpenNext package b/c the adapter relies on the - // "serverless-http" package which is not a dependency in user's - // Next.js app. - - let plugins = - compareSemver(options.nextVersion, "13.4.13") >= 0 - ? [ - openNextPlugin({ - name: "opennext-13.4.13-serverHandler", - target: /plugins\/serverHandler\.js/g, - replacements: ["./serverHandler.replacement.js"], - }), - openNextPlugin({ - name: "opennext-13.4.13-util", - target: /plugins\/util\.js/g, - replacements: ["./util.replacement.js"], - }), - openNextPlugin({ - name: "opennext-13.4.13-default", - target: /plugins\/routing\/default\.js/g, - replacements: ["./default.replacement.js"], - }), - ] - : undefined; - - if (compareSemver(options.nextVersion, "13.5.1") >= 0) { - const isAfter141 = compareSemver(options.nextVersion, "14.1.0") >= 0; - const utilReplacement = isAfter141 ? "./14.1/util.js" : "./13.5/util.js"; - plugins = [ - openNextPlugin({ - name: "opennext-13.5-serverHandler", - target: /plugins\/serverHandler\.js/g, - replacements: ["./13.5/serverHandler.js"], - }), - openNextPlugin({ - name: "opennext-13.5-util", - target: /plugins\/util\.js/g, - replacements: [ - utilReplacement, - ...(isAfter141 ? [] : ["./util.replacement.js"]), - ], - }), - openNextPlugin({ - name: "opennext-13.5-default", - target: /plugins\/routing\/default\.js/g, - replacements: ["./default.replacement.js"], - }), - ]; - } - - if (streaming) { - const streamingPlugin = openNextPlugin({ - name: "opennext-streaming", - target: /plugins\/lambdaHandler\.js/g, - replacements: ["./streaming.replacement.js"], - }); - if (plugins) { - plugins.push(streamingPlugin); - } else { - plugins = [streamingPlugin]; - } - } - - if (plugins && plugins.length > 0) { - logger.debug( - `Applying plugins:: [${plugins - .map(({ name }) => name) - .join(",")}] for Next version: ${options.nextVersion}`, - ); - } - await esbuildAsync({ - entryPoints: [path.join(__dirname, "adapters", "server-adapter.js")], - external: ["next"], - outfile: path.join(outputPath, packagePath, "index.mjs"), - banner: { - js: [ - `globalThis.monorepoPackagePath = "${packagePath}";`, - "import { createRequire as topLevelCreateRequire } from 'module';", - "const require = topLevelCreateRequire(import.meta.url);", - "import bannerUrl from 'url';", - "const __dirname = bannerUrl.fileURLToPath(new URL('.', import.meta.url));", - ].join(""), +function compileCache() { + const outfile = path.join(options.outputDir, ".build", "cache.cjs"); + esbuildSync( + { + external: ["next", "styled-jsx", "react", "@aws-sdk/*"], + entryPoints: [path.join(__dirname, "adapters", "cache.js")], + outfile, + target: ["node18"], + format: "cjs", + banner: { + js: [ + `globalThis.disableIncrementalCache = ${ + config.dangerous?.disableIncrementalCache ?? false + };`, + `globalThis.disableDynamoDBCache = ${ + config.dangerous?.disableTagCache ?? false + };`, + ].join(""), + }, }, - plugins, - }); - - if (isMonorepo) { - addMonorepoEntrypoint(outputPath, packagePath); - } - addPublicFilesList(outputPath, packagePath); - injectMiddlewareGeolocation(outputPath, packagePath); - removeCachedPages(outputPath, packagePath); - addCacheHandler(outputPath, options.dangerous); -} - -function addMonorepoEntrypoint(outputPath: string, packagePath: string) { - // Note: in the monorepo case, the handler file is output to - // `.next/standalone/package/path/index.mjs`, but we want - // the Lambda function to be able to find the handler at - // the root of the bundle. We will create a dummy `index.mjs` - // that re-exports the real handler. - - // Always use posix path for import path - const packagePosixPath = packagePath.split(path.sep).join(path.posix.sep); - fs.writeFileSync( - path.join(outputPath, "index.mjs"), - [ - `export const handler = async (event, context) => {`, - ` const fn = await import("./${packagePosixPath}/index.mjs");`, - ` return fn.handler(event, context);`, - `};`, - ].join(""), - ); -} - -function injectMiddlewareGeolocation(outputPath: string, packagePath: string) { - // WORKAROUND: Set `NextRequest` geolocation data — https://github.com/serverless-stack/open-next#workaround-set-nextrequest-geolocation-data - - const basePath = path.join(outputPath, packagePath, ".next", "server"); - const rootMiddlewarePath = path.join(basePath, "middleware.js"); - const srcMiddlewarePath = path.join(basePath, "src", "middleware.js"); - if (fs.existsSync(rootMiddlewarePath)) { - inject(rootMiddlewarePath); - } else if (fs.existsSync(srcMiddlewarePath)) { - inject(srcMiddlewarePath); - } - - function inject(middlewarePath: string) { - const content = fs.readFileSync(middlewarePath, "utf-8"); - fs.writeFileSync( - middlewarePath, - content.replace( - "geo: init.geo || {}", - `geo: init.geo || { - country: this.headers.get("cloudfront-viewer-country"), - countryName: this.headers.get("cloudfront-viewer-country-name"), - region: this.headers.get("cloudfront-viewer-country-region"), - regionName: this.headers.get("cloudfront-viewer-country-region-name"), - city: this.headers.get("cloudfront-viewer-city"), - postalCode: this.headers.get("cloudfront-viewer-postal-code"), - timeZone: this.headers.get("cloudfront-viewer-time-zone"), - latitude: this.headers.get("cloudfront-viewer-latitude"), - longitude: this.headers.get("cloudfront-viewer-longitude"), - metroCode: this.headers.get("cloudfront-viewer-metro-code"), - }`, - ), - ); - } -} - -function addPublicFilesList(outputPath: string, packagePath: string) { - // Get a list of all files in /public - const { appPublicPath } = options; - const acc: PublicFiles = { files: [] }; - - function processDirectory(pathInPublic: string) { - const files = fs.readdirSync(path.join(appPublicPath, pathInPublic), { - withFileTypes: true, - }); - - for (const file of files) { - file.isDirectory() - ? processDirectory(path.join(pathInPublic, file.name)) - : acc.files.push(path.posix.join(pathInPublic, file.name)); - } - } - - if (fs.existsSync(appPublicPath)) { - processDirectory("/"); - } - - // Save the list - const outputOpenNextPath = path.join(outputPath, packagePath, ".open-next"); - fs.mkdirSync(outputOpenNextPath, { recursive: true }); - fs.writeFileSync( - path.join(outputOpenNextPath, "public-files.json"), - JSON.stringify(acc), + options, ); + return outfile; } -function removeCachedPages(outputPath: string, packagePath: string) { - // Pre-rendered pages will be served out from S3 by the cache handler - const dotNextPath = path.join(outputPath, packagePath); - const isFallbackTruePage = /\[.*\]/; - const htmlPages = getHtmlPages(dotNextPath); - [".next/server/pages", ".next/server/app"] - .map((dir) => path.join(dotNextPath, dir)) - .filter(fs.existsSync) - .forEach((dir) => - removeFiles( - dir, - (file) => - file.endsWith(".json") || - file.endsWith(".rsc") || - file.endsWith(".meta") || - (file.endsWith(".html") && - // do not remove static HTML files - !htmlPages.has(file) && - // do not remove HTML files with "[param].html" format - // b/c they are used for "fallback:true" pages - !isFallbackTruePage.test(file)), - ), - ); -} - -function addCacheHandler(outputPath: string, options?: DangerousOptions) { - esbuildSync({ - external: ["next", "styled-jsx", "react"], - entryPoints: [path.join(__dirname, "adapters", "cache.js")], - outfile: path.join(outputPath, "cache.cjs"), - target: ["node18"], - format: "cjs", - banner: { - js: [ - `globalThis.disableIncrementalCache = ${ - options?.disableIncrementalCache ?? false - };`, - `globalThis.disableDynamoDBCache = ${ - options?.disableDynamoDBCache ?? false - };`, - ].join(""), - }, - }); -} - -/********************/ -/* Helper Functions */ -/********************/ - -function esbuildSync(esbuildOptions: ESBuildOptions) { - const { openNextVersion, debug } = options; - const result = buildSync({ - target: "esnext", - format: "esm", - platform: "node", - bundle: true, - minify: debug ? false : true, - sourcemap: debug ? "inline" : false, - ...esbuildOptions, - banner: { - ...esbuildOptions.banner, - js: [ - esbuildOptions.banner?.js || "", - `globalThis.openNextDebug = ${debug};`, - `globalThis.openNextVersion = "${openNextVersion}";`, - ].join(""), - }, - }); +async function createMiddleware() { + console.info(`Bundling middleware function...`); - if (result.errors.length > 0) { - result.errors.forEach((error) => logger.error(error)); - throw new Error( - `There was a problem bundling ${ - (esbuildOptions.entryPoints as string[])[0] - }.`, - ); - } -} - -async function esbuildAsync(esbuildOptions: ESBuildOptions) { - const { openNextVersion, debug } = options; - const result = await buildAsync({ - target: "esnext", - format: "esm", - platform: "node", - bundle: true, - minify: debug ? false : true, - sourcemap: debug ? "inline" : false, - ...esbuildOptions, - banner: { - ...esbuildOptions.banner, - js: [ - esbuildOptions.banner?.js || "", - `globalThis.openNextDebug = ${debug};`, - `globalThis.openNextVersion = "${openNextVersion}";`, - ].join(""), - }, - }); + const { appBuildOutputPath, outputDir } = options; - if (result.errors.length > 0) { - result.errors.forEach((error) => logger.error(error)); - throw new Error( - `There was a problem bundling ${ - (esbuildOptions.entryPoints as string[])[0] - }.`, - ); + // Get middleware manifest + const middlewareManifest = JSON.parse( + readFileSync( + path.join(appBuildOutputPath, ".next/server/middleware-manifest.json"), + "utf8", + ), + ) as MiddlewareManifest; + + const entry = middlewareManifest.middleware["/"]; + if (!entry) { + return; } -} - -function removeFiles( - root: string, - conditionFn: (file: string) => boolean, - searchingDir: string = "", -) { - traverseFiles( - root, - conditionFn, - (filePath) => fs.rmSync(filePath, { force: true }), - searchingDir, - ); -} - -function traverseFiles( - root: string, - conditionFn: (file: string) => boolean, - callbackFn: (filePath: string) => void, - searchingDir: string = "", -) { - fs.readdirSync(path.join(root, searchingDir)).forEach((file) => { - const filePath = path.join(root, searchingDir, file); - - if (fs.statSync(filePath).isDirectory()) { - traverseFiles( - root, - conditionFn, - callbackFn, - path.join(searchingDir, file), - ); - return; - } - - if (conditionFn(path.join(searchingDir, file))) { - callbackFn(filePath); - } - }); -} - -function getHtmlPages(dotNextPath: string) { - // Get a list of HTML pages - // - // sample return value: - // Set([ - // '404.html', - // 'csr.html', - // 'image-html-tag.html', - // ]) - const manifestPath = path.join( - dotNextPath, - ".next/server/pages-manifest.json", - ); - const manifest = fs.readFileSync(manifestPath, "utf-8"); - return Object.entries(JSON.parse(manifest)) - .filter(([_, value]) => (value as string).endsWith(".html")) - .map(([_, value]) => (value as string).replace(/^pages\//, "")) - .reduce((acc, page) => { - acc.add(page); - return acc; - }, new Set()); -} - -function getBuildId(dotNextPath: string) { - return fs - .readFileSync(path.join(dotNextPath, ".next/BUILD_ID"), "utf-8") - .trim(); -} -function getOpenNextVersion() { - return require(path.join(__dirname, "../package.json")).version; -} + // Create output folder + let outputPath = path.join(outputDir, "server-function"); -function getNextVersion(nextPackageJsonPath: string) { - const version = require(nextPackageJsonPath)?.dependencies?.next; - // require('next/package.json').version + const commonMiddlewareOptions = { + middlewareInfo: entry, + options, + appBuildOutputPath, + }; - if (!version) { - throw new Error("Failed to find Next version"); - } + if (config.middleware?.external) { + outputPath = path.join(outputDir, "middleware"); + fs.mkdirSync(outputPath, { recursive: true }); - // Drop the -canary.n suffix - return version.split("-")[0]; -} + // Copy open-next.config.mjs + copyOpenNextConfig(options.tempDir, outputPath); -function compareSemver(v1: string, v2: string): number { - if (v1 === "latest") return 1; - if (/^[^\d]/.test(v1)) { - v1 = v1.substring(1); - } - if (/^[^\d]/.test(v2)) { - v2 = v2.substring(1); + // Bundle middleware + await buildEdgeBundle({ + entrypoint: path.join(__dirname, "adapters", "middleware.js"), + outfile: path.join(outputPath, "handler.mjs"), + ...commonMiddlewareOptions, + overrides: config.middleware?.override, + defaultConverter: "aws-cloudfront", + }); + } else { + await buildEdgeBundle({ + entrypoint: path.join(__dirname, "core", "edgeFunctionHandler.js"), + outfile: path.join(outputDir, ".build", "middleware.mjs"), + ...commonMiddlewareOptions, + }); } - const [major1, minor1, patch1] = v1.split(".").map(Number); - const [major2, minor2, patch2] = v2.split(".").map(Number); - - if (major1 !== major2) return major1 - major2; - if (minor1 !== minor2) return minor1 - minor2; - return patch1 - patch2; } diff --git a/packages/open-next/src/build/bundleNextServer.ts b/packages/open-next/src/build/bundleNextServer.ts new file mode 100644 index 00000000..5c3f5f8f --- /dev/null +++ b/packages/open-next/src/build/bundleNextServer.ts @@ -0,0 +1,104 @@ +import { createRequire as topLevelCreateRequire } from "node:module"; + +import { build } from "esbuild"; +import path from "path"; + +const externals = [ + // This one was causing trouble, don't know why + "../experimental/testmode/server", + + // sharedExternals + "styled-jsx", + "styled-jsx/style", + "@opentelemetry/api", + "next/dist/compiled/@next/react-dev-overlay/dist/middleware", + "next/dist/compiled/@ampproject/toolbox-optimizer", + "next/dist/compiled/edge-runtime", + "next/dist/compiled/@edge-runtime/ponyfill", + "next/dist/compiled/undici", + "next/dist/compiled/raw-body", + "next/dist/server/capsize-font-metrics.json", + "critters", + "next/dist/compiled/node-html-parser", + "next/dist/compiled/compression", + "next/dist/compiled/jsonwebtoken", + "next/dist/compiled/@opentelemetry/api", + "next/dist/compiled/@mswjs/interceptors/ClientRequest", + "next/dist/compiled/ws", + + // externalsMap + // In the config they replace it, but we don't use this one inside NextServer anymore 13.4.12+ + // For earlier versions we might have to alias it + "./web/sandbox", + + // pagesExternal + "react", + "react-dom", + "react-server-dom-webpack", + "react-server-dom-turbopack", + + // We need to remove this since this is what webpack is building + // Adding it cause to add a lot of unnecessary deps + "next/dist/compiled/next-server", +]; + +export async function bundleNextServer(outputDir: string, appPath: string) { + const require = topLevelCreateRequire(`${appPath}/package.json`); + const entrypoint = require.resolve("next/dist/esm/server/next-server.js"); + + await build({ + entryPoints: [entrypoint], + bundle: true, + platform: "node", + target: ["node18"], + // packages: "external", + format: "cjs", + external: externals, + minify: true, + outfile: path.join(outputDir, "next-server.runtime.prod.js"), + sourcemap: false, + plugins: [ + { + name: "opennext-next-server", + setup(build) { + // This was an attempt at reducing server bundle size + // It might be the better way to go in the future + build.onResolve({ filter: /\.\/module.compiled/ }, (args) => { + const dir = args.resolveDir.split("/").slice(-1); + return { + path: path.join( + "next/dist/compiled/next-server/", + `${dir}.runtime.prod.js`, + ), + external: true, + }; + }); + + build.onResolve({ filter: /[\\/]react-server\.node/ }, (args) => { + return { + path: args.path, + external: true, + }; + }); + + build.onResolve( + { filter: /vendored[\\/]rsc[\\/]entrypoints/ }, + (args) => { + return { + path: args.path, + external: true, + }; + }, + ); + + build.onResolve({ filter: /\.external/ }, (args) => { + return { + path: args.path.replace(/\.\./, "next/dist"), + external: true, + }; + }); + }, + }, + ], + }); +} diff --git a/packages/open-next/src/build/copyTracedFiles.ts b/packages/open-next/src/build/copyTracedFiles.ts new file mode 100644 index 00000000..bf7f2fe0 --- /dev/null +++ b/packages/open-next/src/build/copyTracedFiles.ts @@ -0,0 +1,269 @@ +import { + copyFileSync, + existsSync, + mkdirSync, + readdirSync, + readFileSync, + readlinkSync, + statSync, + symlinkSync, + writeFileSync, +} from "fs"; +import path from "path"; +import { NextConfig, PrerenderManifest } from "types/next-types"; + +import logger from "../logger.js"; + +export async function copyTracedFiles( + buildOutputPath: string, + packagePath: string, + outputDir: string, + routes: string[], + bundledNextServer: boolean, +) { + const tsStart = Date.now(); + const dotNextDir = path.join(buildOutputPath, ".next"); + const standaloneDir = path.join(dotNextDir, "standalone"); + const standaloneNextDir = path.join(standaloneDir, packagePath, ".next"); + const outputNextDir = path.join(outputDir, packagePath, ".next"); + + const extractFiles = (files: string[], from = standaloneNextDir) => { + return files.map((f) => path.resolve(from, f)); + }; + + // On next 14+, we might not have to include those files + // For next 13, we need to include them otherwise we get runtime error + const requiredServerFiles = JSON.parse( + readFileSync( + path.join( + dotNextDir, + bundledNextServer + ? "next-minimal-server.js.nft.json" + : "next-server.js.nft.json", + ), + "utf8", + ), + ); + + const filesToCopy = new Map(); + + // Files necessary by the server + extractFiles(requiredServerFiles.files).forEach((f) => { + filesToCopy.set(f, f.replace(standaloneDir, outputDir)); + }); + + // create directory for pages + if (existsSync(path.join(standaloneDir, ".next/server/pages"))) { + mkdirSync(path.join(outputNextDir, "server/pages"), { + recursive: true, + }); + } + if (existsSync(path.join(standaloneDir, ".next/server/app"))) { + mkdirSync(path.join(outputNextDir, "server/app"), { + recursive: true, + }); + } + + mkdirSync(path.join(outputNextDir, "server/chunks"), { + recursive: true, + }); + + const computeCopyFilesForPage = (pagePath: string) => { + const fullFilePath = `server/${pagePath}.js`; + let requiredFiles; + try { + requiredFiles = JSON.parse( + readFileSync( + path.join(standaloneNextDir, `${fullFilePath}.nft.json`), + "utf8", + ), + ); + } catch (e) { + //TODO: add a link to the docs + throw new Error( + ` +-------------------------------------------------------------------------------- +${pagePath} cannot use the edge runtime. +OpenNext requires edge runtime function to be defined in a separate function. +See the docs for more information on how to bundle edge runtime functions. +-------------------------------------------------------------------------------- + `, + ); + } + const dir = path.dirname(fullFilePath); + extractFiles( + requiredFiles.files, + path.join(standaloneNextDir, dir), + ).forEach((f) => { + filesToCopy.set(f, f.replace(standaloneDir, outputDir)); + }); + + filesToCopy.set( + path.join(standaloneNextDir, fullFilePath), + path.join(outputNextDir, fullFilePath), + ); + }; + + const safeComputeCopyFilesForPage = ( + pagePath: string, + alternativePath?: string, + ) => { + try { + computeCopyFilesForPage(pagePath); + } catch (e) { + if (alternativePath) { + computeCopyFilesForPage(alternativePath); + } + } + }; + + const hasPageDir = routes.some((route) => route.startsWith("pages/")); + const hasAppDir = routes.some((route) => route.startsWith("app/")); + + // We need to copy all the base files like _app, _document, _error, etc + // One thing to note, is that next try to load every routes that might be needed in advance + // So if you have a [slug].tsx at the root, this route will always be loaded for 1st level request + // along with _app and _document + if (hasPageDir) { + //Page dir + computeCopyFilesForPage("pages/_app"); + computeCopyFilesForPage("pages/_document"); + computeCopyFilesForPage("pages/_error"); + + // These files can be present or not depending on if the user uses getStaticProps + safeComputeCopyFilesForPage("pages/404"); + safeComputeCopyFilesForPage("pages/500"); + } + + if (hasAppDir) { + //App dir + // In next 14.2.0, _not-found is at 'app/_not-found/page' + safeComputeCopyFilesForPage("app/_not-found", "app/_not-found/page"); + } + + //Files we actually want to include + routes.forEach((route) => { + computeCopyFilesForPage(route); + }); + + //Actually copy the files + filesToCopy.forEach((to, from) => { + if ( + from.includes("node_modules") && + //TODO: we need to figure which packages we could safely remove + (from.includes("caniuse-lite") || + // from.includes("jest-worker") || This ones seems necessary for next 12 + from.includes("sharp")) + ) { + return; + } + mkdirSync(path.dirname(to), { recursive: true }); + let symlink = null; + // For pnpm symlink we need to do that + // see https://github.com/vercel/next.js/blob/498f342b3552d6fc6f1566a1cc5acea324ce0dec/packages/next/src/build/utils.ts#L1932 + try { + symlink = readlinkSync(from); + } catch (e) { + //Ignore + } + if (symlink) { + try { + symlinkSync(symlink, to); + } catch (e: any) { + if (e.code !== "EEXIST") { + throw e; + } + } + } else { + copyFileSync(from, to); + } + }); + + readdirSync(standaloneNextDir).forEach((f) => { + if (statSync(path.join(standaloneNextDir, f)).isDirectory()) return; + copyFileSync(path.join(standaloneNextDir, f), path.join(outputNextDir, f)); + }); + + // We then need to copy all the files at the root of server + + mkdirSync(path.join(outputNextDir, "server"), { recursive: true }); + + readdirSync(path.join(standaloneNextDir, "server")).forEach((f) => { + if (statSync(path.join(standaloneNextDir, "server", f)).isDirectory()) + return; + if (f !== "server.js") { + copyFileSync( + path.join(standaloneNextDir, "server", f), + path.join(path.join(outputNextDir, "server"), f), + ); + } + }); + + // TODO: Recompute all the files. + // vercel doesn't seem to do it, but it seems wasteful to have all those files + // we replace the pages-manifest.json with an empty one if we don't have a pages dir so that + // next doesn't try to load _app, _document + if (!hasPageDir) { + writeFileSync(path.join(outputNextDir, "server/pages-manifest.json"), "{}"); + } + + //TODO: Find what else we need to copy + const copyStaticFile = (filePath: string) => { + if (existsSync(path.join(standaloneNextDir, filePath))) { + mkdirSync(path.dirname(path.join(outputNextDir, filePath)), { + recursive: true, + }); + copyFileSync( + path.join(standaloneNextDir, filePath), + path.join(outputNextDir, filePath), + ); + } + }; + // Get all the static files - Should be only for pages dir + // Ideally we would filter only those that might get accessed in this specific functions + // Maybe even move this to s3 directly + if (hasPageDir) { + // First we get truly static files - i.e. pages without getStaticProps + const staticFiles: Array = Object.values( + JSON.parse( + readFileSync( + path.join(standaloneNextDir, "server/pages-manifest.json"), + "utf8", + ), + ), + ); + // Then we need to get all fallback: true dynamic routes html + const prerenderManifest = JSON.parse( + readFileSync( + path.join(standaloneNextDir, "prerender-manifest.json"), + "utf8", + ), + ) as PrerenderManifest; + const config = JSON.parse( + readFileSync( + path.join(standaloneNextDir, "required-server-files.json"), + "utf8", + ), + ).config as NextConfig; + const locales = config.i18n?.locales; + Object.values(prerenderManifest.dynamicRoutes).forEach((route) => { + if (typeof route.fallback === "string") { + if (locales) { + locales.forEach((locale) => { + staticFiles.push(`pages/${locale}${route.fallback}`); + }); + } else { + staticFiles.push(`pages${route.fallback}`); + } + } + }); + + staticFiles.forEach((f: string) => { + if (f.endsWith(".html")) { + copyStaticFile(`server/${f}`); + } + }); + } + + logger.debug("copyTracedFiles:", Date.now() - tsStart, "ms"); +} diff --git a/packages/open-next/src/build/createServerBundle.ts b/packages/open-next/src/build/createServerBundle.ts new file mode 100644 index 00000000..e1e0474c --- /dev/null +++ b/packages/open-next/src/build/createServerBundle.ts @@ -0,0 +1,306 @@ +import { existsSync } from "node:fs"; +import { createRequire as topLevelCreateRequire } from "node:module"; + +import fs from "fs"; +import path from "path"; +import { + FunctionOptions, + OpenNextConfig, + SplittedFunctionOptions, +} from "types/open-next"; +import url from "url"; + +import logger from "../logger.js"; +import { minifyAll } from "../minimize-js.js"; +import { openNextReplacementPlugin } from "../plugins/replacement.js"; +import { openNextResolvePlugin } from "../plugins/resolve.js"; +import { bundleNextServer } from "./bundleNextServer.js"; +import { copyTracedFiles } from "./copyTracedFiles.js"; +import { generateEdgeBundle } from "./edge/createEdgeBundle.js"; +import type { BuildOptions } from "./helper.js"; +import { + compareSemver, + copyOpenNextConfig, + esbuildAsync, + traverseFiles, +} from "./helper.js"; + +const require = topLevelCreateRequire(import.meta.url); +const __dirname = url.fileURLToPath(new URL(".", import.meta.url)); + +export async function createServerBundle( + config: OpenNextConfig, + options: BuildOptions, +) { + const foundRoutes = new Set(); + // Get all functions to build + const defaultFn = config.default; + const functions = Object.entries(config.functions ?? {}); + + const promises = functions.map(async ([name, fnOptions]) => { + const routes = fnOptions.routes; + routes.forEach((route) => foundRoutes.add(route)); + if (fnOptions.runtime === "edge") { + await generateEdgeBundle(name, options, fnOptions); + } else { + await generateBundle(name, config, options, fnOptions); + } + }); + + //TODO: throw an error if not all edge runtime routes has been bundled in a separate function + + // We build every other function than default before so we know which route there is left + await Promise.all(promises); + + const remainingRoutes = new Set(); + + const { monorepoRoot, appBuildOutputPath } = options; + + const packagePath = path.relative(monorepoRoot, appBuildOutputPath); + + // Find remaining routes + const serverPath = path.join( + appBuildOutputPath, + ".next", + "standalone", + packagePath, + ".next", + "server", + ); + + // Find app dir routes + if (existsSync(path.join(serverPath, "app"))) { + const appPath = path.join(serverPath, "app"); + traverseFiles( + appPath, + (file) => { + if (file.endsWith("page.js") || file.endsWith("route.js")) { + const route = `app/${file.replace(/\.js$/, "")}`; + if (!foundRoutes.has(route)) { + remainingRoutes.add(route); + } + } + return false; + }, + () => {}, + ); + } + + // Find pages dir routes + if (existsSync(path.join(serverPath, "pages"))) { + const pagePath = path.join(serverPath, "pages"); + traverseFiles( + pagePath, + (file) => { + if (file.endsWith(".js")) { + const route = `pages/${file.replace(/\.js$/, "")}`; + if (!foundRoutes.has(route)) { + remainingRoutes.add(route); + } + } + return false; + }, + () => {}, + ); + } + + // Generate default function + await generateBundle("default", config, options, { + ...defaultFn, + // @ts-expect-error - Those string are RouteTemplate + routes: Array.from(remainingRoutes), + patterns: ["*"], + }); +} + +async function generateBundle( + name: string, + config: OpenNextConfig, + options: BuildOptions, + fnOptions: SplittedFunctionOptions, +) { + const { appPath, appBuildOutputPath, outputDir, monorepoRoot } = options; + logger.info(`Building server function: ${name}...`); + + // Create output folder + const outputPath = path.join(outputDir, "server-functions", name); + + // Resolve path to the Next.js app if inside the monorepo + // note: if user's app is inside a monorepo, standalone mode places + // `node_modules` inside `.next/standalone`, and others inside + // `.next/standalone/package/path` (ie. `.next`, `server.js`). + // We need to output the handler file inside the package path. + const isMonorepo = monorepoRoot !== appPath; + const packagePath = path.relative(monorepoRoot, appBuildOutputPath); + fs.mkdirSync(path.join(outputPath, packagePath), { recursive: true }); + + fs.copyFileSync( + path.join(outputDir, ".build", "cache.cjs"), + path.join(outputPath, packagePath, "cache.cjs"), + ); + + // Bundle next server if necessary + const isBundled = fnOptions.experimentalBundledNextServer ?? false; + if (isBundled) { + bundleNextServer(path.join(outputPath, packagePath), appPath); + } + + // // Copy middleware + if ( + !config.middleware?.external && + existsSync(path.join(outputDir, ".build", "middleware.mjs")) + ) { + fs.copyFileSync( + path.join(outputDir, ".build", "middleware.mjs"), + path.join(outputPath, packagePath, "middleware.mjs"), + ); + } + + // Copy open-next.config.mjs + copyOpenNextConfig( + path.join(outputDir, ".build"), + path.join(outputPath, packagePath), + ); + + // Copy all necessary traced files + copyTracedFiles( + appBuildOutputPath, + packagePath, + outputPath, + fnOptions.routes ?? ["app/page.tsx"], + isBundled, + ); + + // Build Lambda code + // note: bundle in OpenNext package b/c the adapter relies on the + // "serverless-http" package which is not a dependency in user's + // Next.js app. + + const disableNextPrebundledReact = + compareSemver(options.nextVersion, "13.5.1") >= 0 || + compareSemver(options.nextVersion, "13.4.1") <= 0; + + const overrides = fnOptions.override ?? {}; + + const isBefore13413 = compareSemver(options.nextVersion, "13.4.13") <= 0; + const isAfter141 = compareSemver(options.nextVersion, "14.0.4") >= 0; + + const disableRouting = isBefore13413 || config.middleware?.external; + const plugins = [ + openNextReplacementPlugin({ + name: `requestHandlerOverride ${name}`, + target: /core\/requestHandler.js/g, + deletes: disableNextPrebundledReact ? ["applyNextjsPrebundledReact"] : [], + replacements: disableRouting + ? [ + require.resolve( + "../adapters/plugins/without-routing/requestHandler.js", + ), + ] + : [], + }), + openNextReplacementPlugin({ + name: `utilOverride ${name}`, + target: /core\/util.js/g, + deletes: [ + ...(disableNextPrebundledReact ? ["requireHooks"] : []), + ...(disableRouting ? ["trustHostHeader"] : []), + ...(!isBefore13413 ? ["requestHandlerHost"] : []), + ...(!isAfter141 ? ["stableIncrementalCache"] : []), + ...(isAfter141 ? ["experimentalIncrementalCacheHandler"] : []), + ], + }), + + openNextResolvePlugin({ + fnName: name, + overrides: { + converter: overrides.converter, + wrapper: overrides.wrapper, + }, + }), + ]; + + if (plugins && plugins.length > 0) { + logger.debug( + `Applying plugins:: [${plugins + .map(({ name }) => name) + .join(",")}] for Next version: ${options.nextVersion}`, + ); + } + await esbuildAsync( + { + entryPoints: [path.join(__dirname, "../adapters", "server-adapter.js")], + external: ["next", "./middleware.mjs", "./next-server.runtime.prod.js"], + outfile: path.join(outputPath, packagePath, "index.mjs"), + banner: { + js: [ + `globalThis.monorepoPackagePath = "${packagePath}";`, + "import { createRequire as topLevelCreateRequire } from 'module';", + "const require = topLevelCreateRequire(import.meta.url);", + "import bannerUrl from 'url';", + "const __dirname = bannerUrl.fileURLToPath(new URL('.', import.meta.url));", + ].join(""), + }, + plugins, + alias: { + "next/dist/server/next-server.js": isBundled + ? "./next-server.runtime.prod.js" + : "next/dist/server/next-server.js", + }, + }, + options, + ); + + if (isMonorepo) { + addMonorepoEntrypoint(outputPath, packagePath); + } + + if (fnOptions.minify) { + await minifyServerBundle(outputPath); + } + + const shouldGenerateDocker = shouldGenerateDockerfile(fnOptions); + if (shouldGenerateDocker) { + fs.writeFileSync( + path.join(outputPath, "Dockerfile"), + typeof shouldGenerateDocker === "string" + ? shouldGenerateDocker + : ` +FROM node:18-alpine +WORKDIR /app +COPY . /app +EXPOSE 3000 +CMD ["node", "index.mjs"] + `, + ); + } +} + +function shouldGenerateDockerfile(options: FunctionOptions) { + return options.override?.generateDockerfile ?? false; +} + +//TODO: check if this PR is still necessary https://github.com/sst/open-next/pull/341 +function addMonorepoEntrypoint(outputPath: string, packagePath: string) { + // Note: in the monorepo case, the handler file is output to + // `.next/standalone/package/path/index.mjs`, but we want + // the Lambda function to be able to find the handler at + // the root of the bundle. We will create a dummy `index.mjs` + // that re-exports the real handler. + + // Always use posix path for import path + const packagePosixPath = packagePath.split(path.sep).join(path.posix.sep); + fs.writeFileSync( + path.join(outputPath, "index.mjs"), + [`export * from "./${packagePosixPath}/index.mjs";`].join(""), + ); +} + +async function minifyServerBundle(outputDir: string) { + logger.info(`Minimizing server function...`); + + await minifyAll(outputDir, { + compress_json: true, + mangle: true, + }); +} diff --git a/packages/open-next/src/build/edge/createEdgeBundle.ts b/packages/open-next/src/build/edge/createEdgeBundle.ts new file mode 100644 index 00000000..03e6af27 --- /dev/null +++ b/packages/open-next/src/build/edge/createEdgeBundle.ts @@ -0,0 +1,165 @@ +import { mkdirSync } from "node:fs"; +import url from "node:url"; + +import fs from "fs"; +import path from "path"; +import { MiddlewareInfo, MiddlewareManifest } from "types/next-types"; +import { + DefaultOverrideOptions, + IncludedConverter, + RouteTemplate, + SplittedFunctionOptions, +} from "types/open-next"; + +import logger from "../../logger.js"; +import { openNextEdgePlugins } from "../../plugins/edge.js"; +import { openNextResolvePlugin } from "../../plugins/resolve.js"; +import { BuildOptions, copyOpenNextConfig, esbuildAsync } from "../helper.js"; + +const __dirname = url.fileURLToPath(new URL(".", import.meta.url)); + +interface BuildEdgeBundleOptions { + appBuildOutputPath: string; + middlewareInfo: MiddlewareInfo; + entrypoint: string; + outfile: string; + options: BuildOptions; + overrides?: DefaultOverrideOptions; + defaultConverter?: IncludedConverter; + additionalInject?: string; +} + +export async function buildEdgeBundle({ + appBuildOutputPath, + middlewareInfo, + entrypoint, + outfile, + options, + defaultConverter, + overrides, + additionalInject, +}: BuildEdgeBundleOptions) { + await esbuildAsync( + { + entryPoints: [entrypoint], + // inject: , + bundle: true, + outfile, + external: ["node:*", "next", "@aws-sdk/*"], + target: "es2022", + platform: "neutral", + plugins: [ + openNextResolvePlugin({ + overrides: { + wrapper: + typeof overrides?.wrapper === "string" + ? overrides.wrapper + : "aws-lambda", + converter: + typeof overrides?.converter === "string" + ? overrides.converter + : defaultConverter, + }, + }), + openNextEdgePlugins({ + middlewareInfo, + nextDir: path.join(appBuildOutputPath, ".next"), + edgeFunctionHandlerPath: path.join( + __dirname, + "../../core", + "edgeFunctionHandler.js", + ), + isInCloudfare: overrides?.wrapper === "cloudflare", + }), + ], + treeShaking: true, + alias: { + path: "node:path", + stream: "node:stream", + fs: "node:fs", + }, + conditions: ["module"], + mainFields: ["module", "main"], + banner: { + js: ` + ${ + overrides?.wrapper === "cloudflare" + ? "" + : ` + const require = (await import("node:module")).createRequire(import.meta.url); + const __filename = (await import("node:url")).fileURLToPath(import.meta.url); + const __dirname = (await import("node:path")).dirname(__filename); + ` + } + ${additionalInject ?? ""} + `, + }, + }, + options, + ); +} + +export function copyMiddlewareAssetsAndWasm({}) {} + +export async function generateEdgeBundle( + name: string, + options: BuildOptions, + fnOptions: SplittedFunctionOptions, +) { + const { appBuildOutputPath, outputDir } = options; + logger.info(`Generating edge bundle for: ${name}`); + + // Create output folder + const outputPath = path.join(outputDir, "server-functions", name); + fs.mkdirSync(outputPath, { recursive: true }); + + // Copy open-next.config.mjs + copyOpenNextConfig(path.join(outputDir, ".build"), outputPath); + + // Load middleware manifest + const middlewareManifest = JSON.parse( + fs.readFileSync( + path.join(appBuildOutputPath, ".next/server/middleware-manifest.json"), + "utf8", + ), + ) as MiddlewareManifest; + + // Find functions + const functions = Object.values(middlewareManifest.functions).filter((fn) => + fnOptions.routes.includes(fn.name as RouteTemplate), + ); + + if (functions.length > 1) { + throw new Error("Only one function is supported for now"); + } + const fn = functions[0]; + + //Copy wasm files + const wasmFiles = fn.wasm; + mkdirSync(path.join(outputPath, "wasm"), { recursive: true }); + for (const wasmFile of wasmFiles) { + fs.copyFileSync( + path.join(appBuildOutputPath, ".next", wasmFile.filePath), + path.join(outputPath, `wasm/${wasmFile.name}.wasm`), + ); + } + + // Copy assets + const assets = fn.assets; + mkdirSync(path.join(outputPath, "assets"), { recursive: true }); + for (const asset of assets) { + fs.copyFileSync( + path.join(appBuildOutputPath, ".next", asset.filePath), + path.join(outputPath, `assets/${asset.name}`), + ); + } + + await buildEdgeBundle({ + appBuildOutputPath, + middlewareInfo: fn, + entrypoint: path.join(__dirname, "../../adapters", "edge-adapter.js"), + outfile: path.join(outputPath, "index.mjs"), + options, + overrides: fnOptions.override, + }); +} diff --git a/packages/open-next/src/build/generateOutput.ts b/packages/open-next/src/build/generateOutput.ts new file mode 100644 index 00000000..7247c7f9 --- /dev/null +++ b/packages/open-next/src/build/generateOutput.ts @@ -0,0 +1,373 @@ +import * as fs from "node:fs"; +import path from "node:path"; + +import { NextConfig } from "types/next-types.js"; +import { + BaseOverride, + DefaultOverrideOptions, + FunctionOptions, + LazyLoadedOverride, + OpenNextConfig, + OverrideOptions, +} from "types/open-next"; + +import { getBuildId } from "./helper.js"; + +type BaseFunction = { + handler: string; + bundle: string; +}; + +type OpenNextFunctionOrigin = { + type: "function"; + streaming?: boolean; + wrapper: string; + converter: string; +} & BaseFunction; + +type OpenNextECSOrigin = { + type: "ecs"; + bundle: string; + wrapper: string; + converter: string; + dockerfile: string; +}; + +type CommonOverride = { + queue: string; + incrementalCache: string; + tagCache: string; +}; + +type OpenNextServerFunctionOrigin = OpenNextFunctionOrigin & CommonOverride; +type OpenNextServerECSOrigin = OpenNextECSOrigin & CommonOverride; + +type OpenNextS3Origin = { + type: "s3"; + originPath: string; + copy: { + from: string; + to: string; + cached: boolean; + versionedSubDir?: string; + }[]; +}; + +type OpenNextOrigins = + | OpenNextServerFunctionOrigin + | OpenNextServerECSOrigin + | OpenNextS3Origin; + +type ImageFnOrigins = OpenNextFunctionOrigin & { imageLoader: string }; +type ImageECSOrigins = OpenNextECSOrigin & { imageLoader: string }; + +type ImageOrigins = ImageFnOrigins | ImageECSOrigins; + +type DefaultOrigins = { + s3: OpenNextS3Origin; + default: OpenNextServerFunctionOrigin | OpenNextServerECSOrigin; + imageOptimizer: ImageOrigins; +}; + +interface OpenNextOutput { + edgeFunctions: { + [key: string]: BaseFunction; + } & { + middleware?: BaseFunction & { pathResolver: string }; + }; + origins: DefaultOrigins & { + [key: string]: OpenNextOrigins; + }; + behaviors: { + pattern: string; + origin?: string; + edgeFunction?: string; + }[]; + additionalProps?: { + disableIncrementalCache?: boolean; + disableTagCache?: boolean; + initializationFunction?: BaseFunction; + warmer?: BaseFunction; + revalidationFunction?: BaseFunction; + }; +} + +async function canStream(opts: FunctionOptions) { + if (!opts.override?.wrapper) { + return false; + } else { + if (typeof opts.override.wrapper === "string") { + return opts.override.wrapper === "aws-lambda-streaming"; + } else { + const wrapper = await opts.override.wrapper(); + return wrapper.supportStreaming; + } + } +} + +async function extractOverrideName( + defaultName: string, + override?: LazyLoadedOverride | string, +) { + if (!override) { + return defaultName; + } + if (typeof override === "string") { + return override; + } else { + const overrideModule = await override(); + return overrideModule.name; + } +} + +async function extractOverrideFn(override?: DefaultOverrideOptions) { + if (!override) { + return { + wrapper: "aws-lambda", + converter: "aws-apigw-v2", + }; + } + const wrapper = await extractOverrideName("aws-lambda", override.wrapper); + const converter = await extractOverrideName( + "aws-apigw-v2", + override.converter, + ); + return { wrapper, converter }; +} + +async function extractCommonOverride(override?: OverrideOptions) { + if (!override) { + return { + queue: "sqs", + incrementalCache: "s3", + tagCache: "dynamodb", + }; + } + const queue = await extractOverrideName("sqs", override.queue); + const incrementalCache = await extractOverrideName( + "s3", + override.incrementalCache, + ); + const tagCache = await extractOverrideName("dynamodb", override.tagCache); + return { queue, incrementalCache, tagCache }; +} + +function prefixPattern(basePath: string) { + // Prefix CloudFront distribution behavior path patterns with `basePath` if configured + return (pattern: string) => { + return basePath && basePath.length > 0 + ? `${basePath.slice(1)}/${pattern}` + : pattern; + }; +} + +export async function generateOutput( + appPath: string, + outputPath: string, + config: OpenNextConfig, +) { + const edgeFunctions: OpenNextOutput["edgeFunctions"] = {}; + const isExternalMiddleware = config.middleware?.external ?? false; + if (isExternalMiddleware) { + edgeFunctions.middleware = { + bundle: ".open-next/middleware", + handler: "handler.handler", + pathResolver: await extractOverrideName( + "pattern-env", + config.middleware!.originResolver, + ), + ...(await extractOverrideFn(config.middleware?.override)), + }; + } + // Add edge functions + Object.entries(config.functions ?? {}).forEach(async ([key, value]) => { + if (value.placement === "global") { + edgeFunctions[key] = { + bundle: `.open-next/functions/${key}`, + handler: "index.handler", + ...(await extractOverrideFn(value.override)), + }; + } + }); + + const defaultOriginCanstream = await canStream(config.default); + + //Load required-server-files.json + const requiredServerFiles = JSON.parse( + fs.readFileSync( + path.join(appPath, ".next", "required-server-files.json"), + "utf-8", + ), + ).config as NextConfig; + const prefixer = prefixPattern(requiredServerFiles.basePath ?? ""); + + // First add s3 origins and image optimization + + const defaultOrigins: DefaultOrigins = { + s3: { + type: "s3", + originPath: "_assets", + copy: [ + { + from: ".open-next/assets", + to: requiredServerFiles.basePath + ? `_assets${requiredServerFiles.basePath}` + : "_assets", + cached: true, + versionedSubDir: prefixer("_next"), + }, + ...(config.dangerous?.disableIncrementalCache + ? [] + : [ + { + from: ".open-next/cache", + to: "_cache", + cached: false, + }, + ]), + ], + }, + imageOptimizer: { + type: "function", + handler: "index.handler", + bundle: ".open-next/image-optimization-function", + streaming: false, + imageLoader: await extractOverrideName( + "s3", + config.imageOptimization?.loader, + ), + ...(await extractOverrideFn(config.imageOptimization?.override)), + }, + default: config.default.override?.generateDockerfile + ? { + type: "ecs", + bundle: ".open-next/server-functions/default", + dockerfile: ".open-next/server-functions/default/Dockerfile", + ...(await extractOverrideFn(config.default.override)), + ...(await extractCommonOverride(config.default.override)), + } + : { + type: "function", + handler: "index.handler", + bundle: ".open-next/server-functions/default", + streaming: defaultOriginCanstream, + ...(await extractOverrideFn(config.default.override)), + ...(await extractCommonOverride(config.default.override)), + }, + }; + + //@ts-expect-error - Not sure how to fix typing here, it complains about the type of imageOptimizer and s3 + const origins: OpenNextOutput["origins"] = defaultOrigins; + + // Then add function origins + await Promise.all( + Object.entries(config.functions ?? {}).map(async ([key, value]) => { + if (!value.placement || value.placement === "regional") { + if (value.override?.generateDockerfile) { + origins[key] = { + type: "ecs", + bundle: `.open-next/server-functions/${key}`, + dockerfile: `.open-next/server-functions/${key}/Dockerfile`, + ...(await extractOverrideFn(value.override)), + ...(await extractCommonOverride(value.override)), + }; + } else { + const streaming = await canStream(value); + origins[key] = { + type: "function", + handler: "index.handler", + bundle: `.open-next/server-functions/${key}`, + streaming, + ...(await extractOverrideFn(value.override)), + ...(await extractCommonOverride(value.override)), + }; + } + } + }), + ); + + // Then we need to compute the behaviors + const behaviors: OpenNextOutput["behaviors"] = [ + { pattern: prefixer("_next/image*"), origin: "imageOptimizer" }, + ]; + + // Then we add the routes + Object.entries(config.functions ?? {}).forEach(([key, value]) => { + const patterns = "patterns" in value ? value.patterns : ["*"]; + patterns.forEach((pattern) => { + behaviors.push({ + pattern: prefixer(pattern.replace(/BUILD_ID/, getBuildId(outputPath))), + origin: value.placement === "global" ? undefined : key, + edgeFunction: + value.placement === "global" + ? key + : isExternalMiddleware + ? "middleware" + : undefined, + }); + }); + }); + + // We finish with the default behavior so that they don't override the others + behaviors.push({ + pattern: prefixer("_next/data/*"), + origin: "default", + edgeFunction: isExternalMiddleware ? "middleware" : undefined, + }); + behaviors.push({ + pattern: "*", // This is the default behavior + origin: "default", + edgeFunction: isExternalMiddleware ? "middleware" : undefined, + }); + + //Compute behaviors for assets files + const assetPath = path.join(outputPath, ".open-next", "assets"); + fs.readdirSync(assetPath).forEach((item) => { + if (fs.statSync(path.join(assetPath, item)).isDirectory()) { + behaviors.push({ + pattern: prefixer(`${item}/*`), + origin: "s3", + }); + } else { + behaviors.push({ + pattern: prefixer(item), + origin: "s3", + }); + } + }); + + // Check if we produced a dynamodb provider output + const isTagCacheDisabled = + config.dangerous?.disableTagCache || + !fs.existsSync(path.join(outputPath, ".open-next", "dynamodb-provider")); + + const output: OpenNextOutput = { + edgeFunctions, + origins, + behaviors, + additionalProps: { + disableIncrementalCache: config.dangerous?.disableIncrementalCache, + disableTagCache: config.dangerous?.disableTagCache, + warmer: { + handler: "index.handler", + bundle: ".open-next/warmer-function", + }, + initializationFunction: isTagCacheDisabled + ? undefined + : { + handler: "index.handler", + bundle: ".open-next/dynamodb-provider", + }, + revalidationFunction: config.dangerous?.disableIncrementalCache + ? undefined + : { + handler: "index.handler", + bundle: ".open-next/revalidation-function", + }, + }, + }; + fs.writeFileSync( + path.join(outputPath, ".open-next", "open-next.output.json"), + JSON.stringify(output), + ); +} diff --git a/packages/open-next/src/build/helper.ts b/packages/open-next/src/build/helper.ts new file mode 100644 index 00000000..9dfab8d8 --- /dev/null +++ b/packages/open-next/src/build/helper.ts @@ -0,0 +1,246 @@ +import fs from "node:fs"; +import { createRequire as topLevelCreateRequire } from "node:module"; +import path from "node:path"; +import url from "node:url"; + +import { + build as buildAsync, + BuildOptions as ESBuildOptions, + buildSync, +} from "esbuild"; +import { OpenNextConfig } from "types/open-next.js"; + +import logger from "../logger.js"; + +const require = topLevelCreateRequire(import.meta.url); +const __dirname = url.fileURLToPath(new URL(".", import.meta.url)); + +export type BuildOptions = ReturnType; + +export function normalizeOptions(config: OpenNextConfig, root: string) { + const appPath = path.join(process.cwd(), config.appPath || "."); + const buildOutputPath = path.join( + process.cwd(), + config.buildOutputPath || ".", + ); + const outputDir = path.join(buildOutputPath, ".open-next"); + + let nextPackageJsonPath: string; + if (config.packageJsonPath) { + const _pkgPath = path.join(process.cwd(), config.packageJsonPath); + nextPackageJsonPath = _pkgPath.endsWith("package.json") + ? _pkgPath + : path.join(_pkgPath, "./package.json"); + } else { + nextPackageJsonPath = findNextPackageJsonPath(appPath, root); + } + return { + openNextVersion: getOpenNextVersion(), + nextVersion: getNextVersion(nextPackageJsonPath), + nextPackageJsonPath, + appPath, + appBuildOutputPath: buildOutputPath, + appPublicPath: path.join(appPath, "public"), + outputDir, + tempDir: path.join(outputDir, ".build"), + debug: Boolean(process.env.OPEN_NEXT_DEBUG) ?? false, + monorepoRoot: root, + }; +} + +function findNextPackageJsonPath(appPath: string, root: string) { + // This is needed for the case where the app is a single-version monorepo and the package.json is in the root of the monorepo + return fs.existsSync(path.join(appPath, "./package.json")) + ? path.join(appPath, "./package.json") + : path.join(root, "./package.json"); +} + +export function esbuildSync( + esbuildOptions: ESBuildOptions, + options: BuildOptions, +) { + const { openNextVersion, debug } = options; + const result = buildSync({ + target: "esnext", + format: "esm", + platform: "node", + bundle: true, + minify: debug ? false : true, + mainFields: ["module", "main"], + sourcemap: debug ? "inline" : false, + sourcesContent: false, + ...esbuildOptions, + external: ["./open-next.config.mjs", ...(esbuildOptions.external ?? [])], + banner: { + ...esbuildOptions.banner, + js: [ + esbuildOptions.banner?.js || "", + `globalThis.openNextDebug = ${debug};`, + `globalThis.openNextVersion = "${openNextVersion}";`, + ].join(""), + }, + }); + + if (result.errors.length > 0) { + result.errors.forEach((error) => logger.error(error)); + throw new Error( + `There was a problem bundling ${ + (esbuildOptions.entryPoints as string[])[0] + }.`, + ); + } +} + +export async function esbuildAsync( + esbuildOptions: ESBuildOptions, + options: BuildOptions, +) { + const { openNextVersion, debug } = options; + const result = await buildAsync({ + target: "esnext", + format: "esm", + platform: "node", + bundle: true, + minify: debug ? false : true, + mainFields: ["module", "main"], + sourcemap: debug ? "inline" : false, + sourcesContent: false, + ...esbuildOptions, + external: [ + ...(esbuildOptions.external ?? []), + "next", + "./open-next.config.mjs", + ], + banner: { + ...esbuildOptions.banner, + js: [ + esbuildOptions.banner?.js || "", + `globalThis.openNextDebug = ${debug};`, + `globalThis.openNextVersion = "${openNextVersion}";`, + ].join(""), + }, + }); + + if (result.errors.length > 0) { + result.errors.forEach((error) => logger.error(error)); + throw new Error( + `There was a problem bundling ${ + (esbuildOptions.entryPoints as string[])[0] + }.`, + ); + } +} + +export function removeFiles( + root: string, + conditionFn: (file: string) => boolean, + searchingDir: string = "", +) { + traverseFiles( + root, + conditionFn, + (filePath) => fs.rmSync(filePath, { force: true }), + searchingDir, + ); +} + +/** + * Recursively traverse files in a directory and call `callbackFn` when `conditionFn` returns true + * @param root - Root directory to search + * @param conditionFn - Called to determine if `callbackFn` should be called + * @param callbackFn - Called when `conditionFn` returns true + * @param searchingDir - Directory to search (used for recursion) + */ +export function traverseFiles( + root: string, + conditionFn: (file: string) => boolean, + callbackFn: (filePath: string) => void, + searchingDir: string = "", +) { + fs.readdirSync(path.join(root, searchingDir)).forEach((file) => { + const filePath = path.join(root, searchingDir, file); + + if (fs.statSync(filePath).isDirectory()) { + traverseFiles( + root, + conditionFn, + callbackFn, + path.join(searchingDir, file), + ); + return; + } + + if (conditionFn(path.join(searchingDir, file))) { + callbackFn(filePath); + } + }); +} + +export function getHtmlPages(dotNextPath: string) { + // Get a list of HTML pages + // + // sample return value: + // Set([ + // '404.html', + // 'csr.html', + // 'image-html-tag.html', + // ]) + const manifestPath = path.join( + dotNextPath, + ".next/server/pages-manifest.json", + ); + const manifest = fs.readFileSync(manifestPath, "utf-8"); + return Object.entries(JSON.parse(manifest)) + .filter(([_, value]) => (value as string).endsWith(".html")) + .map(([_, value]) => (value as string).replace(/^pages\//, "")) + .reduce((acc, page) => { + acc.add(page); + return acc; + }, new Set()); +} + +export function getBuildId(dotNextPath: string) { + return fs + .readFileSync(path.join(dotNextPath, ".next/BUILD_ID"), "utf-8") + .trim(); +} + +export function getOpenNextVersion(): string { + return require(path.join(__dirname, "../../package.json")).version; +} + +export function getNextVersion(nextPackageJsonPath: string): string { + const version = require(nextPackageJsonPath)?.dependencies?.next; + // require('next/package.json').version + + if (!version) { + throw new Error("Failed to find Next version"); + } + + // Drop the -canary.n suffix + return version.split("-")[0]; +} + +export function compareSemver(v1: string, v2: string): number { + if (v1 === "latest") return 1; + if (/^[^\d]/.test(v1)) { + v1 = v1.substring(1); + } + if (/^[^\d]/.test(v2)) { + v2 = v2.substring(1); + } + const [major1, minor1, patch1] = v1.split(".").map(Number); + const [major2, minor2, patch2] = v2.split(".").map(Number); + + if (major1 !== major2) return major1 - major2; + if (minor1 !== minor2) return minor1 - minor2; + return patch1 - patch2; +} + +export function copyOpenNextConfig(tempDir: string, outputPath: string) { + // Copy open-next.config.mjs + fs.copyFileSync( + path.join(tempDir, "open-next.config.mjs"), + path.join(outputPath, "open-next.config.mjs"), + ); +} diff --git a/packages/open-next/src/build/validateConfig.ts b/packages/open-next/src/build/validateConfig.ts new file mode 100644 index 00000000..7fc2f147 --- /dev/null +++ b/packages/open-next/src/build/validateConfig.ts @@ -0,0 +1,56 @@ +import { + FunctionOptions, + OpenNextConfig, + SplittedFunctionOptions, +} from "types/open-next"; + +import logger from "../logger.js"; + +function validateFunctionOptions(fnOptions: FunctionOptions) { + if (fnOptions.runtime === "edge" && fnOptions.experimentalBundledNextServer) { + logger.warn( + "experimentalBundledNextServer has no effect for edge functions", + ); + } + if ( + fnOptions.override?.generateDockerfile && + fnOptions.override.converter !== "node" && + fnOptions.override.wrapper !== "node" + ) { + logger.warn( + "You've specified generateDockerfile without node converter and wrapper. Without custom converter and wrapper the dockerfile will not work", + ); + } +} + +function validateSplittedFunctionOptions( + fnOptions: SplittedFunctionOptions, + name: string, +) { + validateFunctionOptions(fnOptions); + if (fnOptions.routes.length === 0) { + throw new Error(`Splitted function ${name} must have at least one route`); + } + if (fnOptions.runtime === "edge" && fnOptions.routes.length > 1) { + throw new Error(`Edge function ${name} can only have one route`); + } +} + +export function validateConfig(config: OpenNextConfig) { + validateFunctionOptions(config.default); + Object.entries(config.functions ?? {}).forEach(([name, fnOptions]) => { + validateSplittedFunctionOptions(fnOptions, name); + }); + if (config.dangerous?.disableIncrementalCache) { + logger.warn( + "You've disabled incremental cache. This means that ISR and SSG will not work.", + ); + } + if (config.dangerous?.disableTagCache) { + logger.warn( + `You've disabled tag cache. + This means that revalidatePath and revalidateTag from next/cache will not work. + It is safe to disable if you only use page router`, + ); + } +} diff --git a/packages/open-next/src/cache/incremental/s3.ts b/packages/open-next/src/cache/incremental/s3.ts new file mode 100644 index 00000000..2e277ba1 --- /dev/null +++ b/packages/open-next/src/cache/incremental/s3.ts @@ -0,0 +1,78 @@ +import { + DeleteObjectCommand, + GetObjectCommand, + PutObjectCommand, + S3Client, + S3ClientConfig, +} from "@aws-sdk/client-s3"; +import path from "path"; + +import { awsLogger } from "../../adapters/logger"; +import { parseNumberFromEnv } from "../../adapters/util"; +import { Extension } from "../next-types"; +import { IncrementalCache } from "./types"; + +const { + CACHE_BUCKET_REGION, + CACHE_BUCKET_KEY_PREFIX, + NEXT_BUILD_ID, + CACHE_BUCKET_NAME, +} = process.env; + +function parseS3ClientConfigFromEnv(): S3ClientConfig { + return { + region: CACHE_BUCKET_REGION, + logger: awsLogger, + maxAttempts: parseNumberFromEnv(process.env.AWS_SDK_S3_MAX_ATTEMPTS), + }; +} + +const s3Client = new S3Client(parseS3ClientConfigFromEnv()); + +function buildS3Key(key: string, extension: Extension) { + return path.posix.join( + CACHE_BUCKET_KEY_PREFIX ?? "", + extension === "fetch" ? "__fetch" : "", + NEXT_BUILD_ID ?? "", + extension === "fetch" ? key : `${key}.${extension}`, + ); +} + +const incrementalCache: IncrementalCache = { + async get(key, isFetch) { + const result = await s3Client.send( + new GetObjectCommand({ + Bucket: CACHE_BUCKET_NAME, + Key: buildS3Key(key, isFetch ? "fetch" : "cache"), + }), + ); + + const cacheData = JSON.parse( + (await result.Body?.transformToString()) ?? "{}", + ); + return { + value: cacheData, + lastModified: result.LastModified?.getTime(), + }; + }, + async set(key, value, isFetch): Promise { + await s3Client.send( + new PutObjectCommand({ + Bucket: CACHE_BUCKET_NAME, + Key: buildS3Key(key, isFetch ? "fetch" : "cache"), + Body: JSON.stringify(value), + }), + ); + }, + async delete(key): Promise { + await s3Client.send( + new DeleteObjectCommand({ + Bucket: CACHE_BUCKET_NAME, + Key: buildS3Key(key, "cache"), + }), + ); + }, + name: "s3", +}; + +export default incrementalCache; diff --git a/packages/open-next/src/cache/incremental/types.ts b/packages/open-next/src/cache/incremental/types.ts new file mode 100644 index 00000000..60217cf5 --- /dev/null +++ b/packages/open-next/src/cache/incremental/types.ts @@ -0,0 +1,50 @@ +import { Meta } from "../next-types"; + +export type S3CachedFile = + | { + type: "redirect"; + props?: Object; + meta?: Meta; + } + | { + type: "page"; + html: string; + json: Object; + meta?: Meta; + } + | { + type: "app"; + html: string; + rsc: string; + meta?: Meta; + } + | { + type: "route"; + body: string; + meta?: Meta; + }; + +export type S3FetchCache = Object; + +export type WithLastModified = { + lastModified?: number; + value?: T; +}; + +export type CacheValue = IsFetch extends true + ? S3FetchCache + : S3CachedFile; + +export type IncrementalCache = { + get( + key: string, + isFetch?: IsFetch, + ): Promise>>; + set( + key: string, + value: CacheValue, + isFetch?: IsFetch, + ): Promise; + delete(key: string): Promise; + name: string; +}; diff --git a/packages/open-next/src/cache/next-types.ts b/packages/open-next/src/cache/next-types.ts new file mode 100644 index 00000000..58fe9565 --- /dev/null +++ b/packages/open-next/src/cache/next-types.ts @@ -0,0 +1,76 @@ +interface CachedFetchValue { + kind: "FETCH"; + data: { + headers: { [k: string]: string }; + body: string; + url: string; + status?: number; + tags?: string[]; + }; + revalidate: number; +} + +interface CachedRedirectValue { + kind: "REDIRECT"; + props: Object; +} + +interface CachedRouteValue { + kind: "ROUTE"; + // this needs to be a RenderResult so since renderResponse + // expects that type instead of a string + body: Buffer; + status: number; + headers: Record; +} + +interface CachedImageValue { + kind: "IMAGE"; + etag: string; + buffer: Buffer; + extension: string; + isMiss?: boolean; + isStale?: boolean; +} + +interface IncrementalCachedPageValue { + kind: "PAGE"; + // this needs to be a string since the cache expects to store + // the string value + html: string; + pageData: Object; + status?: number; + headers?: Record; +} + +type IncrementalCacheValue = + | CachedRedirectValue + | IncrementalCachedPageValue + | CachedImageValue + | CachedFetchValue + | CachedRouteValue; + +export interface CacheHandlerContext { + fs?: never; + dev?: boolean; + flushToDisk?: boolean; + serverDistDir?: string; + maxMemoryCacheSize?: number; + _appDir: boolean; + _requestHeaders: never; + fetchCacheKeyPrefix?: string; +} + +export interface CacheHandlerValue { + lastModified?: number; + age?: number; + cacheState?: string; + value: IncrementalCacheValue | null; +} + +export type Extension = "cache" | "fetch"; + +export interface Meta { + status?: number; + headers?: Record; +} diff --git a/packages/open-next/src/adapters/constants.ts b/packages/open-next/src/cache/tag/constants.ts similarity index 100% rename from packages/open-next/src/adapters/constants.ts rename to packages/open-next/src/cache/tag/constants.ts diff --git a/packages/open-next/src/cache/tag/dynamoDb.ts b/packages/open-next/src/cache/tag/dynamoDb.ts new file mode 100644 index 00000000..dc2a61a2 --- /dev/null +++ b/packages/open-next/src/cache/tag/dynamoDb.ts @@ -0,0 +1,156 @@ +import { + BatchWriteItemCommand, + DynamoDBClient, + DynamoDBClientConfig, + QueryCommand, +} from "@aws-sdk/client-dynamodb"; +import path from "path"; + +import { awsLogger, debug, error } from "../../adapters/logger"; +import { chunk, parseNumberFromEnv } from "../../adapters/util"; +import { + getDynamoBatchWriteCommandConcurrency, + MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT, +} from "./constants"; +import { TagCache } from "./types"; + +const { CACHE_BUCKET_REGION, CACHE_DYNAMO_TABLE, NEXT_BUILD_ID } = process.env; + +function parseDynamoClientConfigFromEnv(): DynamoDBClientConfig { + return { + region: CACHE_BUCKET_REGION, + logger: awsLogger, + maxAttempts: parseNumberFromEnv(process.env.AWS_SDK_DYNAMODB_MAX_ATTEMPTS), + }; +} + +const dynamoClient = new DynamoDBClient(parseDynamoClientConfigFromEnv()); + +function buildDynamoKey(key: string) { + // FIXME: We should probably use something else than path.join here + // this could transform some fetch cache key into a valid path + return path.posix.join(NEXT_BUILD_ID ?? "", key); +} + +function buildDynamoObject(path: string, tags: string, revalidatedAt?: number) { + return { + path: { S: buildDynamoKey(path) }, + tag: { S: buildDynamoKey(tags) }, + revalidatedAt: { N: `${revalidatedAt ?? Date.now()}` }, + }; +} + +const tagCache: TagCache = { + async getByPath(path) { + try { + if (globalThis.disableDynamoDBCache) return []; + const result = await dynamoClient.send( + new QueryCommand({ + TableName: CACHE_DYNAMO_TABLE, + IndexName: "revalidate", + KeyConditionExpression: "#key = :key", + ExpressionAttributeNames: { + "#key": "path", + }, + ExpressionAttributeValues: { + ":key": { S: buildDynamoKey(path) }, + }, + }), + ); + const tags = result.Items?.map((item) => item.tag.S ?? "") ?? []; + debug("tags for path", path, tags); + return tags; + } catch (e) { + error("Failed to get tags by path", e); + return []; + } + }, + async getByTag(tag) { + try { + if (globalThis.disableDynamoDBCache) return []; + const { Items } = await dynamoClient.send( + new QueryCommand({ + TableName: CACHE_DYNAMO_TABLE, + KeyConditionExpression: "#tag = :tag", + ExpressionAttributeNames: { + "#tag": "tag", + }, + ExpressionAttributeValues: { + ":tag": { S: buildDynamoKey(tag) }, + }, + }), + ); + return ( + // We need to remove the buildId from the path + Items?.map( + ({ path: { S: key } }) => key?.replace(`${NEXT_BUILD_ID}/`, "") ?? "", + ) ?? [] + ); + } catch (e) { + error("Failed to get by tag", e); + return []; + } + }, + async getLastModified(key, lastModified) { + try { + if (globalThis.disableDynamoDBCache) return lastModified ?? Date.now(); + const result = await dynamoClient.send( + new QueryCommand({ + TableName: CACHE_DYNAMO_TABLE, + IndexName: "revalidate", + KeyConditionExpression: + "#key = :key AND #revalidatedAt > :lastModified", + ExpressionAttributeNames: { + "#key": "path", + "#revalidatedAt": "revalidatedAt", + }, + ExpressionAttributeValues: { + ":key": { S: buildDynamoKey(key) }, + ":lastModified": { N: String(lastModified ?? 0) }, + }, + }), + ); + const revalidatedTags = result.Items ?? []; + debug("revalidatedTags", revalidatedTags); + // If we have revalidated tags we return -1 to force revalidation + return revalidatedTags.length > 0 ? -1 : lastModified ?? Date.now(); + } catch (e) { + error("Failed to get revalidated tags", e); + return lastModified ?? Date.now(); + } + }, + async writeTags(tags) { + try { + if (globalThis.disableDynamoDBCache) return; + const dataChunks = chunk(tags, MAX_DYNAMO_BATCH_WRITE_ITEM_COUNT).map( + (Items) => ({ + RequestItems: { + [CACHE_DYNAMO_TABLE ?? ""]: Items.map((Item) => ({ + PutRequest: { + Item: { + ...buildDynamoObject(Item.path, Item.tag, Item.revalidatedAt), + }, + }, + })), + }, + }), + ); + const toInsert = chunk( + dataChunks, + getDynamoBatchWriteCommandConcurrency(), + ); + for (const paramsChunk of toInsert) { + await Promise.all( + paramsChunk.map(async (params) => + dynamoClient.send(new BatchWriteItemCommand(params)), + ), + ); + } + } catch (e) { + error("Failed to batch write dynamo item", e); + } + }, + name: "dynamoDb", +}; + +export default tagCache; diff --git a/packages/open-next/src/cache/tag/types.ts b/packages/open-next/src/cache/tag/types.ts new file mode 100644 index 00000000..533e725c --- /dev/null +++ b/packages/open-next/src/cache/tag/types.ts @@ -0,0 +1,9 @@ +export type TagCache = { + getByTag(tag: string): Promise; + getByPath(path: string): Promise; + getLastModified(path: string, lastModified?: number): Promise; + writeTags( + tags: { tag: string; path: string; revalidatedAt?: number }[], + ): Promise; + name: string; +}; diff --git a/packages/open-next/src/converters/aws-apigw-v1.ts b/packages/open-next/src/converters/aws-apigw-v1.ts new file mode 100644 index 00000000..dabd88bc --- /dev/null +++ b/packages/open-next/src/converters/aws-apigw-v1.ts @@ -0,0 +1,109 @@ +import { APIGatewayProxyEvent, APIGatewayProxyResult } from "aws-lambda"; +import type { Converter, InternalEvent, InternalResult } from "types/open-next"; + +import { debug } from "../adapters/logger"; +import { removeUndefinedFromQuery } from "./utils"; + +function normalizeAPIGatewayProxyEventHeaders( + event: APIGatewayProxyEvent, +): Record { + event.multiValueHeaders; + const headers: Record = {}; + + for (const [key, values] of Object.entries(event.multiValueHeaders || {})) { + if (values) { + headers[key.toLowerCase()] = values.join(","); + } + } + for (const [key, value] of Object.entries(event.headers || {})) { + if (value) { + headers[key.toLowerCase()] = value; + } + } + return headers; +} + +function normalizeAPIGatewayProxyEventQueryParams( + event: APIGatewayProxyEvent, +): string { + // Note that the same query string values are returned in both + // "multiValueQueryStringParameters" and "queryStringParameters". + // We only need to use one of them. + // For example: + // "?name=foo" appears in the event object as + // { + // ... + // queryStringParameters: { name: 'foo' }, + // multiValueQueryStringParameters: { name: [ 'foo' ] }, + // ... + // } + const params = new URLSearchParams(); + for (const [key, value] of Object.entries( + event.multiValueQueryStringParameters || {}, + )) { + if (value !== undefined) { + for (const v of value) { + params.append(key, v); + } + } + } + const value = params.toString(); + return value ? `?${value}` : ""; +} + +async function convertFromAPIGatewayProxyEvent( + event: APIGatewayProxyEvent, +): Promise { + const { path, body, httpMethod, requestContext, isBase64Encoded } = event; + return { + type: "core", + method: httpMethod, + rawPath: path, + url: path + normalizeAPIGatewayProxyEventQueryParams(event), + body: Buffer.from(body ?? "", isBase64Encoded ? "base64" : "utf8"), + headers: normalizeAPIGatewayProxyEventHeaders(event), + remoteAddress: requestContext.identity.sourceIp, + query: removeUndefinedFromQuery( + event.multiValueQueryStringParameters ?? {}, + ), + cookies: + event.multiValueHeaders?.cookie?.reduce((acc, cur) => { + const [key, value] = cur.split("="); + return { ...acc, [key]: value }; + }, {}) ?? {}, + }; +} + +function convertToApiGatewayProxyResult( + result: InternalResult, +): APIGatewayProxyResult { + const headers: Record = {}; + const multiValueHeaders: Record = {}; + Object.entries(result.headers).forEach(([key, value]) => { + if (Array.isArray(value)) { + multiValueHeaders[key] = value; + } else { + if (value === null) { + headers[key] = ""; + return; + } + headers[key] = value; + } + }); + + const response: APIGatewayProxyResult = { + statusCode: result.statusCode, + headers, + body: result.body, + isBase64Encoded: result.isBase64Encoded, + multiValueHeaders, + }; + debug(response); + return response; +} + +export default { + convertFrom: convertFromAPIGatewayProxyEvent, + convertTo: convertToApiGatewayProxyResult, + name: "aws-apigw-v1", +} as Converter; diff --git a/packages/open-next/src/converters/aws-apigw-v2.ts b/packages/open-next/src/converters/aws-apigw-v2.ts new file mode 100644 index 00000000..02787ed9 --- /dev/null +++ b/packages/open-next/src/converters/aws-apigw-v2.ts @@ -0,0 +1,92 @@ +import { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from "aws-lambda"; +import { parseCookies } from "http/util"; +import type { Converter, InternalEvent, InternalResult } from "types/open-next"; + +import { debug } from "../adapters/logger"; +import { convertToQuery } from "../core/routing/util"; +import { removeUndefinedFromQuery } from "./utils"; + +function normalizeAPIGatewayProxyEventV2Body( + event: APIGatewayProxyEventV2, +): Buffer { + const { body, isBase64Encoded } = event; + if (Buffer.isBuffer(body)) { + return body; + } else if (typeof body === "string") { + return Buffer.from(body, isBase64Encoded ? "base64" : "utf8"); + } else if (typeof body === "object") { + return Buffer.from(JSON.stringify(body)); + } else { + return Buffer.from("", "utf8"); + } +} + +function normalizeAPIGatewayProxyEventV2Headers( + event: APIGatewayProxyEventV2, +): Record { + const { headers: rawHeaders, cookies } = event; + + const headers: Record = {}; + + if (Array.isArray(cookies)) { + headers["cookie"] = cookies.join("; "); + } + + for (const [key, value] of Object.entries(rawHeaders || {})) { + headers[key.toLowerCase()] = value!; + } + + return headers; +} + +async function convertFromAPIGatewayProxyEventV2( + event: APIGatewayProxyEventV2, +): Promise { + const { rawPath, rawQueryString, requestContext } = event; + return { + type: "core", + method: requestContext.http.method, + rawPath, + url: rawPath + (rawQueryString ? `?${rawQueryString}` : ""), + body: normalizeAPIGatewayProxyEventV2Body(event), + headers: normalizeAPIGatewayProxyEventV2Headers(event), + remoteAddress: requestContext.http.sourceIp, + query: removeUndefinedFromQuery(convertToQuery(rawQueryString)), + cookies: + event.cookies?.reduce((acc, cur) => { + const [key, value] = cur.split("="); + return { ...acc, [key]: value }; + }, {}) ?? {}, + }; +} + +function convertToApiGatewayProxyResultV2( + result: InternalResult, +): APIGatewayProxyResultV2 { + const headers: Record = {}; + Object.entries(result.headers) + .filter(([key]) => key.toLowerCase() !== "set-cookie") + .forEach(([key, value]) => { + if (value === null) { + headers[key] = ""; + return; + } + headers[key] = Array.isArray(value) ? value.join(", ") : value.toString(); + }); + + const response: APIGatewayProxyResultV2 = { + statusCode: result.statusCode, + headers, + cookies: parseCookies(result.headers["set-cookie"]), + body: result.body, + isBase64Encoded: result.isBase64Encoded, + }; + debug(response); + return response; +} + +export default { + convertFrom: convertFromAPIGatewayProxyEventV2, + convertTo: convertToApiGatewayProxyResultV2, + name: "aws-apigw-v2", +} as Converter; diff --git a/packages/open-next/src/converters/aws-cloudfront.ts b/packages/open-next/src/converters/aws-cloudfront.ts new file mode 100644 index 00000000..ba969e5c --- /dev/null +++ b/packages/open-next/src/converters/aws-cloudfront.ts @@ -0,0 +1,220 @@ +import { + CloudFrontCustomOrigin, + CloudFrontHeaders, + CloudFrontRequest, + CloudFrontRequestEvent, + CloudFrontRequestResult, +} from "aws-lambda"; +import { OutgoingHttpHeader } from "http"; +import { parseCookies } from "http/util"; +import type { Converter, InternalEvent, InternalResult } from "types/open-next"; + +import { debug } from "../adapters/logger"; +import { + convertRes, + convertToQuery, + convertToQueryString, + createServerResponse, + proxyRequest, +} from "../core/routing/util"; +import { MiddlewareOutputEvent } from "../core/routingHandler"; + +const CloudFrontBlacklistedHeaders = [ + // Disallowed headers, see: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-function-restrictions-all.html#function-restrictions-disallowed-headers + "connection", + "expect", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "proxy-connection", + "trailer", + "upgrade", + "x-accel-buffering", + "x-accel-charset", + "x-accel-limit-rate", + "x-accel-redirect", + /x-amz-cf-(.*)/, + "x-amzn-auth", + "x-amzn-cf-billing", + "x-amzn-cf-id", + "x-amzn-cf-xff", + "x-amzn-errortype", + "x-amzn-fle-profile", + "x-amzn-header-count", + "x-amzn-header-order", + "x-amzn-lambda-integration-tag", + "x-amzn-requestid", + /x-edge-(.*)/, + "x-cache", + "x-forwarded-proto", + "x-real-ip", + + // Read-only headers, see: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-function-restrictions-all.html#function-restrictions-read-only-headers + "accept-encoding", + "content-length", + "if-modified-since", + "if-none-match", + "if-range", + "if-unmodified-since", + "transfer-encoding", + "via", +]; + +function normalizeCloudFrontRequestEventHeaders( + rawHeaders: CloudFrontHeaders, +): Record { + const headers: Record = {}; + + for (const [key, values] of Object.entries(rawHeaders)) { + for (const { value } of values) { + if (value) { + headers[key.toLowerCase()] = value; + } + } + } + + return headers; +} + +async function convertFromCloudFrontRequestEvent( + event: CloudFrontRequestEvent, +): Promise { + const { method, uri, querystring, body, headers, clientIp } = + event.Records[0].cf.request; + return { + type: "core", + method, + rawPath: uri, + url: uri + (querystring ? `?${querystring}` : ""), + body: Buffer.from( + body?.data ?? "", + body?.encoding === "base64" ? "base64" : "utf8", + ), + headers: normalizeCloudFrontRequestEventHeaders(headers), + remoteAddress: clientIp, + query: convertToQuery(querystring), + cookies: + headers.cookie?.reduce((acc, cur) => { + const { key, value } = cur; + return { ...acc, [key ?? ""]: value }; + }, {}) ?? {}, + }; +} + +type MiddlewareEvent = { + type: "middleware"; +} & MiddlewareOutputEvent; + +function convertToCloudfrontHeaders( + headers: Record, +) { + const cloudfrontHeaders: CloudFrontHeaders = {}; + Object.entries(headers) + .filter( + ([key]) => + !CloudFrontBlacklistedHeaders.some((header) => + typeof header === "string" ? header === key : header.test(key), + ), + ) + .forEach(([key, value]) => { + if (key === "set-cookie") { + const cookies = parseCookies(`${value}`); + if (cookies) { + cloudfrontHeaders[key] = cookies.map((cookie) => ({ + key, + value: cookie, + })); + } + return; + } + cloudfrontHeaders[key] = [ + ...(cloudfrontHeaders[key] || []), + ...(Array.isArray(value) + ? value.map((v) => ({ key, value: v })) + : [{ key, value: value.toString() }]), + ]; + }); + return cloudfrontHeaders; +} + +async function convertToCloudFrontRequestResult( + result: InternalResult | MiddlewareEvent, + originalRequest: CloudFrontRequestEvent, +): Promise { + let responseHeaders = + result.type === "middleware" + ? result.internalEvent.headers + : result.headers; + if (result.type === "middleware") { + const { method, clientIp, origin } = originalRequest.Records[0].cf.request; + + // Handle external rewrite + if (result.isExternalRewrite) { + const serverResponse = createServerResponse(result.internalEvent, {}); + await proxyRequest(result.internalEvent, serverResponse); + const externalResult = convertRes(serverResponse); + const cloudfrontResult = { + status: externalResult.statusCode.toString(), + statusDescription: "OK", + headers: convertToCloudfrontHeaders(externalResult.headers), + bodyEncoding: externalResult.isBase64Encoded + ? ("base64" as const) + : ("text" as const), + body: externalResult.body, + }; + debug("externalResult", cloudfrontResult); + return cloudfrontResult; + } + let customOrigin = origin?.custom as CloudFrontCustomOrigin; + let host = responseHeaders["host"] ?? responseHeaders["Host"]; + if (result.origin) { + customOrigin = { + ...customOrigin, + domainName: result.origin.host, + port: result.origin.port ?? 443, + protocol: result.origin.protocol ?? "https", + customHeaders: {}, + }; + host = result.origin.host; + } + + const response: CloudFrontRequest = { + clientIp, + method, + uri: result.internalEvent.rawPath, + querystring: convertToQueryString(result.internalEvent.query).replace( + "?", + "", + ), + headers: convertToCloudfrontHeaders({ + ...responseHeaders, + host, + }), + origin: origin?.custom + ? { + custom: customOrigin, + } + : origin, + }; + + debug("response rewrite", response); + + return response; + } + + const response: CloudFrontRequestResult = { + status: result.statusCode.toString(), + statusDescription: "OK", + headers: convertToCloudfrontHeaders(responseHeaders), + bodyEncoding: result.isBase64Encoded ? "base64" : "text", + body: result.body, + }; + debug(response); + return response; +} + +export default { + convertFrom: convertFromCloudFrontRequestEvent, + convertTo: convertToCloudFrontRequestResult, + name: "aws-cloudfront", +} as Converter; diff --git a/packages/open-next/src/converters/dummy.ts b/packages/open-next/src/converters/dummy.ts new file mode 100644 index 00000000..917b50e1 --- /dev/null +++ b/packages/open-next/src/converters/dummy.ts @@ -0,0 +1,24 @@ +import { Converter } from "types/open-next"; + +type DummyEventOrResult = { + type: "dummy"; + original: any; +}; + +const converter: Converter = { + convertFrom(event) { + return Promise.resolve({ + type: "dummy", + original: event, + }); + }, + convertTo(internalResult) { + return Promise.resolve({ + type: "dummy", + original: internalResult, + }); + }, + name: "dummy", +}; + +export default converter; diff --git a/packages/open-next/src/converters/edge.ts b/packages/open-next/src/converters/edge.ts new file mode 100644 index 00000000..32d5923e --- /dev/null +++ b/packages/open-next/src/converters/edge.ts @@ -0,0 +1,86 @@ +import { parseCookies } from "http/util"; +import { Converter, InternalEvent, InternalResult } from "types/open-next"; + +import { MiddlewareOutputEvent } from "../core/routingHandler"; + +const converter: Converter< + InternalEvent, + InternalResult | ({ type: "middleware" } & MiddlewareOutputEvent) +> = { + convertFrom: async (event: Request) => { + const searchParams = new URL(event.url).searchParams; + const query: Record = {}; + for (const [key, value] of searchParams.entries()) { + if (query[key]) { + if (Array.isArray(query[key])) { + (query[key] as string[]).push(value); + } else { + query[key] = [query[key] as string, value]; + } + } else { + query[key] = value; + } + } + //Transform body into Buffer + const body = await event.arrayBuffer(); + const headers: Record = {}; + event.headers.forEach((value, key) => { + headers[key] = value; + }); + const rawPath = new URL(event.url).pathname; + + return { + type: "core", + method: event.method, + rawPath, + url: event.url, + body: event.method !== "GET" ? Buffer.from(body) : undefined, + headers: headers, + remoteAddress: (event.headers.get("x-forwarded-for") as string) ?? "::1", + query, + cookies: Object.fromEntries( + parseCookies(event.headers.get("cookie") ?? "")?.map((cookie) => { + const [key, value] = cookie.split("="); + return [key, value]; + }) ?? [], + ), + }; + }, + convertTo: async (result) => { + if ("internalEvent" in result) { + let url = result.internalEvent.url; + if (!result.isExternalRewrite) { + if (result.origin) { + url = `${result.origin.protocol}://${result.origin.host}${ + result.origin.port ? `:${result.origin.port}` : "" + }${url}`; + } else { + url = `https://${result.internalEvent.headers.host}${url}`; + } + } + + const req = new Request(url, { + body: result.internalEvent.body, + method: result.internalEvent.method, + headers: { + ...result.internalEvent.headers, + "x-forwarded-host": result.internalEvent.headers.host, + }, + }); + + return fetch(req); + } else { + const headers = new Headers(); + for (const [key, value] of Object.entries(result.headers)) { + headers.set(key, Array.isArray(value) ? value.join(",") : value); + } + return new Response(result.body, { + status: result.statusCode, + headers: headers, + }); + } + }, + name: "edge", +}; + +export default converter; diff --git a/packages/open-next/src/converters/node.ts b/packages/open-next/src/converters/node.ts new file mode 100644 index 00000000..662c0cb1 --- /dev/null +++ b/packages/open-next/src/converters/node.ts @@ -0,0 +1,55 @@ +import { IncomingMessage } from "http"; +import { parseCookies } from "http/util"; +import type { Converter, InternalResult } from "types/open-next"; + +const converter: Converter = { + convertFrom: async (req: IncomingMessage) => { + const body = await new Promise((resolve) => { + const chunks: Uint8Array[] = []; + req.on("data", (chunk) => { + chunks.push(chunk); + }); + req.on("end", () => { + resolve(Buffer.concat(chunks)); + }); + }); + + const url = new URL(req.url!, `http://${req.headers.host}`); + const query = Object.fromEntries(url.searchParams.entries()); + return { + type: "core", + method: req.method ?? "GET", + rawPath: url.pathname, + url: url.pathname + url.search, + body, + headers: Object.fromEntries( + Object.entries(req.headers ?? {}) + .map(([key, value]) => [ + key.toLowerCase(), + Array.isArray(value) ? value.join(",") : value, + ]) + .filter(([key]) => key), + ), + remoteAddress: + (req.headers["x-forwarded-for"] as string) ?? + req.socket.remoteAddress ?? + "::1", + query, + cookies: Object.fromEntries( + parseCookies(req.headers["cookie"])?.map((cookie) => { + const [key, value] = cookie.split("="); + return [key, value]; + }) ?? [], + ), + }; + }, + // Nothing to do here, it's streaming + convertTo: (internalResult: InternalResult) => ({ + body: internalResult.body, + headers: internalResult.headers, + statusCode: internalResult.statusCode, + }), + name: "node", +}; + +export default converter; diff --git a/packages/open-next/src/converters/sqs-revalidate.ts b/packages/open-next/src/converters/sqs-revalidate.ts new file mode 100644 index 00000000..459a93be --- /dev/null +++ b/packages/open-next/src/converters/sqs-revalidate.ts @@ -0,0 +1,25 @@ +import { SQSEvent } from "aws-lambda"; +import { Converter } from "types/open-next"; + +import { RevalidateEvent } from "../adapters/revalidate"; + +const converter: Converter = { + convertFrom(event: SQSEvent) { + const records = event.Records.map((record) => { + const { host, url } = JSON.parse(record.body); + return { host, url }; + }); + return Promise.resolve({ + type: "revalidate", + records, + }); + }, + convertTo() { + return Promise.resolve({ + type: "revalidate", + }); + }, + name: "sqs-revalidate", +}; + +export default converter; diff --git a/packages/open-next/src/converters/utils.ts b/packages/open-next/src/converters/utils.ts new file mode 100644 index 00000000..13b7e810 --- /dev/null +++ b/packages/open-next/src/converters/utils.ts @@ -0,0 +1,11 @@ +export function removeUndefinedFromQuery( + query: Record, +) { + const newQuery: Record = {}; + for (const [key, value] of Object.entries(query)) { + if (value !== undefined) { + newQuery[key] = value; + } + } + return newQuery; +} diff --git a/packages/open-next/src/core/createGenericHandler.ts b/packages/open-next/src/core/createGenericHandler.ts new file mode 100644 index 00000000..a9a3fa08 --- /dev/null +++ b/packages/open-next/src/core/createGenericHandler.ts @@ -0,0 +1,56 @@ +import type { + BaseEventOrResult, + DefaultOverrideOptions, + InternalEvent, + InternalResult, + OpenNextConfig, + OpenNextHandler, +} from "types/open-next"; + +import { debug } from "../adapters/logger"; +import { resolveConverter, resolveWrapper } from "./resolve"; + +declare global { + var openNextConfig: Partial; +} + +type HandlerType = + | "imageOptimization" + | "revalidate" + | "warmer" + | "middleware" + | "initializationFunction"; + +type GenericHandler< + Type extends HandlerType, + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> = { + handler: OpenNextHandler; + type: Type; +}; + +export async function createGenericHandler< + Type extends HandlerType, + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +>(handler: GenericHandler) { + //First we load the config + // @ts-expect-error + const config: OpenNextConfig = await import("./open-next.config.mjs").then( + (m) => m.default, + ); + + globalThis.openNextConfig = config; + const override = config[handler.type] + ?.override as any as DefaultOverrideOptions; + + // From the config, we create the adapter + const adapter = await resolveConverter(override?.converter); + + // Then we create the handler + const wrapper = await resolveWrapper(override?.wrapper); + debug("Using wrapper", wrapper.name); + + return wrapper.wrapper(handler.handler, adapter); +} diff --git a/packages/open-next/src/core/createMainHandler.ts b/packages/open-next/src/core/createMainHandler.ts new file mode 100644 index 00000000..f09ba070 --- /dev/null +++ b/packages/open-next/src/core/createMainHandler.ts @@ -0,0 +1,78 @@ +import type { AsyncLocalStorage } from "node:async_hooks"; + +import type { OpenNextConfig, OverrideOptions } from "types/open-next"; + +import { debug } from "../adapters/logger"; +import { generateUniqueId } from "../adapters/util"; +import type { IncrementalCache } from "../cache/incremental/types"; +import type { Queue } from "../queue/types"; +import { openNextHandler } from "./requestHandler.js"; +import { resolveConverter, resolveTagCache, resolveWrapper } from "./resolve"; + +declare global { + var queue: Queue; + var incrementalCache: IncrementalCache; + var fnName: string | undefined; + var serverId: string; + var __als: AsyncLocalStorage; +} + +async function resolveQueue(queue: OverrideOptions["queue"]) { + if (typeof queue === "string") { + const m = await import(`../queue/${queue}.js`); + return m.default; + } else if (typeof queue === "function") { + return queue(); + } else { + const m_1 = await import("../queue/sqs.js"); + return m_1.default; + } +} + +async function resolveIncrementalCache( + incrementalCache: OverrideOptions["incrementalCache"], +) { + if (typeof incrementalCache === "string") { + const m = await import(`../cache/incremental/${incrementalCache}.js`); + return m.default; + } else if (typeof incrementalCache === "function") { + return incrementalCache(); + } else { + const m_1 = await import("../cache/incremental/s3.js"); + return m_1.default; + } +} + +export async function createMainHandler() { + //First we load the config + const config: OpenNextConfig = await import( + process.cwd() + "/open-next.config.mjs" + ).then((m) => m.default); + + const thisFunction = globalThis.fnName + ? config.functions![globalThis.fnName] + : config.default; + + globalThis.serverId = generateUniqueId(); + + // Default queue + globalThis.queue = await resolveQueue(thisFunction.override?.queue); + + globalThis.incrementalCache = await resolveIncrementalCache( + thisFunction.override?.incrementalCache, + ); + + globalThis.tagCache = await resolveTagCache(thisFunction.override?.tagCache); + + globalThis.lastModified = {}; + + // From the config, we create the adapter + const adapter = await resolveConverter(thisFunction.override?.converter); + + // Then we create the handler + const wrapper = await resolveWrapper(thisFunction.override?.wrapper); + + debug("Using wrapper", wrapper.name); + + return wrapper.wrapper(openNextHandler, adapter); +} diff --git a/packages/open-next/src/core/edgeFunctionHandler.ts b/packages/open-next/src/core/edgeFunctionHandler.ts new file mode 100644 index 00000000..9924b9b4 --- /dev/null +++ b/packages/open-next/src/core/edgeFunctionHandler.ts @@ -0,0 +1,82 @@ +// Necessary files will be imported here with banner in esbuild + +import type { OutgoingHttpHeaders } from "http"; + +interface RequestData { + geo?: { + city?: string; + country?: string; + region?: string; + latitude?: string; + longitude?: string; + }; + headers: OutgoingHttpHeaders; + ip?: string; + method: string; + nextConfig?: { + basePath?: string; + i18n?: any; + trailingSlash?: boolean; + }; + page?: { + name?: string; + params?: { [key: string]: string | string[] }; + }; + url: string; + body?: ReadableStream; + signal: AbortSignal; +} + +interface Entries { + [k: string]: { + default: (props: { page: string; request: RequestData }) => Promise<{ + response: Response; + waitUntil: Promise; + }>; + }; +} +declare global { + var _ENTRIES: Entries; + var _ROUTES: EdgeRoute[]; + var __storage__: Map; + var AsyncContext: any; + //@ts-ignore + var AsyncLocalStorage: any; +} + +export interface EdgeRoute { + name: string; + page: string; + regex: string[]; +} + +type EdgeRequest = Omit; + +export default async function edgeFunctionHandler( + request: EdgeRequest, +): Promise { + const path = new URL(request.url).pathname; + const routes = globalThis._ROUTES; + const correspondingRoute = routes.find((route) => + route.regex.some((r) => new RegExp(r).test(path)), + ); + + if (!correspondingRoute) { + throw new Error(`No route found for ${request.url}`); + } + + const result = await self._ENTRIES[ + `middleware_${correspondingRoute.name}` + ].default({ + page: correspondingRoute.page, + request: { + ...request, + page: { + name: correspondingRoute.name, + }, + }, + }); + await result.waitUntil; + const response = result.response; + return response; +} diff --git a/packages/open-next/src/core/requestHandler.ts b/packages/open-next/src/core/requestHandler.ts new file mode 100644 index 00000000..f4891283 --- /dev/null +++ b/packages/open-next/src/core/requestHandler.ts @@ -0,0 +1,187 @@ +import { AsyncLocalStorage } from "node:async_hooks"; + +import { + IncomingMessage, + OpenNextNodeResponse, + StreamCreator, +} from "http/index.js"; +import { InternalEvent, InternalResult } from "types/open-next"; + +import { debug, error, warn } from "../adapters/logger"; +import { convertRes, createServerResponse, proxyRequest } from "./routing/util"; +import routingHandler, { MiddlewareOutputEvent } from "./routingHandler"; +import { requestHandler, setNextjsPrebundledReact } from "./util"; + +// This is used to identify requests in the cache +globalThis.__als = new AsyncLocalStorage(); + +export async function openNextHandler( + internalEvent: InternalEvent, + responseStreaming?: StreamCreator, +): Promise { + if (internalEvent.headers["x-forwarded-host"]) { + internalEvent.headers.host = internalEvent.headers["x-forwarded-host"]; + } + debug("internalEvent", internalEvent); + + //#override withRouting + let preprocessResult: InternalResult | MiddlewareOutputEvent = { + internalEvent: internalEvent, + isExternalRewrite: false, + origin: false, + }; + try { + preprocessResult = await routingHandler(internalEvent); + } catch (e) { + warn("Routing failed.", e); + } + //#endOverride + + const headers = + "type" in preprocessResult + ? preprocessResult.headers + : preprocessResult.internalEvent.headers; + + const overwrittenResponseHeaders = Object.entries( + "type" in preprocessResult + ? preprocessResult.headers + : preprocessResult.internalEvent.headers, + ).reduce((acc, [key, value]) => { + if (!key.startsWith("x-middleware-response-")) { + return acc; + } else { + const newKey = key.replace("x-middleware-response-", ""); + delete headers[key]; + headers[newKey] = value; + return { ...acc, [newKey]: value }; + } + }, {}); + + if ("type" in preprocessResult) { + // res is used only in the streaming case + const res = createServerResponse(internalEvent, headers, responseStreaming); + res.statusCode = preprocessResult.statusCode; + res.flushHeaders(); + res.write(preprocessResult.body); + res.end(); + return preprocessResult; + } else { + const preprocessedEvent = preprocessResult.internalEvent; + debug("preprocessedEvent", preprocessedEvent); + const reqProps = { + method: preprocessedEvent.method, + url: preprocessedEvent.url, + //WORKAROUND: We pass this header to the serverless function to mimic a prefetch request which will not trigger revalidation since we handle revalidation differently + // There is 3 way we can handle revalidation: + // 1. We could just let the revalidation go as normal, but due to race condtions the revalidation will be unreliable + // 2. We could alter the lastModified time of our cache to make next believe that the cache is fresh, but this could cause issues with stale data since the cdn will cache the stale data as if it was fresh + // 3. OUR CHOICE: We could pass a purpose prefetch header to the serverless function to make next believe that the request is a prefetch request and not trigger revalidation (This could potentially break in the future if next changes the behavior of prefetch requests) + headers: { ...headers, purpose: "prefetch" }, + body: preprocessedEvent.body, + remoteAddress: preprocessedEvent.remoteAddress, + }; + const requestId = Math.random().toString(36); + const internalResult = await globalThis.__als.run(requestId, async () => { + const preprocessedResult = preprocessResult as MiddlewareOutputEvent; + const req = new IncomingMessage(reqProps); + const res = createServerResponse( + preprocessedEvent, + overwrittenResponseHeaders, + responseStreaming, + ); + + await processRequest( + req, + res, + preprocessedEvent, + preprocessedResult.isExternalRewrite, + ); + + const { statusCode, headers, isBase64Encoded, body } = convertRes(res); + + const internalResult = { + type: internalEvent.type, + statusCode, + headers, + body, + isBase64Encoded, + }; + + // reset lastModified. We need to do this to avoid memory leaks + delete globalThis.lastModified[requestId]; + + return internalResult; + }); + return internalResult; + } +} + +async function processRequest( + req: IncomingMessage, + res: OpenNextNodeResponse, + internalEvent: InternalEvent, + isExternalRewrite?: boolean, +) { + // @ts-ignore + // Next.js doesn't parse body if the property exists + // https://github.com/dougmoscrop/serverless-http/issues/227 + delete req.body; + + try { + // `serverHandler` is replaced at build time depending on user's + // nextjs version to patch Nextjs 13.4.x and future breaking changes. + + const { rawPath } = internalEvent; + + if (isExternalRewrite) { + return proxyRequest(internalEvent, res); + } else { + //#override applyNextjsPrebundledReact + setNextjsPrebundledReact(rawPath); + //#endOverride + + // Next Server + await requestHandler(req, res); + } + } catch (e: any) { + // This might fail when using bundled next, importing won't do the trick either + if (e.constructor.name === "NoFallbackError") { + // Do we need to handle _not-found + // Ideally this should never get triggered and be intercepted by the routing handler + tryRenderError("404", res, internalEvent); + } else { + error("NextJS request failed.", e); + tryRenderError("500", res, internalEvent); + } + } +} + +async function tryRenderError( + type: "404" | "500", + res: OpenNextNodeResponse, + internalEvent: InternalEvent, +) { + try { + const _req = new IncomingMessage({ + method: "GET", + url: `/${type}`, + headers: internalEvent.headers, + body: internalEvent.body, + remoteAddress: internalEvent.remoteAddress, + }); + await requestHandler(_req, res); + } catch (e) { + error("NextJS request failed.", e); + res.setHeader("Content-Type", "application/json"); + res.end( + JSON.stringify( + { + message: "Server failed to respond.", + details: e, + }, + null, + 2, + ), + ); + } +} diff --git a/packages/open-next/src/adapters/require-hooks.ts b/packages/open-next/src/core/require-hooks.ts similarity index 98% rename from packages/open-next/src/adapters/require-hooks.ts rename to packages/open-next/src/core/require-hooks.ts index 255fe2f0..963dbfa9 100644 --- a/packages/open-next/src/adapters/require-hooks.ts +++ b/packages/open-next/src/core/require-hooks.ts @@ -2,8 +2,9 @@ // This is needed for userland plugins to attach to the same webpack instance as Next.js'. // Individually compiled modules are as defined for the compilation in bundles/webpack/packages/*. -import { error } from "./logger.js"; -import type { NextConfig } from "./types/next-types.js"; +import type { NextConfig } from "types/next-types.js"; + +import { error } from "../adapters/logger.js"; // This module will only be loaded once per process. diff --git a/packages/open-next/src/core/resolve.ts b/packages/open-next/src/core/resolve.ts new file mode 100644 index 00000000..918d85d6 --- /dev/null +++ b/packages/open-next/src/core/resolve.ts @@ -0,0 +1,58 @@ +import { + BaseEventOrResult, + Converter, + DefaultOverrideOptions, + InternalEvent, + InternalResult, + OverrideOptions, + Wrapper, +} from "types/open-next.js"; + +import { TagCache } from "../cache/tag/types.js"; + +export async function resolveConverter< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +>( + converter: DefaultOverrideOptions["converter"], +): Promise> { + if (typeof converter === "function") { + return converter(); + } else { + const m_1 = await import(`../converters/aws-apigw-v2.js`); + // @ts-expect-error + return m_1.default; + } +} + +export async function resolveWrapper< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +>(wrapper: DefaultOverrideOptions["wrapper"]): Promise> { + if (typeof wrapper === "function") { + return wrapper(); + } else { + // This will be replaced by the bundler + const m_1 = await import("../wrappers/aws-lambda.js"); + // @ts-expect-error + return m_1.default; + } +} + +/** + * + * @param tagCache + * @returns + * @__PURE__ + */ +export async function resolveTagCache( + tagCache: OverrideOptions["tagCache"], +): Promise { + if (typeof tagCache === "function") { + return tagCache(); + } else { + // This will be replaced by the bundler + const m_1 = await import("../cache/tag/dynamoDb.js"); + return m_1.default; + } +} diff --git a/packages/open-next/src/adapters/routing/matcher.ts b/packages/open-next/src/core/routing/matcher.ts similarity index 75% rename from packages/open-next/src/adapters/routing/matcher.ts rename to packages/open-next/src/core/routing/matcher.ts index 39dcb4bc..20073a37 100644 --- a/packages/open-next/src/adapters/routing/matcher.ts +++ b/packages/open-next/src/core/routing/matcher.ts @@ -1,17 +1,22 @@ +import { NextConfig } from "config/index"; import { compile, Match, match, PathFunction } from "path-to-regexp"; - -import { NextConfig } from "../config"; -import { InternalEvent, InternalResult } from "../event-mapper"; -import { debug } from "../logger"; -import { +import type { Header, PrerenderManifest, RedirectDefinition, RewriteDefinition, RouteHas, -} from "../types/next-types"; -import { escapeRegex, unescapeRegex } from "../util"; -import { convertToQueryString, getUrlParts, isExternal } from "./util"; +} from "types/next-types"; +import { InternalEvent, InternalResult } from "types/open-next"; + +import { debug } from "../../adapters/logger"; +import { + convertToQueryString, + escapeRegex, + getUrlParts, + isExternal, + unescapeRegex, +} from "./util"; const routeHasMatcher = ( @@ -113,6 +118,14 @@ export function addNextConfigHeaders( debug("Error matching header ", h.key, " with value ", h.value); requestHeaders[h.key] = h.value; } + try { + const key = convertMatch(_match, compile(h.key), h.key); + const value = convertMatch(_match, compile(h.value), h.value); + requestHeaders[key] = value; + } catch { + debug("Error matching header ", h.key, " with value ", h.value); + requestHeaders[h.key] = h.value; + } }); } } @@ -170,49 +183,53 @@ export function handleRewrites( } function handleTrailingSlashRedirect(event: InternalEvent) { - if (!NextConfig.skipTrailingSlashRedirect) { - const url = new URL(event.url, "http://localhost"); + const url = new URL(event.url, "http://localhost"); + + if ( // Someone is trying to redirect to a different origin, let's not do that - if (url.host !== "localhost") { - return false; - } - if ( - NextConfig.trailingSlash && - !event.headers["x-nextjs-data"] && - !event.rawPath.endsWith("/") && - !event.rawPath.match(/[\w-]+\.[\w]+$/g) - ) { - const headersLocation = event.url.split("?"); - return { - type: event.type, - statusCode: 308, - headers: { - Location: `${headersLocation[0]}/${ - headersLocation[1] ? `?${headersLocation[1]}` : "" - }`, - }, - body: "", - isBase64Encoded: false, - }; - // eslint-disable-next-line sonarjs/elseif-without-else - } else if ( - !NextConfig.trailingSlash && - event.rawPath.endsWith("/") && - event.rawPath !== "/" - ) { - const headersLocation = event.url.split("?"); - return { - type: event.type, - statusCode: 308, - headers: { - Location: `${headersLocation[0].replace(/\/$/, "")}${ - headersLocation[1] ? `?${headersLocation[1]}` : "" - }`, - }, - body: "", - isBase64Encoded: false, - }; - } + url.host !== "localhost" || + NextConfig.skipTrailingSlashRedirect || + // We should not apply trailing slash redirect to API routes + event.rawPath.startsWith("/api/") + ) { + return false; + } + if ( + NextConfig.trailingSlash && + !event.headers["x-nextjs-data"] && + !event.rawPath.endsWith("/") && + !event.rawPath.match(/[\w-]+\.[\w]+$/g) + ) { + const headersLocation = event.url.split("?"); + return { + type: event.type, + statusCode: 308, + headers: { + Location: `${headersLocation[0]}/${ + headersLocation[1] ? `?${headersLocation[1]}` : "" + }`, + }, + body: "", + isBase64Encoded: false, + }; + // eslint-disable-next-line sonarjs/elseif-without-else + } else if ( + !NextConfig.trailingSlash && + event.rawPath.endsWith("/") && + event.rawPath !== "/" + ) { + const headersLocation = event.url.split("?"); + return { + type: event.type, + statusCode: 308, + headers: { + Location: `${headersLocation[0].replace(/\/$/, "")}${ + headersLocation[1] ? `?${headersLocation[1]}` : "" + }`, + }, + body: "", + isBase64Encoded: false, + }; } else return false; } @@ -285,7 +302,15 @@ export function handleFallbackFalse( const routeRegexExp = new RegExp(routeRegex); return routeRegexExp.test(rawPath); }); - if (routeFallback && !Object.keys(routes).includes(rawPath)) { + const locales = NextConfig.i18n?.locales; + const routesAlreadyHaveLocale = + (locales !== undefined && locales.includes(rawPath.split("/")[1])) || + // If we don't use locales, we don't need to add the default locale + locales === undefined; + const localizedPath = routesAlreadyHaveLocale + ? rawPath + : `/${NextConfig.i18n?.defaultLocale}${rawPath}`; + if (routeFallback && !Object.keys(routes).includes(localizedPath)) { return { ...internalEvent, rawPath: "/404", diff --git a/packages/open-next/src/adapters/routing/middleware.ts b/packages/open-next/src/core/routing/middleware.ts similarity index 51% rename from packages/open-next/src/adapters/routing/middleware.ts rename to packages/open-next/src/core/routing/middleware.ts index 85eb02b3..895a4d04 100644 --- a/packages/open-next/src/adapters/routing/middleware.ts +++ b/packages/open-next/src/core/routing/middleware.ts @@ -1,26 +1,21 @@ -import path from "node:path"; +import { MiddlewareManifest, NextConfig } from "config/index.js"; +import { InternalEvent, InternalResult } from "types/open-next.js"; -import { NEXT_DIR, NextConfig } from "../config/index.js"; -import { InternalEvent, InternalResult } from "../event-mapper.js"; -import { IncomingMessage } from "../http/request.js"; -import { ServerlessResponse } from "../http/response.js"; +//NOTE: we should try to avoid importing stuff from next as much as possible +// every release of next could break this +// const { run } = require("next/dist/server/web/sandbox"); +// const { getCloneableBody } = require("next/dist/server/body-streams"); +// const { +// signalFromNodeResponse, +// } = require("next/dist/server/web/spec-extension/adapters/next-request"); import { - convertRes, + convertBodyToReadableStream, convertToQueryString, getMiddlewareMatch, isExternal, - loadMiddlewareManifest, } from "./util.js"; -const middlewareManifest = loadMiddlewareManifest(NEXT_DIR); - -//NOTE: we should try to avoid importing stuff from next as much as possible -// every release of next could break this -const { run } = require("next/dist/server/web/sandbox"); -const { getCloneableBody } = require("next/dist/server/body-streams"); -const { - signalFromNodeResponse, -} = require("next/dist/server/web/spec-extension/adapters/next-request"); +const middlewareManifest = MiddlewareManifest; const middleMatch = getMiddlewareMatch(middlewareManifest); @@ -34,62 +29,68 @@ type MiddlewareOutputEvent = InternalEvent & { // and res.body prior to processing the next-server. // @returns undefined | res.end() -interface MiddlewareResult { - response: Response; -} +// NOTE: We need to normalize the locale path before passing it to the middleware +// See https://github.com/vercel/next.js/blob/39589ff35003ba73f92b7f7b349b3fdd3458819f/packages/next/src/shared/lib/i18n/normalize-locale-path.ts#L15 +function normalizeLocalePath(pathname: string) { + // first item will be empty string from splitting at first char + const pathnameParts = pathname.split("/"); + const locales = NextConfig.i18n?.locales; + + (locales || []).some((locale) => { + if ( + pathnameParts[1] && + pathnameParts[1].toLowerCase() === locale.toLowerCase() + ) { + pathnameParts.splice(1, 1); + pathname = pathnameParts.join("/") || "/"; + return true; + } + return false; + }); + return locales && !pathname.endsWith("/") ? `${pathname}/` : pathname; +} // if res.end() is return, the parent needs to return and not process next server export async function handleMiddleware( internalEvent: InternalEvent, ): Promise { const { rawPath, query } = internalEvent; - const hasMatch = middleMatch.some((r) => r.test(rawPath)); + const normalizedPath = normalizeLocalePath(rawPath); + const hasMatch = middleMatch.some((r) => r.test(normalizedPath)); if (!hasMatch) return internalEvent; // We bypass the middleware if the request is internal if (internalEvent.headers["x-isr"]) return internalEvent; - const req = new IncomingMessage(internalEvent); - const res = new ServerlessResponse({ - method: req.method ?? "GET", - headers: {}, - }); - - // NOTE: Next middleware was originally developed to support nested middlewares - // but that was discarded for simplicity. The MiddlewareInfo type still has the original - // structure, but as of now, the only useful property on it is the "/" key (ie root). - const middlewareInfo = middlewareManifest.middleware["/"]; - middlewareInfo.paths = middlewareInfo.files.map((file) => - path.join(NEXT_DIR, file), - ); - - const host = req.headers.host - ? `https://${req.headers.host}` + const host = internalEvent.headers.host + ? `https://${internalEvent.headers.host}` : "http://localhost:3000"; - const initialUrl = new URL(rawPath, host); + const initialUrl = new URL(normalizedPath, host); initialUrl.search = convertToQueryString(query); const url = initialUrl.toString(); - - const result: MiddlewareResult = await run({ - distDir: NEXT_DIR, - name: middlewareInfo.name || "/", - paths: middlewareInfo.paths || [], - edgeFunctionEntry: middlewareInfo, - request: { - headers: req.headers, - method: req.method || "GET", - nextConfig: { - basePath: NextConfig.basePath, - i18n: NextConfig.i18n, - trailingSlash: NextConfig.trailingSlash, - }, - url, - body: getCloneableBody(req), - signal: signalFromNodeResponse(res), + // console.log("url", url, normalizedPath); + + // @ts-expect-error - This is bundled + const middleware = await import("./middleware.mjs"); + + const result: Response = await middleware.default({ + geo: { + city: internalEvent.headers["x-open-next-city"], + country: internalEvent.headers["x-open-next-country"], + region: internalEvent.headers["x-open-next-region"], + latitude: internalEvent.headers["x-open-next-latitude"], + longitude: internalEvent.headers["x-open-next-longitude"], }, - useCache: true, - onWarning: console.warn, + headers: internalEvent.headers, + method: internalEvent.method || "GET", + nextConfig: { + basePath: NextConfig.basePath, + i18n: NextConfig.i18n, + trailingSlash: NextConfig.trailingSlash, + }, + url, + body: convertBodyToReadableStream(internalEvent.method, internalEvent.body), }); - res.statusCode = result.response.status; + const statusCode = result.status; /* Apply override headers from middleware NextResponse.next({ @@ -103,9 +104,9 @@ export async function handleMiddleware( We can delete `x-middleware-override-headers` and check if the key starts with x-middleware-request- to set the req headers */ - const responseHeaders = result.response.headers as Headers; + const responseHeaders = result.headers as Headers; const reqHeaders: Record = {}; - const resHeaders: Record = {}; + const resHeaders: Record = {}; responseHeaders.delete("x-middleware-override-headers"); const xMiddlewareKey = "x-middleware-request-"; @@ -113,25 +114,32 @@ export async function handleMiddleware( if (key.startsWith(xMiddlewareKey)) { const k = key.substring(xMiddlewareKey.length); reqHeaders[k] = value; - req.headers[k] = value; } else { - resHeaders[key] = value; - res.setHeader(key, value); + if (key.toLowerCase() === "set-cookie") { + resHeaders[key] = resHeaders[key] + ? [...resHeaders[key], value] + : [value]; + } else { + resHeaders[key] = value; + } } }); // If the middleware returned a Redirect, we set the `Location` header with // the redirected url and end the response. - if (res.statusCode >= 300 && res.statusCode < 400) { - resHeaders.location = resHeaders.location?.replace( - "http://localhost:3000", - `https://${req.headers.host}`, - ); + if (statusCode >= 300 && statusCode < 400) { + resHeaders.location = + responseHeaders + .get("location") + ?.replace( + "http://localhost:3000", + `https://${internalEvent.headers.host}`, + ) ?? resHeaders.location; // res.setHeader("Location", location); return { body: "", type: internalEvent.type, - statusCode: res.statusCode, + statusCode: statusCode, headers: resHeaders, isBase64Encoded: false, }; @@ -143,14 +151,16 @@ export async function handleMiddleware( let rewritten = false; let externalRewrite = false; let middlewareQueryString = internalEvent.query; + let newUrl = internalEvent.url; if (rewriteUrl) { - if (isExternal(rewriteUrl, req.headers.host)) { - req.url = rewriteUrl; + // If not a string, it should probably throw + if (isExternal(rewriteUrl, internalEvent.headers.host as string)) { + newUrl = rewriteUrl; rewritten = true; externalRewrite = true; } else { const rewriteUrlObject = new URL(rewriteUrl); - req.url = rewriteUrlObject.pathname; + newUrl = rewriteUrlObject.pathname; //reset qs middlewareQueryString = {}; rewriteUrlObject.searchParams.forEach((v: string, k: string) => { @@ -162,24 +172,27 @@ export async function handleMiddleware( // If the middleware returned a `NextResponse`, pipe the body to res. This will return // the body immediately to the client. - if (result.response.body) { + if (result.body) { // transfer response body to res - const arrayBuffer = await result.response.arrayBuffer(); + const arrayBuffer = await result.arrayBuffer(); const buffer = Buffer.from(arrayBuffer); - res.end(buffer); + // res.end(buffer); // await pipeReadable(result.response.body, res); return { type: internalEvent.type, - ...convertRes(res), + statusCode: statusCode, + headers: resHeaders, + body: buffer.toString(), + isBase64Encoded: false, }; } return { responseHeaders: resHeaders, - url: req.url ?? internalEvent.url, + url: newUrl, rawPath: rewritten - ? req.url ?? internalEvent.rawPath + ? newUrl ?? internalEvent.rawPath : internalEvent.rawPath, type: internalEvent.type, headers: { ...internalEvent.headers, ...reqHeaders }, diff --git a/packages/open-next/src/core/routing/util.ts b/packages/open-next/src/core/routing/util.ts new file mode 100644 index 00000000..48d562d3 --- /dev/null +++ b/packages/open-next/src/core/routing/util.ts @@ -0,0 +1,480 @@ +import crypto from "node:crypto"; +import { OutgoingHttpHeaders } from "node:http"; + +import { BuildId, HtmlPages } from "config/index.js"; +import type { IncomingMessage, StreamCreator } from "http/index.js"; +import { OpenNextNodeResponse } from "http/openNextResponse.js"; +import { parseHeaders } from "http/util.js"; +import type { MiddlewareManifest } from "types/next-types"; +import { InternalEvent } from "types/open-next.js"; + +import { isBinaryContentType } from "../../adapters/binary.js"; +import { debug, error } from "../../adapters/logger.js"; + +/** + * + * @__PURE__ + */ +export function isExternal(url?: string, host?: string) { + if (!url) return false; + const pattern = /^https?:\/\//; + if (host) { + return pattern.test(url) && !url.includes(host); + } + return pattern.test(url); +} + +/** + * + * @__PURE__ + */ +export function getUrlParts(url: string, isExternal: boolean) { + // NOTE: when redirect to a URL that contains search query params, + // compile breaks b/c it does not allow for the '?' character + // We can't use encodeURIComponent because modal interception contains + // characters that can't be encoded + url = url.replaceAll("?", "%3F"); + if (!isExternal) { + return { + hostname: "", + pathname: url, + protocol: "", + }; + } + const { hostname, pathname, protocol } = new URL(url); + return { + hostname, + pathname, + protocol, + }; +} + +/** + * + * @__PURE__ + */ +export function convertRes(res: OpenNextNodeResponse) { + // Format Next.js response to Lambda response + const statusCode = res.statusCode || 200; + const headers = parseHeaders(res.headers); + const isBase64Encoded = isBinaryContentType( + Array.isArray(headers["content-type"]) + ? headers["content-type"][0] + : headers["content-type"], + ); + const encoding = isBase64Encoded ? "base64" : "utf8"; + const body = res.body.toString(encoding); + return { + statusCode, + headers, + body, + isBase64Encoded, + }; +} + +/** + * Make sure that multi-value query parameters are transformed to + * ?key=value1&key=value2&... so that Next converts those parameters + * to an array when reading the query parameters + * @__PURE__ + */ +export function convertToQueryString(query: Record) { + const urlQuery = new URLSearchParams(); + Object.entries(query).forEach(([key, value]) => { + if (Array.isArray(value)) { + value.forEach((entry) => urlQuery.append(key, entry)); + } else { + urlQuery.append(key, value); + } + }); + const queryString = urlQuery.toString(); + + return queryString ? `?${queryString}` : ""; +} + +/** + * Given a raw query string, returns a record with key value-array pairs + * similar to how multiValueQueryStringParameters are structured + * @__PURE__ + */ +export function convertToQuery(querystring: string) { + const query = new URLSearchParams(querystring); + const queryObject: Record = {}; + + for (const key of query.keys()) { + const queries = query.getAll(key); + queryObject[key] = queries.length > 1 ? queries : queries[0]; + } + + return queryObject; +} + +/** + * + * @__PURE__ + */ +export function getMiddlewareMatch(middlewareManifest: MiddlewareManifest) { + const rootMiddleware = middlewareManifest.middleware["/"]; + if (!rootMiddleware?.matchers) return []; + return rootMiddleware.matchers.map(({ regexp }) => new RegExp(regexp)); +} + +/** + * + * @__PURE__ + */ +export function escapeRegex(str: string) { + let path = str.replace(/\(\.\)/g, "_µ1_"); + + path = path.replace(/\(\.{2}\)/g, "_µ2_"); + + path = path.replace(/\(\.{3}\)/g, "_µ3_"); + + return path; +} + +/** + * + * @__PURE__ + */ +export function unescapeRegex(str: string) { + let path = str.replace(/_µ1_/g, "(.)"); + + path = path.replace(/_µ2_/g, "(..)"); + + path = path.replace(/_µ3_/g, "(...)"); + + return path; +} + +/** + * + * @__PURE__ + */ +function filterHeadersForProxy( + headers: Record, +) { + const filteredHeaders: Record = {}; + const disallowedHeaders = [ + "host", + "connection", + "via", + "x-cache", + "transfer-encoding", + "content-encoding", + ]; + Object.entries(headers).forEach(([key, value]) => { + const lowerKey = key.toLowerCase(); + if (disallowedHeaders.includes(lowerKey) || lowerKey.startsWith("x-amz")) + return; + else { + filteredHeaders[key] = value?.toString() ?? ""; + } + }); + return filteredHeaders; +} + +/** + * @__PURE__ + */ +export function convertBodyToReadableStream( + method: string, + body?: string | Buffer, +) { + if (method === "GET" || method === "HEAD") return undefined; + if (!body) return undefined; + const readable = new ReadableStream({ + start(controller) { + controller.enqueue(body); + controller.close(); + }, + }); + return readable; +} + +/** + * + * @__PURE__ + */ +export async function proxyRequest( + internalEvent: InternalEvent, + res: OpenNextNodeResponse, +) { + const { url, headers, method, body } = internalEvent; + const request = await import("node:https").then((m) => m.request); + debug("proxyRequest", url); + await new Promise((resolve, reject) => { + const filteredHeaders = filterHeadersForProxy(headers); + debug("filteredHeaders", filteredHeaders); + const req = request( + url, + { + headers: filteredHeaders, + method, + rejectUnauthorized: false, + }, + (_res) => { + res.writeHead( + _res.statusCode ?? 200, + filterHeadersForProxy(_res.headers), + ); + if (_res.headers["content-encoding"] === "br") { + _res.pipe(require("node:zlib").createBrotliDecompress()).pipe(res); + } else if (_res.headers["content-encoding"] === "gzip") { + _res.pipe(require("node:zlib").createGunzip()).pipe(res); + } else { + _res.pipe(res); + } + + _res.on("error", (e) => { + error("proxyRequest error", e); + res.end(); + reject(e); + }); + _res.on("end", () => { + resolve(); + }); + }, + ); + + if (body && method !== "GET" && method !== "HEAD") { + req.write(body); + } + req.end(); + }); + // console.log("result", result); + // res.writeHead(result.status, resHeaders); + // res.end(await result.text()); +} + +declare global { + var openNextDebug: boolean; + var openNextVersion: string; + var lastModified: Record; +} + +enum CommonHeaders { + CACHE_CONTROL = "cache-control", + NEXT_CACHE = "x-nextjs-cache", +} + +/** + * + * @__PURE__ + */ +export function fixCacheHeaderForHtmlPages( + rawPath: string, + headers: OutgoingHttpHeaders, +) { + // WORKAROUND: `NextServer` does not set cache headers for HTML pages — https://github.com/serverless-stack/open-next#workaround-nextserver-does-not-set-cache-headers-for-html-pages + if (HtmlPages.includes(rawPath)) { + headers[CommonHeaders.CACHE_CONTROL] = + "public, max-age=0, s-maxage=31536000, must-revalidate"; + } +} + +/** + * + * @__PURE__ + */ +export function fixSWRCacheHeader(headers: OutgoingHttpHeaders) { + // WORKAROUND: `NextServer` does not set correct SWR cache headers — https://github.com/sst/open-next#workaround-nextserver-does-not-set-correct-swr-cache-headers + let cacheControl = headers[CommonHeaders.CACHE_CONTROL]; + if (!cacheControl) return; + if (Array.isArray(cacheControl)) { + cacheControl = cacheControl.join(","); + } + if (typeof cacheControl !== "string") return; + headers[CommonHeaders.CACHE_CONTROL] = cacheControl.replace( + /\bstale-while-revalidate(?!=)/, + "stale-while-revalidate=2592000", // 30 days + ); +} + +/** + * + * @__PURE__ + */ +export function addOpenNextHeader(headers: OutgoingHttpHeaders) { + headers["X-OpenNext"] = "1"; + if (globalThis.openNextDebug) { + headers["X-OpenNext-Version"] = globalThis.openNextVersion; + headers["X-OpenNext-RequestId"] = globalThis.__als.getStore(); + } +} + +/** + * + * @__PURE__ + */ +export async function revalidateIfRequired( + host: string, + rawPath: string, + headers: OutgoingHttpHeaders, + req?: IncomingMessage, +) { + if (headers[CommonHeaders.NEXT_CACHE] === "STALE") { + // If the URL is rewritten, revalidation needs to be done on the rewritten URL. + // - Link to Next.js doc: https://nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration#on-demand-revalidation + // - Link to NextInternalRequestMeta: https://github.com/vercel/next.js/blob/57ab2818b93627e91c937a130fb56a36c41629c3/packages/next/src/server/request-meta.ts#L11 + // @ts-ignore + const internalMeta = req?.[Symbol.for("NextInternalRequestMeta")]; + + // When using Pages Router, two requests will be received: + // 1. one for the page: /foo + // 2. one for the json data: /_next/data/BUILD_ID/foo.json + // The rewritten url is correct for 1, but that for the second request + // does not include the "/_next/data/" prefix. Need to add it. + const revalidateUrl = internalMeta?._nextDidRewrite + ? rawPath.startsWith("/_next/data/") + ? `/_next/data/${BuildId}${internalMeta?._nextRewroteUrl}.json` + : internalMeta?._nextRewroteUrl + : rawPath; + + // We need to pass etag to the revalidation queue to try to bypass the default 5 min deduplication window. + // https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html + // If you need to have a revalidation happen more frequently than 5 minutes, + // your page will need to have a different etag to bypass the deduplication window. + // If data has the same etag during these 5 min dedup window, it will be deduplicated and not revalidated. + try { + const hash = (str: string) => + crypto.createHash("md5").update(str).digest("hex"); + const requestId = globalThis.__als.getStore() ?? ""; + + const lastModified = + globalThis.lastModified[requestId] > 0 + ? globalThis.lastModified[requestId] + : ""; + + // For some weird cases, lastModified is not set, haven't been able to figure out yet why + // For those cases we add the etag to the deduplication id, it might help + const etag = headers["etag"] ?? headers["ETag"] ?? ""; + + await globalThis.queue.send({ + MessageBody: { host, url: revalidateUrl }, + MessageDeduplicationId: hash(`${rawPath}-${lastModified}-${etag}`), + MessageGroupId: generateMessageGroupId(rawPath), + }); + } catch (e) { + debug(`Failed to revalidate stale page ${rawPath}`); + debug(e); + } + } +} + +// Since we're using a FIFO queue, every messageGroupId is treated sequentially +// This could cause a backlog of messages in the queue if there is too much page to +// revalidate at once. To avoid this, we generate a random messageGroupId for each +// revalidation request. +// We can't just use a random string because we need to ensure that the same rawPath +// will always have the same messageGroupId. +// https://stackoverflow.com/questions/521295/seeding-the-random-number-generator-in-javascript#answer-47593316 +function generateMessageGroupId(rawPath: string) { + let a = cyrb128(rawPath); + // We use mulberry32 to generate a random int between 0 and MAX_REVALIDATE_CONCURRENCY + var t = (a += 0x6d2b79f5); + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + const randomFloat = ((t ^ (t >>> 14)) >>> 0) / 4294967296; + // This will generate a random int between 0 and MAX_REVALIDATE_CONCURRENCY + // This means that we could have 1000 revalidate request at the same time + const maxConcurrency = parseInt( + process.env.MAX_REVALIDATE_CONCURRENCY ?? "10", + ); + const randomInt = Math.floor(randomFloat * maxConcurrency); + return `revalidate-${randomInt}`; +} + +// Used to generate a hash int from a string +function cyrb128(str: string) { + let h1 = 1779033703, + h2 = 3144134277, + h3 = 1013904242, + h4 = 2773480762; + for (let i = 0, k; i < str.length; i++) { + k = str.charCodeAt(i); + h1 = h2 ^ Math.imul(h1 ^ k, 597399067); + h2 = h3 ^ Math.imul(h2 ^ k, 2869860233); + h3 = h4 ^ Math.imul(h3 ^ k, 951274213); + h4 = h1 ^ Math.imul(h4 ^ k, 2716044179); + } + h1 = Math.imul(h3 ^ (h1 >>> 18), 597399067); + h2 = Math.imul(h4 ^ (h2 >>> 22), 2869860233); + h3 = Math.imul(h1 ^ (h3 >>> 17), 951274213); + h4 = Math.imul(h2 ^ (h4 >>> 19), 2716044179); + (h1 ^= h2 ^ h3 ^ h4), (h2 ^= h1), (h3 ^= h1), (h4 ^= h1); + return h1 >>> 0; +} + +/** + * + * @__PURE__ + */ +export function fixISRHeaders(headers: OutgoingHttpHeaders) { + if (headers[CommonHeaders.NEXT_CACHE] === "REVALIDATED") { + headers[CommonHeaders.CACHE_CONTROL] = + "private, no-cache, no-store, max-age=0, must-revalidate"; + return; + } + const requestId = globalThis.__als.getStore() ?? ""; + const _lastModified = globalThis.lastModified[requestId] ?? 0; + if (headers[CommonHeaders.NEXT_CACHE] === "HIT" && _lastModified > 0) { + // calculate age + const age = Math.round((Date.now() - _lastModified) / 1000); + // extract s-maxage from cache-control + const regex = /s-maxage=(\d+)/; + const cacheControl = headers[CommonHeaders.CACHE_CONTROL]; + debug("cache-control", cacheControl, globalThis.lastModified, Date.now()); + if (typeof cacheControl !== "string") return; + const match = cacheControl.match(regex); + const sMaxAge = match ? parseInt(match[1]) : undefined; + + // 31536000 is the default s-maxage value for SSG pages + if (sMaxAge && sMaxAge !== 31536000) { + const remainingTtl = Math.max(sMaxAge - age, 1); + headers[ + CommonHeaders.CACHE_CONTROL + ] = `s-maxage=${remainingTtl}, stale-while-revalidate=2592000`; + } + } + if (headers[CommonHeaders.NEXT_CACHE] !== "STALE") return; + + // If the cache is stale, we revalidate in the background + // In order for CloudFront SWR to work, we set the stale-while-revalidate value to 2 seconds + // This will cause CloudFront to cache the stale data for a short period of time while we revalidate in the background + // Once the revalidation is complete, CloudFront will serve the fresh data + headers[CommonHeaders.CACHE_CONTROL] = + "s-maxage=2, stale-while-revalidate=2592000"; +} + +/** + * + * @param internalEvent + * @param headers + * @param responseStream + * @returns + * @__PURE__ + */ +export function createServerResponse( + internalEvent: InternalEvent, + headers: Record, + responseStream?: StreamCreator, +) { + return new OpenNextNodeResponse( + (_headers) => { + fixCacheHeaderForHtmlPages(internalEvent.rawPath, _headers); + fixSWRCacheHeader(_headers); + addOpenNextHeader(_headers); + fixISRHeaders(_headers); + }, + async (_headers) => { + await revalidateIfRequired( + internalEvent.headers.host, + internalEvent.rawPath, + _headers, + ); + }, + responseStream, + headers, + ); +} diff --git a/packages/open-next/src/core/routingHandler.ts b/packages/open-next/src/core/routingHandler.ts new file mode 100644 index 00000000..9b71fcdd --- /dev/null +++ b/packages/open-next/src/core/routingHandler.ts @@ -0,0 +1,109 @@ +import { + BuildId, + ConfigHeaders, + PrerenderManifest, + RoutesManifest, +} from "config/index"; +import { InternalEvent, InternalResult, Origin } from "types/open-next"; + +import { debug } from "../adapters/logger"; +import { + addNextConfigHeaders, + fixDataPage, + handleFallbackFalse, + handleRedirects, + handleRewrites, +} from "./routing/matcher"; +import { handleMiddleware } from "./routing/middleware"; + +export interface MiddlewareOutputEvent { + internalEvent: InternalEvent; + isExternalRewrite: boolean; + origin: Origin | false; +} + +export default async function routingHandler( + event: InternalEvent, +): Promise { + const nextHeaders = addNextConfigHeaders(event, ConfigHeaders) ?? {}; + + let internalEvent = fixDataPage(event, BuildId); + if ("statusCode" in internalEvent) { + return internalEvent; + } + + const redirect = handleRedirects(internalEvent, RoutesManifest.redirects); + if (redirect) { + debug("redirect", redirect); + return redirect; + } + + const middleware = await handleMiddleware(internalEvent); + let middlewareResponseHeaders: Record = {}; + if ("statusCode" in middleware) { + return middleware; + } else { + middlewareResponseHeaders = middleware.responseHeaders || {}; + internalEvent = middleware; + } + + let isExternalRewrite = middleware.externalRewrite ?? false; + if (!isExternalRewrite) { + // First rewrite to be applied + const beforeRewrites = handleRewrites( + internalEvent, + RoutesManifest.rewrites.beforeFiles, + ); + internalEvent = beforeRewrites.internalEvent; + isExternalRewrite = beforeRewrites.isExternalRewrite; + } + const isStaticRoute = RoutesManifest.routes.static.some((route) => + new RegExp(route.regex).test(event.rawPath), + ); + + if (!isStaticRoute && !isExternalRewrite) { + // Second rewrite to be applied + const afterRewrites = handleRewrites( + internalEvent, + RoutesManifest.rewrites.afterFiles, + ); + internalEvent = afterRewrites.internalEvent; + isExternalRewrite = afterRewrites.isExternalRewrite; + } + + // We want to run this just before the dynamic route check + internalEvent = handleFallbackFalse(internalEvent, PrerenderManifest); + + const isDynamicRoute = RoutesManifest.routes.dynamic.some((route) => + new RegExp(route.regex).test(event.rawPath), + ); + if (!isDynamicRoute && !isStaticRoute && !isExternalRewrite) { + // Fallback rewrite to be applied + const fallbackRewrites = handleRewrites( + internalEvent, + RoutesManifest.rewrites.fallback, + ); + internalEvent = fallbackRewrites.internalEvent; + isExternalRewrite = fallbackRewrites.isExternalRewrite; + } + + // We apply the headers from the middleware response last + Object.entries({ + ...middlewareResponseHeaders, + ...nextHeaders, + }).forEach(([key, value]) => { + if (value) { + internalEvent.headers[`x-middleware-response-${key}`] = Array.isArray( + value, + ) + ? value.join(",") + : value; + } + }); + + return { + internalEvent, + isExternalRewrite, + origin: false, + }; +} diff --git a/packages/open-next/src/adapters/plugins/util.ts b/packages/open-next/src/core/util.ts similarity index 78% rename from packages/open-next/src/adapters/plugins/util.ts rename to packages/open-next/src/core/util.ts index 5f846897..e9782210 100644 --- a/packages/open-next/src/adapters/plugins/util.ts +++ b/packages/open-next/src/core/util.ts @@ -1,20 +1,20 @@ import fs from "node:fs"; import path from "node:path"; -// @ts-ignore -import NextServer from "next/dist/server/next-server.js"; - import { AppPathsManifestKeys, NextConfig, RoutesManifest, -} from "../config/index.js"; -import { debug } from "../logger.js"; +} from "config/index.js"; +// @ts-ignore +import NextServer from "next/dist/server/next-server.js"; +import type { MiddlewareManifest } from "types/next-types.js"; + +import { debug } from "../adapters/logger.js"; import { applyOverride as applyNextjsRequireHooksOverride, overrideHooks as overrideNextjsRequireHooks, -} from "../require-hooks.js"; -import { MiddlewareManifest } from "../types/next-types.js"; +} from "./require-hooks.js"; // WORKAROUND: Set `__NEXT_PRIVATE_PREBUNDLED_REACT` to use prebundled React — https://github.com/serverless-stack/open-next#workaround-set-__next_private_prebundled_react-to-use-prebundled-react // Step 1: Need to override the require hooks for React before Next.js server @@ -27,12 +27,13 @@ import { MiddlewareManifest } from "../types/next-types.js"; overrideNextjsRequireHooks(NextConfig); applyNextjsRequireHooksOverride(); //#endOverride - -//#override requestHandler +const cacheHandlerPath = require.resolve("./cache.cjs"); // @ts-ignore export const requestHandler = new NextServer.default({ + //#override requestHandlerHost hostname: "localhost", port: 3000, + //#endOverride conf: { ...NextConfig, // Next.js compression should be disabled because of a bug in the bundled @@ -40,16 +41,26 @@ export const requestHandler = new NextServer.default({ compress: false, // By default, Next.js uses local disk to store ISR cache. We will use // our own cache handler to store the cache on S3. + //#override stableIncrementalCache + cacheHandler: cacheHandlerPath, + cacheMaxMemorySize: 0, // We need to disable memory cache + //#endOverride experimental: { ...NextConfig.experimental, - incrementalCacheHandlerPath: `${process.env.LAMBDA_TASK_ROOT}/cache.cjs`, + // This uses the request.headers.host as the URL + // https://github.com/vercel/next.js/blob/canary/packages/next/src/server/next-server.ts#L1749-L1754 + //#override trustHostHeader + trustHostHeader: true, + //#endOverride + //#override experimentalIncrementalCacheHandler + incrementalCacheHandlerPath: cacheHandlerPath, + //#endOverride }, }, customServer: false, dev: false, dir: __dirname, }).getRequestHandler(); -//#endOverride export function getMiddlewareMatch(middlewareManifest: MiddlewareManifest) { const rootMiddleware = middlewareManifest.middleware["/"]; diff --git a/packages/open-next/src/helpers/withCloudflare.ts b/packages/open-next/src/helpers/withCloudflare.ts new file mode 100644 index 00000000..37f486d2 --- /dev/null +++ b/packages/open-next/src/helpers/withCloudflare.ts @@ -0,0 +1,118 @@ +import { + FunctionOptions, + OpenNextConfig, + RouteTemplate, + SplittedFunctionOptions, +} from "types/open-next"; + +type CloudflareCompatibleFunction = + Placement extends "regional" + ? FunctionOptions & { + placement: "regional"; + } + : { placement: "global" }; + +type CloudflareCompatibleRoutes = + Placement extends "regional" + ? { + placement: "regional"; + routes: RouteTemplate[]; + patterns: string[]; + } + : { + placement: "global"; + routes: `app/${string}/route`; + patterns: string; + }; + +type CloudflareCompatibleSplittedFunction< + Placement extends "regional" | "global" = "regional", +> = CloudflareCompatibleRoutes & + CloudflareCompatibleFunction; + +type CloudflareConfig< + Fn extends Record< + string, + CloudflareCompatibleSplittedFunction<"global" | "regional"> + >, +> = { + default: CloudflareCompatibleFunction<"regional">; + functions?: Fn; +} & Omit; + +type InterpolatedSplittedFunctionOptions< + Fn extends Record< + string, + CloudflareCompatibleSplittedFunction<"global" | "regional"> + >, +> = { + [K in keyof Fn]: SplittedFunctionOptions; +}; + +/** + * This function makes it easier to use Cloudflare with OpenNext. + * All options are already restricted to Cloudflare compatible options. + * @example + * ```ts + export default withCloudflare({ + default: { + placement: "regional", + runtime: "node", + }, + functions: { + api: { + placement: "regional", + runtime: "node", + routes: ["app/api/test/route", "page/api/otherApi"], + patterns: ["/api/*"], + }, + global: { + placement: "global", + runtime: "edge", + routes: "app/test/page", + patterns: "/page", + }, + }, +}); + * ``` + */ +export function withCloudflare< + Fn extends Record< + string, + CloudflareCompatibleSplittedFunction<"global" | "regional"> + >, + Key extends keyof Fn, +>(config: CloudflareConfig) { + const functions = Object.entries(config.functions ?? {}).reduce( + (acc, [name, fn]) => { + const _name = name as Key; + acc[_name] = + fn.placement === "global" + ? { + placement: "global", + runtime: "edge", + routes: [fn.routes], + patterns: [fn.patterns], + override: { + wrapper: "cloudflare", + converter: "edge", + }, + } + : { ...fn, placement: "regional" }; + return acc; + }, + {} as InterpolatedSplittedFunctionOptions, + ); + return { + default: config.default, + functions: functions, + middleware: { + external: true, + originResolver: "pattern-env", + override: { + wrapper: "cloudflare", + converter: "edge", + }, + }, + } satisfies OpenNextConfig; +} diff --git a/packages/open-next/src/helpers/withSST.ts b/packages/open-next/src/helpers/withSST.ts new file mode 100644 index 00000000..afe11b33 --- /dev/null +++ b/packages/open-next/src/helpers/withSST.ts @@ -0,0 +1,65 @@ +import { + FunctionOptions, + OpenNextConfig, + RouteTemplate, +} from "types/open-next"; + +type SSTCompatibleFunction = FunctionOptions & { + override?: { + wrapper?: "aws-lambda-streaming" | "aws-lambda"; + converter?: "aws-apigw-v2" | "aws-apigw-v1" | "aws-cloudfront"; + }; +}; + +type SSTCompatibleSplittedFunction = { + routes: RouteTemplate[]; + patterns: string[]; +} & SSTCompatibleFunction; + +type SSTCompatibleConfig< + Fn extends Record, +> = { + default: SSTCompatibleFunction; + functions?: Fn; + middleware?: { + external: true; + }; +} & Pick< + OpenNextConfig, + | "dangerous" + | "appPath" + | "buildCommand" + | "buildOutputPath" + | "packageJsonPath" +>; + +/** + * This function makes it more straightforward to use SST with OpenNext. + * All options are already restricted to SST compatible options only. + * Some options not present here can be used in SST, but it's an advanced use case that + * can easily break the deployment. If you need to use those options, you should just provide a + * compatible OpenNextConfig inside your `open-next.config.ts` file. + * @example + * ```ts + export default withSST({ + default: { + override: { + wrapper: "aws-lambda-streaming", + }, + }, + functions: { + "api/*": { + routes: ["app/api/test/route", "page/api/otherApi"], + patterns: ["/api/*"], + }, + }, + }); + * ``` + */ +export function withSST< + Fn extends Record, +>(config: SSTCompatibleConfig) { + return { + ...config, + } satisfies OpenNextConfig; +} diff --git a/packages/open-next/src/http/index.ts b/packages/open-next/src/http/index.ts new file mode 100644 index 00000000..49efb2fe --- /dev/null +++ b/packages/open-next/src/http/index.ts @@ -0,0 +1,4 @@ +// @__PURE__ +export * from "./openNextResponse.js"; +// @__PURE__ +export * from "./request.js"; diff --git a/packages/open-next/src/http/openNextResponse.ts b/packages/open-next/src/http/openNextResponse.ts new file mode 100644 index 00000000..c2ca5aae --- /dev/null +++ b/packages/open-next/src/http/openNextResponse.ts @@ -0,0 +1,285 @@ +import type { + IncomingMessage, + OutgoingHttpHeader, + OutgoingHttpHeaders, + ServerResponse, +} from "http"; +import { Socket } from "net"; +import { Transform, TransformCallback, Writable } from "stream"; + +import { parseCookies, parseHeaders } from "./util"; + +const SET_COOKIE_HEADER = "set-cookie"; +const CANNOT_BE_USED = "This cannot be used in OpenNext"; + +export interface StreamCreator { + writeHeaders(prelude: { + statusCode: number; + cookies: string[]; + headers: Record; + }): Writable; + // Just to fix an issue with aws lambda streaming with empty body + onWrite?: () => void; + onFinish: () => void; +} + +// We only need to implement the methods that are used by next.js +export class OpenNextNodeResponse extends Transform implements ServerResponse { + statusCode!: number; + statusMessage: string = ""; + headers: OutgoingHttpHeaders = {}; + private _cookies: string[] = []; + private responseStream?: Writable; + headersSent: boolean = false; + _chunks: Buffer[] = []; + + // To comply with the ServerResponse interface : + strictContentLength: boolean = false; + assignSocket(_socket: Socket): void { + throw new Error(CANNOT_BE_USED); + } + detachSocket(_socket: Socket): void { + throw new Error(CANNOT_BE_USED); + } + // We might have to revisit those 3 in the future + writeContinue(_callback?: (() => void) | undefined): void { + throw new Error(CANNOT_BE_USED); + } + writeEarlyHints( + _hints: Record, + _callback?: (() => void) | undefined, + ): void { + throw new Error(CANNOT_BE_USED); + } + writeProcessing(): void { + throw new Error(CANNOT_BE_USED); + } + /** + * This is a dummy request object to comply with the ServerResponse interface + * It will never be defined + */ + req!: IncomingMessage; + chunkedEncoding: boolean = false; + shouldKeepAlive: boolean = true; + useChunkedEncodingByDefault: boolean = true; + sendDate: boolean = false; + connection: Socket | null = null; + socket: Socket | null = null; + setTimeout(_msecs: number, _callback?: (() => void) | undefined): this { + throw new Error(CANNOT_BE_USED); + } + addTrailers( + _headers: OutgoingHttpHeaders | readonly [string, string][], + ): void { + throw new Error(CANNOT_BE_USED); + } + + constructor( + private fixHeaders: (headers: OutgoingHttpHeaders) => void, + onEnd: (headers: OutgoingHttpHeaders) => Promise, + private streamCreator?: StreamCreator, + private initialHeaders?: OutgoingHttpHeaders, + ) { + super(); + if (initialHeaders && initialHeaders[SET_COOKIE_HEADER]) { + this._cookies = parseCookies( + initialHeaders[SET_COOKIE_HEADER] as string | string[], + ) as string[]; + } + this.once("finish", () => { + if (!this.headersSent) { + this.flushHeaders(); + } + onEnd(this.headers); + this.streamCreator?.onFinish(); + }); + } + + // Necessary for next 12 + // We might have to implement all the methods here + get originalResponse() { + return this; + } + + get finished() { + return Boolean( + this.writableFinished && this.responseStream?.writableFinished, + ); + } + + setHeader(name: string, value: string | string[]): this { + const key = name.toLowerCase(); + if (key === SET_COOKIE_HEADER) { + if (Array.isArray(value)) { + this._cookies = value; + } else { + this._cookies = [value]; + } + } + // We should always replace the header + // See https://nodejs.org/docs/latest-v18.x/api/http.html#responsesetheadername-value + this.headers[key] = value; + + return this; + } + + removeHeader(name: string): this { + const key = name.toLowerCase(); + if (key === SET_COOKIE_HEADER) { + this._cookies = []; + } else { + delete this.headers[key]; + } + return this; + } + + hasHeader(name: string): boolean { + const key = name.toLowerCase(); + if (key === SET_COOKIE_HEADER) { + return this._cookies.length > 0; + } + return this.headers[key] !== undefined; + } + + getHeaders(): OutgoingHttpHeaders { + return this.headers; + } + + getHeader(name: string): OutgoingHttpHeader | undefined { + return this.headers[name.toLowerCase()]; + } + + getHeaderNames(): string[] { + return Object.keys(this.headers); + } + + // Only used directly in next@14+ + flushHeaders() { + this.headersSent = true; + this.fixHeaders(this.headers); + // Initial headers should be merged with the new headers + // These initial headers are the one created either in the middleware or in next.config.js + // We choose to override response headers with middleware headers + // This is different than the default behavior in next.js, but it allows more customization + // TODO: We probably want to change this behavior in the future to follow next + // We could add a prefix header that would allow to force the middleware headers + // Something like open-next-force-cache-control would override the cache-control header + if (this.initialHeaders) { + this.headers = { + ...this.headers, + ...this.initialHeaders, + }; + } + if (this._cookies.length > 0) { + // For cookies we cannot do the same as for other headers + // We need to merge the cookies, and in this case, cookies generated by the routes or pages + // should be added after the ones generated by the middleware + // This prevents the middleware from overriding the cookies, especially for server actions + // which uses the same pathnames as the pages they're being called on + this.headers[SET_COOKIE_HEADER] = [ + ...(parseCookies( + this.initialHeaders?.[SET_COOKIE_HEADER] as string | string[], + ) ?? []), + ...this._cookies, + ]; + } + + if (this.streamCreator) { + this.responseStream = this.streamCreator?.writeHeaders({ + statusCode: this.statusCode ?? 200, + cookies: this._cookies, + headers: parseHeaders(this.headers), + }); + this.pipe(this.responseStream); + } + } + + appendHeader(name: string, value: string | string[]): this { + const key = name.toLowerCase(); + if (!this.hasHeader(key)) { + return this.setHeader(key, value); + } else { + const existingHeader = this.getHeader(key) as string | string[]; + const toAppend = Array.isArray(value) ? value : [value]; + const newValue = Array.isArray(existingHeader) + ? [...existingHeader, ...toAppend] + : [existingHeader, ...toAppend]; + return this.setHeader(key, newValue); + } + } + + // Might be used in next page api routes + writeHead( + statusCode: number, + statusMessage?: string | undefined, + headers?: OutgoingHttpHeaders | OutgoingHttpHeader[] | undefined, + ): this; + writeHead( + statusCode: number, + headers?: OutgoingHttpHeaders | OutgoingHttpHeader[] | undefined, + ): this; + writeHead( + statusCode: unknown, + statusMessage?: unknown, + headers?: unknown, + ): this { + let _headers = headers as + | OutgoingHttpHeaders + | OutgoingHttpHeader[] + | undefined; + let _statusMessage: string | undefined; + if (typeof statusMessage === "string") { + _statusMessage = statusMessage; + } else { + _headers = statusMessage as + | OutgoingHttpHeaders + | OutgoingHttpHeader[] + | undefined; + } + const finalHeaders: OutgoingHttpHeaders = this.headers; + if (_headers) { + if (Array.isArray(_headers)) { + // headers may be an Array where the keys and values are in the same list. It is not a list of tuples. So, the even-numbered offsets are key values, and the odd-numbered offsets are the associated values. + for (let i = 0; i < _headers.length; i += 2) { + finalHeaders[_headers[i] as string] = _headers[i + 1] as + | string + | string[]; + } + } else { + for (const key of Object.keys(_headers)) { + finalHeaders[key] = _headers[key]; + } + } + } + + this.statusCode = statusCode as number; + if (headers) { + this.headers = finalHeaders; + } + this.flushHeaders(); + return this; + } + + get body() { + return Buffer.concat(this._chunks); + } + + private _internalWrite(chunk: any, encoding: BufferEncoding) { + this._chunks.push(Buffer.from(chunk, encoding)); + this.push(chunk, encoding); + this.streamCreator?.onWrite?.(); + } + + _transform( + chunk: any, + encoding: BufferEncoding, + callback: TransformCallback, + ): void { + if (!this.headersSent) { + this.flushHeaders(); + } + + this._internalWrite(chunk, encoding); + callback(); + } +} diff --git a/packages/open-next/src/adapters/http/request.ts b/packages/open-next/src/http/request.ts similarity index 98% rename from packages/open-next/src/adapters/http/request.ts rename to packages/open-next/src/http/request.ts index 18013256..d33b1395 100644 --- a/packages/open-next/src/adapters/http/request.ts +++ b/packages/open-next/src/http/request.ts @@ -16,7 +16,7 @@ export class IncomingMessage extends http.IncomingMessage { method: string; url: string; headers: Record; - body: Buffer; + body?: Buffer; remoteAddress: string; }) { super({ diff --git a/packages/open-next/src/http/util.ts b/packages/open-next/src/http/util.ts new file mode 100644 index 00000000..0e3be5fd --- /dev/null +++ b/packages/open-next/src/http/util.ts @@ -0,0 +1,42 @@ +import http from "node:http"; + +export const parseHeaders = ( + headers?: http.OutgoingHttpHeader[] | http.OutgoingHttpHeaders, +) => { + const result: Record = {}; + if (!headers) { + return result; + } + + for (const [key, value] of Object.entries(headers)) { + if (value === undefined) { + continue; + } else { + result[key] = convertHeader(value); + } + } + + return result; +}; + +export const convertHeader = (header: http.OutgoingHttpHeader) => { + if (typeof header === "string") { + return header; + } else if (Array.isArray(header)) { + return header.join(","); + } else { + return String(header); + } +}; + +export function parseCookies( + cookies?: string | string[], +): string[] | undefined { + if (!cookies) return; + + if (typeof cookies === "string") { + return cookies.split(/(? c.trim()); + } + + return cookies; +} diff --git a/packages/open-next/src/index.ts b/packages/open-next/src/index.ts index eb66a719..ec28ccf6 100755 --- a/packages/open-next/src/index.ts +++ b/packages/open-next/src/index.ts @@ -8,22 +8,7 @@ if (command !== "build") printHelp(); const args = parseArgs(); if (Object.keys(args).includes("--help")) printHelp(); -build({ - buildCommand: args["--build-command"], - buildOutputPath: args["--build-output-path"], - appPath: args["--app-path"], - packageJsonPath: args["--package-json"], - minify: Object.keys(args).includes("--minify"), - streaming: Object.keys(args).includes("--streaming"), - dangerous: { - disableDynamoDBCache: Object.keys(args).includes( - "--dangerously-disable-dynamodb-cache", - ), - disableIncrementalCache: Object.keys(args).includes( - "--dangerously-disable-incremental-cache", - ), - }, -}); +build(args["--config-path"]); function parseArgs() { return process.argv.slice(2).reduce( @@ -48,7 +33,9 @@ function printHelp() { console.log(""); console.log("Usage:"); console.log(" npx open-next build"); - console.log(" npx open-next build --build-command 'npm run custom:build'"); + console.log( + " npx open-next build --config-path ./path/to/open-next.config.ts", + ); console.log(""); process.exit(1); diff --git a/packages/open-next/src/logger.ts b/packages/open-next/src/logger.ts index b44dc254..2d56fb9e 100644 --- a/packages/open-next/src/logger.ts +++ b/packages/open-next/src/logger.ts @@ -1,3 +1,5 @@ +import chalk from "chalk"; + type LEVEL = "info" | "debug"; let logLevel: LEVEL = "info"; @@ -6,9 +8,9 @@ export default { setLevel: (level: LEVEL) => (logLevel = level), debug: (...args: any[]) => { if (logLevel !== "debug") return; - console.log("DEBUG", ...args); + console.log(chalk.magenta("DEBUG"), ...args); }, info: console.log, - warn: console.warn, - error: console.error, + warn: (...args: any[]) => console.warn(chalk.yellow("WARN"), ...args), + error: (...args: any[]) => console.error(chalk.red("ERROR"), ...args), }; diff --git a/packages/open-next/src/plugins/edge.ts b/packages/open-next/src/plugins/edge.ts new file mode 100644 index 00000000..e22e8d1d --- /dev/null +++ b/packages/open-next/src/plugins/edge.ts @@ -0,0 +1,185 @@ +import { readFileSync } from "node:fs"; +import path from "node:path"; + +import { Plugin } from "esbuild"; +import { MiddlewareInfo } from "types/next-types.js"; + +import { + loadAppPathsManifestKeys, + loadBuildId, + loadConfig, + loadConfigHeaders, + loadHtmlPages, + loadMiddlewareManifest, + loadPrerenderManifest, + loadRoutesManifest, +} from "../adapters/config/util.js"; + +export interface IPluginSettings { + nextDir: string; + edgeFunctionHandlerPath?: string; + middlewareInfo: MiddlewareInfo; + isInCloudfare?: boolean; +} + +/** + * @param opts.nextDir - The path to the .next directory + * @param opts.edgeFunctionHandlerPath - The path to the edgeFunctionHandler.js file that we'll use to bundle the routing + * @param opts.entryFiles - The entry files that we'll inject into the edgeFunctionHandler.js file + * @returns + */ +export function openNextEdgePlugins({ + nextDir, + edgeFunctionHandlerPath, + middlewareInfo, + isInCloudfare, +}: IPluginSettings): Plugin { + const entryFiles = middlewareInfo.files.map((file: string) => + path.join(nextDir, file), + ); + const routes = [ + { + name: middlewareInfo.name || "/", + page: middlewareInfo.page, + regex: middlewareInfo.matchers.map((m) => m.regexp), + }, + ]; + const wasmFiles = middlewareInfo.wasm ?? []; + return { + name: "opennext-edge", + setup(build) { + if (edgeFunctionHandlerPath) { + // If we bundle the routing, we need to resolve the middleware + build.onResolve({ filter: /\.\/middleware.mjs/g }, () => { + return { + path: edgeFunctionHandlerPath, + }; + }); + } + + build.onResolve({ filter: /\.(mjs|wasm)$/g }, (args) => { + return { + external: true, + }; + }); + + //Copied from https://github.com/cloudflare/next-on-pages/blob/7a18efb5cab4d86c8e3e222fc94ea88ac05baffd/packages/next-on-pages/src/buildApplication/processVercelFunctions/build.ts#L86-L112 + + build.onResolve({ filter: /^node:/ }, ({ kind, path }) => { + // this plugin converts `require("node:*")` calls, those are the only ones that + // need updating (esm imports to "node:*" are totally valid), so here we tag with the + // node-buffer namespace only imports that are require calls + return kind === "require-call" + ? { path, namespace: "node-built-in-modules" } + : undefined; + }); + + // we convert the imports we tagged with the node-built-in-modules namespace so that instead of `require("node:*")` + // they import from `export * from "node:*";` + build.onLoad( + { filter: /.*/, namespace: "node-built-in-modules" }, + ({ path }) => { + return { + contents: `export * from '${path}'`, + loader: "js", + }; + }, + ); + + // We inject the entry files into the edgeFunctionHandler + build.onLoad({ filter: /\/edgeFunctionHandler.js/g }, async (args) => { + let contents = readFileSync(args.path, "utf-8"); + contents = ` +globalThis._ENTRIES = {}; +globalThis.self = globalThis; +if(!globalThis.process){ + globalThis.process = {env: {}}; +} +globalThis._ROUTES = ${JSON.stringify(routes)}; + +import {Buffer} from "node:buffer"; +globalThis.Buffer = Buffer; +import crypto from "node:crypto"; +if(!globalThis.crypto){ + globalThis.crypto = crypto; +} + +import {AsyncLocalStorage} from "node:async_hooks"; +globalThis.AsyncLocalStorage = AsyncLocalStorage; +${ + isInCloudfare + ? `` + : ` +import {readFileSync} from "node:fs"; +import path from "node:path"; +function addDuplexToInit(init) { + return typeof init === 'undefined' || + (typeof init === 'object' && init.duplex === undefined) + ? { duplex: 'half', ...init } + : init +} +// We need to override Request to add duplex to the init, it seems Next expects it to work like this +class OverrideRequest extends Request { + constructor(input, init) { + super(input, addDuplexToInit(init)) + } +} +globalThis.Request = OverrideRequest; +` +} +${wasmFiles + .map((file) => + isInCloudfare + ? `import ${file.name} from './wasm/${file.name}.wasm';` + : `const ${file.name} = readFileSync(path.join(__dirname,'/wasm/${file.name}.wasm'));`, + ) + .join("\n")} +${entryFiles?.map((file) => `require("${file}");`).join("\n")} +${contents} + `; + return { + contents, + }; + }); + + build.onLoad({ filter: /adapters\/config\/index/g }, async () => { + const NextConfig = loadConfig(nextDir); + const BuildId = loadBuildId(nextDir); + const HtmlPages = loadHtmlPages(nextDir); + const RoutesManifest = loadRoutesManifest(nextDir); + const ConfigHeaders = loadConfigHeaders(nextDir); + const PrerenderManifest = loadPrerenderManifest(nextDir); + const AppPathsManifestKeys = loadAppPathsManifestKeys(nextDir); + const MiddlewareManifest = loadMiddlewareManifest(nextDir); + + const contents = ` + import path from "path"; + + import { debug } from "../logger"; + + if(!globalThis.__dirname) { + globalThis.__dirname = "" + } + + export const NEXT_DIR = path.join(__dirname, ".next"); + export const OPEN_NEXT_DIR = path.join(__dirname, ".open-next"); + + debug({ NEXT_DIR, OPEN_NEXT_DIR }); + + export const NextConfig = ${JSON.stringify(NextConfig)}; + export const BuildId = ${JSON.stringify(BuildId)}; + export const HtmlPages = ${JSON.stringify(HtmlPages)}; + export const RoutesManifest = ${JSON.stringify(RoutesManifest)}; + export const ConfigHeaders = ${JSON.stringify(ConfigHeaders)}; + export const PrerenderManifest = ${JSON.stringify(PrerenderManifest)}; + export const AppPathsManifestKeys = ${JSON.stringify(AppPathsManifestKeys)}; + export const MiddlewareManifest = ${JSON.stringify(MiddlewareManifest)}; + + `; + return { + contents, + }; + }); + }, + }; +} diff --git a/packages/open-next/src/plugin.ts b/packages/open-next/src/plugins/replacement.ts similarity index 62% rename from packages/open-next/src/plugin.ts rename to packages/open-next/src/plugins/replacement.ts index 90060f36..1e648fe1 100644 --- a/packages/open-next/src/plugin.ts +++ b/packages/open-next/src/plugins/replacement.ts @@ -1,13 +1,14 @@ import { readFile } from "node:fs/promises"; -import path from "node:path"; +import chalk from "chalk"; import { Plugin } from "esbuild"; -import logger from "./logger.js"; +import logger from "../logger.js"; export interface IPluginSettings { target: RegExp; - replacements: string[]; + replacements?: string[]; + deletes?: string[]; name?: string; } @@ -18,7 +19,8 @@ const importPattern = /\/\/#import([\s\S]*?)\n\/\/#endImport/gm; * * openNextPlugin({ * target: /plugins\/default\.js/g, - * replacements: ["./13_4_13.js"], + * replacements: [require.resolve("./plugins/default.js")], + * deletes: ["id1"], * }) * * To inject arbritary code by using (import at top of file): @@ -43,12 +45,14 @@ const importPattern = /\/\/#import([\s\S]*?)\n\/\/#endImport/gm; * * @param opts.target - the target file to replace * @param opts.replacements - list of files used to replace the imports/overrides in the target - * - the path is relative to the target path + * - the path is absolute + * @param opts.deletes - list of ids to delete from the target * @returns */ -export default function openNextPlugin({ +export function openNextReplacementPlugin({ target, replacements, + deletes, name, }: IPluginSettings): Plugin { return { @@ -57,9 +61,19 @@ export default function openNextPlugin({ build.onLoad({ filter: target }, async (args) => { let contents = await readFile(args.path, "utf-8"); - await Promise.all( - replacements.map(async (fp) => { - const p = path.join(args.path, "..", fp); + await Promise.all([ + ...(deletes ?? []).map(async (id) => { + const pattern = new RegExp( + `\/\/#override (${id})\n([\\s\\S]*?)\/\/#endOverride`, + ); + logger.debug( + chalk.blue(`Open-next replacement plugin ${name}`), + ` -- Deleting override for ${id}`, + ); + contents = contents.replace(pattern, ""); + }), + ...(replacements ?? []).map(async (fp) => { + const p = fp; const replacementFile = await readFile(p, "utf-8"); const matches = replacementFile.matchAll(overridePattern); @@ -72,15 +86,17 @@ export default function openNextPlugin({ const replacement = match[2]; const id = match[1]; const pattern = new RegExp( - `\/\/#override (${id})\n([\\s\\S]*?)\n\/\/#endOverride`, + `\/\/#override (${id})\n([\\s\\S]*?)\/\/#endOverride`, + "g", ); logger.debug( - `Open-next plugin ${name} -- Applying override for ${id} from ${fp}`, + chalk.blue(`Open-next replacement plugin ${name}`), + `-- Applying override for ${id} from ${fp}`, ); contents = contents.replace(pattern, replacement); } }), - ); + ]); return { contents, diff --git a/packages/open-next/src/plugins/resolve.ts b/packages/open-next/src/plugins/resolve.ts new file mode 100644 index 00000000..1767d309 --- /dev/null +++ b/packages/open-next/src/plugins/resolve.ts @@ -0,0 +1,64 @@ +import { readFileSync } from "node:fs"; + +import { Plugin } from "esbuild"; +import type { + DefaultOverrideOptions, + IncludedIncrementalCache, + IncludedQueue, + IncludedTagCache, +} from "types/open-next"; + +import logger from "../logger.js"; + +export interface IPluginSettings { + overrides?: { + wrapper?: DefaultOverrideOptions["wrapper"]; + converter?: DefaultOverrideOptions["converter"]; + // Right now theses do nothing since there is only one implementation + tag?: IncludedTagCache; + queue?: IncludedQueue; + incrementalCache?: IncludedIncrementalCache; + }; + fnName?: string; +} + +/** + * @param opts.overrides - The name of the overrides to use + * @returns + */ +export function openNextResolvePlugin({ + overrides, + fnName, +}: IPluginSettings): Plugin { + return { + name: "opennext-resolve", + setup(build) { + logger.debug(`OpenNext Resolve plugin for ${fnName}`); + build.onLoad({ filter: /core\/resolve.js/g }, async (args) => { + let contents = readFileSync(args.path, "utf-8"); + if (overrides?.wrapper && typeof overrides.wrapper === "string") { + contents = contents.replace( + "../wrappers/aws-lambda.js", + `../wrappers/${overrides.wrapper}.js`, + ); + } + if (overrides?.converter) { + if (typeof overrides.converter === "function") { + contents = contents.replace( + "../converters/aws-apigw-v2.js", + `../converters/dummy.js`, + ); + } else { + contents = contents.replace( + "../converters/aws-apigw-v2.js", + `../converters/${overrides.converter}.js`, + ); + } + } + return { + contents, + }; + }); + }, + }; +} diff --git a/packages/open-next/src/queue/sqs.ts b/packages/open-next/src/queue/sqs.ts new file mode 100644 index 00000000..cbc2bdfe --- /dev/null +++ b/packages/open-next/src/queue/sqs.ts @@ -0,0 +1,28 @@ +import { SendMessageCommand, SQSClient } from "@aws-sdk/client-sqs"; + +import { awsLogger } from "../adapters/logger"; +import { Queue } from "./types"; + +// Expected environment variables +const { REVALIDATION_QUEUE_REGION, REVALIDATION_QUEUE_URL } = process.env; + +const sqsClient = new SQSClient({ + region: REVALIDATION_QUEUE_REGION, + logger: awsLogger, +}); + +const queue: Queue = { + send: async ({ MessageBody, MessageDeduplicationId, MessageGroupId }) => { + await sqsClient.send( + new SendMessageCommand({ + QueueUrl: REVALIDATION_QUEUE_URL, + MessageBody: JSON.stringify(MessageBody), + MessageDeduplicationId, + MessageGroupId, + }), + ); + }, + name: "sqs", +}; + +export default queue; diff --git a/packages/open-next/src/queue/types.ts b/packages/open-next/src/queue/types.ts new file mode 100644 index 00000000..eaba1cb6 --- /dev/null +++ b/packages/open-next/src/queue/types.ts @@ -0,0 +1,13 @@ +export interface QueueMessage { + MessageDeduplicationId: string; + MessageBody: { + host: string; + url: string; + }; + MessageGroupId: string; +} + +export interface Queue { + send(message: QueueMessage): Promise; + name: string; +} diff --git a/packages/open-next/src/adapters/types/aws-lambda.ts b/packages/open-next/src/types/aws-lambda.ts similarity index 100% rename from packages/open-next/src/adapters/types/aws-lambda.ts rename to packages/open-next/src/types/aws-lambda.ts diff --git a/packages/open-next/src/adapters/types/next-types.ts b/packages/open-next/src/types/next-types.ts similarity index 82% rename from packages/open-next/src/adapters/types/next-types.ts rename to packages/open-next/src/types/next-types.ts index 99312216..f1f3f807 100644 --- a/packages/open-next/src/adapters/types/next-types.ts +++ b/packages/open-next/src/types/next-types.ts @@ -1,8 +1,8 @@ // NOTE: add more next config typings as they become relevant -import { InternalEvent } from "../event-mapper.js"; -import { IncomingMessage } from "../http/request.js"; -import { ServerlessResponse } from "../http/response.js"; +import type { IncomingMessage, OpenNextNodeResponse } from "http/index.js"; + +import { InternalEvent } from "./open-next"; type RemotePattern = { protocol?: "http" | "https"; @@ -66,6 +66,7 @@ export interface NextConfig { skipTrailingSlashRedirect?: boolean; i18n?: { locales: string[]; + defaultLocale: string; }; experimental: { serverActions?: boolean; @@ -113,23 +114,31 @@ export interface RoutesManifest { headers?: Header[]; } +export interface MiddlewareInfo { + files: string[]; + paths?: string[]; + name: string; + page: string; + matchers: { + regexp: string; + originalSource: string; + }[]; + wasm: { + filePath: string; + name: string; + }[]; + assets: { + filePath: string; + name: string; + }[]; +} + export interface MiddlewareManifest { sortedMiddleware: string[]; middleware: { - [key: string]: { - files: string[]; - paths?: string[]; - name: string; - page: string; - matchers: { - regexp: string; - originalSource: string; - }[]; - wasm: string[]; - assets: string[]; - }; + [key: string]: MiddlewareInfo; }; - functions: { [key: string]: any }; + functions: { [key: string]: MiddlewareInfo }; version: number; } @@ -152,7 +161,7 @@ export type Options = { export interface PluginHandler { ( req: IncomingMessage, - res: ServerlessResponse, + res: OpenNextNodeResponse, options: Options, - ): Promise; + ): Promise; } diff --git a/packages/open-next/src/types/open-next.ts b/packages/open-next/src/types/open-next.ts new file mode 100644 index 00000000..89b5fcd2 --- /dev/null +++ b/packages/open-next/src/types/open-next.ts @@ -0,0 +1,339 @@ +import type { Readable } from "node:stream"; + +import type { StreamCreator } from "http/index.js"; + +import type { WarmerEvent, WarmerResponse } from "../adapters/warmer-function"; +import type { IncrementalCache } from "../cache/incremental/types"; +import type { TagCache } from "../cache/tag/types"; +import type { Queue } from "../queue/types"; + +export type BaseEventOrResult = { + type: T; +}; + +export type InternalEvent = { + readonly method: string; + readonly rawPath: string; + readonly url: string; + readonly body?: Buffer; + readonly headers: Record; + readonly query: Record; + readonly cookies: Record; + readonly remoteAddress: string; +} & BaseEventOrResult<"core">; + +export type InternalResult = { + statusCode: number; + headers: Record; + body: string; + isBase64Encoded: boolean; +} & BaseEventOrResult<"core">; + +export interface DangerousOptions { + /** + * The tag cache is used for revalidateTags and revalidatePath. + * @default false + */ + disableTagCache?: boolean; + /** + * The incremental cache is used for ISR and SSG. + * Disable this only if you use only SSR + * @default false + */ + disableIncrementalCache?: boolean; +} + +export type BaseOverride = { + name: string; +}; +export type LazyLoadedOverride = () => Promise; + +export type OpenNextHandler< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> = (event: E, responseStream?: StreamCreator) => Promise; + +export type Converter< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> = BaseOverride & { + convertFrom: (event: any) => Promise; + convertTo: (result: R, originalRequest?: any) => any; +}; + +export type WrapperHandler< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> = ( + handler: OpenNextHandler, + converter: Converter, +) => Promise<(...args: any[]) => any>; + +export type Wrapper< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> = BaseOverride & { + wrapper: WrapperHandler; + supportStreaming: boolean; +}; + +export type Warmer = BaseOverride & { + invoke: (warmerId: string) => Promise; +}; + +export type ImageLoader = BaseOverride & { + load: (url: string) => Promise<{ + body?: Readable; + contentType?: string; + cacheControl?: string; + }>; +}; + +export interface Origin { + host: string; + protocol: "http" | "https"; + port?: number; + customHeaders?: Record; +} +export type OriginResolver = BaseOverride & { + resolve: (path: string) => Promise; +}; + +export type IncludedWrapper = + | "aws-lambda" + | "aws-lambda-streaming" + | "node" + | "cloudflare"; + +export type IncludedConverter = + | "aws-apigw-v2" + | "aws-apigw-v1" + | "aws-cloudfront" + | "edge" + | "node" + | "sqs-revalidate" + | "dummy"; + +export type IncludedQueue = "sqs"; + +export type IncludedIncrementalCache = "s3"; + +export type IncludedTagCache = "dynamodb"; + +export interface DefaultOverrideOptions< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> { + /** + * This is the main entrypoint of your app. + * @default "aws-lambda" + */ + wrapper?: IncludedWrapper | LazyLoadedOverride>; + + /** + * This code convert the event to InternalEvent and InternalResult to the expected output. + * @default "aws-apigw-v2" + */ + converter?: IncludedConverter | LazyLoadedOverride>; + /** + * Generate a basic dockerfile to deploy the app. + * If a string is provided, it will be used as the base dockerfile. + * @default false + */ + generateDockerfile?: boolean | string; +} + +export interface OverrideOptions extends DefaultOverrideOptions { + /** + * Add possibility to override the default s3 cache. Used for fetch cache and html/rsc/json cache. + * @default "s3" + */ + incrementalCache?: "s3" | LazyLoadedOverride; + + /** + * Add possibility to override the default tag cache. Used for revalidateTags and revalidatePath. + * @default "dynamodb" + */ + tagCache?: "dynamodb" | LazyLoadedOverride; + + /** + * Add possibility to override the default queue. Used for isr. + * @default "sqs" + */ + queue?: "sqs" | LazyLoadedOverride; +} + +export interface DefaultFunctionOptions< + E extends BaseEventOrResult = InternalEvent, + R extends BaseEventOrResult = InternalResult, +> { + /** + * Minify the server bundle. + * @default false + */ + minify?: boolean; + /** + * Print debug information. + * @default false + */ + debug?: boolean; + /** + * Enable overriding the default lambda. + */ + override?: DefaultOverrideOptions; +} + +export interface FunctionOptions extends DefaultFunctionOptions { + /** + * Runtime used + * @default "node" + */ + runtime?: "node" | "edge"; + /** + * @default "regional" + */ + placement?: "regional" | "global"; + /** + * Enable overriding the default lambda. + */ + override?: OverrideOptions; + + /** + * Bundle Next server into a single file. + * This results in a way smaller bundle but it might break for some cases. + * This option will probably break on every new Next.js version. + * @default false + * @deprecated This is not supported in 14.2+ + */ + experimentalBundledNextServer?: boolean; +} + +export type RouteTemplate = + | `app/${string}/route` + | `app/${string}/page` + | `app/page` + | `app/route` + | `pages/${string}`; + +export interface SplittedFunctionOptions extends FunctionOptions { + /** + * Here you should specify all the routes you want to use. + * For app routes, you should use the `app/${name}/route` format or `app/${name}/page` for pages. + * For pages, you should use the `page/${name}` format. + * @example + * ```ts + * routes: ["app/api/test/route", "app/page", "pages/admin"] + * ``` + */ + routes: RouteTemplate[]; + + /** + * Cloudfront compatible patterns. + * i.e. /api/* + * @default [] + */ + patterns: string[]; +} + +export interface OpenNextConfig { + default: FunctionOptions; + functions?: Record; + + /** + * Override the default middleware + * If you set this options, the middleware need to be deployed separately. + * It supports both edge and node runtime. + * @default undefined + */ + middleware?: DefaultFunctionOptions & { + //We force the middleware to be a function + external: true; + + /** + * Origin resolver is used to resolve the origin for internal rewrite. + * By default, it uses the pattern-env origin resolver. + * Pattern env uses pattern set in split function options and an env variable OPEN_NEXT_ORIGIN + * OPEN_NEXT_ORIGIN should be a json stringified object with the key of the splitted function as key and the origin as value + * @default "pattern-env" + */ + originResolver?: "pattern-env" | LazyLoadedOverride; + }; + + /** + * Override the default warmer + * By default, works for lambda only. + * If you override this, you'll need to handle the warmer event in the wrapper + * @default undefined + */ + warmer?: DefaultFunctionOptions & { + invokeFunction: "aws-lambda" | LazyLoadedOverride; + }; + + /** + * Override the default revalidate function + * By default, works for lambda and on SQS event. + * Supports only node runtime + */ + revalidate?: DefaultFunctionOptions< + { host: string; url: string; type: "revalidate" }, + { type: "revalidate" } + >; + + /** + * Override the default revalidate function + * By default, works on lambda and for S3 key. + * Supports only node runtime + */ + imageOptimization?: DefaultFunctionOptions & { + loader?: "s3" | LazyLoadedOverride; + /** + * @default "arm64" + */ + arch?: "x64" | "arm64"; + /** + * @default "18" + */ + + nodeVersion?: "18" | "20"; + }; + + /** + * Override the default initialization function + * By default, works for lambda and on SQS event. + * Supports only node runtime + */ + initializationFunction?: DefaultFunctionOptions & { + tagCache?: "dynamodb" | LazyLoadedOverride; + }; + + /** + * The command to build the Next.js app. + * @default `npm run build`, `yarn build`, or `pnpm build` based on the lock file found in the app's directory or any of its parent directories. + * @example + * ```ts + * build({ + * buildCommand: "pnpm custom:build", + * }); + * ``` + */ + /** + * Dangerous options. This break some functionnality but can be useful in some cases. + */ + dangerous?: DangerousOptions; + buildCommand?: string; + /** + * The path to the target folder of build output from the `buildCommand` option (the path which will contain the `.next` and `.open-next` folders). This path is relative from the current process.cwd(). + * @default "." + */ + buildOutputPath?: string; + /** + * The path to the root of the Next.js app's source code. This path is relative from the current process.cwd(). + * @default "." + */ + appPath?: string; + /** + * The path to the package.json file of the Next.js app. This path is relative from the current process.cwd(). + * @default "." + */ + packageJsonPath?: string; +} diff --git a/packages/open-next/src/wrappers/aws-lambda-streaming.ts b/packages/open-next/src/wrappers/aws-lambda-streaming.ts new file mode 100644 index 00000000..6b0ad02c --- /dev/null +++ b/packages/open-next/src/wrappers/aws-lambda-streaming.ts @@ -0,0 +1,111 @@ +import { Writable } from "node:stream"; +import zlib from "node:zlib"; + +import { APIGatewayProxyEventV2 } from "aws-lambda"; +import { StreamCreator } from "http/index.js"; +import { WrapperHandler } from "types/open-next"; + +import { error } from "../adapters/logger"; +import { WarmerEvent, WarmerResponse } from "../adapters/warmer-function"; + +type AwsLambdaEvent = APIGatewayProxyEventV2 | WarmerEvent; + +type AwsLambdaReturn = void; + +function formatWarmerResponse(event: WarmerEvent) { + const result = new Promise((resolve) => { + setTimeout(() => { + resolve({ serverId, type: "warmer" } satisfies WarmerResponse); + }, event.delay); + }); + return result; +} + +const handler: WrapperHandler = async (handler, converter) => + awslambda.streamifyResponse( + async (event: AwsLambdaEvent, responseStream): Promise => { + if ("type" in event) { + const result = await formatWarmerResponse(event); + responseStream.end(Buffer.from(JSON.stringify(result)), "utf-8"); + return; + } + + const internalEvent = await converter.convertFrom(event); + let _hasWriten = false; + + //Handle compression + const acceptEncoding = + internalEvent.headers["Accept-Encoding"] ?? + internalEvent.headers["accept-encoding"] ?? + ""; + let contentEncoding: string; + let compressedStream: Writable | undefined; + + responseStream.on("error", (err) => { + error(err); + responseStream.end(); + }); + + if (acceptEncoding.includes("br")) { + contentEncoding = "br"; + compressedStream = zlib.createBrotliCompress({ + flush: zlib.constants.BROTLI_OPERATION_FLUSH, + finishFlush: zlib.constants.BROTLI_OPERATION_FINISH, + }); + compressedStream.pipe(responseStream); + } else if (acceptEncoding.includes("gzip")) { + contentEncoding = "gzip"; + compressedStream = zlib.createGzip({ + flush: zlib.constants.Z_SYNC_FLUSH, + }); + compressedStream.pipe(responseStream); + } else if (acceptEncoding.includes("deflate")) { + contentEncoding = "deflate"; + compressedStream = zlib.createDeflate({ + flush: zlib.constants.Z_SYNC_FLUSH, + }); + compressedStream.pipe(responseStream); + } else { + contentEncoding = "identity"; + compressedStream = responseStream; + } + + const streamCreator: StreamCreator = { + writeHeaders: (_prelude) => { + _prelude.headers["content-encoding"] = contentEncoding; + + responseStream.setContentType( + "application/vnd.awslambda.http-integration-response", + ); + _prelude.headers["content-encoding"] = contentEncoding; + // We need to remove the set-cookie header as otherwise it will be set twice, once with the cookies in the prelude, and a second time with the set-cookie headers + delete _prelude.headers["set-cookie"]; + const prelude = JSON.stringify(_prelude); + + responseStream.write(prelude); + + responseStream.write(new Uint8Array(8)); + + return compressedStream ?? responseStream; + }, + onWrite: () => { + _hasWriten = true; + }, + onFinish: () => { + if (!_hasWriten) { + compressedStream?.end(new Uint8Array(8)); + } + }, + }; + + const response = await handler(internalEvent, streamCreator); + + return converter.convertTo(response); + }, + ); + +export default { + wrapper: handler, + name: "aws-lambda-streaming", + supportStreaming: true, +}; diff --git a/packages/open-next/src/wrappers/aws-lambda.ts b/packages/open-next/src/wrappers/aws-lambda.ts new file mode 100644 index 00000000..8e5785d5 --- /dev/null +++ b/packages/open-next/src/wrappers/aws-lambda.ts @@ -0,0 +1,72 @@ +import type { + APIGatewayProxyEvent, + APIGatewayProxyEventV2, + APIGatewayProxyResult, + APIGatewayProxyResultV2, + CloudFrontRequestEvent, + CloudFrontRequestResult, +} from "aws-lambda"; +import { StreamCreator } from "http/openNextResponse"; +import { Writable } from "stream"; +import type { WrapperHandler } from "types/open-next"; + +import { WarmerEvent, WarmerResponse } from "../adapters/warmer-function"; + +type AwsLambdaEvent = + | APIGatewayProxyEventV2 + | CloudFrontRequestEvent + | APIGatewayProxyEvent + | WarmerEvent; + +type AwsLambdaReturn = + | APIGatewayProxyResultV2 + | APIGatewayProxyResult + | CloudFrontRequestResult + | WarmerResponse; + +function formatWarmerResponse(event: WarmerEvent) { + return new Promise((resolve) => { + setTimeout(() => { + resolve({ serverId, type: "warmer" } satisfies WarmerResponse); + }, event.delay); + }); +} + +const handler: WrapperHandler = + async (handler, converter) => + async (event: AwsLambdaEvent): Promise => { + // Handle warmer event + if ("type" in event) { + return formatWarmerResponse(event); + } + + const internalEvent = await converter.convertFrom(event); + + //TODO: create a simple reproduction and open an issue in the node repo + //This is a workaround, there is an issue in node that causes node to crash silently if the OpenNextNodeResponse stream is not consumed + //This does not happen everytime, it's probably caused by suspended component in ssr (either via or loading.tsx) + //Everyone that wish to create their own wrapper without a StreamCreator should implement this workaround + //This is not necessary if the underlying handler does not use OpenNextNodeResponse (At the moment, OpenNextNodeResponse is used by the node runtime servers and the image server) + const fakeStream: StreamCreator = { + writeHeaders: () => { + return new Writable({ + write: (_chunk, _encoding, callback) => { + callback(); + }, + }); + }, + onFinish: () => { + // Do nothing + }, + }; + + const response = await handler(internalEvent, fakeStream); + + return converter.convertTo(response, event); + }; + +export default { + wrapper: handler, + name: "aws-lambda", + supportStreaming: false, +}; diff --git a/packages/open-next/src/wrappers/cloudflare.ts b/packages/open-next/src/wrappers/cloudflare.ts new file mode 100644 index 00000000..96d6d77f --- /dev/null +++ b/packages/open-next/src/wrappers/cloudflare.ts @@ -0,0 +1,30 @@ +import type { + InternalEvent, + InternalResult, + WrapperHandler, +} from "types/open-next"; + +import { MiddlewareOutputEvent } from "../core/routingHandler"; + +const handler: WrapperHandler< + InternalEvent, + InternalResult | ({ type: "middleware" } & MiddlewareOutputEvent) +> = + async (handler, converter) => + async (event: Request, env: Record): Promise => { + //@ts-expect-error - process is not defined in cloudflare workers + globalThis.process = { env }; + const internalEvent = await converter.convertFrom(event); + + const response = await handler(internalEvent); + + const result: Response = converter.convertTo(response); + + return result; + }; + +export default { + wrapper: handler, + name: "cloudflare", + supportStreaming: true, +}; diff --git a/packages/open-next/src/wrappers/node.ts b/packages/open-next/src/wrappers/node.ts new file mode 100644 index 00000000..2aeff358 --- /dev/null +++ b/packages/open-next/src/wrappers/node.ts @@ -0,0 +1,67 @@ +import { createServer } from "node:http"; + +import { StreamCreator } from "http/index.js"; +import type { WrapperHandler } from "types/open-next"; + +import { debug, error } from "../adapters/logger"; + +const wrapper: WrapperHandler = async (handler, converter) => { + const server = createServer(async (req, res) => { + const internalEvent = await converter.convertFrom(req); + const _res: StreamCreator = { + writeHeaders: (prelude) => { + res.writeHead(prelude.statusCode, prelude.headers); + res.uncork(); + return res; + }, + onFinish: () => { + // Is it necessary to do something here? + }, + }; + if (internalEvent.rawPath === "/__health") { + res.writeHead(200, { + "Content-Type": "text/plain", + }); + res.end("OK"); + } else { + await handler(internalEvent, _res); + } + }); + + await new Promise((resolve) => { + server.on("listening", () => { + const cleanup = (code: number) => { + debug(`Closing server`); + server.close(() => { + debug(`Server closed`); + process.exit(code); + }); + }; + console.log(`Listening on port ${process.env.PORT ?? "3000"}`); + debug(`Open Next version: ${process.env.OPEN_NEXT_VERSION}`); + + process.on("exit", (code) => cleanup(code)); + + process.on("SIGINT", () => cleanup(0)); + process.on("SIGTERM", () => cleanup(0)); + + resolve(); + }); + + server.listen(parseInt(process.env.PORT ?? "3000", 10)); + }); + + server.on("error", (err) => { + error(err); + }); + + return () => { + server.close(); + }; +}; + +export default { + wrapper, + name: "node", + supportStreaming: true, +}; diff --git a/packages/open-next/tsconfig.json b/packages/open-next/tsconfig.json index 6d832295..c4c987fd 100644 --- a/packages/open-next/tsconfig.json +++ b/packages/open-next/tsconfig.json @@ -5,6 +5,11 @@ "module": "esnext", "lib": ["DOM", "ESNext"], "outDir": "./dist", - "allowSyntheticDefaultImports": true + "allowSyntheticDefaultImports": true, + "paths": { + "types/*": ["./src/types/*"], + "config/*": ["./src/adapters/config/*"], + "http/*": ["./src/http/*"], + } } } diff --git a/packages/tests-e2e/tests/appRouter/revalidateTag.test.ts b/packages/tests-e2e/tests/appRouter/revalidateTag.test.ts index ae3d1aa4..b0fc3c49 100644 --- a/packages/tests-e2e/tests/appRouter/revalidateTag.test.ts +++ b/packages/tests-e2e/tests/appRouter/revalidateTag.test.ts @@ -2,10 +2,20 @@ import { expect, test } from "@playwright/test"; test("Revalidate tag", async ({ page, request }) => { test.setTimeout(45000); + // We need to hit the page twice to make sure it's properly cached + // Turbo might cache next build result, resulting in the tag being newer than the page + // This can lead to the cache thinking that revalidate tag has been called when it hasn't + // This is because S3 cache files are not uploaded if they have the same BuildId let responsePromise = page.waitForResponse((response) => { return response.status() === 200; }); await page.goto("/revalidate-tag"); + await responsePromise; + + responsePromise = page.waitForResponse((response) => { + return response.status() === 200; + }); + await page.goto("/revalidate-tag"); let elLayout = page.getByText("Fetched time:"); let time = await elLayout.textContent(); let newTime; diff --git a/packages/tests-unit/tests/event-mapper.test.ts b/packages/tests-unit/tests/event-mapper.test.ts index fad2fd7e..c85df409 100644 --- a/packages/tests-unit/tests/event-mapper.test.ts +++ b/packages/tests-unit/tests/event-mapper.test.ts @@ -1,5 +1,6 @@ import { CloudFrontRequestResult } from "aws-lambda"; +//TODO: rewrite this test to use converter instead of event-mapper import { convertTo } from "../../open-next/src/adapters/event-mapper"; describe("convertTo", () => { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 26e54d7a..7d9ba4b3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -146,8 +146,8 @@ importers: specifier: workspace:* version: link:../../packages/utils next: - specifier: ^14.0.3 - version: 14.0.3(react-dom@18.2.0)(react@18.2.0) + specifier: ^14.1.4 + version: 14.1.4(react-dom@18.2.0)(react@18.2.0) open-next: specifier: workspace:* version: link:../../packages/open-next @@ -267,6 +267,9 @@ importers: '@tsconfig/node18': specifier: ^1.0.1 version: 1.0.3 + chalk: + specifier: ^5.3.0 + version: 5.3.0 esbuild: specifier: 0.19.2 version: 0.19.2 @@ -283,6 +286,9 @@ importers: '@types/node': specifier: ^18.16.1 version: 18.17.13 + tsc-alias: + specifier: ^1.8.8 + version: 1.8.8 typescript: specifier: ^4.9.3 version: 4.9.3 @@ -4735,6 +4741,10 @@ packages: resolution: {integrity: sha512-7xRqh9nMvP5xrW4/+L0jgRRX+HoNRGnfJpD+5Wq6/13j3dsdzxO3BCXn7D3hMqsDb+vjZnJq+vI7+EtgrYZTeA==} dev: false + /@next/env@14.1.4: + resolution: {integrity: sha512-e7X7bbn3Z6DWnDi75UWn+REgAbLEqxI8Tq2pkFOFAMpWAWApz/YCUhtWMWn410h8Q2fYiYL7Yg5OlxMOCfFjJQ==} + dev: false + /@next/eslint-plugin-next@13.4.19: resolution: {integrity: sha512-N/O+zGb6wZQdwu6atMZHbR7T9Np5SUFUjZqCbj0sXm+MwQO35M8TazVB4otm87GkXYs2l6OPwARd3/PUWhZBVQ==} dependencies: @@ -4759,6 +4769,15 @@ packages: dev: false optional: true + /@next/swc-darwin-arm64@14.1.4: + resolution: {integrity: sha512-ubmUkbmW65nIAOmoxT1IROZdmmJMmdYvXIe8211send9ZYJu+SqxSnJM4TrPj9wmL6g9Atvj0S/2cFmMSS99jg==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [darwin] + requiresBuild: true + dev: false + optional: true + /@next/swc-darwin-x64@13.4.12: resolution: {integrity: sha512-WRvH7RxgRHlC1yb5oG0ZLx8F7uci9AivM5/HGGv9ZyG2Als8Ij64GC3d+mQ5sJhWjusyU6T6V1WKTUoTmOB0zQ==} engines: {node: '>= 10'} @@ -4777,6 +4796,15 @@ packages: dev: false optional: true + /@next/swc-darwin-x64@14.1.4: + resolution: {integrity: sha512-b0Xo1ELj3u7IkZWAKcJPJEhBop117U78l70nfoQGo4xUSvv0PJSTaV4U9xQBLvZlnjsYkc8RwQN1HoH/oQmLlQ==} + engines: {node: '>= 10'} + cpu: [x64] + os: [darwin] + requiresBuild: true + dev: false + optional: true + /@next/swc-linux-arm64-gnu@13.4.12: resolution: {integrity: sha512-YEKracAWuxp54tKiAvvq73PUs9lok57cc8meYRibTWe/VdPB2vLgkTVWFcw31YDuRXdEhdX0fWS6Q+ESBhnEig==} engines: {node: '>= 10'} @@ -4795,6 +4823,15 @@ packages: dev: false optional: true + /@next/swc-linux-arm64-gnu@14.1.4: + resolution: {integrity: sha512-457G0hcLrdYA/u1O2XkRMsDKId5VKe3uKPvrKVOyuARa6nXrdhJOOYU9hkKKyQTMru1B8qEP78IAhf/1XnVqKA==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + requiresBuild: true + dev: false + optional: true + /@next/swc-linux-arm64-musl@13.4.12: resolution: {integrity: sha512-LhJR7/RAjdHJ2Isl2pgc/JaoxNk0KtBgkVpiDJPVExVWA1c6gzY57+3zWuxuyWzTG+fhLZo2Y80pLXgIJv7g3g==} engines: {node: '>= 10'} @@ -4813,6 +4850,15 @@ packages: dev: false optional: true + /@next/swc-linux-arm64-musl@14.1.4: + resolution: {integrity: sha512-l/kMG+z6MB+fKA9KdtyprkTQ1ihlJcBh66cf0HvqGP+rXBbOXX0dpJatjZbHeunvEHoBBS69GYQG5ry78JMy3g==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + requiresBuild: true + dev: false + optional: true + /@next/swc-linux-x64-gnu@13.4.12: resolution: {integrity: sha512-1DWLL/B9nBNiQRng+1aqs3OaZcxC16Nf+mOnpcrZZSdyKHek3WQh6j/fkbukObgNGwmCoVevLUa/p3UFTTqgqg==} engines: {node: '>= 10'} @@ -4831,6 +4877,15 @@ packages: dev: false optional: true + /@next/swc-linux-x64-gnu@14.1.4: + resolution: {integrity: sha512-BapIFZ3ZRnvQ1uWbmqEGJuPT9cgLwvKtxhK/L2t4QYO7l+/DxXuIGjvp1x8rvfa/x1FFSsipERZK70pewbtJtw==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + requiresBuild: true + dev: false + optional: true + /@next/swc-linux-x64-musl@13.4.12: resolution: {integrity: sha512-kEAJmgYFhp0VL+eRWmUkVxLVunn7oL9Mdue/FS8yzRBVj7Z0AnIrHpTIeIUl1bbdQq1VaoOztnKicAjfkLTRCQ==} engines: {node: '>= 10'} @@ -4849,6 +4904,15 @@ packages: dev: false optional: true + /@next/swc-linux-x64-musl@14.1.4: + resolution: {integrity: sha512-mqVxTwk4XuBl49qn2A5UmzFImoL1iLm0KQQwtdRJRKl21ylQwwGCxJtIYo2rbfkZHoSKlh/YgztY0qH3wG1xIg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + requiresBuild: true + dev: false + optional: true + /@next/swc-win32-arm64-msvc@13.4.12: resolution: {integrity: sha512-GMLuL/loR6yIIRTnPRY6UGbLL9MBdw2anxkOnANxvLvsml4F0HNIgvnU3Ej4BjbqMTNjD4hcPFdlEow4XHPdZA==} engines: {node: '>= 10'} @@ -4867,6 +4931,15 @@ packages: dev: false optional: true + /@next/swc-win32-arm64-msvc@14.1.4: + resolution: {integrity: sha512-xzxF4ErcumXjO2Pvg/wVGrtr9QQJLk3IyQX1ddAC/fi6/5jZCZ9xpuL9Tzc4KPWMFq8GGWFVDMshZOdHGdkvag==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [win32] + requiresBuild: true + dev: false + optional: true + /@next/swc-win32-ia32-msvc@13.4.12: resolution: {integrity: sha512-PhgNqN2Vnkm7XaMdRmmX0ZSwZXQAtamBVSa9A/V1dfKQCV1rjIZeiy/dbBnVYGdj63ANfsOR/30XpxP71W0eww==} engines: {node: '>= 10'} @@ -4885,6 +4958,15 @@ packages: dev: false optional: true + /@next/swc-win32-ia32-msvc@14.1.4: + resolution: {integrity: sha512-WZiz8OdbkpRw6/IU/lredZWKKZopUMhcI2F+XiMAcPja0uZYdMTZQRoQ0WZcvinn9xZAidimE7tN9W5v9Yyfyw==} + engines: {node: '>= 10'} + cpu: [ia32] + os: [win32] + requiresBuild: true + dev: false + optional: true + /@next/swc-win32-x64-msvc@13.4.12: resolution: {integrity: sha512-Z+56e/Ljt0bUs+T+jPjhFyxYBcdY2RIq9ELFU+qAMQMteHo7ymbV7CKmlcX59RI9C4YzN8PgMgLyAoi916b5HA==} engines: {node: '>= 10'} @@ -4903,6 +4985,15 @@ packages: dev: false optional: true + /@next/swc-win32-x64-msvc@14.1.4: + resolution: {integrity: sha512-4Rto21sPfw555sZ/XNLqfxDUNeLhNYGO2dlPqsnuCg8N8a2a9u1ltqBOPQ4vj1Gf7eJC0W2hHG2eYUHuiXgY2w==} + engines: {node: '>= 10'} + cpu: [x64] + os: [win32] + requiresBuild: true + dev: false + optional: true + /@node-minify/core@8.0.6: resolution: {integrity: sha512-/vxN46ieWDLU67CmgbArEvOb41zlYFOkOtr9QW9CnTrBLuTyGgkyNWC2y5+khvRw3Br58p2B5ZVSx/PxCTru6g==} engines: {node: '>=16.0.0'} @@ -7503,6 +7594,10 @@ packages: /caniuse-lite@1.0.30001525: resolution: {integrity: sha512-/3z+wB4icFt3r0USMwxujAqRvaD/B7rvGTsKhbhSQErVrJvkZCLhgNLJxU8MevahQVH6hCU9FsHdNUFbiwmE7Q==} + /caniuse-lite@1.0.30001608: + resolution: {integrity: sha512-cjUJTQkk9fQlJR2s4HMuPMvTiRggl0rAVMtthQuyOlDWuqHXqN8azLq+pi8B2TjwKJ32diHjUqRIKeFX4z1FoA==} + dev: false + /case@1.6.3: resolution: {integrity: sha512-mzDSXIPaFwVDvZAHqZ9VlbyF4yyXRuX6IvB06WvPYkqJVO24kX1PPhv9bfpKNFZyxYFmmgo03HUiD8iklmJYRQ==} engines: {node: '>= 0.8.0'} @@ -7585,7 +7680,6 @@ packages: /chalk@5.3.0: resolution: {integrity: sha512-dLitG79d+GV1Nb/VYcCDFivJeK1hiukt9QjRNVOsUtTy1rR1YJsmpGGTZ3qJos+uw7WmWF4wUwBd9jxjocFC2w==} engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} - dev: true /character-entities-html4@1.1.4: resolution: {integrity: sha512-HRcDxZuZqMx3/a+qrzxdBKBPUpxWEq9xw2OPZ3a/174ihfrQKVsFhqtthBInFy1zZ9GgZyFXOatNujm8M+El3g==} @@ -7790,6 +7884,11 @@ packages: engines: {node: '>= 12'} dev: false + /commander@9.5.0: + resolution: {integrity: sha512-KRs7WVDKg86PWiuAqhDrAQnTXZKraVcCc6vFdL14qrZ/DcWwuRo7VoiYXalXO7S5GKpqYiVEwCbgFDfxNHKJBQ==} + engines: {node: ^12.20.0 || >=14} + dev: true + /commist@1.1.0: resolution: {integrity: sha512-rraC8NXWOEjhADbZe9QBNzLAN5Q3fsTPQtBV+fEVj6xKIgDgNiEVE6ZNfHpZOqfQ21YUzfVNUXLOEZquYvQPPg==} dependencies: @@ -12464,6 +12563,11 @@ packages: resolution: {integrity: sha512-nnbWWOkoWyUsTjKrhgD0dcz22mdkSnpYqbEjIm2nhwhuxlSkpywJmBo8h0ZqJdkp73mb90SssHkN4rsRaBAfAA==} dev: true + /mylas@2.1.13: + resolution: {integrity: sha512-+MrqnJRtxdF+xngFfUUkIMQrUUL0KsxbADUkn23Z/4ibGg192Q+z+CQyiYwvWTsYjJygmMR8+w3ZDa98Zh6ESg==} + engines: {node: '>=12.0.0'} + dev: true + /mz@2.7.0: resolution: {integrity: sha512-z81GNO7nnYMEhrGh9LeymoE4+Yr0Wn5McHIZMK5cfQCl+NDX08sCZgUc9/6MHni9IWuFLm1Z3HTCXu2z9fN62Q==} dependencies: @@ -12633,6 +12737,45 @@ packages: - babel-plugin-macros dev: false + /next@14.1.4(react-dom@18.2.0)(react@18.2.0): + resolution: {integrity: sha512-1WTaXeSrUwlz/XcnhGTY7+8eiaFvdet5z9u3V2jb+Ek1vFo0VhHKSAIJvDWfQpttWjnyw14kBeq28TPq7bTeEQ==} + engines: {node: '>=18.17.0'} + hasBin: true + peerDependencies: + '@opentelemetry/api': ^1.1.0 + react: ^18.2.0 + react-dom: ^18.2.0 + sass: ^1.3.0 + peerDependenciesMeta: + '@opentelemetry/api': + optional: true + sass: + optional: true + dependencies: + '@next/env': 14.1.4 + '@swc/helpers': 0.5.2 + busboy: 1.6.0 + caniuse-lite: 1.0.30001608 + graceful-fs: 4.2.11 + postcss: 8.4.31 + react: 18.2.0 + react-dom: 18.2.0(react@18.2.0) + styled-jsx: 5.1.1(@babel/core@7.22.11)(react@18.2.0) + optionalDependencies: + '@next/swc-darwin-arm64': 14.1.4 + '@next/swc-darwin-x64': 14.1.4 + '@next/swc-linux-arm64-gnu': 14.1.4 + '@next/swc-linux-arm64-musl': 14.1.4 + '@next/swc-linux-x64-gnu': 14.1.4 + '@next/swc-linux-x64-musl': 14.1.4 + '@next/swc-win32-arm64-msvc': 14.1.4 + '@next/swc-win32-ia32-msvc': 14.1.4 + '@next/swc-win32-x64-msvc': 14.1.4 + transitivePeerDependencies: + - '@babel/core' + - babel-plugin-macros + dev: false + /nextra-theme-docs@2.13.1(next@13.4.12)(nextra@2.13.1)(react-dom@18.2.0)(react@18.2.0): resolution: {integrity: sha512-mckNuKa0AmBbRdPCJ/OQ55KZx5MGH8moomMHYB3XVGXQqmXimOq1/2WZQiBdFx9u43KtfEvqZbQE8oGDIrfIsQ==} peerDependencies: @@ -13202,6 +13345,13 @@ packages: hasBin: true dev: true + /plimit-lit@1.6.1: + resolution: {integrity: sha512-B7+VDyb8Tl6oMJT9oSO2CW8XC/T4UcJGrwOVoNGwOQsQYhlpfajmrMj5xeejqaASq3V/EqThyOeATEOMuSEXiA==} + engines: {node: '>=12'} + dependencies: + queue-lit: 1.5.2 + dev: true + /postcss-import@15.1.0(postcss@8.4.27): resolution: {integrity: sha512-hpr+J05B2FVYUAXHeK1YyI267J/dDDhMU6B6civm8hSY1jYJnBXxzKDKDswzJmtLHryrjhnDjqqp/49t8FALew==} engines: {node: '>=14.0.0'} @@ -13457,6 +13607,11 @@ packages: resolution: {integrity: sha512-FIqgj2EUvTa7R50u0rGsyTftzjYmv/a3hO345bZNrqabNqjtgiDMgmo4mkUjd+nzU5oF3dClKqFIPUKybUyqoQ==} dev: true + /queue-lit@1.5.2: + resolution: {integrity: sha512-tLc36IOPeMAubu8BkW8YDBV+WyIgKlYU7zUNs0J5Vk9skSZ4JfGlPOqplP0aHdfv7HL0B2Pg6nwiq60Qc6M2Hw==} + engines: {node: '>=12'} + dev: true + /queue-microtask@1.2.3: resolution: {integrity: sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==} @@ -14919,6 +15074,18 @@ packages: yn: 3.1.1 dev: true + /tsc-alias@1.8.8: + resolution: {integrity: sha512-OYUOd2wl0H858NvABWr/BoSKNERw3N9GTi3rHPK8Iv4O1UyUXIrTTOAZNHsjlVpXFOhpJBVARI1s+rzwLivN3Q==} + hasBin: true + dependencies: + chokidar: 3.5.3 + commander: 9.5.0 + globby: 11.1.0 + mylas: 2.1.13 + normalize-path: 3.0.0 + plimit-lit: 1.6.1 + dev: true + /tsconfig-paths@3.14.2: resolution: {integrity: sha512-o/9iXgCYc5L/JxCHPe3Hvh8Q/2xm5Z+p18PESBU6Ff33695QnCHBEjcytY2q19ua7Mbl/DavtBOLq+oG0RCL+g==} dependencies: