OpenNext V3 (#402)

* created basic config file * basic wrapper and converter implementation * Minimal response writable * build config * change response to transform to allow to use pipeline * fix streaming for v3 * compression support * better docker handler * add converter for apigw-v1 & cloudfront * overridable queue * overridable s3 cache * overridable tag cache * prebuild middleware * refactor routing and middleware * big refactoring moved files around so that it makes more sense deleted a bunch of useless files added todo to remind myself of what i still need to do * refactor: cleanup plugins added a deletes options in open-next plugin * make other lambdas overridable as well * externalMiddleware * improve plugins * fix proxy request and make it work with streaming * bugfix * fix host * refactor wrapper * generate basic dockerfile * Only build open-next config once * generate basic output file for IAC to use * basic splitting * bundled next server * fix external middleware cloudfront * fix image adapter rebase * couple of fix for node * package version * support for warmer with splitted fn * basic support for edge runtime There is some restriction: Only 1 route per function Support only app route and page No streaming * external middleware support rewrite between splitted servers * fix alias * update package.json * use AsyncLocalStorage to scope lastModified to a single request * merge upstream/main * Add basic validation * fix EISDIR issue with copying traced symlink * added override name to the output for better IAC support * rename BuildOptions remove some unused options properly handle minify * normalize locale path before passing to middleware * Copy necessary static files * fix issues with fallback and i18n in page router * Add a big warning for build on windows * fix for cloudflare workers * add wasm fils and assets * fix 14.1 cache * fix wasm import node * update version * merge upstream * make open-next.config.ts optional * Fix cannot write default config file b/c folder not created (#364) * Fix cannot write default config file b/c folder not created * Removed copyTracedFiles debug log * fix for monorepo * fix for output for dynamodb provider * fix dynamoProvider, skipTrailingSlash, weird ISR deduplication issue * little improvement to streaming in lambda * fix another monorepo error * e2e fixes for v3 rc * update version * Not use custom-resource converter for dynamodb seeding adapter (#365) * Not use custom-resource converter for dynamodb seeding adapter * fix e2e --------- Co-authored-by: Dorseuil Nicolas <[email protected]> * fix fallback false for route without i18n * version package update * Squashed commit of the following: commit ff37de2 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed Mar 6 15:37:07 2024 +0100 Version Packages (#378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 3235392 Author: Iakhub Seitasanov <[email protected]> Date: Wed Mar 6 17:29:46 2024 +0300 fix: prevent duplication of location header (#369) * fix: prevent duplication of location header * changeset * fix linting --------- Co-authored-by: conico974 <[email protected]> commit af2d3ce Author: Chung Wei Leong <[email protected]> Date: Wed Mar 6 22:06:33 2024 +0800 Fix image optimization support for Next 14.1.1 (#377) * Move image optimization to plugin * Refactor image optimization code * Added image optimization plugin for 14.1.1 * Fix image optimization plugin * Add changeset * Revert default sharp version to 0.32.6 * e2e test for image optimization * change one of the test to use an external image --------- Co-authored-by: Dorseuil Nicolas <[email protected]> commit 3deb202 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Feb 13 08:39:35 2024 -0800 Version Packages (#363) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit f9b90b6 Author: khuezy <[email protected]> Date: Tue Feb 13 08:35:10 2024 -0800 changeset/2.3.6 (#362) commit 40c2b36 Author: Patrick Ufer <[email protected]> Date: Tue Feb 13 09:23:40 2024 -0700 security fix: upgrade sharp version to 0.32.6 (#361) * upgrade sharp version commit 63fab05 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Feb 2 00:14:11 2024 +0100 Version Packages (#359) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit c80f1be Author: conico974 <[email protected]> Date: Fri Feb 2 00:00:56 2024 +0100 Fix trailing slash redirect to external domain (#358) * fix trailing slash redirect to external domain * changeset commit 186e28f Author: Jaden VanEckhout <[email protected]> Date: Thu Feb 1 16:49:14 2024 -0600 fix(open-next): correctly set cache control for html pages (#353) * fix(open-next): correctly set cache control for html pages * changeset --------- Co-authored-by: conico974 <[email protected]> commit b9eefca Author: Manuel Antunes <[email protected]> Date: Thu Feb 1 19:41:47 2024 -0300 Fix Cache Support for [email protected] (#356) * feat: add cache support for [email protected] * fix: lint files * chore: apply the proposed changes * Fix typo * changeset --------- Co-authored-by: conico974 <[email protected]> commit afd9605 Author: conico974 <[email protected]> Date: Sat Jan 27 15:19:11 2024 +0100 update docs for V3 (#351) commit 46241fe Author: Abhishek Malik <[email protected]> Date: Sat Jan 27 19:45:18 2024 +0530 Update bundle_size.mdx for excluding pdfjs-dist optional dependency docs (#346) * Update bundle_size.mdx for excluding pdfjs-dist optional dependency docs The current fix didn't work, but this updated fix did work for me. Hence proposing this as another solution. * Update docs/pages/common_issues/bundle_size.mdx Co-authored-by: khuezy <[email protected]> --------- Co-authored-by: khuezy <[email protected]> commit 9a6473a Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Jan 5 16:56:42 2024 +0100 Version Packages (#345) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit bbf9b30 Author: Lucas Vieira <[email protected]> Date: Fri Jan 5 12:45:13 2024 -0300 fix(open-next): use dynamic import handler for monorepo entrypoint (#341) * fix(open-next): use dynamic import handler for monorepo entrypoint * changeset --------- Co-authored-by: Dorseuil Nicolas <[email protected]> commit 83b0838 Author: santiperone <[email protected]> Date: Fri Jan 5 12:38:12 2024 -0300 add suport for bun lockfile in monorepo (#337) * add suport for bun lockfile in monorepo * changeset --------- Co-authored-by: Dorseuil Nicolas <[email protected]> commit e773e67 Author: Jan Stevens <[email protected]> Date: Fri Jan 5 16:31:27 2024 +0100 fix: try to match errors, fall back to just adding raw key / value pare (#336) * fix: try to match errors, fall back to just adding raw key / value pair instead * changeset * fix lint --------- Co-authored-by: Dorseuil Nicolas <[email protected]> commit fd90b26 Author: Dylan Irion <[email protected]> Date: Fri Jan 5 17:22:28 2024 +0200 Changes encoding on cache.body from utf8 to base64 (#329) * changes encoding on cache.body from utf8 to base64 * retain utf8 for json content-type * opting for less greedy base64 * use isBinaryContentType * changeset --------- Co-authored-by: Dorseuil Nicolas <[email protected]> commit eb08980 Author: sommeeeR <[email protected]> Date: Fri Jan 5 16:02:47 2024 +0100 fix: make invalidateCFPaths function async in docs (#344) commit 83207d8 Author: conico974 <[email protected]> Date: Thu Dec 14 16:59:15 2023 +0100 updated docs for v3 (#334) commit 0e827ce Author: conico974 <[email protected]> Date: Fri Dec 8 17:57:51 2023 +0100 ci: update node e2e commit 36da819 Author: conico974 <[email protected]> Date: Thu Dec 7 17:44:06 2023 +0100 Initial docs for V3 (#330) * docs for V3 * fix link * clearer routes in config * fix for next 12 * add support for basePath * allow customization of sharp runtime * updated edge converter to match behaviour of lambda * update version * fix monorepo * improved streaming aws/aws-lambda-nodejs-runtime-interface-client#94 (comment) * update version * fix open-next config build that depends on node * fix crypto middleware node 20 * Sync * fix resolve in image optimization also fix image opt not using streaming * add better error when edge runtime is used inside node * update version * fix null error on lambda hopefully * update version * fix 500 on aws-lambda wrapper * update version * fix duplex for request in node * fix & refactor middleware response headers * update version * Sync * update version * removed specific lamda streaming hack It's been fixed upstream * add geo in middleware * added helpers function for config file Better typing as well * fix for 14.2 * update version * fix redirect lambda streaming * fix e2e tests * test: improve reliability of test for revalidateTag * update version * review fix * fix cookies in streaming also fix an issue when both middleware and page try to set cookies OpenNextNodeResponse also implements ServerResponse * make all write to ddb chunked * changeset * fix e2e --------- Co-authored-by: Frank <[email protected]>
opennextjs · May 3, 2024 · b191ba3 · b191ba3
1 parent 826a37e
commit b191ba3
Show file tree

Hide file tree

Showing 107 changed files with 7,587 additions and 3,227 deletions.
diff --git a/.changeset/rotten-trees-kiss.md b/.changeset/rotten-trees-kiss.md
@@ -0,0 +1,38 @@
+---
+"open-next": major
+---
+
+OpenNext V3
+
+This is the V3 of OpenNext. It includes some breaking changes and cannot be used as a drop-in replacement for V2. If your IAC is using OpenNext V2, you will need to update it to use V3.
+
+If you are using OpenNext V2, please refer to the [migration guide](https://open-next.js.org/migration#from-opennext-v2) to upgrade to V3. 
+
+### New Features
+
+- Add support for function splitting
+- Add support for external middleware
+- Custom config file support : `open-next.config.ts`
+- Support for other deployment targets than lambda (Node.js, Docker and partial support for Cloudflare Workers)
+- Allow for customizing the outputs bundle :
+  - Wrapper
+  - Converter
+  - Incremental Cache (Fetch cache and HTML/JSON/RSC cache)
+  - Tag Cache
+  - Queue (Used to trigger ISR revalidation)
+  - Origin Resolver (Only for external middleware)
+  - Image Loader (Only for image optimization)
+  - Invoke function (For the warmer function)
+- Create an `open-next.output.json` file for easier integration with IAC tools
+
+### Breaking Changes
+
+- Edge runtime don't work out of the box anymore. You need to deploy them on a separate function see [the config for more info](https://open-next.js.org/config)
+- Output directory structure has changed to support function splitting
+- Removed build arguments in favor of `open-next.config.ts`
+
+### Internal Changes
+
+- Use OpenNextNodeResponse instead of ServerResponse (It uses transform stream to properly handle the stream)
+- Big refactor of the codebase to support function splitting
+- Added new plugins to support the new features and make the codebase more modular
diff --git a/docs/pages/inner_workings/_meta.json b/docs/pages/inner_workings/_meta.json
@@ -2,4 +2,4 @@
   "caching": "Caching (ISR/SSG)",
   "components": "Main Components",
   "architecture": "Default Architecture"
-}
+}
diff --git a/docs/pages/v3/_meta.json b/docs/pages/v3/_meta.json
@@ -0,0 +1,7 @@
+{
+  "index": "What's new",
+  "config": "Configuration file",
+  "reference-implementation": "Reference Construct",
+  "requirements": "Requirements",
+  "override": "Advanced - Create your own override"
+}
diff --git a/docs/pages/v3/config.mdx b/docs/pages/v3/config.mdx
@@ -0,0 +1,69 @@
+Here is a simple example of an `open-next.config.ts` file: 
+This file need to be at the same place as your `next.config.js` file
+
+`server` in here could refer to a lambda function, a docker container, a node server or whatever that can support running nodejs code. (Even cloudflare workers in the future)
+
+For more information about the options here, just look at the source file
+
+```ts
+import type { OpenNextConfig } from 'open-next/types/open-next'
+const config = {
+  default: { // This is the default server, similar to the server-function in open-next v2
+    // You don't have to provide the below, by default it will generate an output 
+    // for normal lambda as in open-next v2
+    override: { 
+      wrapper: "aws-lambda-streaming", // This is necessary to enable lambda streaming
+      // You can override any part that is a `LazyLoadedOverride` this way
+      queue: () => Promise.resolve({
+        send: async (message) => {
+          //Your custom code here
+        }
+      })
+    },
+  },
+  // Below we define the functions that we want to deploy in a different server
+  functions: { 
+    ssr: {
+      routes: [
+        "app/api/isr/route", "app/api/sse/route", "app/api/revalidateTag/route", // app dir Api routes
+        "app/route1/page", "app/route2/page", // app dir pages
+        "pages/route3" // page dir pages
+        ], // For app dir, you need to include route|page, no need to include layout or loading
+      patterns: ['api/*', 'route1', 'route2', 'route3'], // patterns needs to be in a cloudfront compatible format, this will be used to generate the output
+      override: {
+        wrapper: "aws-lambda-streaming",
+      },
+      experimentalBundledNextServer: true // This enables the bundled next server which is faster and reduce the size of the server
+    },
+    pageSsr: {
+      routes: ["pages/pageSsr"], // For page dir routes should be in the form `pages/${route}` without the extension, it should match the filesystem
+      patterns: [ 'pageSsr', "_next/data/BUILD_ID/pageSsr.json"],
+      override: {
+        wrapper: "node",
+        converter: "node",
+        // This is necessary to generate the dockerfile and for the implementation to know that it needs to deploy on docker
+        generateDockerfile: true,
+      },
+    },
+    edge: {
+      runtime: "edge",
+      routes: ["app/ssr/page"],
+      patterns: ["ssr"],
+      override: {}
+    }    
+  },
+  // By setting this, it will create another bundle for the middleware, 
+  // and the middleware will be deployed in a separate server. 
+  // If not set middleware will be bundled inside the servers
+  // It could be in lambda@edge, cloudflare workers, or anywhere else
+  // By default it uses lambda@edge
+  // This is not implemented in the reference construct implementation.
+  middleware: {
+    external: true 
+  }
+  buildCommand: "echo 'hello world'",
+} satisfies OpenNextConfig
+
+export default config;
+export type Config = typeof config
+```
diff --git a/docs/pages/v3/index.mdx b/docs/pages/v3/index.mdx
@@ -0,0 +1,67 @@
+import { Callout } from 'nextra/components'
+
+<Callout type="warning" emoji="⚠️">
+
+`[email protected]` is here!!! Please report any issues you find on [discord](https://discord.com/channels/983865673656705025/1164872233223729152) or on the github [PR](https://github.com/sst/open-next/pull/327)
+
+  This is a release candidate, it is mostly ready for production (You might still experience some quirks). We are looking for feedback on this release, so please try it out and let us know what you think. See [getting started](#get-started) to quickly test it.
+
+  It also requires an updated version of the IAC tools that you use, see the sst PR [here](https://github.com/sst/sst/pull/3567) for more information.
+
+  You could also use SST Ion which should support it out of the box pretty soon. See [here for more info](https://github.com/sst/ion) or in the [ion discord](https://discord.com/channels/983865673656705025/1177071497974648952).
+</Callout>
+
+## What's new in V3?
+
+- Rewritten server (We moved from the serverless-http `ServerResponse` into our own version which respect nodejs stream)
+- A new `open-next.config.ts` file to configure your project
+- Native support for aws lambda, aws lambda streaming, lambda@edge, node and docker
+- In this `open-next.config.ts` you can now override a lot of things which allow for more customization:
+  - Wrapper component
+  - Converter component
+  - Incremental Cache
+  - Tag Cache
+  - Queue
+  - Custom warmer function
+  - Custom revalidation function
+  - Custom loader for image optimization
+  - Custom initialization function
+
+- Allow for splitting, you can now split your next app into multiple servers, which could each have their own configuration
+- Allow to move the middleware/routing part in a separate lambda or cloudflare workers in front of your server functions
+- An experimental bundled `NextServer` could be used which can reduce the size of your lambda by up to 24 MB
+- Experimental support for the `edge` runtime of next with some limitations:
+  - Only app router for now
+  - Only 1 route per function 
+  - Works fine in node, only for api route in cloudflare workers
+  - No support for `revalidateTag` or `revalidatePath` for now
+
+## Get started
+
+The easiest way to get started is to use the [reference implementation construct](/v3/reference-implementation). Copy this reference implementation into your project and then use it like that in your sst or cdk project:
+
+```ts
+import { OpenNextCdkReferenceImplementation } from "path/to/reference-implementation"
+
+const site = new OpenNextCdkReferenceImplementation(stack, "site", {
+  openNextPath: ".open-next",
+})
+```
+
+You also need to create an `open-next.config.ts` file in your project root, you can find more info [here](/v3/config).
+
+A very simple example of this file could be:
+
+```ts
+import type { OpenNextConfig } from 'open-next/types/open-next'
+const config = {
+  default: {
+
+  }
+}
+export default config;
+```
+
+Then you need to run `npx [email protected] build` to build your project before running the `sst deploy` or `cdk deploy` command to deploy your project.
+
+In V3 `open-next build` don't accept any arguments, all the args are passed in the `open-next.config.ts` file.
diff --git a/docs/pages/v3/override.mdx b/docs/pages/v3/override.mdx
@@ -0,0 +1,74 @@
+In this version of open-next, you could override a lot of the default behaviour.
+
+For some real example of how to override each behaviour: 
+- [Wrapper](https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/wrappers/aws-lambda.ts)
+- [Converter](https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/converters/aws-apigw-v2.ts)
+- [IncrementalCache](https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/cache/incremental/s3.ts)
+- [TagCache](
+  https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/cache/tag/dynamoDb.ts
+)
+- [Queue](
+  https://github.com/conico974/open-next/blob/feat/splitting/packages/open-next/src/queue/sqs.ts
+)
+
+This means it could allow people to write their own custom open-next.
+For example you could create a custom `withGcp` plugin to allow to deploy open-next on GCP functions
+
+A boilerplate for such a plugin could look like this (This is not real code):
+
+```ts
+
+import { OpenNextConfig } from "open-next/types/open-next";
+
+function withGcp(config: TrimmedDownConfig): OpenNextConfig {
+  return {
+    default: {
+        override: {
+          wrapper: async () => (await import("./gcp-wrapper")).default,
+          converter: async () => (await import("./gcp-converter")).default,
+          incrementalCache: async () => (await import("./gcp-incremental-cache")).default,
+          tagCache: async () => (await import("./gcp-tag-cache")).default,
+          queue: async () => (await import("./gcp-queue")).default,
+        },
+        ...config.default,
+    },
+    functions: {
+      // Same as default but for each splitted function
+      //...
+    }
+    warmer: {
+      override: {
+        wrapper: async () => (await import("./gcp-wrapper")).default,
+        converter: async () => (await import("./gcp-converter")).default,
+      },
+      invokeFunction: async () => (await import("./gcp-invoke-function")).default,
+    },
+    revalidate: {
+      override: {
+        wrapper: async () => (await import("./gcp-wrapper")).default,
+        converter: async () => (await import("./gcp-queue-converter")).default,
+      },
+    },
+    imageOptimization: {
+      override: {
+        wrapper: async () => (await import("./gcp-wrapper")).default,
+        converter: async () => (await import("./gcp-converter")).default,
+      },
+      loader: async () => (await import("./gcp-object-loader")).default,
+    },
+  }
+}
+```
+
+Using this plugin would look like this inside `open-next.config.ts`:
+
+```ts
+import { withGcp } from "./with-gcp";
+const config = withGcp({
+  default: {
+    // ...
+  },
+});
+
+export default config;
+```