Skip to content

Commit

Permalink
merge main
Browse files Browse the repository at this point in the history
  • Loading branch information
@KristopherPaulsen
KristopherPaulsen committed Sep 9, 2024
2 parents f9be8b6 + fd2525b commit a10bc9f
Show file tree
Hide file tree
Showing 4 changed files with 142 additions and 52 deletions.
17 changes: 14 additions & 3 deletions docs/agent/changelog.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,23 @@ title: Changelog

## v3

### ngrok Agent 3.16.0 - \[2024-09-09\]

- Added `--url` agent CLI flag for creation of HTTP(S), TLS, and TCP protocol endpoints.
- Deprecated `--domain`, `--scheme`, and `--remote-addr` agent CLI flags, which have been replaced by `--url`.
- Added `--metadata` and `--description` agent CLI flags when using `--url`.
- Added `endpoints:` as a new field used for [endpoint creation](/docs/agent/config/v3/#full-example).
- A new `agent:` field has been added to the agent configuration format for V3.
- v3 supports _both_ tunnels and endpoints. Tunnels are now considered deprecated when using config v3.
- Some agent configuration fields have been renamed in v3.
- v2 is still supported, but does not support the new `endpoints:` , or `agent:` fields.
- Added support for [endpoint fields](/docs/api/resources/endpoints/#list-endpoints) in ngrok's `api`.

### ngrok Agent 3.15.1 - \[2024-08-29\]

- Fixed a regression with `ngrok config` commands breaking as a result of `3.15.0` agent format changes.
- Updated [documentation](/docs/agent/config/v3/) to reduce confusion around new agent config format changes.
- Restores the functionality and compatibility for version 3.14.0 due to bugs in the previous release. An updated version with the changes from 3.15.0 will be released early next week.

### ngrok Agent 3.15.0 - \[2024-08-29\]
### \[UNSTABLE\] ngrok Agent 3.15.0 - \[2024-08-29\]

- Added `--url` agent CLI flag for creation of HTTP(S), TLS, and TCP protocol endpoints.
- Deprecated `--domain`, `--scheme`, and `--remote-addr` agent CLI flags, which have been replaced by `--url`.
Expand Down
14 changes: 8 additions & 6 deletions docs/agent/config/v3.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ title: "Version 3"

## Breaking Changes

- [Agent specific fields](/docs/agent/config/v3/#agent-configuration) are no longer top level keys, but instead nested under the **required** `agent:` field.
- [Agent specific fields](/docs/agent/config/v3/#agent-configuration) are no longer top level-keys, but instead nested under the **required** `agent:` field.
- There are now _4_ top-level keys (`version`, `agent`, `endpoints`, and `tunnels`.)
- `tunnels` are deprecated in favor of `endpoints`.

```yaml
version: 3
Expand All @@ -13,6 +15,7 @@ agent:
# add nested "Agent Configuration" fields here
# ...
endpoints: ...
tunnels: # deprecated
```
_For a complete list of `agent:` related fields, checkout [agent configuration](/docs/agent/config/v3/#agent-configuration)._
Expand All @@ -21,7 +24,6 @@ _For a complete list of `agent:` related fields, checkout [agent configuration](

Below are a collection of different agent configurations to serve as examples for your `ngrok.yml` file. _See [agent configuration](/docs/agent/config/v3/#agent-configuration) for more details on starting [individual](#start-the-endpoint-named-httpbin) or [all](/docs/agent/config/v3/#start-all-endpoints-defined-in-the-configuration-file) config defined endpoints from the CLI._


### Basic

```yaml
Expand Down Expand Up @@ -97,9 +99,9 @@ endpoints:

## Endpoint configurations

> **Looking for Tunnel configurations?**
> **Looking for _Tunnel_ configurations?**
>
> Tunnel configurations are still supported in the new `Version 3` configuration file format. However, they're being _deprecated_ in favor of endpoints, which provide a simpler and more powerful way of managing ingress. If you wish to see documentation on how to enable and configure tunnels, refer to the [Version 2 Tunnel Configuration](/docs/agent/config/v2#tunnel-configurations) section.
> Tunnel configurations are still supported in the new `Version 3` configuration file format. However, they're being _deprecated_ in favor of endpoints, which provide a simpler and more powerful way of managing ingress. If you wish to see documentation on how to enable and configure tunnels, refer to the [version 2 tunnel configuration](/docs/agent/config/v2#tunnel-configurations) section.

The most common use of the configuration file is to define endpoint
configurations. Defining endpoint configurations is useful because you may then
Expand Down Expand Up @@ -179,7 +181,7 @@ You can configure a single ngrok agent to start multiple endpoints to different
| [`policy.outbound.actions.type`](/docs/http/traffic-policy/actions/index.mdx) | String | `policy` has been renamed to `traffic_policy` in agent version 3.14.0. |
| [`policy.outbound.actions.config`](/docs/http/traffic-policy/actions/index.mdx) | custom | `policy` has been renamed to `traffic_policy` in agent version 3.14.0. |
| `proto` | `http` | The endpoint protocol name. This defines the type of endpoint you would like to start. |
| `proxy_proto` | String | The version of [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) to use with this endpoint, empty if not using. Example values are 1 or 2. |
| `proxy_protocol` | String | The version of [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) to use with this endpoint, empty if not using. Example values are 1 or 2. |
| `request_header.add` | Array of `key:value` Strings | The headers to add to the request in the key:value format. |
| `request_header.remove` | Array of Strings | The header keys to remove from the request. |
| `response_header.add` | Array of Strings | The headers to add to the response in the key:value format. |
Expand Down Expand Up @@ -279,7 +281,7 @@ The following is a list of fields nested under `agent:` in your configuration fi
| [remote_management](#remote_management) | Set this to `true` to allow the ngrok agent to be remotely managed (stop, restart, update). Defaults to `true`. |
| [root_cas](#root_cas) | The root certificate authorities used to validate the TLS connection to the ngrok server. |
| [connect_url](#connect_url) | This is the URL of the ngrok server to connect to. You should only set this if you are using a custom ingress URL. |
| [endpoints](#endpoint-configurations) | A map of names to tunnel definitions. See [V2 agent config tunnel definitions](/docs/agent/config/v2/#tunnel-configurations) for more details. |
| [endpoints](#endpoint-configurations) | A map of names to endpoint definitions. See [endpoint configurations](/docs/agent/config/v3/#endpoint-configurations) for more details. |
| [update_channel](#update_channel) | The update channel determines the stability of released builds to update to. Use `stable` for all production deployments. |
| [update_check](#update_check) | This tells the ngrok agent if it should check for updates. Defaults to `true`. |
| [web_addr](#web_addr) | Network address to bind on for serving the local web interface and api. |
Expand Down
114 changes: 71 additions & 43 deletions docs/http/traffic-policy/gallery.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ pagination_label: Rule Gallery
import {
AddCompression,
AddRobotsTxt,
AddRobotsTxtSpecific,
BlockCountries,
BlockSpecificBots,
CustomResponse,
Deny,
DeprecateVersion,
Expand All @@ -24,107 +26,133 @@ import {
# Rule Gallery

Explore a curated collection of example configurations spanning from common to
unconventional use-cases for the Traffic Policy module.
unconventional use cases for the Traffic Policy module.

A number of these examples come from a longer article about how ngrok [makes policy management accessible](https://ngrok.com/blog-post/api-gateway-policy-management-examples) to developers, including a simple Go-based application for testing these and other configurations.

## Deny non-GET requests:
See the following categories for specific expressions and actions:

This rule denies all inbound traffic that is not a GET request.
- [Authentication](#authentication)
- [Rate limiting](#rate-limiting)
- [Block unwanted requests](#block-unwanted-requests)
- [Other](#other)

<Deny />
## Authentication

## Custom response for unauthorized requests
### Add JWT authentication and key-based rate limiting

This rule sends a custom response with status code `401` and body `Unauthorized`
for requests without an Authorization header.
Building from our [Auth0 guide](https://ngrok.com/docs/integrations/auth0/jwt-action/), these rules also add rate limiting based on your consumers' JWTs.

<CustomResponse />
<JWTsRateLimiting />

## Rate limiting

## Rate limit for specific endpoint
### Rate limit for specific endpoint

This rule applies rate limiting of `30` requests per second to the endpoint
`/api/videos`.

<RateLimit />

## User agent filtering
### Rate limit API consumers based on authentication status

We deliver tailored content to Microsoft Edge users by examining the
`User-Agent` header for the case-insensitive string `(?i)edg/` succeeded by
digits `\d`. To see how this works in practice, explore the following
[regex101 demonstration](https://regex101.com/r/3NPVub/1).
Create a low rate limit for unauthenticated (likely free) users, while allowing authenticated users a higher level of capacity.

To ensure correct decoding from YAML/JSON, it's necessary to properly escape the
`\d` sequence. In YAML, if your string is not enclosed in quotes, use a single
escape: `\\d`. However, when your string is wrapped in quotes, either in YAML or
JSON, you need to double-escape: `\\\\d` for accurate decoding.
<RateLimitAuthentication />

<UserAgentFilter />
### Rate limit API consumers based on pricing tiers

Using a naming scheme in your upstream servers, and API calls using a `tier` header, you can quickly customize access to your API based on any number of pricing tiers.

<RateLimitPricing />

## Block unwanted requests

## Add custom response for robots.txt
### Disallow bots and crawlers with a `robots.txt`

This rule returns a custom response for robots.txt, denying search engine crawlers on all paths.
[What is robots.txt?](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
This rule returns a custom response with a [`robots.txt` file](https://developers.google.com/search/docs/crawling-indexing/robots/intro) to deny search engine or AI crawlers on all paths.

<AddRobotsTxt />

## Add JWT authentication and key-based rate limiting
You can also extend the expression above to create specific rules for crawlers based on their user agent strings, like `ChatGPT-User` and `GPTBot`.

Building from our [Auth0 guide](https://ngrok.com/docs/integrations/auth0/jwt-action/), these rules also add rate limiting based on your consumers' JWTs.
<AddRobotsTxtSpecific />

<JWTsRateLimiting />
### Block bots and crawlers by user agent

## Rate limit API consumers based on authentication status
In addition to, or instead of, denying bots and crawlers with a `robots.txt` file, you can also take action on only incoming requests that contain specific strings in the [`req.user_agent` request variable](/docs/http/traffic-policy/expressions/variables.mdx#requser_agent).

Create a low rate limit for unauthenticated (likely free) users, while allowing authenticated users a higher level of capacity.
You can extend the expression to include additional user agents by extending `(chatgpt-user|gptbot)` like so: `(chatgpt-user|gptbot|anthropic|claude|any|other|user-agent|goes|here)`.

<RateLimitAuthentication />
<BlockSpecificBots />

## Rate limit API consumers based on pricing tiers
### Deny non-GET requests

Using a naming scheme in your upstream servers, and API calls using a `tier` header, you can quickly customize access to your API based on any number of pricing tiers.
This rule denies all inbound traffic that is not a GET request.

<RateLimitPricing />
<Deny />

### Custom response for unauthorized requests

This rule sends a custom response with status code `401` and body `Unauthorized`
for requests without an Authorization header.

<CustomResponse />

## Block traffic from specific countries
### Block traffic from specific countries

Remain compliant with data regulations or sanctions by blocking requests originating from one or more countries using their respective [ISO country codes](https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes).

<BlockCountries />

## Deprecate an API version
### Limit request sizes

Prevent excessively large user uploads, like text or images, that might cause performance or availability issues for your upstream service.

<LimitSize />

## Other

### User agent filtering

We deliver tailored content to Microsoft Edge users by examining the
`User-Agent` header for the case-insensitive string `(?i)edg/` succeeded by
digits `\d`. To see how this works in practice, explore the following
[regex101 demonstration](https://regex101.com/r/3NPVub/1).

To ensure correct decoding from YAML/JSON, it's necessary to properly escape the
`\d` sequence. In YAML, if your string is not enclosed in quotes, use a single
escape: `\\d`. However, when your string is wrapped in quotes, either in YAML or
JSON, you need to double-escape: `\\\\d` for accurate decoding.

<UserAgentFilter />

### Deprecate an API version

By include a `X-Api-Version` header in your API reference or developer documentation, you can quickly return a helpful error message, which encourages them to explore usage of the new version.

<DeprecateVersion />

## Manipulate request headers
### Manipulate request headers

Add new headers to requests to give your upstream service more context about the consumer, which in turn allows for richer functionality, such as localized languages and pricing.

<ManipulateHeaders />

## Add compression
### Add compression

Quickly ensure all JSON responses are [compressed](/docs/http/traffic-policy/actions/compress-response.mdx) en route to your API consumer. If your upstream service already handles compression, ngrok skips this step.

<AddCompression />

## Enforce TLS version
### Enforce TLS version

Prevent obsolete and potentially vulnerable browsers, SDKs, or CLI tools like `curl` from attempting to access your API.

<EnforceTLS />

## Log unsuccessful events
### Log unsuccessful events

Connect your API to ngrok's [event logging system](/docs/obs/index.mdx) for smarter troubleshooting of your API gateway and upstream services.

<LogUnsuccessful />

## Limit request sizes

Prevent excessively large user uploads, like text or images, that might cause performance or availability issues for your upstream service.

<LimitSize />
49 changes: 49 additions & 0 deletions traffic-policy/gallery.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ export const AddRobotsTxt = () => (
config={{
inbound: [
{
name: "Add `robots.txt` to deny all bots and crawlers",
expressions: ["req.url.contains('/robots.txt')"],
actions: [
{
Expand All @@ -106,6 +107,54 @@ export const AddRobotsTxt = () => (
/>
);

export const AddRobotsTxtSpecific = () => (
<ConfigExample
config={{
inbound: [
{
name: "Add `robots.txt` to deny specific bots and crawlers",
expressions: ["req.url.contains('/robots.txt')"],
actions: [
{
type: "custom-response",
config: {
status_code: 200,
content: "User-agent: ChatGPT-User\\r\\nDisallow: /",
headers: {
"content-type": "text/plain",
},
},
},
],
},
],
}}
/>
);

export const BlockSpecificBots = () => (
<ConfigExample
config={{
inbound: [
{
name: "Block specific bots by user agent",
expressions: [
"req.user_agent.matches('(?i).*(chatgpt-user|gptbot)/\\\\d+.*')",
],
actions: [
{
type: "deny",
config: {
status_code: 404,
},
},
],
},
],
}}
/>
);

export const JWTsRateLimiting = () => (
<ConfigExample
config={{
Expand Down

0 comments on commit a10bc9f

Please sign in to comment.