Merge pull request #6481 from TheThingsNetwork/merge/ttsos-v3.27

Merge TTSOS v3.27 into TTSOS v3.28

CHANGELOG.md

@@ -7,14 +7,11 @@ with the exception that this project **does not** follow Semantic Versioning.
 
 For details about compatibility between different releases, see the **Commitments and Releases** section of our README.
 
-## [Unreleased]
+## [3.28.0] - unreleased
 
 ### Added
 
-- Add support for `administrative_contact` and `technical_contact` in the Console.
-- Reimplement move away prompt in payload formatter views in the Console.
-- Add telemetry collection for the CLI. A background process was added to the CLI in order to send the following information: Operating System, Architecture, Binary version and Golang version. The message is sent every 24 hours and it contains an unique random number as an identifier. It is enabled by default and in order to disable it, set `telemetry.enable` to false in the CLI configuration. For more information, consult the documentation [here](https://www.thethingsindustries.com/docs/reference/telemetry/cli).
-- Add telemetry collection for the IdentityServer. A background task was added in the Identity Server which is responsible for collecting information regarding the amount of each entity in the database, this has the purpose of allowing us to better understand how users are interacting with the system, an example being if tenants are using Organizations or just Users. All information is sent every 24 hours and it contains an identifier composed of the URLs present in the following configuration fields `console.ui.[is|gs|ns|as|js].base-url`. It is enabled by default and in order to disable it, set `telemetry.enable` to false in the Stack configuration. For more information, consult the documentation [here](https://www.thethingsindustries.com/docs/reference/telemetry/identity_server).
+- The Network Server ID (NSID, EUI-64) used in LoRaWAN Backend Interfaces is now included in the application uplink message network metadata as well as in the Backend Interfaces `HomeNSAns` message that Identity Server returns to clients. The NSID is configurable via `is.network.ns-id`.
 
 ### Changed
 
@@ -24,28 +21,23 @@ For details about compatibility between different releases, see the **Commitment
 
 ### Fixed
 
-- OAuth clients created by an admin no longer trigger an email requesting approval from one of the tenant's admins.
-- Broken network routing policy links in the Packet Broker panel of the admin panel in the Console.
-- Application Server downlink related events now contain the complete set of end device identifiers, and the received at timestamp is now provided at all times.
-- Wrong order of breadcrumbs in the device views of the Console.
-
 ### Security
 
-## [3.28.0] - unreleased
+## [3.27.1] - unreleased
 
 ### Added
 
-- The Network Server ID (NSID, EUI-64) used in LoRaWAN Backend Interfaces is now included in the application uplink message network metadata as well as in the Backend Interfaces `HomeNSAns` message that Identity Server returns to clients. The NSID is configurable via `is.network.ns-id`.
-
-### Changed
-
-### Deprecated
-
-### Removed
+- Add support for `administrative_contact` and `technical_contact` in the Console.
+- Reimplement move away prompt in payload formatter views in the Console.
+- Add telemetry collection for the CLI. A background process was added to the CLI in order to send the following information: Operating System, Architecture, Binary version and Golang version. The message is sent every 24 hours and it contains an unique random number as an identifier. It is enabled by default and in order to disable it, set `telemetry.enable` to false in the CLI configuration. For more information, consult the documentation [here](https://www.thethingsindustries.com/docs/reference/telemetry/cli).
+- Add telemetry collection for the IdentityServer. A background task was added in the Identity Server which is responsible for collecting information regarding the amount of each entity in the database, this has the purpose of allowing us to better understand how users are interacting with the system, an example being if tenants are using Organizations or just Users. All information is sent every 24 hours and it contains an identifier composed of the URLs present in the following configuration fields `console.ui.[is|gs|ns|as|js].base-url`. It is enabled by default and in order to disable it, set `telemetry.enable` to false in the Stack configuration. For more information, consult the documentation [here](https://www.thethingsindustries.com/docs/reference/telemetry/identity_server).
 
 ### Fixed
 
-### Security
+- OAuth clients created by an admin no longer trigger an email requesting approval from one of the tenant's admins.
+- Broken network routing policy links in the Packet Broker panel of the admin panel in the Console.
+- Application Server downlink related events now contain the complete set of end device identifiers, and the received at timestamp is now provided at all times.
+- Wrong order of breadcrumbs in the device views of the Console.
 
 ## [3.27.0] - 2023-07-31
 
@@ -2673,7 +2665,8 @@ For details about compatibility between different releases, see the **Commitment
 NOTE: These links should respect backports. See https://github.com/TheThingsNetwork/lorawan-stack/pull/1444/files#r333379706.
 -->
 
-[unreleased]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.0...v3.27
+[unreleased]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.1...v3.27
+[3.27.1]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.0...v3.27.1
 [3.27.0]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.26.2...v3.27.0
 [3.26.2]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.26.1...v3.26.2
 [3.26.1]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.26.0...v3.26.1

go.mod

@@ -46,7 +46,7 @@ require (
 	github.com/howeyc/gopass v0.0.0-20210920133722-c8aef6fb66ef
 	github.com/iancoleman/strcase v0.3.0
 	github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa
-	github.com/jackc/pgx/v5 v5.4.2
+	github.com/jackc/pgx/v5 v5.4.3
 	github.com/jacobsa/crypto v0.0.0-20190317225127-9f44e2d11115
 	github.com/jarcoal/httpmock v1.3.0
 	github.com/jaytaylor/html2text v0.0.0-20211105163654-bc68cce691ba
@@ -99,7 +99,7 @@ require (
 	gocloud.dev v0.33.0
 	gocloud.dev/pubsub/natspubsub v0.33.0
 	golang.org/x/crypto v0.12.0
-	golang.org/x/exp v0.0.0-20230321023759-10a507213a29
+	golang.org/x/exp v0.0.0-20230817173708-d852ddb80c63
 	golang.org/x/net v0.14.0
 	golang.org/x/oauth2 v0.10.0
 	golang.org/x/sync v0.3.0

go.sum

@@ -473,8 +473,8 @@ github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsI
 github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
 github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk=
 github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
-github.com/jackc/pgx/v5 v5.4.2 h1:u1gmGDwbdRUZiwisBm/Ky2M14uQyUP65bG8+20nnyrg=
-github.com/jackc/pgx/v5 v5.4.2/go.mod h1:q6iHT8uDNXWiFNOlRqJzBTaSH3+2xCXkokxHZC5qWFY=
+github.com/jackc/pgx/v5 v5.4.3 h1:cxFyXhxlvAifxnkKKdlxv8XqUf59tDlYjnV5YYfsJJY=
+github.com/jackc/pgx/v5 v5.4.3/go.mod h1:Ig06C2Vu0t5qXC60W8sqIthScaEnFvojjj9dSljmHRA=
 github.com/jacobsa/crypto v0.0.0-20190317225127-9f44e2d11115 h1:YuDUUFNM21CAbyPOpOP8BicaTD/0klJEKt5p8yuw+uY=
 github.com/jacobsa/crypto v0.0.0-20190317225127-9f44e2d11115/go.mod h1:LadVJg0XuawGk+8L1rYnIED8451UyNxEMdTWCEt5kmU=
 github.com/jacobsa/oglematchers v0.0.0-20150720000706-141901ea67cd h1:9GCSedGjMcLZCrusBZuo4tyKLpKUPenUUqi34AkuFmA=
@@ -852,8 +852,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
 golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
 golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
 golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
-golang.org/x/exp v0.0.0-20230321023759-10a507213a29 h1:ooxPy7fPvB4kwsA2h+iBNHkAbp/4JxTSwCmvdjEYmug=
-golang.org/x/exp v0.0.0-20230321023759-10a507213a29/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc=
+golang.org/x/exp v0.0.0-20230817173708-d852ddb80c63 h1:m64FZMko/V45gv0bNmrNYoDEq8U5YUhetc9cBWKS1TQ=
+golang.org/x/exp v0.0.0-20230817173708-d852ddb80c63/go.mod h1:0v4NqG35kSWCMzLaMeX+IQrlSnVE/bqGSyC2cz/9Le8=
 golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
 golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
 golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=

pkg/applicationserver/applicationserver_test.go

@@ -266,12 +266,9 @@ func TestApplicationServer(t *testing.T) {
 	c := componenttest.NewComponent(t, &component.Config{
 		ServiceBase: config.ServiceBase{
 			GRPC: config.GRPC{
-				Listen:                      ":9184",
+				Listen:                      ":0",
 				AllowInsecureForCredentials: true,
 			},
-			HTTP: config.HTTP{
-				Listen: ":8099",
-			},
 			Cluster: cluster.Config{
 				IdentityServer: isAddr,
 				JoinServer:     jsAddr,
@@ -833,19 +830,16 @@ func TestApplicationServer(t *testing.T) {
 							chs.downErr <- err
 							continue
 						}
-						url := fmt.Sprintf("http://127.0.0.1:8099/api/v3/as/applications/%s/webhooks/%s/devices/%s/down/%s",
+						url := fmt.Sprintf("/api/v3/as/applications/%s/webhooks/%s/devices/%s/down/%s",
 							data.EndDeviceIds.ApplicationIds.ApplicationId, registeredApplicationWebhookID.WebhookId, data.EndDeviceIds.DeviceId, action,
 						)
-						req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(buf))
-						if err != nil {
-							chs.downErr <- err
-							continue
-						}
+						req := httptest.NewRequest(http.MethodPost, url, bytes.NewReader(buf))
 						req.Header.Set("Content-Type", "application/json")
 						req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", key))
-						res, err := http.DefaultClient.Do(req)
-						if err == nil && (res.StatusCode < 200 || res.StatusCode > 299) {
-							err = errors.FromHTTPStatusCode(res.StatusCode)
+						res := httptest.NewRecorder()
+						c.ServeHTTP(res, req)
+						if res.Code < 200 || res.Code > 299 {
+							err = errors.FromHTTPStatusCode(res.Code)
 						}
 						chs.downErr <- err
 					}
@@ -2905,15 +2899,12 @@ func TestLocationFromPayload(t *testing.T) {
 	c := componenttest.NewComponent(t, &component.Config{
 		ServiceBase: config.ServiceBase{
 			GRPC: config.GRPC{
-				Listen:                      ":9189",
+				Listen:                      ":0",
 				AllowInsecureForCredentials: true,
 			},
 			Cluster: cluster.Config{
 				IdentityServer: isAddr,
 			},
-			HTTP: config.HTTP{
-				Listen: ":8100",
-			},
 		},
 	})
 	config := &applicationserver.Config{
@@ -3093,15 +3084,12 @@ func TestUplinkNormalized(t *testing.T) {
 	c := componenttest.NewComponent(t, &component.Config{
 		ServiceBase: config.ServiceBase{
 			GRPC: config.GRPC{
-				Listen:                      ":9189",
+				Listen:                      ":0",
 				AllowInsecureForCredentials: true,
 			},
 			Cluster: cluster.Config{
 				IdentityServer: isAddr,
 			},
-			HTTP: config.HTTP{
-				Listen: ":8100",
-			},
 		},
 	})
 	config := &applicationserver.Config{

pkg/applicationserver/io/pubsub/integrate_test.go

@@ -93,7 +93,7 @@ func TestIntegrate(t *testing.T) {
 	c := componenttest.NewComponent(t, &component.Config{
 		ServiceBase: config.ServiceBase{
 			GRPC: config.GRPC{
-				Listen:                      ":9185",
+				Listen:                      ":0",
 				AllowInsecureForCredentials: true,
 			},
 			Cluster: cluster.Config{

pkg/applicationserver/io/web/webhooks_test.go

@@ -20,6 +20,7 @@ import (
 	"fmt"
 	stdio "io"
 	"net/http"
+	"net/http/httptest"
 	"testing"
 	"time"
 
@@ -498,7 +499,6 @@ func TestWebhooks(t *testing.T) {
 			ttnpb.Right_RIGHT_APPLICATION_DEVICES_WRITE,
 			ttnpb.Right_RIGHT_APPLICATION_TRAFFIC_READ,
 			ttnpb.Right_RIGHT_APPLICATION_TRAFFIC_DOWN_WRITE)
-		httpAddress := "0.0.0.0:8098"
 		conf := &component.Config{
 			ServiceBase: config.ServiceBase{
 				GRPC: config.GRPC{
@@ -508,9 +508,6 @@ func TestWebhooks(t *testing.T) {
 				Cluster: cluster.Config{
 					IdentityServer: isAddr,
 				},
-				HTTP: config.HTTP{
-					Listen: httpAddress,
-				},
 			},
 		}
 		c := componenttest.NewComponent(t, conf)
@@ -555,21 +552,16 @@ func TestWebhooks(t *testing.T) {
 			} {
 				t.Run(tc.Name, func(t *testing.T) {
 					a := assertions.New(t)
-					url := fmt.Sprintf("http://%s/api/v3/as/applications/%s/webhooks/%s/devices/%s/down/replace",
-						httpAddress, tc.ID.ApplicationId, registeredWebhookID, registeredDeviceID.DeviceId,
+					url := fmt.Sprintf("/api/v3/as/applications/%s/webhooks/%s/devices/%s/down/replace",
+						tc.ID.ApplicationId, registeredWebhookID, registeredDeviceID.DeviceId,
 					)
 					body := bytes.NewReader([]byte(`{"downlinks":[]}`))
-					req, err := http.NewRequest(http.MethodPost, url, body)
-					if !a.So(err, should.BeNil) {
-						t.FailNow()
-					}
+					req := httptest.NewRequest(http.MethodPost, url, body)
 					req.Header.Set("Content-Type", "application/json")
 					req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", tc.Key))
-					res, err := http.DefaultClient.Do(req)
-					if !a.So(err, should.BeNil) {
-						t.FailNow()
-					}
-					a.So(res.StatusCode, should.Equal, tc.ExpectCode)
+					res := httptest.NewRecorder()
+					c.ServeHTTP(res, req)
+					a.So(res.Code, should.Equal, tc.ExpectCode)
 					downlinks, err := io.DownlinkQueueList(ctx, registeredDeviceID)
 					if !a.So(err, should.BeNil) {
 						t.FailNow()

pkg/band/band_test.go

@@ -1148,27 +1148,27 @@ func TestGenerateChMask(t *testing.T) {
 					},
 				},
 				{
-					Cntl: 4,
 					Mask: [16]bool{
 						false, false, false, false, false, false, false, false,
 						false, false, false, false, false, false, false, false,
 					},
 				},
 				{
-					Cntl: 3,
+					Cntl: 1,
 					Mask: [16]bool{
 						false, false, false, false, false, false, false, false,
 						false, false, false, false, false, false, false, false,
 					},
 				},
 				{
-					Cntl: 1,
+					Cntl: 3,
 					Mask: [16]bool{
 						false, false, false, false, false, false, false, false,
 						false, false, false, false, false, false, false, false,
 					},
 				},
 				{
+					Cntl: 4,
 					Mask: [16]bool{
 						false, false, false, false, false, false, false, false,
 						false, false, false, false, false, false, false, false,
@@ -1221,7 +1221,6 @@ func TestGenerateChMask(t *testing.T) {
 					},
 				},
 				{
-					Cntl: 3,
 					Mask: [16]bool{
 						false, false, false, false, false, false, false, false,
 						false, false, false, false, false, false, false, false,
@@ -1235,6 +1234,7 @@ func TestGenerateChMask(t *testing.T) {
 					},
 				},
 				{
+					Cntl: 3,
 					Mask: [16]bool{
 						false, false, false, false, false, false, false, false,
 						false, false, false, false, false, false, false, false,

pkg/band/channel_mask.go

@@ -446,8 +446,8 @@ func generateChMask72Generic(currentChs, desiredChs []bool, atomic bool) ([]ChMa
 		// We also sort the masks descending on the number of enabled channels in order to ensure
 		// that intermediary states through which the end device goes while parsing the masks
 		// are valid (i.e. the total number of enabled channels is always greater than 0).
-		slices.SortFunc(pairs, func(a, b ChMaskCntlPair) bool {
-			return trueCount(a.Mask[:]...) >= trueCount(b.Mask[:]...)
+		slices.SortFunc(pairs, func(a, b ChMaskCntlPair) int {
+			return trueCount(b.Mask[:]...) - trueCount(a.Mask[:]...)
 		})
 		return pairs, nil