Support fetching end device information from the device repository with vendor ID and profile ID

[KrishnaIyer]

Summary

Support fetching end device information from the device repository with vendor ID and profile ID.

Current Situation

When an end device is currently registered via the console using a QR code, the console does the following

1. ListBrands from the DeviceRepository service.
2. Parse the QR code using the Parse RPC of the EndDeviceQRCodeGenerator service.
   • This returns and EndDeviceTemplate message which contains the EndDevice message which in-turn has the important LoRaAllianceProfileIdentifiers.
   • This has a VendorID that uniquely identifies and end device vendor (ex: The Things Industries). We store this in the index file of the device repo.
   • This also contains a VendorProfileID that is currently unused.
3. The console then lists all the end device models defined for this end device (stored as <device-name>.yaml in the folder for that device. ex: https://github.com/TheThingsNetwork/lorawan-devices/tree/master/vendor/the-things-industries) using the ListModels RPC.

• This returns a list of models that contain info such as the firmware/hardware versions, end device profiles (which have some LoRaWAN specific parameters), end device codec ID etc.

4. The user then uses the drop down to choose the model and the firmware/hardware versions and the profile ID.
5. The console fills the necessary other information.
6. The console then fetches the end device profile using the GetTemplate RPC.
7. Now that the generic device information is filled, the console can now move on to specific info like the EUIs, keys etc which is out of scope for this issue.

Why do we need this? Who uses it, and when?

When we parse a LoRa Alliance TR005 QR code, the QR code already contains the VendorID and VendorProfileID fields. We can simplify this flow and directly get the End Device Model and the End Device Profile from it without user intervention.
This makes the onboarding experience very smooth.

Proposed Implementation

New Console flow (After the backend part below is implemented)

Steps 1 and 2 from the Current Situation is the same.
Step 3: The Console calls the new GetEndDeviceProvisioningInfo RPC (see below) and passes the vendor ID and vendor profile ID.
a. If these are not properly defined in the device repository or there's an error, the Console can just use the same flow as now. (OR)
b. If the RPC returns the correct End Device Model and End Device Profile then the console can fill all the necessary info and skip steps 4, 5, 6 in Current Situation and move to Step 7.

Backend implementation

We need the following to make this work.

1. Introduce a new EndDeviceInfo or EndDeviceProvisioningInfo message which contains the EndDeviceModel and EndDeviceTemplate.
2. Add a new index to bleve (ex: https://github.com/TheThingsNetwork/lorawan-stack/blob/v3.32/pkg/devicerepository/store/bleve/init.go#L64) that stores the EndDeviceProvisioningInfo message and has the vendor ID and vendor profile ID for search.
3. Update the Initialize flow to fetch this info from DR (remote or local) and create the index.
4. Introduce a new method to the Store to fetch EndDeviceProvisioningInfo via the vendorID and vendor profile ID. The implementation of this method uses the index from step 2/3.
5. Add a new RPC (ex: GetEndDeviceProvisioningInfo) to the device repository service (ex: https://github.com/TheThingsNetwork/lorawan-stack/blob/v3.32/pkg/devicerepository/grpc.go#L120) which internally uses this new store method.

@johanstokking: Let me know if this is clear/complete.

Contributing

• I can help by doing more research.
• I can help by implementing the feature after the proposal above is approved.
• I can help by testing the feature before it's released.

Validation

• The feature is tested in a staging environment.
• The feature is documented in The Things Stack Documentation

Code of Conduct

• I agree to follow TTN's Community Code of Conduct.

[vlasebian]

Tested on staging1.

Prerequisites:

• Have an app created in staging1 environment

Steps:

• Open the staging1 console
• Click on the + sign for adding an entity
• Select Register end device in an application
• Select the app
• In the Register end device screen, select the 'Select the end device in the LoRaWAN Device Repository'
• Try multiple combinations of End device brand + Model + Hardware Ver. + Firmware Ver. + Profile (Region)
  • Selecting the Profile is what triggers the call to GetTemplates
• Check the call in the Network tab of the Developer tools in the browser

Results:
Tried the following combinations:

• TTI, Generic Node, 1.1, 1.0 EU_863+870

200 OK

{
    "end_device": {
        "version_ids": {
            "brand_id": "the-things-industries",
            "model_id": "generic-node-sensor-edition",
            "firmware_version": "1.0",
            "band_id": "EU_863_870"
        },
        "lorawan_version": "MAC_V1_0_3",
        "lorawan_phy_version": "PHY_V1_0_3_REV_A",
        "supports_join": true,
        "mac_settings": {
            "supports_32_bit_f_cnt": true
        },
        "formatters": {
            "up_formatter": "FORMATTER_REPOSITORY",
            "down_formatter": "FORMATTER_REPOSITORY"
        }
    },
    "field_mask": {
        "paths": [
            "version_ids",
            "supports_join",
            "supports_class_b",
            "supports_class_c",
            "lorawan_version",
            "lorawan_phy_version",
            "formatters",
            "mac_settings.supports_32_bit_f_cnt"
        ]
    }
}

• Bosch, Parking Lot Sensor, 9, 0.23.3, AS_923

200 OK

{
    "end_device": {
        "version_ids": {
            "brand_id": "bosch",
            "model_id": "tps110",
            "firmware_version": "0.23.3",
            "band_id": "AS_923"
        },
        "lorawan_version": "MAC_V1_0_2",
        "lorawan_phy_version": "PHY_V1_0_2_REV_B",
        "supports_join": true,
        "mac_settings": {
            "supports_32_bit_f_cnt": true
        }
    },
    "field_mask": {
        "paths": [
            "version_ids",
            "supports_join",
            "supports_class_b",
            "supports_class_c",
            "lorawan_version",
            "lorawan_phy_version",
            "mac_settings.supports_32_bit_f_cnt"
        ]
    }
}

• Arduino, MKR WAN 1310, 1.0, 1.2.3, US_902_928

200 OK

{
    "end_device": {
        "version_ids": {
            "brand_id": "arduino",
            "model_id": "mkr-wan-1310",
            "firmware_version": "1.2.3",
            "band_id": "US_902_928"
        },
        "lorawan_version": "MAC_V1_0_2",
        "lorawan_phy_version": "PHY_V1_0_2_REV_A",
        "supports_join": true,
        "mac_settings": {
            "supports_32_bit_f_cnt": true
        },
        "formatters": {
            "up_formatter": "FORMATTER_REPOSITORY",
            "down_formatter": "FORMATTER_REPOSITORY"
        }
    },
    "field_mask": {
        "paths": [
            "version_ids",
            "supports_join",
            "supports_class_b",
            "supports_class_c",
            "lorawan_version",
            "lorawan_phy_version",
            "formatters",
            "mac_settings.supports_32_bit_f_cnt"
        ]
    }
}

As the console doesn't use the endpoints that require the vendor profile ID + vendor ID parameters, I also did some API calls:

curl -s --location  -H "Authorization: $TTN_STAGING_AUTH_TOKEN" https://tti.staging1.cloud.thethings.industries/api/v3/dr/vendors/428/profiles/1/template

curl -s --location  -H "Authorization: $TTN_STAGING_AUTH_TOKEN" https://tti.staging1.cloud.thethings.industries/api/v3/dr/vendors/428/profiles/2/template

These return a not found for now. I believe we have to rebuild the bleve.index as a new document type was introduced. I got the same issue locally, but after rebuilding the index the endpoint worked.

EDIT: tested after rc release. the profiles are found now:

curl -s --location  -H "Authorization: $TTN_STAGING_AUTH_TOKEN" https://tti.staging1.cloud.thethings.industries/api/v3/dr/vendors/428/profiles/1/template

{
  "end_device": {
    "version_ids": {
      "brand_id": "the-things-industries",
      "model_id": "generic-node-sensor-edition",
      "hardware_version": "1.1",
      "firmware_version": "1.0",
      "band_id": "EU_863_870"
    },
    "lorawan_version": "MAC_V1_0_3",
    "lorawan_phy_version": "PHY_V1_0_3_REV_A",
    "supports_join": true,
    "mac_settings": {
      "supports_32_bit_f_cnt": true
    },
    "formatters": {
      "up_formatter": "FORMATTER_REPOSITORY",
      "down_formatter": "FORMATTER_REPOSITORY"
    }
  },
  "field_mask": {
    "paths": [
      "version_ids",
      "supports_join",
      "supports_class_b",
      "supports_class_c",
      "lorawan_version",
      "lorawan_phy_version",
      "formatters",
      "mac_settings.supports_32_bit_f_cnt"
    ]
  }
}

curl -s --location  -H "Authorization: $TTN_STAGING_AUTH_TOKEN" https://tti.staging1.cloud.thethings.industries/api/v3/dr/vendors/428/profiles/2/template

{
  "end_device": {
    "version_ids": {
      "brand_id": "the-things-industries",
      "model_id": "generic-node-sensor-edition",
      "hardware_version": "1.1",
      "firmware_version": "1.0",
      "band_id": "US_902_928"
    },
    "lorawan_version": "MAC_V1_0_3",
    "lorawan_phy_version": "PHY_V1_0_3_REV_A",
    "supports_join": true,
    "mac_settings": {
      "supports_32_bit_f_cnt": true
    },
    "formatters": {
      "up_formatter": "FORMATTER_REPOSITORY",
      "down_formatter": "FORMATTER_REPOSITORY"
    }
  },
  "field_mask": {
    "paths": [
      "version_ids",
      "supports_join",
      "supports_class_b",
      "supports_class_c",
      "lorawan_version",
      "lorawan_phy_version",
      "formatters",
      "mac_settings.supports_32_bit_f_cnt"
    ]
  }
}

[KrishnaIyer]

Thanks @vlasebian: This is done and tested.