bdev_nvme_attach_controller: json response error: Could not retrieve PSK from file: /tmp/opikey981191686

[glimchb]

run:

+ docker run --network=opi-spdk-bridge_opi --rm docker.io/namely/grpc-cli call --json_input --json_output opi-spdk-server:50051 CreateNvmeRemoteController '{nvme_remote_controller : {multipath: '\''NVME_MULTIPATH_MULTIPATH'\'', tcp: {psk: '\''TlZNZVRMU2tleS0xOjAxOk1EQXhNVEl5TXpNME5EVTFOalkzTnpnNE9UbGhZV0ppWTJOa1pHVmxabVp3SkVpUTo='\''}}, nvme_remote_controller_id: '\''nvmetls17'\''}'
connecting to opi-spdk-server:50051
{
 "name": "//storage.opiproject.org/volumes/nvmetls17",
 "multipath": "NVME_MULTIPATH_MULTIPATH",
 "tcp": {
  "psk": "TlZNZVRMU2tleS0xOjAxOk1EQXhNVEl5TXpNME5EVTFOalkzTnpnNE9UbGhZV0ppWTJOa1pHVmxabVp3SkVpUTo="
 }
}
Rpc succeeded with OK status
+ docker run --network=opi-spdk-bridge_opi --rm docker.io/namely/grpc-cli call --json_input --json_output opi-spdk-server:50051 CreateNvmePath '{nvme_path : {controller_name_ref: '\''//storage.opiproject.org/volumes/nvmetls17'\'', traddr:"172.21.0.3", trtype:'\''NVME_TRANSPORT_TCP'\'', fabrics: { subnqn:'\''nqn.2022-09.io.spdk:opitest1'\'', trsvcid:'\''7777'\'', adrfam:'\''NVME_ADRFAM_IPV4'\'', hostnqn:'\''nqn.2014-08.org.nvmexpress:uuid:feb98abe-d51f-40c8-b348-2753f3571d3c'\''}}, nvme_path_id: '\''nvmetls17path0'\''}'
connecting to opi-spdk-server:50051
Rpc failed with status code 2, error message: bdev_nvme_attach_controller: json response error: Could not retrieve PSK from file: /tmp/opikey981191686

logs:

opi-spdk-bridge-opi-spdk-server-1  | 2023/09/28 18:42:45 Notice, TLS is used to establish connection: to name:"//storage.opiproject.org/volumes/nvmetls17path0"  controller_name_ref:"//storage.opiproject.org/volumes/nvmetls17"  trtype:NVME_TRANSPORT_TCP  traddr:"172.21.0.3"  fabrics:{trsvcid:7777  subnqn:"nqn.2022-09.io.spdk:opitest1"  adrfam:NVME_ADRFAM_IPV4  hostnqn:"nqn.2014-08.org.nvmexpress:uuid:feb98abe-d51f-40c8-b348-2753f3571d3c"}
opi-spdk-bridge-opi-spdk-server-1  | 2023/09/28 18:42:45 Sending to SPDK: {"jsonrpc":"2.0","method":"bdev_nvme_attach_controller","id":148,"params":{"name":"nvmetls17","trtype":"TCP","traddr":"172.21.0.3","hostnqn":"nqn.2014-08.org.nvmexpress:uuid:feb98abe-d51f-40c8-b348-2753f3571d3c","adrfam":"IPV4","trsvcid":"7777","subnqn":"nqn.2022-09.io.spdk:opitest1","psk":"/tmp/opikey981191686"}}
opi-spdk-bridge-spdk-1             | [2023-09-28 18:42:45.800353] bdev_nvme_rpc.c: 475:rpc_bdev_nvme_attach_controller: *NOTICE*: TLS support is considered experimental
opi-spdk-bridge-spdk-1             | [2023-09-28 18:42:45.800415] bdev_nvme_rpc.c: 329:tcp_load_psk: *ERROR*: Could not read permissions for PSK file
opi-spdk-bridge-opi-spdk-server-1  | 2023/09/28 18:42:45 Received from SPDK: {"jsonrpc":"2.0","id":148,"result":null,"error":{"code":-22,"message":"Could not retrieve PSK from file: /tmp/opikey981191686"}}
opi-spdk-bridge-opi-spdk-server-1  | 2023/09/28 18:42:45 Cleanup key file /tmp/opikey981191686: <nil>

spdk code that returns that log is here:

	if (stat(fname, &statbuf) != 0) {
		SPDK_ERRLOG("Could not read permissions for PSK file\n");
		return -EACCES;
	}

https://github.com/spdk/spdk/blob/32e5ce31e17eb4afc921486b955a6b351186bfda/lib/nvmf/tcp.c#L3537

[glimchb]

this is obvious to me now, we run SPDK and BRIDGE in 2 separate containers, so SPDK can't access same filesystem
but we do:

    volumes_from:
      - spdk

but spdk has

  spdk:
    volumes:
      - /var/tmp:/var/tmp

but key is in /tmp see

opi-spdk-bridge/pkg/backend/nvme_path.go

Line 298 in 202c6ed

keyFile, err := s.psk.createTempFile("", "opikey")

fixing to /var/tmp

 func (s *Server) keyToTemporaryFile(pskKey []byte) (string, error) {
-       keyFile, err := s.psk.createTempFile("", "opikey")
+       keyFile, err := s.psk.createTempFile("/var/tmp", "opikey")

[glimchb]

I pushed a fix to #398 but we can create a separate PR just for this fix

[artek-koltun]

this is obvious to me now, we run SPDK and BRIDGE in 2 separate containers, so SPDK can't access same filesystem but we do:

    volumes_from:
      - spdk

but spdk has

  spdk:
    volumes:
      - /var/tmp:/var/tmp

but key is in /tmp see

opi-spdk-bridge/pkg/backend/nvme_path.go

Line 298 in 202c6ed

keyFile, err := s.psk.createTempFile("", "opikey")

fixing to /var/tmp

 func (s *Server) keyToTemporaryFile(pskKey []byte) (string, error) {
-       keyFile, err := s.psk.createTempFile("", "opikey")
+       keyFile, err := s.psk.createTempFile("/var/tmp", "opikey")

correct reasoning