Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[usb_stream] Bad UVC frame (AEGHB-659) #360

Open
3 tasks done
zry98 opened this issue May 18, 2024 · 10 comments
Open
3 tasks done

[usb_stream] Bad UVC frame (AEGHB-659) #360

zry98 opened this issue May 18, 2024 · 10 comments

Comments

@zry98
Copy link

zry98 commented May 18, 2024

Answers checklist.

  • I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
  • I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
  • I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

I have successfully connected ESP32-S3 (N16R8) to a UVC camera using the example usb_camera_mic_spk, but the obtained frame looks very bad as below:

20240518210402

Related logs
I (643) USB_STREAM: line 151 HCD_PORT_EVENT_CONNECTION
I (644) USB_STREAM: Action: ACTION_DEVICE_CONNECT
I (694) USB_STREAM: Resetting Port
I (754) USB_STREAM: Setting Port FIFO, 1
I (755) USB_STREAM: USB Speed: full-speed
I (755) USB_STREAM: ENUM Stage START, Succeed
I (756) USB_STREAM: ENUM Stage GET_SHORT_DEV_DESC, Succeed
I (768) USB_STREAM: Default pipe endpoint MPS update to 64
I (768) USB_STREAM: ENUM Stage CHECK_SHORT_DEV_DESC, Succeed
I (769) USB_STREAM: ENUM Stage SET_ADDR, Succeed
I (791) USB_STREAM: ENUM Stage CHECK_ADDR, Succeed
I (792) USB_STREAM: ENUM Stage GET_FULL_DEV_DESC, Succeed
*** Device descriptor ***
bcdUSB 2.00
bDeviceClass 0xef
bDeviceSubClass 0x2
bDeviceProtocol 0x1
bMaxPacketSize0 64
idVendor 0x2bdf
idProduct 0x101
bNumConfigurations 1
I (805) USB_STREAM: ENUM Stage CHECK_FULL_DEV_DESC, Succeed
I (806) USB_STREAM: ENUM Stage GET_SHORT_CONFIG_DESC, Succeed
I (818) USB_STREAM: ENUM Stage CHECK_SHORT_CONFIG_DESC, Succeed
I (819) USB_STREAM: ENUM Stage GET_FULL_CONFIG_DESC, Succeed
*** Configuration descriptor ***
wTotalLength 433
bNumInterfaces 2
bConfigurationValue 1
*** Interface Association Descriptor: Video ***
        *** Interface descriptor ***
        bInterfaceNumber 0
        bAlternateSetting 0
        bNumEndpoints 1
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x1
                *** Endpoint descriptor ***
                bEndpointAddress 0x83   EP 3 IN
                bmAttributes 0x3        INT
                wMaxPacketSize 16
                bInterval 8
        *** Interface descriptor ***
        bInterfaceNumber 1
        bAlternateSetting 0
        bNumEndpoints 1
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x2
        *** Class-specific VS Interface Descriptor ***
        bNumFormats 3
        *** VS Format MJPEG Descriptor ***
        bFormatIndex 0x2
        bNumFrameDescriptors 3
        bDefaultFrameIndex 1
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x1
        wWidth 120
        wHeigh 160
        FrameInterval[0] 400000
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x2
        wWidth 240
        wHeigh 320
        FrameInterval[0] 333333
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x3
        wWidth 640
        wHeigh 360
        FrameInterval[0] 333333
                *** Endpoint descriptor ***
                bEndpointAddress 0x81   EP 1 IN
                bmAttributes 0x2        BULK
                wMaxPacketSize 64
                bInterval 0
I (826) USB_STREAM: Actual VS Interface(MPS <= 512) found, interface = 1, alt = 0
I (827) USB_STREAM:     Endpoint(BULK) Addr = 0x81, MPS = 64
I (827) USB_STREAM: Actual MJPEG format index, format index = 2, contains 3 frames
I (828) USB_STREAM: Actual Frame: 7, width*heigh: 65535*65535, frame index = 1
I (828) USB_STREAM: ENUM Stage CHECK_FULL_CONFIG_DESC, Succeed
I (830) USB_STREAM: ENUM Stage SET_CONFIG, Succeed
I (832) uvc_mic_spk_demo: UVC: get frame list size = 3, current = 0
I (832) uvc_mic_spk_demo:       frame[0] = 120x160
I (833) uvc_mic_spk_demo:       frame[1] = 240x320
I (833) uvc_mic_spk_demo:       frame[2] = 640x360
I (833) uvc_mic_spk_demo: Device connected
I (834) USB_STREAM: Probe Format(2), Frame(1) 120*160, interval(400000)
I (834) USB_STREAM: Probe payload size = 1024
bFormatIndex: 2
bFrameIndex: 1
dwFrameInterval: 400000
dwMaxPayloadTransferSize: 16384
bInterfaceNumber: 0
I (865) USB_STREAM: dwMaxPayloadTransferSize set = 1024, probed = 16384
I (865) USB_STREAM: Sample processing task start
I (886) USB_STREAM: USB Device Connected
W (10546) USB_STREAM: line 155 HCD_PORT_EVENT_DISCONNECTION
I (10547) USB_STREAM: Recover Stream Task
I (10547) USB_STREAM: usb stream task wait reset
I (10547) USB_STREAM: usb stream task reset, reason: device recover
I (10547) USB_STREAM: Resetting UVC pipe
I (10547) USB_STREAM: sample processing stop
I (10548) USB_STREAM: Sample processing task deleted
I (10598) uvc_mic_spk_demo: Device disconnected
I (10599) USB_STREAM: Action: ACTION_PIPE_DFLT_DISABLE
I (10599) USB_STREAM: Action: ACTION_PORT_RECOVER
I (10599) USB_STREAM: Action: ACTION_DEVICE_DISCONN
I (10599) USB_STREAM: Waiting USB Connection
I (45973) USB_STREAM: line 151 HCD_PORT_EVENT_CONNECTION
I (45974) USB_STREAM: Action: ACTION_DEVICE_CONNECT
I (46024) USB_STREAM: Resetting Port
I (46084) USB_STREAM: Setting Port FIFO, 1
I (46084) USB_STREAM: USB Speed: full-speed
I (46085) USB_STREAM: ENUM Stage START, Succeed
I (46086) USB_STREAM: ENUM Stage GET_SHORT_DEV_DESC, Succeed
I (46088) USB_STREAM: Default pipe endpoint MPS update to 64
I (46088) USB_STREAM: ENUM Stage CHECK_SHORT_DEV_DESC, Succeed
I (46089) USB_STREAM: ENUM Stage SET_ADDR, Succeed
I (46100) USB_STREAM: ENUM Stage CHECK_ADDR, Succeed
I (46101) USB_STREAM: ENUM Stage GET_FULL_DEV_DESC, Succeed
*** Device descriptor ***
bcdUSB 2.00
bDeviceClass 0xef
bDeviceSubClass 0x2
bDeviceProtocol 0x1
bMaxPacketSize0 64
idVendor 0x2bdf
idProduct 0x101
bNumConfigurations 1
I (46103) USB_STREAM: ENUM Stage CHECK_FULL_DEV_DESC, Succeed
I (46104) USB_STREAM: ENUM Stage GET_SHORT_CONFIG_DESC, Succeed
I (46106) USB_STREAM: ENUM Stage CHECK_SHORT_CONFIG_DESC, Succeed
I (46107) USB_STREAM: ENUM Stage GET_FULL_CONFIG_DESC, Succeed
*** Configuration descriptor ***
wTotalLength 433
bNumInterfaces 2
bConfigurationValue 1
*** Interface Association Descriptor: Video ***
        *** Interface descriptor ***
        bInterfaceNumber 0
        bAlternateSetting 0
        bNumEndpoints 1
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x1
                *** Endpoint descriptor ***
                bEndpointAddress 0x83   EP 3 IN
                bmAttributes 0x3        INT
                wMaxPacketSize 16
                bInterval 8
        *** Interface descriptor ***
        bInterfaceNumber 1
        bAlternateSetting 0
        bNumEndpoints 1
        bInterfaceClass 0xe (Video)
        bInterfaceSubClass 0x2
        *** Class-specific VS Interface Descriptor ***
        bNumFormats 3
        *** VS Format MJPEG Descriptor ***
        bFormatIndex 0x2
        bNumFrameDescriptors 3
        bDefaultFrameIndex 1
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x1
        wWidth 120
        wHeigh 160
        FrameInterval[0] 400000
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x2
        wWidth 240
        wHeigh 320
        FrameInterval[0] 333333
        *** VS MJPEG Frame Descriptor ***
        bFrameIndex 0x3
        wWidth 640
        wHeigh 360
        FrameInterval[0] 333333
                *** Endpoint descriptor ***
                bEndpointAddress 0x81   EP 1 IN
                bmAttributes 0x2        BULK
                wMaxPacketSize 64
                bInterval 0
I (46136) USB_STREAM: Actual VS Interface(MPS <= 512) found, interface = 1, alt = 0
I (46136) USB_STREAM:   Endpoint(BULK) Addr = 0x81, MPS = 64
I (46136) USB_STREAM: Actual MJPEG format index, format index = 2, contains 3 frames
I (46137) USB_STREAM: Actual Frame: 7, width*heigh: 65535*65535, frame index = 1
I (46137) USB_STREAM: ENUM Stage CHECK_FULL_CONFIG_DESC, Succeed
I (46139) USB_STREAM: ENUM Stage SET_CONFIG, Succeed
I (46141) uvc_mic_spk_demo: UVC: get frame list size = 3, current = 0
I (46141) uvc_mic_spk_demo:     frame[0] = 120x160
I (46142) uvc_mic_spk_demo:     frame[1] = 240x320
I (46142) uvc_mic_spk_demo:     frame[2] = 640x360
I (46142) uvc_mic_spk_demo: Device connected
I (46143) USB_STREAM: Probe Format(2), Frame(1) 120*160, interval(400000)
I (46143) USB_STREAM: Probe payload size = 1024
bFormatIndex: 2
bFrameIndex: 1
dwFrameInterval: 400000
dwMaxPayloadTransferSize: 16384
bInterfaceNumber: 0
I (46174) USB_STREAM: dwMaxPayloadTransferSize set = 1024, probed = 16384
I (46174) USB_STREAM: Sample processing task start
I (46218) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 1, width = 120, height = 160, length = 5391, ptr = 0
// ommited some lines of invalid frame (length too short)
// valid frame starts below:
I (47767) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 14, width = 120, height = 160, length = 16734, ptr = 0
I (47783) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 15, width = 120, height = 160, length = 16708, ptr = 0
I (47811) uvc_mic_spk_demo: uvc callback! frame_format = 7, seq = 16, width = 120, height = 160, length = 16746, ptr = 0
// ...

Since the camera is the same as the one in #325, I have set CONFIG_NUM_BULK_BYTES_PER_URB=1024 to make it work.

@github-actions github-actions bot changed the title [usb_stream] Bad UVC frame [usb_stream] Bad UVC frame (AEGHB-659) May 18, 2024
@zry98
Copy link
Author

zry98 commented May 18, 2024

@lijunru-hub @leeebo 感谢~

@zry98
Copy link
Author

zry98 commented May 19, 2024

It seems to be caused by:

} else if (strmh->reassembling) {
ESP_LOGV(TAG, "reassembling %u + %u", strmh->got_bytes, payload_len);
data_len = payload_len;

which is setting data_len incorrectly. I'm getting mostly correct frames after changing this part to:

        } else if (strmh->reassembling) {
            header_len = payload[0];
            data_len = payload_len - header_len;
            /* checking the end-of-header */
            variable_offset = 2;
            header_info = payload[1];
            ESP_LOGV(TAG, "reassembling %u + %u", strmh->got_bytes, data_len);
        } else {

But I'm still getting invalid frames constantly (around every 150 frames), I guess this function still needs more fixes.

@leeebo
Copy link
Collaborator

leeebo commented May 20, 2024

Most USB Bulk cameras use Single Transfer per Sample mode, which is usb_stream currently supported.

image

The camera you currently use seems to be using Multiple Transfers per Sample mode, which is not supported now.

image

@zry98
Copy link
Author

zry98 commented May 20, 2024

The camera you currently use seems to be using Multiple Transfers per Sample mode, which is not supported now.

Thanks for the clue! Are you planning to support this?

@leeebo
Copy link
Collaborator

leeebo commented May 21, 2024

@zry98 It has been supported in #361, please help try it with your camera.

zry98 added a commit to zry98/esp-iot-solution that referenced this issue May 21, 2024
@zry98
Copy link
Author

zry98 commented May 21, 2024

Unfortunately, it still requires the changes in #360 (comment) to work. Can you test if zry98@d3fa962 is OK for cameras of single transfer per sample mode? (I don't have one to test against)

Btw, I noticed that with the changes applied, it randomly panics after some time as below:

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x4037ab9a  PS      : 0x00060934  A0      : 0x82004365  A1      : 0x3fc9f030
0x4037ab9a: esp_cpu_wait_for_intr at C:/tools/esp-idf/v5.2.1/esp-idf/components/esp_hw_support/cpu.c:145

A2      : 0x00000000  A3      : 0x00000001  A4      : 0x8037e7b6  A5      : 0x3fce8340
A6      : 0x00060023  A7      : 0x00000003  A8      : 0x8201cc0e  A9      : 0x3fc9f000  
A10     : 0x00000000  A11     : 0x00000001  A12     : 0x8037e7b6  A13     : 0x3fc9f1e0
A14     : 0x3fc9abec  A15     : 0x00000001  SAR     : 0x0000001e  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000


Backtrace: 0x4037ab97:0x3fc9f030 0x42004362:0x3fc9f050 0x4037f50c:0x3fc9f070
0x4037ab97: xt_utils_wait_for_intr at C:/tools/esp-idf/v5.2.1/esp-idf/components/xtensa/include/xt_utils.h:81
 (inlined by) esp_cpu_wait_for_intr at C:/tools/esp-idf/v5.2.1/esp-idf/components/esp_hw_support/cpu.c:132
0x42004362: esp_vApplicationIdleHook at C:/tools/esp-idf/v5.2.1/esp-idf/components/esp_system/freertos_hooks.c:59
0x4037f50c: prvIdleTask at C:/tools/esp-idf/v5.2.1/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4273 (discriminator 1)

@leeebo
Copy link
Collaborator

leeebo commented May 22, 2024

@zry98 zry98@d3fa962 does not work in our test cameras

According to the UVC1.5 protocol, One payload only contains one header. So we can't subtract the length of the header from each packet. If your camera only works this, it may be a custom protocol, can you provide a link to buy the camera?

for the random panics, it could has no direct connection with the patch. Would you please run the example under this repository to test your camera, and please config your log level to verbose to get more details

@zry98
Copy link
Author

zry98 commented May 22, 2024

zry98@d3fa962 does not work in our test cameras

@leeebo Sorry I didn't pay much attention to the header, thought a simple if (bulk_xfer) would do it.

So we still need to find a way to know if it's in single or multi transfer mode, maybe checking the first several transfers to determine? then process the later ones accordingly.

can you provide a link to buy the camera?

It's the Hikvision DS-2TM01-3XF (or TB-4117-3/S for OEM, more popular on the market) thermal camera. (在淘宝和闲鱼能找到很多)

PotPlayer, webcamtests.com on Windows and USB Camera on Android all work flawlessly with it.

For your reference, I've captured some packets with Wireshark: hik4117.zip

@leeebo
Copy link
Collaborator

leeebo commented May 22, 2024

@zry98 thanks for the information. we will buy one for better support

@leeebo
Copy link
Collaborator

leeebo commented May 27, 2024

Hi @zry98 would you please send us the camera for better support? After the fix, we will send it back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants