CPU and RAM utilization although I am using cap = ffmpegcv.toCUDA

[asad1416]

Hello Everyone, I am new user of ffmpeg and I want to use ffmpegcv to minimize my RAM and CPU utilization. But the issue it is still using RAM. My object is to use GPU for reading frame and putting tensor to yolo model so that RAM/and Cpu involment would be mimium. I am sharing my sample code please help me in this regard. when I nvidia-smi -l it shows that ffmpeg porcess running on it .

import ffmpegcv
import os
from ultralytics import YOLO
import torch.nn.functional as F
os.environ['PATH'] += ':/usr/local/cuda-12.3/bin'
W=1920
H=1080

model_p="/UNITY-NFS/Data_B/deployment/AIEngine_CBOR/models/yolov8n.engine"
cap = ffmpegcv.toCUDA(ffmpegcv.VideoCaptureStreamRT(rtsp, pix_fmt='nv12', resize=(W,H),gpu=0),tensor_format='chw')
W=1920
H=1080
person_model = YOLO("./models/yolov8n.pt")
person_model.to("cuda")

while True:
    ret, frame_CHW_CUDA = cap.read_torch()
    print(frame_CHW_CUDA)
    frame_CHW_CUDA = frame_CHW_CUDA.unsqueeze(0)
    print(type(frame_CHW_cuda))
    new_height = 640  
    new_width = 640  
    resized_tensor = F.interpolate(frame_CHW_CUDA, size=(new_height, new_width), mode='bilinear', align_corners=False)
    detections =person_model.predict(resized_tensor, verbose=False)
    print(detections)

[chenxinfeng4]

Well, I suppose you to use ffmpegcv.VideoCaptureStreamRT(resize=(640,640), ...) instead of 'resize + F.interpolate'. The resize can be aligned to center or corner, see the ffmpegcv.

Second, the RAM may be used by the YOLO model, such as post-processing (may be NMS).

Last, the ffmpegcv.toCUDA(ffmpegcv.VideoCaptureStreamRT(...)) will copy the image frame from CPU RAM to GPU RAM, but it has very minimal cost.