Using Cameras In Conjunction with Autopilot

[jmdelahanty]

Hello!

As we further think about getting our systems on autopilot, I was wondering how people handle recording video through USB cameras in a distributed way. We'll be doing something like 12-14 boxes for our setup and, while we can definitely run all the boxes from just one computer using Autopilot, I'm doubting that we could have that many cameras being fed into the head node. Maybe if we upgraded the machine to have a faster processor/NVMe drive we could, but even then it seems like a stretch.

I haven't tested what the datarate of the cameras we're using is, but it just seems like a lot of video feeds to handle.

So I was curious, how do people manage this many boxes/how would you imagine managing this many video streams? I'd think that spawning 14 processes run through Pis would be plenty fast and that the OS of the main machine could handle receiving timestamps of that data. Do you think one machine networked with a big switch could accomplish this?

[horsto]

I have a similar question - how many raspberry pi hq camera streams can you stream at full rest (1080 30 fps) into the head node? Has anybody tested the capability and latencies? And is the assumption correct that the full resolution will be maintained?

[jmdelahanty]

From my understanding through a different discussion on the repo, it will depend on how much CPU time is needed to compress the video. I haven't tested this out myself, but I'm willing to bet that a Pi could handle one video feed at least with H264 encoding! At least if you don't have the crf value too high maybe...

@cxrodgers might know also?

[sneakers-the-rat]

I haven't tested the top end of streaming video, but i think it would be pretty dependent on the power of the receiving computer. I only have ~5 picams to test in parallel, but can give that a shot if it's useful. In any case there are a few ways this could be handled:

• many to one: by default I imagine the main bottleneck will be the message handling -> writing into hdf5 part. I've been surprised by how much it can handle, but definitely not infinite. You can also write an additional networking callback that uses the video writer that lets you drop frames into a queue to be encoded with x264 so you could roughly trade cpu for memory
• many to many: if one receiving machine is the bottleneck, then you could set up additional recipients to store the data. There isn't a builtin way to do this at the moment, but all it would take is setting a different recipient of a .stream() call and booting up a few agents with a Net_Node wrapping either a Subject.prepare_run() + save_data() call or the above linked video writer
• local storage, integrate later: Another strategy is to encode the video locally on the acquiring machines like with write() and then combine them afterwards. This also isn't built in at the moment, but would just be a matter of adding an additional stage to a task that, after the task is completed, streams all the frames of the captured video to the Terminal, or else combines them however you intend on storing them.

To answer the other questions: yes unless you specifically compress or downsample the video it will be maintained in full resolution. There's some customization and tweaking that can be done re: latency, turning on or off blosc compression, batching vs. not batching, but it is on the order of a few milliseconds at 1080p. can do a more official benchmark.