A frame is captured by waggle-image-producer and is pushed into an exchange called image_pipeline in the local RMQ broker. Any processor, subscribing to the exchange, can receive images. The frame may be modified, by the process and put back in the exchange, wherein the routing-key is modified to signify manipulation. Whenever the frame is ready to be transferred to Beehive, it is put in the exchange with the routing-key exporter. The waggle-image-exporter service moves the frame to images queue. The frames in the images queue are asynchronously shipped (best-effort) to beehive by a shovel in NodeController.
The images are shoved into the RMQ images exchange on beehive, and the subscribers of the exchange recieve them. The frames are JPEGformat, with Meta-data stored in the EXIF tags. The meta-data can be acquired by unpacking the EXIF header.
Waggle 2.8.2 or greater supports EXIF tag feature. Former Waggle nodes with version 2.8.1 or less send images using the following format,
{'results':[{'timestamp':timestamp,'node_id':node_id,'cam_location':cam_location},], 'image':image}
To access raw JPEG image, use image key in the message. Use results key for the meta-data of the image.
The image pipeline is designed to accommodate re-publishing a image into the exchange for further manipulations on the image. User plugins use topics to subscribe and publish images. User plugins are also able to create topic as needed (Refer to the example 2 in the above figure). Topic name normally starts with alpabet and can contain numbers and some of special characters such as ., /, -, _, and so on. No use of capital alphbet is recommended.
There are reserved topics (i.e., routing keys) that plugins or applications must not use to publish,
- top is used to publish images captured from Waggle Top Camera
- bottom is used to publish images captured from Waggle Bottom Camera
- export is used to subscribe images that are being exported to Beehive
As user plugins begin receiving and publishing messages within image pipeline on Edge Processor, the plugins will need to interpret those messages correctly. Since Waggle software has been rapidly evolved to accommodate a raw image along with various types of information, there is a few message formats based on Waggle version. The following sub-sections will cover each format such that user plugins know how to handle messages from image pipeline.
NOTE: for Waggle 2.8.2 or greater, messages that are exported to outside of Edge Processor are a metadata-embedded JPEG image, and thus the following formats are only valid within image pipeline except Format 1; former Waggles less then 2.8.2 still use Format 1 shown below, even after message is exported
The message is JSON format plain text as follows,
{'results':[{'timestamp':timestamp,'node_id':node_id,'cam_location':cam_location},], 'image':image}
where timestamp is integer type epoch time when the image was captured, node_id is NodeController's id, cam_location is a string indicating source of the image. image is base64 encoded JPEG image. To get raw JPEG image,
#! /usr/bin/python3
import base64
# image is "b'CONTENTS'" so b' and ' need to be removed
jpg_image = base64.b64decode(image[2:-1])
Message content is an instance of a custom Python class Packet. A packet consists of 3 parts: meta data, result, and image. Meta data includes the followings,
node_id: node_idimage_width: width of the imageimage_height: height onf the imagedevice: device used to capture the imageproducer: name of the program that produced the imagedatetime: datetime when the image was captured (format is '%Y-%m-%d %H:%M:%S')
Result is an array that sequentially accumulates results produced by user plugins. Image is base64 encoded JPEG image. However, load function on the Packet class provides base64 decoding such that user plugin will not need to decode itself.
In this format, message actively uses RabbitMQ properties. Message body contains only image. Format of image is MJPG (Motion JPEG) and may need to be converted for image manipulations. Code snippet for the conversion is,
import numpy as np
import cv2
properties, image = get_message()
nparr_img = np.fromstring(image, np.uint8)
img = cv2.imdecode(nparr_img, cv2.IMREAD_COLOR)
Message headers (through RabbitMQ.properties.headers) store meta data of image. Meta data includes,
node_id: node_idimage_width: width of the imageimage_height: height of the imageimage_format: image format (default is MJPG)image_size: size of the image in byteimage_rotate: orientation of the image; rotate the image as much as the value to see the image in right orientationdevice: device used to capture the imageproducer: name of the program that produced the imagetimestamp: string formatted epoch time
User plugins are welcome to add more header properties such as results. However, when the message is sent to be exported, only pre-defined properties will be encoded using EXIF and included in the image. Pre-defined properties include processing_software as well as all the properties listed above.
Please refer to supported_libraries