Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cut out objects with transparent background in the segmenter #47

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

W7CH
Copy link
Contributor

@W7CH W7CH commented Jul 31, 2024

This PR modifies the output of the segmenter process by cropping objects with transparent background. The previous implementation cropped objects with an entire region of interest from the original image, which often included unnecessary background and parts of other objects. The file affected is processing/segmenter/planktoscope/segmenter/__init__.py to update and add methods for object detection, cropping and saving.

Changes:

  • Define a method _save_object_image to save the object cutout with RGB channels along with the alpha channel (RGBA) for transparency
  • Convert the raw image to RGB mode in the method _open_and_apply_flat
  • Modify the method _stream to be compatible with RGBA mode and PNG file format of the object image
  • Define a method _crop_and_apply_mask that crops the raw image and mask to the bounding box of the detected object to ensure the cutout is as close as possible to the size of the object. This method adds an alpha channel to the cropped image, and sets the alpha channel to the segmentation mask. This will make areas outside the mask completely transparent and areas inside the mask opaque.
  • The nested function __augment_slice is no longer used within the method _slice_image
  • For each object extracted from the raw image, the cropped mask is applied as an alpha channel to create a transparent background. The resulting object image is saved as a PNG file because the JPEG format doesn't support transparency (it ignores the alpha channel info)

Testing:

These changes were tested in an external notebook environment with multiple images to ensure that each detected object is correctly isolated and saved in the expected format. The transparency was preserved in the output PNG files (see example).
replication of result

Instructions for testing this PR on the PlanktoScope

To test new changes to this PR on a PlanktoScope with internet access:

  1. Push your changes in one or more commits up to the feature/segmenter branch
  2. Wait for the CI checks/workflows on this PR to finish rebuilding the Docker container image for the segmenter. You should see a new entry at https://github.com/PlanktoScope/device-backend/pkgs/container/device-backend-processing-segmenter with a label (e.g. sha-55e23d1) corresponding to the latest commit (e.g. 55e23d1 ) which you had pushed to the branch. It should also have the label pr-47 attached.
  3. Go to the Cockpit Terminal of the PlanktoScope (usually this will be at http://planktoscope.local/admin/cockpit/system/terminal, or else you can access Cockpit from the PlanktoScope's landing page and then you can access the terminal from Cockpit's sidebar).
  4. Run the following command: forklift pallet switch github.com/PlanktoScope/pallet-standard@feature/segmentation-transparency (note: the URL given here corresponds to Bump device-pkgs to make segmentation background transparent pallet-standard#15, which deploys the latest pr-47 version of the Docker container image for the segmenter).
  5. Either reboot your PlanktoScope or run: sudo systemctl start forklift-apply. This command will disconnect Cockpit in your browser tab; if you wait ~10 seconds and refresh your browser tab, then you should be able to log in to Cockpit again.

Afterwards, the updated segmenter should be running. You can check its live-updating logs with Dozzle (usually the logs will be at http://planktoscope.local/admin/dozzle/show?name=apps_ps_backend_proc-segmenter-server-1, or else you can access the live-updating logs from a link in the Node-RED dashboard's "Administration" page).

@W7CH W7CH requested a review from tpollina August 1, 2024 15:36
@W7CH
Copy link
Contributor Author

W7CH commented Aug 5, 2024

A few examples of images segmented with transparent background:

Some points are not represented in the objects because they were not detected in the segmentation mask. This behavior had always existed, but segmentation with a region of interest made it possible to correct it.
00_25_56_357156_20_cutout
00_25_56_357156_9_cutout
00_25_54_141982_56_cutout
00_25_51_908646_13_cutout
00_25_49_701693_14_cutout
00_25_45_101719_103_cutout
00_25_45_101719_81_cutout
00_25_45_101719_28_cutout
00_25_42_755452_43_cutout
00_25_42_755452_2_cutout
00_25_40_432352_11_cutout
00_25_40_432352_4_cutout
00_25_38_251991_26_cutout
00_25_38_251991_18_cutout
00_25_35_964166_53_cutout
00_25_35_964166_49_cutout
00_25_35_964166_36_cutout
00_25_35_964166_8_cutout
00_25_35_964166_9_cutout
00_25_33_745790_63_cutout
00_25_33_745790_8_cutout
00_25_31_542437_34_cutout
00_25_31_542437_2_cutout
00_25_29_314502_38_cutout
00_25_29_314502_58_cutout
00_25_29_314502_10_cutout
00_25_27_130201_25_cutout
00_25_29_314502_36_cutout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant