Feature/results in memory #193

meganrm · 2023-09-21T23:28:34Z

Problem

In working on the variable sized ingredient setting we realized how many different places and ways the packed data was being stored. See this wiki

Solution

We created a new class PackedObjects. At first I didn't store the ingredient in the object, but it ended up making more sense to do it (I was calling "get_ingredient_by_name" a lot).

Other changes:

Delete unused functions
Move functions into Environment if all the code was related to that class
change compNum to compartment_id

with @mogres

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Steps to Verify:

pack -r '/Users/meganr/dev/mesoscope/cellpack/cellpack/tests/recipes/v2/test_variable_size.json'

Screenshots (optional):

* update simularium helper to use radius in molecules * update max and min radii * add size options to ingredient initialization * refactor distribution value extraction

…cellpack into feature/results-in-memory

…ture/results-in-memory

github-actions · 2023-09-21T23:31:49Z

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name	Encapsulating radius	Average number packed
ext_A	25	236.0

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key	Pairwise distance distribution
ext_A

meganrm · 2023-09-22T21:36:04Z

cellpack/autopack/plotly_result.py

@@ -59,7 +59,7 @@ def add_square(self, side_length, pos, rotMat, color, opacity=1):
        )

    def add_ingredient_positions(self, env):
-        for pos, rot, ingr, ptInd in env.molecules:
+        for pos, rot, ingr, ptInd, radius in env.molecules:


test and fix 2D plotly results

3d sphere tree looks questionable

* formatting using black

mogres

This was breaking image export so I made some changes to fix that. Looks good otherwise and a super useful update going forward!

cellpack/autopack/ingredient/Ingredient.py

mogres · 2023-10-06T17:33:21Z

cellpack/autopack/ingredient/Ingredient.py

@@ -1339,11 +1242,12 @@ def get_new_pos(self, ingr, pos, rot, positions_to_adjust):
        return self.transformPoints(pos, rot, positions_to_adjust)

    def check_against_one_packed_ingr(self, index, level, search_tree):
-        overlapped_ingr = self.env.rIngr[index]
+        packed_ingredient = self.env.packed_objects.get()[index]


Can maybe get ingredient here directly?

mogres · 2023-10-06T17:57:04Z

cellpack/autopack/ingredient/single_sphere.py

@@ -96,6 +98,28 @@ def __init__(
        self.encapsulating_radius = radius
        self.min_radius = radius

+    @staticmethod
+    def create_voxelization(
+        self, image_data, bounding_box, voxel_size, image_size, position, rotation, radius, hollow=None, mesh_store=None,


Need to test if this still works

Suggested change

self, image_data, bounding_box, voxel_size, image_size, position, rotation, radius, hollow=None, mesh_store=None,

image_data, bounding_box, voxel_size, image_size, position, rotation, radius, hollow=None, mesh_store=None,

I reverted this, we did this because we weren't originally storing the ingredient

cellpack/autopack/utils.py

cellpack/tests/packing-configs/test_variable_count_config.json

mogres · 2023-10-06T18:06:33Z

cellpack/autopack/Graphics.py

@@ -1302,7 +1302,7 @@ def displayCytoplasmIngredients(self):
        else:
            return
        if self.doSpheres:
-            for pos, rot, ingr, ptInd in self.histo.molecules:
+            for pos, rot, ingr, ptInd, _ in self.histo.molecules:


Are we using Graphics.py for anything? These functions are still using histo.molecules -- should get rid of them at some point.

we're not, I can delete it in a separate simple PR

mogres · 2023-10-06T22:47:32Z

cellpack/autopack/Environment.py

-                nbFreePoints = self.onePrevIngredient(
-                    i, mingrs, distance, nbFreePoints, organelle.molecules
-                )
+        # TODO: refactor this to work with new placed_objects data structure


Do we ever have object already packed while building a grid? This could also be removed if unused.

codecov-commenter · 2023-10-06T23:45:55Z

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (2c9db6d) 98.52% compared to head (d35738d) 98.52%.
Report is 4 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #193   +/-   ##
=======================================
  Coverage   98.52%   98.52%           
=======================================
  Files          16       16           
  Lines         476      476           
=======================================
  Hits          469      469           
  Misses          7        7

Files	Coverage Δ
cellpack/tests/test_ingredient.py	`100.00% <ø> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Co-authored-by: Saurabh Mogre <[email protected]>

…ope/cellpack into feature/results-in-memory

* use object instance instead of class to create voxelization * formatting

mogres · 2023-10-10T23:47:26Z

I also made a couple of minor changes to get partner packing to work with variable size since I needed it to create some synthetic data.

…ope/cellpack into feature/results-in-memory

Co-authored-by: Saurabh Mogre <[email protected]>

…ope/cellpack into feature/results-in-memory

mogres

New updates look good!

Image export is working with compartments
2D plots are working
Sphere tree packing is working

mogres · 2023-10-12T18:48:49Z

Closes #178

mogres and others added 16 commits August 16, 2023 15:34

Working variable count distributions

711bd8e

* add radius to molecules array

8a00bbd

* update simularium helper to use radius in molecules * update max and min radii * add size options to ingredient initialization * refactor distribution value extraction

Test recipe and config for variable size

c5aa16b

add radius to ingredient array dict

de95dfe

add test variable size recipe

316fee7

create packed objects class

945eccb

Merge branch 'feature/variable_size' of https://github.com/mesoscope/…

283015e

…cellpack into feature/results-in-memory

removing .molecules

7db8bec

pack variable size ingredient successfully

8eb3190

Merge branch 'main' of https://github.com/mesoscope/cellpack into fea…

a23bdde

…ture/results-in-memory

pack sphere_tree ingredient

fb3dcb9

DRY code

4c47c8c

store ingredient in packed object

3b97426

change to compartment_id

b665e77

cleanup

7836419

add second example

5541d22

meganrm changed the base branch from feature/variable_size to main September 21, 2023 23:30

meganrm requested review from rugeli and mogres September 21, 2023 23:31

meganrm changed the base branch from main to feature/variable_size September 22, 2023 21:26

meganrm changed the base branch from feature/variable_size to main September 22, 2023 21:30

meganrm commented Sep 22, 2023

View reviewed changes

* fixed broken image export

d5207c5

* formatting using black

mogres reviewed Oct 6, 2023

View reviewed changes

mogres and others added 4 commits October 9, 2023 09:35

fix flake8 linting errors

aae4e49

Update cellpack/autopack/ingredient/Ingredient.py

d4084c4

Co-authored-by: Saurabh Mogre <[email protected]>

Update cellpack/autopack/ingredient/Ingredient.py

ca1464d

Co-authored-by: Saurabh Mogre <[email protected]>

Update cellpack/tests/packing-configs/test_variable_count_config.json

41a300e

Co-authored-by: Saurabh Mogre <[email protected]>

meganrm and others added 10 commits October 9, 2023 12:19

Update cellpack/autopack/ingredient/Ingredient.py

9d3e0cd

Co-authored-by: Saurabh Mogre <[email protected]>

Update cellpack/autopack/ingredient/Ingredient.py

75c870c

Co-authored-by: Saurabh Mogre <[email protected]>

Update cellpack/autopack/utils.py

eadb2e6

Co-authored-by: Saurabh Mogre <[email protected]>

get ingredient directly

862d2e5

revert change to static method

c0741ba

Merge branch 'feature/results-in-memory' of https://github.com/mesosc…

67b59e3

…ope/cellpack into feature/results-in-memory

fix test

ec51ab4

* use kwargs to overload voxelization function

d480893

* use object instance instead of class to create voxelization * formatting

fix tests

9065f17

removed afvi from get_partners which was erroring out

e9dc5b6

mogres and others added 10 commits October 11, 2023 09:58

remove afvi argument causing partner packing to break

4be3fed

add get methods for packed objects

ee214e3

Merge branch 'feature/results-in-memory' of https://github.com/mesosc…

6430603

…ope/cellpack into feature/results-in-memory

move when we store compartments in packed_objects

1605221

Co-authored-by: Saurabh Mogre <[email protected]>

change radius

4a887de

use ingredient stored for compartment

25d9839

fix linting

fe89fb7

pack sphere trees successfully and plot in 2D

c975872

Merge branch 'feature/results-in-memory' of https://github.com/mesosc…

5d9ecae

…ope/cellpack into feature/results-in-memory

format fixed

d35738d

mogres approved these changes Oct 12, 2023

View reviewed changes

meganrm merged commit aae7ed1 into main Oct 13, 2023
7 checks passed

meganrm deleted the feature/results-in-memory branch October 13, 2023 21:38

mogres mentioned this pull request Oct 20, 2023

Create an ingredient type/feature that allows for size distribution #178

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/results in memory #193

Feature/results in memory #193

meganrm commented Sep 21, 2023 •

edited

Loading

github-actions bot commented Sep 21, 2023 •

edited

Loading

meganrm Sep 22, 2023 •

edited

Loading

meganrm Oct 11, 2023 •

edited

Loading

mogres left a comment

mogres Oct 6, 2023

mogres Oct 6, 2023

meganrm Oct 9, 2023

mogres Oct 6, 2023

meganrm Oct 9, 2023

mogres Oct 6, 2023

codecov-commenter commented Oct 6, 2023 •

edited

Loading

mogres commented Oct 10, 2023

mogres left a comment

mogres commented Oct 12, 2023

	self, image_data, bounding_box, voxel_size, image_size, position, rotation, radius, hollow=None, mesh_store=None,
	image_data, bounding_box, voxel_size, image_size, position, rotation, radius, hollow=None, mesh_store=None,

Feature/results in memory #193

Feature/results in memory #193

Conversation

meganrm commented Sep 21, 2023 • edited Loading

Problem

Solution

Type of change

Steps to Verify:

Screenshots (optional):

github-actions bot commented Sep 21, 2023 • edited Loading

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Packing image

Distance analysis

meganrm Sep 22, 2023 • edited Loading

Choose a reason for hiding this comment

meganrm Oct 11, 2023 • edited Loading

Choose a reason for hiding this comment

mogres left a comment

Choose a reason for hiding this comment

mogres Oct 6, 2023

Choose a reason for hiding this comment

mogres Oct 6, 2023

Choose a reason for hiding this comment

meganrm Oct 9, 2023

Choose a reason for hiding this comment

mogres Oct 6, 2023

Choose a reason for hiding this comment

meganrm Oct 9, 2023

Choose a reason for hiding this comment

mogres Oct 6, 2023

Choose a reason for hiding this comment

codecov-commenter commented Oct 6, 2023 • edited Loading

Codecov Report

mogres commented Oct 10, 2023

mogres left a comment

Choose a reason for hiding this comment

mogres commented Oct 12, 2023

meganrm commented Sep 21, 2023 •

edited

Loading

github-actions bot commented Sep 21, 2023 •

edited

Loading

meganrm Sep 22, 2023 •

edited

Loading

meganrm Oct 11, 2023 •

edited

Loading

codecov-commenter commented Oct 6, 2023 •

edited

Loading