test: 7.2.0 Analysis Snapshot Protocols #14569

DerekMaggio · 2024-02-29T16:59:18Z

Overview

Add protocols pulled from v7.2.0 pull requests for Analysis Snapshot testing. All work filed under RQA-2434

Test Plan

Add partial tip pickup protocols from Sanitti
Add ABR protocol from RABR-23
Run new protocols locally and verify results
Run in workflow dispatch action and verify success [link]

Changelog

Here are the protocols added and where I found them:

RQA-2098
- Flex_P1000_96_Gripper_TC_TM_HS_AnalysisError_GripperCollisionWithTips.json
fix(api): associate thermocycler with all slots it occupies #14491
- Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots.py
- Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py
- Flex_None_None_TC_2_16_verifyThermocyclerLoadedSlots.py
- Flex_None_None_TC_2_17_verifyThermocyclerLoadedSlots.py
- OT2_None_None_TC_2_14_VerifyThermocyclerLoadedSlots.py
- OT2_None_None_TC_2_15_VerifyThermocyclerLoadedSlots.py
- OT2_None_None_TC_2_16_VerifyThermocyclerLoadedSlots.py
- OT2_None_None_TC_2_17_VerifyThermocyclerLoadedSlots.py
fix(api): OT-2 trash bins in 2.16 get added to engine at the start of protocol #14475
- Flex_None_None_TC_2_16_AnalysisError_TrashBinAndThermocyclerConflict.py
- OT2_None_None_2_16_verifyDoesNotDeadlock.py
- OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin1.py
- OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin2.py
feat(api): add pipette outer dimensions when checking movement conflict #14547
- Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLid.py
feat(api): raise an error if pipette nozzles might collide with thermocycler lid clips #14522
- Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLidClips.py
Dispense functionality validation
- OT2_P300M_P20S_TC_HS_TM_2_15_dispense_changes.py
- OT2_P300M_P20S_TC_HS_TM_2_16_dispense_changes.py
- OT2_P300M_P20S_TC_HS_TM_2_17_dispense_changes.py
fix(api): Fix floating point errors and backwards incompatibility in volume validation #14253
- OT2_P300S_None_2_16_verifyNoFloatingPointErrorInPipetting.py

Review requests

There are a few pull requests that I had questions about. I will @ the people that worked on them to answer the questions

fix(api): home before retrieving gantry position for disengaged Z mounts in FLEX #14437
- @sfoster1, Can I use the protocol found in RABR-23 to get this to happen? Is there another way?
fix(robot-server): Fix error parsing persisted loadModule commands #14509
- @SyntaxColoring, can analysis capture this change or is this only relavant during actual protocol runtime?
fix(robot-server): append error to analysis error list in case an internal error occurred #14510
- @TamarZanzouri or @SyntaxColoring, can I raise a generic python exception inside the protocol to trigger this functionality?
refactor(api): Save and load tip length calibrations from tiprack uri #14512
- @Laura-Danielle or @SyntaxColoring, Iant to validate my logic. This not a good canidate for snapshot testing because you have to run LPC and Calibration which is not taken into account during analysis. Can you confirm?

Risk assessment

None

codecov · 2024-02-29T16:59:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.50%. Comparing base (b533692) to head (5f0959b).

Additional details and impacted files

@@           Coverage Diff           @@
##             edge   #14569   +/-   ##
=======================================
  Coverage   67.50%   67.50%           
=======================================
  Files        2514     2514           
  Lines       72375    72375           
  Branches     9317     9317           
=======================================
  Hits        48856    48856           
  Misses      21314    21314           
  Partials     2205     2205

Flag	Coverage Δ
app	`63.99% <ø> (ø)`
components	`48.40% <ø> (ø)`
labware-library	`41.11% <ø> (ø)`
protocol-designer	`37.90% <ø> (ø)`
step-generation	`86.88% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

sfoster1 · 2024-03-04T18:01:52Z

I'd expect you can use RABR-23 to test #14437 yeah

SyntaxColoring · 2024-03-04T18:21:44Z

fix(robot-server): Fix error parsing persisted loadModule commands #14509

@SyntaxColoring, can analysis capture this change or is this only relavant during actual protocol runtime?

Analysis can't capture it. It's only relevant when robot-server retrieves details about runs and analyses stored from prior boots.

fix(robot-server): append error to analysis error list in case an internal error occurred #14510

@TamarZanzouri or @SyntaxColoring, can I raise a generic python exception inside the protocol to trigger this functionality?

Unfortunately no. The hole that that PR plugged is only for exceptions raised by certain Opentrons internal code.

refactor(api): Save and load tip length calibrations from tiprack uri #14512

@Laura-Danielle or @SyntaxColoring, Iant to validate my logic. This not a good canidate for snapshot testing because you have to run LPC and Calibration which is not taken into account during analysis. Can you confirm?

Yep, I agree. That fix is for robot runtime behavior, which we can't cover with these analysis tests. Theoretically, we could do a similar snapshot thing for robot runtime behavior. I guess that would be g-code-testing.

SyntaxColoring

Here are some possible apiLevel mistakes, but other than that, this looks great to me. Thank you!

app-testing/files/protocols/py/Flex_None_None_TC_2_16_verifyThermocyclerLoadedSlots.py

app-testing/files/protocols/py/Flex_None_None_TC_2_17_verifyThermocyclerLoadedSlots.py

app-testing/files/protocols/py/Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py

y3rsh

One step to add in the README otherwise great.

y3rsh · 2024-03-05T18:28:28Z

app-testing/example.env

+# Not running this file because the error handling to catch that 
+# the flex is only supported on API versions 2.15 and above, does not exist
+# Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots


Adding back into the test with this task https://opentrons.atlassian.net/browse/RDEVOPS-71

y3rsh · 2024-03-05T18:29:17Z

app-testing/files/protocols/py/Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py

@@ -0,0 +1,14 @@
+# Pulled from: https://github.com/Opentrons/opentrons/pull/14491


❤️ these comments.

y3rsh · 2024-03-05T18:42:07Z

app-testing/README.md

+
+#### Running Analysis Snapshot Tests Locally
+
+To run analysis snapshot tests locally, you run 


Add the step to build the docker container since it must be built before running the test

TARGET="<target-branch-or-tag>" make build-opentrons-analysis

All line number changes

@sfoster1

Add protocols pulled from v7.2.0 pull requests for Analysis Snapshot testing. All work filed under [RQA-2434](https://opentrons.atlassian.net/browse/RQA-2434) - [x] Add partial tip pickup protocols from Sanitti - [x] Add ABR protocol from [RABR-23](https://opentrons.atlassian.net/browse/RABR-23) - [x] Run new protocols locally and verify results - [x] Run in workflow dispatch action and verify success [[link]](https://github.com/Opentrons/opentrons/actions/runs/8160792870/job/22308167177) Here are the protocols added and where I found them: - [RQA-2098](https://opentrons.atlassian.net/browse/RQA-2098) - Flex_P1000_96_Gripper_TC_TM_HS_AnalysisError_GripperCollisionWithTips.json - #14491 - Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots.py - Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py - Flex_None_None_TC_2_16_verifyThermocyclerLoadedSlots.py - Flex_None_None_TC_2_17_verifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_14_VerifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_15_VerifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_16_VerifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_17_VerifyThermocyclerLoadedSlots.py - #14475 - Flex_None_None_TC_2_16_AnalysisError_TrashBinAndThermocyclerConflict.py - OT2_None_None_2_16_verifyDoesNotDeadlock.py - OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin1.py - OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin2.py - #14547 - Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLid.py - #14522 - Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLidClips.py - Dispense functionality validation - OT2_P300M_P20S_TC_HS_TM_2_15_dispense_changes.py - OT2_P300M_P20S_TC_HS_TM_2_16_dispense_changes.py - OT2_P300M_P20S_TC_HS_TM_2_17_dispense_changes.py - #14253 - OT2_P300S_None_2_16_verifyNoFloatingPointErrorInPipetting.py There are a few pull requests that I had questions about. I will @ the people that worked on them to answer the questions - #14437 - @sfoster1, Can I use the protocol found in [RABR-23](https://opentrons.atlassian.net/browse/RABR-23) to get this to happen? Is there another way? - #14509 - @SyntaxColoring, can analysis capture this change or is this only relavant during actual protocol runtime? - #14510 - @TamarZanzouri or @SyntaxColoring, can I raise a generic python exception inside the protocol to trigger this functionality? - #14512 - @Laura-Danielle or @SyntaxColoring, Iant to validate my logic. This not a good canidate for snapshot testing because you have to run LPC and Calibration which is not taken into account during analysis. Can you confirm? None [RQA-2434]: https://opentrons.atlassian.net/browse/RQA-2434?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [RQA-2098]: https://opentrons.atlassian.net/browse/RQA-2098?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [RABR-23]: https://opentrons.atlassian.net/browse/RABR-23?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

@sfoster1

Add protocols pulled from v7.2.0 pull requests for Analysis Snapshot testing. All work filed under [RQA-2434](https://opentrons.atlassian.net/browse/RQA-2434) - [x] Add partial tip pickup protocols from Sanitti - [x] Add ABR protocol from [RABR-23](https://opentrons.atlassian.net/browse/RABR-23) - [x] Run new protocols locally and verify results - [x] Run in workflow dispatch action and verify success [[link]](https://github.com/Opentrons/opentrons/actions/runs/8160792870/job/22308167177) Here are the protocols added and where I found them: - [RQA-2098](https://opentrons.atlassian.net/browse/RQA-2098) - Flex_P1000_96_Gripper_TC_TM_HS_AnalysisError_GripperCollisionWithTips.json - #14491 - Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots.py - Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py - Flex_None_None_TC_2_16_verifyThermocyclerLoadedSlots.py - Flex_None_None_TC_2_17_verifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_14_VerifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_15_VerifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_16_VerifyThermocyclerLoadedSlots.py - OT2_None_None_TC_2_17_VerifyThermocyclerLoadedSlots.py - #14475 - Flex_None_None_TC_2_16_AnalysisError_TrashBinAndThermocyclerConflict.py - OT2_None_None_2_16_verifyDoesNotDeadlock.py - OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin1.py - OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin2.py - #14547 - Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLid.py - #14522 - Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLidClips.py - Dispense functionality validation - OT2_P300M_P20S_TC_HS_TM_2_15_dispense_changes.py - OT2_P300M_P20S_TC_HS_TM_2_16_dispense_changes.py - OT2_P300M_P20S_TC_HS_TM_2_17_dispense_changes.py - #14253 - OT2_P300S_None_2_16_verifyNoFloatingPointErrorInPipetting.py There are a few pull requests that I had questions about. I will @ the people that worked on them to answer the questions - #14437 - @sfoster1, Can I use the protocol found in [RABR-23](https://opentrons.atlassian.net/browse/RABR-23) to get this to happen? Is there another way? - #14509 - @SyntaxColoring, can analysis capture this change or is this only relavant during actual protocol runtime? - #14510 - @TamarZanzouri or @SyntaxColoring, can I raise a generic python exception inside the protocol to trigger this functionality? - #14512 - @Laura-Danielle or @SyntaxColoring, Iant to validate my logic. This not a good canidate for snapshot testing because you have to run LPC and Calibration which is not taken into account during analysis. Can you confirm? None [RQA-2434]: https://opentrons.atlassian.net/browse/RQA-2434?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [RQA-2098]: https://opentrons.atlassian.net/browse/RQA-2098?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [RABR-23]: https://opentrons.atlassian.net/browse/RABR-23?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

DerekMaggio changed the title ~~test: add test cases around dispense logic over different api versions~~ test: 7.2.0 Analysis Snapshot Protocols Feb 29, 2024

DerekMaggio force-pushed the 7.2.0-analysis-snapshot-protocols branch from cc89cc8 to 56c0390 Compare March 4, 2024 16:47

DerekMaggio requested a review from y3rsh March 4, 2024 16:47

DerekMaggio marked this pull request as ready for review March 4, 2024 17:21

DerekMaggio requested review from a team as code owners March 4, 2024 17:21

SyntaxColoring approved these changes Mar 4, 2024

View reviewed changes

y3rsh approved these changes Mar 5, 2024

View reviewed changes

DerekMaggio added the DO NOT MERGE Indicates a PR should not be merged, even if there's a shiny green merge button available label Mar 5, 2024

y3rsh changed the base branch from chore_release-7.2.0 to edge March 6, 2024 19:13

y3rsh removed the DO NOT MERGE Indicates a PR should not be merged, even if there's a shiny green merge button available label Mar 7, 2024

DerekMaggio added 14 commits March 7, 2024 11:58

test: add test cases around dispense logic over different api versions

a15c7c9

add more protocols

1dbc4de

Remove smoke test comment

beaed06

Formatting

defda79

Add analysis results entries

1170992

update existing snapshots

b69c120

All line number changes

add new snapshots

da25e25

formatting

4dc5149

Update example env

c86fd36

add more protocols

57f13a1

formatting

ddb1a5d

Update README

9e4f331

Forgot to add update

e66a7f5

Fix formatting

5f0959b

DerekMaggio force-pushed the 7.2.0-analysis-snapshot-protocols branch from 257a8f7 to 5f0959b Compare March 7, 2024 19:59

DerekMaggio merged commit 0b40719 into edge Mar 7, 2024
32 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: 7.2.0 Analysis Snapshot Protocols #14569

test: 7.2.0 Analysis Snapshot Protocols #14569

DerekMaggio commented Feb 29, 2024 •

edited

Loading

codecov bot commented Feb 29, 2024 •

edited

Loading

sfoster1 commented Mar 4, 2024 •

edited by jira bot

Loading

SyntaxColoring commented Mar 4, 2024

SyntaxColoring left a comment

y3rsh left a comment

y3rsh Mar 5, 2024

y3rsh Mar 5, 2024

y3rsh Mar 5, 2024

		@@ -0,0 +1,14 @@
		# Pulled from: https://github.com/Opentrons/opentrons/pull/14491


		#### Running Analysis Snapshot Tests Locally

		To run analysis snapshot tests locally, you run

test: 7.2.0 Analysis Snapshot Protocols #14569

test: 7.2.0 Analysis Snapshot Protocols #14569

Conversation

DerekMaggio commented Feb 29, 2024 • edited Loading

Overview

Test Plan

Changelog

Review requests

Risk assessment

codecov bot commented Feb 29, 2024 • edited Loading

Codecov Report

sfoster1 commented Mar 4, 2024 • edited by jira bot Loading

SyntaxColoring commented Mar 4, 2024

SyntaxColoring left a comment

Choose a reason for hiding this comment

y3rsh left a comment

Choose a reason for hiding this comment

y3rsh Mar 5, 2024

Choose a reason for hiding this comment

y3rsh Mar 5, 2024

Choose a reason for hiding this comment

y3rsh Mar 5, 2024

Choose a reason for hiding this comment

DerekMaggio commented Feb 29, 2024 •

edited

Loading

codecov bot commented Feb 29, 2024 •

edited

Loading

sfoster1 commented Mar 4, 2024 •

edited by jira bot

Loading