MarkDuplicates strategy of flow based reads that looks only at the qualities close to the end of the read #1942

ilyasoifer · 2024-02-10T17:55:49Z

Description

We found that when the duplication rate is very high, we sometimes also get variability at the start position (although usually it is more precise than the end position). In this version we added a parameter that also allows for an uncertainty in the read start position to be counted as duplicate. We introduced a parameter UNPAIRED_START_UNCERTAINTY in flow mode.
When duplication rate is very high for large cluster of reads when all reads (start,end) are (A,B) we will sometimes get read with (start,end) at (A-1,B). In the default configuration, we will select the read with the best quality sum over all the reads. However, we found that if it is desirable to select read that starts and ends on (A,B) it is better to only look at the qualities close to the end. We thus introduced a new strategy: FLOW_END_QUALITY_STRATEGY.

Checklist (never delete this)

Never delete this, it is our record that procedure was followed. If you find that for whatever reason one of the checklist points doesn't apply to your PR, you can leave it unchecked but please add an explanation below.

Content

Added or modified tests to cover changes and any new functionality
Edited the README / documentation (if applicable)
All tests passing on github actions

Review

Final thumbs-up from reviewer
Rebase, squash and reword as applicable

For more detailed guidelines, see https://github.com/broadinstitute/picard/wiki/Guidelines-for-pull-requests

…of the read

.gitignore

meganshand · 2024-02-12T14:58:12Z

src/main/java/picard/sam/markduplicates/MarkDuplicatesForFlowArgumentCollection.java

+    public enum FLOW_DUPLICATE_SELECTION_STRATEGY {
+        FLOW_QUALITY_SUM_STRATEGY,
+        FLOW_END_QUALITY_STRATEGY
+    }
    @Argument(doc = "enable parameters and behavior specific to flow based reads.", optional = true)
    public boolean FLOW_MODE = false;

    @Argument(doc = "Use specific quality summing strategy for flow based reads. The strategy ensures that the same " +


Is this doc still up to date with the new enum?

src/main/java/picard/sam/markduplicates/MarkDuplicatesForFlowHelper.java

meganshand · 2024-02-12T16:02:43Z

src/main/java/picard/sam/markduplicates/MarkDuplicatesForFlowHelper.java

@@ -236,8 +273,8 @@ static protected int getFlowSumOfBaseQualities(final SAMRecord rec, final int th
        final byte[]  bases = rec.getReadBases();

        // create iteration range and direction
-        final int startingOffset = !rec.getReadNegativeStrandFlag() ? 0 : bases.length;
-        final int endOffset = !rec.getReadNegativeStrandFlag() ? bases.length : 0;
+        final int startingOffset = !rec.getReadNegativeStrandFlag() ? 0 : bases.length - 1;


was this previously a bug? I don't see how this change is related to allowing a range in the start position or changing the quality score calculation.

Was a bug indeed

meganshand · 2024-02-12T16:14:41Z

src/main/java/picard/sam/markduplicates/MarkDuplicatesForFlowHelper.java

@@ -59,6 +60,7 @@ public class MarkDuplicatesForFlowHelper implements MarkDuplicatesHelper {
    public static final char[]  CLIPPING_TAG_CONTAINS_AQ = {'A', 'Q'};
    public static final char[]  CLIPPING_TAG_CONTAINS_QZ = {'Q', 'Z'};

+    public static final int DIST_FROM_END = 10;


It's worth mentioning that the distance from the end is hard coded as 10 bases for the FLOW_END_QUALITY_STRATEGY

Mentioned in the parameter description. Is this OK?

Yup, perfect

ilyasoifer

@meganshand - comments addressed, please take another look

meganshand

👍

meganshand · 2024-02-14T21:00:06Z

src/main/java/picard/sam/markduplicates/MarkDuplicatesForFlowHelper.java

@@ -59,6 +60,7 @@ public class MarkDuplicatesForFlowHelper implements MarkDuplicatesHelper {
    public static final char[]  CLIPPING_TAG_CONTAINS_AQ = {'A', 'Q'};
    public static final char[]  CLIPPING_TAG_CONTAINS_QZ = {'Q', 'Z'};

+    public static final int DIST_FROM_END = 10;


Yup, perfect

ilyasoifer added 15 commits February 10, 2024 16:42

Fixed a bug in getFlowSumOfBaseQualities

1c27af5

uncertain start position parameter in MarkDuplicates

d996754

Modified algorithm to determine the best read

e28bbdf

Modified algorithm to determine the best read

3375618

Oops

c93fc33

Oops2

2861c0a

WIP

2316ec6

Added a new duplicate selection strategy that looks close to the end …

0b58cb3

…of the read

Added a new duplicate selection strategy that looks close to the end …

a735022

…of the read

Test fixed and strategy test added

527fe48

More tests added

0e1c2b7

Bug fixed

be14d43

added .vscode

0757ed5

Test fixed

0b0e6ca

Reverted whitespace changes in MarkDuplicates

0ce00d1

meganshand reviewed Feb 12, 2024

View reviewed changes

PR comments

80f0d4c

ilyasoifer commented Feb 14, 2024

View reviewed changes

ilyasoifer requested review from lbergelson and meganshand February 14, 2024 20:24

meganshand approved these changes Feb 14, 2024

View reviewed changes

kockan merged commit c026f94 into broadinstitute:master Feb 14, 2024
6 checks passed

ilyasoifer deleted the ilyasoifer/md.improvements.flow branch June 1, 2024 12:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MarkDuplicates strategy of flow based reads that looks only at the qualities close to the end of the read #1942

MarkDuplicates strategy of flow based reads that looks only at the qualities close to the end of the read #1942

ilyasoifer commented Feb 10, 2024 •

edited

Loading

meganshand Feb 12, 2024

ilyasoifer Feb 14, 2024

meganshand Feb 12, 2024

ilyasoifer Feb 14, 2024

meganshand Feb 12, 2024

ilyasoifer Feb 14, 2024

meganshand Feb 14, 2024

ilyasoifer left a comment

meganshand left a comment

meganshand Feb 14, 2024

MarkDuplicates strategy of flow based reads that looks only at the qualities close to the end of the read #1942

MarkDuplicates strategy of flow based reads that looks only at the qualities close to the end of the read #1942

Conversation

ilyasoifer commented Feb 10, 2024 • edited Loading

Description

Checklist (never delete this)

Content

Review

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyasoifer left a comment

Choose a reason for hiding this comment

meganshand left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyasoifer commented Feb 10, 2024 •

edited

Loading