Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

First pass and correct T2w segmentations #15

Open
jcohenadad opened this issue Mar 14, 2024 · 30 comments
Open

First pass and correct T2w segmentations #15

jcohenadad opened this issue Mar 14, 2024 · 30 comments

Comments

@jcohenadad
Copy link
Member

jcohenadad commented Mar 14, 2024 •
edited
Loading

@Kaonashi22 While waiting for the vertebral labeling #9 I suggest to:

  • run the pipeline on all subjects
  • visually QC the segmentation of the T2w
  • manually correct the segmentation when needed using https://github.com/spinalcordtoolbox/manual-correction. Manual correction should only be done for levels that will be used for morphometric calculations (C2-T5 if I remember correctly)
  • save the manually corrected segmentations under the derivatives/labels folder inside the source database. Alternatively, we could also upload the manual corrections as part of a release on this repository (to make sure to track the modifications, to make sure it is not lost, and to facilitate collaboration-- eg: for me to try running the pipeline with your manual corrections).
@jcohenadad jcohenadad assigned jcohenadad and unassigned jcohenadad Mar 14, 2024
@Kaonashi22
Copy link

Kaonashi22 commented Mar 16, 2024 via email

@jcohenadad
Copy link
Member Author

@Kaonashi22 let me know when you have corrected 2-3 segmentations. At this point, you can send me the derivatives folder, and I can modify the processing script so that it accounts for these manual corrections. That way we can work in parallel (ie: you correcting the seg, and me updating the analysis script)

@Kaonashi22
Copy link

Kaonashi22 commented Mar 20, 2024 via email

@Kaonashi22
Copy link

Overall, the segmentations of T2-w images are accurate. Sometimes, two or three voxels are missing on some slices; how precise the segmentation should be? Is it worth correcting manually these masks?
Also, the lowest slice of the T2 volume usually has a poor signal; would you recommend discarding the segmentation at this level?

@Kaonashi22
Copy link

Kaonashi22 commented Mar 22, 2024 •
edited
Loading

I attached the derivatives folder with the masks manually corrected.

After running the pipeline on all subjects, the script exited with errors at different steps after the segmentation of T2w images; here are some log files.
err.batch_processing_sub-CG176.log
err.batch_processing_sub-DEV148Sujet01.log
err.batch_processing_sub-DEV203Sujet08.log
err.batch_processing_sub-RC194.log

derivatives.zip

@jcohenadad
Copy link
Member Author

Overall, the segmentations of T2-w images are accurate. Sometimes, two or three voxels are missing on some slices; how precise the segmentation should be? Is it worth correcting manually these masks?

It depends how precise you want the results to be. For example, if a slice is missing 2-3 pixels, and you average the CSA across, let's say, 20 slices, assuming a CSA of 80 mm2 at a resolution of 0.8mm2, 3 pixels correspond to 1.92 mm2, which represents 2.4% of the CSA computed on a single slice, or 0.14% of the CSA computed over 20 slices (so quite negligible, I would say).

 Also, the lowest slice of the T2 volume usually has a poor signal; would you recommend discarding the segmentation at this level?

It depends how important the lowest slice is for your analysis (eg: are you considering it? if not, not important)

@jcohenadad jcohenadad mentioned this issue Apr 4, 2024
Closed
@jcohenadad
Copy link
Member Author

About the issues:

  • err.batch_processing_sub-CG176.log: error during registration of T2w to template. Could you send me this subjects?
  • err.batch_processing_sub-DEV148Sujet01.log: The script is looking for the file warp_template2T1w.nii.gz, but it does not exist. Strangely, the latest version of the script is looking for a different file name (../anat/warp_template2anat.nii.gz):

    spine-park/batch_processing.sh

    Lines 253 to 259 in 8e541e9

    # Register template->dwi (using template-T1w as initial transformation)
    sct_register_multimodal -i $SCT_DIR/data/PAM50/template/PAM50_t1.nii.gz \
    -iseg $SCT_DIR/data/PAM50/template/PAM50_cord.nii.gz \
    -d ${file_dwi_mean}.nii.gz -dseg ${file_dwi_seg}.nii.gz \
    -param step=1,type=seg,algo=centermass:step=2,type=im,algo=bsplinesyn,metric=CC,slicewise=1,iter=3,gradStep=0.5 \
    -initwarp ../anat/warp_template2anat.nii.gz -initwarpinv ../anat/warp_anat2template.nii.gz \
    -qc "${PATH_QC}"

    What version of the script did you run? If you send me this subject I can also try to reproduce.
  • err.batch_processing_sub-DEV203Sujet08.log: Error generated during QC report (portalocker.exceptions.LockException: [Errno 5] Input/output error). I am wondering if this is related to the parallel writing (due to multiple jobs) on a file on a mounted disk. Could you send me this image as well so I can try to reproduce? Tagging @joshuacwnewton so he is aware
  • err.batch_processing_sub-RC194.log: We get an OSError: [Errno 9] Bad file descriptor. I'm not sure how to interpret this. Could you also send me this subjects?

@joshuacwnewton

This comment was marked as off-topic.

Sign in to view
@jcohenadad

This comment was marked as off-topic.

Sign in to view
@joshuacwnewton

This comment was marked as off-topic.

Sign in to view
@jcohenadad

This comment was marked as off-topic.

Sign in to view
@joshuacwnewton joshuacwnewton mentioned this issue Apr 5, 2024
Closed
@joshuacwnewton
Copy link
Member

joshuacwnewton commented Apr 5, 2024 •
edited
Loading

I've summarized our QC-related discussion in a new issue on the SCT repo (spinalcordtoolbox/spinalcordtoolbox#4423), just so that it doesn't distract from the spine-park related discussion in this issue. :)

@Kaonashi22
Copy link

Thanks @jcohenadad, I'll send you the images by email
For sub-DEV148Sujet01, I was indeed using on older version of the script (1b378e0)

@Kaonashi22
Copy link

Kaonashi22 commented Apr 5, 2024 via email

@jcohenadad
Copy link
Member Author

@Kaonashi22 I've ran the script version 78129ca on the data you sent me (incl. those that produced the errors #15 (comment)) and I did not observe any error.

Terminal output 
julien-macbook:~/temp/Lydia/results_20240405_125118/log $ ls -1
batch_processing_sub-BB277.log
batch_processing_sub-BJ170.log
batch_processing_sub-CG176.log
batch_processing_sub-DEV148Sujet01.log
batch_processing_sub-DEV203Sujet08.log
batch_processing_sub-DEV206Sujet10.log
batch_processing_sub-LC164.log
batch_processing_sub-LM166.log
batch_processing_sub-RC194.log

Therefore, I suspect the issues you've observed were caused by parallel writing on the locked QC file. One way to overcome this is to use the flag -jobs 1. Can you please try to see if it solves the issue?

@Kaonashi22
Copy link

Kaonashi22 commented Apr 8, 2024 via email

@jcohenadad
Copy link
Member Author

The number refers to a git SHA that points to a state of the repository. Here is the link to that version of the repository, that includes the file: https://github.com/sct-pipeline/spine-park/tree/78129ca6e3fe90f39b20af565a5e1918d2f6754e

But what you should do instead of manually downloading the file from this repository, is go to your local repository and run:

git pull

Then you can make sure you are running the proper version by running:

git log --pretty=oneline

which indicates the version (top item in the list):

julien-macbook:~/code/spine-park $ git log --pretty=oneline
78129ca6e3fe90f39b20af565a5e1918d2f6754e (HEAD -> main, origin/main, origin/HEAD) Added .gitignore  <--- THIS ONE
9bfb14077c49237b83535a326357f81d7b9a57de Sort DWI chunks from top to bottom (#18)
8e541e9064c9be264a85f867ed93371df80c506b (jca/17-manual-corr) Create analysis script (#8)
ca8104e9f430e1f7e47605aabae21c1c86790308 Update README.md
2c078e6887ad5a3be9eff721e405269e45d728bb Added doc to convert to BIDS
183e7b90ae59f84e715ae159f26c001469adb709 Refactored zip_and_move_file()
eb9998ba7cc1ffb18f2cbfd0c3f65f74ee86657d Cleanup
5d2a8d06b356378de590b20a1d619950d86c0fc9 Convert to function with input arguments
7ab56392eed3d2789e9ab046b301d93381905ebc Added printouts
b395dd28d215490cb26f617cd25591687c16fd24 Cleanup, added docstrings
44936815a372514e5b587070614feaf07d6219b4 Create directory inside the zip_and_move_nifti() function
96bb57438fe0f1519e12c67ace7af74fbccd2aff Put back .gz extension on output file name
be206ec4d5dcbe11949d2414a734b3b34224bb64 Fixed duplicated 'sub-' prefix, removed .gz
a332bb5c40e905de181cd3b3ab256e434f2580c3 Added printout for ignored file
85202a6f64775c18845c8757033f1fc7090b1671 Added docstrings, added output path
ea22e2539ec86bcde6dee6ea18b4214b817861f5 Pushed first prototype that parses subject directory
0d348fc71c7fca496d8d0bd1756bf85bb3f6eee4 Initial commit

@Kaonashi22
Copy link

Kaonashi22 commented Apr 8, 2024 via email

@Kaonashi22
Copy link

I ran the pipeline on all the subjects. The processing time is quite longer (~2h30 for 15 subjects). I split the subjects into groups of 15 to make it smoother. I also had storage space issues, which didn't help...

Here are some comments:
-When I re-run the analysis, all the segmentation files are overwritten. I'm not sure if the "segment_if_does_not_exist "${file_t2}" "t2"" is working properly
-I got errors for some subjects; the log files are attached
-At the end of the processing, the error.log file indicates that the "_T2_seg.nii.gz" and "mtr.nii.gz" don't exit. I guess the path to the "anat" folder is missing
-The diffusion maps (FA, MD,...) are only generated for one DWI chunk.
-Regarding vertebral labeling before registration to template, should we call the SPINEPS function from the script?

error.log
err.batch_processing_sub-LD214.log
err.batch_processing_sub-ER240.log
err.batch_processing_sub-GE200.log

err.batch_processing_sub-GB300.log

@joshuacwnewton
Copy link
Member

Oh dear! In that case, I can try to prioritize the SCT issue in order to avoid - jobs 1 and reduce the processing time.

@jcohenadad jcohenadad mentioned this issue Apr 10, 2024
Closed
@jcohenadad
Copy link
Member Author

The processing time is quite longer (~2h30 for 15 subjects). I split the subjects into groups of 15 to make it smoother.

Sorry about that. We'll work on a fix which hopefully will solve the underlying issue

When I re-run the analysis, all the segmentation files are overwritten. I'm not sure if the "segment_if_does_not_exist "${file_t2}" "t2"" is working properly

It should not be the case, as per:

spine-park/batch_processing.sh

Lines 78 to 79 in 8e541e9

echo "Found! Using manual segmentation."
rsync -avzh "${FILESEGMANUAL}" "${FILESEG}".nii.gz

Can you please give an example of the full path where the segmentation that is being overwritten is located originally? I suspect that you did not locate the manual segmentation in the right folder (it works on my end #17)

At the end of the processing, the error.log file indicates that the "_T2_seg.nii.gz" and "mtr.nii.gz" don't exit. I guess the path to the "anat" folder is missing

Good catch! I've opened an issue: #19

The diffusion maps (FA, MD,...) are only generated for one DWI chunk.

Ah! This is because the name for DTI metrics is generic (eg: dti_FA.nii.gz), so it gets overwritten. My bad. I've opened an issue, it is easy to fix that. #20

Regarding vertebral labeling before registration to template, should we call the SPINEPS function from the script?

Yes, we should. Issue opened #21

err.batch_processing_sub-LD214.log
err.batch_processing_sub-GB300.log

Issue opened on SCT: spinalcordtoolbox/spinalcordtoolbox#4431
Can you please share these subjects with me?

err.batch_processing_sub-ER240.log

The log file says "could the file be damaged?". I'm wondering if this isn't a disk issue? You mentioned you had such issues.

err.batch_processing_sub-GE200.log

Possibly caused by #22. I'll fix it.

@Kaonashi22
Copy link

I answered to the segmentation files being overwritten issue in #17)

Sending you the subject images by email.
Thanks!

@joshuacwnewton
Copy link
Member

err.batch_processing_sub-LD214.log
err.batch_processing_sub-GB300.log

Issue opened on SCT: spinalcordtoolbox/spinalcordtoolbox#4431

Note: These two subjects that @jcohenadad has highlighted have different errors:

  • LD214: OSError: [Errno 5] Input/output error

  • GB300: itk::ERROR: [...] failed to read file

    export02/data/lydiac/spinalcordtoolbox/bin/isct_ComposeMultiTransform 2 warp2d_00390Warp.nii.gz -R dest_Z0039.nii warp2d_null0Warp.nii.gz warp2d_00390GenericAffine.mat # in /tmp/sct_2024-04-10_08-42-18_register-slicewise_nx0liske
Exception occurred. 
 output_image_filename: warp2d_00390Warp.nii.gz
reference_image_filename: dest_Z0039.nii
[0/2]: FIELD: warp2d_null0Warp.nii.gz
[1/2]: AFFINE: warp2d_00390GenericAffine.mat
terminate called after throwing an instance of 'itk::ExceptionObject'
what():  /__w/build_ANTs/build_ANTs/antsbin/ITKv5/Modules/IO/TransformBase/src/itkTransformFileReader.cxx:144:
itk::ERROR: TransformFileReaderTemplate(0x22dfc60): Transform IO: MatlabTransformIOTemplate
failed to read file: warp2d_00390GenericAffine.mat

Given that the GB300 error occurred A) inside isct_ComposeMultiTransform and B) while trying to read the file (rather than write files), I'm wondering if this is indeed another disk issue, as @jcohenadad suspected.

@Kaonashi22
Copy link

Then, I'll rerun the processing in a folder with more space. Though I had the error "no storage space left" in some other subjects, which was more straightforward.

@joshuacwnewton
Copy link
Member

@Kaonashi22 Do you know if any of the filesystems you're working with (e.g. /export02/data/lydiac/, /dagher/dagher11/lydia11/) are "NFS mounts" specifically? You can check this by running:

mount | grep nfs

If so, this might be the key to the QC locking issues: spinalcordtoolbox/spinalcordtoolbox#4423 (comment)

@Kaonashi22
Copy link

Kaonashi22 commented Apr 10, 2024 via email

@joshuacwnewton
Copy link
Member

Aha! I think that explains it then! (/dagher/dagher11 -- where the data lives and where we perform locking for the QC index file -- is an NFS mount.)

I'll take a look at trying portalocker's proposed fix for handling NFS mounted drives. :)

@Kaonashi22
Copy link

Kaonashi22 commented Apr 11, 2024 via email

@jcohenadad
Copy link
Member Author

jcohenadad commented Apr 11, 2024 •
edited
Loading

All 4 subjects ran without error, so I suspect the issue was related to disk space and/or NFS portalocker issue

Terminal output 
Processing 4 subjects in parallel. (Worker processes used: 4).
Started at 11h44m21s: sub-ER240. See log file /Users/julien/temp/Lydia/results_20240411_114414/log/batch_processing_sub-ER240.log
Started at 11h44m21s: sub-GB300. See log file /Users/julien/temp/Lydia/results_20240411_114414/log/batch_processing_sub-GB300.log
Started at 11h44m21s: sub-GE200. See log file /Users/julien/temp/Lydia/results_20240411_114414/log/batch_processing_sub-GE200.log
Started at 11h44m21s: sub-LD214. See log file /Users/julien/temp/Lydia/results_20240411_114414/log/batch_processing_sub-LD214.log

Hooray! your batch completed successfully :-)

Started: 2024-04-11 11h44m21s | Ended: 12h57m38s | Duration: 01h13m16s

@Kaonashi22
Copy link

Kaonashi22 commented Apr 11, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jcohenadad @joshuacwnewton @Kaonashi22