Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Improve datavzrd tables #93

Merged
merged 109 commits into from
Aug 15, 2024
Merged

feat: Improve datavzrd tables #93

merged 109 commits into from
Aug 15, 2024

Conversation

Addimator
Copy link
Collaborator

No description provided.

@Addimator Addimator changed the title Locotact p18 feat: Improve datavzrd tables May 24, 2024
Copy link
Contributor

@johanneskoester johanneskoester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

general: get rid of commented out code, tidy scripts (double empty lines after comments), ensure that scripts are named after what they actually do. Bad example: copy_study_items_sig_terms. The script seems to do more, it is unclear on what kind of data it operates. Sometimes, less is more, e.g. postprocess_go_enrichment.py.

workflow/resources/datavzrd/go-enrichment-template.yaml Outdated Show resolved Hide resolved
workflow/rules/datavzrd.smk Outdated Show resolved Hide resolved
workflow/rules/datavzrd.smk Outdated Show resolved Hide resolved
workflow/rules/datavzrd.smk Show resolved Hide resolved
@dlaehnemann
Copy link
Collaborator

Just wanted to give you a heads-up that I am planning to merge another pull request tomorrow, which will change the datavzrd templates and update to the latest version of the datavzrd wrapper: #92

If you need me to look into updating this branch after the merge, let me know. All the changes are still fresh in my mind, so I should be reasonably confident in resolving conflicts that might arise.

@dlaehnemann
Copy link
Collaborator

dlaehnemann commented Jun 24, 2024

So, I think I have reasonably resolved the conflicts with the latest main branch. Let's see if the tests pass -- if they don't, I'll try to resolve remaining issues.

Afterwards, I think we should still:

  • double-check that the rules are wired as intended
  • make sure the new meta comparison datavzrd reports are requested as output somewhere in common.smk's all_input() function
  • have a look at actual reports using the functionality, to make sure everything works as intended

Copy link
Collaborator

@dlaehnemann dlaehnemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, and thanks for fixing the __use__yte__: trues that I forgot.

For now, this is just another round of questions and suggestions. But it looks like we are almost there.

workflow/scripts/postprocess_spia.py Outdated Show resolved Hide resolved
workflow/report/meta_compare.rst Outdated Show resolved Hide resolved
config/config.yaml Show resolved Hide resolved
workflow/scripts/postprocess_diffexp.py Outdated Show resolved Hide resolved
workflow/rules/datavzrd.smk Outdated Show resolved Hide resolved
workflow/scripts/postprocess_go_enrichment.py Outdated Show resolved Hide resolved
workflow/scripts/postprocess_go_enrichment.py Outdated Show resolved Hide resolved
Copy link
Collaborator Author

@Addimator Addimator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have now edited most of David's comments. In general, I still have a problem understanding the review process, because I see a lot of outdated comments in the conversation tab and for every comment I write on an outdated comment a new box is opened elsewhere. I find this very confusing. Not all of your comments appear in the Files Changed tab. I have now gone through both tabs several times and hope I have not forgotten anything.
If I am generally acting incorrectly during the Git workflow, I would be happy to receive suggestions for improvement!

Comments that are still pending and need to be fixed before merging. I'll try to do that on Monday (and today):

Questions:

  • workflow/resources/datavzrd/spia-template.yaml: How to show gene_ratio, should I use bar plot or circle_plots
  • config/config.yaml: What comments are missing for meta_comparisons

ToDo:

  • workflow/report/workflow.rst: Include "For sample metadata, see {{ snakemake.config["samples"] }}_." Maybe linkout to sample.tsv in the report.
  • workflow/envs/pystats.yaml: Look which dependencies are really needed
  • workflow/scripts/compare_diffexp.py.ipynb: Why is it so big? Are those artifacts?
  • Let it run and create report to verify everything

workflow/scripts/postprocess_go_enrichment.py Outdated Show resolved Hide resolved
.test/config/config.yaml Outdated Show resolved Hide resolved
config/config.yaml Show resolved Hide resolved
workflow/report/workflow.rst Show resolved Hide resolved
workflow/scripts/compare_diffexp.py.ipynb Outdated Show resolved Hide resolved
config/README.md Outdated Show resolved Hide resolved
workflow/envs/pystats.yaml Outdated Show resolved Hide resolved
workflow/rules/datavzrd.smk Outdated Show resolved Hide resolved
config/README.md Outdated Show resolved Hide resolved
config/README.md Outdated Show resolved Hide resolved
@dlaehnemann
Copy link
Collaborator

Just had a quick look, and apart from the things you list, I think you caught all of the other comments. I think all this weirdness in where this appear or don't really comes from this whole pending issue. I think I initially also had some review comments pending before I did the last round of reviews. And with some of your comments also pending and you applying fixes in the meantime, I think that a bunch of the references to lines and other comments just got jumbled.

So maybe you can go through this on Monday, until you are satisfied with the result. And then I will do a fresh round of review, that hopefully offers a less confusing experience for us both. Sorry for my part in causing the confusion here, but I think we're almost through... 😅

Copy link
Collaborator Author

@Addimator Addimator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the open problems. For each of the big jupyter notebooks I created a new python file. The pystats env had really way too much dependencies in it. I removed the unnecessary ones. I have problems linking to the specific input samples file in the workflow.rst, so right now I am linking to the input folder containing the samples file.

workflow/rules/datavzrd.smk Outdated Show resolved Hide resolved
workflow/report/workflow.rst Show resolved Hide resolved
Copy link
Collaborator

@dlaehnemann dlaehnemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're definitely getting there, just some smaller things that should be quick to fix, many with a direct commit suggestion.

.test/config/config.yaml Outdated Show resolved Hide resolved
.test/config/config.yaml Outdated Show resolved Hide resolved
.test/three_prime/config/config.yaml Outdated Show resolved Hide resolved
config/config.yaml Outdated Show resolved Hide resolved
workflow/rules/diffexp.smk Outdated Show resolved Hide resolved
workflow/report/units.rst Outdated Show resolved Hide resolved
workflow/report/workflow.rst Show resolved Hide resolved
workflow/report/workflow.rst Show resolved Hide resolved
workflow/rules/datavzrd.smk Outdated Show resolved Hide resolved
workflow/scripts/compare_diffexp.py Show resolved Hide resolved
Addimator and others added 3 commits August 13, 2024 09:56
Apply Davids suggestions for better code quality

Co-authored-by: David Laehnemann <[email protected]>
Copy link
Collaborator

@dlaehnemann dlaehnemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me! Just 3 little suggestions, one a typo, two of them more descriptive titles for the report (but which will only be used, once we have wildcards available in snakemake report() captions, see snakemake/snakemake#3021).

Feel free to merge, once you have incorporated the suggestions (or something similar). And ping me, in case you don't have the necessary permissions to merge.

config/README.md Outdated Show resolved Hide resolved
workflow/resources/datavzrd/units-template.yaml Outdated Show resolved Hide resolved
workflow/resources/datavzrd/samples-template.yaml Outdated Show resolved Hide resolved
@dlaehnemann dlaehnemann dismissed johanneskoester’s stale review August 14, 2024 18:46

All requests have been addressed, no need to bother Johannes.

Addimator and others added 3 commits August 15, 2024 08:33
@dlaehnemann dlaehnemann merged commit 93512b8 into main Aug 15, 2024
6 checks passed
@dlaehnemann dlaehnemann deleted the locotact_p18 branch August 15, 2024 08:32
johanneskoester pushed a commit that referenced this pull request Aug 16, 2024
🤖 I have created a release *beep* *boop*
---


##
[2.7.0](v2.6.0...v2.7.0)
(2024-08-15)


### Features

* Improve datavzrd tables
([#93](#93))
([93512b8](93512b8))


### Bug Fixes

* Fix missing output in spia.R when no significant genes are found
([#103](#103))
([bc0d017](bc0d017))
* Handle missing bam columns in units.tsv
([#105](#105))
([bae88d0](bae88d0))
* Remove non-existent outputs in spia rule
([#102](#102))
([0fbb930](0fbb930))
* update to latest datavzrd
([417ec3b](417ec3b))


### Performance Improvements

* datavzrd wrapper `v3.12.1`, offer-excel configurable, free disk space
for CI, dynamic sleuth_init mem_mb, pure download rules as localrules
([#92](#92))
([70850fb](70850fb))
* Update datavzrd wrapper
([#98](#98))
([e5eb0e0](e5eb0e0))
* Update samtools fast separate wrapper
([#100](#100))
([65d8f41](65d8f41))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants