Upgrade externally managed content #628

matentzn · 2024-08-05T16:32:18Z

Overview

This PR:

Adds all missing pieces of externally managed content to Mondo Ingest (EFO, ClinGen, MedGen)
And make goals to update them

Pre-merge checklist

Documentation

Was the documentation added/updated under docs/?

Yes
No, updates to the docs were not necessary after careful consideration

QC

Was the full pipeline run before submitting this PR using sh run.sh make build-mondo-ingest on this branch (after
docker pull obolibrary/odkfull:dev), and no errors occurred?

Yes
No, there are no functional (code-related) changes to the pipeline in the PR, so no re-run is necessary

New Packages

Were any new Python packages added?

Yes, requirements.txt.full was updated and an ODK issue submitted
No

Were any other non-Python packages added?

Yes, Dockerfile updated and an ODK issue submitted
No

PR Review and Conversations Resolved

Has the PR been sufficiently reviewed by at least 1 team member of the Mondo Technical team and all threads resolved?

Yes

Unfortunately, I cant find the orginal TSV file, but it is highly unlikely this file will ever change.

src/ontology/mondo-ingest.Makefile

src/ontology/external/efo-proxy-merges.robot.owl

in src/ontology ``` wget http://purl.obolibrary.org/obo/mondo.owl -O mondo.owl sh run.sh make tmp/mondo-incl-robot-report.tsv ```

@twhetzel

…ged content cc @twhetzel check this commit to look at the emc error fixing pipeline in isolation

matentzn · 2024-08-19T20:35:16Z

Next steps:

@twhetzel to update develop from main
@twhetzel to update externalclingenmedgenefo from develop
@twhetzel to update mondo-ingest/src/ontology/config/robot_report_external_content.txt with all important checks she can think of
@matentzn to run full build of pipeline and _commit changes to all files in src/ontology/external to this branch (only those files)
@twhetzel to approve PR and merge
@matentzn CHANGE emc pipeline to point to main branch
@twhetzel agreed?

src/ontology/lexmatch/README.md

.gitignore

src/ontology/config/robot_report_external_content.txt

src/ontology/mondo-ingest.Makefile

src/scripts/post_process_externally_managed_content.py

src/ontology/mondo-ingest.Makefile

src/scripts/post_process_externally_managed_content.py

Redesign of CLI, function names ectc.

src/ontology/medgen-xrefs.robot.template.tsv

src/ontology/mondo-ingest.Makefile

…refreshing repo See #592

matentzn

I have one change request (related to EFO proxy merges, see comments) which I would like your opinion on @twhetzel, else I think this is good to go. You can run the pipeline with:

sh run.sh make update-externally-managed-content -B

However, it does take a long time, so strap in. The biggest downside of this PR is the fact that I chose to properly simulate the EMC pipeline, which adds a big overhead on the runtime.

EDIT: Not that long in the end, at least if you discount the mondo repo refresh:

### DEBUG STATS ###
Elapsed time: 0:13.96
Peak memory: 1669976 kb

I just (24.08.2024, 2pm EEST) finished a complete run of the external pipeline, and all looks good to me. So remains to be seen what you think of my comments and your general review.

matentzn · 2024-08-24T09:31:22Z

src/ontology/external/nord-qc-failures.md

+| MONDO:0007030 |              |                     | Aarskog Syndrome                          |                         |                |                                                |                 |                  | nord     | duplicate_scoped_synonym (oboInOwl:hasExactSynonym) |
+| MONDO:0007306 |              |                     | Klippel-Feil Syndrome                     |                         |                |                                                |                 |                  | nord     | duplicate_scoped_synonym (oboInOwl:hasBroadSynonym) |
+| MONDO:0007389 |              |                     | Spondylothoracic Dysplasia                |                         |                |                                                |                 |                  | nord     | duplicate_scoped_synonym (oboInOwl:hasExactSynonym) |
+| MONDO:0005676 |              |                     |                                           |                         |                | http://purl.obolibrary.org/obo/mondo#nord_rare |                 |                  | nord     | qc-animal-disease-rare (oboInOwl:inSubset)          |


I am glad this made it through after some fiddling!

matentzn · 2024-08-24T09:54:56Z

src/ontology/mondo-ingest.Makefile

+
+$(TMPDIR)/mondo-with-simulated-emc-run.owl: $(TMPDIR)/mondo_repo_built
+	mkdir -p $(TMPDIR)/mondo/src/ontology/tmp/external/
+	$(foreach n,$(EXTERNAL_FILES), cp -f $(EXTERNAL_CONTENT_DIR)/$(n).robot.owl $(TMPDIR)/mondo/src/ontology/tmp/external/processed-$(n).robot.owl;) \


This line here is the crux of the whole PR. Instead of creating our own EMC pipeline here, we are letting the real EMC pipeline in the Mondo repo run. This works basically by copying the unprocessed files over into the mondo repo, running the EMC pipeline there as it is, and then copying the mondo edit back here for running our new QC system.

matentzn · 2024-08-24T09:58:40Z

src/ontology/mondo-ingest.Makefile

+# Unfortunately, @matentzn cant find the orginal TSV file, but it is highly unlikely this file will ever change.
+
+$(EXTERNAL_CONTENT_DIR)/efo-proxy-merges.robot.owl:
+	echo "WARNING: $@ is currently manually curated!"


In my self review, I am a bit inclined now to drop this file entirely from the pipeline and have this being fixed on the EFO side - like with all other content we do. Here is what this file does:

There are 3 cases or 4 where two EFO ids map to the same Mondo ID

This file makes that valid by adding "preferred external" to one of the EFO ids in each pair

I feel now on second thought that we should not easily validate externally provided proxy merged IDs, and work with the upstream sources to fix them.

@matentzn I agree that we should not fix these on our side, but work with the provider of the source to fix these or a a minimum have the provider determine which should be tagged as preferred. There is also usually some curator work to determine which one should be the "preferred external" so automatically choosing which ID skips this needed work.

Ok, I have fixed it now. I made a test here: https://github.com/monarch-initiative/mondo/pull/8120/files#r1734450583

This commit describes the removal of that part from the pipeline: 52fdd09

Note that I have yet deleted the efo-merges owl file from this PR, but I have removed all its content, as there is a cyclic dependency here that was too cumbersome to remove. If you like you can remove these files after first removing references to this file from mondo.Makefile, e.g.:

https://github.com/monarch-initiative/mondo/blob/49291cff03da830ec8826c623f042b2ecfafd6d4/src/ontology/mondo.Makefile#L457

src/ontology/mondo-ingest.Makefile

twhetzel · 2024-08-27T00:35:00Z

src/ontology/mondo-ingest.Makefile

+	cp mondo-edit.owl mondo-edit.owl.bak && \
+	cp mondo-edit.obo mondo-edit.obo.bak && \
+	make update-external-content-incl-rare -B MIR=false IMP=false MIR=false DOWNLOAD_EXTERNAL=false &&\
+	make mondo-edit.owl MIR=false IMP=false MIR=false


where is this goal in the Makefile?

Looks like it has already made its way into mondo makefile on main.

Yes, all that you see here is in the context of the mondo, not mondo-ingest pipeline!

twhetzel

I will Approve for now, but I do think that proxy merges from EFO should not be fixed automatically related to this comment. I don't know if there is still time to change this.

twhetzel · 2024-09-06T16:01:26Z

The full build does not work for this branch. I am looking into this.

Error:

if [ false = true ]; then wget "https://raw.githubusercontent.com/monarch-initiative/mondo-ingest/externalclingenmedgenefo/src/ontology/external/processed-mondo-efo.robot.owl" -O tmp/external/processed-mondo-efo.robot.owl; fi
mkdir -p tmp/external
if [ false = true ]; then wget "https://raw.githubusercontent.com/monarch-initiative/mondo-ingest/externalclingenmedgenefo/src/ontology/external/processed-efo-proxy-merges.robot.owl" -O tmp/external/processed-efo-proxy-merges.robot.owl; fi
make[4]: Leaving directory '/work/src/ontology/tmp/mondo/src/ontology'
grep -vE '^(xref: EFO:|subset: otar)' mondo-edit.obo > tmp/mondo-edit.tmp || true
mv tmp/mondo-edit.tmp mondo-edit.obo
robot --catalog catalog-v001.xml merge -i mondo-edit.obo -i tmp/external/processed-mondo-otar-subset.robot.owl -i tmp/external/processed-mondo-efo.robot.owl -i tmp/external/processed-efo-proxy-merges.robot.owl --collapse-import-closure false \
	query --use-graphs false --update ../sparql/update/update-equivalent-obsolete.ru \
	convert -f obo --check false -o mondo-edit.obo.obo
tmp/external/processed-efo-proxy-merges.robot.owl (No such file or directory)
Use the -vvv option to show the stack trace.
Use the --help option to see usage information.
make[3]: *** [mondo.Makefile:460: update-efo-subset] Error 1
make[3]: Leaving directory '/work/src/ontology/tmp/mondo/src/ontology'
make[2]: *** [mondo.Makefile:319: update-external-content-incl-rare] Error 2
make[2]: Leaving directory '/work/src/ontology/tmp/mondo/src/ontology'
make[1]: *** [mondo-ingest.Makefile:617: tmp/mondo-with-simulated-emc-run.owl] Error 2
rm imports/ro_terms_combined.txt
make[1]: Leaving directory '/work/src/ontology'
make: *** [mondo-ingest.Makefile:346: build-mondo-ingest] Error 2
Command exited with non-zero status 2
### DEBUG STATS ###
Elapsed time: 3:48:09

twhetzel

The full build no longer works after further changes were added after this was already reviewed and previously approved.

The GH Action in the Mondo repo "Update Externally Provided Content" does still work so this PR will be left unmerged until the issue with the full build can be sorted out and fixed.

joeflack4

Build is failing.

twhetzel · 2024-09-20T00:30:06Z

src/ontology/config/robot_report_external_content.txt

+ERROR	duplicate_scoped_synonym
+ERROR	file:tmp/mondo/src/sparql/qc/general/qc-duplicate-exact-synonym-no-abbrev.sparql
+ERROR	file:tmp/mondo/src/sparql/qc/mondo/qc-proxy-merges.sparql
+ERROR	file:tmp/mondo/src/sparql/qc/general/qc-related-exact-synonym.sparql


Consider removing this check since it looks like it is covered by the default ROBOT check of duplicate_scoped_synonym already included in this file.

- This provides a necessary bug fix

src/ontology/mondo-ingest.Makefile

* data build on main following mondo release * remove logs file

Add EFO, MedGen and ClinGen externally managed content

25c48b8

matentzn changed the base branch from main to develop August 5, 2024 16:32

matentzn added 2 commits August 5, 2024 19:45

Adding linkouts to Clingen external managed content

e85f62c

Add EFO proxy merges OWL file to external content

7b81599

Unfortunately, I cant find the orginal TSV file, but it is highly unlikely this file will ever change.

joeflack4 added the enhancement New feature or request label Aug 5, 2024

joeflack4 reviewed Aug 5, 2024

View reviewed changes

src/ontology/mondo-ingest.Makefile Outdated Show resolved Hide resolved

src/ontology/mondo-ingest.Makefile Outdated Show resolved Hide resolved

src/ontology/external/efo-proxy-merges.robot.owl Outdated Show resolved Hide resolved

matentzn added 2 commits August 12, 2024 13:48

Create robot_report_external_content.txt

882b4d9

Merge branch 'main' into externalclingenmedgenefo

85ca737

matentzn marked this pull request as ready for review August 18, 2024 19:17

matentzn assigned twhetzel Aug 18, 2024

matentzn added 7 commits August 19, 2024 16:59

Update mondo-clingen.robot.tsv

cb16d35

Normalise externally managed content

ce933fc

Draft report file

dddf9ee

Update .gitignore

729e760

Add pipeline to run robot report for EMC

474e17a

in src/ontology ``` wget http://purl.obolibrary.org/obo/mondo.owl -O mondo.owl sh run.sh make tmp/mondo-incl-robot-report.tsv ```

Update resources

3b630c9

Finalise makefile and add postprocessing pipeline for externally mana…

4a45d4e

…ged content cc @twhetzel check this commit to look at the emc error fixing pipeline in isolation

matentzn added 2 commits August 20, 2024 00:16

Update NORD

81e4171

Fix the the EMC scripts and update report template

3f5fb87

joeflack4 reviewed Aug 19, 2024

View reviewed changes

src/ontology/lexmatch/README.md Outdated Show resolved Hide resolved

Add "processed" versions of externally managed content files

0bb5542

joeflack4 reviewed Aug 19, 2024

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

src/ontology/config/robot_report_external_content.txt Show resolved Hide resolved

src/ontology/mondo-ingest.Makefile Outdated Show resolved Hide resolved

joeflack4 reviewed Aug 19, 2024

View reviewed changes

src/scripts/post_process_externally_managed_content.py Show resolved Hide resolved

matentzn added 3 commits August 20, 2024 13:03

Merge branch 'develop' into externalclingenmedgenefo

7335419

Update data files again

1349361

Fix various things from @joeflack4 review

7fea2ba

Redesign of CLI, function names ectc.

matentzn commented Aug 20, 2024

View reviewed changes

src/ontology/medgen-xrefs.robot.template.tsv Outdated Show resolved Hide resolved

src/ontology/mondo-ingest.Makefile Outdated Show resolved Hide resolved

include check for proxy merges

da4ca3d

joeflack4 reviewed Aug 23, 2024

View reviewed changes

src/ontology/mondo-ingest.Makefile Outdated Show resolved Hide resolved

twhetzel self-requested a review August 24, 2024 03:35

matentzn added 2 commits August 24, 2024 12:20

Adds a conditional to the mondo repo checkout that allows us to SKIP …

822becd

…refreshing repo See #592

Change the EMC simulation comment to the right place

30926f2

matentzn commented Aug 24, 2024

View reviewed changes

Add dependency on OWL versions of robot templates into pipeline

67bbfc6

joeflack4 reviewed Aug 24, 2024

View reviewed changes

src/ontology/mondo-ingest.Makefile Show resolved Hide resolved

twhetzel reviewed Aug 27, 2024

View reviewed changes

twhetzel approved these changes Aug 27, 2024

View reviewed changes

matentzn mentioned this pull request Aug 27, 2024

Change EMC update source branch to m̶a̶s̶t̶e̶r̶ main monarch-initiative/mondo#8117

Draft

1 task

Update datafile after removing efo proxy merges

19a603d

matentzn mentioned this pull request Aug 28, 2024

Externally managed content update monarch-initiative/mondo#8120

Closed

Remove efo proxy merges from pipeline

52fdd09

twhetzel self-requested a review September 6, 2024 18:28

twhetzel requested changes Sep 6, 2024

View reviewed changes

twhetzel requested a review from joeflack4 September 6, 2024 20:41

joeflack4 requested changes Sep 6, 2024

View reviewed changes

twhetzel added the bug Something isn't working label Sep 19, 2024

twhetzel reviewed Sep 20, 2024

View reviewed changes

joeflack4 mentioned this pull request Sep 27, 2024

Upgrade externally managed content: DOWNLOAD_EXTERNAL=true #672

Closed

9 tasks

Merge branch 'develop' into externalclingenmedgenefo

b91758c

- This provides a necessary bug fix

joeflack4 mentioned this pull request Sep 28, 2024

DO NOT MERGE - EMC: test build #674

Closed

joeflack4 reviewed Sep 28, 2024

View reviewed changes

src/ontology/mondo-ingest.Makefile Show resolved Hide resolved

data build on main following mondo release (#676)

66cd017

* data build on main following mondo release * remove logs file

This was referenced Oct 13, 2024

Externally Managed Content - FileNotFound bug fix monarch-initiative/mondo#8236

Merged

EMC outputs update #679

Open

twhetzel and others added 3 commits October 17, 2024 00:05

add new qc check

bd27e2d

updated dir path for sparql query

e4ae33f

Merge branch 'main' into externalclingenmedgenefo

cfb4968

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade externally managed content #628

Upgrade externally managed content #628

matentzn commented Aug 5, 2024 •

edited

Loading

matentzn commented Aug 19, 2024 •

edited

Loading

matentzn left a comment •

edited

Loading

matentzn Aug 24, 2024

matentzn Aug 24, 2024

matentzn Aug 24, 2024

twhetzel Aug 26, 2024 •

edited

Loading

matentzn Aug 28, 2024

twhetzel Aug 27, 2024

joeflack4 Aug 27, 2024 •

edited

Loading

matentzn Aug 28, 2024

twhetzel left a comment

twhetzel commented Sep 6, 2024

twhetzel left a comment •

edited

Loading

joeflack4 left a comment

twhetzel Sep 20, 2024

Upgrade externally managed content #628

Are you sure you want to change the base?

Upgrade externally managed content #628

Conversation

matentzn commented Aug 5, 2024 • edited Loading

Overview

Pre-merge checklist

Documentation

QC

New Packages

PR Review and Conversations Resolved

matentzn commented Aug 19, 2024 • edited Loading

matentzn left a comment • edited Loading

Choose a reason for hiding this comment

matentzn Aug 24, 2024

Choose a reason for hiding this comment

matentzn Aug 24, 2024

Choose a reason for hiding this comment

matentzn Aug 24, 2024

Choose a reason for hiding this comment

twhetzel Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

matentzn Aug 28, 2024

Choose a reason for hiding this comment

twhetzel Aug 27, 2024

Choose a reason for hiding this comment

joeflack4 Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

matentzn Aug 28, 2024

Choose a reason for hiding this comment

twhetzel left a comment

Choose a reason for hiding this comment

twhetzel commented Sep 6, 2024

twhetzel left a comment • edited Loading

Choose a reason for hiding this comment

joeflack4 left a comment

Choose a reason for hiding this comment

twhetzel Sep 20, 2024

Choose a reason for hiding this comment

matentzn commented Aug 5, 2024 •

edited

Loading

matentzn commented Aug 19, 2024 •

edited

Loading

matentzn left a comment •

edited

Loading

twhetzel Aug 26, 2024 •

edited

Loading

joeflack4 Aug 27, 2024 •

edited

Loading

twhetzel left a comment •

edited

Loading