Ensure all released repo dependencies are in the `openedx` GitHub organization #238

kdmccormick · 2022-12-08T15:51:34Z

Context

Starting with Palm, if a repo is in the named release, then its first-party dependencies must be in the openedx GH org.

More details: https://discuss.openedx.org/t/interim-guidance-on-the-openedx-github-organization/8518

This is related to, but not the same as: #203

Known issues

Deps to edx-platform:

Deps to default frontend-apps (learning, account, gradebook, profile, authn, discussions, course-authoring):

TBD

Acceptance Criteria

Write tooling to find all offending dependencies.

Then, for each dependency, either:

move it to the openedx org; or
remove it from edx-platform's requirements list, and tell interested operators (notably 2U) that those packages must be added to extra/private requirements in order to be installed with the platform.

The text was updated successfully, but these errors were encountered:

kdmccormick · 2022-12-08T19:01:50Z

The "Known Issues" list as of right now was based on some initial discovery by @nedbat . We have not yet checked the dependencies of the default frontend-apps.

Also: We have not checked the dep lists for course-discovery, ecommerce, and other release-tagged IDAs. Tthis is OK, because none of those IDAs are run by default in the named release.

ghassanmas · 2022-12-11T13:12:26Z

I think you might want to add https://github.com/edx/mongoid-tree, which is used by https://github.com/openedx/cs_comments_service

refs:

Or may be not, because forum is not run by defualt...

nedbat · 2023-01-17T21:56:15Z

I've added a check to edx-repo-health to ensure all main repos are in the correct organization: openedx/edx-repo-health#340

kdmccormick · 2023-02-01T18:54:32Z

Thanks Ned!

mariajgrimaldi · 2023-03-20T13:29:23Z

Hi folks! Is there something we could help you with so we can move this forward?

kdmccormick · 2023-03-20T14:29:01Z

@nedbat , had you been working on a script to check whether dependencies of released repos are in the openedx org? Or was that just for top-level released repos?

nedbat · 2023-03-21T22:44:01Z

I worked on it back in December: https://github.com/openedx/repo-tools/blob/master/edx_repo_tools/find_dependencies/README.rst

kdmccormick · 2023-04-13T20:11:36Z

Nice, thanks @nedbat .

@mariajgrimaldi this issue remains important, but I think everyone involved is just busy right now. I'm not sure if Feanil or Ned plan on picking up it; if not, I'd probably be able to work more on it in July-ish.

feanil · 2023-04-13T20:36:21Z

I think the next step is to re-run Ned's script and get a concrete list of all the repos that are outside the openedx organization. Once that's done we can ticket each one individually and start picking them off as it makes sense. I'm in the same boat as Kyle and can't spend a lot of time on it now but @mariajgrimaldi if you have time to do that and update this ticket with the findings, that would be a great help.

mariajgrimaldi · 2023-08-07T14:02:20Z

Do you folks still need help with this? I've now allocated the time for it -- sorry for the delay!

kdmccormick · 2023-08-10T14:35:50Z

@mariajgrimaldi no worries. If you have time and are interested in working on this, then certainly we could still use the help!

nedbat · 2023-08-17T15:16:00Z

I re-ran the dependency analyzer last week and got these results:

These are the 47 repos tagged for Palm:

openedx/blockstore
openedx/configuration
openedx/course-discovery
openedx/credentials
openedx/cs_comments_service
openedx/devstack
openedx/docs.openedx.org
openedx/DoneXBlock
openedx/ecommerce
openedx/ecommerce-worker
openedx/edx-analytics-configuration
openedx/edx-analytics-dashboard
openedx/edx-analytics-data-api
openedx/edx-analytics-pipeline
openedx/edx-app-android
openedx/edx-app-ios
openedx/edx-documentation
openedx/edx-notes-api
openedx/edx-platform
openedx/enterprise-access
openedx/enterprise-catalog
openedx/frontend-app-account
openedx/frontend-app-authn
openedx/frontend-app-communications
openedx/frontend-app-course-authoring
openedx/frontend-app-discussions
openedx/frontend-app-ecommerce
openedx/frontend-app-gradebook
openedx/frontend-app-learner-dashboard
openedx/frontend-app-learner-record
openedx/frontend-app-learning
openedx/frontend-app-library-authoring
openedx/frontend-app-ora-grading
openedx/frontend-app-payment
openedx/frontend-app-profile
openedx/frontend-app-publisher
openedx/frontend-app-support-tools
openedx/frontend-template-application
openedx/license-manager
openedx/openedx-aspects
openedx/openedx-demo-course
openedx/openedx-i18n
openedx/openedx-test-course
openedx/repo-tools
openedx/testeng-ci
openedx/tubular
openedx/xqueue

These are the 54 repos in the openedx organization in the transitive dependencies:

openedx/api-doc-tools
openedx/auth-backends
openedx/blockstore
openedx/brand-openedx
openedx/code-annotations
openedx/codejail
openedx/completion
openedx/credentials-themes
openedx/django-config-models
openedx/django-lang-pref-middleware
openedx/django-pyfs
openedx/django-splash
openedx/django-user-tasks
openedx/ecommerce-worker
openedx/edx-ace
openedx/edx-analytics-data-api-client
openedx/edx-bulk-grades
openedx/edx-celeryutils
openedx/edx-custom-a11y-rules
openedx/edx-django-utils
openedx/edx-drf-extensions
openedx/edx-enterprise
openedx/edx-enterprise-data
openedx/edx-enterprise-subsidy-client
openedx/edx-milestones
openedx/edx-organizations
openedx/edx-proctoring
openedx/edx-rbac
openedx/edx-rest-api-client
openedx/edx-search
openedx/edx-toggles
openedx/edx-when
openedx/eslint-config
openedx/event-bus-kafka
openedx/event-bus-redis
openedx/frontend-build
openedx/frontend-component-footer
openedx/frontend-component-header
openedx/frontend-enterprise
openedx/frontend-platform
openedx/i18n-tools
openedx/olxcleaner
openedx/opaque-keys
openedx/openedx-events
openedx/openedx-filters
openedx/openedx-learning
openedx/paragon
openedx/react-unit-test-utils
openedx/reactifex
openedx/stylelint-config-edx
openedx/XBlock
openedx/xblock-lti-consumer
openedx/xblock-utils
openedx/xss-utils

These are the 9 repos in adjacent organizations that are in the transitive dependencies:

edx/brand-edx.org
edx/braze-client
edx/edx-name-affirmation
edx/frontend-component-footer-edx
edx/getsmarter-api-clients
edx/new-relic-source-map-webpack-plugin
edx/token-utils
edx/ux-pattern-library
mitodl/edx-sga

BUT: Ed noticed that ora2 isn't mentioned. We aren't yet analyzing dependencies that get installed via other mechanisms. We still have some work to do on the analyzer.

kdmccormick · 2023-08-17T16:30:07Z

Thanks @nedbat.

By "other mechanisms" I assumed you mean things like EDXAPP_PRIVATE_REQUIREMENTS, but ora2 is installed in base.txt: https://github.com/openedx/edx-platform/blob/ddb092c07c6e2b663dc3d52e7197b42f7f017988/requirements/edx/base.txt#L776

nedbat · 2023-08-17T16:46:43Z

Yeah, I'm digging into this now. I don't understand why ora2 doesn't end up in the results yet.

nedbat · 2023-08-17T21:45:04Z

Updated results (from this repo-tools fix):

These are the 80 repos in the openedx organization in the transitive dependencies:

These are the 10 repos in adjacent organizations that are in the transitive dependencies:

https://github.com/edx-unsupported/eslint-config-edx
https://github.com/edx/brand-edx.org
https://github.com/edx/braze-client
https://github.com/edx/edx-name-affirmation
https://github.com/edx/frontend-component-footer-edx
https://github.com/edx/getsmarter-api-clients
https://github.com/edx/new-relic-source-map-webpack-plugin
https://github.com/edx/token-utils
https://github.com/edx/ux-pattern-library
https://github.com/mitodl/edx-sga

nedbat · 2023-08-21T21:03:18Z

Hmm, there are still some missing repos because some don't declare their URL in their metadata:

Repo URL is UNKNOWN in files/openedx_django_wiki-2.0.1-py3-none-any.whl
No repo URL in files/staff_graded_xblock-2.1.1-py3-none-any.whl
Repo URL is UNKNOWN in files/pynliner-0.8.0-py2.py3-none-any.whl
Repo URL is UNKNOWN in files/xblock_poll-1.13.0-py3-none-any.whl
No repo URL in files/django_multi_email_field-0.7.0-py3-none-any.whl
No repo URL in files/shapely-2.0.1-cp38-cp38-macosx_10_9_x86_64.whl
Repo URL is UNKNOWN in files/done_xblock-2.1.0-py3-none-any.whl
No repo URL in files/xblock_drag_and_drop_v2-3.2.0-py3-none-any.whl
No repo URL in files/click_didyoumean-0.3.0-py3-none-any.whl
Repo URL is UNKNOWN in files/xblock_google_drive-0.4.0-py2.py3-none-any.whl
Repo URL is UNKNOWN in files/edx_user_state_client-1.3.2-py3-none-any.whl

(I think the difference between "No repo URL" and "Repo URL is UNKNOWN" is due to how the packages are built?)

I've made some pull requests to add the URL to our packages. Thumbs appreciated:

ghassanmas · 2023-09-06T13:54:52Z

I don't if this also releavnt while working on something I found a dependency in frontend-app-learning called frontend-lib-learning-assistant ref in pacakge.json whichs exits in edx https://github.com/edx/frontend-lib-learning-assistant

kdmccormick self-assigned this Dec 8, 2022

kdmccormick mentioned this issue Dec 8, 2022

Remove the edx GitHub organization from the release process #203

Closed

kdmccormick added the enhancement Relates to new features or improvements to existing features label Dec 8, 2022

feanil self-assigned this Dec 8, 2022

arbrandes added this to the Palm.1 milestone Dec 8, 2022

feanil assigned nedbat Jan 4, 2023

mariajgrimaldi modified the milestones: Palm.1, Palm.2 Jul 24, 2023

mariajgrimaldi modified the milestones: Palm.2, Palm.3 Aug 16, 2023

nedbat mentioned this issue Aug 17, 2023

fix(find-dependencies): http->https redirections weren't followed openedx/repo-tools#431

Merged

mariajgrimaldi modified the milestones: Palm.3, Quince.1 Oct 16, 2023

cmltaWt0 modified the milestones: Quince.1, Quince.2 Jan 8, 2024

mariajgrimaldi modified the milestones: Quince.2, Quince.3 Feb 12, 2024

mariajgrimaldi modified the milestones: Quince.3, Redwood.1 Apr 26, 2024

mariajgrimaldi modified the milestones: Redwood.1, Sumac.1 Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure all released repo dependencies are in the `openedx` GitHub organization #238

Ensure all released repo dependencies are in the `openedx` GitHub organization #238

kdmccormick commented Dec 8, 2022 •

edited

Loading

kdmccormick commented Dec 8, 2022 •

edited

Loading

ghassanmas commented Dec 11, 2022 •

edited

Loading

nedbat commented Jan 17, 2023

kdmccormick commented Feb 1, 2023

mariajgrimaldi commented Mar 20, 2023

kdmccormick commented Mar 20, 2023

nedbat commented Mar 21, 2023

kdmccormick commented Apr 13, 2023

feanil commented Apr 13, 2023

mariajgrimaldi commented Aug 7, 2023

kdmccormick commented Aug 10, 2023

nedbat commented Aug 17, 2023

kdmccormick commented Aug 17, 2023

nedbat commented Aug 17, 2023

nedbat commented Aug 17, 2023 •

edited

Loading

nedbat commented Aug 21, 2023 •

edited

Loading

ghassanmas commented Sep 6, 2023 •

edited

Loading

Ensure all released repo dependencies are in the openedx GitHub organization #238

Ensure all released repo dependencies are in the openedx GitHub organization #238

Comments

kdmccormick commented Dec 8, 2022 • edited Loading

Context

Known issues

Acceptance Criteria

kdmccormick commented Dec 8, 2022 • edited Loading

ghassanmas commented Dec 11, 2022 • edited Loading

nedbat commented Jan 17, 2023

kdmccormick commented Feb 1, 2023

mariajgrimaldi commented Mar 20, 2023

kdmccormick commented Mar 20, 2023

nedbat commented Mar 21, 2023

kdmccormick commented Apr 13, 2023

feanil commented Apr 13, 2023

mariajgrimaldi commented Aug 7, 2023

kdmccormick commented Aug 10, 2023

nedbat commented Aug 17, 2023

kdmccormick commented Aug 17, 2023

nedbat commented Aug 17, 2023

nedbat commented Aug 17, 2023 • edited Loading

nedbat commented Aug 21, 2023 • edited Loading

ghassanmas commented Sep 6, 2023 • edited Loading

Ensure all released repo dependencies are in the `openedx` GitHub organization #238

Ensure all released repo dependencies are in the `openedx` GitHub organization #238

kdmccormick commented Dec 8, 2022 •

edited

Loading

kdmccormick commented Dec 8, 2022 •

edited

Loading

ghassanmas commented Dec 11, 2022 •

edited

Loading

nedbat commented Aug 17, 2023 •

edited

Loading

nedbat commented Aug 21, 2023 •

edited

Loading

ghassanmas commented Sep 6, 2023 •

edited

Loading