[DEPR]: Course Discovery Service #4449
Comments
|
I'm all for the end-goal of this proposal. I think every provider has met issues with course-discovery and would like to see a replacement that is actually part of the core, is organization agnostic and can be extended. I know as a provider I definitely want that. I even proposed a pattern of hooks to address this issue among others when I was looking at the possible roadmap for hooks for this year and beyond.
Taken from this hooks roadmap document Something that this proposal is not considering is how the information of the catalog will be displayed. I understand that 2U uses the catalog as an API to feed into their marketing site which is a whole other system also not in the core. Perhaps we could leverage a Course search MFE such as the one proposed here Some specific comments:
|
@felipemontoya My thinking is that a basic "Find Courses MFE" should be included with the platform (similar to how the LMS currently has a super basic page to browse available courses):
It would just use the new core catalog service to show courses to the current user (or to users who aren't yet logged in). I expect that it would be super basic. However, because it's using the new catalog service, it would correctly account for any plugins (subscriptions, enterprise, permissions) that are affecting the catalog - it just might not have any ability to show details of those plugins in the UI. So anyone who needs something more elaborate or custom can either (A) extend it using FPF, (B) replace it with their own custom MFE, or (C) turn it off completely and just integrate the REST API data into their existing external website or external LXP. |
|
Yes, this makes a lot of sense. In that case it would be even more important to lay the correct groundwork during the MFE conversion (platform-roadmap/issues/375) . CC @GlugovGrGlib |
|
@bradenmacdonald A huge thanks for this proposal. As someone provision course-discovery and edx-enterprise suite, the recommendations for a new catalogue app and extension of the data models makes a lot of sense.
200% agree with this. If I boil down course-discovery and enterprise-catalog, it is simply a set of data and business rules made for a single business, that everyone else trying to use the software have to agree to or work around to satisfy their requirements. I am spending an unorthodox amount of hours just trying to get courses to show up for the learner in enterprise-learner-portal MFE. Business rules of which course gets displayed for which user should be left to the individual organisations to maintain as simple openedx-filter plugins. (side thought: maybe someone should build a low-code/GUI tool that will allow this to be done dynamically without having to put things down in Python config and requiring redeployment) Some of the relevant work at OpenCraftWhile Braden's proposal comes with the broader experience with the stack. I want to highlight a couple of things that are relevant to this proposal. Note: I don't want this to be taken as an OpenCraft ad of sorts. Trust me, I am not authorised to do it. I merely want to highlight these things so, someone will tell me if there is a better way to do these things or at least provide an opportunity for the community to say, instead of doing this individually, let's do this together.
There is a plugin that does this from existing courses. - https://github.com/open-craft/section-to-course. It takes a section of a course and publishes it as an independent course. I believe it also allows syncing the content if the parent course is ever updated (might be manual at the moment, but could be automatic).
https://github.com/open-craft/learning-paths-plugin/. This is a very early stage project that tried to implement "Programs" as a plugin. "Programs" is a merely a collection of a course, whereas with learning-path we are trying to implement a sequential -or- non-sequential collection with possible inclusion of micro-credentials like Badges. |
|
Here is a recent proposal from the community that gives an example of wanting to extend the core platform course model with custom fields: openedx/edx-platform#35593 - seems like a very relevant example. |
|
Thanks a lot for putting this together @bradenmacdonald! I'm looking forward to not having to run a separate discovery service. The great thing about this DEPR announcement is that it doesn't really require us to build anything new -- that is, until someone steps forward and tells us that Discovery is a required feature for them. Let's use this announcement to gauge community usage. We can then decide whether we want to backport some features in the core, and how. Wouldn't you want to name a specific release as a deadline? Otherwise I'm afraid there won't be a sense of urgency in the community. I propose to target Teak. |
|
I support this removal. Discovery, credentials, edx-enterprise and licensing-manager has been all 2U-specific. It's more likely than not, that Discovery can be better implemented in a customer Marketing Site (e.g. Drupal or similar) than into an edx-platform tied service. |
|
Thanks Braden! Could you clarify what happens if this is Accepted? Would we archive course-discovery, so that it is not part of Teak? |
|
@kdmccormick At a minimum, there will be a ban on developing any new functionality on top of the discovery service. I'd like to say that it won't be officially supported at all as of Teak (but perhaps an org like 2U may continue to maintain and use it if needed for their org's requirements). But I want to hear about who's using it, how they'd be affected, and how willing they are to help build replacement(s) for it and on what timeframe. |
|
I think we should plan on dropping course-discovery by Teak unless there is a lot of community interest in reforming it. If 2U wants to continue to use it and maintain it, I think that can be done off of a fork like with some other services. |
|
Apologies for coming to this conversation late. While I generally agree with the proposal I think it's important that there is a migration path off the current Discovery service before it is deprecated. In particular I am not in favor of dropping official support until such time that the proposed replacement is available. |
|
@scottrish Why not? Are there features of course-discovery that are required for you? |
|
Hi. Representing 2U Phoenix/Inspire squad (@openedx/2u-phoenix) currently managing and contributing to course-discovery, this service is an integral part of 2U business needs and powers major parts of edx.org marketplace. 2U has been committing code to this codebase for a while. There have not been any major community-driven initiatives/features solely for course-discovery in quite a while. Most of the additions have been by edX/2U teams. It is understandable why Axim/Open edX community would want to deprecate this service. We don't have immediate plans to fork this repository unless Axim / Community decides to make changes that will impact 2U's business needs. We will closely follow this ticket for deprecation status and share any action we are taking regarding forking or updating maintainership. cc: @julrusak |
|
@regisb We currently use eComm coupons (we don't support purchasing but do use it for fulfilling purchases made elsewhere), enterprise catalogs, enterprise enrollment URLs, enterprise coupons and also offer the enterprise enrollment APIs although nobody is currently using them. |
Proposal Date
23 September 2024
Ticket Acceptance Date
23 October 2024
Technology Removal Date
2025-2027
First Open edX Named Release Without This Functionality
TBD
Rationale
The Course Discovery Service provides a variety of optional functionality in Open edX, some of which is required for other services like the edx-enterprise suite. However, the fundamental architecture of the discovery service has proven to be problematic, and it is currently a limiting factor that makes it much more difficult than it should be to develop and integrate custom functionality for the Open edX platform such as ecommerce, enterprise integrations, subscriptions, and programs.
Here are some of the problems with its current architecture:
Non-default: The discovery service is not installed by default on most instances of Open edX, which means that "core" functionality is never built using its APIs.
Out of Sync Course Data: Fundamentally, the discovery service maintains at least two separate sets of models about courses (i.e. the
Coursemodel in its MySQL database, and theCourseRunDocumentin Elasticsearch). Though there are various systems for keeping this data in sync with Studio (which holds the actual "source of truth"), they are imperfect and synchronization issues are a frequent source of bugs and errors.Bi-directional sync: Because the discovery service is the source of truth for some pieces of data, it actually syncs its data to Studio as well as from Studio. This has caused a variety of issues, including bugs with an infinite cycle of updates.
Other Redundant/Conflicting Data Models: The discovery service has its own database tables for storing important data that is already stored by Studio/LMS (Discovery
Organizationvs. CoreOrganization,Modevs.CourseMode,Personvs.UserProfile,CourseEditorvs.CourseAccessRole). Often, these models have either subtle or substantial differences from the Studio/LMS version, and there is no guarantee that they are in sync at all.Competing APIs: some operations like "list courses available to me" can be done with the core platform APIs, the discovery service APIs, or the edx-enterprise APIs. Needless to say, they have different behavior, functionality, caching, etc.
Kitchen Sink / edX-Specific Functionality: The Course Discovery Service includes, in no particular order:
paid-bootcampandbootcamp-2u(not to be confused with Tracks, Modes, or Seat Types which are all related)SubjectTopicFactCertificateInfoIt may be the case that the platform lacked ready-to-use extension points at the time these features were required, but they should have been built as extensions regardless.
Problematic Programs implementation: If you want to group courses together into a "Program", you can do so using the discovery service programs feature. However: it is not well documented; it is limited in functionality; its only management UI is the Django admin; it is not flexible and only supports 2U's original use cases; and it depends on various other apps like a marketing site, ecommerce, etc.
Removal
The entire Course Discovery microservice and the corresponding code in edx-platform that integrates with it. However, see below - this is not going away anytime soon.
Replacement
To rectify this situation, this proposal calls for a series of improvements to the Open edX core (edx-platform and openedx-learning, etc.) that add a "catalog" feature and focus on improving platform extensibility, so that the kitchen sink of features listed above could be largely built using plugins. At the same time, the Discovery service would be officially deprecated.
New Flexible Catalog App
To replace one of the core features of the Discovery Service, a new "Catalog" application should be built into the core platform. The purpose of this app is to provide a unified API for answering the question of "What courses are available to me?". It should be extensible using plugins so that things like Subscriptions and Enterprise Licensing can be accounted for.
This catalog app may also have a flexible concept of "programs" which are essentially groups of related courses, but where each program can define requirements like "pass course A then B, then either C and/or D in order to complete the program." Significant product research and planning is required to define such requirements in more detail.
The catalog app should be developed with "micro-courses" in mind as well - courses significantly smaller than the average MOOC and more like a "Section" of today's typical courses.
The detailed design of the catalog app will happen in future ADRs (and/or an OEP) if this DEPR proposal is accepted. There is already a draft of one initial ADR to add an edx-platform representation of "Catalog Courses".
Extending Core Models
Many of the other "kitchen sink" features listed above were developed in the Discovery service because there wasn't a better place to put them at the time. Instead of finding the least bad place to put specialized functionality like this, we should modify the platform so that the core models can be extended by plugins.
For example, a plugin should be able to add custom fields to the
UserProfilemodel, and have these fields (optionall) show up anywhere user profiles are seen or edited. Likewise the core Course model should be extensible, so a plugin could add (e.g.) a "Price" field to each course and choose to have it show up in the catalog API data, the default catalog UI (when user's browse for courses), and the default course "Advanced Settings" UI. The Open edX Hooks Extension Framework already provides some mechanisms for this, but others will need to be proposed in future ADRs/OEPs.Roles & Permissions
Achieving a good implementation of a core catalog app and extensible core models will almost definitely depend on a major refactor of roles and permissions, the lack of which has been a long-standing pain point in Open edX.
Deprecation
Deprecating the Discovery Service now means three things:
First, proposing any new functionality for the Open edX Project whose implementation depends on the Discovery Service would be rejected.
Second, maintenance of the Discovery Service would become entirely the responsibility of those organizations that [still] use it (e.g. 2U).
Third, eventually the corresponding hooks and integration code in
edx-platform(e.g. sync code, "programs cache, etc.) would be removed. This is likely many years away (see below). However, if this DEPR is accepted, we will make an effort to flag all of this code and corresponding APIs so that it's clear it will be eventually removed, and should not be used for implementing new functionality.Migration
For any functionality that currently depends on the Discovery Service (e.g. "Programs"), there will be no effort to remove corresponding integration code from edx-platform until an alternative approach along the lines proposed has been developed, stablized, and released for enough time that organizations have a reasonable migration path. Details of that will depend on the implementation. Those who currently depend on the discovery service for functionality critical to their organization may continue to use it for foreseeable future, provided they are able to contribute to its ongoing maintenance.
The goal of this DEPR proposal is not to take the discovery service away as soon as possible, but to focus community resources on developing a better alternative, and to encourage stakeholders to avoid building any new features on top of the discovery service from this point forward.
The text was updated successfully, but these errors were encountered: