Product Discovery - Aspects. MVP

[jmakowski1123]

Panels for quick reads of real-time learner and course data:

• instructor view of real-time learner progress for all learners in a course
• learner view of real-time self progress
• aggregated stats of all courses

[github-actions]

Thanks for your submission, @openedx/open-edx-project-managers will review shortly.

[jmakowski1123]

Next steps: Jenna and Brian can use this ticket to refine OARS MVP

[jmakowski1123]

Collaborators:
-Felipe to help bring the big needs from Spanish proposal into fine-grained use cases

[e0d]

@felipemontoya I understand from the DWG meeting that you are planning to adopt OARs as the data platform for the Spanish Consortium project. So, their campus related analytics would be based on that foundation. Is that accurate?

[felipemontoya]

Yes that is accurate. We started to put engineering hours into making OARS land a V1 that is suitable for prod environments.

I need to transfer the use cases from the Spanish tender here and Brian told me it would be good to have this by the end of the month because it will be compiled with use cases from other sources.

[felipemontoya]

Abstract

The Spanish OpenedX for Campus Project aims to improve the analytics capabilities of the Open edX platform by supporting the OARS project and creating custom metrics and dashboards for specific use cases. The project requests aggregated data dashboards that show metrics grouped by category and filtered by courses and time, individual access dashboards that allow students to compare themselves with other students in the course, and other custom metrics that can be created from xblocks or plugins. The project also plans to write optional extensions that can plug into OARS without forking the underlying plugins.

Context & Background

The Open edX platform has used Insights in the past to calculate analytics for educators who want to examine one course at a time. However, Insights has some limitations when it comes to analyzing multiple courses together, more granular data, or individual students compared with the course average.

Insights also relies on batch data processing, which prevents near-real-time analytics from reaching the educators for timely intervention.

Scope & Approach

The Spanish consortium tender requested a mechanism to publish real-time log events in a data lake for big-data processing. This coincided with the launch of the Open edX Analytics Reference System (OARS) project. As the developers selected by the consortium, we decided to support the OARS project and try to fit our requirements into its capabilities or extensibility options.

However, we understand that some of the dashboard, graphs and metrics that we outline in the next section may be too specific for OARS. Therefore, we plan to write them as optional extensions that can plug into OARS without forking the underlying repositories. It is possible that more extension points will be required to achieve some of this goals.

Requested Dashboard and Metrics

1. Aggregated data dashboards:

A series of dashboards with metrics that can be filtered to one or more courses, with start and end times. Filtering this should make all the stats below be updated.

The metrics that follow would be grouped by category. Here are some examples of metrics. The complete list is published in the doc: https://docs.google.com/document/d/1NPYkn7QCJ1f6JUSz0Wr3uCjswLYFOI1o6BKXjAjk3Aw/edit#

• Course Data. E.g: # of courses, # of videos per course, total min of video, ....

• Enrollments data. E.g: total enrollments, active enrollments, enrollments per mode, ....

• Student activity. E.g: students that interacted with courseware, students that tried an exam, unique minutes of video watched, repeated minutes of video, attempts at solving a problem, ....

• Demographic data. Eg: country graph, pie chart of gender distribution, age pyramid.

• temporal evolution data. E.g: time series with data of enrollments, active participants in course, verified enrollments, video watchs, forum posts, ...

• course comparison data. E.g: bar graphs that compare all the courses in the filter in the following parameters; enrollments, users with activity, users that tried exams, users with passing grades, verified enrollments, videos watched, unique minutes of video watched, average final grades, ....

2. Individual access dashboards:

This series of dashboards have in essence the same type of data from the aggregated panels, but in this case the idea is that student users are able to compare themselves with the other students enrolled in the course.

For the comparison, we can use different statistical performance metrics such as quartiles. We can also calculate a “score” based on participation and use it as a comparison metric.

We can embed these dashboards in the LMS for students to see their own stats and let instructors see the students individually as well.

3. Other custom metrics:

The requirement includes the capacity to create metrics for things that exist only in the form of xblocks or plugins. We believe that this would be too specific for this description, but we include the capacity to publish events from a plugin/xblock that can be then further processed by logic also residing in the same or a different plugin.

An example of this is the metric of downloaded course material, which requires the presence of a Course Material tab in the course most likely to be implemented as an xblock.

Milestones and/or Epics

Milestone 1: reaching a usable OARS V1.
Milestone 2: extensibility of OARS through plugins.
Milestone 3: Definition of common metrics and dashboards.
Milestone 4: Creation of Spanish Consortium specific dashboards/metrics in open sourced plugins.

Named Release

Although the first milestone of reaching OARS V1 is likely to be reached short after the publication of Palm. We expect to have something more robust by Quince.

Proposed By

Spanish OpenedX for Campus Project

cc: @bmtcril @Ian2012

[jmakowski1123]

@felipemontoya There are some metrics from the Spanish use cases that require some additional context. I've transferred into an excel sheet and added some extra columns. Column K indicates where we need some additional clarification. Column L indicates some scope restrictions from Brian. Thought we could start by sharing notes/comments in here asynch, and we can always schedule a follow up call if we need to chat through some of these use cases in more detail! cc @bmtcril https://docs.google.com/spreadsheets/d/1_lqmXI33Nlb6C37craFrjpD64Od90IX2ev5vI8NAJiM/edit#gid=1943589061

[felipemontoya]

I commented on the ones I had clarity. I'll ask Nacho more info on the ones missing.

[jmakowski1123]

Status update 7/20: Market research complete and use cases have been documented, organized and prioritized: https://docs.google.com/spreadsheets/d/1_lqmXI33Nlb6C37craFrjpD64Od90IX2ev5vI8NAJiM/edit#gid=0

Next step: Create an MVP product spec that includes:

-Admin dashboard experience that includes basic reports on instance/org health, growth, stats, and what those reports include
-Course team dashboard experience that includes reports on course data, learner progress, and to identify at-risk learners, and what those reports include
-Course team experience to pull reports at the problem/assessment level, and what those reports include

Timeline: Drafted by July 28 and presented to the first August Data WG

[jmakowski1123]

Update 8/15 - Completed a draft MVP spec for reports to be included in the MVP for admins and educators: https://openedx.atlassian.net/wiki/spaces/OEPM/pages/3834281985/DRAFT+Aspects+V1+Product+Requirements

Next step: Define MVP requirements for report visualization (MVP will probably be limited to leveraging visualizations currently available via Superset) and report dashboard access.

[sarina]

@jmakowski1123 I found myself in this ticket for some reason, and I notice the ticket is still open although it was moved to Done on the board. I think it would be clearer if tickets marked as Done in the roadmap were closed. If you agree I can set up that automation, but we'll need to manually close tickets that are already done.