"None cannot be handled by custom serializers" when producing event to Kafka

[timmc-edx]

We have intermittently seen errors involving "None cannot be handled by custom serializers" when producing to the event bus.

Example log message that has been pulled apart for readability, and redacted:

Error delivering message to Kafka event bus.
error=None cannot be handled by custom serializers (and default=None was not set)
full_topic='prod-course-catalog-info-changed'
event_key=None
signal=<OpenEdxPublicSignal: org.openedx.content_authoring.course.catalog_info.changed.v1>
initial_topic='course-catalog-info-changed'
event_key_field='catalog_info.course_key'
event_data={'catalog_info': CourseCatalogData(course_key=CourseLocator('...', '...', '...', None, None), name=None, schedule_data=CourseScheduleData(start=datetime.datetime(...), pacing='self', end=datetime.datetime(...), enrollment_start=None, enrollment_end=None), hidden=False, invitation_only=False)}
event_metadata=EventsMetadata(event_type='org.openedx.content_authoring.course.catalog_info.changed.v1', id=UUID('843664a9-6a0a-11ef-a2ab-8fe8a1efeee6'), minorversion=0, source='openedx/cms/web', sourcehost='ip-...', time=datetime.datetime(...), sourcelib=(9, 11, 0))
event_data_as_json=None
event_metadata_as_json=None

Another event on a different topic and with a different signal:

Error delivering message to Kafka event bus.
error=None cannot be handled by custom serializers (and default=None was not set)
full_topic='prod-learning-course-access-role-lifecycle'
event_key=None
signal=<OpenEdxPublicSignal: org.openedx.learning.user.course_access_role.added.v1>
initial_topic='learning-course-access-role-lifecycle'
event_key_field='course_access_role_data.course_key'
event_data={'course_access_role_data': CourseAccessRoleData(user=UserData(id=..., is_active=True, pii=UserPersonalData(username='...', email='...', name='')), org_key='...', course_key=None, role='data_researcher')}
event_metadata=EventsMetadata(event_type='org.openedx.learning.user.course_access_role.added.v1', id=UUID('6ebe1a20-6a06-11ef-b971-59131e46b91e'), minorversion=0, source='openedx/lms/web', sourcehost='ip-...', time=datetime.datetime(...), sourcelib=(9, 11, 0))
event_data_as_json=None
event_metadata_as_json=None

The traceback is the same:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/edx_event_bus_kafka/internal/producer.py", line 296, in send
    context.event_data_as_json = json.dumps(get_signal_serializer(signal).to_dict(event_data))
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 131, in to_dict
    return _event_data_to_avro_record_dict(event_data, serializers=self.serializers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 76, in _event_data_to_avro_record_dict
    json.dumps(
  File "/usr/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 72, in value_to_dict
    return attr.asdict(value, value_serializer=_get_non_attrs_serializer(serializers))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/attr/_funcs.py", line 71, in asdict
    v = value_serializer(inst, a, v)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 44, in _serialize_non_attrs_values
    raise Exception("None cannot be handled by custom serializers (and default=None was not set)")
Exception: None cannot be handled by custom serializers (and default=None was not set)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/edx_event_bus_kafka/internal/producer.py", line 56, in record_producing_error
    raise EventProductionException(error)
edx_event_bus_kafka.internal.producer.EventProductionException: None cannot be handled by custom serializers (and default=None was not set)

[timmc-edx]

I suspect this is actually a bug in the code that creates the event data in the first place. For example, that course_key=None is very suspicious; a spot-check of successfully sent messages on that topic shows a non-None course key, and the code that sends the event does not indicate that the course key is optional.