Skip to content

Commit

Permalink
[Proposal] Improve FREEZING mechanisms and add Representation dep…
Browse files Browse the repository at this point in the history
…recation

Situation
=========

We have lately seen on some LG and Philips TVs what we call "infinite
`FREEZING`" occurences: the playback position (the `HTMLMediaElement`'s
`currentTime` property) was not advancing and most of the time video
media was not playing either (though in some occurences audio was)
despite having largely enough media data in media buffers. Sadly our
usual trick to restart playback, seeking close to the current position,
didn't seem to have an effect.

Most of those freezing cases were happening as playback switched from a
video quality to another (sometimes both had to be specific, sometimes
only the destination one had an impact).

It is probably better to detect which situations causes which problems on
which devices to work-around it either in the RxPlayer (when for example
we KNOW that a device has issues with a given codec in general) or inside
the application (when we only reproduced it on a specific application and
there's many unknowns left). Yet we found out after encountering issues
again and again, especially on some smart TVs, that it may not be
realistic to catch in advance (before production) all potential breakages
that could exist.

So instead, we thought about the possibility to define an heuristic
detecting that we're probably encountering this type of issue, to then
work-around it automatically.

Rough idea of the solution
==========================

So, the idea is to detect when a `FREEZING` case occurs right when
switching from a given video/audio quality to another. When that's the
case:

1. We first try to "un-freeze" playback, by e.g. performing a small seek.
   This is what we already do today in `FREEZING` cases.

2. If that doesn't work (we're still freezing), I added here a second
   mechanism: we "deprecate" (the name can change) the corresponding new
   quality played, so it isn't played anymore in most cases.

So basically once we see that switching to another quality lead to
`FREEZING`, we avoid playing that quality for the rest of playback.

Unlike other scenarios where we prevent playback of qualities (in the
RxPlayer called `Representation`), here the deprecation is only
considered if there's other, non-deprecated, qualities we can play on the
current track: meaning that if all available qualities are deprecated, we
will play them again. This is to protect against errors in cases where
all qualities would be deprecated: because our heuristic is not 100% sure
science, we would here prefer there to keep playback potentially
happening (the alternative would have been stopping playback on an
error).

As of now, the deprecation has no impact on the API: corresponding
`Representation` are still returned as if they are playable through our
`getVideoTrack` API for example, and the user can still choose to call
`lockVideoRepresentation` on deprecated `Representation` without knowing.
As of now, the "main thread" part of the code isn't even aware of
deprecation happening, for it we're just curiously never seem to be
playing some `Representation`. This also means that our `DEBUG_ELEMENT`
feature won't even notify for now if there's deprecation happening.

This may change in the future, but for now, this is because I did not
want to complexify this experiment too much.

PS: I like this "deprecation" concept, as it seems to me to be portable
for much more future cases where we would prefer not playing qualities
because they seem to present issues (e.g. if `MEDIA_ERR_DECODE` or
`BUFFER_APPEND_ERROR` errors seem to only happen with some
`Representation`).

Bonus: also reloading in other cases
====================================

There are several other posibilities for infinite `FREEZING`: Period
changes gone wrong, random decoding issues, etc.

To also provide a better solution for those cases, I choose to "reload"
(in the RxPlayer, "reloading" means removing and re-creating all the
media buffers, leading to a temporary black screen) if the initial
"un-freezing" attempt (the seek-based one) didn't have any effect.

Unlike when the freezing seems to be linked to a quality change though,
there's no deprecation happening in that case.

Implementation
==============

FreezeResolver
--------------

I renamed the very-specific `DecipherabilityFreezeDetector` class - which
only handled freezes linked to DRM - into a more general `FreezeResolver`
class, which will now perform all un-freezing attempts (even the seekinf
one, which previously happened in the `RebufferingController` in main
thread whereas it's now in `core` - thus optionally running in our
`Worker`).

The `FreezeResolver`'s `onObservation` method has to be called even when
playback goes well, at each produced media observation, because it needs
to construct its history of played segments (see next chapter).

Quality switch detection
------------------------

Knowing whether we switched from a given media quality to another is not
straightforward and we can never really know for sure as there's edge
cases where it might be device-specific (e.g. segment replacements,
segment pushing still pending etc.).

However, we have a rough idea by inspecting the `HTMLMediaElement`'s
`currentTime` property (which is roughly the current media position being
played), and the content of our `SegmentInventory` class, which stores
metadata about all media segments available in the buffer.

From there, I maintain a short-term history in the `FreezeResolver` of
what seemed to be the current video AND audio quality played. If a
`FREEZING`, which we do not seem to be able to fix by seeking, seems to
coincide with a quality switch, we propose to deprecate the
`Representation`.

For now this also lead to a `RELOADING` operation, though it may not even
be required in some cases (e.g. replacing segments and seeking in place
might do the trick). Yet reloading should have more chance of fixing it
(even though it leads to a temporary black screen, where a still video
frame would have been less disruptive for users).

Avoiding deprecated `Representation`
------------------------------------

Then, an `AdaptationStream` is able to filter out deprecated
`Representation` (unless it has no choice), when asking the
`AdaptiveRepresentationSelector` (managing our Adaptive BitRate logic)
which `Representation` should be played (though I'm still not sure
whether the `AdaptiveRepresentationSelector` should actually be the one
doing that?).

Where we go from there
======================

This heuristic seems somewhat risky. as we're essentially blacklisting
qualities from ever be playing again on the current content.

So what I thought was to put the deprecation mechanism behind an
experimental `experimental.enableRepresentationDeprecation` `loadVideo`
option, that applications would have to enable (making it also possible
to disable at any time easily through some config).

However the reload mechanism if un-freezing fails seems OK to me for
enabling it by default.

If we do notice clear improvements, we might think of enabling the
deprecation mechanism by default, removing the
`experimental.enableRepresentationDeprecation` option.

Thoughts?
  • Loading branch information
peaBerberian committed Sep 2, 2024
1 parent b428f0e commit 2c9eb7d
Show file tree
Hide file tree
Showing 23 changed files with 733 additions and 242 deletions.
147 changes: 0 additions & 147 deletions src/core/main/common/DecipherabilityFreezeDetector.ts

This file was deleted.

Loading

0 comments on commit 2c9eb7d

Please sign in to comment.