Time-related problems inside Virtual Box, collection of possible solutions

[lenaschimmel]

There are several problems when running Home Assistant Operating System inside Virtual Box:

1. time based automations triggering late (minutes, hours, maybe never)
2. sensor.date_time_iso from platform time_date is only updated infrequently (mostly alternating between 3 and 5 minutes between updates) and values were off (by random amounts of several minutes)
3. in pyscript, task.sleep(2) returned after 2 seconds most of the time, but sometimes slept for more than 30 seconds or even 10 minutes. On average, 100 calls to task.sleep(2) took about 600 seconds, instead of the expected 200 seconds.
4. stalls, crashes and boot failures caused by rcu_sched, which may have led to file system corruption in some cases

I only experienced symptoms 1, 2 and 3, and for me all of them were solved by setting Paravirtualization = None in Virtual Box.

I did not yet come across reports of the symptoms 2 and 3, but since they might have the same cause, I don't think it's particularly helpful to add more details on them.

I found several issues relating to symptoms 1 and 4, which I'll summarize below. I suspect that all four symptoms may be caused by the same root cause, since they are fixed by the same workaround.

Possible workarounds / solutions

Setting Paravirtualization = None in Virtual Box

• worked for me and most other users
• very easy fix, fully revertable
• could impact performance negatively
• is not documented in the installation instructions for Virtual Box
• I made a PR to include this in the installation instructions, but it was rejected

Include virtual box additions in the vdi

• seems to be non-trivial
• @agners has started to work on this but not sure if there was progress
• unsure if that would even solve the problem (see last summarized issue at the bottom of this post)

Enable High Precision Event Timer (HPET) in Virtual Box

• solved the rcu_sched for someone, but probably not related to Home Assistant
• probably no one tried yet if it fixes the various time problems with Home Assistant
  • I might find some time to try this on my machine, but maybe others could try it as well?
• very easy fix, fully revertable
• should not have negative performance impact, in contrast to Paravirtualization = None
• also needs to be applied by the user
• not sure if this would be considered a workaround or a solution, and if it could be added to the install documentation

Add code to to detect the problem

• not sure if there is an easy, reliable way to detect this
• if detected, a more meaningful error message could be generated, possibly pointing the user to the existing issues and/or a documentation of the known workaround(s)

Other issues and discussions about the same / similar issues

rcu_sched detected stalls on VirtualBox #1705

• several theories about hardware problems
• theory that timer issues might be the root cause
• for some users it is fixed by setting CPU count from 2 to 1, or from 1 to 2
• Paravirtualization = None / KVM was mentioned, but not sure if it solve the issue for anyone

Home assistant 7.2 not working in Virtual box #1737

• similar error as the one above
• Paravirtualization = None / KVM was mentioned
• Paravirtualization = KVM did not solve it
• but Paravirtualization = None solved it

Virtualbox time drift due to missing guest additions #1540

• theory that adding VirtualBox guest additions would fix it
• @agners started working on this
• discussion mentions workaround: Paravirtualization = None
• Also mentions Paravirtualization = KVM, but that did not solve it

Automations triggering late (issue #71278 in home-assistant/core)

• mentions workaround: Paravirtualization = None
• Links to 7 community discussions and 1 reddit post with similar problems
• issue is closed and locked, so that I can't comment there

rcu_sched detected stalls on CPUs/tasks: linux guest and host ?ryzen issue (Ticket #20131 in VirtualBox)

• This issue is not about Home Assistant, but rcu_sched was also mentioned in the Home Assistant issues above
• Reporter said: "Adding guest additions didn't ultimately fix the problem - it just delayed the onset of the problem."
• Someone else said "vboxmanage modifyvm foo --hpet on" solved it. (According to the manual, this enables (...) a High Precision Event Timer (HPET) that can replace a legacy system timer)