Dual boot support for bootupd with static GRUB config installations #530

AdamWill · 2024-01-24T18:49:01Z

bootupd & static GRUB configs don't support dual boot yet.

Previous title: Install of Silverblue installer image with bootupd enabled fails due to grub2-mkconfig failure

Describe the bug
After https://pagure.io/workstation-ostree-config/c/ad61c4d56e45eef8a7ac8af47f72fb4f463892e9?branch=main landed and the openQA ostree installer test started generating ostrees with bootupd, install of the Silverblue image built around the ostree started to fail. The ostree admin instutil set-kargs command that is run to configure the kernel args on the bootupd path fails, apparently because a grub2-mkconfig command it runs fails. It does not tell me why the grub2-mkconfig run fails.

To Reproduce
Please describe the steps needed to reproduce the bug:

Get or build a Silverblue installer image containing an ostree with bootupd enabled, like this one
Try and install it. Observe the failure during post-install bootloader configuration

Expected behavior
The install should complete successfully.

Additional context
I tried reproducing the issue locally, then switching to a VT and running the failing ostree admin instutil set-kargs command locally, after chroot /mnt/sysroot. That does indeed fail, with the error error: Bootloader write config: grub2-mkconfig: Child process exited with code 1. It does not pass along anything about the grub2-mkconfig failure, and /mnt/sysroot is read-only so I can't do any hacks to try and get the output from grub2-mkconfig dumped anywhere, really. If I run the ostree command with -v it gives me more verbose output before the grub2-mkconfig command fails, but nothing more useful about the actual failure:

OT: using fuse: 0
OT: Deployment (long id) unlocked=0
OT: Using bootloader configuration: auto
OT: Using bootloader: OstreeBootloaderGrub2
OT: Using grub2-mkconfig chroot: (null)

@travier has reverted the bootupd change once more for now, but we should figure out the failure so it can be turned on. Again.

The text was updated successfully, but these errors were encountered:

travier · 2024-01-25T09:45:10Z

Looks like it failed because it used a previous commit without bootupd installed thus anaconda could not detect it and use the new bootupd aware installation method.

We'll give it a try again now that the compose build failures should be fixed.

travier · 2024-01-25T09:46:08Z

Audit PR: https://src.fedoraproject.org/rpms/audit/pull-request/11
Bootupd PR: https://pagure.io/workstation-ostree-config/pull-request/453

AdamWill · 2024-01-25T17:01:59Z

No, that diagnosis definitely does not sound right.

I checked the last pass and first fail in openQA when this started happening, and bootupd being present in the ostree is exactly the difference: bootupd was not in the ostree in the last pass, and was in the ostree in the first failure.

anaconda uses exactly the presence of bootupd in the ostree as its basis for deciding whether to go down this path at all: rhinstaller/anaconda@8e690d5

if bootupd isn't there, it won't go down this path, it will use its conventional bootloader install code.

And the openQA test builds an ostree before it builds the ostree installer image. It cannot really 'use an old ostree' like the compose process can, it doesn't have any. The only ostree it can use is the one it builds. If that step fails, the test will die and never reach the ostree installer build phase.

AdamWill · 2024-01-25T17:03:44Z

The ISO I uploaded for testing is from this openQA test. If you check its ostree build log, you will see bootupd listed there.

travier · 2024-01-26T10:55:56Z

Ah thanks, I had missed that this was from an image built during the openqa test run. Then this likely means that we'll have to do another Anaconda fix. CC @jkonecny12

jkonecny12 · 2024-01-26T14:09:41Z

I see. I wonder how to resolve this. I guess we need to add a check if it is container source?

jkonecny12 · 2024-01-26T15:04:37Z

I wonder @travier should we use the bootupd for the ostree standard installation or only for containers?

jkonecny12 · 2024-01-26T15:51:54Z

Also would it be possible to remove bootupd from the ostree base layer? What is the purpose.

We are currently over capacity, if possible we need to minimize our work.

travier · 2024-01-29T09:52:36Z

I wonder @travier should we use the bootupd for the ostree standard installation or only for containers?

We don't have that option as the same manifests are used for both.

Also would it be possible to remove bootupd from the ostree base layer? What is the purpose.

We are currently over capacity, if possible we need to minimize our work.

The goal of https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd is to fix:

We've reverted the change so it's not in right now, but it's a core part of the Ostree Native Containers work.

jkonecny12 · 2024-01-30T10:28:23Z

@AdamWill could you please provide us the ISO which were failing? We need something to debug the issue.

AdamWill · 2024-01-30T16:24:06Z

I already did, in the initial issue description, where it says "Get or build a Silverblue installer image containing an ostree with bootupd enabled, like this one".

jkonecny12 · 2024-01-30T16:34:24Z

I see now, I guess I'm just blind :( . Thanks!

travier · 2024-01-31T10:33:48Z

DEBUG:anaconda.modules.payloads.payload.rpm_ostree.rpm_ostree:Internal mounts are set to: ['/mnt/sysroot/usr', '/mnt/sysroot/dev', '/mnt/sysroot/proc', '/mnt/sysroot/run', '/mnt/sysroot/sys', '/mnt/sysroot/var', '/mnt/sysroot/boot', '/mnt/sysroot/home', '/mnt/sysroot/sysroot']
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-PrepareOSTreeMountTargetsTask-1 (140269177538240)                                                                                                                                                                                                                        
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/10.                                                          
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/11.                                                                                                                                                                                                                          
DEBUG:dasbus.connection:Publishing an object at /org/fedoraproject/Anaconda/Modules/Payloads/Task/12.                                                                                                                                                                                                                          
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-CopyDriverDisksFilesTask-1 (140269177538240)                                                          
INFO:anaconda.modules.common.task.task:Copy driver disks files                                                                                                 
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-CopyDriverDisksFilesTask-1 (140269177538240)                                                             
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-ChangeOSTreeRemoteTask-2 (140269177538240)                                                                                                                                                                                                                            
INFO:anaconda.modules.common.task.task:Change OSTree remote                                                                                                     
INFO:anaconda.core.threads:Thread Done: AnaTaskThread-ChangeOSTreeRemoteTask-2 (140269177538240)                                                                                                                                                                                                                               
INFO:anaconda.core.threads:Running Thread: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                
INFO:anaconda.modules.common.task.task:Configure OSTree bootloader
INFO:program:Running in chroot '/mnt/sysroot'... bootupctl backend install --auto --write-uuid --device /dev/vda /                                                                                                                                                                                                             
INFO:program:Installed: grub.cfg                                                                                                                                
DEBUG:program:Return code: 0                                                                                                                                    
INFO:program:Running in chroot '/mnt/sysroot'... ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw
INFO:program:error: Bootloader write config: grub2-mkconfig: Child process exited with code 1                                                                  
DEBUG:program:Return code: 1   
INFO:anaconda.core.threads:Thread Failed: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                                                                                                                                                                                
ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-ConfigureBootloader-1 has failed: Traceback (most recent call last):                              
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 280, in run                                                                                                                                                                                                                                      
    threading.Thread.run(self)                                                  
  File "/usr/lib64/python3.12/threading.py", line 1010, in run                                                                                                  
    self._target(*self._args, **self._kwargs)                                                                                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 94, in _thread_run_callback                                          
    self._task_run_callback()                                                   
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 107, in _task_run_callback                                           
    self._set_result(self.run())                                               
                     ^^^^^^^^^^                                                
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 505, in run                                                                                                                                                                                                  
    self._set_kargs()                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 568, in _set_kargs                                                                                                                                                                                           
    safe_exec_with_redirect("ostree", set_kargs_args, root=self._sysroot)                                                                                      
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 54, in safe_exec_with_redirect                                                                                                                                                                               
    raise PayloadInstallationError(                                            
pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw' exited with the code 1.

INFO:anaconda.core.threads:Thread Done: AnaTaskThread-ConfigureBootloader-1 (140269177538240)                                                                                                                                                                                                                                  
WARNING:dasbus.server.handler:The call org.fedoraproject.Anaconda.Task.Finish has failed with an exception:                                                                                                                                                                                                                    
Traceback (most recent call last):                                             
  File "/usr/lib/python3.12/site-packages/dasbus/server/handler.py", line 455, in _method_callback                                                                                                                                                                                                                             
    result = self._handle_call(                                                
             ^^^^^^^^^^^^^^^^^^                                                
  File "/usr/lib/python3.12/site-packages/dasbus/server/handler.py", line 265, in _handle_call                                                                                                                                                                                                                                 
    return handler(*parameters, **additional_args)                                                                                                             
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                             
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task_interface.py", line 114, in Finish                                                                                                                                                                                                             
    self.implementation.finish()                                               
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 173, in finish                                                                                                                                                                                                                       
    thread_manager.raise_if_error(self._thread_name)                                                                                                           
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 171, in raise_if_error                                                                                                                                                                                                                           
    raise exc_info[1]                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/core/threads.py", line 280, in run                                                                                                                                                                                                                                      
    threading.Thread.run(self)                                                 
  File "/usr/lib64/python3.12/threading.py", line 1010, in run                                                                                                 
    self._target(*self._args, **self._kwargs)                                                                                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 94, in _thread_run_callback                                                                                                                                                                                                          
    self._task_run_callback()                                                  
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/common/task/task.py", line 107, in _task_run_callback                                                                                                                                                                                                           
    self._set_result(self.run())                                               
                     ^^^^^^^^^^                                                
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 505, in run                                                                                                                                                                                                  
    self._set_kargs()                                                          
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 568, in _set_kargs                                                                                                                                                                                           
    safe_exec_with_redirect("ostree", set_kargs_args, root=self._sysroot)                                                                                      
  File "/usr/lib64/python3.12/site-packages/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py", line 54, in safe_exec_with_redirect                                                                                                                                                                               
    raise PayloadInstallationError(                                            
pyanaconda.modules.common.errors.installation.PayloadInstallationError: The command 'ostree admin instutil set-kargs rhgb quiet root=UUID=d2d7152e-6bdb-4f1d-affc-83f2b7114103 rootflags=subvol=root rw' exited with the code 1.

travier · 2024-01-31T10:37:05Z

$ grub2-mkconfig
### BEGIN /etc/grub.d/15_ostree ###
/etc/grub.d/15_ostree: line 31: /etc/default/grub: No such file or directory

😕

travier · 2024-01-31T10:38:27Z

[anaconda root@fedora /]# rpm -ql grub2-tools | grep default
/etc/default/grub
[anaconda root@fedora /]# rpm -qf /etc/default/grub
grub2-tools-2.06-110.fc40.x86_64

🤔

travier · 2024-01-31T10:42:05Z

$ ostree ls --repo=repo 50ddc44f72905d9b463ffdf27119e670d7911629ae143644ab3809307a8746ff /usr/etc/default
d00755 0 0      0 /usr/etc/default
-00644 0 0    123 /usr/etc/default/useradd

🤔 🤔 🤔

travier · 2024-01-31T10:55:57Z

OK, this is gone in latest Rawhide commits as well so this is not a bootupd issue.

travier · 2024-01-31T11:02:44Z

Somehow, this file is not installed anymore: https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/grub2.spec#_436

travier · 2024-01-31T11:07:21Z

Hum, it's not there on my F39 system either.

travier · 2024-01-31T11:17:00Z

OK, it's anaconda writing this file in the GRUB2 installation path and likely not in the bootupd one: https://github.com/rhinstaller/anaconda/blob/c0d803edeeb8d7797492250353e52180f5ad6a33/pyanaconda/modules/storage/bootloader/grub2.py#L253

travier · 2024-01-31T11:17:05Z

CC @jkonecny12

travier · 2024-01-31T11:23:29Z

I suspect rhinstaller/anaconda#5350

travier · 2024-01-31T13:10:45Z

OK, so we need to setup this config but only if the legacy ostree GRUB hooks/config are there (the ones that we want to remove after this change lands and people have time to update their bootloaders).

travier · 2024-01-31T14:31:18Z

Or we can update this condition / check here: https://github.com/ostreedev/ostree/blob/main/src/boot/grub2/grub2-15_ostree#L31

With the new bootupd installation path in Anaconda, the `/etc/default/grub` config file is not written anymore as we are only using BLS configs with new enough bootloaders. We thus don't need to generate (duplicated) legacy boot entries. We still need to keep this logic in place in Atomic Desktops (Silverblue, etc.) until we've actually landed bootupd there and forced a bootloader update for everybody. See: fedora-silverblue/issue-tracker#530 See: fedora-silverblue/issue-tracker#120 See: https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd

travier · 2024-01-31T14:42:38Z

Not 100% sure yet if my understanding is correct but made: ostreedev/ostree#3150

AdamWill · 2024-01-31T16:59:20Z

you're correct that anaconda writes it, here. GRUB2.write_defaults() is called by GRUB2.write_config(), which is called by GRUB2.write(), which is called by InstallBootloaderTask.run(), using self._storage.bootloader as the bootloader class (so only when that is a GRUB2 instance). That's called by BootloaderModule.install_bootloader_with_tasks(), which has claimed to be "just a temporary method" since March 2019. That's called by BootloaderInterface.InstallBootloaderWithTasks(), which is called by RunInstallationTask._prepare_installation() (using an internal method). See? Simple...:P

The thing that is InstallBootloaderTask._storage is probably an instance of InstallerStorage, whose bootloader property is the result of BootLoaderFactory.create_boot_loader(), which via a slightly funky bit of logic usually winds up autodetecting the bootloader class via get_class_by_platform(): BootloaderModule winds up trying to do BootLoaderFactory.set_default_class("DEFAULT") via BootloaderType.DEFAULT, which the factory's get_class_by_name doesn't have a mapping for, so it returns None, which means BootLoaderFactory.get_class() falls through to get_class_by_platform()...all unless the user specifies extlinux or sdboot via configuration.

AAANYHOOO, what's actually happening here is that all of this is getting skipped because we hit a check in InstallBootloaderTask.run() (remember, that's the thing that calls write(), that calls write_config(), that writes the file). There's a check if self._payload_type == PAYLOAD_TYPE_RPM_OSTREE and have_bootupd(self._sysroot): which bails out of run() before it calls self._bootloader.write().

travier · 2024-01-31T18:00:43Z

Thanks for digging 🕳️ into this!

travier · 2024-02-01T11:57:43Z

I've verified that ostreedev/ostree#3150 fixes the installation

travier · 2024-02-01T12:08:22Z

And I've tested basic bootupd function as well! Will try an update now.

cgwalters · 2024-02-01T13:47:16Z

Adding bootupd to a payload currently also defaults to --with-static-configs which I think is really the correct way to do things by default for dedicated Linux boxes. However...there may be people relying on more esoteric features of dynamic grub configs. For example, grub does this thing where it scans all your block devices and is like "Oh I see a Haiku install here, I'll make an entry for that" etc. And similarly for Windows I think.

But IMO, the right way to handle Windows at least is via the EFI bootloader, not grub.

Having new desktop (silverblue, etc.) installs use bootupd in this way will be awesome because it will align everything and work the same way as FCOS and how bootc does it.

But...transitioning old installs seems tricky and potentially risky.

travier · 2024-02-01T14:04:56Z

Considering that we "already" have dual booting issues on Silverblue (#284), I'm tempted to say that we'll not fix this for Fedora 40. We should be able to document adding a Windows boot entry to the static GRUB config for new installations.

As we're not force upgrading users (yet), systems that were installed before will keep the dynamic GRUB configs for now.

travier · 2024-02-05T14:55:30Z

I've tested installing Fedora Silverblue 37, updating all the way to 39 then rebasing to Rawhide and adopting using bootupd adopt-and-update.

jkonecny12 · 2024-02-06T09:29:52Z

Great work @travier! Sorry that I wasn't able to help you more :(.

I wonder what is the correct behavior here... Should Anaconda create the /etc/default/grub or not? For OSTree based installations I would guess that this file shouldn't be created or it should be handled by bootupd?

travier · 2024-02-06T13:05:24Z

For bootupd installations, we don't need anaconda to create this file right now as we will use a static GRUB config and BLS to get the ostree boot entries.

This will not generate boot entries for Windows, etc. for dual boot but we already don't have that working well so it's not a great loss.

I'll try to document how to manually add boot entries for Windows & Co.

travier · 2024-05-30T14:49:09Z

Related to ostreedev/ostree#3198 for GRUB static config + dual boot with composefs enabled.

osalbahr · 2024-10-31T20:37:03Z

Similar error happened to me by following QA:Testcase dualboot with macOS with the Silverblue 41 ISO (using Fedora Media Writer) on an Intel-based MacBook Pro (2017).

I am not sure if this is a separate issue. It looks the closest to grub-mkconfig fails and aborts installation of Fedora 33 #96

The following error occurred while installing the payload. This is a fatal error and installation will be aborted.

 The command 'ostree admin --sysroot=/mnt/sysimage deploy --os=fedora fedora:fedora/41/x86_64/silverblue' exited with the code 1.

travier · 2024-11-05T11:21:30Z

Make sure to use a distinct, empty /boot & /efi partitions for the Silverblue installation.

AdamWill added the bug Something isn't working label Jan 24, 2024

travier added rawhide f40 Related to Fedora 40 enhancement New feature or request kinoite Also affect Fedora Kinoite fedora-change Needs a Fedora Change and removed bug Something isn't working labels Jan 25, 2024

miabbott mentioned this issue Jan 29, 2024

support bootupd for bootloader management/updates fedora-iot/iot-distro#6

Open

travier mentioned this issue Jan 31, 2024

grub2-15_ostree: Graceful exit if /etc/default/grub doesn't exist ostreedev/ostree#3150

Merged

travier self-assigned this Feb 7, 2024

travier changed the title ~~Install of Silverblue installer image with bootupd enabled fails due to grub2-mkconfig failure~~ Dual boot support for bootup & static GRUB config installations Feb 26, 2024

travier changed the title ~~Dual boot support for bootup & static GRUB config installations~~ Dual boot support for bootupd with static GRUB config installations Feb 26, 2024

travier added f41 Related to Fedora 41 and removed f40 Related to Fedora 40 labels Mar 18, 2024

travier removed their assignment Apr 23, 2024

travier mentioned this issue Sep 19, 2024

Fedora 41 Readiness Tracker ublue-os/main#634

Open

23 tasks

Dual boot support for bootupd with static GRUB config installations #530

Dual boot support for bootupd with static GRUB config installations #530

Comments

AdamWill commented Jan 24, 2024 • edited by travier Loading

Previous title: Install of Silverblue installer image with bootupd enabled fails due to grub2-mkconfig failure

travier commented Jan 25, 2024

travier commented Jan 25, 2024 • edited Loading

AdamWill commented Jan 25, 2024

AdamWill commented Jan 25, 2024

travier commented Jan 26, 2024

jkonecny12 commented Jan 26, 2024

jkonecny12 commented Jan 26, 2024

jkonecny12 commented Jan 26, 2024

travier commented Jan 29, 2024

jkonecny12 commented Jan 30, 2024

AdamWill commented Jan 30, 2024

jkonecny12 commented Jan 30, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024 • edited Loading

travier commented Jan 31, 2024 • edited Loading

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Jan 31, 2024

AdamWill commented Jan 31, 2024

travier commented Jan 31, 2024

travier commented Feb 1, 2024

travier commented Feb 1, 2024

cgwalters commented Feb 1, 2024

travier commented Feb 1, 2024 • edited Loading

travier commented Feb 5, 2024

jkonecny12 commented Feb 6, 2024

travier commented Feb 6, 2024

travier commented May 30, 2024

osalbahr commented Oct 31, 2024 • edited Loading

travier commented Nov 5, 2024

AdamWill commented Jan 24, 2024 •

edited by travier

Loading

travier commented Jan 25, 2024 •

edited

Loading

travier commented Jan 31, 2024 •

edited

Loading

travier commented Jan 31, 2024 •

edited

Loading

travier commented Feb 1, 2024 •

edited

Loading

osalbahr commented Oct 31, 2024 •

edited

Loading