Firmware: Add option to extract firmware directly from macOS!

docs/guides/wifi-bluetooth.md

@@ -22,29 +22,31 @@ Refer to the "Updating Kernel" section on your distro's FAQ for instructions if
 
 We now use a script which can help you set up Wi-Fi and Bluetooth. Click [here](../tools/firmware.sh) to download the script.
 
-There are 4 methods supported by this script to get firmware for Linux, named as **Method 1-4** in this guide. Details of each method are given in the [On macOS](#on-macos) and [On Linux](#on-linux) section.
+There are 5 methods supported by this script to get firmware for Linux, named as **Method 1-5** in this guide. Details of each method are given in the [On macOS](#on-macos) and [On Linux](#on-linux) section. **You have to choose any one of the five methods to get firmware as per your choice.**
 
 **Method 1-3** require macOS installed on your Mac. The initial steps of these methods are to be followed on macOS, and later steps have to be followed on Linux. Thus, if you choose one of these methods, you first need to follow the [On macOS](#on-macos) section and then proceed to the [On Linux](#on-linux) section.
 
-**Method 4** does not require macOS, so you can directly follow the [On Linux](#on-linux) section if you choose it.
+**Method 4** also requires macOS, but doesn't have any step to be followed on macOS. So you can directly follow the [On Linux](#on-linux) section if you choose it.
+
+**Method 5** does not require macOS, so you can directly follow the [On Linux](#on-linux) section if you choose it.
 
 !!! Tip "macOS Removed after installing Linux"
-    In case you have removed macOS after installing Linux, and need the firmware, **Method 4** is the only option for you.
+    In case you have removed macOS after installing Linux, and need the firmware, **Method 5** is the only option for you.
 
 ### On macOS
 
 Run the script on the macOS terminal. After you run the script, it will ask you to choose between 3 methods to move firmware to Linux:
 
-=== "Method 1"
-    **Method 1: Run the same script on Linux**
+=== ":fontawesome-brands-apple: Method 1"
+    **Method 1: Copy the firmware to the EFI partition and run the same script on Linux to retrieve it**
 
     If you choose this method, unlike **Method 2** and **Method 3**, you need not have any specific dependency already installed on your Mac. So if you don't want to install any additional software on macOS, this method is the only option for you.
 
     In this method, the script will copy the firmware to your **EFI** partition.
 
     To retrieve the firmware from **EFI** partition in Linux, you shall have to run the same script on Linux. You have 2 options do so, described in detail in [On Linux](#on-linux) section.
 
-=== "Method 2"
+=== ":fontawesome-brands-apple: Method 2"
     **Method 2: Create a tarball of the firmware and extract it to Linux**
 
     If you choose this method, the script will install the following dependencies, if missing, on macOS:
@@ -57,7 +59,7 @@ Run the script on the macOS terminal. After you run the script, it will ask you
 
     Now you have to extract the firmware in the tarball to Linux. The procedure has been described in detail in [On Linux](#on-linux) section.
 
-=== "Method 3"
+=== ":fontawesome-brands-apple: Method 3"
     **Method 3: Create a Linux specific package which can be installed using a package manager**
 
     If you choose this method, the script will install the following dependencies, if missing, on macOS:
@@ -74,17 +76,12 @@ Run the script on the macOS terminal. After you run the script, it will ask you
 
     Now you have to install the package in Linux using your package manager. The procedure has been described in detail in [On Linux](#on-linux) section.
 
-=== "Method 4"
-    **Method 4: Download a macOS Recovery Image from Apple and extract the firmware from there**
-
-    This method does not have any steps to be followed on macOS. See [On Linux](#on-linux) section.
-
 ### On Linux
 
 Once you have run the script on macOS, depending on the method you chose, the steps to be followed on Linux are described below:
 
-=== "Method 1"
-    **Method 1: Run the same script on Linux**
+=== ":fontawesome-brands-linux: Method 1"
+    **Method 1: Copy the firmware to the EFI partition and run the same script on Linux to retrieve it**
 
     Now we need to retrieve the firmware from the **EFI** partition. You further have 2 options to do so:
 
@@ -96,7 +93,7 @@ Once you have run the script on macOS, depending on the method you chose, the st
         bash /path/to/firmware.sh
         ```
 
-        After you run the script, you have to choose the **"Retrieve the firmware from EFI"** option. After choosing that option, the script will install the firmware on Linux.
+        After you run the script, you have to choose the **"Retrieve the firmware from the EFI partition"** option. After choosing that option, the script will install the firmware on Linux.
 
         !!! note
 
@@ -113,11 +110,11 @@ Once you have run the script on macOS, depending on the method you chose, the st
         sudo umount /tmp/apple-wifi-efi
         ```
 
-        After you run the above commands, you have to choose the **"Retrieve the firmware from EFI"** option. After choosing that option, the script will install the firmware on Linux.
+        After you run the above commands, you have to choose the **"Retrieve the firmware from the EFI partition"** option. After choosing that option, the script will install the firmware on Linux.
 
         This option shall be useful if you are unable to copy the script to Linux.
 
-=== "Method 2"
+=== ":fontawesome-brands-linux: Method 2"
     **Method 2: Create a tarball of the firmware and extract it to Linux**
 
     Now we shall extract the tarball of the firmware which was saved in the **Downloads** folder in macOS as `firmware.tar`. In order to do so, copy `firmware.tar` to Linux and extract the firmware to `/lib/firmware/brcm` by running:
@@ -140,7 +137,7 @@ Once you have run the script on macOS, depending on the method you chose, the st
     sudo modprobe hci_bcm4377
     ```
 
-=== "Method 3"
+=== ":fontawesome-brands-linux: Method 3"
     **Method 3: Create a Linux specific package which can be installed using a package manager**
 
     Now we have to install the firmware package which was saved in the **Downloads** folder in macOS. Copy the package to Linux and follow the instructions below, depending on whether you use `apt`, `dnf` or `rpm`:
@@ -187,16 +184,34 @@ Once you have run the script on macOS, depending on the method you chose, the st
 
             Replace `/path/to/firmware_package.pkg.tar.zst` with the actual path of the package. For example, if `apple-firmware-14.5-1-any.pkg.tar.zst` was created in macOS and has been copied to the Downloads folder in Linux, command to be run would be `sudo pacman -U $HOME/Downloads/apple-firmware-14.5-1-any.pkg.tar.zst`
 
-=== "Method 4"
-    **Method 4: Download a macOS Recovery Image from Apple and extract the firmware from there**
+=== ":fontawesome-brands-linux: Method 4"
+    **Method 4: Retrieve the firmware directly from macOS**
+
+    !!! warning "Internet connection may be required for **Method 4**"
+
+        **Method 4** needs certain dependencies to work. If they are missing, you need to have an active internet connection on Linux to download and install them. You can use Ethernet, USB tethering or an external Wi-Fi adapter to get internet. If you are using a customised ISO made for T2 Macs, then most likely those dependencies shall be shipped alongwith the ISO, so in that case internet shall not be required.
+
+    This method does not have any steps to be followed on macOS. So, you have to run the script directly on Linux. After you run the script on Linux, you have to choose the **"Retrieve the firmware directly from macOS**" option.
+
+    If you choose this method, the script will install the following dependencies, if missing, on Linux:
+
+    1. **APFS driver for Linux** - Enables mounting and reading of the macOS Recovery volume, which has the firmware. The source code of the driver which shall be installed by the script is obtained from [here](https://github.com/linux-apfs/linux-apfs-rw).
+    2. **dmg2img** - Converts the macOS Recovery Image obtained to a Linux readable format.
+
+    The script shall automatically detect if any dependency is missing, and if required, will give you the option of installing it. So you need not worry about having missing dependencies.
+
+    Once the script confirms that you have the necessary dependencies installed, it shall automatically install the firmware.
+
+=== ":fontawesome-brands-linux: Method 5"
+    **Method 5: Download a macOS Recovery Image from Apple and extract the firmware from there**
 
-    !!! warning "Internet connection is required for **Method 4**"
+    !!! warning "Internet connection is required for **Method 5**"
 
-        **Method 4** downloads a macOS Recovery image from Apple. So you need to have an active internet connection on Linux. You can use Ethernet, USB tethering or an external Wi-Fi adapter to get internet.
+        **Method 5** downloads a macOS Recovery image from Apple. So you need to have an active internet connection on Linux. You can use Ethernet, USB tethering or an external Wi-Fi adapter to get internet.
 
     This method does not have any steps to be followed on macOS. So, you have to run the script directly on Linux. After you run the script on Linux, you have to choose the **"Download a macOS Recovery Image from Apple and extract the firmware from there**" option.
 
-    If you choose this method, the script will install the following dependencies, if missing, on Linux:
+    If you choose this method, the script will install the following dependencies, if missing, on Linux: 
 
     1. **curl** - Downloads a [python script](https://github.com/kholia/OSX-KVM/blob/master/fetch-macOS-v2.py) which is used to download the macOS Recovery image from Apple.
     2. **dmg2img** - Converts the downloaded macOS Recovery Image to a Linux readable format.
@@ -222,27 +237,3 @@ Dec 24 22:34:20 hostname kernel: brcmfmac: brcmf_c_process_txcap_blob: TxCap blo
 Dec 24 22:34:20 hostname kernel: brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4377/4 wl0: Jul 16 2021 18:25:13 version 16.20.328.0.3.6.105 FWID 01-30be2b3a
 Dec 24 22:34:20 hostname kernel: brcmfmac 0000:01:00.0 wlp1s0f0: renamed from wlan0
 ```
-
-## Fixing unstable WPA2 using iwd
-
-Using iwd is technically not needed for using Wi-Fi. But if you are facing unstable WPA2 issues and have to reload the Wi-Fi driver every time you connect to a WPA2 network, you will have to follow this section. If your connection is stable, you needn't follow this section.
-
-Instructions in this section might be different for the distribution that you are trying to install.
-
-1. To get WPA2 to work stably, install the `iwd` package (for example `sudo apt install iwd` on Ubuntu).
-
-2. Edit `/etc/NetworkManager/NetworkManager.conf` to look like the following:
-
-    ```ini
-    [device]
-    wifi.backend=iwd
-    ```
-
-3. Set iwd to run on boot with the following commands (assuming that you are using systemd).
-
-    ```sh
-    sudo systemctl enable --now iwd
-    sudo systemctl restart NetworkManager
-    ```
-
-If you Wi-Fi disconnects or has issues otherwise its advised to restart iwd: `sudo systemctl restart iwd`, or reprobe the Wi-Fi kernel module: `sudo modprobe -r brcmfmac_wcc && sudo modprobe -r brcmfmac && sudo modprobe brcmfmac`.

docs/tools/firmware.sh

@@ -351,7 +351,7 @@ dmg2img_check () {
 					git clone https://aur.archlinux.org/dmg2img.git ${dmg2imgdir}
 					makepkg -si --noconfirm
 					cd - >/dev/null
-					sudo rm -r ${dmg2imgdir}
+					sudo rm -r ${verbose} ${dmg2imgdir}
 					;;
 				(*)
 					echo -e "\nUnknown error"
@@ -363,6 +363,46 @@ dmg2img_check () {
 	fi
 }
 
+apfs_install () {
+	echo -e "\nAPFS driver missing!"
+	detect_package_manager
+	echo
+	apfs_driver_link=https://github.com/linux-apfs/linux-apfs-rw.git
+	if [[ ${PACKAGE_MANAGER} = "NONE" ]]
+	then
+		read -p "The script could not detect your package manager. Please install the APFS driver manually from ${apfs_driver_link} and press enter once you have it installed."
+	else
+		read -p "Press enter to install the APFS driver via ${PACKAGE_MANAGER}. Alternatively you can terminate this script by pressing Control+C and install it yourself from ${apfs_driver_link}."
+		echo -e "\nInstalling the APFS driver"
+		case ${PACKAGE_MANAGER} in
+			(apt)
+				sudo apt update
+				sudo apt install -y apfs-dkms
+				;;
+			(dnf)
+				sudo dnf -y copr enable sharpenedblade/t2linux
+				sudo dnf install -y linux-apfs-rw
+				echo -e "\nRunning akmods\n"
+				sudo akmods
+				;;
+			(pacman)
+				apfsdir=$(mktemp -d)
+				cd ${apfsdir}
+				sudo pacman -Sy --noconfirm git base-devel
+				git clone https://aur.archlinux.org/linux-apfs-rw-dkms-git.git ${apfsdir}
+				makepkg -si --noconfirm
+				cd - >/dev/null
+				sudo rm -r ${verbose} ${apfsdir}
+				;;
+			(*)
+				echo -e "\nUnknown error"
+				exit 1
+				;;
+		esac				
+	fi
+	sudo modprobe apfs && echo -e "\nAPFS driver loaded successfully!" || (echo -e "\nAPFS driver could not be loaded. Make sure you have the kernel headers installed. If you are still facing the issue, try again after restarting your Mac, or use some other method to get the firmware" && exit 1)
+}
+
 os=$(uname -s)
 case "$os" in
 	(Darwin)
@@ -385,7 +425,7 @@ case "$os" in
 			EOF
 		fi
 		echo -e "\nHow do you want to copy the firmware to Linux?"
-		echo -e "\n1. Run the same script on Linux."
+		echo -e "\n1. Copy the firmware to the EFI partition and run the same script on Linux to retrieve it."
 		echo "2. Create a tarball of the firmware and extract it to Linux."
 		echo "3. Create a Linux specific package which can be installed using a package manager."
 		echo -e "\nNote: Option 2 and 3 require additional software like python3 and tools specific for your package manager. Requirements will be told as you proceed further."
@@ -484,8 +524,9 @@ case "$os" in
 			exit 1
 		fi
 		echo -e "\nHow do you want to copy the firmware to Linux?"
-		echo -e "\n1. Retrieve the firmware from EFI."
-		echo "2. Download a macOS Recovery Image from Apple and extract the firmware from there."
+		echo -e "\n1. Retrieve the firmware from the EFI partition."
+		echo "2. Retrieve the firmware directly from macOS."
+		echo "3. Download a macOS Recovery Image from Apple and extract the firmware from there."
 		echo -e "\nNote: If you are choosing Option 1, then make sure you have run the same script on macOS before and chose Option 1 (Run the same script on Linux) there."
 		read choice
 		case ${choice} in
@@ -536,14 +577,63 @@ case "$os" in
 				echo -e "\nDone!"
 				;;
 			(2)
+				# Detect whether dmg2img are installed
+				echo -e "\nChecking for missing dependencies"
+				dmg2img_check
+				# Load the apfs driver, and install if missing
+				sudo modprobe apfs >/dev/null 2>&1 || apfs_install
+				unmount_macos_and_cleanup () {
+					sudo rm -r ${verbose} ${workdir} || true
+					sudo umount ${verbose} ${loopdevice} || true
+					sudo umount ${verbose} ${macosvol} || true
+					sudo rm -r ${verbose} ${imgdir} || true
+					sudo rm -r ${verbose} ${macosdir} || true
+					sudo losetup -d /dev/${loopdev} || true
+				}
+
+				echo -e "\nMounting the macOS volume"
+				workdir=$(mktemp -d)
+				imgdir=$(mktemp -d)
+				macosdir=$(mktemp -d)
+				macosvol=/dev/$(lsblk -o NAME,FSTYPE | grep nvme0n1 | grep apfs | head -1 | awk '{print $1'} | rev | cut -c -9 | rev)
+				sudo mount ${verbose} -o vol=2 ${macosvol} ${macosdir}
+				echo "Converting macOS Recovery Image from .dmg to .img"
+				cd ${workdir}
+				if [[ ${verbose} = -v ]]
+				then
+					dmg2img -v ${macosdir}/*/BaseSystem.dmg fw.img || unmount_macos_and_cleanup
+				else
+					dmg2img -s ${macosdir}/*/BaseSystem.dmg fw.img || unmount_macos_and_cleanup
+				fi
+				echo "Mounting image"
+				loopdev=$(losetup -f | cut -d "/" -f 3)
+				sudo losetup -P ${loopdev} fw.img
+				loopdevice=/dev/$(lsblk -o KNAME,TYPE,MOUNTPOINT -n | grep ${loopdev} | tail -1 | awk '{print $1}')
+				sudo mount ${verbose} ${loopdevice} ${imgdir} || unmount_macos_and_cleanup
+				cd - >/dev/null
+				echo "Getting firmware"
+				python3 "$0" ${imgdir}/usr/share/firmware ${workdir}/firmware-renamed.tar ${verbose} || (echo -e "\nCouldn't extract firmware. Try running the script again. If error still persists, try restarting your Mac and then run the script again, or choose some other method." && unmount_macos_and_cleanup && exit 1)
+				sudo tar ${verbose} -xC /lib/firmware/brcm -f ${workdir}/firmware-renamed.tar
+				echo "Reloading Wi-Fi and Bluetooth drivers"
+				sudo modprobe -r brcmfmac_wcc || true
+				sudo modprobe -r brcmfmac || true
+				sudo modprobe brcmfmac || true
+				sudo modprobe -r hci_bcm4377 || true
+				sudo modprobe hci_bcm4377 || true
+				echo "Cleaning up"
+				unmount_macos_and_cleanup
+				echo "Done!"
+				;;
+			(3)
 				# Detect whether curl and dmg2img are installed
+				echo -e "\nChecking for missing dependencies"
 				curl_check
 				dmg2img_check
 				cleanup_dmg () {
 					sudo rm -r ${verbose} ${workdir}
 					sudo umount ${verbose} ${loopdevice}
 					sudo rm -r ${verbose} ${imgdir}
-					sudo losetup -d /dev/loop50
+					sudo losetup -d /dev/${loopdev}
 				}
 
 				echo -e "\nDownloading macOS Recovery Image"
@@ -566,8 +656,9 @@ case "$os" in
 					dmg2img -s BaseSystem.dmg fw.img
 				fi
 				echo "Mounting image"
-				sudo losetup -P loop50 fw.img
-				loopdevice=/dev/$(lsblk -o KNAME,TYPE,MOUNTPOINT -n | grep loop50 | tail -1 | awk '{print $1}')
+				loopdev=$(losetup -f | cut -d "/" -f 3)
+				sudo losetup -P ${loopdev} fw.img
+				loopdevice=/dev/$(lsblk -o KNAME,TYPE,MOUNTPOINT -n | grep ${loopdev} | tail -1 | awk '{print $1}')
 				sudo mount ${verbose} ${loopdevice} ${imgdir}
 				echo "Getting firmware"
 				cd - >/dev/null

mkdocs.yml

@@ -35,6 +35,10 @@ markdown_extensions:
           permalink: true
     - admonition
     - pymdownx.details
+    - attr_list
+    - pymdownx.emoji:
+          emoji_index: !!python/name:material.extensions.emoji.twemoji
+          emoji_generator: !!python/name:material.extensions.emoji.to_svg
 
 plugins:
     -   redirects:

requirements.txt

@@ -2,5 +2,5 @@
 # To install them all run 'pip install -r requirements.txt' in the same directory
 
 mkdocs == 1.6.0
-mkdocs-material == 9.5.25
+mkdocs-material == 9.5.26
 mkdocs-redirects == 1.2.1