Add writeups/hackintosh

1 files changed, 425 insertions, 0 deletions

diff --git a/writeups/hackintosh/cloud+libvirt.md b/writeups/hackintosh/cloud+libvirt.md
new file mode 100644
index 0000000..9bb2a05
--- /dev/null
+++ b/writeups/hackintosh/cloud+libvirt.md
@@ -0,0 +1,425 @@
+# Running Hackintosh on Cloud using QEMU and Libvirt
+![virt-manager graphical console of hackintosh](image-7.png)
+
+```
+                <- Libvirt Domain Control ->
+                <- SPICE/VNC ->
+                <- USB Redirection ->
+LOCAL MACHINE           (INTERNET)           Cloud VM
+ virt-manager                                 libvirtd
+ virt-viewer                                   qemu-kvm
+                                                hackintosh
+```
+
+I've found myself in the mobile dev scene. I thought it was time to buy macbook
+but I was reluctant to make such a financial commitment for something I'll
+rarely use again after I'm done with the project. Hackintosh came up as a viable
+option so I took a deep dive into the rabbit hole.
+
+My choice of tools are Libvirt and qemu-kvm. I chose them because they're the
+ones I use on the daily basis. The fronts like virt-manager and virsh are quite
+nice, too. I've also decided to try running on a cloud VM so my mates can use it
+when they need it.
+
+My goal is to build and test React Native apps for IOS devices. So my mac
+environment has to have all the tools for it: Xcode, pod, brew and all the other
+shebang. Therefore I had to get **Sonoma** working because the latest version of
+Xcode dropped support for older macos.
+
+Apple Silicon is not discussed here because, as of the time of writing, no
+virtualisation software supports Apple's proprietary ARM instructions. Apple's
+recent push to move over to ARM is a real threat to Hackintosh community. Apple
+will eventually drop support for x86 macs and when that time comes, this post
+will no longer be relevant. Reverse engineering Rosetta is definitely a
+challenge. The law suits that come after will be, too.
+
+To those who are new to Hackintosh scene, emulating Mac has always been a
+constant battle between Apple constantly breaking backward compatibility and the
+community trying to fix broken stuff caused by that. The documents get outdated
+pretty quick so if this post looks old or doesn't look right, I suggest quickly
+moving onto new ones online.
+
+## Acknowledgements
+Many thanks to [kholia](https://github.com/kholia/OSX-KVM), the OpenCore
+project, and the Hackintosh communities.
+
+## Disclaimer
+Building a Hackintosh involves use of arbitrary binary from untrusted sources. I
+took the shortcut by using the binaries built by people on the internet since I
+don't have to do anything "serious" with my Hackintosh, other then IOS app
+building and debugging. At any rate, you may want to build the EFI image,
+OpenCore, and all of its extension, reviewing all the code in the process...
+well,
+
+I recommend buying the real hardware rather than hacking your way to save some
+money by using Hackintosh if you need Mac to do serious business. That way,
+there's no funny business between you and Apple, and you will have someone to
+sue when things go wrong - you're also paying for liability along with warranty
+that comes with the products.
+
+Another option is: "if you can't beat 'em, **DON'T** join 'em". This is the
+option many devs choose to take. If you don't condone Apple's business
+practices, show them by not having anything to do with them. Don't support IOS
+and Mac. Don't buy their overpriced hardware. That's how you vote them out of
+the competition.
+
+## Requirements
+When choosing the instance type for your hackintosh, note that at least 8GB of
+RAM is required[^8] if you want to do anything at all with it. For CPU
+configuration, it depends on the type of CPU the instance has.
+
+### Choice of CSP
+In order for kvm_intel or kvm_amd to work, the host must have virtualization CPU
+extensions enabled(`svm` for AMD and `vmx` for Intel). Since we're looking to
+run qemu-kvm on the cloud VM, the keywords we're looking for are "nested
+virtualization" and, more specifically, "L2" because the guest(L2) is run on a
+cloud VM(L1) which in turn runs on a bare metal hypervisor(L0). For stability
+issues[^1], only a handful of CSPs offer VM types with nested virtualisation
+enabled.
+
+1. Azure[^2]: the most painless and cost-effective option as of 2024. Easy to
+   request spot VMs, bargain spot prices, supported on both AMD and Intel
+1. GCP[^3]: supported on Intel instances only
+1. AWS: nested virtualisation support is officially supported only on bare metal
+   instances[^4]. My experience is that you *can* use KVM on compute optimized
+   instances[^5]. Worth a try if you feel so adventurous. Painful to launch spot
+   VMs because spot VMs can only be purchased via fleet requests on AWS
+
+If you look at the references closely, you'll see that nested virtualisation on
+CSP hypervisors is a cutting-edge technology. Your options are pretty much
+limited to the big 3.
+
+### Azure
+Running the vm on an E series instance was not feasible because the vCores are
+hyper-threaded and credit limited.
+
+**D4as_v5** w/ 14GB of RAM allocated to the vm was alright. **D8as_v5** w/ 28GB
+was perfect for my use case(VSCode, building React Native app, Xcode, and one
+iPhone SE 2nd simulator).
+
+The bandwidth of "Standard SSD" on Azure is in fact not "standard" at all. You
+need to use "Premium SSD" to get AWS's gp2/3 performance or the vm will
+bottleneck upon disk IO. At least 50GB is required. After installing all the
+apps for my set up it took up over 50GB meaning 100GB or more is required. So I
+recommend starting with a **premium SSD 128GB**.
+
+#### TCP Idle Timeout
+Azure has aggressive default TCP idle connection cutout(4 minutes)[^6]. Rather
+than fiddling with the cloud config, it'll be best if you just use even a more
+aggressive keepalive interval on the software you have control over(sshd or
+sysctl) so you don't have to worry about changing it every time you deploy a
+public IP on Azure. Since ssh tunneling is used to control libvirt, uncomment
+and change the values of the following settings in sshd_config.
+
+```
+TCPKeepAlive yes
+ClientAliveInterval 30
+ClientAliveCountMax 3
+```
+
+### Building QEMU(for Apple SMC device emulation)
+Some distros don't ship qemu package with Apple SMC device emulation. This is
+strange because this can only be done by patching qemu rather than the build
+config. Someone at Red Hat went a great length to remove it. But the feature
+is included in Fedora's PRM.
+
+So, you're left with 2 options:
+
+1. Pick a distro(**Fedora**) that ships qemu with Apple SMC
+1. Build qemu yourself
+
+To see if the qemu is built with SMC emulation, run:
+
+```sh
+echo -e 'info qtree\nq\n' |
+	qemu-kvm
+		-machine q35
+		-device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc'
+		-display none
+		-monitor stdio
+```
+
+The qemu should run and exit normally. The output should contain something like:
+
+```
+dev: isa-applesmc, id ""
+  iobase = 768 (0x300)
+  osk = "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
+```
+
+If you end up with errors it means that you have to build qemu yourself. If you
+don't have to build qemu yourself, skip to [Download OSX
+Images](#download-osx-images).
+
+#### RHEL
+For longevity of the VM image, I felt that it is necessary to set it up on
+Rocky. Following is how I built mine for use on my Rocky instance. Remember that
+this is a note, not a full-blown guide on how to Hackintosh so please try to
+read between the lines before contacting me for help.
+
+Continued in [build-qemu_rhel.md](build-qemu_rhel.md).
+
+## Download OSX Images
+On the host,
+
+```sh
+# Clone the repo
+git clone https://github.com/kholia/OSX-KVM
+cd OSX-KVM
+# Run the script to download the installer image
+./fetch-macOS-v2.py
+# (choose Sonoma or higher)
+# Convert the image
+dmg2img BaseSystem.dmg
+```
+
+Bring the images down to the libvirt image pool.
+
+```sh
+sudo cp BaseSystem.img OpenCore/OpenCore.qcow2 /var/lib/libvirt/images
+```
+
+## Setting Up
+Install libvirt and its friends on the instance.
+
+```sh
+sudo dnf install \
+	libvirt \
+	libvirt-daemon \
+	libvirt-daemon-kvm \
+	libvirt-daemon-driver-interface \
+	libvirt-daemon-driver-network \
+	libvirt-daemon-driver-nodedev \
+	libvirt-daemon-driver-nwfilter \
+	libvirt-daemon-driver-secret \
+	libvirt-daemon-driver-qemu
+```
+
+Enable and fire them up.
+
+```sh
+sudo systemctl enable --now \
+	libvirtd.service \
+	virtqemud.service \
+	virtlogd.service
+```
+
+Register your account to the groups.
+
+```sh
+sudo usermod -aG libvirt "$USER"
+```
+
+On your machine, install `virt-manager`.
+
+```sh
+# (RPM)
+sudo dnf install virt-manager
+# (DEB)
+sudo apt-get install virt
+# (Gentoo)-manager
+sudo emerge virt-manager
+# (OpenBSD)
+sudo pkg_add virt-manager
+# (mac)
+brew install virt-manager
+```
+
+Run virt-manager.
+
+### Add Libvirt Host Connection
+Add connection to the host(`File` - `Add Connection`).
+
+![virt-manager add connection dialog](image.png)
+
+Connect to the libvirt daemon running on the host by double clicking on the name
+or using the context menu.
+
+## Generate Unique Values for the VM (optional)
+The hackintosh will function just fine without changing the machine serial
+numbers that come with kholia's OpenCore loader. If you don't want to take any
+chances, go ahead and generate the serial for your hackintosh.
+
+https://github.com/sickcodes/osx-serial-generator
+
+Use the script `generate-unique-machine-values.sh` to a generate machine values.
+You can use the script to build the OpenCore boot disks with the values inserted
+to the images. Follow the repo's instruction to do that. Here, I'll demonstrate
+how to set them up manually.
+
+The script writes the values to csv and tsv files. Access them anytime by
+reading the files.
+
+## Create the hackintosh VM
+Add a VM on the host. I'm laying down 2 ways to do this: create one from
+scratch(DIY) using virt-manager and import the one I made.
+
+### Import the XML
+[mac.libvirt.xml](mac.libvirt.xml) is my latest working copy. The Upload the
+domain xml to the host and import using virsh.
+
+```sh
+sudo virsh define mac.libvirt.xml
+# Copying from your machine and pasting using the stdin also works:
+# sudo virsh define /dev/stdin
+```
+
+After importing the xml, edit the memory and the number of vcpu to your needs.
+
+Optional: manually edit the MAC of the net interface to the one generated
+beforehand.
+
+```xml
+<!-- Paste the unique mac generated for your vm(optional) -->
+<qemu:arg value="virtio-net-pci,addr=0x12.0x0,netdev=net0,id=net0,mac=THE_MAC"/>
+```
+
+### DIY(virt-manager)
+See [libvirt-diy.md](libvirt-diy.md).
+
+## Attach Volumes and Start the VM
+Create a system volume for the vm. **50GB** is a good number to start with.
+**100GB** to be on the safe side. Attach it to the vm as a SATA disk. You don't
+have to register the disk in the boot order because only OpenCore can load the
+OS.
+
+Add another SATA disk with the macOS image(`BaseSystem.img`). Again, doesn't
+have to be in the boot order as the image is loaded by OpenCore. The disk can be
+removed from the vm after the installation.
+
+You may now start the VM.
+
+## Setting Serial Number (optional)
+You can set the machine serial numbers by
+
+1. Using the EFI shell
+1. Mounting the OpenCore disk on the host and editing the config file
+
+In the OpenCore boot image is the file `config.plist`, which contains the serial
+numbers for the machine. The first method is illustrated here.
+
+![UEFI Shell selection](image-2.png)
+
+Drop to UEFI shell.
+
+![UEFI shell capture](image-3.png)
+
+Find `config.plist` and run `edit` on it.
+
+![UEFI edit command](image-4.png)
+
+**F4** key to search string `PlatformInfo`. Type in the values you generated in
+the [previous step](#generate-unique-values-for-the-vm-optional).
+
+**F3** to exit. Save before exit.
+
+![Reset NVRAM selection](image-5.png)
+
+`exit` to return to the OpenCore boot menu. Do a NVRAM reset.
+
+## Done!
+Boot to **BaseSystem** and install the OS. Start using the hackintosh.
+
+## Cool Stuff
+### Scaling up and down
+Unlike Windows, macOS doesn't seem to be bothered by hardware change. The VM can
+be scaled and down without any modification to the parameters or reinstallation.
+
+![Hackintosh running with 64 cores and 192GB of RAM](image-6.png)
+
+Meaning, something like this is possible(tested by myself). And, of course, the
+VM can be scaled down once the heavy lifting is done. This can be done
+painlessly on Azure because, unlike AWS, Azure lets you change the size of a
+spot VM without deleting or recreating the instance.
+
+## Caveats/TODOs
+With all the effort, at the end of the day, nothing beats buying a real mac.
+There are many caveats when it comes to using Hackintosh in general. Even more
+from using Libvirt's abstraction and running it on the cloud.
+
+Unlike Windows, Mac(and IOS for that matter) doesn't have to support a wide
+range of products. This enabled Apple devs to make assumptions about underlying
+hardware and take many shortcuts when maintaining the kernel. The virtualization
+software products in the market are not simply cut out for emulating Mac.
+
+The qemu passthrough part of the XML looks ugly but this is the cost of running
+Hackintosh on Libvirt. In my opinion, Libvirt is a good tool and the benefit of
+using it outweighs the cost.
+
+### Q35 pcie-root-port
+When the machine type is set to Q35, libvirt "forcibly" adds pcie-root-port to
+the root PCI-E controller. This is to implement PCI-E device hotplug[^10] and
+modern kernels have no issue walking through these "root ports". For Mac, the
+closest thing it has to hot plug are Thunderbolt devices and supporting these
+"root ports" doesn't make any sense on Mac. All the PCI-E devices Libvirt
+attaches to the VM are attached on root ports, which are non-existent on Mac
+hardware hence rendering them useless for Hackintosh.
+
+To get around this issue, the XHCI controller and the input devices need to be
+attached directly to the PCI-E root via qemu command line passthrough
+
+Someone pointed out the issues of Libvirt forcing root ports, but the libvirt
+devs said no[^9].
+
+### XHCI Related Issue
+The USB 2.0 controller support has been dropped since Sonoma(14) so the input
+devices(usb-kbd and usb-tablet) need to be attached directly to the XHCI
+controller[^11].
+
+There is a discrimination as to types of device that can be attached to the
+controller and the ones that can only be attached to a hub. The XNU kernel is
+unable to enumerate a USB drive when it is attached to the controller. So all
+the usb-redir devices are attached to the usb-hub, which operates at full-speed.
+
+### No Sound
+AppleALC is included in the boot image, but for reasons unknown it doesn't seem
+to be working in Sonoma(14). This might be a simple matter of adding the qemu
+hda codec to AppleALC. It seems that the qemu's hda never made it to
+AppleALC in the first place[^13].
+
+### USB Redirection
+See [usbredir.md](usbredir.md).
+
+### SMM Instruction Crash Bugs
+QEMU crashes when the secboot variant of stock edk2-ovmf is used. When the Mac
+boots up and seconds later, the SMM[^7] instructions are issued and qemu crashes
+with SIGABRT. This appears to be caused by a bug in the kvm module, not QEMU.
+
+https://gitlab.com/qemu-project/qemu/-/issues/1198
+
+So the SMM and secure boot have to be turned off.
+
+```xml
+<loader readonly="yes" secure="no" type="pflash">...</loader>
+...
+<smm state="off"/>
+```
+
+### Notes on Migration/Teleportation
+The instance cannot be saved to disk because of the `invtsc`.
+
+VMWare calls it teleportation. Others call it migration. It being: pausing and
+saving the state of the vm to continue its execution on another bare metal
+hypervisor.
+
+The CSP could migrate your VM running a guest without notice. In such event,
+your Hackintosh will suffer the side effects as there are parameters that make
+the qemu guest difficult to save in order to migrate.
+
+## Links
+- https://hackintosh.com/
+- https://www.tonymacx86.com/
+
+[^1]: https://linux-kvm.org/page/Nested_Guests#Limitations
+[^2]: https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/
+[^3]: https://cloud.google.com/compute/docs/instances/nested-virtualization/overview
+[^4]: https://aws.amazon.com/about-aws/whats-new/2021/02/introducing-amazon-ec2-m5n-m5dn-r5n-and-r5dn-bare-metal-instances/
+[^5]: https://repost.aws/questions/QUkOwmVhagQbOumNhdfc4YcA/
+[^6]: https://carsonip.me/posts/azure-tcp-idle-timeout-tcp-keepalive-python/
+[^7]: https://en.wikipedia.org/wiki/System_Management_Mode
+[^8]: https://en.wikipedia.org/wiki/MacBook_Air_(Intel-based)#Retina_(2018%E2%80%932020)
+[^9]: https://listman.redhat.com/archives/libvir-list/2020-February/198033.html
+[^10]: https://libvirt.org/pci-hotplug.html
+[^11]: https://github.com/kholia/OSX-KVM/pull/238
+[^12]: https://github.com/kholia/OSX-KVM/blob/master/macOS-libvirt-Catalina.xml#L166
+[^13]: https://github.com/acidanthera/AppleALC/wiki/Supported-codecs#currently-supported-codecs-2024-02-05-v189