peter
4 years ago
parent
d82bb73c29
commit
5561b08e6a
@ -0,0 +1,72 @@
|
|||||||
|
#!/usr/bin/docker
|
||||||
|
#
|
||||||
|
# This Dockerfile is to be consumed with the docker_osx helm templates. It consumes the
|
||||||
|
# Ubuntu image so that OpenCore.qcow2 can be re-generated (which happens in Kube itself),
|
||||||
|
# not to mention that OSX-KVM was written for Ubuntu. This was not designed to be run in Docker
|
||||||
|
# by itself.. very well anyway.
|
||||||
|
#
|
||||||
|
|
||||||
|
FROM ubuntu:21.04
|
||||||
|
|
||||||
|
SHELL ["/bin/bash", "-c"]
|
||||||
|
|
||||||
|
# this has to match .Values.image.userName in helm template
|
||||||
|
ARG USER=ubuntu
|
||||||
|
# this installs the kvm linux kernel in the docker container so that OpenCore.qcow2 boot images
|
||||||
|
# can be built.
|
||||||
|
ARG DOCKER_KERNEL_VERSION=linux-image-kvm
|
||||||
|
|
||||||
|
ENV TZ=America/Los_Angeles
|
||||||
|
ARG DEBIAN_FRONTEND=noninteractive
|
||||||
|
|
||||||
|
RUN DEBCONF_FRONTEND=noninteractive apt update \
|
||||||
|
&& apt install \
|
||||||
|
bridge-utils \
|
||||||
|
fish \
|
||||||
|
git wget \
|
||||||
|
libguestfs-tools \
|
||||||
|
libvirt-daemon-system \
|
||||||
|
$DOCKER_KERNEL_VERSION \
|
||||||
|
p7zip-full \
|
||||||
|
qemu \
|
||||||
|
sudo \
|
||||||
|
uml-utilities \
|
||||||
|
virt-manager \
|
||||||
|
-y
|
||||||
|
|
||||||
|
# Configure SSH
|
||||||
|
RUN apt install git vim nano alsa-utils openssh-server -y
|
||||||
|
|
||||||
|
# Create user and grant sudo privledges
|
||||||
|
RUN adduser --disabled-password \
|
||||||
|
--gecos '' $USER \
|
||||||
|
&& echo "$USER ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/$USER \
|
||||||
|
&& chmod 0440 /etc/sudoers.d/$USER
|
||||||
|
|
||||||
|
# Configure VNC for user
|
||||||
|
RUN apt install \
|
||||||
|
dbus-x11 \
|
||||||
|
openbox \
|
||||||
|
tigervnc-common \
|
||||||
|
tigervnc-standalone-server \
|
||||||
|
xfce4 \
|
||||||
|
xfce4-goodies \
|
||||||
|
x11-xserver-utils \
|
||||||
|
xdotool \
|
||||||
|
xorg \
|
||||||
|
xterm \
|
||||||
|
ufw \
|
||||||
|
-y
|
||||||
|
|
||||||
|
USER $USER
|
||||||
|
|
||||||
|
# only create ~/.vnc as helm will build out ~/.vnc/config
|
||||||
|
RUN mkdir -p ${HOME}/.vnc
|
||||||
|
|
||||||
|
RUN git clone --depth 1 https://github.com/kholia/OSX-KVM.git /home/$USER/OSX-KVM
|
||||||
|
|
||||||
|
VOLUME ["/tmp/.X11-unix"]
|
||||||
|
|
||||||
|
WORKDIR /home/$USER/OSX-KVM
|
||||||
|
# helm will build out ./Launch_custom.sh
|
||||||
|
CMD envsubst < ./Launch_custom.sh | bash
|
||||||
@ -0,0 +1,283 @@
|
|||||||
|
# Install macOS Docker Virtualization
|
||||||
|
## Setup
|
||||||
|
This walks through setting up QEMU virtualization for running macOS in Docker & Kubernetes
|
||||||
|
|
||||||
|
Please note, this guide assumes the host operation system is running Centos 7 (or ClearOS 7 more specifically). These commands can mostly be transferred to other distros, but there are a few areas that need commands (i.e. updating )
|
||||||
|
|
||||||
|
### Host configuration
|
||||||
|
|
||||||
|
## Build QEMU and libvirt from source
|
||||||
|
|
||||||
|
Since there is no official QEMU 5.X repo it appears, build from source.
|
||||||
|
|
||||||
|
### QEMU Requirements
|
||||||
|
|
||||||
|
Python, glib2-devel, and pixman
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo yum install python glib2-devel cairo-devel -y
|
||||||
|
```
|
||||||
|
|
||||||
|
Ninja
|
||||||
|
|
||||||
|
```
|
||||||
|
pip3 install ninja
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build QEMU from source steps
|
||||||
|
|
||||||
|
Clone the offical QEMU repo and build from source:
|
||||||
|
|
||||||
|
```
|
||||||
|
git clone git://git.qemu-project.org/qemu.git
|
||||||
|
cd qemu
|
||||||
|
mkdir -p bin/debug/native
|
||||||
|
cd bin/debug/native
|
||||||
|
../../../configure --enable-debug
|
||||||
|
make -j24
|
||||||
|
make install
|
||||||
|
```
|
||||||
|
|
||||||
|
_Note: adjust make to use the desired number of threads avaliable on your system_
|
||||||
|
|
||||||
|
### libvirt Requirements
|
||||||
|
|
||||||
|
Configure repo:
|
||||||
|
|
||||||
|
```
|
||||||
|
yum-config-manager --nogpgcheck --add-repo http://mirror.centos.org/centos/7/virt/x86_64/libvirt-latest/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Install libvirt
|
||||||
|
|
||||||
|
```
|
||||||
|
yum install libvirt -y
|
||||||
|
```
|
||||||
|
|
||||||
|
### Update permissions
|
||||||
|
|
||||||
|
```
|
||||||
|
chmod 660 -R /dev/kvm && chown 1000:1000 /dev/kvm
|
||||||
|
usermod -a -G kvm root
|
||||||
|
```
|
||||||
|
|
||||||
|
_Note: these may not be required_
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
|
||||||
|
Ensure latest version installed
|
||||||
|
|
||||||
|
```
|
||||||
|
virsh -c qemu:///system version --daemon
|
||||||
|
```
|
||||||
|
|
||||||
|
* For example, should output something like:
|
||||||
|
|
||||||
|
```
|
||||||
|
[root@server repos]# virsh -c qemu:///system version --daemon
|
||||||
|
Compiled against library: libvirt 5.0.0
|
||||||
|
Using library: libvirt 5.0.0
|
||||||
|
Using API: QEMU 5.0.0
|
||||||
|
Running hypervisor: QEMU 5.2.50
|
||||||
|
Running against daemon: 5.0.0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Install IMMO for GPU passthrough
|
||||||
|
|
||||||
|
1. Modify GRUB boot args:
|
||||||
|
|
||||||
|
Add the following to `/etc/default/grub` to the end of the `GRUB_CMDLINE_LINUX` parameter:
|
||||||
|
|
||||||
|
```
|
||||||
|
GRUB_CMDLINE_LINUX="... iommu=pt intel_iommu=on"
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Update GRUB2:
|
||||||
|
|
||||||
|
```
|
||||||
|
grub2-mkconfig -o /boot/efi/EFI/clearos/grub.cfg
|
||||||
|
```
|
||||||
|
|
||||||
|
_Note: this command may vary based on location of the grub.cfg for the boot entry_
|
||||||
|
|
||||||
|
1. Reboot system
|
||||||
|
|
||||||
|
1. Ensure that the kernel parameter changes worked:
|
||||||
|
|
||||||
|
```
|
||||||
|
cat /proc/cmdline
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Find GPU hardware ids with `lspci`
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
lspci -nn | grep -i nvidia
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Add the hardware ids to `/etc/modprobe.d/vfio.conf`
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
options vfio-pci ids=10de:1b81,10de:10f0
|
||||||
|
```
|
||||||
|
|
||||||
|
_Note: this is for the NVIDIA GTX 1070_
|
||||||
|
|
||||||
|
1. Enable `vfio-pci`
|
||||||
|
|
||||||
|
```
|
||||||
|
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Make backup and rebuild `initramfs`:
|
||||||
|
|
||||||
|
```
|
||||||
|
cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
|
||||||
|
dracut -f
|
||||||
|
```
|
||||||
|
|
||||||
|
_Note: `dracut -f` may take awhile.._
|
||||||
|
|
||||||
|
1. Increase ulimits
|
||||||
|
|
||||||
|
_This is done to avoid memory issues like `VFIO_MAP_DMA: -12` and etc_
|
||||||
|
|
||||||
|
Append the following to `/etc/security/limits.conf`:
|
||||||
|
|
||||||
|
```
|
||||||
|
@kvm soft memlock unlimited
|
||||||
|
@kvm hard memlock unlimited
|
||||||
|
```
|
||||||
|
|
||||||
|
Append the following to `/etc/docker/daemon.json`:
|
||||||
|
|
||||||
|
```
|
||||||
|
{
|
||||||
|
"default-ulimits": {
|
||||||
|
"nofile": {
|
||||||
|
"name": "nofile",
|
||||||
|
"hard": 65536,
|
||||||
|
"soft": 1024
|
||||||
|
},
|
||||||
|
"memlock":
|
||||||
|
{
|
||||||
|
"name": "memlock",
|
||||||
|
"soft": -1,
|
||||||
|
"hard": -1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `LimitMEMLOCK` to `/etc/systemd/system/multi-user.target.wants/libvirtd.service` like:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Unit]
|
||||||
|
Description=Virtualization daemon
|
||||||
|
...
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
...
|
||||||
|
LimitMEMLOCK=infinity
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Reload systemd after changing config
|
||||||
|
|
||||||
|
```
|
||||||
|
systemctl daemon-reload
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Reboot system
|
||||||
|
|
||||||
|
1. Ensure that `vfio` worked
|
||||||
|
|
||||||
|
```
|
||||||
|
dmesg | grep -i vfio
|
||||||
|
```
|
||||||
|
|
||||||
|
# Issues
|
||||||
|
|
||||||
|
Many issues can rise up as a result of adding the complexity layers involved here. Some of the main areas are improperly loading the `vfio-pci` driver for the GPU and permission issues.
|
||||||
|
|
||||||
|
## Modules for vfio not loading
|
||||||
|
|
||||||
|
When `vfio` does not load, errors such as the following can be seen:
|
||||||
|
|
||||||
|
```
|
||||||
|
error getting device from group *: No such device
|
||||||
|
Verify all devices in group * are bound to vfio-<bus> or pci-stub and not already in use
|
||||||
|
```
|
||||||
|
|
||||||
|
This can show up when `vfio-pci` driver is not loaded for the peripheral. Ensure that `vfio-pci` is loaded.
|
||||||
|
|
||||||
|
```
|
||||||
|
dmesg | grep -i vfio
|
||||||
|
```
|
||||||
|
|
||||||
|
If so, explicitly tell `vfio` modules to start
|
||||||
|
|
||||||
|
```
|
||||||
|
echo 'vfio
|
||||||
|
vfio_iommu_type1
|
||||||
|
vfio_pci
|
||||||
|
vfio_virqfd' > /etc/modules
|
||||||
|
```
|
||||||
|
|
||||||
|
Make backup and rebuild `initramfs`:
|
||||||
|
|
||||||
|
```
|
||||||
|
cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
|
||||||
|
dracut -f
|
||||||
|
```
|
||||||
|
|
||||||
|
_Note: `dracut -f` may take awhile.._
|
||||||
|
|
||||||
|
Do a system reboot
|
||||||
|
|
||||||
|
After rebooting, check on the gpu with `lspci` utilizing your gpu hardware id:
|
||||||
|
|
||||||
|
I.E.
|
||||||
|
|
||||||
|
```
|
||||||
|
[root@server docker-docker-osx]# lspci -vvv -s 09:00.0
|
||||||
|
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev c7) (prog-if 00 [VGA controller])
|
||||||
|
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Radeon RX 480
|
||||||
|
Physical Slot: 5
|
||||||
|
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
|
||||||
|
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
|
||||||
|
Interrupt: pin A routed to IRQ 255
|
||||||
|
...
|
||||||
|
Kernel driver in use: vfio-pci
|
||||||
|
Kernel modules: amdgpu
|
||||||
|
```
|
||||||
|
|
||||||
|
_It does not matter if the host os loads a gpu module as seen with `Kernel modules: amdgpu` in the case above, the important part is that `vfio-pci` is the driver in use._
|
||||||
|
|
||||||
|
## Permissions on vfio and kvm
|
||||||
|
|
||||||
|
One of the biggest areas of pain can be setting permissions on `/dev/kvm`, `/dev/vfio/vfio`, or `/dev/vfio/<iommu_group>`. If permission errors are seen, try the following commands:
|
||||||
|
|
||||||
|
```
|
||||||
|
chmod 660 -R /dev/kvm && chown 1000:1000 /dev/kvm
|
||||||
|
chmod 777 -R /dev/vfio && chown 1000:1000 -R /dev/vfio
|
||||||
|
```
|
||||||
|
|
||||||
|
# References
|
||||||
|
|
||||||
|
https://gist.github.com/dghubble/c2dc319249b156db06aff1d49c15272e
|
||||||
|
|
||||||
|
`Configure IOMMU and vfio`
|
||||||
|
https://www.server-world.info/en/note?os=CentOS_7&p=kvm&f=10
|
||||||
|
|
||||||
|
`Configuring GPU driver with vfio-pci binding`
|
||||||
|
https://github.com/intel/nemu/wiki/Testing-VFIO-with-GPU
|
||||||
|
|
||||||
|
`IOMMU Interrupt Mapping`
|
||||||
|
https://pve.proxmox.com/wiki/Pci_passthrough#IOMMU_Interrupt_Remapping
|
||||||
|
|
||||||
|
`Manual Graphics Driver Binding`
|
||||||
|
https://lwn.net/Articles/143397/
|
||||||
|
|
||||||
|
`QEMU Stdio Example`
|
||||||
|
https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04521.html
|
||||||
Loading…
Reference in new issue