Contribute
Register

DUAL Success: QEMU+KVM+High Sierra/Win10+AMD RX560 Passthrough w/ AsusMaximusIX Code+Intel i7-7700K

Joined
Feb 8, 2014
Messages
8
Mac
Classic Mac
Mobile Phone
***UPDATE***

Good news, I got the EVGA Nvidia GTX 1080 successfully passthrough to Windows 10 Enterprise trial edition. Sound works find as long as the regedit is edited to change the MessageInterrup parameter from 0 to 1 in the windows registry. The sound from the GTX 1080 via the Display Port of my monitor is perfect. Totally high fidelity. No lag at all. Very HD.

So, I got both the AMD and also GTX1080 passthrough to Windows 10.

For High Sierra, I got the AMD RX 560 running in a passthrough successfully; but only managed to boot High Sierra to a stop sign using the Nvidia GTX 1080. This is probably some minor issues with my config.plist, and I think I can get it to work eventually.

Will update this post with the details of how I did it as soon as I can.

Oh yes, I got all three of Gentoo, High Sierra and Windows 10 Enterprise all running at the same time on this single machine right now, and they are all running at very near native or native speed.

***********


"Hardware and Software Hybridization of Guest Operating Systems"

by rev0lt

Experiment's Goals:

(1) High Sierra at near-native speed on Linux QEMU KVM with AMD GPU Passthrough (Success);

(2) Win10 at near-native speed on Linux QEMU KVM with AMD GPU Passthrough (Success);

(3) To achieve (1) and (2) using Nvidia GPU (Not yet successful); and

(4) To achieve (1) and (2) _simultaneously_. (In theory achievable; but would require purchase of a second AMD card).

Rationale ("Why?"):

Can you feel it?

Jokes aside, a bit of a brief background -- this whole trouble started with my Apple Magic Mouse (Series 1). I really love this mouse -- it has been with me almost 8 full years now, and yes, it still looks beautiful; and I really wanted this wonderful mouse to work in an acceptable manner in Gentoo Linux. I managed to get it to work, but somehow the scrolling and movement in X Window just does not "feel right," even if I tried tuning it with xinput.

So, being OCD'd, I tried to get the mouse to work in an acceptable manner in Windows 10 too. It does work somewhat okay-ish, using Apple's Bootcamp driver for the mouse which I got using my MacBook Pro. But the scrolling and "feel" are still somewhat "off".

Which brings us to this point. From OCD my perspective, this is all done just to get the mouse to work "right" on my setup below.

Hardware Setup:

Apple Magic Mouse (Series 1) <3 <3 <3

Asus Maximus Code IX
Intel i7-7700K
EVGA Nvidia GTX 1080 Hybrid
ASUS Strix AMD RX 560 (purchased for testing this setup)
G.Skill TridentZ DDR4-3200 16G
Samsung NVMe SSD 960 EVO M.2 250GB
Samsung SSD 850 PRO 256GB
EVGA Supernova 850w G2 Gold
Dell P4317Q 4K Monitor (43-Inch)
CoolerMaster MasterKeys Pro L (Cherry MX Red)
Sony Playstation 4 PRO
Thermaltake Core X71
Thermaltake Water 3.0
Apple MacBook Pro

Software Setup:

The SSD 850 Pro is the drive of interest here, since it is where I store the Linux host for learning computer science and programming as a hobby. (The NVMe M.2 drive is installed with Windows 10 as my primary OS for daily use, so it is irrelevant here.)

I compiled Linux Gentoo 4.13.8 on the SSD 850 Pro as the host OS, with KVM, IOMMU, VFIO functions enabled in the kernel. I also compiled QEMU 2.10.0.

Discussion:

To get High Sierra (10.13 release) working on Linux with QEMU, I followed the instructions at <https://github.com/kholia/OSX-KVM>. The two OVMF files (OVMF_CODE-pure-efi.fd and OVMF_VARS-pure-efi-1024x768.fd) and also the Clover.qcow2 file there all worked out of the box. I downloaded those 3 files onto the Linux host, then prepared the requisite High Sierra USB installer by using the usual USB+Clover method as described at <https://www.tonymacx86.com/threads/how-to-create-a-macos-high-sierra-public-beta-installation-usb.225520/> (following the instruction regarding the UEFI Mode, not the Legacy Mode). After that, I used the script below (commenting out the 2 VFIO GPU passthrough lines -- the lines mentioning 03:00.0 and 03.00.1 -- for now and use gtk or vnc to output video since GPU passthrough was yet to be done) to get High Sierra installed and running with "soft" video output through gtk, vnc, spice, etc. for the moment, at least until the GPU is passthrough later. (Was I repeating myself there? That is the trouble with OCD, there is this irresistible compulsion to do a certain thing in a specific way). I have tried many other methods, but presently it seems that the instructions in the above GitHub link are the only ones that worked.


It is a bit trickier to get High Sierra to run with GPU passthrough to obtain near-native speed. To achieve that, once I installed and booted into High Sierra with soft video output, I (a) patched the AMD9500Controller.kext in /System/Library/Extensions in the High Sierra guest's hard disk by using the instructions at <http://www.insanelymac.com/forum/topic/303186-how-to-modification-of-amd-fb-clover-injection/>; and then (b) installed the Lilu.kext and WhateverGreen.kext into /System/Library/Extensions using KextBeast. It seems that the binary needed to drive the AMD RX 560 is already included in High Sierra, inside the AMD9500Controller.kext folder. All that is needed is to hex-modify the binary so that the hardware layout of the RX 560 is correctly reflected the binary file in that kext. I modified the "Acre" personality entry in the binary in AMD9500Controller.kext for convenience sake because it has 3 connectors at the back, the same number of connectors as the RX 560. (It is unclear whether (a) is needed if (b) is done -- I have not tested such scenario.)

Specifically, this is what I changed in the binary using xxd:

For the "Acre" personality located at 0x121f80 in the binary file, change the hex from

00040000040300000001010100000000
11020201000000000008000004020000
00010200000000002103050400000000

to this

00040000040300000001010111020101
00080000000200000001020021030204
04000000140200000001030010000305

All the connectors (DP, HDMI, DVI) at the back of the card now work perfect.

At any rate, I did both (a) and then (b), and High Sierra boots successfully with AMD RX 560 passthrough, using the following Linux QEMU command line script adapted from <https://github.com/kholia/OSX-KVM>:

_______________________

#!/bin/bash
MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,avx2,+smep"
export QEMU_AUDIO_DRV=alsa &&
qemu-system-x86_64 \
-enable-kvm \
-m 8192 \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
-machine pc-q35-2.9 \
-smp cpus=8,sockets=1,cores=4,threads=2 \
-device isa-applesmc,osk="<i inserted the key from my Apple Macbook Pro here>" \
-drive if=pflash,format=raw,readonly,file=OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=OVMF_VARS-pure-efi-1024x768.fd \
-smbios type=2 \
-device ich9-intel-hda -device hda-duplex \
-device ide-drive,bus=ide.2,drive=Clover \
-drive id=Clover,if=none,snapshot=on,format=qcow2,file=./'Clover.qcow2' \
-device ide-drive,bus=ide.1,drive=MacHDD \
-drive id=MacHDD,if=none,file=./high.img,format=qcow2 \
-netdev user,id=usr0 -device e1000-82545em,netdev=usr0,id=vnet0 \
-balloon none \
-device vfio-pci,host=03:00.0,multifunction=on \
-device vfio-pci,host=03:00.1 \
-vga none \
-monitor unix:/tmp/monitor.sock,server,nowait \
-nographic \
-device vfio-pci,host=00:14.0,bus=pcie.0 \

_______________________


Once the above is done, the separate AMDRadeonX4250.kext (responsible for 3D acceleration, etc) seems to be loaded by High Sierra and the RX 560 card should be functioning perfectly, directly connected to the guest OS with metal support.

Note that using the above command line, I have passthrough my USB controller as well (vfio-pci,host=00.14.0). The result is that bluetooth and all my USB ports (I have not tested the 3.1 one) worked out of the box, with my Apple Magic Mouse (Series 1) working perfectly with that tight, smooth buttery feel. Also working flawlessly are my CoolerMaster L keyboard and all other USB devices, including USB external drives, etc). I would then control the Linux host underneath via ssh from the High Sierra guest. When the Dell monitor is dedicated to the High Sierra guest at full 4K, the linux host basically becomes transparent and invisible to the user.


Performance-wise, High Sierra runs _buttery smooth_ and _beautifully_ with the AMD RX560 passthrough in QEMU KVM. It is impressively silky, fast and responsive, with QE working and no glitches or hangs or crashes. Apps open almost instantaneously (split second). Ethernet works out of the box and the sound works perfect via the audio output jack of the Dell monitor which is connected to the AMD RX 560 via Display Port. In fact, the setup is so near-native that I'd speculate that a layperson would not notice the difference compared to say, a 2017 iMac (Geekbench 4 benchmark that I ran seem to suggest similar scores) unless the setup is revealed to him. Personally and anecdotally, I do not notice the difference even if I look for them. I mean, this thing is bat-out-of-cave fast. Certainly, it is a whole different league and at a whole different level from the usual slowish virtual box, parallels, vanilla vmware experience. Even compared to my MacBook Pro (also running High Sierra), this setup feels substantially smoother, faster and more responsive.


There are only two very minor noticeable glitches. First, flac audio playback on Fidelia would intermittently "tear" for split seconds if I concurrently run very heavy compile tasks in the Linux Gentoo OS underneath the QEMU/HighSierra. But this is expected. I have not tried CPU-pinning to dedicate specific CPUs to High Sierra yet, but I suppose using CPU-pinning, the lag can be removed since High Sierra would not then have to compete for CPU with the Gentoo Linux host running underneath. Netflix video playback on Chrome/Safari runs smooth, without any lag even under heavy load. Also, the sound in High Sierra via the AMD graphics card works perfectly -- does not suffer from the slight lag as in the case of running Win 10 in QEMU with the same card passed through. Second, in the High Sierra boot screen, the progress bar under the logo would tear slightly during boot up and appear to freeze (but it is still booting underneath) for say 5-6 seconds, before booting into the login/password screen.

With more fine-tuning, I think I can get this experimental set-up to run High Sierra even faster -- but as it is now, it feels like a native machine already. I am super, super impressed with the performance.

Windows 10:

Windows 10 Enterprise (90-trial version) also works with this QEMU KVM GPU passthrough setup. Everything works out of the box without any patching. All that is needed is for one to download and install the AMD Radeon video drivers. Performance is very smooth too and near native -- except that the audio output lags behind the video output during Netflix playback by a second or two. I feel that Win 10 in this setup is not as impressive (the "Wow" factor") as getting High Sierra to work at near-native with GPU passthrough. Both are near-native, but High Sierra just "feels" (to me anecdotally) better, tighter, more finely tuned, more "buttery" and smoother in this setup as compared to Win10 under the same. Maybe it is just a function of graphical user interface design generally. I don't really know why.


Further Experiment:

Note that all of the above is done despite the fact that I have plugged the AMD RX 560 only in the PCIE x4 slot on the motherboard. This is because I already have the Nvidia GTX 1080 installed in the PCIE x8_1 slot. The AMD RX 560 can't be placed in the x8_2 slot because I use the GTX 1080 for the Linux OS and those two x8 slots are in the same IOMMU hardware group, and hardware in the same IOMMU group cannot be passthrough to two different OSes.

Thus, theoretically, performance in such setup above could be further improved if I were to use the Intel HD630 onboard graphics for Linux, disconnect the power to the Nvidia GTX, and place the AMD RX 560 in the x8_2 slot for the passthrough. Additionally, I can try moving the Linux host to the NVMe .M2 drive for testing for a significant disk speed improvement.

A much more interesting next step of the experiment would be to procure a second AMD RX 560 card, and place it in a x4 slot, with the first AMD RX 560 card in the x8_2 slot, have Linux running on the Intel HD 630, and then passthrough both AMD cards to High Sierra and Win 10 simultaneously, such that Linux Gentoo, High Sierra and Win10 all run on the same machine at the same time and all at native or near native-speed!

Alternatively, if anyone here knows how to get the Nvidia GTX 1080 to passthrough to High Sierra or Windows via QEMU, I would appreciate it if you could share your knowledge! I have tried to do so many times, but it all ends in black screen with the Nvidia card.

I intend to achieve this as the final goal of this experiment -- with the Dell P4317Q displaying the 4 machines all at the same time -- four split screens at 1920x1080 resolution each, each split screen for each of linux, high sierra, windows, and also my PS 4 Pro.

I wonder what happens if I install Linux Gentoo on my MacBook Pro, then compile and run QEMU KVM on it, and passthrough the GPU to the guest OS. Without X Window running and just using ssh for control, I'd suppose the Linux overhead should be relatively small.

Conclusion:

With all this running on top of Gentoo, there's really no need to reboot or to troubleshoot incompatible hardware. There are no crashes in the host or the guest, except that sometimes the guest os refuses too fire up (I suspect it has something to do with the GPU needing to power-cycle with a hard shutdown or a reboot in order for the guest to boot again; but I am unsure). Also, backup of guest OSes is easy -- I just copy the qcow2 hard disk file into storage. And I can fire up and power off the guest anytime I wish without interfering with the Gentoo host running underneath, which has basically become transparent. Also, this setup seems, in theory, more resistant to problems when upgrading the OS. And by the way, the messages app seems to work out of the box too.

It is interesting to note that with KVM allowing guest to access many aspects of the host hardware directly, in addition to passing through control of the GPU and USB controllers, discs, other devices etc. to the guest directly, plus pinning CPUs to the guest, the guest is arguably a hybrid of hardware and software, rather than just software. In fact, to me at least, it feels more bare metal than software.

Feel the POWER of Gentoo.

Disclaimer:

*All for fair-use, learning, experimental testing only*

Screenshots:

<https://imgur.com/lxjIFUV> (High Sierra GPU Passthrough at full 4K glory)

<https://imgur.com/Y66Yd8a> (High Sierra + Linux + PS4 Pro)

<https://imgur.com/yOSIQIg> (Win 10 + Linux + PS4 Pro)

<https://imgur.com/y6IgTAm> (Apple Magic Mouse (Series 1))

Credit:

D. Kholia (https://github.com/kholia)
 
Last edited:
Joined
Nov 3, 2017
Messages
6
Hello and thanks for sharing you setup.
I want to achieve something simillar with my more modest pc and I have just a small thing to solve before I get it right,this is my first post here so if I dont provide enough info please tell me.
I have successfully created gpu passthrough on my linux mint with windows 10,everything is working fine.

I found that kholia repository and followed the steps and managed to install osx high sierra
and run it-it works fine.

gpu passthrough also seems to work(I can see the clover boot on my second monitor)
but I cant seem to pass my usb controller to the vm-
when I type lspci -nn I get:
...
00:14.0 USB controller [1c02]: Intel Corporation Device [5076:a1af] # id number is place holder

(its seems I have only one usb controller.)
...
just like your script,so I added to /etc/modprobe.d/vfio.conf "options vfio-pci ids=5076:a1af"
and -device vfio-pci,host=00:14.0 \ to the bash file.

But when I ran the script I get the error -"qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio error: 0000:00:14.0: failed to open /dev/vfio/4: No such file or directory"

I would apreciate some help,I can run synergy to get a mouse working but I need to passthrough my wacom tablet to the vm.
It's a shame you can't passthrough individual usb devices like you can with windows.

my script:
#!/bin/bash

##################################################################################

MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,avx2,+smep"

~/QEMU/bin/qemu-system-x86_64 -enable-kvm -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
-machine pc-q35-2.9 \
-smp 4,cores=2 \
-m 12G \
-mem-path /dev/hugepages \
-mem-prealloc \
-balloon none \
-usb -device usb-kbd -device usb-tablet \
-device isa-applesmc,osk="apple_key_here" \
-drive if=pflash,format=raw,readonly,file=OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=OVMF_VARS-pure-efi-1024x768.fd \
-smbios type=2 \
-device ich9-intel-hda -device hda-duplex \
-device ide-drive,bus=ide.2,drive=Clover \
-drive id=Clover,if=none,snapshot=on,format=qcow2,file=./Clover.qcow2 \
-device ide-drive,bus=ide.1,drive=MacHDD \
-drive id=MacHDD,if=none,file=/media/user//mac_hdd.img,format=qcow2 \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-vga none \
-nographic \
 
Last edited:
Joined
Feb 8, 2014
Messages
8
Mac
Classic Mac
Mobile Phone
I have two USB controllers that show up under lspci -k:

I just pass the 00:14.0 one and all the USB ports in my machine gets send to the VM.

The second USB controller is for the Bluetooth and WIFI I think.

Anyhow -- if I have to guess -- you probably forgot to unbind the USB module (xhc_pci in my case) prior to vfio-pci being able to bind to the device, or you have the module compiled into the kernel as an "y" option in .config for linux such that it seized the device and vfio thus can't bind to the device.

My script under /etc/local.d/vfio-swaps.start (scripts thereunder gets started when system starts) is:

#!/bin/bash

modprobe -r snd_hda_intel &&
modprobe -r xhci_pci &&
modprobe -r ath10k_pci &&
mod probe vfio-pci &&
mod probe snd_hda_intel &&
mod probe xhci_pci

Try something like the above when your system starts and see if the card gets binded to vfio_pci.

Alternative, why not just blacklist whatever module that would have been binded the device had it not been for vfio? Blacklist the driver for the device, and then rebuild initramfs and reboot, and see if vfio_pci now binds to the device.
 
Joined
Nov 3, 2017
Messages
6
It seems that is exactly what happened. when entering lspci -k the driver that is listed is for the usb
is xhci_hcd .

after some googling I enter this command to find if it is compiled in kernel.

Code:
grep -i xhci_hcd /boot/config-$(uname -r)
I get :
Code:
CONFIG_USB_XHCI_HCD=y
creating the script and folder did not work.
trying the command you listed with my module in a terminal I get the message-
"modprobe: FATAL: Module xhci_hcd is builtin."

tried adding in /etc/modprobe.d/blacklist.conf file "blacklist xhci_hcd"
but that produced an error on that line when updating(is it the same as rebuilding?).


strange thing is if I type dmesg | grep vfio
I get it listed with my card:
[ 2.584968] vfio_pci: add [10ds:1301[ffff:ffff]] class 0x000000/00000000
[ 2.605002] vfio_pci: add [10ds:0fba[ffff:ffff]] class 0x000000/00000000
[ 2.605010] vfio_pci: add [1136:a1bf[ffff:ffff]] class 0x000000/00000000


But I guess it wont work if I wont remove that module,my linux knowledge is limited so i am not sure how to proceed.
I will have to research some more,thank you for pointing me in the right direction,any suggetions are highly appreciated though.
 
Last edited:
Joined
Feb 8, 2014
Messages
8
Mac
Classic Mac
Mobile Phone
If the USB module is selected to be built into the kernel rather than a module that is detachable (i.e., by using =y in .config for linux kernel, rather than =m) then you cannot detach it as far as I know. I assume you got your kernel prebuilt with the distribution you are using?

The way to fix this is to edit the .config file in the linux folder (usually /usr/src/linux) by changing the =y to =m for that module (use cat .config | grep -i config_moduleforyourdevice to search for it) for the device, then recompile the kernel and initramfs, and reboot, and your module should now be detachable using the script I included above. Your use of it did not work probably because you are not using Gentoo linux and your distribution probably has another way of initiating scripts at startup. What don't you install cronie, and then do a crontab -e command, and list this line in the crontab "@reboot sh /path/to/vfio-swaps.start" and cronie will automatically fire up the swapping at reboot.
 
Joined
Nov 3, 2017
Messages
6
If the USB module is selected to be built into the kernel rather than a module that is detachable (i.e., by using =y in .config for linux kernel, rather than =m) then you cannot detach it as far as I know. I assume you got your kernel prebuilt with the distribution you are using?
yep the default kernel of linux mint 18.2 sonya(4.8.0-53-generic)
The way to fix this is to edit the .config file in the linux folder (usually /usr/src/linux) by changing the =y to =m for that module (use cat .config | grep -i config_moduleforyourdevice to search for it) for the device, then recompile the kernel and initramfs, and reboot, and your module should now be detachable using the script I included above. Your use of it did not work probably because you are not using Gentoo linux and your distribution probably has another way of initiating scripts at startup. What don't you install cronie, and then do a crontab -e command, and list this line in the crontab "@reboot sh /path/to/vfio-swaps.start" and cronie will automatically fire up the swapping at reboot.
I downloaded and compiled a kernel with the options mentioned in the .config(xhci_hcd and xhci_pci set to "m") I am now able to detach the moduls by running your script or adding "install xhci_hcd /bin/false" to /etc/modprobe.d/blacklist.conf to make it disable the module at startup.
but it seems to disable my usb controller completely so I cant use my mouse or keyboard,the vfio still wont load for whatever reason.
I regressed to the previous kernel for now.



 
Joined
Feb 8, 2014
Messages
8
Mac
Classic Mac
Mobile Phone
@graypersona


You can passthrough individual USB devices though. All you need to do is include this in your qemu command line:

"-usb -device usb-host,host-bus=1,hostaddr=3"

example, since my testing keyboard/trackpad combo is listed under lsusb command as being at bus 1 address 3.

Why don't you do a lsusb command to see the bus and address of the usb device you want to pass through, then adjust the above and put that in the command line?

QEMU in my anecdotal experience is FAR MORE powerful than VMWARE or parallels etc. You can do almost anything on it to make it near native
 
Joined
Nov 3, 2017
Messages
6
@graypersona


You can passthrough individual USB devices though. All you need to do is include this in your qemu command line:

"-usb -device usb-host,host-bus=1,hostaddr=3"

example, since my testing keyboard/trackpad combo is listed under lsusb command as being at bus 1 address 3.

Why don't you do a lsusb command to see the bus and address of the usb device you want to pass through, then adjust the above and put that in the command line?

QEMU in my anecdotal experience is FAR MORE powerful than VMWARE or parallels etc. You can do almost anything on it to make it near native
does it work for you?I tried it because that is the way I do it with my windows passthrough but it does not seem to work with High Sierra.

in the kholia repository I found this note:

"However USB passthrough of EHCI, and XHCI (USB 3.0) devices does not work with
+macOS Sierra. See https://bugs.launchpad.net/qemu/+bug/1509336 for
+confirmation. According to this bug report, USB passthrough does not work with
+versions >= Mac OS X El Capitan guests."
+
 
Joined
Feb 8, 2014
Messages
8
Mac
Classic Mac
Mobile Phone
does it work for you?I tried it because that is the way I do it with my windows passthrough but it does not seem to work with High Sierra.

in the kholia repository I found this note:

"However USB passthrough of EHCI, and XHCI (USB 3.0) devices does not work with
+macOS Sierra. See https://bugs.launchpad.net/qemu/+bug/1509336 for
+confirmation. According to this bug report, USB passthrough does not work with
+versions >= Mac OS X El Capitan guests."
+

It works for me, both of all USB passthrough, or each USB device individually.

I think it is just a matter of tweaking the QEMU command line and figuring out how your motherboard's USB controllers and ports are set up so that things get passthrough correctly.

Generally speaking, my personal view is that Linux (particularly Gentoo) can be tweaked to do just about anything that Windows can do. The problem is figuring out how to do it. But once you got it figured out, Linux gets rock solid at it. Case in point, this whole experiment.
 
Joined
Nov 3, 2017
Messages
6
Well I understand what the problem is.
when I remove the xhci_hcd module by using your scripts or entering the commands
Code:
echo "0000:00:14.0" > /sys/bus/pci/devices/0000:00:14.0/driver/unbind
echo 1046 0xb4af > /sys/bus/pci/drivers/vfio-pci/new_id
it actually does work and loads the vfio module to the controller,
using a virtual keyboard and an ps/2 mouse I was able to log in to the vm
and use my mouse and all other usb ports with my passthrouged controller and install nvidia drivers on my second monitor
with the gpu passthrough.

But it seems that my controller doesn't support resetting which is supposed to give it back to the host as when I pass it through I get the message-
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.

Controllers are cheap so I will probably just buy one that supports resetting.

for future googlers you can read this:
https://www.reddit.com/r/VFIO/comments/4o6wla/cant_get_vfiopci_to_bind_to_usb_30_card_at_boot/
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Passing_through_a_device_that_does_not_support_resetting
 
Top