Introduction

A few weeks ago I wrote this post where I looked at some of the typical issues encountered when installing a Thunderbolt AIC into an unsupported PC. In that adventure we discover that functionality can be improved by patching the firmware in the Thunderbolt controller. We also found that, for Windows users, lack of PCI hot-plug support in the UEFI was a major roadblock to improving functionality further.

My theory was that if we can hack hot-plug support into the UEFI of an unsupported PC, we’d then get useable Thunderbolt. Today (for me, the last 3 weeks’ spare evenings) we’re going to put that to the test.

Context

This post concerns Thunderbolt 3 add-in card usage on PCs running Microsoft Windows which do not have any vendor Thunderbolt support.

There is much related discussion around Thunderbolt hot-plug in the “Mac hacking” community, where Thunderbolt is retrofitted to older Macs, or used with PC hardware running macOS. I do not have sufficient understanding of this area to say how much, if any of it, is relevant to the subject I discuss here. From what I have seen the technical challenges with Mac are quite different.

Development setup

I purchased an older motherboard specifically for this venture: The ASUS CS-B. It’s a straightforward Intel 4^th generation board with a Q87 chipset. Its UEFI is (of course), closed source, and provided by American Megatrends. It is ironic that at the time I write this their website front page reads “Embracing Open Source”.

My selection criteria:

• Two LPC bus attached COM ports for UEFI module debug
• The UEFI lives in a socketed PDIP SPI flash chip
• x4 PCIe slot from chipset (CPU PCIe does not support hot-plug)
• It has no supported PCI hot-plug scenarios, so we’ll get a good taste of how tough this kind of modification is likely to get
• No gamer kitsch to get in my way
• Being 4^th gen there is the possibility of integrating Kernel DMA protection in future. There is an example of that here.

The Thunderbolt AIC is the Gigabyte GC-TITAN RIDGE as used in my previous post.

UEFI PCI Hot-plug done properly

The correct way of implementing hot-plug according to the UEFI specification is to introduce a DXE driver which implements the EFI_PCI_HOT_PLUG_INIT_PROTOCOL interface, which is consumed by the PciBus DXE driver. This technique is not Thunderbolt specific, it is also used for CardBus, ExpressCard, and for proprietary laptop docking stations which contain PCIe peripherals for example.

PCI Hot-plug controllers are not plug and play. The hot-plug driver must be tailored for each individual scenario. For this reason, we cannot just pinch a DXE from a Thunderbolt supporting motherboard and expect it to work.

I knocked up a basic DXE driver implementing EFI_PCI_HOT_PLUG_INIT_PROTOCOL and put some debug statements in the calls I was expecting from the PciBus driver to see what happens. What I found was that the DXE core had loaded my driver, called my driver entry function but alas there were no calls from PciBus driver. Lights on, nobody home.

I then pulled out the PciBus DXE and ran through it with a disassembler to see what was going on. While the EFI_PCI_HOT_PLUG_INIT_PROTOCOL interface GUID does appear in the binary code, it turns out that nothing references it. The certain explanation is that it’s sitting statically defined at the top of a source file, but the code which uses it and would be calling my driver has been #ifdef’d out. This is a very frustrating discovery, but we cannot be surprised as the CS-B has no supported hot-plug scenarios, therefore ASUS had no reason to include it. To progress further, we must replace the CS-B’s stock PciBus driver with one that includes hot-plug support.

Replacing the PciBus DXE driver

My initial thought was to pinch a PciBus driver from a Thunderbolt supporting motherboard from a similar era, it would certainly have to include the needed calls into the hot-plug driver I’m trying to build. That it would have only been a Thunderbolt 2 implementation would not matter, but other things likely would. Might work. Might not. If it didn’t work, I’d have quite a job trying to figure out why or fix it without its source code, especially considering that it executes before the screen turns on.

The bigger prize would be to replace it with an open-source implementation. The EDK2 from TianoCore contains an open source PciBus DXE driver which I thought could probably be modified to do the job. This might seem overly ambitious to attempt but we have a few things working in our favour: The PciBus driver’s job is well defined by both the UEFI and PCI specification. Modifying it is frowned upon and explicitly discouraged by the UEFI development guidelines (but inevitably done in some cases).

Without any further hesitation, I built the EDK2’s PciBusDxe driver, with serial debug enabled, and overwrote the CS-B’s supplied equivalent. Can’t hurt to try right? I was surprised to find that it went through and successfully enumerated the bus. Unfortunately it bombed out when the proprietary AMIBIOS PciHostBridge driver rejected the SubmitResources() call from the EDK2 PciBusDxe driver, which is used to configure the overall system resources for all PCI devices.

It is at this point things get rather difficult. We cannot swap out the PciHostBridge driver in protest as it is certain to contain all sorts of scary board and chipset specific initialisation. We have to get on with it. To fix this we must work out what the original PciBus driver submits through this call so the appropriate modifications can be made. I ended up building a special shim DXE driver which inserts itself between the AMIBIOS PciBus driver, and the PciHostBridge driver. This driver hooks three different interfaces: EFI_DRIVER_BINDING_PROTOCOL, EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL and EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL. It was not trivial to build.

I now have a very nice shim driver which I can use to intercept the original PciBus driver’s calls to the rest of the system, including SubmitResources(). The problem was quickly identified and patched. Just a few flags not set correctly.

The next problem I hit is that once again I’m crashing out in a call to the PciHostBridge driver, but this time it’s the NotifyPhase(EfiPciHostBridgeEndEnumeration) call. It turns out that AMI do not use this event, and their PciHostBridge does not expect to receive it. I commented it out, and we’re moving again.

Next time I powered on the board, the screen turned on! Woohoo!

But instead of the splash screen, I’m seeing garble. Still this is very encouraging. The IGD must be 99% working. Perhaps the frame buffer is pointing to the wrong memory location? I suspected there was probably a missing BAR, so hammered out some more debug in the shim driver to dump out the PCI configuration space and spotted the problem quickly: One of the BARs in the IGD was set to a memory address of 0x00000000 when it was supposed to be 0xE0000000. I am unsure how this address is supposed to be obtained so have just hacked it for now.

On the next boot, the board showed the splash screen correctly, and went ahead and successfully booted Windows 10.

A moment to behold. My ASUS CS-B has just booted Windows 10 with a large critical DXE driver in its UEFI having been swapped out for an open source implementation. I had a quick whip around to check that all of the hardware was setup and working properly. I couldn’t see any immediate problems.

It’s worth briefly mentioning that the AMIBIOS and EDK2’s PciBus drivers are not the same implementation. This was immediately apparent in that the EDK2’s PciBus driver did not route legacy IRQs. AMIBIOS’s is an “everything” implementation, supporting every imaginable legacy/quirky scenario, whereas the EDK2’s is a “best case” implementation supporting only straight forward scenarios, and modern plug-and-play operating systems.

Where were we? Oh yeah, trying to add a hot-plug driver

After an entire week of screwing around, we now have a UEFI which is ready for a hot-plug driver to be added. It’s worth pointing out that developers at ASUS would have been able to do this in minutes by changing the build flags for the PciBus driver. Oh how it sucks to be the end user.

The job of a UEFI PCI hot-plug driver is quite simple. When the PciBus driver calls with the controller’s root bridge device, we return ACPI_ADDRESS_SPACE_TYPE_MEM and ACPI_ADDRESS_SPACE_TYPE_IO descriptors to reserve memory and I/O space, then when it calls in for the Thunderbolt ports, we return an ACPI_ADDRESS_SPACE_TYPE_BUS descriptor to reserve bus indexes. That is it.

I didn’t have any particular difficulty getting this part working. The newly substituted PciBus driver called my hot-plug driver and did the business. A quick check in device manager and HWiNFO64 shows we’ve got the resource allocations as intended.

Enabling hot-plug on the root port

What I do not have at this point is hot-plugging enabled on the root port the Thunderbolt AIC is attached to.

Unfortunately EFI_PCI_HOT_PLUG_INIT_PROTOCOL does cover the configuration of the root port the AIC is attached to. All I knew about how this was configured at the outset is what it says in the Q87 datasheet which is that hot-plug is enabled by the SLCAP register. Unfortunately this register is “write once”, it will have already been written to by something else in the UEFI, so there’s no point in trying to overwrite it in any of my code.

Some time ago I was looking at this guide where the author changes a hidden setting to enable hot-plug on an PCIe slot. I tend to be a bit skeptical about hidden settings. Just because they’re lurking in the descriptors, that doesn’t mean the OEM has included any code which makes use of them.

Instead I went on a fantastical, delightful journey of reverse engineering several DXE drivers. Starting from PchInitDxe which appears to be the first to write this register, then into SbDxe which builds the configuration for the PCH, which its self reads much of that configuration from the Setup data. Argh, so the guide I just linked to was in fact correct. Only difference is that the setting was at offset 0x101 for the CS-B’s x4 slot.

HWiNFO says hot-plug is enabled on the x4 slot, but it’s not working?

The reason for that turns out to be that HWiNFO reads the hot-plug capability from the PCI configuration space registers. Windows does not. It is stated on this page that Windows calls the _OSC() function from the ACPI tables to ascertain whether or not hot-plug can be enabled. The primary purpose of _OSC() appears to be to allow the UEFI to mask off hot-plug support from the O/S, for example if it is enabled at a physical level, _OSC() can lie to the operating system depending on whatever logic is implemented within it.

The logic in question in this particular instance is a global field called “NEXP” whose purpose is to toggle whether or not the operating system is allowed to use PCIe hot-plug. The field comes from the ACPI NVS. I put a cheeky hack into my hot-plug driver which sets it to ‘true’, and that’s it.

The end (of part 2)

After 3 weeks of hacking, I have Thunderbolt hot-plugging working on both ports of the AIC, without breaking secure boot, nor have I had to resort to any hacks to the operating system.

At this point I should mention that this is just the beginning of adding true Thunderbolt support. A full implementation includes a range of other gubbins such as ACPI table entries, SMM handlers and so on. And of course, there is that blasted “header” if we want to implement all of the power management features. Some of this will need to be pulled into this project. Figuring out exactly what will be the next phase.

Can I use this on my setup?

Not yet, no. At this stage this project is purely proof of concept. All of the code in the Github repo is hard coded for the ASUS CS-B. There are still a number of areas which require further investigation. On top of this, developing the process and documentation of how to integrate this work into other boards will take some time. That is assuming that anyone other than me is actually interested in this kind of thing 😉

If you’re interested in this subject, have some hardware to play around with and have a SPI flash programmer and are able to remove SPI flash chips from PCBs and re-program them, Please get in touch.

Posted in PC & Software, Thunderbolt