Raspberry Pi 4 PCI Express: It actually works! USB3, SATA… GPUs?


Recently, Tomasz Mloduchowski posted a popular article on his blog detailing the steps he undertook to get access to the hidden PCIe interface of Raspberry Pi 4: the first Raspberry Pi to include PCIe in its design. After seeing his post, and realizing I was meaning to go buy a Raspberry Pi 4, it just seemed natural to try and replicate his results in the hope of taking it a bit further. I am known for Raspberry Pi Butchery, after all.

Before I tried desoldering anything, I set up my Pi for remote use; enabling SSH, WiFi, serial UART+ boot messages. The USB ports on the Pi board will not function after this modification, so this is super important.

Desoldering the USB3 chipset

As Tomasz lays out in his article, we need to remove the VL805 USB3 chip in order to access the PCIe interface. I used a hot air soldering station at low volume and medium-high temperature, with small nozzle head in order to not disturb the components nearby. I used flux along the edges and after a while the chip came away.

I tried to remove the solder from the large pad by mixing some low-temperature solder paste into whats there, but it’s not needed. Just cover it with capton or poor electrical tape. I used electrical tape just to make seeing the small wires I’d be soldering above it easier to see.

The pins to solder

The VL805 datasheet is confidential which makes posting parts of it here tricky. However, an image search for “VL805-Q6 QFN68” may yield interesting results for those interested in finding out more. The pins we are interested in, are as follows (note differing polarity from Tomasz’s work):

I used 0.1mm enameled wire, with each differential pair cut to nearly the same length. Tinning the end of the wire by scraping with a knife and dipping in a molten solder ball makes soldering to the pads we need easier. Holding the wires down with kapton tape, and using flux, with the smallest iron tip I had made the job just bearable under a microscope.

The rather untidy looking result:

First attempts

I was using a cheap PCIe riser as my “first interface” which was then to be connected to a PCIe switch card, which would then enable another 4 PCie slots. This first socket, as Tomasz mentioned in his article, needs the PCIe Reset pin pulled to 3v3, and both Reset and Wake signal traces to the USB socket cut. Note that whilst we now know where the PCIE Reset line is on the Pi, I have not needed to connect this as yet.

The first attempt to boot with this setup resulted in the Pi not managing to boot at all. After some wiggling of the PCIe slot, the raspberry Pi booted, but no devices were shown when running lspci (lspci can be installed via apt-get). The third attempt, however, after some professional wiggling of the PCIe slot, resulted in success! A booted Pi, with a PCIe switch!

[email protected]:~ $ sudo lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port

However, no devices were detected beyond the ASM1184e switch. Even a USB3 PCIe card using the same VL805 chip that I removed refused to detect. Running dmesg on the pi to get some driver details, I saw that whilst the PCIe link was active, and some busses were being assigned to the switch – it said that devices behind the bridge would not be usable due to bus IDs.

pci 0000:01:00.0: [1b21:1184] type 01 class 0x060400
pci 0000:01:00.0: enabling Extended Tags
pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
PCI: bus1: Fast back to back transfers disabled
pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci_bus 0000:02: busn_res: can not insert [bus 02-01] under [bus 01] (conflicts with (null) [bus 01])
PCI: bus2: Fast back to back transfers enabled
pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
pci 0000:01:00.0: devices behind bridge are unusable because [bus 02] cannot be assigned for them
pci_bus 0000:01: busn_res: [bus 01] end can not be updated to 02

The great thing about linux is that the source code is just there to dig into, and after finding where that “devices behind bridge are unusable” warning is printed I further discovered that the range of assignable busses can be limited by the Device Tree linux uses.

Device Trees

Device trees are simply a description of the hardware which is passed to the Linux kernel on boot. It has all the devices listed; their driver compatabilities, memory mappings, and configuration. It is particularly useful for describing peripherals which may not be discoverable via conventional means. On the root directory of your Raspbian Raspberry Pi boot SD volume, you will find bcm2711-rpi-4-b.dtb – which is the Compiled Device Tree binary. This binary blob is not user readable, but thankfully we can use the Device Tree Compiler to decompile it into a readable form. I did all of this within Windows Subsystem for Linux.

dtc -I dtb -O dts -o bcm2711-rpi-4-b_edit.dts bcm2711-rpi-4-b.dtb

That command will decompile into bcm2711-rpi-4-b_edit.dts. Searching this file for “pci” we find an entry for the PCIe – and it has a definition for bus-range which limits the bus IDs from 0 to 1. I change the entry from “<0x0 0x1>” to “<0x0 0xff>” and recompile to the binary form, and place that on the Raspbian SD card overwriting the default.

Command to recompile:

dtc -I dts -O dtb -o bcm2711-rpi-4-b.dtb bcm2711-rpi-4-b_edit.dts 

Success!

4 More PCIe busses!

And then I plugged some VL805-based USB3 cards in (one with a chained Network Interface). The setup looks as follows:

I have my Motorola LapDock USB hub connected to the Pi via the PCIe USB controller. The keyboard and trackpad work great!

Other Devices

I have a SATA controller based on a JMicron JMB363, which is detected correctly but there is no driver to load. This will require some linux driver/kernel fiddling to get a driver loaded correctly – but it’s very promising!

[email protected]:~$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
03:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 03)
05:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
06:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
[email protected]:~$ lspci -t
-[0000:00]---00.0-[01-06]----00.0-[02-06]--+-01.0-[03]----00.0
                                           +-03.0-[04]--
                                           +-05.0-[05]----00.0
                                           \-07.0-[06]----00.0

I also have tried some other fairly hilarious setups, including the following with a Radeon HD 7990 GPU, and another with a GTX 1060.

I’ll leave this here for now, whilst I read up on the Linux driver stack and how to build kernels for the raspberry pi 🙂

[email protected]:/home/pi# lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10)
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
04:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
05:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
06:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)

Thanks for reading! You can find me, as always, over on twitter @domipheus. Additionally, thanks to Tomasz Mloduchowski for his previous blogs which spurred my interest! I don’t fully understand why Tomasz had kernel panics using a VL805 board, but maybe it’s something to do with the Device Tree, and the fact I also had a PCIe switch.

Updates:

The Boat PC – a marine based Raspberry Pi project

Motivation

In late 2015 I was doing my usual head-scratching about what gifts to get various family members for the holiday season. My wife mentioned making something electronic for my father-in-laws boat, and after a few hours of collecting thoughts came up with an idea:

  • A Raspberry Pi computer, which could be powered off the boats 12v batteries.
  • This computer would have sensors which made sense on a boat. Certainly GPS.
  • I’d have some software which collated the sensor data and displayed it nicely.
  • This could plug into the onboard TV using HDMI.
  • It would all be put into a suitable enclosure.

Excellent – a plan. I expected the hardware part to be easy, the enclosure part fairly straightforward, and the software part to be an absolute disaster. I started searching for an already-existing project to take care of the software side of things.

That’s when I came upon a project called OpenPlotter. It’s a fully-featured linux distribution for Raspberry Pi, specifically for use on a boat, and includes the relevant software for calibrating, collating and transforming data from various sensors into a form that can be used practically. I’ve got to be honest here – OpenPlotter is solid, does exactly what it advertises, and very simple for someone familiar with RPi/Linux to set up and use.

After firmly deciding on OpenPlotter for the software, and knowing I’d be using an old Raspberry Pi 2 I had collecting dust, I looked at what hardware OpenPlotter supported. The list is fairly long, and gave me ideas I had not thought of previously – for example using a USB DVB-T television dongle as an AIS receiver with Software Defined Radio (SDR), allowing real-time data of nearby ships to be displayed. MarineTraffic uses this AIS data, but of course on a boat you can’t rely on an internet connection to pull data from – it’s much better to get the data directly from the VHF signals.

In addition to AIS and GPS, I’d add an Inertial Measurement Unit (IMU – basically an accelerometer, gyroscope and magnetometer in one) in the form of an InvenSense MPU-9150, and also a USB to RS422 converter. RS422 is specified as part of the protocol standard for NMEA 0183, which in turn is the communication specification used in marine electronics. Supporting input and output of direct NMEA using RS422 would allow for some extendibility, for example depth sensors that are already present can feed data into OpenPlotter using this port.

After going and purchasing all of these sensors, I realised that actually using the TV inside the boat isn’t going to be useful, as it’s not visible from the helm. Thankfully, OpenPlotter allows for headless operation, and will automatically set up a WiFi hotspot so you can connect a phone/tablet to the Raspberry Pi and control it using VNC or other software.

The Build

So, to clarify, all the hardware gubbins required:

  • Raspberry Pi 2
  • Invensense MPU9150 board
  • RTL2832U DVB-T USB
  • USB to RS422 Converter
  • USB GPS module
  • USB wifi module

Of course, we need some associated utility to make this into an actual device;

  • 12V to 5V power converter
  • Power switch & connector
  • Status LED
  • Enclosure

When I’ve done projects in the past (the biggest one being PiOnTheWall from years ago), I spend a significant amount searching for the right enclosure to put the hardware in. It’s not just a case of going and getting something that’s big enough to fit the contents, you need to know how thick the sides are, what kind of plastic is it, are there PCB standoffs included, are there vent holes?

After several days, I came up with the following which I got off ebay.

enclosure

I knew already the RTL2832U SDR dongle could run quite hot – so ventilation holes were a must. It’s the hottest part of this hardare, easily 60C+, whilst the Broadcom SOC of the Raspberry Pi will have to be working fairly hard to hit 45C. I did not plan to heatsink anything, and in the end it works fine without them. I did make a concious choice though to have the SDR board at the highest point in the enclosure, closest to the vents.

The design was simple – switch and status LED at the front, RS422, SDR antenna, Power In and Raspberry Pi Mini USB/HDMI/Audio out at the back. I removed all plastic covers from any USB devices, as they just bloated the inside, and I knew removing USB connectors would be a requirement. Laying out the components, I found one which worked well.

layout_annotated

The Raspberry pi would be put on metal standoffs – I used some spares I had from various PC motherboards and cases. I just drilled straight though the bottom of the plastic case with a bit size such that the thread would drive into the plastic.

In my previous Raspberry Pi project I butchered the board, and I’m pleased to say the only thing I had to do in this instance was make the fixing holes on the PCB slightly larger to accommodate the screws for standoffs.

rpi_drill

standoffs

The GPS and Wifi modules remained as dongles, simply connected into one dual header on the Raspberry Pi. To aid fitting all the boards into the enclosure, the male USB connector of the RTL2832U SDR dongle was placed on a ribbon cable. Additionally, the miniUSB cable for the RS422 converter was made small enough to fit in the limited space available. These two boards were physically fixed to the rear panel via bolts, and in the SDR boards case, a little shelf made from spare plastic.

422_sdr_cables

422_sdr_affixed

I’m not very good at making good panel openings, so sadly my HDMI and microUSB ports are very poor. At least they are at the back, where nobody should be able to see them 😉

Internally, all that was left was to connect the 12V->5V DC-DC converter to the Pi, put a power switch inline with the input 12v Power jack, attach the LED to 3v3 (there is a resistor in the LED leg heat-shrink), and fix the rest of it down with the same standoffs. It ended up looking fairly neat and tidy.

complete_internals

For those wondering, I connected the 5V output from the DC-DC converter direct to the 5V rail of the Pi. It bypasses some input protection which exists on the miniUSB power input. For me this is okay, I hoped it would allow the SDR USB dongle to draw more power than is ‘technically’ allowed from the onboard USB ports. I knew that was an issue back in the Raspberry Pi 1 days, and couldn’t remember if that was still the case with RPi 2.

The final rear panel:

complete_rear

The front of the enclosure, unit powered and closed.

complete_front

You will notice the USB socket on the front; I thought it could be useful to trickle charge phones or the tablet that would connect through WiFi to offer controls. I connected the unit to an HDMI monitor to do first-time OpenPlotter setup, making sure the sensors worked, and then switched it into headless mode, with VNC and NMEA 0183 output over it’s own ad-hoc WiFi hotspot.

Testing on the boat!

One thing that I could not test at home and needed to do on the boat was calibrate and test the AIS Receiver. There was a long gap between the hardware being “complete” in summer 2016, and testing it on-board in spring 2017.

AIS runs off VHF frequencies of around 162MHz, a wavelength of 1.85 meters. The boat has a marine antenna already which will work fine, but when I brought the device for testing did not have the correct connector to interface with the SDR dongle.

antenna

Because of this, I made a quick and dirty 1-wire, quarter wavelength antenna. I used a good quality coax, with one end exposing only the inner core to a length of 46 centimeters. I then hooked this around a bit of the boat outside. It wouldn’t get long range, but hoped I’d get some ship signatures in the marina – and it did! After following the calibration instructions on the OpenPlotter guide, I rebooted and after a few minutes the tablet (now connected to the RPi using wifi) displayed the following:

tablet_ais

We used an Android app called SailTracker which takes the collated NMEA datastream and displays the data in an appropriate format. There are several paid apps that come complete with nautical maps, which is neat.

installed

And that’s it! All installed, wired into the 12v, and also now using the VHF antenna at the top of the mast. I’m quite proud with how this one turned out, and I’m very impressed with the OpenPlotter distribution for allowing this project to work as well as it did.

What I’d change

There are 3 things I’d change if I was to do this again:

  1. Changing the front panel LED to RGB, and have it a real status LED rather than power. For example,
    • solid blue: OS booting,
    • flashing green: OpenPlotter starting services,
    • solid green: WiFi hotspot up,
    • red would be an error condition.
  2. Mounting the SDR dongle further in, allowing me to wire up the antenna input from the onboard mini MCX to a PL259 VHF connector on the rear panel. This would have eliminated some of the external complexity of needing various converters.
  3. I’d have a large cover over the microUSB/HDMI/audio raspberry pi connectors, as they are really only needed for debug, and it would have stopped me from making the messy cuts I did 🙂

Thanks for reading. If you have any questions or queries feel free to contact me at @domipheus.

Pi On The Wall – wall mounted home server – Part 4: Putting it together

This is part 4 of the Pi On The Wall build log, concerning modifications to the enclosure and how everything comes together into its final form factor. Part 3 was about power consumption, and optimizing it for low-power and ultimately low-temperature running. Previous parts can be found at Part 2 and Part 1.

41zJZupMBKLThe choice to use a standard (for the UK, anyway) footprint for the Pi On The Wall was made very early on. Ease of mounting to the wall (compatibility with existing wall-fixtures) and a wide range of cheap thermostats to choose from for simply utilising their outer shell.

thecaseAfter removing the guts of the £12 wall thermostat I purchased, the inside was what you’d expect – mounting posts for the PCB, guides for the LCD panel, and a PSU block behind it.

Internally, all I was planning on doing was removing as much of the PCB and LCD supports as possible – there was quite a few and they really restricted the usable space inside the case. A hobby drill and knife were enough to completely re-engineer the inside to suit my needs for maximum possible space (whilst still keeping some of the LCD guides, for the new panel). The following image shows the sort of supports that existed.

front_rear_unmodified On the rear panel, I removed the PSU block and smoothed the pcb supports whilst removing a notch for the flush USB connector.

case_modificationThe front panel ended up looking as follows after installation of the front TFT, after many hours of making sure it was perfectly straight, and holding it down with glue.

front_screen_buttonguidesNote the hole bottom right, for the PIR sensor/lens. It was countersunk to make it ‘neat’ and I messed it up. Which it why there is now a grommet on the opposite side, as the hole is too large for the sensor.

Also of note are the red straw tubes hot-glued into locations around the buttons. The straws are actually cut from the one provided with a can of WD-40, as I didn’t have anything else the correct size. The reason for these straws are made clear with the next image.

button_pcb_guidesThey form the guide for the small off-cut of PCB from the original thermostat which housed the tactile switches for the front buttons. The top one is flush with the PiTFT circuit board giving it a good fit.

frontpcb_wiringThe traces on that board were cut and the switches were directly connected to the 4 GPIO buttons on the PiTFT. There is not much else on this half of the enclosure, the main components (minus the actual panel itself) are laid out below.

raw_front_pcbsThe PIR motion sensor module was rather annoying in that it took 5v input and provided a 4.4v logic level signal when movement was detected. This is voltage divided through to 3v3 for a Pi GPIO input and is designed to switch on the back light of the PiTFT when movement is detected in front of the unit and works well. You can see some foam underneath the tactile switches, there is some on the other side as well when the case is fully closed to keep the pressure correct for the panel buttons. The PIR unit itself was deconstructed for ease of positioning the components, with 0.2mm enamelled wire being used to connect the sensor to the control board, as normal wire was too thick to also allow for the USB connector to fit snug above it.

heatsink_clampSomething I failed to write up well was the heatsink I designed for the Pi On The Wall. It needed to be incredibly low-profile due to the fact the PiTFT board was only 4mm above the Broadcom SoC. Using some very thin metal I cut out a strip which I moulded into a clip-like shape, which would clip either side of the PCB and provide some feedback into the top of the SoC whilst giving enough clearance to other components on the board.

heatsink_sinkAfter that some copper wire was soldered to the metal and arranged so the top of the SoC conducted thermally with the copper, with the help of some paste. The wires were routed to what would be the top side of the board, where the case has ventilation holes. I cut a small aluminium heatsink into two strips and then soldered the wire to those sinks, very poorly. Aluminium does not solder well and I only managed a very weak bond using the oil+scrape method. Soldering a heatsink really is as stupid as it sounds.

Amazingly, however, the heatsink works. It averaged 2 degrees Celsius of a saving over a days worth of burn-in testing. On a very hot day (for UK standards) the SoC was reporting after several hours of burn in that it was 42 degrees Celsius. This was using the ondemand governor with a 350-500MHz working range. I actually don’t think underclocking from 700 to 500 made a massive difference, but I’ve kept it at that value.

BujMHAACcAE0g93The last item is the power supply block. The AC-DC adaptor I choose fit nicely into the case and had enough room to fit the MP2307 5v->3v3 converter, as well as some required capacitors. The capacitors I eventually settled on are different from part 3, as I went from a phone charger 5v rail to this and it had different characteristics. A switch is included to hard-restart the Pi when required. 200uF is provided on the 5v rail mainly to allow for USB hot plugging, which the Pi is spectacularly terrible at.

power_sch_latest powerpsu_cableFinally, I made a video which goes into further detail about how the components all fit together. This was released a few weeks ago, but compliments this part of the build log well, so it is below.

The last part will be on the software running on the PiOnTheWall. Thanks very much for reading, I hope it was interesting!

As always, you can reach me on twitter @domipheus.