I have been learning about the Linux kernel and the most common bus type i always encounter is the PCI bus.
As an electronics Engineer, i decided to look at how this bus works as it seems to be the primary peripheral bus in a modern CPU system. All other buses (ie USB, Disk drives) seem to be connected to the PCI Express bus.
My confusion stems from the role of the ROOT complex. It appears to sit between the CPU and the PCI express ports. Also from a diagram i saw on Wikipedia, it also sits between the CPU and memory.
-
Does all memory access from the CPU pass through the Root Complex ?
-
Is the root complex the main facilitator of memory mapped IO. As in, is the root complex responsible for detecting addresses that map to devices and in turn forward said data to the right PCI express endpoint ?
-
From the above, is it safe to assume that physical RAM is treated as an endpoint ?
Best Answer
Has nothing to do with cpus, memory, etc. For PCIe you are either a root complex or an endpoint. As wikipedia points out and as would other places (many good resources out there) there is one root complex and the rest are endpoints. All point to point connections (through bridges as needed) to the root complex. Once enumerated typically the root complex side is the master, but either side can initiate. Think of it is a topology thing similar to usb, but, both sides can initiate rather than just one.
The normal use case is in computers where the root complex is part of the host side, but it is a peripheral within the processors address space. Typically the use case is to allocate some space within the processors address space for the pcie address space. intel having been beat up by motorola fans and flat address space vs segmented, pcs are strongly tied to this illusion of a flat address space (as are/were motorola and every one else). pcie address space is separate from the host address space, but you can certainly implement your design so that one is part of the other....ish. The non-pcie items, ram, a number of peripherals and support logic is not related to pcie in any way they are simply designed into part of the cpu's address space. There are layers of address decoding that ultimately isolate these items. The root complex pcie peripheral itself is of course not in the pcie address space it is in the hosts address space, for a PC, the BIOS enumerates the pcie endpoints and per their requests and if space is available pcie address space which for a pc matches addresses within the x86 address space then a window into that pcie device is allocated (or multiple, depends on what the endpoint is asking for). In the 32 bit days and bios still support this, a 1 gig window was allocated to pcie, meaning you could only then have 3GBytes for other things, why your computer even if you had a 64 bit and 16GBytes of memory the reality was you had N-1GBytes. Then with 64 bit processors and wider address space provided an opportunity to put the pcie address above the amount of sram (can technically have a machine with that much ram and still have to discard some), and then wisely the window is now typically 2GBytes in size, so all of your pcie endpoints have to live within that window. This is all an illusion of a flat address space, your video card for example may very well have quite a bit of memory but you can only view a small part of it through a window, basically segmented. By the time the operating system boots on an x86 based pc this has all been done and the operating system discovers what the bios has allocated and runs with it.
your processor has a bus, that bus has an address space, the front line address decoders start to narrow down on who the accesses are for, ram, peripherals. Of the peripherals is PCIe, and it is allocated an address space within the processors address space. PCIe itself has its own address space. When you access memory, this front line address decoder aims you essentially toward the cache which then on the backend of that eventually goes through the dram controller to access the dram. pcie is not required nor has anything to do with this. If you access a non pcie peripheral same deal the chip is designed to isolate an address space for this and aim these accesses either to on chip or off chip peripherals, pcie is not related in any way to this either. Then there are the pcie peripherals, and the front end decoders aim those accesses toward the pcie controller on chip or off and then the pcie controller turns those into pcie bus transactions, the host transactions wait, the pcie controller takes the direct path to the endpoint in question, completes the transaction and then releases the hosts transaction. If this happens to be a usb controller, then you get into that address space, wait for those transactions and so on. These new form factor solid state "drives" are or can be pcie and would live in the pcie address space, but the disk controllers that managed spinning media were generally pcie endpoints as well on the motherboard or within the main x86 chip.
If it is not an x86 pc then it is however they choose to design it, no need for the illusion of a flat address space although some semblance of one will still be there, the host and the pcie address spaces dont have to line up though. the designers are free to allocate as much or little of the hosts address space to pcie accesses. It is still wise to have a pretty good sized window but it depends on what the overall design is and how many endpoints are expected. Enumeration can happen before the os or during, up to the system designers. Non x86 pc designs are not required to be influenced by how the pc has evolved.
an x86 design for a long while was a multi-chip deal where you had peripherals off chip from the x86 itself, then the north/south bridge chips with super i/o chips and so on and to some extent they have moved toward putting as much as they can on the main chip (well could be a multi-chip module) including the pcie controller. so have to be a bit vague. there are still many off chip components on the motherboard as you can easily see but the peripherals including the memory controller and pcie controller and items not related to each are on the same chip/module in current implementations. That doesnt make for anything magical you still have a cpu that has an address/data bus (busses) that you need address decoders to slice up that address space and point the accesses to the peripheral/controller that handle those transactions so they can be completed. One of which is the pcie controller and its address space within the cpus address space.
get a datasheet or users/programmers guide for an mcu where they show the address space. its no different for an operating system capable cpu, you have some rom (flash) space for booting or some booting solution based on non-volatile memory, you have some ram space and individual peripherals have their own space often all of the peripherals live with in a chunk of the address space. No different here except that main memory on a pc is not on the processor chip nor are (all of the) pcie peripherals.