Electronic – How does a PCIe device know that its 4K configuration space is mapped to memory address

Each PCI device (when I write PCI, I refer to PCI 3.0, as opposed to PCIe) has two "ranges" - configuration range (CFG) and "memory mapped input-output" range (MMIO). I won't deep dive into the concepts of address spaces and MMIO because it will make the answer too long and complicated. Google them if they are not familiar to you. In short: CFG range is a standard set of registers used to configure the PCI device; MMIO range is a customary set of registers. In other words: CFG ranges are the same across all PCI devices (there might be slight differences, but the majority of registers are standard); MMIO ranges are device specific (NOTE: while the terms "range" and "space" are not synonyms, there is a consensus to call an MMIO range of the device MMIO space. I'll use them interchangeably)

Now, the size of CFG space is standard - there is an upper bound on the number of registers CFG space can contain and it is the same for each PCI device. Usually, the actual number of registers in CFG space is much smaller than the maximal. The size of MMIO space, on the other hand, is not constant. Why? Well, different devices need different number of registers for communication.

Now think about it for a moment: if the size of MMIO space is not constant, then we need to provide the information about this size of a particular device to the computer in some way, right? One option would be to manually define these parameters for each device. It is the way the early computers worked: you really had to configure each device you plug into a computer by hand. Today we are lazy and want the "plug-and-play" functionality - the computer must obtain this info by itself the moment a new device is added.

In order to allow for "plug-and-play" in PCI devices, the concept of MMIO Base Access Registers (MMIO BARs) was introduced. These registers reside in CFG space of each device (I think that there up to five BARs per CFG space are allowed). The flow is as follows:

1. a computer knows to search for these registers during a startup
2. a computer reads the BARs in order to understand what sizes of MMIO ranges does this device require
3. a computer allocates the device's MMIO spaces, which become standard MMIO ranges in the global MMIO space
4. a computer writes back to MMIO BARs the addresses assigned for each device's MMIO range in the global MMIO space.

The above 4 stages are known as "enumeration" of the device - i think that it is usually the BIOS who performs devices' enumeration during the startup.

Except for BARs, CFG space of a device contain many more registers. All other writes to CFG space, which is not part of "enumeration" flow, are called "configuration" of the device. This includes runtime configurations such as: interrupt selection, MSI vectors and addresses, device's power states and many more.

In summary: "enumeration" is the flow executed at startup which allocates MMIO ranges to all the devices. "configuration" is all other writes to CFG space of the device (in general, "enumeration" is included in "configuration").

NOTE: this description is simplified. There are more aspects both to "enumeration" and to "configuration".

Offtopic:

It seems that you are new to PCIe and want to get a fast introduction. While your desire is totally understandable (it will take you a way more time to understand PCIe if you don't ask questions), I believe that nothing can replace the original spec. The problem is that PCIe spec is written in the manner which assumes that you're familiar with PCI (at least it seemed that way to me) - you need to read it first. So, start by googling PCI 3.0 specification and PCIe 2.1 specification. These documents are frustratingly long, but they'll become your bible if you're really going to work with PCIe devices.

It's been awhile since this was asked, but I hate orphaned questions :)

First, let's over-simplify a modern x86 platform and pretend it has 32-bits of address space from 0x00000000 to 0xFFFFFFFF. We'll ignore all the special / reserved areas, TOLUD (top of lower usable DRAM, Intel parlance) holes, etc. We'll call this system memory map.

Second, PCI Express extends PCI. From a software point of view, they are very, very similar.

I'll jump to your 3rd one -- configuration space -- first. Any addresses that point to configuration space are allocated from the system memory map. A PCI device had a 256 byte configuration space -- this is extended to 4KB for PCI express. This 4KB space consumes memory addresses from the system memory map, but the actual values / bits / contents are generally implemented in registers on the peripheral device. For instance, when you read the Vendor ID or Device ID, the target peripheral device will return the data even though the memory address being used is from the system memory map.

You stated these are "allocated into RAM" -- not true, the actual bits / stateful elements are in the peripheral device. However, they are mapped into the system memory map. Next, you asked if it was a common set of registers across all PCIe devices -- yes and no. The way PCI config space works, there is a pointer at the end of each section that indicates if there is more "stuff" to be read. There's a bare minimum that all PCIe devices have to implement, and then the more advanced devices can implement more. As for how useful it is for functional operation, well, it's mandatory and heavily utilized. :)

Now, your question about BARs (base address registers) is a good space to segue into memory space and I/O space. Being somewhat x86 centric, the specification allows the specification of a BAR size, in addition to the type. This allows a device to request a regular memory-mapped BAR, or a IO space BAR, which eats some of the 4K of I/O space a x86 machine has. You'll notice that on PowerPC machines, I/O space BARs are worthless.

A BAR is basically the device's way to tell the host how much memory it needs, and of what type (discussed above). If I ask for say 1MB of memory-mapped space, the BIOS may assign me address 0x10000000 to 0x10100000. This is not consuming physical RAM, just address space (do you see now why 32-bit systems run into issues with expansion cards like high-end GPUs that have GB of RAM?). Now a memory write / read to say 0x10000004 will be sent to the PCI Express device, and that may be a byte-wide register that connects to LEDs. So if I write 0xFF to physical memory address 0x10000004, that will turn on 8 LEDs. This is the basic premise of memory-mapped I/O.

I/O space behaves similarly, except it operates in a separate memory space, the x86 I/O space. Address 0x3F8 (COM1) exists both in I/O space and memory space and are two different things.

Your last question, messages refer to a new type of interrupt mechanism, message signaled interrupts or MSI for short. Legacy PCI devices had four interrupt pins, INTA, INTB, INTC, INTD. These were generally swizzled among slots such that INTA went to INTA on Slot 0, then INTB on Slot 1, then INTC on Slot 2, INTD on Slot 3, and then back to INTA on Slot 4. The reason for this is that most PCI devices implemented only INTA and by swizzling it, having say three devices, each would end up with their own interrupt signal to the interrupt controller. MSI is simply a way of signaling interrupts using the PCI Express protocol layer, and the PCIe root complex (the host) takes care of interrupting the CPU.

This answer might be too late to help you, but maybe it will help some future Googler / Binger.

Finally, I recommend reading this book from Intel to get a good, detailed introduction to PCIe before you go any further. Another reference would be the Linux Device Drivers, an online ebook from LWN.