Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Bjorn Helgaas <helgaas@kernel.org>]

On Sat, Oct 02, 2021 at 10:42:16PM +0530, Ajay Garg wrote:
> Thanks Bjorn, that certainly helped.
> Especially, it is nice to see that ptr->resource[0].start does give
> e2c20000 :)
> 
> Also, idiotic of me of making newbie kernel-format-specifier
> mistakes :-|

If you're referring to %p being hashed, that's bitten me more than
once.  I understand the value of it, but it's annoying in the
day-to-day development process ;)

> So, now the outputs corresponding to
> 
>     printk("Base virtual-address = [%lx]\n", bar0_ptr);
>     printk("Base physical-address (form-1) = [%lx]\n", dev->resource[0].start);
>     printk("Base bus-address = [%lx]\n", pci_bus_address(dev, 0));
>     printk("BAR 0: %pR\n", &dev->resource[0]);
> 
> are :
> 
>     Base virtual-address = [ffffa0c6c02eb000]
>     Base physical-address (form-1) = [e2c20000]
>     Base bus-address = [e2c20000]
>     BAR 0: [mem 0xe2c20000-0xe2c201ff]
> 
> All as expected.
> Plus, all the lower 12 bits are now same everywhere (due to the 4 KB
> page-size alignment in x86, right)?

Yes.

> My major missing understanding regarding this, is that we use the
> iowrite*/ioread* functions, using bar0_ptr as the
> base-(virtual-)address.
> Thus, bar0_ptr *is* very well the kernel-virtual-address, which maps
> to some physical-address (hopefully e2c20000), which directly triggers
> the write/read with the pci-device.

Yes.

> Right now, the physical-address (form-1) we have printed, is via the
> data-structure field.
> However, looking from the virtual-address <=> physical-address
> translation from the usual memory write/read datapath's perspective, I
> am still not able to coalesce things.
> 
> 
> In the same run as above, if I add the following statements :
> 
>     printk("Base physical-address (form-2) = [%lx]\n", virt_to_phys(bar0_ptr));
>     printk("Base physical-address (form-3) = [%lx]\n",
> virt_to_phys(*((uint32_t*)bar0_ptr)));
>     printk("Base physical-address (form-4) = [%lx]\n",
> virt_to_phys(*((uint64_t*)bar0_ptr)));

I don't think these last two make any sense.  You're doing an MMIO
read from the address bar0_ptr, so this value (0x721f4000001e) came
from the PCI devices.  There's no reason to think it's a kernel
virtual address, so you shouldn't call virt_to_phys() on it.

> I get :
> 
>     Base physical-address (form-2) = [12e6002eb000]
>     Base physical-address (form-3) = [721f4000001e]
>     Base physical-address (form-4) = [721f4000001e]
>
> Looking at the function-doc for virt_to_phys(), it states :
> 
> ########################################################
> * This function does not give bus mappings for DMA transfers. In
> * almost all conceivable cases a device driver should not be using
> * this function
> ########################################################
> 
> 
> 
> So, two queries :
> 
> 1)
> Does the above comment apply here too (in MMIO case)?

Yes.  You should not need to use virt_to_phys() for PCI MMIO
addresses.

> 2)
> If yes, then what is the datapath followed for our case (since
> conventional virtual-address <=> physical-address translations via MMU
> / TLB / page-tables is out of the picture I guess)?

This path is IN the picture.  This is exactly the path used by drivers
doing MMIO accesses.

The CPU does a load from a virtual address (0xffffa0c6c02eb000).  The
MMU translates that virtual address to a physical address
(0xe2c20000).  The PCI host bridge decodes that address and generates
a PCIe Memory Read transaction to the bus address 0xe2c20000.

Your dmesg log should have lines similar to this:

  pci_bus 0000:00: root bus resource [mem 0x7f800000-0xefffffff window]
  pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]

that show the parts of the CPU physical address space that result in
MMIO to PCI devices.  0xe2c20000 should be inside one of those
windows.

Bjorn