[Question] Custom MMIO handler - is it possible?

[Question] Custom MMIO handler - is it possible?

[nowicki]

Hello,

I'm trying to implement a fake PCIe device and I'm looking for guidance 
(by fake I mean fully software device).

So far I implemented:
- fake PCIe bus with custom fake pci_ops.read & pci_ops.write functions
- fake PCIe switch
- fake PCIe endpoint

Fake devices have implemented PCIe registers and are visible in user 
space via lspci tool.
Registers can be edited via setpci tool.

Now I'm looking for a way to implement BAR regions with custom memory 
handlers. Is it even possible?
Basically I'd like to capture each MemoryWrite & MemoryRead targeted for 
PCIe endpoint's BAR region and emulate NVMe registers.

I'm in dead-end right now and I'm seeing only two options:
- generate page faults on every access to fake BAR region and execute 
fake PCIe endpoint's callbacks - similar/the same as mmiotrace
- periodically scan fake BAR region for any changes

Both solutions have drawbacks.
Is there other way to implement fake BAR region?


Regards,
Mateusz

Re: [Question] Custom MMIO handler - is it possible?

[Bjorn Helgaas]

On Wed, Jan 31, 2024 at 08:42:18PM +0000, nowicki@posteo.net wrote:
> Hello,
> 
> I'm trying to implement a fake PCIe device and I'm looking for guidance (by
> fake I mean fully software device).
> 
> So far I implemented:
> - fake PCIe bus with custom fake pci_ops.read & pci_ops.write functions
> - fake PCIe switch
> - fake PCIe endpoint
> 
> Fake devices have implemented PCIe registers and are visible in user space
> via lspci tool.
> Registers can be edited via setpci tool.
> 
> Now I'm looking for a way to implement BAR regions with custom memory
> handlers. Is it even possible?
> Basically I'd like to capture each MemoryWrite & MemoryRead targeted for
> PCIe endpoint's BAR region and emulate NVMe registers.
> 
> I'm in dead-end right now and I'm seeing only two options:
> - generate page faults on every access to fake BAR region and execute fake
> PCIe endpoint's callbacks - similar/the same as mmiotrace
> - periodically scan fake BAR region for any changes
> 
> Both solutions have drawbacks.
> Is there other way to implement fake BAR region?

Sounds kind of cool and potentially useful to build kernel test tools.

Is the page fault on access option a problem because you want better
performance?  I assume you really *want* to know about every write and
possibly even every read, so a page fault seems like the way to do
that.

Maybe qemu would have some ideas?  I assume it implements some similar
things.

Bjorn

Re: [Question] Custom MMIO handler - is it possible?

[Mateusz Nowicki]

Thanks for a quick reply Bjorn!

Actually performance is not the biggest concern.
Mmiotrace has documented SMP race condition:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/trace/mmiotrace.rst#n135

Also playing correctly with page fault is quite a challenge. I'm trying 
to find a simpler/easier solution :)

Thanks for Qemu tip! I'll take a look


On Wed, Jan 31 2024 at 15:20:09 -06:00:00, Bjorn Helgaas 
<helgaas@kernel.org> wrote:
> On Wed, Jan 31, 2024 at 08:42:18PM +0000, nowicki@posteo.net wrote:
>>  Hello,
>> 
>>  I'm trying to implement a fake PCIe device and I'm looking for 
>> guidance (by
>>  fake I mean fully software device).
>> 
>>  So far I implemented:
>>  - fake PCIe bus with custom fake pci_ops.read & pci_ops.write 
>> functions
>>  - fake PCIe switch
>>  - fake PCIe endpoint
>> 
>>  Fake devices have implemented PCIe registers and are visible in 
>> user space
>>  via lspci tool.
>>  Registers can be edited via setpci tool.
>> 
>>  Now I'm looking for a way to implement BAR regions with custom 
>> memory
>>  handlers. Is it even possible?
>>  Basically I'd like to capture each MemoryWrite & MemoryRead 
>> targeted for
>>  PCIe endpoint's BAR region and emulate NVMe registers.
>> 
>>  I'm in dead-end right now and I'm seeing only two options:
>>  - generate page faults on every access to fake BAR region and 
>> execute fake
>>  PCIe endpoint's callbacks - similar/the same as mmiotrace
>>  - periodically scan fake BAR region for any changes
>> 
>>  Both solutions have drawbacks.
>>  Is there other way to implement fake BAR region?
> 
> Sounds kind of cool and potentially useful to build kernel test tools.
> 
> Is the page fault on access option a problem because you want better
> performance?  I assume you really *want* to know about every write and
> possibly even every read, so a page fault seems like the way to do
> that.
> 
> Maybe qemu would have some ideas?  I assume it implements some similar
> things.
> 
> Bjorn

Re: [Question] Custom MMIO handler - is it possible?

[Bjorn Helgaas]

On Thu, Feb 01, 2024 at 03:38:42PM +0000, Mateusz Nowicki wrote:
> Thanks for a quick reply Bjorn!
> 
> Actually performance is not the biggest concern.
> Mmiotrace has documented SMP race condition:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/trace/mmiotrace.rst#n135
> 
> Also playing correctly with page fault is quite a challenge. I'm trying to
> find a simpler/easier solution :)

I don't know how to make better handler for this.  Anything we do
would probably involve VM protection to catch accesses, even if that's
just in the kernel and not visible to userspace, and would probably
have the same SMP issue mentioned above.

Bjorn

> On Wed, Jan 31 2024 at 15:20:09 -06:00:00, Bjorn Helgaas
> <helgaas@kernel.org> wrote:
> > On Wed, Jan 31, 2024 at 08:42:18PM +0000, nowicki@posteo.net wrote:
> > >  Hello,
> > > 
> > >  I'm trying to implement a fake PCIe device and I'm looking for
> > > guidance (by
> > >  fake I mean fully software device).
> > > 
> > >  So far I implemented:
> > >  - fake PCIe bus with custom fake pci_ops.read & pci_ops.write
> > > functions
> > >  - fake PCIe switch
> > >  - fake PCIe endpoint
> > > 
> > >  Fake devices have implemented PCIe registers and are visible in
> > > user space
> > >  via lspci tool.
> > >  Registers can be edited via setpci tool.
> > > 
> > >  Now I'm looking for a way to implement BAR regions with custom
> > > memory
> > >  handlers. Is it even possible?
> > >  Basically I'd like to capture each MemoryWrite & MemoryRead
> > > targeted for
> > >  PCIe endpoint's BAR region and emulate NVMe registers.
> > > 
> > >  I'm in dead-end right now and I'm seeing only two options:
> > >  - generate page faults on every access to fake BAR region and
> > > execute fake
> > >  PCIe endpoint's callbacks - similar/the same as mmiotrace
> > >  - periodically scan fake BAR region for any changes
> > > 
> > >  Both solutions have drawbacks.
> > >  Is there other way to implement fake BAR region?
> > 
> > Sounds kind of cool and potentially useful to build kernel test tools.
> > 
> > Is the page fault on access option a problem because you want better
> > performance?  I assume you really *want* to know about every write and
> > possibly even every read, so a page fault seems like the way to do
> > that.
> > 
> > Maybe qemu would have some ideas?  I assume it implements some similar
> > things.
> > 
> > Bjorn
> 
>