If you are trying to get the guest framebuffer onto a physical graphics
device then PVFB is not what you need. Instead you need to do device
passthrough of that device to the guest (which means denying it to
dom0). Passthrough is still a work in progress on Xen on ARM (likely to
land for 4.6) and *requires* an SMMU to be present in the hardware.