* block device backed by DRM buffer object @ 2015-09-21 11:33 Steven Newbury 2015-09-21 13:05 ` Christian König 0 siblings, 1 reply; 7+ messages in thread From: Steven Newbury @ 2015-09-21 11:33 UTC (permalink / raw) To: dri-devel [-- Attachment #1.1: Type: text/plain, Size: 1761 bytes --] I have a mostly* headless server containing a Radeon discrete GPU. It occured to me that having a GiB or two of high speed memory sitting unused is pretty wasteful. Not an original thought; indeed there's a Gentoo wiki which describes how to map the memory as a mtd device: http://www.gentoo-wiki.info/TIP_Use_memory_on_video_card_as_swap There's also a driver (cudaram) written by Piotr Jaroszyński which allocates memory through CUDA and maps it to a block device: http://blog.piotrj.org/2011/03/cudaram-block-device-exposing-nvidia.htm l The Gentoo method is extremely limited, and of course isn't exactly safe, although using pci-stub would probably help. There's no arbitration between the DRM driver and the mtd phram driver. Most of all, it can only expose the memory mapped into the PCI aperture, likely somewhat less than the full VMEM. The second method obviously requires the proprietary NVidia driver and an NVidia gfx card, but it's good proof of concept at utilising a driver managed buffer object as a block device. I'm looking into creating a volatile block device driver which allocates VMEM from DRM, what if any is the right API to use? The DRM userspace API is well documented, but I'd like to make this a kernel module, so using a userspace API would seem inappropriate at best. Ideally I'd like to create something like the bcache flash_vol_create interface except expose a /sys/fs/drm-block/card0/vol_create (where drm-block is a placeholder for the proposed driver name) for each DRM device in the system, and likewise persistence through udev. So is there an API I can (mis-)use for this? Any suggestions? * There's a screen attached for maintainence, but it usually turned off. [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 159 bytes --] _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: block device backed by DRM buffer object 2015-09-21 11:33 block device backed by DRM buffer object Steven Newbury @ 2015-09-21 13:05 ` Christian König 2015-09-22 20:44 ` Steven Newbury 0 siblings, 1 reply; 7+ messages in thread From: Christian König @ 2015-09-21 13:05 UTC (permalink / raw) To: Steven Newbury, dri-devel [-- Attachment #1.1: Type: text/plain, Size: 2550 bytes --] In general a rather interesting idea and actually shouldn't be to hard to implement. The crux is that allocating memory is device driver dependent, so there isn't a general purpose API for doing so. Additional to that the CPU invisible VRAM is only accessible by the GPU so you at least need to be able to submit DMA commands to copy the data back and forth. For an experiment I suggest you do this with Radeon first and if that works generalize from there. Regards, Christian. On 21.09.2015 13:33, Steven Newbury wrote: > I have a mostly* headless server containing a Radeon discrete GPU. It > occured to me that having a GiB or two of high speed memory sitting > unused is pretty wasteful. Not an original thought; indeed there's a > Gentoo wiki which describes how to map the memory as a mtd device: > > http://www.gentoo-wiki.info/TIP_Use_memory_on_video_card_as_swap > > There's also a driver (cudaram) written by Piotr Jaroszyński which > allocates memory through CUDA and maps it to a block device: > > http://blog.piotrj.org/2011/03/cudaram-block-device-exposing-nvidia.htm > l > > The Gentoo method is extremely limited, and of course isn't exactly > safe, although using pci-stub would probably help. There's no > arbitration between the DRM driver and the mtd phram driver. Most of > all, it can only expose the memory mapped into the PCI aperture, likely > somewhat less than the full VMEM. > > The second method obviously requires the proprietary NVidia driver and > an NVidia gfx card, but it's good proof of concept at utilising a > driver managed buffer object as a block device. > > I'm looking into creating a volatile block device driver which > allocates VMEM from DRM, what if any is the right API to use? The DRM > userspace API is well documented, but I'd like to make this a kernel > module, so using a userspace API would seem inappropriate at best. > Ideally I'd like to create something like the bcache flash_vol_create > interface except expose a /sys/fs/drm-block/card0/vol_create (where > drm-block is a placeholder for the proposed driver name) for each DRM > device in the system, and likewise persistence through udev. > > So is there an API I can (mis-)use for this? Any suggestions? > > * There's a screen attached for maintainence, but it usually turned > off. > > > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/dri-devel [-- Attachment #1.2: Type: text/html, Size: 3487 bytes --] [-- Attachment #2: Type: text/plain, Size: 159 bytes --] _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: block device backed by DRM buffer object 2015-09-21 13:05 ` Christian König @ 2015-09-22 20:44 ` Steven Newbury 2015-09-23 20:37 ` drm_device from another device driver? (was: Re: block device backed by DRM buffer object) Steven Newbury 0 siblings, 1 reply; 7+ messages in thread From: Steven Newbury @ 2015-09-22 20:44 UTC (permalink / raw) To: Christian König, dri-devel [-- Attachment #1.1: Type: text/plain, Size: 3980 bytes --] On Mon, 2015-09-21 at 15:05 +0200, Christian König wrote: > In general a rather interesting idea and actually shouldn't be to > hard to implement. > > The crux is that allocating memory is device driver dependent, so > there isn't a general purpose API for doing so. > > Additional to that the CPU invisible VRAM is only accessible by the > GPU so you at least need to be able to submit DMA commands to copy > the data back and forth. > > For an experiment I suggest you do this with Radeon first and if that > works generalize from there. > Thanks for the feedback. I've been studying the code, and it looks like everything I need for Radeon is in radeon_test.c. As a proof of concept I'm going to initially hack it into an mtd device driver with static allocation through module params. This is pushing my knowledge quite a bit. Just to make sure I'm on the right track, this is what I'm attempting: - Grab the pointer to class drm (how? I really need some help here) - Walk the enumerated drm devices - For the specified drm card if it's radeon: - use radeon_bo_create() to create a bo in RADEON_GEM_DOMAIN_VRAM of specified size - create a bo in gtt for DMA to/from GPU memory For reads/writes DMA from/to card VRAM bo and copy from/to gtt bo and userspace I guess it would be easier to do this as an add-on to the radeon drm driver since it wouldn't require walking the drm devices to get the radeon_device pointer but it makes more sense to me to just add a reference count to the drm device as an external module. > Regards, > Christian. > > On 21.09.2015 13:33, Steven Newbury wrote: > > I have a mostly* headless server containing a Radeon discrete GPU. > > It > > occured to me that having a GiB or two of high speed memory sitting > > unused is pretty wasteful. Not an original thought; indeed there's > > a > > Gentoo wiki which describes how to map the memory as a mtd device: > > > > http://www.gentoo-wiki.info/TIP_Use_memory_on_video_card_as_swap > > > > There's also a driver (cudaram) written by Piotr Jaroszyński which > > allocates memory through CUDA and maps it to a block device: > > > > http://blog.piotrj.org/2011/03/cudaram-block-device-exposing-nvidia > > .htm > > l > > > > The Gentoo method is extremely limited, and of course isn't exactly > > safe, although using pci-stub would probably help. There's no > > arbitration between the DRM driver and the mtd phram driver. Most > > of > > all, it can only expose the memory mapped into the PCI aperture, > > likely > > somewhat less than the full VMEM. > > > > The second method obviously requires the proprietary NVidia driver > > and > > an NVidia gfx card, but it's good proof of concept at utilising a > > driver managed buffer object as a block device. > > > > I'm looking into creating a volatile block device driver which > > allocates VMEM from DRM, what if any is the right API to use? The > > DRM > > userspace API is well documented, but I'd like to make this a > > kernel > > module, so using a userspace API would seem inappropriate at best. > > Ideally I'd like to create something like the bcache > > flash_vol_create > > interface except expose a /sys/fs/drm-block/card0/vol_create (where > > drm-block is a placeholder for the proposed driver name) for each > > DRM > > device in the system, and likewise persistence through udev. > > > > So is there an API I can (mis-)use for this? Any suggestions? > > > > * There's a screen attached for maintainence, but it usually turned > > off. > > > > > > _______________________________________________ > > dri-devel mailing list > > dri-devel@lists.freedesktop.org > > http://lists.freedesktop.org/mailman/listinfo/dri-devel > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/dri-devel [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 159 bytes --] _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* drm_device from another device driver? (was: Re: block device backed by DRM buffer object) 2015-09-22 20:44 ` Steven Newbury @ 2015-09-23 20:37 ` Steven Newbury 2015-09-23 21:41 ` Lukas Wunner 0 siblings, 1 reply; 7+ messages in thread From: Steven Newbury @ 2015-09-23 20:37 UTC (permalink / raw) To: steve; +Cc: dri-devel I've been reading up on the device model and studying kernel sources for the last couple of days, but I can't figure out how to get a pointer to the radeon_device struct for a specific card, or the parent drm_device from an external device driver. I imagine I somehow need to take a reference to the drm class kobject for the card in question, and in so doing presumably I should then be able to discover the pointer to device. Can someone point me to an example or provide a snippet to demonstrate? Alternatively, if I'm wandering down the wrong path, a sign to lead my way would be appreciated! _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: drm_device from another device driver? (was: Re: block device backed by DRM buffer object) 2015-09-23 20:37 ` drm_device from another device driver? (was: Re: block device backed by DRM buffer object) Steven Newbury @ 2015-09-23 21:41 ` Lukas Wunner 2015-09-23 22:52 ` Steven Newbury 0 siblings, 1 reply; 7+ messages in thread From: Lukas Wunner @ 2015-09-23 21:41 UTC (permalink / raw) To: Steven Newbury; +Cc: dri-devel Hi, On Wed, Sep 23, 2015 at 08:37:48PM +0000, Steven Newbury wrote: > I can't figure out how to get a pointer to the radeon_device struct > for a specific card, or the parent drm_device from an external device > driver. struct device -> struct drm_device: dev_get_drvdata() struct pci_dev -> struct drm_device: pci_get_drvdata() struct drm_device -> struct radeon_device: drm_device->dev_private > I imagine I somehow need to take a reference to the drm class kobject > for the card in question, and in so doing presumably I should then be > able to discover the pointer to device. It sounds like you want to discover the available radeon cards in the system? If so you could iterate over all pci devices and look for pci->vendor == PCI_VENDOR_ID_ATI || pci->vendor == PCI_VENDOR_ID_AMD, then get to the radeon_device as shown above. However as Christian König pointed out, memory allocation is driver dependent. For an initial proof of concept it may be simplest to hack the radeon driver. Then you'll get an idea which parts are generic and which are driver specific and you can move the generic stuff to a central broker. Rather than discovering the VRAM it probably makes more sense to have drm devices register a set of callbacks with the central broker (e.g. return amount of currently free VRAM, allocate VRAM for use as block device, deallocate VRAM, read vector of blocks, write vector of blocks). The broker could then be controlled from user space via sysfs or ioctls or whatever. Best regards, Lukas _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: drm_device from another device driver? (was: Re: block device backed by DRM buffer object) 2015-09-23 21:41 ` Lukas Wunner @ 2015-09-23 22:52 ` Steven Newbury 2015-09-24 9:41 ` drm_device from another device driver? Christian König 0 siblings, 1 reply; 7+ messages in thread From: Steven Newbury @ 2015-09-23 22:52 UTC (permalink / raw) To: Lukas Wunner; +Cc: dri-devel [-- Attachment #1.1: Type: text/plain, Size: 2672 bytes --] On Wed, 2015-09-23 at 23:41 +0200, Lukas Wunner wrote: > Hi, > > On Wed, Sep 23, 2015 at 08:37:48PM +0000, Steven Newbury wrote: > > I can't figure out how to get a pointer to the radeon_device struct > > for a specific card, or the parent drm_device from an external > > device > > driver. > > struct device -> struct drm_device: dev_get_drvdata() > struct pci_dev -> struct drm_device: pci_get_drvdata() > struct drm_device -> struct radeon_device: drm_device->dev_private > Thanks, that's useful. > > > I imagine I somehow need to take a reference to the drm class > > kobject > > for the card in question, and in so doing presumably I should then > > be > > able to discover the pointer to device. > > It sounds like you want to discover the available radeon cards in the > system? If so you could iterate over all pci devices and look for > pci->vendor == PCI_VENDOR_ID_ATI || pci->vendor == PCI_VENDOR_ID_AMD, > then get to the radeon_device as shown above. > Yes, my plan was to eventually discover all the drm devices on the system and make available a sysfs entry to allocate a volume from VRAM for each capable card selecting an appropriate backend for driver. I was initially just going to implement it for radoen using a static allocation from module params. > However as Christian König pointed out, memory allocation is driver > dependent. For an initial proof of concept it may be simplest to hack > the radeon driver. Then you'll get an idea which parts are generic > and > which are driver specific and you can move the generic stuff to a > central broker. > Hacking the radeon driver was very much something I'd considered; I'd only decided not to because I was thinking too far ahead really. I need to keep it simple, as you say, and only add complexity once I have something working, and hopefully able to demonstrate its utility. > Rather than discovering the VRAM it probably makes more sense to have > drm devices register a set of callbacks with the central broker > (e.g. return amount of currently free VRAM, allocate VRAM for use as > block device, deallocate VRAM, read vector of blocks, write vector > of blocks). The broker could then be controlled from user space via > sysfs or ioctls or whatever. For now I think I'll just take the approach bcache does with "thinly provisioned volumes", and have an entry in the radeon sysfs: /sys/class/drm/card0/device/vram_volume_create which will attempt to allocate a given sized volume and register a vrambd[0]. (or some better name?) Unregistering/deallocation can be triggered from /sys/block/vramd[0]/vrambd/unregister [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 159 bytes --] _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: drm_device from another device driver? 2015-09-23 22:52 ` Steven Newbury @ 2015-09-24 9:41 ` Christian König 0 siblings, 0 replies; 7+ messages in thread From: Christian König @ 2015-09-24 9:41 UTC (permalink / raw) To: Steven Newbury, Lukas Wunner; +Cc: dri-devel [-- Attachment #1.1: Type: text/plain, Size: 3475 bytes --] On 24.09.2015 00:52, Steven Newbury wrote: > On Wed, 2015-09-23 at 23:41 +0200, Lukas Wunner wrote: >> Hi, >> >> On Wed, Sep 23, 2015 at 08:37:48PM +0000, Steven Newbury wrote: >>> I can't figure out how to get a pointer to the radeon_device struct >>> for a specific card, or the parent drm_device from an external >>> device >>> driver. >> struct device -> struct drm_device: dev_get_drvdata() >> struct pci_dev -> struct drm_device: pci_get_drvdata() >> struct drm_device -> struct radeon_device: drm_device->dev_private >> > Thanks, that's useful. > >>> I imagine I somehow need to take a reference to the drm class >>> kobject >>> for the card in question, and in so doing presumably I should then >>> be >>> able to discover the pointer to device. >> It sounds like you want to discover the available radeon cards in the >> system? If so you could iterate over all pci devices and look for >> pci->vendor == PCI_VENDOR_ID_ATI || pci->vendor == PCI_VENDOR_ID_AMD, >> then get to the radeon_device as shown above. >> > Yes, my plan was to eventually discover all the drm devices on the > system and make available a sysfs entry to allocate a volume from VRAM > for each capable card selecting an appropriate backend for driver. I > was initially just going to implement it for radoen using a static > allocation from module params. > >> However as Christian König pointed out, memory allocation is driver >> dependent. For an initial proof of concept it may be simplest to hack >> the radeon driver. Then you'll get an idea which parts are generic >> and >> which are driver specific and you can move the generic stuff to a >> central broker. >> > Hacking the radeon driver was very much something I'd considered; I'd > only decided not to because I was thinking too far ahead really. I > need to keep it simple, as you say, and only add complexity once I have > something working, and hopefully able to demonstrate its utility. > >> Rather than discovering the VRAM it probably makes more sense to have >> drm devices register a set of callbacks with the central broker >> (e.g. return amount of currently free VRAM, allocate VRAM for use as >> block device, deallocate VRAM, read vector of blocks, write vector >> of blocks). The broker could then be controlled from user space via >> sysfs or ioctls or whatever. > For now I think I'll just take the approach bcache does with "thinly > provisioned volumes", and have an entry in the radeon sysfs: > > /sys/class/drm/card0/device/vram_volume_create > > which will attempt to allocate a given sized volume and register a > vrambd[0]. (or some better name?) Unregistering/deallocation can be > triggered from /sys/block/vramd[0]/vrambd/unregister Yeah, that approach sounds reasonable to me I would just go into a different direction with the sysfs interface. It doesn't make much sense to have more than one volume for each card (doesn't it?). So I would rather say instead of having a vram_volume_create sysfs you should rather go with something like vram_volume_size. E.g. setting the size makes the volume available, setting it to zero tries to free it again. If anybody is using it while you try to change the size return -EBUSY. Regards, Christian. > > > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/dri-devel [-- Attachment #1.2: Type: text/html, Size: 4870 bytes --] [-- Attachment #2: Type: text/plain, Size: 159 bytes --] _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-09-24 9:41 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-09-21 11:33 block device backed by DRM buffer object Steven Newbury 2015-09-21 13:05 ` Christian König 2015-09-22 20:44 ` Steven Newbury 2015-09-23 20:37 ` drm_device from another device driver? (was: Re: block device backed by DRM buffer object) Steven Newbury 2015-09-23 21:41 ` Lukas Wunner 2015-09-23 22:52 ` Steven Newbury 2015-09-24 9:41 ` drm_device from another device driver? Christian König
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.