Amdgpu module is references even after unbinding the vtcon

Amdgpu module is references even after unbinding the vtcon

[Christian König]

Hi Thomas,

we ran into a problem with the general fbcon/fbdev implementation and 
though that you might have some idea.

What happens is the following:
1. We load amdgpu and get our normal fbcon.
2. fbcon allocates a dump BO as backing store for the console.
3. GDM/X/Applications start, new framebuffers are created BOs imported, 
exported etc...
4. Somehow X or GDM iterated over all the framebuffer objects the 
kernels knows about and export them as DMA-buf.
5. Application/X/GDM are stopped, handles closed, framebuffers released 
etc...
6. We unbind vtcon.

At this point the amdgpu module usually has a reference count of 0 and 
can be unloaded, but since GDM/X/Whoever iterated over all the known 
framebuffers and exported them as DMA-buf (for whatever reason idk) we 
now still have an exported DMA-buf and with it a reference to the module.

Any idea how we could prevent that?

Thanks,
Christian.

Re: Amdgpu module is references even after unbinding the vtcon

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 1812 bytes --]

Hi

Am 24.01.23 um 15:12 schrieb Christian König:
> Hi Thomas,
> 
> we ran into a problem with the general fbcon/fbdev implementation and 
> though that you might have some idea.
> 
> What happens is the following:
> 1. We load amdgpu and get our normal fbcon.
> 2. fbcon allocates a dump BO as backing store for the console.
> 3. GDM/X/Applications start, new framebuffers are created BOs imported, 
> exported etc...
> 4. Somehow X or GDM iterated over all the framebuffer objects the 
> kernels knows about and export them as DMA-buf.
> 5. Application/X/GDM are stopped, handles closed, framebuffers released 
> etc...
> 6. We unbind vtcon.
> 
> At this point the amdgpu module usually has a reference count of 0 and 
> can be unloaded, but since GDM/X/Whoever iterated over all the known 
> framebuffers and exported them as DMA-buf (for whatever reason idk) we 
> now still have an exported DMA-buf and with it a reference to the module.
> 
> Any idea how we could prevent that?

No real clue, sorry.

But does it change if you use a shadow buffer on top of the amdgpu BO? 
Set prefer_shadow_fbdev = 1. [1]   I once tried to run generic fbdev 
without prefer_shadow_fbdev and it never worked. I suspected that some 
reference counting got wrong, but could never pin it down.  Maybe your 
issue is similar.

That said, generic fbdev is not so super-optimal for TTM-based drivers. 
I'm working on improving that, but it's not there yet.

Best regards
Thomas


[1] 
https://elixir.bootlin.com/linux/v6.0/source/include/drm/drm_mode_config.h#L890

> 
> Thanks,
> Christian.

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Amdgpu module is references even after unbinding the vtcon

[Alex Deucher]

On Tue, Jan 24, 2023 at 11:15 AM Thomas Zimmermann <tzimmermann@suse.de> wrote:
>
> Hi
>
> Am 24.01.23 um 15:12 schrieb Christian König:
> > Hi Thomas,
> >
> > we ran into a problem with the general fbcon/fbdev implementation and
> > though that you might have some idea.
> >
> > What happens is the following:
> > 1. We load amdgpu and get our normal fbcon.
> > 2. fbcon allocates a dump BO as backing store for the console.
> > 3. GDM/X/Applications start, new framebuffers are created BOs imported,
> > exported etc...
> > 4. Somehow X or GDM iterated over all the framebuffer objects the
> > kernels knows about and export them as DMA-buf.
> > 5. Application/X/GDM are stopped, handles closed, framebuffers released
> > etc...
> > 6. We unbind vtcon.
> >
> > At this point the amdgpu module usually has a reference count of 0 and
> > can be unloaded, but since GDM/X/Whoever iterated over all the known
> > framebuffers and exported them as DMA-buf (for whatever reason idk) we
> > now still have an exported DMA-buf and with it a reference to the module.
> >
> > Any idea how we could prevent that?
>
> No real clue, sorry.
>
> But does it change if you use a shadow buffer on top of the amdgpu BO?
> Set prefer_shadow_fbdev = 1. [1]   I once tried to run generic fbdev
> without prefer_shadow_fbdev and it never worked. I suspected that some
> reference counting got wrong, but could never pin it down.  Maybe your
> issue is similar.

Is that equivalent to setting mode_config.prefer_shadow = 1?  If so,
we already do that.

Alex

>
> That said, generic fbdev is not so super-optimal for TTM-based drivers.
> I'm working on improving that, but it's not there yet.
>
> Best regards
> Thomas
>
>
> [1]
> https://elixir.bootlin.com/linux/v6.0/source/include/drm/drm_mode_config.h#L890
>
> >
> > Thanks,
> > Christian.
>
> --
> Thomas Zimmermann
> Graphics Driver Developer
> SUSE Software Solutions Germany GmbH
> Maxfeldstr. 5, 90409 Nürnberg, Germany
> (HRB 36809, AG Nürnberg)
> Geschäftsführer: Ivo Totev

Re: Amdgpu module is references even after unbinding the vtcon

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 2420 bytes --]

Hi

Am 24.01.23 um 17:36 schrieb Alex Deucher:
> On Tue, Jan 24, 2023 at 11:15 AM Thomas Zimmermann <tzimmermann@suse.de> wrote:
>>
>> Hi
>>
>> Am 24.01.23 um 15:12 schrieb Christian König:
>>> Hi Thomas,
>>>
>>> we ran into a problem with the general fbcon/fbdev implementation and
>>> though that you might have some idea.
>>>
>>> What happens is the following:
>>> 1. We load amdgpu and get our normal fbcon.
>>> 2. fbcon allocates a dump BO as backing store for the console.
>>> 3. GDM/X/Applications start, new framebuffers are created BOs imported,
>>> exported etc...
>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>> kernels knows about and export them as DMA-buf.
>>> 5. Application/X/GDM are stopped, handles closed, framebuffers released
>>> etc...
>>> 6. We unbind vtcon.
>>>
>>> At this point the amdgpu module usually has a reference count of 0 and
>>> can be unloaded, but since GDM/X/Whoever iterated over all the known
>>> framebuffers and exported them as DMA-buf (for whatever reason idk) we
>>> now still have an exported DMA-buf and with it a reference to the module.
>>>
>>> Any idea how we could prevent that?
>>
>> No real clue, sorry.
>>
>> But does it change if you use a shadow buffer on top of the amdgpu BO?
>> Set prefer_shadow_fbdev = 1. [1]   I once tried to run generic fbdev
>> without prefer_shadow_fbdev and it never worked. I suspected that some
>> reference counting got wrong, but could never pin it down.  Maybe your
>> issue is similar.
> 
> Is that equivalent to setting mode_config.prefer_shadow = 1?  If so,
> we already do that.

Yes it is, unfortunately.

Best regards
Thomas

> 
> Alex
> 
>>
>> That said, generic fbdev is not so super-optimal for TTM-based drivers.
>> I'm working on improving that, but it's not there yet.
>>
>> Best regards
>> Thomas
>>
>>
>> [1]
>> https://elixir.bootlin.com/linux/v6.0/source/include/drm/drm_mode_config.h#L890
>>
>>>
>>> Thanks,
>>> Christian.
>>
>> --
>> Thomas Zimmermann
>> Graphics Driver Developer
>> SUSE Software Solutions Germany GmbH
>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>> (HRB 36809, AG Nürnberg)
>> Geschäftsführer: Ivo Totev

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Amdgpu module is references even after unbinding the vtcon

[Christian König]

Am 24.01.23 um 17:15 schrieb Thomas Zimmermann:
> Hi
>
> Am 24.01.23 um 15:12 schrieb Christian König:
>> Hi Thomas,
>>
>> we ran into a problem with the general fbcon/fbdev implementation and 
>> though that you might have some idea.
>>
>> What happens is the following:
>> 1. We load amdgpu and get our normal fbcon.
>> 2. fbcon allocates a dump BO as backing store for the console.
>> 3. GDM/X/Applications start, new framebuffers are created BOs 
>> imported, exported etc...
>> 4. Somehow X or GDM iterated over all the framebuffer objects the 
>> kernels knows about and export them as DMA-buf.
>> 5. Application/X/GDM are stopped, handles closed, framebuffers 
>> released etc...
>> 6. We unbind vtcon.
>>
>> At this point the amdgpu module usually has a reference count of 0 
>> and can be unloaded, but since GDM/X/Whoever iterated over all the 
>> known framebuffers and exported them as DMA-buf (for whatever reason 
>> idk) we now still have an exported DMA-buf and with it a reference to 
>> the module.
>>
>> Any idea how we could prevent that?
>
> No real clue, sorry.
>
> But does it change if you use a shadow buffer on top of the amdgpu BO? 
> Set prefer_shadow_fbdev = 1. [1]   I once tried to run generic fbdev 
> without prefer_shadow_fbdev and it never worked. I suspected that some 
> reference counting got wrong, but could never pin it down.  Maybe your 
> issue is similar.
>
> That said, generic fbdev is not so super-optimal for TTM-based 
> drivers. I'm working on improving that, but it's not there yet.

If I'm not completely mistaken this problem is unrelated to TTM and 
could happen with any generic fbdev based driver which supports DMA-buf.

Does VKMS could be used with generic fbdev as well? If yes I think I can 
stitch together a test case demonstrating the issue.

Regards,
Christian.

>
> Best regards
> Thomas
>
>
> [1] 
> https://elixir.bootlin.com/linux/v6.0/source/include/drm/drm_mode_config.h#L890
>
>>
>> Thanks,
>> Christian.
>

Re: Amdgpu module is references even after unbinding the vtcon

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 1966 bytes --]

Hi Christian

Am 24.01.23 um 15:12 schrieb Christian König:
> Hi Thomas,
> 
> we ran into a problem with the general fbcon/fbdev implementation and 
> though that you might have some idea.
> 
> What happens is the following:
> 1. We load amdgpu and get our normal fbcon.
> 2. fbcon allocates a dump BO as backing store for the console.
> 3. GDM/X/Applications start, new framebuffers are created BOs imported, 
> exported etc...
> 4. Somehow X or GDM iterated over all the framebuffer objects the 
> kernels knows about and export them as DMA-buf.
> 5. Application/X/GDM are stopped, handles closed, framebuffers released 
> etc...
> 6. We unbind vtcon.
> 
> At this point the amdgpu module usually has a reference count of 0 and 
> can be unloaded, but since GDM/X/Whoever iterated over all the known 
> framebuffers and exported them as DMA-buf (for whatever reason idk) we 
> now still have an exported DMA-buf and with it a reference to the module.
> 
> Any idea how we could prevent that?

Here's another stab in the dark.

The big difference between old-style fbdev and the new one is that the 
old fbdev setup (e.g., radeon) allocates a GEM object and puts together 
the fbdev data structures from the BO in a fairly hackish way. The new 
style uses an in-kernel client with a file to allocate the BO via dumb 
buffers; and holds a reference to the DRM module.

Maybe the reference comes from the in-kernel DRM client itself. [1] 
Check if the client resources get released [2] when you unbind vtcon.

Best regards
Thomas

[1] 
https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L87
[2] 
https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L160

> 
> Thanks,
> Christian.

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

RE: Amdgpu module is references even after unbinding the vtcon

[Slivka, Danijel]

[AMD Official Use Only - General]

Hi Thomas,

I have checked what you mentioned.
When loading amdgpu we call  drm_client_init() during fbdev setup [1], the refcnt for drm_kms_helper increases from 3 -> 4.
When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the drm_client_release() [2] is not called.
The drm_client_release() is called only when unloading the amdgpu driver.

Is this expected?

There is a comment for drm_client_release with regards to fbdev :
* This function should only be called from the unregister callback. An exception
 * is fbdev which cannot free the buffer if userspace has open file descriptors.

Could this be relevant for our use case, although as Application/X/GDM are stopped at that point and no fd should be open.

Thank you,
BR,
Danijel

>-----Original Message-----
>From: Thomas Zimmermann <tzimmermann@suse.de>
>Sent: Wednesday, January 25, 2023 8:48 PM
>To: Christian König <ckoenig.leichtzumerken@gmail.com>
>Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
><Danijel.Slivka@amd.com>; dri-devel <dri-devel@lists.freedesktop.org>; Sharma,
>Shashank <Shashank.Sharma@amd.com>
>Subject: Re: Amdgpu module is references even after unbinding the vtcon
>
>Hi Christian
>
>Am 24.01.23 um 15:12 schrieb Christian König:
>> Hi Thomas,
>>
>> we ran into a problem with the general fbcon/fbdev implementation and
>> though that you might have some idea.
>>
>> What happens is the following:
>> 1. We load amdgpu and get our normal fbcon.
>> 2. fbcon allocates a dump BO as backing store for the console.
>> 3. GDM/X/Applications start, new framebuffers are created BOs
>> imported, exported etc...
>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>> kernels knows about and export them as DMA-buf.
>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>> released etc...
>> 6. We unbind vtcon.
>>
>> At this point the amdgpu module usually has a reference count of 0 and
>> can be unloaded, but since GDM/X/Whoever iterated over all the known
>> framebuffers and exported them as DMA-buf (for whatever reason idk) we
>> now still have an exported DMA-buf and with it a reference to the module.
>>
>> Any idea how we could prevent that?
>
>Here's another stab in the dark.
>
>The big difference between old-style fbdev and the new one is that the old fbdev
>setup (e.g., radeon) allocates a GEM object and puts together the fbdev data
>structures from the BO in a fairly hackish way. The new style uses an in-kernel
>client with a file to allocate the BO via dumb buffers; and holds a reference to the
>DRM module.
>
>Maybe the reference comes from the in-kernel DRM client itself. [1] Check if the
>client resources get released [2] when you unbind vtcon.
>
>Best regards
>Thomas
>
>[1]
>https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L87
>[2]
>https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L16
>0
>
>>
>> Thanks,
>> Christian.
>
>--
>Thomas Zimmermann
>Graphics Driver Developer
>SUSE Software Solutions Germany GmbH
>Maxfeldstr. 5, 90409 Nürnberg, Germany
>(HRB 36809, AG Nürnberg)
>Geschäftsführer: Ivo Totev

Re: Amdgpu module is references even after unbinding the vtcon

[Christian König]

Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
> [AMD Official Use Only - General]
>
> Hi Thomas,
>
> I have checked what you mentioned.
> When loading amdgpu we call  drm_client_init() during fbdev setup [1], the refcnt for drm_kms_helper increases from 3 -> 4.
> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the drm_client_release() [2] is not called.
> The drm_client_release() is called only when unloading the amdgpu driver.
>
> Is this expected?

Yes, the client can't be released because it is possible that the vtcon 
is bound to this fbdev again.

Please test the handle work around I've send around internally. At least 
for me that approach seems to work.

Regards,
Christian.

>
> There is a comment for drm_client_release with regards to fbdev :
> * This function should only be called from the unregister callback. An exception
>   * is fbdev which cannot free the buffer if userspace has open file descriptors.
>
> Could this be relevant for our use case, although as Application/X/GDM are stopped at that point and no fd should be open.
>
> Thank you,
> BR,
> Danijel
>
>> -----Original Message-----
>> From: Thomas Zimmermann <tzimmermann@suse.de>
>> Sent: Wednesday, January 25, 2023 8:48 PM
>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>> <Danijel.Slivka@amd.com>; dri-devel <dri-devel@lists.freedesktop.org>; Sharma,
>> Shashank <Shashank.Sharma@amd.com>
>> Subject: Re: Amdgpu module is references even after unbinding the vtcon
>>
>> Hi Christian
>>
>> Am 24.01.23 um 15:12 schrieb Christian König:
>>> Hi Thomas,
>>>
>>> we ran into a problem with the general fbcon/fbdev implementation and
>>> though that you might have some idea.
>>>
>>> What happens is the following:
>>> 1. We load amdgpu and get our normal fbcon.
>>> 2. fbcon allocates a dump BO as backing store for the console.
>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>> imported, exported etc...
>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>> kernels knows about and export them as DMA-buf.
>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>> released etc...
>>> 6. We unbind vtcon.
>>>
>>> At this point the amdgpu module usually has a reference count of 0 and
>>> can be unloaded, but since GDM/X/Whoever iterated over all the known
>>> framebuffers and exported them as DMA-buf (for whatever reason idk) we
>>> now still have an exported DMA-buf and with it a reference to the module.
>>>
>>> Any idea how we could prevent that?
>> Here's another stab in the dark.
>>
>> The big difference between old-style fbdev and the new one is that the old fbdev
>> setup (e.g., radeon) allocates a GEM object and puts together the fbdev data
>> structures from the BO in a fairly hackish way. The new style uses an in-kernel
>> client with a file to allocate the BO via dumb buffers; and holds a reference to the
>> DRM module.
>>
>> Maybe the reference comes from the in-kernel DRM client itself. [1] Check if the
>> client resources get released [2] when you unbind vtcon.
>>
>> Best regards
>> Thomas
>>
>> [1]
>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L87
>> [2]
>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L16
>> 0
>>
>>> Thanks,
>>> Christian.
>> --
>> Thomas Zimmermann
>> Graphics Driver Developer
>> SUSE Software Solutions Germany GmbH
>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>> (HRB 36809, AG Nürnberg)
>> Geschäftsführer: Ivo Totev

RE: Amdgpu module is references even after unbinding the vtcon

[Slivka, Danijel]

[AMD Official Use Only - General]

Hi Christian,

I have tested the proposed patch, the issue is not reproducible and everything else seems to work fine.

BR,
Danijel

>-----Original Message-----
>From: Christian König <ckoenig.leichtzumerken@gmail.com>
>Sent: Thursday, January 26, 2023 1:20 PM
>To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann
><tzimmermann@suse.de>
>Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-
>devel@lists.freedesktop.org>; Sharma, Shashank <Shashank.Sharma@amd.com>
>Subject: Re: Amdgpu module is references even after unbinding the vtcon
>
>Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>> [AMD Official Use Only - General]
>>
>> Hi Thomas,
>>
>> I have checked what you mentioned.
>> When loading amdgpu we call  drm_client_init() during fbdev setup [1], the
>refcnt for drm_kms_helper increases from 3 -> 4.
>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the
>drm_client_release() [2] is not called.
>> The drm_client_release() is called only when unloading the amdgpu driver.
>>
>> Is this expected?
>
>Yes, the client can't be released because it is possible that the vtcon is bound to
>this fbdev again.
>
>Please test the handle work around I've send around internally. At least for me
>that approach seems to work.
>
>Regards,
>Christian.
>
>>
>> There is a comment for drm_client_release with regards to fbdev :
>> * This function should only be called from the unregister callback. An exception
>>   * is fbdev which cannot free the buffer if userspace has open file descriptors.
>>
>> Could this be relevant for our use case, although as Application/X/GDM are
>stopped at that point and no fd should be open.
>>
>> Thank you,
>> BR,
>> Danijel
>>
>>> -----Original Message-----
>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>>> <Danijel.Slivka@amd.com>; dri-devel
>>> <dri-devel@lists.freedesktop.org>; Sharma, Shashank
>>> <Shashank.Sharma@amd.com>
>>> Subject: Re: Amdgpu module is references even after unbinding the
>>> vtcon
>>>
>>> Hi Christian
>>>
>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>> Hi Thomas,
>>>>
>>>> we ran into a problem with the general fbcon/fbdev implementation
>>>> and though that you might have some idea.
>>>>
>>>> What happens is the following:
>>>> 1. We load amdgpu and get our normal fbcon.
>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>>> imported, exported etc...
>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>>> kernels knows about and export them as DMA-buf.
>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>>> released etc...
>>>> 6. We unbind vtcon.
>>>>
>>>> At this point the amdgpu module usually has a reference count of 0
>>>> and can be unloaded, but since GDM/X/Whoever iterated over all the
>>>> known framebuffers and exported them as DMA-buf (for whatever reason
>>>> idk) we now still have an exported DMA-buf and with it a reference to the
>module.
>>>>
>>>> Any idea how we could prevent that?
>>> Here's another stab in the dark.
>>>
>>> The big difference between old-style fbdev and the new one is that
>>> the old fbdev setup (e.g., radeon) allocates a GEM object and puts
>>> together the fbdev data structures from the BO in a fairly hackish
>>> way. The new style uses an in-kernel client with a file to allocate
>>> the BO via dumb buffers; and holds a reference to the DRM module.
>>>
>>> Maybe the reference comes from the in-kernel DRM client itself. [1]
>>> Check if the client resources get released [2] when you unbind vtcon.
>>>
>>> Best regards
>>> Thomas
>>>
>>> [1]
>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_cl
>>> ient.c#L87
>>> [2]
>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_cl
>>> ient.c#L16
>>> 0
>>>
>>>> Thanks,
>>>> Christian.
>>> --
>>> Thomas Zimmermann
>>> Graphics Driver Developer
>>> SUSE Software Solutions Germany GmbH
>>> Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg)
>>> Geschäftsführer: Ivo Totev

Re: Amdgpu module is references even after unbinding the vtcon

[Christian König]

Hi Danijel,

can you also double check that GDM/X is still capable of acquiring a 
DMA-buf file descriptor for the buffer (e.g. that we have a DMA-buf 
handle for it while they are started).

And that handover from fbdev to GDM/X is flicker free?

Thanks,
Christian.

Am 26.01.23 um 14:44 schrieb Slivka, Danijel:
> [AMD Official Use Only - General]
>
> Hi Christian,
>
> I have tested the proposed patch, the issue is not reproducible and everything else seems to work fine.
>
> BR,
> Danijel
>
>> -----Original Message-----
>> From: Christian König <ckoenig.leichtzumerken@gmail.com>
>> Sent: Thursday, January 26, 2023 1:20 PM
>> To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann
>> <tzimmermann@suse.de>
>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-
>> devel@lists.freedesktop.org>; Sharma, Shashank <Shashank.Sharma@amd.com>
>> Subject: Re: Amdgpu module is references even after unbinding the vtcon
>>
>> Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>>> [AMD Official Use Only - General]
>>>
>>> Hi Thomas,
>>>
>>> I have checked what you mentioned.
>>> When loading amdgpu we call  drm_client_init() during fbdev setup [1], the
>> refcnt for drm_kms_helper increases from 3 -> 4.
>>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the
>> drm_client_release() [2] is not called.
>>> The drm_client_release() is called only when unloading the amdgpu driver.
>>>
>>> Is this expected?
>> Yes, the client can't be released because it is possible that the vtcon is bound to
>> this fbdev again.
>>
>> Please test the handle work around I've send around internally. At least for me
>> that approach seems to work.
>>
>> Regards,
>> Christian.
>>
>>> There is a comment for drm_client_release with regards to fbdev :
>>> * This function should only be called from the unregister callback. An exception
>>>    * is fbdev which cannot free the buffer if userspace has open file descriptors.
>>>
>>> Could this be relevant for our use case, although as Application/X/GDM are
>> stopped at that point and no fd should be open.
>>> Thank you,
>>> BR,
>>> Danijel
>>>
>>>> -----Original Message-----
>>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>>>> <Danijel.Slivka@amd.com>; dri-devel
>>>> <dri-devel@lists.freedesktop.org>; Sharma, Shashank
>>>> <Shashank.Sharma@amd.com>
>>>> Subject: Re: Amdgpu module is references even after unbinding the
>>>> vtcon
>>>>
>>>> Hi Christian
>>>>
>>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>>> Hi Thomas,
>>>>>
>>>>> we ran into a problem with the general fbcon/fbdev implementation
>>>>> and though that you might have some idea.
>>>>>
>>>>> What happens is the following:
>>>>> 1. We load amdgpu and get our normal fbcon.
>>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>>>> imported, exported etc...
>>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>>>> kernels knows about and export them as DMA-buf.
>>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>>>> released etc...
>>>>> 6. We unbind vtcon.
>>>>>
>>>>> At this point the amdgpu module usually has a reference count of 0
>>>>> and can be unloaded, but since GDM/X/Whoever iterated over all the
>>>>> known framebuffers and exported them as DMA-buf (for whatever reason
>>>>> idk) we now still have an exported DMA-buf and with it a reference to the
>> module.
>>>>> Any idea how we could prevent that?
>>>> Here's another stab in the dark.
>>>>
>>>> The big difference between old-style fbdev and the new one is that
>>>> the old fbdev setup (e.g., radeon) allocates a GEM object and puts
>>>> together the fbdev data structures from the BO in a fairly hackish
>>>> way. The new style uses an in-kernel client with a file to allocate
>>>> the BO via dumb buffers; and holds a reference to the DRM module.
>>>>
>>>> Maybe the reference comes from the in-kernel DRM client itself. [1]
>>>> Check if the client resources get released [2] when you unbind vtcon.
>>>>
>>>> Best regards
>>>> Thomas
>>>>
>>>> [1]
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_cl
>>>> ient.c#L87
>>>> [2]
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_cl
>>>> ient.c#L16
>>>> 0
>>>>
>>>>> Thanks,
>>>>> Christian.
>>>> --
>>>> Thomas Zimmermann
>>>> Graphics Driver Developer
>>>> SUSE Software Solutions Germany GmbH
>>>> Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg)
>>>> Geschäftsführer: Ivo Totev

RE: Amdgpu module is references even after unbinding the vtcon

[Sharma, Shashank]

[AMD Official Use Only - General]

I would also highly recommend this to be tested with another compositor (Like Weston/Sway etc)

Regards
Shashank
-----Original Message-----
From: Christian König <ckoenig.leichtzumerken@gmail.com> 
Sent: 26 January 2023 15:12
To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann <tzimmermann@suse.de>
Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-devel@lists.freedesktop.org>; Sharma, Shashank <Shashank.Sharma@amd.com>
Subject: Re: Amdgpu module is references even after unbinding the vtcon

Hi Danijel,

can you also double check that GDM/X is still capable of acquiring a DMA-buf file descriptor for the buffer (e.g. that we have a DMA-buf handle for it while they are started).

And that handover from fbdev to GDM/X is flicker free?

Thanks,
Christian.

Am 26.01.23 um 14:44 schrieb Slivka, Danijel:
> [AMD Official Use Only - General]
>
> Hi Christian,
>
> I have tested the proposed patch, the issue is not reproducible and everything else seems to work fine.
>
> BR,
> Danijel
>
>> -----Original Message-----
>> From: Christian König <ckoenig.leichtzumerken@gmail.com>
>> Sent: Thursday, January 26, 2023 1:20 PM
>> To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann 
>> <tzimmermann@suse.de>
>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri- 
>> devel@lists.freedesktop.org>; Sharma, Shashank 
>> <Shashank.Sharma@amd.com>
>> Subject: Re: Amdgpu module is references even after unbinding the 
>> vtcon
>>
>> Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>>> [AMD Official Use Only - General]
>>>
>>> Hi Thomas,
>>>
>>> I have checked what you mentioned.
>>> When loading amdgpu we call  drm_client_init() during fbdev setup 
>>> [1], the
>> refcnt for drm_kms_helper increases from 3 -> 4.
>>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but 
>>> the
>> drm_client_release() [2] is not called.
>>> The drm_client_release() is called only when unloading the amdgpu driver.
>>>
>>> Is this expected?
>> Yes, the client can't be released because it is possible that the 
>> vtcon is bound to this fbdev again.
>>
>> Please test the handle work around I've send around internally. At 
>> least for me that approach seems to work.
>>
>> Regards,
>> Christian.
>>
>>> There is a comment for drm_client_release with regards to fbdev :
>>> * This function should only be called from the unregister callback. An exception
>>>    * is fbdev which cannot free the buffer if userspace has open file descriptors.
>>>
>>> Could this be relevant for our use case, although as 
>>> Application/X/GDM are
>> stopped at that point and no fd should be open.
>>> Thank you,
>>> BR,
>>> Danijel
>>>
>>>> -----Original Message-----
>>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel 
>>>> <Danijel.Slivka@amd.com>; dri-devel 
>>>> <dri-devel@lists.freedesktop.org>; Sharma, Shashank 
>>>> <Shashank.Sharma@amd.com>
>>>> Subject: Re: Amdgpu module is references even after unbinding the 
>>>> vtcon
>>>>
>>>> Hi Christian
>>>>
>>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>>> Hi Thomas,
>>>>>
>>>>> we ran into a problem with the general fbcon/fbdev implementation 
>>>>> and though that you might have some idea.
>>>>>
>>>>> What happens is the following:
>>>>> 1. We load amdgpu and get our normal fbcon.
>>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>>> 3. GDM/X/Applications start, new framebuffers are created BOs 
>>>>> imported, exported etc...
>>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the 
>>>>> kernels knows about and export them as DMA-buf.
>>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers 
>>>>> released etc...
>>>>> 6. We unbind vtcon.
>>>>>
>>>>> At this point the amdgpu module usually has a reference count of 0 
>>>>> and can be unloaded, but since GDM/X/Whoever iterated over all the 
>>>>> known framebuffers and exported them as DMA-buf (for whatever 
>>>>> reason
>>>>> idk) we now still have an exported DMA-buf and with it a reference 
>>>>> to the
>> module.
>>>>> Any idea how we could prevent that?
>>>> Here's another stab in the dark.
>>>>
>>>> The big difference between old-style fbdev and the new one is that 
>>>> the old fbdev setup (e.g., radeon) allocates a GEM object and puts 
>>>> together the fbdev data structures from the BO in a fairly hackish 
>>>> way. The new style uses an in-kernel client with a file to allocate 
>>>> the BO via dumb buffers; and holds a reference to the DRM module.
>>>>
>>>> Maybe the reference comes from the in-kernel DRM client itself. [1] 
>>>> Check if the client resources get released [2] when you unbind vtcon.
>>>>
>>>> Best regards
>>>> Thomas
>>>>
>>>> [1]
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_
>>>> cl
>>>> ient.c#L87
>>>> [2]
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_
>>>> cl
>>>> ient.c#L16
>>>> 0
>>>>
>>>>> Thanks,
>>>>> Christian.
>>>> --
>>>> Thomas Zimmermann
>>>> Graphics Driver Developer
>>>> SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, 
>>>> Germany (HRB 36809, AG Nürnberg)
>>>> Geschäftsführer: Ivo Totev

Re: Amdgpu module is references even after unbinding the vtcon

[Christian König]

Oh, yeah. Very good point as well.

Christian.

Am 26.01.23 um 15:13 schrieb Sharma, Shashank:
> [AMD Official Use Only - General]
>
> I would also highly recommend this to be tested with another compositor (Like Weston/Sway etc)
>
> Regards
> Shashank
> -----Original Message-----
> From: Christian König <ckoenig.leichtzumerken@gmail.com>
> Sent: 26 January 2023 15:12
> To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann <tzimmermann@suse.de>
> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-devel@lists.freedesktop.org>; Sharma, Shashank <Shashank.Sharma@amd.com>
> Subject: Re: Amdgpu module is references even after unbinding the vtcon
>
> Hi Danijel,
>
> can you also double check that GDM/X is still capable of acquiring a DMA-buf file descriptor for the buffer (e.g. that we have a DMA-buf handle for it while they are started).
>
> And that handover from fbdev to GDM/X is flicker free?
>
> Thanks,
> Christian.
>
> Am 26.01.23 um 14:44 schrieb Slivka, Danijel:
>> [AMD Official Use Only - General]
>>
>> Hi Christian,
>>
>> I have tested the proposed patch, the issue is not reproducible and everything else seems to work fine.
>>
>> BR,
>> Danijel
>>
>>> -----Original Message-----
>>> From: Christian König <ckoenig.leichtzumerken@gmail.com>
>>> Sent: Thursday, January 26, 2023 1:20 PM
>>> To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann
>>> <tzimmermann@suse.de>
>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-
>>> devel@lists.freedesktop.org>; Sharma, Shashank
>>> <Shashank.Sharma@amd.com>
>>> Subject: Re: Amdgpu module is references even after unbinding the
>>> vtcon
>>>
>>> Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>>>> [AMD Official Use Only - General]
>>>>
>>>> Hi Thomas,
>>>>
>>>> I have checked what you mentioned.
>>>> When loading amdgpu we call  drm_client_init() during fbdev setup
>>>> [1], the
>>> refcnt for drm_kms_helper increases from 3 -> 4.
>>>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but
>>>> the
>>> drm_client_release() [2] is not called.
>>>> The drm_client_release() is called only when unloading the amdgpu driver.
>>>>
>>>> Is this expected?
>>> Yes, the client can't be released because it is possible that the
>>> vtcon is bound to this fbdev again.
>>>
>>> Please test the handle work around I've send around internally. At
>>> least for me that approach seems to work.
>>>
>>> Regards,
>>> Christian.
>>>
>>>> There is a comment for drm_client_release with regards to fbdev :
>>>> * This function should only be called from the unregister callback. An exception
>>>>     * is fbdev which cannot free the buffer if userspace has open file descriptors.
>>>>
>>>> Could this be relevant for our use case, although as
>>>> Application/X/GDM are
>>> stopped at that point and no fd should be open.
>>>> Thank you,
>>>> BR,
>>>> Danijel
>>>>
>>>>> -----Original Message-----
>>>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>>>>> <Danijel.Slivka@amd.com>; dri-devel
>>>>> <dri-devel@lists.freedesktop.org>; Sharma, Shashank
>>>>> <Shashank.Sharma@amd.com>
>>>>> Subject: Re: Amdgpu module is references even after unbinding the
>>>>> vtcon
>>>>>
>>>>> Hi Christian
>>>>>
>>>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>>>> Hi Thomas,
>>>>>>
>>>>>> we ran into a problem with the general fbcon/fbdev implementation
>>>>>> and though that you might have some idea.
>>>>>>
>>>>>> What happens is the following:
>>>>>> 1. We load amdgpu and get our normal fbcon.
>>>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>>>>> imported, exported etc...
>>>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>>>>> kernels knows about and export them as DMA-buf.
>>>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>>>>> released etc...
>>>>>> 6. We unbind vtcon.
>>>>>>
>>>>>> At this point the amdgpu module usually has a reference count of 0
>>>>>> and can be unloaded, but since GDM/X/Whoever iterated over all the
>>>>>> known framebuffers and exported them as DMA-buf (for whatever
>>>>>> reason
>>>>>> idk) we now still have an exported DMA-buf and with it a reference
>>>>>> to the
>>> module.
>>>>>> Any idea how we could prevent that?
>>>>> Here's another stab in the dark.
>>>>>
>>>>> The big difference between old-style fbdev and the new one is that
>>>>> the old fbdev setup (e.g., radeon) allocates a GEM object and puts
>>>>> together the fbdev data structures from the BO in a fairly hackish
>>>>> way. The new style uses an in-kernel client with a file to allocate
>>>>> the BO via dumb buffers; and holds a reference to the DRM module.
>>>>>
>>>>> Maybe the reference comes from the in-kernel DRM client itself. [1]
>>>>> Check if the client resources get released [2] when you unbind vtcon.
>>>>>
>>>>> Best regards
>>>>> Thomas
>>>>>
>>>>> [1]
>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_
>>>>> cl
>>>>> ient.c#L87
>>>>> [2]
>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_
>>>>> cl
>>>>> ient.c#L16
>>>>> 0
>>>>>
>>>>>> Thanks,
>>>>>> Christian.
>>>>> --
>>>>> Thomas Zimmermann
>>>>> Graphics Driver Developer
>>>>> SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg,
>>>>> Germany (HRB 36809, AG Nürnberg)
>>>>> Geschäftsführer: Ivo Totev

RE: Amdgpu module is references even after unbinding the vtcon

[Slivka, Danijel]

[AMD Official Use Only - General]

Hi Christian,

I was not able to check about the handover from fbdev to gdm as I was having issues with setting it up proper driver in the environment with display.
Regarding exporting DMA-buf, when the X starts it acquires file descriptor for the buffer with creating new handle and when closing the file  descriptor it successfully releases the dma buffer that was exported as it is tha last handle on the underlying object.

BR,
Danijel

>-----Original Message-----
>From: Christian König <ckoenig.leichtzumerken@gmail.com>
>Sent: Thursday, January 26, 2023 3:12 PM
>To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann
><tzimmermann@suse.de>
>Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-
>devel@lists.freedesktop.org>; Sharma, Shashank
><Shashank.Sharma@amd.com>
>Subject: Re: Amdgpu module is references even after unbinding the vtcon
>
>Hi Danijel,
>
>can you also double check that GDM/X is still capable of acquiring a DMA-buf
>file descriptor for the buffer (e.g. that we have a DMA-buf handle for it while
>they are started).
>
>And that handover from fbdev to GDM/X is flicker free?
>
>Thanks,
>Christian.
>
>Am 26.01.23 um 14:44 schrieb Slivka, Danijel:
>> [AMD Official Use Only - General]
>>
>> Hi Christian,
>>
>> I have tested the proposed patch, the issue is not reproducible and
>everything else seems to work fine.
>>
>> BR,
>> Danijel
>>
>>> -----Original Message-----
>>> From: Christian König <ckoenig.leichtzumerken@gmail.com>
>>> Sent: Thursday, January 26, 2023 1:20 PM
>>> To: Slivka, Danijel <Danijel.Slivka@amd.com>; Thomas Zimmermann
>>> <tzimmermann@suse.de>
>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; dri-devel <dri-
>>> devel@lists.freedesktop.org>; Sharma, Shashank
>>> <Shashank.Sharma@amd.com>
>>> Subject: Re: Amdgpu module is references even after unbinding the
>>> vtcon
>>>
>>> Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>>>> [AMD Official Use Only - General]
>>>>
>>>> Hi Thomas,
>>>>
>>>> I have checked what you mentioned.
>>>> When loading amdgpu we call  drm_client_init() during fbdev setup
>>>> [1], the
>>> refcnt for drm_kms_helper increases from 3 -> 4.
>>>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but
>>>> the
>>> drm_client_release() [2] is not called.
>>>> The drm_client_release() is called only when unloading the amdgpu
>driver.
>>>>
>>>> Is this expected?
>>> Yes, the client can't be released because it is possible that the
>>> vtcon is bound to this fbdev again.
>>>
>>> Please test the handle work around I've send around internally. At
>>> least for me that approach seems to work.
>>>
>>> Regards,
>>> Christian.
>>>
>>>> There is a comment for drm_client_release with regards to fbdev :
>>>> * This function should only be called from the unregister callback. An
>exception
>>>>    * is fbdev which cannot free the buffer if userspace has open file
>descriptors.
>>>>
>>>> Could this be relevant for our use case, although as
>>>> Application/X/GDM are
>>> stopped at that point and no fd should be open.
>>>> Thank you,
>>>> BR,
>>>> Danijel
>>>>
>>>>> -----Original Message-----
>>>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>>>>> <Danijel.Slivka@amd.com>; dri-devel
>>>>> <dri-devel@lists.freedesktop.org>; Sharma, Shashank
>>>>> <Shashank.Sharma@amd.com>
>>>>> Subject: Re: Amdgpu module is references even after unbinding the
>>>>> vtcon
>>>>>
>>>>> Hi Christian
>>>>>
>>>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>>>> Hi Thomas,
>>>>>>
>>>>>> we ran into a problem with the general fbcon/fbdev implementation
>>>>>> and though that you might have some idea.
>>>>>>
>>>>>> What happens is the following:
>>>>>> 1. We load amdgpu and get our normal fbcon.
>>>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>>>>> imported, exported etc...
>>>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>>>>> kernels knows about and export them as DMA-buf.
>>>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>>>>> released etc...
>>>>>> 6. We unbind vtcon.
>>>>>>
>>>>>> At this point the amdgpu module usually has a reference count of 0
>>>>>> and can be unloaded, but since GDM/X/Whoever iterated over all the
>>>>>> known framebuffers and exported them as DMA-buf (for whatever
>>>>>> reason
>>>>>> idk) we now still have an exported DMA-buf and with it a reference
>>>>>> to the
>>> module.
>>>>>> Any idea how we could prevent that?
>>>>> Here's another stab in the dark.
>>>>>
>>>>> The big difference between old-style fbdev and the new one is that
>>>>> the old fbdev setup (e.g., radeon) allocates a GEM object and puts
>>>>> together the fbdev data structures from the BO in a fairly hackish
>>>>> way. The new style uses an in-kernel client with a file to allocate
>>>>> the BO via dumb buffers; and holds a reference to the DRM module.
>>>>>
>>>>> Maybe the reference comes from the in-kernel DRM client itself. [1]
>>>>> Check if the client resources get released [2] when you unbind vtcon.
>>>>>
>>>>> Best regards
>>>>> Thomas
>>>>>
>>>>> [1]
>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_
>>>>> cl
>>>>> ient.c#L87
>>>>> [2]
>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_
>>>>> cl
>>>>> ient.c#L16
>>>>> 0
>>>>>
>>>>>> Thanks,
>>>>>> Christian.
>>>>> --
>>>>> Thomas Zimmermann
>>>>> Graphics Driver Developer
>>>>> SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409
>Nürnberg,
>>>>> Germany (HRB 36809, AG Nürnberg)
>>>>> Geschäftsführer: Ivo Totev

Re: Amdgpu module is references even after unbinding the vtcon

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 3893 bytes --]

Hi

Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
> [AMD Official Use Only - General]
> 
> Hi Thomas,
> 
> I have checked what you mentioned.
> When loading amdgpu we call  drm_client_init() during fbdev setup [1], the refcnt for drm_kms_helper increases from 3 -> 4.
> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the drm_client_release() [2] is not called.
> The drm_client_release() is called only when unloading the amdgpu driver.
> 
> Is this expected?
> 
> There is a comment for drm_client_release with regards to fbdev :
> * This function should only be called from the unregister callback. An exception
>   * is fbdev which cannot free the buffer if userspace has open file descriptors.
> 
> Could this be relevant for our use case, although as Application/X/GDM are stopped at that point and no fd should be open.

This looks like the bug to me.

I'm not sure why the client code takes the module reference in the first 
place. Drivers invoke client interface directly. Shouldn't that imply 
that they have a module reference already?

Best regards
Thomas

> 
> Thank you,
> BR,
> Danijel
> 
>> -----Original Message-----
>> From: Thomas Zimmermann <tzimmermann@suse.de>
>> Sent: Wednesday, January 25, 2023 8:48 PM
>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>> <Danijel.Slivka@amd.com>; dri-devel <dri-devel@lists.freedesktop.org>; Sharma,
>> Shashank <Shashank.Sharma@amd.com>
>> Subject: Re: Amdgpu module is references even after unbinding the vtcon
>>
>> Hi Christian
>>
>> Am 24.01.23 um 15:12 schrieb Christian König:
>>> Hi Thomas,
>>>
>>> we ran into a problem with the general fbcon/fbdev implementation and
>>> though that you might have some idea.
>>>
>>> What happens is the following:
>>> 1. We load amdgpu and get our normal fbcon.
>>> 2. fbcon allocates a dump BO as backing store for the console.
>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>> imported, exported etc...
>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>> kernels knows about and export them as DMA-buf.
>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>> released etc...
>>> 6. We unbind vtcon.
>>>
>>> At this point the amdgpu module usually has a reference count of 0 and
>>> can be unloaded, but since GDM/X/Whoever iterated over all the known
>>> framebuffers and exported them as DMA-buf (for whatever reason idk) we
>>> now still have an exported DMA-buf and with it a reference to the module.
>>>
>>> Any idea how we could prevent that?
>>
>> Here's another stab in the dark.
>>
>> The big difference between old-style fbdev and the new one is that the old fbdev
>> setup (e.g., radeon) allocates a GEM object and puts together the fbdev data
>> structures from the BO in a fairly hackish way. The new style uses an in-kernel
>> client with a file to allocate the BO via dumb buffers; and holds a reference to the
>> DRM module.
>>
>> Maybe the reference comes from the in-kernel DRM client itself. [1] Check if the
>> client resources get released [2] when you unbind vtcon.
>>
>> Best regards
>> Thomas
>>
>> [1]
>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L87
>> [2]
>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L16
>> 0
>>
>>>
>>> Thanks,
>>> Christian.
>>
>> --
>> Thomas Zimmermann
>> Graphics Driver Developer
>> SUSE Software Solutions Germany GmbH
>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>> (HRB 36809, AG Nürnberg)
>> Geschäftsführer: Ivo Totev

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Amdgpu module is references even after unbinding the vtcon

[Christian König]

Am 26.01.23 um 13:40 schrieb Thomas Zimmermann:
> Hi
>
> Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>> [AMD Official Use Only - General]
>>
>> Hi Thomas,
>>
>> I have checked what you mentioned.
>> When loading amdgpu we call  drm_client_init() during fbdev setup 
>> [1], the refcnt for drm_kms_helper increases from 3 -> 4.
>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the 
>> drm_client_release() [2] is not called.
>> The drm_client_release() is called only when unloading the amdgpu 
>> driver.
>>
>> Is this expected?
>>
>> There is a comment for drm_client_release with regards to fbdev :
>> * This function should only be called from the unregister callback. 
>> An exception
>>   * is fbdev which cannot free the buffer if userspace has open file 
>> descriptors.
>>
>> Could this be relevant for our use case, although as 
>> Application/X/GDM are stopped at that point and no fd should be open.
>
> This looks like the bug to me.
>
> I'm not sure why the client code takes the module reference in the 
> first place. Drivers invoke client interface directly. Shouldn't that 
> imply that they have a module reference already?

It's not the client code who takes the module reference, it's the 
DMA-buf code.

As far as we have narrowed this down GDM/X is inspecting the existing 
configuring during startup, while doing so they export the BO initially 
created by fbdev with DMA-buf (probably to give it to EGL or something 
like this). This DMA-buf export is what's adding the module reference.

The problem is now that when GDM/X exits the DMA-buf should be destroyed 
again, but it isn't because obj->handle_count isn't zero because the 
drm_client interface keeps the handle around even after creating the DRM 
framebuffer object.

Regards,
Christian.

>
> Best regards
> Thomas
>
>>
>> Thank you,
>> BR,
>> Danijel
>>
>>> -----Original Message-----
>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>>> <Danijel.Slivka@amd.com>; dri-devel 
>>> <dri-devel@lists.freedesktop.org>; Sharma,
>>> Shashank <Shashank.Sharma@amd.com>
>>> Subject: Re: Amdgpu module is references even after unbinding the vtcon
>>>
>>> Hi Christian
>>>
>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>> Hi Thomas,
>>>>
>>>> we ran into a problem with the general fbcon/fbdev implementation and
>>>> though that you might have some idea.
>>>>
>>>> What happens is the following:
>>>> 1. We load amdgpu and get our normal fbcon.
>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>>> imported, exported etc...
>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>>> kernels knows about and export them as DMA-buf.
>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>>> released etc...
>>>> 6. We unbind vtcon.
>>>>
>>>> At this point the amdgpu module usually has a reference count of 0 and
>>>> can be unloaded, but since GDM/X/Whoever iterated over all the known
>>>> framebuffers and exported them as DMA-buf (for whatever reason idk) we
>>>> now still have an exported DMA-buf and with it a reference to the 
>>>> module.
>>>>
>>>> Any idea how we could prevent that?
>>>
>>> Here's another stab in the dark.
>>>
>>> The big difference between old-style fbdev and the new one is that 
>>> the old fbdev
>>> setup (e.g., radeon) allocates a GEM object and puts together the 
>>> fbdev data
>>> structures from the BO in a fairly hackish way. The new style uses 
>>> an in-kernel
>>> client with a file to allocate the BO via dumb buffers; and holds a 
>>> reference to the
>>> DRM module.
>>>
>>> Maybe the reference comes from the in-kernel DRM client itself. [1] 
>>> Check if the
>>> client resources get released [2] when you unbind vtcon.
>>>
>>> Best regards
>>> Thomas
>>>
>>> [1]
>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L87 
>>>
>>> [2]
>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L16 
>>>
>>> 0
>>>
>>>>
>>>> Thanks,
>>>> Christian.
>>>
>>> -- 
>>> Thomas Zimmermann
>>> Graphics Driver Developer
>>> SUSE Software Solutions Germany GmbH
>>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>>> (HRB 36809, AG Nürnberg)
>>> Geschäftsführer: Ivo Totev
>

Re: Amdgpu module is references even after unbinding the vtcon

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 5083 bytes --]

Hi

Am 26.01.23 um 13:45 schrieb Christian König:
> Am 26.01.23 um 13:40 schrieb Thomas Zimmermann:
>> Hi
>>
>> Am 26.01.23 um 10:49 schrieb Slivka, Danijel:
>>> [AMD Official Use Only - General]
>>>
>>> Hi Thomas,
>>>
>>> I have checked what you mentioned.
>>> When loading amdgpu we call  drm_client_init() during fbdev setup 
>>> [1], the refcnt for drm_kms_helper increases from 3 -> 4.
>>> When we unbind vtcon, refcnt for drm_kms_helper drops 4 -> 3, but the 
>>> drm_client_release() [2] is not called.
>>> The drm_client_release() is called only when unloading the amdgpu 
>>> driver.
>>>
>>> Is this expected?
>>>
>>> There is a comment for drm_client_release with regards to fbdev :
>>> * This function should only be called from the unregister callback. 
>>> An exception
>>>   * is fbdev which cannot free the buffer if userspace has open file 
>>> descriptors.
>>>
>>> Could this be relevant for our use case, although as 
>>> Application/X/GDM are stopped at that point and no fd should be open.
>>
>> This looks like the bug to me.
>>
>> I'm not sure why the client code takes the module reference in the 
>> first place. Drivers invoke client interface directly. Shouldn't that 
>> imply that they have a module reference already?
> 
> It's not the client code who takes the module reference, it's the 
> DMA-buf code.
> 
> As far as we have narrowed this down GDM/X is inspecting the existing 
> configuring during startup, while doing so they export the BO initially 
> created by fbdev with DMA-buf (probably to give it to EGL or something 
> like this). This DMA-buf export is what's adding the module reference.
> 
> The problem is now that when GDM/X exits the DMA-buf should be destroyed 
> again, but it isn't because obj->handle_count isn't zero because the 
> drm_client interface keeps the handle around even after creating the DRM 
> framebuffer object.

OK, thanks. I saw your patch to address the problem. Let me give it a test.

Best regards
Thomas

> 
> Regards,
> Christian.
> 
>>
>> Best regards
>> Thomas
>>
>>>
>>> Thank you,
>>> BR,
>>> Danijel
>>>
>>>> -----Original Message-----
>>>> From: Thomas Zimmermann <tzimmermann@suse.de>
>>>> Sent: Wednesday, January 25, 2023 8:48 PM
>>>> To: Christian König <ckoenig.leichtzumerken@gmail.com>
>>>> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Slivka, Danijel
>>>> <Danijel.Slivka@amd.com>; dri-devel 
>>>> <dri-devel@lists.freedesktop.org>; Sharma,
>>>> Shashank <Shashank.Sharma@amd.com>
>>>> Subject: Re: Amdgpu module is references even after unbinding the vtcon
>>>>
>>>> Hi Christian
>>>>
>>>> Am 24.01.23 um 15:12 schrieb Christian König:
>>>>> Hi Thomas,
>>>>>
>>>>> we ran into a problem with the general fbcon/fbdev implementation and
>>>>> though that you might have some idea.
>>>>>
>>>>> What happens is the following:
>>>>> 1. We load amdgpu and get our normal fbcon.
>>>>> 2. fbcon allocates a dump BO as backing store for the console.
>>>>> 3. GDM/X/Applications start, new framebuffers are created BOs
>>>>> imported, exported etc...
>>>>> 4. Somehow X or GDM iterated over all the framebuffer objects the
>>>>> kernels knows about and export them as DMA-buf.
>>>>> 5. Application/X/GDM are stopped, handles closed, framebuffers
>>>>> released etc...
>>>>> 6. We unbind vtcon.
>>>>>
>>>>> At this point the amdgpu module usually has a reference count of 0 and
>>>>> can be unloaded, but since GDM/X/Whoever iterated over all the known
>>>>> framebuffers and exported them as DMA-buf (for whatever reason idk) we
>>>>> now still have an exported DMA-buf and with it a reference to the 
>>>>> module.
>>>>>
>>>>> Any idea how we could prevent that?
>>>>
>>>> Here's another stab in the dark.
>>>>
>>>> The big difference between old-style fbdev and the new one is that 
>>>> the old fbdev
>>>> setup (e.g., radeon) allocates a GEM object and puts together the 
>>>> fbdev data
>>>> structures from the BO in a fairly hackish way. The new style uses 
>>>> an in-kernel
>>>> client with a file to allocate the BO via dumb buffers; and holds a 
>>>> reference to the
>>>> DRM module.
>>>>
>>>> Maybe the reference comes from the in-kernel DRM client itself. [1] 
>>>> Check if the
>>>> client resources get released [2] when you unbind vtcon.
>>>>
>>>> Best regards
>>>> Thomas
>>>>
>>>> [1]
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L87
>>>> [2]
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_client.c#L16
>>>> 0
>>>>
>>>>>
>>>>> Thanks,
>>>>> Christian.
>>>>
>>>> -- 
>>>> Thomas Zimmermann
>>>> Graphics Driver Developer
>>>> SUSE Software Solutions Germany GmbH
>>>> Maxfeldstr. 5, 90409 Nürnberg, Germany
>>>> (HRB 36809, AG Nürnberg)
>>>> Geschäftsführer: Ivo Totev
>>
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]