From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9C152C52D7F for ; Thu, 15 Aug 2024 20:26:09 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4AAD810E540; Thu, 15 Aug 2024 20:26:09 +0000 (UTC) X-Greylist: delayed 304 seconds by postgrey-1.36 at gabe; Thu, 15 Aug 2024 20:26:07 UTC Received: from us-smtp-delivery-44.mimecast.com (us-smtp-delivery-44.mimecast.com [205.139.111.44]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9246210E540 for ; Thu, 15 Aug 2024 20:26:07 +0000 (UTC) Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-637-PbOJIVbGN2amJy1OMmbvSQ-1; Thu, 15 Aug 2024 16:19:32 -0400 X-MC-Unique: PbOJIVbGN2amJy1OMmbvSQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1E794187487C; Thu, 15 Aug 2024 20:19:28 +0000 (UTC) Received: from dreadlord.redhat.com (unknown [10.64.136.9]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CA6A51955DD1; Thu, 15 Aug 2024 20:19:25 +0000 (UTC) From: Dave Airlie To: dri-devel@lists.freedesktop.org Cc: nouveau@lists.freedesktop.org, dakr@redhat.com Subject: [PATCH] nouveau/firmware: using dma non-coherent interfaces for fw loading. (v2) Date: Fri, 16 Aug 2024 06:19:23 +1000 Message-ID: <20240815201923.632803-1-airlied@gmail.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: gmail.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=WINDOWS-1252; x-default=true X-BeenThere: nouveau@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Nouveau development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: nouveau-bounces@lists.freedesktop.org Sender: "Nouveau" From: Dave Airlie Currently, enabling SG_DEBUG in the kernel will cause nouveau to hit a BUG() on startup, when the iommu is enabled: kernel BUG at include/linux/scatterlist.h:187! invalid opcode: 0000 [#1] PREEMPT SMP NOPTI CPU: 7 PID: 930 Comm: (udev-worker) Not tainted 6.9.0-rc3Lyude-Test+ #30 Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.I0 01/22/2019 RIP: 0010:sg_init_one+0x85/0xa0 Code: 69 88 32 01 83 e1 03 f6 c3 03 75 20 a8 01 75 1e 48 09 cb 41 89 54 24 08 49 89 1c 24 41 89 6c 24 0c 5b 5d 41 5c e9 7b b9 88 00 <0f> 0b 0f 0b 0f 0b 48 8b 05 5e 46 9a 01 eb b2 66 66 2e 0f 1f 84 00 RSP: 0018:ffffa776017bf6a0 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffffa77600d87000 RCX: 000000000000002b RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffa77680d87000 RBP: 000000000000e000 R08: 0000000000000000 R09: 0000000000000000 R10: ffff98f4c46aa508 R11: 0000000000000000 R12: ffff98f4c46aa508 R13: ffff98f4c46aa008 R14: ffffa77600d4a000 R15: ffffa77600d4a018 FS: 00007feeb5aae980(0000) GS:ffff98f5c4dc0000(0000) knlGS:000000000000000= 0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f22cb9a4520 CR3: 00000001043ba000 CR4: 00000000003506f0 Call Trace: ? die+0x36/0x90 ? do_trap+0xdd/0x100 ? sg_init_one+0x85/0xa0 ? do_error_trap+0x65/0x80 ? sg_init_one+0x85/0xa0 ? exc_invalid_op+0x50/0x70 ? sg_init_one+0x85/0xa0 ? asm_exc_invalid_op+0x1a/0x20 ? sg_init_one+0x85/0xa0 nvkm_firmware_ctor+0x14a/0x250 [nouveau] nvkm_falcon_fw_ctor+0x42/0x70 [nouveau] ga102_gsp_booter_ctor+0xb4/0x1a0 [nouveau] r535_gsp_oneinit+0xb3/0x15f0 [nouveau] ? srso_return_thunk+0x5/0x5f ? srso_return_thunk+0x5/0x5f ? nvkm_udevice_new+0x95/0x140 [nouveau] ? srso_return_thunk+0x5/0x5f ? srso_return_thunk+0x5/0x5f ? ktime_get+0x47/0xb0 Fix this by using the non-coherent allocator instead, I think there might be a better answer to this, but it involve ripping up some of APIs using sg lists. v2: fix build warning Signed-off-by: Dave Airlie Cc: stable@vger.kernel.org --- drivers/gpu/drm/nouveau/nvkm/core/firmware.c | 9 ++++++--- drivers/gpu/drm/nouveau/nvkm/falcon/fw.c | 6 ++++++ 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nvkm/core/firmware.c b/drivers/gpu/drm= /nouveau/nvkm/core/firmware.c index adc60b25f8e6..0af01a0ec601 100644 --- a/drivers/gpu/drm/nouveau/nvkm/core/firmware.c +++ b/drivers/gpu/drm/nouveau/nvkm/core/firmware.c @@ -205,7 +205,8 @@ nvkm_firmware_dtor(struct nvkm_firmware *fw) =09=09break; =09case NVKM_FIRMWARE_IMG_DMA: =09=09nvkm_memory_unref(&memory); -=09=09dma_free_coherent(fw->device->dev, sg_dma_len(&fw->mem.sgl), fw->img= , fw->phys); +=09=09dma_free_noncoherent(fw->device->dev, sg_dma_len(&fw->mem.sgl), +=09=09=09=09 fw->img, fw->phys, DMA_TO_DEVICE); =09=09break; =09case NVKM_FIRMWARE_IMG_SGT: =09=09nvkm_memory_unref(&memory); @@ -236,10 +237,12 @@ nvkm_firmware_ctor(const struct nvkm_firmware_func *f= unc, const char *name, =09=09break; =09case NVKM_FIRMWARE_IMG_DMA: { =09=09dma_addr_t addr; - =09=09len =3D ALIGN(fw->len, PAGE_SIZE); =20 -=09=09fw->img =3D dma_alloc_coherent(fw->device->dev, len, &addr, GFP_KERN= EL); +=09=09fw->img =3D dma_alloc_noncoherent(fw->device->dev, +=09=09=09=09=09=09len, &addr, +=09=09=09=09=09=09DMA_TO_DEVICE, +=09=09=09=09=09=09GFP_KERNEL); =09=09if (fw->img) { =09=09=09memcpy(fw->img, src, fw->len); =09=09=09fw->phys =3D addr; diff --git a/drivers/gpu/drm/nouveau/nvkm/falcon/fw.c b/drivers/gpu/drm/nou= veau/nvkm/falcon/fw.c index 80a480b12174..a1c8545f1249 100644 --- a/drivers/gpu/drm/nouveau/nvkm/falcon/fw.c +++ b/drivers/gpu/drm/nouveau/nvkm/falcon/fw.c @@ -89,6 +89,12 @@ nvkm_falcon_fw_boot(struct nvkm_falcon_fw *fw, struct nv= km_subdev *user, =09=09nvkm_falcon_fw_dtor_sigs(fw); =09} =20 +=09/* after last write to the img, sync dma mappings */ +=09dma_sync_single_for_device(fw->fw.device->dev, +=09=09=09=09 fw->fw.phys, +=09=09=09=09 sg_dma_len(&fw->fw.mem.sgl), +=09=09=09=09 DMA_TO_DEVICE); + =09FLCNFW_DBG(fw, "resetting"); =09fw->func->reset(fw); =20 --=20 2.46.0