* Re: [PATCH] docs: zh_TW: process: localize terminologies and improve fluency in 8.Conclusion
From: Dongliang Mu @ 2026-05-29 11:33 UTC (permalink / raw)
To: 葉宸佑
Cc: Hu Haowen, Jonathan Corbet, Shuah Khan, linux-doc, linux-kernel,
慕冬亮
In-Reply-To: <CAKspUh+5h+MeMVEEGy4Oc+HJteaTE1Ft1XXhDPJEmL-218T4Jg@mail.gmail.com>
On Fri, May 29, 2026 at 7:26 PM 葉宸佑 <chenyou910331@gmail.com> wrote:
>
> Hi Dongliang,
>
> Thank you for reviewing this patch and for your careful observation!
>
> You raised a very valid point. In PRC terminology, "內核" (Kernel) and
> "核心" (Core) are distinctly separated. However, in Taiwan's official
> tech terminology (e.g., by the National Academy for Educational
> Research) and local academic/industry usage, "Kernel" is universally
> translated as "核心" (e.g., Linux Kernel -> Linux 核心, Kernel space ->
> 核心空間).
>
> It is true that "Core" is also translated as "核心" in Taiwan, which
> creates a lexical overlap. However, Taiwanese developers are entirely
> used to relying on the context to differentiate them. In the context
> of operating systems and Linux documentation, local developers will
> natively map "核心 API" to "Kernel API" without any confusion.
>
> If you feel it might still be too ambiguous, another common practice
> in Taiwan is to simply keep the English word for technical precision.
> We could change it to "Linux 核心 API" or just leave it as "kernel API".
>
> For local developers, the current translation "核心 API" is perfectly
> natural, but I am happy to send a v2 patch to adjust this specific
> phrase if you prefer. Please let me know your thoughts!
If local developers have no problem with this part, I am fine too.
Reviewed-by: Dongliang Mu <dzm91@hust.edu.cn>
>
> Best regards,
> Chen You
>
>
> Dongliang Mu <mudongliangabcd@gmail.com> 於 2026年5月29日週五 下午12:34寫道：
> >
> > On Thu, May 28, 2026 at 12:14 PM CHEN-YOU-0331 <chenyou910331@gmail.com> wrote:
> > >
> > > Translate PRC tech terms into Taiwanese tech terms (e.g.,
> > > 內核 -> 核心, 代碼 -> 程式碼, 軟件 -> 軟體) to improve
> > > readability for local developers. Also, rephrase several
> > > awkward sentences to make the document more fluent.
> >
> > I am not very familiar with Taiwanese tech terms. After checking with
> > LLMs, I am almost fine with these term changes, except one place.
> >
> > >
> > > Signed-off-by: CHEN-YOU-0331 <chenyou910331@gmail.com>
> > > ---
> > > .../zh_TW/process/8.Conclusion.rst | 45 +++++++++----------
> > > 1 file changed, 22 insertions(+), 23 deletions(-)
> > >
> > > diff --git a/Documentation/translations/zh_TW/process/8.Conclusion.rst b/Documentation/translations/zh_TW/process/8.Conclusion.rst
> > > index d1634421b62c..823969cf793d 100644
> > > --- a/Documentation/translations/zh_TW/process/8.Conclusion.rst
> > > +++ b/Documentation/translations/zh_TW/process/8.Conclusion.rst
> > > @@ -14,42 +14,41 @@
> > >
> > > .. _tw_development_conclusion:
> > >
> > > -更多信息
> > > +更多資訊
> > > ========
> > >
> > > -關於Linux內核開發和相關主題的信息來源很多。首先是在內核源代碼分發中找到的
> > > -文檔目錄。頂級
> > > +關於Linux核心開發和相關主題的資訊來源很多。首先是在核心原始碼分發中找到的
> > > +文件目錄。頂級
> > > :ref:`Documentation/translations/zh_CN/process/howto.rst <tw_process_howto>`
> > > 文件是一個重要的起點；
> > > :ref:`Documentation/translations/zh_CN/process/submitting-patches.rst <tw_submittingpatches>`
> > > -也是所有內核開發人員都應該閱讀的內容。許多內部內核API都是使用kerneldoc機制
> > > -記錄的；“make htmldocs”或“make pdfdocs”可用於以HTML或PDF格式生成這些文檔
> > > -（儘管某些發行版提供的tex版本會遇到內部限制，無法正確處理文檔）。
> > > -
> > > -不同的網站在各個細節層次上討論內核開發。本文作者想謙虛地建議用 https://lwn.net/
> > > -作爲來源；有關許多特定內核主題的信息可以通過以下網址的 LWN 內核索引找到：
> > > +也是所有核心開發人員都應該閱讀的內容。許多內部核心API都是使用kerneldoc機制
> >
> > "核心API" may cause confusion to me, which means "core API" other than
> > "kernel API".
> >
> > Dongliang Mu
> >
> > > +記錄的；“make htmldocs”或“make pdfdocs”可用於以HTML或PDF格式生成這些文件
> > > +（儘管某些發行版提供的tex版本會遇到內部限制，無法正確處理文件）。
> > >
> > > +不同的網站在各個細節層次上討論核心開發。本文作者想謙虛地建議用 https://lwn.net/
> > > +作爲來源；有關許多特定核心主題的資訊可以通過以下網址的 LWN 核心索引找到:
> > > http://lwn.net/kernel/index/
> > >
> > > -除此之外，內核開發人員的一個寶貴資源是：
> > > +除此之外，核心開發人員的一個寶貴資源是：
> > >
> > > https://kernelnewbies.org/
> > >
> > > -當然，也不應該忘記 https://kernel.org/ ，這是內核發佈信息的最終位置。
> > > +當然，也不應該忘記 https://kernel.org/ ，這是核心發佈資訊的最終位置。
> > >
> > > -關於內核開發有很多書：
> > > +關於核心開發有很多書：
> > >
> > > 《Linux設備驅動程序》第三版（Jonathan Corbet、Alessandro Rubini和Greg Kroah Hartman）
> > > 線上版本在 http://lwn.net/kernel/ldd3/
> > >
> > > - 《Linux內核設計與實現》（Robert Love）
> > > + 《Linux核心設計與實現》（Robert Love）
> > >
> > > - 《深入理解Linux內核》(Daniel Bovet和Marco Cesati）
> > > + 《深入理解Linux核心》(Daniel Bovet和Marco Cesati）
> > >
> > > 然而，所有這些書都有一個共同的缺點：它們上架時就往往有些過時，而且已經上架
> > > -一段時間了。不過，在那裏還是可以找到相當多的好信息。
> > > +一段時間了。不過，在那裏還是可以找到相當多的好資訊。
> > >
> > > -有關git的文檔，請訪問：
> > > +有關git的文件，請訪問：
> > >
> > > https://www.kernel.org/pub/software/scm/git/docs/
> > >
> > > @@ -58,16 +57,16 @@
> > > 結論
> > > ====
> > >
> > > -祝賀所有通過這篇冗長的文檔的人。希望它能夠幫助您理解Linux內核是如何開發的，
> > > +祝賀所有通過這篇冗長的文件的人。希望它能夠幫助您理解Linux核心是如何開發的，
> > > 以及您如何參與這個過程。
> > >
> > > -最後，重要的是參與。任何開源軟件項目都不會超過其貢獻者投入其中的總和。Linux
> > > -內核的發展速度和以前一樣快，因爲它得到了大量開發人員的幫助，他們都在努力使它
> > > -變得更好。內核是一個最成功的例子，說明了當成千上萬的人爲了一個共同的目標一起
> > > +最後，重要的是參與。任何開源軟體專案都不會超過其貢獻者投入其中的總和。Linux
> > > +核心的發展速度和以前一樣快，因爲它得到了大量開發人員的幫助，他們都在努力使它
> > > +變得更好。核心是一個最成功的例子，說明了當成千上萬的人爲了一個共同的目標一起
> > > 工作時，可以做出什麼。
> > >
> > > -不過，內核總是可以從更大的開發人員基礎中獲益。總有更多的工作要做。但是同樣
> > > -重要的是，Linux生態系統中的大多數其他參與者可以通過爲內核做出貢獻而受益。使
> > > -代碼進入主線是提高代碼質量、降低維護和分發成本、提高對內核開發方向的影響程度
> > > +不過，核心總是可以從更大的開發人員基礎中獲益。總有更多的工作要做。但是同樣
> > > +重要的是，Linux生態系統中的大多數其他參與者可以通過爲核心做出貢獻而受益。使
> > > +程式碼進入主線是提高程式碼品質、降低維護和分發成本、提高對核心開發方向的影響程度
> > > 等的關鍵。這是一種共贏的局面。啓動你的編輯器，來加入我們吧；你會非常受歡迎的。
> > >
> > > --
> > > 2.43.0
> > >
> > >
^ permalink raw reply
* Re: [PATCH] docs: zh_TW: process: localize terminologies and improve fluency in 8.Conclusion
From: 葉宸佑 @ 2026-05-29 11:25 UTC (permalink / raw)
To: Dongliang Mu
Cc: Hu Haowen, Jonathan Corbet, Shuah Khan, linux-doc, linux-kernel
In-Reply-To: <CAD-N9QVD+L6Pgx7dVkoYMra1v02tOE7P8gCsuSMa2Nyk8H1DgQ@mail.gmail.com>
Hi Dongliang,
Thank you for reviewing this patch and for your careful observation!
You raised a very valid point. In PRC terminology, "內核" (Kernel) and
"核心" (Core) are distinctly separated. However, in Taiwan's official
tech terminology (e.g., by the National Academy for Educational
Research) and local academic/industry usage, "Kernel" is universally
translated as "核心" (e.g., Linux Kernel -> Linux 核心, Kernel space ->
核心空間).
It is true that "Core" is also translated as "核心" in Taiwan, which
creates a lexical overlap. However, Taiwanese developers are entirely
used to relying on the context to differentiate them. In the context
of operating systems and Linux documentation, local developers will
natively map "核心 API" to "Kernel API" without any confusion.
If you feel it might still be too ambiguous, another common practice
in Taiwan is to simply keep the English word for technical precision.
We could change it to "Linux 核心 API" or just leave it as "kernel API".
For local developers, the current translation "核心 API" is perfectly
natural, but I am happy to send a v2 patch to adjust this specific
phrase if you prefer. Please let me know your thoughts!
Best regards,
Chen You
Dongliang Mu <mudongliangabcd@gmail.com> 於 2026年5月29日週五 下午12:34寫道：
>
> On Thu, May 28, 2026 at 12:14 PM CHEN-YOU-0331 <chenyou910331@gmail.com> wrote:
> >
> > Translate PRC tech terms into Taiwanese tech terms (e.g.,
> > 內核 -> 核心, 代碼 -> 程式碼, 軟件 -> 軟體) to improve
> > readability for local developers. Also, rephrase several
> > awkward sentences to make the document more fluent.
>
> I am not very familiar with Taiwanese tech terms. After checking with
> LLMs, I am almost fine with these term changes, except one place.
>
> >
> > Signed-off-by: CHEN-YOU-0331 <chenyou910331@gmail.com>
> > ---
> > .../zh_TW/process/8.Conclusion.rst | 45 +++++++++----------
> > 1 file changed, 22 insertions(+), 23 deletions(-)
> >
> > diff --git a/Documentation/translations/zh_TW/process/8.Conclusion.rst b/Documentation/translations/zh_TW/process/8.Conclusion.rst
> > index d1634421b62c..823969cf793d 100644
> > --- a/Documentation/translations/zh_TW/process/8.Conclusion.rst
> > +++ b/Documentation/translations/zh_TW/process/8.Conclusion.rst
> > @@ -14,42 +14,41 @@
> >
> > .. _tw_development_conclusion:
> >
> > -更多信息
> > +更多資訊
> > ========
> >
> > -關於Linux內核開發和相關主題的信息來源很多。首先是在內核源代碼分發中找到的
> > -文檔目錄。頂級
> > +關於Linux核心開發和相關主題的資訊來源很多。首先是在核心原始碼分發中找到的
> > +文件目錄。頂級
> > :ref:`Documentation/translations/zh_CN/process/howto.rst <tw_process_howto>`
> > 文件是一個重要的起點；
> > :ref:`Documentation/translations/zh_CN/process/submitting-patches.rst <tw_submittingpatches>`
> > -也是所有內核開發人員都應該閱讀的內容。許多內部內核API都是使用kerneldoc機制
> > -記錄的；“make htmldocs”或“make pdfdocs”可用於以HTML或PDF格式生成這些文檔
> > -（儘管某些發行版提供的tex版本會遇到內部限制，無法正確處理文檔）。
> > -
> > -不同的網站在各個細節層次上討論內核開發。本文作者想謙虛地建議用 https://lwn.net/
> > -作爲來源；有關許多特定內核主題的信息可以通過以下網址的 LWN 內核索引找到：
> > +也是所有核心開發人員都應該閱讀的內容。許多內部核心API都是使用kerneldoc機制
>
> "核心API" may cause confusion to me, which means "core API" other than
> "kernel API".
>
> Dongliang Mu
>
> > +記錄的；“make htmldocs”或“make pdfdocs”可用於以HTML或PDF格式生成這些文件
> > +（儘管某些發行版提供的tex版本會遇到內部限制，無法正確處理文件）。
> >
> > +不同的網站在各個細節層次上討論核心開發。本文作者想謙虛地建議用 https://lwn.net/
> > +作爲來源；有關許多特定核心主題的資訊可以通過以下網址的 LWN 核心索引找到:
> > http://lwn.net/kernel/index/
> >
> > -除此之外，內核開發人員的一個寶貴資源是：
> > +除此之外，核心開發人員的一個寶貴資源是：
> >
> > https://kernelnewbies.org/
> >
> > -當然，也不應該忘記 https://kernel.org/ ，這是內核發佈信息的最終位置。
> > +當然，也不應該忘記 https://kernel.org/ ，這是核心發佈資訊的最終位置。
> >
> > -關於內核開發有很多書：
> > +關於核心開發有很多書：
> >
> > 《Linux設備驅動程序》第三版（Jonathan Corbet、Alessandro Rubini和Greg Kroah Hartman）
> > 線上版本在 http://lwn.net/kernel/ldd3/
> >
> > - 《Linux內核設計與實現》（Robert Love）
> > + 《Linux核心設計與實現》（Robert Love）
> >
> > - 《深入理解Linux內核》(Daniel Bovet和Marco Cesati）
> > + 《深入理解Linux核心》(Daniel Bovet和Marco Cesati）
> >
> > 然而，所有這些書都有一個共同的缺點：它們上架時就往往有些過時，而且已經上架
> > -一段時間了。不過，在那裏還是可以找到相當多的好信息。
> > +一段時間了。不過，在那裏還是可以找到相當多的好資訊。
> >
> > -有關git的文檔，請訪問：
> > +有關git的文件，請訪問：
> >
> > https://www.kernel.org/pub/software/scm/git/docs/
> >
> > @@ -58,16 +57,16 @@
> > 結論
> > ====
> >
> > -祝賀所有通過這篇冗長的文檔的人。希望它能夠幫助您理解Linux內核是如何開發的，
> > +祝賀所有通過這篇冗長的文件的人。希望它能夠幫助您理解Linux核心是如何開發的，
> > 以及您如何參與這個過程。
> >
> > -最後，重要的是參與。任何開源軟件項目都不會超過其貢獻者投入其中的總和。Linux
> > -內核的發展速度和以前一樣快，因爲它得到了大量開發人員的幫助，他們都在努力使它
> > -變得更好。內核是一個最成功的例子，說明了當成千上萬的人爲了一個共同的目標一起
> > +最後，重要的是參與。任何開源軟體專案都不會超過其貢獻者投入其中的總和。Linux
> > +核心的發展速度和以前一樣快，因爲它得到了大量開發人員的幫助，他們都在努力使它
> > +變得更好。核心是一個最成功的例子，說明了當成千上萬的人爲了一個共同的目標一起
> > 工作時，可以做出什麼。
> >
> > -不過，內核總是可以從更大的開發人員基礎中獲益。總有更多的工作要做。但是同樣
> > -重要的是，Linux生態系統中的大多數其他參與者可以通過爲內核做出貢獻而受益。使
> > -代碼進入主線是提高代碼質量、降低維護和分發成本、提高對內核開發方向的影響程度
> > +不過，核心總是可以從更大的開發人員基礎中獲益。總有更多的工作要做。但是同樣
> > +重要的是，Linux生態系統中的大多數其他參與者可以通過爲核心做出貢獻而受益。使
> > +程式碼進入主線是提高程式碼品質、降低維護和分發成本、提高對核心開發方向的影響程度
> > 等的關鍵。這是一種共贏的局面。啓動你的編輯器，來加入我們吧；你會非常受歡迎的。
> >
> > --
> > 2.43.0
> >
> >
^ permalink raw reply
* RE: [PATCH v13 2/6] iio: adc: ad4691: add initial driver for AD4691 family
From: Sabau, Radu bogdan @ 2026-05-29 10:06 UTC (permalink / raw)
To: Jonathan Cameron, Radu Sabau via B4 Relay
Cc: Lars-Peter Clausen, Hennerich, Michael, David Lechner, Sa, Nuno,
Andy Shevchenko, Rob Herring, Krzysztof Kozlowski, Conor Dooley,
Uwe Kleine-König, Liam Girdwood, Mark Brown, Linus Walleij,
Bartosz Golaszewski, Philipp Zabel, Jonathan Corbet, Shuah Khan,
linux-iio@vger.kernel.org, devicetree@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-pwm@vger.kernel.org,
linux-gpio@vger.kernel.org, linux-doc@vger.kernel.org
In-Reply-To: <20260527190713.1cba1e89@jic23-huawei>
> -----Original Message-----
> From: Jonathan Cameron <jic23@kernel.org>
> Sent: Wednesday, May 27, 2026 9:07 PM
...
> > +static bool ad4691_volatile_reg(struct device *dev, unsigned int reg)
> > +{
> > + switch (reg) {
> > + case AD4691_STATUS_REG:
> > + case AD4691_CLAMP_STATUS1_REG:
> > + case AD4691_CLAMP_STATUS2_REG:
> > + case AD4691_GPIO_READ:
> > + case AD4691_ACC_STATUS_FULL1_REG ...
> AD4691_ACC_STATUS_SAT2_REG:
> > + case AD4691_ACC_SAT_OVR_REG(0) ...
> AD4691_ACC_SAT_OVR_REG(15):
> > + return true;
> > + default:
> > + break;
> > + }
> > +
> > + /*
> > + * Multi-byte registers have non-unit strides; only accept base
> > + * addresses to prevent debugfs from triggering reads that cross
> > + * register boundaries.
> What does this have to do with preventing debugfs from doing anything.
> It should allow reading of volatile registers. Why is it not enough to
> make the non base aligned registers neither readable nor writeable?
>
You're right, the comment is wrong and the stride logic is misplaced.
readable_reg is what gates debugfs access, so the stride checks in
volatile_reg are redundant and the comment is incorrect.
^ permalink raw reply
* [PATCH] cpufreq: Documentation: fix freq_step description
From: Pengjie Zhang @ 2026-05-29 11:11 UTC (permalink / raw)
To: rafael, viresh.kumar, corbet, skhan
Cc: linux-pm, linux-doc, linux-kernel, zhanjie9, prime.zeng,
wanghuiqiang, xuwei5, lihuisong, zhenglifeng1, yubowen8,
zhangpengjie2, wangzhi12
The conservative governor documentation incorrectly states that setting
freq_step to 0 will use the default 5% frequency step. In reality, since
the governor's initial implementation
commit b9170836d1aa ("[CPUFREQ] Conservative cpufreq governer"),
freq_step=0 has always caused the governor to skip frequency updates
entirely.
Correct the documentation to reflect the actual behavior: freq_step=0
disables frequency changes by the governor entirely.
Fixes: 2a0e49279850 ("cpufreq: User/admin documentation update and consolidation")
Signed-off-by: Pengjie Zhang <zhangpengjie2@huawei.com>
---
Documentation/admin-guide/pm/cpufreq.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/admin-guide/pm/cpufreq.rst b/Documentation/admin-guide/pm/cpufreq.rst
index dbe6d23a5d67..98c724d49047 100644
--- a/Documentation/admin-guide/pm/cpufreq.rst
+++ b/Documentation/admin-guide/pm/cpufreq.rst
@@ -586,8 +586,8 @@ This governor exposes the following tunables:
100 (5 by default).
This is how much the frequency is allowed to change in one go. Setting
- it to 0 will cause the default frequency step (5 percent) to be used
- and setting it to 100 effectively causes the governor to periodically
+ it to 0 disables frequency changes by the governor entirely and setting
+ it to 100 effectively causes the governor to periodically
switch the frequency between the ``scaling_min_freq`` and
``scaling_max_freq`` policy limits.
--
2.33.0
^ permalink raw reply related
* [PATCH 11/11] net: wwan: t9xx: Add maintainers and documentation
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Adds maintainers and documentation for MediaTek T9XX 5G WWAN modem
device driver.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
.../networking/device_drivers/wwan/t9xx.rst | 48 ++++++++++++++++++++++
MAINTAINERS | 10 +++++
2 files changed, 58 insertions(+)
diff --git a/Documentation/networking/device_drivers/wwan/t9xx.rst b/Documentation/networking/device_drivers/wwan/t9xx.rst
new file mode 100644
index 000000000000..eaba82f18c1b
--- /dev/null
+++ b/Documentation/networking/device_drivers/wwan/t9xx.rst
@@ -0,0 +1,48 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+.. Copyright (c) 2025, MediaTek Inc.
+
+.. _t9xx_driver_doc:
+
+====================================================
+T9XX driver for MTK PCIe based T-series 5G Modem
+====================================================
+The T9XX driver is a WWAN PCIe host driver developed
+for data exchange over PCIe interface between Host platform and MediaTek's
+T-series 5G modem. The driver exposes control plane and data plane interfaces
+to applications. The control plane provides device node interfaces for control
+data transactions. The data plane provides network link interfaces for IP data
+transactions.
+
+Control channel userspace ABI
+=============================
+/dev/wwan0at0 character device
+------------------------------
+The driver exposes an AT port by implementing AT WWAN Port.
+The userspace end of the control channel pipe is a /dev/wwan0at0 character
+device. Application shall use this interface to issue AT commands.
+
+/dev/wwan0mbim0 character device
+--------------------------------
+The driver exposes an MBIM interface to the MBIM function by implementing
+MBIM WWAN Port. The userspace end of the control channel pipe is a
+/dev/wwan0mbim0 character device. Applications shall use this interface
+for MBIM protocol communication.
+
+Data channel userspace ABI
+==========================
+wwan0-X network device
+----------------------
+The T9XX driver exposes IP link interfaces "wwan0-X" of type "wwan" for IP
+traffic. Iproute network utility is used for creating "wwan0-X" network
+interfaces and for associating it with the MBIM IP session.
+
+The userspace management application is responsible for creating a new IP link
+prior to establishing an MBIM IP session where the SessionId is greater than 0.
+
+For example, creating a new IP link for an MBIM IP session with SessionId 1:
+
+ ip link add dev wwan0-1 parentdev wwan0 type wwan linkid 1
+
+The driver will automatically map the "wwan0-1" network device to MBIM IP
+session 1.
diff --git a/MAINTAINERS b/MAINTAINERS
index 461a3eed6129..3fab2b47f531 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16494,6 +16494,16 @@ L: netdev@vger.kernel.org
S: Supported
F: drivers/net/wwan/t7xx/
+MEDIATEK T9XX 5G WWAN MODEM DRIVER
+M: Jack Wu <jackbb_wu@compal.com>
+R: Wen-Zhi Huang <wen-zhi.huang@mediatek.com>
+R: Shi-Wei Yeh <shi-wei.yeh@mediatek.com>
+R: Minano Tseng <Minano.tseng@mediatek.com>
+L: netdev@vger.kernel.org
+S: Supported
+F: Documentation/networking/device_drivers/wwan/t9xx.rst
+F: drivers/net/wwan/t9xx/
+
MEDIATEK USB3 DRD IP DRIVER
M: Chunfeng Yun <chunfeng.yun@mediatek.com>
L: linux-usb@vger.kernel.org
--
2.34.1
^ permalink raw reply related
* [PATCH 10/11] net: wwan: t9xx: Add power management support
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Add s2idle (S0ix) power management support for the t9xx WWAN driver.
In s2idle the modem remains powered. The driver must quiesce host-side
DMA engines and service threads before the platform enters low-power
state, then restore them on resume.
- Suspend: park TRB service threads, stop CLDMA TX/RX queues,
disable DPMAIF data path, mask MHCCIF and MSIX interrupts,
save PCIe state
- Resume: restore PCIe state, re-initialize ATR, unmask MHCCIF,
resume CLDMA queues, re-enable DPMAIF data path, unpark TRB
service threads
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 94 +++++++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 3 ++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c | 35 +++++++++++-
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h | 2 +
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 85 ++++++++++++++++++++++++++++-
5 files changed, 216 insertions(+), 3 deletions(-)
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
index aacb4177d914..a5227eb546f4 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
@@ -1113,6 +1113,100 @@ int mtk_cldma_exit(struct mtk_ctrl_trans *trans)
return 0;
}
+void mtk_cldma_pm_suspend(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_trans *trans = ((struct mtk_ctrl_blk *)mdev->ctrl_blk)->ctrl_hw_priv;
+ struct cldma_dev *cd = trans->dev;
+ struct cldma_drv_info *drv_info;
+ struct cldma_drv_ops *drv_ops;
+ struct rxq *rxq;
+ struct txq *txq;
+ int i, q;
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ drv_info = cd->cldma_drv_info[i];
+ if (!drv_info)
+ continue;
+
+ drv_ops = drv_info->drv_ops;
+
+ /* Stop TX queues and flush pending tx_done_work (suspend phase) */
+ drv_ops->cldma_stop_queue(drv_info, DIR_TX, ALLQ);
+ for (q = 0; q < HW_QUEUE_NUM; q++) {
+ txq = drv_info->txq[q];
+ if (txq)
+ flush_work(&txq->tx_done_work);
+ }
+
+ for (q = 0; q < HW_QUEUE_NUM; q++) {
+ rxq = drv_info->rxq[q];
+ if (!rxq)
+ continue;
+ atomic_set(&rxq->need_exit, 1);
+ drv_ops->cldma_stop_queue(drv_info, DIR_RX, q);
+ flush_work(&rxq->rx_done_work);
+ }
+ mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id);
+ }
+}
+
+void mtk_cldma_pm_resume_early(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_trans *trans = ((struct mtk_ctrl_blk *)mdev->ctrl_blk)->ctrl_hw_priv;
+ struct cldma_dev *cd = trans->dev;
+ struct cldma_drv_info *drv_info;
+ struct cldma_drv_ops *drv_ops;
+ struct rxq *rxq;
+ int i, q;
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ drv_info = cd->cldma_drv_info[i];
+ if (!drv_info)
+ continue;
+
+ drv_ops = drv_info->drv_ops;
+
+ /* Resume RX queues from current HW ring position (no addr reset) */
+ for (q = 0; q < HW_QUEUE_NUM; q++) {
+ rxq = drv_info->rxq[q];
+ if (!rxq)
+ continue;
+ atomic_set(&rxq->need_exit, 0);
+ drv_ops->cldma_resume_queue(drv_info, DIR_RX, q);
+ }
+
+ /* Unmask CLDMA L1 interrupt */
+ mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id);
+ }
+}
+
+void mtk_cldma_pm_resume(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_trans *trans = ((struct mtk_ctrl_blk *)mdev->ctrl_blk)->ctrl_hw_priv;
+ struct cldma_dev *cd = trans->dev;
+ struct cldma_drv_info *drv_info;
+ struct cldma_drv_ops *drv_ops;
+ struct txq *txq;
+ int i, q;
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ drv_info = cd->cldma_drv_info[i];
+ if (!drv_info)
+ continue;
+
+ drv_ops = drv_info->drv_ops;
+
+ /* Restart TX queues that have pending descriptors */
+ for (q = 0; q < HW_QUEUE_NUM; q++) {
+ txq = drv_info->txq[q];
+ if (!txq)
+ continue;
+ if (atomic_read(&txq->req_budget) < txq->nr_gpds)
+ mtk_cldma_start_xfer(drv_info, q);
+ }
+ }
+}
+
static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb)
{
struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
index 04f83ff0e37d..fd39985f75e7 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
@@ -163,6 +163,9 @@ int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno);
int mtk_cldma_trb_process(void *dev, struct sk_buff *skb);
void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_ctrl_trans *trans);
int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que);
+void mtk_cldma_pm_suspend(struct mtk_md_dev *mdev);
+void mtk_cldma_pm_resume_early(struct mtk_md_dev *mdev);
+void mtk_cldma_pm_resume(struct mtk_md_dev *mdev);
#define drv_ops_name(NAME) cldma_drv_ops_##NAME
#define cldma_regs_name(NAME) mtk_cldma_regs_##NAME
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
index 43803587bfc3..63273a85e532 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
@@ -205,6 +205,7 @@ struct mtk_dpmaif_ctlb {
struct dpmaif_irq_param *irq_params;
bool dpmaif_sw_reset;
+ bool trans_enabled;
unsigned char rxq_cnt;
unsigned char txq_cnt;
};
@@ -1687,10 +1688,16 @@ static void mtk_dpmaif_trans_disable(struct mtk_dpmaif_ctlb *dcb)
static void mtk_dpmaif_trans_ctl(struct mtk_dpmaif_ctlb *dcb, bool enable)
{
if (enable) {
- if (dcb->dpmaif_state == DPMAIF_STATE_PWRON)
+ if (!dcb->trans_enabled &&
+ dcb->dpmaif_state == DPMAIF_STATE_PWRON) {
+ dcb->trans_enabled = true;
mtk_dpmaif_trans_enable(dcb);
+ }
} else {
- mtk_dpmaif_trans_disable(dcb);
+ if (dcb->trans_enabled) {
+ dcb->trans_enabled = false;
+ mtk_dpmaif_trans_disable(dcb);
+ }
}
}
@@ -2060,6 +2067,30 @@ static int mtk_dpmaif_stop(struct mtk_md_dev *mdev)
return 0;
}
+void mtk_dpmaif_pm_suspend(struct mtk_md_dev *mdev)
+{
+ struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
+
+ if (!dcb)
+ return;
+
+ mutex_lock(&dcb->trans_ctl_lock);
+ mtk_dpmaif_trans_ctl(dcb, false);
+ mutex_unlock(&dcb->trans_ctl_lock);
+}
+
+void mtk_dpmaif_pm_resume(struct mtk_md_dev *mdev)
+{
+ struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
+
+ if (!dcb)
+ return;
+
+ mutex_lock(&dcb->trans_ctl_lock);
+ mtk_dpmaif_trans_ctl(dcb, true);
+ mutex_unlock(&dcb->trans_ctl_lock);
+}
+
static void mtk_dpmaif_clear(struct mtk_md_dev *mdev)
{
struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h
index e7e2f333141c..20fd53fd44b5 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h
@@ -10,5 +10,7 @@
int mtk_pcie_data_init(struct mtk_md_dev *mdev);
int mtk_pcie_data_exit(struct mtk_md_dev *mdev);
+void mtk_dpmaif_pm_suspend(struct mtk_md_dev *mdev);
+void mtk_dpmaif_pm_resume(struct mtk_md_dev *mdev);
#endif /* __MTK_DPMAIF_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index baac3692f1e3..f659c9a7aa96 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -11,8 +11,10 @@
#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/kernel.h>
+#include <linux/kthread.h>
#include <linux/module.h>
+#include "mtk_cldma.h"
#include "mtk_dev.h"
#include "mtk_dpmaif.h"
#include "mtk_trans_ctrl.h"
@@ -199,7 +201,6 @@ int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id,
}
priv->irq_cb_list[irq_id] = irq_cb;
priv->irq_cb_data[irq_id] = data;
-
return 0;
}
@@ -970,11 +971,93 @@ static const struct pci_error_handlers mtk_pci_err_handler = {
.error_detected = mtk_pci_error_detected,
};
+static void mtk_pci_pm_trb_park(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_trans *trans = ((struct mtk_ctrl_blk *)mdev->ctrl_blk)->ctrl_hw_priv;
+ int i;
+
+ for (i = 0; i < trans->trb_srv_num; i++)
+ kthread_park(trans->trb_srv[i]->trb_thread);
+}
+
+static void mtk_pci_pm_trb_unpark(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_trans *trans = ((struct mtk_ctrl_blk *)mdev->ctrl_blk)->ctrl_hw_priv;
+ int i;
+
+ for (i = 0; i < trans->trb_srv_num; i++)
+ kthread_unpark(trans->trb_srv[i]->trb_thread);
+}
+
+static int __maybe_unused mtk_pci_pm_suspend(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ mtk_pci_pm_trb_park(mdev);
+
+ mtk_cldma_pm_suspend(mdev);
+
+ mtk_dpmaif_pm_suspend(mdev);
+
+ /* Mask MHCCIF interrupt */
+ mtk_pci_mask_irq(mdev, priv->mhccif_irq_id);
+
+ /* Mask all MSI-X interrupts at the device level */
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX);
+
+ /* Save PCI configuration space */
+ pci_save_state(pdev);
+
+ return 0;
+}
+
+static int __maybe_unused mtk_pci_pm_resume(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int ret;
+
+ /* Restore PCIe configuration space (including MSI-X enable bits) */
+ pci_restore_state(pdev);
+
+ /* Re-enable bus mastering for DMA */
+ pci_set_master(pdev);
+
+ /* Restore ATR (address translation registers in MMIO BAR space) */
+ ret = priv->cfg->atr_init(mdev);
+ if (ret) {
+ dev_err(mdev->dev, "PM: failed to re-init ATR on resume\n");
+ return ret;
+ }
+
+ /* Unmask MHCCIF interrupt */
+ mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
+
+ mtk_cldma_pm_resume_early(mdev);
+
+ /* Restart CLDMA TX queues that have pending descriptors */
+ mtk_cldma_pm_resume(mdev);
+
+ mtk_dpmaif_pm_resume(mdev);
+
+ mtk_pci_pm_trb_unpark(mdev);
+
+ return 0;
+}
+
+static const struct dev_pm_ops mtk_pci_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(mtk_pci_pm_suspend, mtk_pci_pm_resume)
+};
+
static struct pci_driver mtk_pci_drv = {
.name = "mtk_pci_drv",
.id_table = t9xx_pci_table,
.probe = mtk_pci_probe,
.remove = mtk_pci_remove,
+ .driver.pm = &mtk_pci_pm_ops,
.err_handler = &mtk_pci_err_handler
};
--
2.34.1
^ permalink raw reply related
* [PATCH 07/11] net: wwan: t9xx: Introduce data plane hardware
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Data Plane Modem AP Interface (DPMAIF) hardware layer
provides hardware abstraction for the upper layer
(DPMAIF HIF). It implements functions to do the data plane
hardware's configuration, TX/RX control and interrupt
handling.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/pcie/Makefile | 5 +-
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c | 1586 ++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h | 259 ++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv_m9xx.c | 687 ++++++++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg.h | 387 ++++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg_m9xx.h | 37 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c | 168 +++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.h | 161 +++
drivers/net/wwan/t9xx/pcie/mtk_pci.h | 1 +
drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c | 1 +
10 files changed, 3291 insertions(+), 1 deletion(-)
diff --git a/drivers/net/wwan/t9xx/pcie/Makefile b/drivers/net/wwan/t9xx/pcie/Makefile
index 5252f158b058..78a765f8e79a 100644
--- a/drivers/net/wwan/t9xx/pcie/Makefile
+++ b/drivers/net/wwan/t9xx/pcie/Makefile
@@ -7,9 +7,12 @@ obj-$(CONFIG_MTK_T9XX_PCI) += mtk_t9xx_pcie.o
mtk_t9xx_pcie-y := \
mtk_pci_drv_m9xx.o \
+ mtk_dpmaif_drv_m9xx.o \
mtk_cldma_drv_m9xx.o \
mtk_ctrl_cfg_m9xx.o \
mtk_pci.o \
mtk_trans_ctrl.o \
mtk_cldma.o \
- mtk_cldma_drv.o
+ mtk_dpmaif_drv.o \
+ mtk_cldma_drv.o \
+ mtk_dpmaif_ring.o
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
new file mode 100644
index 000000000000..3fd2b33d1199
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
@@ -0,0 +1,1586 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#define pr_fmt(fmt) "DATA_DRV:" fmt
+
+#include <linux/delay.h>
+
+#include "mtk_dev.h"
+#include "mtk_dpmaif_drv.h"
+#include "mtk_dpmaif_reg.h"
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+
+static int mtk_dpmaif_drv_dl_add_bat_cnt(struct dpmaif_drv_info *drv_info,
+ u8 bat_id, u32 bat_entry_cnt);
+
+u32 mtk_dpmaif_drv_get_ul_intr_mask(struct dpmaif_drv_info *drv_info)
+{
+ if (drv_info->priv_ops && drv_info->priv_ops->get_ul_intr_mask)
+ return drv_info->priv_ops->get_ul_intr_mask(drv_info);
+
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0);
+}
+
+u32 mtk_dpmaif_drv_get_dl_intr_mask(struct dpmaif_drv_info *drv_info, u8 q_id)
+{
+ if (q_id != DPMAIF_DLQ2)
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base +
+ DPMAIF_PD_AP_DL_L2TIMR0);
+ else
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_RW);
+}
+
+static int mtk_dpmaif_drv_init_mode(struct dpmaif_drv_info *drv_info)
+{
+ u32 val, cnt = 0;
+ int ret;
+
+ /* Initialize dpmaif sram. */
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_MEM_CLR);
+ val |= DPMAIF_MEM_CLR_MASK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_MEM_CLR, val);
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_MEM_CLR) &
+ DPMAIF_MEM_CLR_MASK))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev, "Failed to initialize sram\n");
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ if (drv_info->priv_ops && drv_info->priv_ops->dynamic_sram_init) {
+ ret = drv_info->priv_ops->dynamic_sram_init(drv_info);
+ if (unlikely(ret < 0))
+ return ret;
+ }
+
+ /* Set DPMAIF AP port mode. */
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES);
+ val &= ~DPMAIF_PORT_MODE_MSK;
+ val |= DPMAIF_PORT_MODE_PCIE;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES, val);
+
+ /* Set CG enable. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_CG_EN, 0x7F);
+
+ /* Config SW PCIe mode. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_UL_RESERVE_AO_RW,
+ DPMAIF_PCIE_MODE_SET_VALUE);
+ if (drv_info->priv_ops && drv_info->priv_ops->set_pcie_domain)
+ drv_info->priv_ops->set_pcie_domain(drv_info);
+
+ return 0;
+}
+
+int mtk_dpmaif_drv_ul_intr_init(struct dpmaif_drv_info *drv_info)
+{
+ const struct dpmaif_intr_cfg *intr_cfg = &drv_info->cfg->intr_cfg;
+ u32 cnt = 0;
+
+ /* clear UL interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISAR0,
+ 0xFFFFFFFF);
+ /* unmask ul_l2intrs_enable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TICR0,
+ intr_cfg->ul_l2intrs_enable);
+ /* mask ul_l2intrs_disable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TISR0,
+ intr_cfg->ul_l2intrs_disable);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TISR0);
+ do {
+ if (((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0) &
+ intr_cfg->ul_l2intrs_disable) ==
+ intr_cfg->ul_l2intrs_disable))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to set UL interrupt mask, mask=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0));
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_init_intr(struct dpmaif_drv_info *drv_info)
+{
+ const struct dpmaif_intr_cfg *intr_cfg = &drv_info->cfg->intr_cfg;
+ u32 cnt = 0, cfg;
+ int ret;
+
+ if (drv_info->priv_ops && drv_info->priv_ops->ul_intr_init)
+ ret = drv_info->priv_ops->ul_intr_init(drv_info);
+ else
+ ret = mtk_dpmaif_drv_ul_intr_init(drv_info);
+
+ if (unlikely(ret < 0))
+ return ret;
+
+ /* clear DLQ0/1 interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_DL_L2TISAR0,
+ 0xFFFFFFFF);
+ /* unmask dl_l2intrs_enable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TICR0,
+ intr_cfg->dl_l2intrs_enable);
+ /* mask dl_l2intrs_disable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TISR0,
+ intr_cfg->dl_l2intrs_disable);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TISR0);
+
+ do {
+ if (((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TIMR0) &
+ intr_cfg->dl_l2intrs_disable) ==
+ intr_cfg->dl_l2intrs_disable))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to set DL interrupt mask, mask=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TIMR0));
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ if (drv_info->cfg->rx_cfg.rxq_cnt >= DPMAIF_DLQ2 + 1) {
+ /* clear DLQ2 interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AP_MISC_APDL_L2TISAR1, 0xFFFFFFFF);
+ /* unmask dl2_l2intrs_enable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_CLR,
+ intr_cfg->dl2_l2intrs_enable);
+ /* mask dl2_l2intrs_disable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET, intr_cfg->dl2_l2intrs_disable);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET);
+ cnt = 0;
+ do {
+ if (((mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_RW) &
+ intr_cfg->dl2_l2intrs_disable) ==
+ intr_cfg->dl2_l2intrs_disable))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to set DL2 interrupt mask, mask=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_RW));
+ return -DATA_HW_REG_TIMEOUT;
+ }
+ }
+
+ /* init IP busy */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_IP_BUSY, 0xFFFFFFFF);
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DLUL_IP_BUSY_MASK,
+ intr_cfg->udl_ip_busy_disable);
+
+ /* init HPC */
+ cfg = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_AP_L1TIMR0);
+ cfg |= intr_cfg->hpc_disable;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_AP_L1TIMR0,
+ cfg);
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_HPC_INTR_MASK,
+ 0xFFFF);
+
+ return 0;
+}
+
+static void mtk_dpmaif_drv_set_hpc_cntl(struct dpmaif_drv_info *drv_info)
+{
+ u32 cfg;
+
+ if (drv_info->priv_ops && drv_info->priv_ops->set_hpc_cntl) {
+ drv_info->priv_ops->set_hpc_cntl(drv_info);
+ return;
+ }
+
+ cfg = DPMAIF_HPC_LRO_PATH_DF & 0x3;
+ cfg |= (DPMAIF_HPC_ADD_MODE_DF & 0x3) << 2;
+ cfg |= (DPMAIF_HASH_PRIME_DF & 0xF) << 4;
+ cfg |= (DPMAIF_HPC_NUM_DF & 0xFF) << 8;
+
+ /* Configuration include hpc dlq path, hpc add mode, hash prime, hpc total number. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AO_DL_HPC_CNTL, cfg);
+}
+
+static void mtk_dpmaif_drv_set_agg_cfg(struct dpmaif_drv_info *drv_info, bool enable)
+{
+ u32 cfg;
+
+ cfg = DPMAIF_AGG_MAX_LEN_DF & 0xFFFF;
+ cfg |= (DPMAIF_AGG_TBL_ENT_NUM_DF & 0xFFFF) << 16;
+
+ /* Configuration include agg max length, agg table number. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AO_DL_LRO_AGG_CFG,
+ cfg);
+ cfg = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_AO_DL_RDY_CHK_FRG_THRES);
+ if (enable)
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AO_DL_RDY_CHK_FRG_THRES, cfg | (0xFF << 20));
+ else
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AO_DL_RDY_CHK_FRG_THRES, cfg & 0xF00FFFFF);
+}
+
+static int mtk_dpmaif_drv_init_features(struct dpmaif_drv_info *drv_info)
+{
+ if (drv_info->cfg->cap & DATA_HW_F_HPC)
+ mtk_dpmaif_drv_set_hpc_cntl(drv_info);
+
+ /* Explicitly disable LRO aggregation at init time. */
+ mtk_dpmaif_drv_set_agg_cfg(drv_info, false);
+
+ return 0;
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_remain_minsz(struct dpmaif_drv_info *drv_info,
+ u32 sz)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CONO);
+ val &= ~DPMAIF_BAT_REMAIN_MINSZ_MSK;
+ val |= ((sz / DPMAIF_BAT_REMAIN_SZ_BASE) << 8) & DPMAIF_BAT_REMAIN_MINSZ_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CONO, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_bat_bufsz(struct dpmaif_drv_info *drv_info,
+ u32 buf_sz)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON2);
+ val &= ~DPMAIF_BAT_BUF_SZ_MSK;
+ val |= ((buf_sz / DPMAIF_BAT_BUFFER_SZ_BASE) << 8) & DPMAIF_BAT_BUF_SZ_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON2, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_bat_rsv_length(struct dpmaif_drv_info *drv_info,
+ u32 length)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON2);
+ val &= ~DPMAIF_BAT_RSV_LEN_MSK;
+ val |= length & DPMAIF_BAT_RSV_LEN_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON2, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_bid_maxcnt(struct dpmaif_drv_info *drv_info, u32 cnt)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CONO);
+ val &= ~DPMAIF_BAT_BID_MAXCNT_MSK;
+ val |= (cnt << 16) & DPMAIF_BAT_BID_MAXCNT_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CONO, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_pkt_alignment(struct dpmaif_drv_info *drv_info,
+ bool enable, u32 mode)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES);
+ val &= ~DPMAIF_PKT_ALIGN_MSK;
+ if (enable) {
+ val |= DPMAIF_PKT_ALIGN_EN;
+ val |= (mode << 22) & DPMAIF_PKT_ALIGN_MSK;
+ }
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_pit_seqnum(struct dpmaif_drv_info *drv_info, u32 seq)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_AO_DL_PIT_SEQ_END);
+ val &= ~DPMAIF_DL_PIT_SEQ_MSK;
+ val |= seq & DPMAIF_DL_PIT_SEQ_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AO_DL_PIT_SEQ_END,
+ val);
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_mtu(struct dpmaif_drv_info *drv_info, u32 mtu_sz)
+{
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON1,
+ mtu_sz);
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_pit_chknum(struct dpmaif_drv_info *drv_info,
+ u32 number)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON2);
+ val &= ~DPMAIF_PIT_CHK_NUM_MSK;
+ val |= (number << 24) & DPMAIF_PIT_CHK_NUM_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PKTINFO_CON2, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_ao_bat_check_threshold(struct dpmaif_drv_info *drv_info,
+ u32 size)
+{
+ u32 val;
+
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES);
+ val &= ~DPMAIF_BAT_CHECK_THRES_MSK;
+ val |= (size << 16) & DPMAIF_BAT_CHECK_THRES_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES, val);
+}
+
+static void mtk_dpmaif_drv_dl_set_bat_base_addr(struct dpmaif_drv_info *drv_info,
+ u64 addr, u8 bat_id)
+{
+ u32 lb_addr = (u32)(addr & 0xFFFFFFFF);
+ u32 hb_addr = (u32)(addr >> 32);
+ u64 addr1, addr2;
+
+ if (bat_id == DPMAIF_BAT0) {
+ addr1 = drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT_CON0;
+ addr2 = drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT_CON3;
+ } else {
+ addr1 = drv_info->regs->pd2_base + DPMAIF_DL_2_BAT_INIT_CON0;
+ addr2 = drv_info->regs->pd2_base + DPMAIF_DL_2_BAT_INIT_CON3;
+ }
+
+ mtk_pci_write32(drv_info->mdev, addr1, lb_addr);
+ mtk_pci_write32(drv_info->mdev, addr2, hb_addr);
+}
+
+static void mtk_dpmaif_drv_dl_set_bat_size(struct dpmaif_drv_info *drv_info, u32 size, u8 bat_id)
+{
+ u64 addr;
+ u32 val;
+
+ if (bat_id == DPMAIF_BAT0)
+ addr = drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT_CON1;
+ else
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_BAT_INIT_CON1;
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+ val &= ~DPMAIF_BAT_SIZE_MSK;
+ val |= size & DPMAIF_BAT_SIZE_MSK;
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+static void mtk_dpmaif_drv_dl_bat_en(struct dpmaif_drv_info *drv_info, bool enable, u8 bat_id)
+{
+ u64 addr;
+ u32 val;
+
+ if (bat_id == DPMAIF_BAT0)
+ addr = drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT_CON1;
+ else
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_BAT_INIT_CON1;
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+ if (enable)
+ val |= DPMAIF_BAT_EN_MSK;
+ else
+ val &= ~DPMAIF_BAT_EN_MSK;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+ mtk_pci_read32(drv_info->mdev, addr);
+}
+
+static int mtk_dpmaif_drv_dl_bat_init_done(struct dpmaif_drv_info *drv_info,
+ u8 bat_id, u32 init_mode)
+{
+ u32 cnt = 0, dl_bat_init;
+ u64 addr;
+
+ dl_bat_init = init_mode;
+ dl_bat_init |= DPMAIF_DL_BAT_INIT_EN;
+
+ if (bat_id == DPMAIF_BAT0)
+ addr = drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT;
+ else
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_BAT_INIT;
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_BAT_INIT_NOT_READY)) {
+ mtk_pci_write32(drv_info->mdev, addr, dl_bat_init);
+ break;
+ }
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to initialize bat,init_mode=%u\n", init_mode);
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ cnt = 0;
+ do {
+ if (!((mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_BAT_INIT_NOT_READY)))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev, "Failed to initialize bat done,init_mode=%u\n", init_mode);
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static void mtk_dpmaif_drv_dl_set_pit_base_addr(struct dpmaif_drv_info *drv_info, u64 addr, u8 q_id)
+{
+ u32 lb_addr = (u32)(addr & 0xFFFFFFFF);
+ u32 hb_addr = (u32)(addr >> 32);
+
+ if (q_id != DPMAIF_DLQ2) {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON0,
+ lb_addr);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON4,
+ hb_addr);
+ } else {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_INIT_CON0,
+ lb_addr);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_INIT_CON4,
+ hb_addr);
+ }
+}
+
+static void mtk_dpmaif_drv_dl_set_pit_size(struct dpmaif_drv_info *drv_info, u32 size, u8 q_id)
+{
+ u32 val;
+
+ if (q_id != DPMAIF_DLQ2) {
+ val = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base +
+ NRL2_DPMAIF_DL_LROPIT_INIT_CON1);
+ val &= ~DPMAIF_PIT_SIZE_MSK;
+ val |= size & DPMAIF_PIT_SIZE_MSK;
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON1,
+ val);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON2,
+ 0);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON3,
+ 0);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON5,
+ 0);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON6,
+ 0);
+ } else {
+ val = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_INIT_CON1);
+ val &= ~DPMAIF_PIT_SIZE_MSK;
+ val |= size & DPMAIF_PIT_SIZE_MSK;
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_INIT_CON1,
+ val);
+ }
+}
+
+static void mtk_dpmaif_drv_dl_pit_en(struct dpmaif_drv_info *drv_info, bool enable, u8 q_id)
+{
+ u32 val, addr, mask;
+
+ if (q_id != DPMAIF_DLQ2) {
+ addr = drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT_CON3;
+ mask = DPMAIF_LROPIT_EN_MSK;
+ } else {
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_INIT_CON3;
+ mask = DPMAIF_DL2_PIT_EN_MSK;
+ }
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+ if (enable)
+ val |= mask;
+ else
+ val &= ~mask;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+static int mtk_dpmaif_drv_dl_pit_init_done(struct dpmaif_drv_info *drv_info, u32 pit_idx)
+{
+ int cnt = 0, dl_pit_init;
+ u32 addr;
+
+ if (pit_idx != DPMAIF_DLQ2) {
+ dl_pit_init = DPMAIF_DL_PIT_INIT_ALLSET;
+ dl_pit_init |= pit_idx << DPMAIF_LROPIT_CHAN_OFS;
+ dl_pit_init |= DPMAIF_DL_PIT_INIT_EN;
+ addr = drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_INIT;
+ } else {
+ dl_pit_init = DPMAIF_DL_PIT_INIT_ALLSET;
+ dl_pit_init |= DPMAIF_DL_PIT_INIT_EN;
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_INIT;
+ }
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_PIT_INIT_NOT_READY)) {
+ mtk_pci_write32(drv_info->mdev, addr, dl_pit_init);
+ break;
+ }
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev, "Failed to initialize pit%u\n", pit_idx);
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ cnt = 0;
+ do {
+ if (!((mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_PIT_INIT_NOT_READY)))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev, "Failed to initialize pit%u done\n", pit_idx);
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static int mtk_dpmaif_drv_config_dlq_pit_hw(struct dpmaif_drv_info *drv_info, u8 q_num,
+ struct dpmaif_rxq_cfg *dlq)
+{
+ mtk_dpmaif_drv_dl_set_pit_base_addr(drv_info, (u64)dlq->pit_base, q_num);
+ mtk_dpmaif_drv_dl_set_pit_size(drv_info, dlq->pit_cnt, q_num);
+ mtk_dpmaif_drv_dl_pit_en(drv_info, true, q_num);
+
+ return mtk_dpmaif_drv_dl_pit_init_done(drv_info, q_num);
+}
+
+static int mtk_dpmaif_drv_dlq_all_en(struct dpmaif_drv_info *drv_info, bool enable)
+{
+ int ret;
+ u8 i;
+
+ for (i = 0; i < drv_info->cfg->rx_cfg.bat_ring_num; i++) {
+ mtk_dpmaif_drv_dl_bat_en(drv_info, enable, i);
+ ret = mtk_dpmaif_drv_dl_bat_init_done(drv_info, i,
+ DPMAIF_DL_BAT_INIT_ONLY_ENABLE_BIT);
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_init_dlq(struct dpmaif_drv_info *drv_info)
+{
+ struct dpmaif_rx_cfg *rx_cfg = &drv_info->cfg->rx_cfg;
+ u32 i, val;
+ int ret;
+
+ /* common dl init */
+ if (rx_cfg->pkt_alignment == 64)
+ mtk_dpmaif_drv_dl_set_pkt_alignment(drv_info, true, DPMAIF_PKT_ALIGN64_MODE);
+ else if (rx_cfg->pkt_alignment == 128)
+ mtk_dpmaif_drv_dl_set_pkt_alignment(drv_info, true, DPMAIF_PKT_ALIGN128_MODE);
+ else
+ mtk_dpmaif_drv_dl_set_pkt_alignment(drv_info, false, 0);
+
+ mtk_dpmaif_drv_dl_set_ao_mtu(drv_info, rx_cfg->mtu);
+ mtk_dpmaif_drv_dl_set_ao_remain_minsz(drv_info, DPMAIF_HW_BAT_REMAIN);
+ mtk_dpmaif_drv_dl_set_ao_bat_rsv_length(drv_info, rx_cfg->normal_bat_rsv_length);
+ mtk_dpmaif_drv_dl_set_ao_bid_maxcnt(drv_info, DPMAIF_HW_PKT_BIDCNT);
+ /* Bat cache enable. */
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT_CON1);
+ val |= DPMAIF_DL_BAT_CACHE_PRI;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_INIT_CON1, val);
+
+ /* Common initialize normal bat. */
+ mtk_dpmaif_drv_dl_set_ao_bat_check_threshold(drv_info, DPMAIF_HW_CHK_BAT_NUM);
+ mtk_dpmaif_drv_dl_set_ao_bat_bufsz(drv_info, rx_cfg->bats[0].buf_size);
+
+ /* Initialize BAT rings. */
+ for (i = 0; i < rx_cfg->bat_ring_num; i++) {
+ mtk_dpmaif_drv_dl_set_bat_base_addr(drv_info,
+ (u64)rx_cfg->bats[i].bat_base, i);
+ mtk_dpmaif_drv_dl_set_bat_size(drv_info, rx_cfg->bats[i].bat_cnt, i);
+ mtk_dpmaif_drv_dl_bat_en(drv_info, false, i);
+ ret = mtk_dpmaif_drv_dl_bat_init_done(drv_info, i, DPMAIF_DL_BAT_INIT_ALLSET);
+ if (ret < 0)
+ return ret;
+
+ ret = mtk_dpmaif_drv_dl_add_bat_cnt(drv_info, i, rx_cfg->bats[i].real_reload_cnt);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* Initialize pit information. */
+ /* Pit burst enable. */
+ val = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES);
+ val |= DPMAIF_DL_BURST_PIT_EN;
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_AO_DL_RDY_CHK_THRES, val);
+ mtk_dpmaif_drv_dl_set_ao_pit_chknum(drv_info, DPMAIF_HW_CHK_PIT_NUM);
+ /* Currently, use rxqs[0] to config. */
+ mtk_dpmaif_drv_dl_set_pit_seqnum(drv_info, rx_cfg->rxqs[0].pit_seq_max);
+
+ for (i = 0; i < rx_cfg->rxq_cnt; i++) {
+ ret = mtk_dpmaif_drv_config_dlq_pit_hw(drv_info, i, &rx_cfg->rxqs[i]);
+ if (ret < 0)
+ return ret;
+ }
+
+ ret = mtk_dpmaif_drv_dlq_all_en(drv_info, true);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static void mtk_dpmaif_drv_ul_update_drb_size(struct dpmaif_drv_info *drv_info,
+ u8 q_num, u32 size)
+{
+ u32 old_size;
+ u64 addr;
+
+ addr = drv_info->regs->pd_base + DPMAIF_PD_UL_CHNL0_CON1 + 0x10 * q_num;
+ old_size = mtk_pci_read32(drv_info->mdev, addr);
+ old_size &= ~DPMAIF_DRB_SIZE_MSK;
+ old_size |= size & DPMAIF_DRB_SIZE_MSK;
+ mtk_pci_write32(drv_info->mdev, addr, old_size);
+}
+
+static void mtk_dpmaif_drv_ul_update_drb_base_addr(struct dpmaif_drv_info *drv_info,
+ u8 q_num, u64 addr)
+{
+ u32 lb_addr = (u32)(addr & 0xFFFFFFFF);
+ u32 hb_addr = (u32)(addr >> 32);
+
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_UL_CHNL0_CON0 + 0x10 * q_num, lb_addr);
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_UL_CHNL0_CON2 + 0x10 * q_num, hb_addr);
+}
+
+static void mtk_dpmaif_drv_ul_rdy_en(struct dpmaif_drv_info *drv_info, u8 q_num, bool ready)
+{
+ u32 ul_rdy_en;
+
+ ul_rdy_en = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0);
+ if (ready)
+ ul_rdy_en |= BIT(q_num);
+ else
+ ul_rdy_en &= ~BIT(q_num);
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0,
+ ul_rdy_en);
+}
+
+static void mtk_dpmaif_drv_ul_arb_en(struct dpmaif_drv_info *drv_info, u8 q_num, bool enable)
+{
+ u32 ul_arb_en;
+
+ ul_arb_en = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0);
+ if (enable)
+ ul_arb_en |= BIT(q_num + 8);
+ else
+ ul_arb_en &= ~BIT(q_num + 8);
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0,
+ ul_arb_en);
+}
+
+static void mtk_dpmaif_drv_init_ulq(struct dpmaif_drv_info *drv_info)
+{
+ struct dpmaif_tx_cfg *tx_cfg = &drv_info->cfg->tx_cfg;
+ struct dpmaif_txq_cfg *ulq;
+ u32 i;
+
+ for (i = 0; i < tx_cfg->txq_cnt; i++) {
+ ulq = &tx_cfg->txqs[i];
+ mtk_dpmaif_drv_ul_update_drb_size(drv_info, i,
+ ulq->drb_cnt * DPMAIF_UL_DRB_ENTRY_WORD);
+ mtk_dpmaif_drv_ul_update_drb_base_addr(drv_info, i, (u64)ulq->drb_base);
+ mtk_dpmaif_drv_ul_rdy_en(drv_info, i, true);
+ mtk_dpmaif_drv_ul_arb_en(drv_info, i, true);
+ }
+}
+
+static int mtk_dpmaif_drv_init_done(struct dpmaif_drv_info *drv_info)
+{
+ u32 val, cnt = 0;
+
+ /* Sync default value to SRAM. */
+ val = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_OVERWRITE_CFG);
+ val |= DPMAIF_SRAM_SYNC_MASK;
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_OVERWRITE_CFG, val);
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + NRL2_DPMAIF_AP_MISC_OVERWRITE_CFG) &
+ DPMAIF_SRAM_SYNC_MASK))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev, "Failed to sync default value to sram\n");
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ /* UL configure done. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_INIT_SET,
+ DPMAIF_UL_INIT_DONE_MASK);
+ drv_info->cfg->tx_cfg.txq_all_enable = true;
+
+ /* DL configure done. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + NRL2_DPMAIF_AO_DL_INIT_SET,
+ DPMAIF_DL_INIT_DONE_MASK);
+ drv_info->cfg->rx_cfg.rxq_all_enable = true;
+
+ /* clear dummy interrupts */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISAR0,
+ 0xFFFFFFFF);
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_DL_L2TISAR0,
+ 0xFFFFFFFF);
+
+ return 0;
+}
+
+static void mtk_dpmaif_drv_ulq_all_en(struct dpmaif_drv_info *drv_info, bool enable)
+{
+ u32 ul_arb_en;
+
+ ul_arb_en = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0);
+ if (enable)
+ ul_arb_en |= DPMAIF_UL_ALL_QUE_ARB_EN;
+ else
+ ul_arb_en &= ~DPMAIF_UL_ALL_QUE_ARB_EN;
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0,
+ ul_arb_en);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_UL_CHNL_ARB0);
+}
+
+static bool mtk_dpmaif_drv_ul_all_idle_check(struct dpmaif_drv_info *drv_info)
+{
+ bool is_idle = false;
+ u32 ul_dbg_sta;
+
+ ul_dbg_sta = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_UL_DBG_STA2);
+ if ((ul_dbg_sta & DPMAIF_UL_IDLE_STS_MSK) == DPMAIF_UL_IDLE_STS)
+ is_idle = true;
+
+ return is_idle;
+}
+
+static int mtk_dpmaif_drv_stop_ulq(struct dpmaif_drv_info *drv_info)
+{
+ int cnt = 0;
+
+ /* Disable HW arb and check idle. */
+ mtk_dpmaif_drv_ulq_all_en(drv_info, false);
+
+ do {
+ if (mtk_dpmaif_drv_ul_all_idle_check(drv_info))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to stop ul queue, sta=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_UL_DBG_STA2));
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static bool mtk_dpmaif_drv_dl_is_idle(struct dpmaif_drv_info *drv_info)
+{
+ bool is_idle = false;
+ u32 dl_dbg_sta;
+
+ dl_dbg_sta = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_DL_DBG_STA1);
+ if ((dl_dbg_sta & DPMAIF_DL_IDLE_STS) == DPMAIF_DL_IDLE_STS)
+ is_idle = true;
+
+ return is_idle;
+}
+
+static u32 mtk_dpmaif_drv_dl_get_wridx(struct dpmaif_drv_info *drv_info)
+{
+ return ((mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PIT_STA3)) &
+ DPMAIF_DL_PIT_WRIDX_MSK);
+}
+
+static u32 mtk_dpmaif_drv_dl_get_pit_ridx(struct dpmaif_drv_info *drv_info)
+{
+ return ((mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_AO_DL_PIT_STA2)) &
+ DPMAIF_DL_PIT_WRIDX_MSK);
+}
+
+static int mtk_dpmaif_drv_stop_dlq(struct dpmaif_drv_info *drv_info)
+{
+ u32 cnt = 0, wridx, ridx;
+ int ret;
+
+ ret = mtk_dpmaif_drv_dlq_all_en(drv_info, false);
+ if (ret < 0)
+ return ret;
+
+ /* check idle */
+ do {
+ if (mtk_dpmaif_drv_dl_is_idle(drv_info))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to stop dl queue, sta=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_DL_DBG_STA1));
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ /* check middle pit sync done. */
+ cnt = 0;
+ do {
+ wridx = mtk_dpmaif_drv_dl_get_wridx(drv_info);
+ ridx = mtk_dpmaif_drv_dl_get_pit_ridx(drv_info);
+ if (wridx == ridx)
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev, "Failed to check middle pit sync\n");
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+u32 mtk_dpmaif_drv_get_dl_lv2_sts(struct dpmaif_drv_info *drv_info, u8 q_id)
+{
+ if (q_id != DPMAIF_DLQ2)
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ DPMAIF_PD_AP_DL_L2TISAR0);
+ else
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AP_MISC_APDL_L2TISAR1);
+}
+
+u32 mtk_dpmaif_drv_get_ul_lv2_sts(struct dpmaif_drv_info *drv_info)
+{
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISAR0);
+}
+
+static int mtk_dpmaif_drv_mask_ulq_intr(struct dpmaif_drv_info *drv_info, u32 q_num)
+{
+ u32 cnt = 0, ui_que_done_mask;
+
+ if (drv_info->priv_ops && drv_info->priv_ops->mask_ulq_intr)
+ return drv_info->priv_ops->mask_ulq_intr(drv_info, q_num);
+
+ ui_que_done_mask = BIT(q_num + DP_UL_INT_DONE_OFFSET) & DPMAIF_UL_INT_QDONE_MSK;
+
+ do {
+ if (!(cnt++ % REWRITE_TIMES)) {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TISR0,
+ ui_que_done_mask);
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TISR0);
+ }
+
+ if ((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0) &
+ ui_que_done_mask)) {
+ return 0;
+ }
+
+ udelay(POLL_INTERVAL_US);
+ } while (cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to mask ulq%u interrupt done, sta=0x%08x\n",
+ q_num, mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0));
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static int mtk_dpmaif_drv_ul_mask_all_tx_done_intr(struct dpmaif_drv_info *drv_info)
+{
+ int ret = 0;
+ u8 i;
+
+ for (i = 0; i < drv_info->cfg->tx_cfg.txq_cnt; i++) {
+ ret = mtk_dpmaif_drv_mask_ulq_intr(drv_info, i);
+ if (ret < 0)
+ break;
+ }
+
+ return ret;
+}
+
+void mtk_dpmaif_drv_ul_mask_multi_tx_done_intr(struct dpmaif_drv_info *drv_info,
+ u8 q_mask)
+{
+ u32 i;
+
+ for (i = 0; i < drv_info->cfg->tx_cfg.txq_cnt; i++) {
+ if (q_mask & BIT(i))
+ mtk_dpmaif_drv_mask_ulq_intr(drv_info, i);
+ }
+}
+
+static void mtk_dpmaif_drv_unmask_ulq_intr(struct dpmaif_drv_info *drv_info, u32 q_num)
+{
+ u32 ui_que_done_mask;
+
+ if (drv_info->priv_ops && drv_info->priv_ops->unmask_ulq_intr) {
+ drv_info->priv_ops->unmask_ulq_intr(drv_info, q_num);
+ return;
+ }
+
+ ui_que_done_mask = BIT(q_num + DP_UL_INT_DONE_OFFSET) & DPMAIF_UL_INT_QDONE_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TICR0,
+ ui_que_done_mask);
+}
+
+static void mtk_dpmaif_drv_ul_unmask_all_tx_done_intr(struct dpmaif_drv_info *drv_info)
+{
+ u8 i;
+
+ for (i = 0; i < drv_info->cfg->tx_cfg.txq_cnt; i++)
+ mtk_dpmaif_drv_unmask_ulq_intr(drv_info, i);
+}
+
+static void mtk_dpmaif_drv_clr_ul_done_status(struct dpmaif_drv_info *drv_info, u8 qno)
+{
+ u32 val, l2tisar0;
+
+ /* get TX interrupt status. */
+ l2tisar0 = mtk_dpmaif_drv_get_ul_lv2_sts(drv_info);
+ val = l2tisar0 & DPMAIF_UL_INT_QDONE & BIT(DP_UL_INT_DONE_OFFSET + qno);
+
+ /* ulq status. */
+ if (val) {
+ /* clear ulq done status */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISAR0,
+ val);
+ }
+}
+
+void mtk_dpmaif_drv_mask_dl_batcnt_len_err_intr(struct dpmaif_drv_info *drv_info, u8 bat_id)
+{
+ if (bat_id == DPMAIF_BAT0) {
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TISR0,
+ DPMAIF_DL_INT_BATCNT_LEN_ERR_MSK);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TISR0);
+ } else {
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET, DPMAIF_DL2_INT_BATCNT_LEN_ERR_MSK);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET);
+ }
+}
+
+static void mtk_dpmaif_drv_unmask_dl_batcnt_len_err_intr(struct dpmaif_drv_info *drv_info,
+ u8 bat_id)
+{
+ if (bat_id == DPMAIF_BAT0)
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TICR0,
+ DPMAIF_DL_INT_BATCNT_LEN_ERR_MSK);
+ else
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_CLR,
+ DPMAIF_DL2_INT_BATCNT_LEN_ERR_MSK);
+}
+
+void mtk_dpmaif_drv_dlq_mask_pit_cnt_len_err_intr(struct dpmaif_drv_info *drv_info, u8 qno)
+{
+ if (qno == DPMAIF_DLQ0) {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_APDL_L2TIMSR0,
+ DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR_MSK);
+ } else if (qno == DPMAIF_DLQ1) {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_APDL_L2TIMSR0,
+ DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR_MSK);
+ } else {
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET,
+ DPMAIF_DL_INT_DLQ2_PITCNT_LEN_ERR_MSK);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET);
+ return;
+ }
+
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_APDL_L2TIMSR0);
+}
+
+static void
+ mtk_dpmaif_drv_dlq_unmask_pit_cnt_len_err_intr(struct dpmaif_drv_info *drv_info,
+ u8 qno)
+{
+ if (qno == DPMAIF_DLQ0) {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_APDL_L2TIMCR0,
+ DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR_MSK);
+ } else if (qno == DPMAIF_DLQ1) {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->ao_base + NRL2_DPMAIF_AO_UL_APDL_L2TIMCR0,
+ DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR_MSK);
+ } else {
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base +
+ NRL2_DPMAIF_MISC_PD_APDL12_MASK_CLR,
+ DPMAIF_DL_INT_DLQ2_PITCNT_LEN_ERR_MSK);
+ }
+}
+
+int mtk_dpmaif_drv_dlq_mask_rx_done_intr(struct dpmaif_drv_info *drv_info, u8 q_id)
+{
+ u64 mask_addr = drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TIMR0;
+ u64 addr = drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TISR0;
+ u32 cnt = 0, di_que_done_mask;
+
+ if (q_id == DPMAIF_DLQ0) {
+ di_que_done_mask = DPMAIF_DL_INT_DLQ0_QDONE_MSK;
+ } else if (q_id == DPMAIF_DLQ1) {
+ di_que_done_mask = DPMAIF_DL_INT_DLQ1_QDONE_MSK;
+ } else {
+ di_que_done_mask = DPMAIF_DL_INT_DLQ2_QDONE_MSK;
+ addr = drv_info->regs->pd_base + NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET;
+ mask_addr = drv_info->regs->pd_base + NRL2_DPMAIF_MISC_PD_APDL12_MASK_RW;
+ }
+
+ /* Check mask status. */
+ do {
+ if (!(cnt++ % REWRITE_TIMES)) {
+ mtk_pci_write32(drv_info->mdev, addr, di_que_done_mask);
+ mtk_pci_read32(drv_info->mdev, addr);
+ }
+
+ if ((mtk_pci_read32(drv_info->mdev, mask_addr) & di_que_done_mask))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to mask dlq%u interrupt, sta=0x%08x\n",
+ q_id, mtk_pci_read32(drv_info->mdev, mask_addr));
+ WARN_ON_ONCE(true);
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static int mtk_dpmaif_drv_dl_mask_all_rx_done_intr(struct dpmaif_drv_info *drv_info)
+{
+ int ret = 0;
+ u8 i;
+
+ for (i = 0; i < drv_info->cfg->rx_cfg.rxq_cnt; i++) {
+ ret = mtk_dpmaif_drv_dlq_mask_rx_done_intr(drv_info, i);
+ if (ret < 0)
+ break;
+ }
+
+ return ret;
+}
+
+static void mtk_dpmaif_drv_dl_unmask_rx_done_intr(struct dpmaif_drv_info *drv_info, u8 qno)
+{
+ u64 addr = drv_info->regs->ao_base + DPMAIF_PD_AP_DL_L2TICR0;
+ u32 di_que_done_mask;
+
+ if (qno == DPMAIF_DLQ0) {
+ di_que_done_mask = DPMAIF_DL_INT_DLQ0_QDONE_MSK;
+ } else if (qno == DPMAIF_DLQ1) {
+ di_que_done_mask = DPMAIF_DL_INT_DLQ1_QDONE_MSK;
+ } else {
+ di_que_done_mask = DPMAIF_DL2_INT_DLQ2_QDONE;
+ addr = drv_info->regs->pd_base + NRL2_DPMAIF_MISC_PD_APDL12_MASK_CLR;
+ }
+
+ mtk_pci_write32(drv_info->mdev, addr, di_que_done_mask);
+ mtk_pci_read32(drv_info->mdev, addr);
+}
+
+static void mtk_dpmaif_drv_dl_unmask_all_rx_done_intr(struct dpmaif_drv_info *drv_info)
+{
+ u8 i;
+
+ for (i = 0; i < drv_info->cfg->rx_cfg.rxq_cnt; i++)
+ mtk_dpmaif_drv_dl_unmask_rx_done_intr(drv_info, i);
+}
+
+static int mtk_dpmaif_drv_dl_add_pit_cnt(struct dpmaif_drv_info *drv_info, u32 qno,
+ u32 pit_remain_cnt)
+{
+ u32 cnt = 0, dl_update, addr;
+
+ dl_update = pit_remain_cnt & 0x3FFFF;
+
+ if (qno != DPMAIF_DLQ2) {
+ dl_update |= DPMAIF_DL_ADD_UPDATE | (qno << DPMAIF_ADD_LRO_PIT_CHAN_OFS);
+ addr = drv_info->regs->pd_base + NRL2_DPMAIF_DL_LROPIT_ADD;
+ } else {
+ dl_update |= DPMAIF_DL_ADD_UPDATE;
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_PIT_ADD;
+ }
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_ADD_NOT_READY)) {
+ mtk_pci_write32(drv_info->mdev, addr, dl_update);
+ break;
+ }
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to add dlq%u pit, cnt=%u\n", qno, pit_remain_cnt);
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ cnt = 0;
+ do {
+ if (!((mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_ADD_NOT_READY)))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to add dlq%u pit done, cnt=%u\n", qno, pit_remain_cnt);
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static int mtk_dpmaif_drv_dl_add_bat_cnt(struct dpmaif_drv_info *drv_info,
+ u8 bat_id, u32 bat_entry_cnt)
+{
+ u32 cnt = 0, dl_bat_update;
+ u64 addr;
+
+ dl_bat_update = bat_entry_cnt & 0xFFFF;
+ dl_bat_update |= DPMAIF_DL_ADD_UPDATE;
+
+ if (bat_id == DPMAIF_BAT0)
+ addr = drv_info->regs->pd_base + DPMAIF_PD_DL_BAT_ADD;
+ else
+ addr = drv_info->regs->pd2_base + DPMAIF_DL_2_BAT_ADD;
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_ADD_NOT_READY)) {
+ mtk_pci_write32(drv_info->mdev, addr, dl_bat_update);
+ break;
+ }
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev, "Failed to add bat, cnt=%u\n", bat_entry_cnt);
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ cnt = 0;
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_DL_ADD_NOT_READY))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev, "Failed to add bat done, cnt=%u\n", bat_entry_cnt);
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static int mtk_dpmaif_drv_ul_add_drb(struct dpmaif_drv_info *drv_info, u8 q_num, u32 drb_cnt)
+{
+ u32 drb_entry_cnt = drb_cnt * DPMAIF_UL_DRB_ENTRY_WORD;
+ u32 cnt = 0, ul_update;
+ u64 addr;
+
+ ul_update = drb_entry_cnt & 0xFFFF;
+ ul_update |= DPMAIF_UL_ADD_UPDATE;
+
+ if (q_num == 4)
+ addr = drv_info->regs->pd_base + NRL2_DPMAIF_UL_ADD_DESC_CH4;
+ else
+ addr = drv_info->regs->pd_base + DPMAIF_PD_UL_ADD_DESC_CH + 0x4 * q_num;
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_UL_ADD_NOT_READY)) {
+ mtk_pci_write32(drv_info->mdev, addr, ul_update);
+ break;
+ }
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev, "Failed to add ulq%u drb, cnt=%u\n", q_num, drb_cnt);
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ cnt = 0;
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, addr) & DPMAIF_UL_ADD_NOT_READY))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev, "Failed to add ulq%u drb done, cnt=%u\n", q_num, drb_cnt);
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static int mtk_dpmaif_drv_dl_get_pit_wridx(struct dpmaif_drv_info *drv_info, u32 qno)
+{
+ u32 pit_wridx;
+
+ if (qno != DPMAIF_DLQ2) {
+ pit_wridx = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AO_DL_LRO_STA5 + qno * 0x20) & DPMAIF_DL_PIT_WRIDX_MSK;
+ } else {
+ pit_wridx = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd2_base +
+ DPMAIF_DL_2_STA13) & DPMAIF_DL_PIT_WRIDX_MSK;
+ }
+
+ if (unlikely(pit_wridx >= drv_info->cfg->rx_cfg.rxqs[qno].pit_cnt))
+ return -DATA_HW_REG_CHK_FAIL;
+
+ return pit_wridx;
+}
+
+static int mtk_dpmaif_drv_dl_get_pit_rdidx(struct dpmaif_drv_info *drv_info, u32 qno)
+{
+ u32 pit_rdidx;
+
+ if (qno != DPMAIF_DLQ2) {
+ pit_rdidx = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AO_DL_LRO_STA6 + qno * 0x20) & DPMAIF_DL_PIT_WRIDX_MSK;
+ } else {
+ pit_rdidx = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd2_base +
+ DPMAIF_DL_2_STA14) & DPMAIF_DL_PIT_WRIDX_MSK;
+ }
+
+ if (unlikely(pit_rdidx >= drv_info->cfg->rx_cfg.rxqs[qno].pit_cnt))
+ return -DATA_HW_REG_CHK_FAIL;
+
+ return pit_rdidx;
+}
+
+static int mtk_dpmaif_drv_dl_get_bat_ridx(struct dpmaif_drv_info *drv_info, u8 bat_id)
+{
+ u32 bat_ridx;
+
+ if (bat_id == DPMAIF_BAT0)
+ bat_ridx = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_AO_DL_BAT_STA2) &
+ DPMAIF_DL_BAT_WRIDX_MSK;
+ else
+ bat_ridx = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_2_STA4) &
+ DPMAIF_DL_2_BAT_WRIDX_MSK;
+
+ if (unlikely(bat_ridx >= drv_info->cfg->rx_cfg.bats[bat_id].bat_cnt))
+ return -DATA_HW_REG_CHK_FAIL;
+
+ return bat_ridx;
+}
+
+static int mtk_dpmaif_drv_dl_get_bat_wridx(struct dpmaif_drv_info *drv_info, u8 bat_id)
+{
+ u32 bat_wridx;
+
+ if (bat_id == DPMAIF_BAT0)
+ bat_wridx = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_AO_DL_BAT_STA3) &
+ DPMAIF_DL_BAT_WRIDX_MSK;
+ else
+ bat_wridx = (mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_2_STA4) >> 16) &
+ DPMAIF_DL_2_BAT_WRIDX_MSK;
+
+ if (unlikely(bat_wridx >= drv_info->cfg->rx_cfg.bats[bat_id].bat_cnt))
+ return -DATA_HW_REG_CHK_FAIL;
+
+ return bat_wridx;
+}
+
+static int mtk_dpmaif_drv_ul_get_drb_ridx(struct dpmaif_drv_info *drv_info, u8 qno)
+{
+ u32 drb_ridx;
+ u64 addr;
+
+ addr = drv_info->regs->pd_base + drv_info->regs->ao_ul_ch0_sta + 0x4 * qno;
+
+ drb_ridx = mtk_pci_read32(drv_info->mdev, addr) >> 16;
+ drb_ridx = drb_ridx / DPMAIF_UL_DRB_ENTRY_WORD;
+
+ if (unlikely(drb_ridx >= drv_info->cfg->tx_cfg.txqs[qno].drb_cnt))
+ return -DATA_HW_REG_CHK_FAIL;
+
+ return drb_ridx;
+}
+
+int mtk_dpmaif_drv_init_com(struct dpmaif_drv_info *drv_info, void *data)
+{
+ if (mtk_dpmaif_drv_init_mode(drv_info) < 0)
+ return -DATA_HW_REG_CHK_FAIL;
+
+ if (mtk_dpmaif_drv_init_intr(drv_info) < 0)
+ return -DATA_HW_REG_CHK_FAIL;
+
+ if (mtk_dpmaif_drv_init_features(drv_info) < 0)
+ return -DATA_HW_REG_CHK_FAIL;
+
+ if (mtk_dpmaif_drv_init_dlq(drv_info) < 0)
+ return -DATA_HW_REG_CHK_FAIL;
+
+ mtk_dpmaif_drv_init_ulq(drv_info);
+
+ if (mtk_dpmaif_drv_init_done(drv_info) < 0)
+ return -DATA_HW_REG_CHK_FAIL;
+
+ return 0;
+}
+
+int mtk_dpmaif_drv_start_queue(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_dir dir)
+{
+ int ret;
+
+ if (dir == DPMAIF_TX) {
+ if (unlikely(drv_info->cfg->tx_cfg.txq_all_enable))
+ return 0;
+
+ mtk_dpmaif_drv_ulq_all_en(drv_info, true);
+ mtk_dpmaif_drv_ul_unmask_all_tx_done_intr(drv_info);
+ drv_info->cfg->tx_cfg.txq_all_enable = true;
+ } else {
+ if (unlikely(drv_info->cfg->rx_cfg.rxq_all_enable))
+ return 0;
+
+ ret = mtk_dpmaif_drv_dlq_all_en(drv_info, true);
+ if (ret < 0)
+ return ret;
+
+ mtk_dpmaif_drv_dl_unmask_all_rx_done_intr(drv_info);
+ drv_info->cfg->rx_cfg.rxq_all_enable = true;
+ }
+
+ return 0;
+}
+
+int mtk_dpmaif_drv_stop_queue(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_dir dir)
+{
+ int ret;
+
+ if (dir == DPMAIF_TX) {
+ if (unlikely(!drv_info->cfg->tx_cfg.txq_all_enable))
+ return 0;
+
+ ret = mtk_dpmaif_drv_stop_ulq(drv_info);
+ if (ret < 0)
+ return ret;
+
+ ret = mtk_dpmaif_drv_ul_mask_all_tx_done_intr(drv_info);
+ if (ret < 0)
+ return ret;
+
+ drv_info->cfg->tx_cfg.txq_all_enable = false;
+ } else {
+ if (unlikely(!drv_info->cfg->rx_cfg.rxq_all_enable))
+ return 0;
+
+ ret = mtk_dpmaif_drv_stop_dlq(drv_info);
+ if (ret < 0)
+ return ret;
+
+ ret = mtk_dpmaif_drv_dl_mask_all_rx_done_intr(drv_info);
+ if (ret < 0)
+ return ret;
+
+ drv_info->cfg->rx_cfg.rxq_all_enable = false;
+ }
+
+ return 0;
+}
+
+void mtk_dpmaif_drv_clr_ip_busy_sts(struct dpmaif_drv_info *drv_info)
+{
+ u32 ip_busy_sts;
+
+ /* Get AP IP busy status. */
+ ip_busy_sts = mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_AP_IP_BUSY);
+
+ /* Clear AP IP busy. */
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_AP_IP_BUSY, ip_busy_sts);
+}
+
+int mtk_dpmaif_drv_intr_handle_com(struct dpmaif_drv_info *drv_info, void *data, u8 irq_id)
+{
+ return drv_info->cfg->intr_cfg.irqs[irq_id].handle(drv_info, data);
+}
+
+int mtk_dpmaif_drv_intr_complete_com(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_intr_type type, u8 id, u64 data)
+{
+ switch (type) {
+ case DPMAIF_INTR_UL_DONE:
+ if (data == DPMAIF_CLEAR_INTR)
+ mtk_dpmaif_drv_clr_ul_done_status(drv_info, id);
+ else
+ mtk_dpmaif_drv_unmask_ulq_intr(drv_info, id);
+ break;
+ case DPMAIF_INTR_DL_BATCNT_LEN_ERR:
+ mtk_dpmaif_drv_unmask_dl_batcnt_len_err_intr(drv_info, id);
+ break;
+ case DPMAIF_INTR_DL_PITCNT_LEN_ERR:
+ mtk_dpmaif_drv_dlq_unmask_pit_cnt_len_err_intr(drv_info, id);
+ break;
+ case DPMAIF_INTR_DL_DONE:
+ mtk_dpmaif_drv_dl_unmask_rx_done_intr(drv_info, id);
+ break;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+int mtk_dpmaif_drv_send_doorbell_com(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_ring_type type, u8 id, u32 cnt)
+{
+ int ret = 0;
+
+ switch (type) {
+ case DPMAIF_PIT:
+ ret = mtk_dpmaif_drv_dl_add_pit_cnt(drv_info, id, cnt);
+ break;
+ case DPMAIF_BAT:
+ ret = mtk_dpmaif_drv_dl_add_bat_cnt(drv_info, id, cnt);
+ break;
+ case DPMAIF_DRB:
+ ret = mtk_dpmaif_drv_ul_add_drb(drv_info, id, cnt);
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
+int mtk_dpmaif_drv_get_ring_idx(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_ring_idx index, u8 q_id)
+{
+ int ret = 0;
+
+ switch (index) {
+ case DPMAIF_PIT_WIDX:
+ ret = mtk_dpmaif_drv_dl_get_pit_wridx(drv_info, q_id);
+ break;
+ case DPMAIF_PIT_RIDX:
+ ret = mtk_dpmaif_drv_dl_get_pit_rdidx(drv_info, q_id);
+ break;
+ case DPMAIF_BAT_WIDX:
+ ret = mtk_dpmaif_drv_dl_get_bat_wridx(drv_info, q_id);
+ break;
+ case DPMAIF_BAT_RIDX:
+ ret = mtk_dpmaif_drv_dl_get_bat_ridx(drv_info, q_id);
+ break;
+
+ case DPMAIF_DRB_WIDX:
+ break;
+
+ case DPMAIF_DRB_RIDX:
+ ret = mtk_dpmaif_drv_ul_get_drb_ridx(drv_info, q_id);
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
+void mtk_dpmaif_drv_ul_mask_intr(struct dpmaif_drv_info *drv_info, u32 mask)
+{
+ u32 cnt = 0;
+
+ if (drv_info->priv_ops && drv_info->priv_ops->mask_ul_intr) {
+ drv_info->priv_ops->mask_ul_intr(drv_info, mask);
+ return;
+ }
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TISR0,
+ mask);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TISR0);
+
+ do {
+ if ((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0) &
+ mask) == mask)
+ return;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to mask interrupt done, sta=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->ao_base + DPMAIF_PD_AP_UL_L2TIMR0));
+
+ WARN_ON_ONCE(true);
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h
new file mode 100644
index 000000000000..5a68d2431063
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h
@@ -0,0 +1,259 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DPMAIF_DRV_H__
+#define __MTK_DPMAIF_DRV_H__
+
+#include "mtk_dpmaif_ring.h"
+
+#define DATA_HW_F_HPC BIT(7)
+
+#define DPMAIF_RXQ_CNT_MAX 3
+#define DPMAIF_TXQ_CNT_MAX 5
+#define DPMAIF_IRQ_CNT_MAX 5
+
+enum dpmaif_drv_dir {
+ DPMAIF_TX,
+ DPMAIF_RX,
+};
+
+struct dpmaif_drv_intr {
+ enum dpmaif_drv_dir dir;
+ unsigned int q_mask;
+ unsigned int mode;
+};
+
+enum mtk_drv_err {
+ DATA_ERR_STOP_MAX = 10,
+ DATA_HW_REG_TIMEOUT,
+ DATA_HW_REG_CHK_FAIL,
+ DATA_FLOW_CHK_ERR,
+ DATA_DMA_MAP_ERR,
+ DATA_DL_ONCE_MORE,
+ DATA_PIT_SEQ_CHK_FAIL,
+ DATA_LOW_MEM_TYPE_MAX,
+ DATA_LOW_MEM_DRB,
+ DATA_LOW_MEM_BAT,
+ DATA_LOW_MEM_PIT,
+ DATA_LOW_MEM_SKB,
+ DATA_HW_UNK_PKT,
+};
+
+enum {
+ DPMAIF_CLEAR_INTR,
+ DPMAIF_UNMASK_INTR,
+};
+
+enum dpmaif_drv_dlq_id {
+ DPMAIF_DLQ0 = 0,
+ DPMAIF_DLQ1,
+ DPMAIF_DLQ2,
+ DPMAIF_DLQ_MAX
+};
+
+enum dpmaif_drv_bat_id {
+ DPMAIF_BAT0 = 0,
+ DPMAIF_BAT1,
+};
+
+enum dpmaif_drv_ring_type {
+ DPMAIF_PIT,
+ DPMAIF_BAT,
+ DPMAIF_DRB,
+};
+
+enum dpmaif_drv_ring_idx {
+ DPMAIF_PIT_WIDX,
+ DPMAIF_PIT_RIDX,
+ DPMAIF_BAT_WIDX,
+ DPMAIF_BAT_RIDX,
+ DPMAIF_DRB_WIDX,
+ DPMAIF_DRB_RIDX,
+};
+
+struct dpmaif_drv_regs {
+ unsigned long ao_base;
+ unsigned long pd_base;
+ unsigned long pd2_base;
+ unsigned long ao_ul_ch0_sta;
+};
+
+struct dpmaif_tx_srv_cfg {
+ unsigned char vq_cnt;
+ const unsigned char *vqs;
+ int nice;
+};
+
+struct dpmaif_tx_srvs_cfg {
+ unsigned char tx_vq_cnt;
+ unsigned char tx_srv_cnt;
+ const struct dpmaif_tx_srv_cfg *tx_srvs;
+};
+
+struct dpmaif_txq_cfg {
+ dma_addr_t drb_base;
+ const unsigned int drb_cnt;
+};
+
+struct dpmaif_tx_cfg {
+ bool txq_all_enable;
+ const unsigned char txq_cnt;
+ struct dpmaif_txq_cfg *txqs;
+};
+
+struct dpmaif_rxq_cfg {
+ dma_addr_t pit_base;
+ const unsigned int pit_cnt;
+ const unsigned int pit_seq_max;
+ const unsigned char bat_ring_id;
+};
+
+struct dpmaif_bat_cfg {
+ dma_addr_t bat_base;
+ const unsigned int bat_cnt;
+ unsigned int buf_size;
+ unsigned int reload_cnt;
+ /* actual buffer count allocated during init */
+ unsigned int real_reload_cnt;
+};
+
+struct dpmaif_rx_cfg {
+ bool rxq_all_enable;
+ unsigned int mtu;
+ const unsigned int normal_bat_rsv_length;
+ const unsigned int pkt_alignment;
+ const unsigned char bat_ring_num;
+ struct dpmaif_bat_cfg *bats;
+ const unsigned char rxq_cnt;
+ struct dpmaif_rxq_cfg *rxqs;
+ /* Prefetch count required for DPMAIF Read */
+ const unsigned char bat_wrap_cnt;
+};
+
+struct dpmaif_drv_info;
+struct dpmaif_drv_intr_info;
+
+struct dpmaif_irq_cfg {
+ const unsigned int id;
+ int (*handle)(struct dpmaif_drv_info *drv_info, struct dpmaif_drv_intr_info *intr_info);
+};
+
+struct dpmaif_intr_cfg {
+ u32 ul3_l2intrs_enable;
+ u32 ul3_l2intrs_disable;
+ u32 ul_l2intrs_enable;
+ u32 ul_l2intrs_disable;
+ u32 dl_l2intrs_enable;
+ u32 dl_l2intrs_disable;
+ u32 dl2_l2intrs_enable;
+ u32 dl2_l2intrs_disable;
+ u32 udl_ip_busy_disable;
+ u32 hpc_disable;
+ unsigned char irq_cnt;
+ const struct dpmaif_irq_cfg *irqs;
+};
+
+struct dpmaif_drv_cfg {
+ const u32 cap;
+ const struct dpmaif_tx_srvs_cfg tx_srvs_cfg;
+ struct dpmaif_tx_cfg tx_cfg;
+ struct dpmaif_rx_cfg rx_cfg;
+ const struct dpmaif_intr_cfg intr_cfg;
+};
+
+struct dpmaif_priv_ops {
+ void (*set_pcie_domain)(struct dpmaif_drv_info *drv_info);
+ u32 (*get_ul_intr_mask)(struct dpmaif_drv_info *drv_info);
+ int (*dynamic_sram_init)(struct dpmaif_drv_info *drv_info);
+ int (*ul_intr_init)(struct dpmaif_drv_info *drv_info);
+ int (*mask_ulq_intr)(struct dpmaif_drv_info *drv_info, u32 q_num);
+ void (*unmask_ulq_intr)(struct dpmaif_drv_info *drv_info, u32 q_num);
+ void (*mask_ul_intr)(struct dpmaif_drv_info *drv_info, u32 mask);
+ void (*set_hpc_cntl)(struct dpmaif_drv_info *drv_info);
+ void (*set_dlq_timeout)(struct dpmaif_drv_info *drv_info);
+ void (*clr_dlq_timeout)(struct dpmaif_drv_info *drv_info);
+};
+
+struct dpmaif_drv_info {
+ struct mtk_md_dev *mdev;
+ struct dpmaif_drv_ops *drv_ops;
+ struct dpmaif_priv_ops *priv_ops;
+ const struct dpmaif_drv_regs *regs;
+ struct dpmaif_drv_cfg *cfg;
+};
+
+enum dpmaif_drv_intr_type {
+ DPMAIF_INTR_MIN = 0,
+ /* uplink part */
+ DPMAIF_INTR_UL_DONE,
+ DPMAIF_INTR_UL_DRB_EMPTY,
+ DPMAIF_INTR_UL_MD_NOTREADY,
+ DPMAIF_INTR_UL_MD_PWR_NOTREADY,
+ DPMAIF_INTR_UL_LEN_ERR,
+
+ /* downlink part */
+ DPMAIF_INTR_DL_LEGACY_DONE,
+ DPMAIF_INTR_DL_SKB_LEN_ERR,
+ DPMAIF_INTR_DL_BATCNT_LEN_ERR,
+ DPMAIF_INTR_DL_PKT_EMPTY_SET,
+ DPMAIF_INTR_DL_MTU_ERR,
+
+ DPMAIF_INTR_DL_PITCNT_LEN_ERR,
+ DPMAIF_INTR_DL_HPC_ENT_TYPE_ERR,
+ DPMAIF_INTR_DL_DONE,
+
+ /* traffic sync */
+ DPMAIF_INTR_TRAS_SYNC,
+ DPMAIF_INTR_MAX
+};
+
+#define DPMAIF_INTR_COUNT ((DPMAIF_INTR_MAX) - (DPMAIF_INTR_MIN) - 1)
+
+struct dpmaif_drv_intr_info {
+ unsigned char intr_cnt;
+ enum dpmaif_drv_intr_type intr_types[DPMAIF_INTR_COUNT];
+ /* it's a queue mask or queue index */
+ u32 intr_queues[DPMAIF_INTR_COUNT];
+};
+
+/* This structure defines the management hooks for dpmaif devices. */
+struct dpmaif_drv_ops {
+ /* Initialize dpmaif hardware. */
+ int (*init)(struct dpmaif_drv_info *drv_info, void *data);
+ /* Start dpmaif hardware transaction and unmask dpmaif interrupt. */
+ int (*start_queue)(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_dir dir);
+ /* Stop dpmaif hardware transaction and mask dpmaif interrupt. */
+ int (*stop_queue)(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_dir dir);
+ int (*intr_handle)(struct dpmaif_drv_info *drv_info, void *data, u8 irq_id);
+ /* Unmask or clear dpmaif interrupt. */
+ int (*intr_complete)(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_intr_type type,
+ u8 q_id, u64 data);
+ int (*send_doorbell)(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_ring_type type,
+ u8 q_id, u32 cnt);
+ int (*get_ring_idx)(struct dpmaif_drv_info *drv_info, enum dpmaif_drv_ring_idx index,
+ u8 q_id);
+ int (*get_rx_info)(void *pit, struct dpmaif_rx_info *rx_info, u32 pit_seq_expect, u8 q_id);
+ void (*fill_tx_info)(void *drb, struct dpmaif_tx_info *tx_info, int type);
+};
+
+static inline int mtk_dpmaif_drv_intr_handle(struct dpmaif_drv_info *drv_info,
+ void *data, u8 irq_id)
+{
+ return drv_info->drv_ops->intr_handle(drv_info, data, irq_id);
+}
+
+static inline int mtk_dpmaif_drv_intr_complete(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_intr_type type, u8 q_id, u64 data)
+{
+ return drv_info->drv_ops->intr_complete(drv_info, type, q_id, data);
+}
+
+static inline int mtk_dpmaif_drv_send_doorbell(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_ring_type type, u8 q_id, u32 cnt)
+{
+ return drv_info->drv_ops->send_doorbell(drv_info, type, q_id, cnt);
+}
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv_m9xx.c
new file mode 100644
index 000000000000..a1b86424852d
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv_m9xx.c
@@ -0,0 +1,687 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#define pr_fmt(fmt) "DATA_DRV_M9XX: " fmt
+
+#include <linux/delay.h>
+
+#include "mtk_dev.h"
+#include "mtk_dpmaif_drv.h"
+#include "mtk_dpmaif_reg_m9xx.h"
+#include "mtk_dpmaif_ring.h"
+#include "mtk_pci.h"
+
+enum dpmaif_drv_irq_src {
+ DPMAIF_IRQ_SRC0_DLQ0,
+ DPMAIF_IRQ_SRC1_DLQ1,
+ DPMAIF_IRQ_SRC2_UL_DONE,
+ DPMAIF_IRQ_SRC3_DLQ2,
+ DPMAIF_IRQ_SRC4_TRANS_SYNC
+};
+
+static int mtk_dpmaif_drv_irq_src0(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info);
+static int mtk_dpmaif_drv_irq_src1(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info);
+static int mtk_dpmaif_drv_irq_src2(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info);
+static int mtk_dpmaif_drv_irq_src3(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info);
+static int mtk_dpmaif_drv_irq_src4(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info);
+
+static const struct dpmaif_drv_regs regs = {
+ .ao_base = DPMAIF_DEV_AO_BASE,
+ .pd_base = DPMAIF_DEV_PD_BASE,
+ .pd2_base = DPMAIF_DEV_PD2_BASE,
+ .ao_ul_ch0_sta = NRL2_DPMAIF_AO_UL_CH0_STA,
+};
+
+static const unsigned char tx_srv0_vqs[] = {3};
+static const unsigned char tx_srv1_vqs[] = {1};
+static const unsigned char tx_srv2_vqs[] = {0, 4};
+static const struct dpmaif_tx_srv_cfg tx_srvs[] = {
+ {
+ .vq_cnt = ARRAY_SIZE(tx_srv0_vqs),
+ .vqs = tx_srv0_vqs,
+ .nice = -20,
+ },
+ {
+ .vq_cnt = ARRAY_SIZE(tx_srv1_vqs),
+ .vqs = tx_srv1_vqs,
+ .nice = -15,
+ },
+ {
+ .vq_cnt = ARRAY_SIZE(tx_srv2_vqs),
+ .vqs = tx_srv2_vqs,
+ .nice = -10,
+ },
+};
+
+static struct dpmaif_txq_cfg txqs[] = {
+ {
+ .drb_cnt = 2048,
+ },
+ {
+ .drb_cnt = 2048,
+ },
+ {
+ .drb_cnt = 128,
+ },
+ {
+ .drb_cnt = 1024,
+ },
+ {
+ .drb_cnt = 2048,
+ },
+};
+
+static struct dpmaif_bat_cfg bats[] = {
+ {
+ .bat_cnt = 32768,
+ .reload_cnt = 3072,
+ },
+ {
+ .bat_cnt = 1024,
+ .reload_cnt = 512,
+ },
+};
+
+static struct dpmaif_rxq_cfg rxqs[] = {
+ {
+ .pit_cnt = 32768,
+ .bat_ring_id = 0,
+ .pit_seq_max = 251,
+ },
+ {
+ .pit_cnt = 32768,
+ .bat_ring_id = 0,
+ .pit_seq_max = 251,
+ },
+ {
+ .pit_cnt = 2048,
+ .bat_ring_id = 1,
+ .pit_seq_max = 251,
+ },
+};
+
+static const struct dpmaif_irq_cfg irqs[] = {
+ {
+ .id = MTK_IRQ_SRC_DPMAIF,
+ .handle = mtk_dpmaif_drv_irq_src0,
+ },
+ {
+ .id = MTK_IRQ_SRC_DPMAIF2,
+ .handle = mtk_dpmaif_drv_irq_src1,
+ },
+ {
+ .id = MTK_IRQ_SRC_DPMAIF3,
+ .handle = mtk_dpmaif_drv_irq_src2,
+ },
+ {
+ .id = MTK_IRQ_SRC_DPMAIF6,
+ .handle = mtk_dpmaif_drv_irq_src3,
+ },
+ {
+ .id = MTK_IRQ_SRC_TRAS_SYNC,
+ .handle = mtk_dpmaif_drv_irq_src4,
+ },
+};
+
+static struct dpmaif_drv_cfg drv_cfg = {
+ .cap = DATA_HW_F_HPC,
+ .tx_srvs_cfg = {
+ .tx_vq_cnt = ARRAY_SIZE(txqs),
+ .tx_srv_cnt = ARRAY_SIZE(tx_srvs),
+ .tx_srvs = tx_srvs,
+ },
+ .tx_cfg = {
+ .txq_cnt = ARRAY_SIZE(txqs),
+ .txqs = txqs,
+ },
+ .rx_cfg = {
+ .normal_bat_rsv_length = 0,
+ .pkt_alignment = 64,
+ .mtu = 9000,
+ .bat_ring_num = ARRAY_SIZE(bats),
+ .bats = bats,
+ .rxq_cnt = ARRAY_SIZE(rxqs),
+ .rxqs = rxqs,
+ .bat_wrap_cnt = 48,
+ },
+ .intr_cfg = {
+ .ul3_l2intrs_disable = DPMAIF_UL3_INT_VALID_MSK & ~DPMAIF_UL_INT_QDONE_MSK,
+ .ul3_l2intrs_enable = DPMAIF_UL_INT_QDONE_MSK,
+ .ul_l2intrs_disable = DPMAIF_UL_INT_VALID_MSK & ~DPMAIF_UL_INT_QDONE_MSK,
+ .ul_l2intrs_enable = DPMAIF_UL_INT_QDONE_MSK,
+ .dl_l2intrs_disable = DPMAIF_DL_INT_VALID_MSK & ~(DPMAIF_DL_INT_DLQ0_QDONE_MSK |
+ DPMAIF_DL_INT_DLQ1_QDONE_MSK | DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR_MSK |
+ DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR_MSK | DPMAIF_DL_INT_BATCNT_LEN_ERR_MSK),
+ .dl_l2intrs_enable = DPMAIF_DL_INT_DLQ0_QDONE_MSK |
+ DPMAIF_DL_INT_DLQ1_QDONE_MSK | DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR_MSK |
+ DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR_MSK | DPMAIF_DL_INT_BATCNT_LEN_ERR_MSK,
+ .dl2_l2intrs_disable = DPMAIF_DL2_INT_OTHER_MSK & ~(DPMAIF_DL_INT_DLQ2_QDONE_MSK |
+ DPMAIF_DL_INT_DLQ2_PITCNT_LEN_ERR_MSK | DPMAIF_DL2_INT_BATCNT_LEN_ERR_MSK),
+ .dl2_l2intrs_enable = DPMAIF_DL_INT_DLQ2_QDONE_MSK |
+ DPMAIF_DL_INT_DLQ2_PITCNT_LEN_ERR_MSK | DPMAIF_DL2_INT_BATCNT_LEN_ERR_MSK,
+ .udl_ip_busy_disable = DPMAIF_UDL_IP_BUSY_MSK,
+ .hpc_disable = DPMAIF_DL_INT_Q2APTOP_MSK |
+ DPMAIF_DL_INT_Q2TOQ1_MSK | DPMAIF_UL_TOP0_INT_MSK,
+ .irq_cnt = ARRAY_SIZE(irqs),
+ .irqs = irqs,
+ },
+};
+
+static u32 mtk_dpmaif_drv_get_ul_intr_mask_m9xx(struct dpmaif_drv_info *drv_info)
+{
+ return mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ DPMAIF_PD_AP_UL_L2TIMR0_NEXT);
+}
+
+static int mtk_dpmaif_drv_dynamic_sram_init(struct dpmaif_drv_info *drv_info)
+{
+ u32 sram_addr_tbl[][2] = {
+ {0xFFFF0100, 0x8033F000},
+ {0xFFFF0200, 0x801DF000},
+ {0xFFFF0400, 0x805FF000},
+ {0xFFFF1000, 0x8033F000},
+ {0xFFFF4000, 0x80443000},
+ {0xFFFF8000, 0x80177000},
+ };
+ u32 sram_cnt;
+ u32 cnt = 0;
+ u32 i;
+
+ sram_cnt = ARRAY_SIZE(sram_addr_tbl);
+ for (i = 0; i < sram_cnt; i++) {
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AP_MISC_SRAM_INIT_SET1, sram_addr_tbl[i][0]);
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AP_MISC_SRAM_INIT_SET0, sram_addr_tbl[i][1]);
+
+ do {
+ if (!(mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AP_MISC_SRAM_INIT_SET0) & DPMAIF_AP_PD_SRAM_EN_BIT))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev, "Failed to initialize dynamic sram%d\n", i);
+ return -DATA_HW_REG_TIMEOUT;
+ }
+ cnt = 0;
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_ul_intr_init_m9xx(struct dpmaif_drv_info *drv_info)
+{
+ const struct dpmaif_intr_cfg *intr_cfg = &drv_info->cfg->intr_cfg;
+ u32 cnt = 0;
+ int ret;
+
+ ret = mtk_dpmaif_drv_ul_intr_init(drv_info);
+ if (unlikely(ret < 0))
+ return ret;
+
+ /* clear UL interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISAR0,
+ 0xFFFFFFFF);
+ /* unmask ul3_l2intrs_enable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TICR0_NEXT,
+ intr_cfg->ul3_l2intrs_enable);
+ /* mask ul3_l2intrs_disable interrupt */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISR0_NEXT,
+ intr_cfg->ul3_l2intrs_disable);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISR0_NEXT);
+ do {
+ if (((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TIMR0_NEXT) &
+ intr_cfg->ul3_l2intrs_disable) ==
+ intr_cfg->ul3_l2intrs_disable))
+ break;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ if (cnt >= POLL_MAX_TIMES) {
+ dev_err((drv_info->mdev)->dev,
+ "Failed to set UL interrupt mask, mask=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base +
+ DPMAIF_PD_AP_UL_L2TIMR0_NEXT));
+ return -DATA_HW_REG_TIMEOUT;
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_mask_ulq_intr(struct dpmaif_drv_info *drv_info, u32 q_num)
+{
+ u32 cnt = 0, ui_que_done_mask;
+
+ ui_que_done_mask = BIT(q_num + DP_UL_INT_DONE_OFFSET) & DPMAIF_UL_INT_QDONE_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISR0_NEXT,
+ ui_que_done_mask);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISR0_NEXT);
+
+ do {
+ if ((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TIMR0_NEXT) &
+ ui_que_done_mask))
+ return 0;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to mask ulq%u interrupt done, sta=0x%08x\n",
+ q_num, mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base +
+ DPMAIF_PD_AP_UL_L2TIMR0_NEXT));
+
+ return -DATA_HW_REG_TIMEOUT;
+}
+
+static void mtk_dpmaif_drv_unmask_ulq_intr(struct dpmaif_drv_info *drv_info, u32 q_num)
+{
+ u32 ui_que_done_mask;
+
+ ui_que_done_mask = BIT(q_num + DP_UL_INT_DONE_OFFSET) & DPMAIF_UL_INT_QDONE_MSK;
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TICR0_NEXT,
+ ui_que_done_mask);
+}
+
+static void mtk_dpmaif_drv_mask_ul_intr(struct dpmaif_drv_info *drv_info, u32 mask)
+{
+ u32 cnt = 0;
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISR0_NEXT,
+ mask);
+ mtk_pci_read32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISR0_NEXT);
+
+ do {
+ if ((mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TIMR0_NEXT) &
+ mask) == mask)
+ return;
+
+ udelay(POLL_INTERVAL_US);
+ } while (++cnt < POLL_MAX_TIMES);
+
+ dev_err((drv_info->mdev)->dev,
+ "Failed to mask interrupt done, sta=0x%08x\n",
+ mtk_pci_read32(drv_info->mdev,
+ drv_info->regs->pd_base +
+ DPMAIF_PD_AP_UL_L2TIMR0_NEXT));
+
+ WARN_ON_ONCE(true);
+}
+
+static void mtk_dpmaif_drv_set_hpc_cntl_m9xx(struct dpmaif_drv_info *drv_info)
+{
+ u32 cfg;
+
+ cfg = DPMAIF_HPC_LRO_PATH_DF & 0x3;
+ cfg |= (DPMAIF_HPC_ADD_MODE_DF & 0x3) << 2;
+ cfg |= (DPMAIF_HASH_PRIME_DF & 0xF) << 4;
+ cfg |= (DPMAIF_HPC_NUM_M9XX & 0xFF) << 8;
+
+ /* Configuration include hpc dlq path, hpc add mode, hash prime, hpc total number. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + NRL2_DPMAIF_AO_DL_HPC_CNTL, cfg);
+}
+
+static void mtk_dpmaif_drv_set_dlq_timeout_m9xx(struct dpmaif_drv_info *drv_info)
+{
+ u32 val, i;
+
+ for (i = 0; i < DPMAIF_HPC_NUM_EXT; i++) {
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd2_base +
+ NRL2_DPMAIF_PD_DL_LROPIT_TIMEOUT5 + ((i >> 1) << 2));
+
+ if (i % 2)
+ val = (val & 0xFFFF) | (DPMAIF_LRO_TIMEOUT_THRES_DF << 16);
+ else
+ val = (val & 0xFFFF0000) | DPMAIF_LRO_TIMEOUT_THRES_DF;
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd2_base +
+ NRL2_DPMAIF_PD_DL_LROPIT_TIMEOUT5 + ((i >> 1) << 2), val);
+ }
+}
+
+static void mtk_dpmaif_drv_clr_dlq_timeout_m9xx(struct dpmaif_drv_info *drv_info)
+{
+ u32 val, i;
+
+ for (i = 0; i < DPMAIF_HPC_NUM_EXT; i++) {
+ val = mtk_pci_read32(drv_info->mdev, drv_info->regs->pd2_base +
+ NRL2_DPMAIF_PD_DL_LROPIT_TIMEOUT5 + ((i >> 1) << 2));
+
+ if (i % 2)
+ val = (val & 0xFFFF) | BIT(16);
+ else
+ val = (val & 0xFFFF0000) | 1;
+
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd2_base +
+ NRL2_DPMAIF_PD_DL_LROPIT_TIMEOUT5 + ((i >> 1) << 2), val);
+ }
+}
+
+static void mtk_dpmaif_drv_set_pcie_domain(struct dpmaif_drv_info *drv_info)
+{
+ /* set HW pcie domain */
+ mtk_pci_write32(drv_info->mdev,
+ drv_info->regs->pd2_base + DPMAIF_DL_RESOURCE_DOMAIN, 0x000A9AAA);
+}
+
+static struct dpmaif_priv_ops dpmaif_priv_ops_m9xx = {
+ .set_pcie_domain = mtk_dpmaif_drv_set_pcie_domain,
+ .get_ul_intr_mask = mtk_dpmaif_drv_get_ul_intr_mask_m9xx,
+ .dynamic_sram_init = mtk_dpmaif_drv_dynamic_sram_init,
+ .ul_intr_init = mtk_dpmaif_drv_ul_intr_init_m9xx,
+ .mask_ulq_intr = mtk_dpmaif_drv_mask_ulq_intr,
+ .unmask_ulq_intr = mtk_dpmaif_drv_unmask_ulq_intr,
+ .mask_ul_intr = mtk_dpmaif_drv_mask_ul_intr,
+ .set_hpc_cntl = mtk_dpmaif_drv_set_hpc_cntl_m9xx,
+ .set_dlq_timeout = mtk_dpmaif_drv_set_dlq_timeout_m9xx,
+ .clr_dlq_timeout = mtk_dpmaif_drv_clr_dlq_timeout_m9xx,
+};
+
+static void mtk_dpmaif_drv_cfg_get(struct dpmaif_drv_info *drv_info)
+{
+ drv_info->regs = ®s;
+ drv_info->cfg = &drv_cfg;
+ drv_info->priv_ops = &dpmaif_priv_ops_m9xx;
+}
+
+static void mtk_dpmaif_drv_reset(struct dpmaif_drv_info *drv_info)
+{
+ u32 mask;
+ /* Set DPMAIF infra CG en */
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_INFRA_CG_SET, DPMAIF_AP_INFRA_CG_BIT);
+ udelay(2);
+
+ /* Glitch protect on */
+ mask = mtk_pci_read32(drv_info->mdev, DPMAIF_AP_INFRA_GLITCH_PORT);
+ mask &= ~DPMAIF_AP_GLITCH_PROT_BIT;
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_INFRA_GLITCH_PORT, mask);
+ udelay(2);
+
+ /* AO&PD reset assert&de-assert */
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_AO_RGU_ASSERT, DPMAIF_AP_AO_RST_BIT);
+ udelay(2);
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_RGU_ASSERT, DPMAIF_AP_RST_BIT);
+ udelay(2);
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_AO_RGU_DEASSERT, DPMAIF_AP_AO_RST_BIT);
+ udelay(2);
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_RGU_DEASSERT, DPMAIF_AP_RST_BIT);
+ udelay(2);
+
+ /* Glitch protect off */
+ mask = mtk_pci_read32(drv_info->mdev, DPMAIF_AP_INFRA_GLITCH_PORT);
+ mask |= DPMAIF_AP_GLITCH_PROT_BIT;
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_INFRA_GLITCH_PORT, mask);
+ udelay(2);
+
+ /* Set DPMAIF infra CG off */
+ mtk_pci_write32(drv_info->mdev, DPMAIF_AP_INFRA_CG_CLR, DPMAIF_AP_INFRA_CG_BIT);
+ udelay(2);
+}
+
+static u32 mtk_dpmaif_drv_irq_src0_filter(struct dpmaif_drv_info *drv_info, u32 l2risar0,
+ u32 l2rimr0)
+{
+ if (l2rimr0 & DPMAIF_DL_INT_DLQ0_QDONE_MSK)
+ l2risar0 &= ~DPMAIF_DL_INT_DLQ0_QDONE;
+
+ if (l2rimr0 & DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR_MSK)
+ l2risar0 &= ~DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR;
+
+ if (l2rimr0 & DPMAIF_DL_INT_BATCNT_LEN_ERR_MSK)
+ l2risar0 &= ~DPMAIF_DL_INT_BATCNT_LEN_ERR;
+
+ return l2risar0;
+}
+
+static int mtk_dpmaif_drv_irq_src0(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info)
+{
+ u32 val, ori_l2risar0, l2risar0, l2rimr0;
+
+ ori_l2risar0 = mtk_dpmaif_drv_get_dl_lv2_sts(drv_info, DPMAIF_DLQ0);
+ l2rimr0 = mtk_dpmaif_drv_get_dl_intr_mask(drv_info, DPMAIF_DLQ0);
+
+ /* filter care interrupt status. */
+ l2risar0 = ori_l2risar0 & (DPMAIF_DL_INT_DLQ0_QDONE | DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR |
+ DPMAIF_DL_INT_BATCNT_LEN_ERR |
+ DPMAIF_DL_INT_DLQ_QDONE);
+ if (l2risar0) {
+ /* Filter to get DL unmasked interrupts */
+ l2risar0 = mtk_dpmaif_drv_irq_src0_filter(drv_info, l2risar0, l2rimr0);
+
+ val = l2risar0 & DPMAIF_DL_INT_BATCNT_LEN_ERR;
+ if (val) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_BATCNT_LEN_ERR;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_BAT0;
+ intr_info->intr_cnt++;
+ mtk_dpmaif_drv_mask_dl_batcnt_len_err_intr(drv_info, DPMAIF_BAT0);
+ }
+
+ val = l2risar0 & DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR;
+ if (val) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_PITCNT_LEN_ERR;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_DLQ0;
+ intr_info->intr_cnt++;
+ mtk_dpmaif_drv_dlq_mask_pit_cnt_len_err_intr(drv_info, DPMAIF_DLQ0);
+ }
+
+ val = l2risar0 & DPMAIF_DL_INT_DLQ0_QDONE;
+ if (val) {
+ if (!mtk_dpmaif_drv_dlq_mask_rx_done_intr(drv_info, DPMAIF_DLQ0)) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_DONE;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_DLQ0;
+ intr_info->intr_cnt++;
+ }
+ }
+
+ /* Clear interrupt status. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_DL_L2TISAR0,
+ l2risar0);
+ }
+
+ mtk_dpmaif_drv_clr_ip_busy_sts(drv_info);
+
+ return 0;
+}
+
+static u32 mtk_dpmaif_drv_irq_src1_filter(struct dpmaif_drv_info *drv_info, u32 l2risar0,
+ u32 l2rimr0)
+{
+ if (l2rimr0 & DPMAIF_DL_INT_DLQ1_QDONE_MSK)
+ l2risar0 &= ~DPMAIF_DL_INT_DLQ1_QDONE;
+
+ if (l2rimr0 & DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR_MSK)
+ l2risar0 &= ~DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR;
+
+ return l2risar0;
+}
+
+static int mtk_dpmaif_drv_irq_src1(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info)
+{
+ u32 val, ori_l2risar0, l2risar0, l2rimr0;
+
+ ori_l2risar0 = mtk_dpmaif_drv_get_dl_lv2_sts(drv_info, DPMAIF_DLQ1);
+ l2rimr0 = mtk_dpmaif_drv_get_dl_intr_mask(drv_info, DPMAIF_DLQ1);
+
+ /* filter care interrupt status. */
+ l2risar0 = ori_l2risar0 & (DPMAIF_DL_INT_DLQ1_QDONE | DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR);
+ if (l2risar0) {
+ /* Filter to get DL unmasked interrupts */
+ l2risar0 = mtk_dpmaif_drv_irq_src1_filter(drv_info, l2risar0, l2rimr0);
+
+ val = l2risar0 & DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR;
+ if (val) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_PITCNT_LEN_ERR;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_DLQ1;
+ intr_info->intr_cnt++;
+ mtk_dpmaif_drv_dlq_mask_pit_cnt_len_err_intr(drv_info, DPMAIF_DLQ1);
+ }
+
+ val = l2risar0 & DPMAIF_DL_INT_DLQ1_QDONE;
+ if (val) {
+ if (!mtk_dpmaif_drv_dlq_mask_rx_done_intr(drv_info, DPMAIF_DLQ1)) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_DONE;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_DLQ1;
+ intr_info->intr_cnt++;
+ }
+ }
+
+ /* Clear interrupt status. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_DL_L2TISAR0,
+ l2risar0);
+ }
+
+ mtk_dpmaif_drv_clr_ip_busy_sts(drv_info);
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_irq_src2(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info)
+{
+ u32 ori_l2tisar0, l2tisar0, l2timr0;
+ static u32 cnt;
+ u8 q_mask;
+ u32 val;
+
+ ori_l2tisar0 = mtk_dpmaif_drv_get_ul_lv2_sts(drv_info);
+ l2timr0 = mtk_dpmaif_drv_get_ul_intr_mask(drv_info);
+
+ /* Check and process interrupt. */
+ l2tisar0 = ori_l2tisar0 & (~l2timr0);
+ if (l2tisar0) {
+ cnt = 0;
+ val = l2tisar0 & DPMAIF_UL_INT_QDONE;
+ if (val) {
+ q_mask = val >> DP_UL_INT_DONE_OFFSET & DPMAIF_ULQS;
+ mtk_dpmaif_drv_ul_mask_multi_tx_done_intr(drv_info, q_mask);
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_UL_DONE;
+ intr_info->intr_queues[intr_info->intr_cnt] = val >> DP_UL_INT_DONE_OFFSET;
+ intr_info->intr_cnt++;
+ } else {
+ mtk_dpmaif_drv_ul_mask_intr(drv_info, l2tisar0);
+ }
+
+ /* clear interrupt status */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base + DPMAIF_PD_AP_UL_L2TISAR0,
+ l2tisar0);
+ }
+
+ mtk_dpmaif_drv_clr_ip_busy_sts(drv_info);
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_irq_src4(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info)
+{
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_TRAS_SYNC;
+ intr_info->intr_cnt++;
+
+ return 0;
+}
+
+static u32 mtk_dpmaif_drv_irq_src6_filter(struct dpmaif_drv_info *drv_info,
+ u32 l2risar1, u32 l2rimr1)
+{
+ if (l2rimr1 & DPMAIF_DL_INT_DLQ2_QDONE_MSK)
+ l2risar1 &= ~DPMAIF_DL2_INT_DLQ2_QDONE;
+
+ if (l2rimr1 & DPMAIF_DL_INT_DLQ2_PITCNT_LEN_ERR_MSK)
+ l2risar1 &= ~DPMAIF_DL2_INT_DLQ2_PITCNT_LEN_ERR;
+
+ if (l2rimr1 & DPMAIF_DL2_INT_BATCNT_LEN_ERR_MSK)
+ l2risar1 &= ~DPMAIF_DL2_INT_BATCNT_LEN_ERR;
+
+ return l2risar1;
+}
+
+static int mtk_dpmaif_drv_irq_src3(struct dpmaif_drv_info *drv_info,
+ struct dpmaif_drv_intr_info *intr_info)
+{
+ u32 val, ori_l2risar1, l2risar1, l2rimr1;
+
+ ori_l2risar1 = mtk_dpmaif_drv_get_dl_lv2_sts(drv_info, DPMAIF_DLQ2);
+ l2rimr1 = mtk_dpmaif_drv_get_dl_intr_mask(drv_info, DPMAIF_DLQ2);
+
+ /* filter care interrupt status. */
+ l2risar1 = ori_l2risar1 & (DPMAIF_DL2_INT_DLQ2_QDONE | DPMAIF_DL2_INT_BATCNT_LEN_ERR |
+ DPMAIF_DL2_INT_DLQ2_PITCNT_LEN_ERR);
+
+ if (l2risar1) {
+ /* Filter to get DL unmasked interrupts */
+ l2risar1 = mtk_dpmaif_drv_irq_src6_filter(drv_info, l2risar1, l2rimr1);
+
+ val = l2risar1 & DPMAIF_DL2_INT_BATCNT_LEN_ERR;
+ if (val) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_BATCNT_LEN_ERR;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_BAT1;
+ intr_info->intr_cnt++;
+ mtk_dpmaif_drv_mask_dl_batcnt_len_err_intr(drv_info, DPMAIF_BAT1);
+ }
+
+ val = l2risar1 & DPMAIF_DL2_INT_DLQ2_PITCNT_LEN_ERR;
+ if (val) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_PITCNT_LEN_ERR;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_DLQ2;
+ intr_info->intr_cnt++;
+ mtk_dpmaif_drv_dlq_mask_pit_cnt_len_err_intr(drv_info, DPMAIF_DLQ2);
+ }
+
+ val = l2risar1 & DPMAIF_DL2_INT_DLQ2_QDONE;
+ if (val) {
+ if (!mtk_dpmaif_drv_dlq_mask_rx_done_intr(drv_info, DPMAIF_DLQ2)) {
+ intr_info->intr_types[intr_info->intr_cnt] = DPMAIF_INTR_DL_DONE;
+ intr_info->intr_queues[intr_info->intr_cnt] = DPMAIF_DLQ2;
+ intr_info->intr_cnt++;
+ }
+ }
+
+ /* Clear interrupt status. */
+ mtk_pci_write32(drv_info->mdev, drv_info->regs->pd_base +
+ NRL2_DPMAIF_AP_MISC_APDL_L2TISAR1, l2risar1);
+ }
+
+ mtk_dpmaif_drv_clr_ip_busy_sts(drv_info);
+
+ return 0;
+}
+
+static int mtk_dpmaif_drv_init_m9xx(struct dpmaif_drv_info *drv_info, void *data)
+{
+ int ret;
+
+ mtk_dpmaif_drv_cfg_get(drv_info);
+ mtk_dpmaif_drv_reset(drv_info);
+ ret = mtk_dpmaif_drv_init_com(drv_info, data);
+
+ return ret;
+}
+
+struct dpmaif_drv_ops dpmaif_drv_ops_m9xx = {
+ .init = mtk_dpmaif_drv_init_m9xx,
+ .start_queue = mtk_dpmaif_drv_start_queue,
+ .stop_queue = mtk_dpmaif_drv_stop_queue,
+ .intr_handle = mtk_dpmaif_drv_intr_handle_com,
+ .intr_complete = mtk_dpmaif_drv_intr_complete_com,
+ .send_doorbell = mtk_dpmaif_drv_send_doorbell_com,
+ .get_ring_idx = mtk_dpmaif_drv_get_ring_idx,
+ .get_rx_info = mtk_dpmaif_get_rx_info,
+ .fill_tx_info = mtk_dpmaif_fill_tx_info,
+};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg.h
new file mode 100644
index 000000000000..6c40d4ff5a04
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg.h
@@ -0,0 +1,387 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DPMAIF_REG_H__
+#define __MTK_DPMAIF_REG_H__
+
+/* 2ms -> 2 * 1000 / 10 = 200 */
+#define POLL_MAX_TIMES 200
+#define POLL_INTERVAL_US 10
+#define REWRITE_TIMES 20
+
+/* dpmaif pd domain regisers */
+#define DPMAIF_AO_BASE 0
+#define BASE_NADDR_NRL2_DPMAIF_AO_UL ((unsigned long)(DPMAIF_AO_BASE))
+#define BASE_NADDR_NRL2_DPMAIF_AO_DL ((unsigned long)(DPMAIF_AO_BASE + 0x400))
+
+/* dpmaif uplink */
+#define NRL2_DPMAIF_AO_UL_INIT_SET (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x0)
+#define NRL2_DPMAIF_AO_UL_CHNL_ARB0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x1C)
+#define NRL2_DPMAIF_AO_UL_AP_L2TIMR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x80)
+#define NRL2_DPMAIF_AO_UL_AP_L2TIMCR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x84)
+#define NRL2_DPMAIF_AO_UL_AP_L2TIMSR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x88)
+#define NRL2_DPMAIF_AO_UL_AP_L1TIMR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x8C)
+#define NRL2_DPMAIF_AO_UL_APDL_L2TIMR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x90)
+#define NRL2_DPMAIF_AO_UL_APDL_L2TIMCR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x94)
+#define NRL2_DPMAIF_AO_UL_APDL_L2TIMSR0 (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x98)
+#define NRL2_DPMAIF_AO_UL_AP_DL_UL_IP_BUSY_MASK (BASE_NADDR_NRL2_DPMAIF_AO_UL + 0x9C)
+
+/* dpmaif downlink. */
+#define NRL2_DPMAIF_AO_DL_INIT_SET (BASE_NADDR_NRL2_DPMAIF_AO_DL + 0x0)
+#define NRL2_DPMAIF_AO_DL_LROPIT_INIT_CON5 (BASE_NADDR_NRL2_DPMAIF_AO_DL + 0x28)
+#define NRL2_DPMAIF_AO_DL_LROPIT_TRIG_THRES (BASE_NADDR_NRL2_DPMAIF_AO_DL + 0x34)
+
+/* dpmaif uplink config. */
+#define DPMAIF_PD_UL_CHNL_ARB0 NRL2_DPMAIF_AO_UL_CHNL_ARB0
+
+#define DPMAIF_PD_AP_UL_L2TIMR0 NRL2_DPMAIF_AO_UL_AP_L2TIMR0
+#define DPMAIF_PD_AP_UL_L2TICR0 NRL2_DPMAIF_AO_UL_AP_L2TIMCR0
+#define DPMAIF_PD_AP_UL_L2TISR0 NRL2_DPMAIF_AO_UL_AP_L2TIMSR0
+
+#define DPMAIF_PD_AP_DL_L2TIMR0 NRL2_DPMAIF_AO_UL_APDL_L2TIMR0
+#define DPMAIF_PD_AP_DL_L2TICR0 NRL2_DPMAIF_AO_UL_APDL_L2TIMCR0
+#define DPMAIF_PD_AP_DL_L2TISR0 NRL2_DPMAIF_AO_UL_APDL_L2TIMSR0
+
+#define DPMAIF_PD_AP_DLUL_IP_BUSY_MASK NRL2_DPMAIF_AO_UL_AP_DL_UL_IP_BUSY_MASK
+
+/* dpmaif pd domain regisers */
+#define DPMAIF_PD_BASE 0
+#define BASE_NADDR_NRL2_DPMAIF_UL ((unsigned long)(DPMAIF_PD_BASE))
+#define BASE_NADDR_NRL2_DPMAIF_DL ((unsigned long)(DPMAIF_PD_BASE + 0x100))
+#define BASE_NADDR_NRL2_DPMAIF_AP_MISC ((unsigned long)(DPMAIF_PD_BASE + 0x400))
+#define BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL ((unsigned long)(DPMAIF_PD_BASE + 0xD00))
+#define BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL ((unsigned long)(DPMAIF_PD_BASE + 0xC00))
+#define BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX ((unsigned long)(DPMAIF_PD_BASE + 0x900))
+#define BASE_NADDR_NRL2_DPMAIF_MMW_HPC ((unsigned long)(DPMAIF_PD_BASE + 0x600))
+
+/* dpmaif uplink . */
+#define NRL2_DPMAIF_UL_DBG_STA2 (BASE_NADDR_NRL2_DPMAIF_UL + 0x88)
+#define NRL2_DPMAIF_UL_RESERVE_AO_RW (BASE_NADDR_NRL2_DPMAIF_UL + 0xAC)
+#define NRL2_DPMAIF_UL_ADD_DESC_CH0 (BASE_NADDR_NRL2_DPMAIF_UL + 0xB0)
+#define NRL2_DPMAIF_UL_ADD_DESC_CH4 (BASE_NADDR_NRL2_DPMAIF_UL + 0xE0)
+
+/* dpmaif downlink. */
+#define NRL2_DPMAIF_DL_BAT_INIT (BASE_NADDR_NRL2_DPMAIF_DL + 0x00)
+#define NRL2_DPMAIF_DL_BAT_ADD (BASE_NADDR_NRL2_DPMAIF_DL + 0x04)
+#define NRL2_DPMAIF_DL_BAT_INIT_CON0 (BASE_NADDR_NRL2_DPMAIF_DL + 0x08)
+#define NRL2_DPMAIF_DL_BAT_INIT_CON1 (BASE_NADDR_NRL2_DPMAIF_DL + 0x0C)
+#define NRL2_DPMAIF_DL_PIT_INIT_CON5 (BASE_NADDR_NRL2_DPMAIF_DL + 0x14)
+#define NRL2_DPMAIF_DL_PIT_INIT (BASE_NADDR_NRL2_DPMAIF_DL + 0x20)
+#define NRL2_DPMAIF_DL_PIT_ADD (BASE_NADDR_NRL2_DPMAIF_DL + 0x24)
+#define NRL2_DPMAIF_DL_PIT_INIT_CON0 (BASE_NADDR_NRL2_DPMAIF_DL + 0x28)
+#define NRL2_DPMAIF_DL_PIT_INIT_CON1 (BASE_NADDR_NRL2_DPMAIF_DL + 0x2C)
+#define NRL2_DPMAIF_DL_PIT_INIT_CON2 (BASE_NADDR_NRL2_DPMAIF_DL + 0x30)
+#define NRL2_DPMAIF_DL_PIT_INIT_CON3 (BASE_NADDR_NRL2_DPMAIF_DL + 0x34)
+#define NRL2_DPMAIF_DL_BAT_INIT_CON3 (BASE_NADDR_NRL2_DPMAIF_DL + 0x50)
+#define NRL2_DPMAIF_DL_PIT_INIT_CON4 (BASE_NADDR_NRL2_DPMAIF_DL + 0x54)
+#define NRL2_DPMAIF_DL_UL_INTR_TRAS_CON (BASE_NADDR_NRL2_DPMAIF_DL + 0xAC)
+#define NRL2_DPMAIF_DL_DBG_STA1 (BASE_NADDR_NRL2_DPMAIF_DL + 0xB4)
+
+/* dpmaif misc. */
+#define NRL2_DPMAIF_AP_MISC_AP_L2TISAR0 (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0x00)
+#define NRL2_DPMAIF_AP_MISC_APDL_L2TISAR0 (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0x50)
+#define NRL2_DPMAIF_AP_MISC_AP_IP_BUSY (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0x60)
+#define NRL2_DPMAIF_AP_MISC_CG_EN (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0x68)
+#define NRL2_DPMAIF_AP_MISC_OVERWRITE_CFG (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0x90)
+#define NRL2_DPMAIF_AP_MISC_MEM_CLR (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0x94)
+
+/* dpmaif uplink sram. */
+#define NRL2_DPMAIF_AO_UL_CHNL0_CON0 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x10)
+#define NRL2_DPMAIF_AO_UL_CHNL0_CON1 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x14)
+#define NRL2_DPMAIF_AO_UL_CHNL0_CON2 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x18)
+
+/* dpmaif downlink sram. */
+#define NRL2_DPMAIF_AO_DL_PKTINFO_CON0 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x0)
+#define NRL2_DPMAIF_AO_DL_PKTINFO_CON1 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x4)
+#define NRL2_DPMAIF_AO_DL_PKTINFO_CON2 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x8)
+#define NRL2_DPMAIF_AO_DL_RDY_CHK_THRES (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0xC)
+#define NRL2_DPMAIF_AO_DL_RDY_CHK_FRG_THRES (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x10)
+#define NRL2_DPMAIF_AO_DL_LRO_AGG_CFG (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x20)
+#define NRL2_DPMAIF_AO_DL_LROPIT_TIMEOUT0 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x24)
+#define NRL2_DPMAIF_AO_DL_LROPIT_TIMEOUT1 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x28)
+#define NRL2_DPMAIF_AO_DL_HPC_CNTL (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x38)
+#define NRL2_DPMAIF_AO_DL_PIT_SEQ_END (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x40)
+#define NRL2_DPMAIF_AO_DL_BAT_STA2 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0xD8)
+#define NRL2_DPMAIF_AO_DL_BAT_STA3 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0xDC)
+#define NRL2_DPMAIF_AO_DL_PIT_STA2 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0xEC)
+#define NRL2_DPMAIF_AO_DL_PIT_STA3 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x60)
+#define NRL2_DPMAIF_AO_DL_FRGBAT_STA2 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x78)
+#define NRL2_DPMAIF_AO_DL_LRO_STA5 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0xA4)
+#define NRL2_DPMAIF_AO_DL_LRO_STA6 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0xA8)
+
+/* dpmaif hpc. */
+#define NRL2_DPMAIF_HPC_INTR_MASK (BASE_NADDR_NRL2_DPMAIF_MMW_HPC + 0xF4)
+
+/* dpmaif LRO. */
+#define NRL2_DPMAIF_DL_LROPIT_INIT (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x0)
+#define NRL2_DPMAIF_DL_LROPIT_ADD (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x10)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON0 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x14)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON1 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x18)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON2 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x1C)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON3 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x20)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON4 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x24)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON5 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x28)
+#define NRL2_DPMAIF_DL_LROPIT_INIT_CON6 (BASE_NADDR_NRL2_DPMAIF_DL_LRO_REMOVEAO_IDX + 0x2C)
+
+/* dpmaif uplink config. */
+#define DPMAIF_PD_UL_CHNL0_CON0 NRL2_DPMAIF_AO_UL_CHNL0_CON0
+#define DPMAIF_PD_UL_CHNL0_CON1 NRL2_DPMAIF_AO_UL_CHNL0_CON1
+#define DPMAIF_PD_UL_CHNL0_CON2 NRL2_DPMAIF_AO_UL_CHNL0_CON2
+#define DPMAIF_PD_UL_ADD_DESC_CH NRL2_DPMAIF_UL_ADD_DESC_CH0
+#define DPMAIF_PD_UL_DBG_STA2 NRL2_DPMAIF_UL_DBG_STA2
+
+/* dpmaif downlink config. */
+#define DPMAIF_PD_DL_BAT_INIT NRL2_DPMAIF_DL_BAT_INIT
+#define DPMAIF_PD_DL_BAT_ADD NRL2_DPMAIF_DL_BAT_ADD
+#define DPMAIF_PD_DL_BAT_INIT_CON0 NRL2_DPMAIF_DL_BAT_INIT_CON0
+#define DPMAIF_PD_DL_BAT_INIT_CON1 NRL2_DPMAIF_DL_BAT_INIT_CON1
+#define DPMAIF_PD_DL_BAT_INIT_CON3 NRL2_DPMAIF_DL_BAT_INIT_CON3
+#define DPMAIF_PD_DL_DBG_STA1 NRL2_DPMAIF_DL_DBG_STA1
+#define DPMAIF_PD_DL_UL_TRAS_INTR_CON NRL2_DPMAIF_DL_UL_INTR_TRAS_CON
+
+/* dpmaif pd ap misc, ul misc config. */
+#define DPMAIF_PD_AP_UL_L2TISAR0 NRL2_DPMAIF_AP_MISC_AP_L2TISAR0
+#define DPMAIF_PD_AP_DL_L2TISAR0 NRL2_DPMAIF_AP_MISC_APDL_L2TISAR0
+#define DPMAIF_PD_AP_IP_BUSY NRL2_DPMAIF_AP_MISC_AP_IP_BUSY
+
+/* dpmaif dl config. */
+#define DPMAIF_AO_DL_PKTINFO_CONO NRL2_DPMAIF_AO_DL_PKTINFO_CON0
+#define DPMAIF_AO_DL_PKTINFO_CON1 NRL2_DPMAIF_AO_DL_PKTINFO_CON1
+#define DPMAIF_AO_DL_PKTINFO_CON2 NRL2_DPMAIF_AO_DL_PKTINFO_CON2
+#define DPMAIF_AO_DL_RDY_CHK_THRES NRL2_DPMAIF_AO_DL_RDY_CHK_THRES
+#define DPMAIF_AO_DL_BAT_STA2 NRL2_DPMAIF_AO_DL_BAT_STA2
+#define DPMAIF_AO_DL_BAT_STA3 NRL2_DPMAIF_AO_DL_BAT_STA3
+#define DPMAIF_AO_DL_PIT_STA2 NRL2_DPMAIF_AO_DL_PIT_STA2
+#define DPMAIF_AO_DL_PIT_STA3 NRL2_DPMAIF_AO_DL_PIT_STA3
+#define DPMAIF_AO_DL_FRG_CHK_THRES NRL2_DPMAIF_AO_DL_RDY_CHK_FRG_THRES
+#define DPMAIF_AO_DL_FRG_STA2 NRL2_DPMAIF_AO_DL_FRGBAT_STA2
+
+#define NRL2_DPMAIF_DLY_IRQ_TIMER3 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x1C)
+#define NRL2_DPMAIF_DLY_IRQ_TIMER4 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x2C)
+#define NRL2_DPMAIF_DLY_IRQ_TIMER5 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x3C)
+#define NRL2_DPMAIF_DLY_IRQ_TIMER6 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x60)
+#define NRL2_DPMAIF_DLY_IRQ_TIMER7 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0x64)
+#define NRL2_DPMAIF_AO_DL_DLY_IRQ_TIMER1 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x58)
+#define NRL2_DPMAIF_AO_DL_DLY_IRQ_TIMER2 (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_DL + 0x5C)
+
+/* dpmaif dl2 config. */
+#define BASE_ADDR_NRL2_DPMAIF_DL_2_CFG 0x1000UL
+
+#define BASE_ADDR_MMW_DPMAIF_PD_MISC2_CFG 0xF00UL
+
+#define BASE_ADDR_NRL2_DPMAIF_AP_MISC_CFG 0x400UL
+#define BASE_ADDR_MMW_DPMAIF_PD_MISC_CFG 0xE00UL
+#define BASE_ADDR_NRL2_DPMAIF_DL_2_CFG_SRAM 0UL
+
+#define DPMAIF_DL_RESOURCE_DOMAIN (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG_SRAM + 0x8)
+
+#define DPMAIF_DL_2_BAT_INIT_CON0 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x08)
+#define DPMAIF_DL_2_BAT_INIT_CON3 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x50)
+#define DPMAIF_DL_2_BAT_INIT_CON1 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x0C)
+#define DPMAIF_DL_2_BAT_INIT (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x00)
+#define DPMAIF_DL_2_PIT_INIT_CON0 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x28)
+#define DPMAIF_DL_2_PIT_INIT_CON4 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x54)
+#define DPMAIF_DL_2_PIT_INIT_CON1 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x2C)
+#define DPMAIF_DL_2_PIT_INIT_CON3 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x34)
+#define DPMAIF_DL_2_PIT_INIT (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x20)
+#define DPMAIF_DL_2_STA13 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x18)
+#define DPMAIF_DL_2_STA14 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x1C)
+#define DPMAIF_DL_2_STA4 (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x90)
+#define DPMAIF_DL_2_BAT_ADD (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x04)
+#define DPMAIF_DL_2_PIT_ADD (BASE_ADDR_NRL2_DPMAIF_DL_2_CFG + 0x24)
+
+#define NRL2_DPMAIF_AP_MISC_APDL_L2TISAR1 (BASE_ADDR_NRL2_DPMAIF_AP_MISC_CFG + 0x54)
+
+#define NRL2_DPMAIF_MISC_PD_APDL12_MASK_SET (BASE_ADDR_MMW_DPMAIF_PD_MISC_CFG + 0x60)
+#define NRL2_DPMAIF_MISC_PD_APDL12_MASK_CLR (BASE_ADDR_MMW_DPMAIF_PD_MISC_CFG + 0x64)
+#define NRL2_DPMAIF_MISC_PD_APDL12_MASK_RW (BASE_ADDR_MMW_DPMAIF_PD_MISC_CFG + 0x68)
+
+#define DPMAIF_ULQS 0x1F
+#define DPMAIF_UL_ADD_NOT_READY BIT(31)
+#define DPMAIF_UL_ADD_UPDATE BIT(31)
+#define DPMAIF_UL_ALL_QUE_ARB_EN (DPMAIF_ULQS << 8)
+
+#define DPMAIF_DL_ADD_UPDATE BIT(31)
+#define DPMAIF_DL_ADD_NOT_READY BIT(31)
+#define DPMAIF_DL_FRG_ADD_UPDATE BIT(16)
+
+#define DPMAIF_DL_BAT_INIT_ALLSET BIT(0)
+#define DPMAIF_DL_BAT_FRG_INIT BIT(16)
+#define DPMAIF_DL_BAT_INIT_EN BIT(31)
+#define DPMAIF_DL_BAT_INIT_NOT_READY BIT(31)
+#define DPMAIF_DL_BAT_INIT_ONLY_ENABLE_BIT 0
+
+#define DPMAIF_DL_PIT_INIT_ALLSET BIT(0)
+#define DPMAIF_DL_PIT_INIT_EN BIT(31)
+#define DPMAIF_DL_PIT_INIT_NOT_READY BIT(31)
+
+#define DPMAIF_PKT_ALIGN64_MODE 0
+#define DPMAIF_PKT_ALIGN128_MODE 1
+
+#define DPMAIF_BAT_REMAIN_SZ_BASE 16
+#define DPMAIF_BAT_BUFFER_SZ_BASE 128
+#define DPMAIF_FRG_BUFFER_SZ_BASE 128
+
+#define DPMAIF_PIT_SIZE_MSK 0x3FFFF
+
+#define DPMAIF_BAT_EN_MSK BIT(16)
+#define DPMAIF_FRG_EN_MSK BIT(28)
+#define DPMAIF_BAT_SIZE_MSK 0xFFFF
+
+#define DPMAIF_BAT_BID_MAXCNT_MSK 0xFFFF0000
+#define DPMAIF_BAT_REMAIN_MINSZ_MSK 0x0000FF00
+#define DPMAIF_PIT_CHK_NUM_MSK 0xFF000000
+#define DPMAIF_BAT_BUF_SZ_MSK 0x0001FF00
+#define DPMAIF_FRG_BUF_SZ_MSK 0x0001FF00
+#define DPMAIF_BAT_RSV_LEN_MSK 0x000000FF
+#define DPMAIF_PKT_ALIGN_MSK (0x3 << 22)
+
+#define DPMAIF_BAT_CHECK_THRES_MSK (0x3F << 16)
+#define DPMAIF_FRG_CHECK_THRES_MSK 0xFF
+#define DPMAIF_PKT_ALIGN_EN BIT(23)
+#define DPMAIF_DRB_SIZE_MSK 0x0000FFFF
+
+#define DPMAIF_DL_PIT_WRIDX_MSK 0x3FFFF
+#define DPMAIF_DL_BAT_WRIDX_MSK 0x3FFFF
+#define DPMAIF_DL_FRG_WRIDX_MSK 0x3FFFF
+
+/* DPMAIF_PD_UL_DBG_STA2 */
+#define DPMAIF_UL_IDLE_STS_MSK BIT(11)
+#define DPMAIF_UL_IDLE_STS BIT(11)
+
+/* DPMAIF_PD_DL_DBG_STA1 */
+#define DPMAIF_DL_IDLE_STS BIT(23)
+#define DPMAIF_DL_PKT_CHECKSUM_EN BIT(31)
+#define DPMAIF_PORT_MODE_MSK BIT(30)
+#define DPMAIF_PORT_MODE_PCIE BIT(30)
+
+/* BASE_NADDR_NRL2_DPMAIF_WDMA */
+#define DPMAIF_DL_BAT_CACHE_PRI BIT(22)
+#define DPMAIF_DL_BURST_PIT_EN BIT(13)
+#define DPMAIF_MEM_CLR_MASK BIT(0)
+#define DPMAIF_SRAM_SYNC_MASK BIT(0)
+#define DPMAIF_UL_INIT_DONE_MASK BIT(0)
+#define DPMAIF_DL_INIT_DONE_MASK BIT(0)
+
+#define DPMAIF_DL_PIT_SEQ_MSK 0xFF
+#define DPMAIF_PCIE_MODE_SET_VALUE 0x55
+
+#define DPMAIF_UDL_IP_BUSY_MSK BIT(0)
+
+#define DP_UL_INT_DONE_OFFSET 0
+#define DP_UL_INT_EMPTY_OFFSET 5
+#define DP_UL_INT_MD_NOTRDY_OFFSET 10
+#define DP_UL_INT_PWR_NOTRDY_OFFSET 15
+#define DP_UL_INT_LEN_ERR_OFFSET 20
+
+/* Enable and mask/unmaks UL interrupt mask */
+#define DPMAIF_UL_INT_QDONE_MSK (DPMAIF_ULQS << DP_UL_INT_DONE_OFFSET)
+#define DPMAIF_UL_TOP0_INT_MSK BIT(9)
+#define DPMAIF_UL_INT_VALID_MSK 0x1FFFFFF
+#define DPMAIF_UL3_INT_VALID_MSK 0x3FFFFFFF
+
+/* UL interrupt status */
+#define DPMAIF_UL_INT_QDONE (DPMAIF_ULQS << DP_UL_INT_DONE_OFFSET)
+
+/* Enable and Mask/unmask DL interrupt mask */
+#define DPMAIF_DL_INT_DLQ_QDONE_MSK BIT(0)
+#define DPMAIF_DL_INT_BATCNT_LEN_ERR_MSK BIT(2)
+#define DPMAIF_DL_INT_DLQ_PITCNT_LEN_ERR_MSK BIT(3)
+#define DPMAIF_DL_INT_DLQ0_QDONE_MSK BIT(8)
+#define DPMAIF_DL_INT_DLQ1_QDONE_MSK BIT(9)
+#define DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR_MSK BIT(10)
+#define DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR_MSK BIT(11)
+#define DPMAIF_DL_INT_Q2TOQ1_MSK BIT(24)
+#define DPMAIF_DL_INT_Q2APTOP_MSK BIT(25)
+#define DPMAIF_DL_INT_VALID_MSK 0x1FFF
+
+/* DL interrupt status */
+#define DPMAIF_DL_INT_DLQ_QDONE BIT(0)
+#define DPMAIF_DL_INT_BATCNT_LEN_ERR BIT(2)
+#define DPMAIF_DL_INT_DLQ_PITCNT_LEN_ERR BIT(3)
+#define DPMAIF_DL_INT_DLQ0_PITCNT_LEN_ERR BIT(8)
+#define DPMAIF_DL_INT_DLQ1_PITCNT_LEN_ERR BIT(9)
+#define DPMAIF_DL_INT_DLQ0_QDONE BIT(13)
+#define DPMAIF_DL_INT_DLQ1_QDONE BIT(14)
+
+/* DPMAIF LRO HW configure */
+#define DPMAIF_HPC_LRO_PATH_DF 3
+
+/* 0: HPC rules add by HW; 1: HPC rules add by Host */
+#define DPMAIF_HPC_ADD_MODE_DF 0
+#define DPMAIF_HPC_NUM_DF 8
+#define DPMAIF_AGG_MAX_LEN_DF 65535
+#define DPMAIF_AGG_TBL_ENT_NUM_DF 50
+#define DPMAIF_HASH_PRIME_DF 13
+#define DPMAIF_MID_TIMEOUT_THRES_DF 100
+#define DPMAIF_LRO_TIMEOUT_THRES_DF 100
+#define DPMAIF_LRO_PRS_THRES_DF 10
+#define DPMAIF_LRO_HASH_BIT_CHOOSE_DF 0
+
+#define DPMAIF_LROPIT_EN_MSK 0x100000
+#define DPMAIF_LROPIT_CHAN_OFS 16
+#define DPMAIF_ADD_LRO_PIT_CHAN_OFS 20
+
+#define DPMAIF_DL_PIT_BYTE_SIZE 16
+#define DPMAIF_DL_BAT_BYTE_SIZE 8
+#define DPMAIF_DL_FRG_BYTE_SIZE 8
+#define DPMAIF_UL_DRB_BYTE_SIZE 16
+
+#define DPMAIF_UL_DRB_ENTRY_WORD (DPMAIF_UL_DRB_BYTE_SIZE >> 2)
+#define DPMAIF_DL_PIT_ENTRY_WORD (DPMAIF_DL_PIT_BYTE_SIZE >> 2)
+#define DPMAIF_DL_BAT_ENTRY_WORD (DPMAIF_DL_BAT_BYTE_SIZE >> 2)
+
+#define DPMAIF_HW_BAT_REMAIN 64
+#define DPMAIF_HW_PKT_BIDCNT 1
+
+#define DPMAIF_HW_CHK_BAT_NUM 62
+#define DPMAIF_HW_CHK_FRG_NUM DPMAIF_HW_CHK_BAT_NUM
+#define DPMAIF_HW_CHK_PIT_NUM (2 * DPMAIF_HW_CHK_BAT_NUM)
+
+#define DPMAIF_DL_2_BAT_WRIDX_MSK 0xFFFF
+#define DPMAIF_DL2_PIT_EN_MSK BIT(0)
+
+/* Mask/unmask DL2 interrupt mask */
+#define DPMAIF_DL_INT_DLQ2_QDONE_MSK BIT(0)
+#define DPMAIF_DL2_INT_BATCNT_LEN_ERR_MSK BIT(2)
+#define DPMAIF_DL_INT_DLQ2_PITCNT_LEN_ERR_MSK BIT(3)
+#define DPMAIF_DL2_INT_OTHER_MSK 0xFF00FF
+
+/* DL2 interrupt status */
+#define DPMAIF_DL2_INT_DLQ2_QDONE BIT(0)
+#define DPMAIF_DL2_INT_BATCNT_LEN_ERR BIT(2)
+#define DPMAIF_DL2_INT_DLQ2_PITCNT_LEN_ERR BIT(3)
+#define DLQ2_INTR_MSK 0xFF
+
+#define DPMAIF_HASH_DEFAULT_VALUE 0
+#define DPMAIF_HASH_DEFAULT_V_MASK 0xFFFFFF00
+
+void mtk_dpmaif_drv_ul_mask_intr(struct dpmaif_drv_info *drv_info, u32 mask);
+void mtk_dpmaif_drv_dlq_mask_pit_cnt_len_err_intr(struct dpmaif_drv_info *drv_info, u8 qno);
+void mtk_dpmaif_drv_mask_dl_batcnt_len_err_intr(struct dpmaif_drv_info *drv_info, u8 bat_id);
+void mtk_dpmaif_drv_ul_mask_multi_tx_done_intr(struct dpmaif_drv_info *drv_info, u8 q_mask);
+u32 mtk_dpmaif_drv_get_dl_lv2_sts(struct dpmaif_drv_info *drv_info, u8 q_id);
+u32 mtk_dpmaif_drv_get_dl_intr_mask(struct dpmaif_drv_info *drv_info, u8 q_id);
+int mtk_dpmaif_drv_dlq_mask_rx_done_intr(struct dpmaif_drv_info *drv_info, u8 q_id);
+u32 mtk_dpmaif_drv_get_ul_lv2_sts(struct dpmaif_drv_info *drv_info);
+void mtk_dpmaif_drv_clr_ip_busy_sts(struct dpmaif_drv_info *drv_info);
+int mtk_dpmaif_drv_ul_intr_init(struct dpmaif_drv_info *drv_info);
+
+u32 mtk_dpmaif_drv_get_ul_intr_mask(struct dpmaif_drv_info *drv_info);
+
+int mtk_dpmaif_drv_init_com(struct dpmaif_drv_info *drv_info, void *data);
+int mtk_dpmaif_drv_start_queue(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_dir dir);
+int mtk_dpmaif_drv_stop_queue(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_dir dir);
+int mtk_dpmaif_drv_intr_complete_com(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_intr_type type, u8 id, u64 data);
+int mtk_dpmaif_drv_send_doorbell_com(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_ring_type type, u8 id, u32 cnt);
+int mtk_dpmaif_drv_get_ring_idx(struct dpmaif_drv_info *drv_info,
+ enum dpmaif_drv_ring_idx index, u8 q_id);
+int mtk_dpmaif_drv_intr_handle_com(struct dpmaif_drv_info *drv_info, void *data, u8 irq_id);
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg_m9xx.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg_m9xx.h
new file mode 100644
index 000000000000..808d0cc0e6dc
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg_m9xx.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __MTK_DPMAIF_DRV_M9XX_H__
+#define __MTK_DPMAIF_DRV_M9XX_H__
+
+#include "mtk_dpmaif_reg.h"
+
+#define DPMAIF_DEV_PD_BASE 0x1022D000
+#define DPMAIF_DEV_PD2_BASE 0x10260000
+#define DPMAIF_DEV_AO_BASE 0x10011000
+
+#define NRL2_DPMAIF_AO_UL_CH0_STA (BASE_NADDR_NRL2_DPMAIF_PD_SRAM_UL + 0xE0)
+
+/* DPMAIF AO register */
+#define DPMAIF_AP_RGU_ASSERT 0x10001120
+#define DPMAIF_AP_RGU_DEASSERT 0x10001124
+#define DPMAIF_AP_RST_BIT BIT(4)
+#define DPMAIF_AP_AO_RGU_ASSERT 0x10001140
+#define DPMAIF_AP_AO_RGU_DEASSERT 0x10001144
+#define DPMAIF_AP_AO_RST_BIT BIT(3)
+#define DPMAIF_AP_INFRA_CG_CLR 0x100010E0
+#define DPMAIF_AP_INFRA_CG_SET 0x100010E4
+#define DPMAIF_AP_INFRA_CG_BIT BIT(15)
+#define DPMAIF_AP_INFRA_GLITCH_PORT 0x10001208
+#define DPMAIF_AP_GLITCH_PROT_BIT BIT(15)
+#define DPMAIF_AP_PD_SRAM_EN_BIT BIT(31)
+
+#define NRL2_DPMAIF_AP_MISC_SRAM_INIT_SET0 (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0xC0)
+#define NRL2_DPMAIF_AP_MISC_SRAM_INIT_SET1 (BASE_NADDR_NRL2_DPMAIF_AP_MISC + 0xC4)
+#define DPMAIF_PD_AP_UL_L2TIMR0_NEXT (BASE_ADDR_MMW_DPMAIF_PD_MISC2_CFG + 0x60)
+#define DPMAIF_PD_AP_UL_L2TICR0_NEXT (BASE_ADDR_MMW_DPMAIF_PD_MISC2_CFG + 0x5C)
+#define DPMAIF_PD_AP_UL_L2TISR0_NEXT (BASE_ADDR_MMW_DPMAIF_PD_MISC2_CFG + 0x58)
+
+#define NRL2_DPMAIF_PD_DL_LROPIT_TIMEOUT5 0x40
+
+#define DPMAIF_HPC_NUM_EXT 8
+#define DPMAIF_HPC_NUM_M9XX (DPMAIF_HPC_NUM_DF + DPMAIF_HPC_NUM_EXT)
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c
new file mode 100644
index 000000000000..a8e183b5f4ee
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#define pr_fmt(fmt) "DATA_RING:" fmt
+
+#include <linux/delay.h>
+#include <linux/bitfield.h>
+#include <linux/kernel.h>
+#include <linux/printk.h>
+#include "mtk_dpmaif_drv.h"
+#include "mtk_dpmaif_ring.h"
+
+#define PIT_PD_DATA_LEN GENMASK(31, 16) /* Indicates the data length of current packet. */
+#define PIT_PD_BUF_ID GENMASK(15, 3) /* The low order of buffer index */
+#define PIT_PD_BUF_TYPE BIT(2) /* 0b: normal BAT entry; 1b: fragment BAT entry */
+#define PIT_PD_CONT BIT(1) /* 0b: last entry; 1b: more entry */
+#define PIT_PD_PKT_TYPE BIT(0) /* 0b: normal PIT entry; 1b: message PIT entry */
+
+#define PIT_PD_DLQ_DONE GENMASK(31, 30)
+#define PIT_PD_ULQ_DONE GENMASK(29, 24)
+/* The header length of transport layer and internet layer. */
+#define PIT_PD_HD_OFFSET GENMASK(23, 19)
+#define PIT_PD_BI_F GENMASK(18, 17)
+#define PIT_PD_IG BIT(16)
+#define PIT_PD_RSV GENMASK(15, 11)
+#define PIT_PD_H_BID GENMASK(10, 8) /* The high order of buffer index */
+#define PIT_PD_SEQ GENMASK(7, 0) /* PIT sequence */
+
+#define PIT_MSG_DP BIT(31) /* Indicates software to drop this packet if set. */
+#define PIT_MSG_DW1_RSV1 GENMASK(30, 27)
+#define PIT_MSG_NET_TYPE GENMASK(26, 24)
+#define PIT_MSG_CHNL_ID GENMASK(23, 16) /* channel index */
+#define PIT_MSG_DW1_RSV2 GENMASK(15, 12)
+#define PIT_MSG_HPC_IDX GENMASK(11, 8)
+#define PIT_MSG_SRC_QID GENMASK(7, 5)
+#define PIT_MSG_ERR BIT(4)
+#define PIT_MSG_CHECKSUM GENMASK(3, 2)
+#define PIT_MSG_CONT BIT(1) /* 0b: last entry; 1b: more entry */
+#define PIT_MSG_PKT_TYPE BIT(0) /* 0b: normal PIT entry; 1b: message PIT entry */
+
+#define PIT_MSG_HP_IDX GENMASK(31, 27)
+#define PIT_MSG_CMD GENMASK(26, 24)
+#define PIT_MSG_DW2_RSV GENMASK(23, 21)
+#define PIT_MSG_FLOW GENMASK(20, 16)
+#define PIT_MSG_COUNT_L GENMASK(15, 0)
+
+#define PIT_MSG_HASH GENMASK(31, 24) /* Hash value calculated by Hardware using packet */
+#define PIT_MSG_DW3_RSV1 GENMASK(23, 18)
+#define PIT_MSG_PRO GENMASK(17, 16)
+#define PIT_MSG_VBID GENMASK(15, 3)
+#define PIT_MSG_DW3_RSV2 GENMASK(2, 0)
+
+#define PIT_MSG_DLQ_DONE GENMASK(31, 30)
+#define PIT_MSG_ULQ_DONE GENMASK(29, 24)
+#define PIT_MSG_IP BIT(23)
+#define PIT_MSG_DW4_RSV1 BIT(22)
+#define PIT_MSG_MR GENMASK(21, 20)
+#define PIT_MSG_DW4_RSV2 GENMASK(19, 17)
+#define PIT_MSG_IG BIT(16)
+#define PIT_MSG_DW4_RSV3 GENMASK(15, 11)
+#define PIT_MSG_H_BID GENMASK(10, 8)
+#define PIT_MSG_PIT_SEQ GENMASK(7, 0)
+
+#define DPMAIF_POLL_STEP 20
+#define DPMAIF_POLL_PIT_CNT_MAX 100
+
+#define DRB_MSG_PKT_LEN GENMASK(31, 16) /* The length of a whole packet. */
+#define DRB_MSG_DW1_RSV GENMASK(15, 3)
+#define DRB_MSG_CONT BIT(2) /* 0b: last entry; 1b: more entry */
+#define DRB_MSG_DTYP GENMASK(1, 0) /* 00b: normal DRB entry; 01b: message DRB entry */
+
+#define DRB_MSG_DW2_RSV1 GENMASK(31, 30)
+#define DRB_MSG_L4_CHK BIT(29) /* 0b: disable layer4 checksum offload; 1b: enable */
+#define DRB_MSG_IP_CHK BIT(28) /* 0b: disable IP checksum, 1b: enable IP checksum */
+#define DRB_MSG_DW2_RSV2 BIT(27)
+#define DRB_MSG_NET_TYPE GENMASK(26, 24)
+#define DRB_MSG_CHNL_ID GENMASK(23, 16) /* channel index */
+#define DRB_MSG_COUNT_L GENMASK(15, 0)
+
+#define DRB_PD_DATA_LEN GENMASK(31, 16) /* the length of a payload. */
+#define DRB_PD_RSV GENMASK(15, 3)
+#define DRB_PD_CONT BIT(2)/* 0b: last entry; 1b: more entry */
+#define DRB_PD_DTYP GENMASK(1, 0) /* 00b: normal DRB entry; 01b: message DRB entry. */
+
+int mtk_dpmaif_get_rx_info(void *pit, struct dpmaif_rx_info *rx_info, u32 pit_seq_expect,
+ u8 q_id)
+{
+ struct dpmaif_pd_pit *pd_pit = (struct dpmaif_pd_pit *)pit;
+ int ret = -DATA_PIT_SEQ_CHK_FAIL;
+ struct dpmaif_msg_pit *msg_pit;
+ u64 dma_addr;
+ u32 cnt = 0;
+
+ /* The longest check time is 2ms, step is 20us */
+ do {
+ rx_info->pit_pd_seq = FIELD_GET(PIT_PD_SEQ, le32_to_cpu(pd_pit->pd_footer));
+ if (rx_info->pit_pd_seq == pit_seq_expect) {
+ ret = 0;
+ break;
+ }
+
+ udelay(DPMAIF_POLL_STEP);
+ } while (++cnt < DPMAIF_POLL_PIT_CNT_MAX);
+
+ if (unlikely(ret < 0))
+ goto out;
+
+ rx_info->msg_pit = FIELD_GET(PIT_PD_PKT_TYPE, le32_to_cpu(pd_pit->pd_header));
+ if (rx_info->msg_pit) {
+ msg_pit = (struct dpmaif_msg_pit *)pit;
+ rx_info->pit_msg_chnl_id = FIELD_GET(PIT_MSG_CHNL_ID, le32_to_cpu(msg_pit->dword1));
+ rx_info->pit_msg_checksum = FIELD_GET(PIT_MSG_CHECKSUM,
+ le32_to_cpu(msg_pit->dword1));
+ rx_info->pit_msg_err = FIELD_GET(PIT_MSG_ERR, le32_to_cpu(msg_pit->dword1));
+ rx_info->pit_msg_dp = FIELD_GET(PIT_MSG_DP, le32_to_cpu(msg_pit->dword1));
+ rx_info->pit_msg_hash = FIELD_GET(PIT_MSG_HASH, le32_to_cpu(msg_pit->dword3));
+ rx_info->pit_msg_pro = FIELD_GET(PIT_MSG_PRO, le32_to_cpu(msg_pit->dword3));
+ rx_info->pit_msg_ip = FIELD_GET(PIT_MSG_IP, le32_to_cpu(msg_pit->dword4));
+ } else {
+ rx_info->normal_bat = FIELD_GET(PIT_PD_BUF_TYPE, le32_to_cpu(pd_pit->pd_header));
+ rx_info->pit_pd_cur_bid = (FIELD_GET(PIT_PD_H_BID,
+ le32_to_cpu(pd_pit->pd_footer)) << 13) +
+ FIELD_GET(PIT_PD_BUF_ID, le32_to_cpu(pd_pit->pd_header));
+ rx_info->pit_pd_data_len = FIELD_GET(PIT_PD_DATA_LEN,
+ le32_to_cpu(pd_pit->pd_header));
+ rx_info->pit_pd_hd_offset = FIELD_GET(PIT_PD_HD_OFFSET,
+ le32_to_cpu(pd_pit->pd_footer)) << 2;
+ rx_info->pit_continue = FIELD_GET(PIT_PD_CONT,
+ le32_to_cpu(pd_pit->pd_header));
+ dma_addr = le32_to_cpu(pd_pit->addr_high);
+ rx_info->pit_pd_dma_addr = (dma_addr << 32) + le32_to_cpu(pd_pit->addr_low);
+ }
+
+out:
+ return ret;
+}
+
+void mtk_dpmaif_fill_tx_info(void *drb, struct dpmaif_tx_info *tx_info, int type)
+{
+ struct dpmaif_msg_drb *msg_drb;
+ struct dpmaif_pd_drb *pd_drb;
+
+ if (type == MSG_DRB) {
+ msg_drb = (struct dpmaif_msg_drb *)drb;
+ msg_drb->msg_header1 = cpu_to_le32(FIELD_PREP(DRB_MSG_DTYP, MSG_DRB) |
+ FIELD_PREP(DRB_MSG_CONT, DPMAIF_DRB_MORE) |
+ FIELD_PREP(DRB_MSG_PKT_LEN, tx_info->msg_pkt_len));
+ msg_drb->msg_header2 = cpu_to_le32(FIELD_PREP(DRB_MSG_COUNT_L,
+ tx_info->msg_count_l) |
+ FIELD_PREP(DRB_MSG_CHNL_ID, tx_info->msg_channel_id) |
+ FIELD_PREP(DRB_MSG_L4_CHK, tx_info->msg_txcsum) |
+ FIELD_PREP(DRB_MSG_NET_TYPE, tx_info->msg_network_type));
+ } else {
+ pd_drb = (struct dpmaif_pd_drb *)drb;
+ pd_drb->pd_header = cpu_to_le32(FIELD_PREP(DRB_PD_DTYP, PD_DRB));
+ if (tx_info->pd_is_last)
+ pd_drb->pd_header |= cpu_to_le32(FIELD_PREP(DRB_PD_CONT,
+ DPMAIF_DRB_LASTONE));
+ else
+ pd_drb->pd_header |= cpu_to_le32(FIELD_PREP(DRB_PD_CONT, DPMAIF_DRB_MORE));
+
+ pd_drb->pd_header |= cpu_to_le32(FIELD_PREP(DRB_PD_DATA_LEN, tx_info->pd_data_len));
+ pd_drb->addr_low = cpu_to_le32(lower_32_bits(tx_info->pd_data_dma_addr));
+ pd_drb->addr_high = cpu_to_le32(upper_32_bits(tx_info->pd_data_dma_addr));
+ }
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.h
new file mode 100644
index 000000000000..20db9be37cac
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.h
@@ -0,0 +1,161 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DPMAIF_RING_H__
+#define __MTK_DPMAIF_RING_H__
+
+#include <linux/bitops.h>
+#include <linux/types.h>
+
+/* drb c_bit */
+#define DPMAIF_DRB_LASTONE 0x00
+#define DPMAIF_DRB_MORE 0x01
+
+/* pit c_bit */
+#define DPMAIF_PIT_LASTONE 0x00
+#define DPMAIF_PIT_MORE 0x01
+
+/* pit Pro_bit */
+#define DPMAIF_PIT_TCP 0x01
+#define DPMAIF_PIT_UDP 0x02
+
+/* pit IP_bit */
+#define DPMAIF_PIT_IPV4 0x00
+#define DPMAIF_PIT_IPV6 0x01
+
+enum dpmaif_rcsum_state {
+ CS_RESULT_INVALID = -1,
+ CS_RESULT_PASS = 0,
+ CS_RESULT_FAIL = 1,
+ CS_RESULT_NOTSUPP = 2,
+ CS_RESULT_RSV = 3
+};
+
+/* pit type */
+enum dpmaif_pit_type {
+ PD_PIT = 0,
+ MSG_PIT,
+};
+
+/* RX: buffer address table */
+struct dpmaif_bat {
+ __le32 buf_addr_low;
+ __le32 buf_addr_high;
+};
+
+struct dpmaif_rx_info {
+ u32 pit_pd_seq;
+ u32 msg_pit;
+ u32 pit_msg_chnl_id;
+ u32 pit_msg_checksum;
+ u32 pit_msg_err;
+ u32 pit_msg_dp;
+ u32 pit_msg_hash;
+ u32 pit_msg_pro;
+ u32 pit_msg_ip;
+ u32 normal_bat;
+ u32 pit_pd_cur_bid;
+ u32 pit_pd_data_len;
+ u32 pit_pd_hd_offset;
+ u32 pit_continue;
+ u64 pit_pd_dma_addr;
+};
+
+struct dpmaif_pd_pit {
+ __le32 pd_header;
+ __le32 addr_low;
+ __le32 addr_high;
+ __le32 pd_footer;
+};
+
+struct dpmaif_msg_pit {
+ __le32 dword1;
+ __le32 dword2;
+ __le32 dword3;
+ __le32 dword4;
+};
+
+struct dpmaif_tx_info {
+ u32 msg_pkt_len;
+ u16 msg_count_l;
+ u16 msg_network_type;
+ u8 msg_channel_id;
+ u8 msg_txcsum;
+ dma_addr_t pd_data_dma_addr;
+ u32 pd_data_len;
+ u8 pd_is_last;
+};
+
+struct dpmaif_msg_drb {
+ __le32 msg_header1;
+ __le32 msg_header2;
+ __le32 msg_rsv1;
+ __le32 msg_rsv2;
+};
+
+struct dpmaif_pd_drb {
+ __le32 pd_header;
+ __le32 addr_low;
+ __le32 addr_high;
+ __le32 pd_rsv;
+};
+
+/* drb->type */
+enum dpmaif_drb_type {
+ PD_DRB,
+ MSG_DRB,
+};
+
+static inline unsigned int mtk_dpmaif_ring_buf_get_next_idx(unsigned int buf_len,
+ unsigned int buf_idx)
+{
+ buf_idx++;
+
+ return buf_idx < buf_len ? buf_idx : 0;
+}
+
+static inline unsigned int mtk_dpmaif_ring_buf_readable(unsigned int total_cnt, unsigned int rd_idx,
+ unsigned int wr_idx)
+{
+ unsigned int pkt_cnt;
+
+ if (wr_idx >= rd_idx)
+ pkt_cnt = wr_idx - rd_idx;
+ else
+ pkt_cnt = total_cnt + wr_idx - rd_idx;
+
+ return pkt_cnt;
+}
+
+static inline unsigned int mtk_dpmaif_ring_buf_writable(unsigned int total_cnt,
+ unsigned int rel_idx, unsigned int wr_idx)
+{
+ unsigned int pkt_cnt;
+
+ if (wr_idx < rel_idx)
+ pkt_cnt = rel_idx - wr_idx - 1;
+ else
+ pkt_cnt = total_cnt + rel_idx - wr_idx - 1;
+
+ return pkt_cnt;
+}
+
+static inline unsigned int mtk_dpmaif_ring_buf_releasable(unsigned int total_cnt,
+ unsigned int rel_idx, unsigned int rd_idx)
+{
+ unsigned int pkt_cnt;
+
+ if (rel_idx <= rd_idx)
+ pkt_cnt = rd_idx - rel_idx;
+ else
+ pkt_cnt = total_cnt + rd_idx - rel_idx;
+
+ return pkt_cnt;
+}
+
+int mtk_dpmaif_get_rx_info(void *pit, struct dpmaif_rx_info *rx_info, u32 pit_seq_expect, u8 q_id);
+void mtk_dpmaif_fill_tx_info(void *drb, struct dpmaif_tx_info *tx_info, int type);
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.h b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
index 700879f9b21e..bb657aa786b6 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
@@ -22,6 +22,7 @@ enum mtk_irq_src {
MTK_IRQ_SRC_PM_LOCK,
MTK_IRQ_SRC_DPMAIF3,
MTK_IRQ_SRC_DPMAIF6,
+ MTK_IRQ_SRC_TRAS_SYNC,
MTK_IRQ_SRC_MAX
};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c
index 88b44142afb7..560a3b81a279 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c
@@ -64,6 +64,7 @@ const struct mtk_pci_dev_cfg mtk_dev_cfg_0900 = {
[MTK_IRQ_SRC_PM_LOCK] = 0,
[MTK_IRQ_SRC_DPMAIF3] = 7,
[MTK_IRQ_SRC_DPMAIF6] = 10,
+ [MTK_IRQ_SRC_TRAS_SYNC] = 9,
},
.atr_init = mtk_pci_atr_init_m9xx,
};
--
2.34.1
^ permalink raw reply related
* [PATCH 09/11] net: wwan: t9xx: Introduce WWAN interface
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Creates the WWAN interface which implements the wwan_ops
for registration with the WWAN framework. WWAN interface
also implements the net_device_ops functions used by the
network devices. Network device operations include open,
stop, start transmission and get states.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 3 +-
drivers/net/wwan/t9xx/mtk_data_plane.c | 14 +-
drivers/net/wwan/t9xx/mtk_data_plane.h | 2 +
drivers/net/wwan/t9xx/mtk_wwan.c | 475 ++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_wwan.h | 17 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c | 15 +-
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c | 1 -
7 files changed, 521 insertions(+), 6 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index 3776ccb4952f..6927216021cd 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -12,4 +12,5 @@ mtk_t9xx-y := \
mtk_port.o \
mtk_port_io.o \
mtk_fsm.o \
- mtk_data_plane.o
+ mtk_data_plane.o \
+ mtk_wwan.o
diff --git a/drivers/net/wwan/t9xx/mtk_data_plane.c b/drivers/net/wwan/t9xx/mtk_data_plane.c
index d11c7de45e17..6266915f08bb 100644
--- a/drivers/net/wwan/t9xx/mtk_data_plane.c
+++ b/drivers/net/wwan/t9xx/mtk_data_plane.c
@@ -6,10 +6,13 @@
#include "mtk_data_plane.h"
#include "mtk_dev.h"
#include "mtk_fsm.h"
+#include "mtk_wwan.h"
static void mtk_data_stop(struct mtk_data_blk *data_blk, struct mtk_md_dev *mdev)
{
+ mtk_wwan_notify(data_blk, DATA_EVT_TX_STOP, 0xff);
data_blk->hif_ops->stop(mdev);
+ mtk_wwan_notify(data_blk, DATA_EVT_UNREG_DEV, 0);
data_blk->hif_ops->clear(mdev);
}
@@ -31,6 +34,7 @@ static void mtk_data_fsm_callback(struct mtk_fsm_param *fsm_param, void *data)
data_blk->hif_ops->start(data);
break;
case FSM_STATE_READY:
+ mtk_wwan_notify(data_blk, DATA_EVT_REG_DEV, 0);
break;
default:
break;
@@ -54,15 +58,21 @@ int mtk_data_init(struct mtk_md_dev *mdev, struct mtk_data_hif_ops *ops)
if (ret < 0)
goto data_blk_free;
+ ret = mtk_wwan_init(data_blk);
+ if (ret < 0)
+ goto hif_exit;
+
ret = mtk_fsm_notifier_register(mdev, MTK_USER_DATA, mtk_data_fsm_callback, mdev,
FSM_PRIO_1, false);
if (ret < 0) {
dev_err(mdev->dev, "Failed to register FSM notifier\n");
- goto hif_exit;
+ goto wwan_exit;
}
return 0;
+wwan_exit:
+ mtk_wwan_exit(data_blk);
hif_exit:
data_blk->hif_ops->exit(mdev);
data_blk_free:
@@ -82,6 +92,8 @@ int mtk_data_exit(struct mtk_md_dev *mdev)
mtk_fsm_notifier_unregister(mdev, MTK_USER_DATA);
+ mtk_wwan_exit(data_blk);
+
data_blk->hif_ops->exit(mdev);
devm_kfree(mdev->dev, data_blk);
diff --git a/drivers/net/wwan/t9xx/mtk_data_plane.h b/drivers/net/wwan/t9xx/mtk_data_plane.h
index 1464fab544f1..351b30ebbbe5 100644
--- a/drivers/net/wwan/t9xx/mtk_data_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_data_plane.h
@@ -31,6 +31,7 @@ struct mtk_data_trans_info {
struct mtk_data_blk {
struct mtk_md_dev *mdev;
+ struct mtk_wwan_ctlb *wcb;
void *dcb;
struct mtk_data_hif_ops *hif_ops;
struct mtk_data_trans_info trans_info;
@@ -80,6 +81,7 @@ enum mtk_data_evt {
DATA_EVT_MIN,
DATA_EVT_TX_START,
DATA_EVT_TX_STOP,
+ DATA_EVT_RX_START,
DATA_EVT_RX_STOP,
DATA_EVT_REG_DEV,
DATA_EVT_UNREG_DEV,
diff --git a/drivers/net/wwan/t9xx/mtk_wwan.c b/drivers/net/wwan/t9xx/mtk_wwan.c
new file mode 100644
index 000000000000..a1534c312358
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_wwan.c
@@ -0,0 +1,475 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/if_arp.h>
+#include <linux/netdevice.h>
+#include <linux/skbuff.h>
+#include <linux/types.h>
+#include <linux/wwan.h>
+#include <net/pkt_sched.h>
+
+#include "mtk_data_plane.h"
+#include "mtk_dev.h"
+#include "mtk_wwan.h"
+
+#define MTK_NETDEV_MAX 20
+#define MTK_DFLT_INTF_ID 0
+#define MTK_NETDEV_WDT (HZ)
+#define MTK_CMD_WDT (HZ)
+#define MTK_MAX_INTF_ID (MTK_NETDEV_MAX - 1)
+#define MTK_NAPI_POLL_WEIGHT 128
+
+static unsigned int napi_budget = MTK_NAPI_POLL_WEIGHT;
+
+struct mtk_wwan_instance {
+ struct mtk_wwan_ctlb *wcb;
+ struct net_device *netdev;
+ unsigned int intf_id;
+ struct mtk_data_hif_ops *hif_ops;
+};
+
+struct mtk_wwan_ctlb {
+ struct mtk_data_blk *data_blk;
+ struct mtk_md_dev *mdev;
+ struct mtk_wwan_instance __rcu *wwan_inst[MTK_NETDEV_MAX];
+ struct net_device *dummy_dev;
+ struct napi_struct **gro_napis;
+ atomic_t napi_enabled;
+ unsigned int active_cnt;
+ bool reg_done;
+};
+
+int mtk_wwan_recv(struct mtk_data_blk *data_blk, struct sk_buff *skb)
+{
+ union mtk_data_pkt_info *pkt_info = DATA_SKB_CB(skb);
+ struct mtk_wwan_instance *wwan_inst;
+ unsigned char q_id;
+
+ if (unlikely(pkt_info->rx.ch_id > MTK_MAX_INTF_ID)) {
+ dev_warn(data_blk->mdev->dev,
+ "Invalid interface id=%d\n", pkt_info->rx.ch_id);
+ goto free_skb;
+ }
+
+ q_id = pkt_info->rx.q_id;
+
+ rcu_read_lock();
+ wwan_inst = rcu_dereference(data_blk->wcb->wwan_inst[pkt_info->rx.ch_id]);
+ if (unlikely(!wwan_inst)) {
+ rcu_read_unlock();
+ goto free_skb;
+ }
+
+ skb->dev = wwan_inst->netdev;
+
+ napi_gro_receive(data_blk->wcb->gro_napis[q_id], skb);
+
+ rcu_read_unlock();
+ return 0;
+
+free_skb:
+ dev_kfree_skb_any(skb);
+ return -EINVAL;
+}
+EXPORT_SYMBOL(mtk_wwan_recv);
+
+static void mtk_wwan_napi_enable(struct mtk_wwan_ctlb *wcb)
+{
+ int i;
+
+ if (atomic_cmpxchg(&wcb->napi_enabled, 0, 1) == 0) {
+ for (i = 0; i < wcb->data_blk->trans_info.rxq_cnt; i++)
+ napi_enable(wcb->data_blk->trans_info.napis[i]);
+ }
+}
+
+static void mtk_wwan_napi_disable(struct mtk_wwan_ctlb *wcb)
+{
+ int i;
+
+ if (atomic_cmpxchg(&wcb->napi_enabled, 1, 0) == 1) {
+ for (i = 0; i < wcb->data_blk->trans_info.rxq_cnt; i++) {
+ napi_synchronize(wcb->data_blk->trans_info.napis[i]);
+ napi_disable(wcb->data_blk->trans_info.napis[i]);
+ }
+ }
+}
+
+static int mtk_wwan_open(struct net_device *dev)
+{
+ struct mtk_wwan_instance *wwan_inst = wwan_netdev_drvpriv(dev);
+ struct mtk_wwan_ctlb *wcb = wwan_inst->wcb;
+ struct mtk_data_trans_ctl trans_ctl;
+ int ret;
+
+ if (wcb->active_cnt == 0) {
+ trans_ctl.enable = true;
+ ret = mtk_wwan_cmd_execute(dev, DATA_CMD_TRANS_CTL, &trans_ctl);
+ if (ret < 0) {
+ dev_err(wcb->mdev->dev, "Failed to enable trans\n");
+ return ret;
+ }
+ }
+
+ wcb->active_cnt++;
+
+ netif_tx_start_all_queues(dev);
+ netif_carrier_on(dev);
+
+ return 0;
+}
+
+static int mtk_wwan_stop(struct net_device *dev)
+{
+ struct mtk_wwan_instance *wwan_inst = wwan_netdev_drvpriv(dev);
+ struct mtk_wwan_ctlb *wcb = wwan_inst->wcb;
+ struct mtk_data_trans_ctl trans_ctl;
+ int ret;
+
+ netif_carrier_off(dev);
+ netif_tx_disable(dev);
+
+ if (wcb->active_cnt == 1) {
+ trans_ctl.enable = false;
+ ret = mtk_wwan_cmd_execute(dev, DATA_CMD_TRANS_CTL, &trans_ctl);
+ if (ret < 0)
+ dev_err(wcb->mdev->dev, "Failed to disable trans\n");
+ }
+ wcb->active_cnt--;
+
+ return 0;
+}
+
+static netdev_tx_t mtk_wwan_start_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct mtk_wwan_instance *wwan_inst = wwan_netdev_drvpriv(dev);
+ union mtk_data_pkt_info *pkt_info = DATA_SKB_CB(skb);
+ int ret;
+
+ skb_set_queue_mapping(skb, 0);
+ pkt_info->tx.intf_id = wwan_inst->intf_id;
+
+ ret = wwan_inst->hif_ops->send(wwan_inst->wcb->data_blk, DATA_PKT, skb);
+ if (ret == -EBUSY)
+ return NETDEV_TX_BUSY;
+ else if (ret == -EINVAL)
+ dev_kfree_skb_any(skb);
+
+ return NETDEV_TX_OK;
+}
+
+static const struct net_device_ops mtk_netdev_ops = {
+ .ndo_open = mtk_wwan_open,
+ .ndo_stop = mtk_wwan_stop,
+ .ndo_start_xmit = mtk_wwan_start_xmit,
+};
+
+static int mtk_wwan_cmd_check(struct net_device *dev, enum mtk_data_cmd_type cmd)
+{
+ int ret = 0;
+
+ switch (cmd) {
+ case DATA_CMD_TRANS_CTL:
+ break;
+ default:
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
+static struct sk_buff *mtk_wwan_cmd_alloc(enum mtk_data_cmd_type cmd, unsigned int len)
+{
+ struct mtk_data_cmd *event;
+ struct sk_buff *skb;
+
+ skb = dev_alloc_skb(sizeof(*event) + len);
+ if (unlikely(!skb))
+ return NULL;
+
+ skb_put(skb, len + sizeof(*event));
+ event = (struct mtk_data_cmd *)skb->data;
+ event->cmd = cmd;
+ event->len = len;
+
+ return skb;
+}
+
+static int mtk_wwan_cmd_send(struct net_device *dev, struct sk_buff *skb)
+{
+ struct mtk_wwan_instance *wwan_inst = wwan_netdev_drvpriv(dev);
+
+ return wwan_inst->hif_ops->send(wwan_inst->wcb->data_blk, DATA_CMD, skb);
+}
+
+int mtk_wwan_cmd_execute(struct net_device *dev,
+ enum mtk_data_cmd_type cmd, void *data)
+{
+ struct mtk_wwan_instance *wwan_inst;
+ struct sk_buff *skb;
+ int ret;
+
+ if (mtk_wwan_cmd_check(dev, cmd))
+ return -EOPNOTSUPP;
+
+ skb = mtk_wwan_cmd_alloc(cmd, sizeof(void *));
+ if (unlikely(!skb))
+ return -ENOMEM;
+
+ SKB_TO_CMD_DATA(skb) = data;
+
+ ret = mtk_wwan_cmd_send(dev, skb);
+ if (ret < 0) {
+ wwan_inst = wwan_netdev_drvpriv(dev);
+ dev_err(wwan_inst->wcb->mdev->dev,
+ "Failed to excute command:ret=%d,cmd=%d\n", ret, cmd);
+ }
+
+ dev_consume_skb_any(skb);
+
+ return ret;
+}
+
+static int mtk_wwan_start_txq(struct mtk_wwan_ctlb *wcb, u32 qmask)
+{
+ struct mtk_wwan_instance *wwan_inst;
+ struct net_device *dev;
+ int i;
+
+ rcu_read_lock();
+ for (i = 0; i < MTK_NETDEV_MAX; i++) {
+ wwan_inst = rcu_dereference(wcb->wwan_inst[i]);
+ if (!wwan_inst)
+ continue;
+
+ dev = wwan_inst->netdev;
+
+ if (!(dev->flags & IFF_UP))
+ continue;
+
+ netif_tx_wake_all_queues(dev);
+ netif_carrier_on(dev);
+ }
+ rcu_read_unlock();
+
+ return 0;
+}
+
+static int mtk_wwan_stop_txq(struct mtk_wwan_ctlb *wcb, u32 qmask)
+{
+ struct mtk_wwan_instance *wwan_inst;
+ struct net_device *dev;
+ int i;
+
+ rcu_read_lock();
+ for (i = 0; i < MTK_NETDEV_MAX; i++) {
+ wwan_inst = rcu_dereference(wcb->wwan_inst[i]);
+ if (!wwan_inst)
+ continue;
+
+ dev = wwan_inst->netdev;
+
+ if (!(dev->flags & IFF_UP))
+ continue;
+
+ netif_carrier_off(dev);
+ netif_tx_stop_all_queues(dev);
+ }
+ rcu_read_unlock();
+
+ return 0;
+}
+
+static void mtk_wwan_napi_exit(struct mtk_wwan_ctlb *wcb)
+{
+ int i;
+
+ for (i = 0; i < wcb->data_blk->trans_info.rxq_cnt; i++) {
+ if (!wcb->data_blk->trans_info.napis[i])
+ continue;
+ netif_napi_del(wcb->data_blk->trans_info.napis[i]);
+ }
+}
+
+static int mtk_wwan_napi_init(struct mtk_wwan_ctlb *wcb, struct net_device *dev)
+{
+ int i;
+
+ for (i = 0; i < wcb->data_blk->trans_info.rxq_cnt; i++) {
+ if (!wcb->data_blk->trans_info.napis[i]) {
+ dev_err(wcb->mdev->dev, "Invalid napi pointer, napi=%d\n", i);
+ goto out;
+ }
+ netif_napi_add_weight(dev, wcb->data_blk->trans_info.napis[i],
+ wcb->data_blk->hif_ops->poll, napi_budget);
+ }
+
+ return 0;
+
+out:
+ for (--i; i >= 0; i--)
+ netif_napi_del(wcb->data_blk->trans_info.napis[i]);
+ return -EINVAL;
+}
+
+static void mtk_wwan_setup(struct net_device *dev)
+{
+ dev->watchdog_timeo = MTK_NETDEV_WDT;
+ dev->mtu = ETH_DATA_LEN;
+ dev->min_mtu = ETH_MIN_MTU;
+
+ dev->features = NETIF_F_SG;
+ dev->hw_features = NETIF_F_SG;
+
+ dev->features |= NETIF_F_GRO;
+ dev->hw_features |= NETIF_F_GRO;
+
+ dev->tx_queue_len = DEFAULT_TX_QUEUE_LEN;
+
+ dev->flags = IFF_NOARP;
+ dev->type = ARPHRD_NONE;
+
+ dev->needs_free_netdev = true;
+
+ dev->netdev_ops = &mtk_netdev_ops;
+}
+
+static int mtk_wwan_newlink(void *ctxt, struct net_device *dev, u32 intf_id,
+ struct netlink_ext_ack *extack)
+{
+ struct mtk_wwan_instance *wwan_inst = wwan_netdev_drvpriv(dev);
+ struct mtk_wwan_ctlb *wcb = ctxt;
+ int ret;
+
+ if (intf_id > MTK_MAX_INTF_ID)
+ return -EINVAL;
+
+ if (rcu_access_pointer(wcb->wwan_inst[intf_id]))
+ return -EBUSY;
+
+ dev->max_mtu = wcb->data_blk->trans_info.max_mtu;
+
+ wwan_inst->wcb = wcb;
+ wwan_inst->netdev = dev;
+ wwan_inst->intf_id = intf_id;
+ wwan_inst->hif_ops = wcb->data_blk->hif_ops;
+
+ ret = register_netdevice(dev);
+ if (ret)
+ return ret;
+
+ rcu_assign_pointer(wcb->wwan_inst[intf_id], wwan_inst);
+
+ netif_device_attach(dev);
+
+ return 0;
+}
+
+static void mtk_wwan_dellink(void *ctxt, struct net_device *dev,
+ struct list_head *head)
+{
+ struct mtk_wwan_instance *wwan_inst = wwan_netdev_drvpriv(dev);
+ int intf_id = wwan_inst->intf_id;
+ struct mtk_wwan_ctlb *wcb = ctxt;
+
+ if (WARN_ON(rcu_access_pointer(wcb->wwan_inst[intf_id]) != wwan_inst))
+ return;
+
+ RCU_INIT_POINTER(wcb->wwan_inst[intf_id], NULL);
+ unregister_netdevice_queue(dev, head);
+}
+
+static const struct wwan_ops mtk_wwan_ops = {
+ .priv_size = sizeof(struct mtk_wwan_instance),
+ .setup = mtk_wwan_setup,
+ .newlink = mtk_wwan_newlink,
+ .dellink = mtk_wwan_dellink,
+};
+
+void mtk_wwan_notify(struct mtk_data_blk *data_blk, enum mtk_data_evt evt, u64 data)
+{
+ struct mtk_wwan_ctlb *wcb;
+
+ if (unlikely(!data_blk || !data_blk->wcb))
+ return;
+
+ wcb = data_blk->wcb;
+
+ switch (evt) {
+ case DATA_EVT_TX_START:
+ mtk_wwan_start_txq(wcb, data);
+ break;
+ case DATA_EVT_TX_STOP:
+ mtk_wwan_stop_txq(wcb, data);
+ break;
+ case DATA_EVT_RX_START:
+ mtk_wwan_napi_enable(wcb);
+ break;
+ case DATA_EVT_RX_STOP:
+ mtk_wwan_napi_disable(wcb);
+ break;
+ case DATA_EVT_REG_DEV:
+ if (!wcb->reg_done) {
+ wwan_register_ops(wcb->mdev->dev, &mtk_wwan_ops, wcb, MTK_DFLT_INTF_ID);
+ wcb->reg_done = true;
+ }
+ break;
+ case DATA_EVT_UNREG_DEV:
+ if (wcb->reg_done) {
+ wwan_unregister_ops(wcb->mdev->dev);
+ wcb->reg_done = false;
+ }
+ break;
+ default:
+ break;
+ }
+}
+EXPORT_SYMBOL(mtk_wwan_notify);
+
+int mtk_wwan_init(struct mtk_data_blk *data_blk)
+{
+ struct mtk_wwan_ctlb *wcb;
+ int ret;
+
+ wcb = devm_kzalloc(data_blk->mdev->dev, sizeof(*wcb), GFP_KERNEL);
+ if (unlikely(!wcb))
+ return -ENOMEM;
+
+ wcb->mdev = data_blk->mdev;
+ wcb->data_blk = data_blk;
+
+ wcb->dummy_dev = alloc_netdev_dummy(0);
+ if (!wcb->dummy_dev) {
+ devm_kfree(data_blk->mdev->dev, wcb);
+ return -ENOMEM;
+ }
+
+ data_blk->wcb = wcb;
+
+ wcb->gro_napis = data_blk->trans_info.napis;
+ ret = mtk_wwan_napi_init(wcb, wcb->dummy_dev);
+ if (ret < 0) {
+ free_netdev(wcb->dummy_dev);
+ devm_kfree(data_blk->mdev->dev, wcb);
+ data_blk->wcb = NULL;
+ return ret;
+ }
+
+ return 0;
+}
+
+void mtk_wwan_exit(struct mtk_data_blk *data_blk)
+{
+ struct mtk_wwan_ctlb *wcb = data_blk->wcb;
+
+ if (unlikely(!wcb))
+ return;
+
+ mtk_wwan_napi_exit(wcb);
+ free_netdev(wcb->dummy_dev);
+ devm_kfree(data_blk->mdev->dev, wcb);
+ data_blk->wcb = NULL;
+}
diff --git a/drivers/net/wwan/t9xx/mtk_wwan.h b/drivers/net/wwan/t9xx/mtk_wwan.h
new file mode 100644
index 000000000000..8005aeb1ed97
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_wwan.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_WWAN_H__
+#define __MTK_WWAN_H__
+#include <linux/netdevice.h>
+#include "mtk_data_plane.h"
+
+int mtk_wwan_init(struct mtk_data_blk *data_blk);
+void mtk_wwan_exit(struct mtk_data_blk *data_blk);
+int mtk_wwan_recv(struct mtk_data_blk *data_blk, struct sk_buff *skb);
+void mtk_wwan_notify(struct mtk_data_blk *data_blk, enum mtk_data_evt evt, u64 data);
+int mtk_wwan_cmd_execute(struct net_device *dev, enum mtk_data_cmd_type cmd, void *data);
+
+#endif /* __MTK_WWAN_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
index 1c58dba738fa..43803587bfc3 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
@@ -20,9 +20,11 @@
#include "mtk_dev.h"
#include "mtk_dpmaif.h"
#include "mtk_dpmaif_drv.h"
+#include "mtk_dpmaif_reg.h"
#include "mtk_dpmaif_ring.h"
#include "mtk_pci_reg.h"
#include "mtk_pci.h"
+#include "mtk_wwan.h"
#define MTK_DATA_WS_NAME_LEN 32
#define DPMAIF_PIT_CNT_UPDATE_THRESHOLD 60
@@ -741,6 +743,7 @@ static void mtk_dpmaif_sw_stop_rx(struct mtk_dpmaif_ctlb *dcb)
}
/* Stop PIT polling NAPI. */
+ mtk_wwan_notify(dcb->data_blk, DATA_EVT_RX_STOP, 0xff);
}
static void mtk_dpmaif_sw_start_rx(struct mtk_dpmaif_ctlb *dcb)
@@ -749,6 +752,7 @@ static void mtk_dpmaif_sw_start_rx(struct mtk_dpmaif_ctlb *dcb)
int i;
/* Start PIT polling NAPI. */
+ mtk_wwan_notify(dcb->data_blk, DATA_EVT_RX_START, 0xff);
for (i = 0; i < dcb->rxq_cnt; i++) {
rxq = &dcb->rxqs[i];
@@ -910,6 +914,7 @@ static int mtk_dpmaif_tx_rel_internal(struct dpmaif_txq *txq,
tx_srv = &dcb->tx_srvs[srv_id];
clear_bit(txq->id, &tx_srv->txq_drb_lack_sta);
wake_up(&tx_srv->wait);
+ mtk_wwan_notify(dcb->data_blk, DATA_EVT_TX_START, (u64)1 << txq->id);
}
return 0;
@@ -1426,6 +1431,7 @@ static void mtk_dpmaif_tx_update_ring(struct mtk_dpmaif_ctlb *dcb, struct dpmaif
if (drb_available_cnt < skb_drb_cnt) {
skb_queue_head(&vq->list, skb);
set_bit(q_id, &tx_srv->txq_drb_lack_sta);
+ mtk_wwan_notify(dcb->data_blk, DATA_EVT_TX_STOP, (u64)1 << q_id);
break;
}
@@ -2346,6 +2352,7 @@ static int mtk_dpmaif_update_rx_skb_info(struct sk_buff *skb,
static int mtk_dpmaif_rx_skb(struct dpmaif_rxq *rxq, struct dpmaif_rx_record *rx_record)
{
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
struct sk_buff *new_skb;
int ret;
@@ -2362,7 +2369,7 @@ static int mtk_dpmaif_rx_skb(struct dpmaif_rxq *rxq, struct dpmaif_rx_record *rx
continue;
/* Send skb to data port. */
- /* Data would be sent to network stack here */
+ mtk_wwan_recv(dcb->data_blk, new_skb);
} while (!skb_queue_empty(&rx_record->rx_list));
return ret;
@@ -2600,10 +2607,12 @@ static int mtk_dpmaif_send_pkt(struct mtk_dpmaif_ctlb *dcb, struct sk_buff *skb)
int ret = 0;
vq = &dcb->tx_vqs[vq_id];
- if (likely(skb_queue_len(&vq->list) < vq->max_len))
+ if (likely(skb_queue_len(&vq->list) < vq->max_len)) {
skb_queue_tail(&vq->list, skb);
- else
+ } else {
+ mtk_wwan_notify(dcb->data_blk, DATA_EVT_TX_STOP, (u64)1 << vq_id);
ret = -EBUSY;
+ }
wake_up(&dcb->tx_srvs[vq->srv_id].wait);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
index 077600389ab4..3fd2b33d1199 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
@@ -8,7 +8,6 @@
#include <linux/delay.h>
#include "mtk_dev.h"
-#include "mtk_data_plane.h"
#include "mtk_dpmaif_drv.h"
#include "mtk_dpmaif_reg.h"
#include "mtk_pci.h"
--
2.34.1
^ permalink raw reply related
* [PATCH 05/11] net: wwan: t9xx: Add FSM thread
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
The FSM (Finite-state Machine) thread is responsible for
synchronizing the actions of different modules. The
asynchronous events from the device or the OS will trigger
a state transition.
The FSM thread will append it to the event queue when an
event arrives. It handles the events sequentially. After
processing the event, the FSM thread notifies other modules
before and after the state transition.
Seven FSM states are defined. They can transition from one
state to another, self-transition in some states, and
transition in some sub-states.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 3 +-
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 46 ++
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 2 +
drivers/net/wwan/t9xx/mtk_dev.h | 1 +
drivers/net/wwan/t9xx/mtk_fsm.c | 931 ++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_fsm.h | 140 ++++
drivers/net/wwan/t9xx/mtk_port.c | 65 ++
drivers/net/wwan/t9xx/mtk_port.h | 2 +
drivers/net/wwan/t9xx/mtk_utility.h | 33 +
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 215 +++++-
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 3 +
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 3 -
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c | 7 +-
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h | 2 -
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 16 +-
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 10 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 1 -
17 files changed, 1464 insertions(+), 16 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index db3b1aa1928b..75760b2039dc 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -10,4 +10,5 @@ mtk_t9xx-y := \
mtk_dev.o \
mtk_ctrl_plane.o \
mtk_port.o \
- mtk_port_io.o
+ mtk_port_io.o \
+ mtk_fsm.o
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
index 9264b59783d7..448ab7cdc933 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -5,10 +5,46 @@
*/
#include <linux/device.h>
+#include <linux/freezer.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/pm_runtime.h>
+#include <linux/sched.h>
+#include <linux/wait.h>
#include "mtk_ctrl_plane.h"
#include "mtk_port.h"
+#define TAG "CTRL"
+
+static void mtk_ctrl_trans_fsm_state_handler(struct mtk_fsm_param *param,
+ struct mtk_ctrl_blk *ctrl_blk)
+{
+ struct mtk_md_dev *mdev = ctrl_blk->mdev;
+
+ switch (param->to) {
+ case FSM_STATE_OFF:
+ ctrl_blk->ops->fsm_indication(mdev, param);
+ ctrl_blk->ops->exit(mdev);
+ break;
+ case FSM_STATE_ON:
+ ctrl_blk->ops->init(mdev);
+ fallthrough;
+ default:
+ ctrl_blk->ops->fsm_indication(mdev, param);
+ break;
+ }
+}
+
+static void mtk_ctrl_fsm_state_listener(struct mtk_fsm_param *param, void *data)
+{
+ struct mtk_ctrl_blk *ctrl_blk = data;
+
+ mtk_port_mngr_fsm_state_handler(param, ctrl_blk->port_mngr);
+ mtk_ctrl_trans_fsm_state_handler(param, ctrl_blk);
+ mtk_port_mngr_fsm_state_handler_late(param, ctrl_blk->port_mngr);
+}
+
int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops, struct mtk_ctrl_cfg *cfg)
{
struct mtk_ctrl_blk *ctrl_blk;
@@ -28,8 +64,17 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops, struct
if (err)
goto err_free_mem;
+ err = mtk_fsm_notifier_register(mdev, MTK_USER_CTRL, mtk_ctrl_fsm_state_listener,
+ ctrl_blk, FSM_PRIO_1, false);
+ if (err) {
+ dev_err((mdev)->dev, "Fail to register fsm notification(ret = %d)\n", err);
+ goto err_port_exit;
+ }
+
return 0;
+err_port_exit:
+ mtk_port_mngr_exit(ctrl_blk);
err_free_mem:
devm_kfree(mdev->dev, ctrl_blk);
@@ -41,6 +86,7 @@ int mtk_ctrl_exit(struct mtk_md_dev *mdev)
{
struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ mtk_fsm_notifier_unregister(mdev, MTK_USER_CTRL);
mtk_port_mngr_exit(ctrl_blk);
devm_kfree(mdev->dev, ctrl_blk);
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index 91b623191221..c4530a9ba36c 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -10,6 +10,7 @@
#include <linux/skbuff.h>
#include "mtk_dev.h"
+#include "mtk_fsm.h"
#define Q_MTU_2K (0x800)
#define Q_MTU_3_5K (0xE00)
@@ -62,6 +63,7 @@ struct mtk_ctrl_hif_ops {
int (*init)(struct mtk_md_dev *mdev);
int (*exit)(struct mtk_md_dev *mdev);
int (*submit_skb)(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send);
+ void (*fsm_indication)(struct mtk_md_dev *mdev, struct mtk_fsm_param *param);
int (*send_cmd)(struct mtk_md_dev *mdev, int cmd, void *data);
};
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
index 37eec1a358fa..bf412b49b19d 100644
--- a/drivers/net/wwan/t9xx/mtk_dev.h
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -58,6 +58,7 @@ struct mtk_md_dev {
u32 hw_ver;
char dev_str[MTK_DEV_STR_LEN];
void *ctrl_blk;
+ struct mtk_md_fsm *fsm;
};
static inline u32 mtk_dev_get_dev_state(struct mtk_md_dev *mdev)
diff --git a/drivers/net/wwan/t9xx/mtk_fsm.c b/drivers/net/wwan/t9xx/mtk_fsm.c
new file mode 100644
index 000000000000..2a0373901d05
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_fsm.c
@@ -0,0 +1,931 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/kref.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/pci.h>
+#include <linux/sched/signal.h>
+#include <linux/skbuff.h>
+#include <linux/wait.h>
+
+#include "mtk_fsm.h"
+#include "mtk_port.h"
+#include "mtk_port_io.h"
+#include "mtk_utility.h"
+
+#define EVT_TF_GATECLOSED (1)
+#define MTK_FSM_INFO_LEN (64)
+
+#define FSM_HS_START_MASK (FSM_F_SAP_HS_START | FSM_F_MD_HS_START)
+#define FSM_HS2_DONE_MASK (FSM_F_SAP_HS2_DONE | FSM_F_MD_HS2_DONE)
+
+#define RTFT_DATA_SIZE (3 * 1024)
+#define EVT_HANDLER_TIMEOUT (HZ * 30)
+#define BLOCKING_EVT_TIMEOUT (2 * EVT_HANDLER_TIMEOUT)
+
+#define REGION_BITMASK 0xF
+#define DEVICE_CFG_SHIFT 24
+#define DEVICE_CFG_REGION_MASK 0x3
+
+enum device_stage {
+ DEV_STAGE_IDLE = 4,
+ DEV_STAGE_MAX
+};
+
+enum device_cfg {
+ DEV_CFG_NORMAL = 0,
+ DEV_CFG_MD_ONLY,
+};
+
+enum runtime_feature_support_type {
+ RTFT_TYPE_NOT_EXIST = 0,
+ RTFT_TYPE_NOT_SUPPORT = 1,
+ RTFT_TYPE_MUST_SUPPORT = 2,
+ RTFT_TYPE_OPTIONAL_SUPPORT = 3,
+ RTFT_TYPE_SUPPORT_BACKWARD_COMPAT = 4,
+};
+
+enum query_runtime_feature_id {
+ QUERY_RTFT_ID_MD_PORT_ENUM = 0,
+ QUERY_RTFT_ID_SAP_PORT_ENUM = 1,
+ QUERY_RTFT_ID_MD_PORT_CFG = 2,
+ QUERY_RTFT_ID_MAX
+};
+
+enum ctrl_msg_id {
+ CTRL_MSG_HS1 = 0,
+ CTRL_MSG_HS2 = 1,
+ CTRL_MSG_HS3 = 2,
+};
+
+struct ctrl_msg_header {
+ __le32 id;
+ __le32 ex_msg;
+ __le32 data_len;
+ u8 reserved[];
+} __packed;
+
+struct runtime_feature_entry {
+ u8 feature_id;
+ struct runtime_feature_info support_info;
+ u8 reserved[2];
+ __le32 data_len;
+ u8 data[];
+};
+
+struct feature_query {
+ __le32 head_pattern;
+ struct runtime_feature_info ft_set[FEATURE_CNT];
+ __le32 tail_pattern;
+};
+
+static int mtk_fsm_send_hs1_msg(struct fsm_hs_info *hs_info)
+{
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct feature_query *ft_query;
+ struct sk_buff *skb;
+ int ret, msg_size;
+
+ msg_size = sizeof(*ctrl_msg_h) + sizeof(*ft_query);
+ skb = __dev_alloc_skb(msg_size, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ skb_put(skb, msg_size);
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ ctrl_msg_h->id = cpu_to_le32(CTRL_MSG_HS1);
+ ctrl_msg_h->ex_msg = 0;
+ ctrl_msg_h->data_len = cpu_to_le32(sizeof(*ft_query));
+
+ ft_query = (struct feature_query *)(skb->data + sizeof(*ctrl_msg_h));
+ ft_query->head_pattern = cpu_to_le32(FEATURE_QUERY_PATTERN);
+ memcpy(ft_query->ft_set, hs_info->query_ft_set, sizeof(hs_info->query_ft_set));
+ ft_query->tail_pattern = cpu_to_le32(FEATURE_QUERY_PATTERN);
+
+ /* send handshake1 message to device */
+ ret = mtk_port_internal_write(hs_info->ctrl_port, skb);
+ if (ret <= 0)
+ return ret;
+
+ return 0;
+}
+
+static int mtk_fsm_feature_set_match(enum runtime_feature_support_type *cur_ft_spt,
+ struct runtime_feature_info rtft_info_st,
+ struct runtime_feature_info rtft_info_cfg)
+{
+ int ret = 0;
+
+ switch (FIELD_GET(FEATURE_TYPE, rtft_info_st.feature)) {
+ case RTFT_TYPE_NOT_EXIST:
+ fallthrough;
+ case RTFT_TYPE_NOT_SUPPORT:
+ *cur_ft_spt = RTFT_TYPE_NOT_EXIST;
+ break;
+ case RTFT_TYPE_MUST_SUPPORT:
+ if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_EXIST ||
+ FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_SUPPORT)
+ ret = -EPROTO;
+ else
+ *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
+ break;
+ case RTFT_TYPE_OPTIONAL_SUPPORT:
+ if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_EXIST ||
+ FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_SUPPORT) {
+ *cur_ft_spt = RTFT_TYPE_NOT_SUPPORT;
+ } else {
+ if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) ==
+ FIELD_GET(FEATURE_VER, rtft_info_cfg.feature))
+ *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
+ else
+ *cur_ft_spt = RTFT_TYPE_NOT_SUPPORT;
+ }
+ break;
+ case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT:
+ if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) >=
+ FIELD_GET(FEATURE_VER, rtft_info_cfg.feature))
+ *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
+ else
+ *cur_ft_spt = RTFT_TYPE_NOT_EXIST;
+ break;
+ default:
+ ret = -EPROTO;
+ }
+
+ return ret;
+}
+
+static int (*query_rtft_action[FEATURE_CNT])(struct mtk_md_dev *mdev, void *rt_data) = {
+ [QUERY_RTFT_ID_MD_PORT_ENUM] = mtk_port_status_update,
+ [QUERY_RTFT_ID_SAP_PORT_ENUM] = mtk_port_status_update,
+};
+
+static int mtk_fsm_parse_hs2_msg(struct fsm_hs_info *hs_info)
+{
+ struct mtk_md_fsm *fsm = container_of(hs_info, struct mtk_md_fsm, hs_info[hs_info->id]);
+ char *rt_data = ((struct sk_buff *)hs_info->rt_data)->data;
+ enum runtime_feature_support_type cur_ft_spt;
+ struct runtime_feature_entry *rtft_entry;
+ int ft_id, ret = 0, offset;
+
+ offset = sizeof(struct feature_query);
+ for (ft_id = 0; ft_id < FEATURE_CNT && offset < hs_info->rt_data_len; ft_id++) {
+ rtft_entry = (struct runtime_feature_entry *)(rt_data + offset);
+ ret = mtk_fsm_feature_set_match(&cur_ft_spt,
+ rtft_entry->support_info,
+ hs_info->query_ft_set[ft_id]);
+ if (ret < 0)
+ break;
+
+ if (cur_ft_spt == RTFT_TYPE_MUST_SUPPORT)
+ if (query_rtft_action[ft_id])
+ ret = query_rtft_action[ft_id](fsm->mdev, rtft_entry->data);
+ if (ret < 0)
+ break;
+
+ offset += sizeof(*rtft_entry) + le32_to_cpu(rtft_entry->data_len);
+ }
+
+ if (ft_id != FEATURE_CNT) {
+ dev_err((fsm->mdev)->dev, "Unable to handle mistake hs2 msg, ft_id=%d\n", ft_id);
+ ret = -EPROTO;
+ }
+
+ return ret;
+}
+
+static int mtk_fsm_append_rtft_entries(struct mtk_md_dev *mdev, void *feature_data,
+ unsigned int *len, struct fsm_hs_info *hs_info)
+{
+ char *rt_data = ((struct sk_buff *)hs_info->rt_data)->data;
+ struct runtime_feature_entry *rtft_entry;
+ int ft_id, ret = 0, rtdata_len = 0;
+ struct feature_query *ft_query;
+
+ ft_query = (struct feature_query *)rt_data;
+ if (le32_to_cpu(ft_query->head_pattern) != FEATURE_QUERY_PATTERN ||
+ le32_to_cpu(ft_query->tail_pattern) != FEATURE_QUERY_PATTERN) {
+ ret = -EPROTO;
+ goto hs_err;
+ }
+
+ /* parse runtime feature query and fill runtime feature entry */
+ rtft_entry = feature_data;
+ for (ft_id = 0; ft_id < FEATURE_CNT && rtdata_len < RTFT_DATA_SIZE; ft_id++) {
+ rtft_entry->feature_id = ft_id;
+ rtft_entry->data_len = 0;
+
+ switch (FIELD_GET(FEATURE_TYPE, ft_query->ft_set[ft_id].feature)) {
+ case RTFT_TYPE_NOT_EXIST:
+ fallthrough;
+ case RTFT_TYPE_NOT_SUPPORT:
+ fallthrough;
+ case RTFT_TYPE_MUST_SUPPORT:
+ rtft_entry->support_info = ft_query->ft_set[ft_id];
+ break;
+ case RTFT_TYPE_OPTIONAL_SUPPORT:
+ fallthrough;
+ case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT:
+ rtft_entry->support_info.feature = FEATURE_TYPE_NOT;
+ rtft_entry->support_info.feature |= FEATURE_VER_0;
+ break;
+ }
+
+ rtdata_len += sizeof(*rtft_entry) + le32_to_cpu(rtft_entry->data_len);
+ rtft_entry = (struct runtime_feature_entry *)(feature_data + rtdata_len);
+ }
+ *len = rtdata_len;
+ return 0;
+
+hs_err:
+ *len = 0;
+ return ret;
+}
+
+static int mtk_fsm_send_hs3_msg(struct fsm_hs_info *hs_info)
+{
+ struct mtk_md_fsm *fsm = container_of(hs_info, struct mtk_md_fsm, hs_info[hs_info->id]);
+ unsigned int data_len, msg_size = 0;
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct sk_buff *skb;
+ int ret;
+
+ skb = __dev_alloc_skb(RTFT_DATA_SIZE, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ msg_size += sizeof(*ctrl_msg_h);
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ ctrl_msg_h->id = cpu_to_le32(CTRL_MSG_HS3);
+ ctrl_msg_h->ex_msg = 0;
+ ret = mtk_fsm_append_rtft_entries(fsm->mdev,
+ skb->data + sizeof(*ctrl_msg_h),
+ &data_len, hs_info);
+ if (ret) {
+ dev_kfree_skb(skb);
+ return ret;
+ }
+
+ ctrl_msg_h->data_len = cpu_to_le32(data_len);
+ msg_size += data_len;
+ skb_put(skb, msg_size);
+ ret = mtk_port_internal_write(hs_info->ctrl_port, skb);
+ if (ret <= 0)
+ return ret;
+
+ return 0;
+}
+
+static int mtk_fsm_sap_ctrl_msg_handler(void *__fsm, struct sk_buff *skb)
+{
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct mtk_md_fsm *fsm = __fsm;
+ struct fsm_hs_info *hs_info;
+ int ret;
+
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ skb_pull(skb, sizeof(*ctrl_msg_h));
+
+ hs_info = &fsm->hs_info[HS_ID_SAP];
+ if (le32_to_cpu(ctrl_msg_h->id) != CTRL_MSG_HS2)
+ return -EPROTO;
+
+ hs_info->rt_data = skb;
+ hs_info->rt_data_len = skb->len;
+ ret = mtk_fsm_evt_submit(fsm->mdev, FSM_EVT_STARTUP,
+ hs_info->fsm_flag_hs2, hs_info, sizeof(*hs_info), 0);
+ if (ret == FSM_EVT_RET_FAIL)
+ dev_kfree_skb(skb);
+
+ return 0;
+}
+
+static int mtk_fsm_md_ctrl_msg_handler(void *__fsm, struct sk_buff *skb)
+{
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct mtk_md_fsm *fsm = __fsm;
+ struct fsm_hs_info *hs_info;
+ bool consumed_skb = false;
+ int ret;
+
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ hs_info = &fsm->hs_info[HS_ID_MD];
+ switch (le32_to_cpu(ctrl_msg_h->id)) {
+ case CTRL_MSG_HS2:
+ skb_pull(skb, sizeof(*ctrl_msg_h));
+ hs_info->rt_data = skb;
+ hs_info->rt_data_len = skb->len;
+ ret = mtk_fsm_evt_submit(fsm->mdev, FSM_EVT_STARTUP,
+ hs_info->fsm_flag_hs2, hs_info, sizeof(*hs_info), 0);
+ if (ret != FSM_EVT_RET_FAIL)
+ consumed_skb = true;
+ break;
+ default:
+ dev_err(fsm->mdev->dev, "Invalid ctrl msg id\n");
+ }
+
+ if (!consumed_skb)
+ dev_kfree_skb(skb);
+
+ return 0;
+}
+
+static int (*ctrl_msg_handler[HS_ID_MAX])(void *__fsm, struct sk_buff *skb) = {
+ [HS_ID_MD] = mtk_fsm_md_ctrl_msg_handler,
+ [HS_ID_SAP] = mtk_fsm_sap_ctrl_msg_handler,
+};
+
+static void mtk_fsm_idle_evt_handler(struct mtk_md_dev *mdev,
+ u32 dev_state, struct mtk_md_fsm *fsm)
+{
+ u32 dev_cfg = dev_state >> DEVICE_CFG_SHIFT & DEVICE_CFG_REGION_MASK;
+ int hs_id;
+
+ if (dev_cfg == DEV_CFG_MD_ONLY)
+ fsm->hs_done_flag = FSM_F_MD_HS_START | FSM_F_MD_HS2_DONE;
+ else
+ fsm->hs_done_flag = FSM_HS_START_MASK | FSM_HS2_DONE_MASK;
+
+ mtk_fsm_evt_submit(mdev, FSM_EVT_STARTUP, FSM_F_DFLT, NULL, 0, 0);
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++)
+ mtk_dev_unmask_dev_evt(mdev, fsm->hs_info[hs_id].mhccif_ch);
+}
+
+static int mtk_fsm_early_bootup_handler(u32 status, void *__fsm)
+{
+ struct mtk_md_fsm *fsm = __fsm;
+ struct mtk_md_dev *mdev;
+ u32 dev_state, dev_stage;
+
+ mdev = fsm->mdev;
+ mtk_dev_mask_dev_evt(mdev, status);
+ mtk_dev_clear_dev_evt(mdev, status);
+
+ dev_state = mtk_dev_get_dev_state(mdev);
+ dev_stage = dev_state & REGION_BITMASK;
+ if (dev_stage >= DEV_STAGE_MAX) {
+ dev_err(mdev->dev, "Invalid dev state 0x%x\n", dev_state);
+ return -ENXIO;
+ }
+
+ if (dev_state == fsm->last_dev_state)
+ goto exit;
+ fsm->last_dev_state = dev_state;
+
+ if (dev_stage == DEV_STAGE_IDLE)
+ mtk_fsm_idle_evt_handler(mdev, dev_state, fsm);
+
+exit:
+ return 0;
+}
+
+static int mtk_fsm_ctrl_ch_start(struct mtk_md_fsm *fsm, struct fsm_hs_info *hs_info, int flag)
+{
+ if (!hs_info->ctrl_port) {
+ hs_info->ctrl_port = mtk_port_internal_open(fsm->mdev, hs_info->port_name, flag);
+ if (!hs_info->ctrl_port) {
+ dev_err(fsm->mdev->dev, "Failed to open ctrl port(%s)\n",
+ hs_info->port_name);
+ return -ENODEV;
+ }
+
+ mtk_port_internal_recv_register(hs_info->ctrl_port,
+ ctrl_msg_handler[hs_info->id], fsm);
+ }
+
+ return 0;
+}
+
+static void mtk_fsm_ctrl_ch_stop(struct mtk_md_fsm *fsm)
+{
+ struct fsm_hs_info *hs_info;
+ int hs_id;
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++) {
+ hs_info = &fsm->hs_info[hs_id];
+ if (hs_info->ctrl_port) {
+ mtk_port_internal_close(hs_info->ctrl_port);
+ hs_info->ctrl_port = NULL;
+ }
+ }
+}
+
+static void mtk_fsm_switch_state(struct mtk_md_fsm *fsm,
+ enum mtk_fsm_state to_state, struct mtk_fsm_evt *event)
+{
+ char fsm_info[MTK_FSM_INFO_LEN];
+ struct mtk_fsm_notifier *nt;
+ struct mtk_fsm_param param;
+
+ param.from = fsm->state;
+ param.to = to_state;
+ param.evt_id = event ? event->id : FSM_EVT_MAX;
+ param.fsm_flag = event ? event->fsm_flag : FSM_F_DFLT;
+
+ list_for_each_entry(nt, &fsm->pre_notifiers, entry)
+ nt->cb(¶m, nt->data);
+
+ fsm->state = to_state;
+ fsm->fsm_flag |= event ? event->fsm_flag : FSM_F_DFLT;
+
+ snprintf(fsm_info, MTK_FSM_INFO_LEN,
+ "state=%d, fsm_flag=0x%x", to_state, fsm->fsm_flag);
+ mtk_uevent_notify(fsm->mdev->dev, MTK_UEVENT_FSM, fsm_info);
+
+ list_for_each_entry(nt, &fsm->post_notifiers, entry)
+ nt->cb(¶m, nt->data);
+}
+
+static int mtk_fsm_startup_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ enum mtk_fsm_state to_state = FSM_STATE_BOOTUP;
+ struct fsm_hs_info *hs_info = event->data;
+ struct mtk_md_dev *mdev = fsm->mdev;
+ int ret = 0;
+
+ if (fsm->state != FSM_STATE_ON && fsm->state != FSM_STATE_BOOTUP) {
+ ret = -EPROTO;
+ goto free_rt_data;
+ }
+
+ if (fsm->state != FSM_STATE_BOOTUP) {
+ mtk_fsm_switch_state(fsm, to_state, event);
+ return 0;
+ }
+
+ if (event->fsm_flag & FSM_HS_START_MASK) {
+ mtk_fsm_switch_state(fsm, to_state, event);
+
+ ret = mtk_fsm_ctrl_ch_start(fsm, hs_info, O_NONBLOCK);
+ if (!ret)
+ ret = mtk_fsm_send_hs1_msg(hs_info);
+ if (ret)
+ goto hs_err;
+ } else if (event->fsm_flag & FSM_HS2_DONE_MASK) {
+ ret = mtk_fsm_parse_hs2_msg(hs_info);
+ if (!ret) {
+ mtk_fsm_switch_state(fsm, to_state, event);
+ ret = mtk_fsm_send_hs3_msg(hs_info);
+ }
+ dev_kfree_skb(hs_info->rt_data);
+ hs_info->rt_data = NULL;
+ if (ret)
+ goto hs_err;
+ }
+
+ if (((fsm->fsm_flag | event->fsm_flag) & fsm->hs_done_flag) == fsm->hs_done_flag) {
+ to_state = FSM_STATE_READY;
+ mtk_fsm_switch_state(fsm, to_state, NULL);
+ }
+
+ return 0;
+
+free_rt_data:
+ if (hs_info && hs_info->rt_data) {
+ dev_kfree_skb(hs_info->rt_data);
+ hs_info->rt_data = NULL;
+ }
+hs_err:
+ dev_err((mdev)->dev, "Failed to hs with device %d:0x%x, ret=%d",
+ fsm->state, fsm->fsm_flag, ret);
+ return ret;
+}
+
+static void mtk_fsm_evt_release(struct kref *kref)
+{
+ struct mtk_fsm_evt *event = container_of(kref, struct mtk_fsm_evt, kref);
+
+ devm_kfree(event->mdev->dev, event);
+}
+
+static void mtk_fsm_evt_put(struct mtk_fsm_evt *event)
+{
+ kref_put(&event->kref, mtk_fsm_evt_release);
+}
+
+static void mtk_fsm_evt_finish(struct mtk_md_fsm *fsm,
+ struct mtk_fsm_evt *event, int retval)
+{
+ if (event->mode & EVT_MODE_BLOCKING) {
+ event->status = retval;
+ wake_up(&fsm->evt_waitq);
+ }
+ mtk_fsm_evt_put(event);
+}
+
+static void mtk_fsm_evt_cleanup(struct mtk_md_fsm *fsm, struct list_head *evtq)
+{
+ struct mtk_fsm_evt *event, *tmp;
+
+ list_for_each_entry_safe(event, tmp, evtq, entry) {
+ list_del(&event->entry);
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_FAIL);
+ }
+}
+
+static int mtk_fsm_enter_off_state(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ struct mtk_md_dev *mdev = fsm->mdev;
+ int hs_id;
+
+ if (fsm->state == FSM_STATE_OFF || fsm->state == FSM_STATE_INVALID)
+ return -EPROTO;
+
+ mtk_dev_mask_dev_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++)
+ mtk_dev_mask_dev_evt(mdev, fsm->hs_info[hs_id].mhccif_ch);
+
+ mtk_fsm_ctrl_ch_stop(fsm);
+ mtk_fsm_switch_state(fsm, FSM_STATE_OFF, event);
+
+ return 0;
+}
+
+static int mtk_fsm_dev_rm_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ set_bit(EVT_TF_GATECLOSED, &fsm->t_flag);
+ mtk_fsm_evt_cleanup(fsm, &fsm->evtq);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ return mtk_fsm_enter_off_state(fsm, event);
+}
+
+static int mtk_fsm_hs1_handler(u32 status, void *__hs_info)
+{
+ struct fsm_hs_info *hs_info = __hs_info;
+ struct mtk_md_dev *mdev;
+ struct mtk_md_fsm *fsm;
+
+ fsm = container_of(hs_info, struct mtk_md_fsm, hs_info[hs_info->id]);
+ mdev = fsm->mdev;
+ mtk_fsm_evt_submit(mdev, FSM_EVT_STARTUP,
+ hs_info->fsm_flag_hs1, hs_info, sizeof(*hs_info), 0);
+ mtk_dev_mask_dev_evt(mdev, hs_info->mhccif_ch);
+ mtk_dev_clear_dev_evt(mdev, hs_info->mhccif_ch);
+
+ return 0;
+}
+
+static void mtk_fsm_hs_info_init_by_hsid(struct mtk_md_fsm *fsm, int hs_id)
+{
+ struct fsm_hs_info *hs_info;
+
+ if (hs_id < 0 || hs_id >= HS_ID_MAX) {
+ dev_warn((fsm->mdev)->dev, "hs_id = %d, invalid.\n", hs_id);
+ return;
+ }
+
+ hs_info = &fsm->hs_info[hs_id];
+ hs_info->id = hs_id;
+ hs_info->ctrl_port = NULL;
+ hs_info->rt_data = NULL;
+ switch (hs_id) {
+ case HS_ID_MD:
+ snprintf(hs_info->port_name, PORT_NAME_LEN, "MDCTRL");
+ hs_info->mhccif_ch = DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD;
+ hs_info->fsm_flag_hs1 = FSM_F_MD_HS_START;
+ hs_info->fsm_flag_hs2 = FSM_F_MD_HS2_DONE;
+ hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM].feature =
+ FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT);
+ hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM].feature |=
+ FIELD_PREP(FEATURE_VER, 0);
+ hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_CFG].feature =
+ FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_NOT_SUPPORT);
+ break;
+ case HS_ID_SAP:
+ snprintf(hs_info->port_name, PORT_NAME_LEN, "SAPCTRL");
+ hs_info->mhccif_ch = DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP;
+ hs_info->fsm_flag_hs1 = FSM_F_SAP_HS_START;
+ hs_info->fsm_flag_hs2 = FSM_F_SAP_HS2_DONE;
+ hs_info->query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM].feature =
+ FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT);
+ hs_info->query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM].feature |=
+ FIELD_PREP(FEATURE_VER, 0);
+ break;
+ }
+}
+
+static void mtk_fsm_hs_info_init(struct mtk_md_fsm *fsm)
+{
+ struct mtk_md_dev *mdev = fsm->mdev;
+ struct fsm_hs_info *hs_info;
+ int hs_id;
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++) {
+ mtk_fsm_hs_info_init_by_hsid(fsm, hs_id);
+ hs_info = &fsm->hs_info[hs_id];
+ mtk_dev_register_dev_evt(mdev, hs_info->mhccif_ch,
+ mtk_fsm_hs1_handler, hs_info);
+ }
+}
+
+static void mtk_fsm_hs_info_exit(struct mtk_md_fsm *fsm)
+{
+ struct mtk_md_dev *mdev = fsm->mdev;
+ struct fsm_hs_info *hs_info;
+ int hs_id;
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++) {
+ hs_info = &fsm->hs_info[hs_id];
+ mtk_dev_unregister_dev_evt(mdev, hs_info->mhccif_ch);
+ }
+}
+
+static int mtk_fsm_dev_add_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ if (fsm->state != FSM_STATE_OFF && fsm->state != FSM_STATE_INVALID)
+ return -EPROTO;
+
+ mtk_fsm_switch_state(fsm, FSM_STATE_ON, event);
+ mtk_dev_unmask_dev_evt(fsm->mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+
+ return 0;
+}
+
+static int (*evts_act_tbl[FSM_EVT_MAX])(struct mtk_md_fsm *__fsm, struct mtk_fsm_evt *event) = {
+ [FSM_EVT_STARTUP] = mtk_fsm_startup_act,
+ [FSM_EVT_DEV_RM] = mtk_fsm_dev_rm_act,
+ [FSM_EVT_DEV_ADD] = mtk_fsm_dev_add_act,
+};
+
+int mtk_fsm_start(struct mtk_md_dev *mdev)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+
+ if (!fsm)
+ return -EINVAL;
+
+ if (!fsm->fsm_handler)
+ return -EFAULT;
+
+ wake_up_process(fsm->fsm_handler);
+ return 0;
+}
+EXPORT_SYMBOL(mtk_fsm_start);
+
+static void mkt_fsm_notifier_cleanup(struct mtk_md_dev *mdev, struct list_head *ntq)
+{
+ struct mtk_fsm_notifier *nt, *tmp;
+
+ list_for_each_entry_safe(nt, tmp, ntq, entry) {
+ list_del(&nt->entry);
+ dev_warn((mdev)->dev, "Having to free notifier(%d) by FSM!\n", nt->id);
+ devm_kfree(mdev->dev, nt);
+ }
+}
+
+static void mtk_fsm_notifier_insert(struct mtk_fsm_notifier *notifier, struct list_head *head)
+{
+ struct mtk_fsm_notifier *nt;
+
+ list_for_each_entry(nt, head, entry) {
+ if (notifier->prio > nt->prio) {
+ list_add(¬ifier->entry, nt->entry.prev);
+ return;
+ }
+ }
+ list_add_tail(¬ifier->entry, head);
+}
+
+int mtk_fsm_notifier_register(struct mtk_md_dev *mdev, enum mtk_user_id id,
+ void (*cb)(struct mtk_fsm_param *, void *data),
+ void *data, enum mtk_fsm_prio prio, bool is_pre)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct mtk_fsm_notifier *notifier;
+
+ if (!fsm)
+ return -EINVAL;
+
+ if (id >= MTK_USER_MAX || !cb || prio >= FSM_PRIO_MAX)
+ return -EINVAL;
+
+ notifier = devm_kzalloc(mdev->dev, sizeof(*notifier), GFP_KERNEL);
+ if (!notifier)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(¬ifier->entry);
+ notifier->id = id;
+ notifier->cb = cb;
+ notifier->data = data;
+ notifier->prio = prio;
+
+ if (is_pre)
+ mtk_fsm_notifier_insert(notifier, &fsm->pre_notifiers);
+ else
+ mtk_fsm_notifier_insert(notifier, &fsm->post_notifiers);
+
+ return 0;
+}
+EXPORT_SYMBOL(mtk_fsm_notifier_register);
+
+int mtk_fsm_notifier_unregister(struct mtk_md_dev *mdev, enum mtk_user_id id)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct mtk_fsm_notifier *nt, *tmp;
+
+ if (!fsm)
+ return -EINVAL;
+
+ list_for_each_entry_safe(nt, tmp, &fsm->pre_notifiers, entry) {
+ if (nt->id == id) {
+ list_del(&nt->entry);
+ devm_kfree(mdev->dev, nt);
+ break;
+ }
+ }
+ list_for_each_entry_safe(nt, tmp, &fsm->post_notifiers, entry) {
+ if (nt->id == id) {
+ list_del(&nt->entry);
+ devm_kfree(mdev->dev, nt);
+ break;
+ }
+ }
+ return 0;
+}
+EXPORT_SYMBOL(mtk_fsm_notifier_unregister);
+
+int mtk_fsm_evt_submit(struct mtk_md_dev *mdev,
+ enum mtk_fsm_evt_id id, enum mtk_fsm_flag flag,
+ void *data, unsigned int len, unsigned char mode)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct mtk_fsm_evt *event;
+ unsigned long flags;
+ int ret = 0;
+
+ if (!fsm || id >= FSM_EVT_MAX) {
+ dev_err((mdev)->dev, "Invalid param!\n");
+ return FSM_EVT_RET_FAIL;
+ }
+
+ if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag)) {
+ dev_err((mdev)->dev, "Failed to submit evt, fsm has been removed!\n");
+ return FSM_EVT_RET_FAIL;
+ }
+
+ event = devm_kzalloc(mdev->dev, sizeof(*event),
+ (in_hardirq() || in_softirq() || irqs_disabled()) ?
+ GFP_ATOMIC : GFP_KERNEL);
+ if (!event)
+ return FSM_EVT_RET_FAIL;
+
+ kref_init(&event->kref);
+ event->mdev = mdev;
+ event->id = id;
+ event->fsm_flag = flag;
+ event->status = FSM_EVT_RET_ONGOING;
+ event->data = data;
+ event->len = len;
+ event->mode = mode;
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag)) {
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+ mtk_fsm_evt_put(event);
+ dev_err(mdev->dev, "Failed to add event, fsm dev has been removed!\n");
+ return FSM_EVT_RET_FAIL;
+ }
+
+ kref_get(&event->kref);
+ if (mode & EVT_MODE_TOHEAD)
+ list_add(&event->entry, &fsm->evtq);
+ else
+ list_add_tail(&event->entry, &fsm->evtq);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ wake_up_process(fsm->fsm_handler);
+ if (mode & EVT_MODE_BLOCKING) {
+ ret = wait_event_timeout(fsm->evt_waitq,
+ (event->status != 0), BLOCKING_EVT_TIMEOUT);
+ if (!ret && event->status != FSM_EVT_RET_DONE)
+ dev_err((mdev)->dev, "Handling fsm blocking event timeout!\n");
+
+ ret = event->status;
+ }
+ mtk_fsm_evt_put(event);
+
+ return ret;
+}
+EXPORT_SYMBOL(mtk_fsm_evt_submit);
+
+static int mtk_fsm_evt_handler(void *__fsm)
+{
+ struct mtk_md_fsm *fsm = __fsm;
+ struct mtk_fsm_evt *event;
+ unsigned long flags;
+ int ret;
+
+wake_up:
+ set_current_state(TASK_INTERRUPTIBLE);
+ while (!kthread_should_stop() && !list_empty(&fsm->evtq)) {
+ set_current_state(TASK_RUNNING);
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ event = list_first_entry(&fsm->evtq, struct mtk_fsm_evt, entry);
+ list_del(&event->entry);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ if (event->id < FSM_EVT_MAX) {
+ ret = evts_act_tbl[event->id](fsm, event);
+ if (ret) {
+ dev_err((fsm->mdev)->dev,
+ "Failed to handle evt, fsm state = %d, ret = %d\n",
+ fsm->state, ret);
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_FAIL);
+ } else {
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_DONE);
+ }
+ } else {
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_DONE);
+ }
+ }
+
+ if (kthread_should_stop()) {
+ set_current_state(TASK_RUNNING);
+ return 0;
+ }
+
+ schedule();
+
+ if (fatal_signal_pending(current)) {
+ mtk_fsm_evt_cleanup(fsm, &fsm->evtq);
+ return -ERESTARTSYS;
+ }
+ goto wake_up;
+}
+
+int mtk_fsm_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_md_fsm *fsm;
+ int ret;
+
+ fsm = devm_kzalloc(mdev->dev, sizeof(*fsm), GFP_KERNEL);
+ if (!fsm)
+ return -ENOMEM;
+
+ fsm->fsm_handler = kthread_create(mtk_fsm_evt_handler, fsm, "fsm_evt_thread%d_%s",
+ mdev->hw_ver, mdev->dev_str);
+ if (IS_ERR(fsm->fsm_handler)) {
+ ret = PTR_ERR(fsm->fsm_handler);
+ goto exit;
+ }
+
+ fsm->mdev = mdev;
+ fsm->state = FSM_STATE_INVALID;
+ fsm->fsm_flag = FSM_F_DFLT;
+
+ INIT_LIST_HEAD(&fsm->evtq);
+ spin_lock_init(&fsm->evtq_lock);
+ init_waitqueue_head(&fsm->evt_waitq);
+
+ INIT_LIST_HEAD(&fsm->pre_notifiers);
+ INIT_LIST_HEAD(&fsm->post_notifiers);
+
+ mtk_fsm_hs_info_init(fsm);
+ mtk_dev_register_dev_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC,
+ mtk_fsm_early_bootup_handler, fsm);
+ mdev->fsm = fsm;
+ return 0;
+exit:
+ devm_kfree(mdev->dev, fsm);
+ return ret;
+}
+EXPORT_SYMBOL(mtk_fsm_init);
+
+int mtk_fsm_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ unsigned long flags;
+
+ if (!fsm)
+ return -EINVAL;
+
+ if (fsm->fsm_handler) {
+ kthread_stop(fsm->fsm_handler);
+ fsm->fsm_handler = NULL;
+ }
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ if (WARN_ON(!list_empty(&fsm->evtq)))
+ mtk_fsm_evt_cleanup(fsm, &fsm->evtq);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ mkt_fsm_notifier_cleanup(mdev, &fsm->pre_notifiers);
+ mkt_fsm_notifier_cleanup(mdev, &fsm->post_notifiers);
+
+ mtk_dev_unregister_dev_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+ mtk_fsm_hs_info_exit(fsm);
+
+ devm_kfree(mdev->dev, fsm);
+ return 0;
+}
+EXPORT_SYMBOL(mtk_fsm_exit);
diff --git a/drivers/net/wwan/t9xx/mtk_fsm.h b/drivers/net/wwan/t9xx/mtk_fsm.h
new file mode 100644
index 000000000000..f2fc66bcef61
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_fsm.h
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_FSM_H__
+#define __MTK_FSM_H__
+
+#include "mtk_dev.h"
+
+#define FEATURE_CNT (64)
+#define FEATURE_QUERY_PATTERN (0x49434343)
+
+#define FEATURE_TYPE GENMASK(3, 0)
+#define FEATURE_VER GENMASK(7, 4)
+
+#define FEATURE_TYPE_NOT FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_NOT_SUPPORT)
+#define FEATURE_TYPE_MUST FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT)
+#define FEATURE_TYPE_OPTIONAL FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_OPTIONAL_SUPPORT)
+#define FEATURE_VER_0 FIELD_PREP(FEATURE_VER, 0)
+
+#define EVT_MODE_BLOCKING (0x01)
+#define EVT_MODE_TOHEAD (0x02)
+
+#define FSM_EVT_RET_FAIL (-1)
+#define FSM_EVT_RET_ONGOING (0)
+#define FSM_EVT_RET_DONE (1)
+
+enum mtk_fsm_flag {
+ FSM_F_DFLT = 0,
+ FSM_F_SAP_HS_START = BIT(0),
+ FSM_F_SAP_HS2_DONE = BIT(1),
+ FSM_F_MD_HS_START = BIT(2),
+ FSM_F_MD_HS2_DONE = BIT(3),
+};
+
+enum mtk_fsm_state {
+ FSM_STATE_INVALID = 0,
+ FSM_STATE_OFF,
+ FSM_STATE_ON,
+ FSM_STATE_BOOTUP,
+ FSM_STATE_READY,
+};
+
+enum mtk_fsm_evt_id {
+ FSM_EVT_STARTUP = 0,
+ FSM_EVT_DEV_RM,
+ FSM_EVT_DEV_ADD,
+ FSM_EVT_MAX
+};
+
+enum mtk_fsm_prio {
+ FSM_PRIO_0 = 0,
+ FSM_PRIO_1 = 1,
+ FSM_PRIO_MAX
+};
+
+struct mtk_fsm_param {
+ enum mtk_fsm_state from;
+ enum mtk_fsm_state to;
+ enum mtk_fsm_evt_id evt_id;
+ enum mtk_fsm_flag fsm_flag;
+};
+
+#define PORT_NAME_LEN 20
+
+enum handshake_info_id {
+ HS_ID_MD = 0,
+ HS_ID_SAP,
+ HS_ID_MAX
+};
+
+struct runtime_feature_info {
+ u8 feature;
+};
+
+struct fsm_hs_info {
+ unsigned char id;
+ void *ctrl_port;
+ char port_name[PORT_NAME_LEN];
+ unsigned int mhccif_ch;
+ unsigned int fsm_flag_hs1;
+ unsigned int fsm_flag_hs2;
+ /* the feature that the device should support */
+ struct runtime_feature_info query_ft_set[FEATURE_CNT];
+ /* runtime data from device need to be parsed by host */
+ void *rt_data;
+ unsigned int rt_data_len;
+};
+
+struct mtk_md_fsm {
+ struct mtk_md_dev *mdev;
+ struct task_struct *fsm_handler;
+ struct fsm_hs_info hs_info[HS_ID_MAX];
+ unsigned int hs_done_flag;
+ unsigned long t_flag;
+ u32 last_dev_state;
+ enum mtk_fsm_state state;
+ unsigned int fsm_flag;
+ struct list_head evtq;
+ /* protect evtq */
+ spinlock_t evtq_lock;
+ /* waitq for fsm blocking submit */
+ wait_queue_head_t evt_waitq;
+ struct list_head pre_notifiers;
+ struct list_head post_notifiers;
+};
+
+struct mtk_fsm_evt {
+ struct list_head entry;
+ struct kref kref;
+ struct mtk_md_dev *mdev;
+ enum mtk_fsm_evt_id id;
+ unsigned int fsm_flag;
+ int status;
+ unsigned char mode;
+ unsigned int len;
+ void *data;
+};
+
+struct mtk_fsm_notifier {
+ struct list_head entry;
+ enum mtk_user_id id;
+ void (*cb)(struct mtk_fsm_param *param, void *data);
+ void *data;
+ enum mtk_fsm_prio prio;
+};
+
+int mtk_fsm_init(struct mtk_md_dev *mdev);
+int mtk_fsm_exit(struct mtk_md_dev *mdev);
+int mtk_fsm_start(struct mtk_md_dev *mdev);
+int mtk_fsm_notifier_register(struct mtk_md_dev *mdev, enum mtk_user_id id,
+ void (*cb)(struct mtk_fsm_param *, void *data),
+ void *data, enum mtk_fsm_prio prio, bool is_pre);
+int mtk_fsm_notifier_unregister(struct mtk_md_dev *mdev, enum mtk_user_id id);
+int mtk_fsm_evt_submit(struct mtk_md_dev *mdev,
+ enum mtk_fsm_evt_id id, enum mtk_fsm_flag flag,
+ void *data, unsigned int len, unsigned char mode);
+
+#endif /* __MTK_FSM_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
index e400e4d35f26..dbd279cf2a14 100644
--- a/drivers/net/wwan/t9xx/mtk_port.c
+++ b/drivers/net/wwan/t9xx/mtk_port.c
@@ -818,6 +818,71 @@ int mtk_port_ch_disable(struct mtk_port *port)
return ret;
}
+static void mtk_port_disable(struct mtk_port_mngr *port_mngr)
+{
+ struct mtk_port **ports;
+ int tbl_type;
+ int ret, idx;
+
+ ports = kcalloc(port_mngr->port_cnt, sizeof(struct mtk_port *), GFP_KERNEL);
+ if (!ports)
+ return;
+
+ tbl_type = PORT_TBL_SAP;
+ do {
+ ret = radix_tree_gang_lookup(&port_mngr->port_tbl[tbl_type],
+ (void **)ports, 0, port_mngr->port_cnt);
+ for (idx = 0; idx < ret; idx++)
+ ports_ops[ports[idx]->info.type]->disable(ports[idx]);
+ } while (++tbl_type < PORT_TBL_MAX);
+ kfree(ports);
+}
+
+void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void *arg)
+{
+ struct mtk_port_mngr *port_mngr;
+
+ if (!fsm_param || !arg)
+ return;
+
+ port_mngr = arg;
+
+ switch (fsm_param->to) {
+ case FSM_STATE_OFF:
+ mtk_port_disable(port_mngr);
+ break;
+ default:
+ break;
+ }
+}
+
+void mtk_port_mngr_fsm_state_handler_late(struct mtk_fsm_param *fsm_param, void *arg)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_port *port;
+
+ if (!fsm_param || !arg)
+ return;
+
+ port_mngr = arg;
+
+ switch (fsm_param->to) {
+ case FSM_STATE_BOOTUP:
+ if (fsm_param->fsm_flag & FSM_F_MD_HS_START) {
+ port = mtk_port_search_by_id(port_mngr, CCCI_CONTROL_RX);
+ if (port)
+ ports_ops[port->info.type]->enable(port);
+ } else if (fsm_param->fsm_flag & FSM_F_SAP_HS_START) {
+ port = mtk_port_search_by_id(port_mngr, CCCI_SAP_CONTROL_RX);
+ if (port)
+ ports_ops[port->info.type]->enable(port);
+ }
+ break;
+ default:
+ break;
+ }
+}
+
int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg *port_cfg, int port_cnt)
{
struct mtk_port_mngr *port_mngr;
diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_port.h
index bd4291408bc2..a201c0007878 100644
--- a/drivers/net/wwan/t9xx/mtk_port.h
+++ b/drivers/net/wwan/t9xx/mtk_port.h
@@ -152,6 +152,8 @@ int mtk_port_send_data(struct mtk_port *port, void *data);
int mtk_port_status_update(struct mtk_md_dev *mdev, void *data);
int mtk_port_ch_enable(struct mtk_port *port);
int mtk_port_ch_disable(struct mtk_port *port);
+void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void *arg);
+void mtk_port_mngr_fsm_state_handler_late(struct mtk_fsm_param *fsm_param, void *arg);
int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg *port_cfg, int port_cnt);
void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk);
void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_trb_cmd_type cmd,
diff --git a/drivers/net/wwan/t9xx/mtk_utility.h b/drivers/net/wwan/t9xx/mtk_utility.h
new file mode 100644
index 000000000000..b72db3842d2d
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_utility.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_UTILITY_H__
+#define __MTK_UTILITY_H__
+
+#include <linux/device.h>
+#include "mtk_dev.h"
+
+#define MTK_UEVENT_INFO_LEN 128
+
+/* MTK uevent */
+enum mtk_uevent_id {
+ MTK_UEVENT_UNDEF = 0,
+ MTK_UEVENT_FSM = 1,
+ MTK_UEVENT_MINIDUMP = 2,
+ MTK_UEVENT_LOWPOWER = 3,
+ MTK_UEVENT_MAX
+};
+
+static inline void mtk_uevent_notify(struct device *dev, enum mtk_uevent_id id, const char *info)
+{
+ char buf[MTK_UEVENT_INFO_LEN];
+ char *ext[2] = {NULL, NULL};
+
+ snprintf(buf, MTK_UEVENT_INFO_LEN, "%s:event_id=%d, info=%s",
+ dev->kobj.name, id, info);
+ ext[0] = buf;
+ kobject_uevent_env(&dev->kobj, KOBJ_CHANGE, ext);
+}
+#endif /* __MTK_UTILITY_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
index 48067a010890..aacb4177d914 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
@@ -34,12 +34,164 @@
#define CLDMA_RETRY_DELAY_MS (100)
#define NO_BUDGET (0)
+static struct cldma_drv_info_desc cldma_drv_info_tbl[] = {
+ {0x01CA, &drv_ops_name(m9xx), &cldma_regs_name(m9xx)},
+ {0, NULL},
+};
+
+static void mtk_cldma_get_drv_info(struct cldma_drv_info *drv_info, u32 hw_ver)
+{
+ struct cldma_drv_info_desc *p_drv_info;
+ u8 i;
+
+ for (i = 0; (p_drv_info = &cldma_drv_info_tbl[i]) && p_drv_info &&
+ p_drv_info->drv_ops && p_drv_info->hw_regs; i++)
+ if (p_drv_info->hw_ver == hw_ver) {
+ drv_info->drv_ops = p_drv_info->drv_ops;
+ drv_info->hw_regs = p_drv_info->hw_regs;
+ }
+}
+
+static int mtk_cldma_isr(int irq_id, void *param)
+{
+ struct cldma_drv_info *drv_info = param;
+ struct mtk_md_dev *mdev;
+ u32 tx_done, rx_done;
+ u32 tx_sta, rx_sta;
+ struct txq *txq;
+ struct rxq *rxq;
+ int i;
+
+ mdev = drv_info->mdev;
+ drv_info->drv_ops->cldma_get_intr_status(drv_info, &tx_sta, &rx_sta);
+ tx_done = (tx_sta >> QUEUE_XFER_DONE) & 0xFF;
+ rx_done = (rx_sta >> QUEUE_XFER_DONE) & 0xFF;
+
+ if (tx_done) {
+ for (i = 0; i < HW_QUEUE_NUM; i++) {
+ txq = drv_info->txq[i];
+ if (!(tx_done & BIT(i)) || !txq)
+ continue;
+ queue_work(drv_info->wq, &txq->tx_done_work);
+ }
+ }
+ if (rx_done) {
+ for (i = 0; i < HW_QUEUE_NUM; i++) {
+ rxq = drv_info->rxq[i];
+ if (!(rx_done & BIT(i)) || !rxq)
+ continue;
+ queue_work(drv_info->wq, &rxq->rx_done_work);
+ }
+ }
+
+ mtk_pci_clear_irq(mdev, drv_info->pci_ext_irq_id);
+ mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id);
+
+ return IRQ_HANDLED;
+}
+
static const int mtk_cldma_hw_id_tbl[NR_CLDMA] = {
[CLDMA0] = CLDMA0_HW_ID,
[CLDMA1] = CLDMA1_HW_ID,
- [CLDMA4] = CLDMA4_HW_ID,
};
+static int mtk_cldma_dev_init(struct cldma_dev *cd, int hif_id)
+{
+ char gpd_pool_name[DMA_POOL_NAME_LEN];
+ char bd_pool_name[DMA_POOL_NAME_LEN];
+ struct cldma_drv_info *drv_info;
+ struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ unsigned int flag;
+ int hw_id;
+
+ if (!cd || hif_id >= NR_CLDMA)
+ return -EINVAL;
+
+ if (cd->cldma_drv_info[hif_id])
+ return 0;
+
+ hw_id = mtk_cldma_hw_id_tbl[hif_id];
+ mdev = cd->trans->mdev;
+ drv_info = devm_kzalloc(mdev->dev, sizeof(*drv_info), GFP_KERNEL);
+ if (!drv_info)
+ return -ENOMEM;
+
+ drv_info->cd = cd;
+ drv_info->mdev = mdev;
+ drv_info->hif_id = hif_id;
+ drv_info->hw_id = hw_id;
+ mtk_cldma_get_drv_info(drv_info, mdev->hw_ver);
+
+ if (!drv_info->drv_ops || !drv_info->hw_regs) {
+ dev_err((mdev)->dev, "Failed to find CLDMA Driver for PCI %x\n", mdev->hw_ver);
+ goto err_free_drv_info;
+ }
+
+ hw_regs = drv_info->hw_regs;
+ snprintf(gpd_pool_name, DMA_POOL_NAME_LEN, "cldma%d_gpd_pool_%s",
+ hw_id, mdev->dev_str);
+ snprintf(bd_pool_name, DMA_POOL_NAME_LEN, "cldma%d_bd_pool_%s",
+ hw_id, mdev->dev_str);
+ drv_info->gpd_dma_pool = dma_pool_create(gpd_pool_name, mdev->dev,
+ sizeof(union gpd), 4, 0);
+ if (!drv_info->gpd_dma_pool) {
+ dev_err((mdev)->dev, "Failed to alloc gpd dma pool for cldma%d\n", hw_id);
+ goto err_free_drv_info;
+ }
+ drv_info->bd_dma_pool = dma_pool_create(bd_pool_name, mdev->dev,
+ sizeof(union bd), 4, 0);
+ if (!drv_info->bd_dma_pool) {
+ dev_err((mdev)->dev, "Failed to alloc bd dma pool for cldma%d\n", hw_id);
+ goto err_destroy_gpd_pool;
+ }
+
+ switch (hif_id) {
+ case CLDMA0:
+ drv_info->pci_ext_irq_id = mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_CLDMA0);
+ drv_info->base_addr = hw_regs->cldma0_base_addr;
+ break;
+ case CLDMA1:
+ drv_info->pci_ext_irq_id = mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_CLDMA1);
+ drv_info->base_addr = hw_regs->cldma1_base_addr;
+ break;
+ default:
+ goto err_destroy_dma_pool;
+ }
+
+ flag = WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_HIGHPRI;
+ drv_info->wq = alloc_workqueue("cldma%d_workq_%s", flag, 0, hw_id, mdev->dev_str);
+ if (!drv_info->wq) {
+ dev_err((mdev)->dev, "Failed to alloc work queue for cldma%d\n", hw_id);
+ goto err_destroy_dma_pool;
+ }
+
+ drv_info->drv_ops->cldma_drv_init(drv_info);
+
+ /* mask/clear PCI CLDMA L1 interrupt */
+ mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id);
+ mtk_pci_clear_irq(mdev, drv_info->pci_ext_irq_id);
+
+ /* register CLDMA interrupt handler */
+ mtk_pci_register_irq(mdev, drv_info->pci_ext_irq_id, mtk_cldma_isr, drv_info);
+
+ /* unmask PCI CLDMA L1 interrupt */
+ mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id);
+
+ cd->cldma_drv_info[hif_id] = drv_info;
+ return 0;
+
+ destroy_workqueue(drv_info->wq);
+err_destroy_dma_pool:
+ dma_pool_destroy(drv_info->bd_dma_pool);
+err_destroy_gpd_pool:
+ dma_pool_destroy(drv_info->gpd_dma_pool);
+err_free_drv_info:
+ devm_kfree(mdev->dev, drv_info);
+
+ return -EIO;
+}
+
static inline void mtk_cldma_clr_bd_dsc(struct cldma_drv_info *drv_info,
struct bd_dsc *bd_dsc_pool, int nr_bds)
{
@@ -864,6 +1016,46 @@ static int mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno)
return 0;
}
+static int mtk_cldma_dev_exit(struct cldma_dev *cd, int hif_id)
+{
+ struct cldma_drv_info *drv_info;
+ struct mtk_md_dev *mdev;
+ int virq_id;
+ int hw_id;
+ int i;
+
+ if (!cd || hif_id >= NR_CLDMA)
+ return -EINVAL;
+
+ if (!cd->cldma_drv_info[hif_id])
+ return 0;
+
+ /* free cldma descriptor */
+ drv_info = cd->cldma_drv_info[hif_id];
+ mdev = cd->trans->mdev;
+ virq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
+ mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id);
+ synchronize_irq(virq_id);
+ for (i = 0; i < HW_QUEUE_NUM; i++) {
+ if (drv_info->txq[i])
+ mtk_cldma_txq_free(drv_info, drv_info->txq[i]->txqno);
+ if (drv_info->rxq[i])
+ mtk_cldma_rxq_free(drv_info, drv_info->rxq[i]->rxqno);
+ }
+
+ flush_workqueue(drv_info->wq);
+ destroy_workqueue(drv_info->wq);
+ dma_pool_destroy(drv_info->bd_dma_pool);
+ dma_pool_destroy(drv_info->gpd_dma_pool);
+ mtk_pci_unregister_irq(mdev, drv_info->pci_ext_irq_id);
+
+ hw_id = drv_info->hw_id;
+ devm_kfree(mdev->dev, drv_info);
+ cd->cldma_drv_info[hif_id] = NULL;
+
+ return 0;
+}
+
static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno)
{
struct cldma_drv_ops *drv_ops;
@@ -1183,6 +1375,27 @@ int mtk_cldma_trb_process(void *dev, struct sk_buff *skb)
return trb_act_tbl[trb->cmd](cd, skb);
}
+void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_ctrl_trans *trans)
+{
+ struct cldma_dev *cd = trans->dev;
+ int i;
+
+ switch (param->to) {
+ case FSM_STATE_BOOTUP:
+ if (param->fsm_flag & FSM_F_SAP_HS_START)
+ mtk_cldma_dev_init(cd, CLDMA0);
+ else if (param->fsm_flag & FSM_F_MD_HS_START)
+ mtk_cldma_dev_init(cd, CLDMA1);
+ break;
+ case FSM_STATE_OFF:
+ for (i = 0; i < NR_CLDMA; i++)
+ mtk_cldma_dev_exit(cd, i);
+ break;
+ default:
+ break;
+ }
+}
+
int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que)
{
struct cldma_drv_info *drv_info;
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
index 246d28d3d798..04f83ff0e37d 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
@@ -167,4 +167,7 @@ int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que);
#define drv_ops_name(NAME) cldma_drv_ops_##NAME
#define cldma_regs_name(NAME) mtk_cldma_regs_##NAME
+extern struct cldma_drv_ops cldma_drv_ops_m9xx;
+extern struct cldma_hw_regs mtk_cldma_regs_m9xx;
+
#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
index 8763c23abf54..6de87b7ffd45 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
@@ -11,7 +11,6 @@
#define LINK_ERROR_VAL (0xFFFFFFFF)
#define CLDMA0_HW_ID (0)
#define CLDMA1_HW_ID (1)
-#define CLDMA4_HW_ID (4)
struct cldma_hw_regs {
u8 cldma_rx_skb_pool_max_size;
@@ -36,7 +35,6 @@ struct cldma_hw_regs {
u16 reg_cldma_l2rimsr0;
u16 reg_cldma_l2rimsr1;
u16 reg_cldma_int_mask;
- u16 reg_cldma4_int_mask;
u16 reg_cldma_slp_mem_ctl;
u16 reg_cldma_busy_mask;
u16 reg_cldma_ip_busy_to_pcie_mask;
@@ -58,7 +56,6 @@ struct cldma_hw_regs {
u32 rq_err_int_bitmask;
u32 cldma0_base_addr;
u32 cldma1_base_addr;
- u32 cldma4_base_addr;
u32 rq_active_start_err_int_bitmask;
u32 reg_cldma_ul_start_addrl_0;
u32 reg_cldma_ul_start_addrh_0;
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c
index 240a9f58f658..9041c8f2f99c 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c
@@ -33,7 +33,6 @@
struct cldma_hw_regs mtk_cldma_regs_m9xx = {
.cldma0_base_addr = CLDMA0_BASE_ADDR,
.cldma1_base_addr = CLDMA1_BASE_ADDR,
- .cldma4_base_addr = CLDMA4_BASE_ADDR,
.cldma_rx_skb_pool_max_size = CLDMA_RX_SKB_POOL_MAX_SIZE,
.cldma_rx_skb_reload_threshold = CLDMA_RX_SKB_RELOAD_THRESHOLD,
.tq_err_int_offset = TQ_ERR_INT_OFFSET,
@@ -92,7 +91,6 @@ struct cldma_hw_regs mtk_cldma_regs_m9xx = {
.reg_cldma_l3risar1 = REG_CLDMA_L3RISAR1,
.reg_cldma_ip_busy = REG_CLDMA_IP_BUSY,
.reg_cldma_int_mask = REG_CLDMA_INT_EAP_USIP_MASK,
- .reg_cldma4_int_mask = REG_CLDMA_INT_WF_MASK,
.reg_cldma_ip_busy_to_pcie_mask = REG_CLDMA_IP_BUSY_TO_PCIE_MASK,
.reg_cldma_ip_busy_to_pcie_mask_set = REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SET,
.reg_cldma_ip_busy_to_pcie_mask_clr = REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CLR,
@@ -134,10 +132,7 @@ static void mtk_cldma_drv_init_m9xx(struct cldma_drv_info *drv_info)
ALLQ << 24);
/* enable interrupt to PCIe */
- if (drv_info->hw_id == CLDMA4_HW_ID)
- mtk_pci_write32(mdev, base + hw_regs->reg_cldma4_int_mask, 0);
- else
- mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0);
/* disable illegal memory check */
mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_dummy_0, 1);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h
index 2c63c43ff065..f113c4c1068a 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h
@@ -8,7 +8,6 @@
#define CLDMA0_BASE_ADDR (0x1021C000)
#define CLDMA1_BASE_ADDR (0x1021E000)
-#define CLDMA4_BASE_ADDR (0x10224000)
#define CLDMA_RX_SKB_POOL_MAX_SIZE (64)
#define CLDMA_RX_SKB_RELOAD_THRESHOLD (16)
@@ -80,7 +79,6 @@
#define REG_CLDMA_L2RIMSR1 (0x0800 + 0x00FC)
#define REG_CLDMA_INT_EAP_USIP_MASK (0x0800 + 0x011C)
-#define REG_CLDMA_INT_WF_MASK (0x0800 + 0x0120)
#define REG_CLDMA_RQ1_GPD_DONE_CNT (0x0800 + 0x0174)
#define REG_CLDMA_TQ1_GPD_DONE_CNT (0x0800 + 0x0184)
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index 913705b341e3..aec1c5211c70 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -766,22 +766,34 @@ static int mtk_pci_dev_init(struct mtk_md_dev *mdev)
{
int ret;
- ret = mtk_trans_ctrl_init(mdev);
+ ret = mtk_fsm_init(mdev);
if (ret) {
- dev_err(mdev->dev, "Failed to initialize control plane: %d\n", ret);
+ dev_err(mdev->dev, "Failed to initialize FSM: %d\n", ret);
return ret;
}
+ ret = mtk_trans_ctrl_init(mdev);
+ if (ret)
+ goto free_fsm;
+
return 0;
+free_fsm:
+ mtk_fsm_exit(mdev);
+ return ret;
}
static void mtk_pci_dev_exit(struct mtk_md_dev *mdev)
{
+ mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_RM, 0, NULL, 0,
+ EVT_MODE_BLOCKING | EVT_MODE_TOHEAD);
mtk_trans_ctrl_exit(mdev);
+ mtk_fsm_exit(mdev);
}
static int mtk_pci_dev_start(struct mtk_md_dev *mdev)
{
+ mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_ADD, 0, NULL, 0, 0);
+ mtk_fsm_start(mdev);
return 0;
}
static const struct mtk_dev_ops pci_hw_ops = {
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
index c953729c5137..baa1d69f2e56 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -469,6 +469,15 @@ static int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb,
return 0;
}
+static void mtk_pcie_hif_fsm_indication(struct mtk_md_dev *mdev, struct mtk_fsm_param *param)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+ mtk_cldma_fsm_state_listener(param, trans);
+}
+
static int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data)
{
struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
@@ -498,6 +507,7 @@ static struct mtk_ctrl_hif_ops pcie_ctrl_ops = {
.init = mtk_pcie_hif_init,
.exit = mtk_pcie_hif_exit,
.submit_skb = mtk_pcie_hif_submit_skb,
+ .fsm_indication = mtk_pcie_hif_fsm_indication,
.send_cmd = mtk_pcie_hif_cmd_func,
};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
index cca8e6f1532e..38b0f40d6b90 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -29,7 +29,6 @@
enum mtk_hif_id {
CLDMA0,
CLDMA1,
- CLDMA4,
NR_CLDMA
};
--
2.34.1
^ permalink raw reply related
* [PATCH 08/11] net: wwan: t9xx: Add data plane transaction layer
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Data Path Modem AP Interface (DPMAIF) provides methods for
initialization, ring buffer management, ISR, control and
handling of TX/RX services' flows.
DPMAIF TX
It exposes the function 'mtk_dpmaif_send' which can be
called by the port layer indirectly to transmit packets.
The transaction layer manages uplink data with Descriptor
Ring Buffer (DRB), which includes one message DRB entry
and one or more normal DRB entries. Message DRB holds the
general packet information and each normal DRB entry holds
the address of the packet segment. At the same time, DPMAIF
provides multiple virtual queues with different priorities.
DPMAIF RX
The downlink buffer management uses Buffer Address Table
(BAT), which includes normal BAT and fragment BAT, and
Packet Information Table (PIT) rings. The BAT ring holds
the address of the skb data buffer for the hardware to use,
while the PIT contains metadata about a whole network packet
including a reference to the BAT entry holding the data
buffer address. The driver reads the PIT and BAT entries
written by the modem. When reaching a threshold, the driver
reloads the PIT and BAT rings.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 3 +-
drivers/net/wwan/t9xx/mtk_data_plane.c | 92 +
drivers/net/wwan/t9xx/mtk_data_plane.h | 103 +
drivers/net/wwan/t9xx/mtk_dev.h | 1 +
drivers/net/wwan/t9xx/pcie/Makefile | 1 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c | 2674 ++++++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h | 14 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c | 1 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h | 9 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c | 2 +-
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 8 +
drivers/net/wwan/t9xx/pcie/mtk_pci.h | 3 +-
12 files changed, 2907 insertions(+), 4 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index 75760b2039dc..3776ccb4952f 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -11,4 +11,5 @@ mtk_t9xx-y := \
mtk_ctrl_plane.o \
mtk_port.o \
mtk_port_io.o \
- mtk_fsm.o
+ mtk_fsm.o \
+ mtk_data_plane.o
diff --git a/drivers/net/wwan/t9xx/mtk_data_plane.c b/drivers/net/wwan/t9xx/mtk_data_plane.c
new file mode 100644
index 000000000000..d11c7de45e17
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_data_plane.c
@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#include "mtk_data_plane.h"
+#include "mtk_dev.h"
+#include "mtk_fsm.h"
+
+static void mtk_data_stop(struct mtk_data_blk *data_blk, struct mtk_md_dev *mdev)
+{
+ data_blk->hif_ops->stop(mdev);
+ data_blk->hif_ops->clear(mdev);
+}
+
+static void mtk_data_fsm_callback(struct mtk_fsm_param *fsm_param, void *data)
+{
+ struct mtk_data_blk *data_blk;
+
+ if (!data || !fsm_param)
+ return;
+
+ data_blk = ((struct mtk_md_dev *)data)->data_blk;
+
+ switch (fsm_param->to) {
+ case FSM_STATE_OFF:
+ mtk_data_stop(data_blk, data);
+ break;
+ case FSM_STATE_BOOTUP:
+ if (fsm_param->fsm_flag & FSM_F_MD_HS_START)
+ data_blk->hif_ops->start(data);
+ break;
+ case FSM_STATE_READY:
+ break;
+ default:
+ break;
+ }
+}
+
+int mtk_data_init(struct mtk_md_dev *mdev, struct mtk_data_hif_ops *ops)
+{
+ struct mtk_data_blk *data_blk;
+ int ret;
+
+ data_blk = devm_kzalloc(mdev->dev, sizeof(*data_blk), GFP_KERNEL);
+ if (!data_blk)
+ return -ENOMEM;
+
+ data_blk->mdev = mdev;
+ mdev->data_blk = data_blk;
+ data_blk->hif_ops = ops;
+
+ ret = data_blk->hif_ops->init(mdev);
+ if (ret < 0)
+ goto data_blk_free;
+
+ ret = mtk_fsm_notifier_register(mdev, MTK_USER_DATA, mtk_data_fsm_callback, mdev,
+ FSM_PRIO_1, false);
+ if (ret < 0) {
+ dev_err(mdev->dev, "Failed to register FSM notifier\n");
+ goto hif_exit;
+ }
+
+ return 0;
+
+hif_exit:
+ data_blk->hif_ops->exit(mdev);
+data_blk_free:
+ devm_kfree(mdev->dev, data_blk);
+ mdev->data_blk = NULL;
+
+ return ret;
+}
+EXPORT_SYMBOL(mtk_data_init);
+
+int mtk_data_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_data_blk *data_blk = mdev->data_blk;
+
+ if (!data_blk)
+ return -EINVAL;
+
+ mtk_fsm_notifier_unregister(mdev, MTK_USER_DATA);
+
+ data_blk->hif_ops->exit(mdev);
+
+ devm_kfree(mdev->dev, data_blk);
+ mdev->data_blk = NULL;
+
+ return 0;
+}
+EXPORT_SYMBOL(mtk_data_exit);
diff --git a/drivers/net/wwan/t9xx/mtk_data_plane.h b/drivers/net/wwan/t9xx/mtk_data_plane.h
new file mode 100644
index 000000000000..1464fab544f1
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_data_plane.h
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DATA_PLANE_H__
+#define __MTK_DATA_PLANE_H__
+
+#include <linux/bitops.h>
+#include <linux/completion.h>
+#include <linux/skbuff.h>
+
+#define SKB_TO_CMD(skb) ((struct mtk_data_cmd *)(skb)->data)
+#define CMD_TO_DATA(cmd) (*(void **)(cmd)->data)
+#define SKB_TO_CMD_DATA(skb) (*(void **)SKB_TO_CMD(skb)->data)
+
+#define IPV4_VERSION 0x40
+#define IPV6_VERSION 0x60
+
+enum mtk_data_feature {
+ DATA_F_MULTI_NETDEV = BIT(16),
+};
+
+struct mtk_data_trans_info {
+ u32 cap;
+ unsigned char rxq_cnt;
+ unsigned char txq_cnt;
+ unsigned int max_mtu;
+ struct napi_struct **napis;
+};
+
+struct mtk_data_blk {
+ struct mtk_md_dev *mdev;
+ void *dcb;
+ struct mtk_data_hif_ops *hif_ops;
+ struct mtk_data_trans_info trans_info;
+};
+
+enum mtk_data_type {
+ DATA_PKT,
+ DATA_CMD,
+};
+
+enum mtk_pkt_type {
+ PURE_IP,
+};
+
+enum mtk_data_cmd_type {
+ DATA_CMD_TRANS_CTL,
+};
+
+struct mtk_data_trans_ctl {
+ bool enable;
+};
+
+struct mtk_data_cmd {
+ enum mtk_data_cmd_type cmd;
+ unsigned int len;
+ char data[];
+};
+
+struct mtk_tx_pkt_info {
+ unsigned char intf_id;
+ unsigned char cnt;
+};
+
+struct mtk_rx_pkt_info {
+ unsigned char ch_id;
+ unsigned char q_id;
+};
+
+union mtk_data_pkt_info {
+ struct mtk_tx_pkt_info tx;
+ struct mtk_rx_pkt_info rx;
+};
+
+#define DATA_SKB_CB(__skb) ((union mtk_data_pkt_info *)&((__skb)->cb[0]))
+
+enum mtk_data_evt {
+ DATA_EVT_MIN,
+ DATA_EVT_TX_START,
+ DATA_EVT_TX_STOP,
+ DATA_EVT_RX_STOP,
+ DATA_EVT_REG_DEV,
+ DATA_EVT_UNREG_DEV,
+ DATA_EVT_MAX,
+};
+
+struct mtk_data_hif_ops {
+ int (*init)(struct mtk_md_dev *mdev);
+ int (*exit)(struct mtk_md_dev *mdev);
+ int (*stop)(struct mtk_md_dev *mdev);
+ int (*start)(struct mtk_md_dev *mdev);
+ void (*clear)(struct mtk_md_dev *mdev);
+ int (*poll)(struct napi_struct *napi, int budget);
+ int (*send)(struct mtk_data_blk *data_blk, enum mtk_data_type type,
+ struct sk_buff *skb);
+};
+
+int mtk_data_init(struct mtk_md_dev *mdev, struct mtk_data_hif_ops *ops);
+int mtk_data_exit(struct mtk_md_dev *mdev);
+
+#endif /* __MTK_DATA_PLANE_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
index bf412b49b19d..8913cbeed670 100644
--- a/drivers/net/wwan/t9xx/mtk_dev.h
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -58,6 +58,7 @@ struct mtk_md_dev {
u32 hw_ver;
char dev_str[MTK_DEV_STR_LEN];
void *ctrl_blk;
+ void *data_blk;
struct mtk_md_fsm *fsm;
};
diff --git a/drivers/net/wwan/t9xx/pcie/Makefile b/drivers/net/wwan/t9xx/pcie/Makefile
index 78a765f8e79a..961faed51498 100644
--- a/drivers/net/wwan/t9xx/pcie/Makefile
+++ b/drivers/net/wwan/t9xx/pcie/Makefile
@@ -12,6 +12,7 @@ mtk_t9xx_pcie-y := \
mtk_ctrl_cfg_m9xx.o \
mtk_pci.o \
mtk_trans_ctrl.o \
+ mtk_dpmaif.o \
mtk_cldma.o \
mtk_dpmaif_drv.o \
mtk_cldma_drv.o \
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
new file mode 100644
index 000000000000..1c58dba738fa
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c
@@ -0,0 +1,2674 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#define pr_fmt(fmt) "DATA_TRANS: " fmt
+
+#include <linux/bitfield.h>
+#include <linux/bitmap.h>
+#include <linux/ip.h>
+#include <linux/kthread.h>
+#include <linux/pm_wakeup.h>
+#include <linux/sched/clock.h>
+#include <linux/skbuff.h>
+#include <net/gro.h>
+#include <net/ipv6.h>
+#include <net/pkt_sched.h>
+
+#include "mtk_data_plane.h"
+#include "mtk_dev.h"
+#include "mtk_dpmaif.h"
+#include "mtk_dpmaif_drv.h"
+#include "mtk_dpmaif_ring.h"
+#include "mtk_pci_reg.h"
+#include "mtk_pci.h"
+
+#define MTK_DATA_WS_NAME_LEN 32
+#define DPMAIF_PIT_CNT_UPDATE_THRESHOLD 60
+#define DPMAIF_SKB_TX_WEIGHT 32
+#define DPMAIF_REL_BAT_WEIGHT 128
+#define MIN_BAT_BURST_CNT 64
+
+static struct dpmaif_drv_ops_desc dpmaif_drv_ops_tbl[] = {
+ {0x01CA, &drv_ops_name(m9xx)},
+ {0, NULL},
+};
+
+enum dpmaif_state {
+ DPMAIF_STATE_MIN,
+ DPMAIF_STATE_PWROFF,
+ DPMAIF_STATE_PWRON,
+ DPMAIF_STATE_MAX
+};
+
+struct dpmaif_vq {
+ unsigned char srv_id;
+ unsigned char q_id;
+ u32 max_len; /* align network tx qdisc 1000 */
+ struct sk_buff_head list;
+};
+
+struct dpmaif_tx_srv {
+ struct mtk_dpmaif_ctlb *dcb;
+ unsigned char id;
+ int nice;
+ wait_queue_head_t wait;
+ struct task_struct *srv;
+
+ unsigned long txq_drb_lack_sta;
+ unsigned char cur_vq_id;
+ unsigned char vq_cnt;
+ struct dpmaif_vq *vq[DPMAIF_TXQ_CNT_MAX];
+};
+
+struct dpmaif_drb_skb {
+ struct sk_buff *skb;
+ dma_addr_t data_dma_addr;
+ unsigned short data_len;
+ unsigned short drb_idx:13;
+ unsigned short is_msg:1;
+ unsigned short is_frag:1;
+ unsigned short is_last:1;
+};
+
+struct dpmaif_txq {
+ struct mtk_dpmaif_ctlb *dcb;
+ unsigned char id;
+ atomic_t budget;
+ atomic_t to_submit_cnt;
+ struct dpmaif_pd_drb *drb_base;
+ dma_addr_t drb_dma_addr;
+ unsigned int drb_cnt;
+ unsigned short drb_wr_idx;
+ unsigned short drb_rd_idx;
+ unsigned short drb_rel_rd_idx;
+ struct dpmaif_drb_skb *sw_drb_base;
+ struct delayed_work tx_done_work;
+};
+
+struct dpmaif_rx_record {
+ bool msg_pit_recv;
+ struct sk_buff *cur_skb;
+ struct sk_buff_head rx_list;
+ unsigned short ip_protocol;
+};
+
+struct dpmaif_rxq {
+ struct mtk_dpmaif_ctlb *dcb;
+ unsigned char id;
+ bool started;
+ struct dpmaif_pd_pit *pit_base;
+ dma_addr_t pit_dma_addr;
+ unsigned int pit_cnt;
+ unsigned short pit_wr_idx;
+ unsigned short pit_rd_idx;
+ unsigned short pit_rel_rd_idx;
+ unsigned char pit_seq_expect;
+ atomic_t pit_rel_cnt;
+ atomic_t pit_stats;
+ bool pit_cnt_err_intr_set;
+ unsigned int pit_burst_rel_cnt;
+ unsigned int pit_seq_fail_cnt;
+ struct napi_struct napi;
+ struct dpmaif_rx_record rx_record;
+ /* Record the latest BID polled by this DLQ pit ring. */
+ unsigned int pit_bid;
+ unsigned char bat_ring_id;
+ unsigned int pit_seq_max;
+ struct dpmaif_rx_info *rx_info;
+ struct wakeup_source *ws;
+};
+
+struct skb_mapped_t {
+ struct sk_buff *skb;
+ dma_addr_t data_dma_addr;
+ unsigned int data_len;
+};
+
+union dpmaif_bat_record {
+ struct skb_mapped_t normal;
+};
+
+struct dpmaif_bat_ring {
+ unsigned char id;
+ struct dpmaif_bat *bat_base;
+ dma_addr_t bat_dma_addr;
+ unsigned int bat_cnt;
+ unsigned short bat_wr_idx;
+ unsigned short bat_rd_idx;
+ /* current max relaod bat cnt */
+ unsigned short max_reload_cnt;
+ atomic_t to_reload_cnt;
+ /* reloaded bat cnt, not doorbelled */
+ atomic_t reload_cnt;
+ atomic_t bat_stats;
+ union dpmaif_bat_record *sw_record_base;
+ unsigned int buf_size;
+ unsigned long *mask_tbl;
+ bool bat_cnt_err_intr_set;
+ int (*alloc)(struct mtk_dpmaif_ctlb *dcb,
+ struct dpmaif_bat_ring *bat_ring,
+ unsigned short bat_idx);
+};
+
+struct dpmaif_task_ctlb {
+ wait_queue_head_t wait;
+ bool need_wp;
+};
+
+struct dpmaif_bat_info {
+ struct mtk_dpmaif_ctlb *dcb;
+ unsigned int max_mtu;
+
+ struct dpmaif_bat_ring normal_bat_ring;
+ struct task_struct *reload_task;
+ struct dpmaif_task_ctlb task_ctlb;
+};
+
+struct dpmaif_irq_param {
+ unsigned char idx;
+ struct mtk_dpmaif_ctlb *dcb;
+ enum mtk_irq_src dpmaif_irq_src;
+ int dev_irq_id;
+ int dev_virq_id;
+};
+
+struct dpmaif_tx_evt {
+ unsigned long long ul_done;
+};
+
+struct dpmaif_rx_evt {
+ unsigned long long dl_done;
+};
+
+struct mtk_dpmaif_ctlb {
+ struct mtk_data_blk *data_blk;
+ struct dpmaif_drv_info *drv_info;
+ struct napi_struct *napi[DPMAIF_RXQ_CNT_MAX];
+
+ enum dpmaif_state dpmaif_state;
+ struct mutex trans_ctl_lock; /* protect structure fields */
+
+ struct dpmaif_tx_srv *tx_srvs;
+ struct dpmaif_vq *tx_vqs;
+
+ struct workqueue_struct *tx_done_wq;
+ struct task_struct *db_thread;
+ struct dpmaif_task_ctlb db_task_ctlb;
+ struct dpmaif_txq *txqs;
+ struct dpmaif_rxq *rxqs;
+ struct dpmaif_bat_info *bat_infos;
+ bool irq_enabled;
+ struct dpmaif_irq_param *irq_params;
+
+ bool dpmaif_sw_reset;
+ unsigned char rxq_cnt;
+ unsigned char txq_cnt;
+};
+
+#define DCB_TO_DEV(dcb) ((dcb)->data_blk->mdev->dev)
+#define DCB_TO_MDEV(dcb) ((dcb)->data_blk->mdev)
+#define DCB_TO_DEV_STR(dcb) ((dcb)->data_blk->mdev->dev_str)
+#define DPMAIF_GET_HW_VER(dcb) ((dcb)->data_blk->mdev->hw_ver)
+#define DPMAIF_GET_DRB_CNT(__skb) (skb_shinfo(__skb)->nr_frags + 1 + 1)
+#define DPMAIF_JUMBO_SIZE 9000
+#define DPMAIF_DFLT_MTU 3000
+#define DPMAIF_DL_BUF_MIN_SIZE 128
+
+static int mtk_dpmaif_rx_napi_poll(struct napi_struct *napi, int budget);
+
+static int mtk_dpmaif_send(struct mtk_data_blk *data_blk, enum mtk_data_type type,
+ struct sk_buff *skb);
+
+static bool mtk_pci_mmio_check(struct mtk_md_dev *mdev)
+{
+ return mtk_pci_mac_read32(mdev->hw_priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB) == (u32)-1;
+}
+
+static void mtk_dpmaif_common_err_handle(struct mtk_dpmaif_ctlb *dcb, bool is_hw)
+{
+ if (!is_hw) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "ASSERT file: %s, function: %s, line %d",
+ __FILE__, __func__, __LINE__);
+ return;
+ }
+
+ if (mtk_pci_mmio_check(DCB_TO_MDEV(dcb)))
+ mtk_pci_send_ext_evt(DCB_TO_MDEV(dcb), EXT_EVT_H2D_RESERVED_FOR_DPMAIF);
+ else
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to access mmio\n");
+}
+
+static void mtk_dpmaif_disable_irq(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char irq_cnt = dcb->drv_info->cfg->intr_cfg.irq_cnt;
+ struct dpmaif_irq_param *irq_param;
+ int i;
+
+ if (!dcb->irq_enabled)
+ return;
+
+ dcb->irq_enabled = false;
+ for (i = 0; i < irq_cnt; i++) {
+ irq_param = &dcb->irq_params[i];
+ if (mtk_pci_mask_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id) != 0)
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to mask dev irq%d\n", irq_param->dev_irq_id);
+ synchronize_irq(irq_param->dev_virq_id);
+ }
+}
+
+static void mtk_dpmaif_enable_irq(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char irq_cnt = dcb->drv_info->cfg->intr_cfg.irq_cnt;
+ struct dpmaif_irq_param *irq_param;
+ int i;
+
+ if (dcb->irq_enabled)
+ return;
+
+ dcb->irq_enabled = true;
+ for (i = 0; i < irq_cnt; i++) {
+ irq_param = &dcb->irq_params[i];
+ if (mtk_pci_unmask_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id) != 0)
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to unmask dev irq%d\n", irq_param->dev_irq_id);
+ }
+}
+
+static int mtk_dpmaif_alloc_skb(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_bat_ring *bat_ring,
+ unsigned short bat_idx)
+{
+ union dpmaif_bat_record *cur_bat_record;
+ struct skb_mapped_t *skb_info;
+ struct dpmaif_bat *cur_bat;
+
+ cur_bat_record = bat_ring->sw_record_base + bat_idx;
+ skb_info = &cur_bat_record->normal;
+
+ if (skb_info->skb)
+ return 0;
+
+ skb_info->skb = __dev_alloc_skb(bat_ring->buf_size, GFP_ATOMIC);
+ if (unlikely(!skb_info->skb))
+ return -ENOMEM;
+
+ skb_info->data_len = bat_ring->buf_size;
+ skb_info->data_dma_addr = dma_map_single(DCB_TO_DEV(dcb), skb_info->skb->data,
+ skb_info->data_len, DMA_FROM_DEVICE);
+ if (dma_mapping_error(DCB_TO_DEV(dcb), skb_info->data_dma_addr)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to map dma!\n");
+ dev_kfree_skb_any(skb_info->skb);
+ skb_info->skb = NULL;
+ return -ENOMEM;
+ }
+
+ cur_bat = bat_ring->bat_base + bat_idx;
+ cur_bat->buf_addr_high = cpu_to_le32(upper_32_bits(skb_info->data_dma_addr));
+ cur_bat->buf_addr_low = cpu_to_le32(lower_32_bits(skb_info->data_dma_addr));
+
+ return 0;
+}
+
+static int mtk_dpmaif_reload_rx_buff(struct mtk_dpmaif_ctlb *dcb,
+ struct dpmaif_bat_ring *bat_ring, unsigned int to_reload_cnt)
+{
+ unsigned short cur_bat_idx;
+ int ret = -ENOMEM;
+ unsigned int i;
+
+ /* Pin rx buffers to BAT entries */
+ cur_bat_idx = bat_ring->bat_wr_idx;
+ for (i = 0; i < to_reload_cnt; i++) {
+ if (!test_and_clear_bit(cur_bat_idx, bat_ring->mask_tbl)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ ret = bat_ring->alloc(dcb, bat_ring, cur_bat_idx);
+ if (unlikely(ret)) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev,
+ "Failed to alloc rx buff, bat%u bid=%u\n",
+ bat_ring->id, cur_bat_idx);
+ set_bit(cur_bat_idx, bat_ring->mask_tbl);
+ break;
+ }
+
+ cur_bat_idx = mtk_dpmaif_ring_buf_get_next_idx(bat_ring->bat_cnt, cur_bat_idx);
+ }
+
+ if (unlikely(!i))
+ return ret;
+
+ dma_wmb();
+ atomic_add(i, &bat_ring->reload_cnt);
+ atomic_sub(i, &bat_ring->to_reload_cnt);
+ bat_ring->bat_wr_idx = cur_bat_idx;
+
+ return ret;
+}
+
+static void mtk_dpmaif_preload_rx_buf(struct mtk_dpmaif_ctlb *dcb,
+ struct dpmaif_bat_ring *bat_ring, unsigned int preload_cnt)
+{
+ int i, idx, ret;
+
+ idx = bat_ring->bat_wr_idx;
+ for (i = 0; i < preload_cnt; i++) {
+ /* the next set bit means this buff consumed */
+ idx = find_next_bit(bat_ring->mask_tbl, bat_ring->bat_cnt, idx + 1);
+ if (idx >= bat_ring->bat_cnt)
+ goto find_from_begin;
+
+ ret = bat_ring->alloc(dcb, bat_ring, idx);
+ if (ret) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev,
+ "Failed to preload rx buff, bat%u bid=%u\n",
+ bat_ring->id, idx);
+ break;
+ }
+ }
+
+ goto out;
+
+find_from_begin:
+ idx = -1;
+ for (; i < preload_cnt; i++) {
+ idx = find_next_bit(bat_ring->mask_tbl, bat_ring->bat_wr_idx, idx + 1);
+ if (idx >= bat_ring->bat_wr_idx)
+ break;
+
+ ret = bat_ring->alloc(dcb, bat_ring, idx);
+ if (ret) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev,
+ "Failed to preload rx buff, bat%u bid=%u\n",
+ bat_ring->id, idx);
+ break;
+ }
+ }
+
+out:;
+}
+
+static void mtk_dpmaif_task_wakeup(struct dpmaif_task_ctlb *task_ctlb)
+{
+ task_ctlb->need_wp = true;
+ wake_up(&task_ctlb->wait);
+}
+
+static int mtk_dpmaif_reload_bat(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_bat_ring *bat_ring)
+{
+ int pre_reload_cnt;
+ int to_reload_cnt;
+ int ret = 0;
+
+ to_reload_cnt = atomic_read(&bat_ring->to_reload_cnt);
+ while (to_reload_cnt > 0) {
+ pre_reload_cnt = to_reload_cnt > DPMAIF_REL_BAT_WEIGHT ?
+ DPMAIF_REL_BAT_WEIGHT : to_reload_cnt;
+
+ ret = mtk_dpmaif_reload_rx_buff(dcb, bat_ring, pre_reload_cnt);
+ if (unlikely(ret))
+ break;
+ if (atomic_read(&bat_ring->reload_cnt) >= MIN_BAT_BURST_CNT)
+ mtk_dpmaif_task_wakeup(&dcb->db_task_ctlb);
+ to_reload_cnt -= pre_reload_cnt;
+ }
+
+ if ((ret || bat_ring->bat_cnt_err_intr_set) && atomic_read(&bat_ring->reload_cnt))
+ mtk_dpmaif_task_wakeup(&dcb->db_task_ctlb);
+
+ /* Prefetch rx buff to accelerate next reload work */
+ if (ret == -EINVAL) {
+ mtk_dpmaif_preload_rx_buf(dcb, bat_ring, to_reload_cnt);
+ ret = 0;
+ }
+
+ return ret;
+}
+
+static int mtk_dpmaif_bat_reload_thread(void *arg)
+{
+ struct dpmaif_bat_info *bat_info = arg;
+ struct dpmaif_task_ctlb *task_ctlb;
+ struct dpmaif_bat_ring *bat_ring;
+ int ret;
+
+ task_ctlb = &bat_info->task_ctlb;
+
+ while (1) {
+ ret = wait_event_interruptible(task_ctlb->wait,
+ task_ctlb->need_wp ||
+ kthread_should_stop());
+ task_ctlb->need_wp = false;
+ if (unlikely(ret == -ERESTARTSYS))
+ continue;
+
+ if (kthread_should_stop())
+ break;
+
+ bat_ring = &bat_info->normal_bat_ring;
+ if (atomic_read(&bat_ring->to_reload_cnt) > 0)
+ mtk_dpmaif_reload_bat(bat_info->dcb, bat_ring);
+
+ cond_resched();
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_bat_ring_init(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_bat_ring *bat_ring,
+ int bat_ring_id)
+{
+ int ret;
+
+ bat_ring->bat_cnt = dcb->drv_info->cfg->rx_cfg.bats[bat_ring_id].bat_cnt;
+ bat_ring->max_reload_cnt = dcb->drv_info->cfg->rx_cfg.bats[bat_ring_id].reload_cnt;
+ bat_ring->alloc = mtk_dpmaif_alloc_skb;
+
+ bat_ring->id = bat_ring_id;
+ bat_ring->bat_cnt_err_intr_set = false;
+ atomic_set(&bat_ring->to_reload_cnt, bat_ring->max_reload_cnt);
+
+ /* Allocate BAT memory for HW and SW. */
+ bat_ring->bat_base = dma_alloc_coherent(DCB_TO_DEV(dcb), (bat_ring->bat_cnt +
+ dcb->drv_info->cfg->rx_cfg.bat_wrap_cnt) *
+ sizeof(*bat_ring->bat_base),
+ &bat_ring->bat_dma_addr, GFP_KERNEL);
+ if (!bat_ring->bat_base) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to allocate bat%u buffer\n", bat_ring->id);
+ return -ENOMEM;
+ }
+
+ /* Allocate buffer for SW to record skb information */
+ bat_ring->sw_record_base = devm_kcalloc(DCB_TO_DEV(dcb), bat_ring->bat_cnt,
+ sizeof(*bat_ring->sw_record_base), GFP_KERNEL);
+ if (!bat_ring->sw_record_base) {
+ ret = -ENOMEM;
+ goto free_bat_buf;
+ }
+
+ /* Allocate buffer for SW to recycle BAT. */
+ bat_ring->mask_tbl = bitmap_alloc(bat_ring->bat_cnt, GFP_KERNEL);
+ if (!bat_ring->mask_tbl) {
+ ret = -ENOMEM;
+ goto free_sw_record_base;
+ }
+
+ bitmap_fill(bat_ring->mask_tbl, bat_ring->bat_cnt);
+
+ return 0;
+
+free_sw_record_base:
+ devm_kfree(DCB_TO_DEV(dcb), bat_ring->sw_record_base);
+
+free_bat_buf:
+ dma_free_coherent(DCB_TO_DEV(dcb), (bat_ring->bat_cnt +
+ dcb->drv_info->cfg->rx_cfg.bat_wrap_cnt) *
+ sizeof(*bat_ring->bat_base),
+ bat_ring->bat_base, bat_ring->bat_dma_addr);
+
+ return ret;
+}
+
+static void mtk_dpmaif_bat_ring_exit(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_bat_ring *bat_ring)
+{
+ union dpmaif_bat_record *bat_record;
+ int i;
+
+ bitmap_free(bat_ring->mask_tbl);
+
+ for (i = 0; i < bat_ring->bat_cnt; i++) {
+ bat_record = bat_ring->sw_record_base + i;
+ if (!bat_record->normal.skb)
+ continue;
+
+ dma_unmap_single(DCB_TO_DEV(dcb),
+ bat_record->normal.data_dma_addr,
+ bat_record->normal.data_len,
+ DMA_FROM_DEVICE);
+ dev_kfree_skb_any(bat_record->normal.skb);
+ }
+
+ devm_kfree(DCB_TO_DEV(dcb), bat_ring->sw_record_base);
+
+ dma_free_coherent(DCB_TO_DEV(dcb), (bat_ring->bat_cnt +
+ dcb->drv_info->cfg->rx_cfg.bat_wrap_cnt) *
+ sizeof(*bat_ring->bat_base),
+ bat_ring->bat_base, bat_ring->bat_dma_addr);
+}
+
+static void mtk_dpmaif_bat_ring_reset(struct dpmaif_bat_ring *bat_ring)
+{
+ bat_ring->bat_cnt_err_intr_set = false;
+ bat_ring->bat_wr_idx = 0;
+ bat_ring->bat_rd_idx = 0;
+ atomic_set(&bat_ring->reload_cnt, 0);
+ atomic_set(&bat_ring->bat_stats, 0);
+ atomic_set(&bat_ring->to_reload_cnt, bat_ring->max_reload_cnt);
+
+ bitmap_fill(bat_ring->mask_tbl, bat_ring->bat_cnt);
+}
+
+static void mtk_dpmaif_bat_res_reset(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned int bat_ring_num = dcb->drv_info->cfg->rx_cfg.bat_ring_num;
+ struct dpmaif_bat_ring *bat_ring;
+ int i;
+
+ for (i = 0; i < bat_ring_num; i++) {
+ bat_ring = &dcb->bat_infos[i].normal_bat_ring;
+ bat_ring->max_reload_cnt = dcb->drv_info->cfg->rx_cfg.bats[i].reload_cnt;
+ mtk_dpmaif_bat_ring_reset(bat_ring);
+ }
+}
+
+static void mtk_dpmaif_set_bat_buf_size(struct mtk_dpmaif_ctlb *dcb)
+{
+ int bat_ring_num = dcb->drv_info->cfg->rx_cfg.bat_ring_num;
+ struct dpmaif_bat_info *bat_infos = dcb->bat_infos;
+ unsigned int buf_size;
+ int i;
+
+ for (i = 0; i < bat_ring_num; i++) {
+ bat_infos[i].max_mtu = dcb->drv_info->cfg->rx_cfg.mtu;
+
+ /* Set max mtu, DPMAIF_JUMBO_SIZE. */
+ if (bat_infos[i].max_mtu > DPMAIF_JUMBO_SIZE)
+ bat_infos[i].max_mtu = DPMAIF_JUMBO_SIZE;
+
+ /* Normal BAT buffer size setting. */
+ buf_size = bat_infos[i].max_mtu + dcb->drv_info->cfg->rx_cfg.pkt_alignment +
+ dcb->drv_info->cfg->rx_cfg.normal_bat_rsv_length;
+
+ bat_infos[i].normal_bat_ring.buf_size = ALIGN(buf_size,
+ DPMAIF_DL_BUF_MIN_SIZE);
+ }
+ dcb->drv_info->cfg->rx_cfg.mtu = dcb->bat_infos[0].max_mtu;
+}
+
+static void mtk_dpmaif_bat_info_exit(struct mtk_dpmaif_ctlb *dcb, int bat_ring_id)
+{
+ struct dpmaif_bat_info *bat_info = &dcb->bat_infos[bat_ring_id];
+
+ if (bat_info->reload_task) {
+ kthread_stop(bat_info->reload_task);
+ bat_info->reload_task = NULL;
+ }
+
+ mtk_dpmaif_bat_ring_exit(dcb, &bat_info->normal_bat_ring);
+}
+
+static void mtk_dpmaif_bat_res_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ int bat_ring_num = dcb->drv_info->cfg->rx_cfg.bat_ring_num;
+ int i;
+
+ for (i = 0; i < bat_ring_num; i++)
+ mtk_dpmaif_bat_info_exit(dcb, i);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->bat_infos);
+}
+
+static int mtk_dpmaif_bat_info_init(struct mtk_dpmaif_ctlb *dcb,
+ struct dpmaif_bat_info *bat_info, int bat_ring_id)
+{
+ int ret;
+
+ bat_info->dcb = dcb;
+ ret = mtk_dpmaif_bat_ring_init(dcb, &bat_info->normal_bat_ring, bat_ring_id);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to initialize normal bat ring\n");
+ return ret;
+ }
+
+ init_waitqueue_head(&bat_info->task_ctlb.wait);
+ bat_info->reload_task = kthread_run(mtk_dpmaif_bat_reload_thread, bat_info,
+ "dpmaif_reload_%d", bat_ring_id);
+ if (IS_ERR(bat_info->reload_task)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to create bat reload thread\n");
+ ret = PTR_ERR(bat_info->reload_task);
+ bat_info->reload_task = NULL;
+ mtk_dpmaif_bat_ring_exit(dcb, &bat_info->normal_bat_ring);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_bat_res_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ int bat_ring_num = dcb->drv_info->cfg->rx_cfg.bat_ring_num;
+ int i, j;
+ int ret;
+
+ dcb->bat_infos = devm_kcalloc(DCB_TO_DEV(dcb), bat_ring_num, sizeof(*dcb->bat_infos),
+ GFP_KERNEL);
+ if (!dcb->bat_infos)
+ return -ENOMEM;
+
+ /* Check and set bat buffer size. */
+ mtk_dpmaif_set_bat_buf_size(dcb);
+
+ for (i = 0; i < bat_ring_num; i++) {
+ ret = mtk_dpmaif_bat_info_init(dcb, &dcb->bat_infos[i], i);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to initialize bat_info%d\n", i);
+ goto bat_info_exit;
+ }
+ }
+
+ return 0;
+
+bat_info_exit:
+ for (j = i - 1; j >= 0; j--)
+ mtk_dpmaif_bat_info_exit(dcb, j);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->bat_infos);
+
+ return ret;
+}
+
+static int mtk_dpmaif_rxq_init(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_rxq *rxq)
+{
+ char ws_name[MTK_DATA_WS_NAME_LEN];
+ int ret;
+
+ rxq->pit_seq_max = dcb->drv_info->cfg->rx_cfg.rxqs[rxq->id].pit_seq_max;
+ rxq->bat_ring_id = dcb->drv_info->cfg->rx_cfg.rxqs[rxq->id].bat_ring_id;
+ rxq->pit_cnt = dcb->drv_info->cfg->rx_cfg.rxqs[rxq->id].pit_cnt;
+ rxq->pit_burst_rel_cnt = DPMAIF_PIT_CNT_UPDATE_THRESHOLD;
+
+ snprintf(ws_name, sizeof(ws_name), "dpmaif_rxq%d_ws", rxq->id);
+
+ rxq->ws = wakeup_source_register(NULL, ws_name);
+ if (!rxq->ws) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to register rxq%d wakeup source\n", rxq->id);
+ return -ENOMEM;
+ }
+
+ rxq->pit_base = dma_alloc_coherent(DCB_TO_DEV(dcb),
+ rxq->pit_cnt * sizeof(*rxq->pit_base),
+ &rxq->pit_dma_addr, GFP_KERNEL);
+ if (!rxq->pit_base) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to allocate rxq%u pit resource\n", rxq->id);
+ ret = -ENOMEM;
+ goto unregister_ws;
+ }
+
+ __skb_queue_head_init(&rxq->rx_record.rx_list);
+
+ return 0;
+unregister_ws:
+ wakeup_source_unregister(rxq->ws);
+
+ return ret;
+}
+
+static void mtk_dpmaif_rxq_exit(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_rxq *rxq)
+{
+ dma_free_coherent(DCB_TO_DEV(dcb), rxq->pit_cnt * sizeof(*rxq->pit_base),
+ rxq->pit_base, rxq->pit_dma_addr);
+ wakeup_source_unregister(rxq->ws);
+}
+
+static int mtk_dpmaif_sw_stop_rxq(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_rxq *rxq)
+{
+ /* Rxq done process will check this flag, if rxq->started is false, process will stop. */
+ rxq->started = false;
+
+ /* Make sure rxq->started value update done. */
+ smp_mb();
+
+ /* Wait rxq process done. */
+ napi_synchronize(&rxq->napi);
+
+ return 0;
+}
+
+static void mtk_dpmaif_sw_stop_rx(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_rxq *rxq;
+ int i;
+
+ /* Stop all rx process. */
+ for (i = 0; i < dcb->rxq_cnt; i++) {
+ rxq = &dcb->rxqs[i];
+ mtk_dpmaif_sw_stop_rxq(dcb, rxq);
+ }
+
+ /* Stop PIT polling NAPI. */
+}
+
+static void mtk_dpmaif_sw_start_rx(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_rxq *rxq;
+ int i;
+
+ /* Start PIT polling NAPI. */
+
+ for (i = 0; i < dcb->rxq_cnt; i++) {
+ rxq = &dcb->rxqs[i];
+ rxq->started = true;
+ }
+}
+
+static void mtk_dpmaif_sw_reset_rxq(struct dpmaif_rxq *rxq)
+{
+ memset(rxq->pit_base, 0x00, (rxq->pit_cnt * sizeof(*rxq->pit_base)));
+ memset(&rxq->rx_record, 0x00, sizeof(rxq->rx_record));
+ __skb_queue_head_init(&rxq->rx_record.rx_list);
+
+ rxq->started = false;
+ rxq->pit_wr_idx = 0;
+ rxq->pit_rd_idx = 0;
+ rxq->pit_rel_rd_idx = 0;
+ rxq->pit_seq_expect = 0;
+ atomic_set(&rxq->pit_rel_cnt, 0);
+ rxq->pit_cnt_err_intr_set = false;
+ rxq->pit_seq_fail_cnt = 0;
+}
+
+static void mtk_dpmaif_rx_res_reset(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_rxq *rxq;
+ int i;
+
+ for (i = 0; i < dcb->rxq_cnt; i++) {
+ rxq = &dcb->rxqs[i];
+ mtk_dpmaif_sw_reset_rxq(rxq);
+ }
+}
+
+static int mtk_dpmaif_rx_res_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_rxq *rxq;
+ int i, j;
+ int ret;
+
+ dcb->rxqs = devm_kcalloc(DCB_TO_DEV(dcb), dcb->rxq_cnt, sizeof(*rxq), GFP_KERNEL);
+ if (!dcb->rxqs)
+ return -ENOMEM;
+
+ for (i = 0; i < dcb->rxq_cnt; i++) {
+ rxq = &dcb->rxqs[i];
+ rxq->id = i;
+ rxq->dcb = dcb;
+ ret = mtk_dpmaif_rxq_init(dcb, rxq);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to init rxq%u resource\n", rxq->id);
+ goto exit_rxq;
+ }
+ }
+
+ return 0;
+
+exit_rxq:
+ for (j = i - 1; j >= 0; j--)
+ mtk_dpmaif_rxq_exit(dcb, &dcb->rxqs[j]);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->rxqs);
+
+ return ret;
+}
+
+static void mtk_dpmaif_rx_res_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ int i;
+
+ for (i = 0; i < dcb->rxq_cnt; i++)
+ mtk_dpmaif_rxq_exit(dcb, &dcb->rxqs[i]);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->rxqs);
+}
+
+static unsigned int mtk_dpmaif_poll_tx_drb(struct dpmaif_txq *txq)
+{
+ unsigned short old_sw_rd_idx, new_hw_rd_idx;
+ struct mtk_dpmaif_ctlb *dcb = txq->dcb;
+ unsigned int drb_cnt;
+ int ret;
+
+ old_sw_rd_idx = txq->drb_rd_idx;
+ ret = dcb->drv_info->drv_ops->get_ring_idx(dcb->drv_info, DPMAIF_DRB_RIDX, txq->id);
+ if (unlikely(ret < 0)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to read txq%u drb_rd_idx, ret=%d\n", txq->id, ret);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ return 0;
+ }
+
+ new_hw_rd_idx = ret;
+
+ if (old_sw_rd_idx <= new_hw_rd_idx)
+ drb_cnt = new_hw_rd_idx - old_sw_rd_idx;
+ else
+ drb_cnt = txq->drb_cnt - old_sw_rd_idx + new_hw_rd_idx;
+
+ txq->drb_rd_idx = new_hw_rd_idx;
+
+ return drb_cnt;
+}
+
+static int mtk_dpmaif_tx_rel_internal(struct dpmaif_txq *txq,
+ unsigned int rel_cnt, unsigned int *real_rel_cnt)
+{
+ struct dpmaif_pd_drb *cur_drb = NULL, *drb_base = txq->drb_base;
+ struct mtk_dpmaif_ctlb *dcb = txq->dcb;
+ struct dpmaif_drb_skb *cur_drb_skb;
+ struct dpmaif_tx_srv *tx_srv;
+ struct sk_buff *skb_free;
+ unsigned short cur_idx;
+ unsigned char srv_id;
+ unsigned int i;
+
+ cur_idx = txq->drb_rel_rd_idx;
+ for (i = 0; i < rel_cnt; i++) {
+ cur_drb = drb_base + cur_idx;
+ cur_drb_skb = txq->sw_drb_base + cur_idx;
+ if (cur_drb_skb->is_msg == PD_DRB) {
+ dma_unmap_single(DCB_TO_DEV(dcb), cur_drb_skb->data_dma_addr,
+ cur_drb_skb->data_len, DMA_TO_DEVICE);
+
+ /* The last one drb entry of one tx packet, so, skb will be released. */
+ if (cur_drb_skb->is_last) {
+ skb_free = cur_drb_skb->skb;
+ if (likely(skb_free)) {
+ dev_consume_skb_any(skb_free);
+ } else {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to free skb,txq%u pkt%u,w=%u,r=%u,rel=%u,cnt=%u\n",
+ txq->id, cur_idx, txq->drb_wr_idx,
+ txq->drb_rd_idx, txq->drb_rel_rd_idx, rel_cnt);
+ mtk_dpmaif_common_err_handle(dcb, false);
+ }
+ }
+ }
+
+ cur_drb_skb->skb = NULL;
+ cur_idx = mtk_dpmaif_ring_buf_get_next_idx(txq->drb_cnt, cur_idx);
+ txq->drb_rel_rd_idx = cur_idx;
+ atomic_inc(&txq->budget);
+ }
+
+ *real_rel_cnt = i;
+
+ if (likely(cur_drb)) {
+ if (unlikely(!cur_drb_skb->is_last)) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev,
+ "txq%u done, last one c_bit != 0\n", txq->id);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ }
+ }
+
+ if (atomic_read(&txq->budget) > txq->drb_cnt >> 3) {
+ srv_id = dcb->tx_vqs[txq->id].srv_id;
+ tx_srv = &dcb->tx_srvs[srv_id];
+ clear_bit(txq->id, &tx_srv->txq_drb_lack_sta);
+ wake_up(&tx_srv->wait);
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_tx_rel(struct dpmaif_txq *txq)
+{
+ unsigned int real_rel_cnt = 0;
+ int ret = 0, rel_cnt;
+
+ mtk_dpmaif_poll_tx_drb(txq);
+
+ rel_cnt = mtk_dpmaif_ring_buf_releasable(txq->drb_cnt, txq->drb_rel_rd_idx,
+ txq->drb_rd_idx);
+
+ if (likely(rel_cnt > 0)) {
+ /* Release tx data buffer. */
+ ret = mtk_dpmaif_tx_rel_internal(txq, rel_cnt, &real_rel_cnt);
+ }
+
+ return ret;
+}
+
+static void mtk_dpmaif_tx_done(struct work_struct *work)
+{
+ struct delayed_work *dwork = to_delayed_work(work);
+ struct mtk_dpmaif_ctlb *dcb;
+ struct dpmaif_txq *txq;
+
+ txq = container_of(dwork, struct dpmaif_txq, tx_done_work);
+ dcb = txq->dcb;
+
+ /* Recycle drb and release hardware tx done buffer around drb. */
+ mtk_dpmaif_tx_rel(txq);
+
+ mtk_dpmaif_drv_intr_complete(dcb->drv_info, DPMAIF_INTR_UL_DONE,
+ txq->id, DPMAIF_UNMASK_INTR);
+}
+
+static int mtk_dpmaif_doorbell_thread(void *arg)
+{
+ struct mtk_dpmaif_ctlb *dcb = arg;
+ int bat_ring_num = dcb->drv_info->cfg->rx_cfg.bat_ring_num;
+ int doorbell_cnt, i, ret;
+
+ while (1) {
+ ret = wait_event_interruptible(dcb->db_task_ctlb.wait,
+ dcb->db_task_ctlb.need_wp ||
+ kthread_should_stop());
+ dcb->db_task_ctlb.need_wp = false;
+ if (unlikely(ret == -ERESTARTSYS))
+ continue;
+
+ if (kthread_should_stop())
+ break;
+
+ /* TX DRB doorbell */
+ for (i = 0; i < dcb->txq_cnt; i++) {
+ doorbell_cnt = atomic_read(&dcb->txqs[i].to_submit_cnt);
+ if (doorbell_cnt > 0) {
+ ret = mtk_dpmaif_drv_send_doorbell(dcb->drv_info,
+ DPMAIF_DRB, i,
+ doorbell_cnt);
+ if (unlikely(ret < 0)) {
+ dev_err(DCB_TO_DEV(dcb),
+ "Failed to send txq%d doorbell\n", i);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ break;
+ }
+ atomic_sub(doorbell_cnt, &dcb->txqs[i].to_submit_cnt);
+ }
+ }
+
+ /* BAT doorbell */
+ for (i = 0; i < bat_ring_num; i++) {
+ struct dpmaif_bat_ring *bat_ring = &dcb->bat_infos[i].normal_bat_ring;
+
+ /* Send BAT doorbell to notify HW about newly reloaded BAT entries. */
+ doorbell_cnt = atomic_read(&bat_ring->reload_cnt);
+ if (doorbell_cnt > 0) {
+ ret = mtk_dpmaif_drv_send_doorbell(dcb->drv_info,
+ DPMAIF_BAT,
+ bat_ring->id,
+ doorbell_cnt);
+ if (unlikely(ret < 0)) {
+ dev_err(DCB_TO_DEV(dcb),
+ "Failed to send bat%u doorbell\n", bat_ring->id);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ continue;
+ }
+
+ atomic_sub(doorbell_cnt, &bat_ring->reload_cnt);
+ if (bat_ring->bat_cnt_err_intr_set) {
+ bat_ring->bat_cnt_err_intr_set = false;
+ mtk_dpmaif_drv_intr_complete(dcb->drv_info,
+ DPMAIF_INTR_DL_BATCNT_LEN_ERR,
+ bat_ring->id, 0);
+ }
+ }
+ }
+
+ /* PIT doorbell for all RX queues */
+ for (i = 0; i < dcb->rxq_cnt; i++) {
+ struct dpmaif_rxq *rxq = &dcb->rxqs[i];
+
+ doorbell_cnt = atomic_read(&rxq->pit_rel_cnt);
+ if (doorbell_cnt >= 0) {
+ ret = mtk_dpmaif_drv_send_doorbell(dcb->drv_info,
+ DPMAIF_PIT, rxq->id,
+ doorbell_cnt);
+ if (unlikely(ret < 0)) {
+ dev_err(DCB_TO_DEV(dcb),
+ "Failed to send rxq%d pit doorbell\n", i);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ } else {
+ atomic_sub(doorbell_cnt, &rxq->pit_rel_cnt);
+ }
+ }
+ }
+
+ cond_resched();
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_txq_init(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_txq *txq)
+{
+ struct dpmaif_txq_cfg *txq_cfg = &dcb->drv_info->cfg->tx_cfg.txqs[txq->id];
+ int ret;
+
+ txq->drb_cnt = txq_cfg->drb_cnt;
+ atomic_set(&txq->budget, txq->drb_cnt);
+
+ /* Allocate DRB memory for HW and SW. */
+ txq->drb_base = dma_alloc_coherent(DCB_TO_DEV(dcb), txq->drb_cnt * sizeof(*txq->drb_base),
+ &txq->drb_dma_addr, GFP_KERNEL);
+ if (!txq->drb_base)
+ return -ENOMEM;
+
+ /* Allocate buffer for SW to record the skb information. */
+ txq->sw_drb_base = devm_kcalloc(DCB_TO_DEV(dcb), txq->drb_cnt, sizeof(*txq->sw_drb_base),
+ GFP_KERNEL);
+ if (!txq->sw_drb_base) {
+ ret = -ENOMEM;
+ goto free_drb;
+ }
+
+ INIT_DELAYED_WORK(&txq->tx_done_work, mtk_dpmaif_tx_done);
+
+ return 0;
+
+free_drb:
+ dma_free_coherent(DCB_TO_DEV(dcb), txq->drb_cnt * sizeof(*txq->drb_base),
+ txq->drb_base, txq->drb_dma_addr);
+
+ return ret;
+}
+
+static void mtk_dpmaif_txq_exit(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_txq *txq)
+{
+ struct dpmaif_drb_skb *drb_skb;
+ int i;
+
+ dma_free_coherent(DCB_TO_DEV(dcb), txq->drb_cnt * sizeof(*txq->drb_base),
+ txq->drb_base, txq->drb_dma_addr);
+
+ for (i = 0; i < txq->drb_cnt; i++) {
+ drb_skb = txq->sw_drb_base + i;
+ if (!drb_skb->skb)
+ continue;
+
+ /* Verify msg drb or payload drb, and only payload drb need to unmap dma. */
+ if (drb_skb->data_dma_addr)
+ dma_unmap_single(DCB_TO_DEV(dcb),
+ drb_skb->data_dma_addr,
+ drb_skb->data_len, DMA_TO_DEVICE);
+ if (drb_skb->is_last)
+ dev_kfree_skb_any(drb_skb->skb);
+ }
+
+ devm_kfree(DCB_TO_DEV(dcb), txq->sw_drb_base);
+}
+
+static int mtk_dpmaif_sw_wait_txq_stop(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_txq *txq)
+{
+ /* Wait tx done work done. */
+ flush_delayed_work(&txq->tx_done_work);
+
+ return 0;
+}
+
+static void mtk_dpmaif_sw_wait_tx_stop(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char txq_cnt = dcb->drv_info->cfg->tx_cfg.txq_cnt;
+ int i;
+
+ /* Wait all tx handle complete */
+ for (i = 0; i < txq_cnt; i++)
+ mtk_dpmaif_sw_wait_txq_stop(dcb, &dcb->txqs[i]);
+}
+
+static void mtk_dpmaif_sw_reset_txq(struct dpmaif_txq *txq)
+{
+ struct dpmaif_drb_skb *drb_skb;
+ int i;
+
+ /* Drop all tx buffer around drb. */
+ for (i = 0; i < txq->drb_cnt; i++) {
+ drb_skb = txq->sw_drb_base + i;
+ if (!drb_skb->skb)
+ continue;
+
+ if (drb_skb->data_dma_addr)
+ dma_unmap_single(DCB_TO_DEV(txq->dcb), drb_skb->data_dma_addr,
+ drb_skb->data_len, DMA_TO_DEVICE);
+ if (drb_skb->is_last) {
+ dev_kfree_skb_any(drb_skb->skb);
+ drb_skb->skb = NULL;
+ }
+ }
+
+ /* Reset all txq resource. */
+ memset(txq->drb_base, 0x00, (txq->drb_cnt * sizeof(*txq->drb_base)));
+ memset(txq->sw_drb_base, 0x00, (txq->drb_cnt * sizeof(*txq->sw_drb_base)));
+
+ atomic_set(&txq->budget, txq->drb_cnt);
+ atomic_set(&txq->to_submit_cnt, 0);
+ txq->drb_rd_idx = 0;
+ txq->drb_wr_idx = 0;
+ txq->drb_rel_rd_idx = 0;
+}
+
+static void mtk_dpmaif_tx_res_reset(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char txq_cnt = dcb->drv_info->cfg->tx_cfg.txq_cnt;
+ struct dpmaif_txq *txq;
+ int i;
+
+ for (i = 0; i < txq_cnt; i++) {
+ txq = &dcb->txqs[i];
+ mtk_dpmaif_sw_reset_txq(txq);
+ }
+}
+
+static int mtk_dpmaif_tx_res_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_txq *txq;
+ int i, j;
+ int ret;
+
+ dcb->txqs = devm_kcalloc(DCB_TO_DEV(dcb), dcb->txq_cnt, sizeof(*txq), GFP_KERNEL);
+ if (!dcb->txqs)
+ return -ENOMEM;
+
+ for (i = 0; i < dcb->txq_cnt; i++) {
+ txq = &dcb->txqs[i];
+ txq->id = i;
+ txq->dcb = dcb;
+ ret = mtk_dpmaif_txq_init(dcb, txq);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to init txq%d resource\n", txq->id);
+ goto exit_txq;
+ }
+ }
+
+ dcb->tx_done_wq = alloc_workqueue("dpmaif_tx_done_wq_%s",
+ WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_HIGHPRI,
+ dcb->txq_cnt, DCB_TO_DEV_STR(dcb));
+ if (!dcb->tx_done_wq) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to allocate tx done workqueue\n");
+ ret = -ENOMEM;
+ goto exit_txq;
+ }
+
+ init_waitqueue_head(&dcb->db_task_ctlb.wait);
+ dcb->db_thread = kthread_run(mtk_dpmaif_doorbell_thread, dcb,
+ "dpmaif_db_%s", DCB_TO_DEV_STR(dcb));
+ if (IS_ERR(dcb->db_thread)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to create doorbell thread\n");
+ ret = PTR_ERR(dcb->db_thread);
+ dcb->db_thread = NULL;
+ goto destroy_tx_done_wq;
+ }
+
+ return 0;
+
+destroy_tx_done_wq:
+ flush_workqueue(dcb->tx_done_wq);
+ destroy_workqueue(dcb->tx_done_wq);
+
+exit_txq:
+ for (j = i - 1; j >= 0; j--)
+ mtk_dpmaif_txq_exit(dcb, &dcb->txqs[j]);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->txqs);
+
+ return ret;
+}
+
+static void mtk_dpmaif_tx_res_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_txq *txq;
+ int i;
+
+ if (dcb->db_thread) {
+ kthread_stop(dcb->db_thread);
+ dcb->db_thread = NULL;
+ }
+
+ for (i = 0; i < dcb->txq_cnt; i++) {
+ txq = &dcb->txqs[i];
+ flush_delayed_work(&txq->tx_done_work);
+ }
+
+ if (dcb->tx_done_wq) {
+ flush_workqueue(dcb->tx_done_wq);
+ destroy_workqueue(dcb->tx_done_wq);
+ dcb->tx_done_wq = NULL;
+ }
+
+ for (i = 0; i < dcb->txq_cnt; i++)
+ mtk_dpmaif_txq_exit(dcb, &dcb->txqs[i]);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->txqs);
+}
+
+static int mtk_dpmaif_sw_res_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ int ret;
+
+ ret = mtk_dpmaif_bat_res_init(dcb);
+ if (ret < 0)
+ return ret;
+
+ ret = mtk_dpmaif_rx_res_init(dcb);
+ if (ret < 0)
+ goto bat_res_exit;
+
+ ret = mtk_dpmaif_tx_res_init(dcb);
+ if (ret < 0)
+ goto rx_res_exit;
+
+ return 0;
+
+rx_res_exit:
+ mtk_dpmaif_rx_res_exit(dcb);
+
+bat_res_exit:
+ mtk_dpmaif_bat_res_exit(dcb);
+
+ return ret;
+}
+
+static void mtk_dpmaif_sw_res_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ mtk_dpmaif_tx_res_exit(dcb);
+ mtk_dpmaif_rx_res_exit(dcb);
+ mtk_dpmaif_bat_res_exit(dcb);
+}
+
+static bool mtk_dpmaif_all_vqs_empty_or_busy(struct dpmaif_tx_srv *tx_srv)
+{
+ bool is_empty_or_busy = true;
+ struct dpmaif_vq *vq;
+ int i;
+
+ for (i = 0; i < tx_srv->vq_cnt; i++) {
+ vq = tx_srv->vq[i];
+ if (!skb_queue_empty(&vq->list) && !test_bit(vq->q_id, &tx_srv->txq_drb_lack_sta)) {
+ is_empty_or_busy = false;
+ break;
+ }
+ }
+
+ return is_empty_or_busy;
+}
+
+static void mtk_dpmaif_record_drb_skb(struct mtk_dpmaif_ctlb *dcb, unsigned char q_id,
+ unsigned short cur_idx, struct sk_buff *skb,
+ unsigned short is_msg, unsigned short is_frag,
+ unsigned short is_last, dma_addr_t data_dma_addr,
+ unsigned int data_len)
+{
+ struct dpmaif_drb_skb *drb_skb = dcb->txqs[q_id].sw_drb_base + cur_idx;
+
+ drb_skb->skb = skb;
+ drb_skb->data_dma_addr = data_dma_addr;
+ drb_skb->data_len = data_len;
+ drb_skb->drb_idx = cur_idx;
+ drb_skb->is_msg = is_msg;
+ drb_skb->is_frag = is_frag;
+ drb_skb->is_last = is_last;
+}
+
+static int mtk_dpmaif_tx_fill_drb(struct mtk_dpmaif_ctlb *dcb,
+ unsigned char q_id, struct sk_buff *skb)
+{
+ unsigned short cur_idx, cur_backup_idx, is_frag, is_last;
+ unsigned int send_drb_cnt, wt_cnt, payload_cnt;
+ struct dpmaif_txq *txq = &dcb->txqs[q_id];
+ struct dpmaif_drb_skb *cur_drb_skb;
+ struct dpmaif_msg_drb *msg_drb;
+ struct dpmaif_tx_info tx_info;
+ struct dpmaif_pd_drb *pd_drb;
+ struct skb_shared_info *info;
+ dma_addr_t data_dma_addr;
+ unsigned int data_len;
+ skb_frag_t *frag;
+ void *data_addr;
+ int i, ret;
+
+ info = skb_shinfo(skb);
+ if (unlikely(info->frag_list))
+ dev_warn((DCB_TO_MDEV(dcb))->dev, "txq%d not support skb frag_list\n", q_id);
+
+ send_drb_cnt = DPMAIF_GET_DRB_CNT(skb);
+ payload_cnt = send_drb_cnt - 1;
+ cur_idx = txq->drb_wr_idx;
+ cur_backup_idx = cur_idx;
+
+ tx_info.msg_pkt_len = skb->len;
+ tx_info.msg_channel_id = DATA_SKB_CB(skb)->tx.intf_id;
+ tx_info.msg_txcsum = 1;
+
+ /* Update tx drb, a msg drb first, then payload drb. */
+ /* Update and record payload drb information. */
+ msg_drb = (struct dpmaif_msg_drb *)dcb->txqs[txq->id].drb_base + cur_idx;
+ dcb->drv_info->drv_ops->fill_tx_info(msg_drb, &tx_info, MSG_DRB);
+ mtk_dpmaif_record_drb_skb(dcb, txq->id, cur_idx, skb, 1, 0, 0, 0, 0);
+
+ /* Payload drb: skb->data + frags[]. */
+ cur_idx = mtk_dpmaif_ring_buf_get_next_idx(txq->drb_cnt, cur_idx);
+ for (wt_cnt = 0; wt_cnt < payload_cnt; wt_cnt++) {
+ /* Get data_addr and data_len. */
+ if (wt_cnt == 0) {
+ data_len = skb_headlen(skb);
+ data_addr = skb->data;
+ is_frag = 0;
+ } else {
+ frag = info->frags + wt_cnt - 1;
+ data_len = skb_frag_size(frag);
+ data_addr = skb_frag_address(frag);
+ is_frag = 1;
+ }
+
+ if (wt_cnt == payload_cnt - 1)
+ is_last = 1;
+ else
+ is_last = 0;
+
+ data_dma_addr = dma_map_single(DCB_TO_DEV(dcb),
+ data_addr, data_len, DMA_TO_DEVICE);
+ ret = dma_mapping_error(DCB_TO_DEV(dcb), data_dma_addr);
+ if (unlikely(ret)) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev, "dma mapping fail\n");
+ ret = -DATA_DMA_MAP_ERR;
+ goto unmap_dma;
+ }
+ tx_info.pd_is_last = is_last;
+ tx_info.pd_data_len = data_len;
+ tx_info.pd_data_dma_addr = data_dma_addr;
+
+ /* Update and record payload drb information. */
+ pd_drb = dcb->txqs[txq->id].drb_base + cur_idx;
+ dcb->drv_info->drv_ops->fill_tx_info(pd_drb, &tx_info, PD_DRB);
+ mtk_dpmaif_record_drb_skb(dcb, txq->id, cur_idx, skb, 0, is_frag, is_last,
+ data_dma_addr, data_len);
+
+ cur_idx = mtk_dpmaif_ring_buf_get_next_idx(txq->drb_cnt, cur_idx);
+ }
+
+ dma_wmb();
+ txq->drb_wr_idx = cur_idx;
+
+ return 0;
+
+unmap_dma:
+ mtk_dpmaif_record_drb_skb(dcb, txq->id, cur_backup_idx, NULL, 0, 0, 0, 0, 0);
+ cur_backup_idx = mtk_dpmaif_ring_buf_get_next_idx(txq->drb_cnt, cur_backup_idx);
+ for (i = 0; i < wt_cnt; i++) {
+ cur_drb_skb = txq->sw_drb_base + cur_backup_idx;
+
+ dma_unmap_single(DCB_TO_DEV(dcb),
+ cur_drb_skb->data_dma_addr, cur_drb_skb->data_len,
+ DMA_TO_DEVICE);
+
+ cur_backup_idx = mtk_dpmaif_ring_buf_get_next_idx(txq->drb_cnt, cur_backup_idx);
+ mtk_dpmaif_record_drb_skb(dcb, txq->id, cur_backup_idx, NULL, 0, 0, 0, 0, 0);
+ }
+
+ return ret;
+}
+
+static void mtk_dpmaif_tx_update_ring(struct mtk_dpmaif_ctlb *dcb, struct dpmaif_tx_srv *tx_srv,
+ struct dpmaif_vq *vq)
+{
+ struct dpmaif_txq *txq = &dcb->txqs[vq->q_id];
+ unsigned char q_id = vq->q_id;
+ unsigned char skb_drb_cnt;
+ int i, drb_available_cnt;
+ struct sk_buff *skb;
+
+ drb_available_cnt = mtk_dpmaif_ring_buf_writable(txq->drb_cnt,
+ txq->drb_rel_rd_idx, txq->drb_wr_idx);
+
+ for (i = 0; i < DPMAIF_SKB_TX_WEIGHT; i++) {
+ skb = skb_dequeue(&vq->list);
+ if (!skb)
+ break;
+
+ skb_drb_cnt = DPMAIF_GET_DRB_CNT(skb);
+ if (drb_available_cnt < skb_drb_cnt) {
+ skb_queue_head(&vq->list, skb);
+ set_bit(q_id, &tx_srv->txq_drb_lack_sta);
+ break;
+ }
+
+ if (mtk_dpmaif_tx_fill_drb(dcb, q_id, skb) < 0) {
+ skb_queue_head(&vq->list, skb);
+ break;
+ }
+
+ drb_available_cnt -= skb_drb_cnt;
+ atomic_sub(skb_drb_cnt, &txq->budget);
+ atomic_add(skb_drb_cnt, &txq->to_submit_cnt);
+ }
+}
+
+static struct dpmaif_vq *mtk_dpmaif_srv_select_vq(struct dpmaif_tx_srv *tx_srv)
+{
+ struct dpmaif_vq *vq;
+ int i;
+
+ /* Round robin select tx vqs. */
+ for (i = 0; i < tx_srv->vq_cnt; i++) {
+ tx_srv->cur_vq_id = tx_srv->cur_vq_id % tx_srv->vq_cnt;
+ vq = tx_srv->vq[tx_srv->cur_vq_id];
+ tx_srv->cur_vq_id++;
+ if (!skb_queue_empty(&vq->list) && !test_bit(vq->q_id, &tx_srv->txq_drb_lack_sta))
+ return vq;
+ }
+
+ return NULL;
+}
+
+static void mtk_dpmaif_tx(struct dpmaif_tx_srv *tx_srv)
+{
+ struct mtk_dpmaif_ctlb *dcb = tx_srv->dcb;
+ struct dpmaif_vq *vq;
+
+ while (!kthread_should_stop() && (dcb->dpmaif_state == DPMAIF_STATE_PWRON)) {
+ vq = mtk_dpmaif_srv_select_vq(tx_srv);
+ if (!vq)
+ break;
+
+ mtk_dpmaif_tx_update_ring(dcb, tx_srv, vq);
+
+ if (atomic_read(&dcb->txqs[vq->q_id].to_submit_cnt) > 0)
+ mtk_dpmaif_task_wakeup(&dcb->db_task_ctlb);
+
+ cond_resched();
+ }
+}
+
+static int mtk_dpmaif_tx_thread(void *arg)
+{
+ struct dpmaif_tx_srv *tx_srv = arg;
+ struct mtk_dpmaif_ctlb *dcb;
+ int ret;
+
+ dcb = tx_srv->dcb;
+ set_user_nice(current, tx_srv->nice);
+ while (!kthread_should_stop()) {
+ ret = wait_event_interruptible(tx_srv->wait,
+ (!mtk_dpmaif_all_vqs_empty_or_busy(tx_srv) &&
+ (dcb->dpmaif_state == DPMAIF_STATE_PWRON)) ||
+ kthread_should_stop());
+
+ if (ret == -ERESTARTSYS)
+ continue;
+
+ /* Send packets of all tx virtual queues belong to the tx service. */
+ mtk_dpmaif_tx(tx_srv);
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_tx_srvs_start(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char srvs_cnt = dcb->drv_info->cfg->tx_srvs_cfg.tx_srv_cnt;
+ struct dpmaif_tx_srv *tx_srv;
+ int i, j, ret;
+
+ for (i = 0; i < srvs_cnt; i++) {
+ tx_srv = &dcb->tx_srvs[i];
+ tx_srv->cur_vq_id = 0;
+ tx_srv->txq_drb_lack_sta = 0;
+ if (tx_srv->srv)
+ continue;
+
+ tx_srv->srv = kthread_run(mtk_dpmaif_tx_thread,
+ tx_srv, "dpmaif_tx_srv%u_%s",
+ tx_srv->id, DCB_TO_DEV_STR(dcb));
+ if (IS_ERR(tx_srv->srv)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to alloc dpmaif tx_srv%u\n", tx_srv->id);
+ ret = PTR_ERR(tx_srv->srv);
+ tx_srv->srv = NULL;
+ goto free_tx_srvs;
+ }
+ }
+
+ return 0;
+
+free_tx_srvs:
+ for (j = i - 1; j >= 0; j--) {
+ tx_srv = &dcb->tx_srvs[j];
+ kthread_stop(tx_srv->srv);
+ tx_srv->srv = NULL;
+ }
+
+ return ret;
+}
+
+static void mtk_dpmaif_tx_srvs_stop(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char srvs_cnt = dcb->drv_info->cfg->tx_srvs_cfg.tx_srv_cnt;
+ struct dpmaif_tx_srv *tx_srv;
+ int i;
+
+ for (i = 0; i < srvs_cnt; i++) {
+ tx_srv = &dcb->tx_srvs[i];
+ if (tx_srv->srv) {
+ kthread_stop(tx_srv->srv);
+ tx_srv->srv = NULL;
+ }
+ }
+}
+
+static int mtk_dpmaif_tx_srvs_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ const struct dpmaif_tx_srvs_cfg *tx_srvs_cfg = &dcb->drv_info->cfg->tx_srvs_cfg;
+ struct dpmaif_tx_srv *tx_srv;
+ struct dpmaif_vq *tx_vq;
+ int i, j, vq_id;
+ int ret;
+
+ /* Initialize all data packet tx vitrual queue. */
+ dcb->tx_vqs = devm_kcalloc(DCB_TO_DEV(dcb), tx_srvs_cfg->tx_vq_cnt, sizeof(*dcb->tx_vqs),
+ GFP_KERNEL);
+ if (!dcb->tx_vqs)
+ return -ENOMEM;
+
+ for (i = 0; i < tx_srvs_cfg->tx_vq_cnt; i++) {
+ tx_vq = &dcb->tx_vqs[i];
+ tx_vq->q_id = i;
+ tx_vq->max_len = DEFAULT_TX_QUEUE_LEN;
+ skb_queue_head_init(&tx_vq->list);
+ }
+
+ /* Initialize all data packet tx services. */
+ dcb->tx_srvs = devm_kcalloc(DCB_TO_DEV(dcb), tx_srvs_cfg->tx_srv_cnt, sizeof(*dcb->tx_srvs),
+ GFP_KERNEL);
+ if (!dcb->tx_srvs) {
+ ret = -ENOMEM;
+ goto free_tx_vqs;
+ }
+
+ for (i = 0; i < tx_srvs_cfg->tx_srv_cnt; i++) {
+ tx_srv = &dcb->tx_srvs[i];
+ tx_srv->dcb = dcb;
+ tx_srv->id = i;
+ tx_srv->nice = tx_srvs_cfg->tx_srvs[i].nice;
+ tx_srv->cur_vq_id = 0;
+ tx_srv->txq_drb_lack_sta = 0;
+ init_waitqueue_head(&tx_srv->wait);
+
+ /* Set virtual queues and tx service mapping. */
+ tx_srv->vq_cnt = tx_srvs_cfg->tx_srvs[i].vq_cnt;
+ for (j = 0; j < tx_srv->vq_cnt; j++) {
+ vq_id = tx_srvs_cfg->tx_srvs[i].vqs[j];
+ tx_srv->vq[j] = &dcb->tx_vqs[vq_id];
+ tx_srv->vq[j]->srv_id = tx_srv->id;
+ }
+ }
+
+ return 0;
+
+free_tx_vqs:
+ devm_kfree(DCB_TO_DEV(dcb), dcb->tx_vqs);
+
+ return ret;
+}
+
+static void mtk_dpmaif_tx_vqs_reset(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char vqs_cnt = dcb->drv_info->cfg->tx_srvs_cfg.tx_vq_cnt;
+ struct dpmaif_vq *tx_vq;
+ int i;
+
+ /* Drop all packet in tx virtual queues. */
+ for (i = 0; i < vqs_cnt; i++) {
+ tx_vq = &dcb->tx_vqs[i];
+ if (tx_vq)
+ skb_queue_purge(&tx_vq->list);
+ }
+}
+
+static void mtk_dpmaif_tx_srvs_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ mtk_dpmaif_tx_srvs_stop(dcb);
+ devm_kfree(DCB_TO_DEV(dcb), dcb->tx_srvs);
+ mtk_dpmaif_tx_vqs_reset(dcb);
+ devm_kfree(DCB_TO_DEV(dcb), dcb->tx_vqs);
+}
+
+static void mtk_dpmaif_trans_enable(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_drv_info *drv_info = dcb->drv_info;
+
+ mtk_dpmaif_sw_start_rx(dcb);
+ mtk_dpmaif_enable_irq(dcb);
+
+ if (drv_info->drv_ops->start_queue(drv_info, DPMAIF_RX) < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to start dpmaif hw rx\n");
+ mtk_dpmaif_common_err_handle(dcb, true);
+ return;
+ }
+
+ if (drv_info->drv_ops->start_queue(drv_info, DPMAIF_TX) < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to start dpmaif hw tx\n");
+ mtk_dpmaif_common_err_handle(dcb, true);
+ return;
+ }
+}
+
+static void mtk_dpmaif_trans_disable(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct dpmaif_drv_info *drv_info = dcb->drv_info;
+ bool io_err = false;
+
+ /* Stop dpmaif hw tx and rx. */
+ if (drv_info->drv_ops->stop_queue(drv_info, DPMAIF_TX) < 0) {
+ io_err = true;
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to stop dpmaif hw tx\n");
+ }
+
+ if (drv_info->drv_ops->stop_queue(drv_info, DPMAIF_RX) < 0) {
+ io_err = true;
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to stop dpmaif hw rx\n");
+ }
+
+ if (io_err)
+ mtk_dpmaif_common_err_handle(dcb, true);
+
+ /* Disable all dpmaif L1 interrupt. */
+ mtk_dpmaif_disable_irq(dcb);
+
+ /* Wait tx done work complete */
+ mtk_dpmaif_sw_wait_tx_stop(dcb);
+
+ /* Stop and wait rx handle done */
+ mtk_dpmaif_sw_stop_rx(dcb);
+}
+
+static void mtk_dpmaif_trans_ctl(struct mtk_dpmaif_ctlb *dcb, bool enable)
+{
+ if (enable) {
+ if (dcb->dpmaif_state == DPMAIF_STATE_PWRON)
+ mtk_dpmaif_trans_enable(dcb);
+ } else {
+ mtk_dpmaif_trans_disable(dcb);
+ }
+}
+
+static void mtk_dpmaif_cmd_trans_ctl(struct mtk_dpmaif_ctlb *dcb, void *data)
+{
+ struct mtk_data_trans_ctl *trans_ctl = data;
+
+ /* Try best to drop all tx vq packets when disable trans */
+ if (!trans_ctl->enable)
+ mtk_dpmaif_tx_vqs_reset(dcb);
+
+ mutex_lock(&dcb->trans_ctl_lock);
+ mtk_dpmaif_trans_ctl(dcb, trans_ctl->enable);
+ mutex_unlock(&dcb->trans_ctl_lock);
+}
+
+static struct dpmaif_drv_ops *mtk_dpmaif_get_drv_ops(u32 hw_ver)
+{
+ struct dpmaif_drv_ops_desc *p_drv_ops;
+ unsigned char i;
+
+ for (i = 0; (p_drv_ops = &dpmaif_drv_ops_tbl[i]) && p_drv_ops && p_drv_ops->drv_ops; i++)
+ if (p_drv_ops->hw_ver == hw_ver)
+ return p_drv_ops->drv_ops;
+
+ return NULL;
+}
+
+static int mtk_dpmaif_drv_res_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ dcb->drv_info = devm_kzalloc(DCB_TO_DEV(dcb), sizeof(*dcb->drv_info), GFP_KERNEL);
+ if (!dcb->drv_info)
+ return -ENOMEM;
+
+ dcb->drv_info->mdev = DCB_TO_MDEV(dcb);
+ dcb->drv_info->drv_ops = mtk_dpmaif_get_drv_ops(DPMAIF_GET_HW_VER(dcb));
+ if (!dcb->drv_info->drv_ops) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Unsupported mdev, hw_ver=0x%x\n", DPMAIF_GET_HW_VER(dcb));
+ devm_kfree(DCB_TO_DEV(dcb), dcb->drv_info);
+ return -EFAULT;
+ }
+
+ dcb->drv_info->drv_ops->init(dcb->drv_info, NULL);
+ dcb->rxq_cnt = dcb->drv_info->cfg->rx_cfg.rxq_cnt;
+ dcb->txq_cnt = dcb->drv_info->cfg->tx_cfg.txq_cnt;
+
+ return 0;
+}
+
+static void mtk_dpmaif_drv_res_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ devm_kfree(DCB_TO_DEV(dcb), dcb->drv_info);
+}
+
+static void mtk_dpmaif_irq_tx_done(struct mtk_dpmaif_ctlb *dcb, unsigned int q_mask)
+{
+ unsigned int ulq_done;
+ int drb_rd_idx;
+ int i;
+
+ for (i = 0; i < dcb->drv_info->cfg->tx_cfg.txq_cnt; i++) {
+ ulq_done = q_mask & BIT(i);
+ if (!ulq_done)
+ continue;
+
+ drb_rd_idx = dcb->drv_info->drv_ops->get_ring_idx(dcb->drv_info,
+ DPMAIF_DRB_RIDX, i);
+ if (unlikely(drb_rd_idx < 0)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to read txq%u drb_rd_idx, ret=%d\n",
+ i, drb_rd_idx);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ break;
+ }
+
+ dcb->txqs[i].drb_rd_idx = drb_rd_idx;
+ queue_delayed_work(dcb->tx_done_wq,
+ &dcb->txqs[i].tx_done_work,
+ msecs_to_jiffies(0));
+ }
+}
+
+static void mtk_dpmaif_irq_rx_done(struct mtk_dpmaif_ctlb *dcb, unsigned int q_id)
+{
+ struct dpmaif_rxq *rxq;
+ int pit_widx;
+
+ rxq = &dcb->rxqs[q_id];
+ __pm_stay_awake(rxq->ws);
+
+ pit_widx = dcb->drv_info->drv_ops->get_ring_idx(dcb->drv_info, DPMAIF_PIT_WIDX, q_id);
+ if (unlikely(pit_widx < 0)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to read rxq%u hw pit_wr_idx, ret=%d\n",
+ q_id, pit_widx);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ return;
+ }
+
+ rxq->pit_wr_idx = pit_widx;
+ napi_schedule(&rxq->napi);
+}
+
+static void mtk_dpmaif_irq_pit_len_err(struct mtk_dpmaif_ctlb *dcb, unsigned int q_id)
+{
+ dcb->rxqs[q_id].pit_cnt_err_intr_set = true;
+}
+
+static int mtk_dpmaif_irq_handle(int irq_id, void *data)
+{
+ struct dpmaif_drv_intr_info intr_info;
+ struct dpmaif_irq_param *irq_param;
+ struct dpmaif_bat_ring *bat_ring;
+ struct mtk_dpmaif_ctlb *dcb;
+ int bat_id;
+ int ret;
+ int i;
+
+ irq_param = data;
+ dcb = irq_param->dcb;
+
+ if (unlikely(dcb->dpmaif_state != DPMAIF_STATE_PWRON)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Invalid parameter, unexpected dpmaif irq\n");
+ goto out;
+ }
+
+ memset(&intr_info, 0x00, sizeof(struct dpmaif_drv_intr_info));
+ ret = mtk_dpmaif_drv_intr_handle(dcb->drv_info, &intr_info, irq_param->idx);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to get dpmaif drv irq info\n");
+ goto clean_drv_irq_info;
+ }
+
+ for (i = 0; i < intr_info.intr_cnt; i++) {
+ switch (intr_info.intr_types[i]) {
+ case DPMAIF_INTR_UL_DONE:
+ mtk_dpmaif_irq_tx_done(dcb, intr_info.intr_queues[i]);
+ break;
+ case DPMAIF_INTR_DL_BATCNT_LEN_ERR:
+ bat_id = intr_info.intr_queues[i];
+ bat_ring = &dcb->bat_infos[bat_id].normal_bat_ring;
+ bat_ring->bat_cnt_err_intr_set = true;
+ mtk_dpmaif_task_wakeup(&dcb->bat_infos[bat_id].task_ctlb);
+ break;
+ case DPMAIF_INTR_DL_PITCNT_LEN_ERR:
+ mtk_dpmaif_irq_pit_len_err(dcb, intr_info.intr_queues[i]);
+ break;
+ case DPMAIF_INTR_DL_DONE:
+ mtk_dpmaif_irq_rx_done(dcb, intr_info.intr_queues[i]);
+ break;
+ case DPMAIF_INTR_TRAS_SYNC:
+ break;
+ default:
+ break;
+ }
+ }
+
+clean_drv_irq_info:
+ mtk_pci_clear_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id);
+ mtk_pci_unmask_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id);
+out:
+ return IRQ_HANDLED;
+}
+
+static int mtk_dpmaif_irq_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char irq_cnt = dcb->drv_info->cfg->intr_cfg.irq_cnt;
+ struct dpmaif_irq_param *irq_param;
+ enum mtk_irq_src irq_src;
+ int i, j;
+ int ret;
+
+ dcb->irq_params = devm_kcalloc(DCB_TO_DEV(dcb), irq_cnt, sizeof(*irq_param), GFP_KERNEL);
+ if (!dcb->irq_params)
+ return -ENOMEM;
+
+ for (i = 0; i < irq_cnt; i++) {
+ irq_param = &dcb->irq_params[i];
+ irq_param->idx = i;
+ irq_param->dcb = dcb;
+ irq_src = dcb->drv_info->cfg->intr_cfg.irqs[i].id;
+ irq_param->dpmaif_irq_src = irq_src;
+ irq_param->dev_irq_id = mtk_pci_get_irq_id(DCB_TO_MDEV(dcb), irq_src);
+ if (irq_param->dev_irq_id < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to allocate irq id, irq_src=%d\n", irq_src);
+ ret = -EINVAL;
+ goto unregister_irq;
+ }
+
+ ret = mtk_pci_register_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id,
+ mtk_dpmaif_irq_handle, irq_param);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to register irq, irq_src=%d\n", irq_src);
+ goto unregister_irq;
+ }
+ irq_param->dev_virq_id = mtk_pci_get_virq_id(DCB_TO_MDEV(dcb),
+ irq_param->dev_irq_id);
+ }
+
+ /* HW layer default mask dpmaif interrupt. */
+ dcb->irq_enabled = false;
+
+ return 0;
+
+unregister_irq:
+ for (j = i - 1; j >= 0; j--) {
+ irq_param = &dcb->irq_params[j];
+ mtk_pci_unregister_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id);
+ }
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->irq_params);
+
+ return ret;
+}
+
+static int mtk_dpmaif_irq_exit(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned char irq_cnt = dcb->drv_info->cfg->intr_cfg.irq_cnt;
+ struct dpmaif_irq_param *irq_param;
+ int i;
+
+ for (i = 0; i < irq_cnt; i++) {
+ irq_param = &dcb->irq_params[i];
+ mtk_pci_unregister_irq(DCB_TO_MDEV(dcb), irq_param->dev_irq_id);
+ }
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb->irq_params);
+
+ return 0;
+}
+
+static int mtk_dpmaif_port_cfg(struct mtk_dpmaif_ctlb *dcb)
+{
+ struct mtk_data_trans_info *trans_info = &dcb->data_blk->trans_info;
+ struct dpmaif_drv_cfg *cfg = dcb->drv_info->cfg;
+ struct dpmaif_rxq *rxq;
+ int i;
+
+ memset(trans_info, 0x00, sizeof(struct mtk_data_trans_info));
+
+ trans_info->txq_cnt = cfg->tx_cfg.txq_cnt;
+ trans_info->rxq_cnt = cfg->rx_cfg.rxq_cnt;
+ trans_info->max_mtu = cfg->rx_cfg.mtu;
+
+ for (i = 0; i < trans_info->rxq_cnt; i++) {
+ rxq = &dcb->rxqs[i];
+ dcb->napi[i] = &rxq->napi;
+ }
+ trans_info->napis = dcb->napi;
+
+ return 0;
+}
+
+static int mtk_dpmaif_hw_init(struct mtk_dpmaif_ctlb *dcb)
+{
+ unsigned int bat_ring_num = dcb->drv_info->cfg->rx_cfg.bat_ring_num;
+ struct dpmaif_drv_cfg *cfg = dcb->drv_info->cfg;
+ struct dpmaif_bat_ring *bat_ring;
+ unsigned int bat_reload_cnt;
+ struct dpmaif_rxq *rxq;
+ struct dpmaif_txq *txq;
+ int ret, i;
+
+ for (i = 0; i < bat_ring_num; i++) {
+ bat_ring = &dcb->bat_infos[i].normal_bat_ring;
+ bat_reload_cnt = atomic_read(&bat_ring->to_reload_cnt);
+ mtk_dpmaif_reload_rx_buff(dcb, bat_ring, bat_reload_cnt);
+ if (!atomic_read(&bat_ring->reload_cnt)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to reload normal bat%d buf\n", i);
+ return -ENOMEM;
+ }
+
+ cfg->rx_cfg.bats[i].bat_base = bat_ring->bat_dma_addr;
+ cfg->rx_cfg.bats[i].buf_size = bat_ring->buf_size;
+ cfg->rx_cfg.bats[i].real_reload_cnt = atomic_read(&bat_ring->reload_cnt);
+ }
+
+ for (i = 0; i < cfg->rx_cfg.rxq_cnt; i++) {
+ rxq = &dcb->rxqs[i];
+ cfg->rx_cfg.rxqs[i].pit_base = rxq->pit_dma_addr;
+ }
+
+ for (i = 0; i < cfg->tx_cfg.txq_cnt; i++) {
+ txq = &dcb->txqs[i];
+ cfg->tx_cfg.txqs[i].drb_base = txq->drb_dma_addr;
+ }
+
+ ret = dcb->drv_info->drv_ops->init(dcb->drv_info, NULL);
+ if (ret < 0)
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to initialize dpmaif hw\n");
+
+ for (i = 0; i < bat_ring_num; i++) {
+ bat_ring = &dcb->bat_infos[i].normal_bat_ring;
+ atomic_sub(cfg->rx_cfg.bats[i].real_reload_cnt, &bat_ring->reload_cnt);
+ }
+ return ret;
+}
+
+static int mtk_dpmaif_start(struct mtk_md_dev *mdev)
+{
+ struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
+ int ret;
+
+ if (dcb->dpmaif_state == DPMAIF_STATE_PWRON) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev, "Invalid parameters, dpmaif_state in PWRON\n");
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* Initialize dpmaif hw. */
+ ret = mtk_dpmaif_hw_init(dcb);
+ if (ret < 0) {
+ dev_err((DCB_TO_MDEV(dcb))->dev, "Failed to initialize dpmaif hw\n");
+ goto out;
+ }
+
+ /* Initialize and run all tx services. */
+ ret = mtk_dpmaif_tx_srvs_start(dcb);
+ if (ret) {
+ dev_warn((DCB_TO_MDEV(dcb))->dev, "Failed to start all tx srvs\n");
+ goto out;
+ }
+
+ dcb->dpmaif_state = DPMAIF_STATE_PWRON;
+ dcb->dpmaif_sw_reset = false;
+ mtk_dpmaif_disable_irq(dcb);
+
+ return 0;
+out:
+ return ret;
+}
+
+static void mtk_dpmaif_sw_reset(struct mtk_dpmaif_ctlb *dcb)
+{
+ if (!dcb->dpmaif_sw_reset) {
+ dcb->dpmaif_sw_reset = true;
+ mtk_dpmaif_tx_res_reset(dcb);
+ mtk_dpmaif_rx_res_reset(dcb);
+ mtk_dpmaif_bat_res_reset(dcb);
+ mtk_dpmaif_tx_vqs_reset(dcb);
+ }
+}
+
+static int mtk_dpmaif_stop(struct mtk_md_dev *mdev)
+{
+ struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
+
+ if (dcb->dpmaif_state == DPMAIF_STATE_PWROFF)
+ goto out;
+
+ dcb->dpmaif_state = DPMAIF_STATE_PWROFF;
+
+ /* Stop all tx service. */
+ mtk_dpmaif_tx_srvs_stop(dcb);
+
+ mutex_lock(&dcb->trans_ctl_lock);
+
+ /* Stop dpmaif tx/rx handle. */
+ mtk_dpmaif_trans_ctl(dcb, false);
+
+ mutex_unlock(&dcb->trans_ctl_lock);
+
+out:
+ return 0;
+}
+
+static void mtk_dpmaif_clear(struct mtk_md_dev *mdev)
+{
+ struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
+
+ mtk_dpmaif_sw_reset(dcb);
+}
+
+static int mtk_dpmaif_sw_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_data_blk *data_blk = mdev->data_blk;
+ struct mtk_dpmaif_ctlb *dcb;
+ int ret;
+
+ dcb = devm_kzalloc(data_blk->mdev->dev, sizeof(*dcb), GFP_KERNEL);
+ if (!dcb)
+ return -ENOMEM;
+
+ data_blk->dcb = dcb;
+ dcb->data_blk = data_blk;
+ dcb->dpmaif_state = DPMAIF_STATE_PWROFF;
+ dcb->dpmaif_sw_reset = false;
+ mutex_init(&dcb->trans_ctl_lock);
+
+ ret = mtk_dpmaif_drv_res_init(dcb);
+ if (ret < 0)
+ goto free_dcb;
+
+ ret = mtk_dpmaif_sw_res_init(dcb);
+ if (ret < 0)
+ goto drv_res_exit;
+
+ ret = mtk_dpmaif_tx_srvs_init(dcb);
+ if (ret < 0)
+ goto sw_res_exit;
+
+ ret = mtk_dpmaif_port_cfg(dcb);
+ if (ret < 0)
+ goto tx_srvs_exit;
+
+ ret = mtk_dpmaif_irq_init(dcb);
+ if (ret < 0)
+ goto tx_srvs_exit;
+
+ return 0;
+
+tx_srvs_exit:
+ mtk_dpmaif_tx_srvs_exit(dcb);
+sw_res_exit:
+ mtk_dpmaif_sw_res_exit(dcb);
+drv_res_exit:
+ mtk_dpmaif_drv_res_exit(dcb);
+free_dcb:
+ devm_kfree(DCB_TO_DEV(dcb), dcb);
+
+ return ret;
+}
+
+static int mtk_dpmaif_sw_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_dpmaif_ctlb *dcb = ((struct mtk_data_blk *)(mdev->data_blk))->dcb;
+
+ if (!dcb) {
+ pr_err("Invalid parameter\n");
+ return -EINVAL;
+ }
+
+ mtk_dpmaif_irq_exit(dcb);
+ mtk_dpmaif_tx_srvs_exit(dcb);
+ mtk_dpmaif_sw_res_exit(dcb);
+ mtk_dpmaif_drv_res_exit(dcb);
+
+ devm_kfree(DCB_TO_DEV(dcb), dcb);
+ return 0;
+}
+
+static int mtk_dpmaif_poll_rx_pit(struct dpmaif_rxq *rxq)
+{
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ unsigned int sw_rd_idx, hw_wr_idx;
+ unsigned int pit_cnt;
+ int ret;
+
+ sw_rd_idx = rxq->pit_rd_idx;
+ ret = dcb->drv_info->drv_ops->get_ring_idx(dcb->drv_info, DPMAIF_PIT_WIDX, rxq->id);
+ if (unlikely(ret < 0)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Failed to read rxq%u hw pit_wr_idx, ret=%d\n",
+ rxq->id, ret);
+ mtk_dpmaif_common_err_handle(dcb, true);
+ goto out;
+ }
+
+ hw_wr_idx = ret;
+ pit_cnt = mtk_dpmaif_ring_buf_readable(rxq->pit_cnt, sw_rd_idx, hw_wr_idx);
+ rxq->pit_wr_idx = hw_wr_idx;
+
+ return pit_cnt;
+
+out:
+ return ret;
+}
+
+static int mtk_dpmaif_pit_bid_check(struct dpmaif_rxq *rxq, unsigned int cur_bid)
+{
+ union dpmaif_bat_record *cur_bat_record;
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ struct dpmaif_bat_ring *bat_ring;
+
+ bat_ring = &rxq->dcb->bat_infos[rxq->bat_ring_id].normal_bat_ring;
+ cur_bat_record = bat_ring->sw_record_base + cur_bid;
+
+ if (unlikely(!cur_bat_record->normal.skb || cur_bid >= bat_ring->bat_cnt)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Invalid parameter rxq%u bat%u, bid=%u, bat_cnt=%u\n",
+ rxq->id, bat_ring->id, cur_bid, bat_ring->bat_cnt);
+
+ return -DATA_FLOW_CHK_ERR;
+ }
+
+ rxq->pit_bid = cur_bid;
+
+ return 0;
+}
+
+static int mtk_dpmaif_rx_set_data_to_skb(struct dpmaif_rxq *rxq, struct dpmaif_pd_pit *pit_info,
+ struct dpmaif_rx_record *rx_record)
+{
+ struct dpmaif_bat_ring *bat_ring = &rxq->dcb->bat_infos[rxq->bat_ring_id].normal_bat_ring;
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ unsigned long long data_dma_base_addr;
+ union dpmaif_bat_record *bat_record;
+ unsigned int data_len;
+ struct sk_buff *new_skb;
+ int data_offset;
+
+ bat_record = bat_ring->sw_record_base + rxq->rx_info->pit_pd_cur_bid;
+ new_skb = bat_record->normal.skb;
+ data_dma_base_addr = (unsigned long long)bat_record->normal.data_dma_addr;
+
+ dma_unmap_single(DCB_TO_DEV(dcb), bat_record->normal.data_dma_addr,
+ bat_record->normal.data_len, DMA_FROM_DEVICE);
+
+ /* Calculate data address and data length. */
+ data_offset = (int)(rxq->rx_info->pit_pd_dma_addr - data_dma_base_addr);
+ data_len = rxq->rx_info->pit_pd_data_len;
+
+ /* Check and rebuild skb. */
+ new_skb->len = 0;
+ if (unlikely((new_skb->tail + data_offset + data_len) > new_skb->end)) {
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Invalid packet(%u/%u):data_len=%u,offset=0x%llx-0x%llx,skb(%llx,%llx,%u,%u)\n",
+ rxq->pit_rd_idx, rxq->rx_info->pit_pd_cur_bid,
+ data_len, rxq->rx_info->pit_pd_dma_addr,
+ data_dma_base_addr, (u64)new_skb->head,
+ (u64)new_skb->data, (unsigned int)new_skb->tail,
+ (unsigned int)new_skb->end);
+
+ return -DATA_FLOW_CHK_ERR;
+ }
+
+ skb_put(new_skb, data_offset + data_len);
+ skb_pull(new_skb, data_offset);
+
+ __skb_queue_tail(&rx_record->rx_list, new_skb);
+ rx_record->cur_skb = new_skb;
+
+ bat_record->normal.skb = NULL;
+
+ return 0;
+}
+
+static void mtk_dpmaif_bat_ring_set_mask(struct mtk_dpmaif_ctlb *dcb,
+ unsigned int bat_idx, int bat_ring_id)
+{
+ struct dpmaif_bat_ring *bat_ring;
+
+ bat_ring = &dcb->bat_infos[bat_ring_id].normal_bat_ring;
+
+ set_bit(bat_idx, bat_ring->mask_tbl);
+
+ atomic_inc(&bat_ring->bat_stats);
+ atomic_inc(&bat_ring->to_reload_cnt);
+}
+
+static int mtk_dpmaif_get_rx_pkt(struct dpmaif_rxq *rxq, struct dpmaif_pd_pit *pit_info,
+ struct dpmaif_rx_record *rx_record)
+{
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ int ret;
+
+ /* Check the bid in pit information, don't exceed bat size. */
+ ret = mtk_dpmaif_pit_bid_check(rxq, rxq->rx_info->pit_pd_cur_bid);
+ if (unlikely(ret < 0))
+ goto out;
+
+ /* Receive data from bat and save to rx_record. */
+ ret = mtk_dpmaif_rx_set_data_to_skb(rxq, pit_info, rx_record);
+ if (unlikely(ret < 0))
+ goto out;
+
+ /* Make sure saving data from bat is completed. */
+ wmb();
+
+ /* Set bat mask that have been received. */
+ mtk_dpmaif_bat_ring_set_mask(dcb, rxq->rx_info->pit_pd_cur_bid, rxq->bat_ring_id);
+
+ return 0;
+
+out:
+ return ret;
+}
+
+static int mtk_dpmaif_update_rx_skb_info(struct sk_buff *skb,
+ struct dpmaif_rxq *rxq, struct dpmaif_rx_record *rx_record)
+{
+ union mtk_data_pkt_info *pkt_info = DATA_SKB_CB(skb);
+ unsigned int gso_type = 0;
+ unsigned short l4_proto;
+ int inner_offset;
+ __be16 frag_off;
+ u8 packet_type;
+ u8 nexthdr;
+
+ skb_reset_network_header(skb);
+ skb_reset_mac_header(skb);
+
+ skb_set_hash(skb, rxq->rx_info->pit_msg_hash, PKT_HASH_TYPE_L4);
+ skb_record_rx_queue(skb, rxq->id);
+
+ if (rxq->rx_info->pit_msg_checksum == CS_RESULT_PASS) {
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ if (rxq->rx_info->pit_msg_ip == DPMAIF_PIT_IPV4)
+ skb->protocol = htons(ETH_P_IP);
+ else
+ skb->protocol = htons(ETH_P_IPV6);
+
+ if (rxq->rx_info->pit_msg_pro == DPMAIF_PIT_TCP) {
+ gso_type = SKB_GSO_TCPV4;
+ rx_record->ip_protocol = IPPROTO_TCP;
+ } else if (rxq->rx_info->pit_msg_pro == DPMAIF_PIT_UDP) {
+ rx_record->ip_protocol = IPPROTO_UDP;
+ } else {
+ goto out;
+ }
+ } else {
+ packet_type = skb->data[0] & 0xF0;
+ skb->ip_summed = CHECKSUM_NONE;
+ if (packet_type == IPV4_VERSION) {
+ skb->protocol = htons(ETH_P_IP);
+ l4_proto = ((struct iphdr *)skb->data)->protocol;
+ rx_record->ip_protocol = l4_proto;
+ if (l4_proto == IPPROTO_TCP)
+ gso_type = SKB_GSO_TCPV4;
+ else
+ goto out;
+
+ } else if (packet_type == IPV6_VERSION) {
+ skb->protocol = htons(ETH_P_IPV6);
+ nexthdr = ((struct ipv6hdr *)skb->data)->nexthdr;
+ /* Now skip over extension headers. */
+ inner_offset = ipv6_skip_exthdr(skb, sizeof(struct ipv6hdr),
+ &nexthdr, &frag_off);
+ if (unlikely(inner_offset < 0))
+ goto out;
+
+ rx_record->ip_protocol = nexthdr;
+ if (nexthdr == IPPROTO_TCP)
+ gso_type = SKB_GSO_TCPV6;
+ else
+ goto out;
+
+ } else {
+ dev_err((DCB_TO_MDEV(rxq->dcb))->dev,
+ "Invalid packet type, value=%u\n", packet_type);
+ goto free_skb;
+ }
+ }
+
+out:
+ pkt_info->rx.ch_id = rxq->rx_info->pit_msg_chnl_id;
+ pkt_info->rx.q_id = rxq->id;
+
+ return 0;
+
+free_skb:
+ dev_kfree_skb_any(skb);
+
+ return -DATA_HW_UNK_PKT;
+}
+
+static int mtk_dpmaif_rx_skb(struct dpmaif_rxq *rxq, struct dpmaif_rx_record *rx_record)
+{
+ struct sk_buff *new_skb;
+ int ret;
+
+ if (unlikely(rxq->rx_info->pit_msg_dp || rxq->rx_info->pit_msg_err)) {
+ __skb_queue_purge(&rx_record->rx_list);
+ return 0;
+ }
+
+ do {
+ /* Logically, the rx_list should contain at least one skb. */
+ new_skb = __skb_dequeue(&rx_record->rx_list);
+ ret = mtk_dpmaif_update_rx_skb_info(new_skb, rxq, rx_record);
+ if (unlikely(ret < 0))
+ continue;
+
+ /* Send skb to data port. */
+ /* Data would be sent to network stack here */
+ } while (!skb_queue_empty(&rx_record->rx_list));
+
+ return ret;
+}
+
+static void mtk_dpmaif_recycle_pit_internal(struct dpmaif_rxq *rxq, unsigned short pit_rel_cnt)
+{
+ unsigned short old_sw_rel_idx, new_sw_rel_idx, old_hw_wr_idx;
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+
+ old_sw_rel_idx = rxq->pit_rel_rd_idx;
+ new_sw_rel_idx = old_sw_rel_idx + pit_rel_cnt;
+ old_hw_wr_idx = rxq->pit_wr_idx;
+
+ /* pit_rel_rd_idx should not exceed pit_wr_idx. */
+ if (old_hw_wr_idx < old_sw_rel_idx) {
+ if (new_sw_rel_idx >= rxq->pit_cnt)
+ new_sw_rel_idx = new_sw_rel_idx - rxq->pit_cnt;
+ }
+
+ atomic_add(pit_rel_cnt, &rxq->pit_rel_cnt);
+ rxq->pit_rel_rd_idx = new_sw_rel_idx;
+
+ if (atomic_read(&rxq->pit_rel_cnt) >= rxq->pit_burst_rel_cnt)
+ mtk_dpmaif_task_wakeup(&dcb->db_task_ctlb);
+}
+
+static int mtk_dpmaif_recycle_rx_ring(struct dpmaif_rxq *rxq)
+{
+ unsigned int pit_rel_cnt;
+
+ pit_rel_cnt = mtk_dpmaif_ring_buf_releasable(rxq->pit_cnt,
+ rxq->pit_rel_rd_idx,
+ rxq->pit_rd_idx);
+
+ if (unlikely(pit_rel_cnt > rxq->pit_cnt)) {
+ dev_err((DCB_TO_MDEV(rxq->dcb))->dev,
+ "Invalid rxq%u pit release count, %u>%u\n",
+ rxq->id, pit_rel_cnt, rxq->pit_cnt);
+ mtk_dpmaif_common_err_handle(rxq->dcb, false);
+ return -DATA_FLOW_CHK_ERR;
+ }
+
+ mtk_dpmaif_recycle_pit_internal(rxq, pit_rel_cnt);
+ mtk_dpmaif_task_wakeup(&rxq->dcb->bat_infos[rxq->bat_ring_id].task_ctlb);
+
+ return 0;
+}
+
+#define DPMAIF_PIT_SEQ_CHECK_FAIL_CNT 2500
+
+static int mtk_dpmaif_rx_data_collect_internal(struct dpmaif_rxq *rxq, int pit_cnt,
+ unsigned int *pkt_cnt)
+{
+ unsigned long time_limit = jiffies + msecs_to_jiffies(2);
+ struct dpmaif_rx_record *rx_record = &rxq->rx_record;
+ unsigned int recv_pkt_cnt = 0, pit_rd_cnt = 0;
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ struct dpmaif_pd_pit *pit_info;
+ struct dpmaif_rx_info rx_info;
+ unsigned int rx_cnt, cur_pit;
+ int ret = 0;
+
+ cur_pit = rxq->pit_rd_idx;
+ for (rx_cnt = 0; rx_cnt < pit_cnt; rx_cnt++) {
+ if (!rx_record->msg_pit_recv && time_after_eq(jiffies, time_limit)) {
+ ret = -DATA_DL_ONCE_MORE;
+ break;
+ }
+
+ /* Pit sequence check. */
+ pit_info = rxq->pit_base + cur_pit;
+ ret = dcb->drv_info->drv_ops->get_rx_info(pit_info, &rx_info,
+ rxq->pit_seq_expect, rxq->id);
+ if (likely(!ret)) {
+ rxq->pit_seq_expect++;
+ if (rxq->pit_seq_expect >= rxq->pit_seq_max)
+ rxq->pit_seq_expect = 0;
+
+ rxq->pit_seq_fail_cnt = 0;
+ } else {
+ rxq->pit_seq_fail_cnt++;
+ if (rxq->pit_seq_fail_cnt >= DPMAIF_PIT_SEQ_CHECK_FAIL_CNT) {
+ rxq->pit_seq_fail_cnt = 0;
+ return -DATA_FLOW_CHK_ERR;
+ }
+ break;
+ }
+ rxq->rx_info = &rx_info;
+
+ /* Parse message pit. */
+ if (rx_info.msg_pit) {
+ if (unlikely(rx_record->msg_pit_recv)) {
+ __skb_queue_purge(&rx_record->rx_list);
+ memset(rx_record, 0x00, sizeof(*rx_record));
+ __skb_queue_head_init(&rx_record->rx_list);
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Invalid pit, rxq%u two continuous message pit\n",
+ rxq->id);
+ return -DATA_FLOW_CHK_ERR;
+ }
+
+ rx_record->msg_pit_recv = true;
+
+ } else {
+ /* Parse normal pit. */
+ ret = mtk_dpmaif_get_rx_pkt(rxq, pit_info, rx_record);
+
+ if (unlikely(ret < 0)) {
+ __skb_queue_purge(&rx_record->rx_list);
+
+ dev_err((DCB_TO_MDEV(dcb))->dev,
+ "Invalid packet, error payload, rxq%u pit_idx=%d\n",
+ rxq->id, cur_pit);
+ return ret;
+ }
+
+ /* Last one pit of a packet. */
+ if (!rx_info.pit_continue) {
+ mtk_dpmaif_rx_skb(rxq, rx_record);
+ memset(rx_record, 0x00, sizeof(*rx_record));
+ __skb_queue_head_init(&rx_record->rx_list);
+ recv_pkt_cnt++;
+ }
+ }
+
+ cur_pit = mtk_dpmaif_ring_buf_get_next_idx(rxq->pit_cnt, cur_pit);
+ rxq->pit_rd_idx = cur_pit;
+
+ pit_rd_cnt++;
+ /* Recycle pit/bat in batches (rx packet budget). */
+ if (pit_rd_cnt == DPMAIF_PIT_CNT_UPDATE_THRESHOLD) {
+ atomic_add(DPMAIF_PIT_CNT_UPDATE_THRESHOLD, &rxq->pit_stats);
+ mtk_dpmaif_recycle_rx_ring(rxq);
+ pit_rd_cnt = 0;
+ }
+ }
+
+ if (pit_rd_cnt) {
+ atomic_add(pit_rd_cnt, &rxq->pit_stats);
+ mtk_dpmaif_recycle_rx_ring(rxq);
+ }
+
+ *pkt_cnt = recv_pkt_cnt;
+
+ return ret;
+}
+
+static int mtk_dpmaif_rx_data_collect(struct dpmaif_rxq *rxq, unsigned int *pkt_cnt)
+{
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ unsigned int pit_cnt;
+ int ret;
+
+ ret = mtk_dpmaif_poll_rx_pit(rxq);
+ if (unlikely(ret < 0))
+ return ret;
+
+ pit_cnt = ret;
+
+ /* Collect rx packets. */
+ if (likely(pit_cnt > 0)) {
+ ret = mtk_dpmaif_rx_data_collect_internal(rxq, pit_cnt, pkt_cnt);
+ if (ret <= -DATA_DL_ONCE_MORE) {
+ ret = -DATA_DL_ONCE_MORE;
+ } else if (ret <= -DATA_ERR_STOP_MAX) {
+ ret = -DATA_ERR_STOP_MAX;
+ mtk_dpmaif_common_err_handle(dcb, true);
+ } else {
+ ret = 0;
+ }
+ }
+
+ return ret;
+}
+
+static int mtk_dpmaif_rx_data_collect_more(struct dpmaif_rxq *rxq, int *work_done)
+{
+ unsigned long time_limit = jiffies + msecs_to_jiffies(2);
+ unsigned int total_pkt_cnt = 0, pkt_cnt;
+ int ret = 0;
+
+ do {
+ if (time_after_eq(jiffies, time_limit)) {
+ ret = -DATA_DL_ONCE_MORE;
+ break;
+ }
+
+ pkt_cnt = 0;
+ ret = mtk_dpmaif_rx_data_collect(rxq, &pkt_cnt);
+ total_pkt_cnt += pkt_cnt;
+ if (ret < 0)
+ break;
+ } while (pkt_cnt > 0 && rxq->started);
+
+ *work_done = total_pkt_cnt;
+
+ return ret;
+}
+
+static int mtk_dpmaif_rx_napi_poll(struct napi_struct *napi, int budget)
+{
+ struct dpmaif_rxq *rxq = container_of(napi, struct dpmaif_rxq, napi);
+ struct mtk_dpmaif_ctlb *dcb = rxq->dcb;
+ int work_done = 0;
+ int ret = 0;
+
+ if (likely(rxq->started)) {
+ ret = mtk_dpmaif_rx_data_collect_more(rxq, &work_done);
+
+ if (ret == -DATA_DL_ONCE_MORE) {
+ napi_gro_flush(napi, false);
+ work_done = budget;
+ } else {
+ if (unlikely(ret == -DATA_ERR_STOP_MAX))
+ rxq->started = false;
+ if (work_done > budget)
+ work_done = budget - 1;
+ }
+ }
+
+ if (work_done < budget) {
+ napi_complete_done(napi, work_done);
+ __pm_wakeup_event(rxq->ws, jiffies_to_msecs(HZ));
+ mtk_dpmaif_drv_intr_complete(dcb->drv_info, DPMAIF_INTR_DL_DONE, rxq->id, 0);
+ }
+
+ return work_done;
+}
+
+static int mtk_dpmaif_send_pkt(struct mtk_dpmaif_ctlb *dcb, struct sk_buff *skb)
+{
+ unsigned char vq_id = skb_get_queue_mapping(skb);
+ struct dpmaif_vq *vq;
+ int ret = 0;
+
+ vq = &dcb->tx_vqs[vq_id];
+ if (likely(skb_queue_len(&vq->list) < vq->max_len))
+ skb_queue_tail(&vq->list, skb);
+ else
+ ret = -EBUSY;
+
+ wake_up(&dcb->tx_srvs[vq->srv_id].wait);
+
+ return ret;
+}
+
+static int mtk_dpmaif_send_cmd(struct mtk_dpmaif_ctlb *dcb, struct sk_buff *skb)
+{
+ struct mtk_data_cmd *cmd_info = SKB_TO_CMD(skb);
+
+ if (dcb->dpmaif_state != DPMAIF_STATE_PWRON)
+ return -EINVAL;
+
+ switch (cmd_info->cmd) {
+ case DATA_CMD_TRANS_CTL:
+ mtk_dpmaif_cmd_trans_ctl(dcb, CMD_TO_DATA(cmd_info));
+ break;
+ default:
+ dev_warn((DCB_TO_MDEV(dcb))->dev, "Unknown cmd type=%d\n", cmd_info->cmd);
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int mtk_dpmaif_send(struct mtk_data_blk *data_blk, enum mtk_data_type type,
+ struct sk_buff *skb)
+{
+ struct mtk_dpmaif_ctlb *dcb;
+ int ret;
+
+ if (unlikely(!data_blk || !data_blk->dcb)) {
+ pr_warn("Invalid parameter\n");
+ return -EINVAL;
+ }
+
+ dcb = data_blk->dcb;
+
+ if (unlikely(dcb->dpmaif_state == DPMAIF_STATE_PWROFF))
+ return -EINVAL;
+
+ if (likely(type == DATA_PKT))
+ ret = mtk_dpmaif_send_pkt(dcb, skb);
+ else
+ ret = mtk_dpmaif_send_cmd(dcb, skb);
+
+ return ret;
+}
+
+static struct mtk_data_hif_ops pcie_data_ops = {
+ .init = mtk_dpmaif_sw_init,
+ .exit = mtk_dpmaif_sw_exit,
+ .stop = mtk_dpmaif_stop,
+ .start = mtk_dpmaif_start,
+ .clear = mtk_dpmaif_clear,
+ .poll = mtk_dpmaif_rx_napi_poll,
+ .send = mtk_dpmaif_send,
+};
+
+int mtk_pcie_data_init(struct mtk_md_dev *mdev)
+{
+ return mtk_data_init(mdev, &pcie_data_ops);
+}
+
+int mtk_pcie_data_exit(struct mtk_md_dev *mdev)
+{
+ return mtk_data_exit(mdev);
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h
new file mode 100644
index 000000000000..e7e2f333141c
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DPMAIF_H__
+#define __MTK_DPMAIF_H__
+
+#include "mtk_dev.h"
+
+int mtk_pcie_data_init(struct mtk_md_dev *mdev);
+int mtk_pcie_data_exit(struct mtk_md_dev *mdev);
+
+#endif /* __MTK_DPMAIF_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
index 3fd2b33d1199..077600389ab4 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c
@@ -8,6 +8,7 @@
#include <linux/delay.h>
#include "mtk_dev.h"
+#include "mtk_data_plane.h"
#include "mtk_dpmaif_drv.h"
#include "mtk_dpmaif_reg.h"
#include "mtk_pci.h"
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h
index 5a68d2431063..b2385478abee 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h
@@ -256,4 +256,13 @@ static inline int mtk_dpmaif_drv_send_doorbell(struct dpmaif_drv_info *drv_info,
return drv_info->drv_ops->send_doorbell(drv_info, type, q_id, cnt);
}
+struct dpmaif_drv_ops_desc {
+ u32 hw_ver;
+ struct dpmaif_drv_ops *drv_ops;
+};
+
+#define drv_ops_name(NAME) dpmaif_drv_ops_##NAME
+
+extern struct dpmaif_drv_ops dpmaif_drv_ops_m9xx;
+
#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c
index a8e183b5f4ee..701e50bfe8b1 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c
@@ -14,7 +14,7 @@
#define PIT_PD_DATA_LEN GENMASK(31, 16) /* Indicates the data length of current packet. */
#define PIT_PD_BUF_ID GENMASK(15, 3) /* The low order of buffer index */
-#define PIT_PD_BUF_TYPE BIT(2) /* 0b: normal BAT entry; 1b: fragment BAT entry */
+#define PIT_PD_BUF_TYPE BIT(2) /* 0b: normal BAT entry */
#define PIT_PD_CONT BIT(1) /* 0b: last entry; 1b: more entry */
#define PIT_PD_PKT_TYPE BIT(0) /* 0b: normal PIT entry; 1b: message PIT entry */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index aec1c5211c70..baac3692f1e3 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -14,6 +14,7 @@
#include <linux/module.h>
#include "mtk_dev.h"
+#include "mtk_dpmaif.h"
#include "mtk_trans_ctrl.h"
#include "mtk_pci.h"
#include "mtk_pci_reg.h"
@@ -776,7 +777,13 @@ static int mtk_pci_dev_init(struct mtk_md_dev *mdev)
if (ret)
goto free_fsm;
+ ret = mtk_pcie_data_init(mdev);
+ if (ret)
+ goto free_ctrl_plane;
+
return 0;
+free_ctrl_plane:
+ mtk_trans_ctrl_exit(mdev);
free_fsm:
mtk_fsm_exit(mdev);
return ret;
@@ -786,6 +793,7 @@ static void mtk_pci_dev_exit(struct mtk_md_dev *mdev)
{
mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_RM, 0, NULL, 0,
EVT_MODE_BLOCKING | EVT_MODE_TOHEAD);
+ mtk_pcie_data_exit(mdev);
mtk_trans_ctrl_exit(mdev);
mtk_fsm_exit(mdev);
}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.h b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
index bb657aa786b6..1e88f9c60a11 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
@@ -7,8 +7,7 @@
#define __MTK_PCI_H__
#include <linux/pci.h>
-
-#include "../mtk_dev.h"
+#include "mtk_dev.h"
enum mtk_irq_src {
MTK_IRQ_SRC_MIN,
--
2.34.1
^ permalink raw reply related
* [PATCH 06/11] net: wwan: t9xx: Add AT & MBIM WWAN ports
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Adds AT & MBIM ports to the port infrastructure.
The WWAN initialization method is responsible for creating the
corresponding ports using the WWAN framework infrastructure. The
implemented WWAN port operations are start, stop, tx, tx_blocking
and tx_poll.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/mtk_port.c | 27 ++
drivers/net/wwan/t9xx/mtk_port.h | 15 ++
drivers/net/wwan/t9xx/mtk_port_io.c | 332 ++++++++++++++++++++++++-
drivers/net/wwan/t9xx/mtk_port_io.h | 5 +
drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c | 8 +
5 files changed, 386 insertions(+), 1 deletion(-)
diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
index dbd279cf2a14..4032df99b5b0 100644
--- a/drivers/net/wwan/t9xx/mtk_port.c
+++ b/drivers/net/wwan/t9xx/mtk_port.c
@@ -630,6 +630,7 @@ static int mtk_port_rx_dispatch(struct sk_buff *skb, void *priv, bool force_recv
/* Support scatter gather transmission */
if (port->rx_mtu > port->rx_frag_size) {
ret = mtk_port_rx_dispatch_frag_skb(port, skb);
+ /* -EIO means partial data dispatch complete, does not goto drop flow */
if (ret < 0 && ret != -EIO)
goto drop_frag_skb;
} else {
@@ -818,6 +819,29 @@ int mtk_port_ch_disable(struct mtk_port *port)
return ret;
}
+static int mtk_port_enable_by_type(struct mtk_port_mngr *port_mngr, int tbl_type)
+{
+ struct mtk_port **ports;
+ int ret, idx;
+
+ if (tbl_type < 0 || tbl_type >= PORT_TBL_MAX)
+ return -EINVAL;
+
+ ports = kcalloc(port_mngr->port_cnt, sizeof(struct mtk_port *), GFP_KERNEL);
+ if (!ports)
+ return -ENOMEM;
+
+ ret = radix_tree_gang_lookup(&port_mngr->port_tbl[tbl_type],
+ (void **)ports, 0, port_mngr->port_cnt);
+ for (idx = 0; idx < ret; idx++) {
+ if (ports[idx]->enable)
+ ports_ops[ports[idx]->info.type]->enable(ports[idx]);
+ }
+
+ kfree(ports);
+ return 0;
+}
+
static void mtk_port_disable(struct mtk_port_mngr *port_mngr)
{
struct mtk_port **ports;
@@ -851,6 +875,9 @@ void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void *arg)
case FSM_STATE_OFF:
mtk_port_disable(port_mngr);
break;
+ case FSM_STATE_READY:
+ mtk_port_enable_by_type(port_mngr, PORT_TBL_MD);
+ break;
default:
break;
}
diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_port.h
index a201c0007878..cf561add6318 100644
--- a/drivers/net/wwan/t9xx/mtk_port.h
+++ b/drivers/net/wwan/t9xx/mtk_port.h
@@ -56,6 +56,10 @@ enum mtk_ccci_ch {
/* to MD */
CCCI_CONTROL_RX = 0x2000,
CCCI_CONTROL_TX = 0x2001,
+ CCCI_UART2_RX = 0x200A,
+ CCCI_UART2_TX = 0x200C,
+ CCCI_MBIM_RX = 0x20D0,
+ CCCI_MBIM_TX = 0x20D1,
};
enum mtk_port_flag {
@@ -73,6 +77,7 @@ enum mtk_port_tbl {
enum mtk_port_type {
PORT_TYPE_INTERNAL,
+ PORT_TYPE_WWAN,
PORT_TYPE_MAX
};
@@ -81,6 +86,13 @@ struct mtk_internal_port {
int (*recv_cb)(void *arg, struct sk_buff *skb);
};
+struct mtk_wwan_port {
+ /* w_lock protects wwan_port when recv data and disable port at the same time */
+ struct mutex w_lock;
+ int w_type;
+ void *w_port;
+};
+
struct mtk_port_cfg {
enum mtk_ccci_ch tx_ch;
enum mtk_ccci_ch rx_ch;
@@ -108,8 +120,11 @@ struct mtk_port {
wait_queue_head_t rx_wq;
struct list_head stale_entry;
char dev_str[MTK_DEV_STR_LEN];
+ /* Serializes port write operations */
+ struct mutex write_lock;
struct mtk_port_mngr *port_mngr;
struct mtk_internal_port i_priv;
+ struct mtk_wwan_port w_priv;
};
struct mtk_port_mngr {
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mtk_port_io.c
index 9e7a1207cc03..ab8b1c5157ec 100644
--- a/drivers/net/wwan/t9xx/mtk_port_io.c
+++ b/drivers/net/wwan/t9xx/mtk_port_io.c
@@ -3,6 +3,10 @@
* Copyright (c) 2022, MediaTek Inc.
*/
#include <linux/netdevice.h>
+#include <linux/poll.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+#include <linux/wwan.h>
#include "mtk_port_io.h"
@@ -39,6 +43,149 @@ static void mtk_port_struct_init(struct mtk_port *port)
port->rx_buf_size = MTK_RX_BUF_SIZE;
init_waitqueue_head(&port->trb_wq);
init_waitqueue_head(&port->rx_wq);
+ mutex_init(&port->write_lock);
+}
+
+static int mtk_port_copy_data_from(void *to, union user_buf from, unsigned int len,
+ unsigned int offset, bool from_user_space)
+{
+ int ret = 0;
+
+ if (from_user_space) {
+ ret = copy_from_user(to, from.ubuf + offset, len);
+ if (ret)
+ ret = -EFAULT;
+ } else {
+ memcpy(to, from.kbuf + offset, len);
+ }
+
+ return ret;
+}
+
+static int mtk_port_common_write_frag_skb(struct mtk_port *port, struct sk_buff *skb,
+ union user_buf buf, u32 packet_size,
+ u32 cur_pos, bool from_user_space)
+{
+ struct sk_buff *frag_skb, *tmp = NULL;
+ u32 frag_size;
+ int ret;
+
+ frag_size = min(packet_size, port->tx_frag_size);
+ ret = mtk_port_copy_data_from(skb_put(skb, frag_size),
+ buf, frag_size,
+ cur_pos, from_user_space);
+ if (ret) {
+ dev_err(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to copy skb for port(%s)\n", port->info.name);
+ goto err_reset_skb;
+ }
+ cur_pos += frag_size;
+ packet_size -= frag_size;
+ if (!packet_size)
+ return cur_pos;
+
+ while (packet_size > 0) {
+ frag_skb = __dev_alloc_skb(port->tx_mtu, GFP_KERNEL);
+ if (!frag_skb) {
+ ret = -ENOMEM;
+ goto err_free_frag_list;
+ }
+
+ frag_size = min(packet_size, port->tx_frag_size);
+ ret = mtk_port_copy_data_from(skb_put(frag_skb, frag_size),
+ buf, frag_size,
+ cur_pos, from_user_space);
+ if (ret) {
+ dev_err(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to copy frag_skb for port(%s)\n", port->info.name);
+ dev_kfree_skb_any(frag_skb);
+ goto err_free_frag_list;
+ }
+ skb->data_len += frag_size;
+ skb->len += frag_size;
+ cur_pos += frag_size;
+ packet_size -= frag_size;
+ if (!tmp)
+ skb_shinfo(skb)->frag_list = frag_skb;
+ else
+ tmp->next = frag_skb;
+ tmp = frag_skb;
+ }
+ return cur_pos;
+
+err_free_frag_list:
+ frag_skb = skb_shinfo(skb)->frag_list;
+ while (frag_skb) {
+ tmp = frag_skb->next;
+ frag_skb->next = NULL;
+ dev_kfree_skb_any(frag_skb);
+ frag_skb = tmp;
+ }
+ skb_shinfo(skb)->frag_list = NULL;
+err_reset_skb:
+ skb->data_len = 0;
+ return ret;
+}
+
+static int mtk_port_common_write(struct mtk_port *port, union user_buf buf, unsigned int len,
+ bool from_user_space)
+{
+ u32 packet_size, left_cnt = len, cur_pos;
+ struct sk_buff *skb;
+ int ret;
+
+ if (len == 0)
+ return 0;
+
+start_write:
+ ret = mtk_port_status_check(port);
+ if (ret)
+ goto end_write;
+
+ skb = __dev_alloc_skb(port->tx_mtu, GFP_KERNEL);
+ if (!skb) {
+ ret = -ENOMEM;
+ goto end_write;
+ }
+
+ skb_reserve(skb, sizeof(struct mtk_ccci_header));
+
+ packet_size = min(left_cnt, port->tx_mtu);
+ cur_pos = len - left_cnt;
+ /* Support scatter gather transmission */
+ if (port->tx_mtu > port->tx_frag_size) {
+ ret = mtk_port_common_write_frag_skb(port, skb, buf, packet_size,
+ cur_pos, from_user_space);
+ if (ret < 0)
+ goto err_free_skb;
+ } else {
+ ret = mtk_port_copy_data_from(skb_put(skb, packet_size),
+ buf, packet_size,
+ cur_pos, from_user_space);
+ if (ret) {
+ dev_err(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to copy data for port(%s)\n", port->info.name);
+ goto err_free_skb;
+ }
+ }
+
+ ret = mtk_port_send_data(port, skb);
+ if (ret < 0) {
+ if (ret == -EINTR)
+ left_cnt -= packet_size;
+ goto end_write;
+ }
+
+ left_cnt -= ret;
+ if (left_cnt)
+ goto start_write;
+ else
+ goto end_write;
+
+err_free_skb:
+ dev_kfree_skb_any(skb);
+end_write:
+ return (len > left_cnt) ? (len - left_cnt) : ret;
}
static int mtk_port_internal_init(struct mtk_port *port)
@@ -109,7 +256,6 @@ static int mtk_port_internal_recv(struct mtk_port *port, struct sk_buff *skb)
return ret;
drop_data:
- dev_kfree_skb_any(skb);
return ret;
}
@@ -241,6 +387,190 @@ static const struct port_ops port_internal_ops = {
.recv = mtk_port_internal_recv,
};
+static int mtk_port_wwan_open(struct wwan_port *w_port)
+{
+ struct mtk_port *port;
+ int ret;
+
+ port = wwan_port_get_drvdata(w_port);
+ ret = mtk_port_get_locked(port);
+ if (ret)
+ return ret;
+
+ ret = mtk_port_common_open(port);
+ if (ret)
+ mtk_port_put_locked(port);
+
+ return ret;
+}
+
+static void mtk_port_wwan_close(struct wwan_port *w_port)
+{
+ struct mtk_port *port = wwan_port_get_drvdata(w_port);
+
+ mtk_port_common_close(port);
+ mtk_port_put_locked(port);
+}
+
+static int mtk_port_wwan_write(struct wwan_port *w_port, struct sk_buff *skb)
+{
+ struct mtk_port *port = wwan_port_get_drvdata(w_port);
+ union user_buf user_buf;
+
+ if (unlikely(!skb->len)) {
+ kfree_skb(skb);
+ return 0;
+ }
+
+ port->info.flags &= ~PORT_F_BLOCKING;
+ user_buf.kbuf = (void *)skb->data;
+ return mtk_port_common_write(port, user_buf, skb->len, false);
+}
+
+static int mtk_port_wwan_write_blocking(struct wwan_port *w_port, struct sk_buff *skb)
+{
+ struct mtk_port *port = wwan_port_get_drvdata(w_port);
+ union user_buf user_buf;
+
+ if (unlikely(!skb->len)) {
+ kfree_skb(skb);
+ return 0;
+ }
+
+ port->info.flags |= PORT_F_BLOCKING;
+ user_buf.kbuf = (void *)skb->data;
+ return mtk_port_common_write(port, user_buf, skb->len, false);
+}
+
+static __poll_t mtk_port_wwan_poll(struct wwan_port *w_port, struct file *file,
+ struct poll_table_struct *poll)
+{
+ struct mtk_port *port = wwan_port_get_drvdata(w_port);
+ union ctrl_hif_cmd_data hif_cmd;
+ struct mtk_ctrl_blk *ctrl_blk;
+ __poll_t mask = 0;
+
+ poll_wait(file, &port->trb_wq, poll);
+ if (mtk_port_status_check(port))
+ return EPOLLERR | EPOLLHUP;
+
+ ctrl_blk = port->port_mngr->ctrl_blk;
+ hif_cmd.rx_ch = port->info.rx_ch;
+ if (!ctrl_blk->ops->send_cmd(ctrl_blk->mdev, HIF_CTRL_CMD_CHECK_TX_FULL, &hif_cmd))
+ mask |= EPOLLOUT | EPOLLWRNORM;
+
+ return mask;
+}
+
+static const struct wwan_port_ops wwan_ops = {
+ .start = mtk_port_wwan_open,
+ .stop = mtk_port_wwan_close,
+ .tx = mtk_port_wwan_write,
+ .tx_blocking = mtk_port_wwan_write_blocking,
+ .tx_poll = mtk_port_wwan_poll,
+};
+
+static int mtk_port_wwan_init(struct mtk_port *port)
+{
+ mtk_port_struct_init(port);
+ port->enable = false;
+
+ mutex_init(&port->w_priv.w_lock);
+
+ switch (port->info.rx_ch) {
+ case CCCI_MBIM_RX:
+ port->w_priv.w_type = WWAN_PORT_MBIM;
+ break;
+ case CCCI_UART2_RX:
+ port->w_priv.w_type = WWAN_PORT_AT;
+ break;
+ default:
+ port->w_priv.w_type = WWAN_PORT_UNKNOWN;
+ break;
+ }
+
+ return 0;
+}
+
+static int mtk_port_wwan_exit(struct mtk_port *port)
+{
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ ports_ops[port->info.type]->disable(port);
+
+ return 0;
+}
+
+static int mtk_port_wwan_enable(struct mtk_port *port)
+{
+ struct mtk_port_mngr *port_mngr;
+ int ret = 0;
+
+ port_mngr = port->port_mngr;
+
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ return 0;
+
+ ret = mtk_port_ch_enable(port);
+ if (ret && ret != -EBUSY)
+ return ret;
+
+ port->w_priv.w_port = wwan_create_port(port_mngr->ctrl_blk->mdev->dev,
+ port->w_priv.w_type,
+ &wwan_ops, NULL, port);
+ if (IS_ERR(port->w_priv.w_port)) {
+ dev_warn(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to create wwan port for (%s)\n", port->info.name);
+ return PTR_ERR(port->w_priv.w_port);
+ }
+
+ set_bit(PORT_S_WR, &port->status);
+ set_bit(PORT_S_ENABLE, &port->status);
+
+ return 0;
+}
+
+static int mtk_port_wwan_disable(struct mtk_port *port)
+{
+ struct wwan_port *w_port;
+
+ if (!test_and_clear_bit(PORT_S_ENABLE, &port->status))
+ return 0;
+
+ clear_bit(PORT_S_WR, &port->status);
+ w_port = port->w_priv.w_port;
+ mutex_lock(&port->w_priv.w_lock);
+ port->w_priv.w_port = NULL;
+ mutex_unlock(&port->w_priv.w_lock);
+
+ mtk_port_ch_disable(port);
+ wwan_remove_port(w_port);
+
+ return 0;
+}
+
+static int mtk_port_wwan_recv(struct mtk_port *port, struct sk_buff *skb)
+{
+ mutex_lock(&port->w_priv.w_lock);
+ if (!port->w_priv.w_port) {
+ mutex_unlock(&port->w_priv.w_lock);
+ return -ENXIO;
+ }
+
+ wwan_port_rx(port->w_priv.w_port, skb);
+ mutex_unlock(&port->w_priv.w_lock);
+ return 0;
+}
+
+static const struct port_ops port_wwan_ops = {
+ .init = mtk_port_wwan_init,
+ .exit = mtk_port_wwan_exit,
+ .reset = mtk_port_reset,
+ .enable = mtk_port_wwan_enable,
+ .disable = mtk_port_wwan_disable,
+ .recv = mtk_port_wwan_recv,
+};
+
const struct port_ops *ports_ops[PORT_TYPE_MAX] = {
&port_internal_ops,
+ &port_wwan_ops,
};
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.h b/drivers/net/wwan/t9xx/mtk_port_io.h
index 7d2cfe90334c..12f26d244f1f 100644
--- a/drivers/net/wwan/t9xx/mtk_port_io.h
+++ b/drivers/net/wwan/t9xx/mtk_port_io.h
@@ -23,6 +23,11 @@ struct port_ops {
int (*recv)(struct mtk_port *port, struct sk_buff *skb);
};
+union user_buf {
+ void __user *ubuf;
+ void *kbuf;
+};
+
void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag);
int mtk_port_internal_close(void *i_port);
int mtk_port_internal_write(void *i_port, struct sk_buff *skb);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
index 8611561dd67c..aab09cab360c 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
@@ -16,6 +16,10 @@ static const int mtk_srv_cfg_m9xx[NR_CLDMA][HW_QUE_NUM] = {
/* the number of RX GPDs should be at last two */
static const struct queue_info mtk_queue_info_m9xx[] = {
+ {CCCI_UART2_TX, CCCI_UART2_RX, CLDMA1, TXQ(5), RXQ(5),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K, 0},
+ {CCCI_MBIM_TX, CCCI_MBIM_RX, CLDMA1, TXQ(2), RXQ(2),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K, 0},
{CCCI_CONTROL_TX, CCCI_CONTROL_RX, CLDMA1, TXQ(0), RXQ(0),
Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K, 0},
{CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, CLDMA0, TXQ(0), RXQ(0),
@@ -23,6 +27,10 @@ static const struct queue_info mtk_queue_info_m9xx[] = {
};
static const struct mtk_port_cfg port_cfg_m9xx[] = {
+ {CCCI_UART2_TX, CCCI_UART2_RX, PORT_TYPE_WWAN, "AT",
+ PORT_F_ALLOW_DROP},
+ {CCCI_MBIM_TX, CCCI_MBIM_RX, PORT_TYPE_WWAN, "MBIM",
+ PORT_F_ALLOW_DROP},
{CCCI_CONTROL_TX, CCCI_CONTROL_RX, PORT_TYPE_INTERNAL, "MDCTRL",
PORT_F_ALLOW_DROP},
{CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, PORT_TYPE_INTERNAL, "SAPCTRL",
--
2.34.1
^ permalink raw reply related
* [PATCH 04/11] net: wwan: t9xx: Add control port
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
The control port consists of port I/O and port manager.
Port I/O provides a common operation as defined by "struct port_ops",
and the operation is managed by the "port manager". It provides
interfaces to internal users, the implemented internal interfaces are
open, close, write and recv_register.
The port manager defines and implements port management interfaces and
structures. It is responsible for port creation, destroying, and managing
port states. It sends data from port I/O to CLDMA via TRB ( Transaction
Request Block ), and dispatches received data from CLDMA to port I/O.
The using port will be held in the "stale list" when the driver destroys
it, and after creating it again, the user can continue to use it.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 4 +-
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 16 +-
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 20 +-
drivers/net/wwan/t9xx/mtk_dev.c | 13 +-
drivers/net/wwan/t9xx/mtk_port.c | 875 +++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_port.h | 159 +++++
drivers/net/wwan/t9xx/mtk_port_io.c | 246 +++++++
drivers/net/wwan/t9xx/mtk_port_io.h | 36 +
drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c | 24 +
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 2 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 24 +-
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 1 +
12 files changed, 1410 insertions(+), 10 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index ae9d6f2344ab..db3b1aa1928b 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -8,4 +8,6 @@ obj-$(CONFIG_MTK_T9XX_PCI) += pcie/
mtk_t9xx-y := \
mtk_dev.o \
- mtk_ctrl_plane.o
+ mtk_ctrl_plane.o \
+ mtk_port.o \
+ mtk_port_io.o
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
index ca32827c1a20..9264b59783d7 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -7,10 +7,12 @@
#include <linux/device.h>
#include "mtk_ctrl_plane.h"
+#include "mtk_port.h"
-int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops)
+int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops, struct mtk_ctrl_cfg *cfg)
{
struct mtk_ctrl_blk *ctrl_blk;
+ int err;
ctrl_blk = devm_kzalloc(mdev->dev, sizeof(*ctrl_blk), GFP_KERNEL);
if (!ctrl_blk)
@@ -19,8 +21,19 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops)
ctrl_blk->mdev = mdev;
mdev->ctrl_blk = ctrl_blk;
ctrl_blk->ops = ops;
+ ctrl_blk->cfg = cfg;
+
+ err = mtk_port_mngr_init(ctrl_blk, cfg->port_layer_cfg->port_cfg,
+ cfg->port_layer_cfg->port_cnt);
+ if (err)
+ goto err_free_mem;
return 0;
+
+err_free_mem:
+ devm_kfree(mdev->dev, ctrl_blk);
+
+ return err;
}
EXPORT_SYMBOL(mtk_ctrl_init);
@@ -28,6 +41,7 @@ int mtk_ctrl_exit(struct mtk_md_dev *mdev)
{
struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ mtk_port_mngr_exit(ctrl_blk);
devm_kfree(mdev->dev, ctrl_blk);
return 0;
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index 6d4be89680d6..91b623191221 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -11,6 +11,17 @@
#include "mtk_dev.h"
+#define Q_MTU_2K (0x800)
+#define Q_MTU_3_5K (0xE00)
+#define Q_MTU_7K (0x1C00)
+#define Q_MTU_32K (0x8000)
+#define Q_MTU_63K (0xFC00)
+#define Q_FRAG_2K (0x800)
+#define Q_FRAG_3_5K (0xE00)
+#define Q_FRAG_7K (0x1C00)
+#define Q_FRAG_32K (0x8000)
+#define Q_FRAG_63K (0xFC00)
+
enum mtk_trb_cmd_type {
TRB_CMD_MIN,
TRB_CMD_ENABLE,
@@ -54,17 +65,22 @@ struct mtk_ctrl_hif_ops {
int (*send_cmd)(struct mtk_md_dev *mdev, int cmd, void *data);
};
-struct mtk_ctrl_cfg;
+struct mtk_ctrl_cfg {
+ struct mtk_port_layer_cfg *port_layer_cfg;
+};
+
struct mtk_ctrl_trans;
struct mtk_ctrl_blk {
struct mtk_md_dev *mdev;
+ struct mtk_port_mngr *port_mngr;
struct mtk_ctrl_hif_ops *ops;
void *ctrl_hw_priv;
struct mtk_ctrl_cfg *cfg;
};
-int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops);
+int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops,
+ struct mtk_ctrl_cfg *cfg);
int mtk_ctrl_exit(struct mtk_md_dev *mdev);
#endif /* __MTK_CTRL_PLANE_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_dev.c b/drivers/net/wwan/t9xx/mtk_dev.c
index f254ca7ed877..8ba70d432e6f 100644
--- a/drivers/net/wwan/t9xx/mtk_dev.c
+++ b/drivers/net/wwan/t9xx/mtk_dev.c
@@ -6,6 +6,8 @@
#include <linux/module.h>
#include "mtk_dev.h"
+#include "mtk_port.h"
+#include "mtk_port_io.h"
struct mtk_md_dev *mtk_dev_alloc(struct device *pdev, const struct mtk_dev_ops *dev_ops)
{
@@ -31,12 +33,21 @@ EXPORT_SYMBOL(mtk_dev_free);
static int __init mtk_common_drv_init(void)
{
- return 0;
+ int ret;
+
+ ret = mtk_port_io_init();
+ if (ret)
+ goto err_init_devid;
+
+err_init_devid:
+ return ret;
}
module_init(mtk_common_drv_init);
static void __exit mtk_common_drv_exit(void)
{
+ mtk_port_io_exit();
+ mtk_port_stale_list_grp_cleanup();
}
module_exit(mtk_common_drv_exit);
diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
new file mode 100644
index 000000000000..e400e4d35f26
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port.c
@@ -0,0 +1,875 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/netdevice.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+
+#include "mtk_port.h"
+#include "mtk_port_io.h"
+
+#define MTK_DFLT_TRB_TIMEOUT (5 * HZ)
+#define MTK_DFLT_TRB_STATUS (0x1)
+#define MTK_TRB_HEADER_ADDED (0xADDED)
+#define MTK_CHECK_RX_SEQ_MASK (0x7fff)
+
+#define MTK_PORT_ENUM_VER (0)
+#define MTK_PORT_ENUM_HEAD_PATTERN (0x5a5a5a5a)
+#define MTK_PORT_ENUM_TAIL_PATTERN (0xa5a5a5a5)
+
+#define MTK_PORT_SEARCH_FROM_RADIX_TREE(p, s) ({\
+ struct mtk_port *_p; \
+ _p = rcu_dereference_raw(*(s)); \
+ if (!_p) \
+ continue; \
+ p = _p; \
+})
+
+#define MTK_PORT_INTERNAL_NODE_CHECK(p, s, i) ({\
+ if (radix_tree_is_internal_node(p)) { \
+ s = radix_tree_iter_retry(&(i));\
+ continue; \
+ } \
+})
+
+struct mtk_port_info {
+ __le16 channel;
+ __le16 reserved;
+} __packed;
+
+struct mtk_port_enum_msg {
+ __le32 head_pattern;
+ __le16 port_cnt;
+ __le16 version;
+ __le32 tail_pattern;
+ u8 data[];
+} __packed;
+
+/* global group for stale ports */
+static LIST_HEAD(stale_list_grp);
+/* mutex lock for stale_list_group */
+DEFINE_MUTEX(port_mngr_grp_mtx);
+
+static DEFINE_IDA(ccci_dev_ids);
+
+/* This function working always under mutex lock port_mngr_grp_mtx */
+void mtk_port_release(struct kref *port_kref)
+{
+ struct mtk_stale_list *s_list;
+ struct mtk_port *port;
+
+ port = container_of(port_kref, struct mtk_port, kref);
+ if (!test_bit(PORT_S_ON_STALE_LIST, &port->status))
+ goto port_exit;
+
+ list_del(&port->stale_entry);
+ list_for_each_entry(s_list, &stale_list_grp, entry) {
+ if (!strncmp(s_list->dev_str, port->dev_str, MTK_DEV_STR_LEN) &&
+ list_empty(&s_list->ports) && s_list->dev_id >= 0) {
+ ida_free(&ccci_dev_ids, s_list->dev_id);
+ s_list->dev_id = -1;
+ break;
+ }
+ }
+port_exit:
+ ports_ops[port->info.type]->exit(port);
+ kfree(port);
+}
+
+static int mtk_port_tbl_add(struct mtk_port_mngr *port_mngr, struct mtk_port *port)
+{
+ int ret;
+
+ ret = radix_tree_insert(&port_mngr->port_tbl[MTK_PORT_TBL_TYPE(port->info.rx_ch)],
+ port->info.rx_ch & 0xFFF, port);
+ if (ret)
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "port(%s) add to port_tbl failed, return %d\n",
+ port->info.name, ret);
+ else
+ port_mngr->port_cnt++;
+
+ return ret;
+}
+
+static void mtk_port_tbl_del(struct mtk_port_mngr *port_mngr, struct mtk_port *port)
+{
+ radix_tree_delete(&port_mngr->port_tbl[MTK_PORT_TBL_TYPE(port->info.rx_ch)],
+ port->info.rx_ch & 0xFFF);
+ port_mngr->port_cnt--;
+}
+
+static struct mtk_port *mtk_port_restore_from_stale_list(struct mtk_port_mngr *port_mngr,
+ struct mtk_stale_list *s_list)
+{
+ struct mtk_port *port, *next_port;
+ int ret;
+
+ mutex_lock(&port_mngr_grp_mtx);
+ list_for_each_entry_safe(port, next_port, &s_list->ports, stale_entry) {
+ kref_get(&port->kref);
+ list_del(&port->stale_entry);
+ ret = mtk_port_tbl_add(port_mngr, port);
+ if (ret) {
+ list_add_tail(&port->stale_entry, &s_list->ports);
+ kref_put(&port->kref, mtk_port_release);
+ mutex_unlock(&port_mngr_grp_mtx);
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "Failed when adding (%s) to port mngr\n",
+ port->info.name);
+ return ERR_PTR(ret);
+ }
+
+ port->port_mngr = port_mngr;
+ clear_bit(PORT_S_ON_STALE_LIST, &port->status);
+ ports_ops[port->info.type]->reset(port);
+ }
+ mutex_unlock(&port_mngr_grp_mtx);
+
+ return NULL;
+}
+
+static struct mtk_port *mtk_port_alloc_and_add(struct mtk_port_mngr *port_mngr,
+ struct mtk_port_cfg *dflt_info)
+{
+ struct mtk_port *port;
+ int ret;
+
+ port = kzalloc_obj(*port, GFP_KERNEL);
+ if (!port) {
+ ret = -ENOMEM;
+ goto err_alloc_port;
+ }
+ memcpy(&port->info, dflt_info, sizeof(*dflt_info));
+
+ ret = mtk_port_tbl_add(port_mngr, port);
+ if (ret < 0) {
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to add port(%s) to port tbl\n", dflt_info->name);
+ goto err_free_port;
+ }
+
+ port->port_mngr = port_mngr;
+ ret = ports_ops[port->info.type]->init(port);
+ if (ret < 0) {
+ mtk_port_tbl_del(port_mngr, port);
+ goto err_free_port;
+ }
+
+ memcpy(port->dev_str, port_mngr->ctrl_blk->mdev->dev_str, MTK_DEV_STR_LEN);
+ return port;
+
+err_free_port:
+ kfree(port);
+err_alloc_port:
+ return ERR_PTR(ret);
+}
+
+static void mtk_port_free_or_backup(struct mtk_port_mngr *port_mngr,
+ struct mtk_port *port, struct mtk_stale_list *s_list)
+{
+ mutex_lock(&port_mngr_grp_mtx);
+ mtk_port_tbl_del(port_mngr, port);
+ if (port->info.type != PORT_TYPE_INTERNAL) {
+ if (test_bit(PORT_S_OPEN, &port->status)) {
+ list_add_tail(&port->stale_entry, &s_list->ports);
+ set_bit(PORT_S_ON_STALE_LIST, &port->status);
+ memcpy(port->dev_str, port_mngr->ctrl_blk->mdev->dev_str,
+ MTK_DEV_STR_LEN);
+ port->port_mngr = NULL;
+ }
+ kref_put(&port->kref, mtk_port_release);
+ } else {
+ mtk_port_release(&port->kref);
+ }
+ mutex_unlock(&port_mngr_grp_mtx);
+}
+
+static struct mtk_port *mtk_port_search_by_id(struct mtk_port_mngr *port_mngr, int rx_ch)
+{
+ int tbl_type = MTK_PORT_TBL_TYPE(rx_ch);
+
+ if (tbl_type < PORT_TBL_SAP || tbl_type >= PORT_TBL_MAX)
+ return NULL;
+
+ return radix_tree_lookup(&port_mngr->port_tbl[tbl_type], MTK_CH_ID(rx_ch));
+}
+
+struct mtk_port *mtk_port_search_by_name(struct mtk_port_mngr *port_mngr, char *name)
+{
+ int tbl_type = PORT_TBL_SAP;
+ struct radix_tree_iter iter;
+ struct mtk_port *port;
+ void __rcu **slot;
+
+ do {
+ radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type], &iter, 0) {
+ MTK_PORT_SEARCH_FROM_RADIX_TREE(port, slot);
+ MTK_PORT_INTERNAL_NODE_CHECK(port, slot, iter);
+ if (!strncmp(port->info.name, name, MTK_DFLT_PORT_NAME_LEN))
+ return port;
+ }
+ tbl_type++;
+ } while (tbl_type < PORT_TBL_MAX);
+
+ return NULL;
+}
+
+static int mtk_port_tbl_create(struct mtk_port_mngr *port_mngr, struct mtk_port_cfg *cfg,
+ const int port_cnt, struct mtk_stale_list *s_list)
+{
+ struct mtk_port_cfg *dflt_port;
+ struct mtk_port *port;
+ int i;
+
+ INIT_RADIX_TREE(&port_mngr->port_tbl[PORT_TBL_SAP], GFP_KERNEL);
+ INIT_RADIX_TREE(&port_mngr->port_tbl[PORT_TBL_MD], GFP_KERNEL);
+
+ mtk_port_restore_from_stale_list(port_mngr, s_list);
+
+ /* copy ports from static port cfg table */
+ for (i = 0; i < port_cnt; i++) {
+ dflt_port = cfg + i;
+ if (!mtk_port_search_by_id(port_mngr, dflt_port->rx_ch)) {
+ port = mtk_port_alloc_and_add(port_mngr, dflt_port);
+ if (IS_ERR(port))
+ return PTR_ERR(port);
+ }
+ }
+
+ return 0;
+}
+
+static void mtk_port_tbl_destroy(struct mtk_port_mngr *port_mngr, struct mtk_stale_list *s_list)
+{
+ struct mtk_port **ports;
+ int tbl_type;
+ int ret, idx;
+
+ ports = kcalloc(port_mngr->port_cnt, sizeof(struct mtk_port *), GFP_KERNEL);
+ if (!ports)
+ return;
+
+ tbl_type = PORT_TBL_SAP;
+ do {
+ ret = radix_tree_gang_lookup(&port_mngr->port_tbl[tbl_type],
+ (void **)ports, 0, port_mngr->port_cnt);
+ for (idx = 0; idx < ret; idx++)
+ ports_ops[ports[idx]->info.type]->disable(ports[idx]);
+ for (idx = 0; idx < ret; idx++)
+ mtk_port_free_or_backup(port_mngr, ports[idx], s_list);
+ } while (++tbl_type < PORT_TBL_MAX);
+ kfree(ports);
+}
+
+static struct mtk_stale_list *mtk_port_stale_list_create(struct mtk_ctrl_blk *ctrl_blk)
+{
+ struct mtk_stale_list *s_list;
+
+ s_list = kzalloc_obj(*s_list, GFP_KERNEL);
+ if (!s_list)
+ return NULL;
+
+ memcpy(s_list->dev_str, ctrl_blk->mdev->dev_str, MTK_DEV_STR_LEN);
+ s_list->dev_id = -1;
+ INIT_LIST_HEAD(&s_list->ports);
+ rwlock_init(&s_list->port_mngr_lock);
+
+ mutex_lock(&port_mngr_grp_mtx);
+ list_add_tail(&s_list->entry, &stale_list_grp);
+ mutex_unlock(&port_mngr_grp_mtx);
+
+ return s_list;
+}
+
+static void mtk_port_stale_list_destroy(struct mtk_stale_list *s_list)
+{
+ mutex_lock(&port_mngr_grp_mtx);
+ list_del(&s_list->entry);
+ mutex_unlock(&port_mngr_grp_mtx);
+ kfree(s_list);
+}
+
+static struct mtk_stale_list *mtk_port_stale_list_search(const char *dev_str)
+{
+ struct mtk_stale_list *tmp, *s_list = NULL;
+
+ mutex_lock(&port_mngr_grp_mtx);
+ list_for_each_entry(tmp, &stale_list_grp, entry) {
+ if (!strncmp(tmp->dev_str, dev_str, MTK_DEV_STR_LEN)) {
+ s_list = tmp;
+ break;
+ }
+ }
+ mutex_unlock(&port_mngr_grp_mtx);
+
+ return s_list;
+}
+
+void mtk_port_stale_list_grp_cleanup(void)
+{
+ struct mtk_stale_list *s_list, *next_s_list;
+ struct mtk_port *port, *next_port;
+
+ mutex_lock(&port_mngr_grp_mtx);
+ list_for_each_entry_safe(s_list, next_s_list, &stale_list_grp, entry) {
+ list_del(&s_list->entry);
+
+ list_for_each_entry_safe(port, next_port, &s_list->ports, stale_entry) {
+ list_del(&port->stale_entry);
+ mtk_port_release(&port->kref);
+ }
+
+ kfree(s_list);
+ }
+ mutex_unlock(&port_mngr_grp_mtx);
+}
+
+static struct mtk_stale_list *mtk_port_stale_list_init(struct mtk_ctrl_blk *ctrl_blk, int *dev_id)
+{
+ struct mtk_stale_list *s_list;
+
+ s_list = mtk_port_stale_list_search(ctrl_blk->mdev->dev_str);
+ if (!s_list) {
+ s_list = mtk_port_stale_list_create(ctrl_blk);
+ if (unlikely(!s_list))
+ return NULL;
+ }
+
+ mutex_lock(&port_mngr_grp_mtx);
+ if (s_list->dev_id < 0) {
+ *dev_id = ida_alloc_range(&ccci_dev_ids, 0, MTK_DFLT_MAX_DEV_CNT - 1, GFP_KERNEL);
+ } else {
+ *dev_id = s_list->dev_id;
+ s_list->dev_id = -1;
+ }
+ mutex_unlock(&port_mngr_grp_mtx);
+
+ return s_list;
+}
+
+static void mtk_port_stale_list_exit(struct mtk_ctrl_blk *ctrl_blk,
+ struct mtk_stale_list *s_list, int dev_id)
+{
+ if (!s_list)
+ return;
+ mutex_lock(&port_mngr_grp_mtx);
+ if (list_empty(&s_list->ports)) {
+ ida_free(&ccci_dev_ids, dev_id);
+ mutex_unlock(&port_mngr_grp_mtx);
+ mtk_port_stale_list_destroy(s_list);
+ } else {
+ s_list->dev_id = dev_id;
+ mutex_unlock(&port_mngr_grp_mtx);
+ }
+}
+
+void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_trb_cmd_type cmd,
+ int (*trb_complete)(struct sk_buff *skb))
+{
+ kref_init(&trb->kref);
+ trb->channel_id = port->info.rx_ch;
+ trb->status = MTK_DFLT_TRB_STATUS;
+ trb->priv = port;
+ trb->cmd = cmd;
+ trb->trb_complete = trb_complete;
+}
+
+void mtk_port_trb_free(struct kref *trb_kref)
+{
+ struct trb *trb = container_of(trb_kref, struct trb, kref);
+ struct sk_buff *skb, *frag_skb, *next_skb;
+
+ skb = container_of((char *)trb, struct sk_buff, cb[0]);
+ /* Free frag_list for scatter gather TX */
+ if (trb->cmd == TRB_CMD_TX && skb_has_frag_list(skb)) {
+ frag_skb = skb_shinfo(skb)->frag_list;
+ while (frag_skb) {
+ next_skb = frag_skb->next;
+ frag_skb->next = NULL;
+ dev_kfree_skb_any(frag_skb);
+ frag_skb = next_skb;
+ }
+ skb_shinfo(skb)->frag_list = NULL;
+ skb->data_len = 0;
+ }
+ dev_kfree_skb_any(skb);
+}
+EXPORT_SYMBOL(mtk_port_trb_free);
+
+static int mtk_port_open_trb_complete(struct sk_buff *skb)
+{
+ struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
+ struct trb *trb = (struct trb *)skb->cb;
+ struct mtk_port *port = trb->priv;
+
+ if (!trb->status) {
+ port->tx_mtu = trb_open_priv->tx_mtu;
+ port->rx_mtu = trb_open_priv->rx_mtu;
+ port->tx_frag_size = trb_open_priv->tx_frag_size;
+ port->rx_frag_size = trb_open_priv->rx_frag_size;
+ port->tx_mtu -= MTK_CCCI_H_ELEN;
+ port->rx_mtu -= MTK_CCCI_H_ELEN;
+ }
+
+ wake_up_interruptible_all(&port->trb_wq);
+
+ kref_put(&trb->kref, mtk_port_trb_free);
+ return 0;
+}
+
+static int mtk_port_close_trb_complete(struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct mtk_port *port = trb->priv;
+
+ wake_up_interruptible_all(&port->trb_wq);
+ wake_up_interruptible_all(&port->rx_wq);
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return 0;
+}
+
+static int mtk_port_tx_complete(struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct mtk_port *port = trb->priv;
+
+ if (trb->status < 0)
+ dev_warn(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to send data: status:%d, port:%s\n",
+ trb->status, port->info.name);
+
+ wake_up_interruptible_all(&port->trb_wq);
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return 0;
+}
+
+int mtk_port_status_check(struct mtk_port *port)
+{
+ if (!test_bit(PORT_S_ENABLE, &port->status))
+ return -ENODEV;
+
+ if (!test_bit(PORT_S_OPEN, &port->status) || test_bit(PORT_S_FLUSH, &port->status) ||
+ !test_bit(PORT_S_WR, &port->status))
+ return -EBADF;
+
+ return 0;
+}
+
+int mtk_port_send_data(struct mtk_port *port, void *data)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct sk_buff *skb = data;
+ bool force_send;
+ struct trb *trb;
+ int ret, len;
+
+ port_mngr = port->port_mngr;
+
+ force_send = !!(port->info.flags & (PORT_F_BLOCKING | PORT_F_FORCE_SEND));
+ trb = (struct trb *)skb->cb;
+ mtk_port_trb_init(port, trb, TRB_CMD_TX, mtk_port_tx_complete);
+ len = skb->len;
+ kref_get(&trb->kref); /* kref count 1->2 */
+
+ /* add ccci header */
+ mtk_port_add_header(skb);
+ ret = mtk_port_status_check(port);
+ if (!ret)
+ ret = port_mngr->ctrl_blk->ops->submit_skb(port_mngr->ctrl_blk->mdev,
+ skb, force_send);
+
+ if (ret < 0) {
+ kref_put(&trb->kref, mtk_port_trb_free); /* kref count 2->1 */
+ kref_put(&trb->kref, mtk_port_trb_free); /* kref count 1->0 */
+ port->tx_seq--;
+ goto out;
+ }
+
+ if (!(port->info.flags & PORT_F_BLOCKING)) {
+ kref_put(&trb->kref, mtk_port_trb_free);
+ ret = len;
+ goto out;
+ }
+start_wait:
+
+ /* wait trb done, and no timeout in tx blocking mode */
+ ret = wait_event_interruptible_timeout(port->trb_wq,
+ trb->status <= 0 ||
+ test_bit(PORT_S_FLUSH, &port->status) ||
+ !test_bit(PORT_S_WR, &port->status),
+ MTK_DFLT_TRB_TIMEOUT);
+ if (!ret) {
+ goto start_wait;
+ } else if (ret == -ERESTARTSYS) {
+ ret = -EINTR;
+ } else if (ret > 0) {
+ if (test_bit(PORT_S_FLUSH, &port->status))
+ ret = len;
+ else
+ ret = (!trb->status) ? len : trb->status;
+ }
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+out:
+ return ret;
+}
+
+static int mtk_port_check_rx_seq(struct mtk_port *port, struct mtk_ccci_header *ccci_h)
+{
+ u16 seq_num, assert_bit, channel;
+ struct mtk_md_dev *mdev;
+
+ seq_num = FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status));
+ assert_bit = FIELD_GET(MTK_HDR_FLD_AST, le32_to_cpu(ccci_h->status));
+ if (assert_bit && port->rx_seq &&
+ ((seq_num - port->rx_seq) & MTK_CHECK_RX_SEQ_MASK) != 1) {
+ mdev = port->port_mngr->ctrl_blk->mdev;
+ channel = FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status));
+ dev_warn((mdev)->dev,
+ "<ch: %04x> seq num out-of-order %d->%d, len(%u)\n",
+ channel, seq_num, port->rx_seq,
+ le32_to_cpu(ccci_h->packet_len));
+
+ port->rx_seq = seq_num;
+ return -EPROTO;
+ }
+
+ return 0;
+}
+
+static int mtk_port_rx_dispatch_frag_skb(struct mtk_port *port, struct sk_buff *skb)
+{
+ struct sk_buff *frag_skb, *frag_next;
+ int ret;
+
+ frag_skb = skb_shinfo(skb)->frag_list;
+ skb->len -= skb->data_len;
+ skb->data_len = 0;
+ skb_shinfo(skb)->frag_list = NULL;
+
+ ret = ports_ops[port->info.type]->recv(port, skb);
+ if (ret < 0) {
+ skb_shinfo(skb)->frag_list = frag_skb;
+ return ret;
+ }
+
+ while (frag_skb) {
+ frag_next = frag_skb->next;
+ if (!frag_skb->len) {
+ frag_skb->next = NULL;
+ dev_kfree_skb_any(frag_skb);
+ frag_skb = frag_next;
+ continue;
+ }
+ frag_skb->next = NULL;
+ ret = ports_ops[port->info.type]->recv(port, frag_skb);
+ if (ret < 0) {
+ frag_skb->next = frag_next;
+ while (frag_skb) {
+ frag_next = frag_skb->next;
+ frag_skb->next = NULL;
+ dev_kfree_skb_any(frag_skb);
+ frag_skb = frag_next;
+ }
+ return -EIO;
+ }
+ frag_skb = frag_next;
+ }
+
+ return 0;
+}
+
+static int mtk_port_rx_dispatch(struct sk_buff *skb, void *priv, bool force_recv)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_ccci_header *ccci_h;
+ struct mtk_port *port = priv;
+ int ret = -EPROTO;
+ u16 channel;
+
+ if (!skb || !priv) {
+ pr_err("Invalid input value in rx dispatch\n");
+ return -EINVAL;
+ }
+
+ port_mngr = port->port_mngr;
+
+ ccci_h = mtk_port_strip_header(skb);
+ if (unlikely(!ccci_h)) {
+ dev_warn(port_mngr->ctrl_blk->mdev->dev,
+ "Unsupported: skb length(%d) is less than ccci header\n",
+ skb->len);
+ goto drop_data;
+ }
+
+ channel = FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status));
+ port = mtk_port_search_by_id(port_mngr, channel);
+ if (unlikely(!port)) {
+ dev_warn(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to find port by channel:%d\n", channel);
+ goto drop_data;
+ }
+
+ ret = mtk_port_check_rx_seq(port, ccci_h);
+ if (unlikely(ret))
+ goto drop_data;
+
+ port->rx_seq = FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status));
+ skb_pull(skb, sizeof(*ccci_h));
+
+ /* Support scatter gather transmission */
+ if (port->rx_mtu > port->rx_frag_size) {
+ ret = mtk_port_rx_dispatch_frag_skb(port, skb);
+ if (ret < 0 && ret != -EIO)
+ goto drop_frag_skb;
+ } else {
+ ret = ports_ops[port->info.type]->recv(port, skb);
+ if (ret < 0)
+ goto drop_data;
+ }
+
+ return ret;
+
+drop_frag_skb:
+ {
+ struct sk_buff *frag_skb, *tmp;
+
+ frag_skb = skb_shinfo(skb)->frag_list;
+ while (frag_skb) {
+ tmp = frag_skb->next;
+ frag_skb->next = NULL;
+ dev_kfree_skb_any(frag_skb);
+ frag_skb = tmp;
+ }
+ skb_shinfo(skb)->frag_list = NULL;
+ }
+drop_data:
+ dev_kfree_skb_any(skb);
+ return ret;
+}
+
+int mtk_port_add_header(struct sk_buff *skb)
+{
+ struct mtk_ccci_header *ccci_h;
+ struct mtk_port *port;
+ struct trb *trb;
+
+ trb = (struct trb *)skb->cb;
+ if (trb->status == MTK_TRB_HEADER_ADDED)
+ return 0;
+
+ port = trb->priv;
+ if (!port)
+ return -EINVAL;
+
+ ccci_h = skb_push(skb, sizeof(*ccci_h));
+
+ ccci_h->packet_header = cpu_to_le32(0);
+ ccci_h->packet_len = cpu_to_le32(skb->len);
+ ccci_h->ex_msg = cpu_to_le32(0);
+ ccci_h->status = cpu_to_le32(FIELD_PREP(MTK_HDR_FLD_CHN, port->info.tx_ch) |
+ FIELD_PREP(MTK_HDR_FLD_SEQ, port->tx_seq++) |
+ FIELD_PREP(MTK_HDR_FLD_AST, 1));
+
+ trb->status = MTK_TRB_HEADER_ADDED;
+
+ return 0;
+}
+
+struct mtk_ccci_header *mtk_port_strip_header(struct sk_buff *skb)
+{
+ struct mtk_ccci_header *ccci_h;
+
+ if (skb->len < sizeof(*ccci_h)) {
+ pr_err("Invalid input value\n");
+ return NULL;
+ }
+
+ ccci_h = (struct mtk_ccci_header *)skb->data;
+
+ return ccci_h;
+}
+
+int mtk_port_status_update(struct mtk_md_dev *mdev, void *data)
+{
+ struct mtk_port_enum_msg *msg = data;
+ struct mtk_port_info *port_info;
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_port *port;
+ int port_id;
+ u16 ch_id;
+
+ if (unlikely(!mdev || !msg))
+ return -EINVAL;
+
+ ctrl_blk = mdev->ctrl_blk;
+ port_mngr = ctrl_blk->port_mngr;
+ if (le16_to_cpu(msg->version) != MTK_PORT_ENUM_VER ||
+ le32_to_cpu(msg->head_pattern) != MTK_PORT_ENUM_HEAD_PATTERN ||
+ le32_to_cpu(msg->tail_pattern) != MTK_PORT_ENUM_TAIL_PATTERN)
+ return -EPROTO;
+
+ for (port_id = 0; port_id < le16_to_cpu(msg->port_cnt); port_id++) {
+ port_info = (struct mtk_port_info *)(msg->data +
+ (sizeof(*port_info) * port_id));
+ ch_id = FIELD_GET(MTK_INFO_FLD_CHID, le16_to_cpu(port_info->channel));
+ port = mtk_port_search_by_id(port_mngr, ch_id);
+ if (!port)
+ continue;
+ port->enable = FIELD_GET(MTK_INFO_FLD_EN, le16_to_cpu(port_info->channel));
+ }
+
+ return 0;
+}
+
+int mtk_port_ch_enable(struct mtk_port *port)
+{
+ struct mtk_port_mngr *port_mngr = port->port_mngr;
+ struct trb_open_priv *trb_open_priv;
+ struct sk_buff *skb;
+ struct trb *trb;
+ int ret;
+
+ skb = __dev_alloc_skb(Q_MTU_3_5K, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ trb_open_priv = (struct trb_open_priv *)skb->data;
+ trb_open_priv->rx_done = mtk_port_rx_dispatch;
+
+ skb_put(skb, sizeof(struct trb_open_priv));
+ trb = (struct trb *)skb->cb;
+ mtk_port_trb_init(port, trb, TRB_CMD_ENABLE, mtk_port_open_trb_complete);
+ kref_get(&trb->kref);
+
+ ret = port_mngr->ctrl_blk->ops->submit_skb(port_mngr->ctrl_blk->mdev, skb, true);
+ if (ret) {
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to submit trb for port(%s), ret=%d\n",
+ port->info.name, ret);
+ kref_put(&trb->kref, mtk_port_trb_free);
+ mtk_port_trb_free(&trb->kref);
+ return ret;
+ }
+
+start_wait:
+ ret = wait_event_interruptible_timeout(port->trb_wq, trb->status <= 0,
+ MTK_DFLT_TRB_TIMEOUT);
+ if (ret == -ERESTARTSYS)
+ goto start_wait;
+ else if (!ret)
+ ret = -ETIMEDOUT;
+ else
+ ret = trb->status;
+
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return ret;
+}
+
+int mtk_port_ch_disable(struct mtk_port *port)
+{
+ struct mtk_port_mngr *port_mngr = port->port_mngr;
+ struct sk_buff *skb;
+ struct trb *trb;
+ int ret;
+
+ skb = __dev_alloc_skb(Q_MTU_3_5K, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ trb = (struct trb *)skb->cb;
+ mtk_port_trb_init(port, trb, TRB_CMD_DISABLE, mtk_port_close_trb_complete);
+ kref_get(&trb->kref);
+
+ ret = port_mngr->ctrl_blk->ops->submit_skb(port_mngr->ctrl_blk->mdev, skb, true);
+ if (ret) {
+ dev_warn(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to submit trb for port(%s), ret=%d\n",
+ port->info.name, ret);
+ kref_put(&trb->kref, mtk_port_trb_free);
+ mtk_port_trb_free(&trb->kref);
+ return ret;
+ }
+
+start_wait:
+ ret = wait_event_interruptible_timeout(port->trb_wq, trb->status <= 0,
+ MTK_DFLT_TRB_TIMEOUT);
+ if (ret == -ERESTARTSYS)
+ goto start_wait;
+ else if (!ret)
+ ret = -ETIMEDOUT;
+ else
+ ret = trb->status;
+
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return ret;
+}
+
+int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg *port_cfg, int port_cnt)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_stale_list *s_list;
+ int ret = -ENOMEM;
+ int dev_id;
+
+ s_list = mtk_port_stale_list_init(ctrl_blk, &dev_id);
+ if (!s_list) {
+ dev_err((ctrl_blk->mdev)->dev, "Failed to init mtk_stale_list\n");
+ goto err_out;
+ }
+
+ port_mngr = devm_kzalloc(ctrl_blk->mdev->dev, sizeof(*port_mngr), GFP_KERNEL);
+ if (unlikely(!port_mngr)) {
+ dev_err((ctrl_blk->mdev)->dev, "Failed to alloc memory for port_mngr\n");
+ goto err_exit_stale_list;
+ }
+
+ port_mngr->ctrl_blk = ctrl_blk;
+ port_mngr->dev_id = dev_id;
+
+ ret = mtk_port_tbl_create(port_mngr, port_cfg, port_cnt, s_list);
+ if (unlikely(ret)) {
+ dev_err((ctrl_blk->mdev)->dev, "Failed to create port_tbl\n");
+ goto err_free_port_mngr;
+ }
+
+ ctrl_blk->port_mngr = port_mngr;
+
+ return ret;
+
+err_free_port_mngr:
+ devm_kfree(ctrl_blk->mdev->dev, port_mngr);
+err_exit_stale_list:
+ mtk_port_stale_list_exit(ctrl_blk, s_list, dev_id);
+err_out:
+ return ret;
+}
+
+void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk)
+{
+ struct mtk_port_mngr *port_mngr = ctrl_blk->port_mngr;
+ struct mtk_stale_list *s_list;
+ int dev_id;
+
+ s_list = mtk_port_stale_list_search(port_mngr->ctrl_blk->mdev->dev_str);
+ dev_id = port_mngr->dev_id;
+
+ mtk_port_tbl_destroy(port_mngr, s_list);
+
+ devm_kfree(ctrl_blk->mdev->dev, port_mngr);
+ ctrl_blk->port_mngr = NULL;
+ mtk_port_stale_list_exit(ctrl_blk, s_list, dev_id);
+}
diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_port.h
new file mode 100644
index 000000000000..bd4291408bc2
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port.h
@@ -0,0 +1,159 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PORT_H__
+#define __MTK_PORT_H__
+
+#include <linux/bits.h>
+#include <linux/device.h>
+#include <linux/radix-tree.h>
+#include <linux/skbuff.h>
+#include <linux/types.h>
+
+#include "mtk_ctrl_plane.h"
+#include "mtk_dev.h"
+
+#define MTK_PEER_ID_MASK (0xF000)
+#define MTK_PEER_ID_SHIFT (12)
+#define MTK_PEER_ID(ch) (((ch) & MTK_PEER_ID_MASK) >> MTK_PEER_ID_SHIFT)
+#define MTK_PEER_ID_SAP (0x1)
+#define MTK_PEER_ID_MD (0x2)
+#define MTK_CH_ID_MASK (0x0FFF)
+#define MTK_CH_ID(ch) ((ch) & MTK_CH_ID_MASK)
+#define MTK_DFLT_MAX_DEV_CNT (10)
+#define MTK_DFLT_PORT_NAME_LEN (20)
+
+/* Mapping MTK_PEER_ID and mtk_port_tbl index */
+#define MTK_PORT_TBL_TYPE(ch) (MTK_PEER_ID(ch) - 1)
+
+/* ccci header length + reserved space that is used in exception flow */
+#define MTK_CCCI_H_ELEN (128)
+
+#define MTK_HDR_FLD_AST ((u32)BIT(31))
+#define MTK_HDR_FLD_SEQ GENMASK(30, 16)
+#define MTK_HDR_FLD_CHN GENMASK(15, 0)
+
+#define MTK_INFO_FLD_EN ((u16)BIT(15))
+#define MTK_INFO_FLD_CHID GENMASK(14, 0)
+
+enum mtk_port_status {
+ PORT_S_DFLT = 0,
+ PORT_S_ENABLE,
+ PORT_S_OPEN,
+ PORT_S_RD,
+ PORT_S_WR,
+ PORT_S_FLUSH,
+ PORT_S_ON_STALE_LIST,
+ PORT_S_STOP,
+};
+
+enum mtk_ccci_ch {
+ /* to sAP */
+ CCCI_SAP_CONTROL_RX = 0x1000,
+ CCCI_SAP_CONTROL_TX = 0x1001,
+ /* to MD */
+ CCCI_CONTROL_RX = 0x2000,
+ CCCI_CONTROL_TX = 0x2001,
+};
+
+enum mtk_port_flag {
+ PORT_F_DFLT = 0,
+ PORT_F_BLOCKING = BIT(1),
+ PORT_F_ALLOW_DROP = BIT(2),
+ PORT_F_FORCE_SEND = BIT(6),
+};
+
+enum mtk_port_tbl {
+ PORT_TBL_SAP,
+ PORT_TBL_MD,
+ PORT_TBL_MAX
+};
+
+enum mtk_port_type {
+ PORT_TYPE_INTERNAL,
+ PORT_TYPE_MAX
+};
+
+struct mtk_internal_port {
+ void *arg;
+ int (*recv_cb)(void *arg, struct sk_buff *skb);
+};
+
+struct mtk_port_cfg {
+ enum mtk_ccci_ch tx_ch;
+ enum mtk_ccci_ch rx_ch;
+ enum mtk_port_type type;
+ char name[MTK_DFLT_PORT_NAME_LEN];
+ unsigned char flags;
+};
+
+struct mtk_port {
+ struct mtk_port_cfg info;
+ struct kref kref;
+ bool enable;
+ unsigned long status;
+ unsigned int minor;
+ unsigned short tx_seq;
+ unsigned short rx_seq;
+ unsigned int tx_mtu;
+ unsigned int rx_mtu;
+ u32 tx_frag_size;
+ u32 rx_frag_size;
+ struct sk_buff_head rx_skb_list;
+ unsigned int rx_data_len;
+ unsigned int rx_buf_size;
+ wait_queue_head_t trb_wq;
+ wait_queue_head_t rx_wq;
+ struct list_head stale_entry;
+ char dev_str[MTK_DEV_STR_LEN];
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_internal_port i_priv;
+};
+
+struct mtk_port_mngr {
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct radix_tree_root port_tbl[PORT_TBL_MAX];
+ unsigned int port_cnt;
+ int dev_id;
+};
+
+struct mtk_stale_list {
+ struct list_head entry;
+ struct list_head ports;
+ char dev_str[MTK_DEV_STR_LEN];
+ int dev_id;
+ rwlock_t port_mngr_lock;
+};
+
+struct mtk_ccci_header {
+ __le32 packet_header;
+ __le32 packet_len;
+ __le32 status;
+ __le32 ex_msg;
+};
+
+struct mtk_port_layer_cfg {
+ struct mtk_port_cfg *port_cfg;
+ int port_cnt;
+};
+
+extern const struct port_ops *ports_ops[PORT_TYPE_MAX];
+
+void mtk_port_release(struct kref *port_kref);
+void mtk_port_trb_free(struct kref *trb_kref);
+struct mtk_port *mtk_port_search_by_name(struct mtk_port_mngr *port_mngr, char *name);
+void mtk_port_stale_list_grp_cleanup(void);
+int mtk_port_add_header(struct sk_buff *skb);
+struct mtk_ccci_header *mtk_port_strip_header(struct sk_buff *skb);
+int mtk_port_status_check(struct mtk_port *port);
+int mtk_port_send_data(struct mtk_port *port, void *data);
+int mtk_port_status_update(struct mtk_md_dev *mdev, void *data);
+int mtk_port_ch_enable(struct mtk_port *port);
+int mtk_port_ch_disable(struct mtk_port *port);
+int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg *port_cfg, int port_cnt);
+void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk);
+void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_trb_cmd_type cmd,
+ int (*trb_complete)(struct sk_buff *skb));
+#endif /* __MTK_PORT_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mtk_port_io.c
new file mode 100644
index 000000000000..9e7a1207cc03
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port_io.c
@@ -0,0 +1,246 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+#include <linux/netdevice.h>
+
+#include "mtk_port_io.h"
+
+static int mtk_port_get_locked(struct mtk_port *port)
+{
+ int ret = 0;
+
+ mutex_lock(&port_mngr_grp_mtx);
+ if (!port) {
+ mutex_unlock(&port_mngr_grp_mtx);
+ pr_err("Port does not exist\n");
+ return -ENODEV;
+ }
+ kref_get(&port->kref);
+ mutex_unlock(&port_mngr_grp_mtx);
+
+ return ret;
+}
+
+static void mtk_port_put_locked(struct mtk_port *port)
+{
+ mutex_lock(&port_mngr_grp_mtx);
+ kref_put(&port->kref, mtk_port_release);
+ mutex_unlock(&port_mngr_grp_mtx);
+}
+
+static void mtk_port_struct_init(struct mtk_port *port)
+{
+ port->tx_seq = 0;
+ port->rx_seq = -1;
+ clear_bit(PORT_S_ENABLE, &port->status);
+ kref_init(&port->kref);
+ skb_queue_head_init(&port->rx_skb_list);
+ port->rx_buf_size = MTK_RX_BUF_SIZE;
+ init_waitqueue_head(&port->trb_wq);
+ init_waitqueue_head(&port->rx_wq);
+}
+
+static int mtk_port_internal_init(struct mtk_port *port)
+{
+ mtk_port_struct_init(port);
+ port->enable = false;
+
+ return 0;
+}
+
+static int mtk_port_internal_exit(struct mtk_port *port)
+{
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ ports_ops[port->info.type]->disable(port);
+
+ return 0;
+}
+
+static int mtk_port_reset(struct mtk_port *port)
+{
+ port->tx_seq = 0;
+ port->rx_seq = -1;
+
+ return 0;
+}
+
+static int mtk_port_internal_enable(struct mtk_port *port)
+{
+ int ret;
+
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ return 0;
+
+ ret = mtk_port_ch_enable(port);
+ if (ret && ret != -EBUSY)
+ return ret;
+
+ set_bit(PORT_S_WR, &port->status);
+ set_bit(PORT_S_ENABLE, &port->status);
+
+ return 0;
+}
+
+static int mtk_port_internal_disable(struct mtk_port *port)
+{
+ if (!test_and_clear_bit(PORT_S_ENABLE, &port->status))
+ return 0;
+
+ clear_bit(PORT_S_WR, &port->status);
+ mtk_port_ch_disable(port);
+
+ return 0;
+}
+
+static int mtk_port_internal_recv(struct mtk_port *port, struct sk_buff *skb)
+{
+ struct mtk_internal_port *priv;
+ int ret = -ENXIO;
+
+ if (!test_bit(PORT_S_OPEN, &port->status))
+ goto drop_data;
+
+ priv = &port->i_priv;
+ if (!priv->recv_cb || !priv->arg)
+ goto drop_data;
+
+ ret = priv->recv_cb(priv->arg, skb);
+ return ret;
+
+drop_data:
+ dev_kfree_skb_any(skb);
+ return ret;
+}
+
+static int mtk_port_common_open(struct mtk_port *port)
+{
+ int ret = 0;
+
+ if (!test_bit(PORT_S_ENABLE, &port->status))
+ return -ENODEV;
+
+ if (test_bit(PORT_S_OPEN, &port->status))
+ return -EBUSY;
+
+ skb_queue_purge(&port->rx_skb_list);
+ set_bit(PORT_S_OPEN, &port->status);
+ clear_bit(PORT_S_FLUSH, &port->status);
+
+ return ret;
+}
+
+static void mtk_port_common_close(struct mtk_port *port)
+{
+ clear_bit(PORT_S_OPEN, &port->status);
+
+ skb_queue_purge(&port->rx_skb_list);
+ port->rx_data_len = 0;
+
+ set_bit(PORT_S_FLUSH, &port->status);
+ wake_up_interruptible_all(&port->trb_wq);
+ wake_up_interruptible_all(&port->rx_wq);
+}
+
+void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_port *port;
+ int ret;
+
+ ctrl_blk = mdev->ctrl_blk;
+ port_mngr = ctrl_blk->port_mngr;
+
+ port = mtk_port_search_by_name(port_mngr, name);
+ if (port && port->info.type != PORT_TYPE_INTERNAL) {
+ port = NULL;
+ goto out;
+ }
+
+ ret = mtk_port_get_locked(port);
+ if (ret)
+ goto out;
+
+ ret = mtk_port_common_open(port);
+ if (ret) {
+ mtk_port_put_locked(port);
+ goto out;
+ }
+
+ if (flag & O_NONBLOCK)
+ port->info.flags &= ~PORT_F_BLOCKING;
+ else
+ port->info.flags |= PORT_F_BLOCKING;
+out:
+ return port;
+}
+
+int mtk_port_internal_close(void *i_port)
+{
+ struct mtk_port *port = i_port;
+ int ret = 0;
+
+ if (!port) {
+ ret = -EINVAL;
+ goto end;
+ }
+
+ if (!test_bit(PORT_S_OPEN, &port->status)) {
+ pr_err("Port(%s) has been closed\n", port->info.name);
+ ret = -EBADF;
+ goto end;
+ }
+
+ mtk_port_common_close(port);
+ mtk_port_put_locked(port);
+end:
+ return ret;
+}
+
+int mtk_port_internal_write(void *i_port, struct sk_buff *skb)
+{
+ struct mtk_port *port = i_port;
+
+ if (!port || !skb) {
+ if (skb)
+ dev_kfree_skb_any(skb);
+ pr_err_ratelimited("Internal write: invalid input\n");
+ return -EINVAL;
+ }
+ return mtk_port_send_data(port, skb);
+}
+
+void mtk_port_internal_recv_register(void *i_port,
+ int (*cb)(void *priv, struct sk_buff *skb),
+ void *arg)
+{
+ struct mtk_port *port = i_port;
+ struct mtk_internal_port *priv;
+
+ priv = &port->i_priv;
+ priv->arg = arg;
+ priv->recv_cb = cb;
+}
+
+int mtk_port_io_init(void)
+{
+ return 0;
+}
+
+void mtk_port_io_exit(void)
+{
+}
+
+static const struct port_ops port_internal_ops = {
+ .init = mtk_port_internal_init,
+ .exit = mtk_port_internal_exit,
+ .reset = mtk_port_reset,
+ .enable = mtk_port_internal_enable,
+ .disable = mtk_port_internal_disable,
+ .recv = mtk_port_internal_recv,
+};
+
+const struct port_ops *ports_ops[PORT_TYPE_MAX] = {
+ &port_internal_ops,
+};
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.h b/drivers/net/wwan/t9xx/mtk_port_io.h
new file mode 100644
index 000000000000..7d2cfe90334c
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port_io.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PORT_IO_H__
+#define __MTK_PORT_IO_H__
+
+#include <linux/skbuff.h>
+
+#include "mtk_port.h"
+
+#define MTK_RX_BUF_SIZE (1024 * 1024)
+
+extern struct mutex port_mngr_grp_mtx;
+
+struct port_ops {
+ int (*init)(struct mtk_port *port);
+ int (*exit)(struct mtk_port *port);
+ int (*reset)(struct mtk_port *port);
+ int (*enable)(struct mtk_port *port);
+ int (*disable)(struct mtk_port *port);
+ int (*recv)(struct mtk_port *port, struct sk_buff *skb);
+};
+
+void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag);
+int mtk_port_internal_close(void *i_port);
+int mtk_port_internal_write(void *i_port, struct sk_buff *skb);
+void mtk_port_internal_recv_register(void *i_port,
+ int (*cb)(void *priv, struct sk_buff *skb),
+ void *arg);
+
+int mtk_port_io_init(void);
+void mtk_port_io_exit(void);
+
+#endif /* __MTK_PORT_IO_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
index bf3f87723167..8611561dd67c 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
@@ -4,6 +4,7 @@
*/
#include "mtk_cldma.h"
+#include "mtk_port.h"
#include "mtk_trans_ctrl.h"
#define TRB_SRV_NUM (1)
@@ -13,10 +14,33 @@ static const int mtk_srv_cfg_m9xx[NR_CLDMA][HW_QUE_NUM] = {
{0},
};
+/* the number of RX GPDs should be at last two */
static const struct queue_info mtk_queue_info_m9xx[] = {
+ {CCCI_CONTROL_TX, CCCI_CONTROL_RX, CLDMA1, TXQ(0), RXQ(0),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K, 0},
+ {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, CLDMA0, TXQ(0), RXQ(0),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K, 0},
+};
+
+static const struct mtk_port_cfg port_cfg_m9xx[] = {
+ {CCCI_CONTROL_TX, CCCI_CONTROL_RX, PORT_TYPE_INTERNAL, "MDCTRL",
+ PORT_F_ALLOW_DROP},
+ {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, PORT_TYPE_INTERNAL, "SAPCTRL",
+ PORT_F_ALLOW_DROP},
+};
+
+static struct mtk_port_layer_cfg port_layer_cfg_m9xx = {
+ .port_cfg = (struct mtk_port_cfg *)port_cfg_m9xx,
+ .port_cnt = ARRAY_SIZE(port_cfg_m9xx),
+};
+
+static struct mtk_ctrl_cfg mtk_ctrl_cfg_m9xx = {
+ .port_layer_cfg = &port_layer_cfg_m9xx,
};
struct mtk_ctrl_info mtk_ctrl_info_m9xx = {
+ .ctrl_cfg = &mtk_ctrl_cfg_m9xx,
+ .srv_cfg = (int **)mtk_srv_cfg_m9xx,
.queue_info = (struct queue_info *)mtk_queue_info_m9xx,
.queue_info_num = ARRAY_SIZE(mtk_queue_info_m9xx),
.trb_srv_num = TRB_SRV_NUM,
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index d604c9cb06ea..913705b341e3 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -17,6 +17,8 @@
#include "mtk_trans_ctrl.h"
#include "mtk_pci.h"
#include "mtk_pci_reg.h"
+#include "mtk_port.h"
+#include "mtk_port_io.h"
#define BAR_NUM 6
#define MTK_PCI_TRANSPARENT_ATR_SIZE (0x3F)
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
index 7fad64d214aa..c953729c5137 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -16,13 +16,14 @@
#include "mtk_ctrl_plane.h"
#include "mtk_dev.h"
#include "mtk_pci.h"
+#include "mtk_port.h"
#include "mtk_trans_ctrl.h"
#define MTK_DFLT_PORT_NAME_LEN (20)
extern struct mtk_ctrl_info ctrl_info_name(m9xx);
static struct mtk_ctrl_info_desc mtk_ctrl_info_tbl[] = {
- {2304, &ctrl_info_name(m9xx)},
+ {0x01CA, &ctrl_info_name(m9xx)},
{0, NULL},
};
@@ -134,6 +135,7 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
if (!skb)
break;
trb = (struct trb *)skb->cb;
+ kref_get(&trb->kref);
switch (trb->cmd) {
case TRB_CMD_ENABLE:
@@ -151,8 +153,10 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
if (err) {
if (trans_list->tx_burst_cnt[qno])
kick = true;
- else if (err == -EAGAIN)
+ else if (err == -EAGAIN) {
+ kref_put(&trb->kref, mtk_port_trb_free);
return;
+ }
break;
}
@@ -179,6 +183,8 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
kick = false;
}
+ kref_put(&trb->kref, mtk_port_trb_free);
+
loop++;
} while (loop < TRB_NUM_PER_ROUND);
}
@@ -508,6 +514,7 @@ static void mtk_trans_get_ctrl_info(struct mtk_ctrl_cfg *cfg,
continue;
ctrl_info = ctrl_info_desc->ctrl_info;
+ cfg->port_layer_cfg = ctrl_info->ctrl_cfg->port_layer_cfg;
memcpy(trans->srv_cfg, ctrl_info->srv_cfg,
sizeof(int) * NR_CLDMA * HW_QUE_NUM);
trans->queue_info = ctrl_info->queue_info;
@@ -520,6 +527,7 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
{
struct mtk_ctrl_blk *ctrl_blk;
struct mtk_ctrl_trans *trans;
+ struct mtk_ctrl_cfg *cfg;
int err;
trans = devm_kzalloc(mdev->dev, sizeof(*trans), GFP_KERNEL);
@@ -528,15 +536,19 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
trans->mdev = mdev;
trans->queues_cnt = 0;
- mtk_trans_get_ctrl_info(NULL, trans, mdev->hw_ver);
- if (!trans->queue_info ||
+ cfg = devm_kzalloc(mdev->dev, sizeof(*cfg), GFP_KERNEL);
+ if (!cfg)
+ goto err_free_trans;
+
+ mtk_trans_get_ctrl_info(cfg, trans, mdev->hw_ver);
+ if (!cfg->port_layer_cfg || !trans->queue_info ||
trans->trb_srv_num <= 0 || trans->trb_srv_num > TRB_SRV_MAX_NUM ||
trans->queue_info_num <= 0) {
dev_err((mdev)->dev, "Failed to get ctrl info!\n");
goto err_free_cfg;
}
- err = mtk_ctrl_init(mdev, &pcie_ctrl_ops);
+ err = mtk_ctrl_init(mdev, &pcie_ctrl_ops, cfg);
if (err)
goto err_free_cfg;
@@ -546,6 +558,8 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
return 0;
err_free_cfg:
+ devm_kfree(mdev->dev, cfg);
+err_free_trans:
devm_kfree(mdev->dev, trans);
return -ENOMEM;
}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
index c2df0bf6ed65..cca8e6f1532e 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -12,6 +12,7 @@
#include <linux/types.h>
#include "mtk_dev.h"
+#include "mtk_port.h"
#define TRB_SRV_MAX_NUM (1)
#define HW_QUE_NUM (8)
--
2.34.1
^ permalink raw reply related
* [PATCH 02/11] net: wwan: t9xx: Add control plane transaction layer
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
The control plane implements TX services that reside in the
transaction layer. The services receive the packets from the
port layer and call the corresponding DMA components to
transmit data to the device. Meanwhile, TX services receive
and manage the port control commands from the port layer.
The control plane implements RX services that reside in the
transaction layer. The services receive the downlink packets
from the modem and transfer the packets to the corresponding
port layer interfaces.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/Kconfig | 5 ++++
drivers/net/wwan/t9xx/Makefile | 5 ++--
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 34 ++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 22 +++++++++++++++
drivers/net/wwan/t9xx/mtk_dev.c | 44 +++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_dev.h | 4 +++
drivers/net/wwan/t9xx/pcie/Makefile | 10 +++++++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 8 ++----
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 21 ++++++++++++++
9 files changed, 146 insertions(+), 7 deletions(-)
diff --git a/drivers/net/wwan/Kconfig b/drivers/net/wwan/Kconfig
index 4cee537c739f..d8be12fb988c 100644
--- a/drivers/net/wwan/Kconfig
+++ b/drivers/net/wwan/Kconfig
@@ -124,6 +124,7 @@ config MTK_T7XX
config MTK_T9XX
tristate "MediaTek PCIe 5G WWAN modem T9xx device"
depends on PCI
+ select MTK_T9XX_PCI
select NET_DEVLINK
help
Enables MediaTek PCIe based 5G WWAN modem (T9xx series) device.
@@ -133,6 +134,10 @@ config MTK_T9XX
If unsure, say N.
+config MTK_T9XX_PCI
+ tristate
+ depends on PCI && MTK_T9XX
+
endif # WWAN
endmenu
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index 6f2dd3f91454..ae9d6f2344ab 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -4,7 +4,8 @@ ccflags-y += -I$(src)/pcie
ccflags-y += -I$(src)
obj-$(CONFIG_MTK_T9XX) += mtk_t9xx.o
+obj-$(CONFIG_MTK_T9XX_PCI) += pcie/
mtk_t9xx-y := \
- pcie/mtk_pci.o \
- pcie/mtk_pci_drv_m9xx.o
+ mtk_dev.o \
+ mtk_ctrl_plane.o
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
new file mode 100644
index 000000000000..ae5e1797b817
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ * Copyright (c) 2022-2023, Intel Corporation.
+ */
+
+#include <linux/device.h>
+
+#include "mtk_ctrl_plane.h"
+
+int mtk_ctrl_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk;
+
+ ctrl_blk = devm_kzalloc(mdev->dev, sizeof(*ctrl_blk), GFP_KERNEL);
+ if (!ctrl_blk)
+ return -ENOMEM;
+
+ ctrl_blk->mdev = mdev;
+ mdev->ctrl_blk = ctrl_blk;
+
+ return 0;
+}
+EXPORT_SYMBOL(mtk_ctrl_init);
+
+int mtk_ctrl_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+
+ devm_kfree(mdev->dev, ctrl_blk);
+
+ return 0;
+}
+EXPORT_SYMBOL(mtk_ctrl_exit);
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
new file mode 100644
index 000000000000..8276be19b456
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_CTRL_PLANE_H__
+#define __MTK_CTRL_PLANE_H__
+
+#include <linux/kref.h>
+#include <linux/skbuff.h>
+
+#include "mtk_dev.h"
+
+struct mtk_ctrl_blk {
+ struct mtk_md_dev *mdev;
+ struct mtk_ctrl_trans *trans;
+};
+
+int mtk_ctrl_init(struct mtk_md_dev *mdev);
+int mtk_ctrl_exit(struct mtk_md_dev *mdev);
+
+#endif /* __MTK_CTRL_PLANE_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_dev.c b/drivers/net/wwan/t9xx/mtk_dev.c
new file mode 100644
index 000000000000..f254ca7ed877
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_dev.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/module.h>
+
+#include "mtk_dev.h"
+
+struct mtk_md_dev *mtk_dev_alloc(struct device *pdev, const struct mtk_dev_ops *dev_ops)
+{
+ struct mtk_md_dev *mdev;
+
+ mdev = devm_kzalloc(pdev, sizeof(*mdev), GFP_KERNEL);
+ if (!mdev)
+ return NULL;
+
+ mdev->dev_ops = dev_ops;
+ mdev->dev = pdev;
+ return mdev;
+}
+EXPORT_SYMBOL(mtk_dev_alloc);
+
+void mtk_dev_free(struct mtk_md_dev *mdev)
+{
+ struct device *dev = mdev->dev;
+
+ devm_kfree(dev, mdev);
+}
+EXPORT_SYMBOL(mtk_dev_free);
+
+static int __init mtk_common_drv_init(void)
+{
+ return 0;
+}
+module_init(mtk_common_drv_init);
+
+static void __exit mtk_common_drv_exit(void)
+{
+}
+module_exit(mtk_common_drv_exit);
+
+MODULE_DESCRIPTION("MediaTek T9xx PCIe WWAN driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
index 8278a0e2875e..37eec1a358fa 100644
--- a/drivers/net/wwan/t9xx/mtk_dev.h
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -57,6 +57,7 @@ struct mtk_md_dev {
void *hw_priv;
u32 hw_ver;
char dev_str[MTK_DEV_STR_LEN];
+ void *ctrl_blk;
};
static inline u32 mtk_dev_get_dev_state(struct mtk_md_dev *mdev)
@@ -105,4 +106,7 @@ static inline int mtk_dev_send_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt)
return mdev->dev_ops->send_dev_evt(mdev, dev_evt);
}
+struct mtk_md_dev *mtk_dev_alloc(struct device *pdev, const struct mtk_dev_ops *dev_ops);
+void mtk_dev_free(struct mtk_md_dev *mdev);
+
#endif /* __MTK_DEV_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/Makefile b/drivers/net/wwan/t9xx/pcie/Makefile
new file mode 100644
index 000000000000..7410d1796d27
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/Makefile
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+ccflags-y += -I$(src)
+ccflags-y += -I$(src)/..
+
+obj-$(CONFIG_MTK_T9XX_PCI) += mtk_t9xx_pcie.o
+
+mtk_t9xx_pcie-y := \
+ mtk_pci_drv_m9xx.o \
+ mtk_pci.o
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index adec3ccdee08..518c32d55643 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -14,6 +14,7 @@
#include <linux/module.h>
#include "mtk_dev.h"
+#include "mtk_trans_ctrl.h"
#include "mtk_pci.h"
#include "mtk_pci_reg.h"
@@ -385,8 +386,7 @@ static u32 mtk_pci_ext_h2d_evt_hw_bits(u32 chs)
SET_HW_BITS(hw_bits, chs, MHCCIF_RC2EP_EVT_DEVICE_RESET,
DEV_EVT_H2D_DEVICE_RESET);
- SET_HW_BITS(hw_bits, chs, MHCCIF_RC2EP_EVT_DRM_DISABLE_AP,
- EXT_EVT_H2D_DRM_DISABLE_AP);
+
return LE32_TO_U32(cpu_to_le32(hw_bits));
}
@@ -779,13 +779,11 @@ static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
struct mtk_md_dev *mdev;
int ret;
- mdev = devm_kzalloc(dev, sizeof(*mdev), GFP_KERNEL);
+ mdev = mtk_dev_alloc(dev, &pci_hw_ops);
if (!mdev) {
ret = -ENOMEM;
goto out;
}
- mdev->dev_ops = &pci_hw_ops;
- mdev->dev = dev;
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
if (!priv) {
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
new file mode 100644
index 000000000000..d6de4c43b529
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_TRANS_CTRL_H__
+#define __MTK_TRANS_CTRL_H__
+
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/skbuff.h>
+#include <linux/types.h>
+
+#include "mtk_dev.h"
+
+struct mtk_ctrl_trans {
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_md_dev *mdev;
+};
+
+#endif
--
2.34.1
^ permalink raw reply related
* [PATCH 00/11] net: wwan: t9xx: Add MediaTek T9XX WWAN driver
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
T9XX is the PCIe host device driver for MediaTek's
t900 modem. The driver uses the WWAN framework
infrastructure to create the following control ports
and network interfaces for data transactions.
* /dev/wwan0at0 - Interface that supports AT commands.
* /dev/wwan0mbim0 - Interface conforming to the MBIM
protocol.
* wwan0-X - Primary network interface for IP traffic.
The main blocks in the T9XX driver are:
* HW layer - Abstracts the hardware bus operations for
the device, and provides generic interfaces for the
transaction layer to get the device's information and
control the device's behavior. It includes:
* PCIe - Implements probe, removal and interrupt
handling.
* MHCCIF (Modem Host Cross-Core Interface) - Provides
interrupt channels for bidirectional event
notification such as handshake and port enumeration.
* Transaction layer - Implements data transactions for
the control plane and the data plane. It includes:
* DPMAIF (Data Plane Modem AP Interface) - Controls
the hardware that provides uplink and downlink
queues for the data path. The data exchange takes
place using circular buffers to share data buffer
addresses and metadata to describe the packets.
* CLDMA (Cross Layer DMA) - Manages the hardware
used by the port layer to send control messages to
the device using MediaTek's CCCI (Cross-Core
Communication Interface) protocol.
* TX Services - Dispatch packets from the port layer
to the device.
* RX Services - Dispatch packets to the port layer
when receiving packets from the device.
* Port layer - Provides control plane and data plane
interfaces to userspace. It includes:
* Control Plane - Provides device node interfaces
for controlling data transactions.
* Data Plane - Provides network link interfaces
wwanX (0, 1, 2...) for IP data transactions.
* Core logic - Contains the core logic to keep the
device working. It includes:
* FSM (Finite State Machine) - Monitors the state
of the device, and notifies each module when the
state changes.
The compilation of the T9XX driver is enabled by the
CONFIG_MTK_T9XX and CONFIG_MTK_T9XX_PCI config option
which depends on CONFIG_WWAN.
---
Jack Wu (11):
net: wwan: t9xx: Add PCIe core
net: wwan: t9xx: Add control plane transaction layer
net: wwan: t9xx: Add control DMA interface
net: wwan: t9xx: Add control port
net: wwan: t9xx: Add FSM thread
net: wwan: t9xx: Add AT & MBIM WWAN ports
net: wwan: t9xx: Introduce data plane hardware
net: wwan: t9xx: Add data plane transaction layer
net: wwan: t9xx: Introduce WWAN interface
net: wwan: t9xx: Add power management support
net: wwan: t9xx: Add maintainers and documentation
.../networking/device_drivers/wwan/t9xx.rst | 48 +
MAINTAINERS | 10 +
drivers/net/wwan/Kconfig | 17 +
drivers/net/wwan/Makefile | 1 +
drivers/net/wwan/t9xx/Makefile | 16 +
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 95 +
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 88 +
drivers/net/wwan/t9xx/mtk_data_plane.c | 104 +
drivers/net/wwan/t9xx/mtk_data_plane.h | 105 +
drivers/net/wwan/t9xx/mtk_dev.c | 55 +
drivers/net/wwan/t9xx/mtk_dev.h | 114 +
drivers/net/wwan/t9xx/mtk_fsm.c | 931 +++++++
drivers/net/wwan/t9xx/mtk_fsm.h | 140 +
drivers/net/wwan/t9xx/mtk_port.c | 967 +++++++
drivers/net/wwan/t9xx/mtk_port.h | 176 ++
drivers/net/wwan/t9xx/mtk_port_io.c | 576 +++++
drivers/net/wwan/t9xx/mtk_port_io.h | 41 +
drivers/net/wwan/t9xx/mtk_utility.h | 33 +
drivers/net/wwan/t9xx/mtk_wwan.c | 475 ++++
drivers/net/wwan/t9xx/mtk_wwan.h | 17 +
drivers/net/wwan/t9xx/pcie/Makefile | 19 +
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 1527 +++++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 176 ++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c | 373 +++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 174 ++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c | 177 ++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h | 101 +
drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c | 55 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.c | 2714 ++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif.h | 16 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.c | 1586 ++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv.h | 268 ++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_drv_m9xx.c | 687 +++++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg.h | 387 +++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_reg_m9xx.h | 37 +
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.c | 168 ++
drivers/net/wwan/t9xx/pcie/mtk_dpmaif_ring.h | 161 ++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 1067 ++++++++
drivers/net/wwan/t9xx/pcie/mtk_pci.h | 219 ++
drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c | 70 +
drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 72 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 593 +++++
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 105 +
43 files changed, 14761 insertions(+)
---
base-commit: eb3f4b7426cfd2b79d65b7d37155480b32259a11
change-id: 20260529-t9xx_driver_v1-1744f8af7739
Best regards,
--
Jack Wu <jackbb_wu@compal.com>
^ permalink raw reply
* [PATCH 01/11] net: wwan: t9xx: Add PCIe core
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Registers the T900 device driver with the kernel. Set up all
the fundamental configurations for the device: PCIe layer,
Modem Host Cross Core Interface (MHCCIF), Reset Generation
Unit (RGU), modem common control operations and build
infrastructure.
* PCIe layer code implements driver probe and removal, MSI-X
interrupt initialization and de-initialization, and the way
of resetting the device.
* MHCCIF provides interrupt channels to communicate events
such as handshake, PM and port enumeration.
* RGU provides interrupt channels to generate notifications
from the device so that the T900 driver could get the
device reset.
* Modem common control operations provide the basic read/write
functions of the device's hardware registers,
mask/unmask/get/clear functions of the device's interrupt
registers and inquiry functions of the device's status.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/Kconfig | 12 +
drivers/net/wwan/Makefile | 1 +
drivers/net/wwan/t9xx/Makefile | 10 +
drivers/net/wwan/t9xx/mtk_dev.h | 108 +++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 926 ++++++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_pci.h | 219 ++++++
drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c | 69 ++
drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 71 ++
8 files changed, 1416 insertions(+)
diff --git a/drivers/net/wwan/Kconfig b/drivers/net/wwan/Kconfig
index 88df55d78d90..4cee537c739f 100644
--- a/drivers/net/wwan/Kconfig
+++ b/drivers/net/wwan/Kconfig
@@ -121,6 +121,18 @@ config MTK_T7XX
If unsure, say N.
+config MTK_T9XX
+ tristate "MediaTek PCIe 5G WWAN modem T9xx device"
+ depends on PCI
+ select NET_DEVLINK
+ help
+ Enables MediaTek PCIe based 5G WWAN modem (T9xx series) device.
+
+ To compile this driver as a module, choose M here: the module will be
+ called mtk_t9xx.
+
+ If unsure, say N.
+
endif # WWAN
endmenu
diff --git a/drivers/net/wwan/Makefile b/drivers/net/wwan/Makefile
index 3960c0ae2445..7361eef4c472 100644
--- a/drivers/net/wwan/Makefile
+++ b/drivers/net/wwan/Makefile
@@ -14,3 +14,4 @@ obj-$(CONFIG_QCOM_BAM_DMUX) += qcom_bam_dmux.o
obj-$(CONFIG_RPMSG_WWAN_CTRL) += rpmsg_wwan_ctrl.o
obj-$(CONFIG_IOSM) += iosm/
obj-$(CONFIG_MTK_T7XX) += t7xx/
+obj-$(CONFIG_MTK_T9XX) += t9xx/
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
new file mode 100644
index 000000000000..6f2dd3f91454
--- /dev/null
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+ccflags-y += -I$(src)/pcie
+ccflags-y += -I$(src)
+
+obj-$(CONFIG_MTK_T9XX) += mtk_t9xx.o
+
+mtk_t9xx-y := \
+ pcie/mtk_pci.o \
+ pcie/mtk_pci_drv_m9xx.o
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
new file mode 100644
index 000000000000..8278a0e2875e
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DEV_H__
+#define __MTK_DEV_H__
+
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#define MTK_DEV_STR_LEN 16
+
+enum mtk_user_id {
+ MTK_USER_MIN,
+ MTK_USER_CTRL,
+ MTK_USER_DATA,
+ MTK_USER_MAX
+};
+
+enum mtk_dev_evt_h2d {
+ DEV_EVT_H2D_DEVICE_RESET = BIT(2),
+ DEV_EVT_H2D_MAX = BIT(5)
+};
+
+enum mtk_dev_evt_d2h {
+ DEV_EVT_D2H_BOOT_FLOW_SYNC = BIT(4),
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP = BIT(5),
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD = BIT(6),
+ DEV_EVT_D2H_MAX = BIT(11)
+};
+
+struct mtk_md_dev;
+
+struct mtk_dev_ops {
+ u32 (*get_dev_state)(struct mtk_md_dev *mdev);
+ void (*ack_dev_state)(struct mtk_md_dev *mdev, u32 state);
+ u32 (*get_dev_cfg)(struct mtk_md_dev *mdev);
+ int (*register_dev_evt)(struct mtk_md_dev *mdev, u32 dev_evt,
+ int (*evt_cb)(u32 status, void *data), void *data);
+ void (*unregister_dev_evt)(struct mtk_md_dev *mdev, u32 dev_evt);
+ void (*mask_dev_evt)(struct mtk_md_dev *mdev, u32 dev_evt);
+ void (*unmask_dev_evt)(struct mtk_md_dev *mdev, u32 dev_evt);
+ void (*clear_dev_evt)(struct mtk_md_dev *mdev, u32 dev_evt);
+ int (*send_dev_evt)(struct mtk_md_dev *mdev, u32 dev_evt);
+};
+
+/* mtk_md_dev defines the structure of MTK modem device */
+struct mtk_md_dev {
+ struct device *dev;
+ const struct mtk_dev_ops *dev_ops;
+ void *hw_priv;
+ u32 hw_ver;
+ char dev_str[MTK_DEV_STR_LEN];
+};
+
+static inline u32 mtk_dev_get_dev_state(struct mtk_md_dev *mdev)
+{
+ return mdev->dev_ops->get_dev_state(mdev);
+}
+
+static inline void mtk_dev_ack_dev_state(struct mtk_md_dev *mdev, u32 state)
+{
+ return mdev->dev_ops->ack_dev_state(mdev, state);
+}
+
+static inline u32 mtk_dev_get_dev_cfg(struct mtk_md_dev *mdev)
+{
+ return mdev->dev_ops->get_dev_cfg(mdev);
+}
+
+static inline int mtk_dev_register_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt,
+ int (*evt_cb)(u32 status, void *data), void *data)
+{
+ return mdev->dev_ops->register_dev_evt(mdev, dev_evt, evt_cb, data);
+}
+
+static inline void mtk_dev_unregister_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt)
+{
+ mdev->dev_ops->unregister_dev_evt(mdev, dev_evt);
+}
+
+static inline void mtk_dev_mask_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt)
+{
+ mdev->dev_ops->mask_dev_evt(mdev, dev_evt);
+}
+
+static inline void mtk_dev_unmask_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt)
+{
+ mdev->dev_ops->unmask_dev_evt(mdev, dev_evt);
+}
+
+static inline void mtk_dev_clear_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt)
+{
+ mdev->dev_ops->clear_dev_evt(mdev, dev_evt);
+}
+
+static inline int mtk_dev_send_dev_evt(struct mtk_md_dev *mdev, u32 dev_evt)
+{
+ return mdev->dev_ops->send_dev_evt(mdev, dev_evt);
+}
+
+#endif /* __MTK_DEV_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
new file mode 100644
index 000000000000..adec3ccdee08
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -0,0 +1,926 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/acpi.h>
+#include <linux/aer.h>
+#include <linux/bitfield.h>
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+
+#define BAR_NUM 6
+#define MTK_PCI_TRANSPARENT_ATR_SIZE (0x3F)
+#define MTK_PCI_MINIMUM_ATR_SIZE (0x1000)
+#define LE32_TO_U32(x) ((__force u32)(__le32)(x))
+#define SET_HW_BITS(dest, chs, mhccif, dev) \
+ ({ \
+ if ((chs) & (dev)) \
+ (dest) |= FIELD_PREP(mhccif, 1); \
+ })
+
+extern const struct mtk_pci_dev_cfg mtk_dev_cfg_0900;
+
+struct mtk_mhccif_cb {
+ struct list_head entry;
+ int (*evt_cb)(u32 status, void *data);
+ void *data;
+ u32 chs;
+};
+
+u32 mtk_pci_mac_read32(struct mtk_pci_priv *priv, u64 addr)
+{
+ return ioread32(priv->mac_reg_base + addr);
+}
+
+void mtk_pci_mac_write32(struct mtk_pci_priv *priv, u64 addr, u32 val)
+{
+ iowrite32(val, priv->mac_reg_base + addr);
+}
+
+u32 mtk_pci_read32(struct mtk_md_dev *mdev, u64 addr)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ return ioread32(priv->ext_reg_base + addr);
+}
+
+void mtk_pci_write32(struct mtk_md_dev *mdev, u64 addr, u32 val)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ iowrite32(val, priv->ext_reg_base + addr);
+}
+
+int mtk_pci_setup_atr(struct mtk_md_dev *mdev, struct mtk_atr_cfg *cfg)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 addr, val, size_h, size_l;
+ int atr_size, pos, offset;
+
+ if (cfg->transparent) {
+ atr_size = MTK_PCI_TRANSPARENT_ATR_SIZE; /* No address conversion is performed */
+ } else {
+ if (cfg->size < MTK_PCI_MINIMUM_ATR_SIZE)
+ cfg->size = MTK_PCI_MINIMUM_ATR_SIZE;
+
+ if (cfg->src_addr & (cfg->size - 1)) {
+ dev_err((mdev)->dev, "Invalid atr src addr is not aligned to size\n");
+ return -EFAULT;
+ }
+ if (cfg->trsl_addr & (cfg->size - 1)) {
+ dev_err((mdev)->dev,
+ "Invalid atr trsl addr is not aligned to size, %llx, %llx\n",
+ cfg->trsl_addr, cfg->size - 1);
+ return -EFAULT;
+ }
+ size_l = FIELD_GET(GENMASK_ULL(31, 0), cfg->size);
+ size_h = FIELD_GET(GENMASK_ULL(63, 32), cfg->size);
+ pos = ffs(size_l);
+ if (pos) {
+ atr_size = pos - 2;
+ } else {
+ pos = ffs(size_h);
+ atr_size = pos + 30;
+ }
+ }
+
+ /* Calculate table offset */
+ offset = ATR_PORT_OFFSET * cfg->port + ATR_TABLE_OFFSET * cfg->table;
+ /* SRC_ADDR_H */
+ addr = REG_ATR_PCIE_WIN0_T0_SRC_ADDR_MSB + offset;
+ val = (u32)(cfg->src_addr >> 32);
+ mtk_pci_mac_write32(priv, addr, val);
+ /* SRC_ADDR_L */
+ addr = REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset;
+ val = (u32)(cfg->src_addr & 0xFFFFF000) | (atr_size << 1) | 0x1;
+ mtk_pci_mac_write32(priv, addr, val);
+
+ /* TRSL_ADDR_H */
+ addr = REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_MSB + offset;
+ val = (u32)(cfg->trsl_addr >> 32);
+ mtk_pci_mac_write32(priv, addr, val);
+ /* TRSL_ADDR_L */
+ addr = REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_LSB + offset;
+ val = (u32)(cfg->trsl_addr & 0xFFFFF000);
+ mtk_pci_mac_write32(priv, addr, val);
+
+ /* TRSL_PARAM */
+ addr = REG_ATR_PCIE_WIN0_T0_TRSL_PARAM + offset;
+ val = (cfg->trsl_param << 16) | cfg->trsl_id;
+ mtk_pci_mac_write32(priv, addr, val);
+
+ return 0;
+}
+
+void mtk_pci_atr_disable(struct mtk_pci_priv *priv)
+{
+ int port, tbl, offset;
+ u32 val;
+
+ /* Disable all ATR table for all ports */
+ for (port = ATR_SRC_PCI_WIN0; port <= ATR_SRC_AXIS_3; port++)
+ for (tbl = 0; tbl < ATR_TABLE_NUM_PER_ATR; tbl++) {
+ /* Calculate table offset */
+ offset = ATR_PORT_OFFSET * port + ATR_TABLE_OFFSET * tbl;
+ val = mtk_pci_mac_read32(priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset);
+ val = val & (~BIT(0));
+ /* Disable table by SRC_ADDR_L */
+ mtk_pci_mac_write32(priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset, val);
+ }
+}
+
+static void mtk_pci_set_msix_merged(struct mtk_pci_priv *priv, int irq_cnt)
+{
+ mtk_pci_mac_write32(priv, REG_PCIE_CFG_MSIX, ffs(irq_cnt) * 2 - 1);
+}
+
+u32 mtk_pci_get_dev_state(struct mtk_md_dev *mdev)
+{
+ return mtk_pci_mac_read32(mdev->hw_priv, REG_PCIE_DEBUG_DUMMY_7);
+}
+
+void mtk_pci_ack_dev_state(struct mtk_md_dev *mdev, u32 state)
+{
+ mtk_pci_mac_write32(mdev->hw_priv, REG_PCIE_DEBUG_DUMMY_7, state);
+}
+
+int mtk_pci_get_irq_id(struct mtk_md_dev *mdev, enum mtk_irq_src irq_src)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ const int *irq_tbl = priv->cfg->irq_tbl;
+ int irq_id = -EINVAL;
+
+ if (irq_src > MTK_IRQ_SRC_MIN && irq_src < MTK_IRQ_SRC_MAX) {
+ irq_id = irq_tbl[irq_src];
+ if (unlikely(irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX))
+ irq_id = -EINVAL;
+ }
+
+ return irq_id;
+}
+
+int mtk_pci_get_virq_id(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int nr = 0;
+
+ nr = irq_id % priv->irq_cnt;
+
+ return pci_irq_vector(pdev, nr);
+}
+
+int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id,
+ int (*irq_cb)(int irq_id, void *data), void *data)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (unlikely((irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) || !irq_cb))
+ return -EINVAL;
+
+ if (priv->irq_cb_list[irq_id]) {
+ dev_err((mdev)->dev,
+ "Unable to register irq, irq_id=%d, it's already been register by %ps.\n",
+ irq_id, priv->irq_cb_list[irq_id]);
+ return -EFAULT;
+ }
+ priv->irq_cb_list[irq_id] = irq_cb;
+ priv->irq_cb_data[irq_id] = data;
+
+ return 0;
+}
+
+int mtk_pci_unregister_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (unlikely(irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX))
+ return -EINVAL;
+
+ if (!priv->irq_cb_list[irq_id]) {
+ dev_err((mdev)->dev, "irq_id=%d has not been registered\n", irq_id);
+ return -EFAULT;
+ }
+ priv->irq_cb_list[irq_id] = NULL;
+ priv->irq_cb_data[irq_id] = NULL;
+
+ return 0;
+}
+
+int mtk_pci_mask_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (unlikely((irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) || priv->irq_type != PCI_IRQ_MSIX)) {
+ dev_err(mdev->dev, "Failed to mask irq: input irq_id=%d\n", irq_id);
+ return -EINVAL;
+ }
+
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, BIT(irq_id));
+
+ return 0;
+}
+
+int mtk_pci_unmask_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (unlikely((irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) || priv->irq_type != PCI_IRQ_MSIX)) {
+ dev_err(mdev->dev, "Failed to unmask irq: input irq_id=%d\n", irq_id);
+ return -EINVAL;
+ }
+
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_SET_GRP0_0, BIT(irq_id));
+
+ return 0;
+}
+
+int mtk_pci_clear_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (unlikely((irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) || priv->irq_type != PCI_IRQ_MSIX)) {
+ dev_err(mdev->dev, "Failed to clear irq: input irq_id=%d\n", irq_id);
+ return -EINVAL;
+ }
+
+ mtk_pci_mac_write32(priv, REG_MSIX_ISTATUS_HOST_GRP0_0, BIT(irq_id));
+
+ return 0;
+}
+
+static u32 mtk_pci_ext_d2h_evt_hw_bits(u32 chs)
+{
+ u32 hw_bits = 0;
+
+ SET_HW_BITS(hw_bits, chs, MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC,
+ DEV_EVT_D2H_BOOT_FLOW_SYNC);
+ SET_HW_BITS(hw_bits, chs, MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP,
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP);
+ SET_HW_BITS(hw_bits, chs, MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD,
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD);
+
+ return LE32_TO_U32(cpu_to_le32(hw_bits));
+}
+
+static u32 mtk_pci_ext_d2h_evt_chs(u32 hw_bits)
+{
+ u32 chs = 0;
+
+ if (!hw_bits)
+ return chs;
+
+ chs = FIELD_PREP(DEV_EVT_D2H_BOOT_FLOW_SYNC,
+ FIELD_GET(MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC, hw_bits)) |
+ FIELD_PREP(DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP,
+ FIELD_GET(MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP, hw_bits)) |
+ FIELD_PREP(DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD,
+ FIELD_GET(MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD, hw_bits));
+
+ return chs;
+}
+
+int mtk_pci_register_ext_evt(struct mtk_md_dev *mdev, u32 chs,
+ int (*evt_cb)(u32 status, void *data), void *data)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_mhccif_cb *cb;
+ int ret = 0;
+
+ if (!chs || !evt_cb)
+ return -EINVAL;
+
+ spin_lock_bh(&priv->mhccif_lock);
+ list_for_each_entry(cb, &priv->mhccif_cb_list, entry) {
+ if (cb->chs & chs) {
+ ret = -EFAULT;
+ dev_err((mdev)->dev,
+ "Unable to register evt, intersection: chs=0x%08x&0x%08x registered_cb=%ps\n",
+ chs, cb->chs, cb->evt_cb);
+ goto err_spin_unlock;
+ }
+ }
+ cb = devm_kzalloc(mdev->dev, sizeof(*cb), GFP_ATOMIC);
+ if (!cb) {
+ ret = -ENOMEM;
+ goto err_spin_unlock;
+ }
+ cb->evt_cb = evt_cb;
+ cb->data = data;
+ cb->chs = chs;
+ list_add_tail(&cb->entry, &priv->mhccif_cb_list);
+err_spin_unlock:
+ spin_unlock_bh(&priv->mhccif_lock);
+
+ return ret;
+}
+
+void mtk_pci_unregister_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_mhccif_cb *cb, *next;
+
+ if (!chs)
+ return;
+
+ spin_lock_bh(&priv->mhccif_lock);
+ list_for_each_entry_safe(cb, next, &priv->mhccif_cb_list, entry) {
+ if (cb->chs == chs) {
+ list_del(&cb->entry);
+ devm_kfree(mdev->dev, cb);
+ goto out;
+ }
+ }
+ dev_warn((mdev)->dev,
+ "Unable to unregister evt, no chs=0x%08x has been registered.\n", chs);
+out:
+ spin_unlock_bh(&priv->mhccif_lock);
+}
+
+void mtk_pci_mask_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 hw_bits;
+
+ hw_bits = mtk_pci_ext_d2h_evt_hw_bits(chs);
+
+ mtk_pci_write32(mdev, priv->cfg->mhccif_rc_base_addr +
+ MHCCIF_EP2RC_SW_INT_EAP_MASK_SET, hw_bits);
+}
+
+void mtk_pci_unmask_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 hw_bits;
+
+ hw_bits = mtk_pci_ext_d2h_evt_hw_bits(chs);
+
+ mtk_pci_write32(mdev, priv->cfg->mhccif_rc_base_addr +
+ MHCCIF_EP2RC_SW_INT_EAP_MASK_CLR, hw_bits);
+}
+
+void mtk_pci_clear_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 hw_bits;
+
+ hw_bits = mtk_pci_ext_d2h_evt_hw_bits(chs);
+
+ mtk_pci_write32(mdev, priv->cfg->mhccif_rc_base_addr +
+ MHCCIF_EP2RC_SW_INT_ACK, hw_bits);
+}
+
+static u32 mtk_pci_ext_h2d_evt_hw_bits(u32 chs)
+{
+ u32 hw_bits = 0;
+
+ SET_HW_BITS(hw_bits, chs, MHCCIF_RC2EP_EVT_DEVICE_RESET,
+ DEV_EVT_H2D_DEVICE_RESET);
+ SET_HW_BITS(hw_bits, chs, MHCCIF_RC2EP_EVT_DRM_DISABLE_AP,
+ EXT_EVT_H2D_DRM_DISABLE_AP);
+ return LE32_TO_U32(cpu_to_le32(hw_bits));
+}
+
+int mtk_pci_send_ext_evt(struct mtk_md_dev *mdev, u32 ch)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 rc_base;
+ u32 hw_bits;
+
+ rc_base = priv->cfg->mhccif_rc_base_addr;
+
+ /* Only allow one ch to be triggered at a time */
+ if (!is_power_of_2(ch)) {
+ dev_err((mdev)->dev, "Unsupported ext evt ch=0x%08x\n", ch);
+ return -EINVAL;
+ }
+
+ hw_bits = mtk_pci_ext_h2d_evt_hw_bits(ch);
+ mtk_pci_write32(mdev, rc_base + MHCCIF_RC2EP_SW_BSY, hw_bits);
+ mtk_pci_write32(mdev, rc_base + MHCCIF_RC2EP_SW_TCHNUM, ffs(hw_bits) - 1);
+ return 0;
+}
+
+static u32 mtk_pci_get_ext_evt_hw_status(struct mtk_md_dev *mdev)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ return mtk_pci_read32(mdev, priv->cfg->mhccif_rc_base_addr + MHCCIF_EP2RC_SW_INT_STS);
+}
+
+int mtk_pci_fldr(struct mtk_md_dev *mdev)
+{
+#ifdef CONFIG_ACPI
+ struct acpi_buffer buffer = { ACPI_ALLOCATE_BUFFER, NULL };
+ acpi_status acpi_ret;
+ acpi_handle handle;
+
+ if (acpi_disabled) {
+ dev_err((mdev)->dev, "Unsupported, acpi function isn't enable\n");
+ return -ENODEV;
+ }
+ handle = ACPI_HANDLE(mdev->dev);
+ if (!handle) {
+ dev_err((mdev)->dev, "Unsupported, acpi handle isn't found\n");
+ return -ENODEV;
+ }
+ if (!acpi_has_method(handle, "_RST")) {
+ dev_err((mdev)->dev, "Unsupported, _RST method isn't found\n");
+ return -ENODEV;
+ }
+ acpi_ret = acpi_evaluate_object(handle, "_RST", NULL, &buffer);
+ if (ACPI_FAILURE(acpi_ret)) {
+ dev_err((mdev)->dev, "Failed to execute _RST method: %s\n",
+ acpi_format_exception(acpi_ret));
+ return -EFAULT;
+ }
+ acpi_os_free(buffer.pointer);
+
+ return 0;
+#else
+ dev_err((mdev)->dev, "Unsupported, CONFIG ACPI hasn't been set to 'y'\n");
+
+ return -ENODEV;
+#endif
+}
+
+int mtk_pci_pldr(struct mtk_md_dev *mdev)
+{
+#ifdef CONFIG_ACPI
+ struct acpi_buffer buffer = { ACPI_ALLOCATE_BUFFER, NULL };
+ struct pci_dev *bridge;
+ acpi_status acpi_ret;
+ acpi_handle handle;
+
+ if (acpi_disabled) {
+ dev_err((mdev)->dev, "Unsupported, acpi function isn't enable\n");
+ return -ENODEV;
+ }
+
+ bridge = pci_upstream_bridge(to_pci_dev(mdev->dev));
+ if (!bridge) {
+ dev_err((mdev)->dev, "Unable to find bridge\n");
+ return -ENODEV;
+ }
+
+ handle = ACPI_HANDLE(&bridge->dev);
+ if (!handle) {
+ dev_err((mdev)->dev, "Unsupported, acpi handle isn't found\n");
+ return -ENODEV;
+ }
+ if (!acpi_has_method(handle, "PXP._OFF") ||
+ !acpi_has_method(handle, "PXP._ON")) {
+ dev_err((mdev)->dev, "Unsupported, pldr method isn't supported\n");
+ return -ENODEV;
+ }
+ acpi_ret = acpi_evaluate_object(handle, "PXP._OFF", NULL, &buffer);
+ if (ACPI_FAILURE(acpi_ret)) {
+ dev_err((mdev)->dev, "Failed to execute _OFF method: %s\n",
+ acpi_format_exception(acpi_ret));
+ return -EFAULT;
+ }
+ msleep(500);
+ acpi_ret = acpi_evaluate_object(handle, "PXP._ON", NULL, &buffer);
+ if (ACPI_FAILURE(acpi_ret)) {
+ dev_err((mdev)->dev, "Failed to execute _ON method: %s\n",
+ acpi_format_exception(acpi_ret));
+ return -EFAULT;
+ }
+ acpi_os_free(buffer.pointer);
+
+ return 0;
+#else
+ dev_err((mdev)->dev, "Unsupported, CONFIG ACPI hasn't been set to 'y'\n");
+
+ return -ENODEV;
+#endif
+}
+
+u32 mtk_pci_get_dev_cfg(struct mtk_md_dev *mdev)
+{
+ u32 val;
+
+ val = mtk_pci_mac_read32(mdev->hw_priv, REG_PCIE_DEBUG_DUMMY_4);
+ return (val >> MTK_CFG_INFO_BIT_SHIFT);
+}
+
+static int mtk_pci_dev_reset(struct mtk_md_dev *mdev, enum mtk_reset_type type)
+{
+ switch (type) {
+ case RESET_MHCCIF:
+ return mtk_pci_send_ext_evt(mdev, DEV_EVT_H2D_DEVICE_RESET);
+ case RESET_FLDR:
+ return mtk_pci_fldr(mdev);
+ case RESET_PLDR:
+ return mtk_pci_pldr(mdev);
+ default:
+ break;
+ }
+
+ return -EINVAL;
+}
+
+int mtk_pci_reset(struct mtk_md_dev *mdev, enum mtk_reset_type type)
+{
+ return mtk_pci_dev_reset(mdev, type);
+}
+
+bool mtk_pci_link_check(struct mtk_md_dev *mdev)
+{
+ return pci_device_is_present(to_pci_dev(mdev->dev));
+}
+
+static void mtk_mhccif_isr_work(struct work_struct *work)
+{
+ struct mtk_pci_priv *priv = container_of(work, struct mtk_pci_priv, mhccif_work);
+ struct mtk_md_dev *mdev = priv->irq_desc->mdev;
+ struct mtk_mhccif_cb *cb;
+ u32 stat, mask, chs;
+
+ stat = mtk_pci_get_ext_evt_hw_status(mdev);
+ mask = mtk_pci_read32(mdev, priv->cfg->mhccif_rc_base_addr
+ + MHCCIF_EP2RC_SW_INT_EAP_MASK);
+ if (unlikely(stat == U32_MAX && !(mtk_pci_link_check(mdev)))) {
+ /* When link failed, we don't need to unmask/clear. */
+ dev_err((mdev)->dev, "Failed to check link in MHCCIF handler.\n");
+ return;
+ }
+
+ stat &= ~mask;
+ chs = mtk_pci_ext_d2h_evt_chs(stat);
+ spin_lock_bh(&priv->mhccif_lock);
+ list_for_each_entry(cb, &priv->mhccif_cb_list, entry) {
+ if (cb->chs & chs)
+ cb->evt_cb(cb->chs & chs, cb->data);
+ }
+ spin_unlock_bh(&priv->mhccif_lock);
+
+ mtk_pci_clear_irq(mdev, priv->mhccif_irq_id);
+ mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
+}
+
+static const struct pci_device_id t9xx_pci_table[] = {
+ MTK_PCI_DEV_CFG(0x0900, mtk_dev_cfg_0900),
+ CEI_PCI_DEV_CFG(0x01CA, mtk_dev_cfg_0900),
+ {/* end: all zeroes */}
+};
+
+MODULE_DEVICE_TABLE(pci, t9xx_pci_table);
+
+static int mtk_pci_bar_init(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 bar[BAR_NUM];
+ int i, ret;
+
+ for (i = 0; i < BAR_NUM; i++)
+ pci_read_config_dword(to_pci_dev(mdev->dev),
+ PCI_BASE_ADDRESS_0 + (i << 2), bar + i);
+
+ ret = pcim_iomap_regions(pdev, MTK_REQUESTED_BARS, mdev->dev_str);
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to init MMIO. ret=%d\n", ret);
+ return ret;
+ }
+
+ /* get ioremapped memory */
+ priv->mac_reg_base = pcim_iomap_table(pdev)[MTK_BAR_0_1_IDX];
+ priv->bar23_addr = pcim_iomap_table(pdev)[MTK_BAR_2_3_IDX];
+ if (!priv->mac_reg_base || !priv->bar23_addr) {
+ dev_err((mdev)->dev, "Failed to init BAR.\n");
+ return -EINVAL;
+ }
+ /* We use MD view base address "0" to observe registers */
+ priv->ext_reg_base = priv->bar23_addr - ATR_PCIE_REG_TRSL_ADDR;
+
+ return 0;
+}
+
+static void mtk_pci_bar_exit(struct mtk_md_dev *mdev)
+{
+ pcim_iounmap_region(to_pci_dev(mdev->dev), MTK_REQUESTED_BARS);
+}
+
+static int mtk_mhccif_irq_cb(int irq_id, void *data)
+{
+ struct mtk_md_dev *mdev = data;
+ struct mtk_pci_priv *priv;
+
+ priv = mdev->hw_priv;
+ queue_work(system_highpri_wq, &priv->mhccif_work);
+
+ return 0;
+}
+
+static int mtk_mhccif_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int ret;
+
+ INIT_LIST_HEAD(&priv->mhccif_cb_list);
+ spin_lock_init(&priv->mhccif_lock);
+ INIT_WORK(&priv->mhccif_work, mtk_mhccif_isr_work);
+
+ ret = mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_MHCCIF);
+ if (ret < 0) {
+ dev_err((mdev)->dev, "Failed to get mhccif_irq_id. ret=%d\n", ret);
+ goto err;
+ }
+ priv->mhccif_irq_id = ret;
+
+ ret = mtk_pci_register_irq(mdev, priv->mhccif_irq_id, mtk_mhccif_irq_cb, mdev);
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to register mhccif_irq callback\n");
+ goto err;
+ }
+
+err:
+ return ret;
+}
+
+static void mtk_mhccif_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ mtk_pci_unregister_irq(mdev, priv->mhccif_irq_id);
+ cancel_work_sync(&priv->mhccif_work);
+}
+
+static irqreturn_t mtk_pci_irq_handler(struct mtk_md_dev *mdev, u32 irq_state)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int irq_id;
+
+ /* Check whether each set bit has a callback, if has, call it */
+ do {
+ irq_id = fls(irq_state) - 1;
+ irq_state &= ~BIT(irq_id);
+ if (likely(priv->irq_cb_list[irq_id]))
+ priv->irq_cb_list[irq_id](irq_id, priv->irq_cb_data[irq_id]);
+ else
+ dev_err((mdev)->dev, "Unhandled irq_id=%d, no callback for it.\n", irq_id);
+ } while (irq_state);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t mtk_pci_irq_msix(int irq, void *data)
+{
+ struct mtk_pci_irq_desc *irq_desc = data;
+ struct mtk_md_dev *mdev = irq_desc->mdev;
+ struct mtk_pci_priv *priv;
+ u32 irq_state, irq_enable;
+
+ priv = mdev->hw_priv;
+ irq_state = mtk_pci_mac_read32(priv, REG_MSIX_ISTATUS_HOST_GRP0_0);
+ irq_enable = mtk_pci_mac_read32(priv, REG_IMASK_HOST_MSIX_GRP0_0);
+ irq_state &= irq_enable;
+
+ if (unlikely(!irq_state) ||
+ unlikely(!((irq_state & GENMASK(priv->irq_cnt - 1, 0)) & irq_desc->msix_bits)))
+ return IRQ_NONE;
+
+ /* Mask the bit and user needs to unmask by itself */
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, irq_state & ~BIT(30));
+
+ return mtk_pci_irq_handler(mdev, irq_state);
+}
+
+static int mtk_pci_request_irq_msix(struct mtk_md_dev *mdev, int irq_cnt_allocated)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_pci_irq_desc *irq_desc;
+ struct pci_dev *pdev;
+ int irq_cnt;
+ int ret, i;
+
+ /* calculate the nearest 2's power number */
+ irq_cnt = BIT(fls(irq_cnt_allocated) - 1);
+ pdev = to_pci_dev(mdev->dev);
+ irq_desc = priv->irq_desc;
+ for (i = 0; i < irq_cnt; i++) {
+ irq_desc[i].mdev = mdev;
+ irq_desc[i].msix_bits = BIT(i);
+ snprintf(irq_desc[i].name, MTK_IRQ_NAME_LEN, "msix%d-%s", i, mdev->dev_str);
+ ret = pci_request_irq(pdev, i, mtk_pci_irq_msix, NULL,
+ &irq_desc[i], irq_desc[i].name);
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to request %s: ret=%d\n",
+ irq_desc[i].name, ret);
+ for (i--; i >= 0; i--)
+ pci_free_irq(pdev, i, &irq_desc[i]);
+ return ret;
+ }
+ }
+ priv->irq_cnt = irq_cnt;
+ priv->irq_type = PCI_IRQ_MSIX;
+
+ if (irq_cnt != MTK_IRQ_CNT_MAX)
+ mtk_pci_set_msix_merged(priv, irq_cnt);
+
+ return 0;
+}
+
+static int mtk_pci_request_irq(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ int irq_cnt;
+
+ irq_cnt = pci_alloc_irq_vectors(pdev, MTK_IRQ_CNT_MIN, MTK_IRQ_CNT_MAX, PCI_IRQ_MSIX);
+
+ if (irq_cnt < MTK_IRQ_CNT_MIN) {
+ dev_err(mdev->dev,
+ "Unable to alloc pci irq vectors. ret=%d maxirqcnt=%d irqtype=0x%x\n",
+ irq_cnt, MTK_IRQ_CNT_MAX, PCI_IRQ_MSIX);
+ return -EFAULT;
+ }
+
+ return mtk_pci_request_irq_msix(mdev, irq_cnt);
+}
+
+static void mtk_pci_free_irq(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int i;
+
+ for (i = 0; i < priv->irq_cnt; i++)
+ pci_free_irq(pdev, i, &priv->irq_desc[i]);
+
+ pci_free_irq_vectors(pdev);
+}
+
+static const struct mtk_dev_ops pci_hw_ops = {
+ .get_dev_state = mtk_pci_get_dev_state,
+ .ack_dev_state = mtk_pci_ack_dev_state,
+ .get_dev_cfg = mtk_pci_get_dev_cfg,
+ .register_dev_evt = mtk_pci_register_ext_evt,
+ .unregister_dev_evt = mtk_pci_unregister_ext_evt,
+ .mask_dev_evt = mtk_pci_mask_ext_evt,
+ .unmask_dev_evt = mtk_pci_unmask_ext_evt,
+ .clear_dev_evt = mtk_pci_clear_ext_evt,
+ .send_dev_evt = mtk_pci_send_ext_evt,
+};
+
+static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ struct device *dev = &pdev->dev;
+ struct mtk_pci_priv *priv;
+ struct mtk_md_dev *mdev;
+ int ret;
+
+ mdev = devm_kzalloc(dev, sizeof(*mdev), GFP_KERNEL);
+ if (!mdev) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ mdev->dev_ops = &pci_hw_ops;
+ mdev->dev = dev;
+
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv) {
+ ret = -ENOMEM;
+ goto free_cntx_data;
+ }
+
+ pci_set_drvdata(pdev, mdev);
+ priv->cfg = (void *)id->driver_data;
+ priv->mdev = mdev;
+ mdev->hw_ver = pdev->device;
+ mdev->hw_priv = priv;
+ mdev->dev = dev;
+ snprintf(mdev->dev_str, MTK_DEV_STR_LEN, "%02x%02x%d",
+ pdev->bus->number, PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
+ if (pdev->state_saved)
+ pci_restore_state(pdev);
+
+ ret = pcim_enable_device(pdev);
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to enable pci device.\n");
+ goto free_priv_data;
+ }
+
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to set DMA Mask and Coherent. (ret=%d)\n", ret);
+ goto disable_device;
+ }
+
+ ret = mtk_pci_bar_init(mdev);
+ if (ret)
+ goto disable_device;
+
+ ret = priv->cfg->atr_init(mdev);
+ if (ret)
+ goto free_bar;
+
+ ret = mtk_mhccif_init(mdev);
+ if (ret)
+ goto free_bar;
+
+ /* mask all irqs */
+ if (priv->cfg->flag & MTK_CFG_IRQ_DFLT_MASK)
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX);
+
+ ret = mtk_pci_request_irq(mdev);
+ if (ret)
+ goto free_mhccif;
+
+ pci_set_master(pdev);
+ mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
+
+ if (mtk_pci_link_check(mdev)) {
+ pci_save_state(pdev);
+ } else {
+ ret = -EFAULT;
+ goto clear_master;
+ }
+
+ priv->saved_state = pci_store_saved_state(pdev);
+ if (!priv->saved_state) {
+ ret = -EFAULT;
+ goto clear_master;
+ }
+
+ return 0;
+
+clear_master:
+ pci_clear_master(pdev);
+ mtk_pci_free_irq(mdev);
+free_mhccif:
+ mtk_mhccif_exit(mdev);
+free_bar:
+ mtk_pci_bar_exit(mdev);
+disable_device:
+ pci_disable_device(pdev);
+free_priv_data:
+ devm_kfree(dev, priv);
+free_cntx_data:
+ devm_kfree(dev, mdev);
+out:
+ dev_err(dev, "Failed to probe device, ret=%d\n", ret);
+
+ return ret;
+}
+
+static void mtk_pci_remove(struct pci_dev *pdev)
+{
+ struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct device *dev = &pdev->dev;
+
+ mtk_pci_mask_irq(mdev, priv->mhccif_irq_id);
+
+ if (mtk_pci_pldr(mdev)) {
+ dev_warn(dev, "Failed to execute PLDR, try external event\n");
+ mtk_pci_reset(mdev, RESET_MHCCIF);
+ }
+
+ pci_clear_master(pdev);
+ mtk_mhccif_exit(mdev);
+ mtk_pci_free_irq(mdev);
+ mtk_pci_bar_exit(mdev);
+ pci_disable_device(pdev);
+ pci_load_and_free_saved_state(pdev, &priv->saved_state);
+
+ devm_kfree(dev, priv);
+ devm_kfree(dev, mdev);
+}
+
+static pci_ers_result_t mtk_pci_error_detected(struct pci_dev *pdev,
+ pci_channel_state_t state)
+{
+ struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
+
+ dev_err((mdev)->dev, "AER detected: pci_channel_state_t=%d\n", state);
+
+ /* Request a slot reset. */
+ return PCI_ERS_RESULT_CAN_RECOVER;
+}
+
+static const struct pci_error_handlers mtk_pci_err_handler = {
+ .error_detected = mtk_pci_error_detected,
+};
+
+static struct pci_driver mtk_pci_drv = {
+ .name = "mtk_pci_drv",
+ .id_table = t9xx_pci_table,
+ .probe = mtk_pci_probe,
+ .remove = mtk_pci_remove,
+ .err_handler = &mtk_pci_err_handler
+};
+
+module_pci_driver(mtk_pci_drv);
+
+MODULE_DESCRIPTION("MediaTek T9xx PCIe WWAN driver pcie layer");
+MODULE_LICENSE("GPL");
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.h b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
new file mode 100644
index 000000000000..700879f9b21e
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
@@ -0,0 +1,219 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PCI_H__
+#define __MTK_PCI_H__
+
+#include <linux/pci.h>
+
+#include "../mtk_dev.h"
+
+enum mtk_irq_src {
+ MTK_IRQ_SRC_MIN,
+ MTK_IRQ_SRC_MHCCIF,
+ MTK_IRQ_SRC_DPMAIF,
+ MTK_IRQ_SRC_DPMAIF2,
+ MTK_IRQ_SRC_CLDMA0,
+ MTK_IRQ_SRC_CLDMA1,
+ MTK_IRQ_SRC_CLDMA2,
+ MTK_IRQ_SRC_CLDMA3,
+ MTK_IRQ_SRC_PM_LOCK,
+ MTK_IRQ_SRC_DPMAIF3,
+ MTK_IRQ_SRC_DPMAIF6,
+ MTK_IRQ_SRC_MAX
+};
+
+enum mtk_reset_type {
+ RESET_FLDR,
+ RESET_PLDR,
+ RESET_MHCCIF,
+};
+
+enum mtk_atr_type {
+ ATR_PCI2AXI = 0,
+ ATR_AXI2PCI,
+};
+
+enum mtk_atr_src_port {
+ ATR_SRC_PCI_WIN0 = 0,
+ ATR_SRC_PCI_WIN1,
+ ATR_SRC_AXIS_0,
+ ATR_SRC_AXIS_1,
+ ATR_SRC_AXIS_2,
+ ATR_SRC_AXIS_3,
+};
+
+enum mtk_atr_dst_port {
+ ATR_DST_PCI_TRX = 0,
+ ATR_DST_AXIM_0 = 4,
+ ATR_DST_AXIM_1,
+ ATR_DST_AXIM_2,
+ ATR_DST_AXIM_3,
+};
+
+enum mtk_pci_evt_h2d {
+ DEV_EVT_H2D_EXTEND_BASE = DEV_EVT_H2D_MAX,
+ EXT_EVT_H2D_RESERVED_FOR_CLDMA0 = DEV_EVT_H2D_EXTEND_BASE << 1,
+ EXT_EVT_H2D_RESERVED_FOR_CLDMA1 = DEV_EVT_H2D_EXTEND_BASE << 2,
+ EXT_EVT_H2D_RESERVED_FOR_CLDMA3 = DEV_EVT_H2D_EXTEND_BASE << 3,
+ EXT_EVT_H2D_RESERVED_FOR_CLDMA2 = DEV_EVT_H2D_EXTEND_BASE << 4,
+ EXT_EVT_H2D_RESERVED_FOR_DPMAIF = DEV_EVT_H2D_EXTEND_BASE << 5,
+ EXT_EVT_H2D_PCIE_PM_SUSPEND_REQ = DEV_EVT_H2D_EXTEND_BASE << 6,
+ EXT_EVT_H2D_PCIE_PM_RESUME_REQ = DEV_EVT_H2D_EXTEND_BASE << 7,
+ EXT_EVT_H2D_PCIE_PM_SUSPEND_REQ_AP = DEV_EVT_H2D_EXTEND_BASE << 8,
+ EXT_EVT_H2D_PCIE_PM_RESUME_REQ_AP = DEV_EVT_H2D_EXTEND_BASE << 9,
+ EXT_EVT_H2D_DRM_DISABLE_AP = DEV_EVT_H2D_EXTEND_BASE << 10,
+ EXT_EVT_H2D_RESERVED_FOR_TEST = DEV_EVT_H2D_EXTEND_BASE << 11,
+};
+
+enum mtk_pci_evt_d2h {
+ DEV_EVT_D2H_EXTEND_BASE = DEV_EVT_D2H_MAX,
+ EXT_EVT_D2H_RESERVED_FOR_CLDMA0 = DEV_EVT_D2H_EXTEND_BASE << 1,
+ EXT_EVT_D2H_RESERVED_FOR_CLDMA1 = DEV_EVT_D2H_EXTEND_BASE << 2,
+ EXT_EVT_D2H_RESERVED_FOR_CLDMA3 = DEV_EVT_D2H_EXTEND_BASE << 3,
+ EXT_EVT_D2H_RESERVED_FOR_CLDMA2 = DEV_EVT_D2H_EXTEND_BASE << 4,
+ EXT_EVT_D2H_RESERVED_FOR_DPMAIF = DEV_EVT_D2H_EXTEND_BASE << 5,
+ EXT_EVT_D2H_PCIE_PM_SUSPEND_ACK = DEV_EVT_D2H_EXTEND_BASE << 6,
+ EXT_EVT_D2H_PCIE_PM_RESUME_ACK = DEV_EVT_D2H_EXTEND_BASE << 7,
+ EXT_EVT_D2H_PCIE_PM_SUSPEND_ACK_AP = DEV_EVT_D2H_EXTEND_BASE << 8,
+ EXT_EVT_D2H_PCIE_PM_RESUME_ACK_AP = DEV_EVT_D2H_EXTEND_BASE << 9,
+ EXT_EVT_D2H_SOFT_OFF_NOTIFY = DEV_EVT_D2H_EXTEND_BASE << 10,
+ EXT_EVT_D2H_FRC_DONE_NOTIFY = DEV_EVT_D2H_EXTEND_BASE << 11,
+ EXT_EVT_D2H_RESERVED_FOR_TEST1 = DEV_EVT_D2H_EXTEND_BASE << 12,
+ EXT_EVT_D2H_RESERVED_FOR_TEST2 = DEV_EVT_D2H_EXTEND_BASE << 13,
+};
+
+#define MTK_PCI_CLASS 0x0D4000
+#define MTK_PCI_VENDOR_ID 0x14C3
+#define CEI_PCI_VENDOR_ID 0x03F0
+
+#define MTK_CFG_INFO_BIT_SHIFT 4
+
+#define MTK_PCI_DEV_CFG(id, cfg) \
+{ \
+ PCI_DEVICE(MTK_PCI_VENDOR_ID, id), \
+ MTK_PCI_CLASS, PCI_ANY_ID, \
+ .driver_data = (kernel_ulong_t)&(cfg), \
+}
+
+#define CEI_PCI_DEV_CFG(id, cfg) \
+{ \
+ PCI_DEVICE(CEI_PCI_VENDOR_ID, id), \
+ MTK_PCI_CLASS, PCI_ANY_ID, \
+ .driver_data = (kernel_ulong_t)&(cfg), \
+}
+
+#define MTK_CFG_IRQ_DFLT_MASK BIT(0)
+#define MTK_CFG_DISABLE_AP_DRM BIT(2)
+#define MTK_CFG_PM_SW_IRQ BIT(6)
+
+#define MTK_BAR_0_1_IDX 0
+#define MTK_BAR_2_3_IDX 2
+
+#define MTK_REQUESTED_BARS \
+ ((1 << MTK_BAR_0_1_IDX) | \
+ (1 << MTK_BAR_2_3_IDX))
+
+#define MTK_IRQ_CNT_MIN 1
+#define MTK_IRQ_CNT_MAX 32
+#define MTK_IRQ_NAME_LEN 20
+
+#define ATR_PORT_OFFSET 0x100
+#define ATR_TABLE_OFFSET 0x20
+#define ATR_TABLE_NUM_PER_ATR 8
+#define ATR_PCIE_REG_TRSL_ADDR 0x10000000
+#define ATR_PCIE_REG_SIZE 0x00400000
+#define ATR_PCIE_REG_PORT ATR_SRC_PCI_WIN0
+#define ATR_PCIE_REG_TABLE_NUM 1
+#define ATR_PCIE_REG_TRSL_PORT ATR_DST_AXIM_0
+#define ATR_PCIE_DEV_DMA_SRC_ADDR 0x00000000
+#define ATR_PCIE_DEV_DMA_TRANSPARENT 1
+#define ATR_PCIE_DEV_DMA_SIZE 0
+#define ATR_PCIE_DEV_DMA_TABLE_NUM 0
+#define ATR_PCIE_DEV_DMA_TRSL_ADDR 0x00000000
+
+struct mtk_pci_irq_desc {
+ struct mtk_md_dev *mdev;
+ u32 msix_bits;
+ char name[MTK_IRQ_NAME_LEN];
+};
+
+struct mtk_pci_dev_cfg {
+ u32 flag;
+ u32 mhccif_rc_base_addr;
+ u32 istatus_host_ctrl_addr;
+ int irq_tbl[MTK_IRQ_SRC_MAX];
+ int (*atr_init)(struct mtk_md_dev *mdev);
+};
+
+struct mtk_pci_priv {
+ struct mtk_md_dev *mdev;
+ const struct mtk_pci_dev_cfg *cfg;
+ void __iomem *bar23_addr;
+ void __iomem *mac_reg_base;
+ void __iomem *ext_reg_base;
+ int irq_cnt;
+ int irq_type;
+ void *irq_cb_data[MTK_IRQ_CNT_MAX];
+
+ int (*irq_cb_list[MTK_IRQ_CNT_MAX])(int irq_id, void *data);
+ struct mtk_pci_irq_desc irq_desc[MTK_IRQ_CNT_MAX];
+ struct list_head mhccif_cb_list;
+ /* mhccif_lock: lock to protect mhccif_cb_list */
+ spinlock_t mhccif_lock;
+ struct work_struct mhccif_work;
+ int mhccif_irq_id;
+ struct pci_saved_state *saved_state;
+};
+
+struct mtk_atr_cfg {
+ u64 src_addr;
+ u64 trsl_addr;
+ u64 size;
+ u32 type; /* Port type */
+ u32 port; /* Port number */
+ u32 table; /* Table number (8 tables for each port) */
+ u32 trsl_id;
+ u32 trsl_param;
+ u32 transparent;
+};
+
+/* Read value from MD. For PCIe, it's BAR 0/1 MMIO read */
+u32 mtk_pci_mac_read32(struct mtk_pci_priv *priv, u64 addr);
+/* Write value to MD. For PCIe, it's BAR 0/1 MMIO write */
+void mtk_pci_mac_write32(struct mtk_pci_priv *priv, u64 addr, u32 val);
+/* Read value from MD. For PCIe, it's BAR 2/3 MMIO read */
+u32 mtk_pci_read32(struct mtk_md_dev *mdev, u64 addr);
+/* Write value to MD. For PCIe, it's BAR 2/3 MMIO write */
+void mtk_pci_write32(struct mtk_md_dev *mdev, u64 addr, u32 val);
+/* Device operations */
+u32 mtk_pci_get_dev_state(struct mtk_md_dev *mdev);
+void mtk_pci_ack_dev_state(struct mtk_md_dev *mdev, u32 state);
+u32 mtk_pci_get_dev_cfg(struct mtk_md_dev *mdev);
+/* IRQ Related operations */
+int mtk_pci_get_irq_id(struct mtk_md_dev *mdev, enum mtk_irq_src irq_src);
+int mtk_pci_get_virq_id(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id,
+ int (*irq_cb)(int irq_id, void *data), void *data);
+int mtk_pci_unregister_irq(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_mask_irq(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_unmask_irq(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_clear_irq(struct mtk_md_dev *mdev, int irq_id);
+/* External event related */
+int mtk_pci_register_ext_evt(struct mtk_md_dev *mdev, u32 chs,
+ int (*evt_cb)(u32 status, void *data), void *data);
+void mtk_pci_unregister_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+void mtk_pci_mask_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+void mtk_pci_unmask_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+void mtk_pci_clear_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+int mtk_pci_send_ext_evt(struct mtk_md_dev *mdev, u32 ch);
+int mtk_pci_fldr(struct mtk_md_dev *mdev);
+int mtk_pci_pldr(struct mtk_md_dev *mdev);
+int mtk_pci_reset(struct mtk_md_dev *mdev, enum mtk_reset_type type);
+bool mtk_pci_link_check(struct mtk_md_dev *mdev);
+int mtk_pci_setup_atr(struct mtk_md_dev *mdev, struct mtk_atr_cfg *cfg);
+void mtk_pci_atr_disable(struct mtk_pci_priv *priv);
+
+#endif /* __MTK_PCI_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c
new file mode 100644
index 000000000000..88b44142afb7
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_drv_m9xx.c
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+#include <linux/types.h>
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+
+static int mtk_pci_atr_init_m9xx(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_atr_cfg cfg;
+ int port, ret;
+
+ mtk_pci_atr_disable(priv);
+
+ /* Config ATR for RC to access device's register */
+ cfg.src_addr = pci_resource_start(pdev, MTK_BAR_2_3_IDX);
+ cfg.size = ATR_PCIE_REG_SIZE;
+ cfg.trsl_addr = ATR_PCIE_REG_TRSL_ADDR;
+ cfg.type = ATR_PCI2AXI;
+ cfg.port = ATR_PCIE_REG_PORT;
+ cfg.table = ATR_PCIE_REG_TABLE_NUM;
+ cfg.trsl_id = ATR_PCIE_REG_TRSL_PORT;
+ cfg.trsl_param = 0x0;
+ cfg.transparent = 0x0;
+ ret = mtk_pci_setup_atr(mdev, &cfg);
+ if (ret)
+ return ret;
+
+ /* Config ATR for EP to access RC's memory */
+ for (port = ATR_SRC_AXIS_0; port <= ATR_SRC_AXIS_3; port++) {
+ cfg.src_addr = ATR_PCIE_DEV_DMA_SRC_ADDR;
+ cfg.size = ATR_PCIE_DEV_DMA_SIZE;
+ cfg.trsl_addr = ATR_PCIE_DEV_DMA_TRSL_ADDR;
+ cfg.type = ATR_AXI2PCI;
+ cfg.port = port;
+ cfg.table = ATR_PCIE_DEV_DMA_TABLE_NUM;
+ cfg.trsl_id = ATR_DST_PCI_TRX;
+ cfg.trsl_param = 0x0;
+ /* Enable transparent translation */
+ cfg.transparent = ATR_PCIE_DEV_DMA_TRANSPARENT;
+ ret = mtk_pci_setup_atr(mdev, &cfg);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+const struct mtk_pci_dev_cfg mtk_dev_cfg_0900 = {
+ .flag = MTK_CFG_PM_SW_IRQ,
+ .mhccif_rc_base_addr = 0x1000A000,
+ .istatus_host_ctrl_addr = REG_ISTATUS_HOST_CTRL_NEW,
+ .irq_tbl = {
+ [MTK_IRQ_SRC_DPMAIF] = 24,
+ [MTK_IRQ_SRC_CLDMA0] = 27,
+ [MTK_IRQ_SRC_CLDMA1] = 26,
+ [MTK_IRQ_SRC_CLDMA2] = 25,
+ [MTK_IRQ_SRC_MHCCIF] = 28,
+ [MTK_IRQ_SRC_DPMAIF2] = 29,
+ [MTK_IRQ_SRC_CLDMA3] = 31,
+ [MTK_IRQ_SRC_PM_LOCK] = 0,
+ [MTK_IRQ_SRC_DPMAIF3] = 7,
+ [MTK_IRQ_SRC_DPMAIF6] = 10,
+ },
+ .atr_init = mtk_pci_atr_init_m9xx,
+};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
new file mode 100644
index 000000000000..d033dbf4b0af
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PCI_REG_H__
+#define __MTK_PCI_REG_H__
+
+#define REG_ISTATUS_HOST_CTRL_NEW 0x031C
+#define REG_PCIE_MISC_CTRL 0x0348
+#define REG_PCIE_CFG_MSIX 0x03EC
+#define REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB 0x0600
+#define REG_ATR_PCIE_WIN0_T0_SRC_ADDR_MSB 0x0604
+#define REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_LSB 0x0608
+#define REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_MSB 0x060C
+#define REG_ATR_PCIE_WIN0_T0_TRSL_PARAM 0x0610
+#define REG_PCIE_DEBUG_DUMMY_3 0x0D0C
+#define REG_PCIE_DEBUG_DUMMY_4 0x0D10
+#define REG_PCIE_DEBUG_DUMMY_7 0x0D1C
+#define REG_MSIX_ISTATUS_HOST_GRP0_0 0x0F00
+#define REG_IMASK_HOST_MSIX_SET_GRP0_0 0x3000
+#define REG_IMASK_HOST_MSIX_CLR_GRP0_0 0x3080
+#define REG_IMASK_HOST_MSIX_GRP0_0 0x3100
+
+/* mhccif registers */
+#define MHCCIF_RC2EP_SW_BSY 0x4
+#define MHCCIF_RC2EP_SW_TCHNUM 0xC
+#define MHCCIF_RC2EP_EVT_RESERVED_FOR_CLDMA0 BIT(4)
+#define MHCCIF_RC2EP_EVT_RESERVED_FOR_CLDMA1 BIT(5)
+#define MHCCIF_RC2EP_EVT_RESERVED_FOR_CLDMA3 BIT(6)
+#define MHCCIF_RC2EP_EVT_RESERVED_FOR_CLDMA2 BIT(7)
+#define MHCCIF_RC2EP_EVT_RESERVED_FOR_DPMAIF BIT(8)
+#define MHCCIF_RC2EP_EVT_PCIE_PM_SUSPEND_REQ BIT(9)
+#define MHCCIF_RC2EP_EVT_PCIE_PM_RESUME_REQ BIT(10)
+#define MHCCIF_RC2EP_EVT_PCIE_PM_SUSPEND_REQ_AP BIT(11)
+#define MHCCIF_RC2EP_EVT_PCIE_PM_RESUME_REQ_AP BIT(12)
+#define MHCCIF_RC2EP_EVT_DEVICE_RESET BIT(13)
+#define MHCCIF_RC2EP_EVT_DRM_DISABLE_AP BIT(14)
+#define MHCCIF_RC2EP_EVT_RESERVED_FOR_TEST BIT(31)
+
+#define MHCCIF_EP2RC_SW_INT_STS 0x10
+#define MHCCIF_EP2RC_SW_INT_ACK 0x14
+#define MHCCIF_EP2RC_SW_INT_EAP_MASK 0x20
+#define MHCCIF_EP2RC_SW_INT_EAP_MASK_SET 0x30
+#define MHCCIF_EP2RC_SW_INT_EAP_MASK_CLR 0x40
+#define MHCCIF_EP2RC_SPARE_REG_1 0x0104
+#define MHCCIF_EP2RC_SPARE_REG_5 0x0114
+#define MHCCIF_EP2RC_SPARE_REG_13 0x0134
+#define MHCCIF_EP2RC_SPARE_REG_14 0x0138
+#define MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC BIT(5)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_CLDMA0 BIT(6)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_CLDMA1 BIT(7)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_CLDMA3 BIT(8)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_CLDMA2 BIT(9)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_DPMAIF BIT(10)
+#define MHCCIF_EP2RC_EVT_PCIE_PM_SUSPEND_ACK BIT(11)
+#define MHCCIF_EP2RC_EVT_PCIE_PM_RESUME_ACK BIT(12)
+#define MHCCIF_EP2RC_EVT_PCIE_PM_SUSPEND_ACK_AP BIT(13)
+#define MHCCIF_EP2RC_EVT_PCIE_PM_RESUME_ACK_AP BIT(14)
+#define MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP BIT(15)
+#define MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD BIT(16)
+#define MHCCIF_EP2RC_EVT_SOFT_OFF_NOTIFY BIT(17)
+#define MHCCIF_EP2RC_EVT_MD_REBOOT BIT(19)
+#define MHCCIF_EP2RC_EVT_MD_POWEROFF BIT(20)
+#define MHCCIF_EP2RC_EVT_GNSS_ENABLE BIT(21)
+#define MHCCIF_EP2RC_EVT_GNSS_DISABLE BIT(22)
+#define MHCCIF_EP2RC_EVT_FRC_DONE_NOTIFY BIT(24)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_TEST1 BIT(30)
+#define MHCCIF_EP2RC_EVT_RESERVED_FOR_TEST2 BIT(31)
+
+#endif /* __MTK_PCI_REG_H__ */
--
2.34.1
^ permalink raw reply related
* [PATCH 03/11] net: wwan: t9xx: Add control DMA interface
From: Jack Wu via B4 Relay @ 2026-05-29 10:31 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
In-Reply-To: <20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com>
From: Jack Wu <jackbb_wu@compal.com>
Cross Layer Direct Memory Access(CLDMA) is the hardware
interface used by the control plane and designated to
translate data between the host and the device. It supports
8 hardware queues for the device AP and modem respectively.
CLDMA driver uses General Purpose Descriptor (GPD) to
describe transaction information that can be recognized by
CLDMA hardware. Once CLDMA hardware transaction is started,
it would fetch and parse GPD to transfer data correctly.
To facilitate the CLDMA transaction, a GPD ring for each
queue is used. Once the transaction is started, CLDMA
hardware will traverse the GPD ring to transfer data between
the host and the device until no GPD is available.
CLDMA TX flow:
Once a TX service receives the TX data from the port layer,
it uses APIs exported by the CLDMA driver to configure GPD
with the DMA address of TX data. After that, the service
triggers CLDMA to fetch the first available GPD to transfer
data.
CLDMA RX flow:
When there is RX data from the MD, CLDMA hardware asserts an
interrupt to notify the host to fetch data and dispatch it
to FSM (for handshake messages) or the port layer.
After CLDMA opening is finished, All RX GPDs are fulfilled
and ready to receive data from the device.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 3 +-
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 52 +-
drivers/net/wwan/t9xx/pcie/Makefile | 7 +-
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 1220 +++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 170 ++++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c | 373 +++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 177 ++++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c | 182 ++++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h | 103 ++
drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c | 23 +
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 38 +
drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 1 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 569 +++++++++++
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 84 ++
14 files changed, 2998 insertions(+), 4 deletions(-)
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
index ae5e1797b817..ca32827c1a20 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -8,7 +8,7 @@
#include "mtk_ctrl_plane.h"
-int mtk_ctrl_init(struct mtk_md_dev *mdev)
+int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops)
{
struct mtk_ctrl_blk *ctrl_blk;
@@ -18,6 +18,7 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev)
ctrl_blk->mdev = mdev;
mdev->ctrl_blk = ctrl_blk;
+ ctrl_blk->ops = ops;
return 0;
}
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index 8276be19b456..6d4be89680d6 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -11,12 +11,60 @@
#include "mtk_dev.h"
+enum mtk_trb_cmd_type {
+ TRB_CMD_MIN,
+ TRB_CMD_ENABLE,
+ TRB_CMD_TX,
+ TRB_CMD_DISABLE,
+ TRB_CMD_STOP,
+ TRB_CMD_RECOVER,
+ TRB_CMD_MAX,
+};
+
+enum mtk_hif_dev_ctrl_cmd {
+ HIF_CTRL_CMD_CHECK_TX_FULL,
+};
+
+struct trb_open_priv {
+ u8 log_rg_offset;
+ u32 tx_mtu;
+ u32 rx_mtu;
+ u32 tx_frag_size;
+ u32 rx_frag_size;
+ int (*rx_done)(struct sk_buff *skb, void *priv, bool force_recv);
+};
+
+struct trb {
+ u32 channel_id;
+ enum mtk_trb_cmd_type cmd;
+ int status;
+ struct kref kref;
+ void *priv;
+ int (*trb_complete)(struct sk_buff *skb);
+};
+
+union ctrl_hif_cmd_data {
+ u32 rx_ch;
+};
+
+struct mtk_ctrl_hif_ops {
+ int (*init)(struct mtk_md_dev *mdev);
+ int (*exit)(struct mtk_md_dev *mdev);
+ int (*submit_skb)(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send);
+ int (*send_cmd)(struct mtk_md_dev *mdev, int cmd, void *data);
+};
+
+struct mtk_ctrl_cfg;
+struct mtk_ctrl_trans;
+
struct mtk_ctrl_blk {
struct mtk_md_dev *mdev;
- struct mtk_ctrl_trans *trans;
+ struct mtk_ctrl_hif_ops *ops;
+ void *ctrl_hw_priv;
+ struct mtk_ctrl_cfg *cfg;
};
-int mtk_ctrl_init(struct mtk_md_dev *mdev);
+int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_ctrl_hif_ops *ops);
int mtk_ctrl_exit(struct mtk_md_dev *mdev);
#endif /* __MTK_CTRL_PLANE_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/Makefile b/drivers/net/wwan/t9xx/pcie/Makefile
index 7410d1796d27..5252f158b058 100644
--- a/drivers/net/wwan/t9xx/pcie/Makefile
+++ b/drivers/net/wwan/t9xx/pcie/Makefile
@@ -7,4 +7,9 @@ obj-$(CONFIG_MTK_T9XX_PCI) += mtk_t9xx_pcie.o
mtk_t9xx_pcie-y := \
mtk_pci_drv_m9xx.o \
- mtk_pci.o
+ mtk_cldma_drv_m9xx.o \
+ mtk_ctrl_cfg_m9xx.o \
+ mtk_pci.o \
+ mtk_trans_ctrl.o \
+ mtk_cldma.o \
+ mtk_cldma_drv.o
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
new file mode 100644
index 000000000000..48067a010890
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
@@ -0,0 +1,1220 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kdev_t.h>
+#include <linux/kernel.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/netdevice.h>
+#include <linux/sched.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+#include "mtk_pci.h"
+#include "mtk_cldma.h"
+#include "mtk_cldma_drv.h"
+#include "mtk_dev.h"
+
+#define cldma_drv_ops_null NULL
+#define DMA_POOL_NAME_LEN (64)
+#define WAIT_HWO_ROUND (10)
+#define WAIT_HWO_TIME (5)
+#define CLDMA_RETRY_DELAY_MS (100)
+#define NO_BUDGET (0)
+
+static const int mtk_cldma_hw_id_tbl[NR_CLDMA] = {
+ [CLDMA0] = CLDMA0_HW_ID,
+ [CLDMA1] = CLDMA1_HW_ID,
+ [CLDMA4] = CLDMA4_HW_ID,
+};
+
+static inline void mtk_cldma_clr_bd_dsc(struct cldma_drv_info *drv_info,
+ struct bd_dsc *bd_dsc_pool, int nr_bds)
+{
+ struct bd_dsc *bd_dsc;
+ int i;
+
+ for (i = 0; i < nr_bds; i++) {
+ bd_dsc = bd_dsc_pool + i;
+ dma_unmap_single(drv_info->mdev->dev, bd_dsc->data_dma_addr,
+ bd_dsc->data_len, DMA_TO_DEVICE);
+ bd_dsc->data_dma_addr = 0;
+ bd_dsc->data_len = 0;
+ if (bd_dsc->bd->tx_bd.bd_flags & CLDMA_BD_FLAG_EOL) {
+ bd_dsc->bd->tx_bd.bd_flags &= ~CLDMA_BD_FLAG_EOL;
+ break;
+ }
+ }
+}
+
+static void mtk_cldma_tx_done_work(struct work_struct *work)
+{
+ struct txq *txq = container_of(work, struct txq, tx_done_work);
+ struct cldma_drv_info *drv_info;
+ struct cldma_drv_ops *drv_ops;
+ struct mtk_ctrl_trans *trans;
+ struct mtk_md_dev *mdev;
+ struct tx_req *req;
+ unsigned int state;
+ int i, hif_id;
+ struct trb *trb;
+ u32 txqno;
+
+ drv_info = txq->drv_info;
+ hif_id = drv_info->hif_id;
+ txqno = txq->txqno;
+ mdev = drv_info->mdev;
+ drv_ops = drv_info->drv_ops;
+ trans = drv_info->cd->trans;
+
+again:
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + txq->free_idx;
+
+ rmb(); /* ensure HWO setup done before HWO read */
+
+ if (!req->data_vm_addr || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO))
+ break;
+
+ if (txq->nr_bds)
+ mtk_cldma_clr_bd_dsc(drv_info, req->bd_dsc_pool, txq->nr_bds);
+ else
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->data_len, DMA_TO_DEVICE);
+
+ trb = (struct trb *)req->skb->cb;
+ trb->status = 0;
+ trb->trb_complete(req->skb);
+
+ req->data_vm_addr = NULL;
+ req->data_dma_addr = 0;
+ req->data_len = 0;
+ req->skb = NULL;
+
+ txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
+ if (atomic_fetch_inc(&txq->req_budget) == NO_BUDGET)
+ wake_up(&trans->trb_srv[trans->srv_cfg[hif_id][txqno]]->trb_waitq);
+ }
+
+ state = drv_ops->cldma_check_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+ if (state) {
+ if (unlikely(state == LINK_ERROR_VAL))
+ goto out;
+
+ drv_ops->cldma_clr_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+
+ cond_resched();
+
+ goto again;
+ }
+
+out:
+ drv_ops->cldma_unmask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+}
+
+static void mtk_cldma_rx_skb_adjust(struct mtk_md_dev *mdev, struct rxq *rxq,
+ struct rx_req *req)
+{
+ struct bd_dsc *bd_dsc;
+ int i;
+
+ for (i = 0; i < rxq->nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ if (bd_dsc->data_dma_addr) {
+ dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr,
+ req->frag_size, DMA_FROM_DEVICE);
+ bd_dsc->data_dma_addr = 0;
+ }
+ bd_dsc->skb->len = 0;
+ skb_reset_tail_pointer(bd_dsc->skb);
+ skb_put(bd_dsc->skb,
+ le16_to_cpu(bd_dsc->bd->rx_bd.data_recv_len));
+ if (req->skb != bd_dsc->skb) {
+ req->skb->len += bd_dsc->skb->len;
+ req->skb->data_len += bd_dsc->skb->len;
+ }
+ bd_dsc->bd->rx_bd.data_recv_len = 0;
+ bd_dsc->skb = NULL;
+ }
+ if (!rxq->nr_bds) {
+ if (req->data_dma_addr) {
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->mtu, DMA_FROM_DEVICE);
+ req->data_dma_addr = 0;
+ }
+ req->skb->len = 0;
+ skb_reset_tail_pointer(req->skb);
+ skb_put(req->skb, le16_to_cpu(req->gpd->rx_gpd.data_recv_len));
+ }
+
+ req->gpd->rx_gpd.data_recv_len = 0;
+}
+
+static int mtk_cldma_reload_rx_skb(struct mtk_md_dev *mdev, struct rxq *rxq,
+ struct rx_req *req)
+{
+ struct sk_buff *tail = NULL;
+ struct bd_dsc *bd_dsc;
+ int nr_bds;
+ int i, err;
+
+ nr_bds = rxq->nr_bds;
+
+ for (i = 0; i < nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ bd_dsc->skb = __dev_alloc_skb(req->frag_size, GFP_KERNEL);
+ if (!bd_dsc->skb) {
+ dev_warn((mdev)->dev, "Failed to alloc SKB\n");
+ err = -ENOMEM;
+ goto err_free_skb;
+ }
+ bd_dsc->skb->next = NULL;
+ bd_dsc->data_dma_addr = dma_map_single(mdev->dev, bd_dsc->skb->data,
+ req->frag_size, DMA_FROM_DEVICE);
+ err = dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr);
+ if (unlikely(err)) {
+ dev_warn((mdev)->dev, "Failed to map SKB data\n");
+ err = -EFAULT;
+ goto err_free_skb;
+ }
+ bd_dsc->bd->rx_bd.data_buff_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->data_dma_addr) >> 32);
+ bd_dsc->bd->rx_bd.data_buff_ptr_l =
+ cpu_to_le32(bd_dsc->data_dma_addr);
+ if (tail) {
+ tail->next = bd_dsc->skb;
+ tail = bd_dsc->skb;
+ continue;
+ }
+ if (!req->skb) {
+ req->skb = bd_dsc->skb;
+ } else {
+ skb_shinfo(req->skb)->frag_list = bd_dsc->skb;
+ tail = bd_dsc->skb;
+ }
+ }
+ if (!nr_bds) {
+ req->skb = __dev_alloc_skb(req->mtu, GFP_KERNEL);
+ if (!req->skb) {
+ err = -ENOMEM;
+ goto err_free_skb;
+ }
+
+ req->data_dma_addr = dma_map_single(mdev->dev, req->skb->data,
+ req->mtu, DMA_FROM_DEVICE);
+ err = dma_mapping_error(mdev->dev, req->data_dma_addr);
+ if (unlikely(err)) {
+ dev_warn((mdev)->dev, "Failed to map SKB data\n");
+ err = -EFAULT;
+ goto err_free_skb;
+ }
+ req->gpd->rx_gpd.data_buff_ptr_h = cpu_to_le32((u64)req->data_dma_addr >> 32);
+ req->gpd->rx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
+ }
+ return 0;
+
+err_free_skb:
+ if (nr_bds) {
+ if (req->skb)
+ skb_shinfo(req->skb)->frag_list = NULL;
+ for (i = 0; i < nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ if (!bd_dsc->skb)
+ break;
+ if (!dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr))
+ dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr,
+ req->frag_size, DMA_FROM_DEVICE);
+ bd_dsc->data_dma_addr = 0;
+ bd_dsc->skb->next = NULL;
+ dev_kfree_skb_any(bd_dsc->skb);
+ }
+ } else {
+ req->data_dma_addr = 0;
+ if (req->skb)
+ dev_kfree_skb_any(req->skb);
+ }
+ req->skb = NULL;
+
+ return err;
+}
+
+static int mtk_cldma_check_rx_req(struct cldma_drv_info *drv_info, struct rxq *rxq)
+{
+ struct rx_req *req = rxq->req_pool + rxq->free_idx;
+ u64 curr_addr;
+ int i;
+
+ curr_addr = drv_info->drv_ops->cldma_get_rx_curr_addr(drv_info, rxq->rxqno);
+ if (unlikely(!curr_addr))
+ return -ENXIO;
+
+ if (req->gpd_dma_addr == curr_addr)
+ return -EAGAIN;
+ for (i = 0; i < WAIT_HWO_ROUND; i++) {
+ udelay(WAIT_HWO_TIME);
+ if (!(req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO))
+ break;
+ }
+ if (i == WAIT_HWO_ROUND) {
+ dev_err((drv_info->mdev)->dev, "Failed to check HWO=0\n");
+ return -EAGAIN;
+ }
+
+ return 0;
+}
+
+static bool mtk_cldma_rx_check_again(struct rxq *rxq)
+{
+ struct cldma_drv_info *drv_info;
+ struct cldma_drv_ops *drv_ops;
+ bool need_check_again = false;
+ struct mtk_md_dev *mdev;
+ int rxqno;
+ u32 state;
+
+ drv_info = rxq->drv_info;
+ drv_ops = drv_info->drv_ops;
+ mdev = drv_info->mdev;
+ rxqno = rxq->rxqno;
+
+ do {
+ state = drv_ops->cldma_check_intr_status(drv_info, DIR_RX,
+ rxqno, QUEUE_XFER_DONE);
+ if (state) {
+ if (unlikely(state == LINK_ERROR_VAL))
+ break;
+
+ drv_ops->cldma_clr_intr_status(drv_info, DIR_RX,
+ rxqno, QUEUE_XFER_DONE);
+ cond_resched();
+ return true;
+ }
+ } while (need_check_again);
+
+ return false;
+}
+
+static void mtk_cldma_rx_done_work(struct work_struct *work)
+{
+ struct rxq *rxq = container_of(work, struct rxq, rx_done_work);
+ struct rx_req *req = NULL, *pre_req = NULL;
+ struct cldma_drv_info *drv_info;
+ struct cldma_drv_ops *drv_ops;
+ struct mtk_md_dev *mdev;
+ int i, err, idx;
+
+ drv_info = rxq->drv_info;
+ mdev = drv_info->mdev;
+ drv_ops = drv_info->drv_ops;
+
+again:
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + rxq->free_idx;
+ if (!req->skb) {
+ dev_err((mdev)->dev,
+ "Failed to get valid req cldma%d rxq%d req%d\n",
+ drv_info->hw_id, rxq->rxqno, rxq->free_idx);
+ goto err_out;
+ }
+
+ if (req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)
+ break;
+
+ mtk_cldma_rx_skb_adjust(mdev, rxq, req);
+ do {
+ err = rxq->rx_done(req->skb, rxq->arg,
+ atomic_read(&rxq->need_exit) ? true : false);
+ if (err == -EAGAIN)
+ usleep_range(1000, 2000);
+ else
+ req->skb = NULL;
+ } while (err == -EAGAIN);
+
+ err = mtk_cldma_reload_rx_skb(mdev, rxq, req);
+ if (err)
+ goto err_out;
+
+ wmb(); /* ensure addr set done before HWO setup done */
+
+ idx = rxq->free_idx == 0 ? rxq->nr_gpds - 1 : rxq->free_idx - 1;
+ pre_req = rxq->req_pool + idx;
+ pre_req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
+ rxq->free_idx = (rxq->free_idx + 1) % rxq->nr_gpds;
+ }
+
+ err = mtk_cldma_check_rx_req(drv_info, rxq);
+ if (!err)
+ goto again;
+ else if (err == -ENXIO)
+ goto out;
+
+ if (!atomic_read(&rxq->need_exit))
+ drv_ops->cldma_resume_queue(drv_info, DIR_RX, rxq->rxqno);
+
+ if (mtk_cldma_rx_check_again(rxq))
+ goto again;
+
+out:
+ drv_ops->cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
+ drv_ops->cldma_clear_ip_busy(drv_info);
+err_out:
+ ;
+}
+
+static int mtk_cldma_alloc_tx_bd(struct cldma_drv_info *drv_info, struct txq *txq,
+ struct tx_req *req)
+{
+ struct bd_dsc *bd_dsc, *last_bd_dsc = NULL;
+ int i;
+
+ req->bd_dsc_pool = devm_kcalloc(drv_info->mdev->dev, txq->nr_bds,
+ sizeof(*bd_dsc), GFP_KERNEL);
+ if (!req->bd_dsc_pool)
+ return -ENOMEM;
+
+ for (i = 0; i < txq->nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ bd_dsc->bd = dma_pool_zalloc(drv_info->bd_dma_pool, GFP_KERNEL,
+ &bd_dsc->bd_dma_addr);
+ if (!bd_dsc->bd)
+ return -ENOMEM;
+ if (!last_bd_dsc) {
+ req->gpd->tx_gpd.data_buff_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32);
+ req->gpd->tx_gpd.data_buff_ptr_l =
+ cpu_to_le32(bd_dsc->bd_dma_addr);
+ } else {
+ last_bd_dsc->bd->tx_bd.next_bd_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32);
+ last_bd_dsc->bd->tx_bd.next_bd_ptr_l =
+ cpu_to_le32(bd_dsc->bd_dma_addr);
+ }
+ last_bd_dsc = bd_dsc;
+ }
+ return 0;
+}
+
+static struct txq *mtk_cldma_txq_alloc(struct cldma_drv_info *drv_info, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_ops *drv_ops;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct mtk_md_dev *mdev;
+ struct bd_dsc *bd_dsc;
+ struct tx_req *next;
+ struct tx_req *req;
+ u16 tx_frag_size;
+ struct txq *txq;
+ int i, j, err;
+
+ mdev = drv_info->mdev;
+ ctrl_blk = mdev->ctrl_blk;
+ trans = ctrl_blk->ctrl_hw_priv;
+ drv_ops = drv_info->drv_ops;
+
+ txq = devm_kzalloc(mdev->dev, sizeof(*txq), GFP_KERNEL);
+ if (!txq)
+ return NULL;
+
+ txq->que = radix_tree_lookup(&trans->queue_tbl, trb->channel_id & 0xFFFF);
+ txq->drv_info = drv_info;
+ txq->txqno = txq->que->txqno;
+ txq->nr_gpds = txq->que->tx_nr_gpds;
+ atomic_set(&txq->req_budget, txq->que->tx_nr_gpds);
+ txq->is_stopping = false;
+ tx_frag_size = txq->que->tx_frag_size;
+ if (txq->que->tx_mtu > tx_frag_size && tx_frag_size)
+ txq->nr_bds = (txq->que->tx_mtu + tx_frag_size - 1) / tx_frag_size;
+
+ txq->req_pool = devm_kcalloc(mdev->dev, txq->nr_gpds, sizeof(*req), GFP_KERNEL);
+ if (!txq->req_pool)
+ goto err_free_txq;
+
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + i;
+ req->mtu = txq->que->tx_mtu;
+ req->frag_size = tx_frag_size;
+ req->gpd = dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->gpd_dma_addr);
+ if (!req->gpd)
+ goto err_free_req;
+ if (txq->nr_bds) {
+ err = mtk_cldma_alloc_tx_bd(drv_info, txq, req);
+ if (err)
+ goto err_free_req;
+ req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_BDP;
+ }
+ }
+
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + i;
+ next = txq->req_pool + ((i + 1) % txq->nr_gpds);
+ req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_IOC;
+ req->gpd->tx_gpd.next_gpd_ptr_h = cpu_to_le32((u64)(next->gpd_dma_addr) >> 32);
+ req->gpd->tx_gpd.next_gpd_ptr_l = cpu_to_le32(next->gpd_dma_addr);
+ }
+
+ INIT_WORK(&txq->tx_done_work, mtk_cldma_tx_done_work);
+
+ drv_ops->cldma_stop_queue(drv_info, DIR_TX, txq->txqno);
+ txq->tx_started = false;
+ drv_ops->cldma_setup_start_addr(drv_info, DIR_TX, txq->txqno,
+ txq->req_pool[0].gpd_dma_addr);
+ drv_ops->cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_ERROR);
+ drv_ops->cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_XFER_DONE);
+
+ drv_info->txq[txq->txqno] = txq;
+ return txq;
+
+err_free_req:
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + i;
+ if (!req->gpd)
+ break;
+ if (req->bd_dsc_pool) {
+ for (j = 0; j < txq->nr_bds; j++) {
+ bd_dsc = req->bd_dsc_pool + j;
+ if (!bd_dsc->bd)
+ break;
+ dma_pool_free(drv_info->bd_dma_pool, bd_dsc->bd,
+ bd_dsc->bd_dma_addr);
+ }
+ devm_kfree(mdev->dev, req->bd_dsc_pool);
+ }
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+ }
+ devm_kfree(mdev->dev, txq->req_pool);
+err_free_txq:
+ devm_kfree(mdev->dev, txq);
+ return NULL;
+}
+
+static int mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno)
+{
+ struct cldma_drv_ops *drv_ops;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct mtk_md_dev *mdev;
+ struct bd_dsc *bd_dsc;
+ struct tx_req *req;
+ struct txq *txq;
+ struct trb *trb;
+ int irq_id;
+ int i, j;
+
+ mdev = drv_info->mdev;
+ ctrl_blk = mdev->ctrl_blk;
+ trans = ctrl_blk->ctrl_hw_priv;
+ drv_ops = drv_info->drv_ops;
+
+ txq = drv_info->txq[txqno];
+ drv_info->txq[txqno] = NULL;
+ /* stop HW tx transaction */
+ drv_ops->cldma_stop_queue(drv_info, DIR_TX, txqno);
+ txq->tx_started = false;
+
+ irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
+ synchronize_irq(irq_id);
+ /* flush on-going work */
+ flush_work(&txq->tx_done_work);
+ drv_ops->cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+ drv_ops->cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR);
+
+ /* free tx req resource */
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + txq->free_idx;
+ if (req->skb && req->data_len) {
+ if (!txq->nr_bds)
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->data_len, DMA_TO_DEVICE);
+ for (j = 0; j < txq->nr_bds; j++) {
+ bd_dsc = req->bd_dsc_pool + j;
+ dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr,
+ bd_dsc->data_len, DMA_TO_DEVICE);
+ }
+ trb = (struct trb *)req->skb->cb;
+ trb->status = -EPIPE;
+ trb->trb_complete(req->skb);
+ }
+ for (j = 0; j < txq->nr_bds; j++) {
+ bd_dsc = req->bd_dsc_pool + j;
+ dma_pool_free(drv_info->bd_dma_pool, bd_dsc->bd,
+ bd_dsc->bd_dma_addr);
+ }
+ if (req->bd_dsc_pool)
+ devm_kfree(mdev->dev, req->bd_dsc_pool);
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+ txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
+ }
+
+ devm_kfree(mdev->dev, txq->req_pool);
+ devm_kfree(mdev->dev, txq);
+
+ return 0;
+}
+
+static int mtk_cldma_alloc_rx_bd(struct cldma_drv_info *drv_info, struct rx_req *req,
+ int nr_bds)
+{
+ struct bd_dsc *bd_dsc, *last_bd_dsc = NULL;
+ struct sk_buff *tail = NULL;
+ struct mtk_md_dev *mdev;
+ u32 left_size;
+ int err;
+ int i;
+
+ mdev = drv_info->mdev;
+ left_size = req->mtu;
+
+ req->bd_dsc_pool = devm_kcalloc(mdev->dev, nr_bds,
+ sizeof(*bd_dsc), GFP_KERNEL);
+ if (!req->bd_dsc_pool)
+ return -ENOMEM;
+ for (i = 0; i < nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ bd_dsc->bd = dma_pool_zalloc(drv_info->bd_dma_pool, GFP_KERNEL,
+ &bd_dsc->bd_dma_addr);
+ if (!bd_dsc->bd)
+ return -ENOMEM;
+
+ bd_dsc->skb = __dev_alloc_skb(req->frag_size, GFP_KERNEL);
+ if (!bd_dsc->skb)
+ return -ENOMEM;
+ bd_dsc->skb->next = NULL;
+ bd_dsc->data_dma_addr =
+ dma_map_single(mdev->dev, bd_dsc->skb->data,
+ req->frag_size, DMA_FROM_DEVICE);
+ err = dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr);
+ if (unlikely(err))
+ return -ENOMEM;
+
+ bd_dsc->bd->rx_bd.data_buff_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->data_dma_addr) >> 32);
+ bd_dsc->bd->rx_bd.data_buff_ptr_l =
+ cpu_to_le32(bd_dsc->data_dma_addr);
+ bd_dsc->bd->rx_bd.data_allow_len =
+ cpu_to_le16(min(req->frag_size, left_size));
+ left_size -= min(req->frag_size, left_size);
+ if (!last_bd_dsc) {
+ req->gpd->rx_gpd.data_buff_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32);
+ req->gpd->rx_gpd.data_buff_ptr_l =
+ cpu_to_le32(bd_dsc->bd_dma_addr);
+ } else {
+ last_bd_dsc->bd->rx_bd.next_bd_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32);
+ last_bd_dsc->bd->rx_bd.next_bd_ptr_l =
+ cpu_to_le32(bd_dsc->bd_dma_addr);
+ }
+ last_bd_dsc = bd_dsc;
+ if (tail) {
+ tail->next = bd_dsc->skb;
+ tail = bd_dsc->skb;
+ continue;
+ }
+ if (!req->skb) {
+ req->skb = bd_dsc->skb;
+ } else {
+ skb_shinfo(req->skb)->frag_list = bd_dsc->skb;
+ tail = bd_dsc->skb;
+ }
+ }
+ last_bd_dsc->bd->rx_bd.bd_flags |= CLDMA_BD_FLAG_EOL;
+ return 0;
+}
+
+static void mtk_cldma_rxq_alloc_cancel(struct cldma_drv_info *drv_info, struct rx_req *req,
+ int nr_bds)
+{
+ struct mtk_md_dev *mdev;
+ struct bd_dsc *bd_dsc;
+ int i;
+
+ mdev = drv_info->mdev;
+
+ if (nr_bds) {
+ if (req->skb)
+ skb_shinfo(req->skb)->frag_list = NULL;
+ if (req->bd_dsc_pool) {
+ for (i = 0; i < nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ if (!bd_dsc->bd)
+ break;
+ if (bd_dsc->skb) {
+ if (!dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr))
+ dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr,
+ req->frag_size, DMA_FROM_DEVICE);
+ bd_dsc->data_dma_addr = 0;
+ bd_dsc->skb->next = NULL;
+ dev_kfree_skb_any(bd_dsc->skb);
+ }
+ dma_pool_free(drv_info->bd_dma_pool, bd_dsc->bd,
+ bd_dsc->bd_dma_addr);
+ }
+ devm_kfree(mdev->dev, req->bd_dsc_pool);
+ }
+ } else {
+ if (req->skb) {
+ if (!dma_mapping_error(mdev->dev, req->data_dma_addr))
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->mtu, DMA_FROM_DEVICE);
+ req->data_dma_addr = 0;
+ dev_kfree_skb_any(req->skb);
+ }
+ }
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+}
+
+static struct rxq *mtk_cldma_rxq_alloc(struct cldma_drv_info *drv_info, struct sk_buff *skb)
+{
+ struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_ops *drv_ops;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct mtk_md_dev *mdev;
+ struct rx_req *next;
+ struct rx_req *req;
+ u16 rx_frag_size;
+ struct rxq *rxq;
+ int err;
+ int i;
+
+ mdev = drv_info->mdev;
+ ctrl_blk = mdev->ctrl_blk;
+ trans = ctrl_blk->ctrl_hw_priv;
+ drv_ops = drv_info->drv_ops;
+
+ rxq = devm_kzalloc(mdev->dev, sizeof(*rxq), GFP_KERNEL);
+ if (!rxq)
+ return NULL;
+
+ rxq->que = radix_tree_lookup(&trans->queue_tbl, trb->channel_id & 0xFFFF);
+ if (rxq->que->rx_nr_gpds < MIN_GPD_NUM) {
+ dev_err((mdev)->dev,
+ "Failed to alloc cldma%d rxq%d due to gpd number < 2\n",
+ drv_info->hw_id, rxq->rxqno);
+ goto err_free_rxq;
+ }
+ rxq->drv_info = drv_info;
+ rxq->rxqno = rxq->que->rxqno;
+ rxq->nr_gpds = rxq->que->rx_nr_gpds;
+ rxq->arg = trb->priv;
+ rxq->rx_done = trb_open_priv->rx_done;
+ atomic_set(&rxq->need_exit, 0);
+ rx_frag_size = rxq->que->rx_frag_size;
+ if (rxq->que->rx_mtu > rx_frag_size && rx_frag_size)
+ rxq->nr_bds = (rxq->que->rx_mtu + rx_frag_size - 1) / rx_frag_size;
+
+ rxq->req_pool = devm_kcalloc(mdev->dev, rxq->nr_gpds, sizeof(*req), GFP_KERNEL);
+ if (!rxq->req_pool)
+ goto err_free_rxq;
+
+ /* setup rx request */
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + i;
+ req->mtu = rxq->que->rx_mtu;
+ req->frag_size = rx_frag_size;
+ req->gpd = dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->gpd_dma_addr);
+ if (!req->gpd)
+ goto err_free_req;
+ if (rxq->nr_bds) {
+ err = mtk_cldma_alloc_rx_bd(drv_info, req, rxq->nr_bds);
+ if (err)
+ goto err_free_req;
+ req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_BDP;
+ } else {
+ req->skb = __dev_alloc_skb(req->mtu, GFP_KERNEL);
+ if (!req->skb)
+ goto err_free_req;
+ req->data_dma_addr = dma_map_single(mdev->dev, req->skb->data,
+ req->mtu, DMA_FROM_DEVICE);
+ err = dma_mapping_error(mdev->dev, req->data_dma_addr);
+ if (unlikely(err))
+ goto err_free_req;
+ }
+ }
+
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + i;
+ next = rxq->req_pool + ((i + 1) % rxq->nr_gpds);
+ req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_IOC;
+ req->gpd->rx_gpd.data_allow_len = cpu_to_le16(req->mtu);
+ req->gpd->rx_gpd.next_gpd_ptr_h = cpu_to_le32((u64)(next->gpd_dma_addr) >> 32);
+ req->gpd->rx_gpd.next_gpd_ptr_l = cpu_to_le32(next->gpd_dma_addr);
+ if (!rxq->nr_bds) {
+ req->gpd->rx_gpd.data_buff_ptr_h =
+ cpu_to_le32((u64)(req->data_dma_addr) >> 32);
+ req->gpd->rx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
+ }
+ if (i != rxq->nr_gpds - 1)
+ req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
+ }
+
+ INIT_WORK(&rxq->rx_done_work, mtk_cldma_rx_done_work);
+
+ drv_info->rxq[rxq->rxqno] = rxq;
+ drv_ops->cldma_stop_queue(drv_info, DIR_RX, rxq->rxqno);
+ drv_ops->cldma_setup_start_addr(drv_info, DIR_RX,
+ rxq->rxqno, rxq->req_pool[0].gpd_dma_addr);
+ drv_ops->cldma_start_queue(drv_info, DIR_RX, rxq->rxqno);
+ drv_ops->cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_ERROR);
+ drv_ops->cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
+
+ return rxq;
+
+err_free_req:
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + i;
+ if (!req->gpd)
+ break;
+ mtk_cldma_rxq_alloc_cancel(drv_info, req, rxq->nr_bds);
+ }
+
+ devm_kfree(mdev->dev, rxq->req_pool);
+err_free_rxq:
+ devm_kfree(mdev->dev, rxq);
+ return NULL;
+}
+
+static int mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno)
+{
+ struct cldma_drv_ops *drv_ops;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct mtk_md_dev *mdev;
+ struct bd_dsc *bd_dsc;
+ struct rx_req *req;
+ struct rxq *rxq;
+ int irq_id;
+ int i, j;
+
+ mdev = drv_info->mdev;
+ ctrl_blk = mdev->ctrl_blk;
+ trans = ctrl_blk->ctrl_hw_priv;
+ drv_ops = drv_info->drv_ops;
+
+ rxq = drv_info->rxq[rxqno];
+ drv_info->rxq[rxqno] = NULL;
+
+ /* stop HW rx transaction */
+ atomic_set(&rxq->need_exit, 1);
+ drv_ops->cldma_stop_queue(drv_info, DIR_RX, rxqno);
+
+ irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
+ synchronize_irq(irq_id);
+ /* flush on-going work */
+ flush_work(&rxq->rx_done_work);
+ /* mask L2 RX interrupt again to avoid race condition causing use-after-free issue */
+ drv_ops->cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
+ drv_ops->cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR);
+
+ /* free rx req resource */
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + rxq->free_idx;
+ if (!(req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO) &&
+ le16_to_cpu(req->gpd->rx_gpd.data_recv_len)) {
+ mtk_cldma_rx_skb_adjust(mdev, rxq, req);
+ rxq->rx_done(req->skb, rxq->arg, true);
+ req->skb = NULL;
+ }
+ if (req->skb) {
+ if (rxq->nr_bds) {
+ skb_shinfo(req->skb)->frag_list = NULL;
+ } else {
+ if (req->data_dma_addr)
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->mtu, DMA_FROM_DEVICE);
+ dev_kfree_skb_any(req->skb);
+ }
+ }
+ for (j = 0; j < rxq->nr_bds; j++) {
+ bd_dsc = req->bd_dsc_pool + j;
+ if (bd_dsc->skb) {
+ if (bd_dsc->data_dma_addr)
+ dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr,
+ req->frag_size, DMA_FROM_DEVICE);
+ bd_dsc->skb->next = NULL;
+ dev_kfree_skb_any(bd_dsc->skb);
+ }
+ dma_pool_free(drv_info->bd_dma_pool,
+ bd_dsc->bd, bd_dsc->bd_dma_addr);
+ }
+ if (req->bd_dsc_pool)
+ devm_kfree(mdev->dev, req->bd_dsc_pool);
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+ rxq->free_idx = (rxq->free_idx + 1) % rxq->nr_gpds;
+ }
+
+ devm_kfree(mdev->dev, rxq->req_pool);
+ devm_kfree(mdev->dev, rxq);
+
+ return 0;
+}
+
+static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno)
+{
+ struct cldma_drv_ops *drv_ops;
+ struct txq *txq;
+ int ret = 0;
+ u32 val;
+
+ txq = drv_info->txq[qno];
+ drv_ops = drv_info->drv_ops;
+
+ val = drv_ops->cldma_get_tx_start_addr(drv_info, qno);
+ if (unlikely(!val)) {
+ drv_ops->cldma_drv_init(drv_info);
+ txq = drv_info->txq[qno];
+ drv_ops->cldma_setup_start_addr(drv_info, DIR_TX, qno,
+ txq->req_pool[txq->free_idx].gpd_dma_addr);
+ drv_ops->cldma_start_queue(drv_info, DIR_TX, qno);
+ txq->tx_started = true;
+ } else if (unlikely(val == LINK_ERROR_VAL)) {
+ ret = -EIO;
+ } else {
+ if (unlikely(!txq->tx_started)) {
+ drv_ops->cldma_start_queue(drv_info, DIR_TX, qno);
+ txq->tx_started = true;
+ } else {
+ drv_ops->cldma_resume_queue(drv_info, DIR_TX, qno);
+ }
+ }
+
+ return ret;
+}
+
+int mtk_cldma_init(struct mtk_ctrl_trans *trans)
+{
+ struct cldma_dev *cd;
+
+ cd = devm_kzalloc(trans->mdev->dev, sizeof(*cd), GFP_KERNEL);
+ if (!cd)
+ return -ENOMEM;
+
+ cd->trans = trans;
+ trans->dev = cd;
+
+ return 0;
+}
+
+int mtk_cldma_exit(struct mtk_ctrl_trans *trans)
+{
+ if (!trans->dev)
+ return 0;
+
+ devm_kfree(trans->mdev->dev, trans->dev);
+ trans->dev = NULL;
+
+ return 0;
+}
+
+static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb)
+{
+ struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct queue_info *que;
+ struct txq *txq;
+ struct rxq *rxq;
+ int err = 0;
+
+ que = radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (!drv_info) {
+ err = -EIO;
+ goto out;
+ }
+
+ if (que->tx_mtu == 0 || que->rx_mtu == 0) {
+ dev_err((cd->trans->mdev)->dev,
+ "Failed to enable cldma%d txq%d rxq%d due to wrong mtu\n",
+ drv_info->hw_id, que->txqno, que->rxqno);
+ err = -EINVAL;
+ goto out;
+ }
+
+ trb_open_priv->tx_mtu = que->tx_mtu;
+ trb_open_priv->rx_mtu = que->rx_mtu;
+ trb_open_priv->tx_frag_size = que->tx_frag_size;
+ trb_open_priv->rx_frag_size = que->rx_frag_size;
+
+ if (drv_info->txq[que->txqno] || drv_info->rxq[que->rxqno]) {
+ err = -EBUSY;
+ goto out;
+ }
+
+ txq = mtk_cldma_txq_alloc(drv_info, skb);
+ if (!txq) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ rxq = mtk_cldma_rxq_alloc(drv_info, skb);
+ if (!rxq) {
+ err = -ENOMEM;
+ mtk_cldma_txq_free(drv_info, txq->txqno);
+ goto out;
+ }
+
+out:
+ trb->status = err;
+ trb->trb_complete(skb);
+
+ return err;
+}
+
+static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct mtk_md_dev *mdev;
+ struct queue_info *que;
+ struct txq *txq;
+ int err = 0;
+
+ que = radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (unlikely(!drv_info))
+ return -EPIPE;
+ txq = drv_info->txq[que->txqno];
+ if (unlikely(!txq) || txq->is_stopping)
+ return -EPIPE;
+
+ mdev = drv_info->mdev;
+
+ err = mtk_cldma_start_xfer(drv_info, que->txqno);
+ if (unlikely(err))
+ dev_err((mdev)->dev, "Failed to trigger cldma tx\n");
+
+ return err;
+}
+
+static int mtk_cldma_close(struct cldma_dev *cd, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct queue_info *que;
+
+ que = radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (unlikely(!drv_info))
+ return -EPIPE;
+
+ if (drv_info->txq[que->txqno])
+ mtk_cldma_txq_free(drv_info, que->txqno);
+ if (drv_info->rxq[que->rxqno])
+ mtk_cldma_rxq_free(drv_info, que->rxqno);
+
+ trb->status = 0;
+ trb->trb_complete(skb);
+
+ return 0;
+}
+
+static int mtk_cldma_txbuf_set(struct cldma_drv_info *drv_info, struct sk_buff *skb,
+ struct tx_req *req, int nr_bds)
+{
+ struct sk_buff *curr_skb, *next_skb;
+ struct mtk_md_dev *mdev;
+ struct bd_dsc *bd_dsc;
+ int err;
+ int i;
+
+ mdev = drv_info->mdev;
+
+ if (nr_bds) {
+ bd_dsc = req->bd_dsc_pool;
+ curr_skb = skb;
+ for (i = 0; i < nr_bds && curr_skb; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ if (req->bd_dsc_pool == bd_dsc) {
+ bd_dsc->data_len = skb->len - skb->data_len;
+ next_skb = skb_shinfo(skb)->frag_list;
+ } else {
+ bd_dsc->data_len = curr_skb->len;
+ next_skb = curr_skb->next;
+ }
+ bd_dsc->data_dma_addr = dma_map_single(mdev->dev, curr_skb->data,
+ bd_dsc->data_len, DMA_TO_DEVICE);
+ err = dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr);
+ if (unlikely(err))
+ goto err_unmap_buffer;
+
+ bd_dsc->bd->tx_bd.data_buff_ptr_h =
+ cpu_to_le32((u64)(bd_dsc->data_dma_addr) >> 32);
+ bd_dsc->bd->tx_bd.data_buff_ptr_l = cpu_to_le32(bd_dsc->data_dma_addr);
+ bd_dsc->bd->tx_bd.data_buffer_len = cpu_to_le16(bd_dsc->data_len);
+ curr_skb = next_skb;
+ }
+ bd_dsc->bd->tx_bd.bd_flags = CLDMA_BD_FLAG_EOL;
+ } else {
+ req->data_dma_addr = dma_map_single(mdev->dev, skb->data,
+ skb->len, DMA_TO_DEVICE);
+ err = dma_mapping_error(mdev->dev, req->data_dma_addr);
+ if (unlikely(err)) {
+ req->data_dma_addr = 0;
+ goto err_exit;
+ }
+
+ req->gpd->tx_gpd.data_buff_ptr_h = cpu_to_le32((u64)(req->data_dma_addr) >> 32);
+ req->gpd->tx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
+ }
+
+ return 0;
+
+err_unmap_buffer:
+ for (i = 0; i < nr_bds; i++) {
+ bd_dsc = req->bd_dsc_pool + i;
+ if (dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr)) {
+ bd_dsc->data_dma_addr = 0;
+ break;
+ }
+ dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr,
+ bd_dsc->data_len, DMA_TO_DEVICE);
+ bd_dsc->data_dma_addr = 0;
+ }
+err_exit:
+ dev_err((mdev)->dev, "Failed to map dma! error:%d\n", err);
+ return -EAGAIN;
+}
+
+int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = dev;
+ struct queue_info *que;
+ struct tx_req *req;
+ struct txq *txq;
+ int ret;
+
+ que = radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (unlikely(!drv_info)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ txq = drv_info->txq[que->txqno];
+ if (unlikely(!txq)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (!atomic_read(&txq->req_budget)) {
+ ret = -EAGAIN;
+ goto out;
+ }
+
+ req = txq->req_pool + txq->wr_idx;
+ req->gpd->tx_gpd.debug_id = 0x01;
+ ret = mtk_cldma_txbuf_set(drv_info, skb, req, txq->nr_bds);
+ if (ret)
+ goto out;
+ req->gpd->tx_gpd.data_buff_len = cpu_to_le16(skb->len);
+
+ wmb(); /* ensure data msg set done before HWO setup */
+
+ req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
+
+ wmb(); /* ensure HWO setup done before req msg setup */
+
+ req->data_len = skb->len;
+ req->skb = skb;
+ req->data_vm_addr = skb->data;
+ txq->wr_idx = (txq->wr_idx + 1) % txq->nr_gpds;
+ atomic_dec(&txq->req_budget);
+
+out:
+ return ret;
+}
+
+int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno)
+{
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = dev;
+ struct txq *txq;
+
+ if (unlikely(hif_id >= NR_CLDMA || qno >= HW_QUE_NUM || !cd))
+ return -EINVAL;
+
+ drv_info = cd->cldma_drv_info[hif_id];
+ if (!drv_info)
+ return -EINVAL;
+ txq = drv_info->txq[qno];
+ if (!txq)
+ return -EINVAL;
+ return atomic_read(&txq->req_budget);
+}
+
+static int (*trb_act_tbl[TRB_CMD_MAX])(struct cldma_dev *cd, struct sk_buff *skb) = {
+ [TRB_CMD_ENABLE] = mtk_cldma_open,
+ [TRB_CMD_TX] = mtk_cldma_tx,
+ [TRB_CMD_DISABLE] = mtk_cldma_close,
+};
+
+int mtk_cldma_trb_process(void *dev, struct sk_buff *skb)
+{
+ struct cldma_dev *cd;
+ struct trb *trb;
+
+ if (!dev || !skb)
+ return -EINVAL;
+
+ cd = (struct cldma_dev *)dev;
+ trb = (struct trb *)skb->cb;
+
+ if (!(trb->cmd > TRB_CMD_MIN && trb->cmd < TRB_CMD_STOP))
+ return -EINVAL;
+
+ return trb_act_tbl[trb->cmd](cd, skb);
+}
+
+int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que)
+{
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = dev;
+ struct mtk_md_dev *mdev;
+ struct txq *txq;
+ struct rxq *rxq;
+
+ mdev = cd->trans->mdev;
+ drv_info = cd->cldma_drv_info[que->hif_id];
+
+ if (unlikely(!drv_info)) {
+ dev_err((mdev)->dev, "CLDMA%d has not been initialized\n",
+ mtk_cldma_hw_id_tbl[que->hif_id]);
+ return -EINVAL;
+ }
+
+ txq = drv_info->txq[que->txqno];
+ rxq = drv_info->rxq[que->rxqno];
+ if (unlikely(!txq || !rxq)) {
+ dev_err((mdev)->dev,
+ "CLDMA%d txq%d rxq%d has not been enabled\n",
+ mtk_cldma_hw_id_tbl[que->hif_id], que->txqno, que->rxqno);
+ return -EINVAL;
+ }
+
+ if (que->tx_mtu != txq->que->tx_mtu || que->rx_mtu != rxq->que->rx_mtu) {
+ dev_err((mdev)->dev,
+ "Channel:%08x tx_mtu:%08x rx_mtu:%08x do not match ch cfg\n",
+ que->tx_chl, que->tx_mtu, que->rx_mtu);
+ return -EINVAL;
+ }
+
+ return 0;
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
new file mode 100644
index 000000000000..246d28d3d798
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
@@ -0,0 +1,170 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_CLDMA_H__
+#define __MTK_CLDMA_H__
+
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/interrupt.h>
+#include <linux/list.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#include "mtk_ctrl_plane.h"
+#include "mtk_trans_ctrl.h"
+
+struct mtk_fsm_param;
+
+#define TXQ(N) (N)
+#define RXQ(N) (N)
+
+#define CLDMA_GPD_FLAG_HWO BIT(0)
+#define CLDMA_GPD_FLAG_BDP BIT(1)
+#define CLDMA_GPD_FLAG_BPS BIT(2)
+#define CLDMA_GPD_FLAG_IOC BIT(7)
+#define CLDMA_BD_FLAG_EOL BIT(0)
+
+union gpd {
+ struct {
+ u8 gpd_flags;
+ u8 non_used1;
+ __le16 data_allow_len;
+ __le32 next_gpd_ptr_h;
+ __le32 next_gpd_ptr_l;
+ __le32 data_buff_ptr_h;
+ __le32 data_buff_ptr_l;
+ __le16 data_recv_len;
+ u8 non_used2;
+ u8 debug_id;
+ } rx_gpd;
+
+ struct {
+ u8 gpd_flags;
+ u8 non_used1;
+ u8 non_used2;
+ u8 debug_id;
+ __le32 next_gpd_ptr_h;
+ __le32 next_gpd_ptr_l;
+ __le32 data_buff_ptr_h;
+ __le32 data_buff_ptr_l;
+ __le16 data_buff_len;
+ __le16 non_used3;
+ } tx_gpd;
+} __packed;
+
+union bd {
+ struct {
+ u8 bd_flags;
+ u8 non_used1;
+ __le16 data_allow_len;
+ __le32 next_bd_ptr_h;
+ __le32 next_bd_ptr_l;
+ __le32 data_buff_ptr_h;
+ __le32 data_buff_ptr_l;
+ __le16 data_recv_len;
+ __le16 non_used2;
+ } rx_bd;
+
+ struct {
+ u8 bd_flags;
+ u8 non_used1;
+ __le16 non_used2;
+ __le32 next_bd_ptr_h;
+ __le32 next_bd_ptr_l;
+ __le32 data_buff_ptr_h;
+ __le32 data_buff_ptr_l;
+ __le16 data_buffer_len;
+ u8 extension_len;
+ u8 non_used3;
+ } tx_bd;
+} __packed;
+
+struct bd_dsc {
+ union bd *bd;
+ struct sk_buff *skb;
+ dma_addr_t bd_dma_addr;
+ dma_addr_t data_dma_addr;
+ size_t data_len;
+};
+
+struct rx_req {
+ union gpd *gpd;
+ u32 mtu;
+ struct sk_buff *skb;
+ size_t data_len;
+ dma_addr_t gpd_dma_addr;
+ dma_addr_t data_dma_addr;
+ u32 frag_size;
+ struct bd_dsc *bd_dsc_pool;
+};
+
+struct rxq {
+ struct cldma_drv_info *drv_info;
+ u32 rxqno;
+ struct queue_info *que;
+ struct work_struct rx_done_work;
+ struct rx_req *req_pool;
+ u32 nr_gpds;
+ u32 free_idx;
+ unsigned short rx_done_cnt;
+ void *arg;
+ int (*rx_done)(struct sk_buff *skb, void *priv, bool force_recv);
+ u32 nr_bds;
+ atomic_t need_exit;
+};
+
+struct tx_req {
+ union gpd *gpd;
+ u32 mtu;
+ void *data_vm_addr;
+ size_t data_len;
+ dma_addr_t data_dma_addr;
+ dma_addr_t gpd_dma_addr;
+ struct sk_buff *skb;
+ int (*trb_complete)(struct sk_buff *skb);
+ u32 frag_size;
+ struct bd_dsc *bd_dsc_pool;
+};
+
+struct txq {
+ struct cldma_drv_info *drv_info;
+ u32 txqno;
+ struct queue_info *que;
+ struct work_struct tx_done_work;
+ struct tx_req *req_pool;
+ u32 nr_gpds;
+ atomic_t req_budget;
+ u32 wr_idx;
+ u32 free_idx;
+ bool tx_started;
+ bool is_stopping;
+ unsigned short tx_done_cnt;
+ u32 nr_bds;
+};
+
+struct cldma_dev {
+ struct cldma_drv_info *cldma_drv_info[NR_CLDMA];
+ struct mtk_ctrl_trans *trans;
+};
+
+struct cldma_drv_info_desc {
+ u32 hw_ver;
+ struct cldma_drv_ops *drv_ops;
+ struct cldma_hw_regs *hw_regs;
+};
+
+int mtk_cldma_init(struct mtk_ctrl_trans *trans);
+int mtk_cldma_exit(struct mtk_ctrl_trans *trans);
+int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb);
+int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno);
+int mtk_cldma_trb_process(void *dev, struct sk_buff *skb);
+void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_ctrl_trans *trans);
+int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que);
+
+#define drv_ops_name(NAME) cldma_drv_ops_##NAME
+#define cldma_regs_name(NAME) mtk_cldma_regs_##NAME
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c
new file mode 100644
index 000000000000..d5eb2ab9a425
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c
@@ -0,0 +1,373 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kdev_t.h>
+#include <linux/kernel.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/netdevice.h>
+#include <linux/sched.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+
+#include "mtk_cldma_drv.h"
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+
+#define WAIT_QUEUE_STOP (70)
+
+void mtk_cldma_drv_init(struct cldma_drv_info *drv_info)
+{
+ struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ int base;
+ u32 val;
+
+ mdev = drv_info->mdev;
+ base = drv_info->base_addr;
+ hw_regs = drv_info->hw_regs;
+
+ /* set CLDMA to 64 bit mode GPD */
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_ul_cfg);
+ val = (val & (~(0x7 << 5))) | ((0x4) << 5);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_cfg, val);
+
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_so_cfg);
+ val = (val & (~(0x7 << 10))) | ((0x4) << 10) | (1 << 2);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_cfg, val);
+
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_rx_work_to_reg_mask_set, ALLQ);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_set,
+ ALLQ << 16);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_clr,
+ ALLQ << 24);
+
+ /* enable interrupt to PCIe */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0);
+
+ /* disable illegal memory check */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_dummy_0, 1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_dummy_0, 1);
+}
+
+void mtk_cldma_setup_start_addr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, dma_addr_t addr)
+{
+ struct cldma_hw_regs *hw_regs;
+ unsigned int addr_l;
+ unsigned int addr_h;
+ int base;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX) {
+ addr_l = base + hw_regs->reg_cldma_ul_start_addrl_0 + qno * HW_QUEUE_NUM;
+ addr_h = base + hw_regs->reg_cldma_ul_start_addrh_0 + qno * HW_QUEUE_NUM;
+ } else {
+ addr_l = base + hw_regs->reg_cldma_so_start_addrl_0 + qno * HW_QUEUE_NUM;
+ addr_h = base + hw_regs->reg_cldma_so_start_addrh_0 + qno * HW_QUEUE_NUM;
+ }
+
+ mtk_pci_write32(drv_info->mdev, addr_l, (u32)addr);
+ mtk_pci_write32(drv_info->mdev, addr_h, (u32)((u64)addr >> 32));
+}
+
+void mtk_cldma_mask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2timsr0;
+ else
+ addr = base + hw_regs->reg_cldma_l2rimsr0;
+
+ if (qno == ALLQ)
+ val = qno << type;
+ else
+ val = BIT(qno) << type;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+void mtk_cldma_unmask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2timcr0;
+ else
+ addr = base + hw_regs->reg_cldma_l2rimcr0;
+
+ if (qno == ALLQ)
+ val = qno << type;
+ else
+ val = BIT(qno) << type;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+void mtk_cldma_clr_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+ mdev = drv_info->mdev;
+
+ if (type == QUEUE_ERROR) {
+ if (dir == DIR_TX) {
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar0);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar0, val);
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar1, val);
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar2);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar2, val);
+ } else {
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3risar0);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3risar0, val);
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3risar1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3risar1, val);
+ }
+ }
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2tisar0;
+ else
+ addr = base + hw_regs->reg_cldma_l2risar0;
+
+ if (qno == ALLQ)
+ val = qno << type;
+ else
+ val = BIT(qno) << type;
+
+ mtk_pci_write32(mdev, addr, val);
+ val = mtk_pci_read32(mdev, addr);
+}
+
+u32 mtk_cldma_check_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+ u32 sta;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2tisar0;
+ else
+ addr = base + hw_regs->reg_cldma_l2risar0;
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+ if (val == LINK_ERROR_VAL)
+ sta = val;
+ else if (qno == ALLQ)
+ sta = (val >> type) & 0xFF;
+ else
+ sta = (val >> type) & BIT(qno);
+
+ return sta;
+}
+
+void mtk_cldma_start_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ struct cldma_hw_regs *hw_regs;
+ u32 val = BIT(qno);
+ int base;
+ u32 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_start_cmd;
+ else
+ addr = base + hw_regs->reg_cldma_so_start_cmd;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+void mtk_cldma_resume_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ struct cldma_hw_regs *hw_regs;
+ u32 val = BIT(qno);
+ int base;
+ u32 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_resume_cmd;
+ else
+ addr = base + hw_regs->reg_cldma_so_resume_cmd;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+u32 mtk_cldma_queue_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_status;
+ else
+ addr = base + hw_regs->reg_cldma_so_status;
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+
+ if (qno == ALLQ || val == LINK_ERROR_VAL)
+ return val;
+
+ return val & BIT(qno);
+}
+
+u32 mtk_cldma_stop_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ u32 val = (qno == ALLQ) ? qno : BIT(qno);
+ struct cldma_hw_regs *hw_regs;
+ unsigned int active;
+ int cnt = 0;
+ int base;
+ u32 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_stop_cmd;
+ else
+ addr = base + hw_regs->reg_cldma_so_stop_cmd;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+
+ do {
+ active = drv_info->drv_ops->cldma_queue_status(drv_info, dir, qno);
+ if (active == LINK_ERROR_VAL || !active)
+ break;
+ usleep_range(WAIT_QUEUE_STOP, 2 * WAIT_QUEUE_STOP);
+ } while (++cnt < 10);
+
+ return active;
+}
+
+void mtk_cldma_clear_ip_busy(struct cldma_drv_info *drv_info)
+{
+ mtk_pci_write32(drv_info->mdev, drv_info->base_addr +
+ drv_info->hw_regs->reg_cldma_ip_busy, 0x01);
+}
+
+void mtk_cldma_get_intr_status(struct cldma_drv_info *drv_info, u32 *tx_sta, u32 *rx_sta)
+{
+ struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ u32 tx_mask, rx_mask;
+ int base;
+
+ mdev = drv_info->mdev;
+ base = drv_info->base_addr;
+ hw_regs = drv_info->hw_regs;
+
+ *tx_sta = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2tisar0);
+ tx_mask = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2timr0);
+ *rx_sta = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2risar0);
+ rx_mask = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2rimr0);
+
+ *tx_sta = (*tx_sta) & (~tx_mask);
+ *rx_sta = (*rx_sta) & (~rx_mask);
+
+ if (*tx_sta) {
+ /* TX XFER_DONE and QUEUE_ERROR mask */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2timsr0, *tx_sta);
+ /* TX XFER_DONE clear */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2tisar0,
+ (*tx_sta) & (0xFF << QUEUE_XFER_DONE));
+ }
+
+ if (*rx_sta) {
+ /* RX XFER_DONE and QUEUE_ERROR mask */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2rimsr0, *rx_sta);
+ /* RX XFER_DONE clear */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2risar0,
+ (*rx_sta) & (0xFF << QUEUE_XFER_DONE));
+ }
+}
+
+u32 mtk_cldma_get_tx_start_addr(struct cldma_drv_info *drv_info, u32 qno)
+{
+ u32 addr, val;
+
+ addr = drv_info->base_addr + drv_info->hw_regs->reg_cldma_ul_start_addrl_0 +
+ qno * HW_QUEUE_NUM;
+ val = mtk_pci_read32(drv_info->mdev, addr);
+
+ return val;
+}
+
+u64 mtk_cldma_get_rx_curr_addr(struct cldma_drv_info *drv_info, u32 qno)
+{
+ struct cldma_hw_regs *hw_regs;
+ u32 curr_addr_h, curr_addr_l;
+ struct mtk_md_dev *mdev;
+ u64 curr_addr;
+ int base;
+ u64 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+ mdev = drv_info->mdev;
+
+ addr = base + hw_regs->reg_cldma_so_current_addrh_0 +
+ (u64)qno * HW_QUEUE_NUM;
+ curr_addr_h = mtk_pci_read32(mdev, addr);
+ addr = base + hw_regs->reg_cldma_so_current_addrl_0 +
+ (u64)qno * HW_QUEUE_NUM;
+ curr_addr_l = mtk_pci_read32(mdev, addr);
+ curr_addr = ((u64)curr_addr_h << 32) | curr_addr_l;
+ if (curr_addr_h == LINK_ERROR_VAL && curr_addr_l == LINK_ERROR_VAL)
+ curr_addr = 0;
+ return curr_addr;
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
new file mode 100644
index 000000000000..8763c23abf54
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
@@ -0,0 +1,177 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#ifndef __MTK_CLDMA_DRV_H__
+#define __MTK_CLDMA_DRV_H__
+
+#define HW_QUEUE_NUM (8)
+#define ALLQ (0xFF)
+#define LINK_ERROR_VAL (0xFFFFFFFF)
+#define CLDMA0_HW_ID (0)
+#define CLDMA1_HW_ID (1)
+#define CLDMA4_HW_ID (4)
+
+struct cldma_hw_regs {
+ u8 cldma_rx_skb_pool_max_size;
+ u8 cldma_rx_skb_reload_threshold;
+ u8 tq_err_int_offset;
+ u8 tq_active_start_err_int_offset;
+ u8 rq_err_int_offset;
+ u8 rq_active_start_err_int_offset;
+ u16 reg_cldma_so_cfg;
+ u16 reg_cldma_so_start_addrl_0;
+ u16 reg_cldma_so_start_addrh_0;
+ u16 reg_cldma_so_current_addrl_0;
+ u16 reg_cldma_so_current_addrh_0;
+ u16 reg_cldma_so_status;
+ u16 reg_cldma_debug_id_en;
+ u16 reg_cldma_so_last_update_addrl_0;
+ u16 reg_cldma_so_last_update_addrh_0;
+ u16 reg_cldma_l2rimr0;
+ u16 reg_cldma_l2rimr1;
+ u16 reg_cldma_l2rimcr0;
+ u16 reg_cldma_l2rimcr1;
+ u16 reg_cldma_l2rimsr0;
+ u16 reg_cldma_l2rimsr1;
+ u16 reg_cldma_int_mask;
+ u16 reg_cldma4_int_mask;
+ u16 reg_cldma_slp_mem_ctl;
+ u16 reg_cldma_busy_mask;
+ u16 reg_cldma_ip_busy_to_pcie_mask;
+ u16 reg_cldma_ip_busy_to_pcie_mask_set;
+ u16 reg_cldma_ip_busy_to_pcie_mask_clr;
+ u16 reg_cldma_ip_busy_to_ap_mask;
+ u16 reg_cldma_ip_busy_to_ap_mask_set;
+ u16 reg_cldma_ip_busy_to_ap_mask_clr;
+ u16 reg_cldma_ip_busy_to_md_mask_set;
+ u16 reg_cldma_rx_work_to_reg_mask_set;
+ u16 reg_infra_rst4_set;
+ u16 reg_infra_rst4_clr;
+ u16 reg_infra_rst2_set;
+ u16 reg_infra_rst2_clr;
+ u16 reg_infra_rst0_set;
+ u16 reg_infra_rst0_clr;
+ u32 tq_err_int_bitmask;
+ u32 tq_active_start_err_int_bitmask;
+ u32 rq_err_int_bitmask;
+ u32 cldma0_base_addr;
+ u32 cldma1_base_addr;
+ u32 cldma4_base_addr;
+ u32 rq_active_start_err_int_bitmask;
+ u32 reg_cldma_ul_start_addrl_0;
+ u32 reg_cldma_ul_start_addrh_0;
+ u32 reg_cldma_ul_current_addrl_0;
+ u32 reg_cldma_ul_current_addrh_0;
+ u32 reg_cldma_ul_status;
+ u32 reg_cldma_ul_start_cmd;
+ u32 reg_cldma_ul_resume_cmd;
+ u32 reg_cldma_ul_stop_cmd;
+ u32 reg_cldma_ul_error;
+ u32 reg_cldma_ul_cfg;
+ u32 reg_cldma_ul_dummy_0;
+ u32 reg_cldma_so_error;
+ u32 reg_cldma_so_start_cmd;
+ u32 reg_cldma_so_resume_cmd;
+ u32 reg_cldma_so_stop_cmd;
+ u32 reg_cldma_so_dummy_0;
+ u32 reg_cldma_l2tisar0;
+ u32 reg_cldma_l2tisar1;
+ u32 reg_cldma_l2timr0;
+ u32 reg_cldma_l2timr1;
+ u32 reg_cldma_l2timcr0;
+ u32 reg_cldma_l2timcr1;
+ u32 reg_cldma_l2timsr0;
+ u32 reg_cldma_l2timsr1;
+ u32 reg_cldma_l2risar0;
+ u32 reg_cldma_l2risar1;
+ u32 reg_cldma_l3tisar0;
+ u32 reg_cldma_l3tisar1;
+ u32 reg_cldma_l3tisar2;
+ u32 reg_cldma_l3risar0;
+ u32 reg_cldma_l3risar1;
+ u32 reg_cldma_ip_busy;
+};
+
+enum mtk_ip_busy_src {
+ IP_BUSY_TXDONE = 0,
+ IP_BUSY_TXEMPTY = 8,
+ IP_BUSY_TXACTIVE = 16,
+ IP_BUSY_RXDONE = 24
+};
+
+enum mtk_intr_type {
+ QUEUE_XFER_DONE = 0,
+ QUEUE_EMPTY = 8,
+ QUEUE_ERROR = 16,
+ QUEUE_ACTIVE_START = 24,
+ INVALID_TYPE
+};
+
+enum mtk_tx_rx {
+ DIR_TX,
+ DIR_RX,
+ DIR_MAX
+};
+
+struct cldma_drv_info {
+ int hif_id;
+ int hw_id;
+ int base_addr;
+ int pci_ext_irq_id;
+ struct mtk_md_dev *mdev;
+ struct cldma_dev *cd;
+ struct txq *txq[HW_QUEUE_NUM];
+ struct rxq *rxq[HW_QUEUE_NUM];
+ struct dma_pool *gpd_dma_pool;
+ struct dma_pool *bd_dma_pool;
+ struct workqueue_struct *wq;
+ struct cldma_hw_regs *hw_regs;
+ struct cldma_drv_ops *drv_ops;
+};
+
+struct cldma_drv_ops {
+ void (*cldma_drv_init)(struct cldma_drv_info *drv_info);
+ void (*cldma_drv_reset)(struct cldma_drv_info *drv_info);
+ void (*cldma_setup_start_addr)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, dma_addr_t addr);
+ void (*cldma_mask_intr)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+ void (*cldma_unmask_intr)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+ void (*cldma_clr_intr_status)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+ u32 (*cldma_check_intr_status)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+ void (*cldma_start_queue)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+ void (*cldma_resume_queue)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+ u32 (*cldma_queue_status)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+ u32 (*cldma_stop_queue)(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+ void (*cldma_clear_ip_busy)(struct cldma_drv_info *drv_info);
+ void (*cldma_get_intr_status)(struct cldma_drv_info *drv_info, u32 *tx_sta, u32 *rx_sta);
+ u32 (*cldma_get_tx_start_addr)(struct cldma_drv_info *drv_info, u32 qno);
+ u64 (*cldma_get_rx_curr_addr)(struct cldma_drv_info *drv_info, u32 qno);
+};
+
+void mtk_cldma_drv_init(struct cldma_drv_info *drv_info);
+void mtk_cldma_setup_start_addr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, dma_addr_t addr);
+void mtk_cldma_mask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+void mtk_cldma_unmask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+void mtk_cldma_clr_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+u32 mtk_cldma_check_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+void mtk_cldma_start_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+void mtk_cldma_resume_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+u32 mtk_cldma_queue_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+u32 mtk_cldma_stop_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+void mtk_cldma_clear_ip_busy(struct cldma_drv_info *drv_info);
+void mtk_cldma_get_intr_status(struct cldma_drv_info *drv_info, u32 *tx_sta, u32 *rx_sta);
+u32 mtk_cldma_get_tx_start_addr(struct cldma_drv_info *drv_info, u32 qno);
+u64 mtk_cldma_get_rx_curr_addr(struct cldma_drv_info *drv_info, u32 qno);
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c
new file mode 100644
index 000000000000..240a9f58f658
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.c
@@ -0,0 +1,182 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kdev_t.h>
+#include <linux/kernel.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/netdevice.h>
+#include <linux/sched.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+
+#include "mtk_cldma_drv.h"
+#include "mtk_cldma_drv_m9xx.h"
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+#include "mtk_trans_ctrl.h"
+
+struct cldma_hw_regs mtk_cldma_regs_m9xx = {
+ .cldma0_base_addr = CLDMA0_BASE_ADDR,
+ .cldma1_base_addr = CLDMA1_BASE_ADDR,
+ .cldma4_base_addr = CLDMA4_BASE_ADDR,
+ .cldma_rx_skb_pool_max_size = CLDMA_RX_SKB_POOL_MAX_SIZE,
+ .cldma_rx_skb_reload_threshold = CLDMA_RX_SKB_RELOAD_THRESHOLD,
+ .tq_err_int_offset = TQ_ERR_INT_OFFSET,
+ .tq_err_int_bitmask = TQ_ERR_INT_BITMASK,
+ .tq_active_start_err_int_offset = TQ_ACTIVE_START_ERR_INT_OFFSET,
+ .tq_active_start_err_int_bitmask = TQ_ACTIVE_START_ERR_INT_BITMASK,
+ .rq_err_int_offset = RQ_ERR_INT_OFFSET,
+ .rq_err_int_bitmask = RQ_ERR_INT_BITMASK,
+ .rq_active_start_err_int_offset = RQ_ACTIVE_START_ERR_INT_OFFSET,
+ .rq_active_start_err_int_bitmask = RQ_ACTIVE_START_ERR_INT_BITMASK,
+ .reg_cldma_ul_start_addrl_0 = REG_CLDMA_UL_START_ADDRL_0,
+ .reg_cldma_ul_start_addrh_0 = REG_CLDMA_UL_START_ADDRH_0,
+ .reg_cldma_ul_current_addrl_0 = REG_CLDMA_UL_CURRENT_ADDRL_0,
+ .reg_cldma_ul_current_addrh_0 = REG_CLDMA_UL_CURRENT_ADDRH_0,
+ .reg_cldma_ul_status = REG_CLDMA_UL_STATUS,
+ .reg_cldma_ul_start_cmd = REG_CLDMA_UL_START_CMD,
+ .reg_cldma_ul_resume_cmd = REG_CLDMA_UL_RESUME_CMD,
+ .reg_cldma_ul_stop_cmd = REG_CLDMA_UL_STOP_CMD,
+ .reg_cldma_ul_error = REG_CLDMA_UL_ERROR,
+ .reg_cldma_ul_cfg = REG_CLDMA_UL_CFG,
+ .reg_cldma_ul_dummy_0 = REG_CLDMA_UL_DUMMY_0,
+ .reg_cldma_so_error = REG_CLDMA_SO_ERROR,
+ .reg_cldma_so_start_cmd = REG_CLDMA_SO_START_CMD,
+ .reg_cldma_so_resume_cmd = REG_CLDMA_SO_RESUME_CMD,
+ .reg_cldma_so_stop_cmd = REG_CLDMA_SO_STOP_CMD,
+ .reg_cldma_so_dummy_0 = REG_CLDMA_SO_DUMMY_0,
+ .reg_cldma_so_cfg = REG_CLDMA_SO_CFG,
+ .reg_cldma_so_start_addrl_0 = REG_CLDMA_SO_START_ADDRL_0,
+ .reg_cldma_so_start_addrh_0 = REG_CLDMA_SO_START_ADDRH_0,
+ .reg_cldma_so_current_addrl_0 = REG_CLDMA_SO_CUR_ADDRL_0,
+ .reg_cldma_so_current_addrh_0 = REG_CLDMA_SO_CUR_ADDRH_0,
+ .reg_cldma_so_status = REG_CLDMA_SO_STATUS,
+ .reg_cldma_debug_id_en = REG_CLDMA_DEBUG_ID_EN,
+ .reg_cldma_so_last_update_addrl_0 = REG_CLDMA_SO_LAST_UPDATE_ADDRL_0,
+ .reg_cldma_so_last_update_addrh_0 = REG_CLDMA_SO_LAST_UPDATE_ADDRH_0,
+ .reg_cldma_l2tisar0 = REG_CLDMA_L2TISAR0,
+ .reg_cldma_l2tisar1 = REG_CLDMA_L2TISAR1,
+ .reg_cldma_l2timr0 = REG_CLDMA_L2TIMR0,
+ .reg_cldma_l2timr1 = REG_CLDMA_L2TIMR1,
+ .reg_cldma_l2timcr0 = REG_CLDMA_L2TIMCR0,
+ .reg_cldma_l2timcr1 = REG_CLDMA_L2TIMCR1,
+ .reg_cldma_l2timsr0 = REG_CLDMA_L2TIMSR0,
+ .reg_cldma_l2timsr1 = REG_CLDMA_L2TIMSR1,
+ .reg_cldma_l3tisar0 = REG_CLDMA_L3TISAR0,
+ .reg_cldma_l3tisar1 = REG_CLDMA_L3TISAR1,
+ .reg_cldma_l3tisar2 = REG_CLDMA_L3TISAR2,
+ .reg_cldma_l2risar0 = REG_CLDMA_L2RISAR0,
+ .reg_cldma_l2risar1 = REG_CLDMA_L2RISAR1,
+ .reg_cldma_l2rimr0 = REG_CLDMA_L2RIMR0,
+ .reg_cldma_l2rimr1 = REG_CLDMA_L2RIMR1,
+ .reg_cldma_l2rimcr0 = REG_CLDMA_L2RIMCR0,
+ .reg_cldma_l2rimcr1 = REG_CLDMA_L2RIMCR1,
+ .reg_cldma_l2rimsr0 = REG_CLDMA_L2RIMSR0,
+ .reg_cldma_l2rimsr1 = REG_CLDMA_L2RIMSR1,
+ .reg_cldma_l3risar0 = REG_CLDMA_L3RISAR0,
+ .reg_cldma_l3risar1 = REG_CLDMA_L3RISAR1,
+ .reg_cldma_ip_busy = REG_CLDMA_IP_BUSY,
+ .reg_cldma_int_mask = REG_CLDMA_INT_EAP_USIP_MASK,
+ .reg_cldma4_int_mask = REG_CLDMA_INT_WF_MASK,
+ .reg_cldma_ip_busy_to_pcie_mask = REG_CLDMA_IP_BUSY_TO_PCIE_MASK,
+ .reg_cldma_ip_busy_to_pcie_mask_set = REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SET,
+ .reg_cldma_ip_busy_to_pcie_mask_clr = REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CLR,
+ .reg_cldma_ip_busy_to_ap_mask = REG_CLDMA_IP_BUSY_TO_AP_MASK,
+ .reg_cldma_ip_busy_to_ap_mask_set = REG_CLDMA_IP_BUSY_TO_AP_MASK_SET,
+ .reg_cldma_ip_busy_to_ap_mask_clr = REG_CLDMA_IP_BUSY_TO_AP_MASK_CLR,
+ .reg_cldma_ip_busy_to_md_mask_set = REG_CLDMA_IP_BUSY_TO_MD_MASK_SET,
+ .reg_cldma_rx_work_to_reg_mask_set = REG_CLDMA_RX_WORK_TO_REG_MASK_SET,
+ .reg_infra_rst0_set = REG_INFRA_RST0_SET,
+ .reg_infra_rst0_clr = REG_INFRA_RST0_CLR,
+};
+
+static void mtk_cldma_drv_init_m9xx(struct cldma_drv_info *drv_info)
+{
+ struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ int base;
+ u32 val;
+
+ mdev = drv_info->mdev;
+ base = drv_info->base_addr;
+ hw_regs = drv_info->hw_regs;
+
+ /* set CLDMA to 64 bit mode GPD */
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_ul_cfg);
+
+ val = (val & (~(0x7 << 5))) | ((0x4) << 5);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_cfg, val);
+
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_so_cfg);
+ val = (val & (~(0x7 << 10))) | ((0x4) << 10) | (1 << 2);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_cfg, val);
+
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_rx_work_to_reg_mask_set, ALLQ);
+
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_set,
+ ALLQ << 16);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_clr,
+ ALLQ << 24);
+
+ /* enable interrupt to PCIe */
+ if (drv_info->hw_id == CLDMA4_HW_ID)
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma4_int_mask, 0);
+ else
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0);
+
+ /* disable illegal memory check */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_dummy_0, 1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_dummy_0, 1);
+}
+
+static void mtk_cldma_drv_reset_m9xx(struct cldma_drv_info *drv_info)
+{
+ struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ u32 val;
+
+ mdev = drv_info->mdev;
+ hw_regs = drv_info->hw_regs;
+
+ val = mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_set);
+
+ val |= 1 << (REG_CLDMA0_RST_SET_BIT + drv_info->hw_id);
+ mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_set, val);
+ udelay(1);
+ val = mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr);
+ val |= 1 << (REG_CLDMA0_RST_CLR_BIT + drv_info->hw_id);
+ mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr, val);
+}
+
+struct cldma_drv_ops cldma_drv_ops_m9xx = {
+ .cldma_drv_init = mtk_cldma_drv_init_m9xx,
+ .cldma_drv_reset = mtk_cldma_drv_reset_m9xx,
+ .cldma_setup_start_addr = mtk_cldma_setup_start_addr,
+ .cldma_mask_intr = mtk_cldma_mask_intr,
+ .cldma_unmask_intr = mtk_cldma_unmask_intr,
+ .cldma_clr_intr_status = mtk_cldma_clr_intr_status,
+ .cldma_check_intr_status = mtk_cldma_check_intr_status,
+ .cldma_start_queue = mtk_cldma_start_queue,
+ .cldma_resume_queue = mtk_cldma_resume_queue,
+ .cldma_queue_status = mtk_cldma_queue_status,
+ .cldma_stop_queue = mtk_cldma_stop_queue,
+ .cldma_clear_ip_busy = mtk_cldma_clear_ip_busy,
+ .cldma_get_intr_status = mtk_cldma_get_intr_status,
+ .cldma_get_tx_start_addr = mtk_cldma_get_tx_start_addr,
+ .cldma_get_rx_curr_addr = mtk_cldma_get_rx_curr_addr,
+};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h
new file mode 100644
index 000000000000..2c63c43ff065
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv_m9xx.h
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#ifndef __MTK_CLDMA_DRV_M9XX_H__
+#define __MTK_CLDMA_DRV_M9XX_H__
+
+#define CLDMA0_BASE_ADDR (0x1021C000)
+#define CLDMA1_BASE_ADDR (0x1021E000)
+#define CLDMA4_BASE_ADDR (0x10224000)
+
+#define CLDMA_RX_SKB_POOL_MAX_SIZE (64)
+#define CLDMA_RX_SKB_RELOAD_THRESHOLD (16)
+
+/* L2TISAR0 */
+#define TQ_ERR_INT_OFFSET (16)
+#define TQ_ERR_INT_BITMASK (0x00FF0000)
+#define TQ_ACTIVE_START_ERR_INT_OFFSET (24)
+#define TQ_ACTIVE_START_ERR_INT_BITMASK (0xFF000000)
+
+/* L2RISAR0 */
+#define RQ_ERR_INT_OFFSET (16)
+#define RQ_ERR_INT_BITMASK (0x00FF0000)
+#define RQ_ACTIVE_START_ERR_INT_OFFSET (24)
+#define RQ_ACTIVE_START_ERR_INT_BITMASK (0xFF000000)
+
+/* CLDMA IN(Tx) */
+#define REG_CLDMA_UL_START_ADDRL_0 (0x0004)
+#define REG_CLDMA_UL_START_ADDRH_0 (0x0008)
+#define REG_CLDMA_UL_CURRENT_ADDRL_0 (0x0044)
+#define REG_CLDMA_UL_CURRENT_ADDRH_0 (0x0048)
+#define REG_CLDMA_UL_STATUS (0x0084)
+#define REG_CLDMA_UL_START_CMD (0x0088)
+#define REG_CLDMA_UL_RESUME_CMD (0x008C)
+#define REG_CLDMA_UL_STOP_CMD (0x0090)
+#define REG_CLDMA_UL_ERROR (0x0094)
+#define REG_CLDMA_UL_CFG (0x0098)
+#define REG_CLDMA_UL_DUMMY_0 (0x009C)
+
+/* CLDMA OUT(Rx) */
+#define REG_CLDMA_SO_ERROR (0x0400 + 0x0100)
+#define REG_CLDMA_SO_START_CMD (0x0400 + 0x01BC)
+#define REG_CLDMA_SO_RESUME_CMD (0x0400 + 0x01C0)
+#define REG_CLDMA_SO_STOP_CMD (0x0400 + 0x01C4)
+#define REG_CLDMA_SO_DUMMY_0 (0x0400 + 0x0108)
+#define REG_CLDMA_SO_CFG (0x0400 + 0x0004)
+#define REG_CLDMA_SO_START_ADDRL_0 (0x0400 + 0x0078)
+#define REG_CLDMA_SO_START_ADDRH_0 (0x0400 + 0x007C)
+#define REG_CLDMA_SO_CUR_ADDRL_0 (0x0400 + 0x00B8)
+#define REG_CLDMA_SO_CUR_ADDRH_0 (0x0400 + 0x00BC)
+#define REG_CLDMA_SO_STATUS (0x0400 + 0x00F8)
+#define REG_CLDMA_DEBUG_ID_EN (0x0400 + 0x00FC)
+#define REG_CLDMA_SO_LAST_UPDATE_ADDRL_0 (0x0400 + 0x01C8)
+#define REG_CLDMA_SO_LAST_UPDATE_ADDRH_0 (0x0400 + 0x01CC)
+
+/* CLDMA MISC */
+#define REG_CLDMA_L2TISAR0 (0x0800 + 0x0010)
+#define REG_CLDMA_L2TISAR1 (0x0800 + 0x0014)
+#define REG_CLDMA_L2TIMR0 (0x0800 + 0x0018)
+#define REG_CLDMA_L2TIMR1 (0x0800 + 0x001C)
+#define REG_CLDMA_L2TIMCR0 (0x0800 + 0x0020)
+#define REG_CLDMA_L2TIMCR1 (0x0800 + 0x0024)
+#define REG_CLDMA_L2TIMSR0 (0x0800 + 0x0028)
+#define REG_CLDMA_L2TIMSR1 (0x0800 + 0x002C)
+#define REG_CLDMA_L3TISAR0 (0x0800 + 0x0030)
+#define REG_CLDMA_L3TISAR1 (0x0800 + 0x0034)
+#define REG_CLDMA_L2RISAR0 (0x0800 + 0x0050)
+#define REG_CLDMA_L2RISAR1 (0x0800 + 0x0054)
+#define REG_CLDMA_L3RISAR0 (0x0800 + 0x0070)
+#define REG_CLDMA_L3RISAR1 (0x0800 + 0x0074)
+#define REG_CLDMA_IP_BUSY (0x0800 + 0x00B4)
+#define REG_CLDMA_L3TISAR2 (0x0800 + 0x00C0)
+
+#define REG_CLDMA_L2RIMR0 (0x0800 + 0x00E8)
+#define REG_CLDMA_L2RIMR1 (0x0800 + 0x00EC)
+#define REG_CLDMA_L2RIMCR0 (0x0800 + 0x00F0)
+#define REG_CLDMA_L2RIMCR1 (0x0800 + 0x00F4)
+#define REG_CLDMA_L2RIMSR0 (0x0800 + 0x00F8)
+#define REG_CLDMA_L2RIMSR1 (0x0800 + 0x00FC)
+
+#define REG_CLDMA_INT_EAP_USIP_MASK (0x0800 + 0x011C)
+#define REG_CLDMA_INT_WF_MASK (0x0800 + 0x0120)
+#define REG_CLDMA_RQ1_GPD_DONE_CNT (0x0800 + 0x0174)
+#define REG_CLDMA_TQ1_GPD_DONE_CNT (0x0800 + 0x0184)
+
+#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK (0x0800 + 0x0194)
+#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SET (0x0800 + 0x0198)
+#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CLR (0x0800 + 0x019C)
+
+#define REG_CLDMA_IP_BUSY_TO_AP_MASK (0x0800 + 0x0200)
+#define REG_CLDMA_IP_BUSY_TO_AP_MASK_SET (0x0800 + 0x0204)
+#define REG_CLDMA_IP_BUSY_TO_AP_MASK_CLR (0x0800 + 0x0208)
+#define REG_CLDMA_IP_BUSY_TO_MD_MASK_SET (0x0800 + 0x0210)
+#define REG_CLDMA_RX_WORK_TO_REG_MASK_SET (0x0800 + 0x021C)
+
+/* CLDMA RESET */
+#define REG_INFRA_RST0_SET (0x120)
+#define REG_INFRA_RST0_CLR (0x124)
+#define REG_CLDMA0_RST_SET_BIT (8)
+#define REG_CLDMA0_RST_CLR_BIT (8)
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c b/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
new file mode 100644
index 000000000000..bf3f87723167
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_ctrl_cfg_m9xx.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include "mtk_cldma.h"
+#include "mtk_trans_ctrl.h"
+
+#define TRB_SRV_NUM (1)
+
+static const int mtk_srv_cfg_m9xx[NR_CLDMA][HW_QUE_NUM] = {
+ {0},
+ {0},
+};
+
+static const struct queue_info mtk_queue_info_m9xx[] = {
+};
+
+struct mtk_ctrl_info mtk_ctrl_info_m9xx = {
+ .queue_info = (struct queue_info *)mtk_queue_info_m9xx,
+ .queue_info_num = ARRAY_SIZE(mtk_queue_info_m9xx),
+ .trb_srv_num = TRB_SRV_NUM,
+};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index 518c32d55643..d604c9cb06ea 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -760,6 +760,28 @@ static void mtk_pci_free_irq(struct mtk_md_dev *mdev)
pci_free_irq_vectors(pdev);
}
+static int mtk_pci_dev_init(struct mtk_md_dev *mdev)
+{
+ int ret;
+
+ ret = mtk_trans_ctrl_init(mdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to initialize control plane: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void mtk_pci_dev_exit(struct mtk_md_dev *mdev)
+{
+ mtk_trans_ctrl_exit(mdev);
+}
+
+static int mtk_pci_dev_start(struct mtk_md_dev *mdev)
+{
+ return 0;
+}
static const struct mtk_dev_ops pci_hw_ops = {
.get_dev_state = mtk_pci_get_dev_state,
.ack_dev_state = mtk_pci_ack_dev_state,
@@ -834,6 +856,12 @@ static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
if (ret)
goto free_mhccif;
+ ret = mtk_pci_dev_init(mdev);
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to init dev.\n");
+ goto free_irq;
+ }
+
pci_set_master(pdev);
mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
@@ -850,10 +878,20 @@ static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
goto clear_master;
}
+ ret = mtk_pci_dev_start(mdev);
+ if (ret) {
+ dev_err((mdev)->dev, "Failed to start dev.\n");
+ goto free_saved_state;
+ }
+
return 0;
+free_saved_state:
+ pci_load_and_free_saved_state(pdev, &priv->saved_state);
clear_master:
pci_clear_master(pdev);
+ mtk_pci_dev_exit(mdev);
+free_irq:
mtk_pci_free_irq(mdev);
free_mhccif:
mtk_mhccif_exit(mdev);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
index d033dbf4b0af..0f16e6954397 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
@@ -21,6 +21,7 @@
#define REG_IMASK_HOST_MSIX_SET_GRP0_0 0x3000
#define REG_IMASK_HOST_MSIX_CLR_GRP0_0 0x3080
#define REG_IMASK_HOST_MSIX_GRP0_0 0x3100
+#define REG_DEV_INFRA_BASE 0x10001000
/* mhccif registers */
#define MHCCIF_RC2EP_SW_BSY 0x4
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
new file mode 100644
index 000000000000..7fad64d214aa
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -0,0 +1,569 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/device.h>
+#include <linux/freezer.h>
+#include <linux/hashtable.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/nospec.h>
+#include <linux/sched.h>
+#include <linux/wait.h>
+
+#include "mtk_cldma.h"
+#include "mtk_ctrl_plane.h"
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_trans_ctrl.h"
+
+#define MTK_DFLT_PORT_NAME_LEN (20)
+extern struct mtk_ctrl_info ctrl_info_name(m9xx);
+
+static struct mtk_ctrl_info_desc mtk_ctrl_info_tbl[] = {
+ {2304, &ctrl_info_name(m9xx)},
+ {0, NULL},
+};
+
+#define RX_CH_ID_SHIFT 16
+#define PORT_MTU_MASK 0xFFFF
+#define QUEUE_CHL_MASK 0xFFFF
+
+static bool mtk_queue_list_is_full(struct mtk_ctrl_trans *trans, struct queue_info *que)
+{
+ return trans->trans_list[que->hif_id].skb_list[que->txqno].qlen >= SKB_LIST_MAX_LEN;
+}
+
+static bool mtk_ctrl_chs_is_busy_or_empty(struct trb_srv *srv)
+{
+ struct srv_que *srv_que;
+ int i;
+
+ for (i = 0; i < NR_CLDMA; i++)
+ list_for_each_entry(srv_que, &srv->srv_q_list[i], list)
+ if (!skb_queue_empty(&srv->trans->trans_list[i].skb_list[srv_que->qno]) &&
+ mtk_cldma_get_tx_budget(srv->trans->dev, i, srv_que->qno))
+ return false;
+
+ return true;
+}
+
+static void mtk_ctrl_ch_flush(struct sk_buff_head *skb_list)
+{
+ struct sk_buff *skb;
+ struct trb *trb;
+
+ while (!skb_queue_empty(skb_list)) {
+ skb = skb_dequeue(skb_list);
+ trb = (struct trb *)skb->cb;
+ trb->status = -EIO;
+ trb->trb_complete(skb);
+ }
+}
+
+static void mtk_ctrl_chs_flush(struct trb_srv *srv)
+{
+ struct srv_que *srv_que;
+ int i;
+
+ for (i = 0; i < NR_CLDMA; i++)
+ list_for_each_entry(srv_que, &srv->srv_q_list[i], list)
+ mtk_ctrl_ch_flush(&srv->trans->trans_list[i].skb_list[srv_que->qno]);
+}
+
+static int mtk_ch_status_check(struct mtk_ctrl_trans *trans, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct trb_open_priv *trb_open_priv;
+ struct queue_info *que;
+ int ret = 0;
+
+ que = radix_tree_lookup(&trans->queue_tbl, trb->channel_id & QUEUE_CHL_MASK);
+
+ switch (trb->cmd) {
+ case TRB_CMD_ENABLE:
+ trb_open_priv = (struct trb_open_priv *)skb->data;
+ trb_open_priv->log_rg_offset = que->log_rg_offset;
+ trans->usr_cnt[que->hif_id][que->txqno]++;
+ if (trans->usr_cnt[que->hif_id][que->txqno] == 1)
+ break;
+ trb_open_priv->tx_mtu = que->tx_mtu;
+ trb_open_priv->rx_mtu = que->rx_mtu;
+ trb_open_priv->tx_frag_size = que->tx_frag_size;
+ trb_open_priv->rx_frag_size = que->rx_frag_size;
+ if (mtk_cldma_check_ch_cfg(trans->dev, que)) {
+ trb->status = -EINVAL;
+ ret = -EINVAL;
+ } else {
+ trb->status = -EBUSY;
+ ret = -EBUSY;
+ }
+ trb->trb_complete(skb);
+ break;
+ case TRB_CMD_DISABLE:
+ if (trans->usr_cnt[que->hif_id][que->txqno] > 0) {
+ trans->usr_cnt[que->hif_id][que->txqno]--;
+ if (!trans->usr_cnt[que->hif_id][que->txqno])
+ break;
+ }
+ trb->status = -EBUSY;
+ trb->trb_complete(skb);
+ ret = -EBUSY;
+ break;
+ default:
+ dev_err((trans->mdev)->dev, "Invalid trb command(%d)\n", trb->cmd);
+ ret = -EINVAL;
+ break;
+ }
+ return ret;
+}
+
+static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_list, u32 qno)
+{
+ struct sk_buff_head *skb_list = &trans_list->skb_list[qno];
+ struct mtk_ctrl_trans *trans = srv->trans;
+ struct sk_buff *skb, *skb_next;
+ struct trb *trb, *trb_next;
+ bool kick = false;
+ int loop = 0;
+ int err;
+
+ do {
+ skb = skb_peek(skb_list);
+ if (!skb)
+ break;
+ trb = (struct trb *)skb->cb;
+
+ switch (trb->cmd) {
+ case TRB_CMD_ENABLE:
+ case TRB_CMD_DISABLE:
+ skb_unlink(skb, skb_list);
+ err = mtk_ch_status_check(trans, skb);
+ if (!err) {
+ kick = true;
+ if (trb->cmd == TRB_CMD_DISABLE)
+ mtk_ctrl_ch_flush(skb_list);
+ }
+ break;
+ case TRB_CMD_TX:
+ err = mtk_cldma_submit_tx(trans->dev, skb);
+ if (err) {
+ if (trans_list->tx_burst_cnt[qno])
+ kick = true;
+ else if (err == -EAGAIN)
+ return;
+ break;
+ }
+
+ trans_list->tx_burst_cnt[qno]++;
+ if (trans_list->tx_burst_cnt[qno] >= TX_BURST_MAX_CNT ||
+ skb_queue_is_last(skb_list, skb)) {
+ kick = true;
+ } else {
+ skb_next = skb_peek_next(skb, skb_list);
+ trb_next = (struct trb *)skb_next->cb;
+ if (trb_next->cmd != TRB_CMD_TX)
+ kick = true;
+ }
+
+ skb_unlink(skb, skb_list);
+ break;
+ default:
+ skb_unlink(skb, skb_list);
+ }
+
+ if (kick) {
+ mtk_cldma_trb_process(trans->dev, skb);
+ trans_list->tx_burst_cnt[qno] = 0;
+ kick = false;
+ }
+
+ loop++;
+ } while (loop < TRB_NUM_PER_ROUND);
+}
+
+static void mtk_ctrl_trb_process(struct trb_srv *srv)
+{
+ struct mtk_ctrl_trans *trans = srv->trans;
+ struct srv_que *srv_que;
+ int i;
+
+ for (i = 0; i < NR_CLDMA; i++)
+ list_for_each_entry(srv_que, &srv->srv_q_list[i], list)
+ mtk_ctrl_trb_handler(srv, &trans->trans_list[i], srv_que->qno);
+}
+
+static int mtk_ctrl_trb_thread(void *args)
+{
+ struct trb_srv *srv = args;
+
+ for (;;) {
+ wait_event_interruptible(srv->trb_waitq,
+ !mtk_ctrl_chs_is_busy_or_empty(srv) ||
+ kthread_should_stop() || kthread_should_park());
+ if (kthread_should_stop())
+ break;
+
+ if (kthread_should_park())
+ kthread_parkme();
+
+ do {
+ mtk_ctrl_trb_process(srv);
+ cond_resched();
+ } while (!mtk_ctrl_chs_is_busy_or_empty(srv) && !kthread_should_stop() &&
+ !kthread_should_park());
+ }
+ mtk_ctrl_chs_flush(srv);
+ return 0;
+}
+
+static int mtk_ctrl_trb_srv_init(struct mtk_ctrl_trans *trans)
+{
+ struct srv_que *srv_que;
+ struct trb_srv *srv;
+ int i, j;
+ int ret;
+
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ srv = devm_kzalloc(trans->mdev->dev, sizeof(*srv), GFP_KERNEL);
+ if (!srv) {
+ ret = -ENOMEM;
+ goto err_free_srv;
+ }
+
+ srv->trans = trans;
+ srv->srv_id = i;
+ trans->trb_srv[i] = srv;
+
+ init_waitqueue_head(&srv->trb_waitq);
+ for (j = 0; j < NR_CLDMA; j++)
+ INIT_LIST_HEAD(&srv->srv_q_list[j]);
+ }
+
+ for (i = 0; i < NR_CLDMA; i++)
+ for (j = 0; j < HW_QUE_NUM; j++) {
+ if (trans->srv_cfg[i][j] < 0 ||
+ trans->srv_cfg[i][j] >= trans->trb_srv_num)
+ trans->srv_cfg[i][j] = 0;
+ srv_que = devm_kzalloc(trans->mdev->dev, sizeof(*srv_que), GFP_KERNEL);
+ if (!srv_que) {
+ ret = -ENOMEM;
+ goto err_free_srv_que;
+ }
+ srv_que->hif_id = i;
+ srv_que->qno = j;
+ list_add_tail(&srv_que->list,
+ &trans->trb_srv[trans->srv_cfg[i][j]]->srv_q_list[i]);
+ }
+
+ for (i = 0; i < trans->trb_srv_num; i++)
+ trans->trb_srv[i]->trb_thread = kthread_run(mtk_ctrl_trb_thread, trans->trb_srv[i],
+ "mtk_trb_srv%d_%s", i,
+ trans->mdev->dev_str);
+
+ return 0;
+err_free_srv_que:
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ for (j = 0; j < NR_CLDMA; j++) {
+ struct srv_que *next_srv_que;
+
+ list_for_each_entry_safe(srv_que, next_srv_que,
+ &trans->trb_srv[i]->srv_q_list[j], list) {
+ list_del(&srv_que->list);
+ devm_kfree(trans->mdev->dev, srv_que);
+ }
+ }
+ }
+err_free_srv:
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ if (!trans->trb_srv[i])
+ break;
+ devm_kfree(trans->mdev->dev, trans->trb_srv[i]);
+ trans->trb_srv[i] = NULL;
+ }
+
+ return ret;
+}
+
+static void mtk_ctrl_trb_srv_exit(struct mtk_ctrl_trans *trans)
+{
+ struct srv_que *srv_que, *next_srv_que;
+ struct trb_srv *srv;
+ int i, j;
+
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ srv = trans->trb_srv[i];
+ kthread_stop(srv->trb_thread);
+ for (j = 0; j < NR_CLDMA; j++) {
+ list_for_each_entry_safe(srv_que, next_srv_que,
+ &trans->trb_srv[i]->srv_q_list[j], list) {
+ list_del(&srv_que->list);
+ devm_kfree(trans->mdev->dev, srv_que);
+ }
+ }
+ devm_kfree(trans->mdev->dev, srv);
+ trans->trb_srv[i] = NULL;
+ }
+}
+
+static void mtk_ctrl_remove_radix_tree(struct mtk_ctrl_trans *trans)
+{
+ struct queue_info **queues;
+ int ret, idx;
+
+ queues = kcalloc(trans->queues_cnt, sizeof(struct queue_info *), GFP_KERNEL);
+ if (!queues)
+ return;
+
+ ret = radix_tree_gang_lookup(&trans->queue_tbl, (void **)queues,
+ 0, trans->queues_cnt);
+ for (idx = 0; idx < ret; idx++) {
+ radix_tree_delete(&trans->queue_tbl, queues[idx]->rx_chl & QUEUE_CHL_MASK);
+ kfree(queues[idx]);
+ }
+ kfree(queues);
+}
+
+static void mtk_ctrl_queue_info_update(struct radix_tree_root *queue_tbl, u32 port_chl_mtu)
+{
+ struct queue_info *queue;
+ u32 rx_chl, mtu;
+
+ if (!port_chl_mtu)
+ return;
+
+ rx_chl = port_chl_mtu >> RX_CH_ID_SHIFT;
+ mtu = port_chl_mtu & PORT_MTU_MASK;
+ queue = radix_tree_lookup(queue_tbl, rx_chl);
+ if (!queue)
+ return;
+
+ queue->tx_mtu = mtu;
+ queue->rx_mtu = mtu;
+ queue->tx_frag_size = mtu;
+ queue->rx_frag_size = mtu;
+}
+
+static unsigned int ctrl_port_chl_mtu;
+
+static int mtk_pcie_hif_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct queue_info *queue, *queue_info;
+ struct mtk_ctrl_trans *trans;
+ int i, j;
+ int ret;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+ trans->ctrl_blk = ctrl_blk;
+ queue_info = trans->queue_info;
+
+ INIT_RADIX_TREE(&trans->queue_tbl, GFP_KERNEL);
+ for (i = 0; i < trans->queue_info_num; i++) {
+ queue = kmemdup(queue_info + i, sizeof(*queue), GFP_KERNEL);
+ if (!queue) {
+ ret = -ENOMEM;
+ goto err_free_radix_tree;
+ }
+ if (queue->txqno >= HW_QUE_NUM || queue->rxqno >= HW_QUE_NUM ||
+ queue->hif_id >= NR_CLDMA) {
+ dev_err((mdev)->dev, "Failed to get correct queue info %x\n",
+ queue->rx_chl);
+ ret = -EINVAL;
+ goto err_free_radix_tree;
+ }
+ ret = radix_tree_insert(&trans->queue_tbl, queue->rx_chl & QUEUE_CHL_MASK, queue);
+ if (ret) {
+ dev_err((mdev)->dev, "Insert %x fail, ret: %d", queue->rx_chl, ret);
+ kfree(queue);
+ goto err_free_radix_tree;
+ }
+ trans->queues_cnt++;
+ }
+
+ mtk_ctrl_queue_info_update(&trans->queue_tbl, ctrl_port_chl_mtu);
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ for (j = 0; j < HW_QUE_NUM; j++) {
+ skb_queue_head_init(&trans->trans_list[i].skb_list[j]);
+ trans->trans_list[i].tx_burst_cnt[j] = 0;
+ }
+ }
+ ret = mtk_cldma_init(trans);
+ if (ret)
+ goto err_free_radix_tree;
+
+ ret = mtk_ctrl_trb_srv_init(trans);
+ if (ret)
+ goto err_cldma_exit;
+
+ atomic_set(&trans->available, 1);
+
+ return 0;
+
+err_cldma_exit:
+ mtk_cldma_exit(trans);
+err_free_radix_tree:
+ mtk_ctrl_remove_radix_tree(trans);
+
+ return ret;
+}
+
+static int mtk_pcie_hif_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+
+ atomic_set(&trans->available, 0);
+ mtk_ctrl_trb_srv_exit(trans);
+ mtk_ctrl_remove_radix_tree(trans);
+ mtk_cldma_exit(trans);
+
+ return 0;
+}
+
+static int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct queue_info *que;
+ struct trb *trb;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+ trb = (struct trb *)skb->cb;
+
+ if (trb->cmd == TRB_CMD_STOP || trb->cmd == TRB_CMD_RECOVER) {
+ trb->trb_complete(skb);
+ return 0;
+ }
+
+ que = radix_tree_lookup(&trans->queue_tbl, trb->channel_id & QUEUE_CHL_MASK);
+ if (!que) {
+ dev_warn((mdev)->dev, "lookup que fail, ch_id: %x, que: 0x%p\n",
+ trb->channel_id, que);
+ return -EINVAL;
+ }
+
+ if (!atomic_read(&trans->available))
+ return -EIO;
+
+ if (mtk_queue_list_is_full(trans, que) && !force_send)
+ return -EAGAIN;
+
+ if (trb->cmd == TRB_CMD_DISABLE)
+ skb_queue_head(&trans->trans_list[que->hif_id].skb_list[que->txqno], skb);
+ else
+ skb_queue_tail(&trans->trans_list[que->hif_id].skb_list[que->txqno], skb);
+
+ wake_up(&trans->trb_srv[trans->srv_cfg[que->hif_id][que->txqno]]->trb_waitq);
+
+ return 0;
+}
+
+static int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct queue_info *que;
+ int ret = 0;
+
+ switch (cmd) {
+ case HIF_CTRL_CMD_CHECK_TX_FULL:
+ trans = ctrl_blk->ctrl_hw_priv;
+ que = radix_tree_lookup(&trans->queue_tbl,
+ ((union ctrl_hif_cmd_data *)data)->rx_ch & QUEUE_CHL_MASK);
+ if (!que) {
+ dev_warn((mdev)->dev, "Failed to find que to check tx full\n");
+ return -EINVAL;
+ }
+ return mtk_queue_list_is_full(trans, que);
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static struct mtk_ctrl_hif_ops pcie_ctrl_ops = {
+ .init = mtk_pcie_hif_init,
+ .exit = mtk_pcie_hif_exit,
+ .submit_skb = mtk_pcie_hif_submit_skb,
+ .send_cmd = mtk_pcie_hif_cmd_func,
+};
+
+static void mtk_trans_get_ctrl_info(struct mtk_ctrl_cfg *cfg,
+ struct mtk_ctrl_trans *trans, u32 hw_ver)
+{
+ struct mtk_ctrl_info_desc *ctrl_info_desc;
+ struct mtk_ctrl_info *ctrl_info;
+ u8 i;
+
+ for (i = 0; (ctrl_info_desc = &mtk_ctrl_info_tbl[i]) && ctrl_info_desc &&
+ ctrl_info_desc->ctrl_info; i++) {
+ if (ctrl_info_desc->hw_ver != hw_ver)
+ continue;
+
+ ctrl_info = ctrl_info_desc->ctrl_info;
+ memcpy(trans->srv_cfg, ctrl_info->srv_cfg,
+ sizeof(int) * NR_CLDMA * HW_QUE_NUM);
+ trans->queue_info = ctrl_info->queue_info;
+ trans->queue_info_num = ctrl_info->queue_info_num;
+ trans->trb_srv_num = ctrl_info->trb_srv_num;
+ }
+}
+
+int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ int err;
+
+ trans = devm_kzalloc(mdev->dev, sizeof(*trans), GFP_KERNEL);
+ if (!trans)
+ return -ENOMEM;
+ trans->mdev = mdev;
+ trans->queues_cnt = 0;
+
+ mtk_trans_get_ctrl_info(NULL, trans, mdev->hw_ver);
+ if (!trans->queue_info ||
+ trans->trb_srv_num <= 0 || trans->trb_srv_num > TRB_SRV_MAX_NUM ||
+ trans->queue_info_num <= 0) {
+ dev_err((mdev)->dev, "Failed to get ctrl info!\n");
+ goto err_free_cfg;
+ }
+
+ err = mtk_ctrl_init(mdev, &pcie_ctrl_ops);
+ if (err)
+ goto err_free_cfg;
+
+ ctrl_blk = mdev->ctrl_blk;
+ ctrl_blk->ctrl_hw_priv = trans;
+
+ return 0;
+
+err_free_cfg:
+ devm_kfree(mdev->dev, trans);
+ return -ENOMEM;
+}
+
+int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+
+ ctrl_blk = mdev->ctrl_blk;
+ trans = ctrl_blk->ctrl_hw_priv;
+
+ devm_kfree(mdev->dev, ctrl_blk->cfg);
+ mtk_ctrl_exit(mdev);
+ devm_kfree(mdev->dev, trans);
+
+ return 0;
+}
+
+module_param(ctrl_port_chl_mtu, uint, 0644);
+MODULE_PARM_DESC(ctrl_port_chl_mtu, "This is used to config the ctrl port mtu!\n");
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
index d6de4c43b529..c2df0bf6ed65 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -13,9 +13,93 @@
#include "mtk_dev.h"
+#define TRB_SRV_MAX_NUM (1)
+#define HW_QUE_NUM (8)
+#define TX_GPD_NUM (16)
+#define RX_GPD_NUM (TX_GPD_NUM)
+#define MIN_GPD_NUM (2)
+#define SKB_LIST_MAX_LEN (16)
+#define MTU_RSV_ROOM (0x100)
+#define TRB_NUM_PER_ROUND (TX_GPD_NUM)
+#define TX_BURST_MAX_CNT (TX_GPD_NUM / 4 + 1)
+
+#define HIF_ID(peer_id) ((peer_id) - 1)
+
+enum mtk_hif_id {
+ CLDMA0,
+ CLDMA1,
+ CLDMA4,
+ NR_CLDMA
+};
+
+struct queue_info {
+ u32 tx_chl;
+ u32 rx_chl;
+ enum mtk_hif_id hif_id;
+ u32 txqno;
+ u32 rxqno;
+ u32 tx_mtu;
+ u32 rx_mtu;
+ u32 tx_nr_gpds;
+ u32 rx_nr_gpds;
+ u32 tx_frag_size;
+ u32 rx_frag_size;
+ u8 log_rg_offset;
+};
+
+struct trans_list {
+ struct sk_buff_head skb_list[HW_QUE_NUM];
+ u8 tx_burst_cnt[HW_QUE_NUM];
+};
+
struct mtk_ctrl_trans {
struct mtk_ctrl_blk *ctrl_blk;
+ struct trb_srv *trb_srv[TRB_SRV_MAX_NUM];
+ struct trans_list trans_list[NR_CLDMA];
+ void *dev;
+ struct radix_tree_root queue_tbl;
struct mtk_md_dev *mdev;
+ int usr_cnt[NR_CLDMA][HW_QUE_NUM];
+ u32 tx_mtu_cfg[NR_CLDMA][HW_QUE_NUM];
+ u32 rx_mtu_cfg[NR_CLDMA][HW_QUE_NUM];
+ atomic_t available;
+ int queues_cnt;
+ int srv_cfg[NR_CLDMA][HW_QUE_NUM];
+ struct queue_info *queue_info;
+ int queue_info_num;
+ int trb_srv_num;
+};
+
+struct srv_que {
+ u32 hif_id;
+ u32 qno;
+ struct list_head list;
+};
+
+struct trb_srv {
+ u32 srv_id;
+ struct list_head srv_q_list[NR_CLDMA];
+ struct mtk_ctrl_trans *trans;
+ wait_queue_head_t trb_waitq;
+ struct task_struct *trb_thread;
+};
+
+struct mtk_ctrl_info {
+ struct mtk_ctrl_cfg *ctrl_cfg;
+ int **srv_cfg;
+ struct queue_info *queue_info;
+ u32 queue_info_num;
+ u32 trb_srv_num;
};
+struct mtk_ctrl_info_desc {
+ u32 hw_ver;
+ struct mtk_ctrl_info *ctrl_info;
+};
+
+#define ctrl_info_name(NAME) mtk_ctrl_info_##NAME
+
+int mtk_trans_ctrl_init(struct mtk_md_dev *mdev);
+int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev);
+
#endif
--
2.34.1
^ permalink raw reply related
* Re: [PATCH v3 1/5] KVM: PPC: Book3S HV: Validate arch_compat against host compatibility mode
From: Amit Machhiwal @ 2026-05-29 10:28 UTC (permalink / raw)
To: Ritesh Harjani
Cc: Amit Machhiwal, linuxppc-dev, Madhavan Srinivasan, Vaibhav Jain,
Anushree Mathur, Paolo Bonzini, Nicholas Piggin, Michael Ellerman,
Christophe Leroy (CS GROUP), Jonathan Corbet, Shuah Khan, kvm,
linux-kernel, linux-doc, lkp
In-Reply-To: <pl2g6xbz.ritesh.list@gmail.com>
Hi Ritesh,
Thanks for taking a look at this patch. Please find my response inline
below:
On 2026/05/28 08:43 AM, Ritesh Harjani wrote:
> Amit Machhiwal <amachhiw@linux.ibm.com> writes:
>
> > On IBM POWER systems, newer processor generations can operate in
> > compatibility modes corresponding to earlier generations. This becomes
> > relevant for nested virtualization, where nested KVM guests may need to
> > run with a specific processor compatibility level.
> >
> > Currently, when running a nested KVM guest (L2) inside a Power11 pSeries
> > logical partition (L1) booted in Power10 compatibility mode, the guest
> > fails to boot while setting 'arch_compat'. This happens because the CPU
> > class is derived from the hardware PVR (via mfspr()), which reflects the
> > physical processor generation (Power11), rather than the effective
> > compatibility mode (Power10).
> >
> > As a result, userspace may request a Power11 arch_compat for the L2
> > guest. However, the L1 partition, running in Power10 compatibility, has
> > only negotiated support up to Power10 with the Power Hypervisor (L0).
> > When H_SET_STATE is invoked with a Power11 Logical PVR, the hypervisor
>
> s/H_SET_STATE/H_GUEST_SET_STATE
Good catch! I'll fix this in the next version.
>
> > rejects the request, leading to a late guest boot failure:
> >
> > KVM-NESTEDv2: couldn't set guest wide elements
> > [..KVM reg dump..]
> >
>
> I think irrespective of the other UAPI changes, we should still get this
> fixed - so that we don't see a late KVM guest boot failure msgs.
>
> So, in this review, I would like to mainly look at fixing this issue
> first and would request if we can defer the UAPI changes as a separate
> patch series please.
This patch only enables the guest boot to bail out early instead of going upto
making an H_GUEST_SET_STATE hcall with a non supported compatibility mode. In
addition to that, it does not fix the real problem where guest fails to boot on
Power11 LPAR booted in a Power10 compatibility mode.
I understand your point and to make the segregation explicitly clear, I shall
update the cover letter to mention that patch 1 only takes care of failing
earlier as soon as an invalid compatibility mode is detected and Patch 2-5
introduce a new uAPI for evaluating the right compatibility mode.
The actual L2 guest boot fix with L1 booted in a compatibility mode is done via
the next 4 patches which enables correct compatibility mode detection using the
newly introduced uAPI. So, we would still want to prioritize the whole series
instead of just this one patch.
>
>
> > This situation should be detected earlier. Rejecting unsupported
> > 'arch_compat' values in 'kvmppc_set_arch_compat()' avoids issuing an
> > invalid H_SET_STATE hcall and provides a clearer failure mode.
>
> s/H_SET_STATE/H_GUEST_SET_STATE
Will rectify in the next version.
>
> >
> > Add a check to reject Power11 'arch_compat' requests when the host is
> > running in Power10 compatibility mode, returning -EINVAL early instead
> > of deferring the failure to the hypervisor.
> >
> > Suggested-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> > Tested-by: Anushree Mathur <anushree.mathur@linux.ibm.com>
> > Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com>
> > ---
> > arch/powerpc/kvm/book3s_hv.c | 12 ++++++++++++
> > 1 file changed, 12 insertions(+)
> >
> > diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> > index 61dbeea317f3..249d1f2e4e2c 100644
> > --- a/arch/powerpc/kvm/book3s_hv.c
> > +++ b/arch/powerpc/kvm/book3s_hv.c
> > @@ -446,7 +446,19 @@ static int kvmppc_set_arch_compat(struct kvm_vcpu *vcpu, u32 arch_compat)
> > guest_pcr_bit = PCR_ARCH_300;
> > break;
> > case PVR_ARCH_31:
> > + guest_pcr_bit = PCR_ARCH_31;
> > + break;
> > case PVR_ARCH_31_P11:
> > + /*
> > + * Need to check this for ISA 3.1, as Power10 and
> > + * Power11 share the same PCR. For any subsequent ISA
> > + * versions, this will be taken care of by the guest vs
> > + * host PCR comparison below.
> > + */
> > + if ((PVR_ARCH_31 & cur_cpu_spec->pvr_mask) ==
> > + cur_cpu_spec->pvr_value) {
> > + return -EINVAL;
> > + }
>
> Instead of the complicated check can we simply do this?
> if (!cpu_has_feature(CPU_FTR_P11_PVR))
> return -EINVAL;
Sure, I can base this check on CPU features.
Thanks,
Amit
>
> which means that if the Qemu is trying to set the arch_compat with P11
> PVR (arch_compat) and if the host cpu FTR doesn't support P11 PVR, then
> simply return -EINVAL
>
> -ritesh
>
^ permalink raw reply
* Re: [PATCH net-next 3/3] net/mlx5: Apply devlink default eswitch mode during init
From: Jiri Pirko @ 2026-05-29 10:25 UTC (permalink / raw)
To: Mark Bloch
Cc: Tariq Toukan, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Andrew Lunn, David S. Miller, Jonathan Corbet, Shuah Khan,
Simon Horman, Saeed Mahameed, Leon Romanovsky,
Borislav Petkov (AMD), Andrew Morton, Randy Dunlap,
Thomas Gleixner, Petr Mladek, Peter Zijlstra (Intel), Tejun Heo,
Vlastimil Babka, Feng Tang, Christian Brauner, Dave Hansen,
Dapeng Mi, Kees Cook, Marco Elver, Li RongQing, Eric Biggers,
Paul E. McKenney, linux-doc, linux-kernel, netdev, linux-rdma,
Gal Pressman, Dragos Tatulea, Jiri Pirko, Shay Drori,
Moshe Shemesh
In-Reply-To: <e4f4a6a5-9be0-462b-b4d7-8bbf57001cb4@nvidia.com>
Thu, May 28, 2026 at 10:15:44AM +0200, mbloch@nvidia.com wrote:
>
>
>On 27/05/2026 14:18, Jiri Pirko wrote:
>> Wed, May 27, 2026 at 09:03:26AM +0200, mbloch@nvidia.com wrote:
>>>
>>>
>>> On 27/05/2026 8:14, Jiri Pirko wrote:
>>>> Tue, May 26, 2026 at 07:13:46PM +0200, mbloch@nvidia.com wrote:
>>>>>
>>>>>
>>>>> On 26/05/2026 19:23, Jiri Pirko wrote:
>>>>>> Tue, May 26, 2026 at 05:03:57PM +0200, mbloch@nvidia.com wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 26/05/2026 17:07, Jiri Pirko wrote:
>>>>>>>> Tue, May 26, 2026 at 11:44:46AM +0200, mbloch@nvidia.com wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 26/05/2026 10:44, Jiri Pirko wrote:
>>>>>>>>>> Thu, May 21, 2026 at 09:24:34AM +0200, tariqt@nvidia.com wrote:
>>>>>>>>>>> From: Mark Bloch <mbloch@nvidia.com>
>>>>>>>>>>>
>>>>>>>>>>> Apply devlink default eswitch mode for mlx5 devices after successful
>>>>>>>>>>> device initialization while holding the devlink instance lock.
>>>>>>>>>>>
>>>>>>>>>>> At this point the devlink instance is registered and the mlx5 devlink
>>>>>>>>>>> operations are available, so the default eswitch mode can be applied to
>>>>>>>>>>> the matching PCI devlink handle.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Mark Bloch <mbloch@nvidia.com>
>>>>>>>>>>> Reviewed-by: Shay Drori <shayd@nvidia.com>
>>>>>>>>>>> Reviewed-by: Moshe Shemesh <moshe@nvidia.com>
>>>>>>>>>>> Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
>>>>>>>>>>> ---
>>>>>>>>>>> drivers/net/ethernet/mellanox/mlx5/core/main.c | 17 +++++++++++++++++
>>>>>>>>>>> 1 file changed, 17 insertions(+)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>>>>>>>> index 0c6e4efe38c8..4528097f3d84 100644
>>>>>>>>>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>>>>>>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
>>>>>>>>>>> @@ -1391,6 +1391,21 @@ static void mlx5_unload(struct mlx5_core_dev *dev)
>>>>>>>>>>> mlx5_free_bfreg(dev, &dev->priv.bfreg);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> +static void mlx5_devl_apply_default_esw_mode(struct mlx5_core_dev *dev)
>>>>>>>>>>> +{
>>>>>>>>>>> + struct devlink *devlink = priv_to_devlink(dev);
>>>>>>>>>>> + int err;
>>>>>>>>>>> +
>>>>>>>>>>> + if (!MLX5_ESWITCH_MANAGER(dev))
>>>>>>>>>>> + return;
>>>>>>>>>>> +
>>>>>>>>>>> + devl_assert_locked(devlink);
>>>>>>>>>>> + err = devl_apply_default_esw_mode(devlink);
>>>>>>>>>>> + if (err)
>>>>>>>>>>> + mlx5_core_warn(dev, "Couldn't apply default eswitch mode, err %d\n",
>>>>>>>>>>> + err);
>>>>>>>>>>> +}
>>>>>>>>>>> +
>>>>>>>>>>> int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev)
>>>>>>>>>>> {
>>>>>>>>>>> bool light_probe = mlx5_dev_is_lightweight(dev);
>>>>>>>>>>> @@ -1437,6 +1452,7 @@ int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev)
>>>>>>>>>>> mlx5_core_err(dev, "mlx5_hwmon_dev_register failed with error code %d\n", err);
>>>>>>>>>>>
>>>>>>>>>>> mutex_unlock(&dev->intf_state_mutex);
>>>>>>>>>>> + mlx5_devl_apply_default_esw_mode(dev);
>>>>>>>>>>
>>>>>>>>>> I wonder how we can make this work for all. I mean, other driver would
>>>>>>>>>> silently ignore this command like arg, right? Any idea how to make all
>>>>>>>>>> drivers follow the arg from very beginning?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have a follow-up series that adds the call to all drivers which support
>>>>>>>>> setting eswitch mode. When going over the other drivers, what I found is
>>>>>>>>> that the right point to apply the default is driver specific, drivers
>>>>>>>>> I have patch for:
>>>>>>>>>
>>>>>>>>> 46e16c6d9836 net: Apply devlink esw mode defaults
>>>>>>>>> ab4f54102ba9 bnxt_en: Apply devlink default eswitch mode during init
>>>>>>>>> b48cce1607bb liquidio: Apply devlink default eswitch mode during init
>>>>>>>>> 4ea54b0fe04a ice: Apply devlink default eswitch mode during init
>>>>>>>>> b7faddaa1c90 octeontx2-af: Apply devlink default eswitch mode during init
>>>>>>>>> 74b0c22c47b9 octeontx2-pf: Apply devlink default eswitch mode during init
>>>>>>>>> 5000e4c3d768 nfp: Apply devlink default eswitch mode during init
>>>>>>>>> 97a218e95e41 netdevsim: Apply devlink default eswitch mode during init
>>>>>>>>>
>>>>>>>>> I don't think doing this generically from devlink is realistic. devlink
>>>>>>>>> doesn't really know when a given driver is ready to change eswitch mode.
>>>>>>>>> Some drivers need SR-IOV state, representor setup, or other init pieces to
>>>>>>>>> be ready first, and the locking is not identical across drivers either.
>>>>>>>>
>>>>>>>>
>>>>>>>> Low hanging fruit would be just to call ops->eswitch_mode_set at the end
>>>>>>>> of register. Multiple reasons:
>>>>>>>>
>>>>>>>> 1) end of devl_register is exactly the point userspace is free to issue
>>>>>>>> the eswitch mode set. Driver should be ready to handle it.
>>>>>>>> 2) all drivers would transparently get this functionality, without
>>>>>>>> actually knowing this kernel command line arg ever existed, without
>>>>>>>> odd wiring call of related exported function. I prefer that stongly.
>>>>>>>> 3) you should add a there warning for the case this arg is passed yet
>>>>>>>> the driver does not implement eswitch_mode_set. User should
>>>>>>>> get a feedback like this, not silent ignore.
>>>>>>>>
>>>>>>>> The only loose end is see it the void return of devl_register().
>>>>>>>> Multiple ways to handle the possibly failed eswitch_mode_set(). I would
>>>>>>>> probably just go for pr_warn, seems to be the most correct.
>>>>>>>>
>>>>>>>> Make sense?
>>>>>>>
>>>>>>> I see the point, but I don't think devl_register() (at least not the only place)
>>>>>>> is the right place.
>>>>>>>
>>>>>>> There is a small but important difference between userspace doing
>>>>>>> "devlink eswitch set" after register is done, and devlink core calling
>>>>>>> eswitch_mode_set() from inside the register flow.
>>>>>>>
>>>>>>> Some drivers call devlink_register() while holding the device lock.
>>>>>>> liquidio is one example. If devlink core calls ops->eswitch_mode_set() from
>>>>>>> there, we may start the full eswitch mode change while holding that lock.
>>>>>>> That mode change can create representors, register netdevs, take rtnl,
>>>>>>> allocate resources, etc. I don't think we want this to become an implicit
>>>>>>> side effect of devlink registration.
>>>>>>
>>>>>> I believe your AI may untagle liquidio locking :)
>>>>>
>>>>> I didn't try to solve that one with ai. Most drivers were fairly simple
>>>>> so I didn't use ai at all. bnxt was the one where I needed a bit of help :)
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> For mlx5, the placement after intf_state_mutex is also intentional:
>>>>>>>
>>>>>>> mutex_unlock(&dev->intf_state_mutex);
>>>>>>> mlx5_devl_apply_default_esw_mode(dev);
>>>>>>>
>>>>>>> We can't call it while holding intf_state_mutex because the mode set path
>>>>>>> takes it internally, and switchdev mode may also create IB representors.
>>>>>>>
>>>>>>> Also, devl_register() only covers the first registration. The mlx5 call in
>>>>>>> mlx5_load_one_devl_locked() is for reload/fw reset recovery kind of flows.
>>>>>>> In those flows devlink is already registered, so devl_register() is not
>>>>>>> called again, but the driver state was rebuilt and we may need to apply the
>>>>>>> default again.
>>>>>>
>>>>>> Call it from reload too, right?
>>>>>
>>>>> Yes, that was my first thought: apply it from devl_register() for the first
>>>>> registration and from devlink_reload() after a successful DRIVER_REINIT.
>>>>>
>>>>> That covers the clean devlink reload path but....(see bellow)
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Same for reload, fw reset and pci recovery in general. If the driver tears
>>>>>>> down and rebuilds eswitch related state, the place to apply the default is
>>>>>>> in that driver's reinit flow, not in devl_register().
>>>>>>>
>>>>>>> When I went over the other drivers, the right place was not always the same
>>>>>>> as devlink registration. I'm not an expert in any of them, so I hope I got
>>>>>>> the details right, but for example octeontx2 AF needs sr-iov and the
>>>>>>> representor switch state to be initialized first. nfp can do it after
>>>>>>> app/vNIC init while the devlink lock is already held. liquidio should do it
>>>>>>> only after dropping the PCI device lock.
>>>>>>
>>>>>> Idk, perhaps do it from devlink_post_register_work of some kind? That
>>>>>> would allow you to have the same locking ordering as a userspace cal
>>>>> l.
>>>>>
>>>>> I thought about a workqueue too, it was actually my first idea.
>>>>>
>>>>> The problem is that then we race with userspace. In the mlx5 version here the
>>>>> default is applied while the devlink lock is still held, before userspace can
>>>>> come in and issue its own eswitch set. If we defer it to post-register work,
>>>>> the devlink instance is already visible and userspace can get there first
>>>>> and then we might change the user configuration.
>>>>
>>>> Figure that out and expose to user by setting xa_mark only after the
>>>> work is done? This is doable.
>>>
>>> I agree that if devlink can keep the instance hidden/unavailable until the
>>> post register work is done, that solves the initial userspace race.
>>>
>>> The other part is the reinit/recovery case. For that I think devlink core
>>> needs some explicit indication from the driver that the device is now in
>>> reinit. Something like (at least that's the code I had initially, but something
>>> along those lines):
>>>
>>> void devl_dev_reinit_begin(struct devlink *devlink);
>>> void devl_dev_reinit_end(struct devlink *devlink);
>>> void devl_dev_reinit_abort(struct devlink *devlink);
>>>
>>> The core can then mark the instance as temporarily unavailable/in reinit
>>> between begin/end, and the relevant lookup/dump paths, for example
>>> devlink_get_from_attrs_lock() and devlink_nl_inst_iter_dumpit(), can reject
>>> or skip it while reinit is in progress. devlink_reload() can probably mark
>>> this state by itself around DRIVER_REINIT.
>>
>> I believe this is orthogonal to the problem you are trying to solve in
>> this patchset. Not sure why you bring it in to the conversation...
>>
>
>I brought it up because I was also thinking about reinit/recovery flows, but
>I guess I can tackle that later.
>
>For now I can focus on the generic devlink path, move drivers to register
>devlink only after the device is ready. Then devlink core can apply the default
>before exposing the instance to userspace.
>
>I think it is better to fix the ordering for all devlink drivers, not only the
>ones that support eswitch mode set. That gives us a consistent model and makes
>future defaults easier.
>
>Reload can be handled from devlink after successful DRIVER_REINIT.
>
>Does this sound ok?
Yes. Thanks!
>
>Mark
>
>>
>>>
>>> Then mlx5 would look more or less like:
>>> devl_lock(devlink);
>>> devl_dev_reinit_begin(devlink);
>>> ret = mlx5_load_one_devl_locked(dev, recovery);
>>> if (!ret)
>>> devl_dev_reinit_end(devlink);
>>> else
>>> devl_dev_reinit_abort(devlink);
>>> devl_unlock(devlink);
>>>
>>> This gives devlink core a way to know that the devlink instance is registered,
>>> but should not be used by userspace at the moment. It also allows keeping the
>>> default/config apply logic in devlink instead of adding driver specific calls
>>> to apply it in each init path.
>>>
>>> But this still means the generic solution needs some driver help. Drivers need
>>> to register devlink at a point where the post-register default apply is safe,
>>> and full reinit paths need to be marked with this begin/end API.
>>>
>>>>
>>>>
>>>>>
>>>>> Also, the bigger issue for mlx5 is not only initial registration or devlink
>>>>> reload. Some recovery paths, pci resume, and fw reset flows rebuild the driver
>>>>> state without going through devlink at all. I did not find a clean way for
>>>>> devlink core to infer all those points by itself.
>>>>
>>>> If you don't obey current configuration for example in pci resume, it is
>>>> bug and you should fix it. All these flows should obey current eswitch
>>>> mode configuration.
>>>>
>>>
>>> I agree that the device should come back according
>>> to the intended high level policy. But I don't think full reinit can be treated
>>> as restoring the whole previous runtime state. There may be user created
>>> steering rules and other objects which the driver cannot keep or replay. Today
>>> full reinit brings the device back to a clean initialized state, and that is
>>> intentional.
>>>
>>> So the split I have in mind is:
>>>
>>> - full runtime state is not preserved across full reinit;
>>> - high level devlink policy/configuration should be applied when the device is
>>> initialized again;
>>> - the command line default should not blindly override a later explicit
>>> userspace eswitch mode selection.
>>>
>>> I am not against moving this into devlink core, and I am willing to work on it.
>>>
>>> But before I rework the series, I want to make sure we agree on the direction.
>>> As I see it, doing this cleanly needs a devlink state like "registered but
>>> unavailable/in reinit", plus driver annotations for the reinit paths.
>>>
>>> If this is not the direction you want, I prefer to know now rather than spend
>>> time on a version that will be rejected anyway.
>>>
>>> Mark
>>>
>>>>
>>>>>
>>>>> To handle that from devlink I would still need to add some api for the driver
>>>>> to tell devlink "I just reinitialized, apply the default now". but nce I had
>>>>> that driver call , it felt simpler and clearer to let the driver call
>>>>> the helper directly at the points where it knows eswitch mode is safe.
>>>>>
>>>>> I agree that handling all of this inside devlink would be the better option.
>>>>> I just couldn't make it work in a clean way.
>>>>>
>>>>> Mark
>>>>>
>>>>>>
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also, since this knob is only about eswitch mode, I don't think we need to
>>>>>>>>> touch every devlink driver. Drivers that don't implement eswitch_mode_set()
>>>>>>>>> would just ignore it anyway. The follow-up only wires the default into
>>>>>>>>> drivers that actually support changing eswitch mode.
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> return 0;
>>>>>>>>>>>
>>>>>>>>>>> err_register:
>>>>>>>>>>> @@ -1538,6 +1554,7 @@ int mlx5_load_one_devl_locked(struct mlx5_core_dev *dev, bool recovery)
>>>>>>>>>>> goto err_attach;
>>>>>>>>>>>
>>>>>>>>>>> mutex_unlock(&dev->intf_state_mutex);
>>>>>>>>>>> + mlx5_devl_apply_default_esw_mode(dev);
>>>>>>>>>>> return 0;
>>>>>>>>>>>
>>>>>>>>>>> err_attach:
>>>>>>>>>>> --
>>>>>>>>>>> 2.44.0
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
^ permalink raw reply
* [PATCH v14 5/6] iio: adc: ad4691: add oversampling support
From: Radu Sabau via B4 Relay @ 2026-05-29 10:15 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau
In-Reply-To: <20260529-ad4692-multichannel-sar-adc-driver-v14-0-e93c2747dc1f@analog.com>
From: Radu Sabau <radu.sabau@analog.com>
Add oversampling ratio (OSR) support for CNV burst mode. The accumulator
depth register (ACC_DEPTH_IN(0)) is programmed with the selected OSR at
buffer enable time and before each single-shot read.
Supported OSR values: 1, 2, 4, 8, 16, 32.
Introduce AD4691_MANUAL_CHANNEL() for manual mode channels, which do
not expose the oversampling_ratio attribute since OSR is not applicable
in that mode. A separate manual_channels array is added to
struct ad4691_channel_info and selected at probe time.
The OSR is shared across all channels (in_voltage_sampling_frequency
and in_voltage_oversampling_ratio are info_mask_shared_by_all) because
the chip has one internal oscillator and a single accumulator depth
register (ACC_DEPTH_IN(0)) for all channels.
in_voltage_sampling_frequency represents the effective output rate,
defined as osc_freq / osr. Writing it computes needed_osc = freq * osr
and snaps down to the largest oscillator table entry that satisfies both
osc <= needed_osc and osc % osr == 0, guaranteeing an exact integer
read-back. The result is stored in target_osc_freq_Hz and written to
OSC_FREQ_REG at buffer enable and single-shot time, so sampling_frequency
and oversampling_ratio can be set in any order.
in_voltage_sampling_frequency_available is precomputed at probe for
each OSR value, listing only oscillator table entries that divide
evenly by that OSR, expressed as effective rates (osc_freq / osr).
The list becomes sparser as OSR increases, capping at max_rate / osr.
read_avail picks the precomputed list for the current OSR, making the
returned pointer stable and race-free.
Writing oversampling_ratio stores the new shared OSR and snaps
target_osc_freq_Hz to the largest oscillator table entry that is both
<= old_effective_rate * new_osr and evenly divisible by new_osr. This
preserves an integer read-back of in_voltage_sampling_frequency after
the OSR change while keeping the oscillator as close as possible to the
previous effective rate.
OSR defaults to 1 (no accumulation).
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
drivers/iio/adc/ad4691.c | 357 +++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 327 insertions(+), 30 deletions(-)
diff --git a/drivers/iio/adc/ad4691.c b/drivers/iio/adc/ad4691.c
index b1d1b42f724e..043143ae9560 100644
--- a/drivers/iio/adc/ad4691.c
+++ b/drivers/iio/adc/ad4691.c
@@ -122,6 +122,7 @@ enum ad4691_ref_ctrl {
struct ad4691_channel_info {
const struct iio_chan_spec *channels __counted_by_ptr(num_channels);
+ const struct iio_chan_spec *manual_channels __counted_by_ptr(num_channels);
unsigned int num_channels;
};
@@ -132,7 +133,34 @@ struct ad4691_chip_info {
const struct ad4691_channel_info *offload_info;
};
+/* CNV burst mode channel — exposes oversampling ratio. */
#define AD4691_CHANNEL(ch) \
+ { \
+ .type = IIO_VOLTAGE, \
+ .indexed = 1, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW), \
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SCALE) \
+ | BIT(IIO_CHAN_INFO_SAMP_FREQ) \
+ | BIT(IIO_CHAN_INFO_OVERSAMPLING_RATIO), \
+ .info_mask_shared_by_all_available = \
+ BIT(IIO_CHAN_INFO_SAMP_FREQ) \
+ | BIT(IIO_CHAN_INFO_OVERSAMPLING_RATIO), \
+ .channel = ch, \
+ .scan_index = ch, \
+ .scan_type = { \
+ .format = 'u', \
+ .realbits = 16, \
+ .storagebits = 16, \
+ .endianness = IIO_BE, \
+ }, \
+ }
+
+/*
+ * Manual mode channel — no oversampling ratio attribute. OSR is not
+ * supported in manual mode; ACC_DEPTH_IN is not configured during manual
+ * buffer enable.
+ */
+#define AD4691_MANUAL_CHANNEL(ch) \
{ \
.type = IIO_VOLTAGE, \
.indexed = 1, \
@@ -156,8 +184,33 @@ struct ad4691_chip_info {
* bits into native 16-bit words before DMA, so samples are in
* CPU-native byte order (IIO_CPU). storagebits=16 matches the 16-bit
* DMA word size.
+ *
+ * CNV burst offload configures ACC_DEPTH_IN per channel, so the
+ * oversampling_ratio attribute is exposed. Manual offload does not;
+ * use AD4691_OFFLOAD_MANUAL_CHANNEL for that path.
*/
#define AD4691_OFFLOAD_CHANNEL(ch) \
+ { \
+ .type = IIO_VOLTAGE, \
+ .indexed = 1, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW), \
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SCALE) \
+ | BIT(IIO_CHAN_INFO_SAMP_FREQ) \
+ | BIT(IIO_CHAN_INFO_OVERSAMPLING_RATIO), \
+ .info_mask_shared_by_all_available = \
+ BIT(IIO_CHAN_INFO_SAMP_FREQ) \
+ | BIT(IIO_CHAN_INFO_OVERSAMPLING_RATIO), \
+ .channel = ch, \
+ .scan_index = ch, \
+ .scan_type = { \
+ .format = 'u', \
+ .realbits = 16, \
+ .storagebits = 16, \
+ }, \
+ }
+
+/* Manual offload — same IIO_CPU layout but no oversampling_ratio attribute. */
+#define AD4691_OFFLOAD_MANUAL_CHANNEL(ch) \
{ \
.type = IIO_VOLTAGE, \
.indexed = 1, \
@@ -241,23 +294,91 @@ static const struct iio_chan_spec ad4693_offload_channels[] = {
AD4691_OFFLOAD_CHANNEL(7),
};
+static const struct iio_chan_spec ad4691_manual_channels[] = {
+ AD4691_MANUAL_CHANNEL(0),
+ AD4691_MANUAL_CHANNEL(1),
+ AD4691_MANUAL_CHANNEL(2),
+ AD4691_MANUAL_CHANNEL(3),
+ AD4691_MANUAL_CHANNEL(4),
+ AD4691_MANUAL_CHANNEL(5),
+ AD4691_MANUAL_CHANNEL(6),
+ AD4691_MANUAL_CHANNEL(7),
+ AD4691_MANUAL_CHANNEL(8),
+ AD4691_MANUAL_CHANNEL(9),
+ AD4691_MANUAL_CHANNEL(10),
+ AD4691_MANUAL_CHANNEL(11),
+ AD4691_MANUAL_CHANNEL(12),
+ AD4691_MANUAL_CHANNEL(13),
+ AD4691_MANUAL_CHANNEL(14),
+ AD4691_MANUAL_CHANNEL(15),
+ IIO_CHAN_SOFT_TIMESTAMP(16),
+};
+
+static const struct iio_chan_spec ad4693_manual_channels[] = {
+ AD4691_MANUAL_CHANNEL(0),
+ AD4691_MANUAL_CHANNEL(1),
+ AD4691_MANUAL_CHANNEL(2),
+ AD4691_MANUAL_CHANNEL(3),
+ AD4691_MANUAL_CHANNEL(4),
+ AD4691_MANUAL_CHANNEL(5),
+ AD4691_MANUAL_CHANNEL(6),
+ AD4691_MANUAL_CHANNEL(7),
+ IIO_CHAN_SOFT_TIMESTAMP(8),
+};
+
+static const struct iio_chan_spec ad4691_offload_manual_channels[] = {
+ AD4691_OFFLOAD_MANUAL_CHANNEL(0),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(1),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(2),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(3),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(4),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(5),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(6),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(7),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(8),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(9),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(10),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(11),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(12),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(13),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(14),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(15),
+};
+
+static const struct iio_chan_spec ad4693_offload_manual_channels[] = {
+ AD4691_OFFLOAD_MANUAL_CHANNEL(0),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(1),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(2),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(3),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(4),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(5),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(6),
+ AD4691_OFFLOAD_MANUAL_CHANNEL(7),
+};
+
+static const int ad4691_oversampling_ratios[] = { 1, 2, 4, 8, 16, 32 };
+
static const struct ad4691_channel_info ad4691_sw_info = {
.channels = ad4691_channels,
+ .manual_channels = ad4691_manual_channels,
.num_channels = ARRAY_SIZE(ad4691_channels),
};
static const struct ad4691_channel_info ad4693_sw_info = {
.channels = ad4693_channels,
+ .manual_channels = ad4693_manual_channels,
.num_channels = ARRAY_SIZE(ad4693_channels),
};
static const struct ad4691_channel_info ad4691_offload_info = {
.channels = ad4691_offload_channels,
+ .manual_channels = ad4691_offload_manual_channels,
.num_channels = ARRAY_SIZE(ad4691_offload_channels),
};
static const struct ad4691_channel_info ad4693_offload_info = {
.channels = ad4693_offload_channels,
+ .manual_channels = ad4693_offload_manual_channels,
.num_channels = ARRAY_SIZE(ad4693_offload_channels),
};
@@ -324,6 +445,25 @@ struct ad4691_state {
int irq;
int vref_uV;
u32 cnv_period_ns;
+ /*
+ * Snapped oscillator frequency (Hz) shared by all channels. Set when
+ * sampling_frequency or oversampling_ratio is written; written to
+ * OSC_FREQ_REG at buffer enable and single-shot time so both attributes
+ * can be set in any order. Reading in_voltage_sampling_frequency
+ * returns target_osc_freq_Hz / osr — the effective rate given the
+ * shared oversampling ratio.
+ */
+ u32 target_osc_freq_Hz;
+ /* Shared oversampling ratio across all channels; always 1 in manual mode. */
+ unsigned int osr;
+ /*
+ * Precomputed effective-rate lists, one row per entry in
+ * ad4691_oversampling_ratios[]. Populated at probe; read_avail picks
+ * the row for the current shared OSR. The tables are stable after
+ * probe so returning a pointer into them from read_avail is race-free.
+ */
+ int samp_freq_avail[ARRAY_SIZE(ad4691_oversampling_ratios)][ARRAY_SIZE(ad4691_osc_freqs_Hz)];
+ int samp_freq_avail_len[ARRAY_SIZE(ad4691_oversampling_ratios)];
bool manual_mode;
bool irq_enabled;
@@ -580,35 +720,95 @@ static unsigned int ad4691_samp_freq_start(const struct ad4691_chip_info *info)
return (info->max_rate == 1 * HZ_PER_MHZ) ? 0 : 1;
}
-static int ad4691_get_sampling_freq(struct ad4691_state *st, int *val)
+/*
+ * Find the largest oscillator table entry that is both <= needed_osc and
+ * evenly divisible by osr (guaranteeing an integer effective rate on
+ * read-back). Returns 0 if no such entry exists in the chip's valid range.
+ */
+static unsigned int ad4691_find_osc_freq(struct ad4691_state *st,
+ unsigned int needed_osc,
+ unsigned int osr)
{
- unsigned int reg_val;
- int ret;
+ unsigned int start = ad4691_samp_freq_start(st->info);
- guard(mutex)(&st->lock);
+ for (unsigned int i = start; i < ARRAY_SIZE(ad4691_osc_freqs_Hz); i++) {
+ if ((unsigned int)ad4691_osc_freqs_Hz[i] > needed_osc)
+ continue;
+ if (ad4691_osc_freqs_Hz[i] % osr)
+ continue;
+ return ad4691_osc_freqs_Hz[i];
+ }
+ return 0;
+}
- ret = regmap_read(st->regmap, AD4691_OSC_FREQ_REG, ®_val);
- if (ret)
- return ret;
+/* Write target_osc_freq_Hz to OSC_FREQ_REG. Called at use time. */
+static int ad4691_write_osc_freq(struct ad4691_state *st)
+{
+ for (unsigned int i = 0; i < ARRAY_SIZE(ad4691_osc_freqs_Hz); i++) {
+ if (ad4691_osc_freqs_Hz[i] == st->target_osc_freq_Hz)
+ return regmap_write(st->regmap, AD4691_OSC_FREQ_REG, i);
+ }
+ return -EINVAL;
+}
- *val = ad4691_osc_freqs_Hz[FIELD_GET(AD4691_OSC_FREQ_MASK, reg_val)];
- return IIO_VAL_INT;
+/* Return the index of osr in ad4691_oversampling_ratios[], defaulting to 0. */
+static unsigned int ad4691_osr_index(unsigned int osr)
+{
+ for (unsigned int i = 0; i < ARRAY_SIZE(ad4691_oversampling_ratios) - 1; i++) {
+ if ((unsigned int)ad4691_oversampling_ratios[i] == osr)
+ return i;
+ }
+ return ARRAY_SIZE(ad4691_oversampling_ratios) - 1;
+}
+
+/*
+ * Precompute samp_freq_avail[][]: for each OSR value, list the oscillator
+ * table entries that divide evenly by that OSR, expressed as effective rates
+ * (osc_freq / osr). Called once at probe after st->info is set.
+ */
+static void ad4691_precompute_samp_freq_avail(struct ad4691_state *st)
+{
+ unsigned int start = ad4691_samp_freq_start(st->info);
+
+ for (unsigned int i = 0; i < ARRAY_SIZE(ad4691_oversampling_ratios); i++) {
+ unsigned int osr = ad4691_oversampling_ratios[i];
+ int n = 0;
+
+ for (unsigned int j = start; j < ARRAY_SIZE(ad4691_osc_freqs_Hz); j++) {
+ if (ad4691_osc_freqs_Hz[j] % osr)
+ continue;
+ st->samp_freq_avail[i][n++] = ad4691_osc_freqs_Hz[j] / osr;
+ }
+ st->samp_freq_avail_len[i] = n;
+ }
}
static int ad4691_set_sampling_freq(struct ad4691_state *st, int freq)
{
- unsigned int start = ad4691_samp_freq_start(st->info);
+ unsigned int osr, found;
+ /*
+ * Read osr under st->lock: osr and target_osc_freq_Hz are modified
+ * together under the lock; reading after acquiring it ensures we see
+ * a consistent snapshot with no concurrent write racing us.
+ */
guard(mutex)(&st->lock);
+ osr = st->osr;
- for (unsigned int i = start; i < ARRAY_SIZE(ad4691_osc_freqs_Hz); i++) {
- if (ad4691_osc_freqs_Hz[i] != freq)
- continue;
- return regmap_update_bits(st->regmap, AD4691_OSC_FREQ_REG,
- AD4691_OSC_FREQ_MASK, i);
- }
+ if (freq <= 0 || (unsigned int)freq > st->info->max_rate / osr)
+ return -EINVAL;
- return -EINVAL;
+ found = ad4691_find_osc_freq(st, (unsigned int)freq * osr, osr);
+ if (!found)
+ return -EINVAL;
+
+ /*
+ * Store the snapped oscillator frequency; OSC_FREQ_REG is written at
+ * buffer enable and single-shot time so that sampling_frequency and
+ * oversampling_ratio can be set in any order.
+ */
+ st->target_osc_freq_Hz = found;
+ return 0;
}
static int ad4691_read_avail(struct iio_dev *indio_dev,
@@ -617,13 +817,27 @@ static int ad4691_read_avail(struct iio_dev *indio_dev,
int *length, long mask)
{
struct ad4691_state *st = iio_priv(indio_dev);
- unsigned int start = ad4691_samp_freq_start(st->info);
switch (mask) {
- case IIO_CHAN_INFO_SAMP_FREQ:
- *vals = &ad4691_osc_freqs_Hz[start];
+ case IIO_CHAN_INFO_SAMP_FREQ: {
+ unsigned int osr_idx;
+
+ /*
+ * The precomputed tables are stable after probe; only the
+ * current OSR needs to be read under the lock to pick the
+ * right row atomically.
+ */
+ guard(mutex)(&st->lock);
+ osr_idx = ad4691_osr_index(st->osr);
+ *vals = st->samp_freq_avail[osr_idx];
*type = IIO_VAL_INT;
- *length = ARRAY_SIZE(ad4691_osc_freqs_Hz) - start;
+ *length = st->samp_freq_avail_len[osr_idx];
+ return IIO_AVAIL_LIST;
+ }
+ case IIO_CHAN_INFO_OVERSAMPLING_RATIO:
+ *vals = ad4691_oversampling_ratios;
+ *type = IIO_VAL_INT;
+ *length = ARRAY_SIZE(ad4691_oversampling_ratios);
return IIO_AVAIL_LIST;
default:
return -EINVAL;
@@ -634,7 +848,7 @@ static int ad4691_single_shot_read(struct iio_dev *indio_dev,
struct iio_chan_spec const *chan, int *val)
{
struct ad4691_state *st = iio_priv(indio_dev);
- unsigned int reg_val, osc_idx, period_us;
+ unsigned int reg_val, period_us;
int ret;
guard(mutex)(&st->lock);
@@ -654,7 +868,11 @@ static int ad4691_single_shot_read(struct iio_dev *indio_dev,
if (ret)
return ret;
- ret = regmap_read(st->regmap, AD4691_OSC_FREQ_REG, ®_val);
+ ret = regmap_write(st->regmap, AD4691_ACC_DEPTH_IN(0), st->osr);
+ if (ret)
+ return ret;
+
+ ret = ad4691_write_osc_freq(st);
if (ret)
return ret;
@@ -662,9 +880,12 @@ static int ad4691_single_shot_read(struct iio_dev *indio_dev,
if (ret)
return ret;
- osc_idx = FIELD_GET(AD4691_OSC_FREQ_MASK, reg_val);
- /* Wait 2 oscillator periods for the conversion to complete. */
- period_us = DIV_ROUND_UP(2UL * USEC_PER_SEC, ad4691_osc_freqs_Hz[osc_idx]);
+ /*
+ * Wait osr + 1 oscillator periods: osr for accumulation, +1 for the
+ * pipeline margin (one extra period ensures the final result is ready).
+ */
+ period_us = DIV_ROUND_UP((st->osr + 1) * USEC_PER_SEC,
+ st->target_osc_freq_Hz);
fsleep(period_us);
ret = regmap_write(st->regmap, AD4691_OSC_EN_REG, 0);
@@ -698,8 +919,22 @@ static int ad4691_read_raw(struct iio_dev *indio_dev,
return ad4691_single_shot_read(indio_dev, chan, val);
}
- case IIO_CHAN_INFO_SAMP_FREQ:
- return ad4691_get_sampling_freq(st, val);
+ case IIO_CHAN_INFO_SAMP_FREQ: {
+ /*
+ * Read target_osc_freq_Hz and osr under st->lock to get a
+ * consistent snapshot: write_raw for SAMP_FREQ or OSR modifies
+ * both fields under the lock, so a concurrent read without the
+ * lock could observe a new oscillator frequency with the old OSR.
+ */
+ guard(mutex)(&st->lock);
+ *val = st->target_osc_freq_Hz / st->osr;
+ return IIO_VAL_INT;
+ }
+ case IIO_CHAN_INFO_OVERSAMPLING_RATIO: {
+ guard(mutex)(&st->lock);
+ *val = st->osr;
+ return IIO_VAL_INT;
+ }
case IIO_CHAN_INFO_SCALE:
*val = st->vref_uV / (MICRO / MILLI);
*val2 = chan->scan_type.realbits;
@@ -722,6 +957,33 @@ static int ad4691_write_raw(struct iio_dev *indio_dev,
switch (mask) {
case IIO_CHAN_INFO_SAMP_FREQ:
return ad4691_set_sampling_freq(st, val);
+ case IIO_CHAN_INFO_OVERSAMPLING_RATIO: {
+ unsigned int old_effective, found, osr_idx;
+
+ osr_idx = ad4691_osr_index(val);
+ if (ad4691_oversampling_ratios[osr_idx] != val)
+ return -EINVAL;
+
+ /*
+ * Hold st->lock while computing the new oscillator frequency
+ * and updating both target_osc_freq_Hz and osr atomically:
+ * read_raw for SAMP_FREQ reads both fields under the lock and
+ * must see a consistent pair (new osc ↔ new osr).
+ *
+ * Snap target_osc_freq_Hz to the largest table entry that is
+ * both <= old_effective * new_osr and evenly divisible by
+ * new_osr, preserving an integer read-back of
+ * in_voltage_sampling_frequency after the OSR change.
+ */
+ guard(mutex)(&st->lock);
+ old_effective = st->target_osc_freq_Hz / st->osr;
+ found = ad4691_find_osc_freq(st, old_effective * (unsigned int)val, val);
+ if (!found)
+ return -EINVAL;
+ st->target_osc_freq_Hz = found;
+ st->osr = val;
+ return 0;
+ }
default:
return -EINVAL;
}
@@ -776,6 +1038,10 @@ static int ad4691_enter_conversion_mode(struct ad4691_state *st)
return regmap_update_bits(st->regmap, AD4691_DEVICE_SETUP,
AD4691_MANUAL_MODE, AD4691_MANUAL_MODE);
+ ret = ad4691_write_osc_freq(st);
+ if (ret)
+ return ret;
+
ret = regmap_update_bits(st->regmap, AD4691_ADC_SETUP,
AD4691_ADC_MODE_MASK, AD4691_CNV_BURST_MODE);
if (ret)
@@ -943,6 +1209,10 @@ static int ad4691_cnv_burst_buffer_preenable(struct iio_dev *indio_dev)
if (ret)
goto err_unoptimize;
+ ret = regmap_write(st->regmap, AD4691_ACC_DEPTH_IN(0), st->osr);
+ if (ret)
+ goto err_unoptimize;
+
ret = ad4691_enter_conversion_mode(st);
if (ret)
goto err_unoptimize;
@@ -1122,6 +1392,10 @@ static int ad4691_cnv_burst_offload_buffer_postenable(struct iio_dev *indio_dev)
if (ret)
return ret;
+ ret = regmap_write(st->regmap, AD4691_ACC_DEPTH_IN(0), st->osr);
+ if (ret)
+ return ret;
+
ret = ad4691_enter_conversion_mode(st);
if (ret)
return ret;
@@ -1538,11 +1812,15 @@ static int ad4691_config(struct ad4691_state *st)
if (ret)
return dev_err_probe(dev, ret, "Failed to write OSC_FREQ\n");
+ st->target_osc_freq_Hz = ad4691_osc_freqs_Hz[ad4691_samp_freq_start(st->info)];
+
ret = regmap_update_bits(st->regmap, AD4691_ADC_SETUP,
AD4691_ADC_MODE_MASK, AD4691_AUTONOMOUS_MODE);
if (ret)
return dev_err_probe(dev, ret, "Failed to write ADC_SETUP\n");
+ ad4691_precompute_samp_freq_avail(st);
+
return 0;
}
@@ -1554,7 +1832,14 @@ static int ad4691_setup_triggered_buffer(struct iio_dev *indio_dev,
unsigned int i;
int irq, ret;
- indio_dev->channels = st->info->sw_info->channels;
+ /*
+ * Manual mode exposes channels without the oversampling_ratio attribute
+ * because ACC_DEPTH_IN is not configured in manual mode.
+ */
+ if (st->manual_mode)
+ indio_dev->channels = st->info->sw_info->manual_channels;
+ else
+ indio_dev->channels = st->info->sw_info->channels;
indio_dev->num_channels = st->info->sw_info->num_channels;
indio_dev->info = st->manual_mode ? &ad4691_manual_info : &ad4691_cnv_burst_info;
@@ -1636,7 +1921,18 @@ static int ad4691_setup_offload(struct iio_dev *indio_dev,
st->offload = spi_offload;
- indio_dev->channels = st->info->offload_info->channels;
+ /*
+ * CNV burst offload exposes oversampling_ratio (ACC_DEPTH_IN is
+ * configured per channel at buffer enable). Manual offload does not
+ * configure ACC_DEPTH_IN, so it uses a separate channel array
+ * without the oversampling_ratio attribute. Both paths use IIO_CPU
+ * (no .endianness annotation) because bits_per_word=16 causes the
+ * SPI Engine to produce native 16-bit DMA words.
+ */
+ if (st->manual_mode)
+ indio_dev->channels = st->info->offload_info->manual_channels;
+ else
+ indio_dev->channels = st->info->offload_info->channels;
indio_dev->num_channels = st->info->offload_info->num_channels;
/*
* Offload path uses DMA directly; no IIO trigger is involved, so
@@ -1710,6 +2006,7 @@ static int ad4691_probe(struct spi_device *spi)
st->info = spi_get_device_match_data(spi);
if (!st->info)
return -ENODEV;
+ st->osr = 1;
ret = devm_mutex_init(dev, &st->lock);
if (ret)
--
2.43.0
^ permalink raw reply related
* [PATCH v14 6/6] docs: iio: adc: ad4691: add driver documentation
From: Radu Sabau via B4 Relay @ 2026-05-29 10:15 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau
In-Reply-To: <20260529-ad4692-multichannel-sar-adc-driver-v14-0-e93c2747dc1f@analog.com>
From: Radu Sabau <radu.sabau@analog.com>
Add RST documentation for the AD4691 family ADC driver covering
supported devices, IIO channels, operating modes, oversampling,
reference voltage, LDO supply, reset, GP pins, SPI offload support,
and buffer data format.
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
Documentation/iio/ad4691.rst | 227 +++++++++++++++++++++++++++++++++++++++++++
Documentation/iio/index.rst | 1 +
MAINTAINERS | 1 +
3 files changed, 229 insertions(+)
diff --git a/Documentation/iio/ad4691.rst b/Documentation/iio/ad4691.rst
new file mode 100644
index 000000000000..e45733341a4b
--- /dev/null
+++ b/Documentation/iio/ad4691.rst
@@ -0,0 +1,227 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+=============
+AD4691 driver
+=============
+
+ADC driver for Analog Devices Inc. AD4691 family of multichannel SAR ADCs.
+The module name is ``ad4691``.
+
+
+Supported devices
+=================
+
+The following chips are supported by this driver:
+
+* `AD4691 <https://www.analog.com/en/products/ad4691.html>`_ — 16-channel, 500 kSPS
+* `AD4692 <https://www.analog.com/en/products/ad4692.html>`_ — 16-channel, 1 MSPS
+* `AD4693 <https://www.analog.com/en/products/ad4693.html>`_ — 8-channel, 500 kSPS
+* `AD4694 <https://www.analog.com/en/products/ad4694.html>`_ — 8-channel, 1 MSPS
+
+
+IIO channels
+============
+
+Each physical ADC input maps to one IIO voltage channel. The AD4691 and AD4692
+expose 16 channels (``voltage0`` through ``voltage15``); the AD4693 and AD4694
+expose 8 channels (``voltage0`` through ``voltage7``).
+
+All channels share a common scale (``in_voltage_scale``), derived from the
+reference voltage. Each channel exposes:
+
+* ``in_voltageN_raw`` — single-shot ADC result
+
+The following attributes are shared across all channels:
+
+* ``in_voltage_sampling_frequency`` — effective output rate, defined as the
+ internal oscillator frequency divided by the oversampling ratio. Writing this
+ attribute selects the nearest achievable rate for the current OSR; the value
+ read back reflects the actual rate after snapping to the closest valid
+ oscillator entry.
+* ``in_voltage_sampling_frequency_available`` — list of achievable effective
+ rates for the current oversampling ratio. The list updates dynamically when
+ the oversampling ratio changes.
+
+The following attributes are shared across all channels and only available in
+CNV Burst Mode:
+
+* ``in_voltage_oversampling_ratio`` — hardware oversampling depth applied to
+ all channels; see `Oversampling`_ below.
+* ``in_voltage_oversampling_ratio_available`` — valid ratios: 1, 2, 4, 8, 16,
+ 32.
+
+
+Operating modes
+===============
+
+The driver supports two operating modes, selected automatically from the
+device tree at probe time.
+
+Manual Mode
+-----------
+
+Selected when no ``pwms`` property is present in the device tree. The CNV pin
+is tied to the SPI chip-select: every CS assertion triggers a conversion and
+returns the previous result. A user-defined IIO trigger (e.g. hrtimer trigger)
+drives the buffer.
+
+Oversampling is not supported in Manual Mode.
+
+CNV Burst Mode
+--------------
+
+Selected when a ``pwms`` property is present in the device tree. A PWM drives
+the CNV pin at the configured conversion rate. A GP pin wired to the SoC and
+declared in the device tree signals DATA_READY at the end of each burst,
+triggering a readout of all active channel results into the IIO buffer.
+
+The buffer output rate is controlled by the ``sampling_frequency`` attribute
+on the IIO buffer. In practice the PWM rate should be set low enough to allow
+the SPI readout to complete before the next conversion burst begins.
+
+Autonomous Mode (idle / single-shot)
+-------------------------------------
+
+When the IIO buffer is disabled, ``in_voltageN_raw`` reads perform a single
+conversion on the requested channel using the internal oscillator. The
+oscillator is started and stopped around each read to save power.
+
+
+Oversampling
+============
+
+In CNV Burst Mode a shared hardware accumulator averages a configurable number
+of successive conversions across all active channels. The result is always a
+16-bit mean, so the buffer data type (shown in ``buffer0/in_voltageN_type``)
+is unaffected by the oversampling ratio. Valid ratios are 1, 2, 4, 8, 16 and
+32; the default is 1 (no averaging). Oversampling is not supported in Manual
+Mode.
+
+.. code-block:: bash
+
+ # Set oversampling ratio to 16 (shared across all channels)
+ echo 16 > /sys/bus/iio/devices/iio:device0/in_voltage_oversampling_ratio
+
+ # Read the resulting effective sampling frequency
+ cat /sys/bus/iio/devices/iio:device0/in_voltage_sampling_frequency
+
+Writing ``in_voltage_oversampling_ratio`` stores the new shared depth and snaps
+the internal oscillator to the largest valid table entry that is both less than
+or equal to ``old_effective_rate × new_osr`` and evenly divisible by
+``new_osr``. This preserves an integer read-back of
+``in_voltage_sampling_frequency`` after the change and keeps the oscillator as
+close as possible to the previous effective rate.
+
+
+Reference voltage
+=================
+
+The driver supports two reference configurations, mutually exclusive:
+
+* **External reference** (``ref-supply``): a voltage between 2.4 V and 5.25 V
+ supplied externally.
+* **Buffered internal reference** (``refin-supply``): an internal reference
+ buffer is enabled by the driver.
+
+Exactly one of ``ref-supply`` or ``refin-supply`` must be present in the
+device tree. The reference voltage determines the full-scale range reported
+via ``in_voltage_scale``.
+
+
+LDO supply
+==========
+
+The chip contains an internal LDO that powers part of the analog front-end.
+The supply configuration is mutually exclusive:
+
+* **External VDD** (``vdd-supply``): an external 1.8 V supply is used directly;
+ the internal LDO is disabled.
+* **Internal LDO** (``ldo-in-supply``): the internal LDO is enabled and fed
+ from the ``ldo-in`` regulator. Use this when no external 1.8 V VDD is present.
+
+Exactly one of ``vdd-supply`` or ``ldo-in-supply`` must be provided.
+
+
+Reset
+=====
+
+The driver supports two reset mechanisms:
+
+* **Hardware reset** (``reset-gpios`` in device tree): the GPIO line is
+ asserted then deasserted at probe; the driver waits 300 µs for the chip
+ to complete its internal reset sequence before accepting SPI commands.
+* **Software reset** (fallback when ``reset-gpios`` is absent): written
+ automatically at probe.
+
+
+GP pins and interrupts
+======================
+
+The chip exposes up to four general-purpose (GP) pins. In CNV Burst Mode
+(non-offload), one GP pin must be wired to an interrupt-capable SoC input and
+declared in the device tree using the ``interrupts`` and ``interrupt-names``
+properties. The ``interrupt-names`` value identifies which GP pin is used
+(``"gp0"`` through ``"gp3"``).
+
+Example device tree fragment::
+
+ adc@0 {
+ compatible = "adi,ad4692";
+ ...
+ interrupt-parent = <&gpio0>;
+ interrupts = <17 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "gp0";
+ };
+
+
+SPI offload support
+===================
+
+When a SPI offload engine (e.g. the AXI SPI Engine) is present, the driver
+uses DMA-backed transfers for CPU-independent, high-throughput data capture.
+SPI offload is detected automatically at probe; if no offload hardware is
+available the driver falls back to the software triggered-buffer path.
+
+Two SPI offload sub-modes exist:
+
+CNV Burst offload
+-----------------
+
+Used when a ``pwms`` property is present and SPI offload is available. The PWM
+drives CNV at the configured rate; on DATA_READY the offload engine reads all
+active channel results and streams them directly to the IIO DMA buffer with no
+CPU involvement. The GP pin used as DATA_READY trigger is supplied by the
+trigger-source consumer at buffer enable time; no ``interrupt-names`` entry is
+required.
+
+Manual offload
+--------------
+
+Used when no ``pwms`` property is present and SPI offload is available. A
+periodic SPI offload trigger controls the conversion rate and the offload engine
+streams results directly to the IIO DMA buffer.
+
+The ``sampling_frequency`` attribute on the IIO buffer controls the trigger
+rate (in Hz). The initial rate is 100 kHz.
+
+Oversampling is not supported in Manual Mode.
+
+
+Buffer data format
+==================
+
+The sample format in the IIO buffer depends on whether SPI offload is in use.
+
+Software triggered-buffer path (no SPI offload)
+------------------------------------------------
+
+Each active channel occupies one 16-bit big-endian slot (``storagebits=16``,
+``endianness=be``). Active channels are packed densely in scan-index order,
+followed by a 64-bit software timestamp appended by the IIO core.
+
+SPI offload path
+----------------
+
+Each active channel occupies one 16-bit CPU-native slot (``storagebits=16``,
+``endianness=cpu``). The SPI offload engine streams 16-bit words directly from
+the SPI Engine into the DMA buffer; no software timestamp is appended.
diff --git a/Documentation/iio/index.rst b/Documentation/iio/index.rst
index ba3e609c6a13..007e0a1fcc5a 100644
--- a/Documentation/iio/index.rst
+++ b/Documentation/iio/index.rst
@@ -23,6 +23,7 @@ Industrial I/O Kernel Drivers
ad4000
ad4030
ad4062
+ ad4691
ad4695
ad7191
ad7380
diff --git a/MAINTAINERS b/MAINTAINERS
index 020c1ffae31b..3fbac296b667 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1488,6 +1488,7 @@ L: linux-iio@vger.kernel.org
S: Supported
W: https://ez.analog.com/linux-software-drivers
F: Documentation/devicetree/bindings/iio/adc/adi,ad4691.yaml
+F: Documentation/iio/ad4691.rst
F: drivers/iio/adc/ad4691.c
ANALOG DEVICES INC AD4695 DRIVER
--
2.43.0
^ permalink raw reply related
* [PATCH v14 3/6] iio: adc: ad4691: add triggered buffer support
From: Radu Sabau via B4 Relay @ 2026-05-29 10:15 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau
In-Reply-To: <20260529-ad4692-multichannel-sar-adc-driver-v14-0-e93c2747dc1f@analog.com>
From: Radu Sabau <radu.sabau@analog.com>
Add buffered capture support using the IIO triggered buffer framework.
CNV Burst Mode: the GP pin identified by interrupt-names in the device
tree is configured as DATA_READY output. The IRQ handler stops
conversions and fires the IIO trigger; the trigger handler executes a
pre-built SPI message that reads all active channels from the AVG_IN
accumulator registers and then resets accumulator state and restarts
conversions for the next cycle.
Manual Mode: CNV is tied to SPI CS so each transfer simultaneously
reads the previous result and starts the next conversion (pipelined
N+1 scheme). At preenable time a pre-built, optimised SPI message of
N+1 transfers is constructed (N channel reads plus one NOOP to drain
the pipeline). The trigger handler executes the message in a single
spi_sync() call and collects the results. An external trigger (e.g.
iio-trig-hrtimer) is required to drive the trigger at the desired
sample rate.
Both modes share the same trigger handler and push a complete scan —
one big-endian 16-bit (__be16) slot per active channel, densely packed
in scan_index order, followed by a timestamp.
The CNV Burst Mode sampling frequency (PWM period) is exposed as a
buffer-level attribute via IIO_DEVICE_ATTR.
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
drivers/iio/adc/Kconfig | 2 +
drivers/iio/adc/ad4691.c | 574 ++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 572 insertions(+), 4 deletions(-)
diff --git a/drivers/iio/adc/Kconfig b/drivers/iio/adc/Kconfig
index c9aca0d08e41..1d7afde108c0 100644
--- a/drivers/iio/adc/Kconfig
+++ b/drivers/iio/adc/Kconfig
@@ -148,6 +148,8 @@ config AD4691
tristate "Analog Devices AD4691 Family ADC Driver"
depends on SPI
depends on REGULATOR || COMPILE_TEST
+ select IIO_BUFFER
+ select IIO_TRIGGERED_BUFFER
select REGMAP
help
Say yes here to build support for Analog Devices AD4691 Family MuxSAR
diff --git a/drivers/iio/adc/ad4691.c b/drivers/iio/adc/ad4691.c
index e1febf80f21d..175c80129786 100644
--- a/drivers/iio/adc/ad4691.c
+++ b/drivers/iio/adc/ad4691.c
@@ -11,19 +11,29 @@
#include <linux/dev_printk.h>
#include <linux/device/devres.h>
#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kstrtox.h>
#include <linux/limits.h>
#include <linux/math.h>
#include <linux/module.h>
#include <linux/mod_devicetable.h>
+#include <linux/property.h>
+#include <linux/pwm.h>
#include <linux/regmap.h>
#include <linux/regulator/consumer.h>
#include <linux/reset.h>
+#include <linux/string.h>
#include <linux/spi/spi.h>
#include <linux/types.h>
#include <linux/units.h>
#include <linux/unaligned.h>
+#include <linux/iio/buffer.h>
#include <linux/iio/iio.h>
+#include <linux/iio/sysfs.h>
+#include <linux/iio/trigger.h>
+#include <linux/iio/triggered_buffer.h>
+#include <linux/iio/trigger_consumer.h>
#define AD4691_VREF_uV_MIN 2400000
#define AD4691_VREF_uV_MAX 5250000
@@ -57,6 +67,7 @@
#define AD4691_NOOP 0x00
#define AD4691_ADC_CHAN(ch) ((0x10 + (ch)) << 3)
+#define AD4691_EXIT_COMMAND 0x5000
#define AD4691_OSC_EN_REG 0x180
#define AD4691_STATE_RESET_REG 0x181
@@ -120,8 +131,12 @@ struct ad4691_chip_info {
.info_mask_shared_by_all_available = \
BIT(IIO_CHAN_INFO_SAMP_FREQ), \
.channel = ch, \
+ .scan_index = ch, \
.scan_type = { \
+ .format = 'u', \
.realbits = 16, \
+ .storagebits = 16, \
+ .endianness = IIO_BE, \
}, \
}
@@ -142,6 +157,7 @@ static const struct iio_chan_spec ad4691_channels[] = {
AD4691_CHANNEL(13),
AD4691_CHANNEL(14),
AD4691_CHANNEL(15),
+ IIO_CHAN_SOFT_TIMESTAMP(16),
};
static const struct iio_chan_spec ad4693_channels[] = {
@@ -153,6 +169,7 @@ static const struct iio_chan_spec ad4693_channels[] = {
AD4691_CHANNEL(5),
AD4691_CHANNEL(6),
AD4691_CHANNEL(7),
+ IIO_CHAN_SOFT_TIMESTAMP(8),
};
static const struct ad4691_channel_info ad4691_sw_info = {
@@ -189,6 +206,8 @@ static const int ad4691_osc_freqs_Hz[] = {
[0xF] = 1250,
};
+static const char * const ad4691_gp_names[] = { "gp0", "gp1", "gp2", "gp3" };
+
static const struct ad4691_chip_info ad4691_chip_info = {
.name = "ad4691",
.max_rate = 500 * HZ_PER_KHZ,
@@ -218,8 +237,13 @@ struct ad4691_state {
struct regmap *regmap;
struct spi_device *spi;
+ struct pwm_device *conv_trigger;
+ int irq;
int vref_uV;
+ u32 cnv_period_ns;
+ bool manual_mode;
+ bool irq_enabled;
bool refbuf_en;
bool ldo_en;
/*
@@ -227,8 +251,56 @@ struct ad4691_state {
* atomicity of consecutive SPI operations.
*/
struct mutex lock;
+ /*
+ * Per-buffer-enable lifetime resources:
+ * Manual Mode - a pre-built SPI message that clocks out N+1
+ * transfers in one go.
+ * CNV Burst Mode - a pre-built SPI message that clocks out 2*N
+ * transfers in one go.
+ */
+ struct spi_message scan_msg;
+ /*
+ * max 16 + 1 NOOP (manual) or 2*16 + 1 state-reset (CNV burst).
+ */
+ struct spi_transfer scan_xfers[34];
+ /*
+ * CNV burst: 16 AVG_IN addresses = 16. Manual: 16 channel cmds +
+ * 1 NOOP = 17. Stored as native u16; put_unaligned_be16() fills each
+ * slot so the SPI controller (bits_per_word=8) sends bytes MSB-first.
+ */
+ u16 scan_tx[17] __aligned(IIO_DMA_MINALIGN);
+ /*
+ * CNV burst state-reset: 4-byte write [addr_hi, addr_lo,
+ * STATE_RESET_ALL, OSC_EN=1]. CS is asserted throughout, so
+ * ADDR_DESCENDING writes byte[3]=1 to OSC_EN_REG (0x180) as a
+ * deliberate side-write, keeping the oscillator enabled. Shared
+ * with the offload path (mutually exclusive at probe).
+ */
+ u8 scan_tx_reset[4] __aligned(IIO_DMA_MINALIGN);
+ /*
+ * Scan buffer: one BE16 slot per active channel, plus timestamp.
+ * DMA-aligned because scan_xfers point rx_buf directly into vals[].
+ */
+ IIO_DECLARE_DMA_BUFFER_WITH_TS(__be16, vals, 16);
};
+/*
+ * Configure the given GP pin (0-3) as DATA_READY output.
+ * GP0/GP1 → GPIO_MODE1_REG, GP2/GP3 → GPIO_MODE2_REG.
+ * Even pins occupy bits [3:0], odd pins bits [7:4].
+ */
+static int ad4691_gpio_setup(struct ad4691_state *st, unsigned int gp_num)
+{
+ unsigned int bit_off = gp_num % 2;
+ unsigned int reg_off = gp_num / 2;
+ unsigned int shift = 4 * bit_off;
+
+ return regmap_update_bits(st->regmap,
+ AD4691_GPIO_MODE1_REG + reg_off,
+ AD4691_GP_MODE_MASK << shift,
+ AD4691_GP_MODE_DATA_READY << shift);
+}
+
static int ad4691_reg_read(void *context, unsigned int reg, unsigned int *val)
{
struct spi_device *spi = context;
@@ -539,13 +611,411 @@ static int ad4691_reg_access(struct iio_dev *indio_dev, unsigned int reg,
return regmap_write(st->regmap, reg, writeval);
}
-static const struct iio_info ad4691_info = {
+static int ad4691_set_pwm_freq(struct ad4691_state *st, unsigned int freq)
+{
+ if (!freq)
+ return -EINVAL;
+
+ st->cnv_period_ns = DIV_ROUND_UP(NSEC_PER_SEC, freq);
+ return 0;
+}
+
+static int ad4691_sampling_enable(struct ad4691_state *st, bool enable)
+{
+ struct pwm_state conv_state = {
+ .period = st->cnv_period_ns,
+ .duty_cycle = AD4691_CNV_DUTY_CYCLE_NS,
+ .polarity = PWM_POLARITY_NORMAL,
+ .enabled = enable,
+ };
+
+ return pwm_apply_might_sleep(st->conv_trigger, &conv_state);
+}
+
+/*
+ * ad4691_enter_conversion_mode - Switch the chip to its buffer conversion mode.
+ *
+ * Configures the ADC hardware registers for the mode selected at probe
+ * (CNV_BURST or MANUAL). Called from buffer preenable before starting
+ * sampling. The chip is in AUTONOMOUS mode during idle (for read_raw).
+ */
+static int ad4691_enter_conversion_mode(struct ad4691_state *st)
+{
+ int ret;
+
+ if (st->manual_mode)
+ return regmap_update_bits(st->regmap, AD4691_DEVICE_SETUP,
+ AD4691_MANUAL_MODE, AD4691_MANUAL_MODE);
+
+ ret = regmap_update_bits(st->regmap, AD4691_ADC_SETUP,
+ AD4691_ADC_MODE_MASK, AD4691_CNV_BURST_MODE);
+ if (ret)
+ return ret;
+
+ return regmap_write(st->regmap, AD4691_STATE_RESET_REG,
+ AD4691_STATE_RESET_ALL);
+}
+
+static int ad4691_transfer(struct ad4691_state *st, u16 cmd)
+{
+ u8 buf[2];
+
+ put_unaligned_be16(cmd, buf);
+
+ return spi_write(st->spi, buf, sizeof(buf));
+}
+
+/*
+ * ad4691_exit_conversion_mode - Return the chip to AUTONOMOUS mode.
+ *
+ * Called from buffer postdisable to restore the chip to the
+ * idle state used by read_raw. Clears the sequencer and resets state.
+ */
+static int ad4691_exit_conversion_mode(struct ad4691_state *st)
+{
+ if (st->manual_mode)
+ return ad4691_transfer(st, AD4691_EXIT_COMMAND);
+
+ return regmap_update_bits(st->regmap, AD4691_ADC_SETUP,
+ AD4691_ADC_MODE_MASK, AD4691_AUTONOMOUS_MODE);
+}
+
+static int ad4691_manual_buffer_preenable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ unsigned int k, i;
+ int ret;
+
+ memset(st->scan_xfers, 0, sizeof(st->scan_xfers));
+ memset(st->scan_tx, 0, sizeof(st->scan_tx));
+
+ spi_message_init(&st->scan_msg);
+
+ k = 0;
+ iio_for_each_active_channel(indio_dev, i) {
+ /*
+ * Channel-select command occupies the first (high) byte of the
+ * 16-bit DIN frame; the second byte is a don't-care zero pad.
+ * put_unaligned_be16() writes [cmd, 0x00] in memory so the
+ * SPI controller sends the command byte first on the wire.
+ */
+ put_unaligned_be16((u16)(AD4691_ADC_CHAN(i) << 8), &st->scan_tx[k]);
+ st->scan_xfers[k].tx_buf = &st->scan_tx[k];
+ /*
+ * The pipeline means xfer[0] receives the residual from the
+ * previous sequence, not a valid sample. Discard it (rx_buf=NULL)
+ * to avoid aliasing vals[0] across two concurrent DMA mappings.
+ * xfer[1] (or the NOOP when only one channel is active) writes
+ * the real ch[0] result to vals[0]. Subsequent transfers write
+ * into vals[k-1] so each result lands at the next dense slot.
+ */
+ st->scan_xfers[k].rx_buf = (k == 0) ? NULL : &st->vals[k - 1];
+ st->scan_xfers[k].len = sizeof(*st->scan_tx);
+ st->scan_xfers[k].cs_change = 1;
+ st->scan_xfers[k].cs_change_delay.value = AD4691_CNV_HIGH_TIME_NS;
+ st->scan_xfers[k].cs_change_delay.unit = SPI_DELAY_UNIT_NSECS;
+ spi_message_add_tail(&st->scan_xfers[k], &st->scan_msg);
+ k++;
+ }
+
+ /* Final NOOP transfer retrieves the last channel's result. */
+ st->scan_xfers[k].tx_buf = &st->scan_tx[k]; /* scan_tx[k] == 0 == NOOP */
+ st->scan_xfers[k].rx_buf = &st->vals[k - 1];
+ st->scan_xfers[k].len = sizeof(*st->scan_tx);
+ spi_message_add_tail(&st->scan_xfers[k], &st->scan_msg);
+
+ ret = spi_optimize_message(st->spi, &st->scan_msg);
+ if (ret)
+ return ret;
+
+ ret = ad4691_enter_conversion_mode(st);
+ if (ret) {
+ spi_unoptimize_message(&st->scan_msg);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int ad4691_manual_buffer_postdisable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ int ret;
+
+ ret = ad4691_exit_conversion_mode(st);
+ spi_unoptimize_message(&st->scan_msg);
+ return ret;
+}
+
+static const struct iio_buffer_setup_ops ad4691_manual_buffer_setup_ops = {
+ .preenable = ad4691_manual_buffer_preenable,
+ .postdisable = ad4691_manual_buffer_postdisable,
+};
+
+static int ad4691_cnv_burst_buffer_preenable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ unsigned int acc_mask, std_seq_config;
+ unsigned int k, i;
+ int ret;
+
+ memset(st->scan_xfers, 0, sizeof(st->scan_xfers));
+ memset(st->scan_tx, 0, sizeof(st->scan_tx));
+
+ spi_message_init(&st->scan_msg);
+
+ /*
+ * Each AVG_IN read needs two transfers: a 2-byte address write phase
+ * followed by a 2-byte data read phase. CS toggles between channels
+ * (cs_change=1 on the read phase of all but the last channel).
+ */
+ k = 0;
+ iio_for_each_active_channel(indio_dev, i) {
+ put_unaligned_be16(0x8000 | AD4691_AVG_IN(i), &st->scan_tx[k]);
+ st->scan_xfers[2 * k].tx_buf = &st->scan_tx[k];
+ st->scan_xfers[2 * k].len = sizeof(*st->scan_tx);
+ spi_message_add_tail(&st->scan_xfers[2 * k], &st->scan_msg);
+ st->scan_xfers[2 * k + 1].rx_buf = &st->vals[k];
+ st->scan_xfers[2 * k + 1].len = sizeof(*st->scan_tx);
+ st->scan_xfers[2 * k + 1].cs_change = 1;
+ spi_message_add_tail(&st->scan_xfers[2 * k + 1], &st->scan_msg);
+ k++;
+ }
+
+ /*
+ * Append a 4-byte state-reset transfer [addr_hi, addr_lo,
+ * STATE_RESET_ALL, OSC_EN=1]. CS is asserted throughout, so
+ * ADDR_DESCENDING writes byte[3]=1 to OSC_EN_REG (0x180) as a
+ * deliberate side-write, keeping the oscillator enabled.
+ * STATE_RESET_ALL starts the next burst; the hardware does not
+ * accumulate new conversions until after a STATE_RESET pulse, so
+ * no in-progress data is lost. No cs_change here — CS must
+ * deassert normally at end of message to frame the next command.
+ */
+ put_unaligned_be16(AD4691_STATE_RESET_REG, st->scan_tx_reset);
+ st->scan_tx_reset[2] = AD4691_STATE_RESET_ALL;
+ st->scan_tx_reset[3] = 1;
+ st->scan_xfers[2 * k].tx_buf = st->scan_tx_reset;
+ st->scan_xfers[2 * k].len = sizeof(st->scan_tx_reset);
+ spi_message_add_tail(&st->scan_xfers[2 * k], &st->scan_msg);
+
+ ret = spi_optimize_message(st->spi, &st->scan_msg);
+ if (ret)
+ return ret;
+
+ std_seq_config = bitmap_read(indio_dev->active_scan_mask, 0,
+ iio_get_masklength(indio_dev)) & GENMASK(15, 0);
+ ret = regmap_write(st->regmap, AD4691_STD_SEQ_CONFIG, std_seq_config);
+ if (ret)
+ goto err_unoptimize;
+
+ acc_mask = ~std_seq_config & GENMASK(15, 0);
+ ret = regmap_write(st->regmap, AD4691_ACC_MASK_REG, acc_mask);
+ if (ret)
+ goto err_unoptimize;
+
+ ret = ad4691_enter_conversion_mode(st);
+ if (ret)
+ goto err_unoptimize;
+
+ return 0;
+
+err_unoptimize:
+ spi_unoptimize_message(&st->scan_msg);
+ return ret;
+}
+
+static int ad4691_cnv_burst_buffer_postenable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ int ret;
+
+ /*
+ * Start the PWM and unmask the IRQ here in postenable, not in
+ * preenable. The IIO core attaches the trigger poll function between
+ * preenable and postenable; enabling sampling or unmasking the IRQ
+ * before that point risks a DATA_READY assertion landing before the
+ * poll function is registered. iio_trigger_poll() would drop the
+ * event, disable_irq_nosync() would fire, and enable_irq() would
+ * never be called, leaving the IRQ permanently masked.
+ */
+ ret = ad4691_sampling_enable(st, true);
+ if (ret)
+ return ret;
+
+ enable_irq(st->irq);
+ st->irq_enabled = true;
+ return 0;
+}
+
+static int ad4691_cnv_burst_buffer_predisable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ if (st->irq_enabled) {
+ disable_irq(st->irq);
+ st->irq_enabled = false;
+ }
+ return ad4691_sampling_enable(st, false);
+}
+
+static int ad4691_cnv_burst_buffer_postdisable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ int ret;
+
+ ret = ad4691_exit_conversion_mode(st);
+ spi_unoptimize_message(&st->scan_msg);
+ return ret;
+}
+
+static const struct iio_buffer_setup_ops ad4691_cnv_burst_buffer_setup_ops = {
+ .preenable = ad4691_cnv_burst_buffer_preenable,
+ .postenable = ad4691_cnv_burst_buffer_postenable,
+ .predisable = ad4691_cnv_burst_buffer_predisable,
+ .postdisable = ad4691_cnv_burst_buffer_postdisable,
+};
+
+static ssize_t sampling_frequency_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct iio_dev *indio_dev = dev_to_iio_dev(dev);
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ return sysfs_emit(buf, "%lu\n", NSEC_PER_SEC / st->cnv_period_ns);
+}
+
+static ssize_t sampling_frequency_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t len)
+{
+ struct iio_dev *indio_dev = dev_to_iio_dev(dev);
+ struct ad4691_state *st = iio_priv(indio_dev);
+ unsigned int freq;
+ int ret;
+
+ ret = kstrtouint(buf, 10, &freq);
+ if (ret)
+ return ret;
+
+ IIO_DEV_ACQUIRE_DIRECT_MODE(indio_dev, claim);
+ if (IIO_DEV_ACQUIRE_FAILED(claim))
+ return -EBUSY;
+
+ ret = ad4691_set_pwm_freq(st, freq);
+ if (ret)
+ return ret;
+
+ return len;
+}
+
+static IIO_DEVICE_ATTR_RW(sampling_frequency, 0);
+
+static const struct iio_dev_attr *ad4691_buffer_attrs[] = {
+ &iio_dev_attr_sampling_frequency,
+ NULL
+};
+
+static irqreturn_t ad4691_irq(int irq, void *private)
+{
+ struct iio_dev *indio_dev = private;
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ /*
+ * Disable the IRQ before calling iio_trigger_poll(). The IRQ is
+ * re-enabled via the trigger .reenable callback, which the IIO core
+ * calls inside iio_trigger_notify_done() once use_count reaches zero.
+ * Re-enabling here (before notify_done) would race: a DATA_READY
+ * between enable_irq() and notify_done() calls iio_trigger_poll()
+ * while use_count > 0, dropping the event and permanently masking
+ * the IRQ.
+ */
+ disable_irq_nosync(st->irq);
+ iio_trigger_poll(indio_dev->trig);
+
+ return IRQ_HANDLED;
+}
+
+static void ad4691_trigger_reenable(struct iio_trigger *trig)
+{
+ struct ad4691_state *st = iio_trigger_get_drvdata(trig);
+
+ enable_irq(st->irq);
+}
+
+static const struct iio_trigger_ops ad4691_trigger_ops = {
+ .reenable = ad4691_trigger_reenable,
+ .validate_device = iio_trigger_validate_own_device,
+};
+
+static void ad4691_read_scan(struct iio_dev *indio_dev, s64 ts)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ int ret;
+
+ guard(mutex)(&st->lock);
+
+ ret = spi_sync(st->spi, &st->scan_msg);
+ if (ret) {
+ dev_err_ratelimited(regmap_get_device(st->regmap),
+ "SPI scan failed: %d\n", ret);
+ return;
+ }
+
+ /*
+ * rx_buf pointers in scan_xfers point directly into scan.vals, so no
+ * copy is needed. The scan_msg already includes a STATE_RESET at the
+ * end (appended in preenable), so no explicit reset is needed here.
+ */
+ iio_push_to_buffers_with_ts(indio_dev, st->vals, sizeof(st->vals), ts);
+}
+
+static irqreturn_t ad4691_trigger_handler(int irq, void *p)
+{
+ struct iio_poll_func *pf = p;
+ struct iio_dev *indio_dev = pf->indio_dev;
+
+ ad4691_read_scan(indio_dev, pf->timestamp);
+ iio_trigger_notify_done(indio_dev->trig);
+ return IRQ_HANDLED;
+}
+
+/*
+ * CNV burst mode: only allow our own trigger (driven by DATA_READY IRQ).
+ * Manual mode: external triggers (e.g. iio-trig-hrtimer) must be allowed
+ * because manual mode has no DATA_READY IRQ to fire the internal trigger.
+ * iio_trigger_ops.validate_device = iio_trigger_validate_own_device is
+ * correct in both modes — it prevents other devices from hijacking our
+ * internal trigger; the distinction here is only for iio_info.validate_trigger.
+ */
+static const struct iio_info ad4691_cnv_burst_info = {
+ .read_raw = ad4691_read_raw,
+ .write_raw = ad4691_write_raw,
+ .read_avail = ad4691_read_avail,
+ .debugfs_reg_access = ad4691_reg_access,
+ .validate_trigger = iio_validate_own_trigger,
+};
+
+static const struct iio_info ad4691_manual_info = {
.read_raw = ad4691_read_raw,
.write_raw = ad4691_write_raw,
.read_avail = ad4691_read_avail,
.debugfs_reg_access = ad4691_reg_access,
};
+static int ad4691_pwm_setup(struct ad4691_state *st)
+{
+ struct device *dev = regmap_get_device(st->regmap);
+
+ st->conv_trigger = devm_pwm_get(dev, "cnv");
+ if (IS_ERR(st->conv_trigger))
+ return dev_err_probe(dev, PTR_ERR(st->conv_trigger),
+ "Failed to get CNV PWM\n");
+
+ return ad4691_set_pwm_freq(st, st->info->max_rate);
+}
+
static int ad4691_regulator_setup(struct ad4691_state *st)
{
struct device *dev = regmap_get_device(st->regmap);
@@ -649,6 +1119,22 @@ static int ad4691_config(struct ad4691_state *st)
unsigned int val;
int ret;
+ /*
+ * Determine buffer conversion mode from DT: if a PWM is provided it
+ * drives the CNV pin (CNV_BURST_MODE); otherwise CNV is tied to CS
+ * and each SPI transfer triggers a conversion (MANUAL_MODE).
+ * Both modes idle in AUTONOMOUS mode so that read_raw can use the
+ * internal oscillator without disturbing the hardware configuration.
+ */
+ if (device_property_present(dev, "pwms")) {
+ st->manual_mode = false;
+ ret = ad4691_pwm_setup(st);
+ if (ret)
+ return ret;
+ } else {
+ st->manual_mode = true;
+ }
+
switch (st->vref_uV) {
case AD4691_VREF_uV_MIN ... AD4691_VREF_2P5_uV_MAX:
ref_val = AD4691_VREF_2P5;
@@ -702,6 +1188,86 @@ static int ad4691_config(struct ad4691_state *st)
return 0;
}
+static int ad4691_setup_triggered_buffer(struct iio_dev *indio_dev,
+ struct ad4691_state *st)
+{
+ struct device *dev = regmap_get_device(st->regmap);
+ struct iio_trigger *trig;
+ unsigned int i;
+ int irq, ret;
+
+ indio_dev->channels = st->info->sw_info->channels;
+ indio_dev->num_channels = st->info->sw_info->num_channels;
+ indio_dev->info = st->manual_mode ? &ad4691_manual_info : &ad4691_cnv_burst_info;
+
+ /*
+ * Manual mode relies on an external trigger (e.g. iio-trig-hrtimer);
+ * no internal trigger is needed or registered.
+ */
+ if (st->manual_mode)
+ return devm_iio_triggered_buffer_setup(dev, indio_dev,
+ iio_pollfunc_store_time,
+ ad4691_trigger_handler,
+ &ad4691_manual_buffer_setup_ops);
+
+ /*
+ * CNV burst mode: allocate an internal trigger driven by the
+ * DATA_READY IRQ on the GP pin.
+ */
+ trig = devm_iio_trigger_alloc(dev, "%s-dev%d", indio_dev->name,
+ iio_device_id(indio_dev));
+ if (!trig)
+ return -ENOMEM;
+
+ trig->ops = &ad4691_trigger_ops;
+ iio_trigger_set_drvdata(trig, st);
+
+ ret = devm_iio_trigger_register(dev, trig);
+ if (ret)
+ return dev_err_probe(dev, ret, "IIO trigger register failed\n");
+
+ indio_dev->trig = iio_trigger_get(trig);
+
+ /*
+ * The GP pin named in interrupt-names asserts at end-of-conversion.
+ * The IRQ handler fires the IIO trigger so the trigger handler can
+ * read and push the sample to the buffer. The IRQ is kept disabled
+ * until the buffer is enabled.
+ */
+ irq = -ENXIO;
+ for (i = 0; i < ARRAY_SIZE(ad4691_gp_names); i++) {
+ irq = fwnode_irq_get_byname(dev_fwnode(dev),
+ ad4691_gp_names[i]);
+ if (irq > 0 || irq == -EPROBE_DEFER)
+ break;
+ }
+ if (irq < 0)
+ return dev_err_probe(dev, irq, "failed to get GP interrupt\n");
+
+ st->irq = irq;
+
+ ret = ad4691_gpio_setup(st, i);
+ if (ret)
+ return ret;
+
+ /*
+ * The handler only calls disable_irq_nosync() and iio_trigger_poll(),
+ * both safe in hardirq context, so register as a hard IRQ handler.
+ * IRQF_NO_AUTOEN keeps it disabled until the buffer is enabled.
+ */
+ ret = devm_request_irq(dev, irq, ad4691_irq, IRQF_NO_AUTOEN,
+ indio_dev->name, indio_dev);
+ if (ret)
+ return ret;
+
+ return devm_iio_triggered_buffer_setup_ext(dev, indio_dev,
+ iio_pollfunc_store_time,
+ ad4691_trigger_handler,
+ IIO_BUFFER_DIRECTION_IN,
+ &ad4691_cnv_burst_buffer_setup_ops,
+ ad4691_buffer_attrs);
+}
+
static int ad4691_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
@@ -741,11 +1307,11 @@ static int ad4691_probe(struct spi_device *spi)
return ret;
indio_dev->name = st->info->name;
- indio_dev->info = &ad4691_info;
indio_dev->modes = INDIO_DIRECT_MODE;
- indio_dev->channels = st->info->sw_info->channels;
- indio_dev->num_channels = st->info->sw_info->num_channels;
+ ret = ad4691_setup_triggered_buffer(indio_dev, st);
+ if (ret)
+ return ret;
return devm_iio_device_register(dev, indio_dev);
}
--
2.43.0
^ permalink raw reply related
* [PATCH v14 4/6] iio: adc: ad4691: add SPI offload support
From: Radu Sabau via B4 Relay @ 2026-05-29 10:15 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau
In-Reply-To: <20260529-ad4692-multichannel-sar-adc-driver-v14-0-e93c2747dc1f@analog.com>
From: Radu Sabau <radu.sabau@analog.com>
Add SPI offload support to enable DMA-based, CPU-independent data
acquisition using the SPI Engine offload framework.
When an SPI offload is available (devm_spi_offload_get() succeeds),
the driver registers a DMA engine IIO buffer and uses dedicated buffer
setup operations. If no offload is available the existing software
triggered buffer path is used unchanged.
Both CNV Burst Mode and Manual Mode support offload, but use different
trigger mechanisms:
CNV Burst Mode: the SPI Engine is triggered by the ADC's DATA_READY
signal on the GP pin specified by the trigger-source consumer reference
in the device tree (one cell = GP pin number 0-3). For this mode the
driver acts as both an SPI offload consumer (DMA RX stream, message
optimization) and a trigger source provider: it registers the
GP/DATA_READY output via devm_spi_offload_trigger_register() so the
offload framework can match the '#trigger-source-cells' phandle and
automatically fire the SPI Engine DMA transfer at end-of-conversion.
Manual Mode: the SPI Engine is triggered by a periodic trigger at
the configured sampling frequency. The pre-built SPI message uses
the pipelined CNV-on-CS protocol: N+1 16-bit transfers are issued
for N active channels (the first result is discarded as garbage from
the pipeline flush) and the remaining N results are captured by DMA.
All offload transfers use 16-bit frames (bits_per_word=16, len=2).
The SPI Engine assembles received bits into native 16-bit words before
DMA, so offload samples land in CPU-native byte order (IIO_CPU).
Dedicated channel arrays (AD4691_OFFLOAD_CHANNEL) reflect this: they
omit IIO_BE and carry no soft timestamp (DMA delivers data directly to
userspace). The software triggered-buffer path retains its IIO_BE
channels because bits_per_word=8 causes SPI to deliver bytes MSB-first
into memory, making the on-disk layout big-endian. Both paths use
storagebits=16 as transfers are 16 bits wide in both cases.
IIO_BUFFER_DMAENGINE is selected because the offload path uses
devm_iio_dmaengine_buffer_setup_with_handle() to allocate and
attach the DMA RX buffer to the IIO device.
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
drivers/iio/adc/Kconfig | 2 +
drivers/iio/adc/ad4691.c | 444 ++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 443 insertions(+), 3 deletions(-)
diff --git a/drivers/iio/adc/Kconfig b/drivers/iio/adc/Kconfig
index 1d7afde108c0..4d12eeb08020 100644
--- a/drivers/iio/adc/Kconfig
+++ b/drivers/iio/adc/Kconfig
@@ -149,8 +149,10 @@ config AD4691
depends on SPI
depends on REGULATOR || COMPILE_TEST
select IIO_BUFFER
+ select IIO_BUFFER_DMAENGINE
select IIO_TRIGGERED_BUFFER
select REGMAP
+ select SPI_OFFLOAD
help
Say yes here to build support for Analog Devices AD4691 Family MuxSAR
SPI analog to digital converters (ADC).
diff --git a/drivers/iio/adc/ad4691.c b/drivers/iio/adc/ad4691.c
index 175c80129786..b1d1b42f724e 100644
--- a/drivers/iio/adc/ad4691.c
+++ b/drivers/iio/adc/ad4691.c
@@ -10,6 +10,7 @@
#include <linux/delay.h>
#include <linux/dev_printk.h>
#include <linux/device/devres.h>
+#include <linux/dmaengine.h>
#include <linux/err.h>
#include <linux/interrupt.h>
#include <linux/kstrtox.h>
@@ -24,11 +25,15 @@
#include <linux/reset.h>
#include <linux/string.h>
#include <linux/spi/spi.h>
+#include <linux/spi/offload/consumer.h>
+#include <linux/spi/offload/provider.h>
#include <linux/types.h>
#include <linux/units.h>
#include <linux/unaligned.h>
#include <linux/iio/buffer.h>
+#include <linux/iio/buffer-dma.h>
+#include <linux/iio/buffer-dmaengine.h>
#include <linux/iio/iio.h>
#include <linux/iio/sysfs.h>
#include <linux/iio/trigger.h>
@@ -44,6 +49,11 @@
#define AD4691_CNV_DUTY_CYCLE_NS 380
#define AD4691_CNV_HIGH_TIME_NS 430
+/*
+ * Conservative default for the manual offload periodic trigger. Low enough
+ * to work safely out of the box across all OSR and channel count combinations.
+ */
+#define AD4691_OFFLOAD_INITIAL_TRIGGER_HZ (100 * HZ_PER_KHZ)
#define AD4691_SPI_CONFIG_A_REG 0x000
#define AD4691_SW_RESET (BIT(7) | BIT(0))
@@ -119,6 +129,7 @@ struct ad4691_chip_info {
const char *name;
unsigned int max_rate;
const struct ad4691_channel_info *sw_info;
+ const struct ad4691_channel_info *offload_info;
};
#define AD4691_CHANNEL(ch) \
@@ -140,6 +151,30 @@ struct ad4691_chip_info {
}, \
}
+/*
+ * Offload path (bits_per_word=16): the SPI Engine assembles received
+ * bits into native 16-bit words before DMA, so samples are in
+ * CPU-native byte order (IIO_CPU). storagebits=16 matches the 16-bit
+ * DMA word size.
+ */
+#define AD4691_OFFLOAD_CHANNEL(ch) \
+ { \
+ .type = IIO_VOLTAGE, \
+ .indexed = 1, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW), \
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SCALE) \
+ | BIT(IIO_CHAN_INFO_SAMP_FREQ), \
+ .info_mask_shared_by_all_available = \
+ BIT(IIO_CHAN_INFO_SAMP_FREQ), \
+ .channel = ch, \
+ .scan_index = ch, \
+ .scan_type = { \
+ .format = 'u', \
+ .realbits = 16, \
+ .storagebits = 16, \
+ }, \
+ }
+
static const struct iio_chan_spec ad4691_channels[] = {
AD4691_CHANNEL(0),
AD4691_CHANNEL(1),
@@ -172,6 +207,40 @@ static const struct iio_chan_spec ad4693_channels[] = {
IIO_CHAN_SOFT_TIMESTAMP(8),
};
+/*
+ * Offload channel arrays: no IIO_CHAN_SOFT_TIMESTAMP because DMA delivers
+ * data directly to userspace without a software timestamp.
+ */
+static const struct iio_chan_spec ad4691_offload_channels[] = {
+ AD4691_OFFLOAD_CHANNEL(0),
+ AD4691_OFFLOAD_CHANNEL(1),
+ AD4691_OFFLOAD_CHANNEL(2),
+ AD4691_OFFLOAD_CHANNEL(3),
+ AD4691_OFFLOAD_CHANNEL(4),
+ AD4691_OFFLOAD_CHANNEL(5),
+ AD4691_OFFLOAD_CHANNEL(6),
+ AD4691_OFFLOAD_CHANNEL(7),
+ AD4691_OFFLOAD_CHANNEL(8),
+ AD4691_OFFLOAD_CHANNEL(9),
+ AD4691_OFFLOAD_CHANNEL(10),
+ AD4691_OFFLOAD_CHANNEL(11),
+ AD4691_OFFLOAD_CHANNEL(12),
+ AD4691_OFFLOAD_CHANNEL(13),
+ AD4691_OFFLOAD_CHANNEL(14),
+ AD4691_OFFLOAD_CHANNEL(15),
+};
+
+static const struct iio_chan_spec ad4693_offload_channels[] = {
+ AD4691_OFFLOAD_CHANNEL(0),
+ AD4691_OFFLOAD_CHANNEL(1),
+ AD4691_OFFLOAD_CHANNEL(2),
+ AD4691_OFFLOAD_CHANNEL(3),
+ AD4691_OFFLOAD_CHANNEL(4),
+ AD4691_OFFLOAD_CHANNEL(5),
+ AD4691_OFFLOAD_CHANNEL(6),
+ AD4691_OFFLOAD_CHANNEL(7),
+};
+
static const struct ad4691_channel_info ad4691_sw_info = {
.channels = ad4691_channels,
.num_channels = ARRAY_SIZE(ad4691_channels),
@@ -182,6 +251,16 @@ static const struct ad4691_channel_info ad4693_sw_info = {
.num_channels = ARRAY_SIZE(ad4693_channels),
};
+static const struct ad4691_channel_info ad4691_offload_info = {
+ .channels = ad4691_offload_channels,
+ .num_channels = ARRAY_SIZE(ad4691_offload_channels),
+};
+
+static const struct ad4691_channel_info ad4693_offload_info = {
+ .channels = ad4693_offload_channels,
+ .num_channels = ARRAY_SIZE(ad4693_offload_channels),
+};
+
/*
* Internal oscillator frequency table. Index is the OSC_FREQ_REG[3:0] value.
* Index 0 (1 MHz) is only valid for AD4692/AD4694; AD4691/AD4693 support
@@ -212,24 +291,28 @@ static const struct ad4691_chip_info ad4691_chip_info = {
.name = "ad4691",
.max_rate = 500 * HZ_PER_KHZ,
.sw_info = &ad4691_sw_info,
+ .offload_info = &ad4691_offload_info,
};
static const struct ad4691_chip_info ad4692_chip_info = {
.name = "ad4692",
.max_rate = 1 * HZ_PER_MHZ,
.sw_info = &ad4691_sw_info,
+ .offload_info = &ad4691_offload_info,
};
static const struct ad4691_chip_info ad4693_chip_info = {
.name = "ad4693",
.max_rate = 500 * HZ_PER_KHZ,
.sw_info = &ad4693_sw_info,
+ .offload_info = &ad4693_offload_info,
};
static const struct ad4691_chip_info ad4694_chip_info = {
.name = "ad4694",
.max_rate = 1 * HZ_PER_MHZ,
.sw_info = &ad4693_sw_info,
+ .offload_info = &ad4693_offload_info,
};
struct ad4691_state {
@@ -251,6 +334,10 @@ struct ad4691_state {
* atomicity of consecutive SPI operations.
*/
struct mutex lock;
+ /* NULL when no SPI offload hardware is present. */
+ struct spi_offload *offload;
+ struct spi_offload_trigger *offload_trigger;
+ u64 trigger_hz;
/*
* Per-buffer-enable lifetime resources:
* Manual Mode - a pre-built SPI message that clocks out N+1
@@ -265,8 +352,11 @@ struct ad4691_state {
struct spi_transfer scan_xfers[34];
/*
* CNV burst: 16 AVG_IN addresses = 16. Manual: 16 channel cmds +
- * 1 NOOP = 17. Stored as native u16; put_unaligned_be16() fills each
- * slot so the SPI controller (bits_per_word=8) sends bytes MSB-first.
+ * 1 NOOP = 17. Stored as native u16. The non-offload path fills slots
+ * with put_unaligned_be16() (bits_per_word=8, bytes go out in memory
+ * order). The offload path assigns native values directly
+ * (bits_per_word=bpw, SPI reads each slot as a native 16-bit word and
+ * shifts it out MSB-first).
*/
u16 scan_tx[17] __aligned(IIO_DMA_MINALIGN);
/*
@@ -301,6 +391,45 @@ static int ad4691_gpio_setup(struct ad4691_state *st, unsigned int gp_num)
AD4691_GP_MODE_DATA_READY << shift);
}
+static const struct spi_offload_config ad4691_offload_config = {
+ .capability_flags = SPI_OFFLOAD_CAP_TRIGGER |
+ SPI_OFFLOAD_CAP_RX_STREAM_DMA,
+};
+
+static bool ad4691_offload_trigger_match(struct spi_offload_trigger *trigger,
+ enum spi_offload_trigger_type type,
+ u64 *args, u32 nargs)
+{
+ return type == SPI_OFFLOAD_TRIGGER_DATA_READY && nargs == 1 && args[0] <= 3;
+}
+
+static int ad4691_offload_trigger_request(struct spi_offload_trigger *trigger,
+ enum spi_offload_trigger_type type,
+ u64 *args, u32 nargs)
+{
+ struct ad4691_state *st = spi_offload_trigger_get_priv(trigger);
+
+ if (nargs != 1 || args[0] > 3)
+ return -EINVAL;
+
+ return ad4691_gpio_setup(st, args[0]);
+}
+
+static int ad4691_offload_trigger_validate(struct spi_offload_trigger *trigger,
+ struct spi_offload_trigger_config *config)
+{
+ if (config->type != SPI_OFFLOAD_TRIGGER_DATA_READY)
+ return -EINVAL;
+
+ return 0;
+}
+
+static const struct spi_offload_trigger_ops ad4691_offload_trigger_ops = {
+ .match = ad4691_offload_trigger_match,
+ .request = ad4691_offload_trigger_request,
+ .validate = ad4691_offload_trigger_validate,
+};
+
static int ad4691_reg_read(void *context, unsigned int reg, unsigned int *val)
{
struct spi_device *spi = context;
@@ -876,6 +1005,218 @@ static const struct iio_buffer_setup_ops ad4691_cnv_burst_buffer_setup_ops = {
.postdisable = ad4691_cnv_burst_buffer_postdisable,
};
+static int ad4691_manual_offload_buffer_postenable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ struct device *dev = regmap_get_device(st->regmap);
+ struct spi_device *spi = to_spi_device(dev);
+ struct spi_offload_trigger_config config = {
+ .type = SPI_OFFLOAD_TRIGGER_PERIODIC,
+ };
+ unsigned int bpw = indio_dev->channels[0].scan_type.realbits;
+ unsigned int bit, k;
+ int ret;
+
+ ret = ad4691_enter_conversion_mode(st);
+ if (ret)
+ return ret;
+
+ memset(st->scan_xfers, 0, sizeof(st->scan_xfers));
+ memset(st->scan_tx, 0, sizeof(st->scan_tx));
+
+ /*
+ * N+1 transfers for N channels. Each CS-low period triggers
+ * a conversion AND returns the previous result (pipelined).
+ * TX: [AD4691_ADC_CHAN(n), 0x00]
+ * RX: [data_hi, data_lo] (storagebits=16, shift=0)
+ * Transfer 0 RX is garbage; transfers 1..N carry real data.
+ * scan_tx is reused for TX commands (mutually exclusive with the
+ * non-offload triggered-buffer path).
+ *
+ * bits_per_word=bpw: the SPI controller reads tx_buf as a native
+ * 16-bit word and shifts it out MSB-first. Store the exact 16-bit
+ * value we want on the wire as a plain native u16 — no endianness
+ * macro — so the wire bytes are correct on both LE and BE hosts.
+ * The channel-select command is a single byte; shift it to the MSB
+ * position so SPI sends it first, with a zero pad in the LSB.
+ */
+ k = 0;
+ iio_for_each_active_channel(indio_dev, bit) {
+ st->scan_tx[k] = AD4691_ADC_CHAN(bit) << 8;
+ st->scan_xfers[k].tx_buf = &st->scan_tx[k];
+ st->scan_xfers[k].len = sizeof(*st->scan_tx);
+ st->scan_xfers[k].bits_per_word = bpw;
+ st->scan_xfers[k].cs_change = 1;
+ st->scan_xfers[k].cs_change_delay.value = AD4691_CNV_HIGH_TIME_NS;
+ st->scan_xfers[k].cs_change_delay.unit = SPI_DELAY_UNIT_NSECS;
+ /* First transfer RX is garbage — skip it. */
+ if (k > 0)
+ st->scan_xfers[k].offload_flags = SPI_OFFLOAD_XFER_RX_STREAM;
+ k++;
+ }
+
+ /* Final NOOP transfer retrieves the last channel's result. */
+ st->scan_xfers[k].tx_buf = &st->scan_tx[k]; /* scan_tx[k] == 0 == NOOP */
+ st->scan_xfers[k].len = sizeof(*st->scan_tx);
+ st->scan_xfers[k].bits_per_word = bpw;
+ st->scan_xfers[k].offload_flags = SPI_OFFLOAD_XFER_RX_STREAM;
+ k++;
+
+ spi_message_init_with_transfers(&st->scan_msg, st->scan_xfers, k);
+ st->scan_msg.offload = st->offload;
+
+ ret = spi_optimize_message(spi, &st->scan_msg);
+ if (ret)
+ goto err_exit_conversion;
+
+ config.periodic.frequency_hz = st->trigger_hz;
+ ret = spi_offload_trigger_enable(st->offload, st->offload_trigger, &config);
+ if (ret)
+ goto err_unoptimize;
+
+ return 0;
+
+err_unoptimize:
+ spi_unoptimize_message(&st->scan_msg);
+err_exit_conversion:
+ ad4691_exit_conversion_mode(st);
+ return ret;
+}
+
+static int ad4691_manual_offload_buffer_predisable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ spi_offload_trigger_disable(st->offload, st->offload_trigger);
+ spi_unoptimize_message(&st->scan_msg);
+
+ return ad4691_exit_conversion_mode(st);
+}
+
+static const struct iio_buffer_setup_ops ad4691_manual_offload_buffer_setup_ops = {
+ .postenable = ad4691_manual_offload_buffer_postenable,
+ .predisable = ad4691_manual_offload_buffer_predisable,
+};
+
+static int ad4691_cnv_burst_offload_buffer_postenable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ struct device *dev = regmap_get_device(st->regmap);
+ struct spi_device *spi = to_spi_device(dev);
+ struct spi_offload_trigger_config config = {
+ .type = SPI_OFFLOAD_TRIGGER_DATA_READY,
+ };
+ unsigned int bpw = indio_dev->channels[0].scan_type.realbits;
+ unsigned int acc_mask, std_seq_config;
+ unsigned int bit, k;
+ int ret;
+
+ std_seq_config = bitmap_read(indio_dev->active_scan_mask, 0,
+ iio_get_masklength(indio_dev)) & GENMASK(15, 0);
+ ret = regmap_write(st->regmap, AD4691_STD_SEQ_CONFIG, std_seq_config);
+ if (ret)
+ return ret;
+
+ acc_mask = ~std_seq_config & GENMASK(15, 0);
+ ret = regmap_write(st->regmap, AD4691_ACC_MASK_REG, acc_mask);
+ if (ret)
+ return ret;
+
+ ret = ad4691_enter_conversion_mode(st);
+ if (ret)
+ return ret;
+
+ memset(st->scan_xfers, 0, sizeof(st->scan_xfers));
+ memset(st->scan_tx, 0, sizeof(st->scan_tx));
+
+ /*
+ * Each AVG_IN register read uses two transfers:
+ * TX: [reg_hi | 0x80, reg_lo] (address phase, CS stays asserted)
+ * RX: [data_hi, data_lo] (bpw-wide data phase, storagebits=16)
+ * Both TX and RX use bits_per_word=bpw: the SPI controller reads tx_buf
+ * as a native 16-bit word and shifts it out MSB-first. Store the exact
+ * 16-bit wire value as a plain native u16 — no endianness macro — so the
+ * wire bytes are correct on both LE and BE hosts. The read-address
+ * (0x8000 | reg) is already the 16-bit value we want on the wire.
+ * scan_tx is reused for TX addresses (mutually exclusive with the
+ * non-offload triggered-buffer path).
+ */
+ k = 0;
+ iio_for_each_active_channel(indio_dev, bit) {
+ st->scan_tx[k] = 0x8000 | AD4691_AVG_IN(bit);
+
+ /* TX: address phase, CS stays asserted into data phase */
+ st->scan_xfers[2 * k].tx_buf = &st->scan_tx[k];
+ st->scan_xfers[2 * k].len = sizeof(*st->scan_tx);
+ st->scan_xfers[2 * k].bits_per_word = bpw;
+
+ /* RX: data phase, CS toggles after to delimit the next register op */
+ st->scan_xfers[2 * k + 1].len = sizeof(*st->scan_tx);
+ st->scan_xfers[2 * k + 1].bits_per_word = bpw;
+ st->scan_xfers[2 * k + 1].offload_flags = SPI_OFFLOAD_XFER_RX_STREAM;
+ st->scan_xfers[2 * k + 1].cs_change = 1;
+ k++;
+ }
+
+ /*
+ * State reset: single 4-byte write [addr_hi, addr_lo, STATE_RESET_ALL,
+ * OSC_EN=1]. ADDR_DESCENDING writes byte[3]=1 to OSC_EN_REG (0x180) as
+ * a deliberate side-write, keeping the oscillator enabled.
+ * scan_tx_reset is shared with the non-offload path (len=4 here vs
+ * len=3 there) since the two paths are mutually exclusive at probe.
+ */
+ put_unaligned_be16(AD4691_STATE_RESET_REG, st->scan_tx_reset);
+ st->scan_tx_reset[2] = AD4691_STATE_RESET_ALL;
+ st->scan_tx_reset[3] = 1;
+ st->scan_xfers[2 * k].tx_buf = st->scan_tx_reset;
+ st->scan_xfers[2 * k].len = sizeof(st->scan_tx_reset);
+ /*
+ * 4-byte u8 buffer assembled with put_unaligned_be16(); leave
+ * bits_per_word at the default (8) so bytes go out in memory order.
+ */
+
+ spi_message_init_with_transfers(&st->scan_msg, st->scan_xfers, 2 * k + 1);
+ st->scan_msg.offload = st->offload;
+
+ ret = spi_optimize_message(spi, &st->scan_msg);
+ if (ret)
+ goto err_exit_conversion;
+
+ ret = spi_offload_trigger_enable(st->offload, st->offload_trigger, &config);
+ if (ret)
+ goto err_unoptimize;
+
+ ret = ad4691_sampling_enable(st, true);
+ if (ret)
+ goto err_disable_trigger;
+
+ return 0;
+
+err_disable_trigger:
+ spi_offload_trigger_disable(st->offload, st->offload_trigger);
+err_unoptimize:
+ spi_unoptimize_message(&st->scan_msg);
+err_exit_conversion:
+ ad4691_exit_conversion_mode(st);
+ return ret;
+}
+
+static int ad4691_cnv_burst_offload_buffer_predisable(struct iio_dev *indio_dev)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ ad4691_sampling_enable(st, false);
+ spi_offload_trigger_disable(st->offload, st->offload_trigger);
+ spi_unoptimize_message(&st->scan_msg);
+
+ return ad4691_exit_conversion_mode(st);
+}
+
+static const struct iio_buffer_setup_ops ad4691_cnv_burst_offload_buffer_setup_ops = {
+ .postenable = ad4691_cnv_burst_offload_buffer_postenable,
+ .predisable = ad4691_cnv_burst_offload_buffer_predisable,
+};
+
static ssize_t sampling_frequency_show(struct device *dev,
struct device_attribute *attr,
char *buf)
@@ -883,6 +1224,9 @@ static ssize_t sampling_frequency_show(struct device *dev,
struct iio_dev *indio_dev = dev_to_iio_dev(dev);
struct ad4691_state *st = iio_priv(indio_dev);
+ if (st->manual_mode && st->offload)
+ return sysfs_emit(buf, "%llu\n", READ_ONCE(st->trigger_hz));
+
return sysfs_emit(buf, "%lu\n", NSEC_PER_SEC / st->cnv_period_ns);
}
@@ -903,6 +1247,20 @@ static ssize_t sampling_frequency_store(struct device *dev,
if (IIO_DEV_ACQUIRE_FAILED(claim))
return -EBUSY;
+ if (st->manual_mode && st->offload) {
+ struct spi_offload_trigger_config config = {
+ .type = SPI_OFFLOAD_TRIGGER_PERIODIC,
+ .periodic = { .frequency_hz = freq },
+ };
+
+ ret = spi_offload_trigger_validate(st->offload_trigger, &config);
+ if (ret)
+ return ret;
+
+ WRITE_ONCE(st->trigger_hz, config.periodic.frequency_hz);
+ return len;
+ }
+
ret = ad4691_set_pwm_freq(st, freq);
if (ret)
return ret;
@@ -1268,9 +1626,77 @@ static int ad4691_setup_triggered_buffer(struct iio_dev *indio_dev,
ad4691_buffer_attrs);
}
+static int ad4691_setup_offload(struct iio_dev *indio_dev,
+ struct ad4691_state *st,
+ struct spi_offload *spi_offload)
+{
+ struct device *dev = regmap_get_device(st->regmap);
+ struct dma_chan *rx_dma;
+ int ret;
+
+ st->offload = spi_offload;
+
+ indio_dev->channels = st->info->offload_info->channels;
+ indio_dev->num_channels = st->info->offload_info->num_channels;
+ /*
+ * Offload path uses DMA directly; no IIO trigger is involved, so
+ * external triggers are not restricted (no validate_trigger).
+ */
+ indio_dev->info = &ad4691_manual_info;
+
+ if (st->manual_mode) {
+ st->offload_trigger =
+ devm_spi_offload_trigger_get(dev, st->offload,
+ SPI_OFFLOAD_TRIGGER_PERIODIC);
+ if (IS_ERR(st->offload_trigger))
+ return dev_err_probe(dev, PTR_ERR(st->offload_trigger),
+ "Failed to get periodic offload trigger\n");
+
+ st->trigger_hz = AD4691_OFFLOAD_INITIAL_TRIGGER_HZ;
+ } else {
+ struct spi_offload_trigger_info trigger_info = {
+ .fwnode = dev_fwnode(dev),
+ .ops = &ad4691_offload_trigger_ops,
+ .priv = st,
+ };
+
+ ret = devm_spi_offload_trigger_register(dev, &trigger_info);
+ if (ret)
+ return dev_err_probe(dev, ret,
+ "Failed to register offload trigger\n");
+
+ st->offload_trigger =
+ devm_spi_offload_trigger_get(dev, st->offload,
+ SPI_OFFLOAD_TRIGGER_DATA_READY);
+ if (IS_ERR(st->offload_trigger))
+ return dev_err_probe(dev, PTR_ERR(st->offload_trigger),
+ "Failed to get DATA_READY offload trigger\n");
+ }
+
+ rx_dma = devm_spi_offload_rx_stream_request_dma_chan(dev, st->offload);
+ if (IS_ERR(rx_dma))
+ return dev_err_probe(dev, PTR_ERR(rx_dma),
+ "Failed to get offload RX DMA channel\n");
+
+ if (st->manual_mode)
+ indio_dev->setup_ops = &ad4691_manual_offload_buffer_setup_ops;
+ else
+ indio_dev->setup_ops = &ad4691_cnv_burst_offload_buffer_setup_ops;
+
+ ret = devm_iio_dmaengine_buffer_setup_with_handle(dev, indio_dev, rx_dma,
+ IIO_BUFFER_DIRECTION_IN);
+ if (ret)
+ return ret;
+
+ indio_dev->buffer->attrs = ad4691_buffer_attrs;
+
+ return 0;
+}
+
static int ad4691_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct spi_offload *spi_offload;
struct iio_dev *indio_dev;
struct ad4691_state *st;
int ret;
@@ -1306,10 +1732,20 @@ static int ad4691_probe(struct spi_device *spi)
if (ret)
return ret;
+ spi_offload = devm_spi_offload_get(dev, spi, &ad4691_offload_config);
+ ret = PTR_ERR_OR_ZERO(spi_offload);
+ if (ret == -ENODEV)
+ spi_offload = NULL;
+ else if (ret)
+ return dev_err_probe(dev, ret, "Failed to get SPI offload\n");
+
indio_dev->name = st->info->name;
indio_dev->modes = INDIO_DIRECT_MODE;
- ret = ad4691_setup_triggered_buffer(indio_dev, st);
+ if (spi_offload)
+ ret = ad4691_setup_offload(indio_dev, st, spi_offload);
+ else
+ ret = ad4691_setup_triggered_buffer(indio_dev, st);
if (ret)
return ret;
@@ -1347,3 +1783,5 @@ module_spi_driver(ad4691_driver);
MODULE_AUTHOR("Radu Sabau <radu.sabau@analog.com>");
MODULE_DESCRIPTION("Analog Devices AD4691 Family ADC Driver");
MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS("IIO_DMA_BUFFER");
+MODULE_IMPORT_NS("IIO_DMAENGINE_BUFFER");
--
2.43.0
^ permalink raw reply related
* [PATCH v14 1/6] dt-bindings: iio: adc: add AD4691 family
From: Radu Sabau via B4 Relay @ 2026-05-29 10:15 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau, Conor Dooley
In-Reply-To: <20260529-ad4692-multichannel-sar-adc-driver-v14-0-e93c2747dc1f@analog.com>
From: Radu Sabau <radu.sabau@analog.com>
Add DT bindings for the Analog Devices AD4691 family of multichannel
SAR ADCs (AD4691, AD4692, AD4693, AD4694).
The four variants are not compatible with each other: AD4691/AD4692 have
16 analog input channels while AD4693/AD4694 have 8, and AD4691/AD4693
top out at 500 kSPS while AD4692/AD4694 reach 1 MSPS. These differences
in channel count and maximum sample rate require distinct compatible
strings so the driver can select the correct channel configuration and
rate limits.
Acked-by: Conor Dooley <conor.dooley@microchip.com>
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
.../devicetree/bindings/iio/adc/adi,ad4691.yaml | 180 +++++++++++++++++++++
MAINTAINERS | 7 +
2 files changed, 187 insertions(+)
diff --git a/Documentation/devicetree/bindings/iio/adc/adi,ad4691.yaml b/Documentation/devicetree/bindings/iio/adc/adi,ad4691.yaml
new file mode 100644
index 000000000000..af28a0c1cfa9
--- /dev/null
+++ b/Documentation/devicetree/bindings/iio/adc/adi,ad4691.yaml
@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/iio/adc/adi,ad4691.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Analog Devices AD4691 Family Multichannel SAR ADCs
+
+maintainers:
+ - Radu Sabau <radu.sabau@analog.com>
+
+description: |
+ The AD4691 family are high-speed, low-power, multichannel successive
+ approximation register (SAR) analog-to-digital converters (ADCs) with
+ an SPI-compatible serial interface. The ADC supports CNV Burst Mode,
+ where an external PWM drives the CNV pin, and Manual Mode, where CNV
+ is directly tied to the SPI chip-select.
+
+ Datasheets:
+ * https://www.analog.com/en/products/ad4691.html
+ * https://www.analog.com/en/products/ad4692.html
+ * https://www.analog.com/en/products/ad4693.html
+ * https://www.analog.com/en/products/ad4694.html
+
+$ref: /schemas/spi/spi-peripheral-props.yaml#
+
+properties:
+ compatible:
+ enum:
+ - adi,ad4691
+ - adi,ad4692
+ - adi,ad4693
+ - adi,ad4694
+
+ reg:
+ maxItems: 1
+
+ spi-max-frequency:
+ maximum: 40000000
+
+ spi-cpol: true
+ spi-cpha: true
+
+ avdd-supply:
+ description: Analog power supply (4.5V to 5.5V).
+
+ vdd-supply:
+ description:
+ External 1.8V digital core supply. When present, the internal LDO is
+ disabled (LDO_EN = 0). Mutually exclusive with ldo-in-supply.
+
+ ldo-in-supply:
+ description:
+ LDO input supply (2.4V to 5.5V). When present and vdd-supply is absent,
+ the internal LDO generates 1.8V VDD from this input (LDO_EN = 1).
+ Mutually exclusive with vdd-supply.
+
+ vio-supply:
+ description: I/O voltage supply (1.71V to 1.89V or VDD).
+
+ ref-supply:
+ description: External reference voltage supply (2.4V to 5.25V).
+
+ refin-supply:
+ description: Internal reference buffer input supply.
+
+ reset-gpios:
+ description:
+ GPIO line controlling the hardware reset pin (active-low).
+ maxItems: 1
+
+ pwms:
+ description:
+ PWM connected to the CNV pin. When present, selects CNV Burst Mode where
+ the PWM drives the conversion rate. When absent, Manual Mode is used
+ (CNV tied to SPI CS).
+ maxItems: 1
+
+ interrupts:
+ description:
+ Interrupt lines connected to the ADC GP pins. Each GP pin can be
+ physically wired to an interrupt-capable input on the SoC.
+ maxItems: 4
+
+ interrupt-names:
+ description: Names of the interrupt lines, matching the GP pin names.
+ minItems: 1
+ maxItems: 4
+ items:
+ enum:
+ - gp0
+ - gp1
+ - gp2
+ - gp3
+
+ gpio-controller: true
+
+ '#gpio-cells':
+ const: 2
+
+ '#trigger-source-cells':
+ description:
+ This node can act as a trigger source. The single cell in a consumer
+ reference specifies the GP pin number (0-3) used as the trigger output.
+ const: 1
+
+required:
+ - compatible
+ - reg
+ - avdd-supply
+ - vio-supply
+
+allOf:
+ # vdd-supply and ldo-in-supply are mutually exclusive, one is required:
+ # either an external 1.8V VDD is provided or the internal LDO is fed from
+ # ldo-in-supply to generate VDD.
+ - oneOf:
+ - required:
+ - vdd-supply
+ - required:
+ - ldo-in-supply
+ # ref-supply and refin-supply are mutually exclusive, one is required
+ - oneOf:
+ - required:
+ - ref-supply
+ - required:
+ - refin-supply
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+ /* AD4692 in CNV Burst Mode with SPI offload */
+ spi {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ adc@0 {
+ compatible = "adi,ad4692";
+ reg = <0>;
+ spi-cpol;
+ spi-cpha;
+ spi-max-frequency = <40000000>;
+
+ avdd-supply = <&avdd_supply>;
+ ldo-in-supply = <&avdd_supply>;
+ vio-supply = <&vio_supply>;
+ ref-supply = <&ref_5v>;
+
+ reset-gpios = <&gpio0 15 GPIO_ACTIVE_LOW>;
+
+ pwms = <&pwm_gen 0 0>;
+
+ #trigger-source-cells = <1>;
+ };
+ };
+
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+ /* AD4692 in Manual Mode (CNV tied to SPI CS) */
+ spi {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ adc@0 {
+ compatible = "adi,ad4692";
+ reg = <0>;
+ spi-cpol;
+ spi-cpha;
+ spi-max-frequency = <31250000>;
+
+ avdd-supply = <&avdd_supply>;
+ ldo-in-supply = <&avdd_supply>;
+ vio-supply = <&vio_supply>;
+ refin-supply = <&refin_supply>;
+
+ reset-gpios = <&gpio0 15 GPIO_ACTIVE_LOW>;
+ };
+ };
diff --git a/MAINTAINERS b/MAINTAINERS
index c2c6d79275c6..7d31c38921e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1482,6 +1482,13 @@ W: https://ez.analog.com/linux-software-drivers
F: Documentation/devicetree/bindings/iio/adc/adi,ad4170-4.yaml
F: drivers/iio/adc/ad4170-4.c
+ANALOG DEVICES INC AD4691 DRIVER
+M: Radu Sabau <radu.sabau@analog.com>
+L: linux-iio@vger.kernel.org
+S: Supported
+W: https://ez.analog.com/linux-software-drivers
+F: Documentation/devicetree/bindings/iio/adc/adi,ad4691.yaml
+
ANALOG DEVICES INC AD4695 DRIVER
M: Michael Hennerich <michael.hennerich@analog.com>
M: Nuno Sá <nuno.sa@analog.com>
--
2.43.0
^ permalink raw reply related
* [PATCH v14 0/6] iio: adc: ad4691: add driver for AD4691 multichannel SAR ADC family
From: Radu Sabau via B4 Relay @ 2026-05-29 10:14 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau, Conor Dooley
This series adds support for the Analog Devices AD4691 family of
high-speed, low-power multichannel successive approximation register
(SAR) ADCs with an SPI-compatible serial interface.
The family includes:
- AD4691: 16-channel, 500 kSPS
- AD4692: 16-channel, 1 MSPS
- AD4693: 8-channel, 500 kSPS
- AD4694: 8-channel, 1 MSPS
The devices support two operating modes, auto-detected from the device
tree:
- CNV Burst Mode: external PWM drives CNV independently of SPI;
DATA_READY on a GP pin signals end of conversion
- Manual Mode: CNV tied to SPI CS; each SPI transfer reads
the previous conversion result and starts the
next (pipelined N+1 scheme)
A new driver is warranted rather than extending ad4695: the AD4691
data path uses an accumulator-register model — results are read from
AVG_IN registers, with ACC_MASK, ADC_SETUP, DEVICE_SETUP, and
GPIO_MODE registers controlling the sequencer — none of which exist
in AD4695. CNV Burst Mode (PWM drives CNV independently of SPI) and
Manual Mode (pipelined N+1 transfers) also have no equivalent in
AD4695's command-embedded single-cycle protocol.
The series is structured as follows:
1/6 - DT bindings (YAML schema) and MAINTAINERS entry
2/6 - Initial driver: register map via custom regmap callbacks,
IIO read_raw/write_raw, both operating modes, single-channel
reads via internal oscillator (Autonomous Mode)
3/6 - Triggered buffer support: IRQ-driven (DATA_READY on a GP pin
selected via interrupt-names) for CNV Burst Mode; external IIO
trigger for Manual Mode to handle the pipelined N+1 SPI protocol
4/6 - SPI Engine offload support: DMA-backed high-throughput
capture path using the SPI offload subsystem
5/6 - Shared oversampling ratio support for CNV Burst Mode
6/6 - Driver documentation (Documentation/iio/ad4691.rst)
Datasheets:
https://www.analog.com/en/products/ad4691.html
https://www.analog.com/en/products/ad4692.html
https://www.analog.com/en/products/ad4693.html
https://www.analog.com/en/products/ad4694.html
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
Changes in v14:
- clean up volatile_reg — fold result register ranges into the
switch statement (same pattern as ACC_SAT_OVR_REG); remove misleading
comment about debugfs (readable_reg is what gates access); fix commit
message reset description to match the code (assert → deassert → wait
300 µs for internal reset sequence)
- fix manual mode exit bug — add AD4691_EXIT_COMMAND (0x5000)
and ad4691_transfer() to send the raw SPI command that returns the chip
to Autonomous Mode;
- update commit message to reflect single shared OSR
(in_voltage_sampling_frequency / in_voltage_oversampling_ratio are
info_mask_shared_by_all); replace all per-channel osr[N] /
in_voltageN_* references
- Link to v13: https://lore.kernel.org/r/20260525-ad4692-multichannel-sar-adc-driver-v13-0-1b7626d3b35c@analog.com
Changes in v13:
- IIO_CHAN_INFO_SAMP_FREQ and IIO_CHAN_INFO_OVERSAMPLING_RATIO moved to
info_mask_shared_by_all: the hardware uses a single internal oscillator
and a single ACC_DEPTH_IN(0) register shared across all channels, so
per-channel sysfs attributes were incorrect.
- ad4691_reset: refactor to share the fsleep(300) call between hardware
and software reset paths.
- ad4691_read_scan: check spi_sync() return value; skip push and emit
rate-limited error on failure.
- Drop stale info_mask_separate rationale from commit 2 message.
- Documentation updated to reflect shared sampling_frequency and
oversampling_ratio attributes (in_voltage_* instead of in_voltageN_*).
- Link to v12: https://lore.kernel.org/r/20260519-ad4692-multichannel-sar-adc-driver-v12-0-5b335162aa51@analog.com
Changes in v12:
- IIO_DEV_ACQUIRE_DIRECT_MODE moved from set_sampling_freq into write_raw
- enum ad4691_ref_ctrl: remove explicit values
- .sign = 'u' → .format = 'u' throughout — the field has a new name
- device_property_present(dev, "vdd-supply") / device_property_present(dev, "ref-supply")
pattern instead of handling -ENODEV
- AD4691_STATE_RESET_ALL 0x01 → BIT(0)
- spi_device_id: use named initializers instead of (kernel_ulong_t)&... cast
- ad4691_reset: 300 µs sleep belongs after reset_control_deassert, not
between assert and deassert — actual RESETL minimum time (~10 ns) is
covered by overhead; add post-deassert sleep
- Add guard(mutex)(&st->lock) in get_sampling_freq
- Don't register or attach iio_trigger in manual mode
- Remove the if (i >= indio_dev->num_channels - 1) break guards
- ad4691_read_scan return type → void
- Remove (u16) cast: st->scan_tx[k] = AD4691_ADC_CHAN(bit) << 8
- sizeof(st->scan_tx[k]) → sizeof(*st->scan_tx)
- Remove & from iio_buffer_setup_ops in offload structs
- Add args[0] <= 3 check in ad4691_offload_trigger_request
- Embed offload, offload_trigger, trigger_hz directly in struct ad4691_state
- Link to v11: https://lore.kernel.org/r/20260515-ad4692-multichannel-sar-adc-driver-v11-0-eab27d852ac2@analog.com
Changes in v11:
- initial driver: fix commit message — IIO_CHAN_INFO_SAMP_FREQ is
info_mask_separate throughout the series, not info_mask_shared_by_all
- initial driver: readable_reg / volatile_reg: replace open switch ranges
for multi-byte sparse arrays with stride checks; intermediate (unaligned)
addresses are now excluded so debugfs cannot trigger cross-boundary reads
- initial driver: add comment in ad4691_get_sampling_freq noting that
AD4691_OSC_FREQ_REG is non-volatile and served from regcache; no lock
is needed
- triggered buffer: restore .endianness = IIO_BE on AD4691_CHANNEL
scan_type; accidentally dropped in v10
- triggered buffer: add early break in both iio_for_each_active_channel
loops to skip the soft timestamp scan index; prevents out-of-bounds
writes into scan_tx[] and scan_xfers[]
- triggered buffer: fix DMA aliasing in manual mode preenable — set
rx_buf = NULL for the first transfer (pipeline residual) instead of
aliasing it to vals[0] alongside the second transfer
- triggered buffer: add cs_change_delay of 430 ns on channel transfers
to satisfy the minimum CNV high time requirement
- triggered buffer: remove cs_change=1 from the state-reset transfer;
must not be set on the final transfer of a SPI message
- triggered buffer: move enable_irq() from the trigger handler into a
reenable callback on ad4691_trigger_ops, closing the race between
enable_irq and iio_trigger_notify_done; fix reenable return type
(void, not int)
- triggered buffer: use two separate iio_info structs so that
validate_trigger (iio_validate_own_trigger) is enforced only in CNV
burst mode; manual mode must accept external triggers
- triggered buffer: add comment explaining STATE_RESET_ALL sequencing
in CNV burst mode
- triggered buffer: fix STD_SEQ_CONFIG write in both preenable paths —
apply & GENMASK(15, 0) to strip the soft timestamp bit before writing,
matching the existing acc_mask computation
- oversampling: fix commit message — writing oversampling_ratio snaps
target_osc_freq_Hz to preserve integer sampling_frequency read-back;
the two attributes are not orthogonal
- docs: add missing Buffer data format section covering the __be16
software path and the CPU-native offload DMA path
- Link to v10: https://lore.kernel.org/r/20260511-ad4692-multichannel-sar-adc-driver-v10-0-e1fbb1744e38@analog.com
Changes in v10:
- initial driver: depends on REGULATOR || COMPILE_TEST
- triggered buffer: fix vals[] layout — index vals[] with slot counter k,
not channel index i; fixes sparse active_scan_mask producing garbage in
userspace buffer
- triggered buffer: add comment to cnv_burst_buffer_postenable explaining
why sampling_enable()/enable_irq() cannot be called from preenable
- triggered buffer + offload: scan_tx changed from __be16 to u16;
non-offload path uses put_unaligned_be16() (bits_per_word=8); offload
path uses plain native u16 assignments (bits_per_word=16); also fixes
byte-order bug in manual preenable: command byte was in the low byte,
now correctly shifted to the high byte
- oversampling: remove incorrect iio_for_each_active_channel() timestamp
guards; active_scan_mask never includes the timestamp channel
- Link to v9: https://lore.kernel.org/r/20260430-ad4692-multichannel-sar-adc-driver-v9-0-33e439e4fb87@analog.com
Changes in v9:
- devm_regulator_get_enable() → devm_regulator_get_enable_optional() for
vdd-supply. The non-optional variant silently returns a dummy regulator
(ret=0) when the supply is absent from DT, so st->ldo_en was never set
and the internal LDO was never enabled when only ldo-in-supply was provided.
- struct ad4691_channel_info (factoring channels + num_channels out of
struct ad4691_chip_info into a sw_info pointer) is now introduced in
commit 1 instead of commit 2. It is a pure struct cleanup with no
relation to triggered buffers.
- channels and manual_channels fields in struct ad4691_channel_info
are now annotated with __counted_by_ptr(num_channels).
- Link to v8: https://lore.kernel.org/r/20260416-ad4692-multichannel-sar-adc-driver-v8-0-c415bd048fa3@analog.com
Changes in v8:
- dt-bindings: add commit message note explaining why four separate
compatible strings are required (channel count and max rate both
differ between variants);
- initial driver: sizeof(tx) instead of literal 2 in ad4691_reg_read;
U8_MAX/U16_MAX instead of 0xFF/0xFFFF in ad4691_reg_write
- initial driver: extract ad4691_samp_freq_start() helper
- initial driver: fix regulator model — vdd-supply (external 1.8V,
internal LDO disabled) and ldo-in-supply (feeds internal LDO) are
mutually exclusive; add vdd-supply to binding and driver
- initial driver: add comment in ad4691_reset explaining why
devm_reset_control_get_optional_exclusive_deasserted() cannot be
used (datasheet requires ≥300 µs reset pulse)
- initial driver: REF_CTRL and OSC_FREQ_REG: regmap_update_bits /
regmap_assign_bits → regmap_write (reserved bits are 0 at reset)
- initial driver: use dev instead of &spi->dev in devm_iio_device_alloc
- triggered buffer: scan_tx: add __aligned(IIO_DMA_MINALIGN);
scan struct: IIO_DECLARE_DMA_BUFFER_WITH_TS(__be16, vals, 16)
- triggered buffer: full memset of scan_xfers and scan_tx in both
preenable functions; move buffer-dma.h / buffer-dmaengine.h to
commit 4; spi_optimize_message fail path: return ret directly in
cnv_burst_buffer_preenable; reduce devm_iio_trigger_alloc wrapping
- SPI offload: drop AD4691_OFFLOAD_BITS_PER_WORD; use local
bpw = channels[0].scan_type.realbits; num_channels: ARRAY_SIZE - 1
- SPI offload: rename offload_state.spi → .offload; remove spurious
STD_SEQ_CONFIG write from cnv_burst_offload predisable; extract
local acc_mask variable for ACC_MASK_REG write
- SPI offload: sampling_frequency_store: IIO_DEV_ACQUIRE_DIRECT_MODE
for auto-release; remove explicit iio_device_release_direct calls
- oversampling: in_voltageN_sampling_frequency now represents the
effective output rate (osc_freq / osr[N]), matching ad4695
- oversampling: in_voltageN_sampling_frequency_available computed
dynamically from the channel's current OSR; only oscillator entries
divisible by osr[N] shown as effective rates; list becomes sparser
as OSR increases, capping at max_rate / osr[N]
- oversampling: writing sampling_frequency snaps down to the largest
oscillator entry ≤ freq * osr[N] that is divisible by osr[N],
guaranteeing integer read-back; writing oversampling_ratio stores
the new depth only — target_osc_freq_Hz unchanged; the two
attributes are orthogonal
- oversampling: ad4691_write_osc_freq() called from
ad4691_enter_conversion_mode() after manual mode early return,
covering all CNV burst buffer enable paths
- oversampling: (osr + 1) oscillator period wait in single_shot_read
(osr for accumulation, +1 pipeline margin)
- docs: new commit — Documentation/iio/ad4691.rst, userspace-facing
only; oversampling section describes effective-rate SF semantics;
LDO supply section corrected (vdd-supply vs ldo-in-supply)
- Link to v7: https://lore.kernel.org/r/20260409-ad4692-multichannel-sar-adc-driver-v7-0-be375d4df2c5@analog.com
Changes in v7:
- Fix CNV burst triggered-buffer preenable: the state-reset value
transfer had tx_buf assigned the return value of cpu_to_be16()
(an integer) instead of a pointer to a buffer, which would cause
a kernel oops on buffer enable; extend scan_tx[] from 17 to 18
entries to hold the extra slot and fix the pointer assignment
- Extend memset in ad4691_cnv_burst_buffer_preenable to cover the
two state-reset transfer slots (previously left with stale data
across buffer enable/disable cycles if the active channel count
changed)
- Fix format specifier %u -> %lu for NSEC_PER_SEC in
sampling_frequency_show (NSEC_PER_SEC is unsigned long on 32-bit)
- Fix missing iio_device_release_direct() on spi_offload_trigger_-
validate() error path in sampling_frequency_store
- Correct SPI offload commit message: the implementation uses 16-bit
SPI frames (bits_per_word=16, len=2), not 32-bit; storagebits
remains 16 (not promoted to 32); there is no shift=16 for manual
mode; ad4691_manual_channels[] hides IIO_CHAN_INFO_OVERSAMPLING_-
RATIO (not applicable in manual mode), not encodes shift=16
- Link to v6: https://lore.kernel.org/r/20260403-ad4692-multichannel-sar-adc-driver-v6-0-fa2a01a57c4e@analog.com
Changes in v6:
- Replace device.h with dev_printk.h + device/devres.h; add array_size.h
- Rename osc_freqs[] → osc_freqs_Hz[] with explicit [0xN] index designators
- Move loop variable into for() declaration in set_sampling_freq
- Convert multi-line block comment to single-line in single_shot_read
- Replace (u16)~ cast with ~BIT() & GENMASK(15, 0) for ACC_MASK_REG write;
GENMASK(15, 0) is still needed, otherwise maximum value condition line
in reg_write() would fail.
- Extract osc_idx/period_us temporaries in single_shot_read; add comment
- Use devm_regulator_bulk_get_enable() for avdd + vio supplies
- Reformat reset_gpio_probe() comment; remove (GPIOD_OUT_HIGH) detail
- Extract REF_CTRL value into temporary before regmap_update_bits
- Use regmap_assign_bits for OSC_FREQ_REG in config
- Remove ad4691_free_scan_bufs NULL assignments; they are not checked.
- Replace indio_dev->masklength with iio_get_masklength() throughout
- Fix spi_optimize_message error path to use goto err in preenable
- Add iio_buffer_enabled() guard in sampling_frequency_store and
set_oversampling_ratio
- Move ad4691_gpio_setup call from ad4691_config into
setup_triggered_buffer after IRQ lookup; remove duplicate
fwnode_irq_get_byname loop
- Replace oversampling ratio search loop with is_power_of_2 + ilog2
- Link to v5: https://lore.kernel.org/r/20260327-ad4692-multichannel-sar-adc-driver-v5-0-11f789de47b8@analog.com
Changes in v5:
- Reorder datasheets numerically
- Fix interrupt-names: use enum with minItems/maxItems
- Remove if/then block requiring interrupts — driver detail, not hardware constraint
- Remove redundant .shift = 0 from channel macro
- Write max_rate comparison as 1 * HZ_PER_MHZ
- Invert set_sampling_freq loop to use continue
- Fix fsleep() line break; remove blank line in read_raw
- Reorder supply init: vio immediately after avdd
- Move comment rewrites and OSC_FREQ_REG condition into the base driver patch
- Add bit-15 READ comment in reg_read
- Rewrite ldo-in handling with cleaner if/else-if pattern
- Drop redundant refbuf_en = false; invert if (!rst) in reset
- Drop reset_control_assert() — GPIO already asserted at probe
- Use regmap_update_bits/assign_bits in config
- Remove tab-column alignment of state struct members
- Declare osc_freqs[] as const int, eliminating explicit casts
- Drop obvious AUTONOMOUS mode comment
- Rename ACC_COUNT_LIMIT → ACC_DEPTH_IN to match datasheet
- Use bitmap_weight()/bitmap_read() for active_scan_mask access;
add #include <linux/bitmap.h>
- Fix channel macro line-continuation tab alignment
- Use IIO_CHAN_SOFT_TIMESTAMP(8) for 8-channel variants
- Use aligned_s64 ts in scan struct
- Add comment explaining start-index removal in set_sampling_freq
- Remove trailing comma after NULL in buffer_attrs[]
- Add IRQF_NO_AUTOEN rationale comment
- Remove unreachable manual_mode guards in sampling_frequency_show/store
- Remove st->trig; use indio_dev->trig directly
- Move max_speed_hz param to the offload patch where it is used
- Use DIV_ROUND_UP for CNV period; use compound pwm_state initializer
- Move offload fields into a separately allocated sub-struct
- Build TX words via u8* byte-fill; fixes sparse __be32 warnings
- Add three scan types (NORMAL/OFFLOAD_CNV/OFFLOAD_MANUAL) with
get_current_scan_type; triggered buffer path uses storagebits=16
- Fix IIO_CHAN_INFO_SCALE: use iio_get_current_scan_type() for realbits
- Add MODULE_IMPORT_NS("IIO_DMAENGINE_BUFFER")
- Add Documentation/iio/ad4691.rst
- Link to v4: https://lore.kernel.org/r/20260320-ad4692-multichannel-sar-adc-driver-v4-0-052c1050507a@analog.com
Changes in v4:
- dt-bindings: add avdd-supply (required) and ldo-in-supply (optional);
rename vref-supply → ref-supply, vrefin-supply → refin-supply;
corrected reset-gpios polarity (active-high → active-low); remove
clocks and pwm-names; extend interrupts to up to 4 GP pins with
interrupt-names "gp0".."gp3"; reduce #trigger-source-cells to
const: 1 (GP pin number); add gpio-controller / #gpio-cells = <2>;
drop adi,ad4691.h header; update binding examples
- driver: rename CNV Clock Mode → CNV Burst Mode throughout
- driver: add avdd-supply (required) and ldo-in-supply; track ref vs.
refin supply for REFBUF_EN; set LDO_EN in DEVICE_SETUP when ldo-in
is present; add software reset fallback via SPI_CONFIG_A register
- driver: merge ACC_MASK1_REG / ACC_MASK2_REG into ACC_MASK_REG with
a single ADDR_DESCENDING 16-bit SPI write
- driver: remove clocks usage; set PWM rate directly without ref clock
- driver: rename chip info structs (ad4691_chip_info etc.); rename
*chip → *info in state struct; replace adc_mode enum with manual_mode
bool; replace ktime sampling_period with u32 cnv_period_ns
- driver: move IIO_CHAN_INFO_SAMP_FREQ to info_mask_separate with an
available list for the internal oscillator frequency
- driver: use regcache MAPLE instead of RBTREE
- triggered buffer: derive DATA_READY GP pin from interrupt-names in
firmware ("gp0".."gp3") instead of assuming GP0
- triggered buffer: use regmap_update_bits for DEVICE_SETUP mode toggle
to avoid clobbering LDO_EN when toggling MANUAL_MODE bit
- triggered buffer: split buffer setup ops into separate Manual and
CNV Burst variants (mirrors offload path structure)
- SPI offload: promote channel storagebits from 16 to 32 to match DMA
word size; introduce ad4691_manual_channels[] with shift=16 (data in
upper 16 bits of the 32-bit word); update triggered-buffer paths to
the same layout for consistency
- SPI offload: derive GP pin from trigger-source args[0] instead of
hardcoding GP0; split offload buffer setup ops per mode
- replace put_unaligned_be32() + FIELD_PREP() with cpu_to_be32() and
plain bit-shift ops for SPI offload message construction
- multiple reviewer-requested code style and correctness fixes
(Andy Shevchenko, Nuno Sá, Uwe Kleine-König, David Lechner)
- Link to v3: https://lore.kernel.org/r/20260313-ad4692-multichannel-sar-adc-driver-v3-0-b4d14d81a181@analog.com
Changes in v3:
- Replace GPIO reset handling with reset controller framework
- Replace two regmap_write() calls for ACC_MASK1/ACC_MASK2 with regmap_bulk_write()
- Move conv_us declaration closer to its first use
- Derive spi_device/dev from regmap instead of storing st->spi
- ad4691_trigger_handler(): use guard(mutex)() and iio_for_each_active_channel()
- ad4691_setup_triggered_buffer(): return -ENOMEM/-ENOENT directly instead of
wrapping in dev_err_probe(); fix fwnode_irq_get() check (irq <= 0 → irq < 0)
- Add GENMASK defines for SPI offload 32-bit message layout; replace manual
bit-shifts with put_unaligned_be32() + FIELD_PREP()
- Use DIV_ROUND_CLOSEST_ULL() instead of div64_u64()
- ad4691_set_sampling_freq(): fix indentation; drop unnecessary else after return
- ad4691_probe(): use PTR_ERR_OR_ZERO() for devm_spi_offload_get()
- Link to v2: https://lore.kernel.org/r/20260310-ad4692-multichannel-sar-adc-driver-v2-0-d9bb8aeb5e17@analog.com
Changes in v2:
- Drop adi,spi-mode DT property; operating mode now auto-detected
from pwms presence (CNV Clock Mode if present, Manual Mode if not)
- Reduce from 5 operating modes to 2 (CNV Clock Mode, Manual Mode);
Autonomous, SPI Burst and CNV Burst modes removed as user-selectable
modes; Autonomous Mode is now the internal idle/single-shot state
- Single-shot read_raw always uses internal oscillator (Autonomous
Mode), independent of the configured buffer mode
- Replace bulk regulator API with devm_regulator_get_enable() and
devm_regulator_get_enable_read_voltage()
- Use guard(mutex) and IIO_DEV_ACQUIRE_DIRECT_MODE scoped helpers
- Replace enum + indexed chip_info array with named chip_info structs
- Remove product_id field and hardware ID check from probe
- Factor IIO_CHAN_INFO_RAW body into ad4691_single_shot_read() helper
- Use fwnode_irq_get(dev_fwnode(dev), 0); drop interrupt-names from
DT binding
- Use devm_clk_get_enabled(dev, NULL); drop clock-names from DT
binding
- Use spi_write_then_read() for DMA-safe register writes
- Use put_unaligned_be16() for SPI header construction
- fsleep() instead of usleep_range() in single-shot path
- storagebits 24->32 for manual-mode channels (uniform DMA layout)
- Collect full scan into vals[16], single iio_push_to_buffers_with_ts()
- Use pf->timestamp instead of iio_get_time_ns() in trigger handler
- Remove IRQF_TRIGGER_FALLING (comes from firmware/DT)
- Fix offload xfer array size ([17]: N channels + 1 state reset)
- Drop third DT binding example per reviewer request
- Link to v1: https://lore.kernel.org/r/20260305-ad4692-multichannel-sar-adc-driver-v1-0-336229a8dcc7@analog.com
---
Radu Sabau (6):
dt-bindings: iio: adc: add AD4691 family
iio: adc: ad4691: add initial driver for AD4691 family
iio: adc: ad4691: add triggered buffer support
iio: adc: ad4691: add SPI offload support
iio: adc: ad4691: add oversampling support
docs: iio: adc: ad4691: add driver documentation
.../devicetree/bindings/iio/adc/adi,ad4691.yaml | 180 ++
Documentation/iio/ad4691.rst | 227 +++
Documentation/iio/index.rst | 1 +
MAINTAINERS | 9 +
drivers/iio/adc/Kconfig | 16 +
drivers/iio/adc/Makefile | 1 +
drivers/iio/adc/ad4691.c | 2084 ++++++++++++++++++++
7 files changed, 2518 insertions(+)
---
base-commit: 5200f5f493f79f14bbdc349e402a40dfb32f23c8
change-id: 20260302-ad4692-multichannel-sar-adc-driver-78e4d44d24b2
Best regards,
--
Radu Sabau <radu.sabau@analog.com>
^ permalink raw reply
* [PATCH v14 2/6] iio: adc: ad4691: add initial driver for AD4691 family
From: Radu Sabau via B4 Relay @ 2026-05-29 10:15 UTC (permalink / raw)
To: Lars-Peter Clausen, Michael Hennerich, Jonathan Cameron,
David Lechner, Nuno Sá, Andy Shevchenko, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Uwe Kleine-König,
Liam Girdwood, Mark Brown, Linus Walleij, Bartosz Golaszewski,
Philipp Zabel, Jonathan Corbet, Shuah Khan
Cc: linux-iio, devicetree, linux-kernel, linux-pwm, linux-gpio,
linux-doc, Radu Sabau
In-Reply-To: <20260529-ad4692-multichannel-sar-adc-driver-v14-0-e93c2747dc1f@analog.com>
From: Radu Sabau <radu.sabau@analog.com>
Add support for the Analog Devices AD4691 family of high-speed,
low-power multichannel SAR ADCs: AD4691 (16-ch, 500 kSPS),
AD4692 (16-ch, 1 MSPS), AD4693 (8-ch, 500 kSPS) and
AD4694 (8-ch, 1 MSPS).
The driver implements a custom regmap layer over raw SPI to handle the
device's mixed 1/2/3/4-byte register widths and uses the standard IIO
read_raw/write_raw interface for single-channel reads.
The chip idles in Autonomous Mode so that single-shot read_raw can use
the internal oscillator without disturbing the hardware configuration.
Three voltage supply domains are managed: avdd (required), vio, and a
reference supply on either the REF pin (ref-supply, external buffer)
or the REFIN pin (refin-supply, uses the on-chip reference buffer;
REFBUF_EN is set accordingly). Hardware reset is performed by asserting
then deasserting the reset-gpios GPIO line (tRESETL minimum pulse width
is 10 ns, satisfied by function-call overhead); the driver then waits
300 µs for the chip to complete its internal reset sequence. A software
reset via SPI_CONFIG_A is used as fallback when no reset GPIO is
provided.
Accumulator channel masking for single-shot reads uses ACC_MASK_REG via
an ADDR_DESCENDING SPI write, which covers both mask bytes in a single
16-bit transfer.
Reviewed-by: David Lechner <dlechner@baylibre.com>
Signed-off-by: Radu Sabau <radu.sabau@analog.com>
---
MAINTAINERS | 1 +
drivers/iio/adc/Kconfig | 12 +
drivers/iio/adc/Makefile | 1 +
drivers/iio/adc/ad4691.c | 783 +++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 797 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 7d31c38921e8..020c1ffae31b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1488,6 +1488,7 @@ L: linux-iio@vger.kernel.org
S: Supported
W: https://ez.analog.com/linux-software-drivers
F: Documentation/devicetree/bindings/iio/adc/adi,ad4691.yaml
+F: drivers/iio/adc/ad4691.c
ANALOG DEVICES INC AD4695 DRIVER
M: Michael Hennerich <michael.hennerich@analog.com>
diff --git a/drivers/iio/adc/Kconfig b/drivers/iio/adc/Kconfig
index a9dedbb8eb46..c9aca0d08e41 100644
--- a/drivers/iio/adc/Kconfig
+++ b/drivers/iio/adc/Kconfig
@@ -144,6 +144,18 @@ config AD4170_4
To compile this driver as a module, choose M here: the module will be
called ad4170-4.
+config AD4691
+ tristate "Analog Devices AD4691 Family ADC Driver"
+ depends on SPI
+ depends on REGULATOR || COMPILE_TEST
+ select REGMAP
+ help
+ Say yes here to build support for Analog Devices AD4691 Family MuxSAR
+ SPI analog to digital converters (ADC).
+
+ To compile this driver as a module, choose M here: the module will be
+ called ad4691.
+
config AD4695
tristate "Analog Device AD4695 ADC Driver"
depends on SPI
diff --git a/drivers/iio/adc/Makefile b/drivers/iio/adc/Makefile
index 097357d146ba..707dd708912f 100644
--- a/drivers/iio/adc/Makefile
+++ b/drivers/iio/adc/Makefile
@@ -16,6 +16,7 @@ obj-$(CONFIG_AD4080) += ad4080.o
obj-$(CONFIG_AD4130) += ad4130.o
obj-$(CONFIG_AD4134) += ad4134.o
obj-$(CONFIG_AD4170_4) += ad4170-4.o
+obj-$(CONFIG_AD4691) += ad4691.o
obj-$(CONFIG_AD4695) += ad4695.o
obj-$(CONFIG_AD4851) += ad4851.o
obj-$(CONFIG_AD7091R) += ad7091r-base.o
diff --git a/drivers/iio/adc/ad4691.c b/drivers/iio/adc/ad4691.c
new file mode 100644
index 000000000000..e1febf80f21d
--- /dev/null
+++ b/drivers/iio/adc/ad4691.c
@@ -0,0 +1,783 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024-2026 Analog Devices, Inc.
+ * Author: Radu Sabau <radu.sabau@analog.com>
+ */
+#include <linux/array_size.h>
+#include <linux/bitfield.h>
+#include <linux/bitmap.h>
+#include <linux/cleanup.h>
+#include <linux/delay.h>
+#include <linux/dev_printk.h>
+#include <linux/device/devres.h>
+#include <linux/err.h>
+#include <linux/limits.h>
+#include <linux/math.h>
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/regmap.h>
+#include <linux/regulator/consumer.h>
+#include <linux/reset.h>
+#include <linux/spi/spi.h>
+#include <linux/types.h>
+#include <linux/units.h>
+#include <linux/unaligned.h>
+
+#include <linux/iio/iio.h>
+
+#define AD4691_VREF_uV_MIN 2400000
+#define AD4691_VREF_uV_MAX 5250000
+#define AD4691_VREF_2P5_uV_MAX 2750000
+#define AD4691_VREF_3P0_uV_MAX 3250000
+#define AD4691_VREF_3P3_uV_MAX 3750000
+#define AD4691_VREF_4P096_uV_MAX 4500000
+
+#define AD4691_CNV_DUTY_CYCLE_NS 380
+#define AD4691_CNV_HIGH_TIME_NS 430
+
+#define AD4691_SPI_CONFIG_A_REG 0x000
+#define AD4691_SW_RESET (BIT(7) | BIT(0))
+
+#define AD4691_STATUS_REG 0x014
+#define AD4691_CLAMP_STATUS1_REG 0x01A
+#define AD4691_CLAMP_STATUS2_REG 0x01B
+#define AD4691_DEVICE_SETUP 0x020
+#define AD4691_MANUAL_MODE BIT(2)
+#define AD4691_LDO_EN BIT(4)
+#define AD4691_REF_CTRL 0x021
+#define AD4691_REF_CTRL_MASK GENMASK(4, 2)
+#define AD4691_REFBUF_EN BIT(0)
+#define AD4691_OSC_FREQ_REG 0x023
+#define AD4691_OSC_FREQ_MASK GENMASK(3, 0)
+#define AD4691_STD_SEQ_CONFIG 0x025
+#define AD4691_SEQ_ALL_CHANNELS_OFF 0x00
+#define AD4691_SPARE_CONTROL 0x02A
+
+#define AD4691_MAX_CHANNELS 16
+
+#define AD4691_NOOP 0x00
+#define AD4691_ADC_CHAN(ch) ((0x10 + (ch)) << 3)
+
+#define AD4691_OSC_EN_REG 0x180
+#define AD4691_STATE_RESET_REG 0x181
+#define AD4691_STATE_RESET_ALL BIT(0)
+#define AD4691_ADC_SETUP 0x182
+#define AD4691_ADC_MODE_MASK GENMASK(1, 0)
+#define AD4691_CNV_BURST_MODE 0x01
+#define AD4691_AUTONOMOUS_MODE 0x02
+/*
+ * ACC_MASK_REG covers both mask bytes via ADDR_DESCENDING SPI: writing a
+ * 16-bit BE value to 0x185 auto-decrements to 0x184 for the second byte.
+ */
+#define AD4691_ACC_MASK_REG 0x185
+#define AD4691_ACC_DEPTH_IN(n) (0x186 + (n))
+#define AD4691_GPIO_MODE1_REG 0x196
+#define AD4691_GPIO_MODE2_REG 0x197
+#define AD4691_GP_MODE_MASK GENMASK(3, 0)
+#define AD4691_GP_MODE_DATA_READY 0x06
+#define AD4691_GPIO_READ 0x1A0
+#define AD4691_ACC_STATUS_FULL1_REG 0x1B0
+#define AD4691_ACC_STATUS_FULL2_REG 0x1B1
+#define AD4691_ACC_STATUS_OVERRUN1_REG 0x1B2
+#define AD4691_ACC_STATUS_OVERRUN2_REG 0x1B3
+#define AD4691_ACC_STATUS_SAT1_REG 0x1B4
+#define AD4691_ACC_STATUS_SAT2_REG 0x1BE
+#define AD4691_ACC_SAT_OVR_REG(n) (0x1C0 + (n))
+#define AD4691_AVG_IN(n) (0x201 + (2 * (n)))
+#define AD4691_AVG_STS_IN(n) (0x222 + (3 * (n)))
+#define AD4691_ACC_IN(n) (0x252 + (3 * (n)))
+#define AD4691_ACC_STS_DATA(n) (0x283 + (4 * (n)))
+
+
+static const char * const ad4691_supplies[] = { "avdd", "vio" };
+
+enum ad4691_ref_ctrl {
+ AD4691_VREF_2P5,
+ AD4691_VREF_3P0,
+ AD4691_VREF_3P3,
+ AD4691_VREF_4P096,
+ AD4691_VREF_5P0
+};
+
+struct ad4691_channel_info {
+ const struct iio_chan_spec *channels __counted_by_ptr(num_channels);
+ unsigned int num_channels;
+};
+
+struct ad4691_chip_info {
+ const char *name;
+ unsigned int max_rate;
+ const struct ad4691_channel_info *sw_info;
+};
+
+#define AD4691_CHANNEL(ch) \
+ { \
+ .type = IIO_VOLTAGE, \
+ .indexed = 1, \
+ .info_mask_separate = BIT(IIO_CHAN_INFO_RAW), \
+ .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_SCALE) \
+ | BIT(IIO_CHAN_INFO_SAMP_FREQ), \
+ .info_mask_shared_by_all_available = \
+ BIT(IIO_CHAN_INFO_SAMP_FREQ), \
+ .channel = ch, \
+ .scan_type = { \
+ .realbits = 16, \
+ }, \
+ }
+
+static const struct iio_chan_spec ad4691_channels[] = {
+ AD4691_CHANNEL(0),
+ AD4691_CHANNEL(1),
+ AD4691_CHANNEL(2),
+ AD4691_CHANNEL(3),
+ AD4691_CHANNEL(4),
+ AD4691_CHANNEL(5),
+ AD4691_CHANNEL(6),
+ AD4691_CHANNEL(7),
+ AD4691_CHANNEL(8),
+ AD4691_CHANNEL(9),
+ AD4691_CHANNEL(10),
+ AD4691_CHANNEL(11),
+ AD4691_CHANNEL(12),
+ AD4691_CHANNEL(13),
+ AD4691_CHANNEL(14),
+ AD4691_CHANNEL(15),
+};
+
+static const struct iio_chan_spec ad4693_channels[] = {
+ AD4691_CHANNEL(0),
+ AD4691_CHANNEL(1),
+ AD4691_CHANNEL(2),
+ AD4691_CHANNEL(3),
+ AD4691_CHANNEL(4),
+ AD4691_CHANNEL(5),
+ AD4691_CHANNEL(6),
+ AD4691_CHANNEL(7),
+};
+
+static const struct ad4691_channel_info ad4691_sw_info = {
+ .channels = ad4691_channels,
+ .num_channels = ARRAY_SIZE(ad4691_channels),
+};
+
+static const struct ad4691_channel_info ad4693_sw_info = {
+ .channels = ad4693_channels,
+ .num_channels = ARRAY_SIZE(ad4693_channels),
+};
+
+/*
+ * Internal oscillator frequency table. Index is the OSC_FREQ_REG[3:0] value.
+ * Index 0 (1 MHz) is only valid for AD4692/AD4694; AD4691/AD4693 support
+ * up to 500 kHz and use index 1 as their highest valid rate.
+ */
+static const int ad4691_osc_freqs_Hz[] = {
+ [0x0] = 1000000,
+ [0x1] = 500000,
+ [0x2] = 400000,
+ [0x3] = 250000,
+ [0x4] = 200000,
+ [0x5] = 167000,
+ [0x6] = 133000,
+ [0x7] = 125000,
+ [0x8] = 100000,
+ [0x9] = 50000,
+ [0xA] = 25000,
+ [0xB] = 12500,
+ [0xC] = 10000,
+ [0xD] = 5000,
+ [0xE] = 2500,
+ [0xF] = 1250,
+};
+
+static const struct ad4691_chip_info ad4691_chip_info = {
+ .name = "ad4691",
+ .max_rate = 500 * HZ_PER_KHZ,
+ .sw_info = &ad4691_sw_info,
+};
+
+static const struct ad4691_chip_info ad4692_chip_info = {
+ .name = "ad4692",
+ .max_rate = 1 * HZ_PER_MHZ,
+ .sw_info = &ad4691_sw_info,
+};
+
+static const struct ad4691_chip_info ad4693_chip_info = {
+ .name = "ad4693",
+ .max_rate = 500 * HZ_PER_KHZ,
+ .sw_info = &ad4693_sw_info,
+};
+
+static const struct ad4691_chip_info ad4694_chip_info = {
+ .name = "ad4694",
+ .max_rate = 1 * HZ_PER_MHZ,
+ .sw_info = &ad4693_sw_info,
+};
+
+struct ad4691_state {
+ const struct ad4691_chip_info *info;
+ struct regmap *regmap;
+ struct spi_device *spi;
+
+ int vref_uV;
+
+ bool refbuf_en;
+ bool ldo_en;
+ /*
+ * Synchronize access to members of the driver state, and ensure
+ * atomicity of consecutive SPI operations.
+ */
+ struct mutex lock;
+};
+
+static int ad4691_reg_read(void *context, unsigned int reg, unsigned int *val)
+{
+ struct spi_device *spi = context;
+ u8 tx[2], rx[4];
+ int ret;
+
+ /* Set bit 15 to mark the operation as READ. */
+ put_unaligned_be16(0x8000 | reg, tx);
+
+ switch (reg) {
+ case 0 ... AD4691_OSC_FREQ_REG:
+ case AD4691_SPARE_CONTROL ... AD4691_ACC_MASK_REG - 1:
+ case AD4691_ACC_MASK_REG + 1 ... AD4691_ACC_SAT_OVR_REG(15):
+ ret = spi_write_then_read(spi, tx, sizeof(tx), rx, 1);
+ if (ret)
+ return ret;
+ *val = rx[0];
+ return 0;
+ case AD4691_ACC_MASK_REG:
+ case AD4691_STD_SEQ_CONFIG:
+ case AD4691_AVG_IN(0) ... AD4691_AVG_IN(15):
+ ret = spi_write_then_read(spi, tx, sizeof(tx), rx, 2);
+ if (ret)
+ return ret;
+ *val = get_unaligned_be16(rx);
+ return 0;
+ case AD4691_AVG_STS_IN(0) ... AD4691_AVG_STS_IN(15):
+ case AD4691_ACC_IN(0) ... AD4691_ACC_IN(15):
+ ret = spi_write_then_read(spi, tx, sizeof(tx), rx, 3);
+ if (ret)
+ return ret;
+ *val = get_unaligned_be24(rx);
+ return 0;
+ case AD4691_ACC_STS_DATA(0) ... AD4691_ACC_STS_DATA(15):
+ ret = spi_write_then_read(spi, tx, sizeof(tx), rx, 4);
+ if (ret)
+ return ret;
+ *val = get_unaligned_be32(rx);
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int ad4691_reg_write(void *context, unsigned int reg, unsigned int val)
+{
+ struct spi_device *spi = context;
+ u8 tx[4];
+
+ put_unaligned_be16(reg, tx);
+
+ switch (reg) {
+ case 0 ... AD4691_OSC_FREQ_REG:
+ case AD4691_SPARE_CONTROL ... AD4691_ACC_MASK_REG - 1:
+ case AD4691_ACC_MASK_REG + 1 ... AD4691_GPIO_MODE2_REG:
+ if (val > U8_MAX)
+ return -EINVAL;
+ tx[2] = val;
+ return spi_write_then_read(spi, tx, 3, NULL, 0);
+ case AD4691_ACC_MASK_REG:
+ case AD4691_STD_SEQ_CONFIG:
+ if (val > U16_MAX)
+ return -EINVAL;
+ put_unaligned_be16(val, &tx[2]);
+ return spi_write_then_read(spi, tx, 4, NULL, 0);
+ default:
+ return -EINVAL;
+ }
+}
+
+static bool ad4691_volatile_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case AD4691_STATUS_REG:
+ case AD4691_CLAMP_STATUS1_REG:
+ case AD4691_CLAMP_STATUS2_REG:
+ case AD4691_GPIO_READ:
+ case AD4691_ACC_STATUS_FULL1_REG ... AD4691_ACC_STATUS_SAT2_REG:
+ case AD4691_ACC_SAT_OVR_REG(0) ... AD4691_ACC_SAT_OVR_REG(15):
+ case AD4691_AVG_IN(0) ... AD4691_AVG_IN(15):
+ case AD4691_AVG_STS_IN(0) ... AD4691_AVG_STS_IN(15):
+ case AD4691_ACC_IN(0) ... AD4691_ACC_IN(15):
+ case AD4691_ACC_STS_DATA(0) ... AD4691_ACC_STS_DATA(15):
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool ad4691_readable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case 0 ... AD4691_OSC_FREQ_REG:
+ case AD4691_SPARE_CONTROL ... AD4691_ACC_SAT_OVR_REG(15):
+ case AD4691_STD_SEQ_CONFIG:
+ return true;
+ default:
+ break;
+ }
+
+ /*
+ * Multi-byte result registers have non-unit strides; only the base
+ * address of each entry is a valid single-register read.
+ */
+ if (reg >= AD4691_AVG_IN(0) && reg <= AD4691_AVG_IN(15))
+ return (reg - AD4691_AVG_IN(0)) % 2 == 0;
+ if (reg >= AD4691_AVG_STS_IN(0) && reg <= AD4691_AVG_STS_IN(15))
+ return (reg - AD4691_AVG_STS_IN(0)) % 3 == 0;
+ if (reg >= AD4691_ACC_IN(0) && reg <= AD4691_ACC_IN(15))
+ return (reg - AD4691_ACC_IN(0)) % 3 == 0;
+ if (reg >= AD4691_ACC_STS_DATA(0) && reg <= AD4691_ACC_STS_DATA(15))
+ return (reg - AD4691_ACC_STS_DATA(0)) % 4 == 0;
+
+ return false;
+}
+
+static bool ad4691_writeable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case 0 ... AD4691_OSC_FREQ_REG:
+ case AD4691_STD_SEQ_CONFIG:
+ case AD4691_SPARE_CONTROL ... AD4691_GPIO_MODE2_REG:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static const struct regmap_config ad4691_regmap_config = {
+ .reg_bits = 16,
+ .val_bits = 32,
+ .reg_read = ad4691_reg_read,
+ .reg_write = ad4691_reg_write,
+ .volatile_reg = ad4691_volatile_reg,
+ .readable_reg = ad4691_readable_reg,
+ .writeable_reg = ad4691_writeable_reg,
+ .max_register = AD4691_ACC_STS_DATA(15),
+ .cache_type = REGCACHE_MAPLE,
+};
+
+/*
+ * Index 0 in ad4691_osc_freqs_Hz is 1 MHz — valid only for AD4692/AD4694
+ * (max_rate == 1 MHz). AD4691/AD4693 cap at 500 kHz so their valid range
+ * starts at index 1.
+ */
+static unsigned int ad4691_samp_freq_start(const struct ad4691_chip_info *info)
+{
+ return (info->max_rate == 1 * HZ_PER_MHZ) ? 0 : 1;
+}
+
+static int ad4691_get_sampling_freq(struct ad4691_state *st, int *val)
+{
+ unsigned int reg_val;
+ int ret;
+
+ guard(mutex)(&st->lock);
+
+ ret = regmap_read(st->regmap, AD4691_OSC_FREQ_REG, ®_val);
+ if (ret)
+ return ret;
+
+ *val = ad4691_osc_freqs_Hz[FIELD_GET(AD4691_OSC_FREQ_MASK, reg_val)];
+ return IIO_VAL_INT;
+}
+
+static int ad4691_set_sampling_freq(struct ad4691_state *st, int freq)
+{
+ unsigned int start = ad4691_samp_freq_start(st->info);
+
+ guard(mutex)(&st->lock);
+
+ for (unsigned int i = start; i < ARRAY_SIZE(ad4691_osc_freqs_Hz); i++) {
+ if (ad4691_osc_freqs_Hz[i] != freq)
+ continue;
+ return regmap_update_bits(st->regmap, AD4691_OSC_FREQ_REG,
+ AD4691_OSC_FREQ_MASK, i);
+ }
+
+ return -EINVAL;
+}
+
+static int ad4691_read_avail(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan,
+ const int **vals, int *type,
+ int *length, long mask)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ unsigned int start = ad4691_samp_freq_start(st->info);
+
+ switch (mask) {
+ case IIO_CHAN_INFO_SAMP_FREQ:
+ *vals = &ad4691_osc_freqs_Hz[start];
+ *type = IIO_VAL_INT;
+ *length = ARRAY_SIZE(ad4691_osc_freqs_Hz) - start;
+ return IIO_AVAIL_LIST;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int ad4691_single_shot_read(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan, int *val)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+ unsigned int reg_val, osc_idx, period_us;
+ int ret;
+
+ guard(mutex)(&st->lock);
+
+ /* Use AUTONOMOUS mode for single-shot reads. */
+ ret = regmap_write(st->regmap, AD4691_STATE_RESET_REG, AD4691_STATE_RESET_ALL);
+ if (ret)
+ return ret;
+
+ ret = regmap_write(st->regmap, AD4691_STD_SEQ_CONFIG,
+ BIT(chan->channel));
+ if (ret)
+ return ret;
+
+ ret = regmap_write(st->regmap, AD4691_ACC_MASK_REG,
+ ~BIT(chan->channel) & GENMASK(15, 0));
+ if (ret)
+ return ret;
+
+ ret = regmap_read(st->regmap, AD4691_OSC_FREQ_REG, ®_val);
+ if (ret)
+ return ret;
+
+ ret = regmap_write(st->regmap, AD4691_OSC_EN_REG, 1);
+ if (ret)
+ return ret;
+
+ osc_idx = FIELD_GET(AD4691_OSC_FREQ_MASK, reg_val);
+ /* Wait 2 oscillator periods for the conversion to complete. */
+ period_us = DIV_ROUND_UP(2UL * USEC_PER_SEC, ad4691_osc_freqs_Hz[osc_idx]);
+ fsleep(period_us);
+
+ ret = regmap_write(st->regmap, AD4691_OSC_EN_REG, 0);
+ if (ret)
+ return ret;
+
+ ret = regmap_read(st->regmap, AD4691_AVG_IN(chan->channel), ®_val);
+ if (ret)
+ return ret;
+
+ *val = reg_val;
+
+ ret = regmap_write(st->regmap, AD4691_STATE_RESET_REG, AD4691_STATE_RESET_ALL);
+ if (ret)
+ return ret;
+
+ return IIO_VAL_INT;
+}
+
+static int ad4691_read_raw(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan, int *val,
+ int *val2, long info)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ switch (info) {
+ case IIO_CHAN_INFO_RAW: {
+ IIO_DEV_ACQUIRE_DIRECT_MODE(indio_dev, claim);
+ if (IIO_DEV_ACQUIRE_FAILED(claim))
+ return -EBUSY;
+
+ return ad4691_single_shot_read(indio_dev, chan, val);
+ }
+ case IIO_CHAN_INFO_SAMP_FREQ:
+ return ad4691_get_sampling_freq(st, val);
+ case IIO_CHAN_INFO_SCALE:
+ *val = st->vref_uV / (MICRO / MILLI);
+ *val2 = chan->scan_type.realbits;
+ return IIO_VAL_FRACTIONAL_LOG2;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int ad4691_write_raw(struct iio_dev *indio_dev,
+ struct iio_chan_spec const *chan,
+ int val, int val2, long mask)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ IIO_DEV_ACQUIRE_DIRECT_MODE(indio_dev, claim);
+ if (IIO_DEV_ACQUIRE_FAILED(claim))
+ return -EBUSY;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_SAMP_FREQ:
+ return ad4691_set_sampling_freq(st, val);
+ default:
+ return -EINVAL;
+ }
+}
+
+static int ad4691_reg_access(struct iio_dev *indio_dev, unsigned int reg,
+ unsigned int writeval, unsigned int *readval)
+{
+ struct ad4691_state *st = iio_priv(indio_dev);
+
+ guard(mutex)(&st->lock);
+
+ if (readval)
+ return regmap_read(st->regmap, reg, readval);
+
+ return regmap_write(st->regmap, reg, writeval);
+}
+
+static const struct iio_info ad4691_info = {
+ .read_raw = ad4691_read_raw,
+ .write_raw = ad4691_write_raw,
+ .read_avail = ad4691_read_avail,
+ .debugfs_reg_access = ad4691_reg_access,
+};
+
+static int ad4691_regulator_setup(struct ad4691_state *st)
+{
+ struct device *dev = regmap_get_device(st->regmap);
+ int ret;
+
+ ret = devm_regulator_bulk_get_enable(dev, ARRAY_SIZE(ad4691_supplies),
+ ad4691_supplies);
+ if (ret)
+ return dev_err_probe(dev, ret, "Failed to get and enable supplies\n");
+
+ /*
+ * vdd-supply and ldo-in-supply are mutually exclusive:
+ * vdd-supply present → external 1.8V VDD; disable internal LDO.
+ * vdd-supply absent → enable internal LDO fed from ldo-in-supply.
+ * Having both simultaneously is strongly inadvisable per the datasheet.
+ */
+ if (device_property_present(dev, "vdd-supply")) {
+ ret = devm_regulator_get_enable(dev, "vdd");
+ if (ret)
+ return dev_err_probe(dev, ret,
+ "Failed to get and enable VDD\n");
+ } else if (device_property_present(dev, "ldo-in-supply")) {
+ ret = devm_regulator_get_enable(dev, "ldo-in");
+ if (ret)
+ return dev_err_probe(dev, ret,
+ "Failed to get and enable LDO-IN\n");
+ st->ldo_en = true;
+ } else {
+ return dev_err_probe(dev, -EINVAL,
+ "missing one of vdd-supply, ldo-in-supply\n");
+ }
+
+ if (device_property_present(dev, "ref-supply")) {
+ st->vref_uV = devm_regulator_get_enable_read_voltage(dev, "ref");
+ if (st->vref_uV < 0)
+ return dev_err_probe(dev, st->vref_uV,
+ "Failed to get REF supply voltage\n");
+ } else if (device_property_present(dev, "refin-supply")) {
+ st->vref_uV = devm_regulator_get_enable_read_voltage(dev, "refin");
+ if (st->vref_uV < 0)
+ return dev_err_probe(dev, st->vref_uV,
+ "Failed to get REFIN supply voltage\n");
+ st->refbuf_en = true;
+ } else {
+ return dev_err_probe(dev, -EINVAL,
+ "missing one of ref-supply, refin-supply\n");
+ }
+
+ if (st->vref_uV < AD4691_VREF_uV_MIN || st->vref_uV > AD4691_VREF_uV_MAX)
+ return dev_err_probe(dev, -EINVAL,
+ "vref(%d) must be in the range [%u...%u]\n",
+ st->vref_uV, AD4691_VREF_uV_MIN,
+ AD4691_VREF_uV_MAX);
+
+ return 0;
+}
+
+static int ad4691_reset(struct ad4691_state *st)
+{
+ struct device *dev = regmap_get_device(st->regmap);
+ struct reset_control *rst;
+ int ret;
+
+ rst = devm_reset_control_get_optional_exclusive(dev, NULL);
+ if (IS_ERR(rst))
+ return dev_err_probe(dev, PTR_ERR(rst), "Failed to get reset\n");
+
+ if (rst) {
+ /*
+ * Assert the reset line to guarantee a clean reset pulse on
+ * every probe, including driver reloads where the line may
+ * already be deasserted (reset_control_put() does not
+ * re-assert on release). tRESETL (minimum pulse width) = 10 ns
+ * (Table 5); kernel function-call overhead alone exceeds this,
+ * so no explicit delay is needed between assert and deassert.
+ */
+ reset_control_assert(rst);
+ ret = reset_control_deassert(rst);
+ if (ret)
+ return ret;
+ } else {
+ /* No hardware reset available, fall back to software reset. */
+ ret = regmap_write(st->regmap, AD4691_SPI_CONFIG_A_REG,
+ AD4691_SW_RESET);
+ if (ret)
+ return ret;
+ }
+
+ /*
+ * Wait 300 µs (Table 5) for the device to complete its internal reset
+ * sequence before accepting SPI commands.
+ */
+ fsleep(300);
+ return 0;
+}
+
+static int ad4691_config(struct ad4691_state *st)
+{
+ struct device *dev = regmap_get_device(st->regmap);
+ enum ad4691_ref_ctrl ref_val;
+ unsigned int val;
+ int ret;
+
+ switch (st->vref_uV) {
+ case AD4691_VREF_uV_MIN ... AD4691_VREF_2P5_uV_MAX:
+ ref_val = AD4691_VREF_2P5;
+ break;
+ case AD4691_VREF_2P5_uV_MAX + 1 ... AD4691_VREF_3P0_uV_MAX:
+ ref_val = AD4691_VREF_3P0;
+ break;
+ case AD4691_VREF_3P0_uV_MAX + 1 ... AD4691_VREF_3P3_uV_MAX:
+ ref_val = AD4691_VREF_3P3;
+ break;
+ case AD4691_VREF_3P3_uV_MAX + 1 ... AD4691_VREF_4P096_uV_MAX:
+ ref_val = AD4691_VREF_4P096;
+ break;
+ case AD4691_VREF_4P096_uV_MAX + 1 ... AD4691_VREF_uV_MAX:
+ ref_val = AD4691_VREF_5P0;
+ break;
+ default:
+ return dev_err_probe(dev, -EINVAL,
+ "Unsupported vref voltage: %d uV\n",
+ st->vref_uV);
+ }
+
+ val = FIELD_PREP(AD4691_REF_CTRL_MASK, ref_val);
+ if (st->refbuf_en)
+ val |= AD4691_REFBUF_EN;
+
+ ret = regmap_write(st->regmap, AD4691_REF_CTRL, val);
+ if (ret)
+ return dev_err_probe(dev, ret, "Failed to write REF_CTRL\n");
+
+ ret = regmap_assign_bits(st->regmap, AD4691_DEVICE_SETUP,
+ AD4691_LDO_EN, st->ldo_en);
+ if (ret)
+ return dev_err_probe(dev, ret, "Failed to write DEVICE_SETUP\n");
+
+ /*
+ * Set the internal oscillator to the highest rate this chip supports.
+ * Index 0 (1 MHz) exceeds the 500 kHz max of AD4691/AD4693, so those
+ * chips start at index 1 (500 kHz).
+ */
+ ret = regmap_write(st->regmap, AD4691_OSC_FREQ_REG,
+ ad4691_samp_freq_start(st->info));
+ if (ret)
+ return dev_err_probe(dev, ret, "Failed to write OSC_FREQ\n");
+
+ ret = regmap_update_bits(st->regmap, AD4691_ADC_SETUP,
+ AD4691_ADC_MODE_MASK, AD4691_AUTONOMOUS_MODE);
+ if (ret)
+ return dev_err_probe(dev, ret, "Failed to write ADC_SETUP\n");
+
+ return 0;
+}
+
+static int ad4691_probe(struct spi_device *spi)
+{
+ struct device *dev = &spi->dev;
+ struct iio_dev *indio_dev;
+ struct ad4691_state *st;
+ int ret;
+
+ indio_dev = devm_iio_device_alloc(dev, sizeof(*st));
+ if (!indio_dev)
+ return -ENOMEM;
+
+ st = iio_priv(indio_dev);
+ st->spi = spi;
+ st->info = spi_get_device_match_data(spi);
+ if (!st->info)
+ return -ENODEV;
+
+ ret = devm_mutex_init(dev, &st->lock);
+ if (ret)
+ return ret;
+
+ st->regmap = devm_regmap_init(dev, NULL, spi, &ad4691_regmap_config);
+ if (IS_ERR(st->regmap))
+ return dev_err_probe(dev, PTR_ERR(st->regmap),
+ "Failed to initialize regmap\n");
+
+ ret = ad4691_regulator_setup(st);
+ if (ret)
+ return ret;
+
+ ret = ad4691_reset(st);
+ if (ret)
+ return ret;
+
+ ret = ad4691_config(st);
+ if (ret)
+ return ret;
+
+ indio_dev->name = st->info->name;
+ indio_dev->info = &ad4691_info;
+ indio_dev->modes = INDIO_DIRECT_MODE;
+
+ indio_dev->channels = st->info->sw_info->channels;
+ indio_dev->num_channels = st->info->sw_info->num_channels;
+
+ return devm_iio_device_register(dev, indio_dev);
+}
+
+static const struct of_device_id ad4691_of_match[] = {
+ { .compatible = "adi,ad4691", .data = &ad4691_chip_info },
+ { .compatible = "adi,ad4692", .data = &ad4692_chip_info },
+ { .compatible = "adi,ad4693", .data = &ad4693_chip_info },
+ { .compatible = "adi,ad4694", .data = &ad4694_chip_info },
+ { }
+};
+MODULE_DEVICE_TABLE(of, ad4691_of_match);
+
+static const struct spi_device_id ad4691_id[] = {
+ { .name = "ad4691", .driver_data = (kernel_ulong_t)&ad4691_chip_info },
+ { .name = "ad4692", .driver_data = (kernel_ulong_t)&ad4692_chip_info },
+ { .name = "ad4693", .driver_data = (kernel_ulong_t)&ad4693_chip_info },
+ { .name = "ad4694", .driver_data = (kernel_ulong_t)&ad4694_chip_info },
+ { }
+};
+MODULE_DEVICE_TABLE(spi, ad4691_id);
+
+static struct spi_driver ad4691_driver = {
+ .driver = {
+ .name = "ad4691",
+ .of_match_table = ad4691_of_match,
+ },
+ .probe = ad4691_probe,
+ .id_table = ad4691_id,
+};
+module_spi_driver(ad4691_driver);
+
+MODULE_AUTHOR("Radu Sabau <radu.sabau@analog.com>");
+MODULE_DESCRIPTION("Analog Devices AD4691 Family ADC Driver");
+MODULE_LICENSE("GPL");
--
2.43.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox