* Re: [PATCH] hung_task: Allow printing warnings every check interval
From: Tetsuo Handa @ 2019-07-26 13:28 UTC (permalink / raw)
To: Dmitry Safonov, linux-kernel
Cc: Dmitry Safonov, Andrew Morton, Dmitry Vyukov, Ingo Molnar,
Jonathan Corbet, Thomas Gleixner, Peter Zijlstra (Intel),
Vasiliy Khoruzhick, linux-doc, linux-fsdevel
In-Reply-To: <9a919c32-4e0e-ca1b-887f-c329543913d3@i-love.sakura.ne.jp>
On 2019/07/26 20:29, Tetsuo Handa wrote:
> On 2019/07/25 23:25, Dmitry Safonov wrote:
>> Yes, also current distributions already using the counter to print
>> warnings number of times and then silently ignore. I.e., on my Arch
>> Linux setup:
>> hung_task_warnings:10
>
> You can propose changing the default value of hung_task_warnings to -1.
>
> Current patch might be inconvenient because printk() from hung_task_warning(t, false)
> fails to go to consoles when that "t" was blocked for more than "timeout" seconds, for
>
> if (sysctl_hung_task_panic) {
> console_verbose();
> hung_task_show_lock = true;
> hung_task_call_panic = true;
> }
>
> path which is intended to force printk() to go to consoles is ignored by
>
> /* Don't print warings twice */
> if (!sysctl_hung_task_interval_warnings)
> hung_task_warning(t, true);
>
> when panic() should be called. (The vmcore would contain the printk() output which
> was not sent to consoles if kdump is configured. But vmcore is not always available.)
>
>> Yes, that's why it's disabled by default (=0).
>> I tend to agree that printing with KERN_DEBUG may be better, but in my
>> point of view the patch isn't enough justification for patching
>> sched_show_task() and show_stack().
>
> You can propose sched_show_task_log_lvl() and show_stack_log_lvl() like show_trace_log_lvl().
>
> I think that sysctl_hung_task_interval_warnings should not be decremented automatically.
> I guess that that variable should become a boolean which controls whether to report threads
> (with KERN_DEBUG level) which was blocked for more than sysctl_hung_task_check_interval_secs
> seconds (or a tristate which also controls whether the report should be sent to consoles
> (because KERN_DEBUG level likely prevents sending to consoles)), and
> hung_task_warning(t, false) should be called like
>
> if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) {
> if (sysctl_hung_task_interval_warnings)
> hung_task_warning(t, false);
> return;
> }
>
> rather than
>
> if (sysctl_hung_task_interval_warnings)
> hung_task_warning(t, false);
> if (time_is_after_jiffies(t->last_switch_time + timeout * HZ))
> return;
>
> .
>
Well, another direction is to disassociate sysctl_hung_task_panic from
sysctl_hung_task_timeout_secs. Since nobody would want to call panic() when
a thread was blocked for only one second, allow sysctl_hung_task_panic to
specify larger than 1, and interpret it as sysctl_hung_task_timeout_secs for
calling panic(). Roughly speaking:
- if (sysctl_hung_task_panic) {
+ unsigned long panic_timeout = READ_ONCE(sysctl_hung_task_panic)
+ if (panic_timeout == 1 || (panic_timeout > 1 &&
+ (jiffies - t->last_switch_time) / HZ >= panic_timeout)) {
console_verbose();
hung_task_show_lock = true;
hung_task_call_panic = true;
}
If use of different loglevel is not a requirement for you, this would be the simplest.
^ permalink raw reply
* Re: [PATCH 1/7] docs: fix broken doc references due to renames
From: Rob Herring @ 2019-07-26 13:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Paul E. McKenney, Josh Triplett, Steven Rostedt,
Mathieu Desnoyers, Lai Jiangshan, Joel Fernandes, Jonathan Corbet,
Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon,
Alan Stern, Andrea Parri, Boqun Feng, Nicholas Piggin,
David Howells, Jade Alglave, Luc Maranget, Akira Yokosawa,
Daniel Lustig, Jerry Hoemann, Wim Van Sebroeck, Guenter Roeck,
Maarten Lankhorst, Maxime Ripard, Sean Paul, David Airlie,
Daniel Vetter, Ajay Gupta, Don Brace, James E.J. Bottomley,
Martin K. Petersen, rcu, Linux Doc Mailing List, devicetree,
open list:GENERIC INCLUDE/ASM HEADER FILES, LINUX-WATCHDOG,
dri-devel, Linux I2C, esc.storagedev, SCSI, Wolfram Sang
In-Reply-To: <430ed96cb234805d1deb216e8c8559da22cc6bac.1564140865.git.mchehab+samsung@kernel.org>
On Fri, Jul 26, 2019 at 5:47 AM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> Some files got renamed but probably due to some merge conflicts,
> a few references still point to the old locations.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> Acked-by: Wolfram Sang <wsa@the-dreams.de> # I2C part
> Reviewed-by: Jerry Hoemann <jerry.hoemann@hpe.com> # hpwdt.rst
> ---
> Documentation/RCU/rculist_nulls.txt | 2 +-
> Documentation/devicetree/bindings/arm/idle-states.txt | 2 +-
> Documentation/locking/spinlocks.rst | 4 ++--
> Documentation/memory-barriers.txt | 2 +-
> Documentation/translations/ko_KR/memory-barriers.txt | 2 +-
> Documentation/watchdog/hpwdt.rst | 2 +-
> MAINTAINERS | 10 +++++-----
> drivers/gpu/drm/drm_modes.c | 2 +-
> drivers/i2c/busses/i2c-nvidia-gpu.c | 2 +-
> drivers/scsi/hpsa.c | 4 ++--
> 10 files changed, 16 insertions(+), 16 deletions(-)
Acked-by: Rob Herring <robh@kernel.org>
^ permalink raw reply
* Re: [PATCH] hung_task: Allow printing warnings every check interval
From: Dmitry Safonov @ 2019-07-26 13:58 UTC (permalink / raw)
To: Tetsuo Handa, linux-kernel
Cc: Dmitry Safonov, Andrew Morton, Dmitry Vyukov, Ingo Molnar,
Jonathan Corbet, Thomas Gleixner, Peter Zijlstra (Intel),
Vasiliy Khoruzhick, linux-doc, linux-fsdevel
In-Reply-To: <41fd7652-df1f-26f6-aba0-b87ebae07db6@i-love.sakura.ne.jp>
On 7/26/19 2:28 PM, Tetsuo Handa wrote:
> On 2019/07/26 20:29, Tetsuo Handa wrote:
>> On 2019/07/25 23:25, Dmitry Safonov wrote:
>>> Yes, also current distributions already using the counter to print
>>> warnings number of times and then silently ignore. I.e., on my Arch
>>> Linux setup:
>>> hung_task_warnings:10
>>
>> You can propose changing the default value of hung_task_warnings to -1.
>>
>> Current patch might be inconvenient because printk() from hung_task_warning(t, false)
>> fails to go to consoles when that "t" was blocked for more than "timeout" seconds, for
>>
>> if (sysctl_hung_task_panic) {
>> console_verbose();
>> hung_task_show_lock = true;
>> hung_task_call_panic = true;
>> }
>>
>> path which is intended to force printk() to go to consoles is ignored by
>>
>> /* Don't print warings twice */
>> if (!sysctl_hung_task_interval_warnings)
>> hung_task_warning(t, true);
>>
>> when panic() should be called. (The vmcore would contain the printk() output which
>> was not sent to consoles if kdump is configured. But vmcore is not always available.)
Fair enough.
>>> Yes, that's why it's disabled by default (=0).
>>> I tend to agree that printing with KERN_DEBUG may be better, but in my
>>> point of view the patch isn't enough justification for patching
>>> sched_show_task() and show_stack().
>>
>> You can propose sched_show_task_log_lvl() and show_stack_log_lvl() like show_trace_log_lvl().
I'll try, not sure how well it will go..
>>
>> I think that sysctl_hung_task_interval_warnings should not be decremented automatically.
>> I guess that that variable should become a boolean which controls whether to report threads
>> (with KERN_DEBUG level) which was blocked for more than sysctl_hung_task_check_interval_secs
>> seconds (or a tristate which also controls whether the report should be sent to consoles
>> (because KERN_DEBUG level likely prevents sending to consoles)), and
>> hung_task_warning(t, false) should be called like
>>
>> if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) {
>> if (sysctl_hung_task_interval_warnings)
>> hung_task_warning(t, false);
>> return;
>> }
>>
>> rather than
>>
>> if (sysctl_hung_task_interval_warnings)
>> hung_task_warning(t, false);
>> if (time_is_after_jiffies(t->last_switch_time + timeout * HZ))
>> return;
Good point, will do.
> Well, another direction is to disassociate sysctl_hung_task_panic from
> sysctl_hung_task_timeout_secs. Since nobody would want to call panic() when
> a thread was blocked for only one second, allow sysctl_hung_task_panic to
> specify larger than 1, and interpret it as sysctl_hung_task_timeout_secs for
> calling panic(). Roughly speaking:
>
> - if (sysctl_hung_task_panic) {
> + unsigned long panic_timeout = READ_ONCE(sysctl_hung_task_panic)
> + if (panic_timeout == 1 || (panic_timeout > 1 &&
> + (jiffies - t->last_switch_time) / HZ >= panic_timeout)) {
> console_verbose();
> hung_task_show_lock = true;
> hung_task_call_panic = true;
> }
>
> If use of different loglevel is not a requirement for you, this would be the simplest.
No, we consider such messages as notifications/warnings, rather than
complete failures. So, it would be better to hide them from console.
They're also not rate-limited which is a bummer with slow serial
consoles that we've on some devices (9600).
Thanks for the review,
Dmitry
^ permalink raw reply
* Re: [RFC v1 0/4] arm64: MMU enabled kexec kernel relocation
From: James Morse @ 2019-07-26 14:00 UTC (permalink / raw)
To: Pavel Tatashin
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, will,
Linux Doc Mailing List, Linux ARM
In-Reply-To: <CA+CK2bBj9UsQZCLsy-S8c_Kd5SRAPvtdS8s9P_Fdg+VifTbT5w@mail.gmail.com>
Hi Pavel,
On 17/07/2019 20:13, Pavel Tatashin wrote:
>> After a quick skim:
>>
>> This will map 'nomap' regions of memory with cacheable attributes. This is a non-starter.
>> These regions were described by firmware as having content that was/is written with
>> different attributes. The attributes must match whenever it is mapped, otherwise we have a
>> loss of coherency. Mapping this stuff as cacheable means the CPU can prefetch it into the
>> cache whenever it likes.
>
>> It may be important that we do not ever map some of these regions, even though its
>> described as memory. On AMD-Seattle the bottom page of memory is reserved by firmware for
>> its own use; it is made secure-only, and any access causes an
>> external-abort/machine-check. UEFI describes this as 'Reserved', and we preserve this in
>> the kernel as 'nomap'. The equivalent DT support uses memreserve, possibly with the
>> 'nomap' attribute.
>>
>> Mapping a 'new'/unknown region with cacheable attributes can never be safe, even if we
>> trusted kexec-tool to only write the kernel to memory. The host may be using a bigger page
>> size causing more memory to become cacheable than was intended.
>> Linux's EFI support rounds the UEFI memory map to the largest support page size, (and
>> winges about firmware bugs).
>> If we're allowing kexec to load images in a region not described as IORESOURCE_SYSTEM_RAM,
>> that is a bug we should fix.
>
> We are allowing this. If you consider this to be a bug, I will fix it,
> and this will actually simplify the idmap page table. User will
> receive an error during kexec load if a request is made to load into
> !IORESOURCE_SYSTEM_RAM region.
I consider this a bug, but we can see what others think.
This suggests kexec-tools can open /proc/iomem, find a likely looking gap, and try to load
the new kernel between two platform devices.
>> The only way to do this properly is to copy the linear mapping. The arch code has lots of
>> complex code to generate it correctly at boot, we do not want to duplicate it.
>> (this is why hibernate copies the linear mapping)
>
> As I understand, you would like to take a copy of idmap page table,
> and add entries only for segment
> sources and destinations into the new page table?
I don't think there is a need to idmap memory at all. We should copy the linear map so you
know you won't overwrite its page tables as part of loading the new kernel.
> If so, there is a slight problem: arch hook machine_kexec_prepare() is
> called prior to loading segments from userland. We can solve this by
> adding another hook that is called after kimage_terminate().
Yes, all this would need doing as machine_kexec() runs. We preferably need to allocate
memory in this path, or at least have a bitmap of what we can/can't overwrite.
>> These patches do not remove the running page tables from TTBR1. As you overwrite the live
>> page tables you will corrupt the state of the CPU. The page-table walker may access things
>> that aren't memory, cache memory that shouldn't be cached (see above), and allocate
>> conflicting entries in the TLB.
>
> Indeed. However, I was following what is done in create_safe_exec_page():
> https://soleen.com/source/xref/linux/arch/arm64/kernel/hibernate.c?r=af873fce#263
>
> ttbr1 is not removed there. Am I missing something, or is not yet
> configured there?
Hibernate maps a single executable page in ttbr0_el1 that holds its relocation code.
The relocation code then switches ttbr1_el1 to point to the copy of the linear map. See
the 'break_before_make_ttbr_switch' macro in swsusp_arch_suspend_exit().
> I will set ttbr1 to zero page.
>
>> You cannot use the mm page table helpers to build an idmap on arm64. The mm page table
>> helpers have a compile-time VA_BITS, and we support systems where there is no memory below
>> 1<<VA_BITS. (crazy huh!). Picking on AMD-Seattle again: if you boot a 4K 39bit VA kernel,
>> the idmap will have more page table levels than the page table helpers can build. This is
>> why there are special helpers to load the idmap, and twiddle TCR_EL1.T0SZ.
>> You already need to copy the linear-map, so using an idmap is extra work. You want to work
>> with linear-map addresses, you probably need to add the field to the appropriate structure.
>
> OK. Makes sense. I will do the way hibernate setup this table. I was
> indeed following x86, hoping that eventually it would be possible to
> unite: kasan, hibernate, and kexec implementation of this page table.
Our kasan and hibernate code already went a different way. I doubt we can bring them back
in to look like x86, they have different problems to solve.
>> The kexec relocation code still runs at EL2. You can't use a copy of the linear map here
>> as there is only one TTBR on v8.0, and you'd need to setup EL2 as its been torn back to
>> the hyp-stub.
>
> As I understand normally on baremetal kexec runs at EL1 not EL2. On my
> machine is_kernel_in_hyp_mode() == false in cpu_soft_restart.
and is_hyp_mode_available() ?
This depends on which exception-level your bootloader set Linux running. You should get a
boot message that tells you:
| CPU: All CPU(s) started at EL2
is_kernel_in_hyp_mode() is for determining if the kernel is running at EL2. This is the
case if you get a message like:
| kvm [1]: VHE mode initialized successfully
VHE is a v8.1 feature that repaints the system-registers so a kernel written to run at EL1
can run almost unmodified at EL2.
We also have is_hyp_mode_available(), which will return true if all the CPUs booted at EL2.
kexec either runs its relocation code at EL1, or at EL2 if that is where the first kernel
booted. If you call to EL2, the MMU was already off as KVM will reset EL2 to the hyp-stub
in response to the reboot notifiers hardware_disable() call.
(kvm_arch_hardware_disable() calls cpu_hyp_reset())
> This is the reason hibernate posts EL2 in a holding pen while it rewrites
>> all of memory, then calls back to fixup EL2. Keeping the rewrite phase at EL1 means it
>> doesn't need independently tweaking/testing. You need to do something similar, either
>> calling EL2 to start the new image, or disabling the MMU at EL1 to start the new image there.
> OK, I will study how hibernate does this. I was thinking that if we
> are running in EL2 we can simply configure TTBR0_EL2 instead of
> TTBR0_EL1. But, I need to understand this better.
Yes, if you've got a VHE system it can skip the jumping around Exception-levels, but we
still need to support the non-VHE systems.
>>> This patch series works in terms, that I can kexec-reboot both in QEMU
>>> and on a physical machine. However, I do not see performance improvement
>>> during relocation. The performance is just as slow as before with disabled
>>> caches.
>>
>>> Am I missing something? Perhaps, there is some flag that I should also
>>> enable in page table? Please provide me with any suggestions.
>>
>> Some information about the physical machine you tested this on would help.
>> I'm guessing its v8.0, and booted at EL2....
> I am using Broadcom's Stingray SoC.
First hit on google is [0]. Which assuming its the same SoC, says its Cortex-A72, this is
a v8.0 part, it doesn't have VHE. The kernel will be running at EL1, if it supports KVM it
must have booted at EL2.
> Because is_kernel_in_hyp_mode()
> returns false, I believe it is EL1. How can I boot it at EL2?
This check is for VHE.
|static inline bool is_kernel_in_hyp_mode(void)
| {
| return read_sysreg(CurrentEL) == CurrentEL_EL2;
| }
The kernel's early startup in head.S detects if its running at EL2 or EL1. If it has VHE,
it enables the feature and stays at EL2. Otherwise it installs the 'hyp-stub' and drops to
EL1.
On v8.0 the kernel has to run at EL1 because we need the two ttbr registers.
(kernel/user-space). EL2 on these parts only has one. v8.1's VHE adds a second ttbr to
EL2, meaning we can run linux at EL2 and KVM avoids the jumping between exception levels.
You can check for that:
| CPU: All CPU(s) started at EL2
message. If your bootloader is starting the OS at EL1, you will need to speak to the
firmware folk about getting that changed.
> So, I am still confused why I do not see performance improvements
> during relocation on this machine. Any theories?
I assume you started at EL2. You moved EL1's mmu-off code, but cpu_soft_restart() will
call the relocation code at EL2 where the MMU is already off. Because you expected to be
using an idmap, nothing goes wrong, but nothing has changed.
Thanks,
James
[0] https://www.broadcom.com/products/storage/ethernet-storage-adapters-ics/ps1100r
^ permalink raw reply
* Re: [PATCH] docs: phy: Drop duplicate 'be made'
From: Jonathan Corbet @ 2019-07-26 14:15 UTC (permalink / raw)
To: Guido Günther; +Cc: Mauro Carvalho Chehab, linux-doc, linux-kernel
In-Reply-To: <37eecffe6ecd9d64bfdb57a32f34b2c0805dd754.1564134855.git.agx@sigxcpu.org>
On Fri, 26 Jul 2019 11:55:34 +0200
Guido Günther <agx@sigxcpu.org> wrote:
> Fix duplicate words.
>
> Signed-off-by: Guido Günther <agx@sigxcpu.org>
Applied, thanks.
jon
^ permalink raw reply
* [GIT PULL] Documentation fixes for 5.3
From: Jonathan Corbet @ 2019-07-26 14:21 UTC (permalink / raw)
To: Linus Torvalds; +Cc: LKML, linux-doc, Mauro Carvalho Chehab
The following changes since commit
5f9e832c137075045d15cd6899ab0505cfb2ca4b:
Linus 5.3-rc1 (2019-07-21 14:05:38 -0700)
are available in the Git repository at:
git://git.lwn.net/linux.git tags/docs-5.3-1
for you to fetch changes up to d2eba640a4b96bc1bdc0f4a500b8b8d5e16725c8:
docs: phy: Drop duplicate 'be made' (2019-07-26 08:15:26 -0600)
----------------------------------------------------------------
This is mostly a set of follow-on fixes from Mauro fixing various fallout
from the massive RST conversion; a few other small fixes as well.
----------------------------------------------------------------
Federico Vaga (3):
doc:it_IT: align translation to mainline
doc:it_IT: rephrase statement
doc:it_IT: translations in process/
Guido Günther (1):
docs: phy: Drop duplicate 'be made'
Jeremy Cline (1):
docs/vm: transhuge: fix typo in madvise reference
Jonathan Corbet (2):
Merge tag 'v5.3-rc1' into docs-next
Merge branch 'pdf_fixes_v1' of https://git.linuxtv.org/mchehab/experimental into mauro
Marcus Folkesson (1):
docs: driver-api: generic-counter: fix file path to ABI doc
Mauro Carvalho Chehab (15):
docs: powerpc: convert docs to ReST and rename to *.rst
docs: power: add it to to the main documentation index
docs: fix broken doc references due to renames
docs: pdf: add all Documentation/*/index.rst to PDF output
docs: conf.py: add CJK package needed by translations
docs: conf.py: only use CJK if the font is available
scripts/sphinx-pre-install: fix script for RHEL/CentOS
scripts/sphinx-pre-install: don't use LaTeX with CentOS 7
scripts/sphinx-pre-install: fix latexmk dependencies
scripts/sphinx-pre-install: cleanup Gentoo checks
scripts/sphinx-pre-install: seek for Noto CJK fonts for pdf output
docs: load_config.py: avoid needing a conf.py just due to LaTeX docs
docs: remove extra conf.py files
docs: virtual: add it to the documentation body
docs: load_config.py: ensure subdirs end with "/"
Documentation/PCI/pci-error-recovery.rst | 5 +-
Documentation/RCU/rculist_nulls.txt | 2 +-
Documentation/admin-guide/conf.py | 10 --
Documentation/admin-guide/mm/transhuge.rst | 2 +-
Documentation/conf.py | 30 +++-
Documentation/core-api/conf.py | 10 --
Documentation/crypto/conf.py | 10 --
Documentation/dev-tools/conf.py | 10 --
.../devicetree/bindings/arm/idle-states.txt | 2 +-
Documentation/doc-guide/conf.py | 10 --
Documentation/driver-api/80211/conf.py | 10 --
Documentation/driver-api/conf.py | 10 --
Documentation/driver-api/generic-counter.rst | 4 +-
Documentation/driver-api/phy/phy.rst | 4 +-
Documentation/driver-api/pm/conf.py | 10 --
Documentation/filesystems/conf.py | 10 --
Documentation/gpu/conf.py | 10 --
Documentation/index.rst | 3 +
Documentation/input/conf.py | 10 --
Documentation/kernel-hacking/conf.py | 10 --
Documentation/locking/spinlocks.rst | 4 +-
Documentation/maintainer/conf.py | 10 --
Documentation/media/conf.py | 12 --
Documentation/memory-barriers.txt | 2 +-
Documentation/networking/conf.py | 10 --
Documentation/power/index.rst | 2 +-
.../powerpc/{bootwrapper.txt => bootwrapper.rst} | 28 +++-
.../powerpc/{cpu_families.txt => cpu_families.rst} | 23 +--
.../powerpc/{cpu_features.txt => cpu_features.rst} | 6 +-
Documentation/powerpc/{cxl.txt => cxl.rst} | 46 ++++--
.../powerpc/{cxlflash.txt => cxlflash.rst} | 10 +-
.../powerpc/{DAWR-POWER9.txt => dawr-power9.rst} | 15 +-
Documentation/powerpc/{dscr.txt => dscr.rst} | 18 ++-
...ror-recovery.txt => eeh-pci-error-recovery.rst} | 108 +++++++-------
...ssisted-dump.txt => firmware-assisted-dump.rst} | 117 +++++++++-------
Documentation/powerpc/{hvcs.txt => hvcs.rst} | 108 +++++++-------
Documentation/powerpc/index.rst | 34 +++++
Documentation/powerpc/isa-versions.rst | 15 +-
Documentation/powerpc/{mpc52xx.txt => mpc52xx.rst} | 12 +-
...powernv.txt => pci_iov_resource_on_powernv.rst} | 15 +-
Documentation/powerpc/{pmu-ebb.txt => pmu-ebb.rst} | 1 +
Documentation/powerpc/ptrace.rst | 156 +++++++++++++++++++++
Documentation/powerpc/ptrace.txt | 151 --------------------
.../powerpc/{qe_firmware.txt => qe_firmware.rst} | 37 ++---
.../{syscall64-abi.txt => syscall64-abi.rst} | 29 ++--
...ctional_memory.txt => transactional_memory.rst} | 45 +++---
Documentation/process/conf.py | 10 --
Documentation/sh/conf.py | 10 --
Documentation/sound/conf.py | 10 --
Documentation/sphinx/load_config.py | 27 +++-
.../translations/it_IT/doc-guide/sphinx.rst | 19 +--
Documentation/translations/it_IT/process/index.rst | 1 +
.../translations/it_IT/process/kernel-docs.rst | 11 +-
.../it_IT/process/maintainer-pgp-guide.rst | 25 ++--
.../it_IT/process/programming-language.rst | 51 +++++++
.../translations/ko_KR/memory-barriers.txt | 2 +-
Documentation/userspace-api/conf.py | 10 --
Documentation/virtual/kvm/index.rst | 1 +
Documentation/vm/conf.py | 10 --
Documentation/watchdog/hpwdt.rst | 2 +-
Documentation/x86/conf.py | 10 --
MAINTAINERS | 14 +-
arch/powerpc/kernel/exceptions-64s.S | 2 +-
drivers/gpu/drm/drm_modes.c | 2 +-
drivers/i2c/busses/i2c-nvidia-gpu.c | 2 +-
drivers/scsi/hpsa.c | 4 +-
drivers/soc/fsl/qe/qe.c | 2 +-
drivers/tty/hvc/hvcs.c | 2 +-
include/soc/fsl/qe/qe.h | 2 +-
scripts/sphinx-pre-install | 118 +++++++++++++---
70 files changed, 830 insertions(+), 703 deletions(-)
delete mode 100644 Documentation/admin-guide/conf.py
delete mode 100644 Documentation/core-api/conf.py
delete mode 100644 Documentation/crypto/conf.py
delete mode 100644 Documentation/dev-tools/conf.py
delete mode 100644 Documentation/doc-guide/conf.py
delete mode 100644 Documentation/driver-api/80211/conf.py
delete mode 100644 Documentation/driver-api/conf.py
delete mode 100644 Documentation/driver-api/pm/conf.py
delete mode 100644 Documentation/filesystems/conf.py
delete mode 100644 Documentation/gpu/conf.py
delete mode 100644 Documentation/input/conf.py
delete mode 100644 Documentation/kernel-hacking/conf.py
delete mode 100644 Documentation/maintainer/conf.py
delete mode 100644 Documentation/media/conf.py
delete mode 100644 Documentation/networking/conf.py
rename Documentation/powerpc/{bootwrapper.txt => bootwrapper.rst} (93%)
rename Documentation/powerpc/{cpu_families.txt => cpu_families.rst} (95%)
rename Documentation/powerpc/{cpu_features.txt => cpu_features.rst} (97%)
rename Documentation/powerpc/{cxl.txt => cxl.rst} (95%)
rename Documentation/powerpc/{cxlflash.txt => cxlflash.rst} (98%)
rename Documentation/powerpc/{DAWR-POWER9.txt => dawr-power9.rst} (95%)
rename Documentation/powerpc/{dscr.txt => dscr.rst} (91%)
rename Documentation/powerpc/{eeh-pci-error-recovery.txt => eeh-pci-error-recovery.rst} (82%)
rename Documentation/powerpc/{firmware-assisted-dump.txt => firmware-assisted-dump.rst} (80%)
rename Documentation/powerpc/{hvcs.txt => hvcs.rst} (91%)
create mode 100644 Documentation/powerpc/index.rst
rename Documentation/powerpc/{mpc52xx.txt => mpc52xx.rst} (91%)
rename Documentation/powerpc/{pci_iov_resource_on_powernv.txt => pci_iov_resource_on_powernv.rst} (97%)
rename Documentation/powerpc/{pmu-ebb.txt => pmu-ebb.rst} (99%)
create mode 100644 Documentation/powerpc/ptrace.rst
delete mode 100644 Documentation/powerpc/ptrace.txt
rename Documentation/powerpc/{qe_firmware.txt => qe_firmware.rst} (95%)
rename Documentation/powerpc/{syscall64-abi.txt => syscall64-abi.rst} (82%)
rename Documentation/powerpc/{transactional_memory.txt => transactional_memory.rst} (93%)
delete mode 100644 Documentation/process/conf.py
delete mode 100644 Documentation/sh/conf.py
delete mode 100644 Documentation/sound/conf.py
create mode 100644 Documentation/translations/it_IT/process/programming-language.rst
delete mode 100644 Documentation/userspace-api/conf.py
delete mode 100644 Documentation/vm/conf.py
delete mode 100644 Documentation/x86/conf.py
^ permalink raw reply
* Re: [PATCH 1/7] docs: fix broken doc references due to renames
From: Daniel Vetter @ 2019-07-26 14:24 UTC (permalink / raw)
To: Rob Herring
Cc: Mauro Carvalho Chehab, Paul E. McKenney, Josh Triplett,
Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan, Joel Fernandes,
Jonathan Corbet, Mark Rutland, Peter Zijlstra, Ingo Molnar,
Will Deacon, Alan Stern, Andrea Parri, Boqun Feng,
Nicholas Piggin, David Howells, Jade Alglave, Luc Maranget,
Akira Yokosawa, Daniel Lustig, Jerry Hoemann, Wim Van Sebroeck,
Guenter Roeck, Maarten Lankhorst, Maxime Ripard, Sean Paul,
David Airlie, Daniel Vetter, Ajay Gupta, Don Brace,
James E.J. Bottomley, Martin K. Petersen, rcu,
Linux Doc Mailing List, devicetree,
open list:GENERIC INCLUDE/ASM HEADER FILES, LINUX-WATCHDOG,
dri-devel, Linux I2C, esc.storagedev, SCSI, Wolfram Sang
In-Reply-To: <CAL_JsqK_rfHehrKW_NS89BOV0=dYoao0H=zOzG=D-724vKduKw@mail.gmail.com>
On Fri, Jul 26, 2019 at 07:41:35AM -0600, Rob Herring wrote:
> On Fri, Jul 26, 2019 at 5:47 AM Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
> >
> > Some files got renamed but probably due to some merge conflicts,
> > a few references still point to the old locations.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> > Acked-by: Wolfram Sang <wsa@the-dreams.de> # I2C part
> > Reviewed-by: Jerry Hoemann <jerry.hoemann@hpe.com> # hpwdt.rst
> > ---
> > Documentation/RCU/rculist_nulls.txt | 2 +-
> > Documentation/devicetree/bindings/arm/idle-states.txt | 2 +-
> > Documentation/locking/spinlocks.rst | 4 ++--
> > Documentation/memory-barriers.txt | 2 +-
> > Documentation/translations/ko_KR/memory-barriers.txt | 2 +-
> > Documentation/watchdog/hpwdt.rst | 2 +-
> > MAINTAINERS | 10 +++++-----
> > drivers/gpu/drm/drm_modes.c | 2 +-
for the drm part:
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
> > drivers/i2c/busses/i2c-nvidia-gpu.c | 2 +-
> > drivers/scsi/hpsa.c | 4 ++--
> > 10 files changed, 16 insertions(+), 16 deletions(-)
>
> Acked-by: Rob Herring <robh@kernel.org>
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
^ permalink raw reply
* [PATCH v2 05/26] docs: writing-schema.md: convert from markdown to ReST
From: Mauro Carvalho Chehab @ 2019-07-26 12:51 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab, Rob Herring, Mark Rutland, devicetree,
Rob Herring
In-Reply-To: <cover.1564145354.git.mchehab+samsung@kernel.org>
The documentation standard is ReST and not markdown.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Acked-by: Rob Herring <robh@kernel.org>
---
.../{writing-schema.md => writing-schema.rst} | 137 ++++++++++--------
1 file changed, 80 insertions(+), 57 deletions(-)
rename Documentation/devicetree/{writing-schema.md => writing-schema.rst} (48%)
diff --git a/Documentation/devicetree/writing-schema.md b/Documentation/devicetree/writing-schema.rst
similarity index 48%
rename from Documentation/devicetree/writing-schema.md
rename to Documentation/devicetree/writing-schema.rst
index dc032db36262..8f71d1e2ac52 100644
--- a/Documentation/devicetree/writing-schema.md
+++ b/Documentation/devicetree/writing-schema.rst
@@ -1,65 +1,81 @@
-# Writing DeviceTree Bindings in json-schema
+:orphan:
+
+Writing DeviceTree Bindings in json-schema
+==========================================
Devicetree bindings are written using json-schema vocabulary. Schema files are
written in a JSON compatible subset of YAML. YAML is used instead of JSON as it
considered more human readable and has some advantages such as allowing
comments (Prefixed with '#').
-## Schema Contents
+Schema Contents
+---------------
Each schema doc is a structured json-schema which is defined by a set of
top-level properties. Generally, there is one binding defined per file. The
top-level json-schema properties used are:
-- __$id__ - A json-schema unique identifier string. The string must be a valid
-URI typically containing the binding's filename and path. For DT schema, it must
-begin with "http://devicetree.org/schemas/". The URL is used in constructing
-references to other files specified in schema "$ref" properties. A $ref values
-with a leading '/' will have the hostname prepended. A $ref value a relative
-path or filename only will be prepended with the hostname and path components
-of the current schema file's '$id' value. A URL is used even for local files,
-but there may not actually be files present at those locations.
-
-- __$schema__ - Indicates the meta-schema the schema file adheres to.
-
-- __title__ - A one line description on the contents of the binding schema.
-
-- __maintainers__ - A DT specific property. Contains a list of email address(es)
-for maintainers of this binding.
-
-- __description__ - Optional. A multi-line text block containing any detailed
-information about this binding. It should contain things such as what the block
-or device does, standards the device conforms to, and links to datasheets for
-more information.
-
-- __select__ - Optional. A json-schema used to match nodes for applying the
-schema. By default without 'select', nodes are matched against their possible
-compatible string values or node name. Most bindings should not need select.
-
-- __allOf__ - Optional. A list of other schemas to include. This is used to
-include other schemas the binding conforms to. This may be schemas for a
-particular class of devices such as I2C or SPI controllers.
-
-- __properties__ - A set of sub-schema defining all the DT properties for the
-binding. The exact schema syntax depends on whether properties are known,
-common properties (e.g. 'interrupts') or are binding/vendor specific properties.
-
- A property can also define a child DT node with child properties defined
+$id
+ A json-schema unique identifier string. The string must be a valid
+ URI typically containing the binding's filename and path. For DT schema, it must
+ begin with "http://devicetree.org/schemas/". The URL is used in constructing
+ references to other files specified in schema "$ref" properties. A $ref values
+ with a leading '/' will have the hostname prepended. A $ref value a relative
+ path or filename only will be prepended with the hostname and path components
+ of the current schema file's '$id' value. A URL is used even for local files,
+ but there may not actually be files present at those locations.
+
+$schema
+ Indicates the meta-schema the schema file adheres to.
+
+title
+ A one line description on the contents of the binding schema.
+
+maintainers
+ A DT specific property. Contains a list of email address(es)
+ for maintainers of this binding.
+
+description
+ Optional. A multi-line text block containing any detailed
+ information about this binding. It should contain things such as what the block
+ or device does, standards the device conforms to, and links to datasheets for
+ more information.
+
+select
+ Optional. A json-schema used to match nodes for applying the
+ schema. By default without 'select', nodes are matched against their possible
+ compatible string values or node name. Most bindings should not need select.
+
+ allOf
+ Optional. A list of other schemas to include. This is used to
+ include other schemas the binding conforms to. This may be schemas for a
+ particular class of devices such as I2C or SPI controllers.
+
+ properties
+ A set of sub-schema defining all the DT properties for the
+ binding. The exact schema syntax depends on whether properties are known,
+ common properties (e.g. 'interrupts') or are binding/vendor specific properties.
+
+A property can also define a child DT node with child properties defined
under it.
- For more details on properties sections, see 'Property Schema' section.
+For more details on properties sections, see 'Property Schema' section.
-- __patternProperties__ - Optional. Similar to 'properties', but names are regex.
+patternProperties
+ Optional. Similar to 'properties', but names are regex.
-- __required__ - A list of DT properties from the 'properties' section that
-must always be present.
+required
+ A list of DT properties from the 'properties' section that
+ must always be present.
-- __examples__ - Optional. A list of one or more DTS hunks implementing the
-binding. Note: YAML doesn't allow leading tabs, so spaces must be used instead.
+examples
+ Optional. A list of one or more DTS hunks implementing the
+ binding. Note: YAML doesn't allow leading tabs, so spaces must be used instead.
Unless noted otherwise, all properties are required.
-## Property Schema
+Property Schema
+---------------
The 'properties' section of the schema contains all the DT properties for a
binding. Each property contains a set of constraints using json-schema
@@ -89,42 +105,49 @@ The YAML Devicetree format also makes all string values an array and scalar
values a matrix (in order to define groupings) even when only a single value
is present. Single entries in schemas are fixed up to match this encoding.
-## Testing
+Testing
+-------
-### Dependencies
+Dependencies
+~~~~~~~~~~~~
The DT schema project must be installed in order to validate the DT schema
binding documents and validate DTS files using the DT schema. The DT schema
-project can be installed with pip:
+project can be installed with pip::
-`pip3 install git+https://github.com/devicetree-org/dt-schema.git@master`
+ pip3 install git+https://github.com/devicetree-org/dt-schema.git@master
dtc must also be built with YAML output support enabled. This requires that
libyaml and its headers be installed on the host system.
-### Running checks
+Running checks
+~~~~~~~~~~~~~~
The DT schema binding documents must be validated using the meta-schema (the
schema for the schema) to ensure they are both valid json-schema and valid
binding schema. All of the DT binding documents can be validated using the
-`dt_binding_check` target:
+``dt_binding_check`` target::
-`make dt_binding_check`
+ make dt_binding_check
-In order to perform validation of DT source files, use the `dtbs_check` target:
+In order to perform validation of DT source files, use the `dtbs_check` target::
-`make dtbs_check`
+ make dtbs_check
This will first run the `dt_binding_check` which generates the processed schema.
It is also possible to run checks with a single schema file by setting the
-'DT_SCHEMA_FILES' variable to a specific schema file.
+``DT_SCHEMA_FILES`` variable to a specific schema file.
-`make dtbs_check DT_SCHEMA_FILES=Documentation/devicetree/bindings/trivial-devices.yaml`
+::
+ make dtbs_check DT_SCHEMA_FILES=Documentation/devicetree/bindings/trivial-devices.yaml
-## json-schema Resources
-[JSON-Schema Specifications](http://json-schema.org/)
+json-schema Resources
+---------------------
-[Using JSON Schema Book](http://usingjsonschema.com/)
+
+`JSON-Schema Specifications <http://json-schema.org/>`_
+
+`Using JSON Schema Book <http://usingjsonschema.com/>`_
--
2.21.0
^ permalink raw reply related
* [PATCH 7/7] docs: dt: fix a sound binding broken reference
From: Mauro Carvalho Chehab @ 2019-07-26 11:47 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab, Liam Girdwood, Mark Brown, Rob Herring,
Mark Rutland, Maxime Ripard, Chen-Yu Tsai, alsa-devel, devicetree,
linux-arm-kernel
In-Reply-To: <cover.1564140865.git.mchehab+samsung@kernel.org>
Address this rename:
Documentation/devicetree/bindings/sound/{sun4i-i2s.txt -> allwinner,sun4i-a10-i2s.yaml}
Fixes: 0a0ca8e94ca3 ("dt-bindings: sound: Convert Allwinner I2S binding to YAML")
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
Documentation/devicetree/bindings/sound/sun8i-a33-codec.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/devicetree/bindings/sound/sun8i-a33-codec.txt b/Documentation/devicetree/bindings/sound/sun8i-a33-codec.txt
index 2ca3d138528e..7ecf6bd60d27 100644
--- a/Documentation/devicetree/bindings/sound/sun8i-a33-codec.txt
+++ b/Documentation/devicetree/bindings/sound/sun8i-a33-codec.txt
@@ -4,7 +4,7 @@ Allwinner SUN8I audio codec
On Sun8i-A33 SoCs, the audio is separated in different parts:
- A DAI driver. It uses the "sun4i-i2s" driver which is
documented here:
- Documentation/devicetree/bindings/sound/sun4i-i2s.txt
+ Documentation/devicetree/bindings/sound/allwinner,sun4i-a10-i2s.yaml
- An analog part of the codec which is handled as PRCM registers.
See Documentation/devicetree/bindings/sound/sun8i-codec-analog.txt
- An digital part of the codec which is documented in this current
--
2.21.0
^ permalink raw reply related
* [PATCH 3/7] MAINTAINERS: fix reference to net phy ABI file
From: Mauro Carvalho Chehab @ 2019-07-26 11:47 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564140865.git.mchehab+samsung@kernel.org>
The file sysfs-bus-mdio got removed in favor of sysfs-class-net-phydev,
with contained a duplicated set of information.
Fixes: a6cd0d2d493a ("Documentation: net-sysfs: Remove duplicate PHY device documentation")
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
MAINTAINERS | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 51bdbd230174..17ec9abcce52 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6061,7 +6061,7 @@ M: Florian Fainelli <f.fainelli@gmail.com>
M: Heiner Kallweit <hkallweit1@gmail.com>
L: netdev@vger.kernel.org
S: Maintained
-F: Documentation/ABI/testing/sysfs-bus-mdio
+F: Documentation/ABI/testing/sysfs-class-net-phydev
F: Documentation/devicetree/bindings/net/ethernet-phy.yaml
F: Documentation/devicetree/bindings/net/mdio*
F: Documentation/networking/phy.rst
--
2.21.0
^ permalink raw reply related
* [PATCH 4/7] MAINTAINERS: fix a renamed DT reference
From: Mauro Carvalho Chehab @ 2019-07-26 11:47 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564140865.git.mchehab+samsung@kernel.org>
Fix this rename:
Documentation/devicetree/bindings/i2c/{i2c-mv64xxx.txt -> marvell,mv64xxx-i2c.yaml}
Fixes: f8bbde72ef44 ("dt-bindings: i2c: mv64xxx: Add YAML schemas")
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
MAINTAINERS | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 17ec9abcce52..ea7ad0d08c96 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7507,7 +7507,7 @@ I2C MV64XXX MARVELL AND ALLWINNER DRIVER
M: Gregory CLEMENT <gregory.clement@bootlin.com>
L: linux-i2c@vger.kernel.org
S: Maintained
-F: Documentation/devicetree/bindings/i2c/i2c-mv64xxx.txt
+F: Documentation/devicetree/bindings/i2c/marvell,mv64xxx-i2c.yaml
F: drivers/i2c/busses/i2c-mv64xxx.c
I2C OVER PARALLEL PORT
--
2.21.0
^ permalink raw reply related
* Fw: [PATCH v2 05/10] scripts/sphinx-pre-install: don't use LaTeX with CentOS 7
From: Mauro Carvalho Chehab @ 2019-07-26 14:43 UTC (permalink / raw)
To: Jonathan Corbet, linux-doc
Forwarded message:
Date: Fri, 26 Jul 2019 08:31:19 -0300
From: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
To:
Cc: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Subject: [PATCH v2 05/10] scripts/sphinx-pre-install: don't use LaTeX with CentOS 7
There aren't enough texlive packages for LaTeX-based builds
to work on CentOS/RHEL <= 7.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
scripts/sphinx-pre-install | 68 ++++++++++++++++++++++++++++----------
1 file changed, 50 insertions(+), 18 deletions(-)
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 101ddd00bf02..33efadd6c0b6 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -83,6 +83,17 @@ sub check_missing(%)
foreach my $prog (sort keys %missing) {
my $is_optional = $missing{$prog};
+ # At least on some LTS distros like CentOS 7, texlive doesn't
+ # provide all packages we need. When such distros are
+ # detected, we have to disable PDF output.
+ #
+ # So, we need to ignore the packages that distros would
+ # need for LaTeX to work
+ if ($is_optional == 2 && !$pdf) {
+ $optional--;
+ next;
+ }
+
if ($is_optional) {
print "Warning: better to also install \"$prog\".\n";
} else {
@@ -333,10 +344,10 @@ sub give_debian_hints()
if ($pdf) {
check_missing_file("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
- "fonts-dejavu", 1);
+ "fonts-dejavu", 2);
}
- check_program("dvipng", 1) if ($pdf);
+ check_program("dvipng", 2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
@@ -371,22 +382,40 @@ sub give_redhat_hints()
#
# Checks valid for RHEL/CentOS version 7.x.
#
+ my $old = 0;
+ my $rel;
+ $rel = $1 if ($system_release =~ /release\s+(\d+)/);
+
if (!($system_release =~ /Fedora/)) {
$map{"virtualenv"} = "python-virtualenv";
- }
- my $release;
+ if ($rel && $rel < 8) {
+ $old = 1;
+ $pdf = 0;
- $release = $1 if ($system_release =~ /Fedora\s+release\s+(\d+)/);
+ printf("Note: texlive packages on RHEL/CENTOS <= 7 are incomplete. Can't support PDF output\n");
+ printf("If you want to build PDF, please read:\n");
+ printf("\thttps://www.systutorials.com/241660/how-to-install-tex-live-on-centos-7-linux/\n");
+ }
+ } else {
+ if ($rel && $rel < 26) {
+ $old = 1;
+ }
+ }
+ if (!$rel) {
+ printf("Couldn't identify release number\n");
+ $old = 1;
+ $pdf = 0;
+ }
- check_rpm_missing(\@fedora26_opt_pkgs, 1) if ($pdf && $release >= 26);
- check_rpm_missing(\@fedora_tex_pkgs, 1) if ($pdf);
- check_missing_tex(1) if ($pdf);
+ check_rpm_missing(\@fedora26_opt_pkgs, 2) if ($pdf && !$old);
+ check_rpm_missing(\@fedora_tex_pkgs, 2) if ($pdf);
+ check_missing_tex(2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
- if ($release >= 18) {
+ if (!$old) {
# dnf, for Fedora 18+
printf("You should run:\n\n\tsudo dnf install -y $install\n");
} else {
@@ -425,8 +454,8 @@ sub give_opensuse_hints()
"texlive-zapfding",
);
- check_rpm_missing(\@suse_tex_pkgs, 1) if ($pdf);
- check_missing_tex(1) if ($pdf);
+ check_rpm_missing(\@suse_tex_pkgs, 2) if ($pdf);
+ check_missing_tex(2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
@@ -450,7 +479,7 @@ sub give_mageia_hints()
"texlive-fontsextra",
);
- check_rpm_missing(\@tex_pkgs, 1) if ($pdf);
+ check_rpm_missing(\@tex_pkgs, 2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
@@ -473,7 +502,8 @@ sub give_arch_linux_hints()
"texlive-latexextra",
"ttf-dejavu",
);
- check_pacman_missing(\@archlinux_tex_pkgs, 1) if ($pdf);
+ check_pacman_missing(\@archlinux_tex_pkgs, 2) if ($pdf);
+
check_missing(\%map);
return if (!$need && !$optional);
@@ -492,7 +522,7 @@ sub give_gentoo_hints()
);
check_missing_file("/usr/share/fonts/dejavu/DejaVuSans.ttf",
- "media-fonts/dejavu", 1) if ($pdf);
+ "media-fonts/dejavu", 2) if ($pdf);
check_missing(\%map);
@@ -560,7 +590,7 @@ sub check_distros()
my %map = (
"sphinx-build" => "sphinx"
);
- check_missing_tex(1) if ($pdf);
+ check_missing_tex(2) if ($pdf);
check_missing(\%map);
print "I don't know distro $system_release.\n";
print "So, I can't provide you a hint with the install procedure.\n";
@@ -589,11 +619,13 @@ sub check_needs()
check_program("make", 0);
check_program("gcc", 0);
check_python_module("sphinx_rtd_theme", 1) if (!$virtualenv);
- check_program("xelatex", 1) if ($pdf);
check_program("dot", 1);
check_program("convert", 1);
- check_program("rsvg-convert", 1) if ($pdf);
- check_program("latexmk", 1) if ($pdf);
+
+ # Extra PDF files - should use 2 for is_optional
+ check_program("xelatex", 2) if ($pdf);
+ check_program("rsvg-convert", 2) if ($pdf);
+ check_program("latexmk", 2) if ($pdf);
check_distros();
--
2.21.0
Thanks,
Mauro
^ permalink raw reply related
* [PATCH v2 08/10] scripts/sphinx-pre-install: seek for Noto CJK fonts for pdf output
From: Mauro Carvalho Chehab @ 2019-07-26 11:31 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564139914.git.mchehab+samsung@kernel.org>
The translations guide need Noto CJK fonts. So, add a logic that
would suggest its install for distros.
It also fix a few other issues while testing the script
with several distributions.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
scripts/sphinx-pre-install | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 0a5c83aa5f44..3b638c0e1a4f 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -345,6 +345,9 @@ sub give_debian_hints()
if ($pdf) {
check_missing_file("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
"fonts-dejavu", 2);
+
+ check_missing_file("/usr/share/fonts/noto-cjk/NotoSansCJK-Regular.ttc",
+ "fonts-noto-cjk", 2);
}
check_program("dvipng", 2) if ($pdf);
@@ -374,6 +377,7 @@ sub give_redhat_hints()
my @fedora_tex_pkgs = (
"texlive-collection-fontsrecommended",
"texlive-collection-latex",
+ "texlive-xecjk",
"dejavu-sans-fonts",
"dejavu-serif-fonts",
"dejavu-sans-mono-fonts",
@@ -408,6 +412,11 @@ sub give_redhat_hints()
$pdf = 0;
}
+ if ($pdf) {
+ check_missing_file("/usr/share/fonts/google-noto-cjk/NotoSansCJK-Regular.ttc",
+ "google-noto-sans-cjk-ttc-fonts", 2);
+ }
+
check_rpm_missing(\@fedora26_opt_pkgs, 2) if ($pdf && !$old);
check_rpm_missing(\@fedora_tex_pkgs, 2) if ($pdf);
check_missing_tex(2) if ($pdf);
@@ -456,6 +465,11 @@ sub give_opensuse_hints()
$map{"latexmk"} = "texlive-latexmk-bin";
+ # FIXME: add support for installing CJK fonts
+ #
+ # I tried hard, but was unable to find a way to install
+ # "Noto Sans CJK SC" on openSUSE
+
check_rpm_missing(\@suse_tex_pkgs, 2) if ($pdf);
check_missing_tex(2) if ($pdf);
check_missing(\%map);
@@ -483,6 +497,11 @@ sub give_mageia_hints()
$map{"latexmk"} = "texlive-collection-basic";
+ if ($pdf) {
+ check_missing_file("/usr/share/fonts/google-noto-cjk/NotoSansCJK-Regular.ttc",
+ "google-noto-sans-cjk-ttc-fonts", 2);
+ }
+
check_rpm_missing(\@tex_pkgs, 2) if ($pdf);
check_missing(\%map);
@@ -508,6 +527,11 @@ sub give_arch_linux_hints()
);
check_pacman_missing(\@archlinux_tex_pkgs, 2) if ($pdf);
+ if ($pdf) {
+ check_missing_file("/usr/share/fonts/noto-cjk/NotoSansCJK-Regular.ttc",
+ "noto-fonts-cjk", 2);
+ }
+
check_missing(\%map);
return if (!$need && !$optional);
@@ -528,6 +552,11 @@ sub give_gentoo_hints()
check_missing_file("/usr/share/fonts/dejavu/DejaVuSans.ttf",
"media-fonts/dejavu", 2) if ($pdf);
+ if ($pdf) {
+ check_missing_file("/usr/share/fonts/noto-cjk/NotoSansCJKsc-Regular.otf",
+ "media-fonts/noto-cjk", 2);
+ }
+
check_missing(\%map);
return if (!$need && !$optional);
--
2.21.0
^ permalink raw reply related
* [PATCH v2 07/10] scripts/sphinx-pre-install: cleanup Gentoo checks
From: Mauro Carvalho Chehab @ 2019-07-26 11:31 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564139914.git.mchehab+samsung@kernel.org>
On Gentoo, the portage changes for ImageMagick to work are
always suggested, even if already applied. While the two
extra commands should be harmless, add a check to avoid
reporting it without need.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
scripts/sphinx-pre-install | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 8dc13fe95ffe..0a5c83aa5f44 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -533,8 +533,19 @@ sub give_gentoo_hints()
return if (!$need && !$optional);
printf("You should run:\n\n");
- printf("\tsudo su -c 'echo \"media-gfx/imagemagick svg png\" > /etc/portage/package.use/imagemagick'\n");
- printf("\tsudo su -c 'echo \"media-gfx/graphviz cairo pdf\" > /etc/portage/package.use/graphviz'\n");
+
+ my $imagemagick = "media-gfx/imagemagick svg png";
+ my $cairo = "media-gfx/graphviz cairo pdf";
+ my $portage_imagemagick = "/etc/portage/package.use/imagemagick";
+ my $portage_cairo = "/etc/portage/package.use/graphviz";
+
+ if (qx(cat $portage_imagemagick) ne "$imagemagick\n") {
+ printf("\tsudo su -c 'echo \"$imagemagick\" > $portage_imagemagick'\n")
+ }
+ if (qx(cat $portage_cairo) ne "$cairo\n") {
+ printf("\tsudo su -c 'echo \"$cairo\" > $portage_cairo'\n");
+ }
+
printf("\tsudo emerge --ask $install\n");
}
--
2.21.0
^ permalink raw reply related
* [PATCH v2 04/10] scripts/sphinx-pre-install: fix script for RHEL/CentOS
From: Mauro Carvalho Chehab @ 2019-07-26 11:31 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564139914.git.mchehab+samsung@kernel.org>
There's a missing parenthesis at the script, with causes it to
fail to detect non-Fedora releases (e. g. RHEL/CentOS).
Tested with Centos 7.6.1810.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
scripts/sphinx-pre-install | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index f230e65329a2..101ddd00bf02 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -371,7 +371,7 @@ sub give_redhat_hints()
#
# Checks valid for RHEL/CentOS version 7.x.
#
- if (! $system_release =~ /Fedora/) {
+ if (!($system_release =~ /Fedora/)) {
$map{"virtualenv"} = "python-virtualenv";
}
--
2.21.0
^ permalink raw reply related
* [PATCH v2 06/10] scripts/sphinx-pre-install: fix latexmk dependencies
From: Mauro Carvalho Chehab @ 2019-07-26 11:31 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564139914.git.mchehab+samsung@kernel.org>
The name of the package with carries latexmk is different
on two distros:
- On OpenSUSE, latexmk is packaged as "texlive-latexmk-bin"
- On Mageia, latexmk is packaged at "texlive-collection-basic"
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
scripts/sphinx-pre-install | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 33efadd6c0b6..8dc13fe95ffe 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -454,6 +454,8 @@ sub give_opensuse_hints()
"texlive-zapfding",
);
+ $map{"latexmk"} = "texlive-latexmk-bin";
+
check_rpm_missing(\@suse_tex_pkgs, 2) if ($pdf);
check_missing_tex(2) if ($pdf);
check_missing(\%map);
@@ -479,6 +481,8 @@ sub give_mageia_hints()
"texlive-fontsextra",
);
+ $map{"latexmk"} = "texlive-collection-basic";
+
check_rpm_missing(\@tex_pkgs, 2) if ($pdf);
check_missing(\%map);
--
2.21.0
^ permalink raw reply related
* [PATCH v2 05/10] scripts/sphinx-pre-install: don't use LaTeX with CentOS 7
From: Mauro Carvalho Chehab @ 2019-07-26 11:31 UTC (permalink / raw)
Cc: Mauro Carvalho Chehab
In-Reply-To: <cover.1564139914.git.mchehab+samsung@kernel.org>
There aren't enough texlive packages for LaTeX-based builds
to work on CentOS/RHEL <= 7.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
scripts/sphinx-pre-install | 68 ++++++++++++++++++++++++++++----------
1 file changed, 50 insertions(+), 18 deletions(-)
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 101ddd00bf02..33efadd6c0b6 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -83,6 +83,17 @@ sub check_missing(%)
foreach my $prog (sort keys %missing) {
my $is_optional = $missing{$prog};
+ # At least on some LTS distros like CentOS 7, texlive doesn't
+ # provide all packages we need. When such distros are
+ # detected, we have to disable PDF output.
+ #
+ # So, we need to ignore the packages that distros would
+ # need for LaTeX to work
+ if ($is_optional == 2 && !$pdf) {
+ $optional--;
+ next;
+ }
+
if ($is_optional) {
print "Warning: better to also install \"$prog\".\n";
} else {
@@ -333,10 +344,10 @@ sub give_debian_hints()
if ($pdf) {
check_missing_file("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
- "fonts-dejavu", 1);
+ "fonts-dejavu", 2);
}
- check_program("dvipng", 1) if ($pdf);
+ check_program("dvipng", 2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
@@ -371,22 +382,40 @@ sub give_redhat_hints()
#
# Checks valid for RHEL/CentOS version 7.x.
#
+ my $old = 0;
+ my $rel;
+ $rel = $1 if ($system_release =~ /release\s+(\d+)/);
+
if (!($system_release =~ /Fedora/)) {
$map{"virtualenv"} = "python-virtualenv";
- }
- my $release;
+ if ($rel && $rel < 8) {
+ $old = 1;
+ $pdf = 0;
- $release = $1 if ($system_release =~ /Fedora\s+release\s+(\d+)/);
+ printf("Note: texlive packages on RHEL/CENTOS <= 7 are incomplete. Can't support PDF output\n");
+ printf("If you want to build PDF, please read:\n");
+ printf("\thttps://www.systutorials.com/241660/how-to-install-tex-live-on-centos-7-linux/\n");
+ }
+ } else {
+ if ($rel && $rel < 26) {
+ $old = 1;
+ }
+ }
+ if (!$rel) {
+ printf("Couldn't identify release number\n");
+ $old = 1;
+ $pdf = 0;
+ }
- check_rpm_missing(\@fedora26_opt_pkgs, 1) if ($pdf && $release >= 26);
- check_rpm_missing(\@fedora_tex_pkgs, 1) if ($pdf);
- check_missing_tex(1) if ($pdf);
+ check_rpm_missing(\@fedora26_opt_pkgs, 2) if ($pdf && !$old);
+ check_rpm_missing(\@fedora_tex_pkgs, 2) if ($pdf);
+ check_missing_tex(2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
- if ($release >= 18) {
+ if (!$old) {
# dnf, for Fedora 18+
printf("You should run:\n\n\tsudo dnf install -y $install\n");
} else {
@@ -425,8 +454,8 @@ sub give_opensuse_hints()
"texlive-zapfding",
);
- check_rpm_missing(\@suse_tex_pkgs, 1) if ($pdf);
- check_missing_tex(1) if ($pdf);
+ check_rpm_missing(\@suse_tex_pkgs, 2) if ($pdf);
+ check_missing_tex(2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
@@ -450,7 +479,7 @@ sub give_mageia_hints()
"texlive-fontsextra",
);
- check_rpm_missing(\@tex_pkgs, 1) if ($pdf);
+ check_rpm_missing(\@tex_pkgs, 2) if ($pdf);
check_missing(\%map);
return if (!$need && !$optional);
@@ -473,7 +502,8 @@ sub give_arch_linux_hints()
"texlive-latexextra",
"ttf-dejavu",
);
- check_pacman_missing(\@archlinux_tex_pkgs, 1) if ($pdf);
+ check_pacman_missing(\@archlinux_tex_pkgs, 2) if ($pdf);
+
check_missing(\%map);
return if (!$need && !$optional);
@@ -492,7 +522,7 @@ sub give_gentoo_hints()
);
check_missing_file("/usr/share/fonts/dejavu/DejaVuSans.ttf",
- "media-fonts/dejavu", 1) if ($pdf);
+ "media-fonts/dejavu", 2) if ($pdf);
check_missing(\%map);
@@ -560,7 +590,7 @@ sub check_distros()
my %map = (
"sphinx-build" => "sphinx"
);
- check_missing_tex(1) if ($pdf);
+ check_missing_tex(2) if ($pdf);
check_missing(\%map);
print "I don't know distro $system_release.\n";
print "So, I can't provide you a hint with the install procedure.\n";
@@ -589,11 +619,13 @@ sub check_needs()
check_program("make", 0);
check_program("gcc", 0);
check_python_module("sphinx_rtd_theme", 1) if (!$virtualenv);
- check_program("xelatex", 1) if ($pdf);
check_program("dot", 1);
check_program("convert", 1);
- check_program("rsvg-convert", 1) if ($pdf);
- check_program("latexmk", 1) if ($pdf);
+
+ # Extra PDF files - should use 2 for is_optional
+ check_program("xelatex", 2) if ($pdf);
+ check_program("rsvg-convert", 2) if ($pdf);
+ check_program("latexmk", 2) if ($pdf);
check_distros();
--
2.21.0
^ permalink raw reply related
* Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues
From: Thomas Gleixner @ 2019-07-26 14:47 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Greg Kroah-Hartman, linux-kernel, security, linux-doc,
Jiri Kosina, Mauro Carvalho Chehab
In-Reply-To: <20190725151302.16a3e0e3@lwn.net>
On Thu, 25 Jul 2019, Jonathan Corbet wrote:
> > Note, this document has gone through numerous reviews by a number of
> > kernel developers, developers at some of the Linux distros, as well as
> > all of the lawyers from almost all open source-related companies. It's
> > been sitting on my local drive with no comments for a few months now,
> > and it's about time to get this out and merged properly.
> >
> > If anyone has any final comments, please let me know.
>
> I do think it could benefit from a pass for basic language issues; I can do
> that if such an effort would be welcome.
Definitely!
> > +
> > +The list is encrypted and email to the list can be sent by either PGP or
> > +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> > +certificate. The list's PGP key and S/MIME certificate are available from
> > +https://www.kernel.org/....
>
> Somebody needs to fill in some dots there...:)
Yes. I need to sort that out with Konstantin before the thing gets merged,
but we wanted to give it a wider audience in general.
> > +The hardware security team identifies the developers (domain experts) which
> > +form the initial response team for a particular issue. The initial response
>
> s/which form/who will form/
>
> > +team can bring in further developers (domain experts) to address the issue
> > +in the best technical way.
>
> Does the reporter get any say in who can be in this group? That should
> probably be made explicit either way.
See below.
> > +The hardware security team will provide the disclosing party a list of
> > +developers (domain experts) who should be informed initially about the
> > +issue after confirming with the developers that they will adhere to this
> > +Memorandum of Understanding and the documented process. These developers
> > +form the initial response team and will be responsible for handling the
> > +issue after initial contact. The hardware security team is supporting the
> > +response team, but is not necessarily involved in the mitigation
> > +development process.
>
> Again, "should be informed" is conditional, suggesting that the reporter
> might have some sort of veto power. But the actual policy is not clear.
Yes and no. That's a tricky field. We surely need some agreement with the
reporter/owner, but of course we want as much freedom here as we can
get. The past issues were always a pain when we had the need to get a
particular expert into the group.
> > +While individual developers might be covered by a non-disclosure agreement
> > +via their employer, they cannot enter individual non-disclosure agreements
> > +in their role as Linux kernel developers. They will, however, adhere to
> > +this documented process and the Memorandum of Understanding.
>
> They will *agree to* adhere ... We expect that actual adherence will be
> the case but there is no way (even if an NDA were involved) to guarantee
> that.
Correct.
> > +Disclosure
> > +""""""""""
> > +
> > +The disclosing party provides detailed information to the initial response
> > +team via the specific encrypted mailing-list.
> > +
> > +From our experience the technical documentation of these issues is usually
> > +a sufficient starting point and further technical clarification is best
> > +done via email.
> > +
> > +Mitigation development
> > +""""""""""""""""""""""
> > +
> > +The initial response team sets up an encrypted mailing-list or repurposes
> > +an existing one if appropriate. The disclosing party should provide a list
> > +of contacts for all other parties who have already been, or should be
> > +informed about the issue. The response team contacts these parties so they
> > +can name experts who should be subscribed to the mailing-list.
> > +
> > +Using a mailing-list is close to the normal Linux development process and
> > +has been successfully used in developing mitigations for various hardware
> > +security issues in the past.
> > +
> > +The mailing-list operates in the same way as normal Linux development.
> > +Patches are posted, discussed and reviewed and if agreed on applied to a
> > +non-public git repository which is only accessible to the participating
> > +developers via a secure connection. The repository contains the main
> > +development branch against the mainline kernel and backport branches for
> > +stable kernel versions as necessary.
>
> Do we want to envision a KPTI-like situation where the mitigation can be
> developed publicly? Or perhaps just handle any such case if and when it
> ever arises?
Yes, we handle that when it happens which is hopefully never.
> > +Process ambassadors
> > +-------------------
> > +
> > +For assistance with this process we have established ambassadors in various
> > +organizations, who can answer questions about or provide guidance on the
> > +reporting process and further handling. Ambassadors are not involved in the
> > +disclosure of a particular issue, unless requested by a response team or by
> > +an involved disclosed party. The current ambassadors list:
> > +
> > + ============== ========================================================
> > + ARM
> > + AMD
> > + IBM
> > + Intel
> > + Qualcomm
> > +
> > + Microsoft
> > + VMware
> > + XEN
> > +
> > + Canonical
> > + Debian
> > + Oracle
> > + Redhat
> > + Suse Jiri Kosina <jkosina@suse.com>
> > +
> > + Amazon
> > + Google
> > + ============== ========================================================
>
> Having companies without names seems a little weird. Unless perhaps you
> have people teed up to add their names in follow-on patches?
We already talked to companies and the names should come forth before this
is finished.
> > +Encrypted mailing-lists
> > +-----------------------
> > +
> > +We use encrypted mailing-lists for communication. The operating principle
> > +of these lists is that email sent to the list is encrypted either with the
> > +list's PGP key or with the list's S/MIME certificate. The mailing-list
> > +software decrypts the email and re-encrypts it individually for each
> > +subscriber with the subscriber's PGP key or S/MIME certificate. Details
> > +about the mailing-list software and the setup which is used to ensure the
> > +security of the lists and protection of the data can be found here:
> > +https://www.kernel.org/....
>
> That URL is also in need of completion.
>
> The topic of encrypted mailing lists is visited several times in this
> document; I wonder if that could be coalesced somehow?
Suggestions welcome.
> > +Each subscriber needs to send a subscription request to the response team
> > +by email. The email must be signed with the subscriber's PGP key or S/MIME
> > +certificate. If a PGP key is used, it must be available from a public key
> > +server and is ideally connected to the Linux kernel's PGP web of trust. See
> > +also: https://www.kernel.org/signature.html.
>
> The "public key server" thing isn't working quite as well as it was; does
> this requirement need to be revisited?
I think so. That was written way before that mess happened.
Thanks,
tglx
^ permalink raw reply
* [PATCH] MAINTAINERS: add entries for some documentation scripts
From: Mauro Carvalho Chehab @ 2019-07-26 14:52 UTC (permalink / raw)
To: Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
Jonathan Corbet
There are some documentation scripts I wrote with doesn't
have any maintainer at maintainer's file.
Add them to the DOCUMENTATION entry, in order to have
Jon and linux-doc ML c/c on those patches, plus a new
entry to ensure that I'll be c/c when people send patches
to those.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
MAINTAINERS | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 4de2f288d1ec..a09355672212 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4947,7 +4947,9 @@ M: Jonathan Corbet <corbet@lwn.net>
L: linux-doc@vger.kernel.org
S: Maintained
F: Documentation/
+F: scripts/documentation-file-ref-check
F: scripts/kernel-doc
+F: scripts/sphinx-pre-install
X: Documentation/ABI/
X: Documentation/firmware-guide/acpi/
X: Documentation/devicetree/
@@ -4963,6 +4965,14 @@ L: linux-doc@vger.kernel.org
S: Maintained
F: Documentation/translations/it_IT
+DOCUMENTATION SCRIPTS
+M: Mauro Carvalho Chehab <mchehab@kernel.org>
+L: linux-doc@vger.kernel.org
+S: Maintained
+F: scripts/documentation-file-ref-check
+F: scripts/sphinx-pre-install
+F: Documentation/sphinx/parse-headers.pl
+
DONGWOON DW9714 LENS VOICE COIL DRIVER
M: Sakari Ailus <sakari.ailus@linux.intel.com>
L: linux-media@vger.kernel.org
--
2.21.0
^ permalink raw reply related
* [PATCH v2 1/2] mm/page_idle: Add per-pid idle page tracking using virtual indexing
From: Joel Fernandes (Google) @ 2019-07-26 15:08 UTC (permalink / raw)
To: linux-kernel
Cc: Joel Fernandes (Google), Alexey Dobriyan, Andrew Morton,
Brendan Gregg, Christian Hansen, dancol, fmayer, joaodias, joelaf,
Jonathan Corbet, Kees Cook, kernel-team, linux-api, linux-doc,
linux-fsdevel, linux-mm, Michal Hocko, Mike Rapoport, minchan,
namhyung, Roman Gushchin, Stephen Rothwell, surenb, tkjos,
Vladimir Davydov, Vlastimil Babka, wvw
The page_idle tracking feature currently requires looking up the pagemap
for a process followed by interacting with /sys/kernel/mm/page_idle.
Looking up PFN from pagemap in Android devices is not supported by
unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.
This patch adds support to directly interact with page_idle tracking at
the PID level by introducing a /proc/<pid>/page_idle file. It follows
the exact same semantics as the global /sys/kernel/mm/page_idle, but now
looking up PFN through pagemap is not needed since the interface uses
virtual frame numbers, and at the same time also does not require
SYS_ADMIN.
In Android, we are using this for the heap profiler (heapprofd) which
profiles and pin points code paths which allocates and leaves memory
idle for long periods of time. This method solves the security issue
with userspace learning the PFN, and while at it is also shown to yield
better results than the pagemap lookup, the theory being that the window
where the address space can change is reduced by eliminating the
intermediate pagemap look up stage. In virtual address indexing, the
process's mmap_sem is held for the duration of the access.
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
v1 -> v2:
Mark swap ptes as idle (Minchan)
Avoid need for GFP_ATOMIC (Andrew)
Get rid of idle_page_list lock by moving list to stack
Internal review -> v1:
Fixes from Suren.
Corrections to change log, docs (Florian, Sandeep)
fs/proc/base.c | 3 +
fs/proc/internal.h | 1 +
fs/proc/task_mmu.c | 57 +++++++
include/linux/page_idle.h | 4 +
mm/page_idle.c | 342 +++++++++++++++++++++++++++++++++-----
5 files changed, 362 insertions(+), 45 deletions(-)
diff --git a/fs/proc/base.c b/fs/proc/base.c
index 77eb628ecc7f..a58dd74606e9 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -3021,6 +3021,9 @@ static const struct pid_entry tgid_base_stuff[] = {
REG("smaps", S_IRUGO, proc_pid_smaps_operations),
REG("smaps_rollup", S_IRUGO, proc_pid_smaps_rollup_operations),
REG("pagemap", S_IRUSR, proc_pagemap_operations),
+#ifdef CONFIG_IDLE_PAGE_TRACKING
+ REG("page_idle", S_IRUSR|S_IWUSR, proc_page_idle_operations),
+#endif
#endif
#ifdef CONFIG_SECURITY
DIR("attr", S_IRUGO|S_IXUGO, proc_attr_dir_inode_operations, proc_attr_dir_operations),
diff --git a/fs/proc/internal.h b/fs/proc/internal.h
index cd0c8d5ce9a1..bc9371880c63 100644
--- a/fs/proc/internal.h
+++ b/fs/proc/internal.h
@@ -293,6 +293,7 @@ extern const struct file_operations proc_pid_smaps_operations;
extern const struct file_operations proc_pid_smaps_rollup_operations;
extern const struct file_operations proc_clear_refs_operations;
extern const struct file_operations proc_pagemap_operations;
+extern const struct file_operations proc_page_idle_operations;
extern unsigned long task_vsize(struct mm_struct *);
extern unsigned long task_statm(struct mm_struct *,
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 4d2b860dbc3f..11ccc53da38e 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1642,6 +1642,63 @@ const struct file_operations proc_pagemap_operations = {
.open = pagemap_open,
.release = pagemap_release,
};
+
+#ifdef CONFIG_IDLE_PAGE_TRACKING
+static ssize_t proc_page_idle_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ int ret;
+ struct task_struct *tsk = get_proc_task(file_inode(file));
+
+ if (!tsk)
+ return -EINVAL;
+ ret = page_idle_proc_read(file, buf, count, ppos, tsk);
+ put_task_struct(tsk);
+ return ret;
+}
+
+static ssize_t proc_page_idle_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ int ret;
+ struct task_struct *tsk = get_proc_task(file_inode(file));
+
+ if (!tsk)
+ return -EINVAL;
+ ret = page_idle_proc_write(file, (char __user *)buf, count, ppos, tsk);
+ put_task_struct(tsk);
+ return ret;
+}
+
+static int proc_page_idle_open(struct inode *inode, struct file *file)
+{
+ struct mm_struct *mm;
+
+ mm = proc_mem_open(inode, PTRACE_MODE_READ);
+ if (IS_ERR(mm))
+ return PTR_ERR(mm);
+ file->private_data = mm;
+ return 0;
+}
+
+static int proc_page_idle_release(struct inode *inode, struct file *file)
+{
+ struct mm_struct *mm = file->private_data;
+
+ if (mm)
+ mmdrop(mm);
+ return 0;
+}
+
+const struct file_operations proc_page_idle_operations = {
+ .llseek = mem_lseek, /* borrow this */
+ .read = proc_page_idle_read,
+ .write = proc_page_idle_write,
+ .open = proc_page_idle_open,
+ .release = proc_page_idle_release,
+};
+#endif /* CONFIG_IDLE_PAGE_TRACKING */
+
#endif /* CONFIG_PROC_PAGE_MONITOR */
#ifdef CONFIG_NUMA
diff --git a/include/linux/page_idle.h b/include/linux/page_idle.h
index 1e894d34bdce..f1bc2640d85e 100644
--- a/include/linux/page_idle.h
+++ b/include/linux/page_idle.h
@@ -106,6 +106,10 @@ static inline void clear_page_idle(struct page *page)
}
#endif /* CONFIG_64BIT */
+ssize_t page_idle_proc_write(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos, struct task_struct *tsk);
+ssize_t page_idle_proc_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos, struct task_struct *tsk);
#else /* !CONFIG_IDLE_PAGE_TRACKING */
static inline bool page_is_young(struct page *page)
diff --git a/mm/page_idle.c b/mm/page_idle.c
index 295512465065..d8a14955c39d 100644
--- a/mm/page_idle.c
+++ b/mm/page_idle.c
@@ -5,12 +5,15 @@
#include <linux/sysfs.h>
#include <linux/kobject.h>
#include <linux/mm.h>
-#include <linux/mmzone.h>
-#include <linux/pagemap.h>
-#include <linux/rmap.h>
#include <linux/mmu_notifier.h>
+#include <linux/mmzone.h>
#include <linux/page_ext.h>
#include <linux/page_idle.h>
+#include <linux/pagemap.h>
+#include <linux/rmap.h>
+#include <linux/sched/mm.h>
+#include <linux/swap.h>
+#include <linux/swapops.h>
#define BITMAP_CHUNK_SIZE sizeof(u64)
#define BITMAP_CHUNK_BITS (BITMAP_CHUNK_SIZE * BITS_PER_BYTE)
@@ -25,18 +28,13 @@
* page tracking. With such an indicator of user pages we can skip isolated
* pages, but since there are not usually many of them, it will hardly affect
* the overall result.
- *
- * This function tries to get a user memory page by pfn as described above.
*/
-static struct page *page_idle_get_page(unsigned long pfn)
+static struct page *page_idle_get_page(struct page *page_in)
{
struct page *page;
pg_data_t *pgdat;
- if (!pfn_valid(pfn))
- return NULL;
-
- page = pfn_to_page(pfn);
+ page = page_in;
if (!page || !PageLRU(page) ||
!get_page_unless_zero(page))
return NULL;
@@ -51,6 +49,18 @@ static struct page *page_idle_get_page(unsigned long pfn)
return page;
}
+/*
+ * This function tries to get a user memory page by pfn as described above.
+ */
+static struct page *page_idle_get_page_pfn(unsigned long pfn)
+{
+
+ if (!pfn_valid(pfn))
+ return NULL;
+
+ return page_idle_get_page(pfn_to_page(pfn));
+}
+
static bool page_idle_clear_pte_refs_one(struct page *page,
struct vm_area_struct *vma,
unsigned long addr, void *arg)
@@ -118,6 +128,47 @@ static void page_idle_clear_pte_refs(struct page *page)
unlock_page(page);
}
+/* Helper to get the start and end frame given a pos and count */
+static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
+ unsigned long *start, unsigned long *end)
+{
+ unsigned long max_frame;
+
+ /* If an mm is not given, assume we want physical frames */
+ max_frame = mm ? (mm->task_size >> PAGE_SHIFT) : max_pfn;
+
+ if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
+ return -EINVAL;
+
+ *start = pos * BITS_PER_BYTE;
+ if (*start >= max_frame)
+ return -ENXIO;
+
+ *end = *start + count * BITS_PER_BYTE;
+ if (*end > max_frame)
+ *end = max_frame;
+ return 0;
+}
+
+static bool page_really_idle(struct page *page)
+{
+ if (!page)
+ return false;
+
+ if (page_is_idle(page)) {
+ /*
+ * The page might have been referenced via a
+ * pte, in which case it is not idle. Clear
+ * refs and recheck.
+ */
+ page_idle_clear_pte_refs(page);
+ if (page_is_idle(page))
+ return true;
+ }
+
+ return false;
+}
+
static ssize_t page_idle_bitmap_read(struct file *file, struct kobject *kobj,
struct bin_attribute *attr, char *buf,
loff_t pos, size_t count)
@@ -125,35 +176,21 @@ static ssize_t page_idle_bitmap_read(struct file *file, struct kobject *kobj,
u64 *out = (u64 *)buf;
struct page *page;
unsigned long pfn, end_pfn;
- int bit;
+ int bit, ret;
- if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
- return -EINVAL;
-
- pfn = pos * BITS_PER_BYTE;
- if (pfn >= max_pfn)
- return 0;
-
- end_pfn = pfn + count * BITS_PER_BYTE;
- if (end_pfn > max_pfn)
- end_pfn = max_pfn;
+ ret = page_idle_get_frames(pos, count, NULL, &pfn, &end_pfn);
+ if (ret == -ENXIO)
+ return 0; /* Reads beyond max_pfn do nothing */
+ else if (ret)
+ return ret;
for (; pfn < end_pfn; pfn++) {
bit = pfn % BITMAP_CHUNK_BITS;
if (!bit)
*out = 0ULL;
- page = page_idle_get_page(pfn);
- if (page) {
- if (page_is_idle(page)) {
- /*
- * The page might have been referenced via a
- * pte, in which case it is not idle. Clear
- * refs and recheck.
- */
- page_idle_clear_pte_refs(page);
- if (page_is_idle(page))
- *out |= 1ULL << bit;
- }
+ page = page_idle_get_page_pfn(pfn);
+ if (page && page_really_idle(page)) {
+ *out |= 1ULL << bit;
put_page(page);
}
if (bit == BITMAP_CHUNK_BITS - 1)
@@ -170,23 +207,16 @@ static ssize_t page_idle_bitmap_write(struct file *file, struct kobject *kobj,
const u64 *in = (u64 *)buf;
struct page *page;
unsigned long pfn, end_pfn;
- int bit;
+ int bit, ret;
- if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
- return -EINVAL;
-
- pfn = pos * BITS_PER_BYTE;
- if (pfn >= max_pfn)
- return -ENXIO;
-
- end_pfn = pfn + count * BITS_PER_BYTE;
- if (end_pfn > max_pfn)
- end_pfn = max_pfn;
+ ret = page_idle_get_frames(pos, count, NULL, &pfn, &end_pfn);
+ if (ret)
+ return ret;
for (; pfn < end_pfn; pfn++) {
bit = pfn % BITMAP_CHUNK_BITS;
if ((*in >> bit) & 1) {
- page = page_idle_get_page(pfn);
+ page = page_idle_get_page_pfn(pfn);
if (page) {
page_idle_clear_pte_refs(page);
set_page_idle(page);
@@ -224,6 +254,228 @@ struct page_ext_operations page_idle_ops = {
};
#endif
+/* page_idle tracking for /proc/<pid>/page_idle */
+
+struct page_node {
+ struct page *page;
+ unsigned long addr;
+ struct list_head list;
+};
+
+struct page_idle_proc_priv {
+ unsigned long start_addr;
+ char *buffer;
+ int write;
+
+ /* Pre-allocate and provide nodes to add_page_idle_list() */
+ struct page_node *page_nodes;
+ int cur_page_node;
+ struct list_head *idle_page_list;
+};
+
+/*
+ * Add a page to the idle page list. page can be NULL if pte is
+ * from a swapped page.
+ */
+static void add_page_idle_list(struct page *page,
+ unsigned long addr, struct mm_walk *walk)
+{
+ struct page *page_get = NULL;
+ struct page_node *pn;
+ int bit;
+ unsigned long frames;
+ struct page_idle_proc_priv *priv = walk->private;
+ u64 *chunk = (u64 *)priv->buffer;
+
+ if (priv->write) {
+ /* Find whether this page was asked to be marked */
+ frames = (addr - priv->start_addr) >> PAGE_SHIFT;
+ bit = frames % BITMAP_CHUNK_BITS;
+ chunk = &chunk[frames / BITMAP_CHUNK_BITS];
+ if (((*chunk >> bit) & 1) == 0)
+ return;
+ }
+
+ if (page) {
+ page_get = page_idle_get_page(page);
+ if (!page_get)
+ return;
+ }
+
+ pn = &(priv->page_nodes[priv->cur_page_node++]);
+ pn->page = page_get;
+ pn->addr = addr;
+ list_add(&pn->list, priv->idle_page_list);
+}
+
+static int pte_page_idle_proc_range(pmd_t *pmd, unsigned long addr,
+ unsigned long end,
+ struct mm_walk *walk)
+{
+ struct vm_area_struct *vma = walk->vma;
+ pte_t *pte;
+ spinlock_t *ptl;
+ struct page *page;
+
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ if (pmd_present(*pmd)) {
+ page = follow_trans_huge_pmd(vma, addr, pmd,
+ FOLL_DUMP|FOLL_WRITE);
+ if (!IS_ERR_OR_NULL(page))
+ add_page_idle_list(page, addr, walk);
+ }
+ spin_unlock(ptl);
+ return 0;
+ }
+
+ if (pmd_trans_unstable(pmd))
+ return 0;
+
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
+ for (; addr != end; pte++, addr += PAGE_SIZE) {
+ /*
+ * We add swapped pages to the idle_page_list so that we can
+ * reported to userspace that they are idle.
+ */
+ if (is_swap_pte(*pte)) {
+ add_page_idle_list(NULL, addr, walk);
+ continue;
+ }
+
+ if (!pte_present(*pte))
+ continue;
+
+ page = vm_normal_page(vma, addr, *pte);
+ if (page)
+ add_page_idle_list(page, addr, walk);
+ }
+
+ pte_unmap_unlock(pte - 1, ptl);
+ return 0;
+}
+
+ssize_t page_idle_proc_generic(struct file *file, char __user *ubuff,
+ size_t count, loff_t *pos,
+ struct task_struct *tsk, int write)
+{
+ int ret;
+ char *buffer;
+ u64 *out;
+ unsigned long start_addr, end_addr, start_frame, end_frame;
+ struct mm_struct *mm = file->private_data;
+ struct mm_walk walk = { .pmd_entry = pte_page_idle_proc_range, };
+ struct page_node *cur, *next;
+ struct page_idle_proc_priv priv;
+ bool walk_error = false;
+ LIST_HEAD(idle_page_list);
+
+ if (!mm || !mmget_not_zero(mm))
+ return -EINVAL;
+
+ if (count > PAGE_SIZE)
+ count = PAGE_SIZE;
+
+ buffer = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!buffer) {
+ ret = -ENOMEM;
+ goto out_mmput;
+ }
+ out = (u64 *)buffer;
+
+ if (write && copy_from_user(buffer, ubuff, count)) {
+ ret = -EFAULT;
+ goto out;
+ }
+
+ ret = page_idle_get_frames(*pos, count, mm, &start_frame, &end_frame);
+ if (ret)
+ goto out;
+
+ start_addr = (start_frame << PAGE_SHIFT);
+ end_addr = (end_frame << PAGE_SHIFT);
+ priv.buffer = buffer;
+ priv.start_addr = start_addr;
+ priv.write = write;
+
+ priv.idle_page_list = &idle_page_list;
+ priv.cur_page_node = 0;
+ priv.page_nodes = kzalloc(sizeof(struct page_node) *
+ (end_frame - start_frame), GFP_KERNEL);
+ if (!priv.page_nodes) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ walk.private = &priv;
+ walk.mm = mm;
+
+ down_read(&mm->mmap_sem);
+
+ /*
+ * idle_page_list is needed because walk_page_vma() holds ptlock which
+ * deadlocks with page_idle_clear_pte_refs(). So we have to collect all
+ * pages first, and then call page_idle_clear_pte_refs().
+ */
+ ret = walk_page_range(start_addr, end_addr, &walk);
+ if (ret)
+ walk_error = true;
+
+ list_for_each_entry_safe(cur, next, &idle_page_list, list) {
+ int bit, index;
+ unsigned long off;
+ struct page *page = cur->page;
+
+ if (unlikely(walk_error))
+ goto remove_page;
+
+ if (write) {
+ if (page) {
+ page_idle_clear_pte_refs(page);
+ set_page_idle(page);
+ }
+ } else {
+ if (!page || page_really_idle(page)) {
+ off = ((cur->addr) >> PAGE_SHIFT) - start_frame;
+ bit = off % BITMAP_CHUNK_BITS;
+ index = off / BITMAP_CHUNK_BITS;
+ out[index] |= 1ULL << bit;
+ }
+ }
+remove_page:
+ if (page)
+ put_page(page);
+ list_del(&cur->list);
+ kfree(cur);
+ }
+
+ if (!write && !walk_error)
+ ret = copy_to_user(ubuff, buffer, count);
+
+ up_read(&mm->mmap_sem);
+ kfree(priv.page_nodes);
+out:
+ kfree(buffer);
+out_mmput:
+ mmput(mm);
+ if (!ret)
+ ret = count;
+ return ret;
+
+}
+
+ssize_t page_idle_proc_read(struct file *file, char __user *ubuff,
+ size_t count, loff_t *pos, struct task_struct *tsk)
+{
+ return page_idle_proc_generic(file, ubuff, count, pos, tsk, 0);
+}
+
+ssize_t page_idle_proc_write(struct file *file, char __user *ubuff,
+ size_t count, loff_t *pos, struct task_struct *tsk)
+{
+ return page_idle_proc_generic(file, ubuff, count, pos, tsk, 1);
+}
+
static int __init page_idle_init(void)
{
int err;
--
2.22.0.709.g102302147b-goog
^ permalink raw reply related
* [PATCH v2 2/2] doc: Update documentation for page_idle virtual address indexing
From: Joel Fernandes (Google) @ 2019-07-26 15:08 UTC (permalink / raw)
To: linux-kernel
Cc: Joel Fernandes (Google), Alexey Dobriyan, Andrew Morton,
Brendan Gregg, Christian Hansen, dancol, fmayer, joaodias, joelaf,
Jonathan Corbet, Kees Cook, kernel-team, linux-api, linux-doc,
linux-fsdevel, linux-mm, Michal Hocko, Mike Rapoport, minchan,
namhyung, Roman Gushchin, Stephen Rothwell, surenb, tkjos,
Vladimir Davydov, Vlastimil Babka, wvw
In-Reply-To: <20190726150845.95720-1-joel@joelfernandes.org>
This patch updates the documentation with the new page_idle tracking
feature which uses virtual address indexing.
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
.../admin-guide/mm/idle_page_tracking.rst | 43 ++++++++++++++++---
1 file changed, 36 insertions(+), 7 deletions(-)
diff --git a/Documentation/admin-guide/mm/idle_page_tracking.rst b/Documentation/admin-guide/mm/idle_page_tracking.rst
index df9394fb39c2..1eeac78c94a7 100644
--- a/Documentation/admin-guide/mm/idle_page_tracking.rst
+++ b/Documentation/admin-guide/mm/idle_page_tracking.rst
@@ -19,10 +19,14 @@ It is enabled by CONFIG_IDLE_PAGE_TRACKING=y.
User API
========
+There are 2 ways to access the idle page tracking API. One uses physical
+address indexing, another uses a simpler virtual address indexing scheme.
-The idle page tracking API is located at ``/sys/kernel/mm/page_idle``.
-Currently, it consists of the only read-write file,
-``/sys/kernel/mm/page_idle/bitmap``.
+Physical address indexing
+-------------------------
+The idle page tracking API for physical address indexing using page frame
+numbers (PFN) is located at ``/sys/kernel/mm/page_idle``. Currently, it
+consists of the only read-write file, ``/sys/kernel/mm/page_idle/bitmap``.
The file implements a bitmap where each bit corresponds to a memory page. The
bitmap is represented by an array of 8-byte integers, and the page at PFN #i is
@@ -74,6 +78,31 @@ See :ref:`Documentation/admin-guide/mm/pagemap.rst <pagemap>` for more
information about ``/proc/pid/pagemap``, ``/proc/kpageflags``, and
``/proc/kpagecgroup``.
+Virtual address indexing
+------------------------
+The idle page tracking API for virtual address indexing using virtual page
+frame numbers (VFN) is located at ``/proc/<pid>/page_idle``. It is a bitmap
+that follows the same semantics as ``/sys/kernel/mm/page_idle/bitmap``
+except that it uses virtual instead of physical frame numbers.
+
+This idle page tracking API does not need deal with PFN so it does not require
+prior lookups of ``pagemap`` in order to find if page is idle or not. This is
+an advantage on some systems where looking up PFN is considered a security
+issue. Also in some cases, this interface could be slightly more reliable to
+use than physical address indexing, since in physical address indexing, address
+space changes can occur between reading the ``pagemap`` and reading the
+``bitmap``, while in virtual address indexing, the process's ``mmap_sem`` is
+held for the duration of the access.
+
+To estimate the amount of pages that are not used by a workload one should:
+
+ 1. Mark all the workload's pages as idle by setting corresponding bits in
+ ``/proc/<pid>/page_idle``.
+
+ 2. Wait until the workload accesses its working set.
+
+ 3. Read ``/proc/<pid>/page_idle`` and count the number of bits set.
+
.. _impl_details:
Implementation Details
@@ -99,10 +128,10 @@ When a dirty page is written to swap or disk as a result of memory reclaim or
exceeding the dirty memory limit, it is not marked referenced.
The idle memory tracking feature adds a new page flag, the Idle flag. This flag
-is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` (see the
-:ref:`User API <user_api>`
-section), and cleared automatically whenever a page is referenced as defined
-above.
+is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` for physical
+addressing or by writing to ``/proc/<pid>/page_idle`` for virtual
+addressing (see the :ref:`User API <user_api>` section), and cleared
+automatically whenever a page is referenced as defined above.
When a page is marked idle, the Accessed bit must be cleared in all PTEs it is
mapped to, otherwise we will not be able to detect accesses to the page coming
--
2.22.0.709.g102302147b-goog
^ permalink raw reply related
* [PATCH v3 1/2] mm/page_idle: Add per-pid idle page tracking using virtual indexing
From: Joel Fernandes (Google) @ 2019-07-26 15:23 UTC (permalink / raw)
To: linux-kernel
Cc: Joel Fernandes (Google), Alexey Dobriyan, Andrew Morton,
Brendan Gregg, Christian Hansen, dancol, fmayer, joaodias, joelaf,
Jonathan Corbet, Kees Cook, kernel-team, linux-api, linux-doc,
linux-fsdevel, linux-mm, Michal Hocko, Mike Rapoport, minchan,
namhyung, Roman Gushchin, Stephen Rothwell, surenb, tkjos,
Vladimir Davydov, Vlastimil Babka, wvw
The page_idle tracking feature currently requires looking up the pagemap
for a process followed by interacting with /sys/kernel/mm/page_idle.
Looking up PFN from pagemap in Android devices is not supported by
unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.
This patch adds support to directly interact with page_idle tracking at
the PID level by introducing a /proc/<pid>/page_idle file. It follows
the exact same semantics as the global /sys/kernel/mm/page_idle, but now
looking up PFN through pagemap is not needed since the interface uses
virtual frame numbers, and at the same time also does not require
SYS_ADMIN.
In Android, we are using this for the heap profiler (heapprofd) which
profiles and pin points code paths which allocates and leaves memory
idle for long periods of time. This method solves the security issue
with userspace learning the PFN, and while at it is also shown to yield
better results than the pagemap lookup, the theory being that the window
where the address space can change is reduced by eliminating the
intermediate pagemap look up stage. In virtual address indexing, the
process's mmap_sem is held for the duration of the access.
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
v2->v3:
Fixed a bug where I was doing a kfree that is not needed due to not
needing to do GFP_ATOMIC allocations.
v1->v2:
Mark swap ptes as idle (Minchan)
Avoid need for GFP_ATOMIC (Andrew)
Get rid of idle_page_list lock by moving list to stack
Internal review -> v1:
Fixes from Suren.
Corrections to change log, docs (Florian, Sandeep)
fs/proc/base.c | 3 +
fs/proc/internal.h | 1 +
fs/proc/task_mmu.c | 57 +++++++
include/linux/page_idle.h | 4 +
mm/page_idle.c | 340 +++++++++++++++++++++++++++++++++-----
5 files changed, 360 insertions(+), 45 deletions(-)
diff --git a/fs/proc/base.c b/fs/proc/base.c
index 77eb628ecc7f..a58dd74606e9 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -3021,6 +3021,9 @@ static const struct pid_entry tgid_base_stuff[] = {
REG("smaps", S_IRUGO, proc_pid_smaps_operations),
REG("smaps_rollup", S_IRUGO, proc_pid_smaps_rollup_operations),
REG("pagemap", S_IRUSR, proc_pagemap_operations),
+#ifdef CONFIG_IDLE_PAGE_TRACKING
+ REG("page_idle", S_IRUSR|S_IWUSR, proc_page_idle_operations),
+#endif
#endif
#ifdef CONFIG_SECURITY
DIR("attr", S_IRUGO|S_IXUGO, proc_attr_dir_inode_operations, proc_attr_dir_operations),
diff --git a/fs/proc/internal.h b/fs/proc/internal.h
index cd0c8d5ce9a1..bc9371880c63 100644
--- a/fs/proc/internal.h
+++ b/fs/proc/internal.h
@@ -293,6 +293,7 @@ extern const struct file_operations proc_pid_smaps_operations;
extern const struct file_operations proc_pid_smaps_rollup_operations;
extern const struct file_operations proc_clear_refs_operations;
extern const struct file_operations proc_pagemap_operations;
+extern const struct file_operations proc_page_idle_operations;
extern unsigned long task_vsize(struct mm_struct *);
extern unsigned long task_statm(struct mm_struct *,
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 4d2b860dbc3f..11ccc53da38e 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1642,6 +1642,63 @@ const struct file_operations proc_pagemap_operations = {
.open = pagemap_open,
.release = pagemap_release,
};
+
+#ifdef CONFIG_IDLE_PAGE_TRACKING
+static ssize_t proc_page_idle_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ int ret;
+ struct task_struct *tsk = get_proc_task(file_inode(file));
+
+ if (!tsk)
+ return -EINVAL;
+ ret = page_idle_proc_read(file, buf, count, ppos, tsk);
+ put_task_struct(tsk);
+ return ret;
+}
+
+static ssize_t proc_page_idle_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ int ret;
+ struct task_struct *tsk = get_proc_task(file_inode(file));
+
+ if (!tsk)
+ return -EINVAL;
+ ret = page_idle_proc_write(file, (char __user *)buf, count, ppos, tsk);
+ put_task_struct(tsk);
+ return ret;
+}
+
+static int proc_page_idle_open(struct inode *inode, struct file *file)
+{
+ struct mm_struct *mm;
+
+ mm = proc_mem_open(inode, PTRACE_MODE_READ);
+ if (IS_ERR(mm))
+ return PTR_ERR(mm);
+ file->private_data = mm;
+ return 0;
+}
+
+static int proc_page_idle_release(struct inode *inode, struct file *file)
+{
+ struct mm_struct *mm = file->private_data;
+
+ if (mm)
+ mmdrop(mm);
+ return 0;
+}
+
+const struct file_operations proc_page_idle_operations = {
+ .llseek = mem_lseek, /* borrow this */
+ .read = proc_page_idle_read,
+ .write = proc_page_idle_write,
+ .open = proc_page_idle_open,
+ .release = proc_page_idle_release,
+};
+#endif /* CONFIG_IDLE_PAGE_TRACKING */
+
#endif /* CONFIG_PROC_PAGE_MONITOR */
#ifdef CONFIG_NUMA
diff --git a/include/linux/page_idle.h b/include/linux/page_idle.h
index 1e894d34bdce..f1bc2640d85e 100644
--- a/include/linux/page_idle.h
+++ b/include/linux/page_idle.h
@@ -106,6 +106,10 @@ static inline void clear_page_idle(struct page *page)
}
#endif /* CONFIG_64BIT */
+ssize_t page_idle_proc_write(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos, struct task_struct *tsk);
+ssize_t page_idle_proc_read(struct file *file,
+ char __user *buf, size_t count, loff_t *ppos, struct task_struct *tsk);
#else /* !CONFIG_IDLE_PAGE_TRACKING */
static inline bool page_is_young(struct page *page)
diff --git a/mm/page_idle.c b/mm/page_idle.c
index 295512465065..86244f7f1faa 100644
--- a/mm/page_idle.c
+++ b/mm/page_idle.c
@@ -5,12 +5,15 @@
#include <linux/sysfs.h>
#include <linux/kobject.h>
#include <linux/mm.h>
-#include <linux/mmzone.h>
-#include <linux/pagemap.h>
-#include <linux/rmap.h>
#include <linux/mmu_notifier.h>
+#include <linux/mmzone.h>
#include <linux/page_ext.h>
#include <linux/page_idle.h>
+#include <linux/pagemap.h>
+#include <linux/rmap.h>
+#include <linux/sched/mm.h>
+#include <linux/swap.h>
+#include <linux/swapops.h>
#define BITMAP_CHUNK_SIZE sizeof(u64)
#define BITMAP_CHUNK_BITS (BITMAP_CHUNK_SIZE * BITS_PER_BYTE)
@@ -25,18 +28,13 @@
* page tracking. With such an indicator of user pages we can skip isolated
* pages, but since there are not usually many of them, it will hardly affect
* the overall result.
- *
- * This function tries to get a user memory page by pfn as described above.
*/
-static struct page *page_idle_get_page(unsigned long pfn)
+static struct page *page_idle_get_page(struct page *page_in)
{
struct page *page;
pg_data_t *pgdat;
- if (!pfn_valid(pfn))
- return NULL;
-
- page = pfn_to_page(pfn);
+ page = page_in;
if (!page || !PageLRU(page) ||
!get_page_unless_zero(page))
return NULL;
@@ -51,6 +49,18 @@ static struct page *page_idle_get_page(unsigned long pfn)
return page;
}
+/*
+ * This function tries to get a user memory page by pfn as described above.
+ */
+static struct page *page_idle_get_page_pfn(unsigned long pfn)
+{
+
+ if (!pfn_valid(pfn))
+ return NULL;
+
+ return page_idle_get_page(pfn_to_page(pfn));
+}
+
static bool page_idle_clear_pte_refs_one(struct page *page,
struct vm_area_struct *vma,
unsigned long addr, void *arg)
@@ -118,6 +128,47 @@ static void page_idle_clear_pte_refs(struct page *page)
unlock_page(page);
}
+/* Helper to get the start and end frame given a pos and count */
+static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
+ unsigned long *start, unsigned long *end)
+{
+ unsigned long max_frame;
+
+ /* If an mm is not given, assume we want physical frames */
+ max_frame = mm ? (mm->task_size >> PAGE_SHIFT) : max_pfn;
+
+ if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
+ return -EINVAL;
+
+ *start = pos * BITS_PER_BYTE;
+ if (*start >= max_frame)
+ return -ENXIO;
+
+ *end = *start + count * BITS_PER_BYTE;
+ if (*end > max_frame)
+ *end = max_frame;
+ return 0;
+}
+
+static bool page_really_idle(struct page *page)
+{
+ if (!page)
+ return false;
+
+ if (page_is_idle(page)) {
+ /*
+ * The page might have been referenced via a
+ * pte, in which case it is not idle. Clear
+ * refs and recheck.
+ */
+ page_idle_clear_pte_refs(page);
+ if (page_is_idle(page))
+ return true;
+ }
+
+ return false;
+}
+
static ssize_t page_idle_bitmap_read(struct file *file, struct kobject *kobj,
struct bin_attribute *attr, char *buf,
loff_t pos, size_t count)
@@ -125,35 +176,21 @@ static ssize_t page_idle_bitmap_read(struct file *file, struct kobject *kobj,
u64 *out = (u64 *)buf;
struct page *page;
unsigned long pfn, end_pfn;
- int bit;
+ int bit, ret;
- if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
- return -EINVAL;
-
- pfn = pos * BITS_PER_BYTE;
- if (pfn >= max_pfn)
- return 0;
-
- end_pfn = pfn + count * BITS_PER_BYTE;
- if (end_pfn > max_pfn)
- end_pfn = max_pfn;
+ ret = page_idle_get_frames(pos, count, NULL, &pfn, &end_pfn);
+ if (ret == -ENXIO)
+ return 0; /* Reads beyond max_pfn do nothing */
+ else if (ret)
+ return ret;
for (; pfn < end_pfn; pfn++) {
bit = pfn % BITMAP_CHUNK_BITS;
if (!bit)
*out = 0ULL;
- page = page_idle_get_page(pfn);
- if (page) {
- if (page_is_idle(page)) {
- /*
- * The page might have been referenced via a
- * pte, in which case it is not idle. Clear
- * refs and recheck.
- */
- page_idle_clear_pte_refs(page);
- if (page_is_idle(page))
- *out |= 1ULL << bit;
- }
+ page = page_idle_get_page_pfn(pfn);
+ if (page && page_really_idle(page)) {
+ *out |= 1ULL << bit;
put_page(page);
}
if (bit == BITMAP_CHUNK_BITS - 1)
@@ -170,23 +207,16 @@ static ssize_t page_idle_bitmap_write(struct file *file, struct kobject *kobj,
const u64 *in = (u64 *)buf;
struct page *page;
unsigned long pfn, end_pfn;
- int bit;
+ int bit, ret;
- if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
- return -EINVAL;
-
- pfn = pos * BITS_PER_BYTE;
- if (pfn >= max_pfn)
- return -ENXIO;
-
- end_pfn = pfn + count * BITS_PER_BYTE;
- if (end_pfn > max_pfn)
- end_pfn = max_pfn;
+ ret = page_idle_get_frames(pos, count, NULL, &pfn, &end_pfn);
+ if (ret)
+ return ret;
for (; pfn < end_pfn; pfn++) {
bit = pfn % BITMAP_CHUNK_BITS;
if ((*in >> bit) & 1) {
- page = page_idle_get_page(pfn);
+ page = page_idle_get_page_pfn(pfn);
if (page) {
page_idle_clear_pte_refs(page);
set_page_idle(page);
@@ -224,6 +254,226 @@ struct page_ext_operations page_idle_ops = {
};
#endif
+/* page_idle tracking for /proc/<pid>/page_idle */
+
+struct page_node {
+ struct page *page;
+ unsigned long addr;
+ struct list_head list;
+};
+
+struct page_idle_proc_priv {
+ unsigned long start_addr;
+ char *buffer;
+ int write;
+
+ /* Pre-allocate and provide nodes to add_page_idle_list() */
+ struct page_node *page_nodes;
+ int cur_page_node;
+ struct list_head *idle_page_list;
+};
+
+/*
+ * Add a page to the idle page list. page can be NULL if pte is
+ * from a swapped page.
+ */
+static void add_page_idle_list(struct page *page,
+ unsigned long addr, struct mm_walk *walk)
+{
+ struct page *page_get = NULL;
+ struct page_node *pn;
+ int bit;
+ unsigned long frames;
+ struct page_idle_proc_priv *priv = walk->private;
+ u64 *chunk = (u64 *)priv->buffer;
+
+ if (priv->write) {
+ /* Find whether this page was asked to be marked */
+ frames = (addr - priv->start_addr) >> PAGE_SHIFT;
+ bit = frames % BITMAP_CHUNK_BITS;
+ chunk = &chunk[frames / BITMAP_CHUNK_BITS];
+ if (((*chunk >> bit) & 1) == 0)
+ return;
+ }
+
+ if (page) {
+ page_get = page_idle_get_page(page);
+ if (!page_get)
+ return;
+ }
+
+ pn = &(priv->page_nodes[priv->cur_page_node++]);
+ pn->page = page_get;
+ pn->addr = addr;
+ list_add(&pn->list, priv->idle_page_list);
+}
+
+static int pte_page_idle_proc_range(pmd_t *pmd, unsigned long addr,
+ unsigned long end,
+ struct mm_walk *walk)
+{
+ struct vm_area_struct *vma = walk->vma;
+ pte_t *pte;
+ spinlock_t *ptl;
+ struct page *page;
+
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ if (pmd_present(*pmd)) {
+ page = follow_trans_huge_pmd(vma, addr, pmd,
+ FOLL_DUMP|FOLL_WRITE);
+ if (!IS_ERR_OR_NULL(page))
+ add_page_idle_list(page, addr, walk);
+ }
+ spin_unlock(ptl);
+ return 0;
+ }
+
+ if (pmd_trans_unstable(pmd))
+ return 0;
+
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
+ for (; addr != end; pte++, addr += PAGE_SIZE) {
+ /*
+ * We add swapped pages to the idle_page_list so that we can
+ * reported to userspace that they are idle.
+ */
+ if (is_swap_pte(*pte)) {
+ add_page_idle_list(NULL, addr, walk);
+ continue;
+ }
+
+ if (!pte_present(*pte))
+ continue;
+
+ page = vm_normal_page(vma, addr, *pte);
+ if (page)
+ add_page_idle_list(page, addr, walk);
+ }
+
+ pte_unmap_unlock(pte - 1, ptl);
+ return 0;
+}
+
+ssize_t page_idle_proc_generic(struct file *file, char __user *ubuff,
+ size_t count, loff_t *pos,
+ struct task_struct *tsk, int write)
+{
+ int ret;
+ char *buffer;
+ u64 *out;
+ unsigned long start_addr, end_addr, start_frame, end_frame;
+ struct mm_struct *mm = file->private_data;
+ struct mm_walk walk = { .pmd_entry = pte_page_idle_proc_range, };
+ struct page_node *cur;
+ struct page_idle_proc_priv priv;
+ bool walk_error = false;
+ LIST_HEAD(idle_page_list);
+
+ if (!mm || !mmget_not_zero(mm))
+ return -EINVAL;
+
+ if (count > PAGE_SIZE)
+ count = PAGE_SIZE;
+
+ buffer = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!buffer) {
+ ret = -ENOMEM;
+ goto out_mmput;
+ }
+ out = (u64 *)buffer;
+
+ if (write && copy_from_user(buffer, ubuff, count)) {
+ ret = -EFAULT;
+ goto out;
+ }
+
+ ret = page_idle_get_frames(*pos, count, mm, &start_frame, &end_frame);
+ if (ret)
+ goto out;
+
+ start_addr = (start_frame << PAGE_SHIFT);
+ end_addr = (end_frame << PAGE_SHIFT);
+ priv.buffer = buffer;
+ priv.start_addr = start_addr;
+ priv.write = write;
+
+ priv.idle_page_list = &idle_page_list;
+ priv.cur_page_node = 0;
+ priv.page_nodes = kzalloc(sizeof(struct page_node) *
+ (end_frame - start_frame), GFP_KERNEL);
+ if (!priv.page_nodes) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ walk.private = &priv;
+ walk.mm = mm;
+
+ down_read(&mm->mmap_sem);
+
+ /*
+ * idle_page_list is needed because walk_page_vma() holds ptlock which
+ * deadlocks with page_idle_clear_pte_refs(). So we have to collect all
+ * pages first, and then call page_idle_clear_pte_refs().
+ */
+ ret = walk_page_range(start_addr, end_addr, &walk);
+ if (ret)
+ walk_error = true;
+
+ list_for_each_entry(cur, &idle_page_list, list) {
+ int bit, index;
+ unsigned long off;
+ struct page *page = cur->page;
+
+ if (unlikely(walk_error))
+ goto remove_page;
+
+ if (write) {
+ if (page) {
+ page_idle_clear_pte_refs(page);
+ set_page_idle(page);
+ }
+ } else {
+ if (!page || page_really_idle(page)) {
+ off = ((cur->addr) >> PAGE_SHIFT) - start_frame;
+ bit = off % BITMAP_CHUNK_BITS;
+ index = off / BITMAP_CHUNK_BITS;
+ out[index] |= 1ULL << bit;
+ }
+ }
+remove_page:
+ if (page)
+ put_page(page);
+ }
+
+ if (!write && !walk_error)
+ ret = copy_to_user(ubuff, buffer, count);
+
+ up_read(&mm->mmap_sem);
+ kfree(priv.page_nodes);
+out:
+ kfree(buffer);
+out_mmput:
+ mmput(mm);
+ if (!ret)
+ ret = count;
+ return ret;
+
+}
+
+ssize_t page_idle_proc_read(struct file *file, char __user *ubuff,
+ size_t count, loff_t *pos, struct task_struct *tsk)
+{
+ return page_idle_proc_generic(file, ubuff, count, pos, tsk, 0);
+}
+
+ssize_t page_idle_proc_write(struct file *file, char __user *ubuff,
+ size_t count, loff_t *pos, struct task_struct *tsk)
+{
+ return page_idle_proc_generic(file, ubuff, count, pos, tsk, 1);
+}
+
static int __init page_idle_init(void)
{
int err;
--
2.22.0.709.g102302147b-goog
^ permalink raw reply related
* [PATCH v3 2/2] doc: Update documentation for page_idle virtual address indexing
From: Joel Fernandes (Google) @ 2019-07-26 15:23 UTC (permalink / raw)
To: linux-kernel
Cc: Joel Fernandes (Google), Alexey Dobriyan, Andrew Morton,
Brendan Gregg, Christian Hansen, dancol, fmayer, joaodias, joelaf,
Jonathan Corbet, Kees Cook, kernel-team, linux-api, linux-doc,
linux-fsdevel, linux-mm, Michal Hocko, Mike Rapoport, minchan,
namhyung, Roman Gushchin, Stephen Rothwell, surenb, tkjos,
Vladimir Davydov, Vlastimil Babka, wvw
In-Reply-To: <20190726152319.134152-1-joel@joelfernandes.org>
This patch updates the documentation with the new page_idle tracking
feature which uses virtual address indexing.
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
.../admin-guide/mm/idle_page_tracking.rst | 43 ++++++++++++++++---
1 file changed, 36 insertions(+), 7 deletions(-)
diff --git a/Documentation/admin-guide/mm/idle_page_tracking.rst b/Documentation/admin-guide/mm/idle_page_tracking.rst
index df9394fb39c2..1eeac78c94a7 100644
--- a/Documentation/admin-guide/mm/idle_page_tracking.rst
+++ b/Documentation/admin-guide/mm/idle_page_tracking.rst
@@ -19,10 +19,14 @@ It is enabled by CONFIG_IDLE_PAGE_TRACKING=y.
User API
========
+There are 2 ways to access the idle page tracking API. One uses physical
+address indexing, another uses a simpler virtual address indexing scheme.
-The idle page tracking API is located at ``/sys/kernel/mm/page_idle``.
-Currently, it consists of the only read-write file,
-``/sys/kernel/mm/page_idle/bitmap``.
+Physical address indexing
+-------------------------
+The idle page tracking API for physical address indexing using page frame
+numbers (PFN) is located at ``/sys/kernel/mm/page_idle``. Currently, it
+consists of the only read-write file, ``/sys/kernel/mm/page_idle/bitmap``.
The file implements a bitmap where each bit corresponds to a memory page. The
bitmap is represented by an array of 8-byte integers, and the page at PFN #i is
@@ -74,6 +78,31 @@ See :ref:`Documentation/admin-guide/mm/pagemap.rst <pagemap>` for more
information about ``/proc/pid/pagemap``, ``/proc/kpageflags``, and
``/proc/kpagecgroup``.
+Virtual address indexing
+------------------------
+The idle page tracking API for virtual address indexing using virtual page
+frame numbers (VFN) is located at ``/proc/<pid>/page_idle``. It is a bitmap
+that follows the same semantics as ``/sys/kernel/mm/page_idle/bitmap``
+except that it uses virtual instead of physical frame numbers.
+
+This idle page tracking API does not need deal with PFN so it does not require
+prior lookups of ``pagemap`` in order to find if page is idle or not. This is
+an advantage on some systems where looking up PFN is considered a security
+issue. Also in some cases, this interface could be slightly more reliable to
+use than physical address indexing, since in physical address indexing, address
+space changes can occur between reading the ``pagemap`` and reading the
+``bitmap``, while in virtual address indexing, the process's ``mmap_sem`` is
+held for the duration of the access.
+
+To estimate the amount of pages that are not used by a workload one should:
+
+ 1. Mark all the workload's pages as idle by setting corresponding bits in
+ ``/proc/<pid>/page_idle``.
+
+ 2. Wait until the workload accesses its working set.
+
+ 3. Read ``/proc/<pid>/page_idle`` and count the number of bits set.
+
.. _impl_details:
Implementation Details
@@ -99,10 +128,10 @@ When a dirty page is written to swap or disk as a result of memory reclaim or
exceeding the dirty memory limit, it is not marked referenced.
The idle memory tracking feature adds a new page flag, the Idle flag. This flag
-is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` (see the
-:ref:`User API <user_api>`
-section), and cleared automatically whenever a page is referenced as defined
-above.
+is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` for physical
+addressing or by writing to ``/proc/<pid>/page_idle`` for virtual
+addressing (see the :ref:`User API <user_api>` section), and cleared
+automatically whenever a page is referenced as defined above.
When a page is marked idle, the Accessed bit must be cleared in all PTEs it is
mapped to, otherwise we will not be able to detect accesses to the page coming
--
2.22.0.709.g102302147b-goog
^ permalink raw reply related
* [PATCH] docs: admin-guide: Adjust title underline length
From: Fabio Estevam @ 2019-07-26 16:27 UTC (permalink / raw)
To: corbet; +Cc: mchehab+samsung, linux-doc, Fabio Estevam
The following warning is seen when building 'make htmldocs':
Documentation/admin-guide/sysctl/kernel.rst:397: WARNING: Title underline too short.
Fix it by adjusting the title underline length appropriately.
Signed-off-by: Fabio Estevam <festevam@gmail.com>
---
Documentation/admin-guide/sysctl/kernel.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 2e36620ec1e4..8af424dd0364 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -394,7 +394,7 @@ This file shows up if CONFIG_DETECT_HUNG_TASK is enabled.
hung_task_interval_warnings:
-===================
+============================
The same as hung_task_warnings, but set the number of interval
warnings to be issued about detected hung tasks during check
--
2.17.1
^ permalink raw reply related
* Re: [PATCH v2 25/26] docs: rcu: convert some articles from html to ReST
From: Joel Fernandes @ 2019-07-26 17:55 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Paul E. McKenney, Josh Triplett, Steven Rostedt,
Mathieu Desnoyers, Lai Jiangshan, Jonathan Corbet, rcu, linux-doc
In-Reply-To: <20190726140028.38abb5fa@coco.lan>
On Fri, Jul 26, 2019 at 02:00:28PM -0300, Mauro Carvalho Chehab wrote:
> Hi Joel,
>
> Em Fri, 26 Jul 2019 12:20:02 -0400
> Joel Fernandes <joel@joelfernandes.org> escreveu:
>
> > On Fri, Jul 26, 2019 at 09:51:35AM -0300, Mauro Carvalho Chehab wrote:
> > > There are 4 RCU articles that are written on html format.
> > >
> > > The way they are, they can't be part of the Linux Kernel
> > > documentation body nor share the styles and pdf output.
> > >
> > > So, convert them to ReST format.
> > >
> > > This way, make htmldocs and make pdfdocs will produce a
> > > documentation output that will be like the original ones, but
> > > will be part of the Linux Kernel documentation body.
> > >
> > > Part of the conversion was done with the help of pandoc, but
> > > the result had some broken things that had to be manually
> > > fixed.
> >
> > This looks Ok to me, but I also nervous something could have been done
> > incorrectly during the conversion.
> >
> > Could you list what were the "some broken things" that you had to manually
> > fix to make reviewing easier?
>
> There are a couple of things.
>
> At least the pandoc's version I used here has a bug: its conversion
> from html to ReST on those files only start after a <body> tag - or
> when the first quiz table starts. I only discovered that adding a
> <body> at the beginning of the file solve this book at the last
> conversions.
>
> So, for most html->ReST conversions, I manually converted the first
> part of the document, basically stripping html paragraph tags and
> by replacing highlights by the ReST syntax.
>
> Also, all the quiz tables seem to assume some javascript macro or
> css style that would be hiding the answer part until the mouse moves
> to it. Such macro/css was not there at the kernel tree. So, the quiz
> answers have the same color as the background, making them invisible.
> Even if we had such macro/css, this is not portable for pdf/LaTeX output
> (and I'm not sure if this would work with ePub).
>
> So, I ended by manually doing the table conversion.
>
> Finally, I double-checked if the conversions ended ok, addressing any
> issues that might have heppened.
>
> So, after both automatic conversion and manual fixes, I opened both the
> html files produced by Sphinx and the original ones and compared them
> line per line (except for the indexes, as Sphinx produces them
> automatically), in order to see if all information from the original
> files will be there on a format close to what we have on other ReST
> files, fixing any pending issues if any.
Thanks, I am in the process of going through these docs today and will let
you know anything I find. It would be nice to include the above challenges in
the changelog as well.
Some reason 'make htmldocs' needed me to install a whole bunch of
dependencies this time around.
By the way, that tools/memory-model/Documentation/explanation.txt is a useful
little document. Well not really little with over 1000 lines ;-). But it
would certainly benefit from ReST's / htmldocs ability to jump to labels and
search, etc since it is so long..
thanks,
- Joel
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox