* [patch V2 00/25] timekeeping/ptp: Expand snapshot functionality
From: Thomas Gleixner @ 2026-05-29 19:59 UTC (permalink / raw)
To: LKML
Cc: David Woodhouse, Miroslav Lichvar, John Stultz, Stephen Boyd,
Anna-Maria Behnsen, Frederic Weisbecker, thomas.weissschuh,
Arthur Kiyanovski, Rodolfo Giometti, Vincent Donnefort,
Marc Zyngier, Oliver Upton, kvmarm, Oliver Upton, Richard Cochran,
netdev, Takashi Iwai, Miri Korenblit, Johannes Berg, Jacob Keller,
Tony Nguyen, Saeed Mahameed, Peter Hilber, Michael S. Tsirkin,
virtualization, linux-wireless, linux-sound, David Woodhouse,
Vadim Fedorenko
This is an update to V1 which can be found here:
https://lore.kernel.org/lkml/20260526165826.392227559@kernel.org
PTP wants to grow new snapshot functionality, which provides not only the
captured CLOCK* values, but also the underlying clocksource counter value.
https://lore.kernel.org/20260515164033.6403-1-akiyano@amazon.com
There was quite some discussion in seemingly related threads how to capture
these values and how to provide core infrastructure so that driver writers
have something to work with
https://lore.kernel.org/20260514225842.110706-1-hramamurthy@google.com
https://lore.kernel.org/20260520135207.37826-1-dwmw2@infradead.org
This series implements the timekeeping related mechanisms to:
1) Capture CLOCK values along with the clocksource counter value for
non-hardware based sampling
2) Expanding the hardware cross time stamp mechanism to hand back the
clocksource counter value, which was captured by the device, along
with the related CLOCK values
3) Adding AUX clock support to the hardware cross timestamping core
4) Add support for derived clocksources to the snapshot mechanism (New
in V2)
Changes vs. V1:
- Fixed the ptp_ocp typo - 0-day, Jakub
- Renamed the system_time_snapshot members sys and raw so systime and
monoraw to make them less ambigous.
- Fixed the error case return values of get_device_system_crosststamp()
- Made ktime_snapshot_id() void as there is no point for the return
value, which is nowhere checked and cannot be propagated.
system_time_snapshot::valid has to be evaluated at the call sites
anyway. - Jacob
- Picked up the first patch from Davids follow up series, which extends
the snapshot mechanism so that derived clocksources (like kvmclock and
Hyper-V scaled TSC) can return the actual underlying hardware counter
value (TSC for the two examples).
- Collected Reviewed/Acked/Tested-by tags
Delta patch against v1 below.
The series is based on v7.1-rc2 and also available from git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git timekeeping-ptp-extend-v2
Thanks,
tglx
---
diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c
index 616062587510..b056c652ff01 100644
--- a/arch/arm64/kvm/hyp_trace.c
+++ b/arch/arm64/kvm/hyp_trace.c
@@ -51,8 +51,8 @@ static void __hyp_clock_work(struct work_struct *work)
hyp_clock = container_of(dwork, struct hyp_trace_clock, work);
- ktime_get_snapshot_id(&snap, CLOCK_BOOTTIME);
- boot = ktime_to_ns(snap.sys);
+ ktime_get_snapshot_id(CLOCK_BOOTTIME, &snap);
+ boot = ktime_to_ns(snap.systime);
delta_boot = boot - hyp_clock->boot;
delta_cycles = snap.cycles - hyp_clock->cycles;
@@ -120,7 +120,7 @@ static void hyp_trace_clock_enable(struct hyp_trace_clock *hyp_clock, bool enabl
ktime_get_snapshot_id(&snap, CLOCK_BOOTTIME);
- hyp_clock->boot = ktime_to_ns(snap.sys);
+ hyp_clock->boot = ktime_to_ns(snap.systime);
hyp_clock->cycles = snap.cycles;
hyp_clock->mult = 0;
diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
index e60cc7ed3e70..b11b8821c9fb 100644
--- a/arch/arm64/kvm/hypercalls.c
+++ b/arch/arm64/kvm/hypercalls.c
@@ -28,7 +28,7 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val)
* system time and counter value must captured at the same
* time to keep consistency and precision.
*/
- ktime_get_snapshot_id(&systime_snapshot, CLOCK_REALTIME);
+ ktime_get_snapshot_id(CLOCK_REALTIME, &systime_snapshot);
/*
* This is only valid if the current clocksource is the
@@ -61,8 +61,8 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val)
* in the future (about 292 years from 1970, and at that stage
* nobody will give a damn about it).
*/
- val[0] = upper_32_bits(systime_snapshot.sys);
- val[1] = lower_32_bits(systime_snapshot.sys);
+ val[0] = upper_32_bits(systime_snapshot.systime);
+ val[1] = lower_32_bits(systime_snapshot.systime);
val[2] = upper_32_bits(cycles);
val[3] = lower_32_bits(cycles);
}
diff --git a/drivers/net/dsa/sja1105/sja1105_main.c b/drivers/net/dsa/sja1105/sja1105_main.c
index 1d5fef4df560..2697073dbf90 100644
--- a/drivers/net/dsa/sja1105/sja1105_main.c
+++ b/drivers/net/dsa/sja1105/sja1105_main.c
@@ -2310,10 +2310,10 @@ int sja1105_static_config_reload(struct sja1105_private *priv,
goto out;
}
- t1 = ktime_to_ns(ptp_sts_before.pre_sts.sys);
- t2 = ktime_to_ns(ptp_sts_before.post_sts.sys);
- t3 = ktime_to_ns(ptp_sts_after.pre_sts.sys);
- t4 = ktime_to_ns(ptp_sts_after.post_sts.sys);
+ t1 = ktime_to_ns(ptp_sts_before.pre_sts.systime);
+ t2 = ktime_to_ns(ptp_sts_before.post_sts.systime);
+ t3 = ktime_to_ns(ptp_sts_after.pre_sts.systime);
+ t4 = ktime_to_ns(ptp_sts_after.post_sts.systime);
/* Mid point, corresponds to pre-reset PTPCLKVAL */
t12 = t1 + (t2 - t1) / 2;
/* Mid point, corresponds to post-reset PTPCLKVAL, aka 0 */
diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c
index 5023fc1587f9..f9e4ec6f7ebb 100644
--- a/drivers/net/ethernet/intel/ice/ice_ptp.c
+++ b/drivers/net/ethernet/intel/ice/ice_ptp.c
@@ -2117,7 +2117,7 @@ static int ice_capture_crosststamp(ktime_t *device,
}
/* Snapshot system time for historic interpolation */
- ktime_get_snapshot_id(&ctx->snapshot, ctx->snapshot_clock_id);
+ ktime_get_snapshot_id(ctx->snapshot_clock_id, &ctx->snapshot);
/* Program cmd to master timer */
ice_ptp_src_cmd(hw, ICE_PTP_READ_TIME);
diff --git a/drivers/net/ethernet/intel/igc/igc_ptp.c b/drivers/net/ethernet/intel/igc/igc_ptp.c
index 9b8b4a04e32d..b40aba9ab685 100644
--- a/drivers/net/ethernet/intel/igc/igc_ptp.c
+++ b/drivers/net/ethernet/intel/igc/igc_ptp.c
@@ -1049,7 +1049,7 @@ static int igc_phc_get_syncdevicetime(ktime_t *device,
*/
do {
/* Get a snapshot of system clocks to use as historic value. */
- ktime_get_snapshot_id(&adapter->snapshot, adapter->snapshot_clock_id);
+ ktime_get_snapshot_id(adapter->snapshot_clock_id, &adapter->snapshot);
igc_ptm_trigger(hw);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
index beb80912b9d5..5df786133e4b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c
@@ -340,7 +340,7 @@ static int mlx5_ptp_getcrosststamp(struct ptp_clock_info *ptp,
goto unlock;
}
- ktime_get_snapshot_id(&history_begin, cts->clock_id);
+ ktime_get_snapshot_id(cts->clock_id, &history_begin);
err = get_device_system_crosststamp(mlx5_mtctr_syncdevicetime, mdev,
&history_begin, cts);
@@ -366,7 +366,7 @@ static int mlx5_ptp_getcrosscycles(struct ptp_clock_info *ptp,
goto unlock;
}
- ktime_get_snapshot_id(&history_begin, cts->clock_id);
+ ktime_get_snapshot_id(cts->clock_id, &history_begin);
err = get_device_system_crosststamp(mlx5_mtctr_syncdevicecyclestime,
mdev, &history_begin, cts);
diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index aed5c13cd1be..dc23cd708cfe 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -392,11 +392,11 @@ static long ptp_sys_offset_extended(struct ptp_clock *ptp, void __user *arg,
extoff->ts[i][1].sec = ts.tv_sec;
extoff->ts[i][1].nsec = ts.tv_nsec;
- ts = ktime_to_timespec64(sts.pre_sts.sys);
+ ts = ktime_to_timespec64(sts.pre_sts.systime);
extoff->ts[i][0].sec = ts.tv_sec;
extoff->ts[i][0].nsec = ts.tv_nsec;
- ts = ktime_to_timespec64(sts.post_sts.sys);
+ ts = ktime_to_timespec64(sts.post_sts.systime);
extoff->ts[i][2].sec = ts.tv_sec;
extoff->ts[i][2].nsec = ts.tv_nsec;
}
diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c
index b7a23936a44d..28b0302c6250 100644
--- a/drivers/ptp/ptp_ocp.c
+++ b/drivers/ptp/ptp_ocp.c
@@ -1492,7 +1492,7 @@ __ptp_ocp_gettime_locked(struct ptp_ocp *bp, struct timespec64 *ts,
ptp_read_system_postts(sts);
if (sts && bp->ts_window_adjust)
- sts->post_ts.sys -= bp->ts_window_adjust;
+ sts->post_sts.systime -= bp->ts_window_adjust;
time_ns = ioread32(&bp->reg->time_ns);
time_sec = ioread32(&bp->reg->time_sec);
@@ -4592,8 +4592,8 @@ ptp_ocp_summary_show(struct seq_file *s, void *data)
struct timespec64 sys_ts;
s64 pre_ns, post_ns, ns;
- pre_ns = ktime_to_ns(sts.pre_sts.sys);
- post_ns = ktime_to_ns(sts.post_sts.sys);
+ pre_ns = ktime_to_ns(sts.pre_sts.systime);
+ post_ns = ktime_to_ns(sts.post_sts.systime);
ns = (pre_ns + post_ns) / 2;
ns += (s64)bp->utc_tai_offset * NSEC_PER_SEC;
sys_ts = ns_to_timespec64(ns);
diff --git a/drivers/ptp/ptp_vmclock.c b/drivers/ptp/ptp_vmclock.c
index cb18c15a4697..d6a5a533164a 100644
--- a/drivers/ptp/ptp_vmclock.c
+++ b/drivers/ptp/ptp_vmclock.c
@@ -263,7 +263,7 @@ static int ptp_vmclock_getcrosststamp(struct ptp_clock_info *ptp,
if (ret == -ENODEV) {
struct system_time_snapshot systime_snapshot;
- ktime_get_snapshot_id(&systime_snapshot, CLOCK_REALTIME);
+ ktime_get_snapshot_id(CLOCK_REALTIME, &systime_snapshot);
if (systime_snapshot.cs_id == CSID_X86_TSC ||
systime_snapshot.cs_id == CSID_X86_KVM_CLK) {
diff --git a/drivers/virtio/virtio_rtc_ptp.c b/drivers/virtio/virtio_rtc_ptp.c
index e15d00aeb01d..ff8d834493dc 100644
--- a/drivers/virtio/virtio_rtc_ptp.c
+++ b/drivers/virtio/virtio_rtc_ptp.c
@@ -139,7 +139,7 @@ static int viortc_ptp_getcrosststamp(struct ptp_clock_info *ptp,
if (ret)
return ret;
- ktime_get_snapshot_id(&history_begin, xtstamp->clock_id);
+ ktime_get_snapshot_id(xtstamp->clock_id, &history_begin);
if (history_begin.cs_id != cs_id)
return -EOPNOTSUPP;
diff --git a/include/linux/clocksource.h b/include/linux/clocksource.h
index 7c38190b10bf..6d9ddf1587a2 100644
--- a/include/linux/clocksource.h
+++ b/include/linux/clocksource.h
@@ -31,6 +31,21 @@ struct module;
#include <vdso/clocksource.h>
+/**
+ * struct clocksource_hw_snapshot - Snapshot for the underlying hardware counter of derived
+ * clocksources like kvmclock or Hyper-V scaled TSC
+ * @hw_cycles: The hardware counter value
+ * @hw_csid: Clocksource ID of the hardware counter
+ *
+ * Such clocksources must implement the read_snapshot() callback and fill in the
+ * hardware counter value, the clocksource ID of the hardware counter and derive
+ * the actual clocksource cycles from @hw_cycles to provide an atomic snapshot
+ */
+struct clocksource_hw_snapshot {
+ u64 hw_cycles;
+ enum clocksource_ids hw_csid;
+};
+
/**
* struct clocksource - hardware abstraction for a free running counter
* Provides mostly state-free accessors to the underlying hardware.
@@ -72,6 +87,14 @@ struct module;
* @flags: Flags describing special properties
* @base: Hardware abstraction for clock on which a clocksource
* is based
+ * @read_snapshot: Extended @read() function for clocksources such as
+ * kvmclock or the Hyper-V scaled TSC where the actual
+ * clocksource value for timekeeping is calculated from an
+ * underlying hardware counter. Returns the timekeeping
+ * relevant cycle value and stores the raw value of the
+ * underlying counter from which it was calculated
+ * including the clocksource ID of that counter in the
+ * clocksource hardware snapshot.
* @enable: Optional function to enable the clocksource
* @disable: Optional function to disable the clocksource
* @suspend: Optional suspend function for the clocksource
@@ -113,6 +136,7 @@ struct clocksource {
unsigned long flags;
struct clocksource_base *base;
+ u64 (*read_snapshot)(struct clocksource *cs, struct clocksource_hw_snapshot *chs);
int (*enable)(struct clocksource *cs);
void (*disable)(struct clocksource *cs);
void (*suspend)(struct clocksource *cs);
diff --git a/include/linux/pps_kernel.h b/include/linux/pps_kernel.h
index cd80f1cb96a9..9f088c9023b1 100644
--- a/include/linux/pps_kernel.h
+++ b/include/linux/pps_kernel.h
@@ -102,9 +102,9 @@ static inline void pps_get_ts(struct pps_event_time *ts)
#ifdef CONFIG_NTP_PPS
struct system_time_snapshot snap;
- ktime_get_snapshot_id(&snap, CLOCK_REALTIME);
- ts->ts_real = ktime_to_timespec64(snap.sys);
- ts->ts_raw = ktime_to_timespec64(snap.raw);
+ ktime_get_snapshot_id(CLOCK_REALTIME, &snap);
+ ts->ts_real = ktime_to_timespec64(snap.systime);
+ ts->ts_raw = ktime_to_timespec64(snap.monoraw);
#else
ktime_get_real_ts64(&ts->ts_real);
#endif
diff --git a/include/linux/ptp_clock_kernel.h b/include/linux/ptp_clock_kernel.h
index df6c9aac458b..36a27a910595 100644
--- a/include/linux/ptp_clock_kernel.h
+++ b/include/linux/ptp_clock_kernel.h
@@ -511,13 +511,13 @@ static inline ktime_t ptp_convert_timestamp(const ktime_t *hwtstamp,
static inline void ptp_read_system_prets(struct ptp_system_timestamp *sts)
{
if (sts)
- ktime_get_snapshot_id(&sts->pre_sts, sts->clockid);
+ ktime_get_snapshot_id(sts->clockid, &sts->pre_sts);
}
static inline void ptp_read_system_postts(struct ptp_system_timestamp *sts)
{
if (sts)
- ktime_get_snapshot_id(&sts->post_sts, sts->clockid);
+ ktime_get_snapshot_id(sts->clockid, &sts->post_sts);
}
#endif
diff --git a/include/linux/timekeeping.h b/include/linux/timekeeping.h
index f7945f1048fc..984a866d293b 100644
--- a/include/linux/timekeeping.h
+++ b/include/linux/timekeeping.h
@@ -279,18 +279,24 @@ static inline bool ktime_get_aux_ts64(clockid_t id, struct timespec64 *kt) { ret
* struct system_time_snapshot - Simultaneous time capture of CLOCK_MONOTONIC_RAW,
* a selected CLOCK_* and the clocksource counter value
* @cycles: Clocksource counter value to produce the system times
- * @sys: The system time of the selected CLOCK ID
- * @raw: Monotonic raw system time
+ * @hw_cycles: For derived clocksources, the hardware counter value from
+ * which @cycles was derived
+ * @systime: The system time of the selected CLOCK ID
+ * @monoraw: Monotonic raw system time
* @cs_id: Clocksource ID
+ * @hw_csid: Clocksource ID of the underlying hardware counter for derived
+ * clocksources which implement the read_snapshot() callback.
* @clock_was_set_seq: The sequence number of clock-was-set events
* @cs_was_changed_seq: The sequence number of clocksource change events
* @valid: True if the snapshot is valid
*/
struct system_time_snapshot {
u64 cycles;
- ktime_t sys;
- ktime_t raw;
+ u64 hw_cycles;
+ ktime_t systime;
+ ktime_t monoraw;
enum clocksource_ids cs_id;
+ enum clocksource_ids hw_csid;
unsigned int clock_was_set_seq;
u8 cs_was_changed_seq;
u8 valid;
@@ -348,8 +354,7 @@ extern int get_device_system_crosststamp(
* Simultaneously snapshot a given clock with MONOTONIC_RAW and the underlying
* clocksource counter value.
*/
-extern bool ktime_get_snapshot_id(struct system_time_snapshot *systime_snapshot,
- clockid_t clock_id);
+extern void ktime_get_snapshot_id(clockid_t clock_id, struct system_time_snapshot *systime_snapshot);
/*
* Persistent clock related interfaces
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index c4fd7229b7da..0d5b67f609bb 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -320,6 +320,7 @@ static __always_inline u64 tk_clock_read(const struct tk_read_base *tkr)
return clock->read(clock);
}
+
static inline void clocksource_disable_inline_read(void) { }
static inline void clocksource_enable_inline_read(void) { }
#endif
@@ -1187,14 +1188,26 @@ noinstr time64_t __ktime_get_real_seconds(void)
return tk->xtime_sec;
}
+static inline u64 tk_clock_read_snapshot(const struct tk_read_base *tkr,
+ struct clocksource_hw_snapshot *chs)
+{
+ struct clocksource *clock = READ_ONCE(tkr->clock);
+
+ if (unlikely(clock->read_snapshot))
+ return clock->read_snapshot(clock, chs);
+
+ return clock->read(clock);
+}
+
+
/**
* ktime_get_snapshot_id - Simultaneously snapshot a given clock ID with
* CLOCK_MONOTONIC_RAW and the underlying
* clocksource counter value.
- * @systime_snapshot: Pointer to struct receiving the system time snapshot
* @clock_id: The clock ID to snapshot
+ * @systime_snapshot: Pointer to struct receiving the system time snapshot
*/
-bool ktime_get_snapshot_id(struct system_time_snapshot *systime_snapshot, clockid_t clock_id)
+void ktime_get_snapshot_id(clockid_t clock_id, struct system_time_snapshot *systime_snapshot)
{
ktime_t base_raw, base_sys, offs_sys, *offs, offs_zero = 0;
u64 nsec_raw, nsec_sys, now;
@@ -1206,7 +1219,7 @@ bool ktime_get_snapshot_id(struct system_time_snapshot *systime_snapshot, clocki
systime_snapshot->valid = false;
if (WARN_ON_ONCE(timekeeping_suspended))
- return false;
+ return;
switch (clock_id) {
case CLOCK_REALTIME:
@@ -1226,25 +1239,31 @@ bool ktime_get_snapshot_id(struct system_time_snapshot *systime_snapshot, clocki
case CLOCK_AUX ... CLOCK_AUX_LAST:
tkd = aux_get_tk_data(clock_id);
if (!tkd)
- return false;
+ return;
offs = &tkd->timekeeper.offs_aux;
break;
default:
WARN_ON_ONCE(1);
- return false;
+ return;
}
tk = &tkd->timekeeper;
do {
+ struct clocksource_hw_snapshot chs = { };
+
seq = read_seqcount_begin(&tkd->seq);
/* Aux clocks can be invalid */
if (!tk->clock_valid)
- return false;
+ return;
- now = tk_clock_read(&tk->tkr_mono);
+ now = tk_clock_read_snapshot(&tk->tkr_mono, &chs);
systime_snapshot->cs_id = tk->tkr_mono.clock->id;
+
+ systime_snapshot->hw_cycles = chs.hw_cycles;
+ systime_snapshot->hw_csid = chs.hw_csid;
+
systime_snapshot->cs_was_changed_seq = tk->cs_was_changed_seq;
systime_snapshot->clock_was_set_seq = tk->clock_was_set_seq;
@@ -1257,18 +1276,17 @@ bool ktime_get_snapshot_id(struct system_time_snapshot *systime_snapshot, clocki
} while (read_seqcount_retry(&tkd->seq, seq));
systime_snapshot->cycles = now;
- systime_snapshot->sys = ktime_add_ns(base_sys, offs_sys + nsec_sys);
- systime_snapshot->raw = ktime_add_ns(base_raw, nsec_raw);
+ systime_snapshot->systime = ktime_add_ns(base_sys, offs_sys + nsec_sys);
+ systime_snapshot->monoraw = ktime_add_ns(base_raw, nsec_raw);
/*
* Special case for PTP. Just transfer the raw time into sys,
- * so the call sites can consistently use snap::sys.
+ * so the call sites can consistently use snap::systime.
*/
if (clock_id == CLOCK_MONOTONIC_RAW)
- systime_snapshot->sys = systime_snapshot->raw;
+ systime_snapshot->systime = systime_snapshot->monoraw;
/* Tell the consumer that this snapshot is valid */
systime_snapshot->valid = true;
- return true;
}
EXPORT_SYMBOL_GPL(ktime_get_snapshot_id);
@@ -1330,7 +1348,7 @@ static int adjust_historical_crosststamp(struct system_time_snapshot *history,
* Scale the monotonic raw time delta by:
* partial_history_cycles / total_history_cycles
*/
- corr_raw = (u64)ktime_to_ns(ktime_sub(ts->sys_monoraw, history->raw));
+ corr_raw = (u64)ktime_to_ns(ktime_sub(ts->sys_monoraw, history->monoraw));
ret = scale64_check_overflow(partial_history_cycles,
total_history_cycles, &corr_raw);
if (ret)
@@ -1347,7 +1365,7 @@ static int adjust_historical_crosststamp(struct system_time_snapshot *history,
if (discontinuity) {
corr_sys = mul_u64_u32_div(corr_raw, tk->tkr_mono.mult, tk->tkr_raw.mult);
} else {
- corr_sys = (u64)ktime_to_ns(ktime_sub(ts->sys_systime, history->sys));
+ corr_sys = (u64)ktime_to_ns(ktime_sub(ts->sys_systime, history->systime));
ret = scale64_check_overflow(partial_history_cycles, total_history_cycles,
&corr_sys);
if (ret)
@@ -1356,8 +1374,8 @@ static int adjust_historical_crosststamp(struct system_time_snapshot *history,
/* Fixup monotonic raw and system time time values */
if (interp_forward) {
- ts->sys_monoraw = ktime_add_ns(history->raw, corr_raw);
- ts->sys_systime = ktime_add_ns(history->sys, corr_sys);
+ ts->sys_monoraw = ktime_add_ns(history->monoraw, corr_raw);
+ ts->sys_systime = ktime_add_ns(history->systime, corr_sys);
} else {
ts->sys_monoraw = ktime_sub_ns(ts->sys_monoraw, corr_raw);
ts->sys_systime = ktime_sub_ns(ts->sys_systime, corr_sys);
@@ -1521,12 +1539,12 @@ int get_device_system_crosststamp(int (*get_time_fn)
case CLOCK_AUX ... CLOCK_AUX_LAST:
tkd = aux_get_tk_data(xtstamp->clock_id);
if (!tkd)
- return false;
+ return -ENODEV;
offs = &tkd->timekeeper.offs_aux;
break;
default:
WARN_ON_ONCE(1);
- return false;
+ return -ENODEV;
}
tk = &tkd->timekeeper;
^ permalink raw reply related
* Re: [PATCH net v2] vsock/virtio: bind uarg before filling zerocopy skb
From: patchwork-bot+netdevbpf @ 2026-05-29 19:50 UTC (permalink / raw)
To: Lin Ma
Cc: kuba, avkrasnov, cenxianlong, chenzhe, cuirongzhen, davem,
edumazet, eperezma, horms, jasowang, kvm, linux-kernel, mst,
netdev, pabeni, sgarzare, stefanha, tanjingguo, virtualization,
xuanzhuo
In-Reply-To: <20260527023301.1075581-1-malin89@huawei.com>
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:
On Wed, 27 May 2026 10:33:01 +0800 you wrote:
> From: Jingguo Tan <tanjingguo@huawei.com>
>
> virtio_transport_send_pkt_info() allocates or reuses the zerocopy uarg
> before entering the send loop, but virtio_transport_alloc_skb() still
> fills the skb before it inherits that uarg. When fixed-buffer vectored
> zerocopy hits MAX_SKB_FRAGS, io_sg_from_iter() may partially attach
> managed frags and return -EMSGSIZE. The rollback path call kfree_skb()
> to free an skb that carries SKBFL_MANAGED_FRAG_REFS but no uarg, so
> skb_release_data() falls through to ordinary frag unref.
>
> [...]
Here is the summary with links:
- [net,v2] vsock/virtio: bind uarg before filling zerocopy skb
https://git.kernel.org/netdev/net/c/1e584c304cfb
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply
* Re: [PATCH v3] media: add virtio-media driver
From: Brian Daniels @ 2026-05-29 16:03 UTC (permalink / raw)
To: mchehab+huawei
Cc: acourbot, adelva, aesteve, changyeon, daniel.almeida, eperezma,
gnurou, gurchetansingh, hverkuil, jasowang, linux-kernel,
linux-media, mchehab, mst, nicolas.dufresne, virtualization,
xuanzhuo
In-Reply-To: <20250909111216.18d5f78c@foz.lan>
Hi there! My name is Brian Daniels and I'll be taking over upstreaming this
driver from Alexandre Courbot.
I've consulted with Alexandre and my plan is to upload a v4 set of patches
shortly based on the feedback from this revision.
Before doing so, I'd like to address some of your comments:
> Hi Alex,
>
> I didn't see on a first glance anything that would cause locking
> issues here, but, as I pointed on my last e-mail, testing with
> qv4l2 at the max res of my C920 camera, it ended keeping 24 CPUs
> busy without showing any results with qv4l2 (via ssh at the same
> machine). So, I suspect that there are issues somewhere, but I didn't
> debug any further.
>
> Please do some tests with a high-res camera, using either ssh or
> GPU emulation to see how this behaves with real apps.
I did some testing with a 1080p USB webcam over ssh using ffmpeg and I was able
to stream video to disk without any issue. Let me know if you'd prefer I test
with a specific setup.
> > +config MEDIA_VIRTIO
> > + tristate "Virtio-media Driver"
> > + depends on VIRTIO && VIDEO_DEV && 64BIT && (X86 || (ARM && CPU_LITTLE_ENDIAN))
>
> Why are you limiting it to x86_64 and arm64 little endian?
The little endian requirement comes for the section 5.22.6.1.5 of the virtio
v1.4 specification [1]. The limitation to x86_64 and arm64 is for two reasons:
1. The specification requires all v4l2 structures to use the 64-bit layout
2. This driver has only been tested on x86_64 and arm64 so far
> > +/**
> > + * enum virtio_media_memory - Memory types supported by virtio-media.
> > + * @VIRTIO_MEDIA_MMAP: memory allocated and managed by device. Can be mapped
> > + * into the guest using VIRTIO_MEDIA_CMD_MMAP.
> > + * @VIRTIO_MEDIA_SHARED_PAGES: memory allocated by the driver. Passed to the
> > + * device using virtio_media_sg_entry.
> > + * @VIRTIO_MEDIA_OBJECT: memory backed by a virtio object.
> > + */
> > +enum virtio_media_memory {
> > + VIRTIO_MEDIA_MMAP = V4L2_MEMORY_MMAP,
> > + VIRTIO_MEDIA_SHARED_PAGES = V4L2_MEMORY_USERPTR,
> > + VIRTIO_MEDIA_OBJECT = V4L2_MEMORY_DMABUF,
> > +};
>
> I'm not a big fan of renaming USERPTR to SHARED_PAGES and
> DMABUF to OBJECT, as it makes harder for reviewers and contributors
> to remember about this mapping. Also, everybody knows exactly what
> DMABUF means, so, it sounds to me that this obfuscates a little bit
> the driver.
>
> Also, why are you encapsulating V4L2 names into VIRTIO_* namespace?
> This just adds extra complexity for reviewers without any real
> benefit.
>
> Besides that, doing a grep on the patch, it sounds that this ma
> is not used anywhere.
>
> So, please drop this mapping.
Agreed I don't see it being used anywhere, I think this was included originally
since it's part of the virtio spec. I will remove it.
> > +#define VIRTIO_MEDIA_EVT_ERROR 0
> > +#define VIRTIO_MEDIA_EVT_DQBUF 1
> > +#define VIRTIO_MEDIA_EVT_EVENT 2
>
> OK, here, media events are different than virtio events, so having
> a virtio-specific events make sense. Yet, better to add a comment
> about that.
>
> Also, V4L events are defined as:
>
> #define V4L2_EVENT_ALL 0
> #define V4L2_EVENT_VSYNC 1
> #define V4L2_EVENT_EOS 2
> #define V4L2_EVENT_CTRL 3
> #define V4L2_EVENT_FRAME_SYNC 4
> #define V4L2_EVENT_SOURCE_CHANGE 5
> #define V4L2_EVENT_MOTION_DET 6
> #define V4L2_EVENT_PRIVATE_START 0x08000000
>
> As one may end wanting to map them on some future, I would change
> the definitions above to:
>
> #define VIRTIO_MEDIA_EVT_ERROR V4L2_EVENT_PRIVATE_START
> #define VIRTIO_MEDIA_EVT_DQBUF (V4L2_EVENT_PRIVATE_START + 1)
> #define VIRTIO_MEDIA_EVT_EVENT (V4L2_EVENT_PRIVATE_START + 2)
These event values (VIRTIO_MEDIA_EVT_*) are set by section 5.22.6.2.1 of the
virtio v1.4 specification [2]. I don't believe we have the ability to change
these values as suggested without changing the specification. That would most
likely be a lengthy process, so I'd prefer to keep it as written if that's not
an issue.
> > +#define VIRTIO_MEDIA_MAX_PLANES VIDEO_MAX_PLANES
>
> Here: why renaming it to VIRTIO_* namespace?
This matches the name in section 5.22.6.2.3 of the virtio v1.4 specification
[3]. And Alexandre stated the reason why it was renamed from VIDEO_* to
VIRTIO_MEDIA_* was from an early piece of feedback to avoid V4L2-specific names.
Even though virtio-media reuses the v4l2 structures and API, its possible to use
virtio-media with a different media implementation other than v4l2.
> > +/**
> > + * struct virtio_media_session - A session on a virtio_media device.
> > + * @fh: file handler for the session.
> > + * @id: session ID used to communicate with the device.
> > + * @nonblocking_dequeue: whether dequeue should block or not (nonblocking if
> > + * file opened with O_NONBLOCK).
> > + * @uses_mplane: whether the queues for this session use the MPLANE API or not.
> > + * @cmd: union of session-related commands. A session can have one command currently running.
> > + * @resp: union of session-related responses. A session can wait on one command only.
> > + * @shadow_buf: shadow buffer where data to be added to the descriptor chain can
> > + * be staged before being sent to the device.
> > + * @command_sgs: SG table gathering descriptors for a given command and its response.
> > + * @queues: state of all the queues for this session.
> > + * @queues_lock: protects all members fo the queues for this session.
> > + * virtio_media_queue_state`.
> > + * @dqbuf_wait: waitqueue for dequeued buffers, if ``VIDIOC_DQBUF`` needs to
> > + * block or when polling.
> > + * @list: link into the list of sessions for the device.
> > + */
> > +struct virtio_media_session {
> > + struct v4l2_fh fh;
> > + u32 id;
> > + bool nonblocking_dequeue;
> > + bool uses_mplane;
> > +
> > + union {
> > + struct virtio_media_cmd_close close;
> > + struct virtio_media_cmd_ioctl ioctl;
> > + struct virtio_media_cmd_mmap mmap;
> > + } cmd;
> > +
> > + union {
> > + struct virtio_media_resp_ioctl ioctl;
> > + struct virtio_media_resp_mmap mmap;
> > + } resp;
> > +
>
> Heh, the above is tricky, as to parse the struct, one needs first
> to check cmd.cmd to identify what values to pick from enums.
> Also, the command is stored as: cmd.[close|ioctl|imap].cmd.
>
> IMO, better to place the headers explicitly there, e.g.
>
> union {
> struct virtio_media_cmd_header hdr;
> struct virtio_media_cmd_close close;
> struct virtio_media_cmd_ioctl ioctl;
> struct virtio_media_cmd_mmap mmap;
> } send;
> union {
> struct virtio_media_resp_header hdr;
> struct virtio_media_resp_ioctl ioctl;
> struct virtio_media_resp_mmap mmap;
> } resp;
>
> Also, currently, there are 5 defined commands:
>
> #define VIRTIO_MEDIA_CMD_OPEN 1
> #define VIRTIO_MEDIA_CMD_CLOSE 2
> #define VIRTIO_MEDIA_CMD_IOCTL 3
> #define VIRTIO_MEDIA_CMD_MMAP 4
> #define VIRTIO_MEDIA_CMD_MUNMAP 5
>
> If the data struct is limited only for close/ioctl/mmap, please
> document it and point what structure(s) other commands use.
Tricky indeed!
However, I don't believe its necessary to add an explict header member to the
union. Whenever the driver parses the union, it already has the necessary
context to determine which union member to use:
- The struct virtio_media_cmd_* instances are always created by the driver and
sent to the device, so there are no unknowns there
- The struct virtio_media_resp_* instances are always parsed in the same
function that sends the corresponding command, so again there's no
uncertainty about which union member to access.
For these reasons, I would hesistate to add the `struct virtio_media_cmd_header
hdr` as previously suggested. Please let me know if you disagree or if I've
misunderstood your concern.
That all being said, I have added comments about which structures use which
commands in the upcoming v4 of the patches.
> > +/**
> > + * struct virtio_media - Virtio-media device.
> > + * @v4l2_dev: v4l2_device for the media device.
> > + * @video_dev: video_device for the media device.
> > + * @virtio_dev: virtio device for the media device.
> > + * @commandq: virtio command queue.
> > + * @eventq: virtio event queue.
> > + * @eventq_work: work to run when events are received on @eventq.
> > + * @mmap_region: region into which MMAP buffers are mapped by the host.
> > + * @event_buffer: buffer for event descriptors.
> > + * @sessions: list of active sessions on the device.
> > + * @sessions_lock: protects @sessions and ``virtio_media_session::list``.
> > + * @events_lock: prevents concurrent processing of events.
> > + * @cmd: union of device-related commands.
> > + * @resp: union of device-related responses.
> > + * @vlock: serializes access to the command queue.
> > + * @wq: waitqueue for host responses on the command queue.
> > + */
> > +struct virtio_media {
> > + struct v4l2_device v4l2_dev;
> > + struct video_device video_dev;
> > +
> > + struct virtio_device *virtio_dev;
> > + struct virtqueue *commandq;
> > + struct virtqueue *eventq;
> > + struct work_struct eventq_work;
> > +
> > + struct virtio_shm_region mmap_region;
> > +
> > + void *event_buffer;
> > +
> > + struct list_head sessions;
> > + struct mutex sessions_lock;
> > +
> > + struct mutex events_lock;
> > +
> > + union {
> > + struct virtio_media_cmd_open open;
> > + struct virtio_media_cmd_munmap munmap;
> > + } cmd;
> > +
> > + union {
> > + struct virtio_media_resp_open open;
> > + struct virtio_media_resp_munmap munmap;
> > + } resp;
>
> Based on struct virtio_media_session, I'm assuming here that
> this struct is used only for two commands, right? Please document
> it at kernel-doc markup and add a point to the other structure used
> for the other commands.
>
> The same comment about headers apply to the union here: place
> the header explicitly at the union.
The same thing I said above applies here as well. I will add the comments as
requested in v4.
[1] https://docs.oasis-open.org/virtio/virtio/v1.4/csprd01/virtio-v1.4-csprd01-diff-from-v1.2-cs01.html#x1-8360005
[2] https://docs.oasis-open.org/virtio/virtio/v1.4/csprd01/virtio-v1.4-csprd01-diff-from-v1.2-cs01.html#x1-8560001
[3] https://docs.oasis-open.org/virtio/virtio/v1.4/csprd01/virtio-v1.4-csprd01-diff-from-v1.2-cs01.html#x1-8600003
^ permalink raw reply
* [PATCH v9 37/37] virtio_balloon: implement VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
From: Michael S. Tsirkin @ 2026-05-29 15:25 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
When the device offers DEVICE_INIT_ON_INFLATE (bit 7), the device
initializes inflated pages and returns a per-page bitmap indicating
which pages were successfully initialized.
The driver appends a device-writable bitmap buffer to each inflate
descriptor chain via virtqueue_add_sgs. After the host acknowledges,
the driver checks bitmap bits (bounded by used_len) and marks pages
with SetPageZeroed.
tell_host() returns used_len from virtqueue_get_buf(). Bitmap reads
are bounded: fill_balloon() and virtballoon_migratepage() only trust
bits within the used_len range.
On deflate, release_pages_balloon checks PageZeroed per page and
uses put_page_zeroed for pages the host initialized, propagating
the zeroed hint to the buddy allocator.
If inflate_vq has fewer than 2 descriptors, probe fails with
-ENOSPC. If PAGE_POISON is negotiated with non-zero poison_val,
the feature is cleared in validate().
See the virtio spec change:
https://lore.kernel.org/all/9c69b992c3dd83dfef3db92cd86b2fd8a0730d48.1777731396.git.mst@redhat.com
__SetPageZeroed on inflated pages is non-atomic but safe:
the balloon owns the page exclusively at this point.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
drivers/virtio/virtio_balloon.c | 107 ++++++++++++++++++++++++----
include/uapi/linux/virtio_balloon.h | 1 +
2 files changed, 96 insertions(+), 12 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index bf1172ad5419..4abc7b3b991c 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -20,6 +20,7 @@
#include <linux/mm.h>
#include <linux/page_reporting.h>
#include <linux/cc_platform.h>
+#include <asm-generic/bitops/le.h>
/*
* Balloon device works in 4K page units. So each page is pointed to by
@@ -122,6 +123,13 @@ struct virtio_balloon {
struct virtqueue *reporting_vq;
struct page_reporting_dev_info pr_dev_info;
+ /* No DMA alignment needed: ACCESS_PLATFORM is cleared,
+ * so virtio bypasses the DMA API. If this ever changes,
+ * add ____dma_from_device_aligned here.
+ */
+ /* Bitmap returned by host for DEVICE_INIT_ON_INFLATE */
+ DECLARE_BITMAP(inflate_bitmap, VIRTIO_BALLOON_ARRAY_PFNS_MAX);
+
/* State for keeping the wakeup_source active while adjusting the balloon */
spinlock_t wakeup_lock;
bool processing_wakeup_event;
@@ -182,22 +190,33 @@ static void balloon_ack(struct virtqueue *vq)
wake_up(&vb->acked);
}
-static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
+static unsigned int tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
{
- struct scatterlist sg;
+ struct scatterlist sg_out, sg_in;
+ struct scatterlist *sgs[] = { &sg_out, &sg_in };
unsigned int len;
- sg_init_one(&sg, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
+ sg_init_one(&sg_out, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
/* We made sure the vq is large enough so we should always
* be able to add one buffer to an empty queue.
*/
- virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+ if (vq == vb->inflate_vq &&
+ virtio_has_feature(vb->vdev,
+ VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE)) {
+ unsigned int bitmap_bytes;
+
+ bitmap_bytes = DIV_ROUND_UP(vb->num_pfns, 8);
+ bitmap_zero(vb->inflate_bitmap, vb->num_pfns);
+ sg_init_one(&sg_in, vb->inflate_bitmap, bitmap_bytes);
+ virtqueue_add_sgs(vq, sgs, 1, 1, vb, GFP_KERNEL);
+ } else {
+ virtqueue_add_outbuf(vq, &sg_out, 1, vb, GFP_KERNEL);
+ }
virtqueue_kick(vq);
- /* When host has read buffer, this completes via balloon_ack */
wait_event(vb->acked, virtqueue_get_buf(vq, &len));
-
+ return len;
}
static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_info,
@@ -300,8 +319,37 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
num_allocated_pages = vb->num_pfns;
/* Did we get any? */
- if (vb->num_pfns != 0)
- tell_host(vb, vb->inflate_vq);
+ if (vb->num_pfns != 0) {
+ unsigned int used_len = tell_host(vb, vb->inflate_vq);
+
+ if (virtio_has_feature(vb->vdev,
+ VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE)) {
+ unsigned int i;
+ unsigned int valid_bits = used_len * 8;
+
+ for (i = 0; i < vb->num_pfns;
+ i += VIRTIO_BALLOON_PAGES_PER_PAGE) {
+ unsigned int pfn, j;
+ bool zeroed = true;
+
+ if (i + VIRTIO_BALLOON_PAGES_PER_PAGE > valid_bits)
+ break;
+ for (j = 0; j < VIRTIO_BALLOON_PAGES_PER_PAGE; j++) {
+ if (!test_bit_le(i + j, vb->inflate_bitmap)) {
+ zeroed = false;
+ break;
+ }
+ }
+ if (zeroed) {
+ pfn = virtio32_to_cpu(vb->vdev,
+ vb->pfns[i]);
+ __SetPageZeroed(pfn_to_page(pfn >>
+ (PAGE_SHIFT -
+ VIRTIO_BALLOON_PFN_SHIFT)));
+ }
+ }
+ }
+ }
mutex_unlock(&vb->balloon_lock);
return num_allocated_pages;
@@ -314,7 +362,12 @@ static void release_pages_balloon(struct virtio_balloon *vb,
list_for_each_entry_safe(page, next, pages, lru) {
list_del(&page->lru);
- put_page(page); /* balloon reference */
+ if (PageZeroed(page)) {
+ __ClearPageZeroed(page);
+ put_page_zeroed(page);
+ } else {
+ put_page(page);
+ }
}
}
@@ -861,8 +914,27 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info,
/* balloon's page migration 1st step -- inflate "newpage" */
vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
set_page_pfns(vb, vb->pfns, newpage);
- tell_host(vb, vb->inflate_vq);
+ {
+ unsigned int used_len = tell_host(vb, vb->inflate_vq);
+ if (virtio_has_feature(vb->vdev,
+ VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE) &&
+ used_len >= DIV_ROUND_UP(VIRTIO_BALLOON_PAGES_PER_PAGE, 8)) {
+ unsigned int j;
+ bool zeroed = true;
+
+ for (j = 0; j < VIRTIO_BALLOON_PAGES_PER_PAGE; j++) {
+ if (!test_bit_le(j, vb->inflate_bitmap)) {
+ zeroed = false;
+ break;
+ }
+ }
+ if (zeroed)
+ __SetPageZeroed(newpage);
+ }
+ }
+
+ __ClearPageZeroed(page);
/* balloon's page migration 2nd step -- deflate "page" */
vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
set_page_pfns(vb, vb->pfns, page);
@@ -966,6 +1038,12 @@ static int virtballoon_probe(struct virtio_device *vdev)
if (err)
goto out_free_vb;
+ if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE) &&
+ virtqueue_get_vring_size(vb->inflate_vq) < 2) {
+ err = -ENOSPC;
+ goto out_del_vqs;
+ }
+
if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
vb->vb_dev_info.adjust_managed_page_count = true;
#ifdef CONFIG_BALLOON_MIGRATION
@@ -1191,11 +1269,15 @@ static int virtballoon_validate(struct virtio_device *vdev)
/* Device fills with poison_val, not zeros; disable zeroed hint */
if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON) &&
- !want_init_on_free())
+ !want_init_on_free()) {
__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
+ __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE);
+ }
- if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
+ __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE);
+ }
/*
* Balloon submits 1-2 sg entries max per buffer, virtqueue
* sizes are 128+. Disable indirect descriptors to avoid
@@ -1215,6 +1297,7 @@ static unsigned int features[] = {
VIRTIO_BALLOON_F_PAGE_POISON,
VIRTIO_BALLOON_F_REPORTING,
VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED,
+ VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE,
};
static struct virtio_driver virtio_balloon_driver = {
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index 13074631f300..cbaf18e0b17c 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -38,6 +38,7 @@
#define VIRTIO_BALLOON_F_PAGE_POISON 4 /* Guest is using page poisoning */
#define VIRTIO_BALLOON_F_REPORTING 5 /* Page reporting virtqueue */
#define VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED 6 /* Device initializes reported pages */
+#define VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE 7 /* Device initializes pages on inflate */
/* Size of a PFN in the balloon interface. */
#define VIRTIO_BALLOON_PFN_SHIFT 12
--
MST
^ permalink raw reply related
* [PATCH v9 36/37] mm: balloon: use put_page_zeroed for zeroed balloon pages
From: Michael S. Tsirkin @ 2026-05-29 15:25 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
When a balloon page marked PageZeroed is freed during migration,
use put_page_zeroed() to propagate the zeroed hint to the buddy
allocator. Previously the hint was silently lost via plain put_page().
No page has PageZeroed set yet; the next patch
(VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE) will set it on
pages the host has zeroed during inflate.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
mm/balloon.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/mm/balloon.c b/mm/balloon.c
index 96a8f1e20bc6..6c9dd8ab0c5d 100644
--- a/mm/balloon.c
+++ b/mm/balloon.c
@@ -324,7 +324,15 @@ static int balloon_page_migrate(struct page *newpage, struct page *page,
balloon_page_finalize(page);
spin_unlock_irqrestore(&balloon_pages_lock, flags);
- put_page(page);
+ if (PageZeroed(page)) {
+ /* Atomic to serialize with memory_failure's
+ * TestSetPageHWPoison; not under zone->lock here.
+ */
+ ClearPageZeroed(page);
+ put_page_zeroed(page);
+ } else {
+ put_page(page);
+ }
return 0;
}
--
MST
^ permalink raw reply related
* [PATCH v9 35/37] mm: add put_page_zeroed and folio_put_zeroed
From: Michael S. Tsirkin @ 2026-05-29 15:25 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli, Qi Zheng,
Shakeel Butt, Youngjun Park
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Add put_page_zeroed() / folio_put_zeroed() for callers that hold
a reference to a page known to be zeroed.
If this drops the last reference, the zeroed hint is
propagated to the buddy allocator. If someone else still holds a
reference, the hint is simply lost - this is best-effort.
This is useful for balloon drivers during deflation: the host
has already zeroed the pages, and the balloon is typically the
sole owner. But if the page happens to be shared, silently
dropping the hint is safe and avoids the need for callers to
check the refcount.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
include/linux/mm.h | 13 +++++++++++++
mm/swap.c | 20 ++++++++++++++++++--
2 files changed, 31 insertions(+), 2 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index d1e768dcda13..86638ffaad01 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1913,6 +1913,7 @@ static inline struct folio *virt_to_folio(const void *x)
}
void __folio_put(struct folio *folio);
+void __folio_put_zeroed(struct folio *folio);
void split_page(struct page *page, unsigned int order);
void folio_copy(struct folio *dst, struct folio *src);
@@ -2090,6 +2091,18 @@ static inline void folio_put(struct folio *folio)
__folio_put(folio);
}
+/* Caller must be sole owner to guarantee page is still zero */
+static inline void folio_put_zeroed(struct folio *folio)
+{
+ if (folio_put_testzero(folio))
+ __folio_put_zeroed(folio);
+}
+
+static inline void put_page_zeroed(struct page *page)
+{
+ folio_put_zeroed(page_folio(page));
+}
+
/**
* folio_put_refs - Reduce the reference count on a folio.
* @folio: The folio.
diff --git a/mm/swap.c b/mm/swap.c
index 5cc44f0de987..ecec780172ad 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -94,13 +94,15 @@ static void page_cache_release(struct folio *folio)
lruvec_unlock_irqrestore(lruvec, flags);
}
-void __folio_put(struct folio *folio)
+static void ___folio_put(struct folio *folio, bool zeroed)
{
+ /* zeroed hint ignored for now, no current user */
if (unlikely(folio_is_zone_device(folio))) {
free_zone_device_folio(folio);
return;
}
+ /* zeroed hint ignored for now, no current user */
if (folio_test_hugetlb(folio)) {
free_huge_folio(folio);
return;
@@ -109,10 +111,24 @@ void __folio_put(struct folio *folio)
page_cache_release(folio);
folio_unqueue_deferred_split(folio);
mem_cgroup_uncharge(folio);
- free_frozen_pages(&folio->page, folio_order(folio));
+ if (zeroed)
+ free_frozen_pages_zeroed(&folio->page, folio_order(folio));
+ else
+ free_frozen_pages(&folio->page, folio_order(folio));
+}
+
+void __folio_put(struct folio *folio)
+{
+ ___folio_put(folio, false);
}
EXPORT_SYMBOL(__folio_put);
+void __folio_put_zeroed(struct folio *folio)
+{
+ ___folio_put(folio, true);
+}
+EXPORT_SYMBOL(__folio_put_zeroed);
+
typedef void (*move_fn_t)(struct lruvec *lruvec, struct folio *folio);
static void lru_add(struct lruvec *lruvec, struct folio *folio)
--
MST
^ permalink raw reply related
* [PATCH v9 34/37] mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
From: Michael S. Tsirkin @ 2026-05-29 15:25 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
In __free_pages_prepare(), when FPI_ZEROED is set the page is already
known to be zero. We can skip kernel_init_pages() if page poisoning is
not enabled (because poison would overwrite the zeroes).
This avoids redundant zeroing work when freeing pages that are already
known to contain all zeros.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
mm/page_alloc.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 399e2038dcd9..90a92be96ebf 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1447,7 +1447,14 @@ __always_inline bool __free_pages_prepare(struct page *page,
if (kasan_has_integrated_init())
init = false;
}
- if (init)
+ /*
+ * Skip redundant zeroing when the page is already known-zero
+ * (FPI_ZEROED) and page poisoning did not overwrite it.
+ * When page_poisoning is enabled, kernel_poison_pages above
+ * wrote PAGE_POISON (0xAA), so we must re-zero.
+ */
+ if (init && !((fpi_flags & FPI_ZEROED) &&
+ !page_poisoning_enabled_static()))
kernel_init_pages(page, 1 << order);
/*
--
MST
^ permalink raw reply related
* [PATCH v9 33/37] mm: add free_frozen_pages_zeroed
From: Michael S. Tsirkin @ 2026-05-29 15:25 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Add free_frozen_pages_zeroed(page, order) to free a frozen page
while marking it as zeroed, so the next allocation can skip
redundant zeroing.
An FPI_ZEROED internal flag carries the hint through the free path.
PageZeroed is set after __free_pages_prepare() clears all flags,
so the hint survives on the free list.
__SetPageZeroed is non-atomic but safe here: the page is frozen
(refcount 0) and not yet on any free list.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
include/linux/gfp.h | 1 +
mm/internal.h | 1 -
mm/page_alloc.c | 29 ++++++++++++++++++++++++++++-
3 files changed, 29 insertions(+), 2 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 73109d4e31a4..d24b61e45861 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -384,6 +384,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas
extern void __free_pages(struct page *page, unsigned int order);
extern void free_pages_nolock(struct page *page, unsigned int order);
extern void free_pages(unsigned long addr, unsigned int order);
+void free_frozen_pages_zeroed(struct page *page, unsigned int order);
#define __free_page(page) __free_pages((page), 0)
#define free_page(addr) free_pages((addr), 0)
diff --git a/mm/internal.h b/mm/internal.h
index fd910743ddc3..4af5e72742ba 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -938,7 +938,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid,
#define __alloc_frozen_pages(...) \
alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
void free_frozen_pages(struct page *page, unsigned int order);
-void free_frozen_pages_zeroed(struct page *page, unsigned int order);
void free_unref_folios(struct folio_batch *fbatch);
#ifdef CONFIG_NUMA
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 702fa2ced1e1..399e2038dcd9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -90,6 +90,13 @@ typedef int __bitwise fpi_t;
/* Free the page without taking locks. Rely on trylock only. */
#define FPI_TRYLOCK ((__force fpi_t)BIT(2))
+/*
+ * The page contents are known to be zero (e.g., the host zeroed them
+ * during balloon deflate). Set PageZeroed after free so the next
+ * allocation can skip redundant zeroing.
+ */
+#define FPI_ZEROED ((__force fpi_t)BIT(3))
+
/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */
static DEFINE_MUTEX(pcp_batch_high_lock);
#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)
@@ -1599,8 +1606,19 @@ static void __free_pages_ok(struct page *page, unsigned int order,
unsigned long pfn = page_to_pfn(page);
struct zone *zone = page_zone(page);
- if (__free_pages_prepare(page, order, fpi_flags))
+ if (__free_pages_prepare(page, order, fpi_flags)) {
+ /*
+ * Only mark as zeroed if the page is actually still zero.
+ * kernel_poison_pages() overwrites with PAGE_POISON (0xAA)
+ * when CONFIG_PAGE_POISONING is enabled. With init_on_free,
+ * kernel_init_pages() re-zeroes after poisoning, so the
+ * page is still zero.
+ */
+ if ((fpi_flags & FPI_ZEROED) &&
+ (!page_poisoning_enabled_static() || want_init_on_free()))
+ __SetPageZeroed(page);
free_one_page(zone, page, pfn, order, fpi_flags);
+ }
}
void __meminit __free_pages_core(struct page *page, unsigned int order,
@@ -3021,6 +3039,9 @@ static void __free_frozen_pages(struct page *page, unsigned int order,
if (!__free_pages_prepare(page, order, fpi_flags))
return;
+ if (fpi_flags & FPI_ZEROED)
+ __SetPageZeroed(page);
+
/*
* We only track unmovable, reclaimable and movable on pcp lists.
* Place ISOLATE pages on the isolated list because they are being
@@ -3059,6 +3080,12 @@ void free_frozen_pages(struct page *page, unsigned int order)
__free_frozen_pages(page, order, FPI_NONE);
}
+void free_frozen_pages_zeroed(struct page *page, unsigned int order)
+{
+ __free_frozen_pages(page, order, FPI_ZEROED);
+}
+EXPORT_SYMBOL(free_frozen_pages_zeroed);
+
void free_frozen_pages_nolock(struct page *page, unsigned int order)
{
__free_frozen_pages(page, order, FPI_TRYLOCK);
--
MST
^ permalink raw reply related
* [PATCH v9 32/37] virtio_balloon: disable reporting zeroed optimization for confidential guests
From: Michael S. Tsirkin @ 2026-05-29 15:25 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
In confidential computing environments (TDX, SEV-SNP), the host
is untrusted and may lie about zeroing reported pages. Clear
DEVICE_INIT_REPORTED in validate() so the guest does not skip
re-zeroing based on hints from an untrusted device.
Note: currently REPORTING remains enabled and
VIRTIO_F_ACCESS_PLATFORM is cleared in CC environments.
This is known to leak free page physical addresses to the
host. Whether that, or ballooning in general, is a security
concern in CC is up to the user. This patch only disables
our new zeroed-page hints where the host is untrusted.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
drivers/virtio/virtio_balloon.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index e3afa6f32ba5..bf1172ad5419 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -19,6 +19,7 @@
#include <linux/wait.h>
#include <linux/mm.h>
#include <linux/page_reporting.h>
+#include <linux/cc_platform.h>
/*
* Balloon device works in 4K page units. So each page is pointed to by
@@ -1193,6 +1194,8 @@ static int virtballoon_validate(struct virtio_device *vdev)
!want_init_on_free())
__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
/*
* Balloon submits 1-2 sg entries max per buffer, virtqueue
* sizes are 128+. Disable indirect descriptors to avoid
--
MST
^ permalink raw reply related
* [PATCH v9 31/37] virtio_balloon: skip zeroing for host-zeroed reported pages
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Implement VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (per virtio spec
proposal): when negotiated, the device initializes reported pages
(zeros, or poison_val if PAGE_POISON).
Check per-page used length returned by the device to determine
which reported pages were zeroed. If used_len matches the page
size, the device successfully initialized the page (e.g. via
MADV_DONTNEED), and we set the corresponding zeroed_bitmap bit.
Gate host_zeroes_pages on the feature bit and page content:
when PAGE_POISON is negotiated with non-zero poison_val, the
device fills with poison not zeros, so pages are not zeroed.
Clear the feature in validate() if REPORTING is not present
or if PAGE_POISON is active with non-zero poison_val.
See the virtio spec change:
https://github.com/oasis-tcs/virtio-spec/issues/244
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
drivers/virtio/virtio_balloon.c | 22 ++++++++++++++++++++--
include/uapi/linux/virtio_balloon.h | 1 +
2 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 1fa1c7fa285f..e3afa6f32ba5 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -207,6 +207,8 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_i
struct virtqueue *vq = vb->reporting_vq;
unsigned int i, err = 0;
+ bitmap_zero(pr_dev_info->zeroed_bitmap, nents);
+
/* We should always be able to add these buffers to an empty queue. */
for (i = 0; i < nents; i++) {
struct scatterlist one;
@@ -226,10 +228,14 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_i
/* When host has read buffer, this completes via balloon_ack */
for (i = 0; i < nents; i++) {
- unsigned int unused;
+ struct scatterlist *entry;
+ unsigned int used_len;
wait_event(vb->acked,
- virtqueue_get_buf(vq, &unused));
+ (entry = virtqueue_get_buf(vq, &used_len)));
+ if (used_len == entry->length)
+ set_bit(entry - sg,
+ pr_dev_info->zeroed_bitmap);
}
}
@@ -1051,6 +1057,9 @@ static int virtballoon_probe(struct virtio_device *vdev)
#endif
vb->pr_dev_info.capacity = capacity;
+ vb->pr_dev_info.host_zeroes_pages =
+ virtio_has_feature(vdev,
+ VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
err = page_reporting_register(&vb->pr_dev_info);
if (err)
goto out_unregister_oom;
@@ -1176,6 +1185,14 @@ static int virtballoon_validate(struct virtio_device *vdev)
else if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON))
__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_REPORTING);
+ if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_REPORTING))
+ __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
+
+ /* Device fills with poison_val, not zeros; disable zeroed hint */
+ if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON) &&
+ !want_init_on_free())
+ __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED);
+
/*
* Balloon submits 1-2 sg entries max per buffer, virtqueue
* sizes are 128+. Disable indirect descriptors to avoid
@@ -1194,6 +1211,7 @@ static unsigned int features[] = {
VIRTIO_BALLOON_F_FREE_PAGE_HINT,
VIRTIO_BALLOON_F_PAGE_POISON,
VIRTIO_BALLOON_F_REPORTING,
+ VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED,
};
static struct virtio_driver virtio_balloon_driver = {
diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h
index ee35a372805d..13074631f300 100644
--- a/include/uapi/linux/virtio_balloon.h
+++ b/include/uapi/linux/virtio_balloon.h
@@ -37,6 +37,7 @@
#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3 /* VQ to report free pages */
#define VIRTIO_BALLOON_F_PAGE_POISON 4 /* Guest is using page poisoning */
#define VIRTIO_BALLOON_F_REPORTING 5 /* Page reporting virtqueue */
+#define VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED 6 /* Device initializes reported pages */
/* Size of a PFN in the balloon interface. */
#define VIRTIO_BALLOON_PFN_SHIFT 12
--
MST
^ permalink raw reply related
* [PATCH v9 30/37] mm: page_alloc: propagate PG_zeroed in split_large_buddy
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
When splitting a large buddy page, propagate the PG_zeroed flag
to each sub-page before freeing it. __free_pages_prepare clears
all flags (including PG_zeroed), so the flag must be re-set on
each fragment after the split. This ensures that the buddy merge
logic can see PG_zeroed on pages that were part of a larger
zeroed block.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 72e52f049cf0..702fa2ced1e1 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1524,6 +1524,7 @@ static void split_large_buddy(struct zone *zone, struct page *page,
{
unsigned long end = pfn + (1 << order);
bool reported = PageReported(page);
+ bool zeroed = PageZeroed(page);
VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
/* Caller removed page from freelist, buddy info cleared! */
@@ -1537,6 +1538,8 @@ static void split_large_buddy(struct zone *zone, struct page *page,
if (reported)
__SetPageReported(page);
+ if (zeroed)
+ __SetPageZeroed(page);
__free_one_page(page, pfn, zone, order, mt, fpi);
pfn += 1 << order;
if (pfn == end)
--
MST
^ permalink raw reply related
* [PATCH v9 29/37] mm: page_reporting: add flush parameter with page budget
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Add a write-only module parameter 'flush' that triggers immediate
page reporting. The value specifies a page budget: at least
this many pages (at page_reporting_order) will be reported,
or all unreported pages if fewer remain. The actual number
reported may exceed the budget since each reporting pass
processes a full cycle across all zones.
This is helpful when there is a lot of memory freed quickly,
and a single cycle may not process all free pages due to
internal budget limits.
echo 512 > /sys/module/page_reporting/parameters/flush
Note: the set callback runs under kernel param_lock,
so writing this parameter blocks other built-in parameter
writes until the flush loop completes. This is acceptable
for a privileged debug/test parameter.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
mm/page_reporting.c | 54 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+)
diff --git a/mm/page_reporting.c b/mm/page_reporting.c
index f68c79e57427..3f584f538c68 100644
--- a/mm/page_reporting.c
+++ b/mm/page_reporting.c
@@ -358,6 +358,60 @@ static void page_reporting_process(struct work_struct *work)
static DEFINE_MUTEX(page_reporting_mutex);
DEFINE_STATIC_KEY_FALSE(page_reporting_enabled);
+static int page_reporting_flush_set(const char *val,
+ const struct kernel_param *kp)
+{
+ struct page_reporting_dev_info *prdev;
+ unsigned int budget;
+ int err;
+
+ err = kstrtouint(val, 0, &budget);
+ if (err)
+ return err;
+ if (!budget)
+ return 0;
+
+ mutex_lock(&page_reporting_mutex);
+ prdev = rcu_dereference_protected(pr_dev_info,
+ lockdep_is_held(&page_reporting_mutex));
+ if (prdev) {
+ unsigned int reported;
+ bool interrupted = false;
+
+ for (reported = 0; reported < budget;
+ reported += min(prdev->capacity, budget - reported)) {
+ /*
+ * First flush completes any previously scheduled
+ * reporting work. Then request a new reporting
+ * cycle and flush again to execute it.
+ */
+ flush_delayed_work(&prdev->work);
+ __page_reporting_request(prdev);
+ flush_delayed_work(&prdev->work);
+ if (atomic_read(&prdev->state) == PAGE_REPORTING_IDLE)
+ break;
+ if (signal_pending(current)) {
+ interrupted = true;
+ break;
+ }
+ }
+ if (interrupted) {
+ mutex_unlock(&page_reporting_mutex);
+ return -EINTR;
+ }
+ }
+ mutex_unlock(&page_reporting_mutex);
+ return 0;
+}
+
+static const struct kernel_param_ops flush_ops = {
+ .set = page_reporting_flush_set,
+ .get = param_get_uint,
+};
+static unsigned int page_reporting_flush;
+module_param_cb(flush, &flush_ops, &page_reporting_flush, 0200);
+MODULE_PARM_DESC(flush, "Report at least N pages at page_reporting_order, or until all reported");
+
int page_reporting_register(struct page_reporting_dev_info *prdev)
{
int err = 0;
--
MST
^ permalink raw reply related
* [PATCH v9 28/37] virtio_balloon: disable indirect descriptors
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
A follow-up patch (DEVICE_INIT_ON_INFLATE) adds an inbuf
(bitmap) alongside the outbuf (pfns), making total_sg=2.
With INDIRECT_DESC negotiated, virtqueue_use_indirect()
triggers for total_sg > 1. Balloon never needs more than
2 entries and virtqueue sizes are 128+, so indirect
descriptors are unnecessary.
Disabling them avoids a GFP_KERNEL allocation inside
virtqueue_add_sgs when VIRTIO_RING_F_INDIRECT_DESC is negotiated.
This allocation could trigger OOM reclaim while balloon_lock is
held, leading to a deadlock: OOM notifier -> leak_balloon ->
balloon_lock.
With single-buffer submissions (previous patch) and no indirect
descriptors, virtqueue_add never allocates memory.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
drivers/virtio/virtio_balloon.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 53b4a3984e7d..1fa1c7fa285f 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -7,6 +7,7 @@
*/
#include <linux/virtio.h>
+#include <uapi/linux/virtio_ring.h>
#include <linux/virtio_balloon.h>
#include <linux/swap.h>
#include <linux/workqueue.h>
@@ -1175,6 +1176,13 @@ static int virtballoon_validate(struct virtio_device *vdev)
else if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON))
__virtio_clear_bit(vdev, VIRTIO_BALLOON_F_REPORTING);
+ /*
+ * Balloon submits 1-2 sg entries max per buffer, virtqueue
+ * sizes are 128+. Disable indirect descriptors to avoid
+ * GFP_KERNEL allocation in virtqueue_add under balloon_lock,
+ * which could deadlock via OOM -> oom_notify -> leak_balloon.
+ */
+ __virtio_clear_bit(vdev, VIRTIO_RING_F_INDIRECT_DESC);
__virtio_clear_bit(vdev, VIRTIO_F_ACCESS_PLATFORM);
return 0;
}
--
MST
^ permalink raw reply related
* [PATCH v9 25/37] mm: page_alloc: clear PG_zeroed on buddy merge if not both zero
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
When two buddy pages merge in __free_one_page(), preserve
PG_zeroed on the merged page only if both buddies have the
flag set. Otherwise clear it.
The merged page would inherit PG_zeroed, and a later __GFP_ZERO
allocation would skip zeroing stale data in the non-zero half.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
include/linux/page-flags.h | 1 +
mm/page_alloc.c | 15 ++++++++++++++-
2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 4ee64134acc3..ff0b192b38e5 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -680,6 +680,7 @@ FOLIO_FLAG_FALSE(idle)
* uses this to skip redundant zeroing in post_alloc_hook().
*/
__PAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
+CLEARPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
#define __PG_ZEROED (1UL << PG_zeroed)
/*
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4cb7e779a6c5..ff614e422eec 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -940,10 +940,14 @@ static inline void __free_one_page(struct page *page,
unsigned long buddy_pfn = 0;
unsigned long combined_pfn;
struct page *buddy;
+ bool buddy_zeroed;
+ bool page_zeroed;
bool to_tail;
VM_BUG_ON(!zone_is_initialized(zone));
- VM_BUG_ON_PAGE(page->flags.f & PAGE_FLAGS_CHECK_AT_PREP, page);
+ /* PG_zeroed (aliased to PG_private) is valid on free-list pages */
+ VM_BUG_ON_PAGE(page->flags.f &
+ (PAGE_FLAGS_CHECK_AT_PREP & ~__PG_ZEROED), page);
VM_BUG_ON(migratetype == -1);
VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page);
@@ -978,6 +982,8 @@ static inline void __free_one_page(struct page *page,
goto done_merging;
}
+ buddy_zeroed = PageZeroed(buddy);
+
/*
* Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page,
* merge with it and move up one order.
@@ -996,10 +1002,17 @@ static inline void __free_one_page(struct page *page,
change_pageblock_range(buddy, order, migratetype);
}
+ page_zeroed = PageZeroed(page);
+ __ClearPageZeroed(page);
+ __ClearPageZeroed(buddy);
+
combined_pfn = buddy_pfn & pfn;
page = page + (combined_pfn - pfn);
pfn = combined_pfn;
order++;
+
+ if (page_zeroed && buddy_zeroed)
+ __SetPageZeroed(page);
}
done_merging:
--
MST
^ permalink raw reply related
* [PATCH v9 27/37] virtio_balloon: submit reported pages as individual buffers
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Submit each reported page as a separate virtqueue buffer instead
of one buffer with an sg list of all pages. This avoids indirect
descriptor allocation (kmalloc in the reporting path) and gives
per-page used length feedback from the device.
On error, the already-queued pages are kicked and drained
before the error is returned. The caller (page_reporting_drain)
then marks the batch as unreported, which is conservative
but correct.
Note: if the virtqueue is broken, wait_event on
virtqueue_get_buf hangs. This is a pre-existing issue:
the old single-buffer code had the same hang. EVENT_IDX
is not a concern: callbacks were never disabled, so the
virtqueue manages used_event automatically.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
drivers/virtio/virtio_balloon.c | 40 +++++++++++++++++++++------------
1 file changed, 26 insertions(+), 14 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 6a1a610c2cb1..53b4a3984e7d 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -187,7 +187,9 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
sg_init_one(&sg, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
- /* We should always be able to add one buffer to an empty queue. */
+ /* We made sure the vq is large enough so we should always
+ * be able to add one buffer to an empty queue.
+ */
virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
virtqueue_kick(vq);
@@ -202,25 +204,35 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_i
struct virtio_balloon *vb =
container_of(pr_dev_info, struct virtio_balloon, pr_dev_info);
struct virtqueue *vq = vb->reporting_vq;
- unsigned int unused, err;
+ unsigned int i, err = 0;
/* We should always be able to add these buffers to an empty queue. */
- err = virtqueue_add_inbuf(vq, sg, nents, vb, GFP_NOWAIT);
+ for (i = 0; i < nents; i++) {
+ struct scatterlist one;
- /*
- * In the extremely unlikely case that something has occurred and we
- * are able to trigger an error we will simply display a warning
- * and exit without actually processing the pages.
- */
- if (WARN_ON_ONCE(err))
- return err;
+ sg_init_table(&one, 1);
+ sg_set_page(&one, sg_page(&sg[i]), sg[i].length,
+ sg[i].offset);
+ err = virtqueue_add_inbuf(vq, &one, 1, &sg[i], GFP_NOWAIT);
+ if (WARN_ON_ONCE(err)) {
+ nents = i;
+ break;
+ }
+ }
- virtqueue_kick(vq);
+ if (nents) {
+ virtqueue_kick(vq);
- /* When host has read buffer, this completes via balloon_ack */
- wait_event(vb->acked, virtqueue_get_buf(vq, &unused));
+ /* When host has read buffer, this completes via balloon_ack */
+ for (i = 0; i < nents; i++) {
+ unsigned int unused;
- return 0;
+ wait_event(vb->acked,
+ virtqueue_get_buf(vq, &unused));
+ }
+ }
+
+ return err;
}
static void set_page_pfns(struct virtio_balloon *vb,
--
MST
^ permalink raw reply related
* [PATCH v9 26/37] mm: page_alloc: preserve PG_zeroed in page_del_and_expand
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Propagate PG_zeroed through buddy splits in page_del_and_expand()
and try_to_claim_block(). When a zeroed high-order page is split
to satisfy a smaller allocation, the sub-pages placed back on the
free lists keep PG_zeroed.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
mm/page_alloc.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ff614e422eec..72e52f049cf0 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1715,7 +1715,8 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,
* -- nyc
*/
static inline unsigned int expand(struct zone *zone, struct page *page, int low,
- int high, int migratetype, bool reported)
+ int high, int migratetype, bool reported,
+ bool zeroed)
{
unsigned int size = 1 << high;
unsigned int nr_added = 0;
@@ -1746,6 +1747,8 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,
*/
if (reported)
__SetPageReported(&page[size]);
+ if (zeroed)
+ __SetPageZeroed(&page[size]);
}
return nr_added;
@@ -1757,10 +1760,12 @@ static __always_inline void page_del_and_expand(struct zone *zone,
{
int nr_pages = 1 << high;
bool was_reported = page_reported(page);
+ bool was_zeroed = PageZeroed(page);
__del_page_from_free_list(page, zone, high, migratetype);
- nr_pages -= expand(zone, page, low, high, migratetype, was_reported);
+ nr_pages -= expand(zone, page, low, high, migratetype, was_reported,
+ was_zeroed);
account_freepages(zone, -nr_pages, migratetype);
}
@@ -2356,11 +2361,12 @@ try_to_claim_block(struct zone *zone, struct page *page,
if (current_order >= pageblock_order) {
unsigned int nr_added;
bool was_reported = page_reported(page);
+ bool was_zeroed = PageZeroed(page);
del_page_from_free_list(page, zone, current_order, block_type);
change_pageblock_range(page, current_order, start_type);
nr_added = expand(zone, page, order, current_order, start_type,
- was_reported);
+ was_reported, was_zeroed);
account_freepages(zone, nr_added, start_type);
return page;
}
--
MST
^ permalink raw reply related
* [PATCH v9 24/37] mm: page_reporting: add per-page zeroed bitmap for host feedback
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
The host may skip zeroing some reported pages (e.g., due to alignment
constraints or bounce buffer fallback in QEMU). Currently, when
host_zeroes_pages is set, all reported pages are unconditionally
marked PG_zeroed - even ones the host did not actually zero.
Add a zeroed_bitmap to page_reporting_dev_info that the report()
callback can use to indicate which pages were actually zeroed.
The driver's report() callback is responsible for managing the
bitmap: zeroing it before sending pages to the host, then setting
bits for pages the host actually zeroed.
page_reporting_drain() checks the bitmap per-page in addition to the
global host_zeroes_pages flag.
No driver sets host_zeroes_pages yet, so the static key is
off and the bitmap is never read. Behavior is unchanged.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
include/linux/page_reporting.h | 7 +++++++
mm/page_reporting.c | 8 ++++++--
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/include/linux/page_reporting.h b/include/linux/page_reporting.h
index c331c6b36687..e2e6a487ddab 100644
--- a/include/linux/page_reporting.h
+++ b/include/linux/page_reporting.h
@@ -17,6 +17,13 @@ struct page_reporting_dev_info {
/* If true, host zeros reported pages on reclaim */
bool host_zeroes_pages;
+ /*
+ * Per-page zeroed status, indexed by scatterlist position.
+ * The driver's report() callback must clear the bitmap,
+ * then set bits for pages that were actually zeroed.
+ */
+ DECLARE_BITMAP(zeroed_bitmap, PAGE_REPORTING_CAPACITY);
+
/* work struct for processing reports */
struct delayed_work work;
diff --git a/mm/page_reporting.c b/mm/page_reporting.c
index 8d52718be876..f68c79e57427 100644
--- a/mm/page_reporting.c
+++ b/mm/page_reporting.c
@@ -108,6 +108,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,
struct scatterlist *sgl, unsigned int nents, bool reported)
{
struct scatterlist *sg = sgl;
+ unsigned int i = 0;
/*
* Drain the now reported pages back into their respective
@@ -122,7 +123,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,
/* If the pages were not reported due to error skip flagging */
if (!reported)
- continue;
+ goto next;
/*
* If page was not commingled with another page we can
@@ -133,9 +134,12 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,
*/
if (PageBuddy(page) && buddy_order(page) == order) {
__SetPageReported(page);
- if (page_reporting_host_zeroes_pages())
+ if (page_reporting_host_zeroes_pages() &&
+ test_bit(i, prdev->zeroed_bitmap))
__SetPageZeroed(page);
}
+next:
+ i++;
} while ((sg = sg_next(sg)));
/* reinitialize scatterlist now that it is empty */
--
MST
^ permalink raw reply related
* [PATCH v9 23/37] mm: page_alloc: use aliasing checks instead of user_alloc_needs_zeroing
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Replace user_alloc_needs_zeroing() with the direct aliasing checks
(cpu_dcache_is_aliasing() || cpu_icache_is_aliasing()) in the
post_alloc_hook aliasing guard.
user_alloc_needs_zeroing() includes a !init_on_alloc term that
means "allocator didn't zero this page." But in this guard's
context (!zeroed && !init && __GFP_ZERO), we already know the page
is zero; init incorporates init_on_alloc via want_init_on_alloc().
The only question left is whether the cache architecture needs
the data re-zeroed through a congruent mapping, which is purely
cpu_dcache_is_aliasing() || cpu_icache_is_aliasing().
On non-aliasing architectures with init_on_free=true and
init_on_alloc=false, this avoids a redundant re-zero of an
already-zero page.
Note on PowerPC: PowerPC overrides clear_user_page to call
flush_dcache_page after clear_page, but on freshly allocated
pages PG_dcache_clean is already clear (cleared by
__free_pages_prepare), so flush_dcache_page is a no-op.
Skipping this here thus has no effect.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
mm/page_alloc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5158f7e23d18..4cb7e779a6c5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1883,7 +1883,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
*/
if (!zeroed && !init && (gfp_flags & __GFP_ZERO) &&
user_addr != USER_ADDR_NONE &&
- user_alloc_needs_zeroing())
+ (cpu_dcache_is_aliasing() || cpu_icache_is_aliasing()))
init = true;
/*
* If memory is still not initialized, initialize it now.
--
MST
^ permalink raw reply related
* [PATCH v9 22/37] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
When a guest reports free pages to the hypervisor via the page reporting
framework (used by virtio-balloon and hv_balloon), the host typically
zeros those pages when reclaiming their backing memory. However, when
those pages are later allocated in the guest, post_alloc_hook()
unconditionally zeros them again if __GFP_ZERO is set. This
double-zeroing is wasteful, especially for large pages.
Avoid redundant zeroing:
- Add a host_zeroes_pages flag to page_reporting_dev_info, allowing
drivers to declare that their host zeros reported pages on reclaim.
A static key (page_reporting_host_zeroes) gates the fast path.
- Add PG_zeroed page flag (sharing PG_private bit) to mark pages
that have been zeroed by the host. Set it in
page_reporting_drain() after the host reports them.
- Thread the zeroed bool through rmqueue -> prep_new_page ->
post_alloc_hook, where it skips redundant zeroing for __GFP_ZERO
allocations.
Currently the PG_zeroed hint can be lost when pages are
split (expand) or merged in the buddy allocator. This is
harmless: losing the hint just means the page gets re-zeroed,
which is correct but suboptimal. Follow-up patches propagate
PG_zeroed across splits and merges to preserve the hint on
common paths.
No driver sets host_zeroes_pages yet; a follow-up patch to
virtio_balloon is needed to opt in.
PG_zeroed pages may pass through PCP lists before being freed.
This is safe: __free_pages_prepare clears all
PAGE_FLAGS_CHECK_AT_PREP flags (including PG_zeroed/PG_private)
before the page re-enters the buddy allocator.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
include/linux/page-flags.h | 9 +++++
include/linux/page_reporting.h | 3 ++
mm/compaction.c | 6 ++-
mm/internal.h | 2 +-
mm/page_alloc.c | 68 +++++++++++++++++++++++-----------
mm/page_reporting.c | 14 ++++++-
mm/page_reporting.h | 12 ++++++
7 files changed, 88 insertions(+), 26 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 0e03d816e8b9..4ee64134acc3 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -135,6 +135,8 @@ enum pageflags {
PG_swapcache = PG_owner_priv_1, /* Swap page: swp_entry_t in private */
/* Some filesystems */
PG_checked = PG_owner_priv_1,
+ /* Page contents are known to be zero */
+ PG_zeroed = PG_private,
/*
* Depending on the way an anonymous folio can be mapped into a page
@@ -673,6 +675,13 @@ FOLIO_TEST_CLEAR_FLAG_FALSE(young)
FOLIO_FLAG_FALSE(idle)
#endif
+/*
+ * PageZeroed() tracks pages known to be zero. The allocator
+ * uses this to skip redundant zeroing in post_alloc_hook().
+ */
+__PAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)
+#define __PG_ZEROED (1UL << PG_zeroed)
+
/*
* PageReported() is used to track reported free pages within the Buddy
* allocator. We can use the non-atomic version of the test and set
diff --git a/include/linux/page_reporting.h b/include/linux/page_reporting.h
index 5ab5be02fa15..c331c6b36687 100644
--- a/include/linux/page_reporting.h
+++ b/include/linux/page_reporting.h
@@ -14,6 +14,9 @@ struct page_reporting_dev_info {
int (*report)(struct page_reporting_dev_info *prdev,
struct scatterlist *sg, unsigned int nents);
+ /* If true, host zeros reported pages on reclaim */
+ bool host_zeroes_pages;
+
/* work struct for processing reports */
struct delayed_work work;
diff --git a/mm/compaction.c b/mm/compaction.c
index 4336e433c99b..8000fc5e0a2e 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -82,7 +82,8 @@ static inline bool is_via_compact_memory(int order) { return false; }
static struct page *mark_allocated_noprof(struct page *page, unsigned int order, gfp_t gfp_flags)
{
- post_alloc_hook(page, order, __GFP_MOVABLE, USER_ADDR_NONE);
+ __ClearPageZeroed(page);
+ post_alloc_hook(page, order, __GFP_MOVABLE, false, USER_ADDR_NONE);
set_page_refcounted(page);
return page;
}
@@ -1849,9 +1850,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da
set_page_private(&freepage[size], start_order);
}
dst = (struct folio *)freepage;
+ __ClearPageZeroed(&dst->page);
if (order)
prep_compound_page(&dst->page, order);
- post_alloc_hook(&dst->page, order, __GFP_MOVABLE, USER_ADDR_NONE);
+ post_alloc_hook(&dst->page, order, __GFP_MOVABLE, false, USER_ADDR_NONE);
set_page_refcounted(&dst->page);
cc->nr_freepages -= 1 << order;
cc->nr_migratepages -= 1 << order;
diff --git a/mm/internal.h b/mm/internal.h
index 389098200aa6..fd910743ddc3 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -928,7 +928,7 @@ static inline void init_compound_tail(struct page *tail,
}
void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags,
- unsigned long user_addr);
+ bool zeroed, unsigned long user_addr);
extern bool free_pages_prepare(struct page *page, unsigned int order);
extern int user_min_free_kbytes;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ad0655387e9d..5158f7e23d18 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1746,6 +1746,7 @@ static __always_inline void page_del_and_expand(struct zone *zone,
bool was_reported = page_reported(page);
__del_page_from_free_list(page, zone, high, migratetype);
+
nr_pages -= expand(zone, page, low, high, migratetype, was_reported);
account_freepages(zone, -nr_pages, migratetype);
}
@@ -1818,8 +1819,10 @@ static inline bool should_skip_init(gfp_t flags)
return (flags & __GFP_SKIP_ZERO);
}
+
inline void post_alloc_hook(struct page *page, unsigned int order,
- gfp_t gfp_flags, unsigned long user_addr)
+ gfp_t gfp_flags, bool zeroed,
+ unsigned long user_addr)
{
bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) &&
!should_skip_init(gfp_flags);
@@ -1828,6 +1831,14 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
set_page_private(page, 0);
+ /*
+ * If the page is zeroed, skip memory initialization.
+ * We still need to handle tag zeroing separately since the host
+ * does not know about memory tags.
+ */
+ if (zeroed && init && !zero_tags)
+ init = false;
+
arch_alloc_page(page, order);
debug_pagealloc_map_pages(page, 1 << order);
@@ -1870,7 +1881,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
* through a user-congruent mapping. Host-zeroed pages
* (zeroed flag) don't need this: physical RAM is clean.
*/
- if (!init && (gfp_flags & __GFP_ZERO) &&
+ if (!zeroed && !init && (gfp_flags & __GFP_ZERO) &&
user_addr != USER_ADDR_NONE &&
user_alloc_needs_zeroing())
init = true;
@@ -1903,13 +1914,13 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
}
static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
- unsigned int alloc_flags,
- unsigned long user_addr)
+ unsigned int alloc_flags, bool zeroed,
+ unsigned long user_addr)
{
if (order && (gfp_flags & __GFP_COMP))
prep_compound_page(page, order);
- post_alloc_hook(page, order, gfp_flags, user_addr);
+ post_alloc_hook(page, order, gfp_flags, zeroed, user_addr);
/*
* page is set pfmemalloc when ALLOC_NO_WATERMARKS was necessary to
@@ -3175,6 +3186,7 @@ int __isolate_free_page(struct page *page, unsigned int order)
}
del_page_from_free_list(page, zone, order, mt);
+ __ClearPageZeroed(page);
/*
* Set the pageblock if the isolated page is at least half of a
@@ -3247,7 +3259,7 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z,
static __always_inline
struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
unsigned int order, unsigned int alloc_flags,
- int migratetype)
+ int migratetype, bool *zeroed)
{
struct page *page;
unsigned long flags;
@@ -3282,6 +3294,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
}
}
spin_unlock_irqrestore(&zone->lock, flags);
+ *zeroed = PageZeroed(page);
+ __ClearPageZeroed(page);
} while (check_new_pages(page, order));
/*
@@ -3350,10 +3364,9 @@ static int nr_pcp_alloc(struct per_cpu_pages *pcp, struct zone *zone, int order)
/* Remove page from the per-cpu list, caller must protect the list */
static inline
struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,
- int migratetype,
- unsigned int alloc_flags,
+ int migratetype, unsigned int alloc_flags,
struct per_cpu_pages *pcp,
- struct list_head *list)
+ struct list_head *list, bool *zeroed)
{
struct page *page;
@@ -3388,6 +3401,8 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,
page = list_first_entry(list, struct page, pcp_list);
list_del(&page->pcp_list);
pcp->count -= 1 << order;
+ *zeroed = PageZeroed(page);
+ __ClearPageZeroed(page);
} while (check_new_pages(page, order));
return page;
@@ -3396,7 +3411,8 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,
/* Lock and remove page from the per-cpu list */
static struct page *rmqueue_pcplist(struct zone *preferred_zone,
struct zone *zone, unsigned int order,
- int migratetype, unsigned int alloc_flags)
+ int migratetype, unsigned int alloc_flags,
+ bool *zeroed)
{
struct per_cpu_pages *pcp;
struct list_head *list;
@@ -3414,7 +3430,8 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone,
*/
pcp->free_count >>= 1;
list = &pcp->lists[order_to_pindex(migratetype, order)];
- page = __rmqueue_pcplist(zone, order, migratetype, alloc_flags, pcp, list);
+ page = __rmqueue_pcplist(zone, order, migratetype, alloc_flags,
+ pcp, list, zeroed);
pcp_spin_unlock(pcp);
if (page) {
__count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order);
@@ -3439,19 +3456,19 @@ static inline
struct page *rmqueue(struct zone *preferred_zone,
struct zone *zone, unsigned int order,
gfp_t gfp_flags, unsigned int alloc_flags,
- int migratetype)
+ int migratetype, bool *zeroed)
{
struct page *page;
if (likely(pcp_allowed_order(order))) {
page = rmqueue_pcplist(preferred_zone, zone, order,
- migratetype, alloc_flags);
+ migratetype, alloc_flags, zeroed);
if (likely(page))
goto out;
}
page = rmqueue_buddy(preferred_zone, zone, order, alloc_flags,
- migratetype);
+ migratetype, zeroed);
out:
/* Separate test+clear to avoid unnecessary atomics */
@@ -3842,6 +3859,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
struct pglist_data *last_pgdat = NULL;
bool last_pgdat_dirty_ok = false;
bool no_fallback;
+ bool zeroed;
bool skip_kswapd_nodes = nr_online_nodes > 1;
bool skipped_kswapd_nodes = false;
@@ -3986,10 +4004,11 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
try_this_zone:
page = rmqueue(zonelist_zone(ac->preferred_zoneref), zone, order,
- gfp_mask, alloc_flags, ac->migratetype);
+ gfp_mask, alloc_flags, ac->migratetype,
+ &zeroed);
if (page) {
prep_new_page(page, order, gfp_mask, alloc_flags,
- ac->user_addr);
+ zeroed, ac->user_addr);
return page;
} else {
@@ -4216,9 +4235,11 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
count_vm_event(COMPACTSTALL);
/* Prep a captured page if available */
- if (page)
- prep_new_page(page, order, gfp_mask, alloc_flags,
+ if (page) {
+ __ClearPageZeroed(page);
+ prep_new_page(page, order, gfp_mask, alloc_flags, false,
ac->user_addr);
+ }
/* Try get a page from the freelist if available */
if (!page)
@@ -5191,6 +5212,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
/* Attempt the batch allocation */
pcp_list = &pcp->lists[order_to_pindex(ac.migratetype, 0)];
while (nr_populated < nr_pages) {
+ bool zeroed = false;
/* Skip existing pages */
if (page_array[nr_populated]) {
@@ -5199,7 +5221,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
}
page = __rmqueue_pcplist(zone, 0, ac.migratetype, alloc_flags,
- pcp, pcp_list);
+ pcp, pcp_list, &zeroed);
if (unlikely(!page)) {
/* Try and allocate at least one page */
if (!nr_account) {
@@ -5210,7 +5232,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
}
nr_account++;
- prep_new_page(page, 0, gfp, 0, USER_ADDR_NONE);
+ prep_new_page(page, 0, gfp, 0, zeroed, USER_ADDR_NONE);
set_page_refcounted(page);
page_array[nr_populated++] = page;
}
@@ -6950,7 +6972,8 @@ static void split_free_frozen_pages(struct list_head *list, gfp_t gfp_mask)
list_for_each_entry_safe(page, next, &list[order], lru) {
int i;
- post_alloc_hook(page, order, gfp_mask, USER_ADDR_NONE);
+ __ClearPageZeroed(page);
+ post_alloc_hook(page, order, gfp_mask, false, USER_ADDR_NONE);
if (!order)
continue;
@@ -7156,8 +7179,9 @@ static int __alloc_contig_frozen_range(unsigned long start, unsigned long end,
} else if (start == outer_start && end == outer_end && is_power_of_2(end - start)) {
struct page *head = pfn_to_page(start);
+ __ClearPageZeroed(head);
check_new_pages(head, order);
- prep_new_page(head, order, gfp_mask, 0, user_addr);
+ prep_new_page(head, order, gfp_mask, 0, false, user_addr);
} else {
ret = -EINVAL;
WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",
diff --git a/mm/page_reporting.c b/mm/page_reporting.c
index a6ddf6fafc20..8d52718be876 100644
--- a/mm/page_reporting.c
+++ b/mm/page_reporting.c
@@ -50,6 +50,8 @@ EXPORT_SYMBOL_GPL(page_reporting_order);
#define PAGE_REPORTING_DELAY (2 * HZ)
static struct page_reporting_dev_info __rcu *pr_dev_info __read_mostly;
+DEFINE_STATIC_KEY_FALSE(page_reporting_host_zeroes);
+
enum {
PAGE_REPORTING_IDLE = 0,
PAGE_REPORTING_REQUESTED,
@@ -129,8 +131,11 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,
* report on the new larger page when we make our way
* up to that higher order.
*/
- if (PageBuddy(page) && buddy_order(page) == order)
+ if (PageBuddy(page) && buddy_order(page) == order) {
__SetPageReported(page);
+ if (page_reporting_host_zeroes_pages())
+ __SetPageZeroed(page);
+ }
} while ((sg = sg_next(sg)));
/* reinitialize scatterlist now that it is empty */
@@ -390,6 +395,10 @@ int page_reporting_register(struct page_reporting_dev_info *prdev)
/* Assign device to allow notifications */
rcu_assign_pointer(pr_dev_info, prdev);
+ /* enable zeroed page optimization if host zeroes reported pages */
+ if (prdev->host_zeroes_pages)
+ static_branch_enable(&page_reporting_host_zeroes);
+
/* enable page reporting notification */
if (!static_key_enabled(&page_reporting_enabled)) {
static_branch_enable(&page_reporting_enabled);
@@ -414,6 +423,9 @@ void page_reporting_unregister(struct page_reporting_dev_info *prdev)
/* Flush any existing work, and lock it out */
cancel_delayed_work_sync(&prdev->work);
+
+ if (prdev->host_zeroes_pages)
+ static_branch_disable(&page_reporting_host_zeroes);
}
mutex_unlock(&page_reporting_mutex);
diff --git a/mm/page_reporting.h b/mm/page_reporting.h
index c51dbc228b94..736ea7b37e9e 100644
--- a/mm/page_reporting.h
+++ b/mm/page_reporting.h
@@ -15,6 +15,13 @@ DECLARE_STATIC_KEY_FALSE(page_reporting_enabled);
extern unsigned int page_reporting_order;
void __page_reporting_notify(void);
+DECLARE_STATIC_KEY_FALSE(page_reporting_host_zeroes);
+
+static inline bool page_reporting_host_zeroes_pages(void)
+{
+ return static_branch_unlikely(&page_reporting_host_zeroes);
+}
+
static inline bool page_reported(struct page *page)
{
return static_branch_unlikely(&page_reporting_enabled) &&
@@ -46,6 +53,11 @@ static inline void page_reporting_notify_free(unsigned int order)
#else /* CONFIG_PAGE_REPORTING */
#define page_reported(_page) false
+static inline bool page_reporting_host_zeroes_pages(void)
+{
+ return false;
+}
+
static inline void page_reporting_notify_free(unsigned int order)
{
}
--
MST
^ permalink raw reply related
* [PATCH v9 21/37] mm: memfd: skip zeroing for zeroed hugetlb pool pages
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Add bool *zeroed output to alloc_hugetlb_folio_reserve() so
callers can check whether the pool page is known-zero. memfd's
memfd_alloc_folio() uses this to skip the explicit folio_zero_user()
when the page is already zero.
This avoids redundant zeroing for memfd hugetlb pages that were
pre-allocated into the pool and never mapped to userspace.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
---
include/linux/cma.h | 3 ++-
include/linux/hugetlb.h | 6 ++++--
mm/cma.c | 6 ++++--
mm/hugetlb.c | 11 +++++++++--
mm/hugetlb_cma.c | 4 ++--
mm/memfd.c | 14 ++++++++------
6 files changed, 29 insertions(+), 15 deletions(-)
diff --git a/include/linux/cma.h b/include/linux/cma.h
index 8555d38a97b1..dee88909cf5d 100644
--- a/include/linux/cma.h
+++ b/include/linux/cma.h
@@ -53,7 +53,8 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long
struct page *cma_alloc_frozen(struct cma *cma, unsigned long count,
unsigned int align, bool no_warn);
-struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order);
+struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order,
+ gfp_t caller_gfp);
bool cma_release_frozen(struct cma *cma, const struct page *pages,
unsigned long count);
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 49e5557d6cc0..3d23267014e6 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -714,7 +714,8 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
nodemask_t *nmask, gfp_t gfp_mask,
bool allow_alloc_fallback);
struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
- nodemask_t *nmask, gfp_t gfp_mask);
+ nodemask_t *nmask, gfp_t gfp_mask,
+ bool *zeroed);
int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping,
pgoff_t idx);
@@ -1134,7 +1135,8 @@ static inline void wait_for_freed_hugetlb_folios(void)
static inline struct folio *
alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
- nodemask_t *nmask, gfp_t gfp_mask)
+ nodemask_t *nmask, gfp_t gfp_mask,
+ bool *zeroed)
{
return NULL;
}
diff --git a/mm/cma.c b/mm/cma.c
index c7ca567f4c5c..27971f6264ab 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -924,9 +924,11 @@ struct page *cma_alloc_frozen(struct cma *cma, unsigned long count,
return __cma_alloc_frozen(cma, count, align, gfp);
}
-struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order)
+struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order,
+ gfp_t caller_gfp)
{
- gfp_t gfp = GFP_KERNEL | __GFP_COMP | __GFP_NOWARN;
+ gfp_t gfp = GFP_KERNEL | __GFP_COMP | __GFP_NOWARN |
+ (caller_gfp & __GFP_ZERO);
return __cma_alloc_frozen(cma, 1 << order, order, gfp);
}
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4ccf4eb91d92..649972101ace 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2208,7 +2208,7 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,
}
struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
- nodemask_t *nmask, gfp_t gfp_mask)
+ nodemask_t *nmask, gfp_t gfp_mask, bool *zeroed)
{
struct folio *folio;
@@ -2224,6 +2224,12 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
h->resv_huge_pages--;
spin_unlock_irq(&hugetlb_lock);
+
+ if (zeroed && folio) {
+ *zeroed = folio_test_hugetlb_zeroed(folio);
+ folio_clear_hugetlb_zeroed(folio);
+ }
+
return folio;
}
@@ -2308,7 +2314,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)
* It is okay to use NUMA_NO_NODE because we use numa_mem_id()
* down the road to pick the current node if that is the case.
*/
- folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h),
+ folio = alloc_surplus_hugetlb_folio(h,
+ htlb_alloc_mask(h),
NUMA_NO_NODE, &alloc_nodemask,
USER_ADDR_NONE);
if (!folio) {
diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index 7693ccefd0c6..c9266b25be3d 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -35,14 +35,14 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
return NULL;
if (hugetlb_cma[nid])
- page = cma_alloc_frozen_compound(hugetlb_cma[nid], order);
+ page = cma_alloc_frozen_compound(hugetlb_cma[nid], order, gfp_mask);
if (!page && !(gfp_mask & __GFP_THISNODE)) {
for_each_node_mask(node, *nodemask) {
if (node == nid || !hugetlb_cma[node])
continue;
- page = cma_alloc_frozen_compound(hugetlb_cma[node], order);
+ page = cma_alloc_frozen_compound(hugetlb_cma[node], order, gfp_mask);
if (page)
break;
}
diff --git a/mm/memfd.c b/mm/memfd.c
index fb425f4e315f..5518f7d2d91f 100644
--- a/mm/memfd.c
+++ b/mm/memfd.c
@@ -69,6 +69,7 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
#ifdef CONFIG_HUGETLB_PAGE
struct folio *folio;
gfp_t gfp_mask;
+ bool zeroed;
if (is_file_hugepages(memfd)) {
/*
@@ -93,17 +94,18 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
folio = alloc_hugetlb_folio_reserve(h,
numa_node_id(),
NULL,
- gfp_mask);
+ gfp_mask,
+ &zeroed);
if (folio) {
u32 hash;
/*
- * Zero the folio to prevent information leaks to userspace.
- * Use folio_zero_user() which is optimized for huge/gigantic
- * pages. Pass 0 as addr_hint since this is not a faulting path
- * and we don't have a user virtual address yet.
+ * Zero the folio to prevent information leaks to
+ * userspace. Skip if the pool page is known-zero
+ * (HPG_zeroed set during pool pre-allocation).
*/
- folio_zero_user(folio, 0);
+ if (!zeroed)
+ folio_zero_user(folio, 0);
/*
* Mark the folio uptodate before adding to page cache,
--
MST
^ permalink raw reply related
* [PATCH v9 20/37] mm: hugetlb: add gfp parameter and skip zeroing for zeroed pages
From: Michael S. Tsirkin @ 2026-05-29 15:24 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Add a gfp_t parameter to alloc_hugetlb_folio(). When __GFP_ZERO
is set, the function guarantees the returned folio is zeroed:
- Fresh allocations (buddy or gigantic): zeroed by
post_alloc_hook via __GFP_ZERO, HPG_zeroed set by
alloc_surplus_hugetlb_folio.
- Pool pages with HPG_zeroed set: already zeroed, skip.
- Pool pages without HPG_zeroed: zeroed via folio_zero_user().
The address parameter is renamed to user_addr; the function
aligns it internally for reservation and NUMA policy lookups.
For pages that need zeroing, user_addr is passed to
folio_zero_user() for cache-friendly zeroing near the faulting
subpage. All callers pass a page-aligned address; the
hugetlb_no_page caller passes vmf->real_address & PAGE_MASK
for consistency.
HPG_zeroed (stored in hugetlb folio->private bits) tracks
known-zero pool pages. It is set when alloc_surplus_hugetlb_folio
allocates with __GFP_ZERO, and cleared in free_huge_folio when
the page returns to the pool after userspace use.
Suggested-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Gregory Price <gourry@gourry.net>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
fs/hugetlbfs/inode.c | 3 +--
include/linux/hugetlb.h | 5 ++++-
mm/hugetlb.c | 42 +++++++++++++++++++++++++++++------------
3 files changed, 35 insertions(+), 15 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 8b05bec08e04..5856a3530c7b 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -810,13 +810,12 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
* folios in these areas, we need to consume the reserves
* to keep reservation accounting consistent.
*/
- folio = alloc_hugetlb_folio(&pseudo_vma, addr, false);
+ folio = alloc_hugetlb_folio(&pseudo_vma, addr, false, __GFP_ZERO);
if (IS_ERR(folio)) {
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
error = PTR_ERR(folio);
goto out;
}
- folio_zero_user(folio, addr);
__folio_mark_uptodate(folio);
error = hugetlb_add_to_page_cache(folio, mapping, index);
if (unlikely(error)) {
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index f016bc2e8936..49e5557d6cc0 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -599,6 +599,7 @@ enum hugetlb_page_flags {
HPG_vmemmap_optimized,
HPG_raw_hwp_unreliable,
HPG_cma,
+ HPG_zeroed,
__NR_HPAGEFLAGS,
};
@@ -659,6 +660,7 @@ HPAGEFLAG(Freed, freed)
HPAGEFLAG(VmemmapOptimized, vmemmap_optimized)
HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable)
HPAGEFLAG(Cma, cma)
+HPAGEFLAG(Zeroed, zeroed)
#ifdef CONFIG_HUGETLB_PAGE
@@ -706,7 +708,8 @@ int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list);
int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn);
void wait_for_freed_hugetlb_folios(void);
struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
- unsigned long addr, bool cow_from_owner);
+ unsigned long user_addr, bool cow_from_owner,
+ gfp_t gfp);
struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
nodemask_t *nmask, gfp_t gfp_mask,
bool allow_alloc_fallback);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3a6afbe99116..4ccf4eb91d92 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1712,6 +1712,9 @@ void free_huge_folio(struct folio *folio)
bool restore_reserve;
unsigned long flags;
+ /* Page was mapped to userspace; no longer known-zero */
+ folio_clear_hugetlb_zeroed(folio);
+
VM_BUG_ON_FOLIO(folio_ref_count(folio), folio);
VM_BUG_ON_FOLIO(folio_mapcount(folio), folio);
@@ -2113,6 +2116,10 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
if (!folio)
return NULL;
+ /* Mark as known-zero only if __GFP_ZERO was requested */
+ if (gfp_mask & __GFP_ZERO)
+ folio_set_hugetlb_zeroed(folio);
+
spin_lock_irq(&hugetlb_lock);
/*
* nr_huge_pages needs to be adjusted within the same lock cycle
@@ -2176,11 +2183,11 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas
*/
static
struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,
- struct vm_area_struct *vma, unsigned long addr)
+ struct vm_area_struct *vma, unsigned long addr, gfp_t gfp)
{
struct folio *folio = NULL;
struct mempolicy *mpol;
- gfp_t gfp_mask = htlb_alloc_mask(h);
+ gfp_t gfp_mask = htlb_alloc_mask(h) | gfp;
int nid;
nodemask_t *nodemask;
@@ -2877,16 +2884,19 @@ typedef enum {
* When it's set, the allocation will bypass all vma level reservations.
*/
struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
- unsigned long addr, bool cow_from_owner)
+ unsigned long user_addr, bool cow_from_owner,
+ gfp_t gfp)
{
struct hugepage_subpool *spool = subpool_vma(vma);
struct hstate *h = hstate_vma(vma);
+ unsigned long addr = user_addr & huge_page_mask(h);
struct folio *folio;
long retval, gbl_chg, gbl_reserve;
map_chg_state map_chg;
int ret, idx;
struct hugetlb_cgroup *h_cg = NULL;
- gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
+
+ gfp |= htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
idx = hstate_index(h);
@@ -2954,13 +2964,12 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
folio = dequeue_hugetlb_folio_vma(h, vma, addr, gbl_chg);
if (!folio) {
spin_unlock_irq(&hugetlb_lock);
- folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, addr);
+ folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, user_addr, gfp);
if (!folio)
goto out_uncharge_cgroup;
spin_lock_irq(&hugetlb_lock);
list_add(&folio->lru, &h->hugepage_activelist);
folio_ref_unfreeze(folio, 1);
- /* Fall through */
}
/*
@@ -2983,6 +2992,10 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
spin_unlock_irq(&hugetlb_lock);
+ if ((gfp & __GFP_ZERO) && !folio_test_hugetlb_zeroed(folio))
+ folio_zero_user(folio, user_addr);
+ folio_clear_hugetlb_zeroed(folio);
+
hugetlb_set_folio_subpool(folio, spool);
if (map_chg != MAP_CHG_ENFORCED) {
@@ -4991,7 +5004,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
spin_unlock(src_ptl);
spin_unlock(dst_ptl);
/* Do not use reserve as it's private owned */
- new_folio = alloc_hugetlb_folio(dst_vma, addr, false);
+ new_folio = alloc_hugetlb_folio(dst_vma, addr, false, 0);
if (IS_ERR(new_folio)) {
folio_put(pte_folio);
ret = PTR_ERR(new_folio);
@@ -5520,7 +5533,7 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)
* be acquired again before returning to the caller, as expected.
*/
spin_unlock(vmf->ptl);
- new_folio = alloc_hugetlb_folio(vma, vmf->address, cow_from_owner);
+ new_folio = alloc_hugetlb_folio(vma, vmf->address, cow_from_owner, 0);
if (IS_ERR(new_folio)) {
/*
@@ -5780,7 +5793,13 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
goto out;
}
- folio = alloc_hugetlb_folio(vma, vmf->address, false);
+ /*
+ * Passing vmf->real_address would work just as well,
+ * but PAGE_MASK helps make sure we never pass
+ * USER_ADDR_NONE by mistake.
+ */
+ folio = alloc_hugetlb_folio(vma, vmf->real_address & PAGE_MASK,
+ false, __GFP_ZERO);
if (IS_ERR(folio)) {
/*
* Returning error will result in faulting task being
@@ -5800,7 +5819,6 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
ret = 0;
goto out;
}
- folio_zero_user(folio, vmf->real_address);
__folio_mark_uptodate(folio);
new_folio = true;
@@ -6239,7 +6257,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
goto out;
}
- folio = alloc_hugetlb_folio(dst_vma, dst_addr, false);
+ folio = alloc_hugetlb_folio(dst_vma, dst_addr, false, 0);
if (IS_ERR(folio)) {
pte_t *actual_pte = hugetlb_walk(dst_vma, dst_addr, PMD_SIZE);
if (actual_pte) {
@@ -6286,7 +6304,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
goto out;
}
- folio = alloc_hugetlb_folio(dst_vma, dst_addr, false);
+ folio = alloc_hugetlb_folio(dst_vma, dst_addr, false, 0);
if (IS_ERR(folio)) {
folio_put(*foliop);
ret = -ENOMEM;
--
MST
^ permalink raw reply related
* [PATCH v9 19/37] mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd
From: Michael S. Tsirkin @ 2026-05-29 15:23 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Convert vma_alloc_anon_folio_pmd() to pass __GFP_ZERO instead of
zeroing at the callsite. post_alloc_hook uses the fault address
passed through vma_alloc_folio for cache-friendly zeroing.
Note: with __GFP_ZERO, the folio is zeroed before
mem_cgroup_charge(). If the charge fails, the zeroing work is
wasted. Previously zeroing was done after a successful charge.
This is inherent to moving zeroing into the allocator.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-6
---
mm/huge_memory.c | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index d689e6491ddb..4978d34532ea 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1333,7 +1333,7 @@ EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,
unsigned long addr)
{
- gfp_t gfp = vma_thp_gfp_mask(vma);
+ gfp_t gfp = vma_thp_gfp_mask(vma) | __GFP_ZERO;
const int order = HPAGE_PMD_ORDER;
struct folio *folio;
@@ -1356,17 +1356,9 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,
}
folio_throttle_swaprate(folio, gfp);
- /*
- * When a folio is not zeroed during allocation (__GFP_ZERO not used)
- * or user folios require special handling, folio_zero_user() is used to
- * make sure that the page corresponding to the faulting address will be
- * hot in the cache after zeroing.
- */
- if (user_alloc_needs_zeroing())
- folio_zero_user(folio, addr);
/*
* The memory barrier inside __folio_mark_uptodate makes sure that
- * folio_zero_user writes become visible before the set_pmd_at()
+ * page zeroing becomes visible before the set_pmd_at()
* write.
*/
__folio_mark_uptodate(folio);
--
MST
^ permalink raw reply related
* [PATCH v9 18/37] mm: vma_alloc_anon_folio_pmd: pass raw fault address to vma_alloc_folio
From: Michael S. Tsirkin @ 2026-05-29 15:23 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Now that vma_alloc_folio aligns the address internally, drop the
redundant HPAGE_PMD_MASK alignment at the callsite.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Gregory Price <gourry@gourry.net>
---
mm/huge_memory.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 970e077019b7..d689e6491ddb 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1337,7 +1337,7 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,
const int order = HPAGE_PMD_ORDER;
struct folio *folio;
- folio = vma_alloc_folio(gfp, order, vma, addr & HPAGE_PMD_MASK);
+ folio = vma_alloc_folio(gfp, order, vma, addr);
if (unlikely(!folio)) {
count_vm_event(THP_FAULT_FALLBACK);
--
MST
^ permalink raw reply related
* [PATCH v9 17/37] mm: use __GFP_ZERO in alloc_anon_folio
From: Michael S. Tsirkin @ 2026-05-29 15:23 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Convert alloc_anon_folio() to pass __GFP_ZERO instead of zeroing
at the callsite. post_alloc_hook uses the fault address passed
through vma_alloc_folio for cache-friendly zeroing.
Note: with __GFP_ZERO, the folio is zeroed before
mem_cgroup_charge(). If the charge fails, the zeroing work is
wasted. Previously zeroing was done after a successful charge.
This is inherent to moving zeroing into the allocator.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-6
---
mm/memory.c | 11 +----------
1 file changed, 1 insertion(+), 10 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index adc228fe3578..895ac72121fe 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5249,7 +5249,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
goto fallback;
/* Try allocating the highest of the remaining orders. */
- gfp = vma_thp_gfp_mask(vma);
+ gfp = vma_thp_gfp_mask(vma) | __GFP_ZERO;
while (orders) {
folio = vma_alloc_folio(gfp, order, vma, vmf->address);
if (folio) {
@@ -5259,15 +5259,6 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
goto next;
}
folio_throttle_swaprate(folio, gfp);
- /*
- * When a folio is not zeroed during allocation
- * (__GFP_ZERO not used) or user folios require special
- * handling, folio_zero_user() is used to make sure
- * that the page corresponding to the faulting address
- * will be hot in the cache after zeroing.
- */
- if (user_alloc_needs_zeroing())
- folio_zero_user(folio, vmf->address);
return folio;
}
next:
--
MST
^ permalink raw reply related
* [PATCH v9 16/37] mm: alloc_swap_folio: pass raw fault address to vma_alloc_folio
From: Michael S. Tsirkin @ 2026-05-29 15:23 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1780067977.git.mst@redhat.com>
Same change as the previous patch but for alloc_swap_folio:
pass vmf->address directly instead of ALIGN_DOWN(vmf->address, ...).
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
mm/memory.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 75bee9501666..adc228fe3578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4734,8 +4734,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
/* Try allocating the highest of the remaining orders. */
gfp = vma_thp_gfp_mask(vma);
while (orders) {
- addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
- folio = vma_alloc_folio(gfp, order, vma, addr);
+ folio = vma_alloc_folio(gfp, order, vma, vmf->address);
if (folio) {
if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
gfp, entry))
--
MST
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox