* Re: [PATCH RFC v2 1/3] io_uring: use an enumeration for io_uring_register(2) opcodes
From: Jens Axboe @ 2020-07-16 20:51 UTC (permalink / raw)
To: Pavel Begunkov, Stefano Garzarella
Cc: Alexander Viro, Kernel Hardening, Kees Cook, Aleksa Sarai,
Stefan Hajnoczi, Christian Brauner, Sargun Dhillon, Jann Horn,
io-uring, linux-fsdevel, Jeff Moyer, linux-kernel
In-Reply-To: <20326d79-fb5a-2480-e52a-e154e056171f@gmail.com>
On 7/16/20 2:47 PM, Pavel Begunkov wrote:
> On 16/07/2020 23:42, Jens Axboe wrote:
>> On 7/16/20 2:16 PM, Pavel Begunkov wrote:
>>> On 16/07/2020 15:48, Stefano Garzarella wrote:
>>>> The enumeration allows us to keep track of the last
>>>> io_uring_register(2) opcode available.
>>>>
>>>> Behaviour and opcodes names don't change.
>>>>
>>>> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
>>>> ---
>>>> include/uapi/linux/io_uring.h | 27 ++++++++++++++++-----------
>>>> 1 file changed, 16 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
>>>> index 7843742b8b74..efc50bd0af34 100644
>>>> --- a/include/uapi/linux/io_uring.h
>>>> +++ b/include/uapi/linux/io_uring.h
>>>> @@ -253,17 +253,22 @@ struct io_uring_params {
>>>> /*
>>>> * io_uring_register(2) opcodes and arguments
>>>> */
>>>> -#define IORING_REGISTER_BUFFERS 0
>>>> -#define IORING_UNREGISTER_BUFFERS 1
>>>> -#define IORING_REGISTER_FILES 2
>>>> -#define IORING_UNREGISTER_FILES 3
>>>> -#define IORING_REGISTER_EVENTFD 4
>>>> -#define IORING_UNREGISTER_EVENTFD 5
>>>> -#define IORING_REGISTER_FILES_UPDATE 6
>>>> -#define IORING_REGISTER_EVENTFD_ASYNC 7
>>>> -#define IORING_REGISTER_PROBE 8
>>>> -#define IORING_REGISTER_PERSONALITY 9
>>>> -#define IORING_UNREGISTER_PERSONALITY 10
>>>> +enum {
>>>> + IORING_REGISTER_BUFFERS,
>>>> + IORING_UNREGISTER_BUFFERS,
>>>> + IORING_REGISTER_FILES,
>>>> + IORING_UNREGISTER_FILES,
>>>> + IORING_REGISTER_EVENTFD,
>>>> + IORING_UNREGISTER_EVENTFD,
>>>> + IORING_REGISTER_FILES_UPDATE,
>>>> + IORING_REGISTER_EVENTFD_ASYNC,
>>>> + IORING_REGISTER_PROBE,
>>>> + IORING_REGISTER_PERSONALITY,
>>>> + IORING_UNREGISTER_PERSONALITY,
>>>> +
>>>> + /* this goes last */
>>>> + IORING_REGISTER_LAST
>>>> +};
>>>
>>> It breaks userspace API. E.g.
>>>
>>> #ifdef IORING_REGISTER_BUFFERS
>>
>> It can, yes, but we have done that in the past. In this one, for
>
> Ok, if nobody on the userspace side cares, then better to do that
> sooner than later.
>
>
>> example:
>>
>> commit 9e3aa61ae3e01ce1ce6361a41ef725e1f4d1d2bf (tag: io_uring-5.5-20191212)
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date: Wed Dec 11 15:55:43 2019 -0700
>>
>> io_uring: ensure we return -EINVAL on unknown opcod
>>
>> But it would be safer/saner to do this like we have the done the IOSQE_
>> flags.
>
> IOSQE_ are a bitmask, but this would look peculiar
>
> enum {
> __IORING_REGISTER_BUFFERS,
> ...
> };
> define IORING_REGISTER_BUFFERS __IORING_REGISTER_BUFFERS
Yeah true of course, that won't really work for this case at all.
That said, I don't think it's a huge deal to turn it into an enum.
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH net-next] net: phy: continue searching for C45 MMDs even if first returned ffff:ffff
From: Vladimir Oltean @ 2020-07-16 20:51 UTC (permalink / raw)
To: Andrew Lunn
Cc: davem, netdev, linux, f.fainelli, hkallweit1, claudiu.manoil,
alexandru.marginean, ioana.ciornei, michael
In-Reply-To: <20200716201210.GE1308244@lunn.ch>
On Thu, Jul 16, 2020 at 10:12:10PM +0200, Andrew Lunn wrote:
> > Then the rest of the code just carried on thinking "ok, MMD 1 (PMA/PMD)
> > says that there are 31 devices in that package, each having a device id
> > of ffff:ffff, that's perfectly fine, let's go ahead and probe this PHY
> > device".
>
> With a device ID of ffff:ffff, what PHY driver was getting loaded?
>
You mean ffff:fffe.
No PHY driver. I am driving this PCS locally from within
drivers/net/dsa/ocelot/felix_vsc9959.c. I call get_phy_device at the
address where I know a PCS is present, for the simple reason that I like
an extra validation that my internal MDIO reads/writes are going
somewhere. I've had situations in the past where the PCS was working
because the bootloader had initialized it, however the internal MDIO
reads/writes from Linux were broken. So, the fact that get_phy_device
can read the PHY ID correctly is giving me some assurance.
> > - MDIO_DEVS1=0x008a, MDIO_DEVS2=0x0000,
> > - MDIO_DEVID1=0x0083, MDIO_DEVID2=0xe400
>
> Now that we have valid IDs, is the same driver getting loaded? Do this
> ID adding somewhere?
>
Not applicable, see above.
> > Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
>
> Reviewed-by: Andrew Lunn <andrew@lunn.ch>
>
> Andrew
Thanks,
-Vladimir
^ permalink raw reply
* Re: [PATCH 0/3] Modernize tasklet callback API
From: Dmitry Torokhov @ 2020-07-16 20:48 UTC (permalink / raw)
To: Kees Cook
Cc: Peter Zijlstra, Greg Kroah-Hartman, Thomas Gleixner, Allen Pais,
Oscar Carter, Romain Perier, Kevin Curtis, David S. Miller,
Jakub Kicinski, Harald Freudenberger, Heiko Carstens,
Vasily Gorbik, Christian Borntraeger, Jiri Slaby, Felipe Balbi,
Jason Wessel, Daniel Thompson, Douglas Anderson,
Mitchell Blank Jr, Julian Wiedmann, Karsten Graul, Ursula Braun,
Jaroslav Kysela, Takashi Iwai, Christian Gromm, Nishka Dasgupta,
Masahiro Yamada, Stephen Boyd, Matthew Wilcox (Oracle),
Wambui Karuga, Guenter Roeck, Chris Packham, Kyungtae Kim,
Kuppuswamy Sathyanarayanan, Sebastian Andrzej Siewior,
Rafael J. Wysocki, Jonathan Corbet, Will Deacon,
linux-input@vger.kernel.org, lkml, netdev, linux-s390, devel,
USB list, kgdb-bugreport, alsa-devel@alsa-project.org,
kernel-hardening
In-Reply-To: <202007161214.102F6E6@keescook>
On Thu, Jul 16, 2020 at 12:14 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Jul 16, 2020 at 09:57:18AM +0200, Peter Zijlstra wrote:
> > On Wed, Jul 15, 2020 at 08:08:44PM -0700, Kees Cook wrote:
> > > Hi,
> > >
> > > This is the infrastructure changes to prepare the tasklet API for
> > > conversion to passing the tasklet struct as the callback argument instead
> > > of an arbitrary unsigned long. The first patch details why this is useful
> > > (it's the same rationale as the timer_struct changes from a bit ago:
> > > less abuse during memory corruption attacks, more in line with existing
> > > ways of doing things in the kernel, save a little space in struct,
> > > etc). Notably, the existing tasklet API use is much less messy, so there
> > > is less to clean up.
> >
> > I would _MUCH_ rather see tasklets go the way of the dodo, esp. given
> > that:
> >
> > > drivers/input/keyboard/omap-keypad.c | 2 +-
> > > drivers/input/serio/hil_mlc.c | 2 +-
> > > drivers/net/wan/farsync.c | 4 +--
> > > drivers/s390/crypto/ap_bus.c | 2 +-
> > > drivers/staging/most/dim2/dim2.c | 2 +-
> > > drivers/staging/octeon/ethernet-tx.c | 2 +-
> > > drivers/tty/vt/keyboard.c | 2 +-
> > > drivers/usb/gadget/udc/snps_udc_core.c | 6 ++---
> > > drivers/usb/host/fhci-sched.c | 2 +-
> > > include/linux/interrupt.h | 37 ++++++++++++++++++++++----
> > > kernel/backtracetest.c | 2 +-
> > > kernel/debug/debug_core.c | 2 +-
> > > kernel/irq/resend.c | 2 +-
> > > kernel/softirq.c | 18 ++++++++++++-
> > > net/atm/pppoatm.c | 2 +-
> > > net/iucv/iucv.c | 2 +-
> > > sound/drivers/pcsp/pcsp_lib.c | 2 +-
> > > 17 files changed, 66 insertions(+), 25 deletions(-)
> >
> > there appear to be hardly any users left.. Can't we stage an extinction
> > event here instead?
>
> Oh, I wish, but no. That's just the ones using DECLARE_TASKLET. There
> are hundred(s?) more (see the referenced tree).
Still, do we really need tasklets? Can we substitute timers executing
immediately in their place?
Thanks.
--
Dmitry
^ permalink raw reply
* Re: RFC: PCI devices passthrough on Arm design proposal
From: Stefano Stabellini @ 2020-07-16 20:51 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel@lists.xenproject.org, nd, Stefano Stabellini,
Roger Pau Monné, Julien Grall
In-Reply-To: <E9CBAA57-5EF3-47F9-8A40-F5D7816DB2A4@arm.com>
[-- Attachment #1: Type: text/plain, Size: 10540 bytes --]
On Thu, 16 Jul 2020, Rahul Singh wrote:
> Hello All,
>
> Following up on discussion on PCI Passthrough support on ARM that we had at the XEN summit, we are submitting a Review For Comment and a design proposal for PCI passthrough support on ARM. Feel free to give your feedback.
>
> The followings describe the high-level design proposal of the PCI passthrough support and how the different modules within the system interacts with each other to assign a particular PCI device to the guest.
I think the proposal is good and I only have a couple of thoughts to
share below.
> # Title:
>
> PCI devices passthrough on Arm design proposal
>
> # Problem statement:
>
> On ARM there in no support to assign a PCI device to a guest. PCI device passthrough capability allows guests to have full access to some PCI devices. PCI device passthrough allows PCI devices to appear and behave as if they were physically attached to the guest operating system and provide full isolation of the PCI devices.
>
> Goal of this work is to also support Dom0Less configuration so the PCI backend/frontend drivers used on x86 shall not be used on Arm. It will use the existing VPCI concept from X86 and implement the virtual PCI bus through IO emulation such that only assigned devices are visible to the guest and guest can use the standard PCI driver.
>
> Only Dom0 and Xen will have access to the real PCI bus, guest will have a direct access to the assigned device itself. IOMEM memory will be mapped to the guest and interrupt will be redirected to the guest. SMMU has to be configured correctly to have DMA transaction.
>
> ## Current state: Draft version
>
> # Proposer(s): Rahul Singh, Bertrand Marquis
>
> # Proposal:
>
> This section will describe the different subsystem to support the PCI device passthrough and how these subsystems interact with each other to assign a device to the guest.
>
> # PCI Terminology:
>
> Host Bridge: Host bridge allows the PCI devices to talk to the rest of the computer.
> ECAM: ECAM (Enhanced Configuration Access Mechanism) is a mechanism developed to allow PCIe to access configuration space. The space available per function is 4KB.
>
> # Discovering PCI Host Bridge in XEN:
>
> In order to support the PCI passthrough XEN should be aware of all the PCI host bridges available on the system and should be able to access the PCI configuration space. ECAM configuration access is supported as of now. XEN during boot will read the PCI device tree node “reg” property and will map the ECAM space to the XEN memory using the “ioremap_nocache ()” function.
>
> If there are more than one segment on the system, XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly. All the PCI device tree nodes should have the “linux,pci-domain” property so that there will be no conflicts. During hardware domain boot Linux will also use the same “linux,pci-domain” property and assign the domain number to the host bridge.
>
> When Dom0 tries to access the PCI config space of the device, XEN will find the corresponding host bridge based on segment number and access the corresponding config space assigned to that bridge.
>
> Limitation:
> * Only PCI ECAM configuration space access is supported.
> * Device tree binding is supported as of now, ACPI is not supported.
> * Need to port the PCI host bridge access code to XEN to access the configuration space (generic one works but lots of platforms will required some specific code or quirks).
>
> # Discovering PCI devices:
>
> PCI-PCIe enumeration is a process of detecting devices connected to its host. It is the responsibility of the hardware domain or boot firmware to do the PCI enumeration and configure the BAR, PCI capabilities, and MSI/MSI-X configuration.
>
> PCI-PCIe enumeration in XEN is not feasible for the configuration part as it would require a lot of code inside Xen which would require a lot of maintenance. Added to this many platforms require some quirks in that part of the PCI code which would greatly improve Xen complexity. Once hardware domain enumerates the device then it will communicate to XEN via the below hypercall.
>
> #define PHYSDEVOP_pci_device_add 25
> struct physdev_pci_device_add {
> uint16_t seg;
> uint8_t bus;
> uint8_t devfn;
> uint32_t flags;
> struct {
> uint8_t bus;
> uint8_t devfn;
> } physfn;
> /*
> * Optional parameters array.
> * First element ([0]) is PXM domain associated with the device (if * XEN_PCI_DEV_PXM is set)
> */
> uint32_t optarr[XEN_FLEX_ARRAY_DIM];
> };
>
> As the hypercall argument has the PCI segment number, XEN will access the PCI config space based on this segment number and find the host-bridge corresponding to this segment number. At this stage host bridge is fully initialized so there will be no issue to access the config space.
>
> XEN will add the PCI devices in the linked list maintain in XEN using the function pci_add_device(). XEN will be aware of all the PCI devices on the system and all the device will be added to the hardware domain.
>
> Limitations:
> * When PCI devices are added to XEN, MSI capability is not initialized inside XEN and not supported as of now.
> * ACS capability is disable for ARM as of now as after enabling it devices are not accessible.
> * Dom0Less implementation will require to have the capacity inside Xen to discover the PCI devices (without depending on Dom0 to declare them to Xen).
I think it is fine to assume that for dom0less the "firmware" has taken
care of setting up the BARs correctly. Starting with that assumption, it
looks like it should be "easy" to walk the PCI topology in Xen when/if
there is no dom0?
> # Enable the existing x86 virtual PCI support for ARM:
>
> The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space, XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.
>
> Limitation:
> * No handler is register for the MSI configuration.
> * Only legacy interrupt is supported and tested as of now, MSI is not implemented and tested.
>
> # Assign the device to the guest:
>
> Assign the PCI device from the hardware domain to the guest is done using the below guest config option. When xl tool create the domain, PCI devices will be assigned to the guest VPCI bus.
> pci=[ "PCI_SPEC_STRING", "PCI_SPEC_STRING", ...]
>
> Guest will be only able to access the assigned devices and see the bridges. Guest will not be able to access or see the devices that are no assigned to him.
>
> Limitation:
> * As of now all the bridges in the PCI bus are seen by the guest on the VPCI bus.
We need to come up with something similar for dom0less too. It could be
exactly the same thing (a list of BDFs as strings as a device tree
property) or something else if we can come up with a better idea.
> # Emulated PCI device tree node in libxl:
>
> Libxl is creating a virtual PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot. We introduced the new config option [vpci="pci_ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.
>
> A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node). A trap handler for the PCI ECAM access from guest has been registered at the defined address and redirects requests to the VPCI driver in Xen.
>
> Limitation:
> * Only one PCI device tree node is supported as of now.
I think vpci="pci_ecam" should be optional: if pci=[ "PCI_SPEC_STRING",
...] is specififed, then vpci="pci_ecam" is implied.
vpci="pci_ecam" is only useful one day in the future when we want to be
able to emulate other non-ecam host bridges. For now we could even skip
it.
> BAR value and IOMEM mapping:
>
> Linux guest will do the PCI enumeration based on the area reserved for ECAM and IOMEM ranges in the VPCI device tree node. Once PCI device is assigned to the guest, XEN will map the guest PCI IOMEM region to the real physical IOMEM region only for the assigned devices.
>
> As of now we have not modified the existing VPCI code to map the guest PCI IOMEM region to the real physical IOMEM region. We used the existing guest “iomem” config option to map the region.
> For example:
> Guest reserved IOMEM region: 0x04020000
> Real physical IOMEM region:0x50000000
> IOMEM size:128MB
> iomem config will be: iomem = ["0x50000,0x8000@0x4020"]
>
> There is no need to map the ECAM space as XEN already have access to the ECAM space and XEN will trap ECAM accesses from the guest and will perform read/write on the VPCI bus.
>
> IOMEM access will not be trapped and the guest will directly access the IOMEM region of the assigned device via stage-2 translation.
>
> In the same, we mapped the assigned devices IRQ to the guest using below config options.
> irqs= [ NUMBER, NUMBER, ...]
>
> Limitation:
> * Need to avoid the “iomem” and “irq” guest config options and map the IOMEM region and IRQ at the same time when device is assigned to the guest using the “pci” guest config options when xl creates the domain.
> * Emulated BAR values on the VPCI bus should reflect the IOMEM mapped address.
> * X86 mapping code should be ported on Arm so that the stage-2 translation is adapted when the guest is doing a modification of the BAR registers values (to map the address requested by the guest for a specific IOMEM to the address actually contained in the real BAR register of the corresponding device).
>
> # SMMU configuration for guest:
>
> When assigning PCI devices to a guest, the SMMU configuration should be updated to remove access to the hardware domain memory and add
> configuration to have access to the guest memory with the proper address translation so that the device can do DMA operations from and to the guest memory only.
>
> # MSI/MSI-X support:
> Not implement and tested as of now.
>
> # ITS support:
> Not implement and tested as of now.
^ permalink raw reply
* [dhowells-fs:keys-acl 1/5] security/smack/smack_lsm.c:4258:3: error: 'auth_can_override' undeclared
From: kernel test robot @ 2020-07-16 20:52 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 6404 bytes --]
tree: https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git keys-acl
head: 2a3c3edfc979691a7dc4733da827f508da90995a
commit: 0c172f6031ad95d98e74806335feb64d461816b8 [1/5] keys: Move permissions checking decisions into the checking code
config: i386-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-14) 9.3.0
reproduce (this is a W=1 build):
git checkout 0c172f6031ad95d98e74806335feb64d461816b8
# save the attached .config to linux build tree
make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
security/smack/smack_lsm.c: In function 'smack_key_permission':
>> security/smack/smack_lsm.c:4258:3: error: 'auth_can_override' undeclared (first use in this function)
4258 | auth_can_override = true;
| ^~~~~~~~~~~~~~~~~
security/smack/smack_lsm.c:4258:3: note: each undeclared identifier is reported only once for each function it appears in
>> security/smack/smack_lsm.c:4309:10: error: dereferencing pointer to incomplete type 'struct request_key_auth'
4309 | if (rka->target_key == key)
| ^~
>> security/smack/smack_lsm.c:4309:26: error: 'key' undeclared (first use in this function)
4309 | if (rka->target_key == key)
| ^~~
>> security/smack/smack_lsm.c:4310:5: error: '_perm' undeclared (first use in this function)
4310 | *_perm = 0;
| ^~~~~
--
security/selinux/hooks.c: In function 'selinux_keyperm_to_av':
>> security/selinux/hooks.c:6548:7: warning: variable 'sysadmin_can_override' set but not used [-Wunused-but-set-variable]
6548 | bool sysadmin_can_override = false;
| ^~~~~~~~~~~~~~~~~~~~~
In file included from arch/x86/include/asm/page_32.h:35,
from arch/x86/include/asm/page.h:14,
from arch/x86/include/asm/thread_info.h:12,
from include/linux/thread_info.h:38,
from arch/x86/include/asm/preempt.h:7,
from include/linux/preempt.h:78,
from include/linux/rcupdate.h:27,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from include/linux/tracehook.h:46,
from security/selinux/hooks.c:27:
In function 'strncpy',
inlined from 'selinux_ib_endport_manage_subnet' at security/selinux/hooks.c:6769:2:
include/linux/string.h:297:30: warning: '__builtin_strncpy' specified bound 64 equals destination size [-Wstringop-truncation]
297 | #define __underlying_strncpy __builtin_strncpy
| ^
include/linux/string.h:307:9: note: in expansion of macro '__underlying_strncpy'
307 | return __underlying_strncpy(p, q, size);
| ^~~~~~~~~~~~~~~~~~~~
vim +/auth_can_override +4258 security/smack/smack_lsm.c
4212
4213 /**
4214 * smack_key_permission - Smack access on a key
4215 * @key_ref: gets to the object
4216 * @cred: the credentials to use
4217 * @need_perm: requested key permission
4218 *
4219 * Return 0 if the task has read and write to the object,
4220 * an error code otherwise
4221 */
4222 static int smack_key_permission(key_ref_t key_ref,
4223 const struct cred *cred,
4224 enum key_need_perm need_perm,
4225 unsigned int flags)
4226 {
4227 struct key *keyp;
4228 struct smk_audit_info ad;
4229 struct smack_known *tkp = smk_of_task(smack_cred(cred));
4230 int request = 0;
4231 int rc;
4232
4233 keyp = key_ref_to_ptr(key_ref);
4234 if (keyp == NULL)
4235 return -EINVAL;
4236 /*
4237 * If the key hasn't been initialized give it access so that
4238 * it may do so.
4239 */
4240 if (keyp->security == NULL)
4241 return 0;
4242 /*
4243 * This should not occur
4244 */
4245 if (tkp == NULL)
4246 return -EACCES;
4247
4248 /*
4249 * Validate requested permissions
4250 */
4251 switch (need_perm) {
4252 case KEY_NEED_ASSUME_AUTHORITY:
4253 return 0;
4254
4255 case KEY_NEED_DESCRIBE:
4256 case KEY_NEED_GET_SECURITY:
4257 request |= MAY_READ;
> 4258 auth_can_override = true;
4259 break;
4260
4261 case KEY_NEED_CHOWN:
4262 case KEY_NEED_INVALIDATE:
4263 case KEY_NEED_JOIN:
4264 case KEY_NEED_LINK:
4265 case KEY_NEED_KEYRING_ADD:
4266 case KEY_NEED_KEYRING_CLEAR:
4267 case KEY_NEED_KEYRING_DELETE:
4268 case KEY_NEED_REVOKE:
4269 case KEY_NEED_SETPERM:
4270 case KEY_NEED_SET_RESTRICTION:
4271 case KEY_NEED_UPDATE:
4272 request |= MAY_WRITE;
4273 break;
4274
4275 case KEY_NEED_INSTANTIATE:
4276 auth_can_override = true;
4277 break;
4278
4279 case KEY_NEED_READ:
4280 case KEY_NEED_SEARCH:
4281 case KEY_NEED_USE:
4282 case KEY_NEED_WATCH:
4283 request |= MAY_READ;
4284 break;
4285
4286 case KEY_NEED_SET_TIMEOUT:
4287 request |= MAY_WRITE;
4288 auth_can_override = true;
4289 break;
4290
4291 case KEY_NEED_UNLINK:
4292 return 0; /* Mustn't prevent this; KEY_FLAG_KEEP is already
4293 * dealt with. */
4294
4295 default:
4296 WARN_ON(1);
4297 return -EINVAL;
4298 }
4299
4300 /* Just allow the operation if the process has an authorisation token.
4301 * The presence of the token means that the kernel delegated
4302 * instantiation of a key to the process - which is problematic if we
4303 * then say that the process isn't allowed to get the description of
4304 * the key or actually instantiate it.
4305 */
4306 if (auth_can_override && cred->request_key_auth) {
4307 struct request_key_auth *rka =
4308 cred->request_key_auth->payload.data[0];
> 4309 if (rka->target_key == key)
> 4310 *_perm = 0;
4311 }
4312
4313 if (smack_privileged_cred(CAP_MAC_OVERRIDE, cred))
4314 return 0;
4315
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 74068 bytes --]
^ permalink raw reply
* Re: [patch V3 01/13] entry: Provide generic syscall entry functionality
From: Kees Cook @ 2020-07-16 20:52 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, linux-arch, Will Deacon, Arnd Bergmann, Mark Rutland,
Keno Fischer, Paolo Bonzini, kvm, Gabriel Krisman Bertazi
In-Reply-To: <20200716185424.011950288@linutronix.de>
On Thu, Jul 16, 2020 at 08:22:09PM +0200, Thomas Gleixner wrote:
> From: Thomas Gleixner <tglx@linutronix.de>
>
> On syscall entry certain work needs to be done:
>
> - Establish state (lockdep, context tracking, tracing)
> - Conditional work (ptrace, seccomp, audit...)
>
> This code is needlessly duplicated and different in all
> architectures.
>
> Provide a generic version based on the x86 implementation which has all the
> RCU and instrumentation bits right.
Ahh! You're reading my mind! I was just thinking about this while
reviewing the proposed syscall redirection series[1], and pondering the
lack of x86 TIF flags, and that nearly everything in the series (and for
seccomp and other things) didn't need to be arch-specific. And now that
series absolutely needs to be rebased and it'll magically work for every
arch that switches to the generic entry code. :)
Notes below...
[1] https://lore.kernel.org/lkml/20200716193141.4068476-2-krisman@collabora.com/
> +/*
> + * Define dummy _TIF work flags if not defined by the architecture or for
> + * disabled functionality.
> + */
When I was thinking about this last week I was pondering having a split
between the arch-agnositc TIF flags and the arch-specific TIF flags, and
that each arch could have a single "there is agnostic work to be done"
TIF in their thread_info, and the agnostic flags could live in
task_struct or something. Anyway, I'll keep reading...
> +/**
> + * syscall_enter_from_user_mode - Check and handle work before invoking
> + * a syscall
> + * @regs: Pointer to currents pt_regs
> + * @syscall: The syscall number
> + *
> + * Invoked from architecture specific syscall entry code with interrupts
> + * disabled. The calling code has to be non-instrumentable. When the
> + * function returns all state is correct and the subsequent functions can be
> + * instrumented.
> + *
> + * Returns: The original or a modified syscall number
> + *
> + * If the returned syscall number is -1 then the syscall should be
> + * skipped. In this case the caller may invoke syscall_set_error() or
> + * syscall_set_return_value() first. If neither of those are called and -1
> + * is returned, then the syscall will fail with ENOSYS.
There's been some recent confusion over "has the syscall changed,
or did seccomp request it be skipped?" that was explored in arm64[2]
(though I see Will and Keno in CC already). There might need to be a
clearer way to distinguish between "wild userspace issued a -1 syscall"
and "seccomp or ptrace asked for the syscall to be skipped". The
difference is mostly about when ENOSYS gets set, with respect to calls
to syscall_set_return_value(), but if the syscall gets changed, the arch
may need to recheck the value and consider ENOSYS, etc. IIUC, what Will
ended up with[3] was having syscall_trace_enter() return the syscall return
value instead of the new syscall.
[2] https://lore.kernel.org/lkml/20200704125027.GB21185@willie-the-truck/
[3] https://lore.kernel.org/lkml/20200703083914.GA18516@willie-the-truck/
> +static long syscall_trace_enter(struct pt_regs *regs, long syscall,
> + unsigned long ti_work)
> +{
> + long ret = 0;
> +
> + /* Handle ptrace */
> + if (ti_work & (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_EMU)) {
> + ret = arch_syscall_enter_tracehook(regs);
> + if (ret || (ti_work & _TIF_SYSCALL_EMU))
> + return -1L;
> + }
> +
> + /* Do seccomp after ptrace, to catch any tracer changes. */
> + if (ti_work & _TIF_SECCOMP) {
> + ret = arch_syscall_enter_seccomp(regs);
> + if (ret == -1L)
> + return ret;
> + }
> +
> + if (unlikely(ti_work & _TIF_SYSCALL_TRACEPOINT))
> + trace_sys_enter(regs, syscall);
> +
> + arch_syscall_enter_audit(regs);
> +
> + return ret ? : syscall;
> +}
Modulo the notes about -1 vs syscall number above, this looks correct to
me for ptrace and seccomp.
--
Kees Cook
^ permalink raw reply
* [PATCH] net: alteon: Avoid some useless memset
From: Christophe JAILLET @ 2020-07-16 20:52 UTC (permalink / raw)
To: davem, kuba, jes
Cc: linux-acenic, netdev, linux-kernel, kernel-janitors,
Christophe JAILLET
Avoid a memset after a call to 'dma_alloc_coherent()'.
This is useless since
commit 518a2f1925c3 ("dma-mapping: zero memory returned from dma_alloc_*")
Replace a kmalloc+memset with a corresponding kzalloc.
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
---
drivers/net/ethernet/alteon/acenic.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/alteon/acenic.c b/drivers/net/ethernet/alteon/acenic.c
index 99431c9a899b..ac86fcae1582 100644
--- a/drivers/net/ethernet/alteon/acenic.c
+++ b/drivers/net/ethernet/alteon/acenic.c
@@ -1151,7 +1151,7 @@ static int ace_init(struct net_device *dev)
/*
* Get the memory for the skb rings.
*/
- if (!(ap->skb = kmalloc(sizeof(struct ace_skb), GFP_KERNEL))) {
+ if (!(ap->skb = kzalloc(sizeof(struct ace_skb), GFP_KERNEL))) {
ecode = -EAGAIN;
goto init_error;
}
@@ -1172,9 +1172,6 @@ static int ace_init(struct net_device *dev)
ap->last_mini_rx = 0;
#endif
- memset(ap->info, 0, sizeof(struct ace_info));
- memset(ap->skb, 0, sizeof(struct ace_skb));
-
ecode = ace_load_firmware(dev);
if (ecode)
goto init_error;
--
2.25.1
^ permalink raw reply related
* [PATCH] net: alteon: Avoid some useless memset
From: Christophe JAILLET @ 2020-07-16 20:52 UTC (permalink / raw)
To: davem, kuba, jes
Cc: linux-acenic, netdev, linux-kernel, kernel-janitors,
Christophe JAILLET
Avoid a memset after a call to 'dma_alloc_coherent()'.
This is useless since
commit 518a2f1925c3 ("dma-mapping: zero memory returned from dma_alloc_*")
Replace a kmalloc+memset with a corresponding kzalloc.
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
---
drivers/net/ethernet/alteon/acenic.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/alteon/acenic.c b/drivers/net/ethernet/alteon/acenic.c
index 99431c9a899b..ac86fcae1582 100644
--- a/drivers/net/ethernet/alteon/acenic.c
+++ b/drivers/net/ethernet/alteon/acenic.c
@@ -1151,7 +1151,7 @@ static int ace_init(struct net_device *dev)
/*
* Get the memory for the skb rings.
*/
- if (!(ap->skb = kmalloc(sizeof(struct ace_skb), GFP_KERNEL))) {
+ if (!(ap->skb = kzalloc(sizeof(struct ace_skb), GFP_KERNEL))) {
ecode = -EAGAIN;
goto init_error;
}
@@ -1172,9 +1172,6 @@ static int ace_init(struct net_device *dev)
ap->last_mini_rx = 0;
#endif
- memset(ap->info, 0, sizeof(struct ace_info));
- memset(ap->skb, 0, sizeof(struct ace_skb));
-
ecode = ace_load_firmware(dev);
if (ecode)
goto init_error;
--
2.25.1
^ permalink raw reply related
* Re: [dpdk-dev] [PATCH v2 2/2] bus/vdev: build on Windows
From: Narcisa Ana Maria Vasile @ 2020-07-16 20:53 UTC (permalink / raw)
To: talshn
Cc: dev, thomas, pallavi.kadam, dmitry.kozliuk, david.marchand, grive,
ranjit.menon, harini.ramakrishnan, ocardona, anatoly.burakov
In-Reply-To: <20200707084823.28160-3-talshn@mellanox.com>
On Tue, Jul 07, 2020 at 11:48:23AM +0300, talshn@mellanox.com wrote:
> From: Tal Shnaiderman <talshn@mellanox.com>
>
> current support will build vdev with empty MP functions
> currently unsupported for Windows.
>
> Signed-off-by: Tal Shnaiderman <talshn@mellanox.com>
> ---
> drivers/bus/vdev/meson.build | 6 ------
> lib/librte_eal/rte_eal_exports.def | 5 +++++
> 2 files changed, 5 insertions(+), 6 deletions(-)
Tested-by: Narcisa Vasile <navasile@linux.microsoft.com>
Acked-by: Narcisa Vasile <navasile@linux.microsoft.com>
Thanks for the instructions on system configuration, Tal.
Compiled with mingw and clang 9.0.0. VDEV is compiled and registered successfully.
C:\Users\User\hw> .\dpdk-helloworld.exe -l 1
EAL: Registered [vdev] bus.
EAL: Registered [pci] bus.
EAL: Detected 20 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process support is requested, but not available.
EAL: Cannot access virt2phys driver, PA will not be available
EAL: WARNING: TSC frequency estimated roughly - clock timings may be less accurate.
hello from core 1
With clang I first got a linker error, that goes away if the symbol is added in the .def file:
bus_vdev_vdev.c.obj : error LNK2019: unresolved external symbol per_lcore__thread_id referenced in function __vdev_logtype_bus
drivers\librte_bus_vdev-20.0.dll : fatal error LNK1120: 1 unresolved externals
^ permalink raw reply
* Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support
From: Richard Cochran @ 2020-07-16 20:53 UTC (permalink / raw)
To: Russell King - ARM Linux admin
Cc: Andrew Lunn, Florian Fainelli, Heiner Kallweit, David S. Miller,
Jakub Kicinski, netdev
In-Reply-To: <20200716113354.GS1605@shell.armlinux.org.uk>
On Thu, Jul 16, 2020 at 12:33:54PM +0100, Russell King - ARM Linux admin wrote:
>
> PTP_1588_CLOCK could then be used as a global enable for PTP support
> where appropriate. We can also avoid all the Kconfig complexity
> introduced by having two independent subsystems either of which could
> be modular.
I'm sorry about the Kconfig confusion. I think it all started with
the tiny-fication work and the new "select" Kconfig keyword. The
purpose of all this was, IIRC, to allow leaving dynamic posix clocks
out of the kernel, but it still makes my head spin.
Thanks,
Richard
^ permalink raw reply
* Re: [dpdk-dev] [PATCH v2 1/2] eal/windows: add needed calls to detect vdev PMD
From: Narcisa Ana Maria Vasile @ 2020-07-16 20:54 UTC (permalink / raw)
To: talshn
Cc: dev, thomas, pallavi.kadam, dmitry.kozliuk, david.marchand, grive,
ranjit.menon, harini.ramakrishnan, ocardona, anatoly.burakov
In-Reply-To: <20200707084823.28160-2-talshn@mellanox.com>
On Tue, Jul 07, 2020 at 11:48:22AM +0300, talshn@mellanox.com wrote:
> From: Tal Shnaiderman <talshn@mellanox.com>
>
> Add needed function calls in rte_eal_init to detect vdev PMD.
>
> eal_option_device_parse()
> rte_service_init()
> rte_bus_probe()
>
> Signed-off-by: Tal Shnaiderman <talshn@mellanox.com>
> ---
> lib/librte_eal/common/meson.build | 1 +
> lib/librte_eal/windows/eal.c | 20 ++++++++++++++++++++
> 2 files changed, 21 insertions(+)
Tested-by: Narcisa Vasile <navasile@linux.microsoft.com>
Acked-by: Narcisa Vasile <navasile@linux.microsoft.com>
^ permalink raw reply
* Re: [patch V3 02/13] entry: Provide generic syscall exit function
From: Kees Cook @ 2020-07-16 20:55 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, linux-arch, Will Deacon, Arnd Bergmann, Mark Rutland,
Keno Fischer, Paolo Bonzini, kvm
In-Reply-To: <20200716185424.116500611@linutronix.de>
On Thu, Jul 16, 2020 at 08:22:10PM +0200, Thomas Gleixner wrote:
> From: Thomas Gleixner <tglx@linutronix.de>
>
> Like syscall entry all architectures have similar and pointlessly different
> code to handle pending work before returning from a syscall to user space.
>
> 1) One-time syscall exit work:
> - rseq syscall exit
> - audit
> - syscall tracing
> - tracehook (single stepping)
>
> 2) Preparatory work
> - Exit to user mode loop (common TIF handling).
> - Architecture specific one time work arch_exit_to_user_mode_prepare()
> - Address limit and lockdep checks
>
> 3) Final transition (lockdep, tracing, context tracking, RCU). Invokes
> arch_exit_to_user_mode() to handle e.g. speculation mitigations
>
> Provide a generic version based on the x86 code which has all the RCU and
> instrumentation protections right.
>
> Provide a variant for interrupt return to user mode as well which shares
> the above #2 and #3 work items.
>
> After syscall_exit_to_user_mode() and irqentry_exit_to_user_mode() the
> architecture code just has to return to user space. The code after
> returning from these functions must not be instrumented.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
This looks correct to me. Did you happen to run the seccomp selftests
under this series?
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply
* [LTP] [PATCH] cpuset_hotplug_test.sh: Fix a race condition
From: Petr Vorel @ 2020-07-16 20:56 UTC (permalink / raw)
To: ltp
In-Reply-To: <20200716153037.3qpeyxjenditkq6k@e107158-lin.cambridge.arm.com>
Hi Qais,
merged your original patch.
Thanks for your time.
Kind regards,
Petr
^ permalink raw reply
* Re: [patch V3 05/13] x86/entry: Consolidate check_user_regs()
From: Kees Cook @ 2020-07-16 20:56 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, linux-arch, Will Deacon, Arnd Bergmann, Mark Rutland,
Keno Fischer, Paolo Bonzini, kvm
In-Reply-To: <20200716185424.442693273@linutronix.de>
On Thu, Jul 16, 2020 at 08:22:13PM +0200, Thomas Gleixner wrote:
> The user register sanity check is sprinkled all over the place. Move it
> into enter_from_user_mode().
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply
* Re: [PATCH v5 10/11] hw/arm: Wire up BMC boot flash for npcm750-evb and quanta-gsj
From: Havard Skinnemoen @ 2020-07-16 20:56 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Markus Armbruster, Kevin Wolf, Peter Maydell, Qemu-block,
QEMU Developers, CS20 KFTing, qemu-arm, Cédric Le Goater,
IS20 Avi Fishman
In-Reply-To: <CAFQmdRa9FiP6yX=XDrJy5KpjifQyVkQiY6DXtSoJSvfoq7Vm6w@mail.gmail.com>
On Wed, Jul 15, 2020 at 1:54 PM Havard Skinnemoen
<hskinnemoen@google.com> wrote:
>
> On Wed, Jul 15, 2020 at 3:57 AM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> >
> > On 7/15/20 11:00 AM, Markus Armbruster wrote:
> > > Now my point. Why first make up user configuration, then use that to
> > > create a BlockBackend, when you could just go ahead and create the
> > > BlockBackend?
> >
> > CLI issue mostly.
> >
> > We can solve it similarly to the recent "sdcard: Do not allow invalid SD
> > card sizes" patch:
> >
> > if (!dinfo) {
> > error_setg(errp, "Missing SPI flash drive");
> > error_append_hint(errp, "You can use a dummy drive using:\n");
> > error_append_hint(errp, "-drive if=mtd,driver=null-co,"
> > "read-ones=on,size=64M\n);
> > return;
> > }
> >
> > having npcm7xx_connect_flash() taking an Error* argument,
> > and MachineClass::init() call it with &error_fatal.
>
> Erroring out if the user specifies a configuration that can't possibly
> boot sounds good to me. Better than trying to come up with defaults
> that are still not going to result in a bootable system.
>
> For testing recovery paths, I think it makes sense to explicitly
> specify a null device as you suggest.
Hmm, one problem. qom-test fails with
qemu-system-aarch64: Missing SPI flash drive
You can add a dummy drive using:
-drive if=mtd,driver=null-co,read-zeroes=on,size=32M
Broken pipe
/usr/local/google/home/hskinnemoen/qemu/for-upstream/tests/qtest/libqtest.c:166:
kill_qemu() tried to terminate QEMU process but encountered exit
status 1 (expected 0)
ERROR qom-test - too few tests run (expected 68, got 7)
So it looks like we might need a different solution to this, unless we
want to make generic tests more machine-aware...
^ permalink raw reply
* Re: [patch V3 07/13] x86/ptrace: Provide pt_regs helpers for entry/exit
From: Kees Cook @ 2020-07-16 20:57 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, linux-arch, Will Deacon, Arnd Bergmann, Mark Rutland,
Keno Fischer, Paolo Bonzini, kvm
In-Reply-To: <20200716185424.658427667@linutronix.de>
On Thu, Jul 16, 2020 at 08:22:15PM +0200, Thomas Gleixner wrote:
> As a preparatory step for moving the syscall and interrupt entry/exit
> handling into generic code, provide pt_regs helpers which allow to:
>
> - Retrieve the syscall number from pt_regs
> - Retrieve the syscall return value from pt_regs
> - Retrieve the interrupt state from pt_regs to check whether interrupts
> are reenabled by return from interrupt/exception.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply
* [patch] mm, memcg: provide an anon_reclaimable stat
From: David Rientjes @ 2020-07-16 20:58 UTC (permalink / raw)
To: SeongJae Park, Andrew Morton
Cc: Yang Shi, Michal Hocko, Shakeel Butt, Yang Shi, Roman Gushchin,
Greg Thelen, Johannes Weiner, Vladimir Davydov,
cgroups-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg
In-Reply-To: <alpine.DEB.2.23.453.2007151031020.2788464-X6Q0R45D7oAcqpCFd4KODRPsWskHk0ljAL8bYrjMMd8@public.gmane.org>
Userspace can lack insight into the amount of memory that can be reclaimed
from a memcg based on values from memory.stat. Two specific examples:
- Lazy freeable memory (MADV_FREE) that are clean anonymous pages on the
inactive file LRU that can be quickly reclaimed under memory pressure
but otherwise shows up as mapped anon in memory.stat, and
- Memory on deferred split queues (thp) that are compound pages that can
be split and uncharged from the memcg under memory pressure, but
otherwise shows up as charged anon LRU memory in memory.stat.
Both of this anonymous usage is also charged to memory.current.
Userspace can currently derive this information but it depends on kernel
implementation details for how this memory is handled for the purposes of
reclaim (anon on inactive file LRU or unmapped anon on the LRU).
For the purposes of writing portable userspace code that does not need to
have insight into the kernel implementation for reclaimable memory, this
exports a stat that reveals the amount of anonymous memory that can be
reclaimed and uncharged from the memcg to start new applications.
As the kernel implementation evolves for memory that can be reclaimed
under memory pressure, this stat can be kept consistent.
Signed-off-by: David Rientjes <rientjes-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
---
Documentation/admin-guide/cgroup-v2.rst | 6 +++++
mm/memcontrol.c | 31 +++++++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1296,6 +1296,12 @@ PAGE_SIZE multiple when read back.
Amount of memory used in anonymous mappings backed by
transparent hugepages
+ anon_reclaimable
+ The amount of charged anonymous memory that can be reclaimed
+ under memory pressure without swap. This currently includes
+ lazy freeable memory (MADV_FREE) and compound pages that can be
+ split and uncharged.
+
inactive_anon, active_anon, inactive_file, active_file, unevictable
Amount of memory, swap-backed and filesystem-backed,
on the internal memory management lists used by the
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1350,6 +1350,32 @@ static bool mem_cgroup_wait_acct_move(struct mem_cgroup *memcg)
return false;
}
+/*
+ * Returns the amount of anon memory that is charged to the memcg that is
+ * reclaimable under memory pressure without swap, in pages.
+ */
+static unsigned long memcg_anon_reclaimable(struct mem_cgroup *memcg)
+{
+ long deferred, lazyfree;
+
+ /*
+ * Deferred pages are charged anonymous pages that are on the LRU but
+ * are unmapped. These compound pages are split under memory pressure.
+ */
+ deferred = max_t(long, memcg_page_state(memcg, NR_ACTIVE_ANON) +
+ memcg_page_state(memcg, NR_INACTIVE_ANON) -
+ memcg_page_state(memcg, NR_ANON_MAPPED), 0);
+ /*
+ * Lazyfree pages are charged clean anonymous pages that are on the file
+ * LRU and can be reclaimed under memory pressure.
+ */
+ lazyfree = max_t(long, memcg_page_state(memcg, NR_ACTIVE_FILE) +
+ memcg_page_state(memcg, NR_INACTIVE_FILE) -
+ memcg_page_state(memcg, NR_FILE_PAGES), 0);
+
+ return deferred + lazyfree;
+}
+
static char *memory_stat_format(struct mem_cgroup *memcg)
{
struct seq_buf s;
@@ -1363,6 +1389,9 @@ static char *memory_stat_format(struct mem_cgroup *memcg)
* Provide statistics on the state of the memory subsystem as
* well as cumulative event counters that show past behavior.
*
+ * All values in this buffer are read individually, so no implied
+ * consistency amongst them.
+ *
* This list is ordered following a combination of these gradients:
* 1) generic big picture -> specifics and details
* 2) reflecting userspace activity -> reflecting kernel heuristics
@@ -1405,6 +1434,8 @@ static char *memory_stat_format(struct mem_cgroup *memcg)
(u64)memcg_page_state(memcg, NR_ANON_THPS) *
HPAGE_PMD_SIZE);
#endif
+ seq_buf_printf(&s, "anon_reclaimable %llu\n",
+ (u64)memcg_anon_reclaimable(memcg) * PAGE_SIZE);
for (i = 0; i < NR_LRU_LISTS; i++)
seq_buf_printf(&s, "%s %llu\n", lru_list_name(i),
^ permalink raw reply
* [patch] mm, memcg: provide an anon_reclaimable stat
From: David Rientjes @ 2020-07-16 20:58 UTC (permalink / raw)
To: SeongJae Park, Andrew Morton
Cc: Yang Shi, Michal Hocko, Shakeel Butt, Yang Shi, Roman Gushchin,
Greg Thelen, Johannes Weiner, Vladimir Davydov, cgroups, linux-mm
In-Reply-To: <alpine.DEB.2.23.453.2007151031020.2788464@chino.kir.corp.google.com>
Userspace can lack insight into the amount of memory that can be reclaimed
from a memcg based on values from memory.stat. Two specific examples:
- Lazy freeable memory (MADV_FREE) that are clean anonymous pages on the
inactive file LRU that can be quickly reclaimed under memory pressure
but otherwise shows up as mapped anon in memory.stat, and
- Memory on deferred split queues (thp) that are compound pages that can
be split and uncharged from the memcg under memory pressure, but
otherwise shows up as charged anon LRU memory in memory.stat.
Both of this anonymous usage is also charged to memory.current.
Userspace can currently derive this information but it depends on kernel
implementation details for how this memory is handled for the purposes of
reclaim (anon on inactive file LRU or unmapped anon on the LRU).
For the purposes of writing portable userspace code that does not need to
have insight into the kernel implementation for reclaimable memory, this
exports a stat that reveals the amount of anonymous memory that can be
reclaimed and uncharged from the memcg to start new applications.
As the kernel implementation evolves for memory that can be reclaimed
under memory pressure, this stat can be kept consistent.
Signed-off-by: David Rientjes <rientjes@google.com>
---
Documentation/admin-guide/cgroup-v2.rst | 6 +++++
mm/memcontrol.c | 31 +++++++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1296,6 +1296,12 @@ PAGE_SIZE multiple when read back.
Amount of memory used in anonymous mappings backed by
transparent hugepages
+ anon_reclaimable
+ The amount of charged anonymous memory that can be reclaimed
+ under memory pressure without swap. This currently includes
+ lazy freeable memory (MADV_FREE) and compound pages that can be
+ split and uncharged.
+
inactive_anon, active_anon, inactive_file, active_file, unevictable
Amount of memory, swap-backed and filesystem-backed,
on the internal memory management lists used by the
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1350,6 +1350,32 @@ static bool mem_cgroup_wait_acct_move(struct mem_cgroup *memcg)
return false;
}
+/*
+ * Returns the amount of anon memory that is charged to the memcg that is
+ * reclaimable under memory pressure without swap, in pages.
+ */
+static unsigned long memcg_anon_reclaimable(struct mem_cgroup *memcg)
+{
+ long deferred, lazyfree;
+
+ /*
+ * Deferred pages are charged anonymous pages that are on the LRU but
+ * are unmapped. These compound pages are split under memory pressure.
+ */
+ deferred = max_t(long, memcg_page_state(memcg, NR_ACTIVE_ANON) +
+ memcg_page_state(memcg, NR_INACTIVE_ANON) -
+ memcg_page_state(memcg, NR_ANON_MAPPED), 0);
+ /*
+ * Lazyfree pages are charged clean anonymous pages that are on the file
+ * LRU and can be reclaimed under memory pressure.
+ */
+ lazyfree = max_t(long, memcg_page_state(memcg, NR_ACTIVE_FILE) +
+ memcg_page_state(memcg, NR_INACTIVE_FILE) -
+ memcg_page_state(memcg, NR_FILE_PAGES), 0);
+
+ return deferred + lazyfree;
+}
+
static char *memory_stat_format(struct mem_cgroup *memcg)
{
struct seq_buf s;
@@ -1363,6 +1389,9 @@ static char *memory_stat_format(struct mem_cgroup *memcg)
* Provide statistics on the state of the memory subsystem as
* well as cumulative event counters that show past behavior.
*
+ * All values in this buffer are read individually, so no implied
+ * consistency amongst them.
+ *
* This list is ordered following a combination of these gradients:
* 1) generic big picture -> specifics and details
* 2) reflecting userspace activity -> reflecting kernel heuristics
@@ -1405,6 +1434,8 @@ static char *memory_stat_format(struct mem_cgroup *memcg)
(u64)memcg_page_state(memcg, NR_ANON_THPS) *
HPAGE_PMD_SIZE);
#endif
+ seq_buf_printf(&s, "anon_reclaimable %llu\n",
+ (u64)memcg_anon_reclaimable(memcg) * PAGE_SIZE);
for (i = 0; i < NR_LRU_LISTS; i++)
seq_buf_printf(&s, "%s %llu\n", lru_list_name(i),
^ permalink raw reply
* Re: [PATCH net-next] net: phy: continue searching for C45 MMDs even if first returned ffff:ffff
From: Vladimir Oltean @ 2020-07-16 20:58 UTC (permalink / raw)
To: Andrew Lunn
Cc: davem, netdev, linux, f.fainelli, hkallweit1, claudiu.manoil,
alexandru.marginean, ioana.ciornei, michael
In-Reply-To: <20200716205137.goazvzvhie5s7ttl@skbuf>
On Thu, Jul 16, 2020 at 11:51:37PM +0300, Vladimir Oltean wrote:
> On Thu, Jul 16, 2020 at 10:12:10PM +0200, Andrew Lunn wrote:
> > > Then the rest of the code just carried on thinking "ok, MMD 1 (PMA/PMD)
> > > says that there are 31 devices in that package, each having a device id
> > > of ffff:ffff, that's perfectly fine, let's go ahead and probe this PHY
> > > device".
> >
> > With a device ID of ffff:ffff, what PHY driver was getting loaded?
> >
>
> You mean ffff:fffe.
Sorry, I was wrong to correct you here. ffff:fffe was the
devices-in-package register, the phy id was ffff:ffff. Doesn't change
the rest of the answer though.
> No PHY driver. I am driving this PCS locally from within
> drivers/net/dsa/ocelot/felix_vsc9959.c. I call get_phy_device at the
> address where I know a PCS is present, for the simple reason that I like
> an extra validation that my internal MDIO reads/writes are going
> somewhere. I've had situations in the past where the PCS was working
> because the bootloader had initialized it, however the internal MDIO
> reads/writes from Linux were broken. So, the fact that get_phy_device
> can read the PHY ID correctly is giving me some assurance.
>
> > > - MDIO_DEVS1=0x008a, MDIO_DEVS2=0x0000,
> > > - MDIO_DEVID1=0x0083, MDIO_DEVID2=0xe400
> >
> > Now that we have valid IDs, is the same driver getting loaded? Do this
> > ID adding somewhere?
> >
>
> Not applicable, see above.
>
> > > Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
> >
> > Reviewed-by: Andrew Lunn <andrew@lunn.ch>
> >
> > Andrew
>
> Thanks,
> -Vladimir
^ permalink raw reply
* [Buildroot] [PATCH 1/1] package/refpolicy: needs python3
From: Fabrice Fontaine @ 2020-07-16 20:59 UTC (permalink / raw)
To: buildroot
Commit 09b94b1e8f94d1f49f7a2349b4c8cbfecefa73da forgot to propagate
python3 dependency from host-setools to refpolicy
Fixes:
- http://autobuild.buildroot.org/results/fdeaa54355aa0741eccc7864b034a7b07ef0c4bc
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
---
package/refpolicy/Config.in | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/package/refpolicy/Config.in b/package/refpolicy/Config.in
index b50b2f09ff..e374ab1901 100644
--- a/package/refpolicy/Config.in
+++ b/package/refpolicy/Config.in
@@ -1,5 +1,6 @@
config BR2_PACKAGE_REFPOLICY
bool "refpolicy"
+ depends on BR2_PACKAGE_PYTHON3 # host-setools
depends on BR2_TOOLCHAIN_HAS_THREADS # libsepol
# Even though libsepol is not necessary for building, we get
# the policy version from libsepol, so we select it, and treat
@@ -58,3 +59,6 @@ endif
comment "refpolicy needs a toolchain w/ threads"
depends on !BR2_TOOLCHAIN_HAS_THREADS
+
+comment "refpolicy needs python3"
+ depends on !BR2_PACKAGE_PYTHON3
--
2.27.0
^ permalink raw reply related
* Re: [dpdk-dev] [PATCH 0/9] python2 deprecation notice
From: Stephen Hemminger @ 2020-07-16 21:01 UTC (permalink / raw)
To: Robin Jarry; +Cc: Bruce Richardson, Louise Kilheeney, dev, david.marchand
In-Reply-To: <20200716144429.tf3agu7tfpy4nfp7@6wind.com>
On Thu, 16 Jul 2020 16:44:29 +0200
Robin Jarry <robin.jarry@6wind.com> wrote:
> 2020-07-13, Bruce Richardson:
> > In the absense of a "proper" solution, is the simplest option to
> > change the shebangs to all be python3, but leave the python2
> > compatibility in place, and add the warnings for anyone running it
> > explicitly using python2?
>
> I have found a hacky[1] but somewhat not too ugly way to dynamically use
> any available python interpreter.
>
> Basically, the scripts shebangs become:
>
> #!/bin/sh
> # shell script hack to dynamically use any available python interpreter
> ''':'
> if command -pv python3 >/dev/null 2>&1; then
> exec python3 "$0" "$@"
> else
> exec python2 "$0" "$@"
> fi
> '''
> # real python code starts here
>
> This hack obviously would remain only for the time where we still have
> python 2 support and can be removed after 20.08 is released.
>
> What do you think?
>
> [1] https://unix.stackexchange.com/a/66242
>
No.
^ permalink raw reply
* [Buildroot] [PATCH 1/1] package/setools: add comment
From: Fabrice Fontaine @ 2020-07-16 21:02 UTC (permalink / raw)
To: buildroot
Commit 09b94b1e8f94d1f49f7a2349b4c8cbfecefa73da forgot to add the usual
comment that python3 dependency is due to python-networkx
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
---
package/setools/Config.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/package/setools/Config.in b/package/setools/Config.in
index 9475bbef65..882c8764f3 100644
--- a/package/setools/Config.in
+++ b/package/setools/Config.in
@@ -4,7 +4,7 @@ config BR2_PACKAGE_SETOOLS
depends on BR2_TOOLCHAIN_HAS_THREADS
depends on BR2_USE_WCHAR
depends on BR2_USE_MMU
- depends on BR2_PACKAGE_PYTHON3
+ depends on BR2_PACKAGE_PYTHON3 # python-networkx
select BR2_PACKAGE_PYTHON_NETWORKX # runtime
select BR2_PACKAGE_PYTHON_SETUPTOOLS
select BR2_PACKAGE_LIBSELINUX
--
2.27.0
^ permalink raw reply related
* [Intel-gfx] ✗ Fi.CI.BAT: failure for series starting with [1/2] drm/i915/selftests: Add compiler paranoia for checking HWSP values
From: Patchwork @ 2020-07-16 21:02 UTC (permalink / raw)
To: Chris Wilson; +Cc: intel-gfx
In-Reply-To: <20200716203201.11977-1-chris@chris-wilson.co.uk>
[-- Attachment #1.1: Type: text/plain, Size: 5709 bytes --]
== Series Details ==
Series: series starting with [1/2] drm/i915/selftests: Add compiler paranoia for checking HWSP values
URL : https://patchwork.freedesktop.org/series/79565/
State : failure
== Summary ==
CI Bug Log - changes from CI_DRM_8757 -> Patchwork_18195
====================================================
Summary
-------
**FAILURE**
Serious unknown changes coming with Patchwork_18195 absolutely need to be
verified manually.
If you think the reported changes have nothing to do with the changes
introduced in Patchwork_18195, please notify your bug team to allow them
to document this new failure mode, which will reduce false positives in CI.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/index.html
Possible new issues
-------------------
Here are the unknown changes that may have been introduced in Patchwork_18195:
### IGT changes ###
#### Possible regressions ####
* igt@runner@aborted:
- fi-tgl-y: NOTRUN -> [FAIL][1]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-tgl-y/igt@runner@aborted.html
- fi-tgl-u2: NOTRUN -> [FAIL][2]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-tgl-u2/igt@runner@aborted.html
#### Suppressed ####
The following results come from untrusted machines, tests, or statuses.
They do not affect the overall result.
* igt@runner@aborted:
- {fi-tgl-dsi}: NOTRUN -> [FAIL][3]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-tgl-dsi/igt@runner@aborted.html
Known issues
------------
Here are the changes found in Patchwork_18195 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy:
- fi-icl-u2: [PASS][4] -> [DMESG-WARN][5] ([i915#1982])
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-icl-u2/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy.html
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-icl-u2/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy.html
#### Possible fixes ####
* igt@i915_module_load@reload:
- fi-icl-y: [DMESG-WARN][6] ([i915#1982]) -> [PASS][7]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-icl-y/igt@i915_module_load@reload.html
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-icl-y/igt@i915_module_load@reload.html
* igt@i915_pm_rpm@module-reload:
- fi-cml-s: [DMESG-WARN][8] ([i915#1982]) -> [PASS][9]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-cml-s/igt@i915_pm_rpm@module-reload.html
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-cml-s/igt@i915_pm_rpm@module-reload.html
* igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1:
- fi-icl-u2: [DMESG-WARN][10] ([i915#1982]) -> [PASS][11]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-icl-u2/igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1.html
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-icl-u2/igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1.html
#### Warnings ####
* igt@i915_pm_rpm@module-reload:
- fi-kbl-x1275: [SKIP][12] ([fdo#109271]) -> [DMESG-FAIL][13] ([i915#62])
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-kbl-x1275/igt@i915_pm_rpm@module-reload.html
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-kbl-x1275/igt@i915_pm_rpm@module-reload.html
* igt@kms_force_connector_basic@force-edid:
- fi-kbl-x1275: [DMESG-WARN][14] ([i915#62] / [i915#92]) -> [DMESG-WARN][15] ([i915#62] / [i915#92] / [i915#95]) +4 similar issues
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-kbl-x1275/igt@kms_force_connector_basic@force-edid.html
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-kbl-x1275/igt@kms_force_connector_basic@force-edid.html
* igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
- fi-kbl-x1275: [DMESG-WARN][16] ([i915#1982] / [i915#62] / [i915#92]) -> [DMESG-WARN][17] ([i915#62] / [i915#92])
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8757/fi-kbl-x1275/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a.html
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/fi-kbl-x1275/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a.html
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
[i915#62]: https://gitlab.freedesktop.org/drm/intel/issues/62
[i915#92]: https://gitlab.freedesktop.org/drm/intel/issues/92
[i915#95]: https://gitlab.freedesktop.org/drm/intel/issues/95
Participating hosts (45 -> 40)
------------------------------
Missing (5): fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-byt-clapper
Build changes
-------------
* Linux: CI_DRM_8757 -> Patchwork_18195
CI-20190529: 20190529
CI_DRM_8757: 6802049b80a49f5f45c2bc2dd3e6d189204dc2bb @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_5738: bc8b56fe177af34fbde7b96f1f66614a0014c6ef @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
Patchwork_18195: 58fd410689871a4b20807aaa0c31c49baa7c2f1f @ git://anongit.freedesktop.org/gfx-ci/linux
== Linux commits ==
58fd41068987 drm/i915/gt: Wait for aux invalidation on Tigerlake
31bcd6fa4b2a drm/i915/selftests: Add compiler paranoia for checking HWSP values
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_18195/index.html
[-- Attachment #1.2: Type: text/html, Size: 7311 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply
* [PATCH] memsize: Make get_ram_size() work with arbitary RAM size
From: Heinrich Schuchardt @ 2020-07-16 21:04 UTC (permalink / raw)
To: u-boot
In-Reply-To: <20200714160916.66735240702@gemini.denx.de>
On 7/14/20 6:09 PM, Wolfgang Denk wrote:
> Dear Heinrich,
>
> In message <53dad1c7-7684-f975-1567-6ec5e03fa4b6@gmx.de> you wrote:
>>
>> If we want a fast algorithm to determine the last supported address even
>> if the start or size is not a power of two:
>
> Are you sure? How is this supposed to work?
>
> Running it with start = 0 and size = 0xC0000000 it will test the
> memory locations
>
> 0x80000000
> 0xA0000000
> 0xB0000000
> 0xB8000000
> 0xBC000000
> 0xBE000000
> 0xBF000000
> 0xBF800000
> 0xBFC00000
> 0xBFE00000
> 0xBFF00000
> 0xBFF80000
> 0xBFFC0000
> 0xBFFE0000
> 0xBFFF0000
> 0xBFFF8000
> 0xBFFFC000
> 0xBFFFE000
> 0xBFFFF000
> 0xBFFFF800
> 0xBFFFFC00
> 0xBFFFFE00
> 0xBFFFFF00
> 0xBFFFFF80
> 0xBFFFFFC0
> 0xBFFFFFE0
> 0xBFFFFFF0
> 0xBFFFFFF8
> 0xBFFFFFFC
> 0xBFFFFFFE
> 0xBFFFFFFF
The last accessible byte is at 0xBFFFFFFF which matches (start + size -
1) in your example.
The difference to the current logic is that it does not require start or
size to be power of two.
If the size input is larger than the actually accessible memory size,
you will get the actual memory size, e.g. lets assume that the last
accessible address is 0x3333:
int test(unsigned long addr)
{
int ret = addr < 0x3333;
printf("0x%lx - %s\n", addr, ret ? "success": "failed");
return ret;
}
unsigned long start = 0x0000555UL;
unsigned long size = 0xC0000013L;
The series of test locations will be:
0x8000000 - failed
0x4000000 - failed
0x2000000 - failed
0x1000000 - failed
0x800000 - failed
0x400000 - failed
0x200000 - failed
0x100000 - failed
0x80000 - failed
0x40000 - failed
0x20000 - failed
0x10000 - failed
0x8000 - failed
0x4000 - failed
0x2000 - success
0x3000 - success
0x3800 - failed
0x3400 - failed
0x3200 - success
0x3300 - success
0x3380 - failed
0x3340 - failed
0x3320 - success
0x3330 - success
0x3338 - failed
0x3334 - failed
0x3332 - success
0x3333 - failed
Now the algorithm returns that the last accessible memory location is
0x3332.
With unsigned long start = 0x0000555UL;
unsigned long size = 0x800L;
0x800 - success
0xc00 - success
0xd00 - success
0xd40 - success
0xd50 - success
0xd54 - success
The last accessible memory location is 0xd54 (0x555 + 0x800 - 0x1).
The algorithm runs in O(log(UINT_MAX)) which is the same time complexity
as for the current algorithm.
Best regards
Heinrich
>
> What do you intend with such a sequence?
>
> Best regards,
>
> Wolfgang Denk
>
^ permalink raw reply
* Re: qemu test-qga failure on mergebuild after VERSION file change: dependency issues??
From: Philippe Mathieu-Daudé @ 2020-07-16 21:03 UTC (permalink / raw)
To: Peter Maydell, Michael Roth; +Cc: Paolo Bonzini, QEMU Developers
In-Reply-To: <CAFEAcA9ejQjNMcChrGH4RMt9pERQpUx2titK527s+PQCmVUEkA@mail.gmail.com>
On 7/16/20 10:15 PM, Peter Maydell wrote:
> On Thu, 16 Jul 2020 at 20:52, Michael Roth <mdroth@linux.vnet.ibm.com> wrote:
>> But is it intermittent, environment-dependent? I'm trying to understand how to
>> replicate Peter's result since it seems like it would be straightforward
>> reproducer.
>
> I blew away all my build trees and recreated them from
> scratch, and the issue went away. I'm suspicious that the
> complete lack of .d files was induced by a failed earlier
> pullreq attempt and left the build tree in a messed up state
> where it wouldn't notice that it needed to rebuild files.
If it happens again, can you try to revert aaa1b70a0b ("Makefile:
simplify MINIKCONF rules") on top of the tag you are testing, and
re-run the testing?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.