* Re: Question/Problem: Mounting of XFS filesystem takes a lot of time
From: Dave Chinner @ 2010-10-19 23:08 UTC (permalink / raw)
To: Andreas Hasenkopf; +Cc: xfs
In-Reply-To: <1287521058.2000.21.camel@x200>
On Tue, Oct 19, 2010 at 10:44:18PM +0200, Andreas Hasenkopf wrote:
> Hello,
> since today I have quite a big problem and I'd like to get some opinions
> from experts on how to proceed in solving the problem.
>
> The Problem:
> Since today mounting and unmounting of any XFS partition on my system
> takes (depending on size of partition) up to 15 minutes and more.
How big is the journal? (xfs_info output will tell us).
> The system in its current configuration was working fine for over 6
> months and I did not change anything recently with respect of upgrades.
>
> I have already tried booting a Linux-LiveCD and mounting the XFS
> partitions in that system: same result.
> Also I have run xfs_check and xfs_repair -n on the partitions on the
> RAID1 configuration: mounting still takes a lot of time (15 mins +).
Was it a clean unmount, or is it doing journal replay?
If you do:
# mount <dev> <mtpt> ; umount <mtpt> ; time mount <dev> <mtpt>
Does the second mount take 15 minutes?
> And now to my question:
> Could my problem be a software related problem (Linux, XFS, something
> else) or should I consider hardware problems?
Sounds suspiciously like failing hardware to me. Any errors in
dmesg?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply
* Bug#595265: linux-image-2.6.32-5-686: Nerwork card fails to come up again after suspend
From: Francois Romieu @ 2010-10-19 23:06 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Ben Hutchings, Arnout Boelens, 595265, netdev
In-Reply-To: <1287442257.2361.11.camel@edumazet-laptop>
Eric Dumazet <eric.dumazet@gmail.com> :
> Le lundi 18 octobre 2010 à 23:45 +0200, Francois Romieu a écrit :
[...]
> > Something like the patch made by Stanislaw at :
> > https://bugzilla.redhat.com/show_bug.cgi?id=502974
> >
>
> Seems to be down at this moment, patch is also here (and included in
> 2.6.36-rc8 ) :
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=aeb19f6052b5e5c8a24aa444fbff73b84341beac
It's from the same author but it is not included in 2.6.36-stg yet :
Subject: [PATCH] r8169 init phy when resume
For unknown reasons some r8169 devices after resume operate at
100 Mb/s speed instead of 1000 Mb/s like before suspend. To fix
reset phy during resume.
Signed-off-by: Stanislaw Gruszka <sgruszka@redhat.com>
---
drivers/net/r8169.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 17382c3..a4ce127 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c
@@ -4847,6 +4847,9 @@ static int rtl8169_resume(struct device *device)
{
struct pci_dev *pdev = to_pci_dev(device);
struct net_device *dev = pci_get_drvdata(pdev);
+ struct rtl8169_private *tp = netdev_priv(dev);
+
+ rtl8169_init_phy(dev, tp);
if (!netif_running(dev))
goto out;
--
1.7.1
^ permalink raw reply related
* Re: [RFC v2 0/7] OMAP4: mux: Add the OMAP4430 ES1 & ES2 support
From: Tony Lindgren @ 2010-10-19 23:06 UTC (permalink / raw)
To: Benoit Cousson; +Cc: linux-omap, nm
In-Reply-To: <1287526956-21853-1-git-send-email-b-cousson@ti.com>
* Benoit Cousson <b-cousson@ti.com> [101019 15:14]:
> Hi Tony,
>
> Upon Nishanth request, here is the updated version...
>
> It takes into account your proposal to store partition
> information in a partition structure instead of inside every pad entries.
> The mechanism relies on the uniqueness of the pad name in each partition to
> find the correct partition during iteration.
OK, using the offset defines won't be unique necessarily..
> Removed as well some cpu_is_xxx calls from the core code by adding a couple of
> flags during partition init.
That's great.
> Please note that due to low level access to mux api from the RX51 board code,
> I have to disable the mux change to build properly with omap2plus_defconfig.
> Look like you do have some idea to fix that :-)
> I was thinking of:
> - brute force approach that consist of keeping all the data after init, and
> thus allowing removing the __init in the omap_mux_init_signal API
> - or adding some flag in the pad entry to keep them after init .
Well I was thinking we just register a board specific dynamic mux table
and then call that when hitting retention or off and restore after that.
Still need to think about that a bit more, I don't know if we want
the automatic muxing of gpio pins still in addition to that.
> Boot tested on SDP4430 with ES1.0 & ES2.0 with omap2plus_defconfig. Still require
> some real driver to use that mux API in order to validate it.
You can test by echoing wrong values after booting via /sys to
mess up the MMC data lines and then type sync :)
> The series is on top of lo/master (2.6.36-rc8) and is available here:
> git://gitorious.org/omap-pm/linux.git ctrl-wip/mux-omap4-v2
For pulling anything in, the git branches need to be based on something
that's immutable. So preferrably v2.6.35 or some recent -rc tag
in this case. Or rebase it on v2.6.36 after that's been released.
Regards,
Tony
^ permalink raw reply
* Re: [Xen-devel] Re: [PATCH v4 00/10] xen: initial domain support
From: Greg KH @ 2010-10-19 23:01 UTC (permalink / raw)
To: Jeremy Fitzhardinge
Cc: Stefano Stabellini, Jeremy Fitzhardinge,
xen-devel@lists.xensource.com, linux-kernel@vger.kernel.org
In-Reply-To: <4CBE139E.3060004@goop.org>
On Tue, Oct 19, 2010 at 02:54:38PM -0700, Jeremy Fitzhardinge wrote:
> On 10/19/2010 02:14 PM, Greg KH wrote:
> > On Tue, Oct 19, 2010 at 12:16:31PM +0100, Stefano Stabellini wrote:
> >> Hi all,
> >> this series implements the basic support needed to boot Linux as initial
> >> domain on Xen: the target is not to add full featured dom0 support in
> >> the kernel but to be able to boot Linux on Xen on native.
> > Nice, but what real use is this?
> >
> > I thought people wanted dom0 support, this doesn't seem to give them
> > that. Is that still the end-goal here, and this is merely a
> > stepping-stone to get there?
>
> Yes, it's a significant step there. Once that's merged the main missing
> piece is backend drivers, which we can probably get into a mergable form
> for the next window. And in the meantime, they're a fairly
> self-contained thing to maintain out of tree (like other drivers).
Great.
> Our strategy has been to put together a cluster of patch series which
> each have their own intrinsic value, but are also all leading to full
> dom0 support. For example, Stefano's pv-hvm patches are useful for
> running Linux as a fully virtualized domain under Xen, but the reworking
> of the interrupt infrastructure in a way that dom0 support requires.
> Likewise, Konrad's work on pci-passthrough for domU domains adds all the
> machinery required for a Xen domain to have direct access to hardware,
> which is also what dom0 requires.
>
> This particular series introduces the pieces needed for the kernel to
> actually boot up to usermode as dom0, which has some value even if it
> doesn't yet allow you to start new domains (well, you could, but they
> wouldn't have any devices).
>
> The net result is that there will be no massive "Xen dom0" patch series,
> since that has been pretty clearly rejected in the past. Instead full
> dom0 support is being implemented - at least to some extent - as the
> emergent result of a number of Xen-related patches. These patches also
> touch very little code outside of the existing Xen codebase, so there
> shouldn't be much scope for controversy.
Wonderful, nice job, sounds like a valid plan.
greg k-h
^ permalink raw reply
* [Qemu-devel] Re: [PATCH v5 00/14] pcie port switch emulators
From: Wei Xu @ 2010-10-19 23:05 UTC (permalink / raw)
To: Isaku Yamahata, Adnan Khaleel
Cc: skandasa, etmartin, qemu-devel, Michael S. Tsirkin
In-Reply-To: <20101019223915.GD30021@valinux.co.jp>
Isaku,
Good work! One question here: any plan to put q35 and seabios into mainline?
FYI: I have made pcie plus q35 working on qemu-kvm, though on a little bit
old code base.
Wei
On 10/19/10 3:39 PM, "Isaku Yamahata" <yamahata@valinux.co.jp> wrote:
> On Tue, Oct 19, 2010 at 11:07:34AM -0500, Adnan Khaleel wrote:
>> Are all these patches going to be integrated into the current qemu
>> repository?
>
> Eventually. Right now some of them are merged into Michael's pci branch.
> It's up to Michael when he requests Anthony to pull the branch.
>
> thanks,
>
>>
>> -AK
>>
>> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
>> From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]
>> To: Michael S. Tsirkin [mailto:mst@redhat.com]
>> Cc: qemu-devel@nongnu.org, skandasa@cisco.com, etmartin@cisco.com,
>> wexu2@cisco.com, adnan@khaleel.us
>> Sent: Tue, 19 Oct 2010 08:55:29 -0500
>> Subject: Re: [PATCH v5 00/14] pcie port switch emulators
>>
>>>> Isaku Yamahata (14):
>>>> pci: introduce helper functions to test-and-{clear, set} mask in
>>>> configuration space
>>>> pci: introduce helper function to handle msi-x and msi.
>>>> pci: use pci_word_test_and_clear_mask() in pci_device_reset()
>>>> pci/bridge: fix pci_bridge_reset()
>>>> msi: implements msi
>>>> pcie: add pcie constants to pcie_regs.h
>>>> pcie: helper functions for pcie capability and extended capability
>>>
>>> I'll apply these.
>>>
>>>> pcie/aer: helper functions for pcie aer capability
>>>
>>> Maybe move this to the end of the series?
>>>
>>>> pcie port: define struct PCIEPort/PCIESlot and helper functions
>>>> ioh3420: pcie root port in X58 ioh
>>>> x3130: pcie upstream port
>>>> x3130: pcie downstream port
>>>> pcie/hotplug: introduce pushing attention button command
>>>
>>> I think the above can be applied - just remove the dependency
>>> on aer for now.
>>
>> Okay. I'll update the patch series and send it tomorrow.
>>
>> --
>> yamahata
>>
^ permalink raw reply
* Re: [PATCH] tracing: Cleanup the convoluted softirq tracepoints
From: Steven Rostedt @ 2010-10-19 23:05 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Mathieu Desnoyers, Thomas Gleixner, Koki Sanagi, Peter Zijlstra,
Ingo Molnar, Frederic Weisbecker, nhorman, scott.a.mcmillan,
laijs, LKML, eric.dumazet, kaneshige.kenji, David Miller,
izumi.taku, kosaki.motohiro, Heiko Carstens, Luck, Tony,
Jason Baron
In-Reply-To: <4CBE206A.20702@zytor.com>
On Tue, 2010-10-19 at 15:49 -0700, H. Peter Anvin wrote:
> On 10/19/2010 03:41 PM, Mathieu Desnoyers wrote:
> >>
> >> OK, first of all, there are some serious WTFs here:
> >>
> >> # define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
> >>
> >> A jump instruction is one of the worst possible NOPs. Why are we doing
> >> this?
> >
> > This code is dynamically patched at boot time (and module load time) with a
> > better nop, just like the function tracer does.
> >
>
> That's just ridiculous... start out with something sane and you at least
> have the chance of not having to patch it.
Yep we can fix this. Jason?
> > So if we were executing tracepoints in a maze of jumps, we could argue that
> > instruction throughput is the most important there. However, if we expect the
> > common case to be surrounded by some non-ALU instructions, latency tends to
> > become the most important criterion.
> >
> > But I feel I might be missing something important that distinguish "jcc" from
> > "jmp".
>
> NOP has a latency of 0.5-1.0 cycle/insns, *but has no consumers*.
>
> JMP/Jcc does have a consumer -- the IP -- and actually measuring shows
> that it is much, much worse than NOP and other dummy instructions.
But how does JMP vs Jcc compare?
-- Steve
^ permalink raw reply
* Re: scsi_wait_scan doesn't wait
From: Carlos O'Donell @ 2010-10-19 23:04 UTC (permalink / raw)
To: Michal Ludvig; +Cc: linux-parisc
In-Reply-To: <4CBD102F.2040705@logix.net.nz>
On Mon, Oct 18, 2010 at 11:27 PM, Michal Ludvig <mludvig@logix.net.nz> wrote:
> I'm having a problem with 2.6.36-rc8+ on HP PARISC machine (HP 9000
> RP3410) where I intend to wait for SCSI enumeration during boot before
> LVM2 kicks in and attempts to activate the root volume. I modprobe
> scsi_wait_scan in the initramdisk before lvm2 script is run. However the
> modprobe doesn't wait. I'm on Debian testing, fwiw.
If you're interested in helping, the best thing you could do is back
up to a kernel that works and then try to track down what commit broke
the wait scan. It's a bit tedious, but fairly easy.
Cheers,
Carlos.
^ permalink raw reply
* avoid mbox file fragmentation
From: Stan Hoeppner @ 2010-10-19 23:04 UTC (permalink / raw)
To: xfs
In an effort to maximize mbox performance and minimize fragmentation I'm
using these mount options in fstab. This is on a Debian Lenny box but
with vanilla 2.6.34.1 rolled from kernel.org source (Lenny ships with
2.6.26). xfsprogs is 2.9.8.
/dev/sda6 /home xfs defaults,logbufs=8,logbsize=256k,allocsize=1m
Since the actual XFS mount defaults aren't consistently published
anywhere that I can find I'm manually specifying logbufs and logbsize.
I added allocsize=1m as my read of the man page suggests this will
preallocate an additional 1MB of extent space at the end of each mbox
file each time it is written, which I would think should eliminate
fragmentation of these files. However, this doesn't seem to be
eliminating the fragmentation. I added allocsize=1m at a date after all
of the mbox files in question already existed. Does allocsize=1m only
affect new files or does it preallocate at the end of existing files?
I've probably totally misread what allocsize= actually does. Please
educate me. If allocsize= doesn't help prevent fragmentation of mbox
files, what can I do to mitigate this, other than regularly running xfs_fsr?
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda6 xfs 94G 1.3G 92G 2% /home
Filesystem Type Inodes IUsed IFree IUse% Mounted on
/dev/sda6 xfs 94M 1.1K 94M 1% /home
actual 1096, ideal 1011, fragmentation factor 7.76%
I've recently run xfs_fsr thus the low 7% figure. Every couple of weeks
the fragmentation reaches ~30%. I save a lot of list mail, dozens to
hundreds per day for each of about 7 foss mailing lists. As I say, in
just a couple of weeks, these mbox files become fragmented, on the order
of a dozen to a few dozen extents per mbox file. With so much free
space available on this filesystem, why aren't/weren't these files being
spread out with sufficient space between them to prevent fragmentation?
P.S.
(Dave or someone has suggested on list that with newer kernels the
defaults for these two (and other) mount options do not match those
suggested in the man pages. I requested a feature some time ago that
would actually put these default values in /proc files to eliminate any
doubt as to what the actual defaults being used are. I don't recall if
anything came of this. I've seen many an OP get "scolded" on list for
manually specifying values that were apparently equal to the "default"
values as stated by the responding dev. This problem would never exist
if the documentation was complete and consistent. If it already is,
then something is wrong, as myself and hoards of other OPs aren't able
to locate this definitive information regarding mount defaults.)
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply
* Re: [PATCH v5] GPIO: add support for 74x164 serial-in/parallel-out 8-bit shift register
From: Andrew Morton @ 2010-10-19 23:00 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Florian Fainelli, David Brownell, Willy Tarreau, linux-kernel,
Samuel Ortiz, Miguel Gaio, dbrownell, Juhos Gabor
In-Reply-To: <AANLkTinS+bH+u-x0jEcMgN+uK-__65TNNGg6uQA55TSv@mail.gmail.com>
On Tue, 19 Oct 2010 09:26:42 +0200
Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:
> On Mon, Oct 18, 2010 at 9:35 PM, Florian Fainelli <florian@openwrt.org> wrote:
> > From: Miguel Gaio <miguel.gaio@efixo.com>
> >
> > This patch adds support for generic 74x164 serial-in/parallel-out 8-bits
> > shift register. This driver can be used as a GPIO output expander.
> >
> ...
> > +struct gen_74x164_chip {
> > + struct spi_device *spi;
> > + struct gpio_chip gpio_chip;
> > + struct mutex lock;
> > + u8 port_config;
> > +};
> ...
> > +static void gen_74x164_set_value(struct gpio_chip *gc,
> > + unsigned offset, int val)
> > +{
> > + struct gen_74x164_chip *chip = gpio_to_chip(gc);
> > + bool refresh;
> > +
> > + mutex_lock(&chip->lock);
> > + if (val)
> > + chip->port_config |= (1 << offset);
> > + else
> > + chip->port_config &= ~(1 << offset);
>
> set_bit(), clear_bit() ?
They're only to be used on `unsigned long' types, and `port_config' is
u8.
^ permalink raw reply
* Re: PATCH[1/1] Add I2S common driver for Intel MID platform
From: Liam Girdwood @ 2010-10-19 23:01 UTC (permalink / raw)
To: pl bossart
Cc: alsa-devel@alsa-project.org, Mark Brown, Peter Ujfalusi,
Le Gall, Louis
In-Reply-To: <AANLkTim8uEuSYhHyedLn3gbaPQLvwo7Gu+eo-Vo23KvC@mail.gmail.com>
On Tue, 2010-10-19 at 17:48 -0500, pl bossart wrote:
> >> As this is used to transfer speech frames from/to cellular modem that have specific timing constraints (and modem also uses the I2S path to send some side-info), the Alsa SoC interface cannot be used.
> >
> > What makes you say this? You have provided no detail on what problems
> > you believe exist and since ASoC is a *very* thin wrapper around the
> > core ALSA for DMA stuff it is very surprising that you would see any
> > issues here that don't also affect plain ALSA. If there are any issues
> > then it would seem better to resolve these within the current framework
> > rather than completely discarding the entire existing infrastructure.
>
> Louis meant ALSA rather than ASOC. We use burst transfers to reduce
> round-trip delay, the whole notion of periods/period elapsed/ring
> buffer doesn't make sense here and there is in-band signalling that
> should not be interpreted as samples. If ALSA had a simplified API for
> byte streams sent over a serial output, we would have no problems
> using it and of course ASOC would be used then.
Is the inband signaling here done in software ? I'd expect this to be
multiplexed in the DAI by the hardware leaving the software to send PCM
bursts to the CODEC/MODEM/whatever like Peter's DAC33 driver.
Liam
--
Freelance Developer, SlimLogic Ltd
ASoC and Voltage Regulator Maintainer.
http://www.slimlogic.co.uk
^ permalink raw reply
* [PATCH 6/6] IMA: use i_writecount rather than a private counter
From: Eric Paris @ 2010-10-19 22:58 UTC (permalink / raw)
To: linux-kernel, linux-security-module, linux-fsdevel
Cc: hch, zohar, warthog9, david, jmorris, kyle, hpa, akpm, torvalds,
mingo, eparis, viro
In-Reply-To: <20101019225813.12396.2564.stgit@paris.rdu.redhat.com>
IMA tracks the number of struct files which are holding a given inode
readonly and the number white are holding the inode write or r/w. It needs
this information so when a new reader or writer comes in it can tell if
this new file will be able to invalidate results it already made about
existing files.
aka if a task is holding a struct file open RO, IMA measured the file and
recorded those measurements and then a task opens the file RW IMA needs to
note in the logs that the old measurement may not be correct. It's called
a "Time of Measure Time of Use" (ToMToU) issue. The same is true is a RO
file is opened to an inode which has an open writer. We cannot, with any
validity, measure the file in question since it could be changing.
This patch attempts to use the i_writecount field to track writers. The
i_writecount field actually embeds more information in it's value that IMA
needs but it should work for our purposes and allow us to shrink the struct
inode even more.
Signed-off-by: Eric Paris <eparis@redhat.com>
---
include/linux/fs.h | 3 -
security/integrity/ima/ima_iint.c | 3 -
security/integrity/ima/ima_main.c | 95 +++++++++++++------------------------
3 files changed, 34 insertions(+), 67 deletions(-)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 8f46e5b..5dbbf6b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -777,9 +777,8 @@ struct inode {
void *i_security;
#endif
#ifdef CONFIG_IMA
- /* all protected by i_mutex */
+ /* all protected by i_lock */
unsigned int i_readers; /* struct files open RO */
- unsigned int i_writers; /* struct files open WR */
#endif
#ifdef CONFIG_FS_POSIX_ACL
struct posix_acl *i_acl;
diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima_iint.c
index bef6e8f..6e82324 100644
--- a/security/integrity/ima/ima_iint.c
+++ b/security/integrity/ima/ima_iint.c
@@ -118,11 +118,8 @@ void ima_check_counters(struct inode *inode)
{
if (inode->i_readers)
printk(KERN_INFO "%s: readcount: %u\n", __func__, inode->i_readers);
- if (inode->i_writers)
- printk(KERN_INFO "%s: writers: %u\n", __func__, inode->i_writers);
inode->i_readers = 0;
- inode->i_writers = 0;
}
/* iint_free - called when the iint refcount goes to zero */
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index 860e161..668ea93 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -85,48 +85,15 @@ out:
return found;
}
-/* ima_read_write_check - reflect possible reading/writing errors in the PCR.
- *
- * When opening a file for read, if the file is already open for write,
- * the file could change, resulting in a file measurement error.
- *
- * Opening a file for write, if the file is already open for read, results
- * in a time of measure, time of use (ToMToU) error.
- *
- * In either case invalidate the PCR.
- */
-enum iint_pcr_error { TOMTOU, OPEN_WRITERS };
-static void ima_read_write_check(enum iint_pcr_error error,
- struct inode *inode,
- const unsigned char *filename)
-{
- BUG_ON(!mutex_is_locked(&inode->i_mutex));
-
- switch (error) {
- case TOMTOU:
- if (inode->i_readers > 0)
- ima_add_violation(inode, filename, "invalid_pcr",
- "ToMToU");
- break;
- case OPEN_WRITERS:
- if (inode->i_writers > 0)
- ima_add_violation(inode, filename, "invalid_pcr",
- "open_writers");
- break;
- }
-}
-
/*
* Update the counts given an fmode_t
*/
static void ima_inc_counts(struct inode *inode, fmode_t mode)
{
- BUG_ON(!mutex_is_locked(&inode->i_mutex));
+ assert_spin_locked(&inode->i_lock);
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
inode->i_readers++;
- if (mode & FMODE_WRITE)
- inode->i_writers++;
}
/*
@@ -146,11 +113,12 @@ void ima_counts_get(struct file *file)
struct inode *inode = dentry->d_inode;
fmode_t mode = file->f_mode;
int rc;
+ bool send_tomtou = false, send_writers = false;
- if (!iint_initialized || !S_ISREG(inode->i_mode))
+ if (!S_ISREG(inode->i_mode))
return;
- mutex_lock(&inode->i_mutex);
+ spin_lock(&inode->i_lock);
if (!ima_initialized)
goto out;
@@ -160,13 +128,23 @@ void ima_counts_get(struct file *file)
goto out;
if (mode & FMODE_WRITE) {
- ima_read_write_check(TOMTOU, inode, dentry->d_name.name);
+ if (inode->i_readers)
+ send_tomtou = true;
goto out;
}
- ima_read_write_check(OPEN_WRITERS, inode, dentry->d_name.name);
+
+ if (atomic_read(&inode->i_writecount) > 0)
+ send_writers = true;
out:
ima_inc_counts(inode, file->f_mode);
- mutex_unlock(&inode->i_mutex);
+ spin_unlock(&inode->i_lock);
+
+ if (send_tomtou)
+ ima_add_violation(inode, dentry->d_name.name, "invalid_pcr",
+ "ToMToU");
+ if (send_writers)
+ ima_add_violation(inode, dentry->d_name.name, "invalid_pcr",
+ "open_writers");
}
/*
@@ -175,25 +153,18 @@ out:
static void ima_dec_counts(struct inode *inode, struct file *file)
{
mode_t mode = file->f_mode;
- bool dump = false;
- BUG_ON(!mutex_is_locked(&inode->i_mutex));
+ assert_spin_locked(&inode->i_lock);
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ) {
- if (unlikely(inode->i_readers == 0))
- dump = true;
- inode->i_readers--;
- }
- if (mode & FMODE_WRITE) {
- if (unlikely(inode->i_writers == 0))
- dump = true;
- inode->i_writers--;
- }
-
- if (dump && !ima_limit_imbalance(file)) {
- printk(KERN_INFO "%s: open/free imbalance (r:%u w:%u)\n",
- __func__, inode->i_readers, inode->i_writers);
- dump_stack();
+ if (unlikely(inode->i_readers == 0) &&
+ !ima_limit_imbalance(file)) {
+ printk(KERN_INFO "%s: open/free imbalance (r:%u)\n",
+ __func__, inode->i_readers);
+ dump_stack();
+ } else {
+ inode->i_readers--;
+ }
}
}
@@ -204,10 +175,10 @@ static void ima_check_last_writer(struct ima_iint_cache *iint,
mode_t mode = file->f_mode;
BUG_ON(!mutex_is_locked(&iint->mutex));
- BUG_ON(!mutex_is_locked(&inode->i_mutex));
+ assert_spin_locked(&inode->i_lock);
if (mode & FMODE_WRITE &&
- inode->i_writers == 0 &&
+ atomic_read(&inode->i_writecount) <= 0 &&
iint->version != inode->i_version)
iint->flags &= ~IMA_MEASURED;
}
@@ -216,12 +187,12 @@ static void ima_file_free_iint(struct ima_iint_cache *iint, struct inode *inode,
struct file *file)
{
mutex_lock(&iint->mutex);
- mutex_lock(&inode->i_mutex);
+ spin_lock(&inode->i_lock);
ima_dec_counts(inode, file);
ima_check_last_writer(iint, inode, file);
- mutex_unlock(&inode->i_mutex);
+ spin_unlock(&inode->i_lock);
mutex_unlock(&iint->mutex);
kref_put(&iint->refcount, iint_free);
@@ -229,11 +200,11 @@ static void ima_file_free_iint(struct ima_iint_cache *iint, struct inode *inode,
static void ima_file_free_noiint(struct inode *inode, struct file *file)
{
- mutex_lock(&inode->i_mutex);
+ spin_lock(&inode->i_lock);
ima_dec_counts(inode, file);
- mutex_unlock(&inode->i_mutex);
+ spin_unlock(&inode->i_lock);
}
/**
@@ -241,7 +212,7 @@ static void ima_file_free_noiint(struct inode *inode, struct file *file)
* @file: pointer to file structure being freed
*
* Flag files that changed, based on i_version;
- * and decrement the i_readers/i_writers.
+ * and decrement the i_readers.
*/
void ima_file_free(struct file *file)
{
^ permalink raw reply related
* [PATCH 5/6] IMA: use rbtree instead of radix tree for inode information cache
From: Eric Paris @ 2010-10-19 22:58 UTC (permalink / raw)
To: linux-kernel, linux-security-module, linux-fsdevel
Cc: hch, zohar, warthog9, david, jmorris, kyle, hpa, akpm, torvalds,
mingo, eparis, viro
In-Reply-To: <20101019225813.12396.2564.stgit@paris.rdu.redhat.com>
The IMA code needs to store the number of tasks which have an open fd
granting permission to write a file even when IMA is not in use. It needs
this information in order to be enabled at a later point in time without
losing it's integrity garantees. At the moment that means we store a
little bit of data about every inode in a cache. We use a radix tree key'd
on the inode's memory address. Dave Chinner pointed out that a radix tree
is a terrible data structure for such a sparse key space. This patch
switches to using an rbtree which should be more efficient.
Bug report from Dave:
I just noticed that slabtop
was reportingi an awfully high usage of radix tree nodes:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4200331 2778082 66% 0.55K 144839 29 2317424K radix_tree_node
2321500 2060290 88% 1.00K 72581 32 2322592K xfs_inode
2235648 2069791 92% 0.12K 69864 32 279456K iint_cache
That is, 2.7M radix tree nodes are allocated, and the cache itself
is consuming 2.3GB of RAM. I know that the XFS inodei caches are
indexed by radix tree node, but for 2 million cached inodes that
would mean a density of 1 inode per radix tree node, which for a
system with 16M inodes in the filsystems is an impossibly low
density. The worst I've seen in a production system like kernel.org
is about 20-25% density, which would mean about 150−200k radix tree
nodes for that many inodes. So it's not the inode cache.
So I looked up what the iint_cache was. It appears to used for storing
per-inode IMA information, and uses a radix tree for indexing.
It uses the *address* of the struct inode as the indexing key. That
means the key space is extremely sparse - for XFS the struct inode
addresses are approximately 1000 bytes apart, which means the
closest the radix tree index keys get is ~1000. Which means
that there is a single entry per radix tree leaf node, so the radix
tree is using roughly 550 bytes for every 120byte structure being
cached. For the above example, it's probably wasting close to 1GB of
RAM....
Reported-by: Dave Chinner <david@fromorbit.com>
Signed-off-by: Eric Paris <eparis@redhat.com>
---
security/integrity/ima/ima.h | 4 +-
security/integrity/ima/ima_iint.c | 85 ++++++++++++++++++++++++++++---------
2 files changed, 67 insertions(+), 22 deletions(-)
diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h
index 0767717..386026a 100644
--- a/security/integrity/ima/ima.h
+++ b/security/integrity/ima/ima.h
@@ -101,6 +101,8 @@ static inline unsigned long ima_hash_key(u8 *digest)
/* integrity data associated with an inode */
struct ima_iint_cache {
+ struct rb_node rb_node; /* rooted in ima_iint_tree */
+ struct inode *inode; /* back pointer to inode in question */
u64 version; /* track inode changes */
unsigned long flags;
u8 digest[IMA_DIGEST_SIZE];
@@ -120,7 +122,7 @@ int ima_store_template(struct ima_template_entry *entry, int violation,
void ima_template_show(struct seq_file *m, void *e,
enum ima_show_type show);
-/* radix tree calls to lookup, insert, delete
+/* rbtree tree calls to lookup, insert, delete
* integrity data associated with an inode.
*/
struct ima_iint_cache *ima_iint_insert(struct inode *inode);
diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima_iint.c
index 0bab052..bef6e8f 100644
--- a/security/integrity/ima/ima_iint.c
+++ b/security/integrity/ima/ima_iint.c
@@ -12,21 +12,48 @@
* File: ima_iint.c
* - implements the IMA hooks: ima_inode_alloc, ima_inode_free
* - cache integrity information associated with an inode
- * using a radix tree.
+ * using a rbtree tree.
*/
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/spinlock.h>
-#include <linux/radix-tree.h>
+#include <linux/rbtree.h>
#include "ima.h"
-RADIX_TREE(ima_iint_store, GFP_ATOMIC);
+static struct rb_root ima_iint_tree = RB_ROOT;
DEFINE_SPINLOCK(ima_iint_lock);
static struct kmem_cache *iint_cache __read_mostly;
int iint_initialized = 0;
-/* ima_iint_find_get - return the iint associated with an inode
+/*
+ * __ima_iint_find - return the iint associated with an inode
+ *
+ * The caller must hold either the rcu_read_lock or the ima_iint_lock
+ */
+static struct ima_iint_cache *__ima_iint_find(struct inode *inode)
+{
+ struct ima_iint_cache *iint;
+ struct rb_node *n = ima_iint_tree.rb_node;
+
+ while (n) {
+ iint = rb_entry(n, struct ima_iint_cache, rb_node);
+
+ if (inode < iint->inode)
+ n = n->rb_left;
+ else if (inode > iint->inode)
+ n = n->rb_right;
+ else
+ break;
+ }
+ if (!n)
+ return NULL;
+
+ return iint;
+}
+
+/*
+ * ima_iint_find_get - return the iint associated with an inode
*
* ima_iint_find_get gets a reference to the iint. Caller must
* remember to put the iint reference.
@@ -36,12 +63,11 @@ struct ima_iint_cache *ima_iint_find_get(struct inode *inode)
struct ima_iint_cache *iint;
rcu_read_lock();
- iint = radix_tree_lookup(&ima_iint_store, (unsigned long)inode);
- if (!iint)
- goto out;
- kref_get(&iint->refcount);
-out:
+ iint = __ima_iint_find(inode);
+ if (iint)
+ kref_get(&iint->refcount);
rcu_read_unlock();
+
return iint;
}
@@ -51,24 +77,39 @@ out:
*/
int ima_inode_alloc(struct inode *inode)
{
- struct ima_iint_cache *iint = NULL;
+ struct rb_node **p;
+ struct rb_node *new_node, *parent = NULL;
+ struct ima_iint_cache *new_iint, *test_iint;
int rc = 0;
- iint = kmem_cache_alloc(iint_cache, GFP_NOFS);
- if (!iint)
+ new_iint = kmem_cache_alloc(iint_cache, GFP_NOFS);
+ if (!new_iint)
return -ENOMEM;
- rc = radix_tree_preload(GFP_NOFS);
- if (rc < 0)
- goto out;
+ new_iint->inode = inode;
+ new_node = &new_iint->rb_node;
spin_lock(&ima_iint_lock);
- rc = radix_tree_insert(&ima_iint_store, (unsigned long)inode, iint);
- spin_unlock(&ima_iint_lock);
- radix_tree_preload_end();
+
+ p = &ima_iint_tree.rb_node;
+ while (*p) {
+ parent = *p;
+ test_iint = rb_entry(parent, struct ima_iint_cache, rb_node);
+
+ if (inode < test_iint->inode)
+ p = &(*p)->rb_left;
+ else if (inode > test_iint->inode)
+ p = &(*p)->rb_right;
+ else {
+ rc = -EEXIST;
+ goto out;
+ }
+ }
+
+ rb_link_node(new_node, parent, p);
+ rb_insert_color(new_node, &ima_iint_tree);
out:
- if (rc < 0)
- kmem_cache_free(iint_cache, iint);
+ spin_unlock(&ima_iint_lock);
return rc;
}
@@ -113,7 +154,9 @@ void ima_inode_free(struct inode *inode)
struct ima_iint_cache *iint;
spin_lock(&ima_iint_lock);
- iint = radix_tree_delete(&ima_iint_store, (unsigned long)inode);
+ iint = __ima_iint_find(inode);
+ if (iint)
+ rb_erase(&iint->rb_node, &ima_iint_tree);
spin_unlock(&ima_iint_lock);
if (iint)
call_rcu(&iint->rcu, iint_rcu_free);
^ permalink raw reply related
* [PATCH 4/6] IMA: only allocate iint when needed
From: Eric Paris @ 2010-10-19 22:58 UTC (permalink / raw)
To: linux-kernel, linux-security-module, linux-fsdevel
Cc: hch, zohar, warthog9, david, jmorris, kyle, hpa, akpm, torvalds,
mingo, eparis, viro
In-Reply-To: <20101019225813.12396.2564.stgit@paris.rdu.redhat.com>
IMA always allocates an integrity structure to hold information about every
inode, but only needed this structure to tract the number of readers and
writers currently accessing a given inode. Since that information was moved
into struct inode instead of the integrity struct this patch stops allocating
the integrity stucture until it is needed. Thurs greatly reducing memory
usage.
Signed-off-by: Eric Paris <eparis@redhat.com>
---
include/linux/ima.h | 6 ---
security/integrity/ima/ima.h | 1
security/integrity/ima/ima_api.c | 2 -
security/integrity/ima/ima_iint.c | 4 --
security/integrity/ima/ima_main.c | 87 ++++++++++++++++++++++++++-----------
security/security.c | 4 --
6 files changed, 65 insertions(+), 39 deletions(-)
diff --git a/include/linux/ima.h b/include/linux/ima.h
index 1c4bdb9..72644a5 100644
--- a/include/linux/ima.h
+++ b/include/linux/ima.h
@@ -15,7 +15,6 @@ struct linux_binprm;
#ifdef CONFIG_IMA
extern int ima_bprm_check(struct linux_binprm *bprm);
-extern int ima_inode_alloc(struct inode *inode);
extern void ima_inode_free(struct inode *inode);
extern int ima_file_check(struct file *file, int mask);
extern void ima_file_free(struct file *file);
@@ -29,11 +28,6 @@ static inline int ima_bprm_check(struct linux_binprm *bprm)
return 0;
}
-static inline int ima_inode_alloc(struct inode *inode)
-{
- return 0;
-}
-
static inline void ima_inode_free(struct inode *inode)
{
return;
diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h
index e500fe3..0767717 100644
--- a/security/integrity/ima/ima.h
+++ b/security/integrity/ima/ima.h
@@ -70,6 +70,7 @@ int ima_init(void);
void ima_cleanup(void);
int ima_fs_init(void);
void ima_fs_cleanup(void);
+int ima_inode_alloc(struct inode *inode);
int ima_add_template_entry(struct ima_template_entry *entry, int violation,
const char *op, struct inode *inode);
int ima_calc_hash(struct file *file, char *digest);
diff --git a/security/integrity/ima/ima_api.c b/security/integrity/ima/ima_api.c
index 52015d0..44f779f 100644
--- a/security/integrity/ima/ima_api.c
+++ b/security/integrity/ima/ima_api.c
@@ -116,7 +116,7 @@ int ima_must_measure(struct ima_iint_cache *iint, struct inode *inode,
{
int must_measure;
- if (iint->flags & IMA_MEASURED)
+ if (iint && (iint->flags & IMA_MEASURED))
return 1;
must_measure = ima_match_policy(inode, function, mask);
diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima_iint.c
index d327840..0bab052 100644
--- a/security/integrity/ima/ima_iint.c
+++ b/security/integrity/ima/ima_iint.c
@@ -123,9 +123,7 @@ static void init_once(void *foo)
{
struct ima_iint_cache *iint = foo;
- memset(iint, 0, sizeof *iint);
- iint->version = 0;
- iint->flags = 0UL;
+ memset(iint, 0, sizeof(*iint));
mutex_init(&iint->mutex);
kref_init(&iint->refcount);
}
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index 68ecb43..860e161 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -145,19 +145,17 @@ void ima_counts_get(struct file *file)
struct dentry *dentry = file->f_path.dentry;
struct inode *inode = dentry->d_inode;
fmode_t mode = file->f_mode;
- struct ima_iint_cache *iint;
int rc;
if (!iint_initialized || !S_ISREG(inode->i_mode))
return;
- iint = ima_iint_find_get(inode);
- if (!iint)
- return;
- mutex_lock(&iint->mutex);
+
mutex_lock(&inode->i_mutex);
+
if (!ima_initialized)
goto out;
- rc = ima_must_measure(iint, inode, MAY_READ, FILE_CHECK);
+
+ rc = ima_must_measure(NULL, inode, MAY_READ, FILE_CHECK);
if (rc < 0)
goto out;
@@ -169,21 +167,16 @@ void ima_counts_get(struct file *file)
out:
ima_inc_counts(inode, file->f_mode);
mutex_unlock(&inode->i_mutex);
- mutex_unlock(&iint->mutex);
-
- kref_put(&iint->refcount, iint_free);
}
/*
* Decrement ima counts
*/
-static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
- struct file *file)
+static void ima_dec_counts(struct inode *inode, struct file *file)
{
mode_t mode = file->f_mode;
bool dump = false;
- BUG_ON(!mutex_is_locked(&iint->mutex));
BUG_ON(!mutex_is_locked(&inode->i_mutex));
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ) {
@@ -195,10 +188,6 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
if (unlikely(inode->i_writers == 0))
dump = true;
inode->i_writers--;
- if (inode->i_writers == 0) {
- if (iint->version != inode->i_version)
- iint->flags &= ~IMA_MEASURED;
- }
}
if (dump && !ima_limit_imbalance(file)) {
@@ -208,6 +197,45 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
}
}
+static void ima_check_last_writer(struct ima_iint_cache *iint,
+ struct inode *inode,
+ struct file *file)
+{
+ mode_t mode = file->f_mode;
+
+ BUG_ON(!mutex_is_locked(&iint->mutex));
+ BUG_ON(!mutex_is_locked(&inode->i_mutex));
+
+ if (mode & FMODE_WRITE &&
+ inode->i_writers == 0 &&
+ iint->version != inode->i_version)
+ iint->flags &= ~IMA_MEASURED;
+}
+
+static void ima_file_free_iint(struct ima_iint_cache *iint, struct inode *inode,
+ struct file *file)
+{
+ mutex_lock(&iint->mutex);
+ mutex_lock(&inode->i_mutex);
+
+ ima_dec_counts(inode, file);
+ ima_check_last_writer(iint, inode, file);
+
+ mutex_unlock(&inode->i_mutex);
+ mutex_unlock(&iint->mutex);
+
+ kref_put(&iint->refcount, iint_free);
+}
+
+static void ima_file_free_noiint(struct inode *inode, struct file *file)
+{
+ mutex_lock(&inode->i_mutex);
+
+ ima_dec_counts(inode, file);
+
+ mutex_unlock(&inode->i_mutex);
+}
+
/**
* ima_file_free - called on __fput()
* @file: pointer to file structure being freed
@@ -223,15 +251,12 @@ void ima_file_free(struct file *file)
if (!iint_initialized || !S_ISREG(inode->i_mode))
return;
iint = ima_iint_find_get(inode);
- if (!iint)
- return;
- mutex_lock(&iint->mutex);
- mutex_lock(&inode->i_mutex);
- ima_dec_counts(iint, inode, file);
- mutex_unlock(&iint->mutex);
- mutex_unlock(&inode->i_mutex);
- kref_put(&iint->refcount, iint_free);
+ if (iint)
+ ima_file_free_iint(iint, inode, file);
+ else
+ ima_file_free_noiint(inode, file);
+
}
static int process_measurement(struct file *file, const unsigned char *filename,
@@ -243,11 +268,21 @@ static int process_measurement(struct file *file, const unsigned char *filename,
if (!ima_initialized || !S_ISREG(inode->i_mode))
return 0;
+
+ rc = ima_must_measure(NULL, inode, mask, function);
+ if (rc != 0)
+ return rc;
+retry:
iint = ima_iint_find_get(inode);
- if (!iint)
- return -ENOMEM;
+ if (!iint) {
+ rc = ima_inode_alloc(inode);
+ if (!rc || rc == -EEXIST)
+ goto retry;
+ return rc;
+ }
mutex_lock(&iint->mutex);
+
rc = ima_must_measure(iint, inode, mask, function);
if (rc != 0)
goto out;
diff --git a/security/security.c b/security/security.c
index ad5ca29..04c098d 100644
--- a/security/security.c
+++ b/security/security.c
@@ -331,9 +331,7 @@ int security_inode_alloc(struct inode *inode)
ret = security_ops->inode_alloc_security(inode);
if (ret)
return ret;
- ret = ima_inode_alloc(inode);
- if (ret)
- security_inode_free(inode);
+
return ret;
}
^ permalink raw reply related
* [PATCH 3/6] IMA: use unsigned int instead of long for counters
From: Eric Paris @ 2010-10-19 22:58 UTC (permalink / raw)
To: linux-kernel, linux-security-module, linux-fsdevel
Cc: hch, zohar, warthog9, david, jmorris, kyle, hpa, akpm, torvalds,
mingo, eparis, viro
In-Reply-To: <20101019225813.12396.2564.stgit@paris.rdu.redhat.com>
Currently IMA uses 2 longs in struct inode. To save space (and as it seems
impossible to overflow 32 bits) we switch these to unsigned int. The
switch to unsigned does require slightly different checks for underflow,
but it isn't complex.
Signed-off-by: Eric Paris <eparis@redhat.com>
---
include/linux/fs.h | 4 ++--
security/integrity/ima/ima_iint.c | 4 ++--
security/integrity/ima/ima_main.c | 15 ++++++++++-----
3 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 593bb4d..8f46e5b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -778,8 +778,8 @@ struct inode {
#endif
#ifdef CONFIG_IMA
/* all protected by i_mutex */
- long i_readers; /* struct files open RO */
- long i_writers; /* struct files open WR */
+ unsigned int i_readers; /* struct files open RO */
+ unsigned int i_writers; /* struct files open WR */
#endif
#ifdef CONFIG_FS_POSIX_ACL
struct posix_acl *i_acl;
diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima_iint.c
index 2dc32d6..d327840 100644
--- a/security/integrity/ima/ima_iint.c
+++ b/security/integrity/ima/ima_iint.c
@@ -76,9 +76,9 @@ out:
void ima_check_counters(struct inode *inode)
{
if (inode->i_readers)
- printk(KERN_INFO "%s: readcount: %ld\n", __func__, inode->i_readers);
+ printk(KERN_INFO "%s: readcount: %u\n", __func__, inode->i_readers);
if (inode->i_writers)
- printk(KERN_INFO "%s: writers: %ld\n", __func__, inode->i_writers);
+ printk(KERN_INFO "%s: writers: %u\n", __func__, inode->i_writers);
inode->i_readers = 0;
inode->i_writers = 0;
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index 92235a0..68ecb43 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -181,12 +181,19 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
struct file *file)
{
mode_t mode = file->f_mode;
+ bool dump = false;
+
BUG_ON(!mutex_is_locked(&iint->mutex));
BUG_ON(!mutex_is_locked(&inode->i_mutex));
- if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
+ if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ) {
+ if (unlikely(inode->i_readers == 0))
+ dump = true;
inode->i_readers--;
+ }
if (mode & FMODE_WRITE) {
+ if (unlikely(inode->i_writers == 0))
+ dump = true;
inode->i_writers--;
if (inode->i_writers == 0) {
if (iint->version != inode->i_version)
@@ -194,10 +201,8 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
}
}
- if (((inode->i_readers < 0) ||
- (inode->i_writers < 0)) &&
- !ima_limit_imbalance(file)) {
- printk(KERN_INFO "%s: open/free imbalance (r:%ld w:%ld)\n",
+ if (dump && !ima_limit_imbalance(file)) {
+ printk(KERN_INFO "%s: open/free imbalance (r:%u w:%u)\n",
__func__, inode->i_readers, inode->i_writers);
dump_stack();
}
^ permalink raw reply related
* [PATCH 1/6] IMA: move read/write counters into struct inode
From: Eric Paris @ 2010-10-19 22:58 UTC (permalink / raw)
To: linux-kernel, linux-security-module, linux-fsdevel
Cc: hch, zohar, warthog9, david, jmorris, kyle, hpa, akpm, torvalds,
mingo, eparis, viro
IMA currently alocated an inode integrity structure for every inode in
core. This stucture is about 120 bytes long. Most files however
(especially on a system which doesn't make use of IMA) will never need any
of this space. The problem is that if IMA is enabled we need to know
information about the number of readers and the number of writers for every
inode on the box. At the moment we collect that information in the per
inode iint structure and waste the rest of the space. This patch moves those
counters into the struct inode so we can eventually stop allocating an IMA
integrity structure except when absolutely needed.
This patch does the minimum needed to move the location of the data. Further
cleanups, especially the location of counter updates, may still be possible.
Signed-off-by: Eric Paris <eparis@redhat.com>
---
fs/inode.c | 2 ++
include/linux/fs.h | 6 +++++
include/linux/ima.h | 5 ++++
security/integrity/ima/ima.h | 3 --
security/integrity/ima/ima_iint.c | 32 +++++++++++--------------
security/integrity/ima/ima_main.c | 48 +++++++++++++++++++++----------------
6 files changed, 54 insertions(+), 42 deletions(-)
diff --git a/fs/inode.c b/fs/inode.c
index 8646433..5d7ce1e 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -24,6 +24,7 @@
#include <linux/mount.h>
#include <linux/async.h>
#include <linux/posix_acl.h>
+#include <linux/ima.h>
/*
* This is needed for the following functions:
@@ -224,6 +225,7 @@ static struct inode *alloc_inode(struct super_block *sb)
void __destroy_inode(struct inode *inode)
{
BUG_ON(inode_has_buffers(inode));
+ ima_check_counters(inode);
security_inode_free(inode);
fsnotify_inode_delete(inode);
#ifdef CONFIG_FS_POSIX_ACL
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 5fb4dfd..3a402b3 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -776,6 +776,12 @@ struct inode {
#ifdef CONFIG_SECURITY
void *i_security;
#endif
+#ifdef CONFIG_IMA
+ /* all protected by i_mutex */
+ long i_readers; /* struct files open RO */
+ long i_writers; /* struct files open WR */
+ long i_opencount; /* total open files (readers + writers) */
+#endif
#ifdef CONFIG_FS_POSIX_ACL
struct posix_acl *i_acl;
struct posix_acl *i_default_acl;
diff --git a/include/linux/ima.h b/include/linux/ima.h
index 975837e..1c4bdb9 100644
--- a/include/linux/ima.h
+++ b/include/linux/ima.h
@@ -21,6 +21,7 @@ extern int ima_file_check(struct file *file, int mask);
extern void ima_file_free(struct file *file);
extern int ima_file_mmap(struct file *file, unsigned long prot);
extern void ima_counts_get(struct file *file);
+extern void ima_check_counters(struct inode *inode);
#else
static inline int ima_bprm_check(struct linux_binprm *bprm)
@@ -58,5 +59,9 @@ static inline void ima_counts_get(struct file *file)
return;
}
+static inline void ima_check_counters(struct inode *inode)
+{
+ return;
+}
#endif /* CONFIG_IMA_H */
#endif /* _LINUX_IMA_H */
diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h
index 3fbcd1d..e500fe3 100644
--- a/security/integrity/ima/ima.h
+++ b/security/integrity/ima/ima.h
@@ -104,9 +104,6 @@ struct ima_iint_cache {
unsigned long flags;
u8 digest[IMA_DIGEST_SIZE];
struct mutex mutex; /* protects: version, flags, digest */
- long readcount; /* measured files readcount */
- long writecount; /* measured files writecount */
- long opencount; /* opens reference count */
struct kref refcount; /* ima_iint_cache reference count */
struct rcu_head rcu;
};
diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima_iint.c
index afba4ae..c584938 100644
--- a/security/integrity/ima/ima_iint.c
+++ b/security/integrity/ima/ima_iint.c
@@ -73,6 +73,20 @@ out:
return rc;
}
+void ima_check_counters(struct inode *inode)
+{
+ if (inode->i_readers)
+ printk(KERN_INFO "%s: readcount: %ld\n", __func__, inode->i_readers);
+ if (inode->i_writers)
+ printk(KERN_INFO "%s: writers: %ld\n", __func__, inode->i_writers);
+ if (inode->i_opencount)
+ printk(KERN_INFO "%s: opencount: %ld\n", __func__, inode->i_opencount);
+
+ inode->i_readers = 0;
+ inode->i_writers = 0;
+ inode->i_opencount = 0;
+}
+
/* iint_free - called when the iint refcount goes to zero */
void iint_free(struct kref *kref)
{
@@ -80,21 +94,6 @@ void iint_free(struct kref *kref)
refcount);
iint->version = 0;
iint->flags = 0UL;
- if (iint->readcount != 0) {
- printk(KERN_INFO "%s: readcount: %ld\n", __func__,
- iint->readcount);
- iint->readcount = 0;
- }
- if (iint->writecount != 0) {
- printk(KERN_INFO "%s: writecount: %ld\n", __func__,
- iint->writecount);
- iint->writecount = 0;
- }
- if (iint->opencount != 0) {
- printk(KERN_INFO "%s: opencount: %ld\n", __func__,
- iint->opencount);
- iint->opencount = 0;
- }
kref_init(&iint->refcount);
kmem_cache_free(iint_cache, iint);
}
@@ -131,9 +130,6 @@ static void init_once(void *foo)
iint->version = 0;
iint->flags = 0UL;
mutex_init(&iint->mutex);
- iint->readcount = 0;
- iint->writecount = 0;
- iint->opencount = 0;
kref_init(&iint->refcount);
}
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index e662b89..a70700b 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -97,18 +97,19 @@ out:
*/
enum iint_pcr_error { TOMTOU, OPEN_WRITERS };
static void ima_read_write_check(enum iint_pcr_error error,
- struct ima_iint_cache *iint,
struct inode *inode,
const unsigned char *filename)
{
+ BUG_ON(!mutex_is_locked(&inode->i_mutex));
+
switch (error) {
case TOMTOU:
- if (iint->readcount > 0)
+ if (inode->i_readers > 0)
ima_add_violation(inode, filename, "invalid_pcr",
"ToMToU");
break;
case OPEN_WRITERS:
- if (iint->writecount > 0)
+ if (inode->i_writers > 0)
ima_add_violation(inode, filename, "invalid_pcr",
"open_writers");
break;
@@ -118,15 +119,15 @@ static void ima_read_write_check(enum iint_pcr_error error,
/*
* Update the counts given an fmode_t
*/
-static void ima_inc_counts(struct ima_iint_cache *iint, fmode_t mode)
+static void ima_inc_counts(struct inode *inode, fmode_t mode)
{
- BUG_ON(!mutex_is_locked(&iint->mutex));
+ BUG_ON(!mutex_is_locked(&inode->i_mutex));
- iint->opencount++;
+ inode->i_opencount++;
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
- iint->readcount++;
+ inode->i_readers++;
if (mode & FMODE_WRITE)
- iint->writecount++;
+ inode->i_writers++;
}
/*
@@ -154,6 +155,7 @@ void ima_counts_get(struct file *file)
if (!iint)
return;
mutex_lock(&iint->mutex);
+ mutex_lock(&inode->i_mutex);
if (!ima_initialized)
goto out;
rc = ima_must_measure(iint, inode, MAY_READ, FILE_CHECK);
@@ -161,12 +163,13 @@ void ima_counts_get(struct file *file)
goto out;
if (mode & FMODE_WRITE) {
- ima_read_write_check(TOMTOU, iint, inode, dentry->d_name.name);
+ ima_read_write_check(TOMTOU, inode, dentry->d_name.name);
goto out;
}
- ima_read_write_check(OPEN_WRITERS, iint, inode, dentry->d_name.name);
+ ima_read_write_check(OPEN_WRITERS, inode, dentry->d_name.name);
out:
- ima_inc_counts(iint, file->f_mode);
+ ima_inc_counts(inode, file->f_mode);
+ mutex_unlock(&inode->i_mutex);
mutex_unlock(&iint->mutex);
kref_put(&iint->refcount, iint_free);
@@ -180,25 +183,26 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
{
mode_t mode = file->f_mode;
BUG_ON(!mutex_is_locked(&iint->mutex));
+ BUG_ON(!mutex_is_locked(&inode->i_mutex));
- iint->opencount--;
+ inode->i_opencount--;
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
- iint->readcount--;
+ inode->i_readers--;
if (mode & FMODE_WRITE) {
- iint->writecount--;
- if (iint->writecount == 0) {
+ inode->i_writers--;
+ if (inode->i_writers == 0) {
if (iint->version != inode->i_version)
iint->flags &= ~IMA_MEASURED;
}
}
- if (((iint->opencount < 0) ||
- (iint->readcount < 0) ||
- (iint->writecount < 0)) &&
+ if (((inode->i_opencount < 0) ||
+ (inode->i_readers < 0) ||
+ (inode->i_writers < 0)) &&
!ima_limit_imbalance(file)) {
printk(KERN_INFO "%s: open/free imbalance (r:%ld w:%ld o:%ld)\n",
- __func__, iint->readcount, iint->writecount,
- iint->opencount);
+ __func__, inode->i_readers, inode->i_writers,
+ inode->i_opencount);
dump_stack();
}
}
@@ -208,7 +212,7 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
* @file: pointer to file structure being freed
*
* Flag files that changed, based on i_version;
- * and decrement the iint readcount/writecount.
+ * and decrement the i_readers/i_writers.
*/
void ima_file_free(struct file *file)
{
@@ -222,8 +226,10 @@ void ima_file_free(struct file *file)
return;
mutex_lock(&iint->mutex);
+ mutex_lock(&inode->i_mutex);
ima_dec_counts(iint, inode, file);
mutex_unlock(&iint->mutex);
+ mutex_unlock(&inode->i_mutex);
kref_put(&iint->refcount, iint_free);
}
^ permalink raw reply related
* [PATCH 2/6] IMA: drop the inode opencount since it isn't needed for operation
From: Eric Paris @ 2010-10-19 22:58 UTC (permalink / raw)
To: linux-kernel, linux-security-module, linux-fsdevel
Cc: hch, zohar, warthog9, david, jmorris, kyle, hpa, akpm, torvalds,
mingo, eparis, viro
In-Reply-To: <20101019225813.12396.2564.stgit@paris.rdu.redhat.com>
The opencount was used to help depugging to make sure that everything which
created a struct file also correctly made the IMA calls. Since we moved
all of that into the VFS this isn't as necessary. We should be able to get
the same amount of debugging out of just the reader and write count.
Signed-off-by: Eric Paris <eparis@redhat.com>
---
include/linux/fs.h | 1 -
security/integrity/ima/ima_iint.c | 3 ---
security/integrity/ima/ima_main.c | 10 +++-------
3 files changed, 3 insertions(+), 11 deletions(-)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 3a402b3..593bb4d 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -780,7 +780,6 @@ struct inode {
/* all protected by i_mutex */
long i_readers; /* struct files open RO */
long i_writers; /* struct files open WR */
- long i_opencount; /* total open files (readers + writers) */
#endif
#ifdef CONFIG_FS_POSIX_ACL
struct posix_acl *i_acl;
diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima_iint.c
index c584938..2dc32d6 100644
--- a/security/integrity/ima/ima_iint.c
+++ b/security/integrity/ima/ima_iint.c
@@ -79,12 +79,9 @@ void ima_check_counters(struct inode *inode)
printk(KERN_INFO "%s: readcount: %ld\n", __func__, inode->i_readers);
if (inode->i_writers)
printk(KERN_INFO "%s: writers: %ld\n", __func__, inode->i_writers);
- if (inode->i_opencount)
- printk(KERN_INFO "%s: opencount: %ld\n", __func__, inode->i_opencount);
inode->i_readers = 0;
inode->i_writers = 0;
- inode->i_opencount = 0;
}
/* iint_free - called when the iint refcount goes to zero */
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index a70700b..92235a0 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -123,7 +123,6 @@ static void ima_inc_counts(struct inode *inode, fmode_t mode)
{
BUG_ON(!mutex_is_locked(&inode->i_mutex));
- inode->i_opencount++;
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
inode->i_readers++;
if (mode & FMODE_WRITE)
@@ -185,7 +184,6 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
BUG_ON(!mutex_is_locked(&iint->mutex));
BUG_ON(!mutex_is_locked(&inode->i_mutex));
- inode->i_opencount--;
if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
inode->i_readers--;
if (mode & FMODE_WRITE) {
@@ -196,13 +194,11 @@ static void ima_dec_counts(struct ima_iint_cache *iint, struct inode *inode,
}
}
- if (((inode->i_opencount < 0) ||
- (inode->i_readers < 0) ||
+ if (((inode->i_readers < 0) ||
(inode->i_writers < 0)) &&
!ima_limit_imbalance(file)) {
- printk(KERN_INFO "%s: open/free imbalance (r:%ld w:%ld o:%ld)\n",
- __func__, inode->i_readers, inode->i_writers,
- inode->i_opencount);
+ printk(KERN_INFO "%s: open/free imbalance (r:%ld w:%ld)\n",
+ __func__, inode->i_readers, inode->i_writers);
dump_stack();
}
}
^ permalink raw reply related
* Re: [PATCH] tracing: Cleanup the convoluted softirq tracepoints
From: H. Peter Anvin @ 2010-10-19 22:57 UTC (permalink / raw)
To: Steven Rostedt
Cc: Jason Baron, Thomas Gleixner, Mathieu Desnoyers, Koki Sanagi,
Peter Zijlstra, Ingo Molnar, Frederic Weisbecker, nhorman,
scott.a.mcmillan, laijs, LKML, eric.dumazet, kaneshige.kenji,
David Miller, izumi.taku, kosaki.motohiro, Heiko Carstens,
Luck, Tony
In-Reply-To: <1287529013.16971.537.camel@gandalf.stny.rr.com>
On 10/19/2010 03:56 PM, Steven Rostedt wrote:
>>
>> There is no excuse for needing the second jump here, obviously...
>
> Now the trick is to tell gcc that.
>
Yes.
-hpa
^ permalink raw reply
* Re: linux-next - multi-omap image fails to boot on omap3/4
From: Tony Lindgren @ 2010-10-19 22:57 UTC (permalink / raw)
To: Gadiyar, Anand; +Cc: linux-omap
In-Reply-To: <AANLkTikYGwH-kQV3b+gqDYOKVvuiXOU=kBDVLy3ryNjR@mail.gmail.com>
* Gadiyar, Anand <gadiyar@ti.com> [101019 11:26]:
> On Tue, Oct 19, 2010 at 11:51 PM, Tony Lindgren <tony@atomide.com> wrote:
> > * Anand Gadiyar <gadiyar@ti.com> [101019 07:41]:
> >> Hi all,
> >>
> >> linux-next, as of 20101019, built with the omap2plus_defconfig fails
> >> to boot on omap3 and omap4. (I've disabled CONFIG_ARCH_OMAP2 or
> >> CONFIG_SWP_EMULATE to get the image to build). Building with only
> >> ARCH_OMAP3 allows the resultant image to boot up on OMAP3. Likewise,
> >> an image built with only ARCH_OMAP4 boots up on OMAP4 boards.
> >>
> >> earlyprintk does not provide any additional prints after
> >> "Uncompressing Linux... done, booting the kernel."
> >>
> >> Any ideas where to look?
> >
> > Hmm I did a quick test merge of linux-omap master and rmk/devel
> > branches and that boots just fine.
> >
> > So it's probably something that already got fixed in rmk/devel
> > but is not yet in next, or something that came from elsewhere,
> > or something that we have in omap-testing branch that's not in
> > for-next for some reason.
>
> I tried bisecting linux-next between v2.6.36-rc8 and HEAD, but
> there's a commit in between that breaks the build. I'll take another
> stab at this in a while.
Looks like current next at 80f8f1f8b33750d954beb386c0c8142d0c01c25c
boots again except on 2430sdp it produces a NULL pointer at ubi_io_write.
Tony
^ permalink raw reply
* Re: [PATCH] tracing: Cleanup the convoluted softirq tracepoints
From: Steven Rostedt @ 2010-10-19 22:56 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Jason Baron, Thomas Gleixner, Mathieu Desnoyers, Koki Sanagi,
Peter Zijlstra, Ingo Molnar, Frederic Weisbecker, nhorman,
scott.a.mcmillan, laijs, LKML, eric.dumazet, kaneshige.kenji,
David Miller, izumi.taku, kosaki.motohiro, Heiko Carstens,
Luck, Tony
In-Reply-To: <4CBE1F5E.8040807@zytor.com>
On Tue, 2010-10-19 at 15:44 -0700, H. Peter Anvin wrote:
> On 10/19/2010 03:38 PM, Jason Baron wrote:
> >
> > I'm not sure I did, the documentation says that we have nop followed by
> > a jmp:
> >
> > +The new code is a 'nopl' followed by a 'jmp'. Thus:
> > +
> > +nopl - 0f 1f 44 00 00 - 5 bytes
> > +jmp - eb 3e - 2 bytes
> >
>
> There is no excuse for needing the second jump here, obviously...
Now the trick is to tell gcc that.
-- Steve
^ permalink raw reply
* [Qemu-devel] Re: [PATCH v5 00/14] pcie port switch emulators
From: Isaku Yamahata @ 2010-10-19 22:55 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: skandasa, etmartin, qemu-devel, wexu2
In-Reply-To: <20101019224048.GA28390@redhat.com>
On Wed, Oct 20, 2010 at 12:40:49AM +0200, Michael S. Tsirkin wrote:
> I think I suggested once a good way to implement this:
>
> - always make the bits w1c
> - after config write:
> if MHR is enabled, and you see that error log is not empty and that bit is 0,
> this means that someone has written 1b.
> so pop the first error from the log, and set bit to 1 if it's not empty.
>
> This way we only touch w1c mask on setup, no tricky range checks.
> Anything wrong with this approach?
Basically okay.
Without the range checks, we need to restore the old value to the register
if 1b isn't written to the bit of the first error.
Probably you prefer restoring it mostly always than the range check.
--
yamahata
^ permalink raw reply
* Re: Committing changes to an ext3_inode structure
From: Greg Freemyer @ 2010-10-19 22:55 UTC (permalink / raw)
To: Andreas Dilger; +Cc: Felipe Franciosi, linux-ext4
In-Reply-To: <B25FFA22-5EBF-472C-B030-22617E4539DE@dilger.ca>
On Tue, Oct 19, 2010 at 5:56 PM, Andreas Dilger <adilger@dilger.ca> wrote:
> On 2010-10-19, at 09:23, Felipe Franciosi wrote:
>> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and evaluate different methods for migrating datablocks for a given inode on the fly.
>>
>> I have created an ioctl call that implements some verifications and invokes a function I wrote in balloc.c for such migration.
>
> Are you aware that there already is the ability to do on-the-fly migration of files in ext4? This was developed quite some time ago and is in ext4 in newer kernels (not sure of the exact version, maybe 2.6.32 or so).
>
> I would strongly recommend that you investigate that code, since it could definitely use some good testing/inspection/enhancement (as needed). It is in fs/ext4/migrate.c, though I'm not sure where the user-space tools are located (maybe e2fsprogs?).
>
> Cheers, Andreas
I'm not familiar with that code, I'm curious what it does?
But there is also the EXT4_IOC_MOVE_EXT ioctl which is called by
e4defrag primarily.
The code is in fs/ext4/move_extent.c
The concept is you allocate new blocks in a donor inode. Once you
have that done, then EXT4_IOC_MOVE_EXT will move the data from the
original data blocks to the new ones and replace the old original
blocks with the new donor blocks.
All done on the fly.
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [refpolicy] load_policy() with upstart on mint 9 fluxbox
From: Justin P. Mattock @ 2010-10-19 22:55 UTC (permalink / raw)
To: refpolicy
I admit it, I'm a total newb at this upstart thing, anyways
I installed all the packages for mint 9 fluxbox, but the policy
is not loading during boot
what/where might I look to read up on this, so I can get the policy
loaded at boot...
Justin P. Mattock
^ permalink raw reply
* load_policy() with upstart on mint 9 fluxbox
From: Justin P. Mattock @ 2010-10-19 22:55 UTC (permalink / raw)
To: selinux, refpolicy
I admit it, I'm a total newb at this upstart thing, anyways
I installed all the packages for mint 9 fluxbox, but the policy
is not loading during boot
what/where might I look to read up on this, so I can get the policy
loaded at boot...
Justin P. Mattock
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.
^ permalink raw reply
* [PATCH 9/9] arch: arm: fixed typos
From: Andrea Gelmini @ 2010-10-19 22:54 UTC (permalink / raw)
To: linux-arm-kernel
In-Reply-To: <alpine.LNX.2.00.1010192224000.6879@pobox.suse.cz>
2010/10/19 Jiri Kosina <jkosina@suse.cz>:
Hi,
and thanks a lot for your reply.
> Could you please refresh it (and probably incorporate Andrea's changes as
> well) and resend it to me?
Well, I already merged all the patches. I don't care about credit, so
they can be commit with Uwe name.
But, some tree had already applied part of my patches.
I guess it would be better to wait for the next push time to re-apply
all of them.
Thanks again,
Andrea
^ permalink raw reply
* Re: [PATCH 9/9] arch: arm: fixed typos
From: Andrea Gelmini @ 2010-10-19 22:54 UTC (permalink / raw)
To: Jiri Kosina
Cc: Nishanth Menon, Leo Chen, Tony Lindgren, Sergio Aguirre,
Russell King, Kevin Hilman, Grazvydas Ignotas,
Uwe Kleine-König, Peter Ujfalusi, Rajendra Nayak,
Maciej Sosnowski, Dan Williams, linux-omap, linux-arm-kernel,
Liam Girdwood, Paul Walmsley, Eric Miao, Linus Walleij,
Scott Branden, Mark Brown, Tero Kristo, Santosh Shilimkar
In-Reply-To: <alpine.LNX.2.00.1010192224000.6879@pobox.suse.cz>
2010/10/19 Jiri Kosina <jkosina@suse.cz>:
Hi,
and thanks a lot for your reply.
> Could you please refresh it (and probably incorporate Andrea's changes as
> well) and resend it to me?
Well, I already merged all the patches. I don't care about credit, so
they can be commit with Uwe name.
But, some tree had already applied part of my patches.
I guess it would be better to wait for the next push time to re-apply
all of them.
Thanks again,
Andrea
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.