Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Werner Almesberger]

[ 3rd try. Apologies to Karim, Thomas, and Roman, who apparently also
  received my previous attempts. For some reason, one of my upstream
  DNS servers decided to send me highly bogus MX records. ]

Karim Yaghmour wrote:
> Might I add that this is part of the problem ... No personal
> offence intended, but there's been _A LOT_ of things said about
> LTT that were based on third-hand account and no direct contact
> with the toolset/code.

Sigh, yes, guilty as charged ...

At least today, I have a good excuse: my cable modem died, and I
couldn't possibly have download things to look at :)

> > As far as kprobes go, then you still need to have some form or another
> > of marking the code for key events, unless you keep maintaining a set
> > of kprobes-able points separately, which really makes it unusable for
> > the rest of us, as the users of LTT have discovered over time (having
> > to create a new patch for every new kernel that comes out.)

Yes, I think you will need some set of "pads" in the code, where you
can attach probes. I'm not sure how many, though. An alternative, at
least in some cases, would be to move such things into separate
functions, so that you could put the probe just at function entry.
Then add a comment that this function isn't supposed to be torn
apart without dire need.

> > Generating new interrupts is simply unacceptable for LTT's functionality.

Absolutely. If I remember correctly, this is in the process of being
addressed in kprobes. You basically have the following choices:

 - if the probe target is an instruction long enough, replace it with
   a jump or call (that's what I think the kprobes folks are working
   on. I remember for sure that they were thinking about it.)
 - if the probe target is in a basic block with enough room after the
   target, see above (needs feedback from compiler or assembler)
 - if all else fails, add some NOPs (i.e. the marker approach)

> I have received very little feedback on this suggestion,

Probably because everybody saw that it was good :-)

> As for the location of ltt trace points, then they are very rarely
> at function boundaries. Here's a classic:
> 		prepare_arch_switch(rq, next);
> 		ltt_ev_schedchange(prev, next);
> 		prev = context_switch(rq, prev, next);

Yes, in some cases, you don't have a choice but to add some marker.

> > Removing this data would require more data for each event to
> > be logged, and require parsing through the trace before reading it in
> > order to obtain markers allowing random access.

So you need seeking, even in the presence of fine-grained control
over what gets traced in the first place ? (As opposed to extracting
the interesting data from the full trace, given that the latter
shouldn't contain too much noise.)

> If I understand you correctly, you are talking about the fact that
> the transport layer's management of the buffers is syncrhonized
> with some user-space entity that consumes the buffers produced
> and talks back to relayfs (albeit indirectly) to let it know that
> said buffers are now available?

Or that they have been consumed. My question is just whether this
kind of aggregation is something you need.

> I have nothing against kprobes. People keep refering to it as if
> it magically made all the related problems go away, and it doesn't.

Yes, I know just too well :-) In umlsim, I have pretty much the
same problems, and the solutions aren't always nice. So far, I've
been lucky enough that I could almost always find a suitable
function entry to abuse.

However, since a kprobes-based mechanism is - in the worst case,
i.e. when needing markup - as good as direct calls to LTT, and gives
you a lot more flexibility if things aren't quite as hostile, I
think it makes sense to focus on such a solution.

> Nothing precludes us to move in this direction once something is
> in the kernel, it's all currently hidden away in a .h, and it would
> be the same with this.

Yup, but you could move even more intelligence outside the kernel.
All you really need in the kernel is a place to put the probe,
plus some debugging information to tell you where you find the
data (the latter possibly combined with gently coercing the
compiler to put it at some accessible place).

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Werner Almesberger wrote:
>  - if the probe target is an instruction long enough, replace it with
>    a jump or call (that's what I think the kprobes folks are working
>    on. I remember for sure that they were thinking about it.)

I heard about this years ago, but I don't know that anything came of
it. I suspect that this is not as simple as it looks and that the
only reliable way to do it is with a trap.

> Probably because everybody saw that it was good :-)

Great, thanks. That's what we'll aim for then. We've already got
the "disable" and "static" implemented, so now we need to figure
out how do we best implement this tagging. IBM's kernel hooks
allowed the NOP solution, so I'm guessing it shouldn't be that
much of a stretch to extend it for marking up the code for kprobes
and friends. I don't know whether this code is still maintained or
not, but I'd like to hear input as to whether this is a good basis,
or whether you're thinking of something like your uml-sim hooks?

> So you need seeking, even in the presence of fine-grained control
> over what gets traced in the first place ? (As opposed to extracting
> the interesting data from the full trace, given that the latter
> shouldn't contain too much noise.)

The problem is that you don't necessarily know beforehand what's
the problem. So here's an actual example:

I had a client who had this box on which a task was always getting
picked up by the OOM killer. Try as they might, the development
team couldn't figure out which part of the code was causing this.
So we put LTT in there and in less than 5 minutes we found the
problem. It turned out that a user-space access to a memory-mapped
FPGA caused an unexpected FP interrupt to occur, and the application
found itself in a recursive signal handler. In this case there was
an application symptom, but it was a hardware problem.

This is just a simple example, but there are plenty of other
examples where a sysadmin will be experiencing some weird
hard to reproduce bugs on some of his systems and he'll spend
a considerable amount of time trying to guess what's happening.
This is especially complicated when there's no indication as to
what's the root of the problem. So at that point being able to
log everything and being able to rapidely browse through it is
critical.

Once you've done such a first trace you _may_ _possibly_ be
able to refine your search requirements and relog with that in
mind, but that's after the fact.

> Or that they have been consumed. My question is just whether this
> kind of aggregation is something you need.

Absolutely. If you're thinking about short 100kb or MBs traces,
then a simpler scheme would be possible. But when we're talking
about GB and 100GBs spaning days, there's got to be a managed
way of doing it.

>>I have nothing against kprobes. People keep refering to it as if
>>it magically made all the related problems go away, and it doesn't.
> 
> 
> Yes, I know just too well :-) In umlsim, I have pretty much the
> same problems, and the solutions aren't always nice. So far, I've
> been lucky enough that I could almost always find a suitable
> function entry to abuse.

Glad you acknowledge as much.

> However, since a kprobes-based mechanism is - in the worst case,
> i.e. when needing markup - as good as direct calls to LTT, and gives
> you a lot more flexibility if things aren't quite as hostile, I
> think it makes sense to focus on such a solution.

You certainly have a lot more experience than I do with that, so
I'd like to solicit your help. As above: what's the best way to
provide this in addition to the static and disable points?

> Yup, but you could move even more intelligence outside the kernel.
> All you really need in the kernel is a place to put the probe,
> plus some debugging information to tell you where you find the
> data (the latter possibly combined with gently coercing the
> compiler to put it at some accessible place).

Right, but then you end up with a mechanism with generalized hooks.
Actually there was a time when LTT was a driver and you could
either build it as a module or keep it built-in. However, when
we published patches to get LTT accepted in 2.5 we were told on
LKML to move LTT into kernel/ and avoid all this driver stuff.
Having it, or parts of it, in the kernel makes it much simpler
and much more likely that the existing ad-hoc tracing code
spreading accross the sources be removed in exchange for a
single agreed upon way of doing things.

It must be said that like I had done with relayfs, the LTT patch
will go through a major redux and I will post the patches for
review like before on LKML.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

2.6.11-rc1-mm1

[Andrew Morton]

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.11-rc1/2.6.11-rc1-mm1/


- Added bk-xfs to the -mm "external trees" lineup.

- Added the Linux Trace Toolkit (and hence relayfs).  Mainly because I
  haven't yet taken as close a look at LTT as I should have.  Probably neither
  have you.

  It needs a bit of work on the kernel<->user periphery, which is not a big
  deal.

  As does relayfs, IMO.  It seems to need some regularised way in which a
  userspace relayfs client can tell relayfs what file(s) to use.  LTT is
  currently using some ghastly stick-a-pathname-in-/proc thing.  Relayfs
  should provide this service.

  relayfs needs a closer look too.  A lot of advanced instrumentation
  projects seem to require it, but none of them have been merged.  Lots of
  people say "use netlink instead" and lots of other people say "err, we think
  relayfs is better".  This is a discussion which needs to be had.

- The 2.6.10-mm3 announcement was munched by the vger filters, sorry.  One of
  the uml patches had an inopportune substring in its name (oh pee tee hyphen
  oh you tee).  Nice trick if you meant it ;)

- Big update to the ext3 extended attribute support.  agruen, tridge and sct
  have been cooking this up for a while.  samba4 proved to be a good
  stress test.

- davej's "2.6 post-Halloween features" document has been added to -mm as
  Documentation/feature-list-2.6.txt in the hope that someone will review it
  and help keep it up-to-date.

- Added FUSE (filesystem in userspace) for people to play with.  Am agnostic
  as to whether it should be merged (haven't read it at all closely yet,
  either), but I am impressed by the amount of care which has obviously gone
  into it.  Opinions sought.




Changes since 2.6.10-mm3:


 linus.patch
 bk-alsa.patch
 bk-arm.patch
 bk-cifs.patch
 bk-cpufreq.patch
 bk-drm-via.patch
 bk-i2c.patch
 bk-ide-dev.patch
 bk-input.patch
 bk-dtor-input.patch
 bk-kbuild.patch
 bk-kconfig.patch
 bk-netdev.patch
 bk-ntfs.patch
 bk-pci.patch
 bk-usb.patch
 bk-xfs.patch

 Latest versions of everyone's bk trees.

-m32r-include-nodemaskh-for-build-fix.patch
-acpi_smp_processor_id-warning-fix.patch
-sn2-trivial-nodemaskh-include-fix.patch
-split-bprm_apply_creds-into-two-functions.patch
-merge-_vm_enough_memorys-into-a-common-helper.patch
-ppc64-fix-iommu-cleanup-regression.patch
-ppc64-rename-perf-counter-register-defines.patch
-dmi_iterate-fix.patch
-arch-i386-kernel-cpu-mtrr-too-many-bits-are-masked-off-from-cr4.patch
-pm-introduce-pm_message_t.patch
-mark-older-power-managment-as-deprecated.patch
-swsusp-device-power-management-fix.patch
-swsusp-properly-suspend-and-resume-all-devices.patch
-m32r-employ-new-kernel-api-abi.patch
-m68k-update-defconfigs-for-2610.patch
-mmc_wbsd-depends-on-isa.patch
-m68k-remove-nowhere-referenced-files.patch
-direct-write-vs-truncate-deadlock.patch
-random-whitespace-cleanups.patch
-random-remove-pool-resizing-sysctl.patch
-cciss-update-to-version-264.patch
-reiserfs-vs-8115-test-adjustment.patch
-export-get_sb_pseudo.patch
-proc_kcore-correct-double-accounting-of-elf_buflen.patch
-remove-intermezzo-maintainers-entry.patch
-3c59x-reload-eeprom-values-at-rmmod-for-needy-cards.patch
-3c59x-remove-eeprom_reset-for-3c905b.patch
-3c59x-add-eeprom_reset-for-3c900-boomerang.patch
-3c59x-pm-fix.patch
-3c59x-missing-pci_disable_device.patch
-3c59x-use-netdev_priv.patch
-3c59x-make-use-of-generic_mii_ioctl.patch
-3c59x-vortex-select-mii.patch
-3c59x-support-more-ethtool_ops.patch
-inux-269-fs-proc-basec-array-size.patch
-linux-269-fs-proc-proc_ttyc-avoid-array.patch
-optimize-prefetch-usage-in-list_for_each_xxx.patch
-signalc-convert-assertion-to-bug_on.patch
-right-severity-level-for-fatal-message.patch
-remove-unused-drivers-char-rio-cdprotoh.patch
-remove-unused-drivers-char-rsf16fmih.patch
-mtd-added-nec-upd29f064115-support.patch
-ide-cd-is-very-noisy.patch
-signedness-fix-in-deadline-ioschedc.patch
-cleanup-virtual-console-selectionc-interface.patch
-warn-about-cli-sti-co-uses-even-on-up.patch
-remove-umsdos-from-tree.patch
-kill-quota_v2c-printk-of-size_t-warning.patch
-silence-numerous-size_t-warnings-in-drivers-acpi-processor_idlec.patch
-make-irda-string-tables-conditional-on-config_irda_debug.patch
-fix-unresolved-mtd-symbols-in-scx200_docflashc.patch
-fix-module_param-type-mismatch-in-drivers-char-n_hdlcc.patch
-drivers-char-misc-cleanups.patch
-pktcdvd-make-two-functions-static.patch
-pktcdvd-grep-friendly-function-prototypes.patch
-pktcdvd-small-documentation-update.patch
-isofs-remove-useless-include.patch
-synaptics-remove-unused-struct-member-variable.patch
-kill-one-if-x-vfreex-usage.patch
-smbfs-make-some-functions-static.patch
-mips-fixed-build-error-about-nec-vr4100-series.patch
-efs-make-a-struct-static-fwd.patch
-fs-ext3-possible-cleanups.patch
-fs-ext2-xattrc-make-ext2_xattr_list-static.patch
-fs-hugetlbfs-inodec-make-4-functions-static.patch
-remove-nr_super-define.patch
-i2o-fix-init-exit-section-usage.patch
-use-modern-format-for-pci-apic-irq-transform-printks.patch
-coda-bounds-checking.patch
-coda-use-list_for_each_entry_safe.patch
-coda-make-global-code-static.patch
-coda-remove-unused-coda_mknod.patch
-coda-rename-coda_psdev-to-coda.patch
-remove-outdated-smbfs-changelog.patch
-update-geerts-address-in-credits.patch
-cputime-introduce-cputime.patch
-cputime-microsecond-based-cputime-for-s390.patch
-4level-swapoff-hang-fix.patch
-snd-intel8x0-ac97-quirk-entries-for-hp-xw6200-xw8000.patch
-igxb-build-fix.patch
-eepro-build-fix.patch
-3c515-warning-fix.patch
-ixgb-whitespace-fix.patch
-fix-expand_stack-smp-race.patch
-ppc-fix-idle-with-interrupts-disabled.patch
-ppc-remove-duplicate-define.patch
-ppc-include-missing-header.patch
-ppc64-move-hotplug-cpu-functions-to-smp_ops.patch
-ppc64-kprobes-breaks-bug-handling.patch
-ppc64-fix-numa-build.patch
-ppc64-enhance-oops-printing.patch
-ppc64-fix-xmon-longjmp-handling.patch
-ppc64-make-xmon-print-bug-warnings.patch
-ppc64-xtime-gettimeofday-can-get-out-of-sync.patch
-ppc64-pci-cleanup.patch
-ppc64-remove-flush_instruction_cache.patch
-ppc64-interrupt-code-cleanup.patch
-ppc64-fix-rtas_set_indicator9005.patch
-ppc64-make-numa-code-handle-unexpected-layouts.patch
-ppc64-semicolon-in-rtasdc.patch
-improved-wait_8254_wraparound.patch
-kprobes-dont-steal-interrupts-from-vm86.patch
-apic-lapic-hanging-problems-on-nforce2-system.patch
-x86_64-work-around-another-aperture-bios-bug-on-opteron.patch
-x86_64-hack-to-disable-clustered-mode-on-amd-systems.patch
-x86_64-updates-for-x86-64-boot-optionstxt.patch
-x86_64-update-defconfig.patch
-x86_64-remove-old-checksumc.patch
-x86_64-fix-sparse-warnings.patch
-x86_64-fix-some-gcc-4-warnings-in-arch-x86_64.patch
-i386-port-missing-cpuid-bits-from-x86-64-to-i386.patch
-i386-amd-dual-core-support-for-i386.patch
-i386-count-both-multi-cores-and-smp-siblings-in.patch
-i386-count-both-multi-cores-and-smp-siblings-in-fix.patch
-i386-export-phys_proc_id.patch
-x86_64-move-memset_io-out-of-line-to-avoid-warnings.patch
-x86_64-fix-ioremap-attribute-restoration-on-i386-and.patch
-x86_64-fix-tlb-reporting-on-k8.patch
-x86_64-change_page_attr-logic-fixes-from-andrea.patch
-x86_64-fix-mptables-printk.patch
-x86_64-add-new-key-syscalls.patch
-x86_64-remove-direct-mem_map-references.patch
-x86_64-remove-check-that-limited-max-number-of-io-apic.patch
-x86_64-prevent-gcc-from-generating-mmx-code-by-mistake.patch
-x86_64-dont-sync-apic-arbs-on-p4s.patch
-x86_64-cleanups-preparing-for-memory-hotplug.patch
-x86_64-remove-unused-prototypes.patch
-x86_64-fix-a-lot-of-broken-white-space-in.patch
-x86_64-fix-signal-fpu-leak-on-i386-and-x86-64.patch
-x86_64-disable-conforming-bit-on-user32_cs-segment.patch
-x86_64-notify-user-of-mce-events.patch
-uml-add-some-pudding.patch
-uml-use-va_end-wherever-va_args-are-used.patch
-uml-split-out-arch-specific-syscalls-from-generic-ones.patch
-uml-three-level-page-table-support.patch
-uml-x86-64-core-support.patch
-uml-x86-64-config-support.patch
-uml-factor-out-register-saving-and-restoring.patch
-uml-x86_64-ptrace-support.patch
-uml-separate-out-signal-reception.patch
-uml-make-a-common-misconfiguration-impossible.patch
-uml-separate-out-the-time-code.patch
-uml-x86-64-headers.patch
-uml-split-out-arch-link-address-definitions.patch
-uml-dont-use-__nr_waitpid-on-arches-which-dont-have-it.patch
-uml-use-va_copy.patch
-uml-code-tidying.patch
-uml-use-for_each_cpu.patch
-uml-2610-ptrace-updates.patch
-uml-add-the-new-syscalls.patch
-uml-64-bit-cleanups.patch
-uml-silence-some-message-from-the-console-driver.patch
-uml-add-a-missing-include.patch
-uml-sparse-annotations.patch
-uml-fix-sys_call_table-syntax.patch
-uml-fix-make-clean.patch
-uml-define-config_input-better.patch
-uml-fix-a-compile-warning.patch
-seclvl-add-missing-dependency.patch
-binfmt_elf-fix-return-error-codes-and-early-corrupt-binary-detection.patch
-fix-setattr-attr_size-locking-for-nfsd.patch
-pcmcia-new-ds-cs-interface.patch
-pcmcia-call-device-drivers-from-ds-not-from-cs.patch
-pcmcia-unify-bind_mtd-and-pcmcia_bind_mtd.patch
-pcmcia-unfiy-bind_device-and-pcmcia_bind_device.patch
-pcmcia-device-model-integration-can-only-be-submitted-under-gpl.patch
-pcmcia-add-pcmcia_devices.patch
-pcmcia-remove-socket_bind_t-use-pcmcia_devices-instead.patch
-pcmcia-remove-internal-module-use-count-use-module_refcount-instead.patch
-pcmcia-set-drivers-owner-field.patch
-pcmcia-move-pcmcia_unregister_client-to-ds.patch
-pcmcia-device-model-integration-can-only-be-submitted-under-gpl-part-2.patch
-pcmcia-use-kref-instead-of-native-atomic-counter.patch
-pcmcia-add-pcmcia_putget_socket.patch
-pcmcia-grab-a-reference-to-the-cs-socket-in-ds.patch
-pcmcia-get-a-reference-to-ds-socket-for-each-pcmcia_device.patch
-pcmcia-add-a-pointer-to-client-in-struct-pcmcia_device.patch
-pcmcia-use-pcmcia_device-in-send_event.patch
-pcmcia-use-pcmcia_device-to-mark-clients-as-stale.patch
-pcmcia-code-moving-in-ds.patch
-pcmcia-use-pcmcia_device-in-register_client.patch
-pcmcia-direct-ordered-unbind-of-devices.patch
-pcmcia-bug-on-dev_list-=-null.patch
-pcmcia-bug-if-clients-are-kept-too-long.patch
-pcmcia-move-struct-client_t-inside-struct-pcmcia_device.patch
-pcmcia-use-driver_find-in-ds.patch
-pcmcia-set_netdev-for-network-devices.patch
-pcmcia-set_netdev-for-wireless-network-devices.patch
-pcmcia-reduce-stack-usage-in-ds_ioctl-randy-dunlap.patch
-pcmcia-add-disable_clkrun-option.patch
-pcmcia-rename-pcmcia-devices.patch
-pcmcia-pd6729-e-mail-update.patch
-pcmcia-pd6729-cleanups.patch
-pcmcia-pd6729-isa_irq-handling.patch
-pcmcia-remove-obsolete-code.patch
-pcmcia-remove-pending_events.patch
-pcmcia-remove-client_attributes.patch
-pcmcia-remove-unneeded-parameter-from-rsrc_mgr.patch
-pcmcia-remove-dev_info-from-client.patch
-pcmcia-remove-mtd-and-bulkmem-replaced-by-pcmciamtd.patch
-pcmcia-per-socket-resource-database.patch
-pcmcia-validate_mem-only-for-non-statically-mapped-sockets.patch
-pcmcia-adjust_io_region-only-for-non-statically-mapped-sockets.patch
-pcmcia-find_io_region-only-for-non-statically-mapped-sockets.patch
-pcmcia-find_mem_region-only-for-non-statically-mapped-sockets.patch
-pcmcia-adjust_-and-release_resources-only-for-non-statically-mapped-sockets.patch
-pcmcia-move-resource-handling-code-only-for-non-statically-mapped-sockets-to-other-file.patch
-pcmcia-make-rsrc_nonstatic-an-independend-module.patch
-pcmcia-allocate-resource-database-per-socket.patch
-pcmcia-remove-typedef.patch
-pcmcia-grab-lock-in-resource_release.patch
-sched-make-preempt_bkl-depend-on-preempt-alone.patch
-use-mmiowb-in-qla1280c.patch
-bug-on-error-handlings-in-ext3-under-i-o-failure.patch
-bug-on-error-handlings-in-ext3-under-i-o-failure-fix.patch
-scsi-aic7xxx-kill-kernel-22-ifdefs.patch

 Merged

+sparc64-nodemask-build-fix.patch

 sparc64 compile fix

+selinux-fix-error-handling-code-for-policy-load.patch

 SELinux fix

+generic-irq-code-missing-export-of-probe_irq_mask.patch

 parisc fix

+infiniband-ipoib-use-correct-static-rate-in-ipoib.patch
+infiniband-mthca-trivial-formatting-fix.patch
+infiniband-mthca-support-rdma-atomic-attributes-in-qp-modify.patch
+infiniband-mthca-clean-up-allocation-mapping-of-hca-context-memory.patch
+infiniband-mthca-add-needed-rmb-in-event-queue-poll.patch
+infiniband-core-remove-debug-printk.patch
+infiniband-make-more-code-static.patch
+infiniband-core-set-byte_cnt-correctly-in-mad-completion.patch
+infiniband-core-add-qp-number-to-work-completion-struct.patch
+infiniband-core-add-node_type-and-phys_state-sysfs-attrs.patch
+infiniband-mthca-clean-up-computation-of-hca-memory-map.patch
+infiniband-core-fix-handling-of-0-hop-directed-route-mads.patch
+infiniband-core-add-more-parameters-to-process_mad.patch
+infiniband-core-add-qp_type-to-struct-ib_qp.patch
+infiniband-core-add-ib_find_cached_gid-function.patch
+infiniband-update-copyrights-for-new-year.patch
+infiniband-ipoib-move-structs-from-stack-to-device-private-struct.patch
+infiniband-core-rename-handle_outgoing_smp.patch

 infiniband updates

+seagate-st3200822as-sata-disk-needs-to-be-in-sil_blacklist-as-well.patch

 SATA blacklist entry

-agpgart-allow-multiple-backends-to-be-initialized-fix.patch
-agpgart-add-bridge-assignment-missed-in-agp_allocate_memory.patch

 Folded into agpgart-allow-multiple-backends-to-be-initialized.patch

+agpgart-add-agp_find_bridge-function.patch
+agpgart-allow-drivers-to-allocate-memory-local-to.patch
-agp-x86_64-build-fix.patch

 More work on the support-multiple-agp-busses patches.

+orphaned-pagecache-memleak-fix.patch

 Fix a weird memory leak on the page LRU.  This isn't right yet.

+mark-page-accessed-in-filemapc-not-quite-right.patch

 Page aging fix

+netpoll-fix-napi-polling-race-on-smp.patch

 netpoll oops fix

+tun-tan-arp-monitor-support.patch

 Make the tun/tap driver play right with ARP monitoring.

+atmel_cs-add-support-lg-lw2100n-wlan-pcmcia-card.patch

 Add firmware support for another wlan card.

+ppc32-fix-mpc8272ads.patch
+ppc32-add-freescale-pq2fads-support.patch

 ppc32 updates

+ppc64-make-hvlpevent_unregisterhandler-work.patch
+ppc64-make-iseries_veth-call-flush_scheduled_work.patch
+ppc64-iommu-avoid-isa-io-space-on-power3.patch

 ppc64 updates

+frv-remove-mandatory-single-step-debugging-diversion.patch
+frv-excess-whitespace-cleanup.patch

 arch/frv updates

+x86_64-i386-increase-command-line-size.patch
+x86_64-add-brackets-to-bitops.patch
+x86_64-move-early-cpu-detection-earlier.patch
+x86_64-disable-uselib-when-possible.patch
+x86_64-optimize-nodemask-operations-slightly.patch
+x86_64-fix-a-bug-in-timer_suspend.patch
+x86-consolidate-code-segment-base-calculation.patch

 x86_64 update

+swsusp-more-small-fixes.patch
+swsusp-dm-use-right-levels-for-device_suspend.patch
+swsusp-update-docs.patch
+acpi-comment-whitespace-updates.patch
+make-suspend-work-with-ioapic.patch
+swsusp-refrigerator-cleanups.patch

 swsusp update

+uml-avoid-null-dereference-in-linec.patch
+uml-readd-config_magic_sysrq-for-uml.patch
+uml-commentary-addition-to-recent-sysemu-fix.patch
+uml-drop-unused-buffer_headh-header-from-hostfs.patch
+uml-delete-unused-header-umnh.patch
+uml-commentary-about-sigwinch-handling-for-consoles.patch
+uml-fail-xterm_open-when-we-have-no-display.patch
+uml-depend-on-usermode-in-drivers-block-kconfig-and-drop-arch-um-kconfig_block.patch
+uml-makefile-simplification-and-correction.patch
+uml-fix-compilation-for-missing-headers.patch
+uml-fix-some-uml-own-initcall-macros.patch
+uml-refuse-to-run-without-skas-if-no-tt-mode-in.patch
+uml-for-ubd-cmdline-param-use-colon-as-delimiter.patch
+uml-allow-free-ubd-flag-ordering.patch
+uml-move-code-from-ubd_user-to-ubd_kern.patch
+uml-fix-and-cleanup-code-in-ubd_kernc-coming-from-ubd_userc.patch
+uml-add-stack-content-to-dumps.patch
+uml-add-stack-addresses-to-dumps.patch
+uml-update-ld-scripts-to-newer-binutils.patch

 UML update

+reintroduce-export_symboltask_nice-for-binfmt_elf32.patch

 s/390 build fix

+csum_and_copy_from_user-gcc4-warning-fixes-m32r-fix.patch

 m32r build fix

+fixups-for-block2mtd.patch

 block2mtd update

+poll-mini-optimisations.patch

 teeny poll() speedup

+file_tableexpand_files-code-cleanup.patch
+file_tableexpand_files-code-cleanup-remove-debug.patch

 code consolidation

+mtrr-size-and-base-debug.patch

 Debug an mtrr bug.

+minor-ext3-speedup.patch

 Reduce ext3 CPU consumption a little.

+move-read-only-and-immutable-checks-into-permission.patch
+factor-out-common-code-around-follow_link-invocation.patch

 Code cleanups/consolidation

+relayfs-doc.patch
+relayfs-common-files.patch
+relayfs-locking-lockless-implementation.patch
+relayfs-headers.patch

 relayfs

+ltt-core-implementation.patch
+ltt-core-headers.patch
+ltt-kconfig-fix.patch
+ltt-kernel-events.patch
+ltt-kernel-events-tidy.patch
+ltt-kernel-events-build-fix.patch
+ltt-fs-events.patch
+ltt-fs-events-tidy.patch
+ltt-ipc-events.patch
+ltt-mm-events.patch
+ltt-net-events.patch
+ltt-architecture-events.patch

 LTT.

+lock-initializer-cleanup-ppc.patch
+lock-initializer-cleanup-m32r.patch
+lock-initializer-cleanup-video.patch
+lock-initializer-cleanup-ide.patch
+lock-initializer-cleanup-sound.patch
+lock-initializer-cleanup-sh.patch
+lock-initializer-cleanup-ppc64.patch
+lock-initializer-cleanup-security.patch
+lock-initializer-cleanup-core.patch
+lock-initializer-cleanup-media-drivers.patch
+lock-initializer-cleanup-networking.patch
+lock-initializer-cleanup-block-devices.patch
+lock-initializer-cleanup-s390.patch
+lock-initializer-cleanup-usermode.patch
+lock-initializer-cleanup-scsi.patch
+lock-initializer-cleanup-sparc.patch
+lock-initializer-cleanup-v850.patch
+lock-initializer-cleanup-i386.patch
+lock-initializer-cleanup-drm.patch
+lock-initializer-cleanup-firewire.patch
+lock-initializer-cleanup-arm26.patch
+lock-initializer-cleanup-m68k.patch
+lock-initializer-cleanup-network-drivers.patch
+lock-initializer-cleanup-mtd.patch
+lock-initializer-cleanup-x86_64.patch
+lock-initializer-cleanup-filesystems.patch
+lock-initializer-cleanup-ia64.patch
+lock-initializer-cleanup-raid.patch
+lock-initializer-cleanup-isdn.patch
+lock-initializer-cleanup-parisc.patch
+lock-initializer-cleanup-sparc64.patch
+lock-initializer-cleanup-arm.patch
+lock-initializer-cleanup-misc-drivers.patch
+lock-initializer-cleanup-alpha.patch
+lock-initializer-cleanup-character-devices.patch
+lock-initializer-cleanup-drivers-serial.patch
+lock-initializer-cleanup-frv.patch

 spinlock and rwlock initialiser clanups

+ext3-ea-revert-cleanup.patch
+ext3-ea-revert-old-ea-in-inode.patch
+ext3-ea-mbcache-cleanup.patch
+ext2-ea-race-in-ext-xattr-sharing-code.patch
+ext3-ea-ext3-do-not-use-journal_release_buffer.patch
+ext3-ea-ext3-factor-our-common-xattr-code-unnecessary-lock.patch
+ext3-ea-ext-no-spare-xattr-handler-slots-needed.patch
+ext3-ea-cleanup-and-prepare-ext3-for-in-inode-xattrs.patch
+ext3-ea-hide-ext3_get_inode_loc-in_mem-option.patch
+ext3-ea-in-inode-extended-attributes-for-ext3.patch

 Big ext3+EA update with various fixes

+fix-race-between-core-dumping-and-exec.patch
+fix-exec-deadlock-when-ptrace-used-inside-the-thread-group.patch
+ptrace-unlocked-access-to-last_siginfo-resending.patch
+clear-false-pending-signal-indication-in-core-dump.patch

 Various ptrace/signal/coredump fixes

+pcmcia-remove-irq_type_time.patch
+pcmcia-ignore-driver-irq-mask.patch
+pcmcia-remove-irq_mask-and-irq_list-parameters-from-pcmcia-drivers.patch
+pcmcia-use-irq_mask-to-mark-irqs-as-unusable.patch
+pcmcia-remove-racy-try_irq.patch
+pcmcia-modify-irq_mask-via-sysfs.patch
+pcmcia-remove-includes-in-rsrc_mgr-which-arent-necessary-any-longer.patch

 pcmcia udpates.

+sched-fix-preemption-race-core-i386.patch
+sched-make-use-of-preempt_schedule_irq-ppc.patch
+sched-make-use-of-preempt_schedule_irq-arm.patch

 CPU scheduler preemption fix

+fbdev-cleanup-broken-edid-fixup-code.patch
+fbcon-catch-blank-events-on-both-device-and-console-level.patch
+fbcon-fix-compile-error.patch
+fbdev-fbmon-cleanup.patch
+i810fb-module-param-fix.patch
+atyfb-fix-module-parameter-descriptions.patch
+radeonfb-fix-init-exit-section-usage.patch
+pxafb-reorder-add_wait_queue-and-set_current_state.patch
+sa1100fb-reorder-add_wait_queue-and-set_current_state.patch
+backlight-add-backlight-lcd-device-basic-support.patch
+fbdev-add-w100-framebuffer-driver.patch

 fbdev/fbcon update

+post-halloween-doc.patch

 davej's 2.6 feature list

+fuse-maintainers-kconfig-and-makefile-changes.patch
+fuse-core.patch
+fuse-device-functions.patch
+fuse-read-only-operations.patch
+fuse-read-write-operations.patch
+fuse-file-operations.patch
+fuse-mount-options.patch
+fuse-extended-attribute-operations.patch
+fuse-readpages-operation.patch
+fuse-nfs-export.patch
+fuse-direct-i-o.patch

 Filesystem in userspace.

+ieee1394-adds-a-disable_irm-option-to-ieee1394ko.patch

 New command line option for firewire.

+fix-typo-in-arch-i386-kconfig.patch

 Fix a tpyo.

+random-whitespace-doh.patch
+random-entropy-debugging-improvements.patch
+random-run-time-configurable-debugging.patch
+random-periodicity-detection-fix.patch
+random-add_input_randomness.patch

 random driver updates

+various-kconfig-fixes.patch

 Fix a huge number of Kconfig typos and brainos.





number of patches in -mm: 434
number of changesets in external trees: 314
number of patches in -mm only: 417
total patches: 731




All 434 patches:


linus.patch

sparc64-nodemask-build-fix.patch
  sparc64: nodemask build fix

selinux-fix-error-handling-code-for-policy-load.patch
  SELinux: fix error handling code for policy load

generic-irq-code-missing-export-of-probe_irq_mask.patch
  generic irq code missing export of probe_irq_mask()

infiniband-ipoib-use-correct-static-rate-in-ipoib.patch
  InfiniBand/IPoIB: use correct static rate in IpoIB

infiniband-mthca-trivial-formatting-fix.patch
  InfiniBand/mthca: trivial formatting fix

infiniband-mthca-support-rdma-atomic-attributes-in-qp-modify.patch
  InfiniBand/mthca: support RDMA/atomic attributes in QP modify

infiniband-mthca-clean-up-allocation-mapping-of-hca-context-memory.patch
  InfiniBand/mthca: clean up allocation mapping of HCA context memory

infiniband-mthca-add-needed-rmb-in-event-queue-poll.patch
  InfiniBand/mthca: add needed rmb() in event queue poll

infiniband-core-remove-debug-printk.patch
  InfiniBand/core: remove debug printk

infiniband-make-more-code-static.patch
  InfiniBand: make more code static

infiniband-core-set-byte_cnt-correctly-in-mad-completion.patch
  InfiniBand/core: set byte_cnt correctly in MAD completion

infiniband-core-add-qp-number-to-work-completion-struct.patch
  InfiniBand/core: add QP number to work completion struct

infiniband-core-add-node_type-and-phys_state-sysfs-attrs.patch
  InfiniBand/core: add node_type and phys_state sysfs attrs

infiniband-mthca-clean-up-computation-of-hca-memory-map.patch
  InfiniBand/mthca: clean up computation of HCA memory map

infiniband-core-fix-handling-of-0-hop-directed-route-mads.patch
  InfiniBand/core: fix handling of 0-hop directed route MADs

infiniband-core-add-more-parameters-to-process_mad.patch
  InfiniBand/core: add more parameters to process_mad

infiniband-core-add-qp_type-to-struct-ib_qp.patch
  InfiniBand/core: add qp_type to struct ib_qp

infiniband-core-add-ib_find_cached_gid-function.patch
  InfiniBand/core: add ib_find_cached_gid function

infiniband-update-copyrights-for-new-year.patch
  InfiniBand: update copyrights for new year

infiniband-ipoib-move-structs-from-stack-to-device-private-struct.patch
  InfiniBand/ipoib: move structs from stack to device private struct

infiniband-core-rename-handle_outgoing_smp.patch
  InfiniBand/core: rename handle_outgoing_smp

ia64-acpi-build-fix.patch
  ia64 acpi build fix

ia64-config_apci_numa-fix.patch
  ia64 CONFIG_APCI_NUMA fix

bk-acpi-revert-20041210.patch
  bk-acpi-revert-20041210

acpi-report-errors-in-fanc.patch
  ACPI: report errors in fan.c

acpi-flush-tlb-when-pagetable-changed.patch
  acpi: flush TLB when pagetable changed

acpi-kfree-fix.patch
  a

bk-alsa.patch

bk-arm.patch

bk-cifs.patch

bk-cpufreq.patch

bk-drm-via.patch

bk-i2c.patch

bk-ide-dev.patch

ide-dev-build-fix.patch
  ide-dev-build-fix

bk-input.patch

bk-dtor-input.patch

alps-touchpad-detection-fix.patch
  ALPS touchpad detection fix

bk-kbuild.patch

bk-kconfig.patch

seagate-st3200822as-sata-disk-needs-to-be-in-sil_blacklist-as-well.patch
  Seagate ST3200822AS SATA disk needs to be in sil_blacklist as well

bk-netdev.patch

bk-ntfs.patch

bk-pci.patch

bk-usb.patch

bk-xfs.patch

mm.patch
  add -mmN to EXTRAVERSION

fix-smm-failures-on-e750x-systems.patch
  fix SMM failures on E750x systems

agpgart-allow-multiple-backends-to-be-initialized.patch
  agpgart: allow multiple backends to be initialized
  agpgart-allow-multiple-backends-to-be-initialized fix
  agpgart: add bridge assignment missed in agp_allocate_memory
  x86_64 agp failure fix

agpgart-add-agp_find_bridge-function.patch
  agpgart: add agp_find_bridge function

agpgart-allow-drivers-to-allocate-memory-local-to.patch
  agpgart: allow drivers to allocate memory local to the bridge

drm-add-support-for-new-multiple-agp-bridge-agpgart-api.patch
  drm: add support for new multiple agp bridge agpgart api

fb-add-support-for-new-multiple-agp-bridge-agpgart-api.patch
  fb: add support for new multiple agp bridge agpgart api

agpgart-add-bridge-parameter-to-driver-functions.patch
  agpgart: add bridge parameter to driver functions

vm-pageout-throttling.patch
  vm: pageout throttling

make-tree_lock-an-rwlock.patch
  make mapping->tree_lock an rwlock

orphaned-pagecache-memleak-fix.patch
  orphaned pagecache memleak fix

mark-page-accessed-in-filemapc-not-quite-right.patch
  mark-page-accessed in filemap.c not quite right

must-fix.patch
  must fix lists update
  must fix list update
  mustfix update
  must-fix update
  mustfix lists

pcnet32-79c976-with-fiber-optic.patch
  pcnet32: 79c976 with fiber optic fix

add-omap-support-to-smc91x-ethernet-driver.patch
  Add OMAP support to smc91x Ethernet driver

restore-net-sched-iptc-after-iptables-kmod-cleanup.patch
  Restore net/sched/ipt.c After iptables Kmod Cleanup

b44-bounce-buffer-fix.patch
  b44 bounce buffering fix

netpoll-fix-napi-polling-race-on-smp.patch
  netpoll: fix NAPI polling race on SMP

tun-tan-arp-monitor-support.patch
  tun/tap ARP monitor support

atmel_cs-add-support-lg-lw2100n-wlan-pcmcia-card.patch
  atmel_cs: Add support LG LW2100N WLAN PCMCIA card

ppc32-fix-mpc8272ads.patch
  ppc32: Fix mpc8272ads

ppc32-add-freescale-pq2fads-support.patch
  ppc32: Add Freescale PQ2FADS support

ppc64-make-hvlpevent_unregisterhandler-work.patch
  ppc64: make HvLpEvent_unregisterHandler() work

ppc64-make-iseries_veth-call-flush_scheduled_work.patch
  ppc64: make iseries_veth call flush_scheduled_work()

ppc64-iommu-avoid-isa-io-space-on-power3.patch
  ppc64: iommu: avoid ISA io space on POWER3

ppc64-reloc_hide.patch

frv-remove-mandatory-single-step-debugging-diversion.patch
  FRV: Remove mandatory single-step debugging diversion

frv-excess-whitespace-cleanup.patch
  FRV: Excess whitespace cleanup

superhyway-bus-support.patch
  SuperHyway bus support

x86_64-i386-increase-command-line-size.patch
  x86_64/i386: increase command line size

x86_64-add-brackets-to-bitops.patch
  x86_64: Add brackets to bitops

x86_64-move-early-cpu-detection-earlier.patch
  x86_64: Move early CPU detection earlier

x86_64-disable-uselib-when-possible.patch
  x86_64: Disable uselib when possible

x86_64-optimize-nodemask-operations-slightly.patch
  x86_64: Optimize nodemask operations slightly

x86_64-fix-a-bug-in-timer_suspend.patch
  Fix a bug in timer_suspend() on x86_64

x86-consolidate-code-segment-base-calculation.patch
  x68: consolidate code segment base calculation

xen-vmm-4-add-ptep_establish_new-to-make-va-available.patch
  Xen VMM #4: add ptep_establish_new to make va available

xen-vmm-4-return-code-for-arch_free_page.patch
  Xen VMM #4: return code for arch_free_page

xen-vmm-4-return-code-for-arch_free_page-fix.patch
  Get rid of arch_free_page() warning

xen-vmm-4-runtime-disable-of-vt-console.patch
  Xen VMM #4: runtime disable of VT console

xen-vmm-4-has_arch_dev_mem.patch
  Xen VMM #4: HAS_ARCH_DEV_MEM

xen-vmm-4-split-free_irq-into-teardown_irq.patch
  Xen VMM #4: split free_irq into teardown_irq

swsusp-more-small-fixes.patch
  swsusp: more small fixes

swsusp-dm-use-right-levels-for-device_suspend.patch
  swsusp/dm: Use right levels for device_suspend()

swsusp-update-docs.patch
  swsusp: update docs

acpi-comment-whitespace-updates.patch
  acpi: comment/whitespace updates

make-suspend-work-with-ioapic.patch
  make suspend work with ioapic

swsusp-refrigerator-cleanups.patch
  swsusp: refrigerator cleanups

uml-avoid-null-dereference-in-linec.patch
  uml: avoid NULL dereference in line.c

uml-readd-config_magic_sysrq-for-uml.patch
  uml: readd CONFIG_MAGIC_SYSRQ for UML

uml-commentary-addition-to-recent-sysemu-fix.patch
  uml: Commentary addition to recent SYSEMU fix.

uml-drop-unused-buffer_headh-header-from-hostfs.patch
  uml: drop unused buffer_head.h header from hostfs

uml-delete-unused-header-umnh.patch
  uml: delete unused header umn.h

uml-commentary-about-sigwinch-handling-for-consoles.patch
  uml: commentary about SIGWINCH handling for consoles

uml-fail-xterm_open-when-we-have-no-display.patch
  uml: fail xterm_open when we have no $DISPLAY

uml-depend-on-usermode-in-drivers-block-kconfig-and-drop-arch-um-kconfig_block.patch
  uml: depend on !USERMODE in drivers/block/Kconfig and drop arch/um/Kconfig_block

uml-makefile-simplification-and-correction.patch
  uml: Makefile simplification and correction.

uml-fix-compilation-for-missing-headers.patch
  uml: fix compilation for missing headers

uml-fix-some-uml-own-initcall-macros.patch
  uml: fix some UML own initcall macros

uml-refuse-to-run-without-skas-if-no-tt-mode-in.patch
  uml: refuse to run without skas if no tt mode in

uml-for-ubd-cmdline-param-use-colon-as-delimiter.patch
  uml: for ubd cmdline param use colon as delimiter

uml-allow-free-ubd-flag-ordering.patch
  uml: allow free ubd flag ordering

uml-move-code-from-ubd_user-to-ubd_kern.patch
  uml: move code from ubd_user to ubd_kern

uml-fix-and-cleanup-code-in-ubd_kernc-coming-from-ubd_userc.patch
  uml: fix and cleanup code in ubd_kern.c coming from ubd_user.c

uml-add-stack-content-to-dumps.patch
  uml: add stack content to dumps

uml-add-stack-addresses-to-dumps.patch
  uml: add stack addresses to dumps

uml-update-ld-scripts-to-newer-binutils.patch
  uml: update ld scripts to newer binutils

reintroduce-export_symboltask_nice-for-binfmt_elf32.patch
  reintroduce task_nice export for binfmt_elf32

wacom-tablet-driver.patch
  wacom tablet driver

force-feedback-support-for-uinput.patch
  Force feedback support for uinput

kmap_atomic-takes-char.patch
  kmap_atomic takes char*

kmap_atomic-takes-char-fix.patch
  kmap_atomic-takes-char-fix

kmap_atomic-fallout.patch
  kmap_atomic fallout

kunmap-fallout-more-fixes.patch
  kunmap-fallout-more-fixes

make-sysrq-f-call-oom_kill.patch
  make sysrq-F call oom_kill()

allow-admin-to-enable-only-some-of-the-magic-sysrq-functions.patch
  Allow admin to enable only some of the Magic-Sysrq functions

sort-out-pci_rom_address_enable-vs-ioresource_rom_enable.patch
  Sort out PCI_ROM_ADDRESS_ENABLE vs IORESOURCE_ROM_ENABLE

csum_and_copy_from_user-gcc4-warning-fixes.patch
  csum_and_copy_from_user gcc4 warning fixes

csum_and_copy_from_user-gcc4-warning-fixes-m32r-fix.patch
  csum_and_copy_from_user-gcc4-warning-fixes m32r fix

smbfs-fixes.patch
  smbfs fixes

irqpoll.patch
  irqpoll

fixups-for-block2mtd.patch
  fixups for block2mtd

poll-mini-optimisations.patch
  poll: mini optimisations

file_tableexpand_files-code-cleanup.patch
  file_table:expand_files() code cleanup

file_tableexpand_files-code-cleanup-remove-debug.patch
  file_tableexpand_files-code-cleanup-remove-debug

mtrr-size-and-base-debug.patch
  mtrr size-and-base debugging

minor-ext3-speedup.patch
  Minor ext3 speedup

move-read-only-and-immutable-checks-into-permission.patch
  move read-only and immutable checks into permission()

factor-out-common-code-around-follow_link-invocation.patch
  factor out common code around ->follow_link invocation

relayfs-doc.patch
  relayfs: doc

relayfs-common-files.patch
  relayfs: common files

relayfs-locking-lockless-implementation.patch
  relayfs: locking/lockless implementation

relayfs-headers.patch
  relayfs: headers

ltt-core-implementation.patch
  ltt: core implementation

ltt-core-headers.patch
  ltt: core headers

ltt-kconfig-fix.patch
  ltt kconfig fix

ltt-kernel-events.patch
  ltt: kernel/ events

ltt-kernel-events-tidy.patch
  ltt-kernel-events tidy

ltt-kernel-events-build-fix.patch
  ltt-kernel-events-build-fix

ltt-fs-events.patch
  ltt: fs/ events

ltt-fs-events-tidy.patch
  ltt-fs-events tidy

ltt-ipc-events.patch
  ltt: ipc/ events

ltt-mm-events.patch
  ltt: mm/ events

ltt-net-events.patch
  ltt: net/ events

ltt-architecture-events.patch
  ltt: architecture events

lock-initializer-cleanup-ppc.patch
  Lock initializer cleanup: PPC

lock-initializer-cleanup-m32r.patch
  Lock initializer cleanup: M32R

lock-initializer-cleanup-video.patch
  Lock initializer cleanup: Video

lock-initializer-cleanup-ide.patch
  Lock initializer cleanup: IDE

lock-initializer-cleanup-sound.patch
  Lock initializer cleanup: sound

lock-initializer-cleanup-sh.patch
  Lock initializer cleanup: SH

lock-initializer-cleanup-ppc64.patch
  Lock initializer cleanup: PPC64

lock-initializer-cleanup-security.patch
  Lock initializer cleanup: Security

lock-initializer-cleanup-core.patch
  Lock initializer cleanup: Core

lock-initializer-cleanup-media-drivers.patch
  Lock initializer cleanup: media drivers

lock-initializer-cleanup-networking.patch
  Lock initializer cleanup: Networking

lock-initializer-cleanup-block-devices.patch
  Lock initializer cleanup: Block devices

lock-initializer-cleanup-s390.patch
  Lock initializer cleanup: S390

lock-initializer-cleanup-usermode.patch
  Lock initializer cleanup: UserMode

lock-initializer-cleanup-scsi.patch
  Lock initializer cleanup: SCSI

lock-initializer-cleanup-sparc.patch
  Lock initializer cleanup: SPARC

lock-initializer-cleanup-v850.patch
  Lock initializer cleanup: V850

lock-initializer-cleanup-i386.patch
  Lock initializer cleanup: I386

lock-initializer-cleanup-drm.patch
  Lock initializer cleanup: DRM

lock-initializer-cleanup-firewire.patch
  Lock initializer cleanup: Firewire

lock-initializer-cleanup-arm26.patch
  Lock initializer cleanup - (ARM26)

lock-initializer-cleanup-m68k.patch
  Lock initializer cleanup: M68K

lock-initializer-cleanup-network-drivers.patch
  Lock initializer cleanup: Network drivers

lock-initializer-cleanup-mtd.patch
  Lock initializer cleanup: MTD

lock-initializer-cleanup-x86_64.patch
  Lock initializer cleanup: X86_64

lock-initializer-cleanup-filesystems.patch
  Lock initializer cleanup: Filesystems

lock-initializer-cleanup-ia64.patch
  Lock initializer cleanup: IA64

lock-initializer-cleanup-raid.patch
  Lock initializer cleanup: Raid

lock-initializer-cleanup-isdn.patch
  Lock initializer cleanup: ISDN

lock-initializer-cleanup-parisc.patch
  Lock initializer cleanup: PARISC

lock-initializer-cleanup-sparc64.patch
  Lock initializer cleanup: SPARC64

lock-initializer-cleanup-arm.patch
  Lock initializer cleanup: ARM

lock-initializer-cleanup-misc-drivers.patch
  Lock initializer cleanup: Misc drivers

lock-initializer-cleanup-alpha.patch
  Lock initializer cleanup - (ALPHA)

lock-initializer-cleanup-character-devices.patch
  Lock initializer cleanup: character devices

lock-initializer-cleanup-drivers-serial.patch
  Lock initializer cleanup: drivers/serial

lock-initializer-cleanup-frv.patch
  Lock initializer cleanup: FRV

ext3-ea-revert-cleanup.patch
  ext3-ea-revert-cleanup

ext3-ea-revert-old-ea-in-inode.patch
  revert old ea-in-inode patch

ext3-ea-mbcache-cleanup.patch
  ext3/EA: mbcache cleanup

ext2-ea-race-in-ext-xattr-sharing-code.patch
  ext3/EA: Race in ext[23] xattr sharing code

ext3-ea-ext3-do-not-use-journal_release_buffer.patch
  ext3/EA: Ext3: do not use journal_release_buffer

ext3-ea-ext3-factor-our-common-xattr-code-unnecessary-lock.patch
  ext3/EA: Ext3: factor our common xattr code; unnecessary lock

ext3-ea-ext-no-spare-xattr-handler-slots-needed.patch
  ext3/EA: Ext[23]: no spare xattr handler slots needed

ext3-ea-cleanup-and-prepare-ext3-for-in-inode-xattrs.patch
  ext3/EA: Cleanup and prepare ext3 for in-inode xattrs

ext3-ea-hide-ext3_get_inode_loc-in_mem-option.patch
  ext3/EA: Hide ext3_get_inode_loc in_mem option

ext3-ea-in-inode-extended-attributes-for-ext3.patch
  ext3/EA: In-inode extended attributes for ext3

speedup-proc-pid-maps.patch
  Speed up /proc/pid/maps

speedup-proc-pid-maps-fix.patch
  Speed up /proc/pid/maps fix

speedup-proc-pid-maps-fix-fix.patch
  speedup-proc-pid-maps fix fix

speedup-proc-pid-maps-fix-fix-fix.patch
  speedup /proc/<pid>/maps(4th version)

inotify.patch
  inotify

ioctl-rework-2.patch
  ioctl rework #2

ioctl-rework-2-fix.patch
  ioctl-rework-2 fix

make-standard-conversions-work-with-compat_ioctl.patch
  make standard conversions work with compat_ioctl.

fget_light-fput_light-for-ioctls.patch
  fget_light/fput_light for ioctls

macros-to-detect-existance-of-unlocked_ioctl-and-ioctl_compat.patch
  macros to detect existance of unlocked_ioctl and ioctl_compat

fix-coredump_wait-deadlock-with-ptracer-tracee-on-shared-mm.patch
  fix coredump_wait deadlock with ptracer & tracee on shared mm

fix-race-between-core-dumping-and-exec.patch
  fix race between core dumping and exec with shared mm

fix-exec-deadlock-when-ptrace-used-inside-the-thread-group.patch
  fix exec deadlock when ptrace used inside the thread group

ptrace-unlocked-access-to-last_siginfo-resending.patch
  ptrace: unlocked access to last_siginfo (resending)

clear-false-pending-signal-indication-in-core-dump.patch
  clear false pending signal indication in core dump

pcmcia-remove-irq_type_time.patch
  pcmcia: remove IRQ_TYPE_TIME

pcmcia-ignore-driver-irq-mask.patch
  pcmcia: ignore driver IRQ mask

pcmcia-remove-irq_mask-and-irq_list-parameters-from-pcmcia-drivers.patch
  pcmcia: remove irq_mask and irq_list parameters from PCMCIA drivers

pcmcia-use-irq_mask-to-mark-irqs-as-unusable.patch
  pcmcia: use irq_mask to mark IRQs as (un)usable

pcmcia-remove-racy-try_irq.patch
  pcmcia: remove racy try_irq()

pcmcia-modify-irq_mask-via-sysfs.patch
  pcmcia: modify irq_mask via sysfs

pcmcia-remove-includes-in-rsrc_mgr-which-arent-necessary-any-longer.patch
  pcmcia: remove #includes in rsrc_mgr which aren't necessary any longer

kgdb-ga.patch
  kgdb stub for ia32 (George Anzinger's one)
  kgdbL warning fix
  kgdb buffer overflow fix
  kgdbL warning fix
  kgdb: CONFIG_DEBUG_INFO fix
  x86_64 fixes
  correct kgdb.txt Documentation link (against  2.6.1-rc1-mm2)
  kgdb: fix for recent gcc
  kgdb warning fixes
  THREAD_SIZE fixes for kgdb
  Fix stack overflow test for non-8k stacks
  kgdb-ga.patch fix for i386 single-step into sysenter
  fix TRAP_BAD_SYSCALL_EXITS on i386
  add TRAP_BAD_SYSCALL_EXITS config for i386
  kgdb-is-incompatible-with-kprobes
  kgdb-ga-build-fix
  kgdb-ga-fixes

kgdb-kill-off-highmem_start_page.patch
  kgdb: kill off highmem_start_page

kgdboe-netpoll.patch
  kgdb-over-ethernet via netpoll
  kgdboe: fix configuration of MAC address

kgdb-x86_64-support.patch
  kgdb-x86_64-support.patch for 2.6.2-rc1-mm3
  kgdb-x86_64-warning-fixes
  kgdb-x86_64-fix
  kgdb-x86_64-serial-fix
  kprobes exception notifier fix

dev-mem-restriction-patch.patch
  /dev/mem restriction patch

dev-mem-restriction-patch-allow-reads.patch
  dev-mem-restriction-patch: allow reads

jbd-remove-livelock-avoidance.patch
  JBD: remove livelock avoidance code in journal_dirty_data()

journal_add_journal_head-debug.patch
  journal_add_journal_head-debug

list_del-debug.patch
  list_del debug check

unplug-can-sleep.patch
  unplug functions can sleep

firestream-warnings.patch
  firestream warnings

perfctr-core.patch
  perfctr: core
  perfctr: remove bogus perfctr_sample_thread() calls

perfctr-i386.patch
  perfctr: i386

perfctr-x86-core-updates.patch
  perfctr x86 core updates

perfctr-x86-driver-updates.patch
  perfctr x86 driver updates

perfctr-x86-driver-cleanup.patch
  perfctr: x86 driver cleanup

perfctr-prescott-fix.patch
  Prescott fix for perfctr

perfctr-x86-update-2.patch
  perfctr x86 update 2

perfctr-x86_64.patch
  perfctr: x86_64

perfctr-x86_64-core-updates.patch
  perfctr x86_64 core updates

perfctr-ppc.patch
  perfctr: PowerPC

perfctr-ppc32-driver-update.patch
  perfctr: ppc32 driver update

perfctr-ppc32-mmcr0-handling-fixes.patch
  perfctr ppc32 MMCR0 handling fixes

perfctr-ppc32-update.patch
  perfctr ppc32 update

perfctr-ppc32-update-2.patch
  perfctr ppc32 update

perfctr-virtualised-counters.patch
  perfctr: virtualised counters

perfctr-remap_page_range-fix.patch

virtual-perfctr-illegal-sleep.patch
  virtual perfctr illegal sleep

make-perfctr_virtual-default-in-kconfig-match-recommendation.patch
  Make PERFCTR_VIRTUAL default in Kconfig match recommendation  in help text

perfctr-ifdef-cleanup.patch
  perfctr ifdef cleanup

perfctr-update-2-6-kconfig-related-updates.patch
  perfctr: Kconfig-related updates

perfctr-virtual-updates.patch
  perfctr virtual updates

perfctr-virtual-cleanup.patch
  perfctr: virtual cleanup

perfctr-ppc32-preliminary-interrupt-support.patch
  perfctr ppc32 preliminary interrupt support

perfctr-update-5-6-reduce-stack-usage.patch
  perfctr: reduce stack usage

perfctr-interrupt-support-kconfig-fix.patch
  perfctr interrupt_support Kconfig fix

perfctr-low-level-documentation.patch
  perfctr low-level documentation

perfctr-inheritance-1-3-driver-updates.patch
  perfctr inheritance: driver updates

perfctr-inheritance-2-3-kernel-updates.patch
  perfctr inheritance: kernel updates

perfctr-inheritance-3-3-documentation-updates.patch
  perfctr inheritance: documentation updates

perfctr-inheritance-locking-fix.patch
  perfctr inheritance locking fix

perfctr-api-changes-first-step.patch
  perfctr API changes: first step

perfctr-virtual-update.patch
  perfctr virtual update

perfctr-x86-64-ia32-emulation-fix.patch
  perfctr x86-64 ia32 emulation fix

perfctr-sysfs-update-1-4-core.patch
  perfctr sysfs update: core

perfctr-sysfs-update.patch
  Perfctr sysfs update

perfctr-sysfs-update-2-4-x86.patch
  perfctr sysfs update: x86

perfctr-sysfs-update-3-4-x86-64.patch
  perfctr sysfs update: x86-64
  perfctr: syscall numbers in x86-64 ia32-emulation
  perfctr x86_64 native syscall numbers fix

perfctr-sysfs-update-4-4-ppc32.patch
  perfctr sysfs update: ppc32

sched-fix-preemption-race-core-i386.patch
  sched: fix preemption race (Core/i386)

sched-make-use-of-preempt_schedule_irq-ppc.patch
  sched: make use of preempt_schedule_irq() (PPC)

sched-make-use-of-preempt_schedule_irq-arm.patch
  sched: make use of preempt_schedule_irq (ARM)

add-do_proc_doulonglongvec_minmax-to-sysctl-functions.patch
  Add do_proc_doulonglongvec_minmax to sysctl functions
  add-do_proc_doulonglongvec_minmax-to-sysctl-functions-fix
  add-do_proc_doulonglongvec_minmax-to-sysctl-functions fix 2

add-sysctl-interface-to-sched_domain-parameters.patch
  Add sysctl interface to sched_domain parameters

allow-modular-ide-pnp.patch
  allow modular ide-pnp

allow-x86_64-to-reenable-interrupts-on-contention.patch
  Allow x86_64 to reenable interrupts on contention

i386-cpu-hotplug-updated-for-mm.patch
  i386 CPU hotplug updated for -mm

ppc64-fix-cpu-hotplug.patch
  ppc64: fix hotplug cpu

serialize-access-to-ide-devices.patch
  serialize access to ide devices

disable-atykb-warning.patch
  disable atykb "too many keys pressed" warning

export-file_ra_state_init-again.patch
  Export file_ra_state_init() again

cachefs-filesystem.patch
  CacheFS filesystem

numa-policies-for-file-mappings-mpol_mf_move-cachefs.patch
  numa-policies-for-file-mappings-mpol_mf_move for cachefs

cachefs-release-search-records-lest-they-return-to-haunt-us.patch
  CacheFS: release search records lest they return to haunt us

fix-64-bit-problems-in-cachefs.patch
  Fix 64-bit problems in cachefs

cachefs-fixed-typos-that-cause-wrong-pointer-to-be-kunmapped.patch
  cachefs: fixed typos that cause wrong pointer to be kunmapped

cachefs-return-the-right-error-upon-invalid-mount.patch
  CacheFS: return the right error upon invalid mount

fix-cachefs-barrier-handling-and-other-kernel-discrepancies.patch
  Fix CacheFS barrier handling and other kernel discrepancies

remove-error-from-linux-cachefsh.patch
  Remove #error from linux/cachefs.h

cachefs-warning-fix-2.patch
  cachefs warning fix 2

cachefs-linkage-fix-2.patch
  cachefs linkage fix

cachefs-build-fix.patch
  cachefs build fix

cachefs-documentation.patch
  CacheFS documentation

add-page-becoming-writable-notification.patch
  Add page becoming writable notification

add-page-becoming-writable-notification-fix.patch
  do_wp_page_mk_pte_writable() fix

add-page-becoming-writable-notification-build-fix.patch
  add-page-becoming-writable-notification build fix

provide-a-filesystem-specific-syncable-page-bit.patch
  Provide a filesystem-specific sync'able page bit

provide-a-filesystem-specific-syncable-page-bit-fix.patch
  provide-a-filesystem-specific-syncable-page-bit-fix

provide-a-filesystem-specific-syncable-page-bit-fix-2.patch
  provide-a-filesystem-specific-syncable-page-bit-fix-2

make-afs-use-cachefs.patch
  Make AFS use CacheFS

afs-cachefs-dependency-fix.patch
  afs-cachefs-dependency-fix

split-general-cache-manager-from-cachefs.patch
  Split general cache manager from CacheFS

turn-cachefs-into-a-cache-backend.patch
  Turn CacheFS into a cache backend

rework-the-cachefs-documentation-to-reflect-fs-cache-split.patch
  Rework the CacheFS documentation to reflect FS-Cache split

update-afs-client-to-reflect-cachefs-split.patch
  Update AFS client to reflect CacheFS split

assign_irq_vector-section-fix.patch
  assign_irq_vector __init section fix

kexec-i8259-shutdowni386.patch
  kexec: i8259-shutdown.i386

kexec-i8259-shutdown-x86_64.patch
  kexec: x86_64 i8259 shutdown

kexec-apic-virtwire-on-shutdowni386patch.patch
  kexec: apic-virtwire-on-shutdown.i386.patch

kexec-apic-virtwire-on-shutdownx86_64.patch
  kexec: apic-virtwire-on-shutdown.x86_64

kexec-ioapic-virtwire-on-shutdowni386.patch
  kexec: ioapic-virtwire-on-shutdown.i386

kexec-apic-virt-wire-fix.patch
  kexec: apic-virt-wire fix

kexec-ioapic-virtwire-on-shutdownx86_64.patch
  kexec: ioapic-virtwire-on-shutdown.x86_64

kexec-e820-64bit.patch
  kexec: e820-64bit

kexec-kexec-generic.patch
  kexec: kexec-generic

kexec-ide-spindown-fix.patch
  kexec-ide-spindown-fix

kexec-ifdef-cleanup.patch
  kexec ifdef cleanup

kexec-machine_shutdownx86_64.patch
  kexec: machine_shutdown.x86_64

kexec-kexecx86_64.patch
  kexec: kexec.x86_64

kexec-kexecx86_64-4level-fix.patch
  kexec-kexecx86_64-4level-fix

kexec-kexecx86_64-4level-fix-unfix.patch
  kexec-kexecx86_64-4level-fix unfix

kexec-machine_shutdowni386.patch
  kexec: machine_shutdown.i386

kexec-kexeci386.patch
  kexec: kexec.i386

kexec-use_mm.patch
  kexec: use_mm

kexec-loading-kernel-from-non-default-offset.patch
  kexec: loading kernel from non-default offset

kexec-loading-kernel-from-non-default-offset-fix.patch
  kdump: fix bss compile error

kexec-enabling-co-existence-of-normal-kexec-kernel-and-panic-kernel.patch
  kexec: nabling co-existence of normal kexec kernel and  panic kernel

kexec-ppc-support.patch
  kexec: ppc support

crashdump-documentation.patch
  crashdump: documentation

crashdump-memory-preserving-reboot-using-kexec.patch
  crashdump: memory preserving reboot using kexec

crashdump-memory-preserving-reboot-using-kexec-fix.patch
  kdump: Fix for boot problems on SMP

kdump-config_discontigmem-fix.patch
  kdump: CONFIG_DISCONTIGMEM fix

crashdump-routines-for-copying-dump-pages.patch
  crashdump: routines for copying dump pages

crashdump-routines-for-copying-dump-pages-kmap-fiddle.patch
  crashdump-routines-for-copying-dump-pages-kmap-fiddle

crashdump-kmap-build-fix.patch
  crashdump kmap build fix

crashdump-register-snapshotting-before-kexec-boot.patch
  crashdump: register snapshotting before kexec boot

crashdump-elf-format-dump-file-access.patch
  crashdump: ELF format dump file access

crashdump-linear-raw-format-dump-file-access.patch
  crashdump: linear/raw format dump file access

crashdump-minor-bug-fixes-to-kexec-crashdump-code.patch
  crashdump: minor bug fixes to kexec crashdump code

crashdump-cleanups-to-the-kexec-based-crashdump-code.patch
  crashdump: cleanups to the kexec based crashdump code

x86-rename-apic_mode_exint.patch
  x86: rename APIC_MODE_EXINT

x86-local-apic-fix.patch
  x86: local apic fix

new-bitmap-list-format-for-cpusets.patch
  new bitmap list format (for cpusets)

cpusets-big-numa-cpu-and-memory-placement.patch
  cpusets - big numa cpu and memory placement

cpusets-config_cpusets-depends-on-smp.patch
  Cpusets: CONFIG_CPUSETS depends on SMP

cpusets-move-cpusets-above-embedded.patch
  move CPUSETS above EMBEDDED

cpusets-fix-cpuset_get_dentry.patch
  cpusets : fix cpuset_get_dentry()

cpusets-fix-race-in-cpuset_add_file.patch
  cpusets: fix race in cpuset_add_file()

cpusets-remove-more-casts.patch
  cpusets: remove more casts

cpusets-make-config_cpusets-the-default-in-sn2_defconfig.patch
  cpusets: make CONFIG_CPUSETS the default in sn2_defconfig

cpusets-document-proc-status-allowed-fields.patch
  cpusets: document proc status allowed fields

cpusets-dont-export-proc_cpuset_operations.patch
  Cpusets - Dont export proc_cpuset_operations

cpusets-display-allowed-masks-in-proc-status.patch
  cpusets: display allowed masks in proc status

cpusets-simplify-cpus_allowed-setting-in-attach.patch
  cpusets: simplify cpus_allowed setting in attach

cpusets-remove-useless-validation-check.patch
  cpusets: remove useless validation check

cpusets-tasks-file-simplify-format-fixes.patch
  Cpusets tasks file: simplify format, fixes

cpusets-simplify-memory-generation.patch
  Cpusets: simplify memory generation

cpusets-interoperate-with-hotplug-online-maps.patch
  cpusets: interoperate with hotplug online maps

cpusets-alternative-fix-for-possible-race-in.patch
  cpusets: alternative fix for possible race in  cpuset_tasks_read()

cpusets-remove-casts.patch
  cpusets: remove void* typecasts

reiser4-sb_sync_inodes.patch
  reiser4: vfs: add super_operations.sync_inodes()

reiser4-allow-drop_inode-implementation.patch
  reiser4: export vfs inode.c symbols

reiser4-truncate_inode_pages_range.patch
  reiser4: vfs: add truncate_inode_pages_range()

reiser4-export-remove_from_page_cache.patch
  reiser4: export pagecache add/remove functions to modules

reiser4-export-page_cache_readahead.patch
  reiser4: export page_cache_readahead to modules

reiser4-reget-page-mapping.patch
  reiser4: vfs: re-check page->mapping after calling try_to_release_page()

reiser4-rcu-barrier.patch
  reiser4: add rcu_barrier() synchronization point

reiser4-export-inode_lock.patch
  reiser4: export inode_lock to modules

reiser4-export-pagevec-funcs.patch
  reiser4: export pagevec functions to modules

reiser4-export-radix_tree_preload.patch
  reiser4: export radix_tree_preload() to modules

reiser4-export-find_get_pages.patch

reiser4-radix-tree-tag.patch
  reiser4: add new radix tree tag

reiser4-radix_tree_lookup_slot.patch
  reiser4: add radix_tree_lookup_slot()

reiser4-perthread-pages.patch
  reiser4: per-thread page pools

reiser4-include-reiser4.patch
  reiser4: add to build system

reiser4-doc.patch
  reiser4: documentation

reiser4-only.patch
  reiser4: main fs

reiser4-recover-read-performance.patch
  reiser4: recover read performance

reiser4-export-find_get_pages_tag.patch
  reiser4-export-find_get_pages_tag

reiser4-add-missing-context.patch

add-acpi-based-floppy-controller-enumeration.patch
  Add ACPI-based floppy controller enumeration.

possible-dcache-bug-debugging-patch.patch
  Possible dcache BUG: debugging patch

serial-add-support-for-non-standard-xtals-to-16c950-driver.patch
  serial: add support for non-standard XTALs to 16c950 driver

add-support-for-possio-gcc-aka-pcmcia-siemens-mc45.patch
  Add support for Possio GCC AKA PCMCIA Siemens MC45

mpsc-driver-patch.patch
  serial: MPSC driver

generic-serial-cli-conversion.patch
  generic-serial cli() conversion

specialix-io8-cli-conversion.patch
  Specialix/IO8 cli() conversion

sx-cli-conversion.patch
  SX cli() conversion

revert-allow-oem-written-modules-to-make-calls-to-ia64-oem-sal-functions.patch
  revert "allow OEM written modules to make calls to ia64 OEM SAL functions"

md-add-interface-for-userspace-monitoring-of-events.patch
  md: add interface for userspace monitoring of events.

make-acpi_bus_register_driver-consistent-with-pci_register_driver-again.patch
  make acpi_bus_register_driver() consistent with pci_register_driver()

remove-lock_section-from-x86_64-spin_lock-asm.patch
  remove LOCK_SECTION from x86_64 spin_lock asm

kfree_skb-dump_stack.patch
  kfree_skb-dump_stack

cancel_rearming_delayed_work.patch
  cancel_rearming_delayed_work()
  make cancel_rearming_delayed_workqueue static

ipvs-deadlock-fix.patch
  ipvs deadlock fix

minimal-ide-disk-updates.patch
  Minimal ide-disk updates

use-find_trylock_page-in-free_swap_and_cache-instead-of-hand-coding.patch
  use find_trylock_page in free_swap_and_cache instead of hand coding

fbdev-cleanup-broken-edid-fixup-code.patch
  fbdev: Cleanup broken edid fixup code

fbcon-catch-blank-events-on-both-device-and-console-level.patch
  fbcon: Catch blank events on both device and console level

fbcon-fix-compile-error.patch
  fbcon: Fix compile error

fbdev-fbmon-cleanup.patch
  fbdev: Fbmon cleanup

i810fb-module-param-fix.patch
  i810fb: Module param fix

atyfb-fix-module-parameter-descriptions.patch
  atyfb: Fix module parameter descriptions

radeonfb-fix-init-exit-section-usage.patch
  radeonfb: Fix init/exit section usage

pxafb-reorder-add_wait_queue-and-set_current_state.patch
  pxafb: Reorder add_wait_queue() and set_current_state()

sa1100fb-reorder-add_wait_queue-and-set_current_state.patch
  sa1100fb: Reorder add_wait_queue() and set_current_state()

backlight-add-backlight-lcd-device-basic-support.patch
  backlight: Add Backlight/LCD device basic support

fbdev-add-w100-framebuffer-driver.patch
  fbdev: Add w100 framebuffer driver

raid5-overlapping-read-hack.patch
  raid5 overlapping read hack

figure-out-who-is-inserting-bogus-modules.patch
  Figure out who is inserting bogus modules

detect-atomic-counter-underflows.patch
  detect atomic counter underflows

waiting-10s-before-mounting-root-filesystem.patch
  retry mounting the root filesystem at boot time

post-halloween-doc.patch
  post halloween doc

periodically-scan-redzone-entries-and-slab-control-structures.patch
  periodically scan redzone entries and slab control structures

fuse-maintainers-kconfig-and-makefile-changes.patch
  Subject: [PATCH 1/11] FUSE - MAINTAINERS, Kconfig and Makefile changes

fuse-core.patch
  Subject: [PATCH 2/11] FUSE - core

fuse-device-functions.patch
  Subject: [PATCH 3/11] FUSE - device functions

fuse-read-only-operations.patch
  Subject: [PATCH 4/11] FUSE - read-only operations

fuse-read-write-operations.patch
  Subject: [PATCH 5/11] FUSE - read-write operations

fuse-file-operations.patch
  Subject: [PATCH 6/11] FUSE - file operations

fuse-mount-options.patch
  Subject: [PATCH 7/11] FUSE - mount options

fuse-extended-attribute-operations.patch
  Subject: [PATCH 8/11] FUSE - extended attribute operations

fuse-readpages-operation.patch
  Subject: [PATCH 9/11] FUSE - readpages operation

fuse-nfs-export.patch
  Subject: [PATCH 10/11] FUSE - NFS export

fuse-direct-i-o.patch
  Subject: [PATCH 11/11] FUSE - direct I/O

ieee1394-adds-a-disable_irm-option-to-ieee1394ko.patch
  ieee1394: add a disable_irm option to ieee1394.ko

fix-typo-in-arch-i386-kconfig.patch
  Fix typo in arch/i386/Kconfig

random-whitespace-doh.patch
  random: whitespace doh

random-entropy-debugging-improvements.patch
  random: entropy debugging improvements

random-run-time-configurable-debugging.patch
  random: run-time configurable debugging

random-periodicity-detection-fix.patch
  random: periodicity detection fix

random-add_input_randomness.patch
  random: add_input_randomness

various-kconfig-fixes.patch
  various Kconfig fixes

Re: 2.6.11-rc1-mm1

[Thomas Gleixner]

On Fri, 2005-01-14 at 00:23 -0800, Andrew Morton wrote:
> - Added the Linux Trace Toolkit (and hence relayfs).  Mainly because I
>   haven't yet taken as close a look at LTT as I should have.  Probably neither
>   have you.

I have. Maybe you should have. I really don't see a good argument to
include this code. 

The "non-locking" claim is nice, but a do { } while loop in the slot
reservation for every event including a do { } while loop in the slow
path is just a replacement of locking without actually using a lock. I
don't care whether this is likely or unlikely to happen, it's just bogus
to add a non constant time path for debugging/tracing purposes. 

Default timestamp measuring with do_gettimeofday is also contrary to the
non locking argument. There is 
a) a lock in there 
b) it might loop because it's a sequential lock. 

If you have no TSC you can do at least a jiffies + event-number based,
not so finegrained tracing which gives you at least the timeline of the
events.

There is also no need to do time diff calculations / conversions, this
can be done in userspace postprocessing. 

Adding 150k relayfs source in order to do tracing is scary. I don't see
any real advantage over a nice implemented per cpu ringbuffer, which is
lock free and does not add variable timed delays in the log path. Don't
tell me that a ringbuffer is not suitable, it's a question of size and
it is the same problem for relayfs. If you don't have enough buffers it
does not work. This applies for every implementation of tracebuffering
you do. In space constraint systems relayfs is even worse as it needs
more memory than the plain ringbuffer.
The ringbuffer has a nice advantage. In case the system crashes you can
retrieve the last and therefor most interesting information from the
ringbuffer without any hassle via BDI or in the worstcase via a serial
dump. You can even copy the tail of the buffer into a permanent storage
like buffered SRAM so it can be retrieved after reboot. 

Splitting the trace into different paths is nice to have but I don't see
a single point which cannot be done by a userspace (hostside)
postprocessing tool. It adds another non time constant component to the
trace path. Even the per CPU ringbuffers can be nicely synchronized by a
userspace postprocessing tool without adding complex synchronization
functions.

Replacing printk by a varags print into an event buffer is a nice idea
to replace serial logging of long lasting debug features. Must we really
include 150k source for this or can we just increase the log buffer size
or improve the printk itself?
In case of time related tracing it's just overkill. The printk
information is mostly a string, which can be replaced by the address on
which the printk is happening. The maybe available arguments can be
dumped in binary form. All this information can be converted into human
readable form by postprocessing. 

I wonder whether the various formatting options of the trace are really
of any value. I need neither strings, HEX strings nor XML formatted
information from the kernel. Max. 8192 Byte of user information makes me
frown. Tracing is not a copy to userspace function or am I missing
something ?

All tracepoints are unconditionally compiled into the kernel, whether
they are enabled or not. Why is it neccecary to check the enabled bit
for information I'm not interested in ? Why can't I compile this away by
not enabling the tracepoint at all.

I don't need to point out the various coding style issues again, but I
question if 
	atomic_set(&var), atomic_read(&var) | bit);
which can be found on several places is really doing what it's suggests
to do.

I did a short test on a 300MHz PIII box and the maximum time spent in
the log path (interrupts disabled during measurement) is about 30us.
Extrapolated to a 74MHz ARM SoC it will sum up to ~ 90-120us, what makes
it purely useless. 

Summary:

1. The code is not doing what it claims to do.
2. The code adds unnecessary overhead
3. It's not useful for low speed systems.

Question:
Why is the code included ?	

tglx

Re: 2.6.11-rc1-mm1

[Tim Bird]

Thomas Gleixner wrote:
> On Fri, 2005-01-14 at 00:23 -0800, Andrew Morton wrote:
> 
>>- Added the Linux Trace Toolkit (and hence relayfs).  Mainly because I
>>  haven't yet taken as close a look at LTT as I should have.  Probably neither
>>  have you.
> 
> I have. Maybe you should have. I really don't see a good argument to
> include this code. 

[ Lots of excellent criticisms omitted.]

I don't want to be argumentative, but possibly (to answer your last
question first), there are twofold reasons to put this in -mm:
 - there's no tracing infrastructure in the kernel now (except for
 kprobes - which provides hooks for creating tracepoints dynamically,
 but not 1) supporting infrastructure for timestamping, managing event
 data, etc., and 2) a static list of generally useful tracepoints.
 - to generate this discussion.

> 
> I did a short test on a 300MHz PIII box and the maximum time spent in
> the log path (interrupts disabled during measurement) is about 30us.
> Extrapolated to a 74MHz ARM SoC it will sum up to ~ 90-120us, what makes
> it purely useless. 

I've used it for various tasks, and I know others who have.  I wouldn't
recommend it in its present form for deep scheduling tweaks or debugging
kernel race conditions (which it is more likely to mask than
it is to find), but inapplicability there hardly makes it worthless for
other things.

> 
> Summary:
> 
> 1. The code is not doing what it claims to do.
I'm guessing the sense of this is in the micro-claims which are implied
(e.g. runs lockless and therefore avoids cache thrashing), rather than
the high-level claim of providing useful information in some situations.
It clearly does the latter.  At least is has for me.

> 2. The code adds unnecessary overhead
I agree it could be improved.  The threshold for "unnecessary" varies
by task.

> 3. It's not useful for low speed systems.
I've used it on low speed systems.

> Question:
> Why is the code included ?	
See above.

By the way, don't think that your comments are not appreciated.
I'm not particularly glued to any specific part of the implementation.
I'm excited to see tracing discussed here, if only to avoid
duplicate efforts and point out danger areas, for multiple tracing
projects that I am aware of.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

[RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Thomas Gleixner]

</Flame off>

On Fri, 2005-01-14 at 15:22 -0800, Tim Bird wrote:
>  but not 1) supporting infrastructure for timestamping, managing event
>  data, etc., and 2) a static list of generally useful tracepoints.

Both points are well taken. Thats the essential minimum what
instrumentation needs.

I'd like to see this infrastructure usable for all kinds of
instrumentation mechanisms which are built in to the kernel already or
functions which are used for similar purposes in experimental trees and
other instrumentation related projects. 

This requires to seperate the backend from the infrastructure, so you
can chose from a set of backends which fit best for the intended use. 

One of those backends is LTT+relayfs. 
I really respect the work you have done there, but please accept that I
just see the limitations and try to figure out a way to make it more
generic and flexible before it is cemented into the kernel and makes it
hard to use for other interesting instrumentation aspects and maybe
enforces redundant implementation of infrastructure related
functionality.

E.g. tracking down timing related issues can make use from such
functionality if the infrastructure is provided seperately.
I guess a lot of developers would be happy to use it when it is already
around in the kernel and it can help testers for giving better
information to developers.

tglx

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Hello Thomas,

I don't mind having a general discussion about instrumentation, but
it has to be understood that the topic is so general and means so
many different things to different people that we are unlikely to
reach any useful consensus. Believe me, it's not for the lack of
trying. More below.

Thomas Gleixner wrote:
> </Flame off>

:D

> One of those backends is LTT+relayfs. 
> I really respect the work you have done there, but please accept that I
> just see the limitations and try to figure out a way to make it more
> generic and flexible before it is cemented into the kernel and makes it
> hard to use for other interesting instrumentation aspects and maybe
> enforces redundant implementation of infrastructure related
> functionality.
> 
> E.g. tracking down timing related issues can make use from such
> functionality if the infrastructure is provided seperately.
> I guess a lot of developers would be happy to use it when it is already
> around in the kernel and it can help testers for giving better
> information to developers.

I would invite you to review the history behind LTT and the history
behind the efforts to get LTT integrated in the kernel (which are
two separate topics.) If you look back, you will see that I worked
very hard trying to get people to think about a common framework
and that I and others made numerous suggestions in this regard. Here
are a few examples:

- DProbes (kprobes ancestor):
Shortly after dprobes came out in 2000, I was one of the first to
suggest that there could be interfacing between both to allow
dynamically added trace points. We worked with, and eventually
joined forces with, the IBM team working on this and very early
on, LTT and DProbes were interfacing:
http://marc.theaimsgroup.com/?l=linux-kernel&m=97079714009328&w=2
- OProfile:
When time came to integrate oprofile in the kernel, I tried to push
for oprofile to use ltt as it's logging engine (to John's utter
horror.) relayfs didn't exist at the time, and obviously oprofile
made it in without relying on ltt.
Here's a posting from July 2002 where I suggested oprofile rely on
ltt. In that same posting I listed a number of drivers/subsystems
that already contained tracing statements. Obviously I was pointing
out that there was an opportunity to create a common, uniform
infrastructure based on ltt:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102624656615567&w=2
- Syscalltrack:
In replying to a posting of someone looking for tracing info, there
was a brief discussion as to how syscalltrack could use ltt instead
of: a) redirecting the syscall table, b) have its own buffering
mechanism. Again, relayfs didn't exist at the time:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102822343523369&w=2
- Event logging:
When there was discussion about event logging, there was suggestion
to use ltt's engine. Again, relayfs wasn't there:
http://marc.theaimsgroup.com/?l=linux-kernel&m=101836133400796&w=2

And there are many other cases. As you can see, it's not as if
I didn't try to have this discussion before. Unfortunately, interest
in this was rather limited.

In addition, and this is a very important issue, quite a few
kernel developers mistook LTT for a kernel debugging tool, which
it was never meant to be. When, in fact, if you ask those who have
looked at using it for that purpose (try Marcelo or Andrea) you will
see that they didn't find it to be appropriate for them. And
rightly so, it was never meant for that purpose. Even lately, when
I suggested Ingo try using relayfs instead of his custom tracing
code for his preemption work, he looked at it and said that it
wasn't suited, but would consider reusing parts of it if it were
in the kernel.

So, in general, one thing I learned over the years is to not touch
the topic of kernel debugging even with a 10 foot poll when
discussing LTT.

What you are hinting at here (mention of developers vs. testers,
for example), and your stated preference for the type of ring-buffer
you described earlier clearly goes in the direction I've learned to
avoid: buffering support for the general purpose of kernel debugging.

Let me say outright that I see the relevance of what you are looking
for, but let me also say that what we tried to achieve with relayfs
is to provide a general mechanism for kernel subsystems that need to
convey large amounts of data to user-space. We did not attempt to
solve the problem of providing a buffering framework for core kernel
debugging. As I mentioned to Ingo in the mail I referred to earlier
regarding the type of buffering you are looking for:
> The above tracer may indeed be very appropriate for kernel development,
> but it doesn't provide enough functionality for the requirements of
> mainstream users.

If there is interest for using either relayfs and/or ltt for that
purpose, then this is an entirely different mandate and a few things
would need to be added for that to happen. For starters, we could
add another mode to relayfs. Currently, it supports a locking and
a lockless buffering scheme. We could also have ring-buffer mode
which would function very much as you, and Ingo before, have
described. But let me be crystal clear about this: don't count on
me to make a case for it on LKML. I've had enough flak as it is.
If you believe this is necessary, then you are welcome to make a
case for it, and obtain support from others on LKML. Obviously, as
the maintainers of relayfs, we see no reason to avoid extending it
for purposes others may find it useful for and/or accepting patches
to that end, if indeed such extensions don't preclude its adoption
in the mainline kernel.

Hope this helps clarify things a little,

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Roman Zippel]

Hi,

On Sat, 15 Jan 2005, Karim Yaghmour wrote:

> In addition, and this is a very important issue, quite a few
> kernel developers mistook LTT for a kernel debugging tool, which
> it was never meant to be. When, in fact, if you ask those who have
> looked at using it for that purpose (try Marcelo or Andrea) you will
> see that they didn't find it to be appropriate for them. And
> rightly so, it was never meant for that purpose. Even lately, when
> I suggested Ingo try using relayfs instead of his custom tracing
> code for his preemption work, he looked at it and said that it
> wasn't suited, but would consider reusing parts of it if it were
> in the kernel.

Well, that's really a core problem. We don't want to duplicate 
infrastructure, which practically does the same. So if relayfs isn't 
usable in this kind of situation, it really raises the question whether 
relayfs is usable at all. We need to make relayfs generally usable, 
otherwise it will join the fate of devfs.

bye, Roman

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Hello Roman,

Roman Zippel wrote:
> On Sat, 15 Jan 2005, Karim Yaghmour wrote:
>>In addition, and this is a very important issue, quite a few
>>kernel developers mistook LTT for a kernel debugging tool, which
>>it was never meant to be. When, in fact, if you ask those who have
>>looked at using it for that purpose (try Marcelo or Andrea) you will
>>see that they didn't find it to be appropriate for them. And
>>rightly so, it was never meant for that purpose. Even lately, when
>>I suggested Ingo try using relayfs instead of his custom tracing
>>code for his preemption work, he looked at it and said that it
>>wasn't suited, but would consider reusing parts of it if it were
>>in the kernel.
> 
> Well, that's really a core problem. We don't want to duplicate 
> infrastructure, which practically does the same. So if relayfs isn't 
> usable in this kind of situation, it really raises the question whether 
> relayfs is usable at all. We need to make relayfs generally usable, 
> otherwise it will join the fate of devfs.

Hmm, coming from you I will take this is a pretty strong endorsement
for what I was suggesting earlier: provide a basic buffering mode
in relayfs to be used in kernel debugging. However, it must be
understood that this is separate from the existing modes and ltt,
for example, could not use such a basic infrastructure. If this is
ok with you, and no one wants to complain too loudly about this, I
will go ahead and add this to our to-do list for relayfs.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Thomas Gleixner]

On Sat, 2005-01-15 at 23:23 -0500, Karim Yaghmour wrote:
> > Well, that's really a core problem. We don't want to duplicate 
> > infrastructure, which practically does the same. So if relayfs isn't 
> > usable in this kind of situation, it really raises the question whether 
> > relayfs is usable at all. We need to make relayfs generally usable, 
> > otherwise it will join the fate of devfs.
> 
> Hmm, coming from you I will take this is a pretty strong endorsement
> for what I was suggesting earlier: provide a basic buffering mode
> in relayfs to be used in kernel debugging. However, it must be
> understood that this is separate from the existing modes and ltt,
> for example, could not use such a basic infrastructure. If this is
> ok with you, and no one wants to complain too loudly about this, I
> will go ahead and add this to our to-do list for relayfs.

This implies to seperate 

- infrastructure 
- event registration
- transport mechanism

tglx

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Thomas Gleixner wrote:
> This implies to seperate 
> 
> - infrastructure 
> - event registration
> - transport mechanism

Like I said in my first response: we can't be everything for everbody,
the requirements are just too broad. ISO tried it with OSI. Have a
look at net/* for the result.

Currently, LTT provides the first two in one piece, and relayfs
provides the third. Like I acknowledged earlier, there is room for
generalizing the transport mechanism, and I'm thinking of amending
the relayfs API proposal further and rename the modes to make them
more straight-forward:
- Managed (locking or lockless.)
- Ad-Hoc (which works like Ingo, yourself, and others have requested.)

If you really want to define layers, then there are actually four
layers:
1- hooking mechanism
2- event definition / registration
3- event management infrastructure
4- transport mechanism

LTT currently does 1, 2 & 3. Clearly, as in the mail I refered to
earlier, there is code in the kernel that already does 1, 2, 3,
and 4 in very hardwired/ad-hoc fashion and there isn't anyone asking
for them to remove it. We're offering 4 separately and are putting
LTT on top of it. If you want to get 1 & 2 separately, have a look
at kernel hooks and genevent:
http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/
http://www.listserv.shafik.org/pipermail/ltt-dev/2003-January/000408.html

We'd gladly take a serious look at using the former if it was
included, and there is work in progress being conducted on getting
the latter being the standard way for declaring LTT events instead
of using a static ltt-events.h.

Five years ago, there was a discussion about integrating GKHI into
the kernel (the kernel hooks ancestor). Have a look for yourself
as to the response to this suggestion (basically people weren't
ready to accept a generalized hooking mechanism without a defined
set of hooks, and then others didn't like the idea at all because
creating general hooks in the kernel which anybody can register
to creates legal and maintenance problems ... basically it's a
can of worms):
http://marc.theaimsgroup.com/?l=linux-kernel&m=97371908916365&w=2

There's only so much we can push into the kernel in the same time.
Not to mention that before you can be generic, you've got to have
some specific implementation to start working off on. I believe
that what we've ironed out through the discussion of the past
two days is a good basis.

There is some irony in all this. For years, we were told that we
couldn't make it into the kernel because we were perceived as
providing a kernel debugging tool, and now that we're starting
to get our things seriously reviewed we're being told that maybe
it ain't really that useful because those who want to do kernel
debugging can't use it as-is ... go figure.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Thomas Gleixner]

On Sun, 2005-01-16 at 20:54 -0500, Karim Yaghmour wrote:

> If you really want to define layers, then there are actually four
> layers:
> 1- hooking mechanism
> 2- event definition / registration
> 3- event management infrastructure
> 4- transport mechanism
> 
> LTT currently does 1, 2 & 3. Clearly, as in the mail I refered to
> earlier, there is code in the kernel that already does 1, 2, 3,
> and 4 in very hardwired/ad-hoc fashion and there isn't anyone asking
> for them to remove it. We're offering 4 separately and are putting
> LTT on top of it. If you want to get 1 & 2 separately, have a look
> at kernel hooks and genevent:

I know that there is enough code which does x,y,z hardcoded/hardwired
already. 

Thats the point. Adding another hardwired implementation does not give
us a possibility to solve the hardwired problem of the already available
stuff.

tglx

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Thomas Gleixner wrote:
> Thats the point. Adding another hardwired implementation does not give
> us a possibility to solve the hardwired problem of the already available
> stuff.

Well then, like I said before, you know what you need to do:
http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Thomas Gleixner]

On Mon, 2005-01-17 at 15:34 -0500, Karim Yaghmour wrote:
> Thomas Gleixner wrote:
> > Thats the point. Adding another hardwired implementation does not give
> > us a possibility to solve the hardwired problem of the already available
> > stuff.
> 
> Well then, like I said before, you know what you need to do:
> http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/

Oh, I guess my English must be really bad.

I was talking about seperation of layers, so why do I need
kernelhooks ? 

The seperation of layers makes it possible to actually reuse
functionality and gives the possibility that existing hardwired stuff
can be cleaned up to use the new functionality too. 

If we add another hardwired implementation then we do not have said
benefits.

tglx

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Thomas Gleixner wrote:
> If we add another hardwired implementation then we do not have said
> benefits.

Please stop handwaving. Folks like Andrew, Christoph, Zwane, Roman,
and others actually made specific requests for changes in the code.
What makes you think you're so special that you think you are
entitled to stay on the side and handwave about concepts.

If there is a limitation with the code, please present actual
snippets that need to be changed and suggest alternatives. That's
what everyone else does on this list.

If you want to clean-up the existing tracing code in the kernel,
then here are some ltt calls you may be interested in:
int ltt_create_event(char *event_type,
		     char *event_desc,
		     int format_type,
		     char *format_data);
int ltt_log_raw_event(int event_id, int event_size, void *event_data);

And here's an actual example:
...
  delta_id = ltt_create_event("Delta",
                              NULL,
                              CUSTOM_EVENT_FORMAT_TYPE_HEX,
                              NULL);
...
  ltt_log_raw_event(delta_id, sizeof(a_delta_event), &a_delta_event);
...
  ltt_destroy_event(delta_id);

You can then use LibLTT to read the trace and extract your custom
events and format your binary data as it suits you.

Save the bandwidth and start cleaning.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Thomas Gleixner]

On Mon, 2005-01-17 at 18:57 -0500, Karim Yaghmour wrote: 
> Thomas Gleixner wrote:
> > If we add another hardwired implementation then we do not have said
> > benefits.
> 
> Please stop handwaving. Folks like Andrew, Christoph, Zwane, Roman,
> and others actually made specific requests for changes in the code.
> What makes you think you're so special that you think you are
> entitled to stay on the side and handwave about concepts.

So the points you added to your todo list which were brought up by me
are worthless ?

I'm not handwaving. I started this RFC to move the discussion into a
general discussion about instrumentation. A couple of people are
seriosly interested to do this. If you are not interested then ignore
the thread, but you're way not in a position to tell me to shut up.

You turned this thread into your LTT prayer wheel.

Roman pointed out your unwillingness to create a common framework
before. But I have to disagree with him in one point. It's not amazing,
it's annoying.

> If there is a limitation with the code, please present actual
> snippets that need to be changed and suggest alternatives. That's
> what everyone else does on this list.

I pointed you to actually broken code and you accused me of throwing
mud.

> Save the bandwidth 

Please remove me from cc, it's a good start to save bandwidth.

> and start cleaning.

Yes, I did already start cleaning

cat ../broken-out/ltt* | patch -p1 -R

tglx

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Thomas,

Thomas Gleixner wrote:
> Yes, I did already start cleaning
> 
> cat ../broken-out/ltt* | patch -p1 -R

:D

If it gives you a warm and fuzzy feeling to have the last
cheap-shot, then I'm all for it, it is of no consequence anyway.
And _please_ don't forget to answer this very email with
something of the same substance.

For my part I consider that I've invested a substantial amount
of time in responding to both your conceptual and practical
feedback, as the archives clearly show.

That being said, I have to thank you for making sure that all
the obvious questions have been asked. I now have more than a
dozen archive links of my answers to those. I'll sure come in
handy when writing an FAQ.

Thanks again,

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Werner Almesberger]

>From all I've heard and seen of LTT (and I have to admit that most
of it comes from reading this thread, not from reading the code),
I have the impression that it may try to be a bit too specialized,
and thus might miss opportunities for synergy. 

You must be getting tired of people trying to redesign things from
scratch, but maybe you'll humor me anyway ;-)

Karim Yaghmour wrote:
> If you really want to define layers, then there are actually four
> layers:
> 1- hooking mechanism
> 2- event definition / registration
> 3- event management infrastructure
> 4- transport mechanism

For 1, kprobes would seem largely sufficient. In cases where you
don't have a usable attachment point (e.g. in the middle of a
function and you need access to variables with unknown location),
you can add lightweight instrumentation that arranges the code
flow suitably. [1, 2]

2 and 3 should be the main domain of LTT, with 2 sitting on top
of kprobes. kprobes currently doesn't have a nice way for
describing handlers, but that can be fixed [3]. But you probably
don't need a "nice" interface right now, but might be satisfied
with one that works and is fast (?)

>From the discussion, it seems that the management is partially
done by relayfs. I find this a little strange. E.g. instead of
filtering events, you may just not generate them in the first
place, e.g. by not placing a probe, or by filtering in LTT,
before submitting the event.

Timestamps may be fine either way. Restoring sequence should be
a task user-space can handle: in the worst case, you'd have to
read and merge from #cpus streams. Seeking works in that context,
too.

Last but not least, 4 should be simple. Particularly since you're
worried about extreme speeds, there should be as little
processing as you can afford. If you need to seek efficiently
(do you, really ?), you may not even want message boundaries at
that level.

Something that isn't entirely clear to me is if you also need to
aggregate information in buffers. E.g. by updating a record until
is has been retrieved by user space, or by updating a record
when there is no space to create a new one. Such functionality
would add complexity and needs tight sychronization with the
transport.

[1] I've seen the argument that kprobes aren't portable. This
    strikes me a highly questionable. Even if an architecture
    doesn't have a trap instruction (or equivalent code sequence)
    that is at least as short as the shortest instruction, you
    can always fall back to adding instrumentation [2]. Also, if
    you know where your basic blocks are, you may be able to
    use traps that span multiple instructions. I recall that
    things of this kind are already planned for kprobes.

[2] See the "reliable markers" of umlsim from umlsim.sf.net.
    Implementation: cd umlsim/lib; make; tail -50 markers_kernel.h
    Examples: cd umlsim/sim/tests; cat sbug.marker
    They're basically extra-light markup in the source code.
    Works on ia32, but I haven't found a way to get the assembler
    to cooperate for amd64, yet.

[3] I've already solved this problem in umlsim: there, I have a
    Perl/C-like scripting language that allows handlers to do
    pretty much anything they want. Of course, kprobes would
    want pre-compiled C code, not some scripts, but I think the
    design could be developped in a direction that would allow
    both. Will take a while, but since I'll eventually have to
    rewrite the "microcode" anyway, ...

So my comments are basically as follows:

1) kprobes seems like a suitable and elegant mechanism for
   placing all the hooks LTT needs, so I think that it would
   be better to build on this basis, and extend it where
   necessary, than to build yet another specialized variant
   in parallel.
2) LTT should do what it is good at, and not have to worry
   about the rest (i.e. supporting infrastructure).
3) relayfs should be lean and fast, as you intend it to be, so
   that non-LTT tracing or fnord debugging fnord code may find
   it useful, too.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: [RFC] Instrumentation (was Re: 2.6.11-rc1-mm1)

[Karim Yaghmour]

Werner Almesberger wrote:
>>From all I've heard and seen of LTT (and I have to admit that most
> of it comes from reading this thread, not from reading the code),

Might I add that this is part of the problem ... No personal
offence intended, but there's been _A LOT_ of things said about
LTT that were based on third-hand account and no direct contact
with the toolset/code. And part of the problem is that _many_
people on this list, and elsewhere, have done some form of
tracing or another as part of their development, so they all
have their idea of how this is best done. Yet, while such
experience can help provide additional ideas to LTT's development,
it also often requires re-explaining to every new suggestor why we
added features he couldn't imagine would be useful to any of
his/her own tracing needs ... Sometimes I wish my interests lied
in some arcane feature that few had ever played with ;)

IOW, while I don't discount anybody else's experience with tracing,
please give us at least the benefit of the doubt by actually:
a) Looking at the code
b) Looking at the mailing list archives
c) Asking us questions directly related to the code

> I have the impression that it may try to be a bit too specialized,
> and thus might miss opportunities for synergy. 

Bare with me on this one ...

> You must be getting tired of people trying to redesign things from
> scratch, but maybe you'll humor me anyway ;-)

Hey, from you Werner I'll take anything. It's always a pleasure
talking with you :)

> Karim Yaghmour wrote:
> 
>>If you really want to define layers, then there are actually four
>>layers:
>>1- hooking mechanism
>>2- event definition / registration
>>3- event management infrastructure
>>4- transport mechanism
> 
> 
> For 1, kprobes would seem largely sufficient. In cases where you
> don't have a usable attachment point (e.g. in the middle of a
> function and you need access to variables with unknown location),
> you can add lightweight instrumentation that arranges the code
> flow suitably. [1, 2]

Let me say outright, as I said to Andi early on in the sister thread,
that I have no problems with having the trace points being fed by
kprobes. In fact, in 2000, way back before kprobes even existed, LTT
was already interfacing with DProbes for dynamic insertion of trace
points.

... There I said it ... now watch me have to repeat this yet again
later on ... :/

However, kprobes is not magic:
a) Like I said to Andi:
> As far as kprobes go, then you still need to have some form or another
> of marking the code for key events, unless you keep maintaining a set
> of kprobes-able points separately, which really makes it unusable for
> the rest of us, as the users of LTT have discovered over time (having
> to create a new patch for every new kernel that comes out.)

b) Like I said to Andrew back in July:
> I've double-checked what I already knew about kprobes and have looked again
> at the site and the patch, and unless there's some feature of kprobes I don't
> know about that allows using something else than the debug interrupt to add
> hooks,
...
> Generating new interrupts is simply unacceptable for LTT's functionality.
> Not to mention that it breaks LTT because tracing something will generate
> events of its own, which will generating tracing events of their own ...
> recursion.

Ok, you can argue about the recursion thing with an "if()", but you'll
have to admit that like in the case I described to Roman:
> ... Say you're getting
> 2MB/s of data (which is not unrealistic on a loaded system.) That means
> that if I'm tracing for 2 days, I've got 345GB of data (~7.5GB/hour).
IOW, something like 200,000events/s (average of 10bytes/event). Do I
really need to explain that 200,000 traps/interrupts per second is
not something you want ... ?

But don't despair, like I said to Andi:
> So lately I've been thinking that there may be a middle-ground here
> where everyone could be happy. Define three states for the hooks:
> disabled, static, marker. The third one just adds some info into
> System.map for allowing the automation of the insertion of kprobes
> hooks (though you would still need the debugging info to find the
> values of the variables that you want to log.) Hence, you get to
> choose which type of poison you prefer. For my part, I think the
> noop/early-check should be sufficient to get better performance from
> the existing hook-set.
I have received very little feedback on this suggestion, though I
really think it's worth entertaining, especially with your mention
of uml-sim markers further below.

As for the location of ltt trace points, then they are very rarely
at function boundaries. Here's a classic:
		prepare_arch_switch(rq, next);
		ltt_ev_schedchange(prev, next);
		prev = context_switch(rq, prev, next);

> 2 and 3 should be the main domain of LTT, with 2 sitting on top
> of kprobes. kprobes currently doesn't have a nice way for
> describing handlers, but that can be fixed [3]. But you probably
> don't need a "nice" interface right now, but might be satisfied
> with one that works and is fast (?)

The functions have been there for DProbes for 5 years:
int ltt_create_event(char *event_type,
		     char *event_desc,
		     int format_type,
		     char *format_data)
int ltt_log_raw_event(int event_id, int event_size, void *event_data)

>>From the discussion, it seems that the management is partially
> done by relayfs. I find this a little strange. E.g. instead of
> filtering events, you may just not generate them in the first
> place, e.g. by not placing a probe, or by filtering in LTT,
> before submitting the event.

Like I said to Andi:
> ... For one thing, the current
> ltt hooks aren't as fast as they should be (i.e. we check whether
> the tracing is enabled for a certain event way too far in the code-path.)
> This should be rather simple to fix.
And I've already got the code snippet to fix this ready.

> Timestamps may be fine either way. Restoring sequence should be
> a task user-space can handle: in the worst case, you'd have to
> read and merge from #cpus streams. Seeking works in that context,
> too.
> 
> Last but not least, 4 should be simple. Particularly since you're
> worried about extreme speeds, there should be as little
> processing as you can afford. If you need to seek efficiently
> (do you, really ?), you may not even want message boundaries at
> that level.

Like I said to Roman:
> Removing this data would require more data for each event to
> be logged, and require parsing through the trace before reading it in
> order to obtain markers allowing random access. This wouldn't be so
> bad if we were expecting users to use LTT sporadically for very short
> periods of time. However, given ltt's target audience (i.e. need to
> run traces for hours, maybe days, weeks), traces would rapidely become
> useless because while plowing through a few hundred KBs of data and
> allocating RAM for building internal structures as you go is fine,
> plowing through tens of GBs of data, possibly hundreds, requires that
> you come up with a format that won't require unreasonable resources
> from your system, while incuring negligeable runtime costs for generating
> it. We believe the format we currently have achieves the right balance
> here.

What we've agreed with Roman is that relayfs won't write anything at
the boundaries. Its clients will provide it with callbacks to be
invoked at buffer boundaries. When invoked, said callbacks can add
whatever they feel is important to the buffer, relayfs doesn't care.

> Something that isn't entirely clear to me is if you also need to
> aggregate information in buffers. E.g. by updating a record until
> is has been retrieved by user space, or by updating a record
> when there is no space to create a new one. Such functionality
> would add complexity and needs tight sychronization with the
> transport.

If I understand you correctly, you are talking about the fact that
the transport layer's management of the buffers is syncrhonized
with some user-space entity that consumes the buffers produced
and talks back to relayfs (albeit indirectly) to let it know that
said buffers are now available? If so, then that's why I suggested
elsewhere that we have two modes for relayfs: managed and adhoc.
In the former, you have the required mechanics for what I just
described. In the latter, you have a very basic buffering scheme
that cares nothing about user-space synchronization.

> [1] I've seen the argument that kprobes aren't portable. This
>     strikes me a highly questionable. Even if an architecture
>     doesn't have a trap instruction (or equivalent code sequence)
>     that is at least as short as the shortest instruction, you
>     can always fall back to adding instrumentation [2]. Also, if
>     you know where your basic blocks are, you may be able to
>     use traps that span multiple instructions. I recall that
>     things of this kind are already planned for kprobes.

I have nothing against kprobes. People keep refering to it as if
it magically made all the related problems go away, and it doesn't.
See above.

> [2] See the "reliable markers" of umlsim from umlsim.sf.net.
>     Implementation: cd umlsim/lib; make; tail -50 markers_kernel.h
>     Examples: cd umlsim/sim/tests; cat sbug.marker
>     They're basically extra-light markup in the source code.
>     Works on ia32, but I haven't found a way to get the assembler
>     to cooperate for amd64, yet.

Nothing precludes us to move in this direction once something is
in the kernel, it's all currently hidden away in a .h, and it would
be the same with this.

> [3] I've already solved this problem in umlsim: there, I have a
>     Perl/C-like scripting language that allows handlers to do
>     pretty much anything they want. Of course, kprobes would
>     want pre-compiled C code, not some scripts, but I think the
>     design could be developped in a direction that would allow
>     both. Will take a while, but since I'll eventually have to
>     rewrite the "microcode" anyway, ...

Like I said, nothing precludes us ...

> So my comments are basically as follows:
> 
> 1) kprobes seems like a suitable and elegant mechanism for
>    placing all the hooks LTT needs, so I think that it would
>    be better to build on this basis, and extend it where
>    necessary, than to build yet another specialized variant
>    in parallel.

Whichever way you look at this, you need to mark the code. What's
in the .h is something we can tweak ad-nauseam.

> 2) LTT should do what it is good at, and not have to worry
>    about the rest (i.e. supporting infrastructure).

I'm guessing that when you're talking about "supporting
infrastructure" you are refering to the trace statements. If so,
please see above. Also note that without the existing marker set
LTT is useless to its users (application developers, sysadmins,
power users, etc.)

> 3) relayfs should be lean and fast, as you intend it to be, so
>    that non-LTT tracing or fnord debugging fnord code may find
>    it useful, too.

relayfs has already been used for many non-LTT related. Ask
Hubertus or Jamal, to name a few.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546