public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* Slow I/O performance on SAS1064
@ 2014-03-05 17:50 Markus
  2014-03-05 18:21 ` Mark Knecht
  0 siblings, 1 reply; 6+ messages in thread
From: Markus @ 2014-03-05 17:50 UTC (permalink / raw)
  To: Nagalakshmi Nandigama, Sreekanth Reddy, support,
	DL-MPTFusionLinux, linux-scsi, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 5346 bytes --]

Hi

I have problem with SATA disks at my Sunfire v245 and its lsi controller they are very slow.

So I test it with dd and while dd in a nother terminal i see dstat results of sda:


Copy from ramdisk to sda.
=== Test ext4 ===
DD

root@outpost:/ramdisk# dd if=debian-7.4.0-sparc-netinst.iso of=/root/test.dd
353344+0 records in
353344+0 records out
180912128 bytes (181 MB) copied, 24.6384 s, 7.3 MB/s

Dstat:

----total-cpu-usage---- --dsk/sda-- ---io/sda--
usr sys idl wai hiq siq| read  writ| read  writ
  1   0 100   0   0   0|   0     0 |   0     0 
  0   0 100   0   0   0|   0     0 |   0     0 
  2   1  97   0   0   0|   0     0 |   0     0 
  1   5  94   0   0   0|   0     0 |   0     0 
  7  43  50   0   0   0|   0     0 |   0     0 
  7  43  50   0   0   0|   0     0 |   0     0 
 10  40  50   0   0   0|   0     0 |   0     0 
 12  45  41   3   0   0|   0     0 |   0     0 
  6  42  36  17   0   1|   0   512k|   0  1.00 
  0   0   0  99   0   0|   0  4096k|   0  8.00 
  0   2   0  98   0   0|   0  5120k|   0  10.0 
  0   1   0  99   0   0|   0  7680k|   0  15.0 
  1   0   0 100   0   0|   0  5632k|   0  11.0 
  0   0   0  99   0   0|   0  7168k|   0  14.0 
  1   1   0  99   0   0|   0  6656k|   0  13.0 
  0   0   0 100   0   0|   0  6144k|   0  12.0 
  1   0   0 100   0   0|   0  7680k|   0  15.0 
  0   1  22  78   0   0|   0  4608k|   0  9.00 
  0   0  50  50   0   0|   0  3072k|   0  6.00 
  1   0  50  50   0   1|   0  7168k|   0  14.0 
  1   1  50  49   0   0|4096B 5120k|1.00  10.0 
  0   0  50  50   0   0|   0  8704k|   0  17.0 
  0   0  50  49   0   0|   0  7680k|   0  15.0 
  0   1  50  50   0   0|   0  7168k|   0  14.0 
  1   0  49  49   0   0|   0  7168k|   0  14.0 
  1   0  50  50   0   0|   0  7680k|   0  15.0 
  0   0  50  50   0   0|   0  7168k|   0  14.0 
  0   0  50  50   0   0|   0  2560k|   0  5.00 
  0   1  30  69   0   0|   0  3624k|   0  8.00 
  1   0   0  99   0   0|   0  4100k|   0  9.00 
  0   0   0 100   0   0|   0  5632k|   0  11.0 
  0   0   0  99   0   0|   0  3584k|   0  7.00 
  1   0   0 100   0   0|   0  4608k|   0  9.00 
  0   0   0  99   0   1|   0  5120k|   0  10.0 
  1   0   4  96   0   0|   0  2560k|   0  6.00 
  0   0  50  50   0   0|   0  3072k|   0  6.00 
  0   0  49  50   0   0|   0  5120k|   0  10.0 
  1   0  50  50   0   0|   0  5120k|   0  10.0 
  0   0  50  50   0   0|   0  4096k|   0  8.00 
  1   0  50  50   0   0|   0  5120k|   0  10.0 
  0   0  50  50   0   0|   0  5120k|   0  10.0 
  0   1  57  42   0   0|   0  1125k|   0  17.0 
  0   0 100   0   0   0|   0     0 |   0     0 
  1   0 100   0   0   0|   0     0 |   0     0 
  1   0 100   0   0   0|   0     0 |   0     0 
  0   0 100   0   0   0|   0     0 |   0     0 
  0   0 100   0   0   0|   0     0 |   0     0

-------------------------------------------------------------------------------
Information:
Kernel 3.13.5 (stable release)
Debian7 (stable)
hdparm v9.39


Controller 
LSI SAS1064
	cat /proc/mpt/ioc0/info 
	ioc0:
	  ProductID = 0x2701 (LSISAS1064 A3)
	  FWVersion = 0x01080400
	  MsgVersion = 0x0105
	  FirstWhoInit = 0x00
	  EventState = 0x00
	  CurrentHostMfaHighAddr = 0x00000000
	  CurrentSenseBufferHighAddr = 0x00000000
	  MaxChainDepth = 0x60 frames
	  MinBlockSize = 0x20 bytes
	  RequestFrames @ 0xfffffc137f602800 (Dma @ 0x00000000c000a800)
	    {CurReqSz=128} x {CurReqDepth=511} = 65408 bytes ^= 0x10000
	    {MaxReqSz=128}   {MaxReqDepth=511}
	  Frames   @ 0xfffffc137f600000 (Dma @ 0x00000000c0008000)
	    {CurRepSz=80} x {CurRepDepth=128} = 10240 bytes ^= 0x2880
	    {MaxRepSz=0}   {MaxRepDepth=511}
	  MaxDevices = 63
	  MaxBuses = 1
	  PortNumber = 1 (of 1)

	cat /proc/mpt/version 
		mptlinux-3.04.20
		  Fusion MPT base driver
		  Fusion MPT SAS host driver
		  Fusion MPT ioctl driver

-------------------------------------------------------------------------------
Hdparm Information:
	hdparm -I /dev/sda
	
	/dev/sda:
	SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	
	ATA device, with non-removable media
	Standards:
	        Likely used: 1
	Configuration:
	        Logical         max     current
	        cylinders       0       0
	        heads           0       0
	        sectors/track   0       0
	        --
	        Logical/Physical Sector size:           512 bytes
	        device size with M = 1024*1024:           0 MBytes
	        device size with M = 1000*1000:           0 MBytes 
	        cache/buffer size  = unknown
	Capabilities:
	        IORDY not likely
	        Cannot perform double-word IO
	        R/W multiple sector transfer: not supported
	        DMA: not supported
			PIO: pio0  

-------------------------------------------------------------------------------
Moduls load:
lsmod | grep mpt
	mptctl                 27359  0 
	mptsas                 45861  5 
	mptscsih               25765  1 mptsas
	mptbase                75530  3 mptctl,mptsas,mptscsih
	scsi_transport_sas     21624  1 mptsas

-------------------------------------------------------------------------------
The hdparm result looks like there is somethink not right . There were no features supported but why ?


Greetings Markus

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Slow I/O performance on SAS1064
  2014-03-05 17:50 Markus
@ 2014-03-05 18:21 ` Mark Knecht
  2014-03-05 21:40   ` markus
  0 siblings, 1 reply; 6+ messages in thread
From: Mark Knecht @ 2014-03-05 18:21 UTC (permalink / raw)
  To: Markus
  Cc: Nagalakshmi Nandigama, Sreekanth Reddy, support,
	DL-MPTFusionLinux, linux-scsi, Linux Kernel List

On Wed, Mar 5, 2014 at 9:50 AM, Markus <markus@kola.li> wrote:
<SNIP>

> The hdparm result looks like there is somethink not right . There were no features supported but why ?


Does the HDD have S.M.A.R.T. features? Possibly

smartctl -a /dev/sda

would provide some additional visibility?

- Mark

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Slow I/O performance on SAS1064
  2014-03-05 18:21 ` Mark Knecht
@ 2014-03-05 21:40   ` markus
  2014-03-06 20:51     ` Mark Knecht
  0 siblings, 1 reply; 6+ messages in thread
From: markus @ 2014-03-05 21:40 UTC (permalink / raw)
  To: Mark Knecht
  Cc: Nagalakshmi Nandigama, Sreekanth Reddy, support,
	DL-MPTFusionLinux, linux-scsi, Linux Kernel List

[-- Attachment #1: Type: text/plain, Size: 4819 bytes --]

On Wed, Mar 05, 2014 at 10:21:07AM -0800, Mark Knecht wrote:
> On Wed, Mar 5, 2014 at 9:50 AM, Markus <markus@kola.li> wrote:
> <SNIP>
> 
> > The hdparm result looks like there is somethink not right . There were no features supported but why ?
> 
> 
> Does the HDD have S.M.A.R.T. features? Possibly
> 
> smartctl -a /dev/sda
root@outpost:~# smartctl -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net


Probable ATA device behind a SAT layer
Try an additional '-d ata' or '-d sat' argument.

so i try it with scsi. 

root@outpost:~# smartctl -d scsi -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Logical Unit id:      0x50000394e2380537
Serial number:                   7365P4KNT
Device type:          disk
Local Time is:        Wed Mar  5 22:32:09 2014 CET
Device supports SMART and is Disabled
Temperature Warning Disabled or Not Supported
SMART Health Status: OK

Error Counter logging not supported
Device does not support Self Test logging


> 
> would provide some additional visibility?
.....
[   98.037209] Uniform Multi-Platform E-IDE driver
[   98.107564] SLAB: Unable to allocate memory on node 0 (gfp=0x40d1)
[   98.188985]   cache: dma-kmalloc-512, object size: 512, order: 0
[   98.268127]   node 0: slabs: 0/0, objs: 0/0, free: 0
[   98.333538] sr: out of memory.
[   98.373628] cdrom: Uniform CD-ROM driver Revision: 3.20
[   98.443390] sr 0:0:0:0: Attached scsi CD-ROM sr0
[   98.448375] sr 0:0:0:0: Attached scsi generic sg0 type 5
[  103.198705] scsi2 : ioc0: LSISAS1064 A3, FwRev=01080400h, Ports=1, MaxQ=511, IRQ=10
[  103.329805] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 0, phy 0, sas_addr 0x1221000000000000
[  103.462762] scsi 2:0:0:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  103.571610] scsi 2:0:0:0: Attached scsi generic sg1 type 0
[  103.645418] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 1, phy 1, sas_addr 0x1221000001000000
[  103.777635] scsi 2:0:1:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  103.886607] scsi 2:0:1:0: Attached scsi generic sg2 type 0
[  103.960517] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 2, phy 2, sas_addr 0x1221000002000000
[  104.093109] scsi 2:0:2:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  104.201973] scsi 2:0:2:0: Attached scsi generic sg3 type 0
[  104.275830] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 3, phy 3, sas_addr 0x1221000003000000
[  104.408412] scsi 2:0:3:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  104.517379] scsi 2:0:3:0: Attached scsi generic sg4 type 0
[  104.618997] sd 2:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.721185] sd 2:0:3:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.823771] sd 2:0:1:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.824017] sd 2:0:2:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.828691] sd 2:0:0:0: [sda] Write Protect is off
[  104.828697] sd 2:0:0:0: [sda] Mode Sense: 67 00 00 08
[  104.829940] sd 2:0:2:0: [sdc] Write Protect is off
[  104.829945] sd 2:0:2:0: [sdc] Mode Sense: 67 00 00 08
[  104.830488] sd 2:0:3:0: [sdd] Write Protect is off
[  104.830493] sd 2:0:3:0: [sdd] Mode Sense: 67 00 00 08
[  104.833225] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  104.834471] sd 2:0:2:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  104.835118] sd 2:0:3:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  105.583175] sd 2:0:1:0: [sdb] Write Protect is off
[  105.623801]  sda: sda1 sda2 sda3
[  105.651118]  sdc: sdc1 sdc9
[  105.654024]  sdd: sdd1 sdd9
[  105.761966] sd 2:0:1:0: [sdb] Mode Sense: 67 00 00 08
[  105.769029] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  105.775293] sd 2:0:2:0: [sdc] Attached SCSI disk
[  105.775941] sd 2:0:3:0: [sdd] Attached SCSI disk
[  106.014357] sd 2:0:0:0: [sda] Attached SCSI disk
[  106.158864]  sdb: sdb1 sdb9
[  106.205879] sd 2:0:1:0: [sdb] Attached SCSI disk
[  106.502387] random: nonblocking pool is initialized
[  106.792978] device-mapper: uevent: version 1.0.3
[  106.861434] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com
[  108.085062] bio: create slab <bio-1> at 1
.....
> 
> - Mark

Markus

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Slow I/O performance on SAS1064
  2014-03-05 21:40   ` markus
@ 2014-03-06 20:51     ` Mark Knecht
  2014-03-07  9:46       ` markus
  0 siblings, 1 reply; 6+ messages in thread
From: Mark Knecht @ 2014-03-06 20:51 UTC (permalink / raw)
  To: markus
  Cc: Nagalakshmi Nandigama, Sreekanth Reddy, support,
	DL-MPTFusionLinux, linux-scsi, Linux Kernel List

On Wed, Mar 5, 2014 at 1:40 PM, markus <markus@kola.li> wrote:
> On Wed, Mar 05, 2014 at 10:21:07AM -0800, Mark Knecht wrote:
>> On Wed, Mar 5, 2014 at 9:50 AM, Markus <markus@kola.li> wrote:
>> <SNIP>
>>
>> > The hdparm result looks like there is somethink not right . There were no features supported but why ?
>>
>>
>> Does the HDD have S.M.A.R.T. features? Possibly
>>
>> smartctl -a /dev/sda
> root@outpost:~# smartctl -a /dev/sda
> smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>
>
> Probable ATA device behind a SAT layer
> Try an additional '-d ata' or '-d sat' argument.
>
> so i try it with scsi.
>
> root@outpost:~# smartctl -d scsi -a /dev/sda
> smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>
> User Capacity:        2,000,398,934,016 bytes [2.00 TB]
> Logical block size:   512 bytes
> Logical Unit id:      0x50000394e2380537
> Serial number:                   7365P4KNT
> Device type:          disk
> Local Time is:        Wed Mar  5 22:32:09 2014 CET
> Device supports SMART and is Disabled
> Temperature Warning Disabled or Not Supported
> SMART Health Status: OK
>
> Error Counter logging not supported
> Device does not support Self Test logging

I am not familiar with this message:

"Device supports SMART and is Disabled"

but there are lots of posts out there with folks asking about it.
Possibly you can figure out what's causing smart not to turn on, get
it turned on and get some info.

Sorry I cannot help more.

Good luck,
Mark

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Slow I/O performance on SAS1064
  2014-03-06 20:51     ` Mark Knecht
@ 2014-03-07  9:46       ` markus
  0 siblings, 0 replies; 6+ messages in thread
From: markus @ 2014-03-07  9:46 UTC (permalink / raw)
  To: Mark Knecht
  Cc: Nagalakshmi Nandigama, Sreekanth Reddy, support,
	DL-MPTFusionLinux, linux-scsi, Linux Kernel List

[-- Attachment #1: Type: text/plain, Size: 4040 bytes --]

On Thu, Mar 06, 2014 at 12:51:27PM -0800, Mark Knecht wrote:
> On Wed, Mar 5, 2014 at 1:40 PM, markus <markus@kola.li> wrote:
> > On Wed, Mar 05, 2014 at 10:21:07AM -0800, Mark Knecht wrote:
> >> On Wed, Mar 5, 2014 at 9:50 AM, Markus <markus@kola.li> wrote:
> >> <SNIP>
> I am not familiar with this message:
> 
> "Device supports SMART and is Disabled"
> 
> but there are lots of posts out there with folks asking about it.
> Possibly you can figure out what's causing smart not to turn on, get
> it turned on and get some info.
right , thats why i writing to the scsi and lsi devs , because die disk supports smart and lots of other features, but the controller driver failes to enable support for this.

Now i get that the mpt driver provides /dev/sg[0-1] 

root@outpost:~# smartctl -d scsi -x /dev/sg1
smartctl 6.2 2013-07-26 r3841 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Logical Unit id:      0x50000394e2380537
Serial number:        7365P4KNT
Device type:          disk
Local Time is:        Fri Mar  7 09:54:47 2014 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Disabled
Temperature Warning:  Disabled or Not Supported
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Error Counter logging not supported

Device does not support Self Test logging
Device does not support Background scan results logging
scsiPrintSasPhy Log Sense Failed [unsupported field in scsi command]

so i had read those posts and i tried but it won't work.

root@outpost:~/lsiutil# smartctl -r scsiioctl,3 -s on -d scsi /dev/sg1
smartctl 6.2 2013-07-26 r3841 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

 [inquiry: 12 01 00 01 fc 00 ]
  scsi_status=0x0, host_status=0x0, driver_status=0x0
  info=0x0  duration=0 milliseconds  resid=500
  Incoming data, len=8:
 00     00 00 00 04 00 80 83 89                                                 
 [inquiry: 12 00 00 00 24 00 ]
  scsi_status=0x0, host_status=0x0, driver_status=0x0
  info=0x0  duration=4 milliseconds  resid=0
  Incoming data, len=36:
 00     00 00 05 12 45 00 00 02  41 54 41 20 20 20 20 20                        
 10     54 4f 53 48 49 42 41 20  4d 51 30 31 41 42 42 32                        
 20     30 55 20 20                                                             
=== START OF ENABLE/DISABLE COMMANDS SECTION ===
 [mode sense(6): 1a 00 1c 00 40 00 ]
  scsi_status=0x0, host_status=0x0, driver_status=0x0
  info=0x0  duration=4 milliseconds  resid=40
  Incoming data, len=24:
 00     17 00 00 08 00 00 00 00  00 00 02 00 1c 0a 08 06                        
 10     08 00 00 00 00 00 00 00                                                 
 [mode sense(6): 1a 00 5c 00 40 00 ]
  scsi_status=0x0, host_status=0x0, driver_status=0x0
  info=0x0  duration=0 milliseconds  resid=40
  Incoming data, len=24:
 00     17 00 00 08 00 00 00 00  00 00 02 00 1c 0a 00 00                        
 10     00 00 00 00 00 00 00 00                                                 
 00     00 00 00 08 00 00 00 00  00 00 02 00 1c 0a 08 06                        
 10     08 00 00 00 00 00 00 00                                                 
 [mode select(6): 15 10 00 00 18 00 ]
  Outgoing data, len=24:
  scsi_status=0x2, host_status=0x0, driver_status=0x8
  info=0x1  duration=0 milliseconds  resid=24
  >>> Sense buffer, len=18:
 00     70 00 0b 00 00 00 00 0a  00 00 00 00 00 00 00 00                        
 10     00 00                                                                   
  status=2: sense_key=b asc=0 ascq=0
unable to enable Exception control and warning [aborted command]


> 
> Sorry I cannot help more.
> 
> Good luck,
> Mark

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Slow I/O performance on SAS1064
       [not found] <CAN7X1Umb_wyTCk9w590AnSTFgL=3CC5x8zGNiGZoAdNEej9TCQ@mail.gmail.com>
@ 2014-03-08 20:14 ` markus
  0 siblings, 0 replies; 6+ messages in thread
From: markus @ 2014-03-08 20:14 UTC (permalink / raw)
  To: Purush Gupta
  Cc: Nagalakshmi Nandigama, Sreekanth Reddy, support,
	DL-MPTFusionLinux, linux-scsi, Linux Kernel List

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On Fri, Mar 07, 2014 at 08:24:48AM -0800, Purush Gupta wrote:
> I noticed you had benchmarked only sda target, is the behavior same on
> other targets?
> 
Yes the other disk are equal to sda , but they are used with zfs and there were other preformance problems because of the multible disk access.
config:

        NAME        STATE     READ WRITE CKSUM
        home        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     3
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0



> Since they are SCSI targets, have you explored sg_utils [
> http://sg.danny.cz/sg/uu_index.html]
> 
> And for benchmarking I also recommend FIO https://github.com/axboe/fio
> 
Thanks for the tips.
> Good luck,
> Purush
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2014-03-08 20:09 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <CAN7X1Umb_wyTCk9w590AnSTFgL=3CC5x8zGNiGZoAdNEej9TCQ@mail.gmail.com>
2014-03-08 20:14 ` Slow I/O performance on SAS1064 markus
2014-03-05 17:50 Markus
2014-03-05 18:21 ` Mark Knecht
2014-03-05 21:40   ` markus
2014-03-06 20:51     ` Mark Knecht
2014-03-07  9:46       ` markus

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox