* can't attach @ 2014-10-03 20:21 Stan Hoeppner 2014-10-03 20:37 ` Slava Pestov 0 siblings, 1 reply; 11+ messages in thread From: Stan Hoeppner @ 2014-10-03 20:21 UTC (permalink / raw) To: linux-bcache # ls -la /sys/fs/bcache total 0 drwxr-xr-x 3 root root 0 Oct 3 14:39 . drwxr-xr-x 6 root root 0 Oct 2 16:32 .. drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 --w------- 1 root root 4096 Oct 3 14:39 register --w------- 1 root root 4096 Oct 3 14:50 register_quiet # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach bash: echo: write error: Invalid argument What's the problem here? Thanks, Stan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 20:21 can't attach Stan Hoeppner @ 2014-10-03 20:37 ` Slava Pestov 2014-10-03 20:50 ` Stan Hoeppner 0 siblings, 1 reply; 11+ messages in thread From: Slava Pestov @ 2014-10-03 20:37 UTC (permalink / raw) To: Stan Hoeppner; +Cc: linux-bcache Please include any bcache messages printed in dmesg when this happens. On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: > # ls -la /sys/fs/bcache > total 0 > drwxr-xr-x 3 root root 0 Oct 3 14:39 . > drwxr-xr-x 6 root root 0 Oct 2 16:32 .. > drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 > --w------- 1 root root 4096 Oct 3 14:39 register > --w------- 1 root root 4096 Oct 3 14:50 register_quiet > > > # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach > bash: echo: write error: Invalid argument > > > What's the problem here? > > Thanks, > Stan > -- > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 20:37 ` Slava Pestov @ 2014-10-03 20:50 ` Stan Hoeppner 2014-10-03 20:57 ` Slava Pestov 0 siblings, 1 reply; 11+ messages in thread From: Stan Hoeppner @ 2014-10-03 20:50 UTC (permalink / raw) To: Slava Pestov; +Cc: linux-bcache On 10/03/2014 03:37 PM, Slava Pestov wrote: > Please include any bcache messages printed in dmesg when this happens. [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 [87579.418947] : cache set not found For completeness, these are the steps are performed, soup to nuts: # wipefs -a /dev/dm-4 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) # wipefs -a /dev/dm-7 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) # wipefs -a /dev/sdc # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc Already a bcache device on /dev/sdc, overwrite with --wipe-bcache # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 Set UUID: 8246b830-95d4-420d-aff8-894efc348801 version: 0 nbuckets: 761856 block_size: 1 bucket_size: 1024 nr_in_set: 1 nr_this_dev: 0 first_bucket: 1 UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 Set UUID: 8246b830-95d4-420d-aff8-894efc348801 version: 1 block_size: 1 data_offset: 16 UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f Set UUID: 8246b830-95d4-420d-aff8-894efc348801 version: 1 block_size: 1 data_offset: 16 # echo /dev/dm-4 > /sys/fs/bcache/register # echo /dev/dm-7 > /sys/fs/bcache/register # echo /dev/sdc > /sys/fs/bcache/register # ls -la /dev/b* brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 # mkfs.xfs /dev/bcache0 meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks = sectsz=512 attr=2, projid32bit=0 data = bsize=4096 blocks=11709285374, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 # mkfs.xfs /dev/bcache1 meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks = sectsz=512 attr=2, projid32bit=0 data = bsize=4096 blocks=11709285374, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A # ls -la /sys/fs/bcache total 0 drwxr-xr-x 3 root root 0 Oct 3 14:39 . drwxr-xr-x 6 root root 0 Oct 2 16:32 .. drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 --w------- 1 root root 4096 Oct 3 14:39 register --w------- 1 root root 4096 Oct 3 14:50 register_quiet # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach bash: echo: write error: Invalid argument > On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >> # ls -la /sys/fs/bcache >> total 0 >> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >> --w------- 1 root root 4096 Oct 3 14:39 register >> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >> >> >> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >> bash: echo: write error: Invalid argument >> >> >> What's the problem here? >> >> Thanks, >> Stan >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 20:50 ` Stan Hoeppner @ 2014-10-03 20:57 ` Slava Pestov 2014-10-03 21:14 ` Stan Hoeppner 0 siblings, 1 reply; 11+ messages in thread From: Slava Pestov @ 2014-10-03 20:57 UTC (permalink / raw) To: Stan Hoeppner; +Cc: linux-bcache It looks like everything went fine, what is the problem? On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: > > On 10/03/2014 03:37 PM, Slava Pestov wrote: >> Please include any bcache messages printed in dmesg when this happens. > > [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached > [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 > [87579.418947] : cache set not found > > For completeness, these are the steps are performed, soup to nuts: > > # wipefs -a /dev/dm-4 > 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) > # wipefs -a /dev/dm-7 > 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) > # wipefs -a /dev/sdc > # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc > Already a bcache device on /dev/sdc, overwrite with --wipe-bcache > # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache > UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 > Set UUID: 8246b830-95d4-420d-aff8-894efc348801 > version: 0 > nbuckets: 761856 > block_size: 1 > bucket_size: 1024 > nr_in_set: 1 > nr_this_dev: 0 > first_bucket: 1 > UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 > Set UUID: 8246b830-95d4-420d-aff8-894efc348801 > version: 1 > block_size: 1 > data_offset: 16 > UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f > Set UUID: 8246b830-95d4-420d-aff8-894efc348801 > version: 1 > block_size: 1 > data_offset: 16 > # echo /dev/dm-4 > /sys/fs/bcache/register > # echo /dev/dm-7 > /sys/fs/bcache/register > # echo /dev/sdc > /sys/fs/bcache/register > > # ls -la /dev/b* > brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 > brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 > > # mkfs.xfs /dev/bcache0 > meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks > = sectsz=512 attr=2, projid32bit=0 > data = bsize=4096 blocks=11709285374, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal log bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > # mkfs.xfs /dev/bcache1 > meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks > = sectsz=512 attr=2, projid32bit=0 > data = bsize=4096 blocks=11709285374, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal log bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A > # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A > > # ls -la /sys/fs/bcache > total 0 > drwxr-xr-x 3 root root 0 Oct 3 14:39 . > drwxr-xr-x 6 root root 0 Oct 2 16:32 .. > drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 > --w------- 1 root root 4096 Oct 3 14:39 register > --w------- 1 root root 4096 Oct 3 14:50 register_quiet > > # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach > bash: echo: write error: Invalid argument > > >> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>> # ls -la /sys/fs/bcache >>> total 0 >>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>> --w------- 1 root root 4096 Oct 3 14:39 register >>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>> >>> >>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>> bash: echo: write error: Invalid argument >>> >>> >>> What's the problem here? >>> >>> Thanks, >>> Stan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 20:57 ` Slava Pestov @ 2014-10-03 21:14 ` Stan Hoeppner 2014-10-03 21:21 ` Slava Pestov 0 siblings, 1 reply; 11+ messages in thread From: Stan Hoeppner @ 2014-10-03 21:14 UTC (permalink / raw) To: Slava Pestov; +Cc: linux-bcache I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. bash: echo: write error: Invalid argument How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. Thanks, Stan On 10/03/2014 03:57 PM, Slava Pestov wrote: > It looks like everything went fine, what is the problem? > > On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >> >> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>> Please include any bcache messages printed in dmesg when this happens. >> >> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >> [87579.418947] : cache set not found >> >> For completeness, these are the steps are performed, soup to nuts: >> >> # wipefs -a /dev/dm-4 >> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >> # wipefs -a /dev/dm-7 >> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >> # wipefs -a /dev/sdc >> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >> version: 0 >> nbuckets: 761856 >> block_size: 1 >> bucket_size: 1024 >> nr_in_set: 1 >> nr_this_dev: 0 >> first_bucket: 1 >> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >> version: 1 >> block_size: 1 >> data_offset: 16 >> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >> version: 1 >> block_size: 1 >> data_offset: 16 >> # echo /dev/dm-4 > /sys/fs/bcache/register >> # echo /dev/dm-7 > /sys/fs/bcache/register >> # echo /dev/sdc > /sys/fs/bcache/register >> >> # ls -la /dev/b* >> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >> >> # mkfs.xfs /dev/bcache0 >> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >> = sectsz=512 attr=2, projid32bit=0 >> data = bsize=4096 blocks=11709285374, imaxpct=5 >> = sunit=0 swidth=0 blks >> naming =version 2 bsize=4096 ascii-ci=0 >> log =internal log bsize=4096 blocks=521728, version=2 >> = sectsz=512 sunit=0 blks, lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> # mkfs.xfs /dev/bcache1 >> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >> = sectsz=512 attr=2, projid32bit=0 >> data = bsize=4096 blocks=11709285374, imaxpct=5 >> = sunit=0 swidth=0 blks >> naming =version 2 bsize=4096 ascii-ci=0 >> log =internal log bsize=4096 blocks=521728, version=2 >> = sectsz=512 sunit=0 blks, lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >> >> # ls -la /sys/fs/bcache >> total 0 >> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >> --w------- 1 root root 4096 Oct 3 14:39 register >> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >> >> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >> bash: echo: write error: Invalid argument >> >> >>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>> # ls -la /sys/fs/bcache >>>> total 0 >>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>> >>>> >>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>> bash: echo: write error: Invalid argument >>>> >>>> >>>> What's the problem here? >>>> >>>> Thanks, >>>> Stan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 21:14 ` Stan Hoeppner @ 2014-10-03 21:21 ` Slava Pestov 2014-10-03 21:37 ` Stan Hoeppner 0 siblings, 1 reply; 11+ messages in thread From: Slava Pestov @ 2014-10-03 21:21 UTC (permalink / raw) To: Stan Hoeppner; +Cc: linux-bcache Hi Stan, The 'register' command attaches the backing device, you don't need to do the separate 'attach' step. Perhaps the instructions you are following are out of date. Slava On Fri, Oct 3, 2014 at 2:14 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: > I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. > > bash: echo: write error: Invalid argument > > How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? > > FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. > > I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. > > Thanks, > Stan > > > > On 10/03/2014 03:57 PM, Slava Pestov wrote: >> It looks like everything went fine, what is the problem? >> >> On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>> >>> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>>> Please include any bcache messages printed in dmesg when this happens. >>> >>> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >>> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >>> [87579.418947] : cache set not found >>> >>> For completeness, these are the steps are performed, soup to nuts: >>> >>> # wipefs -a /dev/dm-4 >>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>> # wipefs -a /dev/dm-7 >>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>> # wipefs -a /dev/sdc >>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >>> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >>> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>> version: 0 >>> nbuckets: 761856 >>> block_size: 1 >>> bucket_size: 1024 >>> nr_in_set: 1 >>> nr_this_dev: 0 >>> first_bucket: 1 >>> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>> version: 1 >>> block_size: 1 >>> data_offset: 16 >>> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>> version: 1 >>> block_size: 1 >>> data_offset: 16 >>> # echo /dev/dm-4 > /sys/fs/bcache/register >>> # echo /dev/dm-7 > /sys/fs/bcache/register >>> # echo /dev/sdc > /sys/fs/bcache/register >>> >>> # ls -la /dev/b* >>> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >>> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >>> >>> # mkfs.xfs /dev/bcache0 >>> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >>> = sectsz=512 attr=2, projid32bit=0 >>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>> = sunit=0 swidth=0 blks >>> naming =version 2 bsize=4096 ascii-ci=0 >>> log =internal log bsize=4096 blocks=521728, version=2 >>> = sectsz=512 sunit=0 blks, lazy-count=1 >>> realtime =none extsz=4096 blocks=0, rtextents=0 >>> >>> # mkfs.xfs /dev/bcache1 >>> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >>> = sectsz=512 attr=2, projid32bit=0 >>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>> = sunit=0 swidth=0 blks >>> naming =version 2 bsize=4096 ascii-ci=0 >>> log =internal log bsize=4096 blocks=521728, version=2 >>> = sectsz=512 sunit=0 blks, lazy-count=1 >>> realtime =none extsz=4096 blocks=0, rtextents=0 >>> >>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >>> >>> # ls -la /sys/fs/bcache >>> total 0 >>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>> --w------- 1 root root 4096 Oct 3 14:39 register >>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>> >>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>> bash: echo: write error: Invalid argument >>> >>> >>>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>> # ls -la /sys/fs/bcache >>>>> total 0 >>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>> >>>>> >>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>> bash: echo: write error: Invalid argument >>>>> >>>>> >>>>> What's the problem here? >>>>> >>>>> Thanks, >>>>> Stan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 21:21 ` Slava Pestov @ 2014-10-03 21:37 ` Stan Hoeppner 2014-10-03 21:49 ` Slava Pestov 0 siblings, 1 reply; 11+ messages in thread From: Stan Hoeppner @ 2014-10-03 21:37 UTC (permalink / raw) To: Slava Pestov; +Cc: linux-bcache Probably so. I'm working from: https://www.kernel.org/doc/Documentation/bcache.txt Do I need to manually register the backing devices? I formatted the LUNs and the SSD on the same command line. Instructions above say you don't need to register manually if doing it that way. Also, I'm not using the udev rules, but doing everything manually at this point. And there is a reason for that. I don't have complete control/access to the development system. The rootfs is NFS mounted read only, and changes such as udev rules take days... Thanks, Stan On 10/03/2014 04:21 PM, Slava Pestov wrote: > Hi Stan, > > The 'register' command attaches the backing device, you don't need to > do the separate 'attach' step. Perhaps the instructions you are > following are out of date. > > Slava > > On Fri, Oct 3, 2014 at 2:14 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >> I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. >> >> bash: echo: write error: Invalid argument >> >> How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? >> >> FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. >> >> I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. >> >> Thanks, >> Stan >> >> >> >> On 10/03/2014 03:57 PM, Slava Pestov wrote: >>> It looks like everything went fine, what is the problem? >>> >>> On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>> >>>> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>>>> Please include any bcache messages printed in dmesg when this happens. >>>> >>>> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >>>> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >>>> [87579.418947] : cache set not found >>>> >>>> For completeness, these are the steps are performed, soup to nuts: >>>> >>>> # wipefs -a /dev/dm-4 >>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>> # wipefs -a /dev/dm-7 >>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>> # wipefs -a /dev/sdc >>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >>>> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >>>> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>> version: 0 >>>> nbuckets: 761856 >>>> block_size: 1 >>>> bucket_size: 1024 >>>> nr_in_set: 1 >>>> nr_this_dev: 0 >>>> first_bucket: 1 >>>> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>> version: 1 >>>> block_size: 1 >>>> data_offset: 16 >>>> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>> version: 1 >>>> block_size: 1 >>>> data_offset: 16 >>>> # echo /dev/dm-4 > /sys/fs/bcache/register >>>> # echo /dev/dm-7 > /sys/fs/bcache/register >>>> # echo /dev/sdc > /sys/fs/bcache/register >>>> >>>> # ls -la /dev/b* >>>> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >>>> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >>>> >>>> # mkfs.xfs /dev/bcache0 >>>> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >>>> = sectsz=512 attr=2, projid32bit=0 >>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>> = sunit=0 swidth=0 blks >>>> naming =version 2 bsize=4096 ascii-ci=0 >>>> log =internal log bsize=4096 blocks=521728, version=2 >>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>> >>>> # mkfs.xfs /dev/bcache1 >>>> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >>>> = sectsz=512 attr=2, projid32bit=0 >>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>> = sunit=0 swidth=0 blks >>>> naming =version 2 bsize=4096 ascii-ci=0 >>>> log =internal log bsize=4096 blocks=521728, version=2 >>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>> >>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >>>> >>>> # ls -la /sys/fs/bcache >>>> total 0 >>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>> >>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>> bash: echo: write error: Invalid argument >>>> >>>> >>>>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>> # ls -la /sys/fs/bcache >>>>>> total 0 >>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>> >>>>>> >>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>> bash: echo: write error: Invalid argument >>>>>> >>>>>> >>>>>> What's the problem here? >>>>>> >>>>>> Thanks, >>>>>> Stan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 21:37 ` Stan Hoeppner @ 2014-10-03 21:49 ` Slava Pestov 2014-10-03 22:05 ` Stan Hoeppner 0 siblings, 1 reply; 11+ messages in thread From: Slava Pestov @ 2014-10-03 21:49 UTC (permalink / raw) To: Stan Hoeppner; +Cc: linux-bcache Hi Stan, It looks like those docs are out of date. Echoing the names of the cache and backing devices to 'register' is sufficient. 'attach' is for adding a new backing device to an existing cache set after the fact. Slava On Fri, Oct 3, 2014 at 2:37 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: > Probably so. I'm working from: > https://www.kernel.org/doc/Documentation/bcache.txt > > Do I need to manually register the backing devices? I formatted the LUNs and the SSD on the same command line. Instructions above say you don't need to register manually if doing it that way. Also, I'm not using the udev rules, but doing everything manually at this point. And there is a reason for that. I don't have complete control/access to the development system. The rootfs is NFS mounted read only, and changes such as udev rules take days... > > > Thanks, > Stan > > > On 10/03/2014 04:21 PM, Slava Pestov wrote: >> Hi Stan, >> >> The 'register' command attaches the backing device, you don't need to >> do the separate 'attach' step. Perhaps the instructions you are >> following are out of date. >> >> Slava >> >> On Fri, Oct 3, 2014 at 2:14 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>> I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. >>> >>> bash: echo: write error: Invalid argument >>> >>> How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? >>> >>> FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. >>> >>> I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. >>> >>> Thanks, >>> Stan >>> >>> >>> >>> On 10/03/2014 03:57 PM, Slava Pestov wrote: >>>> It looks like everything went fine, what is the problem? >>>> >>>> On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>> >>>>> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>>>>> Please include any bcache messages printed in dmesg when this happens. >>>>> >>>>> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >>>>> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >>>>> [87579.418947] : cache set not found >>>>> >>>>> For completeness, these are the steps are performed, soup to nuts: >>>>> >>>>> # wipefs -a /dev/dm-4 >>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>> # wipefs -a /dev/dm-7 >>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>> # wipefs -a /dev/sdc >>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >>>>> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >>>>> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>> version: 0 >>>>> nbuckets: 761856 >>>>> block_size: 1 >>>>> bucket_size: 1024 >>>>> nr_in_set: 1 >>>>> nr_this_dev: 0 >>>>> first_bucket: 1 >>>>> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>> version: 1 >>>>> block_size: 1 >>>>> data_offset: 16 >>>>> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>> version: 1 >>>>> block_size: 1 >>>>> data_offset: 16 >>>>> # echo /dev/dm-4 > /sys/fs/bcache/register >>>>> # echo /dev/dm-7 > /sys/fs/bcache/register >>>>> # echo /dev/sdc > /sys/fs/bcache/register >>>>> >>>>> # ls -la /dev/b* >>>>> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >>>>> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >>>>> >>>>> # mkfs.xfs /dev/bcache0 >>>>> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >>>>> = sectsz=512 attr=2, projid32bit=0 >>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>> = sunit=0 swidth=0 blks >>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>> >>>>> # mkfs.xfs /dev/bcache1 >>>>> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >>>>> = sectsz=512 attr=2, projid32bit=0 >>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>> = sunit=0 swidth=0 blks >>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>> >>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >>>>> >>>>> # ls -la /sys/fs/bcache >>>>> total 0 >>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>> >>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>> bash: echo: write error: Invalid argument >>>>> >>>>> >>>>>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>>> # ls -la /sys/fs/bcache >>>>>>> total 0 >>>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>>> >>>>>>> >>>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>>> bash: echo: write error: Invalid argument >>>>>>> >>>>>>> >>>>>>> What's the problem here? >>>>>>> >>>>>>> Thanks, >>>>>>> Stan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 21:49 ` Slava Pestov @ 2014-10-03 22:05 ` Stan Hoeppner 2014-10-03 22:08 ` Slava Pestov 0 siblings, 1 reply; 11+ messages in thread From: Stan Hoeppner @ 2014-10-03 22:05 UTC (permalink / raw) To: Slava Pestov; +Cc: linux-bcache Would you please kindly provide a link to more recent docs? I notice I have a stripe_size file that's not mentioned in any of the docs I have linked, including the official evil pirate docs. I know Kent was working on stripe write optimization in newer versions. I'm on kernel 3.12.26, so that must have been back ported. Thanks, Stan On 10/03/2014 04:49 PM, Slava Pestov wrote: > Hi Stan, > > It looks like those docs are out of date. Echoing the names of the > cache and backing devices to 'register' is sufficient. 'attach' is for > adding a new backing device to an existing cache set after the fact. > > Slava > > On Fri, Oct 3, 2014 at 2:37 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >> Probably so. I'm working from: >> https://www.kernel.org/doc/Documentation/bcache.txt >> >> Do I need to manually register the backing devices? I formatted the LUNs and the SSD on the same command line. Instructions above say you don't need to register manually if doing it that way. Also, I'm not using the udev rules, but doing everything manually at this point. And there is a reason for that. I don't have complete control/access to the development system. The rootfs is NFS mounted read only, and changes such as udev rules take days... >> >> >> Thanks, >> Stan >> >> >> On 10/03/2014 04:21 PM, Slava Pestov wrote: >>> Hi Stan, >>> >>> The 'register' command attaches the backing device, you don't need to >>> do the separate 'attach' step. Perhaps the instructions you are >>> following are out of date. >>> >>> Slava >>> >>> On Fri, Oct 3, 2014 at 2:14 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>> I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. >>>> >>>> bash: echo: write error: Invalid argument >>>> >>>> How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? >>>> >>>> FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. >>>> >>>> I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. >>>> >>>> Thanks, >>>> Stan >>>> >>>> >>>> >>>> On 10/03/2014 03:57 PM, Slava Pestov wrote: >>>>> It looks like everything went fine, what is the problem? >>>>> >>>>> On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>> >>>>>> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>>>>>> Please include any bcache messages printed in dmesg when this happens. >>>>>> >>>>>> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >>>>>> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >>>>>> [87579.418947] : cache set not found >>>>>> >>>>>> For completeness, these are the steps are performed, soup to nuts: >>>>>> >>>>>> # wipefs -a /dev/dm-4 >>>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>>> # wipefs -a /dev/dm-7 >>>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>>> # wipefs -a /dev/sdc >>>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >>>>>> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >>>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >>>>>> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>> version: 0 >>>>>> nbuckets: 761856 >>>>>> block_size: 1 >>>>>> bucket_size: 1024 >>>>>> nr_in_set: 1 >>>>>> nr_this_dev: 0 >>>>>> first_bucket: 1 >>>>>> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>> version: 1 >>>>>> block_size: 1 >>>>>> data_offset: 16 >>>>>> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>> version: 1 >>>>>> block_size: 1 >>>>>> data_offset: 16 >>>>>> # echo /dev/dm-4 > /sys/fs/bcache/register >>>>>> # echo /dev/dm-7 > /sys/fs/bcache/register >>>>>> # echo /dev/sdc > /sys/fs/bcache/register >>>>>> >>>>>> # ls -la /dev/b* >>>>>> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >>>>>> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >>>>>> >>>>>> # mkfs.xfs /dev/bcache0 >>>>>> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >>>>>> = sectsz=512 attr=2, projid32bit=0 >>>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>>> = sunit=0 swidth=0 blks >>>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>>> >>>>>> # mkfs.xfs /dev/bcache1 >>>>>> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >>>>>> = sectsz=512 attr=2, projid32bit=0 >>>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>>> = sunit=0 swidth=0 blks >>>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>>> >>>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >>>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >>>>>> >>>>>> # ls -la /sys/fs/bcache >>>>>> total 0 >>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>> >>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>> bash: echo: write error: Invalid argument >>>>>> >>>>>> >>>>>>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>>>> # ls -la /sys/fs/bcache >>>>>>>> total 0 >>>>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>>>> >>>>>>>> >>>>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>>>> bash: echo: write error: Invalid argument >>>>>>>> >>>>>>>> >>>>>>>> What's the problem here? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Stan >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 22:05 ` Stan Hoeppner @ 2014-10-03 22:08 ` Slava Pestov 2014-10-03 22:43 ` Stan Hoeppner 0 siblings, 1 reply; 11+ messages in thread From: Slava Pestov @ 2014-10-03 22:08 UTC (permalink / raw) To: Stan Hoeppner; +Cc: linux-bcache Hi Stan, Is there any chance you could upgrade to a more recent kernel? As far as I know, recent bcache bug fixes are not being backported to older kernels because bcache depends on some block layer changes that went in more recently. On Fri, Oct 3, 2014 at 3:05 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: > Would you please kindly provide a link to more recent docs? I notice I have a stripe_size file that's not mentioned in any of the docs I have linked, including the official evil pirate docs. I know Kent was working on stripe write optimization in newer versions. I'm on kernel 3.12.26, so that must have been back ported. > > Thanks, > Stan > > > On 10/03/2014 04:49 PM, Slava Pestov wrote: >> Hi Stan, >> >> It looks like those docs are out of date. Echoing the names of the >> cache and backing devices to 'register' is sufficient. 'attach' is for >> adding a new backing device to an existing cache set after the fact. >> >> Slava >> >> On Fri, Oct 3, 2014 at 2:37 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>> Probably so. I'm working from: >>> https://www.kernel.org/doc/Documentation/bcache.txt >>> >>> Do I need to manually register the backing devices? I formatted the LUNs and the SSD on the same command line. Instructions above say you don't need to register manually if doing it that way. Also, I'm not using the udev rules, but doing everything manually at this point. And there is a reason for that. I don't have complete control/access to the development system. The rootfs is NFS mounted read only, and changes such as udev rules take days... >>> >>> >>> Thanks, >>> Stan >>> >>> >>> On 10/03/2014 04:21 PM, Slava Pestov wrote: >>>> Hi Stan, >>>> >>>> The 'register' command attaches the backing device, you don't need to >>>> do the separate 'attach' step. Perhaps the instructions you are >>>> following are out of date. >>>> >>>> Slava >>>> >>>> On Fri, Oct 3, 2014 at 2:14 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>> I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. >>>>> >>>>> bash: echo: write error: Invalid argument >>>>> >>>>> How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? >>>>> >>>>> FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. >>>>> >>>>> I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. >>>>> >>>>> Thanks, >>>>> Stan >>>>> >>>>> >>>>> >>>>> On 10/03/2014 03:57 PM, Slava Pestov wrote: >>>>>> It looks like everything went fine, what is the problem? >>>>>> >>>>>> On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>>> >>>>>>> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>>>>>>> Please include any bcache messages printed in dmesg when this happens. >>>>>>> >>>>>>> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >>>>>>> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >>>>>>> [87579.418947] : cache set not found >>>>>>> >>>>>>> For completeness, these are the steps are performed, soup to nuts: >>>>>>> >>>>>>> # wipefs -a /dev/dm-4 >>>>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>>>> # wipefs -a /dev/dm-7 >>>>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>>>> # wipefs -a /dev/sdc >>>>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >>>>>>> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >>>>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >>>>>>> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >>>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>>> version: 0 >>>>>>> nbuckets: 761856 >>>>>>> block_size: 1 >>>>>>> bucket_size: 1024 >>>>>>> nr_in_set: 1 >>>>>>> nr_this_dev: 0 >>>>>>> first_bucket: 1 >>>>>>> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >>>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>>> version: 1 >>>>>>> block_size: 1 >>>>>>> data_offset: 16 >>>>>>> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >>>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>>> version: 1 >>>>>>> block_size: 1 >>>>>>> data_offset: 16 >>>>>>> # echo /dev/dm-4 > /sys/fs/bcache/register >>>>>>> # echo /dev/dm-7 > /sys/fs/bcache/register >>>>>>> # echo /dev/sdc > /sys/fs/bcache/register >>>>>>> >>>>>>> # ls -la /dev/b* >>>>>>> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >>>>>>> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >>>>>>> >>>>>>> # mkfs.xfs /dev/bcache0 >>>>>>> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >>>>>>> = sectsz=512 attr=2, projid32bit=0 >>>>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>>>> = sunit=0 swidth=0 blks >>>>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>>>> >>>>>>> # mkfs.xfs /dev/bcache1 >>>>>>> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >>>>>>> = sectsz=512 attr=2, projid32bit=0 >>>>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>>>> = sunit=0 swidth=0 blks >>>>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>>>> >>>>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >>>>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >>>>>>> >>>>>>> # ls -la /sys/fs/bcache >>>>>>> total 0 >>>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>>> >>>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>>> bash: echo: write error: Invalid argument >>>>>>> >>>>>>> >>>>>>>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>>>>> # ls -la /sys/fs/bcache >>>>>>>>> total 0 >>>>>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>>>>> >>>>>>>>> >>>>>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>>>>> bash: echo: write error: Invalid argument >>>>>>>>> >>>>>>>>> >>>>>>>>> What's the problem here? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Stan >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: can't attach 2014-10-03 22:08 ` Slava Pestov @ 2014-10-03 22:43 ` Stan Hoeppner 0 siblings, 0 replies; 11+ messages in thread From: Stan Hoeppner @ 2014-10-03 22:43 UTC (permalink / raw) To: Slava Pestov; +Cc: linux-bcache I'd actually just like information on how to confirm caching is enabled and working for both of the LUNs mentioned in my long output below. I'm seeing normal write behavior for one, and odd behavior (2x streaming throughput and some apparent permissions issues) for the other. This leads me to believe bcache is active for the 2nd LUN and is causing some breakage while the first LUN is not being cached. Thanks, Stan On 10/03/2014 05:08 PM, Slava Pestov wrote: > Hi Stan, > > Is there any chance you could upgrade to a more recent kernel? As far > as I know, recent bcache bug fixes are not being backported to older > kernels because bcache depends on some block layer changes that went > in more recently. > > On Fri, Oct 3, 2014 at 3:05 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >> Would you please kindly provide a link to more recent docs? I notice I have a stripe_size file that's not mentioned in any of the docs I have linked, including the official evil pirate docs. I know Kent was working on stripe write optimization in newer versions. I'm on kernel 3.12.26, so that must have been back ported. >> >> Thanks, >> Stan >> >> >> On 10/03/2014 04:49 PM, Slava Pestov wrote: >>> Hi Stan, >>> >>> It looks like those docs are out of date. Echoing the names of the >>> cache and backing devices to 'register' is sufficient. 'attach' is for >>> adding a new backing device to an existing cache set after the fact. >>> >>> Slava >>> >>> On Fri, Oct 3, 2014 at 2:37 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>> Probably so. I'm working from: >>>> https://www.kernel.org/doc/Documentation/bcache.txt >>>> >>>> Do I need to manually register the backing devices? I formatted the LUNs and the SSD on the same command line. Instructions above say you don't need to register manually if doing it that way. Also, I'm not using the udev rules, but doing everything manually at this point. And there is a reason for that. I don't have complete control/access to the development system. The rootfs is NFS mounted read only, and changes such as udev rules take days... >>>> >>>> >>>> Thanks, >>>> Stan >>>> >>>> >>>> On 10/03/2014 04:21 PM, Slava Pestov wrote: >>>>> Hi Stan, >>>>> >>>>> The 'register' command attaches the backing device, you don't need to >>>>> do the separate 'attach' step. Perhaps the instructions you are >>>>> following are out of date. >>>>> >>>>> Slava >>>>> >>>>> On Fri, Oct 3, 2014 at 2:14 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>> I assumed it did not go fine due to the immediate error on the echo command. FYI, this is my first attempt at bcache. >>>>>> >>>>>> bash: echo: write error: Invalid argument >>>>>> >>>>>> How can I confirm both dm-4 and dm-7 (bcache0 and bcache1) are both attached and that bcache is running? >>>>>> >>>>>> FYI, this is my first attempt at bcache, so I'm following the instructions precisely. Also, dm-4 and dm-7 are 44 TB RAID6 LUNs. If my testing shows substantial benefit in decreasing RMW we will be looking at deploying bcache with one or more mirrored pairs of SSDs to 14 of these arrays. That's per system. And we'll be deploying 100s of such systems. >>>>>> >>>>>> I emailed Kent's gmail address about this about a month ago and never received a response. At some point, probably next year some time, the company I'm working for may want to contract with him for a short time before moving into production with this. >>>>>> >>>>>> Thanks, >>>>>> Stan >>>>>> >>>>>> >>>>>> >>>>>> On 10/03/2014 03:57 PM, Slava Pestov wrote: >>>>>>> It looks like everything went fine, what is the problem? >>>>>>> >>>>>>> On Fri, Oct 3, 2014 at 1:50 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>>>> >>>>>>>> On 10/03/2014 03:37 PM, Slava Pestov wrote: >>>>>>>>> Please include any bcache messages printed in dmesg when this happens. >>>>>>>> >>>>>>>> [87579.418945] bcache: bch_cached_dev_attach() Can't attach dm-4: already attached >>>>>>>> [87579.418947] bcache: __cached_dev_store() Can't attach 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>> [87579.418947] : cache set not found >>>>>>>> >>>>>>>> For completeness, these are the steps are performed, soup to nuts: >>>>>>>> >>>>>>>> # wipefs -a /dev/dm-4 >>>>>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>>>>> # wipefs -a /dev/dm-7 >>>>>>>> 4 bytes [58 46 53 42] erased at offset 0x0 (xfs) >>>>>>>> # wipefs -a /dev/sdc >>>>>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc >>>>>>>> Already a bcache device on /dev/sdc, overwrite with --wipe-bcache >>>>>>>> # ./make-bcache -B /dev/dm-4 /dev/dm-7 -C /dev/sdc --wipe-bcache >>>>>>>> UUID: 501486ea-d614-4a32-a137-bc8e004fcb85 >>>>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>> version: 0 >>>>>>>> nbuckets: 761856 >>>>>>>> block_size: 1 >>>>>>>> bucket_size: 1024 >>>>>>>> nr_in_set: 1 >>>>>>>> nr_this_dev: 0 >>>>>>>> first_bucket: 1 >>>>>>>> UUID: 261cf532-ca93-45b7-a8fb-1eff57078953 >>>>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>> version: 1 >>>>>>>> block_size: 1 >>>>>>>> data_offset: 16 >>>>>>>> UUID: eeca31ab-a632-4d7b-94f6-35e9b878344f >>>>>>>> Set UUID: 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>> version: 1 >>>>>>>> block_size: 1 >>>>>>>> data_offset: 16 >>>>>>>> # echo /dev/dm-4 > /sys/fs/bcache/register >>>>>>>> # echo /dev/dm-7 > /sys/fs/bcache/register >>>>>>>> # echo /dev/sdc > /sys/fs/bcache/register >>>>>>>> >>>>>>>> # ls -la /dev/b* >>>>>>>> brw-rw---- 1 root disk 253, 0 Oct 3 14:39 /dev/bcache0 >>>>>>>> brw-rw---- 1 root disk 253, 1 Oct 3 14:39 /dev/bcache1 >>>>>>>> >>>>>>>> # mkfs.xfs /dev/bcache0 >>>>>>>> meta-data=/dev/bcache0 isize=256 agcount=44, agsize=268435455 blks >>>>>>>> = sectsz=512 attr=2, projid32bit=0 >>>>>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>>>>> = sunit=0 swidth=0 blks >>>>>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>>>>> >>>>>>>> # mkfs.xfs /dev/bcache1 >>>>>>>> meta-data=/dev/bcache1 isize=256 agcount=44, agsize=268435455 blks >>>>>>>> = sectsz=512 attr=2, projid32bit=0 >>>>>>>> data = bsize=4096 blocks=11709285374, imaxpct=5 >>>>>>>> = sunit=0 swidth=0 blks >>>>>>>> naming =version 2 bsize=4096 ascii-ci=0 >>>>>>>> log =internal log bsize=4096 blocks=521728, version=2 >>>>>>>> = sectsz=512 sunit=0 blks, lazy-count=1 >>>>>>>> realtime =none extsz=4096 blocks=0, rtextents=0 >>>>>>>> >>>>>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache0 /mnt/V1A >>>>>>>> # mount -o inode64,nobarrier,noatime,nodiratime /dev/bcache1 /mnt/V2A >>>>>>>> >>>>>>>> # ls -la /sys/fs/bcache >>>>>>>> total 0 >>>>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>>>> >>>>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>>>> bash: echo: write error: Invalid argument >>>>>>>> >>>>>>>> >>>>>>>>> On Fri, Oct 3, 2014 at 1:21 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote: >>>>>>>>>> # ls -la /sys/fs/bcache >>>>>>>>>> total 0 >>>>>>>>>> drwxr-xr-x 3 root root 0 Oct 3 14:39 . >>>>>>>>>> drwxr-xr-x 6 root root 0 Oct 2 16:32 .. >>>>>>>>>> drwxr-xr-x 7 root root 0 Oct 3 14:50 8246b830-95d4-420d-aff8-894efc348801 >>>>>>>>>> --w------- 1 root root 4096 Oct 3 14:39 register >>>>>>>>>> --w------- 1 root root 4096 Oct 3 14:50 register_quiet >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> # echo 8246b830-95d4-420d-aff8-894efc348801 > /sys/block/bcache0/bcache/attach >>>>>>>>>> bash: echo: write error: Invalid argument >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> What's the problem here? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Stan >>>>>>>>>> -- >>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-10-03 22:43 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-10-03 20:21 can't attach Stan Hoeppner 2014-10-03 20:37 ` Slava Pestov 2014-10-03 20:50 ` Stan Hoeppner 2014-10-03 20:57 ` Slava Pestov 2014-10-03 21:14 ` Stan Hoeppner 2014-10-03 21:21 ` Slava Pestov 2014-10-03 21:37 ` Stan Hoeppner 2014-10-03 21:49 ` Slava Pestov 2014-10-03 22:05 ` Stan Hoeppner 2014-10-03 22:08 ` Slava Pestov 2014-10-03 22:43 ` Stan Hoeppner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).