* [dm-crypt] trouble with dm-crypt and NFS: server hangs @ 2014-03-20 14:48 Stephen Cousins 2014-03-20 19:16 ` Milan Broz 0 siblings, 1 reply; 7+ messages in thread From: Stephen Cousins @ 2014-03-20 14:48 UTC (permalink / raw) To: dm-crypt [-- Attachment #1: Type: text/plain, Size: 1393 bytes --] I am testing dm-crypt with a system that serves data over NFS. I'm finding that when serving the data over NFS4 it will go for a little while and then it will just stop copying and networking on the NFS server will stall out so that other connections to other parts of the storage (a non-dm-crypt volume) will become very laggy. Soon after canceling the copy the other connections resume back to normal. I have tried this on two different servers, one with a single disk with ZFS and another with a ZFS raidz2 pool and they both do the same thing. CPU doesn't seem to be an issue. Doing local copies is not a problem. That is, copying to/from these volumes made up of dm-crypted disks from other disks on the server have no problems. One system (the one with the RAID pool) is CentOS 6.4 and the other one is CentOS 6.5. I'm curious if anyone else has run into this before and if so, what can be done to make it work? I'm going to test to see if taking ZFS out of the equation helps. I haven't run into this with ZFS with disks without dm-crypt. Thanks, Steve -- ________________________________________________________________ Steve Cousins Supercomputer Engineer/Administrator Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 561-3574 Orono ME 04469 steve.cousins at maine.edu [-- Attachment #2: Type: text/html, Size: 3056 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs @ 2014-03-20 19:16 ` Milan Broz 0 siblings, 0 replies; 7+ messages in thread From: Milan Broz @ 2014-03-20 19:16 UTC (permalink / raw) To: Stephen Cousins, dm-crypt; +Cc: device-mapper development On 03/20/2014 03:48 PM, Stephen Cousins wrote: > I am testing dm-crypt with a system that serves data over NFS. I'm > finding that when serving the data over NFS4 it will go for a little > while and then it will just stop copying and networking on the NFS > server will stall out so that other connections to other parts of the > storage (a non-dm-crypt volume) will become very laggy. Soon after > canceling the copy the other connections resume back to normal. > > I have tried this on two different servers, one with a single disk > with ZFS and another with a ZFS raidz2 pool and they both do the same > thing. CPU doesn't seem to be an issue. Doing local copies is not a > problem. That is, copying to/from these volumes made up of dm-crypted > disks from other disks on the server have no problems. > > One system (the one with the RAID pool) is CentOS 6.4 and the other > one is CentOS 6.5. > > I'm curious if anyone else has run into this before and if so, what > can be done to make it work? I'm going to test to see if taking ZFS > out of the equation helps. I haven't run into this with ZFS with > disks without dm-crypt. Hi, unfortunately the dmcrypt code in Centos 6.x is not what is upstream kernel (mainly because of per cpu processing changes upstream). (It would be interesting if it is reproducible with upstream kernel, but I guess it is not easy to test...) I am adding cc to dm-devel list, maybe someone there could have some idea how to help here. Milan > > Thanks, > > Steve > > > > -- ________________________________________________________________ > Steve Cousins Supercomputer Engineer/Administrator > Advanced Computing Group University of Maine System 244 > Neville Hall (UMS Data Center) (207) 561-3574 > <tel:%28207%29%20561-3574> Orono ME 04469 > steve.cousins at maine.edu <http://maine.edu> ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: trouble with dm-crypt and NFS: server hangs @ 2014-03-20 19:16 ` Milan Broz 0 siblings, 0 replies; 7+ messages in thread From: Milan Broz @ 2014-03-20 19:16 UTC (permalink / raw) To: Stephen Cousins, dm-crypt-4q3lyFh4P1g; +Cc: device-mapper development On 03/20/2014 03:48 PM, Stephen Cousins wrote: > I am testing dm-crypt with a system that serves data over NFS. I'm > finding that when serving the data over NFS4 it will go for a little > while and then it will just stop copying and networking on the NFS > server will stall out so that other connections to other parts of the > storage (a non-dm-crypt volume) will become very laggy. Soon after > canceling the copy the other connections resume back to normal. > > I have tried this on two different servers, one with a single disk > with ZFS and another with a ZFS raidz2 pool and they both do the same > thing. CPU doesn't seem to be an issue. Doing local copies is not a > problem. That is, copying to/from these volumes made up of dm-crypted > disks from other disks on the server have no problems. > > One system (the one with the RAID pool) is CentOS 6.4 and the other > one is CentOS 6.5. > > I'm curious if anyone else has run into this before and if so, what > can be done to make it work? I'm going to test to see if taking ZFS > out of the equation helps. I haven't run into this with ZFS with > disks without dm-crypt. Hi, unfortunately the dmcrypt code in Centos 6.x is not what is upstream kernel (mainly because of per cpu processing changes upstream). (It would be interesting if it is reproducible with upstream kernel, but I guess it is not easy to test...) I am adding cc to dm-devel list, maybe someone there could have some idea how to help here. Milan > > Thanks, > > Steve > > > > -- ________________________________________________________________ > Steve Cousins Supercomputer Engineer/Administrator > Advanced Computing Group University of Maine System 244 > Neville Hall (UMS Data Center) (207) 561-3574 > <tel:%28207%29%20561-3574> Orono ME 04469 > steve.cousins at maine.edu <http://maine.edu> _______________________________________________ dm-crypt mailing list dm-crypt-4q3lyFh4P1g@public.gmane.org http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs @ 2014-03-20 22:12 ` Stephen Cousins 0 siblings, 0 replies; 7+ messages in thread From: Stephen Cousins @ 2014-03-20 22:12 UTC (permalink / raw) To: Milan Broz; +Cc: dm-crypt, device-mapper development [-- Attachment #1: Type: text/plain, Size: 3016 bytes --] I'm getting mixed results but it is looking like ZFS have something to do with the problem afterall. I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag issue. I say mixed results mainly because I've been using rsync to do the testing and for some reason rsync has trouble with this setup. It works but really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR" then it goes at 60 MB/sec. I use rsync all the time and I've never seen this before so it took a while for me to realize that rsync was acting up. Anyone run into this before? I'll bring this to the ZFSonLinux people and see what they say. Thanks, Steve On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland@gmail.com> wrote: > On 03/20/2014 03:48 PM, Stephen Cousins wrote: > > I am testing dm-crypt with a system that serves data over NFS. I'm > > finding that when serving the data over NFS4 it will go for a little > > while and then it will just stop copying and networking on the NFS > > server will stall out so that other connections to other parts of the > > storage (a non-dm-crypt volume) will become very laggy. Soon after > > canceling the copy the other connections resume back to normal. > > > > I have tried this on two different servers, one with a single disk > > with ZFS and another with a ZFS raidz2 pool and they both do the same > > thing. CPU doesn't seem to be an issue. Doing local copies is not a > > problem. That is, copying to/from these volumes made up of dm-crypted > > disks from other disks on the server have no problems. > > > > One system (the one with the RAID pool) is CentOS 6.4 and the other > > one is CentOS 6.5. > > > > I'm curious if anyone else has run into this before and if so, what > > can be done to make it work? I'm going to test to see if taking ZFS > > out of the equation helps. I haven't run into this with ZFS with > > disks without dm-crypt. > > Hi, > > unfortunately the dmcrypt code in Centos 6.x is not what is upstream > kernel (mainly because of per cpu processing changes upstream). > (It would be interesting if it is reproducible with upstream kernel, > but I guess it is not easy to test...) > > I am adding cc to dm-devel list, maybe someone there could have some idea > how to help here. > > Milan > > > > > Thanks, > > > > Steve > > > > > > > > -- ________________________________________________________________ > > Steve Cousins Supercomputer Engineer/Administrator > > Advanced Computing Group University of Maine System 244 > > Neville Hall (UMS Data Center) (207) 561-3574 > > <tel:%28207%29%20561-3574> Orono ME 04469 > > steve.cousins at maine.edu <http://maine.edu> > -- ________________________________________________________________ Steve Cousins Supercomputer Engineer/Administrator Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 561-3574 Orono ME 04469 steve.cousins at maine.edu [-- Attachment #2: Type: text/html, Size: 5062 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: trouble with dm-crypt and NFS: server hangs @ 2014-03-20 22:12 ` Stephen Cousins 0 siblings, 0 replies; 7+ messages in thread From: Stephen Cousins @ 2014-03-20 22:12 UTC (permalink / raw) To: Milan Broz; +Cc: dm-crypt-4q3lyFh4P1g, device-mapper development [-- Attachment #1.1: Type: text/plain, Size: 3046 bytes --] I'm getting mixed results but it is looking like ZFS have something to do with the problem afterall. I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag issue. I say mixed results mainly because I've been using rsync to do the testing and for some reason rsync has trouble with this setup. It works but really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR" then it goes at 60 MB/sec. I use rsync all the time and I've never seen this before so it took a while for me to realize that rsync was acting up. Anyone run into this before? I'll bring this to the ZFSonLinux people and see what they say. Thanks, Steve On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > On 03/20/2014 03:48 PM, Stephen Cousins wrote: > > I am testing dm-crypt with a system that serves data over NFS. I'm > > finding that when serving the data over NFS4 it will go for a little > > while and then it will just stop copying and networking on the NFS > > server will stall out so that other connections to other parts of the > > storage (a non-dm-crypt volume) will become very laggy. Soon after > > canceling the copy the other connections resume back to normal. > > > > I have tried this on two different servers, one with a single disk > > with ZFS and another with a ZFS raidz2 pool and they both do the same > > thing. CPU doesn't seem to be an issue. Doing local copies is not a > > problem. That is, copying to/from these volumes made up of dm-crypted > > disks from other disks on the server have no problems. > > > > One system (the one with the RAID pool) is CentOS 6.4 and the other > > one is CentOS 6.5. > > > > I'm curious if anyone else has run into this before and if so, what > > can be done to make it work? I'm going to test to see if taking ZFS > > out of the equation helps. I haven't run into this with ZFS with > > disks without dm-crypt. > > Hi, > > unfortunately the dmcrypt code in Centos 6.x is not what is upstream > kernel (mainly because of per cpu processing changes upstream). > (It would be interesting if it is reproducible with upstream kernel, > but I guess it is not easy to test...) > > I am adding cc to dm-devel list, maybe someone there could have some idea > how to help here. > > Milan > > > > > Thanks, > > > > Steve > > > > > > > > -- ________________________________________________________________ > > Steve Cousins Supercomputer Engineer/Administrator > > Advanced Computing Group University of Maine System 244 > > Neville Hall (UMS Data Center) (207) 561-3574 > > <tel:%28207%29%20561-3574> Orono ME 04469 > > steve.cousins at maine.edu <http://maine.edu> > -- ________________________________________________________________ Steve Cousins Supercomputer Engineer/Administrator Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 561-3574 Orono ME 04469 steve.cousins at maine.edu [-- Attachment #1.2: Type: text/html, Size: 5122 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ dm-crypt mailing list dm-crypt-4q3lyFh4P1g@public.gmane.org http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs 2014-03-20 22:12 ` Stephen Cousins (?) @ 2014-03-27 17:56 ` Stephen Cousins 2014-03-27 20:28 ` Arno Wagner -1 siblings, 1 reply; 7+ messages in thread From: Stephen Cousins @ 2014-03-27 17:56 UTC (permalink / raw) To: Milan Broz; +Cc: dm-crypt [-- Attachment #1: Type: text/plain, Size: 3708 bytes --] I'm following up on this to let you know that it has been identified as a ZFS sync issue that has been addressed recently. No need to contact dm-devel. Thanks for your help. Steve On Thu, Mar 20, 2014 at 6:12 PM, Stephen Cousins <steve.cousins@maine.edu>wrote: > I'm getting mixed results but it is looking like ZFS have something to do > with the problem afterall. > > I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag > issue. I say mixed results mainly because I've been using rsync to do the > testing and for some reason rsync has trouble with this setup. It works but > really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR" > then it goes at 60 MB/sec. I use rsync all the time and I've never seen > this before so it took a while for me to realize that rsync was acting up. > Anyone run into this before? > > I'll bring this to the ZFSonLinux people and see what they say. > > Thanks, > > Steve > > > On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland@gmail.com> wrote: > >> On 03/20/2014 03:48 PM, Stephen Cousins wrote: >> > I am testing dm-crypt with a system that serves data over NFS. I'm >> > finding that when serving the data over NFS4 it will go for a little >> > while and then it will just stop copying and networking on the NFS >> > server will stall out so that other connections to other parts of the >> > storage (a non-dm-crypt volume) will become very laggy. Soon after >> > canceling the copy the other connections resume back to normal. >> > >> > I have tried this on two different servers, one with a single disk >> > with ZFS and another with a ZFS raidz2 pool and they both do the same >> > thing. CPU doesn't seem to be an issue. Doing local copies is not a >> > problem. That is, copying to/from these volumes made up of dm-crypted >> > disks from other disks on the server have no problems. >> > >> > One system (the one with the RAID pool) is CentOS 6.4 and the other >> > one is CentOS 6.5. >> > >> > I'm curious if anyone else has run into this before and if so, what >> > can be done to make it work? I'm going to test to see if taking ZFS >> > out of the equation helps. I haven't run into this with ZFS with >> > disks without dm-crypt. >> >> Hi, >> >> unfortunately the dmcrypt code in Centos 6.x is not what is upstream >> kernel (mainly because of per cpu processing changes upstream). >> (It would be interesting if it is reproducible with upstream kernel, >> but I guess it is not easy to test...) >> >> I am adding cc to dm-devel list, maybe someone there could have some idea >> how to help here. >> >> Milan >> >> > >> > Thanks, >> > >> > Steve >> > >> > >> > >> > -- ________________________________________________________________ >> > Steve Cousins Supercomputer Engineer/Administrator >> > Advanced Computing Group University of Maine System 244 >> > Neville Hall (UMS Data Center) (207) 561-3574 >> > <tel:%28207%29%20561-3574> Orono ME 04469 >> > steve.cousins at maine.edu <http://maine.edu> >> > > > > -- > ________________________________________________________________ > Steve Cousins Supercomputer Engineer/Administrator > Advanced Computing Group University of Maine System > 244 Neville Hall (UMS Data Center) (207) 561-3574 > Orono ME 04469 steve.cousins at maine.edu > > -- ________________________________________________________________ Steve Cousins Supercomputer Engineer/Administrator Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 561-3574 Orono ME 04469 steve.cousins at maine.edu [-- Attachment #2: Type: text/html, Size: 7089 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs 2014-03-27 17:56 ` [dm-crypt] " Stephen Cousins @ 2014-03-27 20:28 ` Arno Wagner 0 siblings, 0 replies; 7+ messages in thread From: Arno Wagner @ 2014-03-27 20:28 UTC (permalink / raw) To: dm-crypt Thank you! Arno On Thu, Mar 27, 2014 at 18:56:06 CET, Stephen Cousins wrote: > I'm following up on this to let you know that it has been identified as a > ZFS sync issue that has been addressed recently. No need to contact > dm-devel. > > Thanks for your help. > > Steve > > > On Thu, Mar 20, 2014 at 6:12 PM, Stephen Cousins <steve.cousins@maine.edu>wrote: > > > I'm getting mixed results but it is looking like ZFS have something to do > > with the problem afterall. > > > > I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag > > issue. I say mixed results mainly because I've been using rsync to do the > > testing and for some reason rsync has trouble with this setup. It works but > > really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR" > > then it goes at 60 MB/sec. I use rsync all the time and I've never seen > > this before so it took a while for me to realize that rsync was acting up. > > Anyone run into this before? > > > > I'll bring this to the ZFSonLinux people and see what they say. > > > > Thanks, > > > > Steve > > > > > > On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland@gmail.com> wrote: > > > >> On 03/20/2014 03:48 PM, Stephen Cousins wrote: > >> > I am testing dm-crypt with a system that serves data over NFS. I'm > >> > finding that when serving the data over NFS4 it will go for a little > >> > while and then it will just stop copying and networking on the NFS > >> > server will stall out so that other connections to other parts of the > >> > storage (a non-dm-crypt volume) will become very laggy. Soon after > >> > canceling the copy the other connections resume back to normal. > >> > > >> > I have tried this on two different servers, one with a single disk > >> > with ZFS and another with a ZFS raidz2 pool and they both do the same > >> > thing. CPU doesn't seem to be an issue. Doing local copies is not a > >> > problem. That is, copying to/from these volumes made up of dm-crypted > >> > disks from other disks on the server have no problems. > >> > > >> > One system (the one with the RAID pool) is CentOS 6.4 and the other > >> > one is CentOS 6.5. > >> > > >> > I'm curious if anyone else has run into this before and if so, what > >> > can be done to make it work? I'm going to test to see if taking ZFS > >> > out of the equation helps. I haven't run into this with ZFS with > >> > disks without dm-crypt. > >> > >> Hi, > >> > >> unfortunately the dmcrypt code in Centos 6.x is not what is upstream > >> kernel (mainly because of per cpu processing changes upstream). > >> (It would be interesting if it is reproducible with upstream kernel, > >> but I guess it is not easy to test...) > >> > >> I am adding cc to dm-devel list, maybe someone there could have some idea > >> how to help here. > >> > >> Milan > >> > >> > > >> > Thanks, > >> > > >> > Steve > >> > > >> > > >> > > >> > -- ________________________________________________________________ > >> > Steve Cousins Supercomputer Engineer/Administrator > >> > Advanced Computing Group University of Maine System 244 > >> > Neville Hall (UMS Data Center) (207) 561-3574 > >> > <tel:%28207%29%20561-3574> Orono ME 04469 > >> > steve.cousins at maine.edu <http://maine.edu> > >> > > > > > > > > -- > > ________________________________________________________________ > > Steve Cousins Supercomputer Engineer/Administrator > > Advanced Computing Group University of Maine System > > 244 Neville Hall (UMS Data Center) (207) 561-3574 > > Orono ME 04469 steve.cousins at maine.edu > > > > > > > -- > ________________________________________________________________ > Steve Cousins Supercomputer Engineer/Administrator > Advanced Computing Group University of Maine System > 244 Neville Hall (UMS Data Center) (207) 561-3574 > Orono ME 04469 steve.cousins at maine.edu > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt -- Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: arno@wagner.name GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-03-27 20:28 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-03-20 14:48 [dm-crypt] trouble with dm-crypt and NFS: server hangs Stephen Cousins 2014-03-20 19:16 ` Milan Broz 2014-03-20 19:16 ` Milan Broz 2014-03-20 22:12 ` [dm-crypt] " Stephen Cousins 2014-03-20 22:12 ` Stephen Cousins 2014-03-27 17:56 ` [dm-crypt] " Stephen Cousins 2014-03-27 20:28 ` Arno Wagner
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.