Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

[DENIEL Philippe]

Hi,

I'd like to set up from scratch a test bed using pNFS with block devices 
as I did a few month ago with LAYOUT4_NFSV4_1_FILES.
My question about the kernel and nfs-utils versions to be used ? Which 
one are the right ones ? Where could I get it ?

    Regards

       Philippe

Re: Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

[Benny Halevy]

On 2010-09-14 14:40, DENIEL Philippe wrote:
> Hi,
> 
> I'd like to set up from scratch a test bed using pNFS with block devices 
> as I did a few month ago with LAYOUT4_NFSV4_1_FILES.
> My question about the kernel and nfs-utils versions to be used ? Which 
> one are the right ones ? Where could I get it ?

Hi Philippe,

The instructions are documented here:
http://wiki.linux-nfs.org/wiki/index.php/PNFS_Block_Server_Setup_Instructions

The development kernel is the current one, pnfs-all-latest branch.
and the nfs utils are here:
git://linux-nfs.org/~bhalevy/pnfs-nfs-utils.git

You'll now need the user mode daemon: blkmapd
After make and make install, see man (8) blkmapd

Jim, mind taking a look at the wiki page and update it,
in particular anything regarding blkmapd?

Benny

> 
>     Regards
> 
>        Philippe
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

[Jim Rees]

Benny Halevy wrote:

  Jim, mind taking a look at the wiki page and update it,
  in particular anything regarding blkmapd?

I can't comment on the server setup.  For the client, there are Fedora
specific instructions here:

http://wiki.linux-nfs.org/wiki/index.php/Fedora_pNFS_Client_Setup

If you're not using Fedora, you want to install the kernel and nfs-utils
from git.  I'm working on a wiki page with generic client instructions but
it goes something like this:

iscsiadm -m discovery -t sendtargets -p pnfs-server-iscsi -l
service blkmapd restart
mount -t nfs4 -o minorversion=1 pnfs-server-1:/pnfs /mnt
[ run tests ]
umount /mnt
iscsiadm -m node -U all

Re: Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

[Jim Rees]

Benny Halevy wrote:

  On 2010-09-14 14:40, DENIEL Philippe wrote:
  > Hi,
  > 
  > I'd like to set up from scratch a test bed using pNFS with block devices 
  > as I did a few month ago with LAYOUT4_NFSV4_1_FILES.
  > My question about the kernel and nfs-utils versions to be used ? Which 
  > one are the right ones ? Where could I get it ?
  
  Hi Philippe,
  
  The instructions are documented here:
  http://wiki.linux-nfs.org/wiki/index.php/PNFS_Block_Server_Setup_Instructions

I have added a section on the block layout client to the Fedora client setup
wiki page:

http://wiki.linux-nfs.org/wiki/index.php/Fedora_pNFS_Client_Setup

For now the isntructions apply to Fedora, which is what we test with.
Eventually I will make this more generic.

Re: Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

[DENIEL Philippe]

Hi Jim,

>   The instructions are documented here:
>   http://wiki.linux-nfs.org/wiki/index.php/PNFS_Block_Server_Setup_Instructions
>   
I followed these instructions. Compiling the kernel is OK and so is 
compilation of pnfs-nfs-utils . I run client and server on F13. They 
both are virtual machines managed by virt-manager.

I used your start script to make one of my own (I had to change to 
argument to tgtadm related to the ip name of the iscsi array) but it was ok.
The only issue I have is about the ctl program. It stops as soon as it 
starts with this message:

    # ./ctl -u
    write: File exists

  
I then ran strace to know more about it and it showed this:

    munmap(0x7f5c533ac000, 47451)           = 0
    *open("/proc/fs/pnfs_block/ctl", O_RDWR) = 3*
    *write(3, "\1\0\0\0", 4)                 = -1 EEXIST (File exists)*
    dup(2)                                  = 4
    fcntl(4, F_GETFL)                       = 0x8002 (flags
    O_RDWR|O_LARGEFILE)
    fstat(4, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
    -1, 0) = 0x7f5c533b7000
    lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
    write(4, "write: File exists\n", 19write: File exists
    )    = 19
    close(4)                                = 0
    munmap(0x7f5c533b7000, 4096)            = 0
    exit_group(1)                           = ?

Apparently writing to /proc/fs/pnfs_block/ctl does not work. Do you have 
an idea on what I should do to fix this ?

> I have added a section on the block layout client to the Fedora client setup
> wiki page:
>
> http://wiki.linux-nfs.org/wiki/index.php/Fedora_pNFS_Client_Setup
>   
Is there a layoutdriver to modprobe on the client or is it embedded in 
the kernel itself ?

    Philippe

Re: Deploying a pNFS / LAYOUT4_BLOCK_VOLUME configuration

[Jim Rees]

DENIEL Philippe wrote:

  Apparently writing to /proc/fs/pnfs_block/ctl does not work. Do you
  have an idea on what I should do to fix this ?

I have not worked with the server side but will be in the coming months.
Maybe someone on the list can help.

  >I have added a section on the block layout client to the Fedora client setup
  >wiki page:
  >
  >http://wiki.linux-nfs.org/wiki/index.php/Fedora_pNFS_Client_Setup
  Is there a layoutdriver to modprobe on the client or is it embedded
  in the kernel itself ?

You could build it in but I normally build it as a module.  The startup
script assumes it's a module and probes it:

fedora# grep blocklayoutdriver /etc/init.d/blkmapd
        modprobe -q blocklayoutdriver
fedora#

Right now the nfs-utils rpm doesn't install the startup script.  I'll fix
this but for now just copy it from the source:

cp nfs-utils/utils/blkmapd/etc/initd/initd.redhat /etc/init.d/blkmapd
chmod +x /etc/init.d/blkmapd