route add default fails with ESRCH?

[Arun Sharma <asharma@fb.com>]

I've been trying to debug this strange problem with recent 3.2 kernels:

# boot to single user mode
# service network start
# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr <blah>
           inet addr:10.47.46.39  Bcast:10.47.46.255  Mask:255.255.255.0

<Link is up>
# strace route add default gw 10.47.46.1
ioctl(3, SIOCADDRT, 0x7fff1fd4db80)     = -1 ESRCH (No such process)

Couple of data points:

* This didn't happen with 2.6.3x kernels
* This doesn't happen on all machines. Only a small subset

I tried to debug using perf probe points:

# cmdline : /usr/bin/perf record -e 
probe:fib_get_table,probe:fib_new_table,probe:fib_table_insert route add 
default gw 10.47.46.1

            route  7800 [001]  4170.916261: fib_get_table: 
(ffffffff81406528 <- ffffffff814065a6) ret=ffff8802142605c0
            route  7800 [001]  4170.916265: fib_get_table: 
(ffffffff81406528 <- ffffffff814067d3) ret=ffff880212a36680
            route  7800 [001]  4170.916266: fib_new_table: 
(ffffffff814067b6 <- ffffffff81406f04) ret=ffff880212a36680
            route  7800 [001]  4170.916270: fib_get_table: 
(ffffffff81406528 <- ffffffff8140d7fa) ret=ffff8802142605c0
            route  7800 [001]  4170.916271: fib_get_table: 
(ffffffff81406528 <- ffffffff8140d7fa) ret=ffff880212a36680
            route  7800 [001]  4170.916272: fib_get_table: 
(ffffffff81406528 <- ffffffff8140d7fa) ret=0
            route  7800 [001]  4170.916273: fib_table_insert: 
(ffffffff8140a60f <- ffffffff81406f18) ret=fffffffd

Any ideas on why fib_table_insert() is failing with -ESRCH?

  -Arun