* [Drbd-dev] [CASE 12] 2 primary split-brain is not detected
@ 2015-12-22 1:06 김성은
2016-01-20 9:26 ` Philipp Reisner
0 siblings, 1 reply; 4+ messages in thread
From: 김성은 @ 2015-12-22 1:06 UTC (permalink / raw)
To: drbd-dev
Cc: 박성환, 김재헌,
진현이, Windows DRBD, Philipp Reisner
[-- Attachment #1.1: Type: text/plain, Size: 1352 bytes --]
Hello,
I am Software Manager in Mantech, and testing drbd 9.
some problems are occurred. CASE 12 is one of them.
[CASE 12] 2 primary split-brain is not detected.
Env:
- Linux VM 2 nodes
- Linux drbd version 9.0.0
Test scenario:
- 2 node's status is uptodate
node 1node 2drbdadm disconnect r0 drbdadm primary r0 copy files on
node1 drbdadm
primary r0 copy files on node2
drbdadm connect r0
- at this point, we predicted that 2 primary split-brain is occurred and
drbd9 detect it.
But split-brain is not occurred, not detected. no debug log.
Just retry connection log is showing between two nodes.
Question is :
1) How can I test 2 primary split-brain situation ?
2) Does support 2 primary split-brain in drbd9 ?
PS)
http://drbd.linbit.com/users-guide-9.0/s-configure-split-brain-behavior.html#s-split-brain-notification
drbd9's manual describe 'after-sb-2pri' option.
Thanks,
Best Regards
Sungeun Kim
[image: 설명: logo]
기술연구소 / DRBD팀 Mobile : 010-2813-4843
*김 성 은 **차장 sekim@mantech.co.kr <sekim@mantech.co.kr>*
서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
http://www.mantech.co.kr
[image: 본문 이미지 1]
[-- Attachment #1.2: Type: text/html, Size: 5871 bytes --]
[-- Attachment #2: Type: message/external-body, Size: 198 bytes --]
[-- Attachment #3: Type: message/external-body, Size: 218 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Drbd-dev] [CASE 12] 2 primary split-brain is not detected
2015-12-22 1:06 [Drbd-dev] [CASE 12] 2 primary split-brain is not detected 김성은
@ 2016-01-20 9:26 ` Philipp Reisner
2016-01-21 4:13 ` 김성은
0 siblings, 1 reply; 4+ messages in thread
From: Philipp Reisner @ 2016-01-20 9:26 UTC (permalink / raw)
To: 김성은
Cc: 진현이, Windows DRBD, 김재헌,
박성환, 'lars Ellenberg', drbd-dev
Hi,
I am following up on CASE 12 now.
I was not able to reproduce what you describe in this report.
Please check that you can still reproduce it with current GIT-head.
If yes, please re-send it with a more detailed description, including:
Configuration used.
Logfiles from begin to end of the test.
Thanks!
Best regards,
Phil
> Hello,
>
> I am Software Manager in Mantech, and testing drbd 9.
>
> some problems are occurred. CASE 12 is one of them.
>
> [CASE 12] 2 primary split-brain is not detected.
>
> Env:
> - Linux VM 2 nodes
> - Linux drbd version 9.0.0
>
> Test scenario:
>
> - 2 node's status is uptodate
>
> node 1node 2drbdadm disconnect r0 drbdadm primary r0 copy files on
> node1 drbdadm
> primary r0 copy files on node2
> drbdadm connect r0
>
> - at this point, we predicted that 2 primary split-brain is occurred and
> drbd9 detect it.
>
> But split-brain is not occurred, not detected. no debug log.
> Just retry connection log is showing between two nodes.
>
> Question is :
>
> 1) How can I test 2 primary split-brain situation ?
>
> 2) Does support 2 primary split-brain in drbd9 ?
>
>
> PS)
>
> http://drbd.linbit.com/users-guide-9.0/s-configure-split-brain-behavior.html
> #s-split-brain-notification
>
> drbd9's manual describe 'after-sb-2pri' option.
>
> Thanks,
>
>
> Best Regards
> Sungeun Kim
>
>
>
>
> [image: 설명: logo]
>
> 기술연구소 / DRBD팀 Mobile : 010-2813-4843
>
> *김 성 은 **차장 sekim@mantech.co.kr <sekim@mantech.co.kr>*
>
> 서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
>
> Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
>
> http://www.mantech.co.kr
>
> [image: 본문 이미지 1]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Drbd-dev] [CASE 12] 2 primary split-brain is not detected
2016-01-20 9:26 ` Philipp Reisner
@ 2016-01-21 4:13 ` 김성은
2016-01-23 23:06 ` Philipp Reisner
0 siblings, 1 reply; 4+ messages in thread
From: 김성은 @ 2016-01-21 4:13 UTC (permalink / raw)
To: Philipp Reisner
Cc: 진현이, Windows DRBD, 김재헌,
박성환, lars Ellenberg, drbd-dev
[-- Attachment #1.1.1: Type: text/plain, Size: 3550 bytes --]
Hello,
thank you for your reply. appreciated that
I tested again with current GIT-head version(Commit: [9c711b4]), reproduce
this issue, and same result is happened.
more detail descriptions are below :
1) Env:
- CentOS 7 64bit
- i7 4790K CPU 4Ghz , memory :1G
- 2 node VM
2) drbd version (self build)
[root@drbd9-02 ]# ls -l
/lib/modules/3.10.0-229.7.2.el7.x86_64/updates/drbd.ko
-rw-r--r-- 1 root root 10657876 * Jan 21* 11:01
/lib/modules/3.10.0-229.7.2.el7.x86_64/updates/drbd.ko
3) scenario:
- basically, progressed to 2-primary split-brain scenario.
- first, 2 node status is secondary-secondary uptodate
node 1 node 2
drbdadm disconnect r0
drbdadm primary r0
copy files on node1 volume
drbdadm primary r0
copy files on node2 volume
drbdadm connect r0
4) result
- 2 primary Split-brain did not occurred.
- according to drbd kernel log, I guess that connecting phase is failed and
retry to connect between 2 nodes, repeatedly.
you will see my drbd configuration and kernel log files, attached.
I hope that help for you.
Best Regards
Sung-eun kim
from Seoul
[image: 설명: logo]
기술연구소 / DRBD팀 Mobile : 010-2813-4843
*김 성 은 **차장 sekim@mantech.co.kr <sekim@mantech.co.kr>*
서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
http://www.mantech.co.kr
[image: 본문 이미지 1]
2016-01-20 18:26 GMT+09:00 Philipp Reisner <philipp.reisner@linbit.com>:
> Hi,
>
> I am following up on CASE 12 now.
>
> I was not able to reproduce what you describe in this report.
> Please check that you can still reproduce it with current GIT-head.
> If yes, please re-send it with a more detailed description, including:
>
> Configuration used.
> Logfiles from begin to end of the test.
>
> Thanks!
>
> Best regards,
> Phil
>
> > Hello,
> >
> > I am Software Manager in Mantech, and testing drbd 9.
> >
> > some problems are occurred. CASE 12 is one of them.
> >
> > [CASE 12] 2 primary split-brain is not detected.
> >
> > Env:
> > - Linux VM 2 nodes
> > - Linux drbd version 9.0.0
> >
> > Test scenario:
> >
> > - 2 node's status is uptodate
> >
> > node 1node 2drbdadm disconnect r0 drbdadm primary r0 copy files on
> > node1 drbdadm
> > primary r0 copy files on node2
> > drbdadm connect r0
> >
> > - at this point, we predicted that 2 primary split-brain is occurred and
> > drbd9 detect it.
> >
> > But split-brain is not occurred, not detected. no debug log.
> > Just retry connection log is showing between two nodes.
> >
> > Question is :
> >
> > 1) How can I test 2 primary split-brain situation ?
> >
> > 2) Does support 2 primary split-brain in drbd9 ?
> >
> >
> > PS)
> >
> >
> http://drbd.linbit.com/users-guide-9.0/s-configure-split-brain-behavior.html
> > #s-split-brain-notification
> >
> > drbd9's manual describe 'after-sb-2pri' option.
> >
> > Thanks,
> >
> >
> > Best Regards
> > Sungeun Kim
> >
> >
> >
> >
> > [image: 설명: logo]
> >
> > 기술연구소 / DRBD팀 Mobile : 010-2813-4843
> >
> > *김 성 은 **차장 sekim@mantech.co.kr <sekim@mantech.co.kr>*
> >
> > 서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
> >
> > Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
> >
> > http://www.mantech.co.kr
> >
> > [image: 본문 이미지 1]
>
>
[-- Attachment #1.1.2: Type: text/html, Size: 8794 bytes --]
[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 16229 bytes --]
[-- Attachment #1.3: image001.png --]
[-- Type: image/png, Size: 4544 bytes --]
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 2primary-sb-linbit-node1-2016-01-21.txt --]
[-- Type: text/plain; charset=US-ASCII; name="2primary-sb-linbit-node1-2016-01-21.txt", Size: 56851 bytes --]
[root@drbd9-01 ¹ÙÅÁȸé]# tail -f /var/log/messages | grep 'kernel: drbd'
Jan 21 11:02:13 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
Jan 21 11:02:13 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource )
Jan 21 11:02:13 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: Began resync as SyncSource (will sync 608620 KB [152155 bits set]).
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: updated UUIDs 2DD65F029B357EBC:0000000000000000:16E20D79CF755A32:97F2ECC6D75967D8
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: unexpected
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: In UUIDs from node 1 found equal UUID (16E20D79CF755A32) for nodes 2 3
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have 97F2ECC6D75967D9 for node_id=2
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have 97F2ECC6D75967D9 for node_id=3
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: Resync done (total 42 sec; paused 0 sec; 14488 K/sec)
Jan 21 11:02:55 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established )
Jan 21 11:03:59 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 980419737 (0->-1 3/1)
Jan 21 11:03:59 drbd9-01 kernel: drbd r0: State change 980419737: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:03:59 drbd9-01 kernel: drbd r0: Committing cluster-wide state change 980419737 (0ms)
Jan 21 11:03:59 drbd9-01 kernel: drbd r0: role( Secondary -> Primary )
Jan 21 11:03:59 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: unexpected
Jan 21 11:03:59 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: In UUIDs from node 1 found equal UUID (16E20D79CF755A32) for nodes 2 3
Jan 21 11:03:59 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have 97F2ECC6D75967D9 for node_id=2
Jan 21 11:03:59 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have 97F2ECC6D75967D9 for node_id=3
Jan 21 11:05:42 drbd9-01 kernel: drbd r0: role( Primary -> Secondary )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3350744225 (0->1 496/16)
Jan 21 11:05:53 drbd9-01 kernel: drbd r0: State change 3350744225: primary_nodes=0, weak_nodes=0
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: Cluster is now split
Jan 21 11:05:53 drbd9-01 kernel: drbd r0: Committing cluster-wide state change 3350744225 (0ms)
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( Disconnecting -> StandAlone )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-02: Terminating receiver thread
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-03: conn( Connecting -> Disconnecting )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 tcp:drbd9-03: Closing unexpected connection from 100.100.10.21
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-03: Connection closed
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-03: conn( Disconnecting -> StandAlone )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-04: conn( Connecting -> Disconnecting )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-04: Connection closed
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-04: conn( Disconnecting -> StandAlone )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-05: conn( Connecting -> Disconnecting )
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-05: Connection closed
Jan 21 11:05:53 drbd9-01 kernel: drbd r0 drbd9-05: conn( Disconnecting -> StandAlone )
Jan 21 11:05:54 drbd9-01 kernel: drbd r0: State change failed: Need a connection to start verify or resync
Jan 21 11:05:54 drbd9-01 kernel: drbd r0 drbd9-03: Failed: conn( StandAlone -> Connecting )
Jan 21 11:05:54 drbd9-01 kernel: drbd r0 drbd9-03: Terminating receiver thread
Jan 21 11:05:54 drbd9-01 kernel: drbd r0: State change failed: Need a connection to start verify or resync
Jan 21 11:05:54 drbd9-01 kernel: drbd r0 drbd9-04: Failed: conn( StandAlone -> Connecting )
Jan 21 11:05:54 drbd9-01 kernel: drbd r0 drbd9-04: Terminating receiver thread
Jan 21 11:05:54 drbd9-01 kernel: drbd r0: State change failed: Need a connection to start verify or resync
Jan 21 11:05:54 drbd9-01 kernel: drbd r0 drbd9-05: Failed: conn( StandAlone -> Connecting )
Jan 21 11:05:54 drbd9-01 kernel: drbd r0 drbd9-05: Terminating receiver thread
Jan 21 11:05:57 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2087200364 (0->-1 3/1)
Jan 21 11:05:57 drbd9-01 kernel: drbd r0: Committing cluster-wide state change 2087200364 (0ms)
Jan 21 11:05:57 drbd9-01 kernel: drbd r0: role( Secondary -> Primary )
Jan 21 11:06:19 drbd9-01 kernel: drbd r0/0 drbd1: new current UUID: 3ED55776C1C5ABE9 weak: FFFFFFFFFFFFFFFE
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( StandAlone -> Unconnected )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-02: Starting receiver thread (from drbd_w_r0 [1658])
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-03: conn( StandAlone -> Unconnected )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-03: Starting receiver thread (from drbd_w_r0 [1658])
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-04: conn( StandAlone -> Unconnected )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-04: Starting receiver thread (from drbd_w_r0 [1658])
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-03: conn( Unconnected -> Connecting )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-05: conn( StandAlone -> Unconnected )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-05: Starting receiver thread (from drbd_w_r0 [1658])
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-04: conn( Unconnected -> Connecting )
Jan 21 11:07:33 drbd9-01 kernel: drbd r0 drbd9-05: conn( Unconnected -> Connecting )
Jan 21 11:07:36 drbd9-01 kernel: drbd r0 tcp:drbd9-02: Receiver busy; rejecting incoming connection
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: sock was shut down by peer
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> BrokenPipe )
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:45 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: sock was shut down by peer
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> BrokenPipe )
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: sock was shut down by peer
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> BrokenPipe )
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:47 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:47 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:48 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:07:49 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet M crossed
Jan 21 11:07:49 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:07:50 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:50 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:50 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:51 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1672440206 (0->1 499/145)
Jan 21 11:07:51 drbd9-01 kernel: drbd r0: State change 1672440206: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:51 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1672440206 (1ms) rv = -1
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:51 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3367223114 (0->1 499/145)
Jan 21 11:07:51 drbd9-01 kernel: drbd r0: State change 3367223114: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:51 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3367223114 (1ms) rv = -1
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:51 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:52 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3580016746 (0->1 499/145)
Jan 21 11:07:52 drbd9-01 kernel: drbd r0: State change 3580016746: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:52 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3580016746 (1ms) rv = -1
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:52 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:53 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 4169387318 (0->1 499/145)
Jan 21 11:07:53 drbd9-01 kernel: drbd r0: State change 4169387318: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:53 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 4169387318 (1ms) rv = -1
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:53 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 4045170620 (0->1 499/145)
Jan 21 11:07:53 drbd9-01 kernel: drbd r0: State change 4045170620: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:53 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 4045170620 (1ms) rv = -1
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:53 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:54 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2661986277 (0->1 499/145)
Jan 21 11:07:54 drbd9-01 kernel: drbd r0: State change 2661986277: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:54 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2661986277 (1ms) rv = -1
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:54 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:55 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2169685066 (0->1 499/145)
Jan 21 11:07:55 drbd9-01 kernel: drbd r0: State change 2169685066: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:55 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2169685066 (0ms) rv = -1
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:55 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:07:56 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2605417961 (0->1 499/145)
Jan 21 11:07:56 drbd9-01 kernel: drbd r0: State change 2605417961: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:07:56 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2605417961 (1ms) rv = -1
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:07:56 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:05 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 764946311 (0->1 499/145)
Jan 21 11:08:05 drbd9-01 kernel: drbd r0: State change 764946311: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:05 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 764946311 (1ms) rv = -1
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:05 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:06 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1224111795 (0->1 499/145)
Jan 21 11:08:06 drbd9-01 kernel: drbd r0: State change 1224111795: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:06 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1224111795 (1ms) rv = -1
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:06 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:07 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 552808556 (0->1 499/145)
Jan 21 11:08:07 drbd9-01 kernel: drbd r0: State change 552808556: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:07 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 552808556 (1ms) rv = -1
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:07 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:08 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3965476312 (0->1 499/145)
Jan 21 11:08:08 drbd9-01 kernel: drbd r0: State change 3965476312: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:08 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3965476312 (2ms) rv = -1
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:08 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1963189689 (0->1 499/145)
Jan 21 11:08:08 drbd9-01 kernel: drbd r0: State change 1963189689: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:08 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1963189689 (1ms) rv = -1
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:08 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:09 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet M crossed
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:10 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2562260599 (0->1 499/145)
Jan 21 11:08:10 drbd9-01 kernel: drbd r0: State change 2562260599: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:10 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2562260599 (0ms) rv = -1
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:10 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:11 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2624123659 (0->1 499/145)
Jan 21 11:08:11 drbd9-01 kernel: drbd r0: State change 2624123659: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:11 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2624123659 (1ms) rv = -1
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:11 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:12 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3681398760 (0->1 499/145)
Jan 21 11:08:12 drbd9-01 kernel: drbd r0: State change 3681398760: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:12 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3681398760 (1ms) rv = -1
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:12 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:13 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 727952773 (0->1 499/145)
Jan 21 11:08:13 drbd9-01 kernel: drbd r0: State change 727952773: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:13 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 727952773 (1ms) rv = -1
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:13 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:14 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1773629614 (0->1 499/145)
Jan 21 11:08:14 drbd9-01 kernel: drbd r0: State change 1773629614: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:14 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1773629614 (4ms) rv = -1
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:14 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:15 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 545000671 (0->1 499/145)
Jan 21 11:08:15 drbd9-01 kernel: drbd r0: State change 545000671: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:15 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 545000671 (44ms) rv = -1
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:15 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 198186252 (0->1 499/145)
Jan 21 11:08:15 drbd9-01 kernel: drbd r0: State change 198186252: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:15 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 198186252 (1ms) rv = -1
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:15 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:16 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3738817929 (0->1 499/145)
Jan 21 11:08:16 drbd9-01 kernel: drbd r0: State change 3738817929: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:16 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3738817929 (1ms) rv = -1
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:16 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:26 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 22138955 (0->1 499/145)
Jan 21 11:08:26 drbd9-01 kernel: drbd r0: State change 22138955: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:26 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 22138955 (1ms) rv = -1
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:26 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:27 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3346055353 (0->1 499/145)
Jan 21 11:08:27 drbd9-01 kernel: drbd r0: State change 3346055353: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:27 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3346055353 (1ms) rv = -1
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:27 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:28 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3625008168 (0->1 499/145)
Jan 21 11:08:28 drbd9-01 kernel: drbd r0: State change 3625008168: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:28 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3625008168 (0ms) rv = -1
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:28 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1787935049 (0->1 499/145)
Jan 21 11:08:28 drbd9-01 kernel: drbd r0: State change 1787935049: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:28 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1787935049 (0ms) rv = -1
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:28 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:29 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2225005906 (0->1 499/145)
Jan 21 11:08:29 drbd9-01 kernel: drbd r0: State change 2225005906: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:29 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2225005906 (0ms) rv = -1
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:29 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:30 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2672567104 (0->1 499/145)
Jan 21 11:08:30 drbd9-01 kernel: drbd r0: State change 2672567104: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:30 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2672567104 (1ms) rv = -1
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:30 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:31 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3256706244 (0->1 499/145)
Jan 21 11:08:31 drbd9-01 kernel: drbd r0: State change 3256706244: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:31 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3256706244 (0ms) rv = -1
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:31 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:32 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 347185546 (0->1 499/145)
Jan 21 11:08:32 drbd9-01 kernel: drbd r0: State change 347185546: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:32 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 347185546 (40ms) rv = -1
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:32 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:33 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 3414016638 (0->1 499/145)
Jan 21 11:08:33 drbd9-01 kernel: drbd r0: State change 3414016638: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:33 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 3414016638 (1ms) rv = -1
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:33 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1469925946 (0->1 499/145)
Jan 21 11:08:33 drbd9-01 kernel: drbd r0: State change 1469925946: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:33 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1469925946 (1ms) rv = -1
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.21
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:33 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:34 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 1431293797 (0->1 499/145)
Jan 21 11:08:34 drbd9-01 kernel: drbd r0: State change 1431293797: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:34 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 1431293797 (1ms) rv = -1
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:34 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
^AJan 21 11:08:44 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet M crossed
Jan 21 11:08:44 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:45 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet M crossed
Jan 21 11:08:45 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:46 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 483707917 (0->1 499/145)
Jan 21 11:08:46 drbd9-01 kernel: drbd r0: State change 483707917: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:46 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 483707917 (1ms) rv = -1
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:46 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
Jan 21 11:08:47 drbd9-01 kernel: drbd r0 drbd9-02: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:47 drbd9-01 kernel: drbd r0 drbd9-02: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:47 drbd9-01 kernel: drbd r0 drbd9-02: Starting ack_recv thread (from drbd_r_r0 [2886])
Jan 21 11:08:48 drbd9-01 kernel: drbd r0: Preparing cluster-wide state change 2864291322 (0->1 499/145)
Jan 21 11:08:48 drbd9-01 kernel: drbd r0: State change 2864291322: primary_nodes=3, weak_nodes=FFFFFFFFFFFFFFFC
Jan 21 11:08:48 drbd9-01 kernel: drbd r0: Aborting cluster-wide state change 2864291322 (1ms) rv = -1
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: Failure to connect; retrying
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: conn( Connecting -> NetworkFailure )
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: ack_receiver terminated
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: Terminating ack_recv thread
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: Connection closed
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: Restarting receiver thread
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 drbd9-02: conn( Unconnected -> Connecting )
Jan 21 11:08:48 drbd9-01 kernel: drbd r0 tcp:drbd9-02: initial packet S crossed
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 2primary-sb-linbit-node2-2016-01-21.txt --]
[-- Type: text/plain; charset=US-ASCII; name="2primary-sb-linbit-node2-2016-01-21.txt", Size: 81759 bytes --]
[root@drbd9-02 ¹ÙÅÁȸé]# tail -f /var/log/messages | grep 'kernel: drbd'
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-01: Resync done (total 39 sec; paused 0 sec; 15604 K/sec)
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-01: updated UUIDs 2DD65F029B357EBC:0000000000000000:16E20D79CF755A32:0000000000000000
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1: disk( Inconsistent -> UpToDate )
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-05: pdsk( DUnknown -> Outdated ) resync-susp( connection dependency -> no )
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( DUnknown -> Outdated ) resync-susp( connection dependency -> no )
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-03: pdsk( DUnknown -> Outdated ) resync-susp( connection dependency -> no )
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-01: repl( SyncTarget -> Established )
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-01: helper command: /sbin/drbdadm after-resync-target
Jan 21 11:03:01 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-01: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0)
Jan 21 11:04:00 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 980419737 (primary_nodes=0, weak_nodes=0)
Jan 21 11:04:00 drbd9-02 kernel: drbd r0 drbd9-01: Committing remote state change 980419737
Jan 21 11:04:00 drbd9-02 kernel: drbd r0 drbd9-01: peer( Secondary -> Primary )
Jan 21 11:05:42 drbd9-02 kernel: drbd r0 drbd9-01: peer( Primary -> Secondary )
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3350744225 (primary_nodes=0, weak_nodes=0)
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: Committing remote state change 3350744225
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connected -> TearDown ) peer( Secondary -> Unknown )
Jan 21 11:05:53 drbd9-02 kernel: drbd r0/0 drbd1 drbd9-01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( TearDown -> Unconnected )
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:05:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:11 drbd9-02 kernel: drbd r0: Preparing cluster-wide state change 2314832267 (1->-1 3/1)
Jan 21 11:07:11 drbd9-02 kernel: drbd r0: Committing cluster-wide state change 2314832267 (0ms)
Jan 21 11:07:11 drbd9-02 kernel: drbd r0: role( Secondary -> Primary )
Jan 21 11:07:22 drbd9-02 kernel: drbd r0/0 drbd1: new current UUID: 44CDABF1A064061D weak: FFFFFFFFFFFFFFFD
Jan 21 11:07:35 drbd9-02 kernel: drbd r0 drbd9-01: sock_recvmsg returned -11
Jan 21 11:07:35 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:35 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:35 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:36 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:07:46 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:07:47 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:48 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:07:49 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet M crossed
Jan 21 11:07:49 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:07:50 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:50 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:50 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1672440206 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1672440206
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3367223114 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3367223114
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:51 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3580016746 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3580016746
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:52 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 4169387318 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 4169387318
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 4045170620 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 4045170620
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:53 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2661986277 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2661986277
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:54 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2169685066 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2169685066
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:07:55 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2605417961 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2605417961
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:07:56 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 764946311 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 764946311
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1224111795 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1224111795
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:06 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 552808556 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 552808556
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:07 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3965476312 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3965476312
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1963189689 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1963189689
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:08 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:09 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet M crossed
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2562260599 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2562260599
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:10 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2624123659 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2624123659
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:11 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3681398760 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3681398760
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:12 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 727952773 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 727952773
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:13 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1773629614 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1773629614
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:14 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 545000671 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 545000671
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 198186252 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 198186252
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3738817929 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3738817929
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:16 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 22138955 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 22138955
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:26 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3346055353 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3346055353
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:27 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3625008168 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3625008168
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1787935049 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1787935049
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:28 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2225005906 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2225005906
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:29 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2672567104 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2672567104
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:30 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3256706244 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3256706244
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:31 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 347185546 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 347185546
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:32 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3414016638 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3414016638
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1469925946 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1469925946
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:33 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1431293797 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1431293797
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:34 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:43 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet M crossed
Jan 21 11:08:44 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:45 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet M crossed
Jan 21 11:08:45 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 483707917 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 483707917
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:46 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:08:47 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:08:47 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:08:47 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2864291322 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2864291322
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:08:48 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:00 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet M crossed
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 938555955 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 938555955
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:01 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 824314604 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 824314604
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3693353173 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3693353173
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:02 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2314960084 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2314960084
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:03 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3719877924 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3719877924
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:04 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 913118447 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 913118447
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2934288254 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2934288254
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:05 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2345726076 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2345726076
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:15 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 321208629 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 321208629
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:16 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 4030258388 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 4030258388
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:17 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:18 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet M crossed
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2239819122 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2239819122
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:19 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2806236142 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2806236142
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 1487729360 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 1487729360
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:20 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 4254376853 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 4254376853
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:21 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2035196134 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2035196134
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:22 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 4243977772 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 4243977772
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3876357647 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3876357647
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: meta connection shut down by peer.
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> NetworkFailure )
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: conn( NetworkFailure -> Unconnected )
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:23 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2097234543 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2097234543
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:24 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2993915249 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2993915249
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 tcp:drbd9-05: Closing unexpected connection from 100.100.10.11
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 276857679 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 276857679
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:25 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 666780213 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 666780213
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:26 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3702291070 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3702291070
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:27 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:28 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:28 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:28 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 3248627951 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 3248627951
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:29 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Handshake successful: Agreed network protocol version 110
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Starting ack_recv thread (from drbd_r_r0 [1708])
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Preparing remote state change 2839862951 (primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC)
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Aborting remote state change 2839862951
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: sock was shut down by peer
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: conn( Connecting -> BrokenPipe )
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: ack_receiver terminated
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Terminating ack_recv thread
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Connection closed
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: conn( BrokenPipe -> Unconnected )
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: Restarting receiver thread
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 drbd9-01: conn( Unconnected -> Connecting )
Jan 21 11:09:30 drbd9-02 kernel: drbd r0 tcp:drbd9-01: initial packet S crossed
[-- Attachment #4: global_common.conf --]
[-- Type: application/octet-stream, Size: 2144 bytes --]
# DRBD is the result of over a decade of development by LINBIT.
# In case you need professional services for DRBD or have
# feature requests visit http://www.linbit.com
global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
# cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;
}
common {
handlers {
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
options {
# cpu-mask on-no-data-accessible
}
disk {
# size on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
}
net {
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
protocol A;
sndbuf-size 10M;
on-congestion pull-ahead;
congestion-fill 5M;
}
}
[-- Attachment #5: r0.res --]
[-- Type: application/octet-stream, Size: 457 bytes --]
resource r0 {
device /dev/drbd1;
disk /dev/sdb1;
meta-disk internal;
on drbd9-01 {
address 100.100.10.11:7000;
node-id 0;
}
on drbd9-02 {
address 100.100.10.21:7000;
node-id 1;
}
on drbd9-03 {
address 100.100.10.31:7000;
node-id 2;
}
on drbd9-04 {
address 100.100.10.41:7000;
node-id 3;
}
on drbd9-05 {
address 100.100.10.51:7000;
node-id 4;
}
connection-mesh {
hosts drbd9-01 drbd9-02 drbd9-03 drbd9-04 drbd9-05;
}
}
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Drbd-dev] [CASE 12] 2 primary split-brain is not detected
2016-01-21 4:13 ` 김성은
@ 2016-01-23 23:06 ` Philipp Reisner
0 siblings, 0 replies; 4+ messages in thread
From: Philipp Reisner @ 2016-01-23 23:06 UTC (permalink / raw)
To: 김성은
Cc: 진현이, Windows DRBD, 김재헌,
박성환, lars Ellenberg, drbd-dev
[-- Attachment #1: Type: text/plain, Size: 4065 bytes --]
Hi,
thanks for the detailed description.
Please find attached a proposed fix. I want to think about it for one more
day before I push it onto the master branch.
best regards,
phil
> Hello,
>
> thank you for your reply. appreciated that
>
> I tested again with current GIT-head version(Commit: [9c711b4]), reproduce
> this issue, and same result is happened.
>
> more detail descriptions are below :
>
> 1) Env:
>
> - CentOS 7 64bit
> - i7 4790K CPU 4Ghz , memory :1G
> - 2 node VM
>
> 2) drbd version (self build)
>
> [root@drbd9-02 ]# ls -l
> /lib/modules/3.10.0-229.7.2.el7.x86_64/updates/drbd.ko
> -rw-r--r-- 1 root root 10657876 * Jan 21* 11:01
> /lib/modules/3.10.0-229.7.2.el7.x86_64/updates/drbd.ko
>
> 3) scenario:
> - basically, progressed to 2-primary split-brain scenario.
> - first, 2 node status is secondary-secondary uptodate
> node 1 node 2
> drbdadm disconnect r0
> drbdadm primary r0
> copy files on node1 volume
> drbdadm primary r0
> copy files on node2 volume
> drbdadm connect r0
>
> 4) result
>
> - 2 primary Split-brain did not occurred.
> - according to drbd kernel log, I guess that connecting phase is failed and
> retry to connect between 2 nodes, repeatedly.
>
>
> you will see my drbd configuration and kernel log files, attached.
>
> I hope that help for you.
>
> Best Regards
> Sung-eun kim
> from Seoul
>
>
> [image: 설명: logo]
>
> 기술연구소 / DRBD팀 Mobile : 010-2813-4843
>
> *김 성 은 **차장 sekim@mantech.co.kr <sekim@mantech.co.kr>*
>
> 서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
>
> Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
>
> http://www.mantech.co.kr
>
> [image: 본문 이미지 1]
>
> 2016-01-20 18:26 GMT+09:00 Philipp Reisner <philipp.reisner@linbit.com>:
> > Hi,
> >
> > I am following up on CASE 12 now.
> >
> > I was not able to reproduce what you describe in this report.
> > Please check that you can still reproduce it with current GIT-head.
> >
> > If yes, please re-send it with a more detailed description, including:
> > Configuration used.
> > Logfiles from begin to end of the test.
> >
> > Thanks!
> >
> > Best regards,
> >
> > Phil
> >
> > > Hello,
> > >
> > > I am Software Manager in Mantech, and testing drbd 9.
> > >
> > > some problems are occurred. CASE 12 is one of them.
> > >
> > > [CASE 12] 2 primary split-brain is not detected.
> > >
> > > Env:
> > > - Linux VM 2 nodes
> > > - Linux drbd version 9.0.0
> > >
> > > Test scenario:
> > >
> > > - 2 node's status is uptodate
> > >
> > > node 1node 2drbdadm disconnect r0 drbdadm primary r0 copy files on
> > > node1 drbdadm
> > > primary r0 copy files on node2
> > > drbdadm connect r0
> > >
> > > - at this point, we predicted that 2 primary split-brain is occurred and
> > > drbd9 detect it.
> > >
> > > But split-brain is not occurred, not detected. no debug log.
> > > Just retry connection log is showing between two nodes.
> > >
> > > Question is :
> > >
> > > 1) How can I test 2 primary split-brain situation ?
> > >
> > > 2) Does support 2 primary split-brain in drbd9 ?
> > >
> > >
> > > PS)
> >
> > http://drbd.linbit.com/users-guide-9.0/s-configure-split-brain-behavior.ht
> > ml>
> > > #s-split-brain-notification
> > >
> > > drbd9's manual describe 'after-sb-2pri' option.
> > >
> > > Thanks,
> > >
> > >
> > > Best Regards
> > > Sungeun Kim
> > >
> > >
> > >
> > >
> > > [image: 설명: logo]
> > >
> > > 기술연구소 / DRBD팀 Mobile : 010-2813-4843
> > >
> > > *김 성 은 **차장 sekim@mantech.co.kr <sekim@mantech.co.kr>*
> > >
> > > 서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
> > >
> > > Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
> > >
> > > http://www.mantech.co.kr
> > >
> > > [image: 본문 이미지 1]
[-- Attachment #2: case12_fix.patch --]
[-- Type: text/x-patch, Size: 1854 bytes --]
commit b775310ecb7549122d6e39b88c84b48ea65a5dc5
Author: Philipp Reisner <philipp.reisner@linbit.com>
Date: Sat Jan 23 23:49:13 2016 +0100
drbd: Do not outdate myself if I am primary
diff --git a/drbd/drbd_state.c b/drbd/drbd_state.c
index ee63141..8a3ca9f 100644
--- a/drbd/drbd_state.c
+++ b/drbd/drbd_state.c
@@ -3574,6 +3574,9 @@ void __outdate_myself(struct drbd_resource *resource)
struct drbd_device *device;
int vnr;
+ if (resource->role[NOW] == R_PRIMARY)
+ return;
+
idr_for_each_entry(&resource->devices, device, vnr) {
if (device->disk_state[NOW] > D_OUTDATED)
__change_disk_state(device, D_OUTDATED);
commit c6963c44c48c99e7248265ee4318914e69b4a6fd
Author: Philipp Reisner <philipp.reisner@linbit.com>
Date: Sat Jan 23 22:56:06 2016 +0100
drbd: When connect fails because too many primaries assume split-brain
If there are more primaries than allowed, and connect fails therefore
we do not get a chance to look at the UUIDs.
In order to match expectations (and drbd-8 behaviour) call the
split-brain handler in this case as well.
diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 904d444..bd59827 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -667,6 +667,10 @@ int connect_work(struct drbd_work *work, int cancel)
connection->connect_timer.expires = jiffies + HZ/20;
add_timer(&connection->connect_timer);
return 0; /* Return early. Keep the reference on the connection! */
+ } else if (rv == SS_TWO_PRIMARIES) {
+ change_cstate(connection, C_DISCONNECTING, CS_HARD);
+ drbd_alert(connection, "Split-Brain since more primaries than allowed; dropping connection!\n");
+ drbd_khelper(NULL, connection, "split-brain");
} else {
drbd_info(connection, "Failure to connect; retrying\n");
change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-01-23 23:06 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-12-22 1:06 [Drbd-dev] [CASE 12] 2 primary split-brain is not detected 김성은
2016-01-20 9:26 ` Philipp Reisner
2016-01-21 4:13 ` 김성은
2016-01-23 23:06 ` Philipp Reisner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.