Когда мне говорили, что device mapper реально бажен — не верил. Сегодня представился случай увидеть собственными глазами такую картину:
Jul 23 16:06:42 database multipathd: sdc: couldn't get asymmetric access state
Jul 23 16:06:43 database iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Jul 23 16:06:45 database multipathd: mpathc: load table [0 8589934592 multipath 1 queue_if_no_path 0 2 2 service-time 0 1 1 8:32 1 service-time 0 1 1 8:64 1]
Jul 23 16:06:45 database iscsid: connection1:0 is operational after recovery (1 attempts)
Jul 23 16:07:07 database multipathd: mpathc: load table [0 8589934592 multipath 1 queue_if_no_path 0 2 1 service-time 0 1 1 8:32 1 service-time 0 1 1 8:64 1]
Jul 23 17:20:34 database iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Jul 23 17:20:34 database systemd: Started Session 4234 of user root.
Jul 23 17:20:34 database systemd: Starting Session 4234 of user root.
Jul 23 17:20:35 database iscsid: Kernel reported iSCSI connection 1:0 error (1022 - Invalid or unknown error code) state (3)
Jul 23 17:20:41 database systemd: Deactivating swap /db/swapfile...
Jul 23 17:20:41 database systemd: Stopped target Remote File Systems.
Jul 23 17:20:41 database systemd: Stopping Remote File Systems
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] killing request
Jul 23 17:20:43 database kernel: sd 3:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] killing request
Jul 23 17:20:43 database kernel: sd 3:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] killing request
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] CDB: Write(16) 8a 00 00 00 00 01 00 00 57 40 00 00 00 40 00 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdc, sector 4294989632
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] CDB: Write(16) 8a 00 00 00 00 01 7c 73 2d 40 00 00 00 10 00 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdc, sector 6382890304
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] CDB: Read(16) 88 00 00 00 00 00 75 54 73 00 00 00 01 00 00 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdc, sector 1968468736
Jul 23 17:20:43 database kernel: device-mapper: multipath: Failing path 8:32.
Jul 23 17:20:43 database kernel: sd 4:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 4:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 4:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: device-mapper: multipath: Failing path 8:64.
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 3:0:0:1: [sdb] FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:1: [sdb] CDB: Write(10) 2a 00 06 40 f3 c0 00 00 40 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdb, sector 104920000
Jul 23 17:20:43 database kernel: device-mapper: multipath: Failing path 8:16.
Наше счастье, блочные устройства безрезиновых бабLVM собраны и отдельными устройствами, бо это кусок нормальной промышленной СХД. Так что потери данных не случилось. Но fstab слегка переписал.
Jul 23 16:06:42 database multipathd: sdc: couldn't get asymmetric access state
Jul 23 16:06:43 database iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Jul 23 16:06:45 database multipathd: mpathc: load table [0 8589934592 multipath 1 queue_if_no_path 0 2 2 service-time 0 1 1 8:32 1 service-time 0 1 1 8:64 1]
Jul 23 16:06:45 database iscsid: connection1:0 is operational after recovery (1 attempts)
Jul 23 16:07:07 database multipathd: mpathc: load table [0 8589934592 multipath 1 queue_if_no_path 0 2 1 service-time 0 1 1 8:32 1 service-time 0 1 1 8:64 1]
Jul 23 17:20:34 database iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Jul 23 17:20:34 database systemd: Started Session 4234 of user root.
Jul 23 17:20:34 database systemd: Starting Session 4234 of user root.
Jul 23 17:20:35 database iscsid: Kernel reported iSCSI connection 1:0 error (1022 - Invalid or unknown error code) state (3)
Jul 23 17:20:41 database systemd: Deactivating swap /db/swapfile...
Jul 23 17:20:41 database systemd: Stopped target Remote File Systems.
Jul 23 17:20:41 database systemd: Stopping Remote File Systems
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] killing request
Jul 23 17:20:43 database kernel: sd 3:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] killing request
Jul 23 17:20:43 database kernel: sd 3:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] killing request
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] CDB: Write(16) 8a 00 00 00 00 01 00 00 57 40 00 00 00 40 00 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdc, sector 4294989632
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] CDB: Write(16) 8a 00 00 00 00 01 7c 73 2d 40 00 00 00 10 00 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdc, sector 6382890304
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:2: [sdc] CDB: Read(16) 88 00 00 00 00 00 75 54 73 00 00 00 01 00 00 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdc, sector 1968468736
Jul 23 17:20:43 database kernel: device-mapper: multipath: Failing path 8:32.
Jul 23 17:20:43 database kernel: sd 4:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 4:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: sd 4:0:0:2: rejecting I/O to offline device
Jul 23 17:20:43 database kernel: device-mapper: multipath: Failing path 8:64.
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 2:0:0:0: [sda] abort
Jul 23 17:20:43 database kernel: sd 3:0:0:1: [sdb] FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
Jul 23 17:20:43 database kernel: sd 3:0:0:1: [sdb] CDB: Write(10) 2a 00 06 40 f3 c0 00 00 40 00
Jul 23 17:20:43 database kernel: blk_update_request: I/O error, dev sdb, sector 104920000
Jul 23 17:20:43 database kernel: device-mapper: multipath: Failing path 8:16.
Наше счастье, блочные устройства без
Комментариев нет:
Отправить комментарий