I am having a server running Ubuntu 16.04 with some data writing on external hardisk. Suddenly the server stops working on Nov 6 and when I restart the server it start working without problem, I got the following error from log file Detected IO errors while flushing file data on sdc-8. I have seen that the last writ on disk has happens on Nov 6 12:44:01 and I have attached the log at that time.
Can anyone help me to find what could be happens at that time. So I can apply the fix on my server to prevent such a situation.
Nov 6 12:27:09 user kernel: [10350005.519121] scsi_io_completion: 11 callbacks suppressed
Nov 6 12:27:09 user kernel: [10350005.519132] sd 2:0:0:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519141] sd 2:0:0:0: [sdc] tag#10 CDB: Write(16) 8a 00 00 00 00 00 1d fc f3 28 00 00 01 28 00 00
Nov 6 12:27:09 user kernel: [10350005.519144] print_req_error: 11 callbacks suppressed
Nov 6 12:27:09 user kernel: [10350005.519147] print_req_error: I/O error, dev sdc, sector 503116584
Nov 6 12:27:09 user kernel: [10350005.519156] EXT4-fs warning: 4 callbacks suppressed
Nov 6 12:27:09 user kernel: [10350005.519162] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 85219810 (offset 0 size 151552 starting block 62889610)
Nov 6 12:27:09 user kernel: [10350005.519165] buffer_io_error: 741 callbacks suppressed
Nov 6 12:27:09 user kernel: [10350005.519167] Buffer I/O error on device sdc, logical block 62889573
Nov 6 12:27:09 user kernel: [10350005.519174] Buffer I/O error on device sdc, logical block 62889574
Nov 6 12:27:09 user kernel: [10350005.519179] Buffer I/O error on device sdc, logical block 62889575
Nov 6 12:27:09 user kernel: [10350005.519183] Buffer I/O error on device sdc, logical block 62889576
Nov 6 12:27:09 user kernel: [10350005.519188] Buffer I/O error on device sdc, logical block 62889577
Nov 6 12:27:09 user kernel: [10350005.519192] Buffer I/O error on device sdc, logical block 62889578
Nov 6 12:27:09 user kernel: [10350005.519197] Buffer I/O error on device sdc, logical block 62889579
Nov 6 12:27:09 user kernel: [10350005.519201] Buffer I/O error on device sdc, logical block 62889580
Nov 6 12:27:09 user kernel: [10350005.519206] Buffer I/O error on device sdc, logical block 62889581
Nov 6 12:27:09 user kernel: [10350005.519210] Buffer I/O error on device sdc, logical block 62889582
Nov 6 12:27:09 user kernel: [10350005.519258] sd 2:0:0:0: [sdc] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519263] sd 2:0:0:0: [sdc] tag#9 CDB: Write(16) 8a 00 00 00 00 00 1d fc f0 98 00 00 01 30 00 00
Nov 6 12:27:09 user kernel: [10350005.519266] print_req_error: I/O error, dev sdc, sector 503115928
Nov 6 12:27:09 user kernel: [10350005.519274] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 86550053 (offset 0 size 155648 starting block 62889529)
Nov 6 12:27:09 user kernel: [10350005.519322] sd 2:0:0:0: [sdc] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519327] sd 2:0:0:0: [sdc] tag#8 CDB: Write(16) 8a 00 00 00 00 00 1d fc 1c a0 00 00 01 30 00 00
Nov 6 12:27:09 user kernel: [10350005.519330] print_req_error: I/O error, dev sdc, sector 503061664
Nov 6 12:27:09 user kernel: [10350005.519337] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 85219807 (offset 0 size 155648 starting block 62882746)
Nov 6 12:27:09 user kernel: [10350005.519384] sd 2:0:0:0: [sdc] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519389] sd 2:0:0:0: [sdc] tag#7 CDB: Write(16) 8a 00 00 00 00 01 21 80 00 38 00 00 00 08 00 00
Nov 6 12:27:09 user kernel: [10350005.519391] print_req_error: I/O error, dev sdc, sector 4857004088
Nov 6 12:27:09 user kernel: [10350005.519397] Buffer I/O error on dev sdc, logical block 607125511, lost async page write
Nov 6 12:27:09 user kernel: [10350005.519422] sd 2:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519427] sd 2:0:0:0: [sdc] tag#6 CDB: Write(16) 8a 00 00 00 00 01 21 44 4b c0 00 00 00 18 00 00
Nov 6 12:27:09 user kernel: [10350005.519429] print_req_error: I/O error, dev sdc, sector 4853091264
Nov 6 12:27:09 user kernel: [10350005.519437] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151685307 (offset 0 size 0 starting block 606636409)
Nov 6 12:27:09 user kernel: [10350005.519444] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151688454 (offset 0 size 4096 starting block 606636410)
Nov 6 12:27:09 user kernel: [10350005.519451] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151688455 (offset 0 size 4096 starting block 606636411)
Nov 6 12:27:09 user kernel: [10350005.519464] sd 2:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519468] sd 2:0:0:0: [sdc] tag#5 CDB: Write(16) 8a 00 00 00 00 01 21 44 4b 68 00 00 00 08 00 00
Nov 6 12:27:09 user kernel: [10350005.519471] print_req_error: I/O error, dev sdc, sector 4853091176
Nov 6 12:27:09 user kernel: [10350005.519477] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151688477 (offset 0 size 4096 starting block 606636398)
Nov 6 12:27:09 user kernel: [10350005.519494] sd 2:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519498] sd 2:0:0:0: [sdc] tag#4 CDB: Write(16) 8a 00 00 00 00 01 21 44 40 c8 00 00 00 78 00 00
Nov 6 12:27:09 user kernel: [10350005.519500] print_req_error: I/O error, dev sdc, sector 4853088456
Nov 6 12:27:09 user kernel: [10350005.519507] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151688476 (offset 0 size 4096 starting block 606636058)
Nov 6 12:27:09 user kernel: [10350005.519520] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151688479 (offset 0 size 4096 starting block 606636059)
Nov 6 12:27:09 user kernel: [10350005.519532] EXT4-fs warning (device sdc): ext4_end_bio:323: I/O error 10 writing to inode 151688481 (offset 0 size 4096 starting block 606636060)
Nov 6 12:27:09 user kernel: [10350005.519581] sd 2:0:0:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519586] sd 2:0:0:0: [sdc] tag#3 CDB: Write(16) 8a 00 00 00 00 01 21 41 87 38 00 00 00 08 00 00
Nov 6 12:27:09 user kernel: [10350005.519588] print_req_error: I/O error, dev sdc, sector 4852909880
Nov 6 12:27:09 user kernel: [10350005.519593] Buffer I/O error on dev sdc, logical block 606613735, lost async page write
Nov 6 12:27:09 user kernel: [10350005.519609] sd 2:0:0:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519614] sd 2:0:0:0: [sdc] tag#2 CDB: Write(16) 8a 00 00 00 00 01 21 41 56 60 00 00 00 08 00 00
Nov 6 12:27:09 user kernel: [10350005.519616] print_req_error: I/O error, dev sdc, sector 4852897376
Nov 6 12:27:09 user kernel: [10350005.519620] Buffer I/O error on dev sdc, logical block 606612172, lost async page write
Nov 6 12:27:09 user kernel: [10350005.519637] sd 2:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Nov 6 12:27:09 user kernel: [10350005.519641] sd 2:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 01 21 41 56 40 00 00 00 10 00 00
Nov 6 12:27:09 user kernel: [10350005.519643] print_req_error: I/O error, dev sdc, sector 4852897344
Nov 6 12:27:09 user kernel: [10350005.519648] Buffer I/O error on dev sdc, logical block 606612168, lost async page write
Nov 6 12:27:09 user kernel: [10350005.519653] Buffer I/O error on dev sdc, logical block 606612169, lost async page write
Nov 6 12:27:09 user kernel: [10350005.519664] Buffer I/O error on dev sdc, logical block 606603630, lost async page write
Nov 6 12:27:09 user kernel: [10350005.520026] JBD2: Detected IO errors while flushing file data on sdc-8
Nov 6 12:44:41 user kernel: [10351057.165709] Aborting journal on device sdc-8.
Nov 6 12:44:41 user kernel: [10351057.269153] EXT4-fs error (device sdc): ext4_journal_check_start:61: Detected aborted journal
Nov 6 12:44:41 user kernel: [10351057.269160] EXT4-fs (sdc): Remounting filesystem read-only
Nov 6 12:44:41 user kernel: [10351057.277460] EXT4-fs error (device sdc): ext4_journal_check_start:61: Detected aborted journal
Nov 6 12:44:41 user kernel: [10351057.285798] EXT4-fs error (device sdc): ext4_journal_check_start:61: Detected aborted journal
Nov 6 12:44:41 user kernel: [10351057.344165] EXT4-fs error (device sdc): ext4_journal_check_start:61: Detected aborted journal
Nov 6 12:44:41 user kernel: [10351057.352474] EXT4-fs error (device sdc): ext4_journal_check_start:61: Detected aborted journal
Nov 6 12:44:41 user kernel: [10351057.352483] EXT4-fs (sdc): ext4_writepages: jbd2_start: 1024 pages, ino 86550607; err -301 Answer
This is likely a hardware problem with your disk. The kernel is sending data to the disk, but the disk has timed out in sending a reply. It may happen to work after restarting the system or the disk, but the problem will probably reoccur again and again until the disk dies.
You should copy the data to a new disk using dd if that's possible and then discard the old disk. You can try querying the disk with smartctl as outlined in this document about running SMART self-tests to see if it reports any problems.
