ORA-00206: error in writing (block #, # blocks #) of control file | ORA-00202 ORA-27063

ORA-00206 ORA-00202 ORA-27063 Errors in Alert Log | Issues with Controlfile

This happened with one of our Test Environments where we are having a single instance database and an application configured on the same. The problem was observed when the database was found to be Down and just 5 minutes before it was working fine. There could be many reasons for the same but 2 main guesses were:

1. Either there is a power failure.
2. Check the Alert Log.

There was no power failure. After confirming the same, I checked the Alert Log and the below mentioned errors were there:

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/data/testdb/control03.ctl'

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 5: I/O error

Additional information: -1

Additional information: 16384

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/data/testdb/control02.ctl'

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 5: I/O error

Additional information: -1

Additional information: 16384

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/data/testdb/control01.ctl'

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 5: I/O error

Additional information: -1

Additional information: 16384

Errors in file /oracle/app/oracle/diag/rdbms/testserv/testserv/trace/testserv_ckpt_23265504.trc:

ORA-00221: error on write to control file

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/data/testdb/control03.ctl'

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 5: I/O error

Additional information: -1

Additional information: 16384

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/data/testdb/control02.ctl'

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 5: I/O error

Additional information: -1

Additional information: 16384

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: '/data/testdb/control01.ctl'

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 5: I/O error

Additional information: -1

Additional information: 16384

CKPT (ospid: 23265504): terminating the instance due to error 221

Sat Aug 18 12:16:13 2018

System state dump requested by (instance=1, osid=23265504 (CKPT)), summary=[abnormal instance termination].

System State dumped to trace file /oracle/app/oracle/diag/rdbms/testserv/testserv/trace/testserv_diag_24314088_20180818121613.trc

Dumping diagnostic data in directory=[cdmp_20180818121613], requested by (instance=1, osid=23265504 (CKPT)), summary=[abnormal instance termination].

Termination issued to instance processes. Waiting for the processes to exit

Sat Aug 18 12:16:28 2018

Instance termination failed to kill one or more processes

Instance terminated by CKPT, pid = 23265504

Sat Aug 18 12:21:18 2018

Adjusting the default value of parameter parallel_max_servers

from 160 to 120 due to the value of parameter processes (150)

Reason:

The reason is not inside database but outside on OS/hardware level. This is because of the problem with I/O Subsystem. This can be because of some bad sector on Disk, some bad array cable, issues with Network or can be because of some OS level Bug.

Resolution:

Consult your OS/Hardware Vendor to run some diagnostics test on the Disks where the problematic files are present and ask him/her to investigate the reason for this I/O error.

I hope this helps !!