Exadata: Disk controller was hung. Cell was power cycled
Just another manic magic Monday. After a great weekend, we came to office and performed our daily health checks like every Monday. One of our storage server(cell) of Exadata X2-2 X4270 M2 had lost 11 ASM disks of total 34 ASM disks. We struck it lucky, all databases were still up despite all the lost. Lets examine what happened to our cell server. When i checked the mailbox, i saw an alert mail from the problematic cell stating that "Disk controller was hung. Cell was power cycled". It looks like cell disk controller was not performing well (may be a bug or a peak moment) and forced server to reboot. But normally reboots does not end up with disk losses. I started with checking with cells physicaldisk status. What I got from output was; we had one flashdisk and one harddisk failure (disk number 3) and also one harddisk was in import failure status(disk number 7). But that did not explain 11 ASM disks failure. It should had been 6 ASM disks according to output. There shoul