Posts

Showing posts from September, 2011

Slow database? Check RAID battery!

Executive Summary: 

If your Dell database servers get slow suddenly, and I/O seems sluggish, do yourself a favor and check if the RAID battery is currently going through its 'relearning' cycle. If this is so, then the Write-Back policy is disabled and Write-Through is enabled -- as a result writes become very slow compared to the standard operation.

Details:

This turns out to be a fairly well known problem with RAID controllers in Dell servers, specifically LSI controllers. The default mode of operation for the RAID battery is to periodically go through a so-called 'relearn cycle', where it discharges, then charges and recalibrates itself by finding the current charge. In this timeframe, as I mentioned, Write-Back is disabled and Write-Through is enabled.

For our MySQL servers, we have innodb_flush_log_at_trx_commit set to 1, which means that every commit if flushed to disk. In consequence, the Write-Through mode will severely impact the performance of the writes to the…

Setting up RAID1 with mdadm on a running Ubuntu server

For the last couple of weeks we've been working on setting up a bunch of Dell C2100 servers with Ubuntu 10.04. These servers come with 2 x 500 GB internal disks that can be set up in RAID1 with the on-board RAID controller. However, when we did that during the Ubuntu install, we never managed to get back into the OS after the initial reboot, or in some cases GRUB refused to write to the array (I think it was when we tried 11.04 in a desperation move). To make matters worse, even software RAID via mdadm stopped working, with the servers going to the initramfs BusyBox prompt after the initial reboot. My guess is that it all stems from GRUB not writing properly to /dev/md0 (the root partition RAID-ed during the install) and instead writing to /dev/sda1 and /dev/sdb1. So we decided to install the root partition and the swap space on /dev/sda and leave /dev/sdb alone.

I started to look for articles on how to set up RAID1 post-install, when you have the OS installed on /dev/sda, and you…