LSI MegaRaid – Quick configuration without web based BIOS on SuperMicro IPMIs

Setting up a hardware raid configuration remotely can be a rather tiresome task if you do not happen to perform such on a daily basis. Using LSI’s web based client via IPMI can be an extremely nerve wrecking pastime, and since we have run into this scenario a couple of times already, we thought it might be useful to post a very quick getting started guide.

Getting into the configuration shell

  1. Connect to your server’s IPMI and launch the remote console.
  2. Boot the server and enter the preboot client shell (CLI) when the option is presented on screen.

How to create a virtual drive

In the shell, list all your physical disks this way:

-PDList -aALL

This will list all physical devices on all adapters. There will be a lot of scrollage, so you may need to capture (if pause does not work – which is often the case…) the screen with the inbuilt video capture functionality. You will need the following values:

  1. adapter ID (usually 0 or 1)
  2. enclosure ID (often 252, but not always)
  3. slot ID (per disk, varies)

Next, create a virtual disk with your desired RAID setup, e.g. a simple Raid 1 can be done this way (assuming we have the controller’s ID as 0, enclosure ID 252, and the two disks in question on slots 2 and 3 respectively):

-CfgLdAdd -r1 [252:2,252:3] -a0

You are good to go now – reboot, and your OS should pick up the virtual disk now.

Further reading and information:

Software Raid: readding disks after replacement

As we have recently seen it on a client’s server we manage, hosted in another ISP’s DC: Below please see a quick typical way to get a broken (degraded) software raid array back to healthy and clean:

First, check the raid status as such:

# cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
md0 : active raid1 sdb1[1] sda1[0]
4198976 blocks [3/2] [UU_]

md1 : active raid1 sdb2[1] sda2[0]
2104448 blocks [3/2] [UU_]

md2 : active raid5 sdc3[2] sdb3[1] sda3[0]
2917660800 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

This shows that md0 and md1 have issues.

Go into details for each problem device to get some additional information:

# /sbin/mdadm –detail /dev/md0

/dev/md0:
Version : 0.90
Creation Time : Tue Jun 8 07:47:09 2010
Raid Level : raid1
Array Size : 4198976 (4.00 GiB 4.30 GB)
Used Dev Size : 4198976 (4.00 GiB 4.30 GB)
Raid Devices : 3
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Aug 21 06:24:07 2013
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

UUID : ...
Events : 0.3000

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 0 0 2 removed

Removed devices no longer need to be removed with mdadm (this could be done with mdadm –manage /dev/mdX –fail /dev/sdYY). Repeat for /dev/md1 or any other failed / degraded raid devices.

Replace the failed drive (hot swap, recable, etc. – may have to reboot the machine for the OS to recognise the new disk even in a hot swap scenario).

After replacing the bad disk, clone the partition table to the new disk:

# /sbin/sfdisk -d /dev/sda | sfdisk /dev/sdY

Then, re-add the new devices to the array:

# /sbin/mdadm –manage /dev/mdX –add /dev/sdYY

Repeat for all failed devices – the software raid system will automatically schedule the respective resyncs.

Please also see: http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array

IOPS and RAID considerations

IOPS (input/output operations per second) are still – maybe even more so than ever – the most prominent and important metric to measure storage performance. With SSD technology finding its way into affordable, mainstream server solutions, providers are eager to outdo each other offering ever higher IOPS dedicated servers and virtual private servers.

While SSD based servers will perform vastly better than SATA or SAS based ones, especially for random I/O, the type of storage alone isn’t everything. Vendors will often quote performance figures using lab conditions only, i.e. the best possible environment for their own technology. In reality, however, we are facing different conditions – several clients competing for I/O, as well as a wide ranging mix of random reads and writes along with sequential I/O (imagine 20 VPS doing dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync).

Since most providers won’t offer their servers without RAID storage, let’s have a look at how RAID setups impact IOPS then. Read operations will usually not incur any penalty since they can use any disk in the array (total theoretical read IOPS available therefore being the sum of the individual disks’ read IOPS), whereas the same is not true for write operations as we can see from the following table:

RAID level Backend Write IOPS per incoming write request
0 1
1 2
5 4
6 6
10 2

We can see that RAID 0 offers the best write IOPS performance – a single incoming write request will equate to a single backend write request – but we also know that RAID 0 bears the risk of total array loss in case a single disk fails. RAID 1 and 10, the latter being providers’ typical or most advertised choice, offers a decent tradeoff – 2 backend writes per single incoming write. RAID 5 and RAID 6, with their additional, robust setup, bear the largest penalty.

When calculating the effective IOPS, thus, keep in mind the write penalty individual RAID setups come with.

The effective IOPS performance of your array can be estimated using the following formula:

IOPSeff = n * IOPSdisk / ( R% + W% * FRAID )

with n being the number of disks in the array, R and W being the read and write percentage, and F being the RAID write factor tabled above.

We can also calculate the total IOPS performance needed based on an effective IOPS workload and a given RAID setup:

IOPStotal = ( IOPSeff * R% ) + ( IOPSeff * W% )

So if we need 500 effective IOPS, and expect around 25% read, and 75% write operations in a RAID 10 setup, we’d need:

500 * 0.25 + 500 * 0.75 * 2 =  875 total IOPS

i.e. our array would have to support at least 875 total, theoretical IOPS. How many disks/drives does this equate to? Today’s solid state drives will easily be able to handle that, but what about SATA or SAS based RAID arrays? A typical SAS 10k hard disk drive will give you around 100-140 IOPS. That means we will need 8 SAS 10k drives to achieve our desired IOPS performance.

Conclusion:
All RAID levels except RAID 0 have significant impact on your storage array’s IOPS performance. The decision about which RAID level to use is therefore not only a question about redundancy or data protection, but also about resulting performance for your application’s needs:

  1. Evaluate your application’s performance requirements;
  2. Evaluate your application’s redundacy needs;
  3. Decide which RAID setup to use;
  4. Calculate the resulting IOPS performance necessary;

 

Sources:

Calculate IOPS in a storage array by Scott Lowe, TechRepublic, 2/2010
Getting the hang of IOPS by Symantec, 6/2012