Moving to Soft RAID 5 on Linux

These days, more and more often do I need to access old data that I have remotely (sending out music demos, old pictures, old documents, old codes, etc.), and I quickly learned that my old way of finding the index document and inserting the optical media to access an old file doesn't work well anymore. With the terabyte-level hard drive being available, I decided to move most of my data onto big hard drives that are running live. After a while, I realized the huge time cost in moving the data around. My solution is to have redundancy so if the hardware fails, I can quickly do hard drive to hard drive recovery instead of optical medium to hard drive copying.

I heard many fellow system admins complaining about hardware RAID controller going End of Life and they can't access their data after their last backup controller going dead. Therefore, I chose to go software RAID. Considering performance isn't crucial for my purpose, this should be a reasonable solution.

The hard drives I use are three of 1TB WD10EACS. I chose them for the price and for their low temperature/power design. However, I had only got one drive at first and started to put data on that drive, before I realized that I would like RAID. Due to my linux box's limitation in the number of SATA ports, I can only have two more WD10EACS. RAID 5 is my choice for its nice trade-off between performance and capacity.

Normally, one would build RAID 5 on 3 drives and copy data onto that RAID 5. However, in my case, one of the drive already contains that data that I want to keep so I can't build a 3-disk RAID 5 from the beginning. After some research, I learned that we can actually start from a degraded RAID 5 with 2 drives and add the 3rd one in later on.

Linux Raid is an excellent guide for me to do so. Managing RAID and LVM with Linux also helped a lot. Thanks to those kind minds.

/dev/sda is the drive that already contains the data (in /dev/sda1)
/dev/sdb and /dev/sdc are new drives.

I decided to use build a large partition instead of using the whole disk, in case in the future I am having trouble getting a drive that has slightly larger capacity for replacement. Using a partition that's slightly smaller than the whole drive capacity will save us from having to rely on manufacturer producing 100% identical capacity hard drive in the future.

(As a side note: In one of the posts in Managing RAID and LVM with Linux, the author mentioned a Slashdot solution that involves 10 partitions. They are perfectly right that solution provides a lot of flexibility, but the steps are too tedious and take too long. I can't imagine myself spending that much time in the future just to expand the capacity. With the continuous price dropping of large volume hard drives, I believe I will be able to find and afford larger hard drives for data migration.)

# fdisk /dev/sdb

Create one partition that's slightly smaller then the drive's capacity. In my case, I use cylinder 1-121577, as opposed to 1-121601. That's about 1000.2GB. I also changed the partition ID to "da", Non-FS data. For those who are not familiar with fdisk, the commands are: "new->primary->1->1->121577" and "t->da", and then "w", and then "q".

Do the same for another disk.
# fdisk /dev/sdc

Now we are ready to build a 2-disk RAID 5 across /dev/sdb1 and /dev/sdc1.

# mdadm --create --verbose /dev/md0 -l5 -n2 /dev/sdb1 /dev/sdc1

Now /dev/md0 is ready to be used as a drive. However, for future flexibility, I decide to build LVM on top of RAID 5.

# pvcreate /dev/md0
# vgcreate -s 64M lvm-tb-storage /dev/md0

(Note that I use lvm2 so there is no restriction on the number of extends in each logical volume. However, according to "man vgcreate", too many extends may slow down the tools. Therefore, for a few TB, setting 64M as the physical extent size is probably better.)

# vgdisplay lvm-tb-storage
# lvcreate -L931.32G -n tb lvm-tb-storage
# mkfs.ext3 -b 4096 -R stride=8 /dev/md0

Then we can mount the drive. Here's my /etc/fstab

/dev/lvm-tb-storage/tb /mnt/lvm-tb-storage-tb ext3 defaults 1 1

And then we can copy the data onto that. I used "rsync -a" so I can resume it if it stopped for some reason.

# rsync -a old_1TB_drive_mount_point/* new_raid_mount_point

To make the system bootable, we add the RAID setting into its config file

# mdadm --detail --scan > /etc/mdadm.conf

Now, we have a 2-disk RAID 5 that has all the data we need and the original 1TB drive that we can add to the RAID 5.

First, do the same partitioning for this old drive.

# fdisk /dev/sda

Then we are ready to add it into the array.

# mdadm --grow /dev/md0 -n3 --backup-file=raidbackup
# mdadm -a /dev/md0 /dev/sda1

Although the shell prompt pops up right away, the rebuilding has started in the background. Use this to check.

# mdadm --detail /dev/md0

I could leave a shell open and repeat the following command once in a while:

# date; mdadm --detail /dev/md0 | grep "Reshape Status"

It should show a time stamp and something like "Reshape Status : 0% complete".

However, for convenience, e.g. remotely management, I put the following line in my crontab and execute it hourly.

/usr/bin/perl -le 'print "---------\n" . `/sbin/mdadm --detail /dev/md0 `;' >>/root/rebuild_progress.txt 2>&1

From the log, it operated at a rate of approximately 1% per hour. Therefore, the whole process took about 7-8 days to complete. Of course, I can still read/write to the RAID during this time. Or, if you don't mind system taking heavier load, check out this advice for how to speed up the reconstruction.

After the process is completed (this is after 8 days), we want to update the mdadm.con so next time when we boot it'll have the latest setting.

# mdadm --detail --scan > /etc/mdadm.conf

Now we can resize the LVM.

# pvresize /dev/md0
# vgdisplay lvm-tb-storage

This is what it looks like.

[root@localhost ~]# vgdisplay lvm-tb-storage
  --- Volume group ---
  VG Name               lvm-tb-storage
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               4.00 MB
  Total PE              476839
  Alloc PE / Size       238422 / 931.34 GB
  Free  PE / Size       238417 / 931.32 GB
  VG UUID               wqIXsb-KRZQ-eRnH-JvuP-VdHk-XJTG-DSWimc
   
[root@localhost ~]# lvresize -l +238450 /dev/lvm-tb-storage/tb
  Extending logical volume tb to 1.82 TB
  Insufficient free space: 238450 extents needed, but only 238389 available
[root@localhost ~]# lvresize -l +238389 /dev/lvm-tb-storage/tb
  Extending logical volume tb to 1.82 TB
  Logical volume tb successfully resized
[root@localhost ~]# vgdisplay lvm-tb-storage
  --- Volume group ---
  VG Name               lvm-tb-storage
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               4.00 MB
  Total PE              476839
  Alloc PE / Size       476839 / 1.82 TB
  Free  PE / Size       0 / 0   
  VG UUID               wqIXsb-KRZQ-eRnH-JvuP-VdHk-XJTG-DSWimc

Note that I tried to gave it a bigger PE number than available so it would warn me and give me the max PE number. Also note that my PE size is 4MB, an unfortunately mistake I made, which would require too much time to fix. As I said before, the optimal PE size should be 64MB.

Now we are ready to expand the file system.

Although resize2fs should work online, to be safe, we first unmount the volume

# umount /dev/mapper/lvm--tb--storage-tb

Then do a check before we modify the file system. This takes a while.

# e2fsck -t -f /dev/mapper/lvm--tb--storage-tb

And now resize the file system. (The -p option shows the progress bar. Since we already assigned RAID-stride when we created the file system, we don't need to do it again with -S)

# resize2fs -p /dev/lvm-tb-storage/tb

Now we can re-mount and enjoy the expanded RAID 5 storage.

# mount -all

As a side mark, if you are getting crazy temperature reading from smartd in /var/log/messages, try editing /etc/smartd.conf and add "-R 194" in the option. My /etc/smartd.conf has one line (see smartmontools's FAQ page for instructions):

DEVICESCAN -a -R 194

Game & Music

Here's a talk given by Matsuura (some transcription can be found here) on today's game music, and games themselves. I'm not much of a video gamer (any more) but found his speech very insightful and inspirational.

Modifiers Across Keyboards

I use a MacBook Pro with an external screen on its left (desktop expansion). I'm also using a bluetooth keyboard as a second keyboard on the left. This allows me to put each hand on one keyboard (left on BT, right on the MacBook) when I type. If I were to type on only one keyboard, my neck will be in a constant uncomfortable position. Besides, I need to access the multi-touch pad constantly.

By default, OS X seems to keep track of modifiers (CMD, ALT, CTRL, SHIFT, etc.) on a per-keyboard basis, i.e. pressing the SHIFT on one keyboard while pressing another letter on another won't have any shifting effect.

However, in another PC setup, this is the default behavior so I can comfortably use one keyboard for each hand. (Sort of the split keyboard concept).

I posted my question on Apple Support forum without any response so I had to do it myself. Hopefully this little code would save somebody in the same situation some time.

PS: Later I realized that it doesn't work for password fields in Firefox. I guess Firefox somehow bypasses the standard hooking mechanism (?).

UPDATE: 2008/09/25

The current program does not aggregate modifiers across different keyboards, i.e. if I press CMD on one keyboard and then SHIFT on another, it only recognizes the later one. It's not hard to modify the code (basically apply the CHANGES from a specific keyboard to the universal Modifier flag holder). I'll come back to this if I get some time.

UPDATE: 2009/03/13

Per Nathan Schmidt's suggestion, I'm sharing my compiled binary here for those who don't have convenient access to compilers: Cross Keyboard Modifiers binary.

UPDATE: 2009/07/07

Thanks to Jonathan for asking for detailed steps. Here they are:

1. Download the binary (see the link in the previous update) into your home directory (or anywhere of your choice. I downloaded it into a directory called "bin" under my home directory, i.e. /Users/tk/bin . "tk" is my username, "bin" is a directory I created under my home directory.)

2. Open up Terminal, type "chmod 755 /Users/tk/bin/cross_keyboard_modifiers". This is one time setup. You can close the Terminal after this.

3. Now you should be able to double click to run the program via Finder. Finder will open Terminal to run the program. Note that this way you'll need to keep Terminal open for the program to stay running. If you want to the process to run without going through Terminal, read on.

4. If you want this program to run every time you boot OS X, you can do so in System Preference > Accounts > Select your account > Login Items > + sign > select the program, i.e. "/Users/tk/bin/cross_keyboard_modifiers". After this, you'll need to reboot for this to work.

UPDATE: 2011/08/16

Thanks to brookswift for pointing out a solution that works with newer OS. I haven't tried it myself but it seems others have found it helpful.

Source code:

// Name:    cross_keyboard_modifiers.c
// Version: 1.0.0
// Date:    2008/09/19
// Author:  Tsan-Kuang Lee 
// Site:    http://blog.tklee.org
//
// What?
//
// This program allows OS X users to use multiple keyboards at the same time.  By default (10.5.4, as far as I know), the modifiers
// (CMD, CTL, SHIFT, etc.) only work with other keys ON THE SAME KEYBOARD.  For example, if you press SHIFT key and the key "a" on
// your laptop, you get a capital "A"; if you press SHIFT on your laptop keyboard and the "a" key on an external keyboard (say, a
// bluetooth keyboard), you get a lowercase "a".
//
// Why?
//
// Keyboard is my primary input device.  After long time use, I found it helpful if I can use one keybaord for each hand so my wrist
// can type in a more comfortable position.  Having two keyboards help me a lot; however, I learned to type in a way that whenever
// SHIFT key is needed, I'll use a different hand for it's combined key.  For example, I'll use left shift for SHIFT-P, but right shift
// for SHIFT-A.  This increases the speed, too. I searched everywhere online but couldn't find any solution that can change OS X's
// default behavior.  Therefore I had to learn how OS X handles things.
//
// How?
//
// By default, OS X keeps different modifier flags for different devices.  We tap into the System's event handling system and keep a 
// local copy of the modifiers' flags.  Whenever a non-modifier key is pressed, we reset the modifers' flags with the value we keep.  
//
// Credit
//
// I studied the code from the following sources.  They are very helpful.  Thanks to their authors.
//  * alterkeys.c http://osxbook.com
//  * http://lists.apple.com/archives/quartz-dev/2007/Jan/msg00049.html
//  * http://developer.apple.com/documentation/Carbon/Reference/QuartzEventServicesRef/Reference/reference.html
//
// Usage
//
// Complile using the following command line:
//     gcc -Wall -o cross_keyboard_modifiers cross_keyboard_modifiers.c -framework ApplicationServices
// Then run cross_keyboard_modifiers.  (Be sure not to let the process go into sleep, e.g. ctrl-z at the shell.  
// Otherwise, you may lose your keybaord input because of the never wakened process.  The easiest way to do so is
// put it in the background right away, i.e. "cross_keyboard_modifiers &")
//
// You need superuser privileges to create the event tap, unless accessibility
// is enabled. To do so, select the "Enable access for assistive devices"
// checkbox in the Universal Access system preference pane.

#include <ApplicationServices/ApplicationServices.h>

// This callback will be invoked every time there is a keystroke.
CGEventRef myCGEventCallback(CGEventTapProxy proxy, CGEventType type, CGEventRef event, void *refcon)
{
static CGEventFlags flags = (CGEventFlags) NULL;

if (flags == (CGEventFlags)NULL)
{
flags = CGEventGetFlags(event);
}

switch (type) {
case kCGEventKeyDown:
case kCGEventKeyUp:
{
CGEventSetFlags(event, flags);
break;
}
case kCGEventFlagsChanged:
{
flags = CGEventGetFlags(event);
break;
}
}
return event;
}

int main(void)
{
CFMachPortRef      eventTap;
CGEventMask        eventMask;
CFRunLoopSourceRef runLoopSource;

//CGEventFlags oldFlags = CGEventSourceFlagsState(kCGEventSourceStateCombinedSessionState);
CGEventFlags oldFlags = CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState);

// Create an event tap. We are interested in key presses.
eventMask = CGEventMaskBit(kCGEventKeyDown) | CGEventMaskBit(kCGEventKeyUp) | CGEventMaskBit(kCGEventFlagsChanged);
eventTap = CGEventTapCreate(kCGSessionEventTap, kCGHeadInsertEventTap, 0, eventMask, myCGEventCallback, &oldFlags);
if (!eventTap) {
fprintf(stderr, "failed to create event tap\n");
exit(1);
}

// Create a run loop source.
runLoopSource = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, eventTap, 0);

CFRelease(eventTap);

// Add to the current run loop.
CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSource, kCFRunLoopCommonModes);

// Enable the event tap.
CGEventTapEnable(eventTap, true);

CFRelease(runLoopSource); 

// Set it all running.
CFRunLoopRun();

// In a real program, one would have arranged for cleaning up.
exit(0);
}

MultiClutch Keycodes

On MacBook Pro, I use MultiClutch to add some convenient gestures to my desktop. This application surely opens the potential of the multi-touch pad. However, I'm having trouble bind arbitrary hotkeys to various gestures, so I look into the source code and found that if you input the keycodes directly, you can setup any key bindings.

You can edit the file:
/Users/YourLogin/Library/Preferences/com.wonderboots.MultiClutchBindings.plist

These is my setting, in which you can find Command-Tab which is usually not allowed in the interface.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>bindings</key>
<dict>
<key>Firefox</key>
<dict>
<key>Rotate Left</key>
<string>56,55,33</string>
<key>Rotate Right</key>
<string>56,55,30</string>
<key>Supports_Rotate</key>
<string>true</string>
<key>Supports_Zoom</key>
<string>false</string>
<key>Swipe Left</key>
<string>55,123</string>
<key>Swipe Right</key>
<string>55,124</string>
</dict>
<key>Global</key>
<dict>
<key>Supports_Rotate</key>
<string>false</string>
<key>Supports_Zoom</key>
<string>true</string>
<key>Swipe Down</key>
<string>55048</string>
<key>Swipe Left</key>
<string>56055048</string>
<key>Zoom In</key>
<string>55069</string>
<key>Zoom Out</key>
<string>55078</string>
<key>Zoom Out, Zoom In</key>
<string>100</string>
</dict>
</dict>
<key>donateHidden</key>
<string>NO</string>
<key>version</key>
<string>b4</string>
</dict>
</plist>

Why Blogger?

I had talked about some publishing systems in the post, Why Choose Blog. However, in mid-2008, I decided to move everything to a platform that's managed by a more stable and committed provider, Google, thinking it would save me a lot of time maintaining the technical side of things and enables me to focus more on the contents themselves. This decision involves new technical consideration, life priority, and the trends of the time.

On the technical side, Serendipity is a very good product. Otherwise I wouldn't have chosen it and spent so much time hacking it. However, after a while, it became too time consuming to patch my mods onto every new release.

The other major problem of the old system is MySQL didn't support Chinese search. I highly doubt any PHP (or Perl, or any script) based solution would include a Chinese Segmentation Engine for good Chinese searching. Therefore, I either have to rely on an external search engine, or give up the hope on local Chinese search.

Professionally I have to be very familiar with Drupal and Movable Type, and I do find Drupal very promising. The Drupal community is excellent and I think the framework is very mature and elegant. However, one thing that I learned about myself in the past few months is my current life priority does not allow me to throw in too much time maintaining a production site. Therefore, I made a decision: sacrifice the flexibility of coding things myself in exchange of precious time for my First Life.

Another important lesson that I learned after I started to work is the power of team work. DIABY (Do It All By Yourself) is not the trend in modern professional world. I learned to accept imperfection (or my intolerance of things not being my way) in exchange of the benefit of balance of life -- if I insist on doing everything on my own, I will be too occupied and exhausted to actually live. The trends nowadays are collaboration, outsourcing, and openness. Doing everything on my own is just not feasible at all (although the image of Mac Gyver or any of those heroes came to mind) and heavily customizing my personal blog satisfies nobody but myself. I think I'd be a better internet citizen if I contribute more meaningful contents instead of a very customized look of my blog. By using a more common platform, more people can have access to it with their own familiar tools.

So that's it. A Blogger now I am.

About this Blog (moved to Blogger)

I had to move the site again -- the original hosting space became unavailable about 9 months ago so the original site was down for that long. Unfortunately, I was going through a time too overwhelming to take care of things like that. Now my life finally came back to normal and I decided to use a relatively stable provider to host my writing and to share my thoughts with people. After a careful consideration of my situation and the resources out there, I decided to use Google's Blogger service. (I'll share the technical consideration process in another post.)

Anyway, I managed to convert everything from my old blog system to Google's Blogger. However, a lot of the hyperlinks in the contents are still linking to the old system. I'll fix them one by one. In the mean time, you can easily search within my Blogger to find the intended linked page.

Also see the old About page.