How To Boost mailserver storage performance using Facebook FlashCache

Few months ago we had performance problem with mailserver storage.

Storage has 3 HDD in RAID5 configuration. Average disk utilization per
one disk was about 90 % - 97%.
One of goals was incrase storage performance but with minimum downtime
so -> no data migration.



I'd like to try some cheap solution first and something new.
Facebook relased interesting project called flashcache.
Flashcache give you oportunity tu combine your old storage and new fast
SSD disk. All read and writes operations are cached through SSD disk.

I do some practice on testing enviroment and then implement on our
mailserver. All was really easy to use and setup.

First numbers ware absolutly amazing. Only few hours after enabling
flashcache iowait drop down from 20 - 40% to 1 - 3%. In first 3 hours
cache hit rate grow to 87 %.
Whole performance of storage reise up and imap response drop from
seconds to miliseconds.

Lest's have a look on some graphs.

[caption id="attachment_441" align="alignnone" width="471"] server load - before FlashCache (before Week 50) and after FlashCache implementation[/caption]

[caption id="attachment_442" align="alignnone" width="471"] SSD & HDD Read / Write operations[/caption]

[caption id="attachment_443" align="alignnone" width="471"] FlashCache Hits[/caption]

[caption id="attachment_444" align="alignnone" width="471"] FlashCache Hit Rate (%)[/caption]

[caption id="attachment_445" align="alignnone" width="471"] SSD usage (%)[/caption]

[caption id="attachment_446" align="alignnone" width="471"] FlashCache MetaData[/caption]

I am using Linux Debian and I create my own startup scripts. They are simple but they works. Feel free to use.

/etc/default/flashcache

# FlashCache Configuration
#

# FlashCache configured ?
FLASH_CACHE_START=YES

# FlashCache binaries
FLASH_CACHE_BIN=/usr/local/sbin

# New Flashcache device
FLASH_CACHE_DEV=flashcache0

# Original data device
DATA_DEV=/dev/md1

# Cache device - Fast disk / SSD
CACHE_DEV=/dev/sdd

/etc/init.d/flashcache-init

#!/bin/bash
### BEGIN INIT INFO
# Provides:          flashcache-init
# Required-Start:
# Required-Stop:
# Default-Start:     S
# Default-Stop:      0 6
# Short-Description: Prepare flashcache device during boot time
# Description:       Create flashcache device during boot
### END INIT INFO
# debian lenny
# update-rc.d flashcache-init start 27 S . stop 45 0 6 .

# Load Configuration
. /etc/default/flashcache

RED="^[[31m"
GREEN="^[[32m"
YELOW="^[[33m"
BLUE="^[[34m"
NOC="^[[0m"

export PATH=$PATH:$FLASH_CACHE_BIN:/usr/bin:/usr/sbin/

function do_start {

if ! [[ "$FLASH_CACHE_START" = "YES" ]] ; then

        echo "${RED}Flashcache not configured or turned off${NOC}"
        exit 0

fi
echo "${BLUE}Starting FlashCache${NOC}"
echo -n "${BLUE}Loading kernel module...${NOC}"

if modprobe flashcache ; then
        echo "${GREEN}OK$NOC"
else
        echo "${RED}FAILED$NOC"
        exit 1
fi

echo -n "${BLUE}Loading FlashCache${NOC}${YELOW}"
if flashcache_load $CACHE_DEV; then

        echo "${NOC}${GREEN}Loading FlashCache OK${NOC}"

else

        echo "${RED}ERROR: Problem during FlashCache loading${NOC}"
        exit 1

fi
}

function do_stop {

echo "${BLUE}Stoping FlashCache${NOC}${YELOW}"
if dmsetup remove $FLASH_CACHE_DEV ; then
        echo "${NOC}${GREEN}Successfully removed${NOC}"
        exit
else
        echo "${RED}ERROR: Some problems during remove${NOC}"
        exit 1
fi

}

case "$1" in
"start")
do_start
;;
"stop")
do_stop
;;
*)

cat  << EOF
Initialization script for FlashCache
Usage: $0 <start|stop>
EOF
exit 1

;;
esac

All these scripts counts FlashCache is already initialized.

There is no need to any data migration or manipulate with old filesystem. You can anytime stop flascache, destroy it and mount your old storage.

Simple way to initialize flashcache as writeback cache in our example.

# flashcache_create -v -p back flashcache0 /dev/sdd /dev/md1

# ls -l /dev/mapper/flashcache0
brw-rw---- 1 root disk 253, 0 2011-12-13 03:16 /dev/mapper/flashcache0

• /dev/sdd - whole SSD disk


• /dev/md1 - our data partition with xfs filesystem


• flashcache0 - name of new cache device (combination of SSD disk and our old storage)

Now, in last step, you must change your /etc/fstab and rewrite /dev/md1 to /dev/mapper/flashcache0.

If you are using debian use update-rc.d command to enable our init script at startup and reboot/halt.

Flashcache initialization is easy but please follow steps and recommendations in flashcache wiki/documentation.

You can find very nice tips how to tune up flashcache with XFS filesystem in their documentation.

Links

• FlashCache Project on GitHub


• FlashCache Documentation