Flashcache

From ArchWiki

Tango-edit-clear.pngThis article or section needs language, wiki syntax or style improvements. See Help:Style for reference.Tango-edit-clear.png

Reason: Written in first person (Discuss in Talk:Flashcache)

Related articles

Flashcache is a module originally written and released by Facebook(Mohan Srinivasan, Paul Saab and Vadim Tkachenko) in April of 2010. It is a kernel module that allows Writethrough caching of a drive on another drive. This is most often used for caching a rotational drive on a smaller solid-state drive for performance reasons. This gives you the speed of an SSD and the size of a standard rotational drive for recently cached files. Facebook originally wrote the module to speed up database I/O, but it is easily extended to any I/O.

An alternative to Flashcache is Bcache

Warning: Installation of flashcache at this point is *not* a simple task and should only be attempted by people comfortable with such things. Please read the instructions thoroughly before attempting to install this driver. It is very easy to get a non-booting machine if you are not careful.

It is important to note that this driver is intended to speed up a slower drive with a faster one. It is not required that one of the drives is a solid-state drive. It could be useful in a situation with widely different speed drives such as a fast and small SCSI drive and a larger SATA drive or even a slower "green" drive and a super-fast 10k rpm drive.

Installation

Tip: Throughout the page, /dev/sda will be used to indicate the slow drive and /dev/sdb will used to indicate the fast drive. The two partitions sda3 and sda4 are being cached on sdb1 and sdb2 respectively. Be sure to change these examples to match your setup.

Getting the kernel module

Warning: You must have the full kernel source with device-mapper enabled or built as a module.

At this time the kernel module is not included in the arch packages or AUR, so you must download the file directly from github at https://github.com/facebook/flashcache. Extract the image and compile the module/utils: 

$ tar -xzvf facebook-flashcache-1.0-64-g085b7ba.tar.gz
$ cd facebook-flashcache-085b7ba
$ make KERNEL_TREE=<root of kernel source tree> (most likely in /usr/src)

If all goes well you are ready to install the module(you need root access for this): 

# make KERNEL_TREE=<root of kernel source tree> install

From there I copied the executables to /usr/sbin for ease of use: 

# cp src/utils/{flashcache_create,flashcache_load,flashcache_destroy} /usr/sbin

Preparing the fast drive

To prepare the fast drive for the caching required, we need to simply partition it based on our desired cache sizes. I wanted to cache a good deal of the root partition and some of my home partition, so I made my partitions 4 gigs and 116 gigs respectively. Take a look at your partitions and make an informed decision about the partition sizes you require.

Creating flashcache

# flashcache_create cached_part1 /dev/sdb1 /dev/sda3
# flashcache_create cached_part2 /dev/sdb2 /dev/sda4

Starting Drives on Boot

Depending on where you intend to use flashcache, you can choose either to start the drives at early boot, essential if you are caching the root file-system, or to start the drives later during the boot cycle, for example for a /home drive partition. Starting the drives during the later boot stages is much easier.

Late Boot Start Using systemd

Tip: In this section I am ignoring the root filesystem flashcache, as this method is not appropriate for it

In this method, you will create and activate a systemd script to load the flashcache module and mount the flashcache drives.

First, edit /etc/fstab and make sure that /home is being mounted using its UUID. If it is not, then get the UUID from /dev/disk/by-uuid. Flashcache will give the flashcache drive the same UUID as the parent, so this method will allow booting even if the flashcache drive could not be loaded. The relevant line should look something like: 

...
UUID=5ebee55d-8871-44ea-b159-58c103970f54   /home               ext4        rw,relatime
...

Next, create an init script to actually do the work.

/lib/initcpio/hooks/flashcache: 

#!/bin/bash

# Check if /home is mounted
HOMEMOUNT=$(mount | grep home | sed 's/ .*//' | sed 's#.*/##' )

# Load module if not already loaded
if [[ ! "$(eval lsmod | grep "^flashcache")" ]]; then
  modprobe flashcache
fi

# Mount flashcache /home if not already mounted
if [[ ! "$HOMEMOUNT" == "cached_part2" ]]; then
  umount /home
  if [ ! -e /dev/mapper/cached_part2 ]; then
    flashcache_load /dev/sdb2
  fi  
  mount /home
fi

Now create the systemd script that will use that init script. /lib/systemd/system/flashcache.service 

[Unit]
Description=FlashCache

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/usr/local/bin/flashcache

[Install]
WantedBy=multi-user.target

Finally, you can enable flashcache.service

That should be it!

Early Boot Start with initcpio

This method requires that you edit the kernel's initial ramdisk to load flashcache at early boot time.

Setting up the ramdisk

If you are setting up a cache that uses your root partition or you do not want to unmount your partitions while running, the best option is to change your ramdisk to support flashcache and do your administration with busybox. If you have unmounted filesystems that you are trying to cache, skip down to #Other file changes.

LVM is used as a base for the ramdisk changes, but it is very possible that there is a better way to do this. First, create (be sure to update to match your system) 

/lib/initcpio/hooks/flashcache
# vim:set ft=sh:
run_hook ()
{
    /sbin/modprobe -q dm-mod >/dev/null 2>&1
    if [ -e "/sys/class/misc/device-mapper" ]; then
        if [ ! -e "/dev/mapper/control" ]; then
            /bin/mknod "/dev/mapper/control" c $(cat /sys/class/misc/device-mapper/dev | sed 's|:| |')
        fi

        [ "${quiet}" = "y" ] && LVMQUIET=">/dev/null"

        msg "Activating cache volumes..."
        eval /usr/sbin/flashcache_load cached_part1 /dev/sdb1 /dev/sda3 $LMQUIET
        eval /usr/sbin/flashcache_load cached_part2 /dev/sdb2 /dev/sda4 $LMQUIET
    fi
}

/lib/initcpio/install
# vim: set ft=sh:

install ()
{
    MODULES="dm-mod"
    BINARIES=""
    FILES=""
    SCRIPT="flashcache"

    add_dir "/dev/mapper"
    add_binary "/sbin/dmsetup"
    add_binary "/usr/sbin/flashcache_create"
    add_binary "/usr/sbin/flashcache_load"
    add_binary "/usr/sbin/flashcache_destroy"
    add_file "/lib/udev/rules.d/10-dm.rules"
    add_file "/lib/udev/rules.d/13-dm-disk.rules"
    add_file "/lib/udev/rules.d/95-dm-notify.rules"
    add_file "/lib/udev/rules.d/11-dm-lvm.rules"
}

help ()
{
cat<<HELPEOF
  This hook loads the necessary modules for a flash drive as a cache device for your root device.
HELPEOF
}
Note: The files dm-disk.rules, dm-notify.rules and dm-lvm.rules are left in, but they might not be required.

Update your /etc/mkinitcpio.conf:

Add flashcache to your modules 

# MODULES="flashcache"

Add flashcache to your hooks(be sure to add it before filesystems) 

# HOOKS="... flashcache ..."
Tip: You might need to add usbinput to your hooks as well if you are using a usb keyboard.

Now rebuild your ramdisk image: 

# mkinitcpio -g /boot/<your ramdisk filename>.img

You might want to make this a different ramdisk image than your normal one and create a new entry in your grub configuration to use this ramdisk.

Other file changes

The new partitions will show up in /dev/mapper and you need to mount these partitions instead of the original /dev/sd* partitons. You need to edit grub and your fstab to mount these new partitions. In this example, the root partition is /dev/mapper/cached_part1 and the home directory is /dev/mapper/cached_part2.

/boot/grub/menu.lst 

# (0) Arch Linux
title  Arch Linux(flashcache)
root   (hd0,0)
kernel /vmlinuz26 root=/dev/mapper/cached_part1 ro 5
initrd /kernel-2.6.38.4.img

/etc/fstab 

...
/dev/mapper/cached_part1 / ext3 defaults 0 1
/dev/mapper/cached_part2 /home ext3 defaults,user_xattr 0 1
...

If you are caching your root partition, reboot and press e in grub to edit the kernel command line options. Select the line with the kernel options and press e again. Append 'break=y' without quotes to the end and press enter. Press b to boot. This tells the ramdisk image to stop after loading modules and drop into a shell to allow us to do some work.

Now reboot(make sure you use the flashcache ramdisk) and verify the mounted partitions are the newly created ones.

Tweaks

There are a lot of options for controlling the cache, check the system administrators guide in the documentation for flashcache for all of them. I added the following changes to /etc/sysctl.d/90-flashcache.conf: 

#####################
# flashcache settings
#####################

# disable writing dirty cache data at shutdown
dev.flashcache.fast_remove = 1

# change the reclaim policy to LRU from FIFO 
dev.flashcache.reclaim_policy = 1

# do not write "stale" data to disk until evicted due to lack of space
dev.flashcache.fallow_delay = 0

Troubleshooting

  • If you get an error trying to create the cache 'device-mapper: reload ioctl failed: Invalid argument', you could be trying to create a cache of a mounted filesystem.
  • If boot fails, an easy way to check on the cache is to edit your kernel command line in grub to add break=y and use the ramdisk's shell to poke around.
  • /proc/flashcache_stats has interesting information and can tell you if the cache is properly working.

See also