Tuesday, September 4, 2012

To btrfs or not to btrfs?

Tux
With the new release of openSUSE 12.2, there is an important choice to be made when updating: to stay with the ext4 file system or make the leap ahead with the btrfs file system. While ext4 was a rather minor upgrade of ext3, which in its turn was an upgrade of ext2, btrfs is an entirely new file system. It was originally proposed in 2007 and offers a lot of new features. It is based on B-trees and offers copy-on-write.

Although btrfs is approximately five years old, it is still considered experimental. So why would one choose it as primary file system? Let us read the FAQ on the btrfs website:

Is btrfs stable?

Short answer: No, it's still considered experimental.
Long answer: Nobody is going to magically stick a label on the btrfs code and say "yes, this is now stable and bug-free". Different people have different concepts of stability: a home user who wants to keep their ripped CDs on it will have a different requirement for stability than a large financial institution running their trading system on it. If you are concerned about stability in commercial production use, you should test btrfs on a testbed system under production workloads to see if it will do what you want of it. In any case, you should join the mailing list (and hang out in IRC) and read through problem reports and follow them to their conclusion to give yourself a good idea of the types of issues that come up, and the degree to which they can be dealt with. Whatever you do, we recommend keeping good, tested, off-system (and off-site) backups.
Pragmatic, personal and anecdotal answer: (HugoMills, 2011-08-21) In the last few months, the vast majority of the problems with broken and unmountable file systems I've seen on IRC and the mailing list have been caused by power outages in the middle of a write to the FS, and have been trivially fixable. We also have more filesystem-fixing tools coming along soon, which may make you happier about stability.

This does not sound too bad and after five years of development, it should be fairly stable. This was also what I read on the Internet, so, in the last couple of weeks, I decided to give btrfs a try in a virtual machine running openSUSE 12.2 release candidates. Installation went smoothly and afterwards I ran some tests, creating and deleting subvolumes, reverting an update with snapper, etc. I also shut down the virtual machine abruptly several times, simulating power failures, while writing data. Whatever I tried, the file system remained stable. Even when it would go wrong, there is now a file system check too, btrfsck, that can detect and fix some errors and a recovery tool to retrieve data from a damaged btrfs file system.

Besides that, you have to keep good backups anyway, Even when you have the most stable file system, hard drives can fail, users can accidentally destroy data, etc.

I searched the net for forums, websites, bug reports for more information. This is the overview of my findings:

Advantages:

  • Subvolumes and snapshotting
    • More stable backups: you can take a snapshot and backup that. Even when users work while the backup software is running, the backup will end up in a consistent state.
    • Incremental backups using e.g. rsync and snapshotting.
    • Rollback to an older version is easy and fast, especially in openSUSE with the snapper tool. Ideal for testing newer versions of software and going back when there are issues.
  •  Checksumming of data and metadata
  • Multiple device support
    •  File Striping, File Mirroring, and File Striping+Mirroring implementations 
  • Prepared for performance
    • Online defragmentation
    •  SSD awareness, discard support and several optimizations

Disadvantages:

  •  As mentioned already, still marked experimental in the Linux kernel
    • Although the interfaces and file system format are said to be stable, there will still be many changes under the hood.
  • Quota support will only arrive with Linux 3.6 and has to be tested then
  • For Linux 3.4, Phoronix concludes that for most workloads ext4 remains well ahead of btrfs.
    • So while the developers have thought of performance, it is still not completely there yet.
When looking at this, I have decided to give btrfs a go. Since I will be using Tumbleweed again, starting from openSUSE 12.2, I hope to see more improvements and features of btrfs soon. I will keep you posted on my findings. If you feel like trying it out yourself, know that you can also convert an existing ext2, ext3 or ext4 filesystem to btrfs. As always, make sure you make backups before fiddling with your file systems :-)

Do you have any experience with btrfs or comments you would like to share? Please let me know!



7 comments:

  1. Did you find it was any faster or slower than using ext4?

    ReplyDelete
    Replies
    1. In the tests I did, I did not really notice any difference, but the tests were limited w.r.t. performance and were conducted in a virtual machine. I will come back on this when I install openSUSE 12.2 on my laptop and test it 'for real'. However, it seems unlikely to me that you would notice a difference under a normal workload on a laptop.

      Delete
  2. I have a 12.1 system setup right now with an EXT4 boot volume and a BTRFS system volume (root, home, and an extra storage volume) and I experienced one of those power failure during write events. The system will boot but I'm not having any luck repairing the BRTFS volumes. I'll probably end up using ext4 now for /home until I can be certain an fsck will actually fix the filesystems problems.
    I just downloaded 12.2 and will see what's there.
    Decent article overall, just wanted to make sure that people have a clear picture of what they're getting into. Definitely have some sort of backup solution, which is good advice with any filesystem...

    ReplyDelete
    Replies
    1. I know that there have been issues with btrfs on openSUSE 12.1 or any Linux distribution with a kernel < 3.4 for that matter, but as far as I have read, these issues should be solved by now. I have installed openSUSE 12.2 yesterday, so it is too early to tell anything yet, I'll keep you posted :-)

      Delete
  3. I am running openSUSE 12.1 and 12.2 on our servers used as xen hosts. Root file system is on ext4, but xen images are on btrfs file system, so I can make easily snapshot of whole group of virtual machines. btrfs runs just fine, I had only one failure in 4 months, data was recovered from backup, so no big deal. However what really concerns me is performance. When I start at same time several xen domains (VMs) performance is REALLY bad. I tested difference between openSUSE 12.1 and 12.2, but almost no performance improvement here.
    I also tested btrfs with compression on, but again no significant performace improvement. It looks like accessing btrfs filer system from multiple threads at same time is a problem.

    ReplyDelete
  4. Any more benchmarks between Ext3/4 vs Btrfs ?. We planned to change our file systems to Ext4 for performance improvement but after conversion, we got to know that SuSe no longer supports Ext4 in near future.

    In performance wise like multiple threads writing to BTRFS is it going to show the same performance as in EXT4 ? could anyone please comment on the same

    ReplyDelete
  5. Also, if you do not delete old snapshots and do a frequent "zypper dup", you will run out of space soon. AFAIK btrfs does not have a command to delete more snapshot at a time, so you must delete one by one or make s script. Moreover btrfs performance drops _a lot_ if it is low on free space. So i wouldn't recommend it for users that rely on speed/performance. I'm not saying btrfs is bad, but it has it's pros/cons. Lets hope that performance will be improved soon. Cheers!

    ReplyDelete