Out the box Linux Mint installs without btrfs support. The one big advantage of btrfs is when running a VM it allows for compression and that reduces the about of IO. Storage is almost always the bottleneck for VMs so this is an important factor. It's also useful if you use it on a USB flash drive (slow) as a diagnostic tool.

This assumes that you have a separate /boot mountpoint with an easily readable filesystem like ext4. That ensures you can always get up and running.

So, here's how to do this.

Initial Install

Do your usual install. If you want LVM then you either need to choose that option or manually create the LVs as you want before install.

Once you are up and running, then we can start the conversion. Before we go any further ensure that btrfs-tools are installed:

# apt-get install btrfs-tools

At this point you might light to snapshot the VM.

Conversion

Boot again from the LiveCD again. Once up, install btrfs-tools as above. This is only an install in the LiveCD environment and needed for the conversion.

Next we can run the conversion - you need to work out what your root device is. In my case it's an LV:

# btrfs-convert /dev/vg00/root
creating btrfs metadata.
creating ext2fs image file.
cleaning up system chunk.
conversion complete.

This will sit and thrash the disk for a while as it converts the filesystems. Now we need to get this back to a bootable state.

Mount the filesystem where you can get to it, plus the other virtual filesystems:

# mount /dev/vg00/root /mnt/
# mount /dev/ /mnt/dev/ -o bind
# mount /proc/ /mnt/proc/ -o bind
# mount /sys/ /mnt/sys -o bind

Then chroot into this environment and mount the /boot mountpoint:

# chroot /mnt/
# mount /boot

Next edit /etc/fstab and change the filesystem type to btrfs and set compression:

/dev/mapper/vg00-root /               btrfs    space_cache,compress=zlib 0       1

Additionally, btrfs has an automatic defragmentation feature which might be useful if you are using magnetic storage. You can add the autodefrag option to the mountpoint options above.

At this point everything should be fine, except preparations for boot. For this we need to update the initramfs and grub:

# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.13.0-24-generic
Warning: No support for locale: en_GB.utf8
# update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-3.13.0-24-generic
Found initrd image: /boot/initrd.img-3.13.0-24-generic
Found memtest86+ image: /memtest86+.elf
Found memtest86+ image: /memtest86+.bin
done

And then reboot.....

With a bit of luck it will come up fine else you will need to debug that or roll back the snapshot... you did take a snapshot?

Cleanup and Compress

So far we've got up and running, but conversion creates a subvolume containing the old ext4 filesystem and if everything is fine then that's no longer needed. To remove that:

# btrfs subvolume delete /ext2_saved/
Delete subvolume '//ext2_saved'

Now we are left with a quite full filesystem which is not compressed, but has compression enabled for new writes:

# df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg00-root  6.0G  4.0G  1.7G  70% /

What we need to do now is compress the existing data to reduce space. This is a little bit misleading sometimes due to the imprecise way btrfs calculates space, but goes something like this:

# btrfs filesystem defragment -vrc /

That should run a while compressing all the existing files and show you what it's up to. This generates loads of IO so expect the disk to thrash as it rewrites everything.

After this you should have considerably less space used:

# df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg00-root  6.0G  1.9G  2.4G  45% /

From here you can remove any snapshots created for safety and continue to use it as normal.