QEMU/KVM + virtio-fs - Sharing a host directory with a virtual machine
Before virtio-fs if someone wanted to share files between a virtual machine (VM) and the host the VM runs on there weren’t too much options that worked very well performance-wise. Among other options that were mostly SMB-Share (Samba), NFS or virto-9p. As already mentioned they all have in common that they are not that fast. One reason is that they are all depended on some network related stuff which of course adds overhead that costs performance.
virtio-fs on the other side is designed to offer local file system semantics and performance. virtio-fs takes advantage of the virtual machine’s co-location with the hypervisor to avoid overheads associated with network file systems. virtio-fs uses FUSE as the foundation. Unlike traditional FUSE where the file system daemon runs in userspace, the virtio-fs daemon runs on the host. A VIRTIO device carries FUSE messages and provides extensions for advanced features not available in traditional FUSE.
The prerequisites to use virtio-fs are:
- The guest VM must run a Linux kernel >= 5.4 (which is true for Ubuntu 20.04 e.g. - also Ubuntu 20.04 supports kernel 5.8 meanwhile)
- On the host QEMU 5.0 and libvirt 6.2 must be installed. At least Archlinux supports this requirements. For other distributions please check with your package manager which versions are available.
So lets start with the directory that should be shared on the host. The presentation
A Shared File System for Virtual Machines] (PDF / slides) mentions a few security best practices that should be taken into consideration:
- Guests have full
gidaccess to shared directory
- Guests have no access outside shared directory
- Use dedicated file system for shared directory to prevent inode exhaustion or other Denial-of-Service attacks
- Parent directory of shared directory should have
rwx------permissions to prevent non-owners from accessing untrusted files
- Mount shared directory
It’s perfectly possible to share e.g.
/srv/http which should be available on most Linux installations. This directory is normally located at the root (
/) filesystem. So if you share this directory with the virtual machine it would be possible to create lots of files which could result in running out of inodes on the host root partition. So it depends on your needs if that is an issue you want to avoid because of production requirements or if you just run a VM on your private laptop e.g.
I want to share three host directories with three VMs. To adhere the best practices I decided to create three partitions on a disk that had free space left e.g.:
cfdisk /dev/sda sda 8:0 0 931.5G 0 disk ├─sda1 8:1 0 100G 0 part ├─sda2 8:2 0 100G 0 part └─sda3 8:3 0 100G 0 part
sda(1-3) can be formatted e.g. with a
ext4 filesystem (
-m 1 specifies the percentage of the filesystem blocks reserved for the super-user. In my case one percent is good enough. Additionally I also put a label (
-L) on it which can be used later to mount the disk accordingly:
sudo mkfs.ext4 -m 1 -L shared_vm1 /dev/sda1 sudo mkfs.ext4 -m 1 -L shared_vm2 /dev/sda2 sudo mkfs.ext4 -m 1 -L shared_vm3 /dev/sda3
Now I create a parent directory with
rwx------ permissions for all mountpoints as mentioned above in best practices. E.g.:
sudo mkdir /shared sudo chmod 700 /shared sudo mkdir /shared/vm1 sudo mkdir /shared/vm2 sudo mkdir /shared/vm3 sudo chown 700 /shared/vm1 sudo chown 700 /shared/vm2 sudo chown 700 /shared/vm3
To mount the new partitions I added the following entries in
LABEL=shared_vm1 /shared/vm1 ext4 rw,relatime,data=ordered,noatime,nosuid,nodev 0 2 LABEL=shared_vm2 /shared/vm2 ext4 rw,relatime,data=ordered,noatime,nosuid,nodev 0 2 LABEL=shared_vm3 /shared/vm3 ext4 rw,relatime,data=ordered,noatime,nosuid,nodev 0 2
Now the more interesting part on how to configure QEMU and finally share a host directory with a VM. The document libvirt: Sharing files with virtio-fs describes the process in more detail. Here is the path that I implemented.
QEMU needs to allocate the backing memory for all the guest RAM as shared memory. I have chosen to use
file-backed memory (the other option would be
hugepage-backed memory). This needs to be configured in
# NOTE: big files will be stored here memory_backing_dir = "/var/lib/libvirt/qemu/ram"
Make sure that the directory specified here have enough space left. For a VM that is configured to have 8 GB RAM you’ll see a file
libvirt/qemu/vm1/ram-node0 later e.g. that also has 8 GB file size (in this case for a VM called
vm1). This doesn’t mean that the whole 8 GB of that file will be allocated right from the start. So it’s perfectly possible that
du -sh /var/lib/libvirt/qemu/ram/ might report a way lower value of space used.
sudo systemctl restart libvirtd
Next I changed the configuration of the VM accordingly e.g.
sudo virsh edit vm1:
<domain> ... <cpu ...> <numa> <cell id='0' cpus='0-1' memory='8' unit='GiB' memAccess='shared'/> </numa> </cpu> ... </domain>
<cpu> element probably already exists. So just the
<numa> element is new in my case. The VM in question has 2 CPUs and 8 GB RAM. This numbers are also reflected for the
<cell> properties accordingly. In case of a 4 CPU VM the property value of
cpus would be
file-backed memory also a new element needs to be inserted at the same intention as the
<domain> ... <memoryBacking> <access mode='shared'/> </memoryBacking> ... </domain>
Finally within the
<devices> XML node a new
<filesystem> element needs to be inserted e.g.:
<domain> ... <devices> ... <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs'/> <source dir='/shared/vm1'/> <target dir='vm1'/> </filesystem> ... </devices> </domain>
passthrough is the only supported access mode at the moment (which requires to run
virtiofsd daemon on the host as
root for now - this daemon will be started automatically).
<source dir='/shared/vm1'/> specifies the host directory that should be shared with the VM.
<target dir='vm1'/> is a bit misleading. The value isn’t a directory but just a tag which will be specified when the filesystem is mounted later within the VM (see below).
Now the VM needs to be restarted e.g.:
sudo virsh shutdown vm1 sudo virsh start vm1
After the VM is up again the shared directory can be mounted:
sudo mount -t virtiofs vm1 /mnt
Or put the mount into
/etc/fstab to make it permanent:
vm1 /mnt virtiofs rw,_netdev 0 0
To mount immediately run
sudo mount -a.
Happy file sharing ;-)