Volume layout with root and home in top level not working

[ssendev]

On Fedora the default volume layout is

# btrfs subvolume list -t /
ID	gen	top level	path	
--	---	---------	----	
257	123693	5		root
258	123693	5		home
305	123659	257		var/lib/machines

snazzer --all --dry-run fails with ERROR: /root is not a btrfs subvolume and
snazzer --all --dry-run /home fails with /home/root is not a btrfs subvolume

mount /dev/sdb -o subvolid=5 /mnt and then
snazzer --all --dry-run /mnt will ignore ignores.
while snazzer --all --dry-run /mnt/root will fail with ERROR: /mnt/root is not a filesystem mountpoint same for /mnt/home

[florianjacob]

Thanks for trying snazzer on fedora. 👍 Could you post your /etc/fstab as well, please, and the output of findmnt?

I've a similar layout running on Arch Linux, so we should get this to work somehow. Btw, did you run snazzer --all --dry-run or snazzer --all --dry-run /?

@csirac2 the strange thing is that with my similar setup, I actually get no output to snazzer --all --dry-run. (I always run snazzer with specifying the target mountpoint.)

[ssendev]

#
# /etc/fstab
# Created by anaconda on Sat Oct 15 05:01:16 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=74a743dd-61f2-4773-b168-fc6dc6269d63 /                       btrfs   subvol=root,x-systemd.device-timeout=0,noatime,nodiratime 0 0
UUID=2bf80deb-68cf-461a-a647-0efa9952fe3a /boot                   ext4    defaults        1 2
UUID=B2BB-B996          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=74a743dd-61f2-4773-b168-fc6dc6269d63 /home                   btrfs   subvol=home,x-systemd.device-timeout=0,noatime,nodiratime 0 0
/dev/mapper/luks-71506272-8821-40b8-980c-5a8752268266 swap                    swap    defaults,x-systemd.device-timeout=0 0 0

TARGET                                SOURCE                                                                            FSTYPE          OPTIONS
/                                     /dev/mapper/luks-6d7b5391-2c54-46cc-81f4-065251e55fa3[/root]                      btrfs           rw,noatime,nodiratime,seclabel,ssd,space_cache,subvolid=257,subvol=/root
├─/sys                                sysfs                                                                             sysfs           rw,nosuid,nodev,noexec,relatime,seclabel
│ ├─/sys/kernel/security              securityfs                                                                        securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                    tmpfs                                                                             tmpfs           ro,nosuid,nodev,noexec,seclabel,mode=755
│ │ ├─/sys/fs/cgroup/systemd          cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
│ │ ├─/sys/fs/cgroup/memory           cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/freezer          cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,freezer
│ │ ├─/sys/fs/cgroup/cpuset           cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,cpuset
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
│ │ ├─/sys/fs/cgroup/hugetlb          cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,hugetlb
│ │ ├─/sys/fs/cgroup/pids             cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,pids
│ │ ├─/sys/fs/cgroup/blkio            cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/devices          cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/cpu,cpuacct      cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ └─/sys/fs/cgroup/perf_event       cgroup                                                                            cgroup          rw,nosuid,nodev,noexec,relatime,perf_event
│ ├─/sys/fs/pstore                    pstore                                                                            pstore          rw,nosuid,nodev,noexec,relatime,seclabel
│ ├─/sys/firmware/efi/efivars         efivarfs                                                                          efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/selinux                   selinuxfs                                                                         selinuxfs       rw,relatime
│ ├─/sys/kernel/debug                 debugfs                                                                           debugfs         rw,relatime,seclabel
│ ├─/sys/kernel/config                configfs                                                                          configfs        rw,relatime
│ └─/sys/fs/fuse/connections          fusectl                                                                           fusectl         rw,relatime
├─/proc                               proc                                                                              proc            rw,nosuid,nodev,noexec,relatime
│ ├─/proc/sys/fs/binfmt_misc          systemd-1                                                                         autofs          rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14067
│ └─/proc/fs/nfsd                     nfsd                                                                              nfsd            rw,relatime
├─/dev                                devtmpfs                                                                          devtmpfs        rw,nosuid,seclabel,size=4030340k,nr_inodes=1007585,mode=755
│ ├─/dev/shm                          tmpfs                                                                             tmpfs           rw,nosuid,nodev,seclabel
│ ├─/dev/pts                          devpts                                                                            devpts          rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
│ ├─/dev/hugepages                    hugetlbfs                                                                         hugetlbfs       rw,relatime,seclabel
│ └─/dev/mqueue                       mqueue                                                                            mqueue          rw,relatime,seclabel
├─/run                                tmpfs                                                                             tmpfs           rw,nosuid,nodev,seclabel,mode=755
│ ├─/run/user/42                      tmpfs                                                                             tmpfs           rw,nosuid,nodev,relatime,seclabel,size=808396k,mode=700,uid=42,gid=42
│ └─/run/user/1000                    tmpfs                                                                             tmpfs           rw,nosuid,nodev,relatime,seclabel,size=808396k,mode=700,uid=1000,gid=1000
│   └─/run/user/1000/gvfs             gvfsd-fuse                                                                        fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
├─/tmp                                tmpfs                                                                             tmpfs           rw,nosuid,nodev,seclabel
├─/home                               /dev/mapper/luks-6d7b5391-2c54-46cc-81f4-065251e55fa3[/home]                      btrfs           rw,noatime,nodiratime,seclabel,ssd,space_cache,subvolid=258,subvol=/home
├─/boot                               /dev/sda2                                                                         ext4            rw,relatime,seclabel,data=ordered
│ └─/boot/efi                         /dev/sda1                                                                         vfat            rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
├─/var/lib/nfs/rpc_pipefs             sunrpc                                                                            rpc_pipefs      rw,relatime
├─/mnt                                /dev/mapper/luks-6d7b5391-2c54-46cc-81f4-065251e55fa3                             btrfs           rw,relatime,seclabel,ssd,space_cache,subvolid=5,subvol=/
└─/var/lib/docker/btrfs               /dev/mapper/luks-6d7b5391-2c54-46cc-81f4-065251e55fa3[/root/var/lib/docker/btrfs] btrfs           rw,noatime,nodiratime,seclabel,ssd,space_cache,subvolid=257,subvol=/root/var/lib/docker/btrfs

I ran booth and they gave the same output

[florianjacob]

I'm confused. What I use to circumvent the top level subvolumes issue is to mount my disk via fstab to /run/systemfs, and run snazzer on that mountpoint, so exactly what you tried with your /mnt experiment.

Hopefully @csirac2 can shed some light on what's actually happening there next weekend, and why root is always appended…

To complete the view, you could post df -t btrfs, maybe luks encryption is interfering with that somehow? I don't use luks encryption, which might be the difference between our systems.

[ssendev]

Using /mnt works-ish i modified exclude.patterns var/cache to root/var/cache and now taking snapshots seems to work.

But then snazzer-receive fails

snazzer-receive host --all /mnt/systemfs
subvolume mnt/systemfs:
mkdir: created directory 'mnt'
Create subvolume 'mnt/systemfs'
mkdir: created directory 'mnt/systemfs/.snapshotz'
At subvol 2016-11-27T235638+0100
 1 of 1 snapshots received (0 pruned, 1 considered, 0 skipped)
  appending measurements...
ERROR running:
  ssh "host" "sudo -n grep -srl '^> on host at ' '/mnt/systemfs/.snapshotz/.measurements/'"

as does snazzer-measure

snazzer-measure /mnt/systemfs                     
################################################################################
> on host at 2016-11-28T002058+0100, du bytes:
(du -bs --one-file-system --exclude-from '/mnt/systemfs/.snapshot_measurements.exclude' '/mnt/systemfs')
cat: /mnt/systemfs/.snapshot_measurements.exclude: No such file or directory
80	/mnt/systemfs

tar: /mnt/systemfs/.snapshot_measurements.exclude: No such file or directory
tar: Error is not recoverable: exiting now

list-snapshots seems to work

snazzer --list-snapshots --all /mnt/systemfs
/mnt/systemfs/.snapshotz/2016-11-27T235638+0100
/mnt/systemfs/root/.snapshotz/2016-11-27T235638+0100
/mnt/systemfs/home/.snapshotz/2016-11-27T235638+0100
/mnt/systemfs/root/var/lib/machines/.snapshotz/2016-11-27T235638+0100

42 subvolumes excluded in /mnt/systemfs by /etc/snazzer/exclude.patterns.

root seems to be appended because it is the first subvolume reported by btrfs subvolume list

# sudo btrfs subvolume list /
ID 257 gen 124374 top level 5 path root
ID 258 gen 124374 top level 5 path home
ID 305 gen 124302 top level 257 path var/lib/machines
ID 398 gen 124295 top level 5 path .snapshotz/2016-11-27T235638+0100
ID 399 gen 124298 top level 257 path .snapshotz/2016-11-27T235638+0100
ID 400 gen 124300 top level 258 path home/.snapshotz/2016-11-27T235638+0100
ID 401 gen 124301 top level 305 path var/lib/machines/.snapshotz/2016-11-27T235638+0100

and is even when /home is specified

btrfs subvolume list /home
ID 257 gen 124377 top level 5 path root
ID 258 gen 124377 top level 5 path home
ID 305 gen 124302 top level 257 path root/var/lib/machines
ID 398 gen 124295 top level 5 path .snapshotz/2016-11-27T235638+0100
ID 399 gen 124298 top level 257 path root/.snapshotz/2016-11-27T235638+0100
ID 400 gen 124300 top level 258 path .snapshotz/2016-11-27T235638+0100
ID 401 gen 124301 top level 305 path root/var/lib/machines/.snapshotz/2016-11-27T235638+0100

btrfs subvolume list /home -o returns only actual subvolumes but this causes btrfs subvolume list / -o to not include /home which would probably make --all not work as intended

df -t btrfs
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/dm-0      236752896 48858680 187165944  21% /
/dev/dm-0      236752896 48858680 187165944  21% /home
/dev/dm-0      236752896 48858680 187165944  21% /mnt/systemfs

[florianjacob]

Research results: btrfs subvolume list <path> lists all subvolumes on 's filesystem regardless of which part of the filesystem points to. Only -o reduces that to subvolumes below , but only regarding to the actual subvolume structure on the volume, not the structure realized through mounting.

[csirac2]

Thanks for the contributions - yes indeed, you have discovered that I haven't been testing snazzer with this layout. Now that snazzer appears to have people trying it, I will work on improving the testing :)

Addressing this requires some thought - it seems like a regression I thought I had already fixed. Will work on this once the other two PRs are done. I have some ansible automation I will need to bring into our tests somehow. Thanks again

[sl6xx]

I run into a similar situation. OS is CentOS 7.3. I created a btrfs filesystem named tank. Then I created two subvolumes under tank for var and home. My /etc/fstab is as below:

[root@localhost /]# cat /etc/fstab

UUID=24ac7a7f-4a8b-4a56-814d-f4195d2f5690 / ext4 defaults 1 1
LABEL=tank /tank btrfs defaults 0 0
LABEL=tank /var btrfs defaults,subvolid=258 0 0
LABEL=tank /home btrfs defaults,subvolid=259 0 0
UUID=021b1b78-4b80-44af-b11a-389a62d464ca swap swap defaults 0 0

[root@localhost /]# btrfs subvolume list -t /tank
ID gen top level path
-- --- --------- ----
258 453 5 var
259 455 5 home

[root@localhost /]# snazzer --all --dry-run /home
cat <<EXCL | tee '/home/.snapshot_measurements.exclude' >/dev/null

EXCL
btrfs subvolume snapshot -r '/home' '/home/.snapshotz/2017-05-24T015701+0800'
rm '/home/.snapshot_measurements.exclude'
ERROR: /home/var is not a btrfs subvolume

[root@localhost /]# snazzer --all --dry-run /var
cat <<EXCL | tee '/var/.snapshot_measurements.exclude' >/dev/null

EXCL
btrfs subvolume snapshot -r '/var' '/var/.snapshotz/2017-05-24T015857+0800'
rm '/var/.snapshot_measurements.exclude'
ERROR: /var/var is not a btrfs subvolume