cross-posted from: https://programming.dev/post/9319044

Hey,

I am planning to implement authenticated boot inspired from Pid Eins’ blog. I’ll be using pam mount for /home/user. I need to check integrity of all partitions.

I have been using luks+ext4 till now. I am hesistant hesitant to switch to zfs/btrfs, afraid I might fuck up. A while back I accidently purged ‘/’ trying out timeshift which was my fault.

Should I use zfs/btrfs for /home/user? As for root, I’m considering luks+(zfs/btrfs) to be restorable to blank state.

      • Yes it is great. For me snapper rollback was an awesome onboarding experience to linux. Being eager to try things I read online for tweaks and general explorarion it brought me back to a working system after some custom kernel compiling gone awry, or deleting the wrong file etc.

        • I’ve been on btrfs for so many years, with nightly backups with restic, so I’ve been dragging my feet on snapper. Finally installed it a couple weeks ago, and while I opened the config, I don’t think I changed anything. It’s worked so well, and the Arch package was so well done, that I’d forgotten I had it installed until a few days later I noticed that it was taking snapshots every time before I installed something. It’s shockingly good, and I don’t understand why btrfs+snapper(+grub-btrfs) isn’t the default on installs now.

  •  PenguinCoder   ( @Penguincoder@beehaw.org ) 
    link
    fedilink
    English
    11
    edit-2
    5 months ago

    As a home user I’d recommend btrfs. It has main line kernel support and is way easier to get operational than zfs. I’d you don’t need the more advance raid types of zfs or deduplication, btrfs can do everything you want. Also btrfs is a lot more resource friendly. Zfs, especially with deduplication, takes a ton of RAM.

  •  XTL   ( @XTL@sopuli.xyz ) 
    link
    fedilink
    9
    edit-2
    5 months ago

    My experiences:

    ZFS: never even tried because it’s not integrated (license).

    Btrfs: iirc I’ve tried it three times. Several years ago now. On at least two of those tries, after maybe a month or some of daily driving, suddenly the fs goes totally unresponsive and because it’s the entire system, could only reboot. FS is corrupted and won’t recover. There is no fsck. There is no recovery. Total data loss. Start again from last backup. Haven’t seen that since reiserfs around 2000. Found lots of posts with similar error message. Took btrfs off the list of things I’ll be using in production.

    I like both from a distance, but still use ext*. Never had total data loss that wasn’t a completely electrically dead drive with any version I’ve used since 1995.

    • Ouch, that must have been a pain to recover from…

      I’ve had almost the opposite experience to yours funnily. Several years ago my HDDs would drop out at random during heavy write loads, after a while I narrowed down the cause to some dodgy SATA power cables, which sadly I could not replace at the time. Due to the hardware issue I could not scrub the filesystem successfully either. However I managed to recover all my data to a separate BTRFS filesystem, using some “restore” utility that was mentioned in the docs, and to the best of my knowledge all the recovered data was intact.

      While that past error required a separate filesystem to perform the recovery, my most recent hardware issue with drives dropping out didn’t need any recovery at all - after resolving the hardware issue (a loose power connection) BTRFS pretty much fixed itself during a scheduled scrub and spat out all the repairs in dmesg.

      I would suggest enabling some kind of monitoring on BTRFS’s counters if you haven’t, because the fs will do whatever it can to prevent interruption to operations. In my previous two cases, performance was pretty much unaffected, and I only noticed the hardware problems due to the scheduled scrub & balance taking longer or failing.

      Don’t run a fsck - BTRFS essentially does this to itself during filesystem operations, such as a scrub or a file read. The provided btrfs check tool (fsck) is for the internal B-tree structure specifically AFAIK, and irreversably modifies the filesystem internally in a way that can cause unrecoverable data loss if the user does not know what they are doing. Instead of running fsck, run a scrub - it’s an online operation that can be done while the filesystem is still mounted

      • DO NOT RUN A SCRUB IF YOU SUSPECT HARDWARE FAILURE.

        No seriously. If you are having hardware issues a scrub could make the corruption much worse. You should first make a complete copy of your data and then run btrfs check. Sorry for shouting but it is really important you don’t stub a bad disk.

        •  BCsven   ( @BCsven@lemmy.ca ) 
          link
          fedilink
          5
          edit-2
          5 months ago

          That is a caveat with OS disk tools. Even partition resizing gives this warning, as does Windows checkdisk…something about unnessary disk checks ahould be avoided as they can create issues where none might have existed, so only run when you suspect a problem.

          But as lemann pointed out in this thread btrfs scrub is less risky

    • Btrfs has come a long way in the last few years. I have been using it for a little over 5 years and its rock solid. It now powers all my bare metal machines and I use Raid 1 on my servers.

      There was one time I had a disk unexpectedly go bad (it started returning bad data on read) which lead to the system going read only. It took me about 5min to swap disks and it was fine. Needless to say I was impressed that no data was lost.

      Btrfs will normally won’t get corrupted unless you have a hardware issue. It uses cow so writes can never be half competed. If you do manage to get corruption you can use btrfs check.

  • @unhinge I run a simple 48TiB zpool, and I found it easier to set up than many suggest and trivial to work with. I don’t do anything funky with it though, outside of some playing with snapshots and send/receive when I first built it.

    I think I recall reading about some nuance around using LUKS vs ZFS’s own encryption back then. Might be worth having a read around comparing them for your use case.

  • I did my first BTRFS setup over the weekend. I followed the Arch wiki to set up what I thought was RAID 1 only to find out nearly a TB of copying later that it was splitting the data between the drives, not mirroring them (only the metadata was in R1.) One command later and I’d converted the filesystem to true RAID 1. I feel like any other system would require a total redo of the entire FS, but BTRFS did it flawlessly.

    I’m still confused, however, as it seems RAID 1 only works with two drives from what I’ve read. Is that true? Why?

    •  The Doctor   ( @drwho@beehaw.org ) 
      link
      fedilink
      English
      85 months ago

      That is not the case. In the context of btrfs, RAID-1 means “ensure that two copies of every data block are available in the running volume,” not “ensure that every bit of both of these drives is identical at all times.” For example, I have a btrfs volume in my server with six drives in it (14 TB each) set up as a RAID-1/1 (both data and metadata are mirrored). It doesn’t really matter which two drives of the six have copies of a given data block, only that two copies exist at all.

      Compare it to… three RAID-1 metadevices (mdadm), with LVM over top, and ext4 (let’s say) on top of that. When a file is created in the filesystem (ext4), LVM ensures that it doesn’t matter on which pair of drives it was written, and mdadm’s RAID-1 functionality ensures that there are always two identical copies of the file (on two identical copies of a drive).

  • At some, long ago, the Ubuntu installer was offering to use zfs for the boot and root partitions. That sounded like a good idea and worked great for a long time, automatic snapshots, options to restore state at boot etc.

    Until my generous boot partition started to run out if space with all the snapshots (which were setup automatically and no obvious way to configure) OK no big deal, write a bash script that finds the old snapshots and delete them manually whenever boot is full again.

    Then one day recently my laptop wouldn’t boot anymore, Grub could no longer read the zfs on boot. Managed to boot with USB installation image, read zsf and chroot. Tried alot of things but in the end killed zfs and replace with ext4. Then made it boot again.

    Apparently I’m not the only one with this issue.

  •  floofloof   ( @floofloof@lemmy.ca ) 
    link
    fedilink
    English
    4
    edit-2
    5 months ago

    I haven’t used them professionally but I’ve been using ZFS on my home router (OPNsense) and NAS (TrueNAS with RAID-Z2) for many years without problem. I’ve used Btrfs on laptops and desktops with OpenSUSE Tumbleweed for the past year and a bit, also without problem. Btrfs snapshots have saved me a couple of times when I messed something up. Both seem like solid filesystems for everyday use.

  • Many many years ago I set up btrfs for the disks I write my backups to with a raid 1 config for them. Unfortunately one of those disks went bad and ended up corrupting the whole array. Makes me wonder if I set it up correctly or not.

    Nowadays, I have the following disks in my system set up as btrfs:

    • My backups disk because of compression.
    • My OS drive because of Timeshift.
    • My home folder because it feels safer. COW feels like it’ll handle power failures better, whilst there’s also checksumming so I can identify corrupted files.
    • My SSD Steam library over two drives because life is short and I cba managing the two ssds independently.

    It’s going fine, but it feels like I need to manually run a balance every one in a while when the disk fills up.

    I also like btrfs-assistant for managing the devices.

    Out of interest, since I’ve not used the “recommended partion setup” for any install for a while now, is ext4 still the default on most distros?

  •  Ramin Honary   ( @Ramin_HAL9001@lemmy.ml ) 
    link
    fedilink
    English
    2
    edit-2
    5 months ago

    Linux does not support ZFS as well as operating systems like OpenBSD or OpenIndiana, but I do use it on my Ubuntu box for my backup array. It is not the best setup: RAID-Z over USB is not at all guaranteed to keep your data safe, but it was the most economical thing I was able to build myself, and it gets the job done well enough with regular scrubbing to give me piece of mind about at least having one other reliable copy of my data. And I can write files to it quickly, and take snapshots of the state of the filesystem if need be.

    I used to use Btrfs on my laptop and it worked just fine, but I did have trouble once when I ran out of disk space. A Btrfs filesystem puts itself into read-only mode when that happens, and that makes it tough to delete files to free-up space. There is a magic incantation that can restore read-write functionality, but I never learned what it was, I just decided to stop using it because Btrfs is pretty clearly not for home PC use. Freezing the filesystem in read-only mode makes sense in a data-center scenario, but not for a home user who might want to try to erase data so one can keep using it normally. I might consider using Btrfs in place of ZFS on a file server, though ZFS does seem to provide more features and seems to be somewhat better tested and hardened.

    There is also BCacheFS now as an alternative to Btrfs, but it is still fairly new, and not widely supported by default installations. I don’t know how stable it is or how well it compares to Btrfs, but I thought I would mention it.

  •  The Doctor   ( @drwho@beehaw.org ) 
    link
    fedilink
    English
    2
    edit-2
    5 months ago

    There’s no reason you couldn’t; btrfs is pretty stable.

    Edit: Going on five years of using btrfs on production servers (storing and processing data on a 24x7 basis).

  • I’ve been running ZFS in the form of FreeNAS/TrueNAS in production environments for the past 12 years or so. Started with around 5TB and currently have nearly 300TB across several servers. Mostly NFS nowadays, but have shared out SMB and iSCSI.

    No data loss. Drives have been easy to replace and re-silver. We have had a couple instances where a failing ZIL or L2ARC has crashed a storage server and taken storage offline, but removing/replacing the log device got us up and running without data loss.

    Btrfs I only have experience on home systems. It has reliably stored my data for several years now, but I’m about to put it to the test this weekend. I plan on adding 4x8TB disks to a 4TB mirror to turn it into a 20TB RAID10. Wish me luck!

  • I have had great luck with my users’ home directories on ZFS. No issues in years. Used to have issues, and on those days I was glad root was on ext3.

    I had issues with btrfs about 10 years ago. It is much better now.

    Both experiences with Linux.

    A different ZFS partition per user is really helpful for quota and migration.