SystemD is blamed for long boot times and being heavy and bloated on resources. I tried OpenRC and Runit on real hardware (Ryzen 5000-series laptop) for week each and saw only 1 second faster boot time.

I’m old enough to remember plymouth.service (graphical image) being the most slowest service on boot in Ubuntu 16.04 and 18.04. But I don’t see that as an issue anymore. I don’t have a graphical systemD boot on my Arch but I installed Fedora Sericea and it actually boots faster than my Arch despite the plymouth (or whatever they call it nowadays).

My 2 questions:

  1. Is the current SystemD rant derived from years ago (while they’ve improved a lot)?
  2. Should Linux community rant about bigger problems such as Wayland related things not ready for current needs of normies?
  • Ok, so I have a very unique background in systemd. I worked at Red Hat supporting it basically as the primary support and I’ve worked with the developers of systemd at Red Hat directly. I no longer work there.

    So first off, it’s “systemd” all lower case. I don’t care, but for some reason Lennart Pottering (creator) does.

    systemd was a MASSIVE change. And Red Hat did a TERRIBLE job relaying it. To the point where I’m still trying to get my company to understand that it can NOT be treated like the old init systems. You can NOT just drop an init script in place and walk away and hope it works. Because a LOT of times it doesn’t. Due to forks, switch users, etc.

    systemd is NOT an init system. RHEL 5 and older had sysvinit as it’s init systemd. RHEL 6 had UpStart as it’s init system and looked exactly like sysvinit that no one even noticed. systemd again is NOT an init system. Init system is 1 part of systemd. systemd does a lot of cool things. It bundles applications together, it manages those applications and can restart them or kill children, it can do resource constraints, it separates out users from the system, and lots more.

    Because it is not an init system there is a LOT LOT LOT of bad recommendations out on the internet where someone has X problem and person suggests Y and IT WORKS! … except it doesn’t REALLY work as far as systemd is concerned and you’ll hit other issues or your application takes longer to start or stop and people just blame systemd.

    It is systemd’s fault that it has done an ATROCIOUS job of helping people adapt. It’s a great example of RTFM. systemd’s man pages are INCREDIBLE and extensive, but when you drop so much knowledge it becomes more difficult to find what you want/need. systemd.index and systemd.directives are your best bet.

    So systemd does a lot of amazing things that sysvinit never attempted to do. It’s never attempted to explain anything it expects everyone just learn magically. it’s INCREDIBLY complex, but once you understand it’s basics you can more easily get an application running, but as soon as there’s a problem it’ll just break your brain.

    To give you an example, sshd’s old init script is like 250 lines of bash. systemd’s unit file comparative is like 12. Because systemd handles a LOT of what you manually had to handle before. BUT to get to that 12 you literally have to learn EVERYTHING new.

    There is no “is it good or bad” here really imo. It’s a completely different fundamental design. Red Hat made it for themselves. Other distros picked it up. It can be argued that lots of folks followed Debian and Debian had a few Red Hat board members that were pushing it. Whether they pushed it of their own accord or because they were with Red Hat I don’t have a clue.

    What I can say is at my current company they’re suffering from a LOT of systemd issues and they don’t even realize it. I’ve been working with Red Hat to try to get Insights to alert people to the failures and we’re making progress.

    To see if you have issues just to start run the two following commands:

    # systemctl list-units --failed
    # systemd-cgls
    

    If you have any units that are failed, investigate those. If you don’t need them, disable them. As for the systemd-cgls this shows HOW systemd is grouping things. ANY application that runs as a service (or daemon or application or runs in the background or however you wanna say it) should be under system.slice. ONLY humans logging into the system (meat bags NOT applications switching to users) should be in user.slice. A LOT of times what happens is an old init script is dropped in place, they start it, it has a switch user and systemd assumes it’s a user and puts it into user.slice. systemd does NOT treat anything in user.slice the same as in system.slice and this WILL eventually cause problems.

    So again, is it good or bad? Eh. It does a lot of cool things, but they did a MASSIVE disservice to ALL of us by just expecting to relearn absolutely EVERYTHING.

    •  nyan   ( @nyan@lemmy.cafe ) 
      link
      fedilink
      English
      41 year ago

      sshd’s init script under OpenRC is 87 lines, of which around half are blanks, comments, closing braces, and other boilerplate. Granted, that still makes the real code maybe three times the size of your systemd unit file, but the difference isn’t as impressive as you’re making out.

      95% of people shouldn’t need to poke around in their init scripts or unit files anyway. If you actually need to do that, your use case is already somewhat unusual.

      • As an end user, unless you’re running a server, then no you shouldn’t have to mess with any of it.

        If you’re running a server or a sysadmin you absolutely 100% should be paying attention. Almost every single vendor I’ve seen selling their applications only have initscripts. Which then cause issues. I’ve gone to the vendors and told them and they’ve said go to Red Hat. Well Red Hat doesn’t support that vendor’s init scripts.

        Not naming an application, but it was from a BIG BLUE company and they said their only instructions are to call their script from the user. But it won’t remain running if you do that because systemd will close out the slice when the user logs out. SO it’s obvious they haven’t tried what they’re suggesting.

        And I’m not attempting to state that systemd is impressive in any way. systemd basically took what had been building over 40 years of init scripting and threw it out the window and said our way is better. I don’t think it is. I’m just saying, with a directive based unit file it’ll be simpler to parse than a bash script.

      • When I was training folks that were fresh on Linux they loved systemd more than init scripts. It really does make things easier to approach, but they’re so much more complex under the hood. With a bash script, pretty much what you see is what you get.

    • I’ll also add that systemd is so different and confusing that even Red Hat’s own training team teaches things wrong. If you take the new RHCSA on RHEL 8 they have a section where you have to create a container and have it running. systemd’s design indicates that this SHOULD be a system service so should create a unit file and set the User= to run it as a user. Instead they have you log in, run it as yourself, and then enable linger on loginctl to prevent your user from logging out. That is WRONG according to systemd’s design and since that’s outside the design there’s a chance in the future that something will break that. Or better yet, if your application is there running and you go to shutdown your system, systemd probably won’t stop it correctly and it could cause data corruption.

    • an old init script is dropped in place, they start it, it has a switch user and systemd assumes it’s a user and puts it into user.slice.

      Wait, what? As far as I know, if you drop a System V init script in its usual place, systemd will treat it as a service and place its processes in the control group for services, even if it calls fork and/or setuid. How did it end up in user.slice?

      • If you put an init script in /etc/rc.d/init.d/ systemd looks there, it has the systemd-sysv-generator which will look at the script, do it’s best guess to build a systemd unit file, and then start it via unit file. But it’s guessing at a LOT of things. Even if you have a LSB header in your init script, systemd’s ExecStart is just calling the init script. The problem will calling an init script is you’re almost certainly going to call another script. Well systemd thinks that ExecStart IS the parent. And since children can’t live without the parent, if the script finishes, systemd sees the script finish and it kils all the children, because children shouldn’t be around without a parent.

        Now systemd CAN guess a bit with a fork, but if you’re forking numerous times, init script calls X script calls Y script, it will NOT follow this. Hence why people decided that “RemainAfterExit=true” is the work around. That’s NOT. That’s NOT the answer. That is WRONG and systemd will NOT manage your application appropriately.

        systemd starting a service when User= is set sets the user during exec. There is no change user. So if you use a switch user that has a login, which most everyone do as su invokes login natively, then it falls through the pam stack, which include pam.systemd.so which tells systemd, ok this is a login which means it’s a meatbag move it over to the user.slice. So because it thinks it’s a human, ergo in the user.slice, it will NOT manage resources properly, it will NOT shut things down nicely, it will NOT clean up children, etc.

        systemd’s reality is you don’t use init scripts, you break your script down to the basic parts, you call your binary directly, you do everything via systemd’s directives. Do NOT let it guess at anything, be absolutely explicit.

        • Now systemd CAN guess a bit with a fork, but if you’re forking numerous times, init script calls X script calls Y script, it will NOT follow this.

          It follows it just fine. I have, for example, a Debian machine running the ISC DHCP server, which has only an LSB init script and no systemd service file. Starting it involves at least four forks: once to start the init script, once when the init script runs start-stop-daemon, once when start-stop-daemon runs dhcpd, and once when dhcpd forks itself into the background.

          Systemd correctly recognizes that it’s still running because, although the init script, start-stop-daemon, and the initial dhcpd process have all exited, there is still another process in its control group:

          $ systemctl status isc-dhcp-server
          ● isc-dhcp-server.service - LSB: DHCP server
               Loaded: loaded (/etc/init.d/isc-dhcp-server; generated)
               Active: active (running) since Fri 2023-06-02 23:28:47 PDT; 3 weeks 6 days ago
                 Docs: man:systemd-sysv-generator(8)
              Process: 1513 ExecStart=/etc/init.d/isc-dhcp-server start (code=exited, status=0/SUCCESS)
                Tasks: 1 (limit: 19074)
               Memory: 8.1M
                  CPU: 4.774s
               CGroup: /system.slice/isc-dhcp-server.service
                       └─1550 /usr/sbin/dhcpd -4 -q -cf /etc/dhcp/dhcpd.conf
          

          Similarly, starting Apache involves running apachectl, which then forks httpd, which then forks itself into the background, and if you’re using mpm_prefork, that background process forks itself a whole bunch more times too. Systemd has no problem with this:

          $ systemctl status apache2
          ● apache2.service - The Apache HTTP Server
               Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
               Active: active (running) since Fri 2023-06-02 23:28:43 PDT; 3 weeks 6 days ago
                 Docs: https://httpd.apache.org/docs/2.4/
              Process: 1456 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
              Process: 1144683 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
             Main PID: 1492 (apache2)
                Tasks: 15 (limit: 19074)
               Memory: 40.0M
                  CPU: 58.165s
               CGroup: /system.slice/apache2.service
                       ├─   1492 /usr/sbin/apache2 -k start
                       ├─1144695 /usr/sbin/apache2 -k start
                       ├─1144696 /usr/sbin/apache2 -k start
                       ├─1144697 /usr/sbin/apache2 -k start
                       ├─1144698 /usr/sbin/apache2 -k start
                       ├─1144699 /usr/sbin/apache2 -k start
                       ├─1144731 /usr/sbin/apache2 -k start
                       └─1155330 /usr/sbin/apache2 -k start
          

          That’s one of the neat things about systemd: fork all you want, and it’ll still keep track of which process belongs to which service.

          systemd’s reality is you don’t use init scripts, you break your script down to the basic parts, you call your binary directly, you do everything via systemd’s directives. Do NOT let it guess at anything, be absolutely explicit.

          Or just don’t write messy, incorrect init scripts that use su. Be sensible and use start-stop-daemon. That’s what it’s there for, and it’s been there since the '90s.

          • So just to state, your comment is coming off condescending, I don’t know if that was intentional or not. But saying something is “sensible” sounds rude. Everyone has different experiences and what you think is “sensible” or “common sense” someone else might have never heard of. I’ve never heard of a command called “start-stop-daemon”. I’ve only heard start/stop init scripts. Looking at the daemon, I’ve never seen anyone use this. Vast majority of scripts I’ve looked at were other companies and never saw a single one use it. So maybe start-stop-daemon would work, but I don’t have a clue. But when I talked to the developers of systemd and told them that people were falling into the user.slice because they were running a su in their init scripts they were all surprised as well, because they didn’t think ANYONE did that. Well I was a sysadmin for 5 years before I worked at Red Hat for 6 and that’s how everyone is taught to do it. So again, different experiences.

            The other part I want to address is you saying that those two examples shows I’m wrong. The first instance:

            $ systemctl status isc-dhcp-server
            ● isc-dhcp-server.service - LSB: DHCP server
                 Loaded: loaded (/etc/init.d/isc-dhcp-server; generated)
                 Active: active (running) since Fri 2023-06-02 23:28:47 PDT; 3 weeks 6 days ago
                   Docs: man:systemd-sysv-generator(8)
                Process: 1513 ExecStart=/etc/init.d/isc-dhcp-server start (code=exited, status=0/SUCCESS)
                  Tasks: 1 (limit: 19074)
                 Memory: 8.1M
                    CPU: 4.774s
                 CGroup: /system.slice/isc-dhcp-server.service
                         └─1550 /usr/sbin/dhcpd -4 -q -cf /etc/dhcp/dhcpd.conf
            

            There definitely seems to be something wrong there because sysetmd no longer is showing a Main Pid process which is the parent, so my guess is there’s something like a “RemainAfterExit=true” or something else there, because systemd isn’t showing a parent. And when there isn’t a parent, unless overridden in the unit file, systemd will kill the children.

            The second example:

            $ systemctl status apache2
            ● apache2.service - The Apache HTTP Server
                 Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
                 Active: active (running) since Fri 2023-06-02 23:28:43 PDT; 3 weeks 6 days ago
                   Docs: https://httpd.apache.org/docs/2.4/
                Process: 1456 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
                Process: 1144683 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
               Main PID: 1492 (apache2)
                  Tasks: 15 (limit: 19074)
                 Memory: 40.0M
                    CPU: 58.165s
                 CGroup: /system.slice/apache2.service
                         ├─   1492 /usr/sbin/apache2 -k start
                         ├─1144695 /usr/sbin/apache2 -k start
                         ├─1144696 /usr/sbin/apache2 -k start
                         ├─1144697 /usr/sbin/apache2 -k start
                         ├─1144698 /usr/sbin/apache2 -k start
                         ├─1144699 /usr/sbin/apache2 -k start
                         ├─1144731 /usr/sbin/apache2 -k start
                         └─1155330 /usr/sbin/apache2 -k start
            

            Shows you’re calling “/usr/sbin/apachectl” which then forks to apache and that stays as the main parent, you can see “Main PID” and then yes there are many children because they’re children of 1492. That is called a double fork. systemd can figure out the parent in a double fork, but once you fork and fork and fork, NOT counting CHILDREN but the actual PARENT, then systemd doesn’t know which to follow so usually ends up assuming the wrong PID to follow and will clean up everything when what it thinks is the parent goes away. But your second example is how systemd is supposed to work. You call an application that fork and the parent dies and lets the child take over. That’s a standard double fork, that’s what systemd expects. So that works as you expect, because you’ve called it as you expect. Wrap your /usr/sbin/apachectl in a script and then ExecStart=<script> and unless they’ve made massive improvements you’re gonna run into issues.

            Edit: fixed formatting on the systemctl status output

            • I’ve never heard of a command called “start-stop-daemon”. I’ve only heard start/stop init scripts. Looking at the daemon, I’ve never seen anyone use this.

              It’s not a daemon; it’s a command-line program that starts and stops daemons. It is specifically designed to be called from init scripts. It was originally developed for Debian in the '90s, and has been used in its init scripts ever since.

              I’m not familiar with non-Debian distributions, but I assume they have their own equivalents, if they aren’t using a full-blown service manager like systemd.

              Well I was a sysadmin for 5 years before I worked at Red Hat for 6 and that’s how everyone is taught to do it.

              An old pre-systemd CentOS machine I have access to uses some shell functions defined in /etc/init.d/functions to start and stop daemons. Those shell functions use the runuser program from the util-linux package to change user ID, not su.

              There definitely seems to be something wrong there because sysetmd no longer is showing a Main Pid process which is the parent, so my guess is there’s something like a “RemainAfterExit=true” or something else there, because systemd isn’t showing a parent.

              Per the systemd.service man page for RemainAfterExit: “Takes a boolean value that specifies whether the service shall be considered active even when all its processes exited. Defaults to no.” (Emphasis mine.)

              And when there isn’t a parent, unless overridden in the unit file, systemd will kill the children.

              systemd-sysv-generator generates Type=forking service units.

              Per the systemd.service man page for this type of service: “If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main service process, and the service manager will consider the unit started when the parent process exits. This is the behavior of traditional UNIX services.”

              It would be counterproductive for systemd to kill all of the processes belonging to such a service when the parent process exits, since the whole point of this type of service is that the parent process exits immediately.

              You might be thinking of how logind by default kills all processes belonging to a user session when the session ends. That’s different from services, though.

              Wrap your /usr/sbin/apachectl in a script and then ExecStart=<script> and unless they’ve made massive improvements you’re gonna run into issues.

              Works for me:

              $ systemctl cat apache2
              # /lib/systemd/system/apache2.service
              [Unit]
              Description=The Apache HTTP Server
              After=network.target remote-fs.target nss-lookup.target
              Documentation=https://httpd.apache.org/docs/2.4/
              
              [Service]
              Type=forking
              Environment=APACHE_STARTED_BY_SYSTEMD=true
              ExecStart=/usr/sbin/apachectl start
              ExecStop=/usr/sbin/apachectl graceful-stop
              ExecReload=/usr/sbin/apachectl graceful
              KillMode=mixed
              PrivateTmp=true
              Restart=on-abort
              OOMPolicy=continue
              
              [Install]
              WantedBy=multi-user.target
              
              # /etc/systemd/system/apache2.service.d/override.conf
              [Service]
              ExecStart=
              ExecStart=/usr/local/sbin/just-run-apachectl start
              ExecStop=
              

              $ cat /usr/local/sbin/just-run-apachectl 
              #!/bin/sh -e
              /usr/sbin/apachectl "$@"
              

              $ systemctl status apache2
              ● apache2.service - The Apache HTTP Server
                   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
                  Drop-In: /etc/systemd/system/apache2.service.d
                           └─override.conf
                   Active: active (running) since Fri 2023-06-30 21:42:38 PDT; 1min 30s ago
                     Docs: https://httpd.apache.org/docs/2.4/
                  Process: 1176549 ExecStart=/usr/local/sbin/just-run-apachectl start (code=exited, status=0/SUCCESS)
                 Main PID: 1176554 (apache2)
                    Tasks: 11 (limit: 19074)
                   Memory: 23.2M
                      CPU: 106ms
                   CGroup: /system.slice/apache2.service
                           ├─1176554 /usr/sbin/apache2 -k start
                           ├─1176555 /usr/sbin/apache2 -k start
                           ├─1176556 /usr/sbin/apache2 -k start
                           ├─1176558 /usr/sbin/apache2 -k start
                           ├─1176560 /usr/sbin/apache2 -k start
                           └─1176561 /usr/sbin/apache2 -k start
              

              (systemctl status also shows recent log entries, which I have omitted here.)

              Even with the extra fork, and even without a PidFile setting, it correctly identifies all of the processes belonging to this service, including the main process. And, even without an ExecStop setting telling it to use apachectl, it still correctly stops the service:

              $ systemctl status apache2
              ○ apache2.service - The Apache HTTP Server
                   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
                  Drop-In: /etc/systemd/system/apache2.service.d
                           └─override.conf
                   Active: inactive (dead) since Fri 2023-06-30 21:53:28 PDT; 2min 16s ago
                 Duration: 4.632s
                     Docs: https://httpd.apache.org/docs/2.4/
                  Process: 1177323 ExecStart=/usr/local/sbin/just-run-apachectl start (code=exited, status=0/SUCCESS)
                 Main PID: 1177328 (code=exited, status=0/SUCCESS)
                      CPU: 129ms
              

              (I have once again omitted the recent log entries from this output.)

              $ pgrep -l apache2
              $
              
              • runuser I do know and you’re correct it’s what is expected, and actually you CAN use su you just have to make sure you use the arguement that tells it not to follow a login process, but as a syaadmin myself, none of the guys I worked with, and none of the customers I had ever worked with hitting these issues had ever heard of it. But yes you can switch user without falling through the PAM stack and be fine.

                As for the fork maybe they resolved that or maybe they made it better because you only have 1 child, but I can tell you I wouldn’t trust systemd that way personally from my experience. And yes Type=forking also says you should have a PIDFile and that’s in case there are multiple forks you can be explicit in which pid is thr parent. There are other issues you can get into like not setting ulimits or if you have 1 script that starts multiple daemons.