Is Systemd that bad afterall?

Nuuskis ( @Nuuskis@sopuli.xyz ) · edit-2 1 year ago

Is Systemd that bad afterall?

!ozoned@lemmy.world ( @ozoned@beehaw.org ) · edit-2 1 year ago

So just to state, your comment is coming off condescending, I don’t know if that was intentional or not. But saying something is “sensible” sounds rude. Everyone has different experiences and what you think is “sensible” or “common sense” someone else might have never heard of. I’ve never heard of a command called “start-stop-daemon”. I’ve only heard start/stop init scripts. Looking at the daemon, I’ve never seen anyone use this. Vast majority of scripts I’ve looked at were other companies and never saw a single one use it. So maybe start-stop-daemon would work, but I don’t have a clue. But when I talked to the developers of systemd and told them that people were falling into the user.slice because they were running a su in their init scripts they were all surprised as well, because they didn’t think ANYONE did that. Well I was a sysadmin for 5 years before I worked at Red Hat for 6 and that’s how everyone is taught to do it. So again, different experiences.

The other part I want to address is you saying that those two examples shows I’m wrong. The first instance:

$ systemctl status isc-dhcp-server
● isc-dhcp-server.service - LSB: DHCP server
     Loaded: loaded (/etc/init.d/isc-dhcp-server; generated)
     Active: active (running) since Fri 2023-06-02 23:28:47 PDT; 3 weeks 6 days ago
       Docs: man:systemd-sysv-generator(8)
    Process: 1513 ExecStart=/etc/init.d/isc-dhcp-server start (code=exited, status=0/SUCCESS)
      Tasks: 1 (limit: 19074)
     Memory: 8.1M
        CPU: 4.774s
     CGroup: /system.slice/isc-dhcp-server.service
             └─1550 /usr/sbin/dhcpd -4 -q -cf /etc/dhcp/dhcpd.conf

There definitely seems to be something wrong there because sysetmd no longer is showing a Main Pid process which is the parent, so my guess is there’s something like a “RemainAfterExit=true” or something else there, because systemd isn’t showing a parent. And when there isn’t a parent, unless overridden in the unit file, systemd will kill the children.

The second example:

$ systemctl status apache2
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
     Active: active (running) since Fri 2023-06-02 23:28:43 PDT; 3 weeks 6 days ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 1456 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
    Process: 1144683 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
   Main PID: 1492 (apache2)
      Tasks: 15 (limit: 19074)
     Memory: 40.0M
        CPU: 58.165s
     CGroup: /system.slice/apache2.service
             ├─   1492 /usr/sbin/apache2 -k start
             ├─1144695 /usr/sbin/apache2 -k start
             ├─1144696 /usr/sbin/apache2 -k start
             ├─1144697 /usr/sbin/apache2 -k start
             ├─1144698 /usr/sbin/apache2 -k start
             ├─1144699 /usr/sbin/apache2 -k start
             ├─1144731 /usr/sbin/apache2 -k start
             └─1155330 /usr/sbin/apache2 -k start

Shows you’re calling “/usr/sbin/apachectl” which then forks to apache and that stays as the main parent, you can see “Main PID” and then yes there are many children because they’re children of 1492. That is called a double fork. systemd can figure out the parent in a double fork, but once you fork and fork and fork, NOT counting CHILDREN but the actual PARENT, then systemd doesn’t know which to follow so usually ends up assuming the wrong PID to follow and will clean up everything when what it thinks is the parent goes away. But your second example is how systemd is supposed to work. You call an application that fork and the parent dies and lets the child take over. That’s a standard double fork, that’s what systemd expects. So that works as you expect, because you’ve called it as you expect. Wrap your /usr/sbin/apachectl in a script and then ExecStart=<script> and unless they’ve made massive improvements you’re gonna run into issues.

Edit: fixed formatting on the systemctl status output

argv_minus_one ( @argv_minus_one@beehaw.org ) · 1 year ago

I’ve never heard of a command called “start-stop-daemon”. I’ve only heard start/stop init scripts. Looking at the daemon, I’ve never seen anyone use this.

It’s not a daemon; it’s a command-line program that starts and stops daemons. It is specifically designed to be called from init scripts. It was originally developed for Debian in the '90s, and has been used in its init scripts ever since.

I’m not familiar with non-Debian distributions, but I assume they have their own equivalents, if they aren’t using a full-blown service manager like systemd.

Well I was a sysadmin for 5 years before I worked at Red Hat for 6 and that’s how everyone is taught to do it.

An old pre-systemd CentOS machine I have access to uses some shell functions defined in /etc/init.d/functions to start and stop daemons. Those shell functions use the runuser program from the util-linux package to change user ID, not su.

There definitely seems to be something wrong there because sysetmd no longer is showing a Main Pid process which is the parent, so my guess is there’s something like a “RemainAfterExit=true” or something else there, because systemd isn’t showing a parent.

Per the systemd.service man page for RemainAfterExit: “Takes a boolean value that specifies whether the service shall be considered active even when all its processes exited. Defaults to no.” (Emphasis mine.)

And when there isn’t a parent, unless overridden in the unit file, systemd will kill the children.

systemd-sysv-generator generates Type=forking service units.

Per the systemd.service man page for this type of service: “If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main service process, and the service manager will consider the unit started when the parent process exits. This is the behavior of traditional UNIX services.”

It would be counterproductive for systemd to kill all of the processes belonging to such a service when the parent process exits, since the whole point of this type of service is that the parent process exits immediately.

You might be thinking of how logind by default kills all processes belonging to a user session when the session ends. That’s different from services, though.

Wrap your /usr/sbin/apachectl in a script and then ExecStart=<script> and unless they’ve made massive improvements you’re gonna run into issues.

Works for me:

$ systemctl cat apache2
# /lib/systemd/system/apache2.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Documentation=https://httpd.apache.org/docs/2.4/

[Service]
Type=forking
Environment=APACHE_STARTED_BY_SYSTEMD=true
ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl graceful-stop
ExecReload=/usr/sbin/apachectl graceful
KillMode=mixed
PrivateTmp=true
Restart=on-abort
OOMPolicy=continue

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/apache2.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/local/sbin/just-run-apachectl start
ExecStop=

$ cat /usr/local/sbin/just-run-apachectl 
#!/bin/sh -e
/usr/sbin/apachectl "$@"

$ systemctl status apache2
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/apache2.service.d
             └─override.conf
     Active: active (running) since Fri 2023-06-30 21:42:38 PDT; 1min 30s ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 1176549 ExecStart=/usr/local/sbin/just-run-apachectl start (code=exited, status=0/SUCCESS)
   Main PID: 1176554 (apache2)
      Tasks: 11 (limit: 19074)
     Memory: 23.2M
        CPU: 106ms
     CGroup: /system.slice/apache2.service
             ├─1176554 /usr/sbin/apache2 -k start
             ├─1176555 /usr/sbin/apache2 -k start
             ├─1176556 /usr/sbin/apache2 -k start
             ├─1176558 /usr/sbin/apache2 -k start
             ├─1176560 /usr/sbin/apache2 -k start
             └─1176561 /usr/sbin/apache2 -k start

(systemctl status also shows recent log entries, which I have omitted here.)

Even with the extra fork, and even without a PidFile setting, it correctly identifies all of the processes belonging to this service, including the main process. And, even without an ExecStop setting telling it to use apachectl, it still correctly stops the service:

$ systemctl status apache2
○ apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/apache2.service.d
             └─override.conf
     Active: inactive (dead) since Fri 2023-06-30 21:53:28 PDT; 2min 16s ago
   Duration: 4.632s
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 1177323 ExecStart=/usr/local/sbin/just-run-apachectl start (code=exited, status=0/SUCCESS)
   Main PID: 1177328 (code=exited, status=0/SUCCESS)
        CPU: 129ms

(I have once again omitted the recent log entries from this output.)

$ pgrep -l apache2
$

!ozoned@lemmy.world ( @ozoned@beehaw.org ) · 1 year ago

runuser I do know and you’re correct it’s what is expected, and actually you CAN use su you just have to make sure you use the arguement that tells it not to follow a login process, but as a syaadmin myself, none of the guys I worked with, and none of the customers I had ever worked with hitting these issues had ever heard of it. But yes you can switch user without falling through the PAM stack and be fine.

As for the fork maybe they resolved that or maybe they made it better because you only have 1 child, but I can tell you I wouldn’t trust systemd that way personally from my experience. And yes Type=forking also says you should have a PIDFile and that’s in case there are multiple forks you can be explicit in which pid is thr parent. There are other issues you can get into like not setting ulimits or if you have 1 script that starts multiple daemons.