Good day to all! Over the last 30 minutes or so, I’ve been having issues loading beehaw.org. Sometimes CSS is missing and the page layout is broken, and others there is a server side NGINX error.

Just wanted to make the admins aware this is happening. There are some NGINX settings that can be adjusted to make more threads available to NGINX if it is hitting a worker limit.

      •  ffmike   ( @ffmike@beehaw.org ) 
        link
        fedilink
        English
        41 year ago

        Yes, Beehaw is struggling with uptime. From talking with the admins, this really isn’t an nginx issue. It’s more that the Lemmy code itself is immature, with memory leaks and SQL performance issues, and those issues are becoming more disruptive as the usage explodes.

        If you’ve got development skills, helping out the Lemmy project on Github is probably the best way to help. If not, then just press F5 with the rest of us when the site goes down for a bit.

        •  BitOneZero   ( @BitOneZero@beehaw.org ) 
          link
          fedilink
          English
          1
          edit-2
          1 year ago

          If you’ve got development skills, helping out the Lemmy project on Github is probably the best way to help.

          I have been, I’m RocketDerp on Github. I’ve been watching for weeks how none of the people running the major sites have opened an issue on observable problems, so I have done so myself:

          Major data integrity issues ignored since June 14 issue opened: https://github.com/LemmyNet/lemmy/issues/3101

          Obvious user-interface signs of the same problem reported June 19: https://github.com/LemmyNet/lemmy/issues/3203

          The problems were going on weeks before I created these issues, and they are still being ignored. It wasn’t in the 0.18 announcement today (June 23), etc.

          •  ffmike   ( @ffmike@beehaw.org ) 
            link
            fedilink
            English
            41 year ago

            I’m not an official spokescritter, but I can assure you the Beehaw admins aren’t ignoring the issues. But ultimately it’s going to come down to someone getting PRs in to the code. I hope someone gets some performance-focused PRs in soon.

            •  BitOneZero   ( @BitOneZero@beehaw.org ) 
              link
              fedilink
              English
              1
              edit-2
              1 year ago

              I’m not an official spokescritter, but I can assure you the Beehaw admins aren’t ignoring the issues.

              They are not informing the end-users (and flocking new server installers) of the problem, they are leaving people like me wasting their time calling out the problem. Denial isn’t just a river in Egypt. Lemmy isn’t scaling, it’s falling flat on it’s face, and the federation protocols of doing one single like per https transaction are causing servers to overload peer servers. There isn’t even anything built into lemmy_server to detect that posts&comment are misisng, nor any tools to ‘heal’ missing data.

              Where are the server logs? Why are the crashes not being shared to developers? Do i really have to build up an instance with 5000 users to get access to the data that Beehaw’s servers are logging each hour?

                  •  PenguinCoder   ( @Penguincoder@beehaw.org ) 
                    shield
                    A
                    link
                    fedilink
                    English
                    4
                    edit-2
                    1 year ago

                    Thanks for the suggestions. We are aware of how to review system logs and work to solve the issues. Right now there are a lot of moving parts, some of which we control and are responsible for, but a lot that we cannot.

                    As you know, an NGINX 500 issue is due to the server instance and not the client (you). For our stack, that could be an issue anywhere along the path with varnish, nginx, firewall rules, security/HIDS, host networking, docker networking, one or multiple services of the six containers, the docker service/daemon itself.

                    The issues are being addressed as we are able to troubleshoot, prove it, and verify a solution.

                •  BitOneZero   ( @BitOneZero@beehaw.org ) 
                  link
                  fedilink
                  English
                  1
                  edit-2
                  1 year ago

                  Because not every issue we’re experiencing even the 500’s , are a result of Lemmy or their code.

                  Then what are they, when Nginx is failing to talk to the NodeJS app? I also consider this more than code, as they are also giving recommendations for performance tuning various components, etc.

                  I have a lot of suspicion so far that federation activity is causing 500 and other errors due to how it queues (swarms) other peers. It isn’t just the lemmy-ui webapp, smartphone users, and other end-users.