Call for Assistance
Call for Assistance
Hi all,
Thank you for joining me here! It's great to see that we have a community that wants to grow in such a new and exciting manner.
As it is, I thought it would be a lot easier to do this by myself than it has been. So I'm asking for some help!
I have several things I need assistance with:
- setting up and moderating a chat community, for those times when users are having issues with the website. I think it's up to the community what software we use, but I would probably prefer Discord. Since this is all federation though I completely understand if others want to use something like Zulip or Matrix! So let's just use what everyone wants. If you have an opinion please post below.
- database stuff. I'm absolutely terrible at database stuff, and that is not an exaggeration. If anyone is willing to help it would be much appreciated. Currently I have a need to set up pgbouncer, or we should modify the lemmy source to allow for setting up a bouncer. I also want to set up read replicas so that we can distribute the load a bit more evenly. As it currently is, the site was simply set up with the lemmy-ansible script, so everything is running on a single box 😬. If you know Rust and want to help modify the Lemmy source code for this, or you are a Database Admin and want to help, I'd very much appreciate it.
- instance admins. I cannot be online constantly and I do have a day job. I'm getting messages and applications to join the instance along with needing to set up new communities, create and update rules, moderate, etc. I cannot handle this all alone.
- I also need some general help.
- email admin
- migration of server to larger VPS (will have to bring the entire site down for this, unless someone wants to help set up a load balancer, a brand new box, and have some sort of migration strategy.)
If you want to help out on the server side of things I will want to know your real life persona, but for instance admins, chat mods, etc. I would just like to see some sort of comment history from you elsewhere.
And thank you once again, for helping create an inclusive community.
In terms of chat community Matrix seems like the best option since there's mechanics built into Lemmy to interact with it nicely (for example you can add a matrix profile to your account and then people can click a send secure message button when they view your profile to message you on matrix)
Discord does have the most users though
The database knowledge required for the things relating to databases most likely extends way past what I know but I would be willing to help out as an instance admin to keep things running. Ive got some social medias linked in my Lemmy profile if you wanted to check my history
Is there any nice documentation about integrating Matrix with Lemmy?
I have my own Matrix instance, so that would be very cool to be able to utilize.
They can't interact extremely well as Matrix is using its own protocol rather than ActivityPub but things like bridges should be able to be made. (For the Lemmy mechanics here it basically just opens up a link in matrix rather than interacting with Matrix from Lemmy). Don't really know any documentation as I'm relatively new to matrix but I found a bridge between Matrix and Mastodon when I tried to look that may help https://github.com/ma1uta/mxtoot
I'm not really a fan of Discord, so something else would probably be preferable, but whatever it ends up being, I'd really appreciate it if, whatever it is, I could create an account without an email, or even better, if I could just read what's going on without having to sign into something.
Yay for Matrix!
May want to consult this selfhosted community about managing a lemmy server: https://lemmy.world/c/selfhosted
Agreed. As Lemmy instances start getting slammed, there will be a community of instance hosters there developing best practices and so on. Definitely lean on the community there
Also I'd seriously advise against forking the code. You'll be signing up for a world of hurt keeping your customizations up. Contributing to the core project is a better investment if possible.
Finally, thanks OP for standing up this instance! Costs are likely to keep growing, so I hope there's a way we can contribute financially as well
I'm not a DBA, but I love Postgres, so let me know when I can help with anything.
Nowhere near as experienced as some of the users here, but I use Rust and databases on a daily basis for work. Would be willing to learn if there is a need
Please join us in the discord! https://discord.gg/FXWnavvU
I used to work as a senior linux sysadmin for some webhosting/datacentre before i go back to education to do my engineering degree. So i may be able to give a hand in that direction.
Well that would be great. Can you join us in the discord? https://discord.gg/FXWnavvU
I would like to suggest XMPP for the community chat. Federated+standardized protocols that are practical to self-host should be priorities in a world were lemmy is viable and makes sense. In this area XMPP is much friendlier (in admin, in complexity, in resource-usage) than the alternatives.
I also happen to have experience running ejabberd clusters for a couple communities and can offer some amount of sysadmin help would it be wanted. Another interesting aspect of XMPP is that there are efforts underway to federate with the fediverse, so we could probably enable chatting over XMPP with lemmy accounts and vice-versa at some point.
Any of the xmpp communities open? I've tried matrix for chat and it's kinda janky, I wouldn't mind trying an xmpp server.
You can have a look here: https://providers.xmpp.net/
...to pick a provider from the A or B bucket.
Then, client-side, my recommendation would be to go with gajim as a power user, conversations/cheogram on Android (both listed on froid), siskin/beagle on iOS/mac, and have a look at movim.eu for something web based.
I'd be glad to help out where I can. I have plenty of infrastructure background, so I can help with emails, backends, some database stuff, and other admin needs. I'm not familiar with VPS, but have worked on enough platforms I'm sure I can be of service.
I added my profile here to my linktree, so that you can see I'm a real person. https://linktr.ee/mdowst
I suggestion that someone create a shell client stress-test app for the server API. Do a login, do a listing of postings, do a listing of communities, access a user profile, access a posting with 10 comments and another with 150 comments, etc.. Measure the response time and output it in a simple format that different admins can share and identify slow areas within the lemmy_server code.
The problem seems to be rooted in the lack of generating enough dummy test data for the lemmy-server app... it was only incrementally tested with low activity that was on the Lenny network before May 2023... now it is 100x or more data in the comments, upvotes, users, etc.
Each new comment and posting is making servers slower, the SQL is jut not scaling with the current table structures and JOIN usage.
+1.
This thing needs to be profiled and optimized. It should not be running into the ground with this low activity. Worse yet is federation reduces the speed by a ton too, but without it, instances have low activity.
I can help out here and try to do it in spare time, but no commitments/promises as I'm currently oncall at $dayjob.
Not familiar with most of tech needed to run a lemmy instance, but I'd be glad to help with simpler stuff like digging through email.
Let's try a discord first so that we can at least connect and figure the rest out -> https://discord.gg/T4PjRWSa
I'm biased to matrix, as the integration opportunity would be too cool to pass up. But from a pragmatic standpoint, it's good to have multiple communication channels as a fail over, especially when self-hosting matrix servers. So just as status pages are hosted separately, having discord as a backup chat channel could be wise if we had domain or server issues. Plus, as you said, it may take a while to figure matrix out.
I agree, but let's just connect now the most effect way so we can figure out the whos and whats of hosting and so we can help @snowe a bit. I agree that we can then take the time to setup Matrix and mailing and all the other stuff. We can migrate from there :)
I would be happy to help out with instance admin stuff, if you still need help. Some examples:
I shortly looked through the source code for Lemmy, and it's using a Rust ORM named Diesel. I don't know Rust at all, but it seems like in order to add the ability for pgbouncer, two things would need to be done.
- There would have to be separate environment variables for a direct connection to the database, to run migrations. Prisma (an ORM for Node.js, which I am more familiar with) does something similar: https://www.prisma.io/docs/data-platform/data-proxy/prisma-cli-with-data-proxy#set-a-direct-database-connection-url-in-your-prisma-schema. From my understanding, pgbouncer is essentially a queue for the database. So if you have a bunch of things in the queue, it will delay your migrations from running.
- Once step 1 is completed, the regular database environment variables can point to the pgbouncer connection pool. Based on the pgbouncer documentation: PgBouncer acts as a Postgres server, so simply point your client to the PgBouncer port.
Like I said, I don't know Rust but I would be interested in learning it if I have time. Or maybe someone else in the open-source community could work on this. There's no specific issue for adding pgbouncer support, but there's an issue about slow SQL queries that mentions it: https://github.com/LemmyNet/lemmy/issues/2877
PgBouncer is a connection pooler, it works as a cache for the PostgreSQL database handles. I'm not aware of any queueing outside of that.
Migrations just need read-write handle, which PgB can provide, there is no need to move around it. Edit: also a user with alter privileges :)
Alters need various levels of schema locks, which are a separate domain of problems. If the migrations only add new relations and empty columns, migrations will only lock for a milisecond, but will still wait for the lock in the queue. 'Destructive alters' where you can lose data by dropping columns or 'productive alters' adding lots of values will lock for the entire table rewrite, but I would expect the devs to know this and not write alters like that without warning in the changelog.
Happy to help however I can! No experience with Rust, but have been doing backend web stuff for quite a while. I'm comfortable with friend stuff as needed as well as Postgres. Also just happy to help with admin things as needed.
I'm willing to help out in any way I can to make this happen.