- Head admin @ lemm.ee, a general-purpose Lemmy instance
- Creator of lemmy-ui-next, an alternative Lemmy frontend
- Lemmy contributor
[!ko-fi](https://ko-fi.com/sunaurus)
Sorry, it was a glitch in the Matrix, the comments are actually here: https://lemm.ee/post/45660045?scrollToComments=true
Expected lemm.ee downtime when this post is 13 hours old.
Hey!
Unfortunately, Hetzner (our hosting provider) is currently experiencing some network issues. They are planning to address this with an emergency maintenance in roughly 13 hours from now, which will cause lemm.ee downtime. Hopefully we'll be fully recovered later tomorrow!
----
UPDATE: Sorry for the false alarm, I was on the move when I posted this and missed the fact that the Hetzner notice was actually for next month! So it's not as imminent as I originally understood. As we have a whole month to prepare, I will probably be able to come up with some alternative solution to prevent the downtime while they are conducting this maintenance.
It was the result of a DoS attack yesterday, should be mitigated & recovering now.
Seeking feedback: how should lemm.ee move forward with external images? (related to frequent broken images)
Hey folks!
I am looking for feedback from active lemm.ee users on what you all value when it comes to images on Lemmy. I'll go into a bit of detail about what our options are, and then I would ask you to voice your opinion about the issue in the comments.
First, some context for those who don't know. Lemmy software can be configured to handle images in three different ways:
- Store images locally - whenever an external image is posted somewhere, lemm.ee will download a permanent local copy. When you view posts, you are seeing our local copy of the image.
- Proxy all images - similarly to the first option, lemm.ee will download a local copy of external images, however, this copy is temporary. It will be automatically deleted shortly after, and if users open the relevant post/comment again in the future, there will be another attempt to download a temporary copy at that point.
- Pass through external images directly - lemm.ee never downloads any external images, users will always connect directly to the source servers to load the images.
There are pros and cons to each configuration.
Storing images locally
Benefits:
- Your IP address is never leaked to external image hosts, as you never connect directly to the source server. External image hosts only see the IP address of the lemm.ee server.
- External servers don't become bottlenecks for opening lemm.ee posts. If an external server is slow, it won't matter, because the image is always available locally
Downsides:
- As time goes on, our storage will fill up with hundreds of gigabytes of useless images, most of which will never be viewed again after the relevant posts fall off the front page.
- Many big external image hosts will rate limit bigger Lemmy servers, causing broken images when we fail to make a local copy.
- Crucially: some people love to spend their time uploading illegal content to online servers. There are tools to try and filter out such content, but these are not perfect. The end result is that there is a high chance of some content like this inadvertently reaching lemm.ee storage and staying there permanently. This downside is why lemm.ee has not, and will not, use this particular configuration.
Proxying images
Benefits: In addition to the same benefits as exist for the permanent local storage, by only temporarily making local copies for the moment they are requested by our users, we free up a ton of storage & remove the risk of permanently storing illegal content on our servers.
Downsides: The key downside is that external rate limits hit us much harder, as we will be requesting external images far more often. This results in a lot of constant broken images on lemm.ee.
Passing through external images
Benefits:
- Images are rarely broken, unless the source server goes down.
- The images never touch our servers, removing a lot of risk with illegal content as well as with storage costs.
Downsides:
- Our users lose a degree of privacy. Every external image that is loaded on your browser will result in the remote server getting a request directly from your computer to fetch that image - this is pretty much the same as you had visited that external server directly, which lets them log your IP address if they wish.
- When remote servers are slow, it can slow down the entire page load in some cases.
Current situation
Initially, lemm.ee was using the third option of passing through images. Ever since support for option 2, image proxying, was implemented in Lemmy code, we immediately switched to that option, mainly for the privacy benefits. However, after many months, and being blocked by more and more external servers, it is clear that image proxying is seriously degrading the user experience on lemm.ee. We often end up with broken images, and our users have to deal with the results.
I still believe image proxying is a really valuable feature, but I am starting to believe it is a better fit for small instances which make much less requests to external servers.
As a result, I am now seriously considering switching back to the previous method of passing through external images.
This is where you come in - I would ask you as users to please let me know which do you value more: the privacy that you get from image proxying, or the better user experience you get from directly passing through images from their source. Please let me know in the comments how you feel. If I get enough feedback about people being against image proxying, then I will be switching it off for lemm.ee soon. Thanks for reading & sharing your thoughs, and I hope you have a great weekend!
Hi, there is no free speech policy on lemm.ee, we have very strict moderation when it comes to our rules. We regularly permaban users for breaking our instance rules. We simply don't use defederation as a moderation tool, preferring other tools like user bans, for reasons outlined here: https://lemm.ee/post/35472386
Our pict-rs (image software) was having some issues, but it should be resolved now!
Can you try clearing all your cookies & then logging in again? I'm not sure if clearing the cache also clears cookies in Firefox.
Hey, I saw this ping, but I didn't actually get any message from you about CORS headers. Where did you contact me?
That one was an error on the lemm.ee side, but should be fixed now, thanks for linking it!
I don't see any errors with this image on the lemm.ee server side, most likely it's indeed some kind of client issue.
Interesting! We've had quite a noticeable spike of sign-ups on lemm.ee as well
Hey, the 20 character limit for display names is hardcoded into Lemmy. Even if we changed this for lemm.ee, I'm not sure if it would work through federation, as other instances might not accept such a long display name.
Actually, disregard that, I was looking at the wrong thing - it might be possible to raise this limit after all. I will take a better look in a few hours.
There isn't any way to do this with the default lemmy-ui unfortunately
Hey! I'm not really sure about this at the moment. I can tell you that if the authors (or any legal entity) would contact me about this and ask for links to be removed, then I would comply, rather than try to fight it.
Usually, this is caused by external servers blocking the lemm.ee server from downloading images. This is happening surprisingly regularly, I'm not sure if there's a good fix for this..
The YouTube thing does indeed appear to be a bug, I will look into that separately. Thanks for reporting!
We did indeed have a delay with outgoing federation around the time you made this post, but it already recovered shortly after. Sorry for the inconvenience.
Generally it is indeed possible to take over communities with inactive moderation, but have you tried contacting the mod first?
@[email protected], are you still around?
We had a temporary issue with generating different formats/sizes for existing images. I believe the issue is now fixed, our server is going through a backlog of missing files currently and regenerating them. I believe all broken images will be fixed within the next few hours.
Sorry for the inconvenience!
We had a temporary issue with generating different formats/sizes for existing images. I believe the issue is now fixed, our server is going through a backlog of missing files currently and regenerating them. I believe all broken images will be fixed within the next few hours.
Sorry for the inconvenience!
Hey, I see you luckily already received a relevant response here, but I still want to clarify: this community is for lemm.ee support, but your issue seems to be related to [email protected], so actually lemm.ee admins wouldn't be able to help you with this.
For now, I just redeployed all of our servers completely, but as I don't know the actual root cause of the issue yet, I'm still investigating to figure out if anything more is needed.
2024-07-18 lemm.ee downtime
Hey folks!
Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.
For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!
Why defederation is extremely rare on lemm.ee
Hey folks!
For anybody stumbling on this post from outside lemm.ee: I am the head admin of lemm.ee, a general purpose Lemmy instance, which recently turned 1 year old. I am writing this post to elaborate on how we approach defederation on lemm.ee.
Anybody who has been on Lemmy for a while has most likely seen several public defederation drama posts (most recently regarding lemmy.ml, but there have been many many others previously). As an admin, I have probably seen far more than what is visible publicly, as I regularly receive private messages on the topic, ranging from polite questions about federation, to outright demands that I immediately defederate, and even to threats and personal attacks over the fact that I have not defederated some particular instance. It is definitely a topic that will keep coming up for as long as Lemmy exists, which is why I feel it would be useful to condense my current thoughts about it in a single place.
Note that while I strongly believe everything this post contains, it is definitely a subjective topic, and there is no single right answer here. Other instances have completely different approaches to federation compared to lemm.ee, and that’s of course totally fine. The beauty of Lemmy is that everybody can choose their home instance, and in fact, everybody is free to spin up their own instance and run it however they feel is best. For an absurd example, if you want to create an instance which defederates any instance with an “L” in their name, then nobody can stop you!
Quick intro to the lemm.ee federation policy
Very shortly after creating lemm.ee, I wrote down a federation policy, which basically boils down to “we treat defederation as an absolute last resort, and we do not use it as a generic way to curate content for lemm.ee users”. This policy can always be found in the sidebar of the lemm.ee front page.
In practice, this has meant that we have had extremely few defederations, and that we mostly solve problems with other means. I am very happy with the results, as it means that lemm.ee has become a great entry point into the Lemmy network, with very few artifical limitations on who our users are allowed to interact with.
The benefits of federation
I hope that this part of the post is very uncontroversial, but I firmly believe that federation is the absolute strongest feature of Lemmy. While we all know that the concept of federation can cause confusion for new users, this is usually overcome extremely quickly (for example, using the common e-mail providers analogy to explain Lemmy instances). To me, it’s completely clear that the benefits of federation far outweigh the downsides.
For example, by splitting the Lemmy network between thousands of independent nodes, we ensure that:
- Any single entity is not a single point of failure for the whole network. Even if the biggest instance goes down tomorrow, their content will still be accessible through all the other federated instances.
- The maximum impact of admins is limited to their own instance. As a lemm.ee admin, I can ban a remote user from posting on lemm.ee, but I can’t completely ban them from the entire network.
- Private user data (such as ip addresses, e-mails, etc) are never shared between instances. No single malicious instance can harvest user data for the entire network, and extremely privacy sensitive users can always spin up their own instance if they don’t want to put their trust in any existing admins.
One thing which is probably important to note here is that I tend to view Lemmy instances as infrastructure, rather than as communities. I know that there are alternative approaches, as quite a few large instances are in fact run as mega-communities, but that’s not the approach I take with lemm.ee, because I feel like such an approach encourages centralization and negates some of the benefits of federation (if all communities related to one topic condense on a single instance, then that instance does effectively become a single point of failure for a large number of users).
In general, I feel like it should be a goal to encourage and cultivate decentralizing the network through federation as much as is practical, in order to maximize the above benefits.
The downsides of dedeferation
Conversely, defederation has a lot of downsides.
- It obviously negates all the benefits of federation mentioned above. Every time two instances defederate, the Lemmy network becomes less redundant, some communities become a bit more centralized, and the danger of malicious admins for those communities becomes much greater.
- There is a lot of collateral damage. The most common reason I have personally seen for defederation demands is related to moderation of either a single user, or a handful of users. For example, a lemm.ee user gets into some heated arguments with people from an instance with hundreds of active users, and then links this heated thread to me as proof that the instance should be immediately defederated. However, in this situation, there are hundreds of other users who were not even involved (or even aware of) the thread in question. By defederating, I would be making a decision to cut off every single lemm.ee user from every single one of those hundreds of innocent remote users.
- Ironically, defederation actually makes moderation more difficult. It was recently pointed out to me by a user on another instance that they are afraid they can’t effectively moderate communities on lemm.ee, because their instance has defederated several other instances, which means they would not be able to see posts from those instances on lemm.ee communities.
- It is extremely easy for malicious actors to abuse. In the year I’ve been on Lemmy, I have already seen two separate cases of users creating accounts on another instance and posting garbage, and then going back to their home instance and demanding their admins defederate over the content they themselves created. Basically, if an instance is known to use defederation as a tool to punish misbehaving users on other instances, then it’s actually quite easy for users to manipulate the situation to a place where admins have no alternative except to defederate.
It seems to me that a lot of users don’t think of such downsides when demanding defederation, or they just don’t consider them as important enough. In my opinion, these are all significant issues. I do not want to end up in a fragmented Lemmy network, where users are required to have accounts on 5 different instances in order to be able to access all their communities.
What’s the alternative to defederation? Should Lemmy become some kind of unmoderated free speech abolutism platform?
I want to be very clear that I do NOT believe in unmoderated social networks. Communities should always be free to set and enforce rules which foster healthy discussions. On top of that, instances should always be free to set and enforce rules for all of their users and communities.
In the case of lemm.ee, we have some instance-wide rules, and we will enforce them on all lemm.ee users, as well as all remote users participating in communities hosted on lemm.ee. For example, we never want to offer a platform for bigotry, so we regularly issue permanent bans for users who want to abuse lemm.ee to spread such hate. In practice, site bans have been extremely effective at getting rid of awful users, whether they are remote or local.
On top of site bans, Lemmy admins also have the option of removing entire remote communities. There are certainly cases where a community might be allowed on instance A, but not instance B - rather than defederating (and potentially cutting off a lot of innocent unrelated users), instance A can just “defederate” a single community.
Finally, a lot of issues can be solved through simple communication between instance admins. Often having a discussion with another admin results in pretty clear alignment over whether some user is problematic, and the user will end up being banned on their home instance.
Being one of the most openly federated large instances with such an approach, we have discovered several things:
- If we were to defederate over every rule breaking user or community on the Lemmy network, we would not be federated with any of the large instances at this point
- In the vast majority of cases, remote users who have broken lemm.ee rules have ended up banned on their home instance anyway - there is very little additional moderation workload for our admins from being widely federated
- If a user truly wants to spread some kind of hate, defederation wouldn’t stop them anyway, as they will just create accounts on any instance which they want to “attack”
The longer I run lemm.ee, the more sure I become that in the vast majority of cases of abusive users, the best approach is to simply hand out site bans.
When is defederation the only option?
Having said all of the above, I still believe that there a few cases when defederation is the best option:
- When an instance is abusing the Lemmy network - generating spam, advertising, illegal content, etc - either deliberately, or through inactive admins (this has been the most common reason for lemm.ee to defederate any instance in the past)
- When an instance is just causing too much moderation workload. So far, we haven’t experienced this yet on lemm.ee, but I can’t rule out that it could happen in the future.
Conclusion
I hope this post helps clarify my stance on defederation. Like I said in the beginning, I realize a lot of this is subjective, and there are no right or wrong answers - this is just the way we have been (and will be) doing things on lemm.ee. I intend to save this post and link it in the future when people bring up defederation requests. If you feel like I didn’t address something important, please feel free to raise it in the comments!
When this post is 3 hours old, lemm.ee will briefly go offline for database maintenance
Hey folks
Just a heads up that I will be doing some minor database maintenance shortly. I expect the downtime to last <5 minutes.
Have a nice day!
Update: maintenance is complete!
Happy cakeday, lemm.ee!
Hey, folks!
Today, we can celebrate the first anniversary of the creation of lemm.ee! I thought it would be cool to write down how lemm.ee was born, as well as collect some stats about our first year. Here goes!
A quick recap of the beginning of this instance
As probably many others here, I discovered Lemmy early last summer. I had been aware of the Fediverse previously, and always thought it was an amazing concept, but I had never been super interested in Twitter-style social networks. When I found out that Lemmy combined all the great parts of federation with the best parts of link aggregation, I knew that I had to join immediately.
As I was trying to find an instance to make my account on, I realized that most instances were struggling to keep up with a massive influx of new users. At the time, there was a big explosion in Lemmy user numbers, and the network wasn’t fully ready for it. I have some experience with building software for scale, so it felt natural to set up a new instance and try to help with spreading out the load. I got to work in the evening of the 8th of June, 2023, and I was actually so excited about everything, that I completely skipped sleep that night. By the morning of the 9th of June, lemm.ee was online.
From the very beginning, I always intended for lemm.ee to be a welcoming, reliable, and stable gateway into the Lemmy network. I wrote a welcome post on lemm.ee, which most of you have probably seen, as well as a comment on lemmy.ml inviting new users to lemm.ee (lemmy.ml, as many instances, was extremely overloaded at the time).
We started growing extremely quickly. Thousands of users joined lemm.ee over the first few months. Even during the biggest waves of new users, we never closed our sign-ups. The first month or two were definitely very stressful in terms of just trying to deal with the load, but overall, I think I managed to deal with it well enough, and lemm.ee has been running more or less smoothly (with a few exceptions) ever since.
Some stats about the first year
I promised to collect some statistics about lemm.ee so far. This is what I’ve come up with:
Usage
Overall, lemm.ee has 28,715 registered users. Of course, it’s easy to create an account, and most of these are probably inactive at this point, but it’s still a ridiculous amount.
Of all the registered users, 7903 have made at least one post or comment.
7373 users have never made any posts or comments, but have still been voting. This means that out of users who actually interact on lemm.ee, more than half generate content (through their comments and posts) - this is way more than I expected!
Meanwhile, we also have 13,439 users who have never made a single comment, post or vote. I guess most of these are people who just signed up and never got into Lemmy, but I’m sure there are quite a few hardcore lurkers among this group as well.
As for communities, our users have created 1430 of them. Most of these have not (yet) taken off, as only 491 of these communities have at least one comment in them. In general I am happy to see some great communities appearing on lemm.ee - my hope is that we can spread awesome communities out quite evenly on the network, so that in the end, no instance becomes a single point of failure for Lemmy.
Judging by posts and comments made by lemm.ee users, I feel like we’re definitely on the right track: our users have made 20,898 posts in local communities, and 30,847 posts in communities hosted on other instances. The situation is even better for comments, where lemm.ee users have written 42,785 comments in local communities, and a whopping 569,730 comments on remote communities! This means that lemm.ee is not just its own little closed pocket in the Fediverse, but indeed a proper gateway to the Lemmy network, which is exactly what I always hoped it would be.
Note about comment and post counts: I realize the numbers above don’t match the stats about posts and comments on our front page, I’m guessing something is out of sync there, but the stats I am sharing here are based on actual fresh data, counted directly in our database today.
Lemmy (and lemm.ee) would be quite useless without its users, so a big thanks to all of you for using lemm.ee!
Administration
We have a really awesome volunteer admin team, with admins putting in countless hours of their free time to help weed out bad actors. A lot of the work our admins do is completely invisible to most users. I think the admin team does not really get enough recognition, and in fact in many cases, they actually get some undeserved abuse thrown at them.
I am personally very grateful for everybody who has stepped up to be a part of the team, and I think all lemm.ee users benefit from their work every day. In the past year, our admins have handled 12,329 reports from users. While most reports aren’t too bad, and don’t require harsh action, there is still a significant amount of these reports which contain the absolute worst content which you can find on Lemmy - hate speech, bigotry, gore, even illegal content. Our admins are constantly going through every single report they receive, to ensure that mods are getting admin-level support where needed, and to ensure that malicious users in general can’t use lemm.ee to spread garbage into the Lemmy network.
In terms of admin actions, I think the most interesting statistic might be amount of users banned by lemm.ee admins, grouped by their home instance. I will list the top 10 instances here:
- kbin.social: 581
- lemm.ee: 355
- lemmy.world: 31
- sh.itjust.works: 29
- m.mxin.moe: 28
- discuss.tchncs.de: 26
- kbin.chat: 22
- mastodon.social: 19
- lemmy.ca: 18
- fedia.io: 16
As you can see, with the exception of kbin.social, the vast majority of our instance bans are for our own users. Most of the big instances are actually very good at banning their own abusive users, and once they are banned on their own instance, our admins don’t really need to worry about them, as they have no way to log in at that point. kbin.social is a bit of a special case - they either don’t give out a lot of bans, or those bans just don’t federate to Lemmy properly, and for some reason, a lot of advertisers sign up on that instance all the time.
Financials
I have received some questions every now and then about how much it costs to run lemm.ee. While you can always get a sense for the predicted monthly costs for the current month on https://status.lemm.ee, I thought I might include a full breakdown of our costs for the first year here.
Here are all of our costs for the past year, grouped by service:
- Postmark: 177.06€
- Cloudflare: 222.28€
- DigitalOcean: 1744.27€
- Hetzner: 510.20€ (lemm.ee migrated from DigitalOcean to Hetzner several months ago)
- Backblaze: 3.78€ (we’ve been using Backblaze B2 for a few months now, it’s incredibly cheap)
- Domain registration: 100.70€ (paid for the next 10 years!)
We are currently completely funded by lemm.ee users!
There is a small minority of users who are shouldering the entire cost of lemm.ee for all of us. I am extremely grateful that others find Lemmy useful enough that they have put their own money into ensuring financial stability for lemm.ee.
We currently have 49 active sponsors on GitHub, and 7 active supporters on Ko-Fi. In addition, there have been 62 more sponsors on GitHub over the past year, as well as 49 additional supporters on Ko-Fi. This means that in total, 167 users have supported lemm.ee financially. This has completely exceeded all my expectations, I really think it’s incredible. A huge thanks on behalf of myself (and I think I can speak for all other lemm.ee users here as well) to all the supporters!
Conclusion
Running lemm.ee has certainly been a rollercoaster in many ways. There are a lot more things which happened during the first year that I could write about here. On the other hand, this post is already quite long, and a lot of the things which happened are probably best forgotten about anyway, so I think I should wrap up here 😅.
At its core, Lemmy is really an amazing piece of software. It’s helping real humans connect on the internet, without any corporate bullshit. I am very happy to be here with all of you, thank you for joining lemm.ee for its first year, and I hope you’ll join me here again when I write this post in another year from now!
Update about lemm.ee infrastructure & upcoming cakeday
Hey all!
Upcoming lemm.ee cakeday
Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.
I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!
Infrastructure updates
A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.
With the two changes mentioned above, I believe lemm.ee should now be much more resilient going forwad, and I expect a significantly lower rate of infrastructure-related issues for the rest of the year!
I'll leave a tehcnical overview about the problem & solution below for those interested, but if these details don't interest you, then you can safely skip the rest of this post.
----
For context, lemm.ee has been hosted on Hetzner servers for most of this year (having migrated from DigitalOcean initially), with everything except our database being hosted on the Hetzner Cloud side, and the database itself living on a powerful dedicated Hetzner server. This mix allows a great amount of flexibility for redeploying and horizontally scaling our application servers, while still allowing a really cost-effective way of hosting a quite resource-hungry database.
In order to facilitate networking between the cloud servers and the dedicated database server (which live in different networks), Hetzner provides a service named "vSwitch". This service basically allows you to connect different servers together in a private network. Unfortunately, I discovered quite quickly that this service is very unreliable. During the short few months that we have been using the vSwitch, we have gone through one extended period of downtime (where the service was just completely broken for several hours), as well as dozens (if not hundreds at this point) intermittent disconnects, where servers randomly lose their connections over the vSwitch. After such a disconnect, the connection never recovers without manual intervetion.
For most lemm.ee users, the majority of these vSwitch issues have been mostly invisible, as we have redundancy in our servers - if one server loses its connection to the database, other servers will take over the load. Additionally, I have generally been able to respond quite quickly to issues by redeploying the broken servers (or deploying other temporary workarounds). However, in addition to a huge amount of these issues which lemm.ee users hopefully haven't ever noticed, there have also been a few short periods of downtime this year so far, as well as a few cases of federation delays. These more extreme cases were generally caused by multiple servers losing their vSwitch connections at the same time.
After several attempts to work around these issues, I decided that we need to migrate away from vSwitch.
As of earlier today, lemm.ee is no longer using Hetzner's vSwitch at all!
I finally found enough time earlier today to focus on this migration, and I was able to successfully complete it. None of our networking is relying on the vSwitch anymore.
In the end, I went with quite a simple solution - I configured a host-level firewall (nftables) on our database dedicated server, which will deny all connections by default. Whenever any cloud servers are added/removed, their corresponding public IP addresses are added/removed in the allowlist of our database firewall. It would have been ideal to do this whole logic in Hetzner's own firewall, but that one unfortunately has a limit of only 10 rules per server, which is just not enough for our setup.
Bonus: our pict-rs server has been decommissioned!
Pict-rs is the software which Lemmy uses for everything related to media (image storage mostly). Initially, pict-rs required a local filesystem to store both files as well as metadata about files. Since the beginning, lemm.ee has used a dedicated server just for pict-rs, in order to ensure we could easily redeploy the rest of our servers without losing any images.
Over the past year, pict-rs has gained the ability to store files in object storage, and metadata in a PostgreSQL database. This meant that the server running pict-rs itself no longer contained any of the important data, so it became possible to redeploy without losing any images. Additionally, this meant that it would be possible to run multiple pict-rs servers in parallel.
While we had already migrated our pict-rs server to use object storage and PostgreSQL several months ago, we still had the single dedicated pict-rs server up until today. I have been planning for a while to decommission this server, and start running pict-rs directly on each one of our Lemmy application servers. Earlier today, I was able to complete this plan. This should hopefully mean that our pict-rs server is less likely to get overloaded, and it also means a tiny reduction in our overall monthly infrastructure bill (due to one less server running).
With the above changes, I think our infrastructure has become more robust, and hopefully, we will experience less issues with images, federation, and general downtime going forward.
----
That's all from me for now. Feel free to leave any thoughts or questions in the comments, and as always, I hope you're having a great day!
Notice for all moderators: please check the lemm.ee moderation policy
Hey folks!
This is a quick notice about a change to our moderation policy.
We have had a policy on lemm.ee for administration and federation nearly since the very beginning. This policy has also always included a section about moderator responsibilities. Today, we have made two changes to this policy:
- The policy has been renamed to Policy for administration, moderation, federation - this is to make it clear that the policy is also relevant for mods
- We have introduced a new responsibility for moderators, they must "Ensure that they only provide accurate and clear reasons for mod actions".
The reason for the addition is that mod log actions federate out to other instances, and are more or less permanent (due to how Lemmy and federation works right now). This means that users do not really currently have any easy way to clarify or defend themselves against inaccurate accusations in the mod log.
As always, I am very grateful to all mods for your efforts in building awesome communities on lemm.ee. I hope you can understand why this new policy is necessary - I do not want to make your lives more difficult, the goal is to just try and reduce any mod log related misunderstandings in the future.
Thank you for reading and have a nice day!
2024-05-06 lemm.ee downtime
Hey folks!
We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!
----
Technical details
Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.
I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.
Update: the migration is now complete! You can read more here.
Seeking feedback on lemmy-ui-next! (Milestone 2 is nearing completion!)
Hey folks!
I've been steadily working through the roadmap for lemmy-ui-next (which is a new alternative Lemmy frontend), and it's getting to a point where I think https://next.lemm.ee is becoming quite usable. I've been personally using it as my main Lemmy frontend for several weeks now, and I know there are a few other brave users doing the same, so at this point, I'm confident enough to ask the wider lemm.ee population to try it out and share some honest feedback.
If you're at all interested in this project, I would massively appreciate it if you could spend some time using https://next.lemm.ee and letting me know how you feel about it. I'm interested to hear about things like:
- are you running into any issues or bugs
- are there any things that generally annoy you
- are you missing any features
- what would it take for lemmy-ui-next to become your preferred frontend
- anything else that comes to mind
Please keep in mind that this is still a work in progress - some features are planned but not implemented yet (see the roadmap linked above for more details), other features are half-finished and may be a bit buggy still!
Any feedback would really help me out, so please don't hesitate to share!
When this post is 6 hours old, lemm.ee will undergo database maintenance, which will include a few minutes of downtime
Hey folks
This is just a quick heads up that I need to perform some maintenance & upgrades on our database server, which unfortunately will require downtime. I don't expect the downtime to last for longer than 2-3 minutes, but just wanted to give a heads up first so you know not to be concerned.
That's all, hope you have a great week!
Edit: maintenance complete!
Milestone 1 is complete!
Milestone 1 complete!
This is just a mini-announcement & celebration for the fact that I have completed the scope for the first milestone I set for myself in the roadmap.
Of course, I am still planning to keep improving and tweaking things as I go, but in terms of the raw list of features, the work for milestone 1 is complete. I am now going to take a day or two to clean up the code and work on some performance optimizations, and then in the later half of the week, I will continue working towards milestone 2, starting with commenting features!
If anybody is interested (and brave), please feel free to check it out at https://next.lemm.ee, and feel free to share any thoughts and feedback in the comments!
Intro & roadmap
Intro
This project is an open source alternative frontend for Lemmy. It is built with Next.js.
- Preview environment: next.lemm.ee
- Source code & issue tracker: GitHub
Screenshots (desktop & mobile) !
Goals
- Drop-in replacement for lemmy-ui
- Minimalistic design, following in the footsteps of other timeless link aggregator UIs
- Fast!
- Super basic NextJS architecture, taking advantage of features like the app router & server actions
Motivation
The original lemmy-ui has been extremely important for the growth of Lemmy, and the new lemmy-ui-leptos also looks quite interesting. One issue with both of these is that they are built using quite obscure technologies (Inferno and Leptos).
This project was created as an alternative for contributors who are already familiar with NextJs and want to use those skills on Lemmy. The beauty of open source is that anybody can build what they want, and all these alternative projects can happily coexist!
You can read more in the original announcement post here.
Roadmap
✅ - Completed
Milestone 1 - Lurk (✅ v0.1.0)
Includes read-only functionality, more or less everything you need in order to be a lurker on Lemmy
- Front page (✅ v0.1.0)
- Single post page with comments (✅ v0.1.0)
- Single comment thread page (✅ v0.1.0)
- User profile (✅ v0.1.0)
- Community page (✅ v0.1.0)
- Communities list (✅ v0.1.0)
- Inline expanding media (✅ v0.1.0)
- Separate mobile layout for narrow screens (✅ v0.1.0)
- Search page (✅ v0.1.0)
- Federation page (✅ v0.1.0)
- Full Lemmy markdown support (spoiler tags, custom emoji, etc) (✅ v0.1.0)
- Blur NSFW content (✅ v0.1.0)
Milestone 2 - Participate
Features related to actually participating on Lemmy
- Login page (✅ v0.1.0)
- Sign-up page (✅ v0.9.0)
- Forgot password page (✅ v0.5.0)
- Vote functionality (✅ v0.1.0)
- Post create/edit/delete (✅ v0.3.0)
- Comment create/edit/delete (✅ v0.1.0)
- Inbox (Replies, DMs, mentions) (✅ v0.8.0)
- DM sending (✅ v0.6.0)
- Post/comment sharing (✅ v0.2.1)
- Post/comment saving (✅ v0.2.1)
- Image uploads (✅ v0.10.0)
- User settings page
- User/instance/community blocking
Milestone 3 - Moderate
Features related to moderation & administration
- Report posts/comment/DMs
- Report inbox
- Community create/edit/delete
- Modlog
- "Rap sheet" on user profiles
- Mod toolbar on posts/comments
- Instance settings for admins
- Sign-up applications inbox
Future ideas
- GitHub actions pipeline
- Complete instructions & examples for deployment on other instances
- More themes/layouts?
- More features for markdown editor (more formatting options, emoji picker, @mentions)
Announcing lemmy-ui-next, an alternative Lemmy frontend built with NextJS!
Hello, friends!
TL;DR: I am working on a new Lemmy frontend in nextJS. There is still much work to be done, but you can already have an early look at https://next.lemm.ee
First of all, quick note to lemm.ee users: I am making this announcement post in [email protected], as this is also a notice that I will be hosting an alternative frontend (lemmy-ui-next) for the first time on lemm.ee. Going forward, I will post updates about lemmy-ui-next in a separate dedicated community: [email protected]. If you're interested in future updates, please subscribe there!
What is lemmy-ui-next?
Lemmy is generally accessed through some kind of frontend UI. By default, Lemmy provides its own web interface (lemmy-ui), which you can find on the front page of most Lemmy instances (including lemm.ee). There are also several other independent frontends, for both the web and different mobile platforms, which I'm sure many of you are familiar with.
Lemmy-ui-next is a brand new alternative frontend, built from the ground up with modern and popular tooling - a framework known as NextJS. Lemmy-ui-next has (or aims to have) the following high-level features:
- Open source (AGPL)
- Drop-in replacement for lemmy-ui - same exact URL structure, so all existing links will continue working
- Very plain & minimalistic UI, strongly inspired by other link aggregator sites (of course including the original lemmy-ui!)
- Very basic and "typical" NextJS architecture, to encourage open source contributions
- Fully functional even when JavaScript is disabled (but works better with JS enabled!)
- Optimized data transfer between your browser and the server (filtering out only relevant data from the Lemmy API, caching, memoization)
- Strong focus on privacy and security (all authentication with the Lemmy API is done through secure httpOnly cookies, user IP addresses are not leaked to external image hosts, etc)
What is the current status of lemmy-ui-next?
I have mentally split the initial work I want to complete into 3 milestones:
- Lurk - All read-only features of Lemmy
- Participate - Voting/posting/commenting/DMs/reports, etc
- Moderate - Handling reports, creating & managing communities, etc
I am now nearing completion of the first milestone. It's not 100% there yet, but you can already log in, browse, subscribe to communities and even vote. Some things may still look a bit wonky, and some features are still missing, but the core experience is getting there.
In terms of code contributions, I would ask anybody who is interested in getting involved to contact me first before working on anything. I am not looking for PRs just yet - the code structure is still a bit loose, and I am redefining it as I add more stuff. I would ideally really like to complete the first 3 milestones before opening things up for external contributors.
Who can use lemmy-ui-next?
At the moment, it is only hosted on this instance, at https://next.lemm.ee. I do not yet have any formal instructions for running it on other instances, but generally speaking, it is a simple NextJS app - to deploy it, you just need to do: npm install
, npm run build
and LEMMY_BACKEND=https://<your lemmy api here> npm run start
.
Why not just improve lemmy-ui instead?
Lemmy-ui is an extremely important and valuable project. There has been a significant amount of hard effort put into it so far, and nobody can refute that it is the frontend which has really carried Lemmy to this point.
Unfortunately, there are some architectural problems with lemmy-ui (mostly related to how data is fetched and how sessions are stored in memory), all of which would require quite a significant rewrite to fix. Additionally, I think that the core technical solution used for lemmy-ui is just a bit too obscure, which has been an obstacle to my own contributions, as well as to contributions by others. If a rewrite is on the table anyway, then I believe a different technology is the best way forward.
Why not work on lemmy-ui-leptos instead?
Lemmy-ui-leptos is another rewrite of lemmy-ui, which is being lead by Lemmy maintainers. It is based around a Rust web framework called Leptos. I think this is really cool tech, and will be happy to host lemmy-ui-leptos on lemm.ee in the future as well.
There are a two key reasons why I personally decided to start working on another alternative, though:
- I have heard from several people on Lemmy that they feel like Leptos is a big barrier to entry in terms of them contributing
- Even for myself personally, I am very comfortable (and think I can move very fast) when working on something like NextJS, but with Leptos, I think the learning curve would be quite big and I would get much less done with any time I invest into it
My hope is that by providing a very vanilla alternative, I can provide an outlet for potential open source contributors who would like to work on Lemmy, but aren't prepared to do it with Leptos.
Does this mean that lemm.ee will change in ways I don't like?
First, let me be clear: lemm.ee will always host the default Lemmy frontend. This means lemmy-ui for now, and most likely lemmy-ui-leptos in the future.
I am however considering the possibility of switching things around at some point in the future, so that lemmy-ui-next will be hosted directly on lemm.ee, and lemmy-ui will be accessible on a different subdomain (like ui.lemm.ee). This would only happen once I have completed all 3 milestones for lemmy-ui-next. The main reason I am considering this is that I feel like I will always be in the best position to offer technical support to users on the frontend which I am myself maintaining. If you have any thoughts about this potential change, please let me know in the comments below!
That's about it for now!
This is something I've been thinking of doing for a while now, and I'm very excited to finally get the ball rolling! If you have a chance, please feel free to check out what https://next.lemm.ee looks like so far, and please let me know if you have any thoughts or feedback!
Hetzner networking issues & minor lemm.ee downtime
Hey
This is just a quick heads up that our host, Hetzner, has been experiencing networking issues today, which has caused some downtime for lemm.ee.
I have a workaround in place for now, so we should (fingers crossed) be recovering at the moment, but I am still waiting on the proper solution from Hetzner. You can track their issue here: https://status.hetzner.com/incident/9406c500-9c8b-48be-9591-a73691134096
Also, this is a good opportunity to remind everybody about https://status.lemm.ee - you can be sure that I will provide updates on that page as soon as I am aware of & dealing with any issues. I have been posting status updates for the current issue there as well.
Sorry for the inconvenience and I hope you have an otherwise great day!
UPDATE: Hetzner claims they have fixed the issue, but the problems have not been resolved for lemm.ee servers yet, so I am keeping my temporary workaround active for now. Will continue troubleshooting this tomorrow.
UPDATE 2: Hetzner has now fixed their issue, and our network has been restored to its original optimized state.
I am extremely grateful to everybody involved with Lemmy. That includes you!
Hello!
I am sunaurus, the head admin of lemm.ee. Ever since I created my instance, I have been following a lot of public and private discussion channels between different parties involved with Lemmy. As I’m sure many others have also noticed, the discussions in such channels sometimes get heated, and in fact recently, I feel like there has been a constant trend in these discussions towards a lot of demands, hostility, negativity, and a general lack of empathy between different participants in the Lemmy network.
I am writing this post for a few reasons:
- I would like add a bit of positivity by expressing my gratitude towards every single person who has helped improve Lemmy.
- I want to speak up in defense of different people who have been receiving negativity lately.
- There are a few false rumors spreading on Lemmy, which I would like to try and counteract with very simple evidence.
- I want to remind everybody that at the end of the day, all of us care about building and improving Lemmy. We all have the same goal, and it’s too easy to lose sight of that.
I will split up what I want to say in this post by different user groups - users, mods, admins and developers. I understand that many people belong to several (or even all) of these groups, but I just want to highlight the value of, and express my gratitude to each group separately.
Users
At the end of the day, Lemmy would not be worth anything without the users. Users bring Lemmy to life by posting great content, getting involved in discussions in comments, helping surface interesting content for others through voting and even keeping the platform clean through reports. I am extremely thankful for all the users who have given me so much enjoyment on this platform.
I believe that users often get treated unfairly on Lemmy based on what instance they are participating from. I’m sure so many of you have noticed comments around Lemmy along the lines of “Oh, another user from <instance>, I’m going to completely ignore your stupid takes”. I’ve also many cases of people treating users as second-class citizen if they are not on the same instance - for example, I’ve seen users who are active and valuable participants in communities on another instance receive comments like “why are you participating in our discussions, go back to your own instance”. In my opinion this is completely counterproductive to the whole idea of federation. On a human level, I can understand it - you’re far more likely to notice or remember what instance somebody is posting from if you have a negative experience. As a result, as time goes by, people tend to develop negative views of each instance, despite potentially having had many positive interactions with other users of those same instances. The message I want to put out here is that instances, especially bigger ones, are not monoliths - do not judge users based on what instance they are browsing Lemmy from, judge them by their actual words and actions.
Mods
There are some excellent communities already on Lemmy, and these communities are all continuously being built up and maintained by mods. Mods put in huge amounts of their free time and energy in order to provide spaces for all Lemmy users. They form the first line of defense against bad actors, they keep communities alive and often receive no praise, only criticism. I am very grateful to everybody who has dedicated time to building communities on Lemmy.
Users rarely notice the lengths mods go to in order to keep communities running smoothly - mods more often than not only get noticed when users disagree with some mod actions. I believe mods deserve a lot better than this. Constructive criticism can of course be useful to improve communities, but it must be balanced with empathy and kindness towards people who have been putting in effort to provide something for users. Remember that there is another human being reading your words when you start writing about the mods of any particular community. Users who are not happy with mods of a certain community always have the opportunity to start their own community and run it as they like.
Admins
Admins provide two main key functions for the network:
- Taking care of the actual infrastructure of Lemmy
- Working as a higher level defense against bad actors, in cases where mods are not enough
I can tell from my own experience that being an admin of a bigger instance requires constant energy and attention. I don’t believe that there is a single medium-to-big instance where the admins have not put in hundreds (if not thousands) of hours of their free time, as well as in many cases, probably their own money. This is a service which admins provide for free, and it is necessary in order to keep the Lemmy network healthy. I have endless respect for anybody who is willing to put themselves in the position of a Lemmy admin.
I have seen awful messages towards admins from all the other groups listed here, including other admins. These messages range from condescending and rude, to downright hateful. I have seen admins treated as useless and their work taken for granted. I have seen people getting frustrated with admins for not spending every waking minute on Lemmy. I have seen some users consistently spreading provably false rumors about particular admins in an effort to tarnish their reputation on Lemmy.
Before you take out frustration on admins, please remember that they are also humans who have been working tirelessly to improve Lemmy in their own way.
Also, a reminder: the absolute best feature of Lemmy is that users are free to pick their instance - and as a result, users are also free to pick their admins. Even more than that, users can always become their own admins by spinning up their own instance. Yes, this requires dedication, effort, and research, but that’s exactly my point. It’s not easy running an instance, and mistreating people who do this as a free service is completely unacceptable.
Developers
Lemmy development has been lead by a few key maintainers, with a massive amount of smaller contributors. The software is constantly being improved at a very good pace, and everybody is able to benefit from this effort at no cost whatsoever. I am extremely grateful to everybody who has participated in the development of the Lemmy software, and other related software, as without you folks, none of us would even be here now.
There seems to be a huge amount of people with very little appreciation of the work that has gone into the software. I’m sure many of you have seen countless messages where people express that the devs should be doing more in one way or another. “They should work faster”, “they should prioritize this obviously most important feature”, “they should be available 24/7 to offer support”, etc. I just want to take a moment here and acknowledge what core maintainers have already done for Lemmy:
- Years worth of work on the code itself
- Offering support to the community and other admins
- Reviewing literally thousands of pull requests on GitHub
- Acting fast in stressful situations where the Lemmy network has been overloaded
- Not abandoning the project in the face of constant hateful users
- Sacrificing literally hundreds of thousands of euros in missed salaries which they could have been getting if they were working for a tech company instead of working on Lemmy
I also want to take this moment to discredit some rumors which I have seen repeated too many times:
- Rumor: Lemmy devs do not accept outside code contributions
This is completely false - the maintainers are completely open to (and even constantly asking for) contributions. When somebody starts contributing, they will receive support and code reviews very quickly. I can tell you that I have experienced this myself several times, but that’s anecdotal, so let me also provide evidence:
a. Contributors list for the Lemmy backend: https://github.com/LemmyNet/lemmy/graphs/contributors
b. Contributors list for Lemmy UI: https://github.com/LemmyNet/lemmy-ui/graphs/contributors
Both of these lists include 100 different names, and that’s only because GitHub literally caps these pages to 100 users. Actually, the amount of different contributors is even bigger. If Lemmy devs did not accept and encourage outside contributions, then there would be no way for these lists to be so big.
- Rumor: Lemmy devs work too slowly
This is an extremely entitled and frankly stupid claim. I try to keep on top of the changes made in the Lemmy repo, and let me tell you, the pace of improvement is very good.
I very firmly believe that if the network started downgrading to Lemmy versions from ~8 months ago, the whole network would just collapse, as none of the instances could keep up with the current volume. That is to say, we have come an extremely long way since last summer alone.
Let me provide some more evidence. Take a look at the Pulse page for the Lemmy backend on GitHub: https://github.com/LemmyNet/lemmy/pulse. As of writing this, Lemmy devs have merged 18 pull requests in the week leading up to this post - that’s an average of 2.5 merged PRs per day. This is extremely good for a project with a small underfunded team.
- Rumor: Lemmy devs do not prioritize the important issues
There are two sides to this. First of all, there are endless users who turn to the Lemmy devs with what they believe is the most important issue and should immediately be prioritized - the problem is that almost none of these endless users have the same view of what the most important issue actually is! In that sense, it’s literally impossible to please everybody, because everybody wants different things.
On the other hand, even when Lemmy devs do prioritize things which some users have been desperately asking for, I have on several occasions seen a dismissive response along the lines of “too little too late”. Basically, the demands made are often unrealistic and impossible to meet.
If you are somebody who feels like Lemmy devs are not doing enough, I would ask you to please take a step back, look at the actual contributions which they have made, and consider how you yourself would feel if after making such a massive contribution, you would still need to listen to countless strangers on the internet tell you how you’re not good enough in their opinion.
Conclusion
Lastly, I am very thankful to anybody who took the time to read to the end of this post. Again, my goal is to try and defuse some of the hostility, as well as to put out a message of gratitude and positivity. I am very interested in the success of Lemmy as a whole, and that is much easier to achieve and maintain if we all work together. Thank you, I hope you're doing well, and have a nice weekend!
RFC to split reports into admin and mod reports
The RFC PR is here: https://github.com/LemmyNet/rfcs/pull/6
Reposting RFC contents below:
------
- Feature Name: report-inboxes
- Start Date: 2024-02-20
- RFC PR: LemmyNet/rfcs#0000
- Lemmy Issue: LemmyNet/lemmy#0000
Summary
Rather than combining all reports into a single report inbox, we should allow users to select whether they are reporting to mods or admins, and we should split reports into different inboxes based on that selection.
Motivation
The current approach has some shortcomings:
- Users are not currently able to bypass mods and report directly to admins - this may allow mods to conceal instance rule breaking in specific communities
- Admins are not aware of community rules, so they may wish to take no action for most community rule breaking reports. However, if an admin resolves such a report, the relevant community mods most likely never see it.
- Different instances may have different rules, but somebody resolving a report on one instance will resolve it for other instances as well, thus potentially resulting in missed reports.
- Mods might take local action on a report and mark it as resolved even in cases where a user should be banned from the entire instance. In this case, admins are very unlikely to see the report.
Guide-level explanation
When creating reports, users will be able to select if it's a mod report, or an admin report (or both)
Note: labels on the sreenshot are illustrative, actual labels can be more user-friendy. Maybe something like:
- Breaks community rules (report sent to moderators)
- Breaks instance rules (report sent to admins)
Instead of the current single report inbox, there will be three different kinds of inboxes
- Admin reports - show all reports sent to admins (only visible to admins)
- Mod reports - show all reports sent to mods for any communities the user moderates (visible to admins in case they are explicit mods in any communities)
- This is equivalent to the report view that mods currently have in Lemmy already
- All reports - Shows a view of all (admin and mod) reports, only visible to admins
- This is akin to the current 0.19.3 admin report view, and would allow admins to still keep an eye on mod actions on their instance if they wish
The UI wouldn't need to change for mods, but for admins, there would be a new selection at the top of the reports page (the "mod reports" tab would only be visible if the admin is also a mod in any community): !image
Resolving reports should be more granular
- Reports in the "admin reports" tab can only be manually resolved for admins of the local instance
- To reduce overhead, banning the reported user on the user's home instance + removing reported content should automatically resolve reports for remote admins as well.
- Reports in the "mod reports" tab should be manually resolved by relevant mods (including admins, if they are explicit mods in the relevant community).
- To reduce overhead, admins banning the reported user on the community instance OR the user's home instance + removing reported content should automatically resolve reports for mods as well
- Admins could still resolve reports in the "all reports" tab
- If it's not an admin report, and not a mod report from a community the admin explicitly moderates, then there should be an additional warning/confirmation when resolving a report here. This is to prevent cases of admins accidentally preventing mods from moderating according to their own community rules.
To further clarify automatic resolution of reports: in any case where there is no further action possible, the report should be automatically resolved.
Mods should be able to escalate reports to admins
This would generate a corresponding report in the admin inbox.
Reference-level explanation
- In the UI, changes are needed for both reporting as well as the reports inbox views
- In the database and API, we should split reports by intended audience
- Federation needs to be changed as well in order to allow distinguishing the report target audience
Drawbacks
It might make reporting slightly more confusing for end users - the mod/admin distinction might not be fully clear to all.
Rationale and alternatives
Alternatively, we could make reporting even more granular. It would be possible to allow users to select only a specific instances admins as the intended report audience, for example. However, I think this has several downsides:
- Makes the report UI even more confusing
- Potentially takes away valuable information from other admins (imagine a user only reports CSAM to their own instances admins, while leaving the offending post authors home admins in the dark)
Prior art
Most other social networks allow users to select whether they are reporting a violation of community rules, or site rules as whole.
Unresolved questions
Does ActivityPub properly support splitting up reports like this?
Future possibilities
In the future, it might be a nice addition to have some automation to always escalate to admins, even if they're submitted as mod reports, based on report keywords. For example, "CSAM", "Spam", etc.
Ongoing spam attack on the Fediverse
Hey folks
Some of you may have noticed comments complaining about spam and lack of moderation within the past day or so. Maybe you've even noticed a few spam posts yourself (hopefully not too much, as we have automations in place on lemm.ee to remove the spam as soon as it is posted).
I just wanted to write a quick post with some context about the attack, what we are doing about it, and how you can help.
Context
Allegedly, a group of kids in Japan have created a bot, which signs up on different Fediverse instances and posts spam into different communities. The spam generally consists of Japanese text and/or an image and/or a bunch of random @mentions into different communities. You can check a post on Mastodon with more information here: https://mastodon.de/@ErikUden/111940301222380638
What we are doing about it
Many instances are actively working to limit this spam-wave, and lemm.ee is no different. Thankfully, we have not had to deal with any bot sign-ups on our instance (potentially as a result of different protections we have implemented for sign-ups), but we still suffer the effects of the spam, even if it's posted from other instances. To help us quickly eliminate most of the spam for lemm.ee users, I am continually tuning our @adminbot to automatically detect and remove content posted in this current spam-wave.
We cannot remove content from the wider Fediverse if it's not posted there by a lemm.ee user, so our automated removals won't help users on other instances, but we are at least improving the experience for our own users. For an example, you can compare how /c/[email protected] currently looks like on lemm.ee, to how it looks like on this screenshot I took from another smaller instance:
How you can help
First and foremost, please continue reporting any spam you find, so that relevant mods and admins can deal with it. I am very grateful to users who help us identify spam through reports, and your reports are precisely what allow me to implement automated content removal for more extreme spam-waves such as this current one.
Secondly, I am seeking for a few volunteers to grow the lemm.ee admin team. I am purposely burying this at the bottom of the post, to hopefully pre-filter out some candidates who would want to join for the wrong reasons. If you have read until this point in the post, then I assume you are already quite interested in improving the experience on lemm.ee, so if you feel like you could contribute to the admin team, please read on.
First, I will say a few words about who we are looking for, then I will describe what kind of tasks you would have as an admin, and finally, I will cover some significant downsides of joining the admin team.
We are looking for folks who more or less match the following profile:
- You have already been active on the Fediverse for several months (not necessarily on lemm.ee)
- Previous mod experience would be a huge plus
- You should feel a strong agreement with our basic instance rules and our administration & federation policy
- You should be prepared to be exposed to some vile content through reports
- You are OK with using Discord as the main method of admin communication (that is what we have settled on and will continue using for the foreseeable future)
As volunteers, we don't expect admins to be available 24/7, but as our instance grows, I do think it would be quite important to achieve a state of pretty good timezone coverage with our admin team, so please only consider applying if you are already regularly active on Lemmy.
As for what tasks admins are responsible are for: it's mostly covered in the administration policy post linked above. But in short, you should be prepared to regularly check the report queue, contact users with friendly messages to de-escalate conflicts, issue bans, remove content, and monitor the activity of @adminbot. Additionally, if you're interested in taking a more hands-on approach to any kind of community-building on lemm.ee, then this would be totally welcome as well, but not strictly considered a core responsibility for admins.
Please note that the lemm.ee admin team has an absolute zero tolerance policy against any kind of abuse towards minority communities. If you do not share this mindset, then please do not consider applying.
Finally, let me share some negative aspects about joining the admin team. I think this will probably reduce the amount of any potential candidates, but I still feel it's important to be honest and upfront about this:
Through the report queue, you will regularly see absolutely vile content which you might otherwise never even notice on Lemmy. Many users come to Lemmy to spread hate, post disturbing images, etc, and in order to clean such content up for other users, mods and admins need to actually be exposed to this content in much larger amounts than regular users.
Additionally, while Lemmy is constantly being improved by the developers, the moderation tools are still quite rough around the edges. Lemmy is not at 1.0 yet, and that will most likely become even more obvious to you as you work on admin tasks.
Maybe this is the most important one: no matter what you do, there will always be people unhappy with how you apply our rules. I have seen countless comments complaining about lemm.ee admins specifically. I have been told by complete strangers that they hate me. I have seen many complaints about us moderating too harshly. I have seen complaints about us not moderating enough. I have seen users on Lemmy make up wild stories about our admin team, and share them as facts. There are of course plenty of supportive users, but the negative experiences tend to leave a much more lasting impression.
If after reading all of the above, you are still motivated to help make lemm.ee a better place through offering your help in the admin team, please contact me on Discord (@sunaurus
)!
That's all from me for now. Thank you very much to anybody who went through this whole wall of text, and I hope you are all having a good weekend!
We have a new lemm.ee status page, including federation status, at status.lemm.ee!
Hey folks!
Just a quick update: we now have a dedicated status page for lemm.ee.
You can find it at status.lemm.ee. It currently contains three sections:
- A web status section, which I will update manually to communicate issues about lemm.ee
- A financial status section, which I will update monthly to give an overview of how we're doing financially
- A federation section, which automatically checks the current federation status, both incoming and outgoing, between lemm.ee and other instances. By default it shows 3 large instances, but you can also search for any specific instance you are interested in.
This status page is hosted completely separately from our main servers, so if there is any trouble with our servers, you can expect the status page to still be available!
If you have any issues with this page, or any other thoughts, feel free to comment.