Its now running on a dedicated server with 6 cores/12 threads and 32 gb ram. I hope this will be enough for the near future. Nevertheless, new users should still prefer to signup on other instances.
Please know that your work is genuinely appreciated in fascilitating the migration from Reddit to Lemmy. Your efforts will hopefully ensure a bright future for communities on this platform. Kudos @[email protected] !
I just want to say, you all are doing a great job. Maybe I don’t fully understand it yet, but I have created some communities on the lemmy.ml instance and I don’t really want to move away. Is there anything I can or should do? I think that I’m currently locked in here.
I was getting 502 Bad Gateway. When I pinged Lemmy.ml I got an IPV6 address. It disabled IPV6 on my local computer and now when I ping I get a IPV4 IP address it works now.
I am wondering if DNS is screwed up on the IPV6 network for Lemmy.ml.
~~Note. This could totally be something on my end, I really haven't done much with IPV6 but it did solve the 502 Error so I might do the same for you. ~~
Edit.
Multiple people are reporting the same thing I am seeing. It is defiantly something about IPV6 on the lemmy.ml server end.
Thanks for mentioning the IPv6, I've been banging my head all day trying to figure out why I kept getting the 502 yet no one was complaining anywhere and isitdown was showing the server as Up.
I forces my DNS to resolve only IPv4 for lemmy.ml and now I can use it.
My suspicion is that nginx is misconfigured and not listening via IPv6. Or maybe the AAAA record is pointing to the wrong IPv6 address.
Me and others are getting that issue without that. Seems to be related to its IPV6 configuration or AAAA record since disabling IPV6 seems to make lemmy.ml work
Hey, what do you mean by "scale horizontally"?
There are multiple approaches to tackle this.
Have multiple nodes/pods for the same instance and run them on a cloud-like service provider
have RO-instances to handle to read-load
share/merge bigger communities/subs over multiple instances
...
All of these requiere most likely a major rewrite/change of Lemmy server software I guess.
They are already addressed as issues/feature requests on github
In my opinion the first option would fit the most.
My comment was without knowing the topology of Lemmy at all, but my thoughts were initially that vertically scaling can have diminishing returns past a certain threshold. Since the servers seem to be struggling I'm wondering if that has been surpassed and if it's more cost-effective and reliable to scale this way? But if the application isn't written in that way, or the underlying data store isn't equipped for multiple instances then fair enough, I'd be interested as to why especially if Lemmy grows. I'll take a look at open issues and educate myself a bit more though.
And it's not even a super beefy one!
kbin.social uses a CCX41 with 16 virtual cores (I think 8 of those are hyperthreaded though) and 64gb of ram for its ~25k users
Are there still issues with cross-instance content? Previously, if I signed in to lemmy.ca and subscribed to a channel on .ml, I didn't see all content. Likewise, if I left a comment from my .ca account, it wouldn't necessarily show up for users on .ml.
If this is still a problem, it's a HUGE roadblock in being able to just tell people to join other instances, if we don't want to fracture an existing community.
Edit: That may have simply been a result of the excess load?
Edit 2: Okay I'm testing and it looks like this was just an issue of load; I don't seem to be having any further issues with this.
Edit 3: Comments from users on lemmy.world still don't seem to propagate correctly. I can see them from .ml but not .ca
I was wondering the same thing. I initially created an account on some obscure instance because I didn't fully understand what I was doing. I just abandoned it and set up a new one on lemmy world. I think I'm getting the hang of it now. I'm curious to see how Lemmy grows and matures over time. There is still a learning curve that will keep some people away.
Have we explored the possibility of "porting" the larger communities to other instances? It seems that many of us simply wish to subscribe to the largest (insert type of community here) and can do so from various home instances. Might lower demand on this specific instance at the very least.
Is most of the discussion about the infrastructure done in a community here, or do you folks have an invite only discord/slack or something @[email protected]?
I've been working in devops for 4 years now, but with mostly kubernetes and AWS, but I'd like to throw some ideas and just ask some more specific questions about the infrastructure. Like I'm curious why autoscaling the web servers and moving the database server to a dedicated instance is not the current configuration.
There is really not much to discuss. The server was overloaded so we got a bigger one. And there is no reason to mess with stuff like kubernetes when a single server works fine. After all our job is to improve the Lemmy software for everyone, not build a huge centralized platform only on lemmy.ml.
I agree with the kubernetes point. Not every problem needs it.
I was more curious about having a load balancer infront of an autoscaling group of servers with the existing images that you are running off of, minus the self hosted database server. Then you would be able to handle spikes automatically.
Just curious if that has been thought of.
Having it accessible to everyone is great. For sure.
But I was just thinking that having stronger pieces of infrastructure for instances that handle more traffic might be beneficial.
Exactly. lemmy.ml is lemmy.ml - just a single instance. there should be incentive for other to stand up instances of their own. overload on any instance is good reason for ppl to get out there and build new instances.
I can feel the difference already. Really enjoying it here and while I wouldn’t let some technical difficulties stop me, I think it will help grow the community. Thank you!
Thank you! I can actually participate now!! I have several screen shots saved of my profile showing me logged in under random user names over the last few days. . Shit was weird. I couldn't post, couldn't stay logged in, or I'd see a strange profile name if it actually let me click into a thread. Everything seems to be solid today.
Thank you for your hard work! Although I kinda foresee for the future if Lemmy really would become the new "reddit" with such servers and millions of users, wouldn't that also rise the server costs and ultimately make the hosts dependent on asking money for it, maybe by a paywall or by ads? I think to make this community really be "free" without any host responsible for spending a huge amount of money for servers, the best solution would be to make the actual "servers" be a p2p cluster. Unfortunately I'm not quite sure how to realize that without losing a huge fraction of the model if a lot of nodes (i.e., the actual users) are offline. Sorry, I'm just brainstorming.
You might want to check the Earthstar project: https://earthstar-project.org/ They are working on that. Right now, it's super early; they are building the foundations. Peer-to-peer is unfortunately much more difficult to code then servers, because less people have built the building-blocks required, and because mobile phones are actively making it hard to run peer-to-peer apps.
hi! I would be interested in understaning more the setup of the lemmy.ml instance. Do you use a cloud provider, a SaaS platform or a traditional hosting ? What are the costs that are incurred? Cheers!
Is Lemmy made with horizontal scaling (a.k.a. launching more instances and have a load balancer proxy the requests to the various instances) in mind? It could help larger instances like lemmy.ml managing the load better rather than just putting it on a beefier machine.
I'm still having trouble that any comments I make on other instances through lemmy.ml are not seen on those instances, only when viewed through lemmy.ml
I'm a little curious how difficult it would be to include this status in the sidebar - we see 'users online' statistics, we should also see 'costs vs donations' statistics there plus a link for donations also.
JFYI, it's getting an error 500 here and there when refreshing and sometimes it misses loading the css.
No idea if you are using a cache somewhere, could be that as well if you don't see anything weird in the monitoring.
Probably, but it's significantly more difficult to set up and maintain, and introduces new problems. I'm sure it'll be considered once they reach the limits of what a single node can handle.
I assume it's for the sake of more freedom. Without admins to make troubling choices like in the case of reddit, lemmy can't be corrupt so easily. I'm quite new to this as well, but from what I've seen, the most lemmy can do is unlink certain instances so they don't show up in your search, but instances may live or die, but the social network as a whole lives on (except if the killed instance housed your account).
hey 👋
do you have some Grafana dashboard where we can check user increase?
Given the no. of posts on this instance I guess it has quite a large user base and I’m really surprised how it manages to run on that amount of resources 😄