These log settings are not very good for production. The DB would spend more time logging, then working.
But: We can simulate this kinds of load with local toolings. My first tests look quiet promising, if i would only find the bug in my old pg-exporter i wrote for mal last employer ;) (yes, it is open source).
Yes, i see this issue also. I would assume that the statements used here, tend to get very bad plans, due to overhang (specific id's will have far more entries then others).
This is one of the reasons for my current setup.
But when it comes to optimizing databases, i think i'm pretty skilled in it, and i have seen much worse scenarios (billing systems, processing > 100.000.000 entries per billing run, with tough time constraints).
Nice! This would greatly help to populate the database and get much better results!
Maybe @[email protected] can shed some light on this. Not sure if the find time, but it would be great to get some more input.
In theory, yes. You only need a Activitypub library and some lines of code.
Current workaround for the Rate limit issue: https://gitea.loki.codes/lemmy-performance/load-test/src/branch/main/pkg/instance/prepare.go#L19
This is not "gmails" behavior. This is called "verb" and it is one of the best things ever. I only use verb for signing up at stuff. So, i can sort, and know when someone misuses my mail address.
And, if i would need to create an alt, to post stuff, that i don't want somebody do bring in connection with me (TIFU, NSFW, whatever you like) i would use verb.
In the site settings, rate limiting can be configured.
But, yes there should be some way to see if limits where hit or not. Maybe this could be done via prometheus and just provide the option to gather this data outside of lemmy itself.
Lemmy Benchmarking - Working on tooling
Heyho,
as a PostgreSQL guy, i'm currently working an tooling environment to simulate load on a lemmy instance and measure the database usage.
The tooling is written in Go (just because it is easy to write parallel load generators with it) and i'm using tools like PoWA to have a deep look at what happens on the database.
Currently, i have some troubles with lemmy itself, that make it hard to realy stress the database. For example the worker pool is far to small to bring the database near any real performance bootlenecks. Also, Ratelimiting per IP is an issue.
I though about ignoring the reverse proxy in front of the lemmy API and just spoof Forwarded-For headers to work around it.
Any ideas are welcome. Anyone willing to help is welcome.
Goals if this should be:
- Get a good feeling for the current database usage of lemmy
- Optimize Statements and DB Layout
- Find possible improvements by caching
As your loved core devs for lemmy have large Issue Tracker backlog, some folks that talk rust fluently would be great, so we can work on this dedicated topic and provide finished PR's
Greatings, Loki (@tbe on github)
First of all: I'm posting this from my .ml alt. Because i can not do it from my .world main. That i can't do it, i found out just because i was waiting for a response on a comment where is was sure that the OP would respond. After searching, i found out that my comment and my DM's never where federated to .ml.
So, that said: I'm all for defederating bad instances, i'm all for separation where it makes sense. BUT:
- If an instance is listed on join-lemmy, this should work as the normal user would expect
- We are not ready for this yet. We are missing features (more details below)
- Even instances that officialy require applications, can be spam instances (admins can do what ever they want), so we would need protection against this anyways. Hell, one could just implement spam bots that talk directly federation protocol, and wouldn't even need lemmy for this ...
Minimal features we need:
- Show users that the community they try to interact with is on a server that defederated the users instance
- Forbid sending DM's to servers that are not fully federated
Currently, all we do is: Make lemmy look broken
And before someone starts with: "Then help!", i do. I do in my field of expertice. I'm a PostgreSQL Professional. So i have build a setup to messure the lemmy SQL performance, usage patterns, and will contribute everything i can to make lemmy better.
(I tried rust, but i'm to much C++ guy to bring something usefull to the table beyond database stuff, sry :( )