Search

Firstly, the maintenance for Pawb.Social services has been completed and we've upgraded Lemmy to 0.19.6-beta.13, and our Mastodon instances to v4.3.0.

Thanks for your patience.

----

and now, onto the "Toastify is awesome" (because it is. Totally.) postmortem:

Short summary: Crash did a stupid with the config and emails weren’t getting sent (oops). Emails should now be working again, and registration shouldn't require admin intervention.

Longer story:

Lemmy has several configuration options that need to be updated to correspond to the specific instance that's running it. For example, here's a snippet of Pawb.Social's config:

{ database: { host: "the.best.db.url.internal" user: "pawbsocial_lemmy" database: "pawbsocial_lemmy" password: "pawbsocial_do_you_really_think_i_would_leak_this" pool_size: 128 } hostname: "pawb.social" pictrs: { url: "http://pawb-social-pictrs:8080/" api_key: "sekrit" } email: { smtp_server: "email-smtp.us-east-2.amazonaws.com:587" smtp_login: "pls_send_emails" smtp_password: "jeff_im_begging_you" smtp_from_address: "Pawb.Social Lemmy <{{domain}}>" tls_type: "starttls" } }

Can you spot the problem? (No, it's not my awful password, or the pleading I'm doing with AWS to send emails so y'all stop having email problems.)

Originally, Lemmy was deployed through docker-compose which uses variable replacement to change something like {{domain}} to pawb.social automatically. Crash swapped away from using docker-compose due to repeated issues with trying to deploy the Lemmy changes and opted for a more bare Docker configuration.

Can you guess what didn’t get changed? …Yeah, the {{domain}} part was never removed and replaced with pawb.social so Lemmy correctly caught the error and threw an exception. The unfortunate part is that the thrown exception caused the whole comment process to close the connection to the web proxy server which returned a 502 Bad Gateway, which gave the UI HTML instead of a JSON response. The response handler in the UI then tried to figure out what was wrong, failed to translate the HTML to JSON, and gave the default error message “Toastify is awesome!” (which it is. Objectively.)

HOPEFULLY. This does fix the issue. I can’t guarantee it, but I’ve been able to replicate it by replying to someone else’s comment (which usually triggers an email notification) and after making the change, it successfully sent off the comment with a happy green confirmation toast.

We've successfully upgraded Lemmy from v0.19.3 to v0.19.5 and have a bunch of changes to share!

Bug Fixes

• Previously users were reporting that the web client kept logging them out. This was suspected to be due to a misconfiguration of the cookie storing the session information and should be fixed.

• Some communities were slow to load. Federation changes in v0.19.4 and v0.19.5 should rectify some federation issues. Additionally, we increased the number of database connections for Lemmy and will gradually increase this if we identify any further slowdowns.

(Longer term, we're planning to upgrade the database hardware which should improve the performance for all services)

Local Only Communities

When creating or editing your communities, you'll now see a "Visibility" option that allows you to choose between "Public" and "Local Only".

The "Public" setting will continue to operate as normal and federate your community to all other instances we federate with.

"Local Only" will disable federation for your community and restrict the community to being viewed by local logged in users only; Logged out users may not be able to see your community until they log in.

Image Proxying

Once we've established that the database is no longer under substantial load and the performance issues previously experienced are fixed, we'll be enabling an image proxying mode which will allow us to locally cache images from remote instances rather than serving them directly.

This will allow us to improve the performance of loading images without applying additional load to other instances, and prevent remote bad actors from trying to scrape IP addresses.

Other changes

There are a handful of other miscellaneous changes that you can view in the full change logs:

• https://join-lemmy.org/news/2024-06-07_-Lemmy_Release_v0.19.4-_Image_Proxying_and_Federation_improvements

• https://join-lemmy.org/news/2024-06-19_-Lemmy_Release_v0.19.5-_A_Few_Bugfixes