Meta

I've noticed a lot of broken images around the place and can't quite work out what's happening. It doesn't seem to be a caching issue, at least not on my end, as trying to visit the image URL directly returns a 404.

As noted in a post in the Melbourne daily thread a few weeks ago, there are also a lot of broken community icons about the place. And as I think was raised a while ago, thumbnails seem to be broken most of the time as well.

As an example, here's what the trending communities tab looks like: !

The communities page is a little better, but it's about 50/50 as to whether the icon does or doesn't load: !

Here's a list of my own posts with missing images I've noticed recently | Post | Image | | -------- | ------- | | https://aussie.zone/post/13071169 | https://aussie.zone/pictrs/image/2982b6d9-b84a-4c2f-8f3b-1f38ecd27241.png | | https://aussie.zone/post/12388426 | https://aussie.zone/pictrs/image/8638242e-4d62-4372-9a7f-15c913fc7b83.jpeg | | https://aussie.zone/post/11404847 | https://aussie.zone/pictrs/image/d22d1f8f-e288-4190-8df5-ef4e6756c396.jpeg | | https://aussie.zone/post/11369832 | https://aussie.zone/pictrs/image/a9d77a68-3a9b-4f4d-9508-fc6aefa14434.jpeg | | https://aussie.zone/post/11285649) | https://aussie.zone/pictrs/image/8a64f759-77e6-4e38-a739-2f875e90cbf0.png | | https://aussie.zone/post/10604574 | https://aussie.zone/pictrs/image/39091834-d6fd-45c1-9520-73b9b7be3529.jpeg | | https://aussie.zone/post/10981887 | https://aussie.zone/pictrs/image/b4df332b-82a9-4ec8-b115-ac1b1b1824b0.jpeg |

That last image seems to be super broken at the time of writing. Instead of just returning a generic 404, it returns: {"error":"unknown","message":"Request error: error sending request for url (http://pictrs:8080/image/original/b4df332b-82a9-4ec8-b115-ac1b1b1824b0.jpeg): connection error: Connection reset by peer (os error 104)"}

Additionally, this is how thumbnails on links have been previewing: !

Right-clicking and opening in new tab, it seems to be pulling a random pixel from somewhere in the image and then enlarging it and proxying it (500% zoom): !

Example URL

But then other URL thumbnails don't work at all, for instance this post on trending: !

Which links to Wikipedia, but fails for whatever reason, then tries to load an image called "broken-image-fallback.png", and, ironically, can't even load that (leads to a 404 page).

And some thumbnails on working images don't work, for example this post in c/GardeningAustralia doesn't load - even if I click on the thumbnail (which should normally load the full size/original image), but does work if I click the image URL itself.

One more thing: this new image proxy thing is getting super annoying. I realise it's a Lemmy-side implementation issue, but a couple of sites (Imgur, for example), don't like all the image requests from a single server/IP and end up rate limiting us. This effectively means that Imgur is unusable and no images load.

I actually thought there might have been another mass image purge like last years' incident which required the purging of 48 hours of images, but there hasn't been any announcements about it, and the other broken images dotted around the joint makes me think Pictrs is on its deathbed.

Any ideas what's going on?

Sidenote: image uploads whether in posts or as embedded images within posts/comments seem to fail about half the time for me on both mobile and desktop. I thought this was an issue with Boost again, but it's happening to me on PC as well. On mobile, it usually either returns no error at all, or a 500. On PC, no UI errors are displayed, it just doesn't embed the image, but network tools tell me that something somewhere is returning a 500, likely the image uploader thingy.

I thought the recent patches were supposed to fix the slow federation, but we're still not getting comments or vote tallies through for over a week.

My web dunce question of the week is,

Why would the images url be linked to aussie.zone when the community is blahaj.zone and the poster is lemm.ee?

Is it as simple as the lemm.ee poster might have seen it on aussie.zone and cross posted, or copied over from there?

Edit,

So, just checked this post on the sync app, (using Jerboa for the post), and sync has recognised the image as a URL? I'm confused, i loaded it in Jerboa like i'd normally load a screenshot as an image. Anyway, sorry if the picture doesn't display.

I'm just trying to understand the lemmy system a little better, not trying to highlight a perceived problem.

About to reboot the server, hold onto your hats.

I've been seeing various 5xx error codes (usually 504s, but occasionally 502) sporadically most of the day. It doesn't seem to be a constant thing, just every 5-10 or so loads/clicks, it errors out. I've also noticed it's sometimes taking a while to load. Seems 50/50 whether it takes 1-2 seconds, or 6-10 seconds to load.

Is this something on my end, or is today a lazy Sunday for the server, too?

(I took a screenshot, but I'm getting errors when I try to upload that, too. A classic "ffprobe failed with exit code 1: success")

I'm the developer of Fediverser Project, which is a set of services to make it easy for people on Reddit to migrate to the Fediverse. It lets people use their Reddit credentials (OAuth) to sign up and create an account on a Lemmy server.

It also offers a cool onboarding feature: during signup, we can fetch the user's subscribed subreddits, and we use this information to automatically subscribe them to the corresponding Lemmy (or Kbin/Mbin) community. This "subreddit -> fediverse group" map is crowdsourced and people can sign up if they want to contribute. The "main" site also provides a "Find an instance" feature: it can track all the servers that use the Fediverse software and redirect users to their closest instance.

To enable this service, the Lemmy admin needs to add a couple of docker services to their setup and needs to get their own Reddit API key (which is used only for authentication, so well within the rate limits and certainly not incurring any prices).

I'd really like to see aussie.zone becoming part of the network. I believe this would make it faster and simpler to get more people in the fediverse, and I'm willing to provide all the support and help needed to get the "country-based" services getting started with it.

Any questions, don't hesitate to ask.

This is a simple software bug. You see, the way Lemmy federates is that updates to a community only federate to instances that have a user subscribed to that community. If you take a community like [email protected], I'm the only person on all of Aussie Zone who subscribed to it. I know it because when I subscribed, we started getting updates from Blahaj.Zone about that community.

A few days ago, I was mistakenly banned from that community. And this caused Blahaj.Zone to stop sending our instance updates on the community. I spoke with the admin of the community and got unbanned, but the unban didn't federate. The instance will only let us know I'm unbanned if someone here subscribes over there. And I can't subscribe, because this server thinks I'm banned. It's a deadlock. If it wasn't for the ban, I could subscribe and the server would realise there isn't a ban anymore. But I can't, because this server thinks there is still a ban.

When I try and upload an image for my avatar, I receive the message:

{"data":{"files":null,"msg":"ffprobe Failed with exit status: 1"},"state":"success"}

This happened with both a WebP and a PNG file, so I suspect it's a bug.

I can upload images fine here, and when I export my account details it shows a valid image URL for my avatar.

!

So, as per images below, when you search for an Australian community associated with lemmy, lemmy.world is more likely to come up than Aussie Zone in all i've tried, bar Melbourne our most active community.

My question: Is this a problem we should consider intentional action to correct? And if so what could we do?

!

!

!

Sorry if this has been asked/answered previously, but it just struck me that I hadn't really noticed any impacts of CloudStrike on aussie.zone.

Was wondering if it was a non-event, or maybe there are some war stories with heroes deserving medals!

So, I've just done something that I said I'd be reluctant to do in future without community consultation and blocked an instance.

I don't think anyone will have any issues with it, but I figured I'd let everyone know since this executive decision affects all aussie.zone users.

Hate content

!

Would people be interested in a community dedicated to discussing lemmy's in general but, specifically aussie zone's growth?

I'm thinking a pretty wide ranging set of ideas including,

• exposure and promotion of fediverse-lemmy-aussie zone,
• discussions on server structures and dispersal of user-bases,
• philosophical discussions on growth as an objective in the case of lemmy,
• discussions on different platforms aussie zone users might want to gather round as the potential use cases of the federated social web start to be explored.

I'm thinking like [email protected] , but a sister community that has regard to aussie zone user's corner of the fediverse.

To be used by aussie zone as a general compass setting and project brainstorming community for how we might like to develop, if at all.

Or is [email protected] the better place for these sorts of discussions?

There's no communities I think need better moderation, or any that have no active mods (that I know of), but if there is one day, can somebody make a request to have ownership or moderation powers transferred over to them?

Hey all, following the work over the weekend we're now running Lemmy 0.19.4. please post any comments, questions, feedback or issues in this thread.

One of the major features added has been the ability to proxy third party images, which I've enabled. I'll be keeping a closer eye on our server utilisation to see how this goes...

This weekend I'll be working to upgrade AZ to lemmy 0.19.4, which requires changes to some other back end supporting systems.

Expect occasional errors/slowdowns, broken images etc.

Once complete, I'll be making further changes to enable/tweak some of the new features.

UPDATE: one of the back end component upgrades requires dumping and reimporting the entire lemmy database. This will require ~1 hour of total downtime for the site. I expect this to kick off tonight ~9pm Perth time.

UPDATE2: DB dump/re-import going to happen ~6pm Perth time, ie about 10 minutes from this edit.

UPDATGE3: we're back after the postgres upgrade. Next will be a brief outage for the lemmy upgrade itself... after I've had dinner 🙂

UPDATE34: We're on lemmy 0.19.4 now. I'll be looking at new features/settings and playing around with them.

I know there's been a few posts about this before, but it's been a month since the last one and it's still ongoing. It doesn't seem that any of the LW admins responded to Zag's post on their help community, and the last response from lodion/Nath I'm aware of was from 4 months ago when there were outright federation failures as opposed to just lengthy delays.

@[email protected] posted a comment on the post from last month about the delays stating that it's an issue on our end as our server isn't keeping up. I'm not sure whether this is the case or not, and I'm not sure how to interpret the Grafana dashboard they linked to, but as it's a new reply on an old post, I wanted to note it.

Current federation delays seem to be around 7 days. It doesn't seem to be affecting posts themselves on Lemmy.world communities, but does affect all replies to them (even from users on other instances), and all upvotes on the posts. [Edit: on further investigation, this isn't the case. The current delays are at least 13 days, and this does actually affect posts too]

I don't want to sound too pushy, since the LW admins and Lodion/Nath are all volunteers, but I was hoping we might be able to get an update on what the cause is, and if it's an issue in Lemmy itself, if anybody's opened an issue on GitHub and the developers are aware.

(NB: I don't interact that much with LW, so all of my testing has been on the Boost for Lemmy community.)

Hey all, can anyone tell me the date we're gona land on for the first anniverssary?

I guess it'll be when the instance was spun up? Or i don't know, maybe when it was a twinkle in Lodion's eye?

Its been a little while since I posted stuff :)

CPU: !

Memory: !

Network: !

Storage: !

Cloudflare caching: !

Comments: Not much has changed in quite a while. I still have a cron-job running to restart Lemmy every day due to memory leaks, hopefully this improves with future updates. Outside of that, CPU, memory and network usage are fine. Object storage usage is growing steadily, but we're a long way from paying more than the monthly minimum Wasabi fee.

I know when things first kicked off we would get server updates semi-regularly such as finances and server load. I can’t say I’ve seen one for a while, just wondering how things are going?

Posted a query in the LW support channel to see if anyone there has any idea.

Just thought I'd give people a heads-up. I've just noticed some upvotes and posts haven't federated from lemmy.world to aussie.zone .

Hopefully it's just delays again, and they'll work their way through the queue reasonably soon.

Comments, votes don't appear to be federating for one of the last posts I made over at [email protected].

https://aussie.zone/post/7871931

https://lemmy.world/post/13089616

Can we get a tech com please? I don't mind modding. I know the bigger insts have very large tech coms but I figure a place for locals to post about tech without the whole fv watching could also come in handy. Probably best to call it technology but I don't mind tech either. Whatever yall want to roll with.

I'm still seeing cases where comments don't seem to be federating correctly. For example:

| Other instance | aussie.zone | | ---- | ---- | | https://lemmy.world/comment/6786498 | https://aussie.zone/comment/5886332 | | https://feddit.uk/post/6819925 | https://aussie.zone/post/6020867 | | https://lemmy.ml/comment/7476880 | https://aussie.zone/comment/5881947 | | https://lemmy.blahaj.zone/comment/5759107 | https://aussie.zone/comment/5888459 | | https://midwest.social/comment/6313276 | https://aussie.zone/comment/5889962 |

I'm not sure what the cause is, but the last few days it seems more of my comments are failing to federate than are succeeding.

Hopefully this will be the end of our federation woes

We’ve already defederated one tanky instance, and frankly I think I’ve rarely had a positive interaction with Hexbear users. Their defence of Russia’s invasion of Ukraine, or of China’s genocide against the Uighur people, is hard enough to stomach. But what prompted me to post this is their commentary in this thread where even without any of the propaganda-based reasons to justify their beliefs, they seem to delight in being rude arseholes to people.

Personally, I don’t think they provide us any real value, no we’d be better off without them, just as we are better off without lemmygrad and exploding-heads.

Not sure if this is a .19 bug or has always existed, but on the inbox page, if I have no unreads, click "all" then click back to unreads it shows DMs despite all DMs being read and the DM filter not even being selected

no unreads

!

click all

!

now in unreads DMs show up

!

The upgrade to lemmy 0.19 has introduced some issues that are being investigated.. but we currently have no fixes for:

• thumbnails break after a time. Caused by memory exhaustion killing object storage processes.
• messages to/from lemmy instance not yet running 0.19 are not federating. I believe it requires bugfixes by the devs.

~~I've re-enabled 2 hourly lemmy restarts. Hopefully this will help with both issues, though it will result in a disruption to the site around every couple of hours.

When the hourly restarts are disabled I'll unpin this post. As any other issues are identified I'll post them here too. ~~

Update: I've disabled the 2 hourly restart after upgrading to 0.19.2... lets see how this goes...

Update2: no issues seen since the upgrade, looks to have resolved both the memory leak and the federation issues. Hooray :)

I'll be restarting AZ today for an update to lemmy 0.19.

Upgrade complete.

This is a major upgrade, so I expect there to be some issues. Strap in, enjoy the ride.

Expect:

• further restarts
• bugs
• slowdowns
• logouts
• 2FA being disabled
• possibly issues with images, upgrading pictrs to 0.4 at the same time

Are there plans to switch to lemmy 0.19? we are still on 0.18.4, and the new version seems to have alot of useful features. Are there any problems stopping us from updating?

Meta threads will open federation to the fediverse soon, and while this is mainly to mastodon it will still affect lemmy. They are acting like they won't be evil, but let's be real this is Facebook when have they ever done that.

This article which has been trending lately explains some of the issues. https://ploum.net/2023-06-23-how-to-kill-decentralised-networks.html

This comment here is a simple analogy if you can't be bothered reading the article. https://lemmy.ca/comment/5702922

@[email protected]

This isn't a question specifically about aussie.zone, per se, but I'm a bit confused about how subscriber numbers work on Lemmy. I often see a community that says it only has 40 or whatever subscribers, but then posts get hundreds of upvotes. So is the member number shown on communities just counting subscribers from our instance?

I noticed recently there's some slowness in the mornings. Things taking a while to load. Comments failing due to timeout etc.

When I switch to another instance everything works again so pretty sure it's not my internet.

Is there anything we can do or help with? Are there DOS or bot shenanigans going on?

Heya, could we get a SydneyTrains community created please? Thanks

I used to run R/BrisbaneTrains on spez's bullshit playground. Wondering if I can run it here.

I'm kicking off a storage upgrade on the server, expect it to proceed in the next 15 minutes and be back online shortly after.

Hi,

I was curious, after seeing quite a few spam posts originate from one particular instance, is there a way that I, as a user, can block instances, so that their users' posts don't appear on my feed completely?

Cheers