Tiff @reddthat.com Self Proclaimed Internet user and Administrator of Reddthat
gaining access to anyones browser without them even visiting a website
What a good browser...
a post going over 4 exploits for CVE-2024-20017, a remotely exploitable buffer overflow in a component of the MediaTek MT7622 SDK.
This is sso support as the client. So you could use any backend that supports the oauth backend (I assume, didn't look at it yet).
So you could use a forgejo instance, immediately making your git hosting instance a social platform, if you wanted.
Or use something as self hostable like hydra.
Or you can use the social platforms that already exist such as Google or Microsoft. Allowing faster onboarding to joining the fediverse. While allowing the issues that come with user creation to be passed onto a bigger player who already does verification. All of these features are up for your instance to decide on.
The best part, if you don't agree with what your instance decides on, you can migrate to one that has a policy that coincides with your values.
Hope that gives you an idea behind why this feature is warranted.
Possibly, as it's one generic endpoint, but it also blocked a few other things people in the fediverse created, which are mighty helpful in diagnosis of these and other issues.
So using some AI model or whatever CF uses is probably not going to be the best thing for us as it classified a POST request as a crawler?? 🤷
I'd have to whitelist every regular endpoint as well and then it gets messy as CF only gives you so much control as a free user.
So, for the moment I've blocked the most annoying ones based on UserAgent.
We enabled the CloudFlare AI bots and Crawlers mode around 0:00 UTC (20/Sept).
This was because we had a huge number of AI scrapers that were attempting to scan the whole lemmyverse.
It successfully blocked them... While also blocking federation 😴
I've disabled the block. Within the next hour we should see federation traffic come through.
Sorry for the unfortunate delay in new posts!
Tiff
latchset/clevis: Automated Encryption Framework
Automated Encryption Framework. Contribute to latchset/clevis development by creating an account on GitHub.
to be paired with tang
Social media giants would be forced to ban children under the age of 14 from their platforms or face hefty penalties, under proposed laws in South Australia that could be replicated in other states.
Highly relevant to us (as admins)
broadcast-box: A broadcast, in a box.
A broadcast, in a box. . Contribute to Glimesh/broadcast-box development by creating an account on GitHub.
Not so much a sploit but an easy way to do broadcasting!
Outage: 2/Sept ~29 minutes
We had a brief outage today due to the server running out of space.
I have been tracking our usage but associated it with extra logging and the extra build caches/etc that we've being doing.
Turns out the problem was the frontend Next-UI which has been caching every image since the container was created! All 75GB of cached data!
Once diagnosed it was a simple solution to fix. I'm yet to notify the project of this error/oversight and I'll edit this once Issues/PRs are created. I also haven't looked at turning the caching off yet as my priority was recovering the main Reddthat service.
Thanks all for being here!
Tiff
I can neither confirm or deny for the safety of my pigeons.
I've always wanted to do colocation and looked into it when I first started Reddthat and we had our initial growth. We are lucky we didn't otherwise I would be out of money ages ago!
The aruba.it colocation is about the same price as our OVH server which would be probably the most viable as we are close to having enough donators to have a long enough runway it would make sense.
(Goal being: A$150/m ~ USD$100/€90. And OpenCollective doing some magical nonsense with the Total amount we have left and averaging it across 12 months or something)
The 32GB of RAM we have currently is overkill for an instance of our size. We could get away with using 16GB and still have room to grow for the next year. Which will probably be what will happen in April, but I'll re-evaluate before then and see whether a colocation option is viable.
Our stance on downvotes, and lack of super-popular local communities means we are growing slowly over time and are very stable in requirements. Lemmy still being in it's infancy means there are problems that are no fault of the hardware but the software. An example would be the latest issue we were experiencing where no matter what resources we had, it would result in 100% CPU usage for brief periods, with 10-20s page loads for everyone during that period.
I am very tempted but I still need to make sure I am being responsible with the donations. By next year we'll most likely be over 1TB of object storage and our S3 costs will slowly start increasing MoM. While not a lot I still need to factor it in growing costs.
I low key like crypto too...
Recently I've taken the docker compose example from SChernykh and have started a p2pool for Reddthat!
https://github.com/SChernykh/p2pool/tree/master/docker-compose (many thanks here!). After some minor changes I removed the IP listing from statistics and increase the visibility to 100 "supporters". It's viewable at donate.reddthat.com. (if @[email protected] wants the code change I can provide diff)
The idea was to also allow people to donate to instances via CPU instead of actual $.
My question for the community is whether I am creating a centralised pool or am I still participating in a decentralized fashion?
I recently had solar panels and a battery storage system from GivEnergy installed at my house. A major selling point for me was that they have a local network API which can be used to monitor and…
Yeah! Except it wasn't just images. Instead it was all links. And inside the Lemmy process and it would block all other connections! Which is the why it was an issue! Super weird.
But the whole Lemmy app is full of these issues hiding in the background. People seem to forget it's still in its infancy and expect a 1-1 Reddit experience. We regularly get people signing up and then quitting like the thread from last week regarding the lack of downvotes. Or the lack of X,Y,Z.
</end rant>
It's fixed for Reddthat now, which is the best part! 😉
I do! It's already in the sidebar!
BTC Directly: bc1q8md5gfdr55rn9zh3a30n6vtlktxgg5w4stvvas
Tony Hawk's Pro Strcpy: A game save and RCE exploit for the Tony Hawk game series that can be used to hack Xbox, Playstation 2, Gamecube, and Xbox 360 consoles.
The long loads are because of huge images/content believe it or not 😂. I too can't wait to finally see some nice fast Reddthat
Whoa! Thanks for even considering donating. I won't hold you to it if you happen to donate less later 😉
After testing ko-fi we still end up having the same fees compared with OpenCollective as it's PayPal instead of Stripe. So in the end it's better to go via OpenCollective. As it's a lot more transparent and shows all the donations and will allow me to show all of our bills etc.
Thanks!
Reddthat Update: August-ish 2024
Hello. It is I, Tiff. I am not dead contrary to my lack of Reddthat updates 😅 !
It's been a fun few months since our last update. We've been mainlining those beta releases, found critical performance issues before they made it into the wider Lemmyverse and helped the rest of the Lemmyverse update from Postgres 15 to 16 as part of the updates for Lemmy versions 0.19.4 and 0.19.5!
Thank-you to everyone who helped out in the matrix admin rooms as well as others who have made improvements which will allow us to streamline the setup for all future upgrades.
And a huge thank you to everyone who has stuck around as a Reddthat user too! Without you all this little corner of the world wouldn't have been possible. I havn't been as active as I should be for Reddthat, moderating, diagnosing issues and helping other admins has been taking the majority of my Reddthat allocated time. Creating these "monthly" updates should... be monthly at least! so I'll attempt start posting them monthly, even if nothing is really happening!
High CPU Usages / Long Load Times
Unfortunately you may have noticed some longer page load times with Reddthat, but we are not alone! These issues are with Lemmy as a whole! Since the 0.19.x releases many people have talked about Lemmy having an increase in CPU usage, and they have the monitoring to prove it too. On average there was a 20% increase and for those who have single user instance this was a significant increase. Especially when people were using a raspberry pi or some other small form factor device to run their instance.
This increase was one of the many reasons why our server migrations were required a couple months ago. There is good news believe it or not! We found the issue with the long page load times, and helped the developers find the solution! -
This change looks like it will be merged within the next couple days. Once we've done our own testing, we will backport the commit and start creating our own Lemmy 'version'. Any backporting will be met with scrutiny and I will only cherry-pick the bare minimum to ensure we never get into a situation where we are required to use the beta releases. Stability is one of my core beliefs and ensuring we all have a great time!
Donation Drive
We need some recurring donations!
We currently have $374.10 and our operating costs have slowly been creeping up over the course of the last few months. Especially with the current currency conversions. The current deficit is $74. Even with the amazing 12 current users we will run out of money in 5 months. That's January next year! We need another 15 users to donate $5/month and we'd be breaking even. That's 1 coffee a month.
If you are financially able please see the sidebar for donation options, go to our funding post , or go directly to our Open Collective and signup for recurring donations!
Our finances are viewable to all and you can see the latest expense here: https://opencollective.com/reddthat/expenses/213722
- OVH Server (Main Server) - $119.42 AUD
- Wasabi S3 (Image Hosting) - $16.85 AUD
- Scaleway Server (LemmyWorld Proxy) - $6.62 AUD
Scaleway
Unfortunately until Lemmy optimises their activity sending we still need a proxy in EU, and I havn't found any server that is cheaper than €3.99. If you know of something with 1GB RAM with an IPv4 thats less than that let me know. The good news is that Lemmy.ml is currently testing their new sending capabilities so it's possible that we will be able to eventually remove the server in the next year or so. The biggest cost in scaleway is actually the IPv4. The server itself is less than €1.50 so if lemmy.world had IPv6 we could in theory save €1.50/m. In saying all this, that saving per month is not a lot of money!
Wasabi
Wasabi S3 is also one of those interesting items where in theory it should only be USD$7, but in reality they are charging us closer to USD$11. They charge a premium for any storage that is deleted before 30 days, as they are meant to be an "archive" instead of a hot storage system.
This means that all images that are deleted before 30 days incur a cost. Over the last 30 days that has amounted to 305GB! So while we don't get charged for outbound traffic, we are still paying more than the USD$7 per month.
We've already tried setting the pictrs caching to auto-delete the thumbnails after 30 days rather than the default 7 days, but people still upload and delete files, and close our their accounts and delete everything. I expect this to happen and want people to be able to delete their content if they wish.
OVH Server
When I migrated the server in April we were having database issues. As such we purchased a server with more memory (ram) than the size of the database, which is the general idea when sizing a database. Memory: 32 GB. Unfortunately I was thinking on a purely technical level rather than what we could realistically afford and now we are paying the price. Literally. (I also forgot it was in USD not AUD :| )
Again, having the extra ram gives us the ability to host our other frontends, trial new features, and ensure we are going to be online incase there are other issues. Eventually we will also increase our Lemmy applications from 1 to 2 and this extra headroom will facilitate this.
Donate your CPU! (Trialing)
If you are unable to donate money that is okay too and here is another option! You can donate your CPU power instead to help us mine crypto coins, anonymously! This is a new initiative we are trialing. We have setup a P2Pool at: https://donate.reddthat.com. More information about joining the mining pool can be found there. The general idea is: download a mining program, enter in our pool details, start mining, when our pool finds a "block", we'll get paid.
I've been testing this myself as an option of the past month as a "side hustle" on some laptops. Over the past 30 days I managed to make $5. Which is not terrible if we can scale it out. If it doesn't takes off, that's fine too!
I understand some people will be hesitant for any of the many reasons that exist against crypto, but any option to help us pay our server bills and allow people to donate in an anonymous way is always a boon for me.
Conclusion
These Lemmy bugs have been causing a headache for me in diagnosing and finding solutions. With the upcoming 0.19.6 release I hope that we can put this behind us and Reddthat will be better for it.
Again, thank you all for sticking around in our times of instabilities, posting, commenting and engaging in thoughtful communications.
You are all amazing.
Cheers,
Tiff
Cloudstrike Channel File 291 Incident Root Cause Analysis 08.06.2024
the "official" incident root cause analysis
Keys were labeled "DO NOT TRUST." Nearly 500 device models use them anyway.
An article from July, but I bet you haven't updated your bios! Or you left it open on purpose?
A really good article on how TPMs work
Yeah that's why I included the other "main posts"... Their technical details really didn't say anything technical
Learn more about the July 19, 2024 CrowdStrike outage and the technical details related to it.
The update that broke half of all enterprise servers. One of the official remediation steps is to "Reboot as many as 15 times"
Read more: https://community.intel.com/t5/Intel-vPro-Platform/Remediate-CrowdStrike-Falcon-update-issue-on-Windows-systems/m-p/1616593/thread-id/11795
- https://old.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/
- https://repost.aws/en/knowledge-center/ec2-instance-crowdstrike-agent
- https://azure.status.microsoft/en-gb/status
regreSSHion: CVE-2024-6387
Update your openssh, now
Happy Reddthat Day!
It's our Birthday! 🎂
It's been a wild ride over the past year. I still remember hearing about a federated platform that could be a user driven version of Reddit.
And how we have grown!
Thank you to everyone, old and new who has had an account here. I know we've had our ups and downs, slow servers, botched migrations, and finding out just because we are on the otherside of the world Lemmy can't handle it, but we are still here and making it work!
If I could go back and make the choice again. Honestly, I'd probably make the same choice. While it has been hectic, it has been enjoyable to the n'th degree. I've made friends and learnt a lot. Not just on a technical level but a fair amount on a personal level too.
We have successfully made a community of over 300 people who regularly use Reddthat as their entryway into the Lemmyverse. Those numbers are real people, making real conversations.
Here's to another amazing year!
Tiff
PS: I'm still waiting for that first crypto donation 😜
PPS: Would anyone like a Hoodie with some sort of Reddthat logo/design? I know I would.