You're viewing a single thread.
Ok, that got me. I still remember the days of ZX and that funny noise... But I do have a question for one part of the meme: can someone explain to me why on Earth the updates now weigh these tens of gigs? I can accept that hires textures and other assets can take that space, but these are most likely not the bits that are being updated most of the time. Why don't devs just take the code they actually update and send that our way?
The trick is to download the Fitgirl repack. Cheaper on your wallet and your hard drive.
I am perfectly fine with paying developers, as I buy only the games i do like after some testing ) Going the repack route is unpredictable - no updates, may contain whatever viruses repacker is interested in adding (and given the particular one is likely Russian, I do have my reservations at this crazy time...), etc.
Joking aside, when you download an update, many times it is completely replacing chunks of the game, not just a couple lines of code.
I mean, I understand when they chuck everything into a single file, but they used to know how to make their updaters unpack and replace only the stuff that needed updating, instead of just throwing the whole fucking file at you, redundancy be damned.
For instance, stuff in Quake 3 engine is kept in .pk3 files. You don't need to download the full, newest .pk3, you send a command to remove/replace files X, Y and Z within it and call it a day.
Yeah but then people go completely ballistic when games require you to install their own launches I don't think Steam would necessarily be able to handle the myriad of different formats that would be needed to make that work. So either you have custom lunches or you don't have particularly efficient patches.
I guess most people care more about the launcher.
That's because games don't need "their own launcher" to apply updates like that. Ask anyone that's been playing on PC, patches were these self extracting files or "mini installers" that you just needed to point to the installed game's folder. Even vanilla World of Warcraft let people download the patches for offline install, it even included a text with all the changes applied.
People don't want the fiddling on of that. I just want to be able to install the patch and then it be there.
That's why lunches are a thing there's no other reason to have them.
You might enjoy the technical solution but 99% of people don't care. I never understand why people seem to think that the 1% of the most experienced people are the standard when they are anything but. Most gamers build their own PCs, they don't want to have to understand about file systems and formats and compiling. They just wanted to work and then they want to play their game.
For modern games, from what I've seen, they've taken a more modular approach to how assets are saved. So you'll have large data files which are essentially full of compressed textures or something. Depending on how many textures you're using and how many versions of each textures is available (for different detail levels), it can be a lot of assets, even if all the assets in this file, are all wall textures, as an example.
So the problem becomes that the updaters/installers are not complex enough to update a single texture file in a single compressed texture dataset file. So the solution is to instead, replace the entire dataset with one that contains the new information. So while you're adding an item or changing how something looks, you're basically sending not only the item, but also all similar items (all in the same set) again, even though 90% didn't change. The files can easily reach into the 10s of gigabytes in size due to how many assets are needed. Adding a map? Dataset file for all maps needs to be sent. Adding a weapon or changing the look/feel/animation of a weapon? Here's the entire weapon dataset again.
Though not nearly as horrible, the same can be said for the libraries and executable binaries of the game logic. This variable was added, well, here's that entire binary file with the change (not just the change). Binaries tend to be a lot smaller than the assets so it's less problematic.
The entirety of the game content is likely stored in a handful (maybe a few dozen at most) dataset files, so if any one of them change for any reason, end users now need to download 5-10% of the installed size of the game, to get the update.
Is there a better way? Probably. But it may be too complex to accomplish. Basically write a small patching program to unpack the dataset, replace/insert the new assets, then repack it. It would reduce the download size, but increase the amount of work the end user system needs to do for the update, which may or may not be viable depending on the system you've made the game for. PC games should support it, but what happens if you're coding across PC, Xbox, PlayStation, and Nintendo switch? Do those consoles allow your game the read/write access they need to the storage to do the unpacking and repacking? Do they have the space for that?
It becomes a risk, and doing it the way they are now, if you have enough room to download the update, then no more space is needed, since the update manager will simply copy the updated dataset entirely, over the old one.
It's a game of choices and variables, risks and rewards. Developers definitely don't want to get into the business of custom updates per platform based on capabilities, so you have to find a solution that works for everyone who might be running the game. The current solution wastes bandwidth, but has the merit of being cross compatible, and consistent. The process is the same for every platform.
The console argument does actually make a lot of sense to me, thank you for the detailed response. It would still (seemingly) be possible to structure the project in a way that would allow replacing only what you actually need to replace, but that requires more investment in the architecture and likely cause more errors due to added complexity. Still, i cannot forgive the BG 3 coders for making me redownload these 120gb or so! )
The issue is the compression. There's hundreds of individual assets, the process to compress or more accurately, uncompress the assets for use takes processor resources. Usually it only really needs to be done a few times when the game starts, when it loads the assets required. Basically when you get to a loading screen, the game is unpacking the relevant assets from those dataset files. Every time the game opens one of those datasets, it takes time to create the connection to the dataset file on the host system, then unpack the index of the dataset, and finally go and retrieve the assets needed.
Two things about this process: first, securing access to the file and getting the index is a fairly slow process. Allocating anything takes significant time (relative to the other steps in the process) and accomplishes nothing except preparing to load the relevant assets. It's basically just wasted time. The second thing is that compressed files are most efficient in making the total size smaller when there's more data in the file.
Very basically, the most simple compression, zip (aka "compressed folders" in Windows) basically looks through the files for repeating sections of data, it then replaces all that repeated content with a reference to the original data. The reference is much smaller than the data it replaces. This can also be referred to as de-duplication. In this way if you had a set of files that all contained mostly the same data, say text files with most of the same repeating messages, the resulting compression would be very high (smaller size) and this method is used for things like log files since there are many repeating dates, times, and messages with a few unique variances from line to line. This is an extremely basic concept of one style of compression that's very common, and certainly not the only way, and also not necessarily the method being used, or the only method being used.
If there's less content per compressed dataset file, there's going to be fewer opportunities for the compression to optimize the content to be smaller, so large similar datasets are preferable over smaller ones containing more diverse data.
This, combined with the relatively long open times per file means that programmers will want as few datasets as possible to keep the system from needing to open many files to retrieve the required data during load times, and to boost the efficiency of those compressed files to optimal levels.
If, for example, many smaller files were used, then yes, updates would be smaller. However, loading times could end up being doubled or tripled from their current timing. Given that you would, in theory, be leading data many times over (every time you load into a game or a map or something), compared to how frequently you perform updates, the right choice is to have updates take longer with more data required for download, so when you get into the game, your intra-session loads may be much faster.
With the integration of solid state storage in most modern systems, loading times have also been dramatically reduced due to the sheer speed at which files can be locked, opened, and data streamed out of them into working memory, but it's still a trade-off that needs to be taken into account. This is especially true when considering releases on PC, since PC's can have wildly different hardware and not everyone is using SSDs, or similar (fast) flash storage; perhaps on older systems or if the end user simply prefers the less expensive space available from spinning platter hard disks.
All of this must be counter balanced to provide the best possible experience for the end user and I assure you that all aspects of this process are heavily scrutinized by the people who designed the game. Often, these decisions are made early on so that the rest of the loading system can be designed around these concepts consistently, and it doesn't need to be reworked part way through the lifecycle of the game. It's very likely that even as systems and standards change, the loading system in the game will not, so if the game was designed with optimizations for hard disks (not SSDs) in mind, then that will not change until at least the next major release in that games franchise.
What isn't really excusable is when the next game from a franchise has a large overhaul, and the loading system (with all of its obsolete optimizations) is used for more modern titles; which is something I'm certain happens with most AAA studios. They reuse a lot of the existing systems and code to reduce how much work is required to go from concept to release, and hopefully shorten the duration of time (and the amount of effort required) to get to launch. Such systems should be under scrutiny at all times whenever possible, to further streamline the process and optimize it for the majority of players. If that means outlier customers trying to play the latest game on their WD green spinning disk have a worse time because they haven't purchased an SSD, when more than 90% + have at least a SATA SSD, all of whom get the benefits from the newer load system while obsolete users are detrimented because of their slow platter drives, then so be it.
But I'm starting to cross over into my opinions on it a bit more than I intended to. So I'll stop there. I hope that helps at least make sense of what's happening and why such decisions are made. As always if anyone reads this and knows more than I do, please speak up and correct me. I'm just some guy on the internet, and I'm not perfect. I don't make games, I'm not a developer. I am a systems administrator, so I see these issues constantly; I know how the subsystems work and I have a deep understanding of the underlying technology, but I haven't done any serious coding work for a long long time. I may be wrong or inaccurate on a few points and I welcome any corrections that anyone may have that they can share.
Have a good day.
I've got 2gig fiber, not 56k dialup. It's Steam's bandwidth now. They paid Valve their 30%. Why bother with insane compression that just makes it feel slow for us?
That is also a factor I do not understand. Bandwidth costs the storefront money, would Steam and others not want to decrease this load? And well done you with that fiber, you dog! I also have a fiber line but see no reason to upgrade from my tariff (150mib, i think?) that covers everything just to shave that hour of download time a year.