PDA

View Full Version : DX3 File Archiving+Saved Game Directory



Kevyne-Shandris
29th Jul 2008, 18:50
The loading bar thread got me thinking about a little pet peeve of mine. Not a major issue, but one on system performance and wear and tear on disks: how games are being archived and saved game directory location.

In the bad old days game files weren't archived into a mega file, so a gamer had 200+ files all over the hard drive. If they were fragmented, system performance would dip. Devs got a brilliant idea of keeping all those messy files in a couple archives based on what it contained, which means the files were less likely be fragmented, as it's but a couple of files.

BUT, some of these archives are over 3GB in size.

I use a disk optimizer (O+O Defrag) to improve disk performance on 2x150GB Raptors configured as RAID-0. When it reorganizes files to eliminate free space and defrag files, the scheme is by name (this is the fastest file retrieval scheme I've found, and won't go back to one that uses the modified date type). This means after the Boot record/Layout.ini/Admin files are laid out, the other files are loaded from there alphabetically by directory.

When you have a large archive, and because it's so huge, when it meets the MFT and page file zones, it will fragment and wrap around those zones. Short of renaming the file (which is 99.9% impossible for games), there's nothing much to do; as the alternative of renaming is nixed and stuffing a directory ahead of the game's directory to push it past the MFT zone, is a waste of disk space; and/or not friendly for a gamer to break their own file organization scheme.

So the best at heart effort from devs to keep the game files from fragging, fragments them because the archive is now too huge.

Can DX3 break down these archives into 1024KB or such smaller sizes? It's a couple more files, but there's less chance for them to fragment on a drive due to hitting locked zones.

Secondly, can we have an option, upon installing the game, to relocate the saved game directory from the My Documents folder?

MS is recommending saved games in My Documents, and devs are heeding, but now it's making the My Documents folder a mini-drive in itself with 20+ folders. If you use such a disk optimizer as O+O with the Name scheme, every new save in the My Documents folder will cause the ENTIRE DISK TO BE REORGANIZED. An option to chose your save game destination would eliminate this wear on the HDD (as a whole drive reorganization is the worst wear a disk can have).

Please, break down these large archives into smaller ones; and please let gamers chose their save game destination (for DX3, if it's anything like DX, and you're a hardcore fan, the saved games will need it's own partition anyway [especially if you're saving 999+ files!]).

jordan_a
29th Jul 2008, 18:58
New! Saves (http://forums.eidosgames.com/showthread.php?t=78722) (can include the saving system, maybe you ought to post a poll)

jcp28
29th Jul 2008, 19:57
From what you say about how save games are organized, it seems like either way, the files are going to be fragmented no matter where they are put. I'd almost say put them in a new folder, but it seems to me that would just provide another place for files to fragment, even if the folder was in an isolated directory of some sort(only thing I can think of)
It sounds like the best solution is for developers to break the files down to the 1024 KB size you mentioned.

I have to say, though, it sure sounds like there isn't a whole lot that can be done to prevent files from fragmenting in the method you mentioned. I highly suspect that might have been what played a part in my computer crashing three years back.

Kevyne-Shandris
29th Jul 2008, 20:32
From what you say about how save games are organized, it seems like either way, the files are going to be fragmented no matter where they are put. I'd almost say put them in a new folder, but it seems to me that would just provide another place for files to fragment, even if the folder was in an isolated directory of some sort(only thing I can think of)
It sounds like the best solution is for developers to break the files down to the 1024 KB size you mentioned.

I have to say, though, it sure sounds like there isn't a whole lot that can be done to prevent files from fragmenting in the method you mentioned. I highly suspect that might have been what played a part in my computer crashing three years back.

The way Windows lays them out on the disk will ensure they will be fragmented. :mad2: But the concern is the location of the directory. It's the location of the directory that is the problem, as it pollutes the My Documents folder with content that never was designed to be there (it's My Documents, not My Games for a reason); and in that disk optimization scheme, in using the My Documents as a game save directory forces a WHOLE DISK REORGANIZATION OF FILES!!

This pic illustrates what I mean that happens when a large archive hits the MFT zone, and splits the file...

That's partition O of 20GB, and it has nothing but FEAR/FEAR XP/SDK files on it. Now the reason that file is split is because the main archives ahead of it is 3.7GB in size (taking every square between the 2nd one to the last line before the red fragment seen), and not even a 117MB archive can squeeze past the MFT zone.

If the archives were split to smaller sizes, if it did split, it wouldn't affect almost 2/3 of the game files (especially if the 3.7GB archive had fragmented). Notice where the split goes, too? At the slowest part of the partition (ideally games like this should be at D/E/F partition for speed, but I don't play FEAR often).

It's not a game breaking thing, but it's one of the things devs need to understand as these games aren't getting smaller. Each year these archives are getting bigger, and eventually the main archive is going to split as it hits the MFT zone.

Blade_hunter
29th Jul 2008, 20:50
About saves I think when we change level the game must use autosaves and make a save
For the game we need 3 autosaves slots, 2 quicksaves slots and 30+ normal save slots
(the upper autosave is the last save point)

About the file location I agree by the fact we can change it it can be useful if we want to get an easy location or a secure location

I don't want console checkpoints to make saves, i want to make saves when I want

for the rest nothing to say,

Absentia
29th Jul 2008, 22:51
Yeah, a niggling thing that I noticed with all 3 of the Ion Storm games that I've played (DX, DX:IW, and Thief 3) is that none of them save automatically at any point. And because so many games do that, sometimes i've not bothered to save at the beginning of a level and just assumed it would've done, then after dying, realised i'm all the way back at a previous level.
Autosaving and checkpoints are a very good idea, just make sure it's not extremely frequent. I liked how in Ridd ick: EFBB it archived all the checkpoints you went through and you could load any of them. I guess that might take up a fair amount of disk space, but it was a really nice feature because it gave the game some more replay value (I could just find which bit I wanted to start from, instead of only choosing from my own infrequent saves or starting over)

jcp28
29th Jul 2008, 22:55
The way Windows lays them out on the disk will ensure they will be fragmented. :mad2: But the concern is the location of the directory. It's the location of the directory that is the problem, as it pollutes the My Documents folder with content that never was designed to be there (it's My Documents, not My Games for a reason); and in that disk optimization scheme, in using the My Documents as a game save directory forces a WHOLE DISK REORGANIZATION OF FILES!!

This pic illustrates what I mean that happens when a large archive hits the MFT zone, and splits the file...

http://home.comcast.net/~deus_ex_machina/img/OandO_Archive_Meets_MFT_Zone.png

That's partition O of 20GB, and it has nothing but FEAR/FEAR XP/SDK files on it. Now the reason that file is split is because the main archives ahead of it is 3.7GB in size (taking every square between the 2nd one to the last line before the red fragment seen), and not even a 117MB archive can squeeze past the MFT zone.

If the archives were split to smaller sizes, if it did split, it wouldn't affect almost 2/3 of the game files (especially if the 3.7GB archive had fragmented). Notice where the split goes, too? At the slowest part of the partition (ideally games like this should be at D/E/F partition for speed, but I don't play FEAR often).

It's not a game breaking thing, but it's one of the things devs need to understand as these games aren't getting smaller. Each year these archives are getting bigger, and eventually the main archive is going to split as it hits the MFT zone.

Well, I sure don't need further confirmation that Microsoft sucks, as their terrible browser and crappy security measures have shown. Seriously, they seem so moronic in some of the stuff they do.

It's reasons like these that make me want to get a Mac operating system when I eventually have to get a new computer. The trouble with that is, I'm pretty sure there are fewer games avalible. However, Escape Velocity which I believe was ported to PC some years back, was pretty cool when I last played it five years ago.:cool: Anybody who's into open-ended space combat and trading a la the somewhat disappointing Freelancer should enjoy it(once they get past the outdated graphics, unless there's been some sort of upgrade when I wasn't watching - perfectly possible)

Kevyne-Shandris
30th Jul 2008, 00:51
If I install another OS as a dual boot, it'll be FreeBSD or OpenBSD (or if I had the money, RHE w/HSphere or a direct admin web client), remove Apache 2 and use Lighthttp or some other web server (can't STAND Apache 2, clunky, resource hog, and you have to reboot it more than even Windows). Each lays files down so they don't frag as much -- it still does, but not like how Windows does it. But I don't really like the other alternatives too much, as working on a server with either *BSD and *nix is a navigational nightmare (especially with a webhost client -- WHM/cPanel being the most popular, and the most BROKEN).

As for Macs, I boycott them, as that company gave my tech relative's shop hell (they dropped servicing Macs as they really do live up to their snot label).

PC 100%.

Tsumaru
30th Jul 2008, 01:08
I take Mac hating to the levels of religious extremism.

But anyway, back on the topic. I personally liked it back in the day when they kept save games in the game folder. Sure, maybe there were problems with fragmenting and hard drives or whatever - but it was so much cleaner. I knew if I wanted ANYTHING to do with a certain game, it was in my D:\Games\<game name> folder. Nowadays crap is all over the place. I have no idea where to look. Same as game settings. It's a bloody nightmare.

Kevyne-Shandris
30th Jul 2008, 01:18
Try F.E.A.R.. Not the My Documents folder, but the Shared Documents folder even (and one place you're sure to forget to backup when doing a clean install).

964 folders, 964 now in My Documents. This is crazy devs, crazy to fill it up like that -- it's My Documents! ****

Red
30th Jul 2008, 06:56
I think the good ol' location <dxhome>\Savegames would be the best for saving games. I don't like this "save everything to my documents" either.

Lady_Of_The_Vine
30th Jul 2008, 10:24
Autosaving and checkpoints are a very good idea, just make sure it's not extremely frequent. I liked how in Ridd ick: EFBB it archived all the checkpoints you went through and you could load any of them. I guess that might take up a fair amount of disk space, but it was a really nice feature because it gave the game some more replay value (I could just find which bit I wanted to start from, instead of only choosing from my own infrequent saves or starting over)

Yes, I agree that I like to keep a full archive of all checkpoints in case I want to go back to one. :cool:

Auto-saving/Checkpoints are a good idea if not too frequent AND you still have the option to save manually at any point. I do like to have some control in that area.

The only thing I HATE is a checkpoint you don't know is coming! :(
Tomb Raider: Anniversary is an example of this. Damn annoying when you've spent a serious amount of time climbing and dodging traps to get to a certain high point... then you accidently slip down a slope, land at the lower ground area again, innocently walk to look about... and then you walk into a 'certain area' and hear "DING!" and read the Checkpoint activation!
WTF!!!! :eek: LOL.
Absolutely no visible way to recognise that you are about to walk into a checkpoint area... I really don't like that kind of system. :nut:

Absentia
30th Jul 2008, 12:41
.....Absolutely no visible way to recognise that you are about to walk into a checkpoint area... I really don't like that kind of system. :nut:

Yeah, It wouldn't be hard to display a small message in the corner or something that said "Approaching save point" so you knew that if you kept going in that direction a little more it would save. Also, I remember playing the TR:Legend demo and noticing that all of the checkpoints could strangely be re-visited just by travelling to that certain spot again, which was kinda weird.

Also, yes the game should not totally be reliant on checkpoints, it does need a quicksave or permanent save option too.
Only downside to this idea (which is particularly important for Deus Ex), is that if checkpoints make the game so free that you can just load practically any point you want from the past, then what if you ruthlessly killed an innocent NPC (provided you were given the freedom to do so, as a "moral choice" a la Lebedev) and then had to live with the consequences of these actions, which you didn't like. What's stopping you then from just loading back a checkpoint and doing it otherwise? I think that kinda spoils the whole idea of the game. On the other hand, you have to give the player some freedom, and if the player wants to f**k with the non-linearity and realism of the game, then I suppose he can. After all, he bought it!

jordan_a
30th Jul 2008, 13:05
What do you think about AUTO save and load?

One the one hand I like to save the game whenever I want. So that I can get back to it where I left: there is no interruption in the narration, the story.

On the other, it might be too easy.

Kevyne-Shandris
30th Jul 2008, 14:21
What do you think about AUTO save and load?

How about, "What do you think about game file archiving and the location of the saved game directory"? ;)

I rarely use auto saves and loads (if ever). As I'm more busy saving each area to get back to a spot at any time, and it not be overwritten by a future auto save or checkpoint.

Thus, needing a nice BIG place for saved game files and not in the My Documents folder (more like it's own partition!).

Beauty of DX was even if each save was 15MB a piece (large today, that averages around 2 or 3MB), it was highly compressable into saved game archives. So 5GB of saved games could easily be reduced to but 160MB.

Now I hope the saved games in DX3 can be archived as tightly as well!

Kevyne-Shandris
30th Jul 2008, 14:39
If this thread is going to fork into saved games itself, there's another issue that goes with archiving saved games -- naming convention.

Some games now are including not only the name of the ingame player; the location and even the date stamp (as seen in Oblivion). This is excellent when you need to find a specific area in a saved game archives, months or years later.

In DX you had to look it up in the game itself, and if the bitmap image to tell you were you were at isn't saved (as sometimes it isn't or blocked by the CHEATS label), it's a guess to find which save is in which area.

Hope that this new trend in games that have length and expansive environments, will continue in DX3. Makes it easier to find that perfect spot that was so much fun, and getting back to it with little more than a windows search for the game file number.

Would be even sweeter if, if we can click on it, and the game will launch and we're on our way, too. :)

And PLEASE, PLEASE, PLEASE give us a .ini file, so if we need to change the sequencing of the games (e.g., save #10, #15, #17, it can be changed to #10, #11, #12 onwards). DX left gaps in the saves, so you deleted a game, you couldn't go from #10 to #11, it'll be #12 as #11 was deleted).

Tsumaru
31st Jul 2008, 00:35
In DX you had to look it up in the game itself, and if the bitmap image to tell you were you were at isn't saved (as sometimes it isn't or blocked by the CHEATS label), it's a guess to find which save is in which area.
Guess you shouldn't have cheated then. =P

Kevyne-Shandris
31st Jul 2008, 01:39
Guess you shouldn't have cheated then. =P

Guess you tried it too many times to know. :lol:

Kevyne-Shandris
31st Jul 2008, 02:04
Since the thread has now probably descended into another tit-for-tat ready for another lock; threads were bumped to try the experiment of trying to post something; no new news; and now a Final Word thread emerged, I believe it's about time for a final word myself.

I gave my best. That's all anyone can ask.

Have fun!