[Resolved] File Size/Design Question

[neef]

I've been creating a football game as a basement project. Each play that is called is shown on screen as bitmaps crashing into each other. It's a low tech game- even a throwback, but it's very realistic in the results it produces.

Here's the issue: I would like to save each week's game data to disk but I'm concerned that the file sizes this creates are too big. Each week's data includes the positions of every player on the field for the duration of the play (every 20th of a second) which is stored as 22 List (of point) variables. This data really works great as the play is calculated, but saving the info to disk has resulted in a much larger file size than I thought. An entire seasons wroth of plays (16 weeks * 16 games per week * about 150 plays a game) is 4.3 gigs.

The reason I want to save all this data is to let the user watch "film" of the next opponent he is playing. Having access to every play of the season would allow the user to plan his game plan like a real NFL coach. It's suppose to be fun and challenging, and I don't want to lose that aspect of the game. However, that file size is almost as big as the entire Skyrim game install; it's not something a user of this type of low-tech game is used to! lol

The first thing I would ask if I was to answer this thread is : "Do you need to create all that data?". The answer is yes because of the physics involved in player's colliding and balls being thrown. The frame rate is crucial to create something that looks real. But there is an option to lessen the amount of data needed to be saved (see #1 below).

So, I ask the pros out there what should I do?

1) Tighten up the file size: I can get the file size down a great deal by cutting out 80% of the frames stored in each play. I'd have to give up instant replay on these plays (normal speed only show 1 out of every 5 frames - instant replay shows them all). This brings an entire season to about 1 Gig. Much better, but still pretty big. This would be simple to do as well. Giving up the slow motion replay would be less than ideal as you can really pick out important things with it (just like in real football).

2) Have options to save each week's files: I could let the user decide if the disk space is worth it themselves. They could choose not to save any data, save the data but give up the instant replay, or save everything.

3) Not even worry about the file size. Most PCs don't even blink at this amount of disk space. It's just unusual for this type of game.

4) Throw out the entire idea: Saving every game's data is not a crucial aspect of the game I'm creating. It's more like frosting than cake. Maybe the disk space isn't worth it to begin with.

Finally a programmatic question: Are there any tricks to reduce file sizes saved to disk that maybe I'm unaware of? 99% of the data stored are Point objects. Is there a way to use less memory by breaking a point object up into individual integers maybe?

Thanks for any advice in advance.

neefman

[si_the_geek]

A quick and rough calculation shows you are storing about 100 KB per play... that is much bigger than it should be!


In terms of the coordinates you work with, what is the range of values you need? If they are always between 0 and 255, you can eliminate 75% of the size by simply switching to a Byte based equivalent of Point. For a range of 0 to 65535 (or -32768 to 32767), you can eliminate 50% of the size by using Short/UShort.

Even if the coordinates do need a bigger range of values than Byte allows, presumably the movements are small each time (less than 128 in any direction), in which case you could have one 'start' frame per play (using Integer/Short/UShort), followed by 'movement' frames which use Byte to list the amount of change since the last frame.

If there are times when players don't move, you may be able to save storage by storing a time-offset from the start of the play along with the position change... but this kind of thing varies a lot based on the exact data, and it might make things worse.


In addition to that, presumably you are also storing other data for each play (perhaps player names etc?), and the way you store everything else could have a significant effect on the size too.

[neef]

Thanks for the quick reply. Your ideas have helped me greatly:

I've been saving lists of (single) which represent yards. If I convert it to int16 before I save it there goes 1/2 my space. That's easy to do. I just didn't realize singles take up 4 bytes while int16s just use 2 bytes. I wish I did a little research and found this out myself.

The field width is about 1000 pixels and the height is about 2200, so I can't use bytes straight up. I like the idea of using bytes as increments though. It will take some work, but it can be done. I totally would not have though of that.

The other data stored is pretty trimmed down already, but I did find one bloated member - injured players. If I store the injured players' key (an integer) and not the entire player object itself I'll chop off some space.

One followup: what size would be good for a target as I trim down from 4.3 GBs?

[dunfiddlin]

A quick and rough calculation shows you are storing about 100 KB per play... that is much bigger than it should be!

Well, yes and no. How would it compare with the equivalent in video, avi say, of the play? I compared it to replays from a baseball game I use which come in (albeit with a touch more graphic sophistication, perhaps) at around 80 - 100kb so it's not that far out in terms of what users might expect. Obviously any optimisation is good optimisation and I would certainly offer a choice as to whether to save all replays from all games, replays for just selected teams or games and so on along with the ability to delete at will, of course, but I don't think it's too far from the norm.

[si_the_geek]

Video etc takes far more data, most of what should be needed here is 22 positions (and an ID to refer to the player) plus a position for the ball.

I would guess that the size can be trimmed to under 20 kb per play, but obviously we'll have to see how far we get with it.

Giving options whether to save or not (including quality) and the ability to delete are good ideas.

The field width is about 1000 pixels and the height is about 2200, so I can't use bytes straight up. I like the idea of using bytes as increments though. It will take some work, but it can be done. I totally would not have though of that.

An important point is that you don't need to change your 'main' code (and probably shouldn't, as the size gain comes at the cost of a speed loss), you only need to alter the part that saves/loads the file... which will hopefully reduce the workload a bit.

The other data stored is pretty trimmed down already, but I did find one bloated member - injured players. If I store the injured players' key (an integer) and not the entire player object itself I'll chop off some space.

How often do you save that kind of information?

Injuries tend to be rare (and affect an entire play, perhaps several/all of them), so it might be worth storing a list of injuries separately to the plays - storing just: player/first play missed/last play missed, and perhaps a description of the injury.

One followup: what size would be good for a target as I trim down from 4.3 GBs?

I would aim for less 0.5 GB's, but accept about 1.0 !

[Shaggy Hiker]

Another point that you might consider is that storing every position of every player for every frame isn't really necessary. Technically, if P was at (x,y) at some time, then moved to (U,V) at some later time, do you really need all the intermediate points? I'm a bit unclear as to whether this is a replay of real games, or if this is entirely something you generated, though it sounds like the latter. If that is the case, you might find that there is a way to describe the action as a direction and duration, both of which may be as small as bytes. Storing just that information may cut the total size considerably. You'd be storing vector rather than raster data, so it could drop to only a few percent of what it had been.

[neef]

Thanks everyone again. I've already gotten the memory down by 40%

I have decided to create a new class that condenses everything in my StoredPlay class into a memory friendly format. The singles will convert to bytes. The intervals will be stored as bytes instead of singles and I'll trim down a bunch of other stuff as well. I'll keep the original class because that is 100 times easier to work with in the code that controls while the play is actually executing and I don't want to mess with it.

One last question: if I use bytes to show the change of a position rather than the position itself, how will I deal with negative numbers? So if a player moves to the top of the screen from point 100 to point 95, the difference is -5, but bytes only store from 0 to 255?

[Shaggy Hiker]

You could say that 0-127 is positive, and 128-255 is negative, but that really limits the range. There's no real way around this. Depending on what the actual data is you could pack all the bytes into an array of bytes with a single byte (or a 32-bit integer) that holds the sign for up to 32 bytes. That's a whole different level of encoding, but that's life. If you have only 8 bits to work with, there is a very strict limit as to how much information you can store in there.

[si_the_geek]

An alternative is to treat 128 as 0 (so 127 represents -1, 126 is -2, etc).

Doing it that way would mean the process of transferring data would simply be +128 or -128.

[neef]

I think I can work within that range. The maximum I've ever seen a player move per second is about 10 yards. There are 20 pixels per yard which is 200 pixels either plus or minus, but each seconds is broken up into 5ths knocking that down to 40 pixels per increment max. Plenty of room (fingers crosses).

Thanks again for everyone's help. I'll mark this solved but still report back my results if anyone is interested.

[si_the_geek]

10 yards per second is close to the limit of human running ability, so it will be worrying if you have somebody moving much faster!

[neef]

Yeah the fastest NFL player run a 4.3 40 yard dash is a little less than 10 yds/sec. The ball moves faster than that in the air, but the increments are still small enough to not even come close to the range limit.

File size is less than 1/4th of the original. I'm satisfied with that for video. The files are split into weeks as well making them less cumbersome to save and load, so I think I'm good.

Thanks everyone!