DATA COMPRESSION

DATA COMPRESSION IN RPG MAKER 2003

What does this do?

This tutorial will teach you a technique for storing multiple data values in single variables based on modular arithmetic. It does not require any patches or plugins. It does NOT have anything to do with file archives like .zip, .rar, etc.

Why do it?

The more savvy RPG Maker 2003 users among you may be thinking, you can already transcend the apparent 5000 variable limit by using "Value Reference" manipulations, increasing the working number to a staggering 9999999. Why would you need to bother compressing data when there's so much storage space available?

Well, it all depends on the needs of your project, really. For example, suppose you wanted to create a strategy RPG with an army of 300 characters, and each of those units has, oh, 20 stats. Pretty ambitious, but it's only 6000 variables to track, right? Then you decide you want TEN such armies of opposing forces, facing off against each other in a grand campaign! Now it's 60000 variables, but that's still way less than the limit, right? Then, you get the most ingenious idea of all: every single unit has a relationship stat toward every other unit which affects how well they work together or fight against each other. 8o MIND=BLOWN! Unfortunately it also means our variable usage just blew up too, with 3000 units each needing 3000 variables (technically 2999 but whatever) to track how much they like or hate everyone else, adding up to 9000000 plus the 60000 we were already using. Hum...hope we don't have any more really ambitious ideas to fit into the remaining 939999 variables for this game. c.c

But wait! It's data compression to the rescue! If we decide that the maximum value a given unit stat can have is 200, we can store 3 of those values in a single variable (I'll explain how it works later). That means that the 60000 variables we were using for basic stats can be divided by 3, reducing it to 20000! But it gets even better! Suppose we decide that the relationship stats only need to record a rating from -5 to 5. That can be squished into a single variable 6 times, so the 9000000 variables from before are chopped down to 1500000! WHAT A SAVINGS! Now we can afford around FIVE crazy systems like that and still have room left over for miscellaneous junk! :D

That's a rather silly and extreme example of course, but you see how quickly data storage can become a problem if your concept demands it, and how much of a boon compressing that data can be in such situations. Besides, just knowing such increases in storage capacity are possible might get your gears whirring on potential uses. ;) Where the compression system really shines is when you have extremely small chunks of data to store, like values which are either 0 or 1, true or false, on or off. Sound familiar? Yup, that's exactly what switches do. Using data compression, you can store the equivalent of a whopping 23 switches per variable, and unlike regular switches, you can both set AND check the values by reference! This allows you to implement huge, numerically-driven systems while saving the actual switches for the few things switches do which variables can't, like controlling monster behaviors.

What's the code?

I've created a demo project (which you can download here) containing the common events needed for the data compression system, along with a map event containing the demonstration scripting we'll be making in this tutorial. You can copy the common events and drop them straight into any project you like, although of course you'll have to assign variables within your project and change all the variable references in the data compression events to use them. Also, listed below is the scripting for the common events (edited a bit from the pure copy-paste to make it clearer which variables are being used). I'm not going to go into a detailed explanation of the internal workings of the data compression system; those of you who are particularly code-savvy can probably figure it out from looking at it, and those of you who are puzzled by it don't really need to fully understand it in order to use it.



How do I use it?

Okay, now how do we actually start dumping stuff into compressed data and pulling it back out? First we have to decide a few things about the data we're storing. Most importantly, what size is each chunk of data? Assuming you want to store numbers that range from 0 to N, the "chunk size" of your data is N + 1. For example, if you want to store values from 0 to 10, you'll need a chunk size of 11. I highly recommend creating a variable to hold the chunk size of each possible type of data you'll want to compress and setting that variable at the very start of your game. Using a variable like this will make your code more readable as opposed to plain numbers, and if you ever need to change the chunk size for a data type, you only have to update the value in the variable instead of hunting down and changing every instance of it. Personally I like to put a # at the start of such variables to indicate that they are "constants", and should never be altered after their initial setting.

The reason the chunk size is important is that it dictates how many data values can be stuffed into each variable. The bigger the values, the fewer can be compressed into a variable. Specifically, the number of values you can fit into one variable is the largest power by which you can raise your chunk size without exceeding 10000000 (which is the highest number a variable can hold, plus one for the value 0).

If all that goes over your head, don't worry about it too much. You can find the ideal chunk size for any set of values using the table below.

Allowed value range	Chunk size needed	Values per variable
0-1(simulates switches!)	2	23
0-2	3	14
0-3	4	11
0-6	7	8
0-9(easily readable!)	10	7
0-13	14	6
0-24	25	5
0-55	56	4
0-99(easily readable!)	100	3
0-214	215	3
0-999(easily readable!)	1000	2
0-3161	3162	2

It's allowable to use chunk sizes between the ones prescribed in the table; these ones are just what give you the maximum allowed value range before stepping into the next lower bracket of values per variable (or, in the cases of 100 and 1000, happen to provide you with data that's relatively easy for a human to read, more on that later).

For our example, let's say that we're storing data about a unit's rank, and that rank can go from 0 to 9. We assign a variable in memory the name #RANK_CHUNK_SIZE and set it to 9 + 1, which is 10.





Next, we need to figure out where we're going to store the compressed data. If you're planning to use the variables beyond the 5000 mark normally allowed by RPG Maker, compressed data arrays are a prime way to make use of that real estate, as you typically won't want to manipulate the data directly. Whatever the case, you'll need to know how many variables in a row you'll need to hold your data. The number of actual physical variables needed is:

(number of values being stored) / (number of values which can be stored in each variable), rounded up if there's any decimal remainder

For our example, let's say we need to store rank values for 100 units. The number of values that can be stored in a variable with our chunk size of 10 is 7, and 100 / 7 = 14.286, approximately. As long as there's any decimal remainder at all we have to round up, so that means we need 15 physical variables. Before marking those variables, I'm actually going to create another "constant", this time to hold the numerical variable location where our compressed data will start. We'll call it #RANK_START, then place the actual compressed data variables right after it (although we could put them anywhere with 15 free variables in a row).





Notice that I used asterisks to fill in most of the middle variables for the rank data. It's a lot easier to copy-paste a monotonous "name" like that than to type out the actual name with a number for every single one.

Okay, we're finally ready to start plugging in values! First we need to set COMP_START to the reference of the first variable in the compressed data range, and COMP_CHUNK_SIZE to the chunk size of our data. We can do both of these using the "constants" we created earlier. Next, we set COMP_INDEX to the position in the data array we want to fill (NOT the physical variable reference mind you, but the number of the unit whose rank we're trying to set). The index starts with 0, not 1, so if we want to set the rank of the first unit, we should set COMP_INDEX to 0. Finally, we set COMP_SET_VALUE to the data value we want to store--let's say we're setting the rank of the first unit to 2. Once all that is set up, we simply call the Set compressed value event, and it handles the rest!



Let's set a couple more while we're at it. You can leave the COMP_START and COMP_CHUNK_SIZE variables as they are. Add 1 to COMP_INDEX to aim at the next value position, then call Set compressed value again. The second position (index 1) is given the same value as the first, 2, because that's what's in COMP_SET_VALUE still. Let's change it up now. Add 1 to COMP_INDEX and set COMP_SET_VALUE to 3, then call Set compressed value again.



If we run this code in test mode and then bring up the debug menu (press the F9 key), we can see that the first compressed data variable has the value 322. Hey, if you read those digits from right to left, they're the numbers we stored earlier! :D There's a reason I chose 10 for the chunk size in this example, it makes the compressed data easily human-readable. The same is true for 100 and 1000, so you may want to consider using those numbers for chunk sizes even if they're bigger than what you technically need. It can help you with your debugging efforts.



Just to demonstrate how it works, let's fill the rest of the compressed data array with values. We don't particularly care what the values are at this point, so I'll throw in a loop to fill them with a repeating sequence from 0 to 3.





Okay, we've managed to put a bunch of values into the compressed data array; how do we get them back out again? Say we want to find out what rank the 10th unit has. First we set COMP_START and COMP_CHUNK_SIZE to the appropriate values (they should still be the same as when we used them to enter the values into compressed data, but it's generally a good practice to set them again to make sure). Then we set COMP_INDEX to the compressed data array index we want to look up. Remember that the index always starts at 0, so to look up the 10th unit, we should set COMP_INDEX to 9. Finally, we call the event Get compressed value. After that, the 10th value in the compressed data array should be copied to COMP_GET_VALUE.





And there you have it! COMP_GET_VALUE now holds the rank of the 10th unit, plucked from the compressed data. It's worth noting that the Set compressed value event also calls Get compressed value as part of its own processes, so COMP_GET_VALUE will be changed by calling either event. You should copy the value in COMP_GET_VALUE to another variable if you need to keep it while getting or setting other compressed data values.

What are the downsides?

There's no such thing as a free lunch, of course, and the data compression technique does have some downsides. Most notably, it takes some extra time to get or set compressed data values. It's unnoticable if you're only doing a few, but if you have to do a whole lot at once, it may cause some lag in your game (deleting the comments at the start of the Get compressed value and Set compressed value events may help if this becomes an issue, as comments do actually take a cycle of scripting time for the RPG Maker to read). It also adds complexity to your code, which tends to make development slower and more error-prone. Finally, as with all scripting that manipulates variables by reference, there is a strong potential for data corruption. The events given here have no safeguards to ensure that the data manipulations will stay within the variables you set aside for compressed data; it's up to you to make certain the COMP_START, COMP_CHUNK_SIZE, and COMP_INDEX variables are set correctly and don't go out of bounds.