'data overflow' during script compilation...

[OWHorus]

Hello,

I ran into this problem not for the first time. We have 3 very big scripts, which are important and not easy to change. They also work now for years. So please do not tell me to change the script - but I have a question to core insiders:

What exactly triggers this problem?

What happened: I did a very small update to th huge script - and it bombed with 'data overflow Flog the programmer for using 2 byte offsets in data files'. I managed to correct the problem, btw.

But it was painful: Reducing the length of strings sometimes helped, sometimes not. There were cases, where I shortened messages, etc. but it did not help, or even get worse. In the end I managed to change only the necessary minimum, and it worked.
But one day I will run into the problem and not manage to fix it - so my question is:

Can anybody of the core devs explain to me what exactly triggers this problem? Too many strings? To many variables? To many general data?

I looked into the source - the message is generated in pol-core/bscript/symcont.cpp in a class constructor function named 'void StoredToken::StoredToken( unsigned lengthToAdd )'.
But I do not know enough to tell, what triggers the problem, except that it has to do with huge scripts, and that obviously a data storage overflows.

If somebody could enlighten me, I would be really grateful - and the I can work on our problem scripts step for step. What can be done to a script (except dividing it or rewrite it or other brute force methods) to fix the problem?
The script is quite important and I do not have the time to rewrite it (it also was not written by me), but I could try to reduce this and that. The scripts itself are in service for years now and besides their size they compile without errors or warnings and are well tested.

Thank you.

OWHorus

[Nando]

I've never seen that before, but a quick look suggests that the compiled code is getting too big (more than 65kb). At some point, when a jump is required (end of a function or loop, for example), the offset ends up being larger than 2-bytes and the compiler complains.

Could I have a look into the script? I can't believe that this script really has more than 65kb, so I'd also want to see if it could be a compiler bug. Otherwise, I can only imagine that hard-coded data or MANY excessively large functions are pushing this number up.

[OWHorus]

The script is rather big because it includes a lot other files (we have our own system here and a lot of includes) so I cannot show the script without posting a lot of rather big includes, which are used but only partially, since it uses only called functions.

I could post a list file, which ecompile produces. This is rather big (~1.4 MB), so I do not want to post it, but I could it send to you (zipped by email, with WinRar it is around 250kB). This is obviously a working version, since ecompile produces no listing in case of errors.

The resulting ecl-file after compilation is around 330 kB. It contains numerous gump menus (and message strings) and so on. The only thing I can think of is to try to create several subroutines and so reduce the absolute code size (speed is not an important factor here, it is mainly a player/GM interface).

Since I am interested in this problem I further looked into the source. It seems that the constructor mentioned in my last post is just a sort of 'bitfield', ie. a fixed size marker, which is used in symbol containers. For this reason the developer (seems to be original and has not changed for a long time) defined a short int as 'offset' and since the size is fixed, it is probably not simple to make it long int. If a am not mistaken, ecompile uses this every time it needs to store a token, and if the script is rather big the number of tokens increases. It is not the data in itself, but the number of tokens, and this makes it rather unpredictable. We have a lot of scripts (with many messages, gumps and so on) which are bigger than 64kB, this never was a problem.

I do not think this is a bug, it is simply a limitation. The problem is, that it is really hard to change a working system that big. So I am on the look for doable solutions. The idea to use more subroutines to shrink the code size is probably the way to go, even if not very simple...

So please PM me and I will send you the listing.

Thanks

OWHorus

[blckfire]

Well,in my short experience relating to POL and from the problems you describe, I think you need to split up the code in more files than one.

If you must keep track of several things at the same time you can use a network of scripts that talk between each other through event messages. I'm using this to create what I call a "centralized system", this means that it has several scripts that need to modify things in this main script, and the way to do so is by sending messages to it, this also covers the part of dealing with what came first and after, since the events are cached by order of arrival. In this way you can handle stuff more easily.

I don't know if anything I wrote actually fits your case, since I can't see the code itself, but I think it really might help to separate things by running several scripts instead of the one script.

[OWHorus]

Hello,

I talked to Nando and sent him examples. At the end it was pretty clear, that it is simply a big script, and a certain data object in Escript overflows. It could be changed, but it is a rare thing.

What should help: Write more subroutines and lower the lines of code for a task. Or divide it in more different scripts (programs).

The number of files (i.e. include files) has nothing to do with it, also the calling depth has nothing to do with it. It is simply the number of lines. Reducing the number of (written, not executed) code lines should help.

I will try to find subtasks in our system and put them in subroutines. Speed is not of an essence in this system.

OWHorus