[eCompile] Unused include file

Developer Silver · Post by **Developer Silver** » Thu Nov 30, 2006 1:29 pm

A suggestion based upon ecompile's feature to automatically exclude unused functions from generated .ecl: should be possibile to have a warning like "Unused include file: xxx.inc" if neither functions nor variables/constants are used from a declared include file?

Alternative: the .dep files could have written the number of "hit" of each .inc? This way files with 0 hit could be removed from includes...

This could save so much time compiling scripts!

qrak · Post by **qrak** » Fri Dec 01, 2006 3:52 am

Good feature suggestion. Excluding unused functions would be good to minimize .ecl size

Post by **Austin** » Fri Dec 01, 2006 4:49 am

It already does this.
Make an include with a function that has errors in it, include it in a script but dont call the function, itll still compile because it never uses the function's code.

Developer Silver · Post by **Developer Silver** » Fri Dec 01, 2006 9:33 am

Austin wrote:It already does this.
Make an include with a function that has errors in it, include it in a script but dont call the function, itll still compile because it never uses the function's code.

I don't understand if you replied to qrak or to me

In second case: I know this, in fact I didn't suggest that to reduce .ecl size, but compiling time.

We have about 1300 .src and 200 .inc, and during last months I've heavly modified .inc functions distribution, so now we need a way to delete unused includes to avoid useless script recompiling (even if they're not used, ecompile detects updated .inc and recompiles scripts).

Marilla · Post by **Marilla** » Fri Dec 01, 2006 4:24 pm

You oughtta try to be more careful about how many include... statements you toss in places. I'm not really sure how practical it is to try to have the compiler going through and NEGATIVELY checking for functions/identifiers used from include files; seems like it would make the compile process take longer, every single time, because it would always have to be digging around looking for functions that you -don't- use, which seems a big waste of effort.

Basically, since you can't keep from adding extraneous 'include' statements, you would be making MY compile times take longer by having ecompile looking for things that I don't think it should bother taking the time to look for.

If you really have a problem with this, I think you may want to consider a serious re-organization of your include files. Under most circumstances, it should be fairly intuitive when you need an include or not - because your includes should contain logical groups of functions and identifiers that you obviously either need, or don't need.

I've seen people posting that basically make it sound like their 'debugging' of scripts amounted to commenting in and out include files until stuff compiles. That just suggests such a mess that I wonder how people can expect to make sense out of anything at all.

Post by **MuadDib** » Fri Dec 01, 2006 6:29 pm

If calls are recorded in the files......... someone could just as easily make a script to clean based on those

Developer Silver · Post by **Developer Silver** » Mon Dec 04, 2006 2:23 am

Marilla wrote:I'm not really sure how practical it is to try to have the compiler going through and NEGATIVELY checking for functions/identifiers used from include files; seems like it would make the compile process take longer, every single time, because it would always have to be digging around looking for functions that you -don't- use, which seems a big waste of effort.

While ecompile parses .src file, it already scans included files in order to find used functions... so it should be easy to track how many calls are made at every .inc (this could also be a boolean, it's not interesting exactly how many calls, it's important only 0 or >0)... this way your compile process shouldn't be slower, but you can see which includes have 0 calls and are not necessary.

MuadDib wrote:If calls are recorded in the files......... someone could just as easily make a script to clean based on those

Oh yes, you can also use UltraEdit's search function to find unused includes

Marilla · Post by **Marilla** » Mon Dec 04, 2006 2:45 am

Developer Silver wrote:...you can see which includes have 0 calls and are not necessary.

Perhaps, but I can also see this causing trouble when people don't realize that just because they aren't using a function/identifier in an include, does not mean the include is not needed, as I suspect you might not realize

For instance, perhaps an include exists to aggregate other includes which are frequently - but not always - needed together, to avoid the need for excessive include statements in the source. Or perhaps such an include really does have functions in it, but it just so happens none of them are used. Remove that include based on the fact that none of it's functions/identifiers are used, and you may end up having trouble with includes that file itself defined.

While it could be argued that it would be better to simply include all of your individual files themselves (and I would argue that), I've seen all sorts of twisted and confusing stuff with includes, and it would surprise me NOT to see something like the above. The problem? Without needlessly complex issues like that, and with a generally good structure for how includes are used, such a feature really isn't all that important.

So, it's sort of like... the problem for which you think you want the feature may actually make it difficult to even use the feature in the first place.

SMJ · Post by **SMJ** » Mon Dec 04, 2006 5:16 am

I can understand where you're coming from with the suggestion; saving discspace and whatnot. However, I really don't see how something like this would be especially helpful. If I get the feeling that an include file or a particular function is un-needed, I just stuck "--" or "__" before the file or function name, respectively, then recompile the entire scriptbase and wait for error messages.

Developer Silver · Post by **Developer Silver** » Mon Dec 04, 2006 6:52 am

@Marilla: using this feature in ecompile makes impossible to generate false positive, I think... calling a function in a file included in another include, should make both used in ecompile listing... see example below.

@SMJ: disk space is not our problem, the main goal is reducing compile time deleting unused includes

Your suggestion is what we're doing now, but testing every include in every script everytime we update something, is impossible

Maybe I've not been clear... I'll do an example...

test.src

Code: Select all

include "inc1";
include "inc2";

program test()
	CallInc1b();
endprogram

inc1.inc

Code: Select all

include "inc1b";

function CallInc1()
	Print("Calling CallInc1 from inc1.inc");
endfunction

inc1b.inc

Code: Select all

function CallInc1b()
	Print("Calling CallInc1b from inc1b.inc");
endfunction

inc2.inc

Code: Select all

include "inc2b";

function CallInc2()
	Print("Calling CallInc2 from inc2.inc");
endfunction

inc2b.inc

Code: Select all

function CallInc2b()
	Print("Calling CallInc2b from inc2b.inc");
endfunction

At compiling time ecompile could prints out "Unused inc2.inc in test.src"... or in .dep file:

test.dep

Code: Select all

C:\pol\scripts\test.src
C:\pol\scripts\inc1.inc 1
C:\pol\scripts\inc1b.inc 1
C:\pol\scripts\inc2.inc 0
C:\pol\scripts\inc2b.inc 0

The 0s mean that inc2.inc can be safely removed from test.src

Anyway, this is not so vital... it was only (for us) a usefull feature

Tritan · Post by **Tritan** » Mon Dec 04, 2006 11:00 am

I do this just like SMJ does. When I working on a script I start removing the includes to see is really needed. I know what most my includes hold since they are broken up fairly well in the packages.

Marilla · Post by **Marilla** » Mon Dec 04, 2006 2:02 pm

Developer Silver wrote:@Marilla: using this feature in ecompile makes impossible to generate false positive, I think...

This is what I was talking about:

Code: Select all

include "inc1";

program DoIt(who)
   CallInc2Function();
endprogram

inc1:

Code: Select all

include "inc2";
....other stuff not used in the .src...

inc2:

Code: Select all

function CallInc2Function()
   ...
endfunction

In your example, ecompile would report - correctly - that no functions are used from inc1. However, if you remove inc1 from the .src, you will break it. Obviously, no one would ever do what I've done in the above simple example. But also just as obviously, if you look through the actual includes I've often seen, the above effectively happens a whole lot.

========================

I think that a combination of what Tritan and SMJ have mentioned, coupled with re-organizing your include files, will solve the problem much better. I think people create includes very improperly, a lot of the time. For instance, with apologies to any former distro devs (and I don't know - maybe these includes are no longer in the distro?), I think that includes like 'myutil' and 'std' are just completely horrid. Well, beyond the fact that the large majority of functions in them can probably be killed completely due to features the core now has (especially the std.inc I used to have way back from 092! hehe), I also think that 'catch-all' includes like that make for very messy code.

Now, there may be times when you will have some 'master functions' that you just use ALL the time. But includes like myutil and std don't have functions like that in them, to be honest. They have a bunch of often completely unrelated functions that (if core support had not been added) can be useful from time to time in this or that thing.

The problem with includes like this is simple: Changing one function in them for some purpose will end up with you having lots of .src files recompiled needlessly (which is what I know your issue is here - and I agree that it's a good goal, to eliminate needless recompiles due to includes. I just think you are barking up the wrong tree to accomplish it)

So, maybe myutil has a function in it to, say, open the player's web browser. Fine. (though again, it's now a core function, but stay with me here). Then maybe it also has a function to... I dunno... do anything else at all. Think about this a second:

In what logical way is the "DoAnythingElseAtAll()" function related to the "OpenWebBrowser()" function? The answer is 'in no way'. The logical conclusion should then be, "These two functions do not belong, therefore, in the same include".

My solution was very, very simple: I created an include for the web browser function and it's helper functions alone. There were no other functions at all that were 'related' to them, so I put no other functions in that include.

So, if I ever change the web browser include file, I can rest assured that only .src files that actually use the web browser function will be recompiled.

I do this same thing for all includes: I only put functions/identifiers together in an include when those functions/identifiers are logically connected to each other. This means I end up 'splitting up' includes a lot - I have a lot of includes with only a couple functions, and those includes are named such that when I look at the include area of a .src file, it's obvious what includes are there, and why. I never need to ask myself, "does this .src use that include?"

And then something that has been mentioned related to includes elsewhere; includes including other includes; Some people think that's just plain bad, right from the start. I think it *can* be bad, and confusing, but I also think it's absolutely necessary to accomplish the type of organization I do.

For example, take my WebBrowser include. As you may know, that include needed some functions to convert characters into hex and such, to build the packet to send. But what if I also needed those functions elsewhere?

I would put those functions themselves in another include - again, only with related functions. Then, I would include that include from within the web browser include, and whatever other scripts/includes needed those functions would also include it.

What I would NEVER do: Create duplicate copies of those functions and put them in multiple source files. (That's when you start getting the lovely "Expected identifier, got user function instead" error).

Developer Silver · Post by **Developer Silver** » Mon Dec 04, 2006 2:45 pm

In my suggestion, your example works anyway: DoIt calls CallInc2Function, which is in inc2.inc... but to reach inc2.inc, ecompile must pass through inc1.inc, and so it marks both includes file

About includes organization, I know what you mean, but I was thinking that if this feature is not so difficult to implement (the .dep alternative, at least), it could be an aid for everyone organization...

Marilla · Post by **Marilla** » Mon Dec 04, 2006 3:29 pm

Developer Silver wrote:About includes organization, I know what you mean, but I was thinking that if this feature is not so difficult to implement (the .dep alternative, at least), it could be an aid for everyone organization...

Perhaps insofar as it tells you what you already should know: Your includes are screwed up :razz: I'd rather my eCompile did not take the extra time, personally.

Post by **Yukiko** » Mon Dec 04, 2006 9:25 pm

I have no comment about this really other than if it is implemented please make it a switchable option in ecompile.cfg. I can see the listings getting messier with

"Warning - include file stuff.inc referenced but no calls made to it in line 4.
Warning - variable junk not used in line 24"

etc.

Xadhoom · Post by **Xadhoom** » Tue Dec 05, 2006 12:18 am

You mean you don't want to know if one of your include in unused in one of yours .src files Yukiko?
Even if an unused include doesn't make you ecl bigger, if you modify that include it causes your script to recompile also if not needed.
Ecompile already checkes if function inside an include are used or not, display a warning would not raise your compile time.

Post by **Yukiko** » Tue Dec 05, 2006 12:37 am

First, I don't care if a script recompiles or not. Compiling scripts isn't a big time concern of mine. Frankly I am impressed at the efficiency of ecompile. So that makes little difference to me.

Second, I am tired of seeing all those "unused variable" warnings in the ecompile output. I don't want another, in my humble opinion, unnecessary item cluttering up the output from ecompile.

Third, I am not convinced that having an unused include file causes any inefficiencies in POL executing the compiled files. I can see where an unused variable would take up space unnecessarily in the shards data files but unused includes seem to only take up space in the source files and appear to have no adverse effects on the world data file size or script execution.

Xadhoom · Post by **Xadhoom** » Tue Dec 05, 2006 12:53 am

Maybe if you compile 20 scripts isn't a big time, if you have to recompile more then one thousand it is
As already Silver said, it would be more useful to add a line in the .dep files than having a warning, althought if you can see warnings in the ecompile you can go editing the script (removing the unused include) ad you'll never see it again, easy, no?

Having unused include files DOES NOT causes inefficiencies during execution of POL, but it can take lot of time when compiling scripts

Marilla · Post by **Marilla** » Tue Dec 05, 2006 1:10 am

Xadhoom wrote:Maybe if you compile 20 scripts isn't a big time, if you have to recompile more then one thousand it is
As already Silver said, it would be more useful to add a line in the .dep files than having a warning, althought if you can see warnings in the ecompile you can go editing the script (removing the unused include) ad you'll never see it again, easy, no?

Having unused include files DOES NOT causes inefficiencies during execution of POL, but it can take lot of time when compiling scripts

But what I don't think you are understanding is, what is being asked for here is to add time to the eCompile process to 'fix' something that you should have fixed already. It's sort of like asking eCompile to check for any endless loops in your code, or any of a thousand other coding mistakes. I would -much- prefer eCompile just took me at my word when I write a script, and do what I ask it to do... and not be spending time double-checking my work. You are asking eCompile to take longer for ME, because you can't organize your includes properly... when there are plenty of other ways to do what you are asking for (like simply commenting-out includes if you suspect they aren't used, and try recompiling to see).

As it is, on my shard, due to the extent that the Housing and Town Stone systems are involved in things (there is very little that does not check something in one or the other of those systems), I very often have compiles that do hundreds of scripts at a time. So, it takes a couple minutes. Big deal. However, in my case, because I've organized all of my includes properly from the start (ok; not 'from the start', because the include mess in WoD/Ackadia was horrid), I know very well that ALL of those scripts really do need to be recompiled, because I know that only includes that are necessary are in each script.

Xadhoom · Post by **Xadhoom** » Tue Dec 05, 2006 1:35 am

Not all shards have only one scripter that works on all scripts. If i'm insure of an include into one script i can remove it and check if it compiles, but doing it for every include in every script is way longer than looking for what ecompile already does.
I repeat it: what ecompile already does
It wouldn't take hours to write in the .dep if the include is used because the functions are already checked, if no function is checked the include is useless.
We're asking for a feature, you already said that if implemented it would be useless to you, like other features are for us. I think we already explained why we want or not this to be implemented, now it's up to the developers

Marilla · Post by **Marilla** » Tue Dec 05, 2006 3:04 am

Xadhoom wrote:Not all shards have only one scripter that works on all scripts.

I don't know of any shards that have only one scripter. However, I do know this: having more than one scripter is more reason to keep everything organized... not less.

Xadhoom wrote:I repeat it: what ecompile already does

Just because you repeat something, does not make it true. I just opened a .dep file, and guess what? No counts of how many identifiers/functions are used in each include. If it already did it, I would have to wonder why it's being asked for, then.

Post by **Yukiko** » Tue Dec 05, 2006 3:24 am

Quoting Marilla:
"I don't know of any shards that have only one scripter."

You do now Marilla unless you count the scripts I have borrowed/used from other shards. For most of the life of Hope Lives I have been the sole scripter.

As for the number of scripts and ecompile time...

Output from the last ecompile of my shard:
Compilation Summary:
Compiled 1127 scripts in 141781 ms.

Now unless I am mistaken that's a 141 second compile time or just over two minutes and twenty seconds. I'd say that's not a major delay for 1,127 programs compiling.

There is one thing I think would be neat but only for the sake of curiosity. I wish ecompile gave a total count of all lines compiled in the summary. It would be for nothing else than curiosity. But that's another topic.

Developer Silver · Post by **Developer Silver** » Tue Dec 05, 2006 6:16 am

Marilla wrote:I just opened a .dep file, and guess what? No counts of how many identifiers/functions are used in each include. If it already did it, I would have to wonder why it's being asked for, then.

That's the feature we're asking for (the alternative, at least): writing to .dep files the hit count of include calls (already done by ecompile scripts parsing)

Marilla · Post by **Marilla** » Tue Dec 05, 2006 7:05 am

Developer Silver wrote:That's the feature we're asking for (the alternative, at least): writing to .dep files the hit count of include calls (already done by ecompile scripts parsing)

Xadhoom said (and, let's not forget, repeated) that eCompile 'already does this', referring to counting how many times a function is used. Since it being displayed somewhere is the only way I could imagine he knows that, I looked.

I think he (and also you) is confusing the fact that eCompile never reads functions that are never called, with it having any clue how many times a certain include is used.

What I'm saying is: Why slow down ALL of our eCompiles to count and track this stuff, when the problem is that you have a mess of includes in the first place. That is: You are asking eCompile to go beyond being a compiler, and instead you want it to be an auditor of your code. My hope is that does not happen: I use eCompile too.

Developer Silver · Post by **Developer Silver** » Tue Dec 05, 2006 8:00 am

Try to put an useless include in a .src file: the generated .dep file will list that file anyway.
This means: ecompile already looks for every included file, so it could be easy and fast to track which .inc are really used (while it parses .src file) and which ones are not.

We're asking a feature to improve our compiling optimization, not to slow down yours, obviously.

If a dev will tell that this could slow down ecompile performance, never mind.