[probably fixed] Tool tips memory leak&lag

[Demostenes]

Recently we were facing problems with lags and memory leaks. It usually started after there was more then 80 players. After that, in few hours and jumps (each jump cca 300MB), memory jumped to 3GB and server became totally unpredictible. Lags started when the memory reached over 2.5GB (after first jump).
We shut down toolstips and that problem disappeard. Server is holding around 2GB without any problem.


We have now more then 130 players in the peak and quite big data, pol allocates cca 2GB after start.

[MuadDib]

What core are you using?

[Keryan]

We are using latest RC5. And by shuting down tooltips Demostenes means that we changed UOFeatureEnable from 0x1a0 to 0x00 -> so we disabled little more features, but I think tooltips was the reason for memory leaks and lags.
This memory leaks are quite strange - it just "jump" in few minutes/sec from 2,250GB to 2,550+, then for few hours hold there and then "jump" again to 2850+ .., but this happens only after few hours ( we have auto restarts in 4:00am and first "jump" is in 17:00+, maybe number of players having something to do with it ).

After setting UOFeatureEnable=0x00 is everything OK server is running second day without any problems with lags or memory.

[qrak]

I have same problem after hooking megacliloc packet.

Here is my problem related to memory leak:
http://forums.polserver.com/ftopic1880.php

It happens after ~ 24h uptime.

[MuadDib]

Could you try setting Feature Enable back to how it was, and just disable that packet hook?

[Demostenes]

Small update: Memory is still leaking, but many times slower (cca 10x). With previous version of POL we were running (0.95) there were no problems, so we are suspecting there is some problem with new core.

[tartaros]

MuadDib: out of curiosity, what algorhitm/memory scheme do you use for caching the tooltips? If any...
Or are the packets being generated over and over again every time any client asks for them?

[MuadDib]

Tooltips themselves are generated if a client requests it, on the spot.

Would be nice for someone to report any difference between with a hook, and without. Might help narrow it down some too.

[coltain]

I can report some...

As I tested there was no difference if the packet was hooked or not (I even moved the directory away)

I had UoFeatureenable set 0x1a8 (to show the single click window)

[Keryan]

Tooltips themselves are generated if a client requests it, on the spot.

Would be nice for someone to report any difference between with a hook, and without. Might help narrow it down some too.

It seems that the our problem with memory "jumps" has only little to do with tooltips after all, turning tooltip off has only slown down the problem 10x +- ( as Demostenes write before me ), maybe it is in all packet hooks and tooltips was just one of most used, or maybe it is somewhere else, i dont really know, but now server can work for few days, before first memory jump occurs.

[Demostenes]

Any progress with this?

[Shinigami]

just try latest core...

Shinigami

[Keryan]

just try latest core...

Shinigami

Thx for the new release !

Sure, i am going to test it right now on our shard. Will post results in few days.

[coltain]

Any progress???

This allocation problem makes me tired:(

[guialtran]

The problem in this function -> CreatePacket(...,...);
CreatePacket( ) == Memory leak...
use - > SendPacket( ...,...);


look here
http://forums.polserver.com/viewtopic.p ... eak#p12631

[MuadDib]

Nando and me (nando, don't leave me hangin here, LOL)are gonna try to look into the leaks in CreatePacket(). I have no internet at home, but, I AM BACK BIZNITCHES.

guialtran: thanks for your post with the code that caused the leaks. I just downloaded it and plan on going over it during this week.

[MuadDib]

Just an update, we are working on it. Will take some digging, but it MAY fix a lot more leaks than just this individual one. Don't get your hopes up, but it may involve a bigger leak that affects all Variable Length Packets using the CreatePacket() function and methods.

As of right now, setting the packets in testing to Fixed Length removes the increased leak in my own testing (I set the remove buff to a fixed length of 44) and it cut the leak in half (same as not sending it should do). So there IS progress on this report. Will keep you all updated if I can.

[Luth]

Keep up the good work.

[MuadDib]

I have sent the fix to be added to the CVS for the next 098/097.2 Release. Fix is done, although I plan to do some more rework on the code itself to add some more redundant checks to eliminate a few more possible issues.

So, as it stands, in future versions the Packet Class memory leak with VARIABLE_LEN packets should be resolved (unless more leaks are found).

My last test with the Buff example yielded a virtual ZERO memory loss, while as the original test yielded a 22 meg memory loss with my tests.

[OldnGrey]

So, you are saying that all the ways of setting up and sending packets - whether by SendPacket() or .SendPacket will be working equally well and unlike TaxMan I don't have to rewrite a bunch of script?

I can't tell you how eagerly we are waiting for this fix. It's worth a release on its own in my opinion. Well done, and welcome home Muad'Dib.

[Nando]

And just to make you guys happier: just found the leak in SendAOSTooltips(), will commit for 097.2 and 098

[Yukiko]

Thanks for the hard work guys.
POL developers are the BEST!

[MuadDib]

I been bored lately Yukiko. As you can tell. I released a near final version of that Parser in the POL Tools section, and Nando and I have a brand new project we are jumping in. (Already began it actually).

I MIGHT, MIGHT, MIGHT have found a leak in AuxSvc too (while I was writing a new security feature for the open ports they use). Will have to test later. But in future 098 versions you will be able to set IPMATCH lines in the AuxSvc.cfg like in servers.cfg. Can we say ip specific access only for AuxSvc? YAY!

Please remember, any memory leaks are HARD to find, and we need all the help we can get in finding them. If you have a way to reproduce, SEND IT ON BEBE! Sometimes it's a quick find, others it's not. But every bit helps

[OldnGrey]

A report on 097.2

With 097.1 after 24 hours it was at 1.02GB of RAM usage for pol.exe.
At 24 hours with 097.2 it was at 964MB. The difference between 4 hours uptime and 24 hours uptime was around 50MB.
(All realms activated, 30,000npcs, 87K top level items)

So it's too early to tell for sure, but I already see a huge improvement in RAM usage and will be extending the restart timer to see how far I can go before I hit the old amount of 1.02GB.

[OldnGrey]

Good heavens.
That's a first. POL actually seems to have gone DOWN in ram usage over time. I've extended the uptime by a few hours and so far at 29 hours it's LESS than I reported for 24 hours on the previous run.

Switching to 36 hour restart, then I will increase it gradually and see what we can do. I may not bother letting you know how I went, as far I am concerned it is unlikely to ever reach what it used to reach on 097.1. With no dump files and a stable server it's now other reasons apart from ram usage that will decide uptime for us.

Well done.