[probably fixed] Tool tips memory leak&lag

Bug Reports relating to the 097 core are moved to this forum once solved.

Moderator: POL Developer

Demostenes
New User
Posts: 24
Joined: Wed Aug 29, 2007 2:44 pm
Contact:

[probably fixed] Tool tips memory leak&lag

Post by Demostenes » Wed Oct 24, 2007 4:46 am

Recently we were facing problems with lags and memory leaks. It usually started after there was more then 80 players. After that, in few hours and jumps (each jump cca 300MB), memory jumped to 3GB and server became totally unpredictible. Lags started when the memory reached over 2.5GB (after first jump).
We shut down toolstips and that problem disappeard. Server is holding around 2GB without any problem.


We have now more then 130 players in the peak and quite big data, pol allocates cca 2GB after start.

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Post by MuadDib » Wed Oct 24, 2007 9:59 am

What core are you using?

Keryan
New User
Posts: 11
Joined: Fri Aug 10, 2007 3:30 pm

Post by Keryan » Wed Oct 24, 2007 11:10 am

We are using latest RC5. And by shuting down tooltips Demostenes means that we changed UOFeatureEnable from 0x1a0 to 0x00 -> so we disabled little more features, but I think tooltips was the reason for memory leaks and lags.
This memory leaks are quite strange - it just "jump" in few minutes/sec from 2,250GB to 2,550+, then for few hours hold there and then "jump" again to 2850+ .., but this happens only after few hours ( we have auto restarts in 4:00am and first "jump" is in 17:00+, maybe number of players having something to do with it ).

After setting UOFeatureEnable=0x00 is everything OK server is running second day without any problems with lags or memory.

qrak
Grandmaster Poster
Posts: 198
Joined: Sun Feb 05, 2006 4:35 pm
Location: Poland

Post by qrak » Wed Oct 24, 2007 1:12 pm

I have same problem after hooking megacliloc packet.

Here is my problem related to memory leak:
http://forums.polserver.com/ftopic1880.php

It happens after ~ 24h uptime.

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Post by MuadDib » Wed Oct 24, 2007 1:12 pm

Could you try setting Feature Enable back to how it was, and just disable that packet hook?

Demostenes
New User
Posts: 24
Joined: Wed Aug 29, 2007 2:44 pm
Contact:

Post by Demostenes » Sun Oct 28, 2007 3:49 pm

Small update: Memory is still leaking, but many times slower (cca 10x). With previous version of POL we were running (0.95) there were no problems, so we are suspecting there is some problem with new core.

tartaros
New User
Posts: 28
Joined: Tue Mar 27, 2007 6:30 am
Contact:

Post by tartaros » Wed Oct 31, 2007 7:55 am

MuadDib: out of curiosity, what algorhitm/memory scheme do you use for caching the tooltips? If any...
Or are the packets being generated over and over again every time any client asks for them?

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Post by MuadDib » Wed Oct 31, 2007 8:19 am

Tooltips themselves are generated if a client requests it, on the spot.

Would be nice for someone to report any difference between with a hook, and without. Might help narrow it down some too.

coltain
Grandmaster Poster
Posts: 159
Joined: Tue Mar 20, 2007 7:17 am
Location: Poland

Post by coltain » Wed Oct 31, 2007 8:22 am

I can report some...

As I tested there was no difference if the packet was hooked or not (I even moved the directory away)

I had UoFeatureenable set 0x1a8 (to show the single click window)

Keryan
New User
Posts: 11
Joined: Fri Aug 10, 2007 3:30 pm

Post by Keryan » Thu Nov 01, 2007 4:41 pm

MuadDib wrote:Tooltips themselves are generated if a client requests it, on the spot.

Would be nice for someone to report any difference between with a hook, and without. Might help narrow it down some too.
It seems that the our problem with memory "jumps" has only little to do with tooltips after all, turning tooltip off has only slown down the problem 10x +- :) ( as Demostenes write before me ), maybe it is in all packet hooks and tooltips was just one of most used, or maybe it is somewhere else, i dont really know, but now server can work for few days, before first memory jump occurs.

Demostenes
New User
Posts: 24
Joined: Wed Aug 29, 2007 2:44 pm
Contact:

Post by Demostenes » Mon Feb 18, 2008 11:25 am

Any progress with this?

Shinigami
Former Developer
Posts: 308
Joined: Mon Jan 30, 2006 9:28 am
Location: Germany, Bavaria
Contact:

Post by Shinigami » Wed Feb 27, 2008 1:37 pm

just try latest core...

Shinigami

Keryan
New User
Posts: 11
Joined: Fri Aug 10, 2007 3:30 pm

Post by Keryan » Wed Feb 27, 2008 2:17 pm

Shinigami wrote:just try latest core...

Shinigami
Thx for the new release !

Sure, i am going to test it right now on our shard. Will post results in few days.

coltain
Grandmaster Poster
Posts: 159
Joined: Tue Mar 20, 2007 7:17 am
Location: Poland

Post by coltain » Sun May 04, 2008 10:30 am

Any progress???

This allocation problem makes me tired:(

guialtran
Grandmaster Poster
Posts: 120
Joined: Wed Jul 30, 2008 12:42 pm

Re: Tool tips memory leak&lag

Post by guialtran » Wed Dec 03, 2008 11:46 am

The problem in this function -> CreatePacket(...,...);
CreatePacket( ) == Memory leak...
use - > SendPacket( ...,...);


look here
http://forums.polserver.com/viewtopic.p ... eak#p12631

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Re: Tool tips memory leak&lag

Post by MuadDib » Sun Dec 14, 2008 9:39 am

Nando and me (nando, don't leave me hangin here, LOL)are gonna try to look into the leaks in CreatePacket(). I have no internet at home, but, I AM BACK BIZNITCHES.

guialtran: thanks for your post with the code that caused the leaks. I just downloaded it and plan on going over it during this week.

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Re: Tool tips memory leak&lag

Post by MuadDib » Mon Dec 15, 2008 5:10 pm

Just an update, we are working on it. Will take some digging, but it MAY fix a lot more leaks than just this individual one. Don't get your hopes up, but it may involve a bigger leak that affects all Variable Length Packets using the CreatePacket() function and methods.

As of right now, setting the packets in testing to Fixed Length removes the increased leak in my own testing (I set the remove buff to a fixed length of 44) and it cut the leak in half (same as not sending it should do). So there IS progress on this report. Will keep you all updated if I can.

Luth
Former Developer
Posts: 200
Joined: Mon Jan 30, 2006 8:10 am

Re: Tool tips memory leak&lag

Post by Luth » Mon Dec 15, 2008 5:13 pm

Keep up the good work. :)

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Re: Tool tips memory leak&lag

Post by MuadDib » Tue Dec 16, 2008 4:41 pm

I have sent the fix to be added to the CVS for the next 098/097.2 Release. Fix is done, although I plan to do some more rework on the code itself to add some more redundant checks to eliminate a few more possible issues.

So, as it stands, in future versions the Packet Class memory leak with VARIABLE_LEN packets should be resolved (unless more leaks are found).

My last test with the Buff example yielded a virtual ZERO memory loss, while as the original test yielded a 22 meg memory loss with my tests.

User avatar
OldnGrey
POL Expert
Posts: 657
Joined: Sat Feb 04, 2006 6:26 pm

Re: Tool tips memory leak&lag

Post by OldnGrey » Tue Dec 16, 2008 6:49 pm

So, you are saying that all the ways of setting up and sending packets - whether by SendPacket() or .SendPacket will be working equally well and unlike TaxMan I don't have to rewrite a bunch of script?

I can't tell you how eagerly we are waiting for this fix. It's worth a release on its own in my opinion. Well done, and welcome home Muad'Dib.

Nando
POL Developer
Posts: 267
Joined: Wed Sep 17, 2008 6:53 pm
Contact:

Re: Tool tips memory leak&lag

Post by Nando » Wed Dec 17, 2008 6:19 am

And just to make you guys happier: just found the leak in SendAOSTooltips(), will commit for 097.2 and 098 :)

Yukiko
Distro Developer
Posts: 2741
Joined: Thu Feb 02, 2006 1:41 pm
Location: San Antonio, Texas
Contact:

Re: [probably fixed] Tool tips memory leak&lag

Post by Yukiko » Thu Dec 18, 2008 11:48 pm

Thanks for the hard work guys.
POL developers are the BEST!

MuadDib
Former Developer
Posts: 1090
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Re: [probably fixed] Tool tips memory leak&lag

Post by MuadDib » Fri Dec 19, 2008 4:13 pm

I been bored lately Yukiko. As you can tell. I released a near final version of that Parser in the POL Tools section, and Nando and I have a brand new project we are jumping in. (Already began it actually).

I MIGHT, MIGHT, MIGHT have found a leak in AuxSvc too (while I was writing a new security feature for the open ports they use). Will have to test later. But in future 098 versions you will be able to set IPMATCH lines in the AuxSvc.cfg like in servers.cfg. Can we say ip specific access only for AuxSvc? YAY!

Please remember, any memory leaks are HARD to find, and we need all the help we can get in finding them. If you have a way to reproduce, SEND IT ON BEBE! Sometimes it's a quick find, others it's not. But every bit helps :)

User avatar
OldnGrey
POL Expert
Posts: 657
Joined: Sat Feb 04, 2006 6:26 pm

Re: [probably fixed] Tool tips memory leak&lag

Post by OldnGrey » Tue Jan 20, 2009 10:16 pm

A report on 097.2

With 097.1 after 24 hours it was at 1.02GB of RAM usage for pol.exe.
At 24 hours with 097.2 it was at 964MB. The difference between 4 hours uptime and 24 hours uptime was around 50MB.
(All realms activated, 30,000npcs, 87K top level items)

So it's too early to tell for sure, but I already see a huge improvement in RAM usage and will be extending the restart timer to see how far I can go before I hit the old amount of 1.02GB.

User avatar
OldnGrey
POL Expert
Posts: 657
Joined: Sat Feb 04, 2006 6:26 pm

Re: [probably fixed] Tool tips memory leak&lag

Post by OldnGrey » Thu Jan 22, 2009 4:46 am

Good heavens.
That's a first. POL actually seems to have gone DOWN in ram usage over time. I've extended the uptime by a few hours and so far at 29 hours it's LESS than I reported for 24 hours on the previous run.

Switching to 36 hour restart, then I will increase it gradually and see what we can do. I may not bother letting you know how I went, as far I am concerned it is unlikely to ever reach what it used to reach on 097.1. With no dump files and a stable server it's now other reasons apart from ram usage that will decide uptime for us.

Well done.

Locked