[probably fixed] Tool tips memory leak&lag
Moderator: POL Developer
-
- Neophyte Poster
- Posts: 31
- Joined: Wed Aug 29, 2007 2:44 pm
[probably fixed] Tool tips memory leak&lag
Recently we were facing problems with lags and memory leaks. It usually started after there was more then 80 players. After that, in few hours and jumps (each jump cca 300MB), memory jumped to 3GB and server became totally unpredictible. Lags started when the memory reached over 2.5GB (after first jump).
We shut down toolstips and that problem disappeard. Server is holding around 2GB without any problem.
We have now more then 130 players in the peak and quite big data, pol allocates cca 2GB after start.
We shut down toolstips and that problem disappeard. Server is holding around 2GB without any problem.
We have now more then 130 players in the peak and quite big data, pol allocates cca 2GB after start.
We are using latest RC5. And by shuting down tooltips Demostenes means that we changed UOFeatureEnable from 0x1a0 to 0x00 -> so we disabled little more features, but I think tooltips was the reason for memory leaks and lags.
This memory leaks are quite strange - it just "jump" in few minutes/sec from 2,250GB to 2,550+, then for few hours hold there and then "jump" again to 2850+ .., but this happens only after few hours ( we have auto restarts in 4:00am and first "jump" is in 17:00+, maybe number of players having something to do with it ).
After setting UOFeatureEnable=0x00 is everything OK server is running second day without any problems with lags or memory.
This memory leaks are quite strange - it just "jump" in few minutes/sec from 2,250GB to 2,550+, then for few hours hold there and then "jump" again to 2850+ .., but this happens only after few hours ( we have auto restarts in 4:00am and first "jump" is in 17:00+, maybe number of players having something to do with it ).
After setting UOFeatureEnable=0x00 is everything OK server is running second day without any problems with lags or memory.
I have same problem after hooking megacliloc packet.
Here is my problem related to memory leak:
http://forums.polserver.com/ftopic1880.php
It happens after ~ 24h uptime.
Here is my problem related to memory leak:
http://forums.polserver.com/ftopic1880.php
It happens after ~ 24h uptime.
-
- Neophyte Poster
- Posts: 31
- Joined: Wed Aug 29, 2007 2:44 pm
It seems that the our problem with memory "jumps" has only little to do with tooltips after all, turning tooltip off has only slown down the problem 10x +- ( as Demostenes write before me ), maybe it is in all packet hooks and tooltips was just one of most used, or maybe it is somewhere else, i dont really know, but now server can work for few days, before first memory jump occurs.MuadDib wrote:Tooltips themselves are generated if a client requests it, on the spot.
Would be nice for someone to report any difference between with a hook, and without. Might help narrow it down some too.
Re: Tool tips memory leak&lag
The problem in this function -> CreatePacket(...,...);
CreatePacket( ) == Memory leak...
use - > SendPacket( ...,...);
look here
http://forums.polserver.com/viewtopic.p ... eak#p12631
CreatePacket( ) == Memory leak...
use - > SendPacket( ...,...);
look here
http://forums.polserver.com/viewtopic.p ... eak#p12631
Re: Tool tips memory leak&lag
Nando and me (nando, don't leave me hangin here, LOL)are gonna try to look into the leaks in CreatePacket(). I have no internet at home, but, I AM BACK BIZNITCHES.
guialtran: thanks for your post with the code that caused the leaks. I just downloaded it and plan on going over it during this week.
guialtran: thanks for your post with the code that caused the leaks. I just downloaded it and plan on going over it during this week.
Re: Tool tips memory leak&lag
Just an update, we are working on it. Will take some digging, but it MAY fix a lot more leaks than just this individual one. Don't get your hopes up, but it may involve a bigger leak that affects all Variable Length Packets using the CreatePacket() function and methods.
As of right now, setting the packets in testing to Fixed Length removes the increased leak in my own testing (I set the remove buff to a fixed length of 44) and it cut the leak in half (same as not sending it should do). So there IS progress on this report. Will keep you all updated if I can.
As of right now, setting the packets in testing to Fixed Length removes the increased leak in my own testing (I set the remove buff to a fixed length of 44) and it cut the leak in half (same as not sending it should do). So there IS progress on this report. Will keep you all updated if I can.
Re: Tool tips memory leak&lag
Keep up the good work.
Re: Tool tips memory leak&lag
I have sent the fix to be added to the CVS for the next 098/097.2 Release. Fix is done, although I plan to do some more rework on the code itself to add some more redundant checks to eliminate a few more possible issues.
So, as it stands, in future versions the Packet Class memory leak with VARIABLE_LEN packets should be resolved (unless more leaks are found).
My last test with the Buff example yielded a virtual ZERO memory loss, while as the original test yielded a 22 meg memory loss with my tests.
So, as it stands, in future versions the Packet Class memory leak with VARIABLE_LEN packets should be resolved (unless more leaks are found).
My last test with the Buff example yielded a virtual ZERO memory loss, while as the original test yielded a 22 meg memory loss with my tests.
Re: Tool tips memory leak&lag
So, you are saying that all the ways of setting up and sending packets - whether by SendPacket() or .SendPacket will be working equally well and unlike TaxMan I don't have to rewrite a bunch of script?
I can't tell you how eagerly we are waiting for this fix. It's worth a release on its own in my opinion. Well done, and welcome home Muad'Dib.
I can't tell you how eagerly we are waiting for this fix. It's worth a release on its own in my opinion. Well done, and welcome home Muad'Dib.
Re: Tool tips memory leak&lag
And just to make you guys happier: just found the leak in SendAOSTooltips(), will commit for 097.2 and 098
Re: [probably fixed] Tool tips memory leak&lag
Thanks for the hard work guys.
POL developers are the BEST!
POL developers are the BEST!
Re: [probably fixed] Tool tips memory leak&lag
I been bored lately Yukiko. As you can tell. I released a near final version of that Parser in the POL Tools section, and Nando and I have a brand new project we are jumping in. (Already began it actually).
I MIGHT, MIGHT, MIGHT have found a leak in AuxSvc too (while I was writing a new security feature for the open ports they use). Will have to test later. But in future 098 versions you will be able to set IPMATCH lines in the AuxSvc.cfg like in servers.cfg. Can we say ip specific access only for AuxSvc? YAY!
Please remember, any memory leaks are HARD to find, and we need all the help we can get in finding them. If you have a way to reproduce, SEND IT ON BEBE! Sometimes it's a quick find, others it's not. But every bit helps
I MIGHT, MIGHT, MIGHT have found a leak in AuxSvc too (while I was writing a new security feature for the open ports they use). Will have to test later. But in future 098 versions you will be able to set IPMATCH lines in the AuxSvc.cfg like in servers.cfg. Can we say ip specific access only for AuxSvc? YAY!
Please remember, any memory leaks are HARD to find, and we need all the help we can get in finding them. If you have a way to reproduce, SEND IT ON BEBE! Sometimes it's a quick find, others it's not. But every bit helps
Re: [probably fixed] Tool tips memory leak&lag
A report on 097.2
With 097.1 after 24 hours it was at 1.02GB of RAM usage for pol.exe.
At 24 hours with 097.2 it was at 964MB. The difference between 4 hours uptime and 24 hours uptime was around 50MB.
(All realms activated, 30,000npcs, 87K top level items)
So it's too early to tell for sure, but I already see a huge improvement in RAM usage and will be extending the restart timer to see how far I can go before I hit the old amount of 1.02GB.
With 097.1 after 24 hours it was at 1.02GB of RAM usage for pol.exe.
At 24 hours with 097.2 it was at 964MB. The difference between 4 hours uptime and 24 hours uptime was around 50MB.
(All realms activated, 30,000npcs, 87K top level items)
So it's too early to tell for sure, but I already see a huge improvement in RAM usage and will be extending the restart timer to see how far I can go before I hit the old amount of 1.02GB.
Re: [probably fixed] Tool tips memory leak&lag
Good heavens.
That's a first. POL actually seems to have gone DOWN in ram usage over time. I've extended the uptime by a few hours and so far at 29 hours it's LESS than I reported for 24 hours on the previous run.
Switching to 36 hour restart, then I will increase it gradually and see what we can do. I may not bother letting you know how I went, as far I am concerned it is unlikely to ever reach what it used to reach on 097.1. With no dump files and a stable server it's now other reasons apart from ram usage that will decide uptime for us.
Well done.
That's a first. POL actually seems to have gone DOWN in ram usage over time. I've extended the uptime by a few hours and so far at 29 hours it's LESS than I reported for 24 hours on the previous run.
Switching to 36 hour restart, then I will increase it gradually and see what we can do. I may not bother letting you know how I went, as far I am concerned it is unlikely to ever reach what it used to reach on 097.1. With no dump files and a stable server it's now other reasons apart from ram usage that will decide uptime for us.
Well done.