PenUltima Online Forum Index Official Core: 096.7
Official Core: 097 2008-02-26
Donate towards the POL web hosting bill!
 POL Home   FAQ   Search    Memberlist   Usergroups    Register    Profile   Log in to check your private messages   Log in
Packethook leak
Goto page 1, 2  Next
 
Post new topic   Reply to topic    PenUltima Online Forum Index -> Bug Reports 097
Display posts from previous:   

Author Message
Bracco



Joined: 28 Dec 2006
Posts: 80

PostPosted: Sat Mar 17, 2007 5:39 am    Post subject: Packethook leak Reply with quote

there is a memory leak in packethooks, unfortunately it isn't related to what code it's inside them, it just happens when a packet is sent or received and there's a packethook for it.
it appears to affect both receivefunction and sendfunction

core version: 097 2007-01-04 RC2 Coregina
platform: windows

i did some testing, starting with a very simple and empty packethook:

uopacket.cfg:
Code:
Packet 0x11
{
  Length variable
  SendFunction stat:status
}


stat.src:
Code:

exported function status(who, byref packet)
   return 0;
endfunction


to test it i ran a injection macro that continuously equips and unequips armor, so the status packet is being sent continuously because of the armor value change.
there was only 1 player online, me, and i am sure my server alone doesn't grow in memory usage: i let it run for hours without players without getting even a mb increase

these are the results: (i disabled pagefile to make task manager more readable, with pagefile active windows continuously swaps something from physical ram to pagefile)

beginning:


after 10 minutes:


as you can see, the ram usage grew about 8 MB, but that's with only 1 player doing this and for only 10 minutes. being a packethook leak, and given the fact that most shards will probably have more than 1 packethook, you can imagine how bad could the leak turn out with many players.

then i tried adding some code to the hook, and this is the result:

beginning:


after 10 minutes:


as you can see the memory usage is pratically identical to the one without any code inside the hook

i then disabled the hook and ran the same test:

beginning:

after 10 minutes:


without the hook there's no leak, just a very slight variation probably due to some script running in background

that's as much info as i can, probably this is the leak that people (including me) complained for a while, hope it's easyly fixable so we could hope to see the fix in the next release Wink

Author Message
blkwr



Joined: 08 Feb 2006
Posts: 8

PostPosted: Sun Mar 18, 2007 4:46 pm    Post subject: Reply with quote

anyone that reply here?

Author Message
CWO



Joined: 04 Feb 2006
Posts: 700
Location: Chicago, IL USA

PostPosted: Sun Mar 18, 2007 5:10 pm    Post subject: Reply with quote

Knowing the situation, the core devs might not even reply but I'm sure they're looking at it. A couple things I've reported before never got a reply but got fixed in the next release or two.

Author Message
MontuZ
Distro Developer


Joined: 10 Feb 2006
Posts: 297
Location: Myrtle Beach, South Carolina

PostPosted: Sun Mar 18, 2007 8:25 pm    Post subject: Reply with quote

Do the test over again, only next time run the server for like more than one hour, at least. Maybe an entire day.

If you can. Cool

Author Message
MuadDib
POL Developer


Joined: 13 Feb 2006
Posts: 830
Location: Indiana, USA

PostPosted: Thu Mar 29, 2007 1:08 am    Post subject: Reply with quote

Can you try this with a fixed length hook and see if it still leaks with that? And post the results of course also.

Author Message
OldnGrey



Joined: 04 Feb 2006
Posts: 523

PostPosted: Tue Apr 10, 2007 7:11 am    Post subject: Reply with quote

I introduced the megacliloc packethook this week and watched as server ram usage grew. Before this PH the RAM usage was around 640MB, but now after only 25 hours it's up to 1.2GB and climbing. That's a massive difference and enough to make me reduce my restart timer down to 25 hours.

It seems to be related to the number of times players use tooltips.
My test server has the same code but no players and is only at 700MB after 22 hours. The packethook ecl is around 15KB in size. It has to be to display all the stuff we want.

Author Message
itec



Joined: 10 Feb 2006
Posts: 35
Location: Finland

PostPosted: Sat Jun 23, 2007 11:49 am    Post subject: Reply with quote

Any progress about this? After we updated from 96 to 97 RC3, the system became laggy after it had been running for a day or so. And the cause appears to be memory leek. As many of our systems depends on packethooks, we can't really do much, only way out seems to convert back to 96, and that's not progress at all. Not to mention how much futile work it would need. The whole shard is in somewhat slump at the moment, it is really deadly poison for it, (and eventually for pol community as well).

We also have a few crash dumps most likely caused by this, after the system had been running for several days. If those are of any help.

Author Message
Justae



Joined: 24 May 2007
Posts: 25
Location: South Africa

PostPosted: Sat Jun 23, 2007 1:10 pm    Post subject: Definitive memory leak with hooks. Reply with quote

Exactly the same issue here and it's not just megacliloc. Megacliloc show's up easier as the hook that's "leaking" most probably because it's the most used. Other hooks also slowly, over time, hog memory. To a lesser degree, but it all adds-up.

Author Message
itec



Joined: 10 Feb 2006
Posts: 35
Location: Finland

PostPosted: Mon Jun 25, 2007 4:27 am    Post subject: Reply with quote

Can someone confirm if there's a leak in RC1 as well? At least we can say that 96.6 does not contain any memory leak (worked fine with me), and RC 2 & 3 does.

I wish devs would make more warning announcements about thease things, people who are going to update, would be saved from some serious headache if they would knew about thease things. Althought at the moment seems like all the devs are on vacant from Pol. Sad

You should keep up the good work, it may not always seems like it, but it's really appreciated. Cool

Author Message
OldnGrey



Joined: 04 Feb 2006
Posts: 523

PostPosted: Mon Jun 25, 2007 6:26 am    Post subject: Reply with quote

Are you sure that 09.6 doesn't have memory leaks in packet hooks?

It was only when I enabled the megacliloc packethook on a live shard that I noticed the problem at all. If the packet isn't sent it doesn't grow the RAM. That's why on my test server it seemed perfectly fine.

From what I hear, this won't be looked at until 098 is being developed.

Author Message
Developer Silver



Joined: 05 Feb 2006
Posts: 58
Location: Italy

PostPosted: Tue Jun 26, 2007 5:12 am    Post subject: Reply with quote

I think we have same problem here (097 RC3 Linux): after 2/3 days POL absolutely need to be restarted because WorldSaves raise from 4 to 80 seconds, sysload from 1 to 90%, all scripts become deadly slow and memory usage grows up to fill whole system RAM (anyway this is not too strange under Linux) Sad
We have a lot of hooks (more than 30) and they can't be disabled, so we'll hold on and hope devs can found a solution! Smile

Author Message
itec



Joined: 10 Feb 2006
Posts: 35
Location: Finland

PostPosted: Tue Jun 26, 2007 8:04 am    Post subject: Reply with quote

OldnGrey wrote:
Are you sure that 09.6 doesn't have memory leaks in packet hooks?

I'm certain that problems started when we updated our live server to 97, and if my memory works correctly, we did indeed have 96.6 before that. I've had 97 for a long time as development server, and yeah, you couldn't notice any memory leaks when nobody used the hooks.

I really hope the problem would be solved in 97, since seems like every live 97 shard that uses packet hooks, is troubled about it. Only way to keep the shard playable is to reboot pol every other night (or better, every night) and it still doesn't eliminate the lag totally.

Author Message
OldnGrey



Joined: 04 Feb 2006
Posts: 523

PostPosted: Tue Jun 26, 2007 8:15 am    Post subject: Reply with quote

We are having to reboot every 9 hours, so I'd love to do it every second day! In test I regularly set it to a week with no issues of ram growth.

We use Win2k server, and I spent a lot of time tuning the megacliloc messages to be as brief as possible. Even so, we are growing at 25MB per hour. With only 1GB in the server, I was setting the restart timer very carefully to avoid swapping to hard disk. There was no noticeable lag with short messages and no disk swapping. Win2K isn't too bad at swapping out the least used code, so it got so that POL could use nearly all the RAM, but even so, a lot of players online all using the mouse-hover tooltips is a ram eater like I've never seen before.

We can live with it and I am almost sure the devs said it won't be fixed in 096/097. RAM isn't expensive.

Author Message
blkwr



Joined: 08 Feb 2006
Posts: 8

PostPosted: Wed Jun 27, 2007 12:47 pm    Post subject: Reply with quote

OldnGrey wrote:
We are having to reboot every 9 hours, so I'd love to do it every second day! In test I regularly set it to a week with no issues of ram growth.

We use Win2k server, and I spent a lot of time tuning the megacliloc messages to be as brief as possible. Even so, we are growing at 25MB per hour. With only 1GB in the server, I was setting the restart timer very carefully to avoid swapping to hard disk. There was no noticeable lag with short messages and no disk swapping. Win2K isn't too bad at swapping out the least used code, so it got so that POL could use nearly all the RAM, but even so, a lot of players online all using the mouse-hover tooltips is a ram eater like I've never seen before.

We can live with it and I am almost sure the devs said it won't be fixed in 096/097. RAM isn't expensive.



Yes but if the devs verify all possible memory leak we could use a server that not have to restart itself every day or every X hours..

Author Message
Shinigami
POL Core Developer


Joined: 30 Jan 2006
Posts: 292
Location: Germany, Bavaria

PostPosted: Fri Jun 29, 2007 10:43 pm    Post subject: Reply with quote

Sounds strange... we'll investigate here...

Shinigami

Post new topic   Reply to topic    PenUltima Online Forum Index -> Bug Reports 097 All times are GMT - 4 Hours
Goto page 1, 2  Next
Page 1 of 2

 




Powered by phpBB © 2001, 2005 phpBB Group :: Theme & Graphics by GHS & Scott E. Royalty