Packethook leak

Report core bugs regarding the Ultima Online Emulator Core release (version 097). You can attach your Core Dump. One bug per post.

Moderator: POL Developer

Bracco
Adept Poster
Posts: 80
Joined: Thu Dec 28, 2006 11:52 am

Packethook leak

Post by Bracco »

there is a memory leak in packethooks, unfortunately it isn't related to what code it's inside them, it just happens when a packet is sent or received and there's a packethook for it.
it appears to affect both receivefunction and sendfunction

core version: 097 2007-01-04 RC2 Coregina
platform: windows

i did some testing, starting with a very simple and empty packethook:

uopacket.cfg:

Code: Select all

Packet 0x11
{
  Length variable
  SendFunction stat:status
}
stat.src:

Code: Select all

exported function status(who, byref packet)
   return 0;
endfunction
to test it i ran a injection macro that continuously equips and unequips armor, so the status packet is being sent continuously because of the armor value change.
there was only 1 player online, me, and i am sure my server alone doesn't grow in memory usage: i let it run for hours without players without getting even a mb increase

these are the results: (i disabled pagefile to make task manager more readable, with pagefile active windows continuously swaps something from physical ram to pagefile)

beginning:
Image

after 10 minutes:
Image

as you can see, the ram usage grew about 8 MB, but that's with only 1 player doing this and for only 10 minutes. being a packethook leak, and given the fact that most shards will probably have more than 1 packethook, you can imagine how bad could the leak turn out with many players.

then i tried adding some code to the hook, and this is the result:

beginning:
Image

after 10 minutes:
Image

as you can see the memory usage is pratically identical to the one without any code inside the hook

i then disabled the hook and ran the same test:

beginning:
Image
after 10 minutes:
Image

without the hook there's no leak, just a very slight variation probably due to some script running in background

that's as much info as i can, probably this is the leak that people (including me) complained for a while, hope it's easyly fixable so we could hope to see the fix in the next release ;)
blkwr
New User
Posts: 26
Joined: Wed Feb 08, 2006 1:47 pm

Post by blkwr »

anyone that reply here?
User avatar
CWO
POL Expert
Posts: 1159
Joined: Sat Feb 04, 2006 5:49 pm
Location: Chicago, IL USA

Post by CWO »

Knowing the situation, the core devs might not even reply but I'm sure they're looking at it. A couple things I've reported before never got a reply but got fixed in the next release or two.
User avatar
MontuZ
Forum Regular
Posts: 338
Joined: Fri Feb 10, 2006 8:08 am
Location: Myrtle Beach, South Carolina

Post by MontuZ »

Do the test over again, only next time run the server for like more than one hour, at least. Maybe an entire day.

If you can. 8)
MuadDib
Former Developer
Posts: 1091
Joined: Sun Feb 12, 2006 9:50 pm
Location: Cross Lanes, WV

Post by MuadDib »

Can you try this with a fixed length hook and see if it still leaks with that? And post the results of course also.
User avatar
OldnGrey
POL Expert
Posts: 657
Joined: Sat Feb 04, 2006 6:26 pm

Post by OldnGrey »

I introduced the megacliloc packethook this week and watched as server ram usage grew. Before this PH the RAM usage was around 640MB, but now after only 25 hours it's up to 1.2GB and climbing. That's a massive difference and enough to make me reduce my restart timer down to 25 hours.

It seems to be related to the number of times players use tooltips.
My test server has the same code but no players and is only at 700MB after 22 hours. The packethook ecl is around 15KB in size. It has to be to display all the stuff we want.
User avatar
itec
Novice Poster
Posts: 42
Joined: Thu Feb 09, 2006 11:48 pm
Location: Finland

Post by itec »

Any progress about this? After we updated from 96 to 97 RC3, the system became laggy after it had been running for a day or so. And the cause appears to be memory leek. As many of our systems depends on packethooks, we can't really do much, only way out seems to convert back to 96, and that's not progress at all. Not to mention how much futile work it would need. The whole shard is in somewhat slump at the moment, it is really deadly poison for it, (and eventually for pol community as well).

We also have a few crash dumps most likely caused by this, after the system had been running for several days. If those are of any help.
Justae
Expert Poster
Posts: 79
Joined: Thu May 24, 2007 2:12 pm
Location: South Africa

Definitive memory leak with hooks.

Post by Justae »

Exactly the same issue here and it's not just megacliloc. Megacliloc show's up easier as the hook that's "leaking" most probably because it's the most used. Other hooks also slowly, over time, hog memory. To a lesser degree, but it all adds-up.
User avatar
itec
Novice Poster
Posts: 42
Joined: Thu Feb 09, 2006 11:48 pm
Location: Finland

Post by itec »

Can someone confirm if there's a leak in RC1 as well? At least we can say that 96.6 does not contain any memory leak (worked fine with me), and RC 2 & 3 does.

I wish devs would make more warning announcements about thease things, people who are going to update, would be saved from some serious headache if they would knew about thease things. Althought at the moment seems like all the devs are on vacant from Pol. :sad:

You should keep up the good work, it may not always seems like it, but it's really appreciated. :cool:
User avatar
OldnGrey
POL Expert
Posts: 657
Joined: Sat Feb 04, 2006 6:26 pm

Post by OldnGrey »

Are you sure that 09.6 doesn't have memory leaks in packet hooks?

It was only when I enabled the megacliloc packethook on a live shard that I noticed the problem at all. If the packet isn't sent it doesn't grow the RAM. That's why on my test server it seemed perfectly fine.

From what I hear, this won't be looked at until 098 is being developed.
Developer Silver
Apprentice Poster
Posts: 58
Joined: Sun Feb 05, 2006 1:34 pm
Location: Italy

Post by Developer Silver »

I think we have same problem here (097 RC3 Linux): after 2/3 days POL absolutely need to be restarted because WorldSaves raise from 4 to 80 seconds, sysload from 1 to 90%, all scripts become deadly slow and memory usage grows up to fill whole system RAM (anyway this is not too strange under Linux) :(
We have a lot of hooks (more than 30) and they can't be disabled, so we'll hold on and hope devs can found a solution! :)
User avatar
itec
Novice Poster
Posts: 42
Joined: Thu Feb 09, 2006 11:48 pm
Location: Finland

Post by itec »

OldnGrey wrote:Are you sure that 09.6 doesn't have memory leaks in packet hooks?
I'm certain that problems started when we updated our live server to 97, and if my memory works correctly, we did indeed have 96.6 before that. I've had 97 for a long time as development server, and yeah, you couldn't notice any memory leaks when nobody used the hooks.

I really hope the problem would be solved in 97, since seems like every live 97 shard that uses packet hooks, is troubled about it. Only way to keep the shard playable is to reboot pol every other night (or better, every night) and it still doesn't eliminate the lag totally.
User avatar
OldnGrey
POL Expert
Posts: 657
Joined: Sat Feb 04, 2006 6:26 pm

Post by OldnGrey »

We are having to reboot every 9 hours, so I'd love to do it every second day! In test I regularly set it to a week with no issues of ram growth.

We use Win2k server, and I spent a lot of time tuning the megacliloc messages to be as brief as possible. Even so, we are growing at 25MB per hour. With only 1GB in the server, I was setting the restart timer very carefully to avoid swapping to hard disk. There was no noticeable lag with short messages and no disk swapping. Win2K isn't too bad at swapping out the least used code, so it got so that POL could use nearly all the RAM, but even so, a lot of players online all using the mouse-hover tooltips is a ram eater like I've never seen before.

We can live with it and I am almost sure the devs said it won't be fixed in 096/097. RAM isn't expensive.
blkwr
New User
Posts: 26
Joined: Wed Feb 08, 2006 1:47 pm

Post by blkwr »

OldnGrey wrote:We are having to reboot every 9 hours, so I'd love to do it every second day! In test I regularly set it to a week with no issues of ram growth.

We use Win2k server, and I spent a lot of time tuning the megacliloc messages to be as brief as possible. Even so, we are growing at 25MB per hour. With only 1GB in the server, I was setting the restart timer very carefully to avoid swapping to hard disk. There was no noticeable lag with short messages and no disk swapping. Win2K isn't too bad at swapping out the least used code, so it got so that POL could use nearly all the RAM, but even so, a lot of players online all using the mouse-hover tooltips is a ram eater like I've never seen before.

We can live with it and I am almost sure the devs said it won't be fixed in 096/097. RAM isn't expensive.

Yes but if the devs verify all possible memory leak we could use a server that not have to restart itself every day or every X hours..
Shinigami
Former Developer
Posts: 308
Joined: Mon Jan 30, 2006 9:28 am
Location: Germany, Bavaria
Contact:

Post by Shinigami »

Sounds strange... we'll investigate here...

Shinigami
blkwr
New User
Posts: 26
Joined: Wed Feb 08, 2006 1:47 pm

Post by blkwr »

Shinigami wrote:Sounds strange... we'll investigate here...

Shinigami

thanks :)
Developer Silver
Apprentice Poster
Posts: 58
Joined: Sun Feb 05, 2006 1:34 pm
Location: Italy

Post by Developer Silver »

Thanks you VERY MUCH :D
User avatar
ncrsn
Grandmaster Poster
Posts: 255
Joined: Fri Feb 10, 2006 12:15 am

Post by ncrsn »

Any progress in this?
mr bubbles
Grandmaster Poster
Posts: 120
Joined: Thu Jan 18, 2007 2:34 am

Post by mr bubbles »

Im interested too, crashing every 2nd day because of it.

Although since upgrading to the latest core it doesn't actually 'crash', just drops all players and gives bad allocation to every script in console, and crazy messages like Exception in i/o thread (checkpoint=8, what=badallocation), etc
blkwr
New User
Posts: 26
Joined: Wed Feb 08, 2006 1:47 pm

Post by blkwr »

any news? :?
mr bubbles
Grandmaster Poster
Posts: 120
Joined: Thu Jan 18, 2007 2:34 am

Post by mr bubbles »

Getting annoying now. Any word on a fix yet? Bit hard running a reliable shard when it crashes every few days due to this :sad:
RazorTongue
New User
Posts: 28
Joined: Mon Apr 23, 2007 8:18 am

Post by RazorTongue »

That's why POL 97 is a RC, not a stable release...
mr bubbles
Grandmaster Poster
Posts: 120
Joined: Thu Jan 18, 2007 2:34 am

Post by mr bubbles »

Yes I know that. But this has been a problem for a while now, so thats why im asking when it is being fixed. I want KR support and the other new stuff too, but i think a problem as bad as this should be fixed first :/
Shinigami
Former Developer
Posts: 308
Joined: Mon Jan 30, 2006 9:28 am
Location: Germany, Bavaria
Contact:

Post by Shinigami »

the problem is fixed and tested on a live shard. an updated core will be available asap (it's planned for tomorrow).

Shinigami
User avatar
ncrsn
Grandmaster Poster
Posts: 255
Joined: Fri Feb 10, 2006 12:15 am

Post by ncrsn »

That's! Just! Great!
Locked