PenUltima Online

there is a memory leak in packethooks, unfortunately it isn't related to what code it's inside them, it just happens when a packet is sent or received and there's a packethook for it.
it appears to affect both receivefunction and sendfunction

core version: 097 2007-01-04 RC2 Coregina
platform: windows

i did some testing, starting with a very simple and empty packethook:

uopacket.cfg:


stat.src:


to test it i ran a injection macro that continuously equips and unequips armor, so the status packet is being sent continuously because of the armor value change.
there was only 1 player online, me, and i am sure my server alone doesn't grow in memory usage: i let it run for hours without players without getting even a mb increase

these are the results: (i disabled pagefile to make task manager more readable, with pagefile active windows continuously swaps something from physical ram to pagefile)

beginning:


after 10 minutes:


as you can see, the ram usage grew about 8 MB, but that's with only 1 player doing this and for only 10 minutes. being a packethook leak, and given the fact that most shards will probably have more than 1 packethook, you can imagine how bad could the leak turn out with many players.

then i tried adding some code to the hook, and this is the result:

beginning:


after 10 minutes:


as you can see the memory usage is pratically identical to the one without any code inside the hook

i then disabled the hook and ran the same test:

beginning:

after 10 minutes:


without the hook there's no leak, just a very slight variation probably due to some script running in background

that's as much info as i can, probably this is the leak that people (including me) complained for a while, hope it's easyly fixable so we could hope to see the fix in the next release

anyone that reply here?

Knowing the situation, the core devs might not even reply but I'm sure they're looking at it. A couple things I've reported before never got a reply but got fixed in the next release or two.

Do the test over again, only next time run the server for like more than one hour, at least. Maybe an entire day.

If you can.

Can you try this with a fixed length hook and see if it still leaks with that? And post the results of course also.

I introduced the megacliloc packethook this week and watched as server ram usage grew. Before this PH the RAM usage was around 640MB, but now after only 25 hours it's up to 1.2GB and climbing. That's a massive difference and enough to make me reduce my restart timer down to 25 hours.

It seems to be related to the number of times players use tooltips.
My test server has the same code but no players and is only at 700MB after 22 hours. The packethook ecl is around 15KB in size. It has to be to display all the stuff we want.

Any progress about this? After we updated from 96 to 97 RC3, the system became laggy after it had been running for a day or so. And the cause appears to be memory leek. As many of our systems depends on packethooks, we can't really do much, only way out seems to convert back to 96, and that's not progress at all. Not to mention how much futile work it would need. The whole shard is in somewhat slump at the moment, it is really deadly poison for it, (and eventually for pol community as well).

We also have a few crash dumps most likely caused by this, after the system had been running for several days. If those are of any help.

Exactly the same issue here and it's not just megacliloc. Megacliloc show's up easier as the hook that's "leaking" most probably because it's the most used. Other hooks also slowly, over time, hog memory. To a lesser degree, but it all adds-up.

Can someone confirm if there's a leak in RC1 as well? At least we can say that 96.6 does not contain any memory leak (worked fine with me), and RC 2 & 3 does.

I wish devs would make more warning announcements about thease things, people who are going to update, would be saved from some serious headache if they would knew about thease things. Althought at the moment seems like all the devs are on vacant from Pol.

You should keep up the good work, it may not always seems like it, but it's really appreciated.

Are you sure that 09.6 doesn't have memory leaks in packet hooks?

It was only when I enabled the megacliloc packethook on a live shard that I noticed the problem at all. If the packet isn't sent it doesn't grow the RAM. That's why on my test server it seemed perfectly fine.

From what I hear, this won't be looked at until 098 is being developed.

I think we have same problem here (097 RC3 Linux): after 2/3 days POL absolutely need to be restarted because WorldSaves raise from 4 to 80 seconds, sysload from 1 to 90%, all scripts become deadly slow and memory usage grows up to fill whole system RAM (anyway this is not too strange under Linux)
We have a lot of hooks (more than 30) and they can't be disabled, so we'll hold on and hope devs can found a solution!

I'm certain that problems started when we updated our live server to 97, and if my memory works correctly, we did indeed have 96.6 before that. I've had 97 for a long time as development server, and yeah, you couldn't notice any memory leaks when nobody used the hooks.

I really hope the problem would be solved in 97, since seems like every live 97 shard that uses packet hooks, is troubled about it. Only way to keep the shard playable is to reboot pol every other night (or better, every night) and it still doesn't eliminate the lag totally.

We are having to reboot every 9 hours, so I'd love to do it every second day! In test I regularly set it to a week with no issues of ram growth.

We use Win2k server, and I spent a lot of time tuning the megacliloc messages to be as brief as possible. Even so, we are growing at 25MB per hour. With only 1GB in the server, I was setting the restart timer very carefully to avoid swapping to hard disk. There was no noticeable lag with short messages and no disk swapping. Win2K isn't too bad at swapping out the least used code, so it got so that POL could use nearly all the RAM, but even so, a lot of players online all using the mouse-hover tooltips is a ram eater like I've never seen before.

We can live with it and I am almost sure the devs said it won't be fixed in 096/097. RAM isn't expensive.

Yes but if the devs verify all possible memory leak we could use a server that not have to restart itself every day or every X hours..

Sounds strange... we'll investigate here...

Shinigami

thanks

Thanks you VERY MUCH

Any progress in this?

Im interested too, crashing every 2nd day because of it.

Although since upgrading to the latest core it doesn't actually 'crash', just drops all players and gives bad allocation to every script in console, and crazy messages like Exception in i/o thread (checkpoint=8, what=badallocation), etc

any news?

Getting annoying now. Any word on a fix yet? Bit hard running a reliable shard when it crashes every few days due to this

That's why POL 97 is a RC, not a stable release...

Yes I know that. But this has been a problem for a while now, so thats why im asking when it is being fixed. I want KR support and the other new stuff too, but i think a problem as bad as this should be fixed first :/

the problem is fixed and tested on a live shard. an updated core will be available asap (it's planned for tomorrow).

Shinigami

That's! Just! Great!