'bad allocation' errors

[taxman]

POL097-2008-02-26 Coregina (VS.NET 2003)
compiled on Feb 26 2008 22:16:06

After ~60 hours of uptime have this:

Code: Select all

[03/08 23:38:56] Exception in: scripts/ai/killPCs.eclze() doesn't take parameters.: bad allocation
 PC < nLines: (ndex) requires a parameter.unsigned1399:  1399: get member id 'z' (2)
1400:  1400: global #3
1401:  1401: get member id 'realm' (113)
1402:  1402: 1L
1403:  1403: 7L
1404:  1404: Func(4,28): FindPath
1405: >1405: local2 := 
1406:  1406: local #2
1407:  1407: 1L
1408:  1408: [] 1
1409:  1409: get member id 'x' (0)
Exception in: scripts/ai/killPCs.eclze() doesn't take parameters.: bad allocation
 PC < nLines: (ndex) requires a parameter.unsigned1399:  1399: get member id 'z' (2)
1400:  1400: global #3
1401:  1401: get member id 'realm' (113)
1402:  1402: 1L
1403:  1403: 7L
1404:  1404: Func(4,28): FindPath
1405: >1405: local2 := 
1406:  1406: local #2
1407:  1407: 1L
1408:  1408: [] 1
1409:  1409: get member id 'x' (0)
Exception in: scripts/ai/spellKillPCs.ecle: bad allocation
 PC < nLines: (orted2061:  2061: get member id 'z' (2)
2062:  2062: global #7
2063:  2063: get member id 'realm' (113)
2064:  2064: 1L
2065:  2065: 7L
2066:  2066: Func(4,40): FindPath
2067: >2067: local2 := 
2068:  2068: local #2
2069:  2069: 1L
2070:  2070: [] 1
2071:  2071: get member id 'x' (0)
Exception in: scripts/ai/spellKillPCs.ecle: bad allocation
 PC < nLines: (orted2061:  2061: get member id 'z' (2)
2062:  2062: global #7
2063:  2063: get member id 'realm' (113)
2064:  2064: 1L
2065:  2065: 7L
2066:  2066: Func(4,40): FindPath
2067: >2067: local2 := 
2068:  2068: local #2
2069:  2069: 1L
2070:  2070: [] 1
2071:  2071: get member id 'x' (0)

Seems 097 gives more information. I hope this can help to locate error and to fix it. If neede more info or something - tell me.

[CWO]

Seems something with getting the Z parameter on or before FindPath?

[taxman]

It is symptom, not reason.

Now, after ~55 hours i see the same: 'bad allocation' errors and 'zombie' connections.

Code: Select all

Waiting for 161 child threads to exit
Waiting for 161 child threads to exit
Waiting for 161 child threads to exit
Waiting for 161 child threads to exit
Waiting for 161 child threads to exit
Waiting for 161 child threads to exit
No clock movement in 30 seconds.  Dumping thread status.
*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 53
Last Script: scripts/misc/logoff.ecl PC: 207
Escript Instruction Cycles: 48606999212
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
660 - Main
1104 - ThreadStatus
Waiting for 161 child threads to exit

POL waiting for 161 threads that not exists.

[coltain]

any progress??

[taxman]

POL097.1-2008-12-08 Coregina (VS.NET 2003)
compiled on Dec 8 2008 21:50:11

uo::SendPacket() removed from scripts, replaced by .sendpacket method.
Uptime now 88 hours, server still alive.

[taxman]

for 118 hours of uptime manual restart executed.
no 'bad allocation' errors but there are 33 zombie connections already so core cannot finish normal way and make the save:

Code: Select all

Current Threads:
288 - ThreadStatus
760 - Main
Waiting for 33 child threads to exit

[coltain]

same error:

pol.log

Code: Select all

Client#819: Disconnecting client due to send() error (1): 10038
Client#501 (xx.xxx.xx.xx): disconnected (account xxxxxxxxxxx)

*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 52
Last Script: pkg/pakiety/GumpSerwer/logoff.ecl PC: 35
Escript Instruction Cycles: 16496513622
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
2884 - ThreadStatus
2964 - Main
*Thread Info*

no other disturbing things in other logs, no dmp was ganerated since POL didn`t crushed

(I only try to use .sendpacket(mobile))

[coltain]

after a couple minutes ater restarting I get to messages:

Socket::send_nowait() error:10053
Socket::send_nowait() error:10038
Socket::send_nowait() error:10053
Socket::send_nowait() error:10038

I searched google for a definition of this errors, but since I`m not a specialist in it... simply cant understand it

Can it be possible that this errors affect allocation??

Edit:
Taken from some forum:

Code: Select all

10038 is "Socket operation on non-socket" which usually means that data was being sent through a socket that was already closed.  It looks like there was still something in the send buffer when vortec disconnected, which generated the error, but then vortec connected on a second attempt.

[MuadDib]

Yeah, that error there is in relation to a ghosted connection in the core. Someone disconnected but the core had not caught up yet for some reason (high sysload, or it tried sending right as they disconnected).

Different error, but, might be related to an older Memory Leak, maybe. Once the new cores come out (097.2 and next 098) that massive memory leak should be fixed. (It involves .SendPacket() and Packet Objects not using SendPacket()).

[Nando]

Do you use the internal webserver? I suspect these zombie connections might be artifacts from the internal webserver...

Just fixed a bug where some threads wouldn't be properly reported in windows (but would still show in the "Waiting for xx threads..."). This doesn't fix the problem of threads stopping POL from exiting, but could give us information of where the problem is.

[taxman]

141 hours uptime
Houston, all looks great

[coltain]

3 days uptime and still...

[coltain]

pol.log

Code: Select all

Exception in: scripts/ai/city_unmerchant.eclnout: bad allocation
 PC < nLines: (edex1856:  1856: [] 1
1857:  1857: Func(0,0): CInt
1858:  1858: global #49
1859:  1859: <>
1860:  1860: "stock"
1861:  1861: Func(4,1): GetProperty
1862: >1862: !
1863:  1863: ||
1864:  1864: if false goto 1876
1865:  1865: "undata"
1866:  1866: Func(2,10): GetGlobalProperty
[01/23 13:35:57] Unhandled Exception! Writing Minidump file. 
This file with explanation and last lines from log files should be given to the development team.
Saved dump file to 'POL097.2-2009-01-13 Coregina-20090123033201-0.dmp'
Last Script: pkg/systems/combat/main/mainAttackTimer.ecl PC: 873

added 3 .dmp file

[Nando]

Coltain, how was memory use when this happened? How much memory do you have? Were the 3 dumps the same error?

[coltain]

I don`t know how much memory usage was when it happend. A computer has 3GB ram.

The dmp file from 23.01.2009 is made when this error in pol.log occured. I don`t know about other 2, but I can search pol.log for them (thay were made eariler).

[Nando]

Hm... would you please search the logs for the 21 and 22 dumps?

They all seem to be related to running out of memory... the one from 21 was an exception in the CRT (when loading, I think), 22 and 23 are from the STLport lib that POL uses, in the memory allocation function.

[coltain]

log from 22.01.2009 4:05

Code: Select all

Client#663: Disconnecting client due to send() error (1): 10038
Client#839 (83.27.131.164): disconnected (account wol44473)
Client#839: Disconnecting client due to send() error (1): 10038
[01/22 04:04:39] Client#854 (86.12.246.26): disconnected (account xxxxxxxx)
Client#854: Disconnecting client due to send() error (1): 10038
[01/22 04:04:42] Shutting down...
[01/22 04:04:56] Scheduler passes: 2949475
Script passes:    3088957788
*****Scripts*****
[01/22 04:05:15] Assertion Failed: contents_.empty(), .\containr.cpp, line 42
[01/22 04:05:16] Unhandled Exception! Writing Minidump file. 
This file with explanation and last lines from log files should be given to the development team.
Saved dump file to 'POL097.2-2009-01-13 Coregina-20090121074427-0.dmp'
Last Script:  	2'pakiety/GumpSerwer/logoff.ecl PC: 35

log form 22.01.2009 20:!3

Code: Select all

Client#593: Leaving queued mode (67 bytes xmitted)
[01/22 20:13:47] Exception in: scripts/ai/animal.eclspecified): bad allocation
 PC < nLines: ( '('2549:  2549: 0L
2550:  2550: return
2551:  2551: local #1
2552:  2552: casejmp
2553:  2553: "#events_0"
2554:  2554: Func(3,1): GetProperty
2555: >2555: "#events_0"
2556:  2556: Func(3,1): GetProperty
2557:  2557: 3L
2558:  2558: [] 1
2559:  2559: Func(0,2): CInt
Exception in: pkg/npcs/combat_ai/ai_default_mag.eclśR: bad allocation
 PC < nLines: (Qs@3972:  3972: return
3973:  3973: pop param 'kto'
3974:  3974: 2L
3975:  3975: local #0
3976:  3976: "spells_config"
Unhandled Exception! Writing Minidump file. 
This file with explanation and last lines from log files should be given to the development team.
Saved dump file to 'POL097.2-2009-01-13 Coregina-20090122040516-0.dmp'
Last Script: pkg/systems/combat/CLCombat.ecl PC: 92

the last one dmp file time was: 23.01.2009 13:35

[coltain]

memory usage:
just after restart 1 736 MB
20 minutes after restart 1 032 MB
10h uptime 1 347 MB and gaining

online about 30 players

[coltain]

ram usage about 1300MB and standing still but after 34h online this error is shown

when I forced POL to shutt down the files has been saved but 64 child threads were shown, and some error about my "gumpserver" - packet that controls gumps (virtuses etc.). I disabled this package and restarted server.

I will post some results if it will be shown.

[MuadDib]

coltain, do you have the internal webserver enabled, and is it used much if it is?

[coltain]

Yes I have.
SQL server and ftp,www server by IIS.

I don`t use them so much. (but maybe my web admin does, I`ll ask)

[taxman]

141 hours uptime
Houston, all looks great

Manual restart after 285 hours. No errors, no zombies.

[coltain]

after about 5h online

Code: Select all

[01/27 22:17:59] Exception in: pkg/items/magic_items/zaklecia/tornado.eclash: bad allocation
 PC < nLines: (equires a parameter.1744:  1744: local #0
1745:  1745: get member id 'z' (2)
1746:  1746: 0L
1747:  1747: local #0
1748:  1748: get member id 'realm' (113)
1749:  1749: Func(2,25): CreateNpcFromTemplate
1750: >1750: local10 := 
1751:  1751: local #10
1752:  1752: "Tornado"
1753:  1753: Func(2,30): SetName
1754:  1754: #
[01/27 22:18:03] Exception in: pkg/systems/evspawner/main/spawnuj.eclreach: bad allocation
 PC < nLines: (gal here1977:  1977: local #4
1978:  1978: local #3
1979:  1979: local #2
1980:  1980: local #1
1981:  1981: local #0
1982:  1982: Func(2,23): CreateNpcFromTemplate
1983: >1983: :=
1984:  1984: #
1985:  1985: local #8
1986:  1986: if false goto 1990
1987:  1987: local #8
Exception in: pkg/systems/evspawner/main/spawnuj.eclreach: bad allocation
 PC < nLines: (gal here1977:  1977: local #4
1978:  1978: local #3
1979:  1979: local #2
1980:  1980: local #1
1981:  1981: local #0
1982:  1982: Func(2,23): CreateNpcFromTemplate
1983: >1983: :=
1984:  1984: #
1985:  1985: local #8
1986:  1986: if false goto 1990
1987:  1987: local #8
Unhandled Exception! Writing Minidump file. 
This file with explanation and last lines from log files should be given to the development team.
Saved dump file to 'POL097.2-2009-01-13 Coregina-20090127163438-0.dmp'
Last Script: pkg/npcs/combat_ai/ai_default_cmag.ecl PC: 3808

It`s annoying. Maybe changing to 098 will help?

[coltain]

disabled all www and ftp servers, waiting for a result

[MuadDib]

If you DO switch to 098, be SURE to read the core-changes. Especially when updating to new betas.

I say this, because now not only in 098 are we fixing a LOT of memory leaks (over 50 so far), new systems are being introduced that slightly alter existing systems. (Nothing released yet, but the next beta will show this).