PenUltima Online Forum Index Official Core: 096.7
Official Core: 097 2008-02-26
Donate towards the POL web hosting bill!
 POL Home   FAQ   Search    Memberlist   Usergroups    Register    Profile   Log in to check your private messages   Log in
(POL096-2006-01-18 (gentoo)) Some stress tests
Goto page 1, 2  Next
 
Post new topic   Reply to topic    PenUltima Online Forum Index -> Bug Reports 096
 Display posts from previous 
Display posts from previous:   

 (POL096-2006-01-18 (gentoo)) Some stress tests 
Author Message
AsYlum



Joined: 05 Feb 2006
Posts: 30
PostPosted: Mon Apr 17, 2006 4:22 pm    Post subject: (POL096-2006-01-18 (gentoo)) Some stress tests Reply with quote
Hello, we've organized mass quest for our players, the online count around 170 players avarage. After some time server became very unstable. Here are some crash logs:

Code:
[04/17 16:35:21] Client#1089 connected from 212.162.20.85 (112 connections) on interface 217.8.180.198
Failed to create worker thread for Client#1089
[04/17 16:35:22] Client#1084 (83.8.196.108): disconnected (account sss)
*Thread Info*
Semaphore PID: 19261
  ("kill -SIGUSR2 19261" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: scripts/ai/spellkillpcs.ecl PC: 1549
Escript Instruction Cycles: 1333928309
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:
19226 - Main
19261 - Listen
19262 - Tasks


Code:
[scripts/misc/logofftest.ecl]: xxx/xxx sie. SL: 30 C [4646,1484,26]
[04/17 18:29:01] sysload=40 (158) cputime=0
[04/17 18:29:26] Client#778 connected from 80.55.177.74 (115 connections) on interface 217.8.180.198
Failed to create worker thread for Client#778
*Thread Info*
Semaphore PID: 21368
  ("kill -SIGUSR2 21368" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: scripts/ai/killpcs.ecl PC: 1327
Escript Instruction Cycles: 1096622220
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:


Code:
[04/17 19:57:06] yyy/yyy used out-of-sequence target cursor
[04/17 19:57:16] Client#840 connected from 83.20.202.198 (149 connections) on interface 217.8.180.198
Failed to create worker thread for Client#840
*Thread Info*
Semaphore PID: 22587
  ("kill -SIGUSR2 22587" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: pkg/hooks/attack.ecl PC: 217
Escript Instruction Cycles: 907428731
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:


Code:
[04/17 20:48:12] Account ccc logged in from 83.17.125.162
[04/17 20:48:15] Client#690 (83.17.125.162): disconnected (account ccc)
Client#691 connected from 83.17.125.162 (194 connections) on interface 217.8.180.198
[04/17 20:48:16] Account ccc logged in from 83.17.125.162
[04/17 20:48:19] Client#415: Switching to queued data mode (2)
[04/17 20:48:20] Account ccc selecting character zzz
[04/17 20:48:21] Client#415: Leaving queued mode (264 bytes xmitted)
[04/17 20:48:24] Client#692 connected from 83.10.27.201 (195 connections) on interface 217.8.180.198
Failed to create worker thread for Client#692
[04/17 20:48:55] Client#531 (83.26.19.77): disconnected (account xxx)
*Thread Info*
Semaphore PID: 23789
  ("kill -SIGUSR2 23789" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: pkg/packethooks/SendAscii/sendAscii.ecl PC: 8
Escript Instruction Cycles: 430243405
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116


Code:
Client#106: Switching to queued data mode (2)
[04/17 21:14:47] Client#577: Switching to queued data mode (2)
[04/17 21:14:48] Client#581 connected from 195.177.196.176 (197 connections) on interface 217.8.180.198
Failed to create worker thread for Client#581
[04/17 21:15:17] Client#462 (83.8.32.187): disconnected (account zzz)
*Thread Info*
Semaphore PID: 24712
  ("kill -SIGUSR2 24712" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: pkg/packethooks/movepacket/acceptmove.ecl PC: 8
Escript Instruction Cycles: 214398003
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:


Code:
Client#412: Switching to queued data mode (2)
Client#412: Leaving queued mode (1778 bytes xmitted)
[04/17 22:13:45] Client#592 connected from 213.155.169.110 (173 connections) on interface 217.8.180.198
Failed to create worker thread for Client#592
[04/17 22:14:06] Client#105 (81.129.25.221): disconnected (account xxx)
*Thread Info*
Semaphore PID: 26178
  ("kill -SIGUSR2 26178" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: pkg/hooks/attributeCore.ecl PC: 246
Escript Instruction Cycles: 271055160
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:
26152 - Main
26178 - Listen
26179 - Tasks
26180 - Scripts
26181 - Decay_britannia


Any idea what could possibly cause those errors? Sorry for such big amount of crashes in one post, but i think they all are generated by the same bug/problem.

  
Author Message
qrak



Joined: 05 Feb 2006
Posts: 160
Location: Poland
PostPosted: Mon Apr 17, 2006 8:57 pm    Post subject: Reply with quote
Semaphore PID: 26178
("kill -SIGUSR2 26178" to output backtrace)


You forgot about that. Do kill -sigusr before posting

  
Author Message
Lad



Joined: 05 Feb 2006
Posts: 23
PostPosted: Tue Apr 18, 2006 5:45 pm    Post subject: Reply with quote
Code:

No clock movement in 30 seconds.  Dumping thread status.
*Thread Info*
Semaphore PID: 7365
  ("kill -SIGUSR2 7365" to output backtrace)
[0x81132d4]
[0x8255b59]
[0x8255cfd]
[0x8262334]
[0x82d3ada]
Scripts Thread Checkpoint: 0
Last Script: pkg/skills/armslore/armsLore.ecl PC: 449
Escript Instruction Cycles: 1698081767
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:
7320 - Main
7365 - Listen
7366 - Tasks
7367 - Scripts
7368 - Decay_britannia
7369 - Decay_malas
7370 - Decay_ilshenar
7371 - Reap
7373 - ThreadStatus
7374 - Console
7404 - Client#5
7423 - Client#21
7436 - Client#34
7439 - Client#37
7443 - Client#41
7445 - Client#43
7448 - Client#46
7453 - Client#51
7462 - Client#60
7468 - Client#66
7491 - Client#89
(....)

and after kill pid ...
Code:

Caught SIGUSR2 (On-demand backtrace).  Please mail the following with explanation and last lines from log files to pol-core-test:
=== CUT ===
Build: POL096-2006-01-18 (gentoo)
Last Script: pkg/skills/armslore/armsLore.ecl PC: 449
Stack Backtrace:
[0x8251dcb]
[0x826603b]
[0x82ae828]
[0x8264833]
[0x8264698]
[0x8265ce7]
[0x8263225]
[0x812571f]
[0x8112a28]
[0x8255b59]
[0x8255cfd]
[0x8262334]
[0x82d3ada]
=== CUT ===

  
Author Message
Shinigami
POL Core Developer


Joined: 30 Jan 2006
Posts: 292
Location: Germany, Bavaria
PostPosted: Wed Apr 19, 2006 2:40 am    Post subject: Reply with quote
Lad... do u use AUX-interfaces or the Web-Interface? is it possible that u've sometimes some connection problems using one of em?

Shinigami

  
Author Message
Shinigami
POL Core Developer


Joined: 30 Jan 2006
Posts: 292
Location: Germany, Bavaria
PostPosted: Wed Apr 19, 2006 2:46 am    Post subject: Reply with quote
AsYlum & Lad & all other Linux user please call your pol binary via:

Code:
./pol 2> polerr.log &

or something like that. this will help a lot because u don't need to call "kill pid" to get a backtrace... it will be printed to stderr in each case (polerr.log).

Shinigami

  
Author Message
AsYlum



Joined: 05 Feb 2006
Posts: 30
PostPosted: Wed Apr 19, 2006 6:33 am    Post subject: Reply with quote
OK. Thanks for advice. Maybe it's time to add those hints to sticked topic on this forum : rules or something?

  
Author Message
AsYlum



Joined: 05 Feb 2006
Posts: 30
PostPosted: Wed Apr 19, 2006 5:24 pm    Post subject: Reply with quote
Again... hmm nothing unusual in console, client connected - disconnected, connected... etc.

Code:
Caught SIGUSR2 (On-demand backtrace).  Please mail the following with explanation and last lines from log files to pol-core-test:
=== CUT ===
Build: POL096-2006-01-18 (gentoo)
Last Script: pkg/hooks/attributeCore.ecl PC: 1130
Stack Backtrace:
[0x8251dcb]
[0x826603b]
[0x82ae828]
[0x8264833]
[0x8264698]
[0x8265ce7]
[0x8263225]
[0x812571f]
[0x8112a28]
[0x8255b59]
[0x8255cfd]
[0x8262334]
[0x82d3ada]
=== CUT ===


Any idea or hint what to search for? Maybe our custom attack-hook could cause those problems?

And in reply to previous posts, we don't use any aux pkgs, or pol-www server.

  
Author Message
Shinigami
POL Core Developer


Joined: 30 Jan 2006
Posts: 292
Location: Germany, Bavaria
PostPosted: Thu Apr 20, 2006 2:30 am    Post subject: Reply with quote
AsYlum wrote:
Maybe our custom attack-hook could cause those problems?

that's possible...

would be nice to see which lines of code are

AsYlum wrote:
pkg/hooks/attributeCore.ecl PC: 1130
pkg/skills/armslore/armsLore.ecl PC: 449

(just ecompile with -l, open the *.lst files and take a look around lines)

Shinigami

  
Author Message
Lad



Joined: 05 Feb 2006
Posts: 23
PostPosted: Thu Apr 20, 2006 11:17 am    Post subject: Reply with quote
Code:
Account Chester logged in from 83.xx.xx.xx
Abnormal end after checkpoint: File pol/charactr.cpp, line 1706
Exception in pkg/systems/combat/mainhitscript.ecl, PC=3957: St9bad_alloc
3952:  3952: makelocal
3953:  3953: jmp userfunc @5573
3954:  3954: #
3955:  3955: local #3
3956:  3956: local #2
3957:  3957: Func(3,15):
3958: >3958: #
3959:  3959: 0L
3960:  3960: return
3961: đĚJ1: pop param '
3962: x(92: pop param '
Client#836: Exception in message handler 0x07: St9bad_alloc
0000: 07 55 09 e3 ce 00 01                               .U...... ........
Client#836: Exception in i/o thread: St9bad_alloc! (checkpoint=4)
Client#836 (62.xx.xx.xx): disconnected (account Jaca)
Exception in scripts/misc/logofftest.ecl, PC=389: St9bad_alloc
384:  384: progend
385:  385: leave block(7)
Caught SIGSEGV (Segfault).  Please mail the following to pol-core-test:
=== CUT ===
Build: POL096-2006-01-18 (gentoo)
Last Script: scripts/misc/logofftest.ecl PC: 389
Stack Backtrace:
[0x8251dcb]
[0x826603b]
[0x82ae828]
[0x8056b10]
[0x8227558]
[0x8227644]
[0x82274bb]
[0x814c990]
[0x814cc4b]
[0x8111a88]
[0x8255bb2]
[0x8112330]
[0x8262334]
[0x82d3ada]
=== CUT ===

  
Author Message
Lad



Joined: 05 Feb 2006
Posts: 23
PostPosted: Thu Apr 20, 2006 7:02 pm    Post subject: Reply with quote
look at this topic :
http://forums.polserver.com/viewtopic.php?t=29

there are the same "Stack Backtraces" we recompile all scripts, maybe you have any suggestions what we can do with this. With 100-150 users online server crashe every 2h, it's imposible to play on it

  
Author Message
tekproxy
Distro Developer


Joined: 06 Apr 2006
Posts: 350
Location: Nederland, Texas
PostPosted: Thu Apr 20, 2006 8:46 pm    Post subject: Reply with quote
You know I can crash my server just by using a last target maco if the target doesn't exist anymore. Ex: while making arrows with last object (shafts) and last target (feathers) you run out of feathers and press the macro key anyway and then crash.

Does this happen on your server? If so, you might somehow be able to hook that packet and handle the logic yourself until a dev makes a fix.

  
Author Message
AsYlum



Joined: 05 Feb 2006
Posts: 30
PostPosted: Fri Apr 21, 2006 3:21 am    Post subject: Reply with quote
tekproxy wrote:
You know I can crash my server just by using a last target maco if the target doesn't exist anymore. Ex: while making arrows with last object (shafts) and last target (feathers) you run out of feathers and press the macro key anyway and then crash.

Does this happen on your server? If so, you might somehow be able to hook that packet and handle the logic yourself until a dev makes a fix.


Hmm... with 1-18 or 3-17 core? I'll try to reproduce it on our server but i think this crash is only in 3-17.

  
Author Message
qrak



Joined: 05 Feb 2006
Posts: 160
Location: Poland
PostPosted: Fri Apr 21, 2006 6:14 am    Post subject: Reply with quote
yes, is only in 3-17

  
Author Message
AsYlum



Joined: 05 Feb 2006
Posts: 30
PostPosted: Sun Apr 23, 2006 10:53 am    Post subject: Reply with quote
Now we've set polcfg: ListenPort=0 and uoclient.cfg Listener {Port 5001}

and this came out:

Code:
[pkg/commands/textcmd/admin/sd.ecl]: CmdSd: ShutDown started
Shutting down...
[04/23 02:33:00] Client#2: Disconnecting client due to send() error (1): 9
Caught SIGSEGV (Segfault).  Please mail the following to pol-core-test:
=== CUT ===
Build: POL096-2006-01-18 (gentoo)
Last Script: scripts/ai/townguard.ecl PC: 1408
Stack Backtrace:
[04/23 02:33:01] [0x8251dcb]
[0x826603b]
[0x82ae828]
[0x8170528]
[0x81701b1]
[0x81751bf]
[0x8170202]
[0x808441c]
[0x80840e4]
[0x817527a]
[0x81cc84e]
[0x81153f1]
[0x811586b]
[0x8259372]
[0x82a79e8]
[0x8048131]
=== CUT ===


Code:
Assertion Failed: pthread_create( &thread, &create_detached_attr, thread_stub2, td ) == 0, clib/threadhelp.cpp, line 251
Aborting due to assertion failure.


Code:
Account xxx logged in from 62.121.83.127
[04/23 16:36:05] Account xxx selecting character yyy
[04/23 16:36:13] [scripts/misc/logoff.ecl]: zzz/zzz wylogowal sie. x: 2467 y: 516 z: 15
[04/23 16:36:27] Client#732 (217.172.242.163): disconnected (account ccc)
[scripts/misc/logofftest.ecl]: ccc/ccc wylogowal sie. SL: 0 C [2871,697,0]
[04/23 16:36:30] sysload=45 (268) cputime=0
[04/23 16:36:43] Assertion Failed: pthread_create( &thread, &create_detached_attr, thread_stub2, td ) == 0, clib/threadhelp.cpp, line 251
Aborting due to assertion failure.
[04/23 16:36:45] Client#46 (62.233.180.141): disconnected (account sss)


Maybe libc problem?

and this (afer executing server shutdown via in-game command):

Code:
Client#1904: Disconnecting client due to send() error (1): 9
Client#1350: Disconnecting client due to send() error (1): 9
Client#1839: Disconnecting client due to send() error (1): 9
Client#1787: Disconnecting client due to send() error (1): 9
Client#1613: Disconnecting client due to send() error (1): 9
Client#1755: Disconnecting client due to send() error (1): 9
Client#1690: Disconnecting client due to send() error (1): 9
Client#1177: Disconnecting client due to send() error (1): 9
Client#1586: Disconnecting client due to send() error (1): 9
Client#1705: Disconnecting client due to send() error (1): 9
Client#1751: Disconnecting client due to send() error (1): 9
Client#1873: Disconnecting client due to send() error (1): 9
Client#857: Disconnecting client due to send() error (1): 9
Client#1510: Disconnecting client due to send() error (1): 9
Client#1749: Disconnecting client due to send() error (1): 9
Client#662: Disconnecting client due to send() error (1): 9
Client#1846: Disconnecting client due to send() error (1): 9
Client#1897: Disconnecting client due to send() error (1): 9
Client#1687: Disconnecting client due to send() error (1): 9
Client#1359: Disconnecting client due to send() error (1): 9
Client#1828: Disconnecting client due to send() error (1): 9
Client#1453: Disconnecting client due to send() error (1): 9
Client#1886: Disconnecting client due to send() error (1): 9
Client#1563: Disconnecting client due to send() error (1): 9
Client#1570: Disconnecting client due to send() error (1): 9
Client#1877: Disconnecting client due to send() error (1): 9
Client#1767: Disconnecting client due to send() error (1): 9
*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 55
Last Script: scripts/misc/logoff.ecl PC: 230
Escript Instruction Cycles: -1007044966
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
9803 - Main
9870 - UO Client Listener Port 5001
9878 - ThreadStatus
*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 55
Last Script: scripts/misc/logoff.ecl PC: 230
Escript Instruction Cycles: -1007044966
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
9803 - Main
9870 - UO Client Listener Port 5001
9878 - ThreadStatus
*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 55
Last Script: scripts/misc/logoff.ecl PC: 230
Escript Instruction Cycles: -1007044966
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
9803 - Main
9870 - UO Client Listener Port 5001
9878 - ThreadStatus
*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 55
Last Script: scripts/misc/logoff.ecl PC: 230
Escript Instruction Cycles: -1007044966
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
9803 - Main
9870 - UO Client Listener Port 5001
9878 - ThreadStatus
*Thread Info*
Semaphore PID: 0
Scripts Thread Checkpoint: 55
Last Script: scripts/misc/logoff.ecl PC: 230
Escript Instruction Cycles: -1007044966
Tasks Thread Checkpoint: 9
Active Client Thread Checkpoint: 116
Current Threads:
9803 - Main
9870 - UO Client Listener Port 5001
9878 - ThreadStatus

  
Author Message
Lad



Joined: 05 Feb 2006
Posts: 23
PostPosted: Mon May 22, 2006 6:10 pm    Post subject: Reply with quote
We still have this problem with create new thread any suggestion what we can do with this ?

and the last one:

Code:

[05/23 00:00:16] Client#2905 connected from 85.13.251.42 (82 connections) on interface 217.8.180.198
Failed to create worker thread for Client#2905
*Thread Info*
Semaphore PID: 15467
  ("kill -SIGUSR2 15467" to output backtrace)
Scripts Thread Checkpoint: 0
Last Script: scripts/ai/killpcs.ecl PC: 430
Escript Instruction Cycles: -1260961961
Tasks Thread Checkpoint: 1
Active Client Thread Checkpoint: 116
Current Threads:
15444 - Main
15467 - Listen
15468 - Tasks
15469 - Scripts
15470 - Decay_britannia
15471 - Decay_ilshenar
15472 - Decay_malas
15473 - Reap
15474 - DbgListn
15475 - ThreadStatus
15476 - Console
16087 - Client#473

(...)


[05/23 00:02:24] Caught SIGUSR2 (On-demand backtrace).  Please mail the following with explanation and last lines from log files to pol-core-test:
=== CUT ===
Build: POL096-2006-01-18 (gentoo)
Last Script: scripts/ai/killpcs.ecl PC: 430
Stack Backtrace:
[0x8251dcb]
[0x826603b]
[0xffffe420]
[0x8264833]
[0x8264698]
[0x8265ce7]
[0x8263225]
[0x812571f]
[0x8112a28]
[0x8255b59]
[0x8255cfd]
[0x8262334]
[0x82d3ada]
=== CUT ===

Post new topic   Reply to topic    PenUltima Online Forum Index -> Bug Reports 096 All times are GMT - 4 Hours
Goto page 1, 2  Next
Page 1 of 2

 




Powered by phpBB © 2001, 2005 phpBB Group :: Theme & Graphics by GHS & Scott E. Royalty