Pol crashing for "Unexpected messages" flood

[Terciob]

Hi folks, we got a problem here were pol are receiving a lot of invalid packets (isn't a ddos attack) and this are creating a way to crash.
I'm using an old version (commit 334).
Normally we got 3 crash per month, today got 4 in 5 hours.
pol.log recorded thousands of lines like this (a zip with full pol.log report is attached).

Client#7275 connected from 187.22.67.82 (1091 connections) on interface 177.70.7.26
Client#7276 connected from 187.22.67.82 (1092 connections) on interface 177.70.7.26
Client#7277 connected from 187.22.67.82 (1093 connections) on interface 177.70.7.26
Client#7278 connected from 187.22.67.82 (1094 connections) on interface 177.70.7.26
Client#7279 connected from 187.22.67.82 (1095 connections) on interface 177.70.7.26
Client#7280 connected from 187.22.67.82 (1096 connections) on interface 177.70.7.26
Client#7281 connected from 187.22.67.82 (1097 connections) on interface 177.70.7.26
Client#7282 connected from 187.22.67.82 (1098 connections) on interface 177.70.7.26
Client#7283 connected from 187.22.67.82 (1099 connections) on interface 177.70.7.26
Client#7284 connected from 187.22.67.82 (1100 connections) on interface 177.70.7.26
Client#7285 connected from 187.22.67.82 (1101 connections) on interface 177.70.7.26
Client#7286 connected from 187.22.67.82 (1102 connections) on interface 177.70.7.26
Client#7287 connected from 187.22.67.82 (1103 connections) on interface 177.70.7.26
Client#7288 connected from 187.22.67.82 (1104 connections) on interface 177.70.7.26
Client#7289 connected from 187.22.67.82 (1105 connections) on interface 177.70.7.26
Client#7290 connected from 187.22.67.82 (1106 connections) on interface 177.70.7.26
Client#7291 connected from 187.22.67.82 (1107 connections) on interface 177.70.7.26
Client#7292 connected from 187.22.67.82 (1108 connections) on interface 177.70.7.26
Client#7293 connected from 187.22.67.82 (1109 connections) on interface 177.70.7.26
Client#7294 connected from 187.22.67.82 (1110 connections) on interface 177.70.7.26
Client#7295 connected from 187.22.67.82 (1111 connections) on interface 177.70.7.26
Client#7296 connected from 187.22.67.82 (1112 connections) on interface 177.70.7.26
Client#7297 connected from 187.22.67.82 (1113 connections) on interface 177.70.7.26
Client#7298 connected from 187.22.67.82 (1114 connections) on interface 177.70.7.26
Client#7299 connected from 187.22.67.82 (1115 connections) on interface 177.70.7.26
Client#7300 connected from 187.22.67.82 (1116 connections) on interface 177.70.7.26
Client#7301 connected from 187.22.67.82 (1117 connections) on interface 177.70.7.26
Client#7302 connected from 187.22.67.82 (1118 connections) on interface 177.70.7.26
Client#7303 connected from 187.22.67.82 (1119 connections) on interface 177.70.7.26
Client#7304 connected from 187.22.67.82 (1120 connections) on interface 177.70.7.26
Client#7305 connected from 187.22.67.82 (1121 connections) on interface 177.70.7.26
Client#7306 connected from 187.22.67.82 (1122 connections) on interface 177.70.7.26
Client#7307 connected from 187.22.67.82 (1123 connections) on interface 177.70.7.26
Client#7308 connected from 187.22.67.82 (1124 connections) on interface 177.70.7.26
Client#7309 connected from 187.22.67.82 (1125 connections) on interface 177.70.7.26
Client#7310 connected from 187.22.67.82 (1126 connections) on interface 177.70.7.26
[11/09 01:12:08] Client#7311 connected from 187.22.67.82 (1127 connections) on interface 177.70.7.26
Client#7312 connected from 187.22.67.82 (1128 connections) on interface 177.70.7.26
Client#7313 connected from 187.22.67.82 (1129 connections) on interface 177.70.7.26
Client#7314 connected from 187.22.67.82 (1130 connections) on interface 177.70.7.26
Client#7315 connected from 187.22.67.82 (1131 connections) on interface 177.70.7.26
error in create_thread: 12 8 "Not enough space" "Not enough space" 283 6302704 SocketClientThread 6316976 0 456712128
error in create_thread: 12 8 "Not enough space" "Not enough space" 284 6302704 SocketClientThread 6316976 0 456713344

Client#8070: Unexpected message type 77, 1280 bytes (IP:187.22.67.82, Account:None)
0000: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
0010: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
0020: 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 ogirao w inz!cogi
0030: 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f rao winz !cogirao
0040: 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 winz!co girao wi
0050: 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 nz!cogir ao winz!
0060: 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 cogirao winz!cog
0070: 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 irao win z!cogira
0080: 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 o winz!c ogirao w
0090: 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a inz!cogi rao winz
00a0: 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f !cogirao winz!co
00b0: 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 girao wi nz!cogir
00c0: 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 ao winz! cogirao
00d0: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
00e0: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
00f0: 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 ogirao w inz!cogi
0100: 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f rao winz !cogirao
0110: 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 winz!co girao wi
0120: 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 nz!cogir ao winz!
0130: 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 cogirao winz!cog
0140: 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 irao win z!cogira
0150: 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 o winz!c ogirao w
0160: 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a inz!cogi rao winz
0170: 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f !cogirao winz!co
0180: 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 girao wi nz!cogir
0190: 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 ao winz! cogirao
01a0: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
01b0: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
01c0: 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 ogirao w inz!cogi
01d0: 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f rao winz !cogirao
01e0: 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 winz!co girao wi
01f0: 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 nz!cogir ao winz!
0200: 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 cogirao winz!cog
0210: 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 irao win z!cogira
0220: 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 o winz!c ogirao w
0230: 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a inz!cogi rao winz
0240: 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f !cogirao winz!co
0250: 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 girao wi nz!cogir
0260: 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 ao winz! cogirao
0270: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
0280: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
0290: 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 ogirao w inz!cogi
02a0: 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f rao winz !cogirao
02b0: 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 winz!co girao wi
02c0: 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 nz!cogir ao winz!
02d0: 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 cogirao winz!cog
02e0: 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 irao win z!cogira
02f0: 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 o winz!c ogirao w
0300: 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a inz!cogi rao winz
0310: 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f !cogirao winz!co
0320: 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 girao wi nz!cogir
0330: 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 ao winz! cogirao
0340: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
0350: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
0360: 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 ogirao w inz!cogi
0370: 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f rao winz !cogirao
0380: 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 winz!co girao wi
0390: 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 nz!cogir ao winz!
03a0: 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 cogirao winz!cog
03b0: 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 irao win z!cogira
03c0: 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 o winz!c ogirao w
03d0: 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a inz!cogi rao winz
03e0: 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f !cogirao winz!co
03f0: 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 girao wi nz!cogir
0400: 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 ao winz! cogirao
0410: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
0420: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
0430: 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 ogirao w inz!cogi
0440: 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f rao winz !cogirao
0450: 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 winz!co girao wi
0460: 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 nz!cogir ao winz!
0470: 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 cogirao winz!cog
0480: 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 irao win z!cogira
0490: 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 o winz!c ogirao w
04a0: 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a inz!cogi rao winz
04b0: 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f !cogirao winz!co
04c0: 67 69 72 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 girao wi nz!cogir
04d0: 61 6f 20 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 ao winz! cogirao
04e0: 77 69 6e 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e winz!cog irao win
04f0: 7a 21 63 6f 67 69 72 61 6f 20 77 69 6e 7a 21 63 z!cogira o winz!c
Client#8070: Too-long message type 111 length 26473
Unhandled Exception! Writing Minidump file.
Post this file with explanation and last lines from log files on http://forums.polserver.com/tracker.php for the development team.
Saved dump file to 'POL099 334 speedhack 102-20121108163922-0.dmp'
Last Script: pkg/npcs/old_ai/barker.ecl PC: 177

So, have someway somehow block this issue? or perhaps this was fixed in a nearly version.
Thanks

[RusseL]

that is a ddos attack it could be easily done with one tool.
i will not upload link here. that's dangerous for pol project.

i have the same prob. it must be fixed in core, i think.

possible fix as i understand:
10-15 identical or wrong packets in a second - disconnect client (must be added in core)
and then you must control incoming connections with firewall. 5 connections per 15seconds for example.


Iptables rules to control incoming connections (4 per 60seconds):

Code: Select all

iptables -I INPUT -p tcp --dport 5003 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 5003 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

[Terciob]

Thanks Russel.
I agree with you, i think pol is a little bit fragile against invalid packets, the guys which are doing this are cooperating with us, they told that pol crash after 3 seconds of flood with one single pc.
This can be done in any language which can open sockets and the frustration part is that only need one computer (ddos normally have a horde of computers).

[RusseL]

ok, yes, it's not a ddos, it's just a flood, but it doesn't matter
it could be crashed in 1 sec, it depends on internet speed of attacker

[Turley]

This problem should be somehow fixed (a 100% fix for floods will i think never be possible) with the latest svn revision. I added the pol.cfg option to not create an extra thread for each login process, only fully logged in clients get an extra thread.

[Terciob]

Thanks Turley, i'm going upgrade my server from rev334 to last revision.
OP will deliver the results soon.

[RusseL]

Can not compile last revision

Linux version compiled after removing some *.o files.

Windows:

Code: Select all

2>------ Build started: Project: ecompile, Configuration: Release Win32 ------
2>LINK : fatal error LNK1104: cannot open file 'VCOMP.lib'

[Turley]

I guess you ar compiling with VC express. There is the open mp library missing. After some Google help here is a website to install open mp: its for VC express 2008 but should work for all versions if you use the specific redist pkg. After this you should be also able to compile 64bit.

http://kenny-tm.xanga.com/651048063/par ... 8-express/

[Terciob]

So, with UseSingleThreadLogin=1 pol seems to handle floods pretty well with no crashes, still have some issues with keeping a lot of invalid connections alive and cpu usage too high while dealing with this. We are still working on it to provide a better and clean feedback.

About our upgrade to last revision, we had some troubles with mounts where we had to insert mount.movable := 1 inside a ondie hook, otherwise player keep mounted after die and the mount item still is moved to corpse, so mount is duped.

We found a small bug with "+=" operator where "result += (one + weapon.dmg_mod);" doesn't work:

Code: Select all

use uo;
program test (chr)
	var weapon := target (chr); //select a weapon
	if (!weapon or !weapon.isA (POLCLASS_WEAPON))
		return;
	endif

	weapon.dmg_mod := 5;
	
//----------------------------
	var one := 1;
	var result := 0;	
	broadcast (weapon.dmg_mod); //print 5
	broadcast (one); //print 1

//----------------------------
	result += (one + weapon.dmg_mod);
	broadcast (result); //print 0 // should print 6

//----------------------------
	result += one;
	result += weapon.dmg_mod;
	broadcast (result); //print 6

//----------------------------
	result := 0;
	var dmg_mod := weapon.dmg_mod;
	result += (one + dmg_mod);
	broadcast (result); //print 6

endprogram 

But the major annoying problem still are "array operation MultiSubscript" where old behavior was used to get subindexes values of an array and the new return a splice of an array broking a lot of scripts where i need to convert [index, subindex] to [index][subindex].
edit: Anyone know a search program which makes logical search where a can find syntaxes like this "[x, x]" ?

[Turley]

As a quickfix set OptimizeObjectMembers to 0 in ecompile.cfg there is the problem.
Currently I dont have time to fix it, so give a moment.

[Harley]

that is a ddos attack it could be easily done with one tool.
i will not upload link here. that's dangerous for pol project.

i have the same prob. it must be fixed in core, i think.

possible fix as i understand:
10-15 identical or wrong packets in a second - disconnect client (must be added in core)
and then you must control incoming connections with firewall. 5 connections per 15seconds for example.


Iptables rules to control incoming connections (4 per 60seconds):

Code: Select all

iptables -I INPUT -p tcp --dport 5003 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 5003 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

RusseL, good day! Some days ago, my server have been attacked by DDoS.. I searched some progs, but didn't find..
Can u send me private tool, that u use? I'll be very grateful 4u!

[Harley]

Is it hard to tell?