Multithread save

[Terciob]

I saw inside the pol source that saves are handled in steps, is possible instead of this launch 4 threads and save separately on each: storage // npcs, pcs // itens // multis, parties, globals, guilds, etc?

Another random question, i just loaded my world into a 64bit core and memory usage jumps from 2.7gb (32bits usage) to 4.2gb, my question is what's the main difference between pol 32 and 64bits? This high memory usage can decrease the process speed?
Thanks.

[kevin]

This is a good idea, but I'm not sure about the logistics about world-save as I've never really worked on it. I'll investigate when I get some time.

Thanks for the idea.

[Turley]

Parallel saving wouldn't give you a mayor speedup since the bottleneck is your hard drive. It would be a speedup if the core would do some calculations during storage of the data, but it doesn't...there are only simple file write operations, it can even be a slowdown after the change since the hard drive now has to switch positions between several file positions.
We could speedup the storage if we drop the human readable format and store it eg inside a binary format. But there the trick is only to reduce the resulting filesize.

Since I already included openmp support you can easily try it with something like that, inside of the existing storage function:
#pragma omp parallel sections
{
#pragma omp section
{
//store NPC
}
#pragma omp section
{
//store items
}
...
}

[Terciob]

Kevin, Turley, thanks for reply.

About HD speed, i'm using a SSD with a up to 250mb/s write speed, our world size are 904mb and world save takes about 22 seconds to complete.

Guys, i think this is just a dream, but now with Sql support is possible pol save dynamically ("on the fly") to Sql when a character/item change positions or cprops? and never more use saveworldstate.

[Yukiko]

Parallel saving wouldn't give you a mayor speedup since the bottleneck is your hard drive. It would be a speedup if the core would do some calculations during storage of the data, but it doesn't...there are only simple file write operations, it can even be a slowdown after the change since the hard drive now has to switch positions between several file positions.
We could speedup the storage if we drop the human readable format and store it eg inside a binary format. But there the trick is only to reduce the resulting filesize.

Since I already included openmp support you can easily try it with something like that, inside of the existing storage function:
#pragma omp parallel sections
{
#pragma omp section
{
//store NPC
}
#pragma omp section
{
//store items
}
...
}

I would vote against binary blob files for world data. There have been many times when I've had to edit the actual data files for one reason or another. I don't find the file size of the world data all that prohibitive. The only way I would be happy with switching to a binary file format is if there were a way to either convert between binary and human readable formats or an editor that would allow editing the files in a human readable form. I would include the data format as one of the elegant aspects of POL. That may sound strange but it does make fixing problems easier when somehow the data gets corrupted or you have to remove a wayward NPC or item from the world.

About HD speed, i'm using a SSD with a up to 250mb/s write speed, our world size are 904mb and world save takes about 22 seconds to complete.

Guys, i think this is just a dream, but now with Sql support is possible pol save dynamically ("on the fly") to Sql when a character/item change positions or cprops? and never more use saveworldstate.

I wonder if it would be possible to perform the world save of all the files in a separate thread, not splitting each type of world data, ie. NPCs, Storage, items etc., into separate threads but just have the whole world save run in its own thread while the rest of the POL VM runs happily along as usual?

I was initially thinking about a staggered save where POL writes one file, say pcs.txt, then after 5 seconds or so writes another data file and continues until all data is saved but that would or could create a terrible mess of discontinuity if the server crashed or the system went down with only half the data files saved. Everything would be out of sync. Maybe the POL gurus could see a way of making that a fail safe operation but I can't see how.

Anyway, those are my two pence worth.

Thanks to the devs for your hard work.

[Turley]

For any choose format of cause a converter/viewer has to be made. That was always the fixed point in discussions between Nando and me, we had lots of talks about our storage format

An independent thread for storage is no sollution , cause you need a clean fixed state during saving. What happens if someone drops an item during saving... The saves needs to be a fixed snapshot of the world, thus it has to be a blocking operation.

SQL seems to be at the first look the solution cause then you could directly store any changes. But then you have to deal with database fragmentation etc.. So also no easy drop in change.

[AsYlum]

I was thinking about json/ubjson as alternative formats for POL.

Plain JSON is just formatted text file. Dunno how much bigger data files would be. From what i know JSON has a pretty low footprint. What do you think?

http://json.org/
http://ubjson.org/
http://www.w3schools.com/json/json_syntax.asp

[Terciob]

I think exist ways to defrag Sql and server admins could make weekly maintenance for this.
Is too hard, too much work, make sql save? because it's so amazing, mmorpg servers like world of warcraft uses this method.

[kevin]

i don't know about instantaneous/on-the-fly saving... just a guess (because i have no great knowledge on sql performance) but there would be tons of calls on large servers. sql call for each char movement... or imagine a PC/NPC death => create corpse item, corpse item properties, all items within the corpse that were newly created (for NPCs) would be created, setting of those properties, etc etc.

maybe not a real way to find out unless it's just coded... ;p