Question:
if one has to regulary access a certain item-specific information of (mostly) unchanging extended properties, would it be more efficient (CPU-wise) to store this data straight in the itemdesc, rereading them as properties from the cfg-file on each ocation, or would it be better, to store these values in a seperate datafile and access them there?
I'd asume the datafile access to be faster, is that correct?
And as a second question, how does the same thing look memory wise? Are itemdesc entries more memory-saving than seperate datafiles? Or does it simply not matter, as the complete itemdesc is likely to be kept in memory anyway?
efficiency & performance: cfgfile vs. datafile
-
Firedancer
- Grandmaster Poster
- Posts: 104
- Joined: Fri Feb 03, 2006 6:32 am
-
Marilla
I think you have your assumption backwards, there...
For raw speed in reading unchanging data, config files are by far the best. That's why all the stuff is in config files to begin with. If data files were faster, the definitions of items and NPCs would be in datafiles, instead.
DataFiles, on the other hand, are best for data that you need to change with some regularity... well; anything that you need to change and can't only do it when the shard is down.
I had a thread I posted here a while back explaining the general theories behind why there is a difference, which centered on why editing/removing items is not allowed from information in config files while the shard is running... but I don't remember where it was, now..
For raw speed in reading unchanging data, config files are by far the best. That's why all the stuff is in config files to begin with. If data files were faster, the definitions of items and NPCs would be in datafiles, instead.
DataFiles, on the other hand, are best for data that you need to change with some regularity... well; anything that you need to change and can't only do it when the shard is down.
I had a thread I posted here a while back explaining the general theories behind why there is a difference, which centered on why editing/removing items is not allowed from information in config files while the shard is running... but I don't remember where it was, now..
-
Firedancer
- Grandmaster Poster
- Posts: 104
- Joined: Fri Feb 03, 2006 6:32 am
hmm....this is partly true, but I'm not sure on the performance point of view of yours....Marilla wrote:I think you have your assumption backwards, there...
For raw speed in reading unchanging data, config files are by far the best. That's why all the stuff is in config files to begin with. If data files were faster, the definitions of items and NPCs would be in datafiles, instead.
DataFiles, on the other hand, are best for data that you need to change with some regularity... well; anything that you need to change and can't only do it when the shard is down.
I had a thread I posted here a while back explaining the general theories behind why there is a difference, which centered on why editing/removing items is not allowed from information in config files while the shard is running... but I don't remember where it was, now..
Since datafiles are commonly kept in memory and aren't re-read from disk, I was asuming them to be faster but more memory consuming. Also they can easily store complex variables like e.g. structs... cfgfiles have no neat commands to access such, which further strengthened my theory, even though I'm not completely sure.
ps: true enough, options to change cfgfiles from script are very limited indeed...instead they are more easy to change offline, but for what I need it would do. I'm more worried on memory and cpu usuage though, as this question will influence most all my crafts.
The best way is the method that takes the least time of reading.
If you have a config file of 10,000 items (making each item 10 lines) is 100,000 lines to read of maximum size.
If you have say... a file for weapons that had 2000 items in it (10 lines each) you only have 20,000 lines for max read.
Depending on what you want to do, making it so the server has the least amount of reading is the best choice. If your looking to improve the item grabbing methods you can try making seperate files like weapons, world items, armor, misc, etc.. to reduce excess reading when not needed.
Im currently redoing the UOXC emulator and this is one thing ive come across. The data files had 8900 entries of items for weapons and armors and I coded them hardwise in a few variables so that I could completely eliminate 8900 items from being read everytime a item is added (give est. 10 lines, thats 89,000 lines being read I just took out).
Its definatly worth while to seperate into files like you mentioned but you may never see the increase in speed until the server has to preform those seeking enough to lag the computer =\.
If you have a config file of 10,000 items (making each item 10 lines) is 100,000 lines to read of maximum size.
If you have say... a file for weapons that had 2000 items in it (10 lines each) you only have 20,000 lines for max read.
Depending on what you want to do, making it so the server has the least amount of reading is the best choice. If your looking to improve the item grabbing methods you can try making seperate files like weapons, world items, armor, misc, etc.. to reduce excess reading when not needed.
Im currently redoing the UOXC emulator and this is one thing ive come across. The data files had 8900 entries of items for weapons and armors and I coded them hardwise in a few variables so that I could completely eliminate 8900 items from being read everytime a item is added (give est. 10 lines, thats 89,000 lines being read I just took out).
Its definatly worth while to seperate into files like you mentioned but you may never see the increase in speed until the server has to preform those seeking enough to lag the computer =\.
-
Marilla
Some more info; my source will remain unnamed for now 
Probably has something to do with no mail-order beer!
Essentially, there will be very little difference at all in the speed of accessing a single record in either system, as they both use the same type of memory structure for accessing the information. That is, as long as you are not modifying the information often.
It should also be pointed out that if you have 8900 records in a data file, POL does not have to read all of them just to find one record! That's what indexers are for. All this type of thing would likely be stored in fairly efficient and low-ish-level STL containers that are able to find particular objects fairly quickly by their 'keys' (UO objects, config entries and data file entries all have unique keys); again, to emphasize: In NO WAY does POL have to read any number of records beyond the one you are looking for; it just has to check the particular container's indexer.
The only difference here would then be if you actually do insert, delete or perhaps modify items. Depending on the exact container type being used to store the information in question, inserting or deleting records can cause a need for the container itself to internally have to re-index itself. For a small container, this is fast and easy - it's essentially 'instant'. For a larger container, it's still pretty fast, but depending on how much larger we're talking about, it might take a barely perceptible amount of time. I would expect this to be an issue likely only with entries in :*:itemdesc.cfg or :*:npcdesc.cfg... however, those cannot be modified while running, anyway; they can only have items appended to them.
It should also be noted that I'm referring to the 'files' still, but in reality, that's not at all what's important here. In ALL cases, the information is being loaded into memory the first time it's requested, and it's read from there... not from the file. If it was doing all reads from the file, this would all be too slow to make the game playable. However, using the 'file' is the interface we have, so that's why I'm referring to them as 'files'.
Basically, if the information in question is already being stored in an existing config, such as itemdesc or npcdesc, I wouldn't bother changing it, especially if it's a significant amount of information. Loading from itemdesc/npcdesc works plenty fast enough to access NPC and Weapon information during a combat hook, for example, while having that combat hook script take up no appreciable sysload, according to the profiles. I'd say loading info during a combat hook is likely one of the potentially most 'critical' operations that can be done (using the term differently from the meaning related to 'critical blocks' of code, thread-wise); if something works well there, then it's good.
But, if you have NEW information that must be modified now and again, the new information I've learned, suggesting that the data from both config files and from data files is stored in memory by the same means, then by all means, feel free to use Datafiles, as well. Just keep in mind that unlike config files, datafiles must be written to disk at every world save, regardless whether they have been changed... that's also likely a total non-issue.
Probably has something to do with no mail-order beer!Essentially, there will be very little difference at all in the speed of accessing a single record in either system, as they both use the same type of memory structure for accessing the information. That is, as long as you are not modifying the information often.
It should also be pointed out that if you have 8900 records in a data file, POL does not have to read all of them just to find one record! That's what indexers are for. All this type of thing would likely be stored in fairly efficient and low-ish-level STL containers that are able to find particular objects fairly quickly by their 'keys' (UO objects, config entries and data file entries all have unique keys); again, to emphasize: In NO WAY does POL have to read any number of records beyond the one you are looking for; it just has to check the particular container's indexer.
The only difference here would then be if you actually do insert, delete or perhaps modify items. Depending on the exact container type being used to store the information in question, inserting or deleting records can cause a need for the container itself to internally have to re-index itself. For a small container, this is fast and easy - it's essentially 'instant'. For a larger container, it's still pretty fast, but depending on how much larger we're talking about, it might take a barely perceptible amount of time. I would expect this to be an issue likely only with entries in :*:itemdesc.cfg or :*:npcdesc.cfg... however, those cannot be modified while running, anyway; they can only have items appended to them.
It should also be noted that I'm referring to the 'files' still, but in reality, that's not at all what's important here. In ALL cases, the information is being loaded into memory the first time it's requested, and it's read from there... not from the file. If it was doing all reads from the file, this would all be too slow to make the game playable. However, using the 'file' is the interface we have, so that's why I'm referring to them as 'files'.
Basically, if the information in question is already being stored in an existing config, such as itemdesc or npcdesc, I wouldn't bother changing it, especially if it's a significant amount of information. Loading from itemdesc/npcdesc works plenty fast enough to access NPC and Weapon information during a combat hook, for example, while having that combat hook script take up no appreciable sysload, according to the profiles. I'd say loading info during a combat hook is likely one of the potentially most 'critical' operations that can be done (using the term differently from the meaning related to 'critical blocks' of code, thread-wise); if something works well there, then it's good.
But, if you have NEW information that must be modified now and again, the new information I've learned, suggesting that the data from both config files and from data files is stored in memory by the same means, then by all means, feel free to use Datafiles, as well. Just keep in mind that unlike config files, datafiles must be written to disk at every world save, regardless whether they have been changed... that's also likely a total non-issue.