Technology & data - Current Towns Data

Print Page | Close Window

Current Towns Data

Printed From: Illyriad
Category: Miscellaneous
Forum Name: Technology & data
Forum Description: Discussions on data dumps, downloads, and third party applications.
URL: http://forum.illyriad.co.uk/forum_posts.asp?TID=77
Printed Date: 17 Apr 2022 at 04:01
Software Version: Web Wiz Forums 12.03 - http://www.webwizforums.com

Topic: Current Towns Data

Posted By: GM Stormcrow
Subject: Current Towns Data
Date Posted: 08 Mar 2010 at 21:49

As with all these datafiles, please make sure you read and understand the terms of use, in the Sticky on this forum, http://forum.illyriad.co.uk/data-requests_topic67.html - here . Please be aware that these terms of use may change at any time.

But here's the first datafile, in XML format in the format:

http://[servername].illyriad.co.uk/data_downloads/datafile_towns.xml

So, for server UK1, this is available at:

http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml

It generates once daily, at midnight, and gives you much of the publicly accessible data about player towns ingame currently. If you schedule a download, please leave sufficient time between midnight and when it should complete generation - as the server grows it will take longer to generate, and so I'd suggest maybe 00:30 is a good time to grab it.

There are some data fields in here from which you can derive other lookup tables for your own use:

PlayerID <--> Playername
AllianceID <--> AllianceName, Ticker, Tax Rate
TownID <--> TownName

We're happy to expand the dataset out to encompass other fields should people want them (I'm thinking maybe maptile IDs and URLs etc), so if you code against this XML doc, please make sure you can handle/ignore fields you don't care about. We'd rather have a single feed that encompasses everyone's requests rather than multiple feeds.

We hope you find this a useful first stab, and please give us feedback on what else you might like to see in it.

Best wishes,

Replies:

Posted By: GM Stormcrow
Date Posted: 09 Mar 2010 at 10:47

Like it? Hate it?

Anyone care at all...?

Cry

-------------
GM Stormcrow | http://bit.ly/rLKfoT" rel="nofollow - Twitter | http://on.fb.me/uvfajA" rel="nofollow - Facebook | http://bit.ly/rBzlzf" rel="nofollow - G+

Posted By: Wuzzel
Date Posted: 09 Mar 2010 at 11:06

[22:50:34] <recendent> naive just saw this: http://forum.illyriad.co.uk/current-towns-data_topic77.html
[22:51:36] <recendent> Not going to read anymore posts by GM Stormcrow; they give me epilepsy
[22:53:01] <NotiCon> awesome
[22:53:32] <recendent> epilepsy?
[22:53:39] <recendent> :P
[22:54:29] <LauraChristine_> lol wtf
[22:55:14] <Braque> wow concentrating on it makes me nuts!
[22:56:30] <LauraChristine_> ahh i get it now lol
[22:57:37] <Braque> ^^

The above people cared SC :)

Posted By: GM Stormcrow
Date Posted: 09 Mar 2010 at 12:04

-------------
GM Stormcrow | http://bit.ly/rLKfoT" rel="nofollow - Twitter | http://on.fb.me/uvfajA" rel="nofollow - Facebook | http://bit.ly/rBzlzf" rel="nofollow - G+

Posted By: fluffy
Date Posted: 10 Mar 2010 at 00:52

There's a couple of player names that cause the xml to be parsed incorrectly.
I've found the culplits to be:
Alâkay
Agustín

The accents are causing the parse error, i've just made them their none accented versions for now.

Posted By: GM Stormcrow
Date Posted: 10 Mar 2010 at 11:25

fluffy wrote:

There's a couple of player names that cause the xml to be parsed incorrectly.
I've found the culplits to be:
Alâkay
Agustín

The accents are causing the parse error, i've just made them their none accented versions for now.

Thanks for the heads up - we need to think about what to do (and whether to allow) non-standard ASCII characters in Player / Alliance / Town / Commander / Army / Division names, and will get back to you on this one.

Posted By: naive
Date Posted: 10 Mar 2010 at 13:04

GM Stormcrow wrote:

Like it? Hate it?

Anyone care at all...?

Cry

i'm pretty sure that's exactly what we're looking for (at least to start with). now it's just a matter of finding time to play with the data Big smile

Posted By: GM ThunderCat
Date Posted: 10 Mar 2010 at 14:17

GM Stormcrow wrote:

Thanks for the heads up - we need to think about what to do (and whether to allow) non-standard ASCII characters in Player / Alliance / Town / Commander / Army / Division names, and will get back to you on this one.

Nose to the gridstone... Will be sorted shortly....

Posted By: GM ThunderCat
Date Posted: 10 Mar 2010 at 18:34

Ok, give that a go...

Hopefully syntax not changed much - but should work correctly now.

Posted By: fluffy
Date Posted: 10 Mar 2010 at 23:51

Maybe its just me, but i dont think its working still. can't figure out why though.
Also, could you maybe add the race of the person to it?

Posted By: GM ThunderCat
Date Posted: 11 Mar 2010 at 00:10

Should work this time - decided to rewrite the file IO as well

Added playerrace as a bonus...

Its turning out to be quite a lightwieght peice so am considering if a faster refresh cycle. Would that be of any use?

Posted By: fluffy
Date Posted: 11 Mar 2010 at 00:41

well as of right now, the only thing thats changing almost constantly would be the population if im not mistaken. well and alliances I guess too.

Seems to be working now as well :D thanks

Posted By: GM ThunderCat
Date Posted: 11 Mar 2010 at 00:45

Excellent! :-)

Posted By: ajdub
Date Posted: 15 Mar 2010 at 19:41

score?

Posted By: GM Stormcrow
Date Posted: 15 Mar 2010 at 20:38

ajdub wrote:

score?

2-Nil to the Dev team.

<rimshot>

Next! Tongue

Srsly tho, this is a good one. Attached to the Towns data, in the player record? Or separate?

Though I'd want the "Ranking" changes we discussed a while back in Wuzzel's thread to be in place prior to this.

-------------
GM Stormcrow | http://bit.ly/rLKfoT" rel="nofollow - Twitter | http://on.fb.me/uvfajA" rel="nofollow - Facebook | http://bit.ly/rBzlzf" rel="nofollow - G+

Posted By: GM Stormcrow
Date Posted: 07 Jul 2010 at 19:16

Hi all,

Just a heads up during preparations for sovereignty.

Please be aware that we are changing some of the names of the Terrain Type descriptions shortly.

If you have lookups tables generated from these values, you may wish to regenerate the Lookup tables.

For example, with...

<terraintype id="4">Light woods</terraintype>
<terrainoveralltype id="6">Small Forest</terrainoveralltype>

... TerrainTypeID 4 may be renamed "Small Copse" in the future.

The TerrainTypeID number will remain the same, the TerrainOverallTypeID remains the same and the TerrainOverallType description remains the same, it's just the description of TerrainTypeID that might change.

Best wishes,

GM Stormcrow

Posted By: GM Stormcrow
Date Posted: 07 Jul 2010 at 19:17

Oh, and we'll post in here when we've changed the descriptions.

It's likely to be tomorrow sometime.

Apologies for the short notice.

Posted By: GM Stormcrow
Date Posted: 08 Jul 2010 at 16:04

Terrain Names have been updated.

Basically, any second-word non-capitalisation has been fixed, plus some "Plains" and "Clay" tiles have been renamed.

TerrainTypeID 10 is now "Turned Clay"
TerrainTypeID 11 is now "Heavy Clay Seam"
TerrainTypeID 30 is now "Scrubland"
TerrainTypeID 31 is now "Clearing"
TerrainTypeID 32 is now "Tundra"
TerrainTypeID 33 is now "Open Plains"
TerrainTypeID 34 is now "Moor"

Again, apologies for the short notice. Was suddenly required for Sovereignty update.

Posted By: HonoredMule
Date Posted: 08 Jul 2010 at 17:13

Sweet, I always wanted to build a city on a moor. Ferns and moss for every dinner table.

Posted By: Larry
Date Posted: 08 Jul 2010 at 19:57

I feel rather strongly that the format the xml file ought to be restructured. As it is there is an immense amount of duplication of data, particularly among players with multiple towns. Each and every one of their towns duplicates not only all of the player data but also (as a consequence) all of the alliance data. The downsides to this are rather obvious from both a resource usage and parsing speed perspective.

Instead you could break it up into 3 sections, Alliances, Players, and Towns. Players are tied to the given alliance by adding a <alliancegameid>18</alliancegameid> tag inside of a given <player> that corrosponds with the id found in <alliancename id="18">Council Of Illyriad</alliancename>. Same with towns. Pastebin example below

http://pastebin.com/HNYJWMiP - http://pastebin.com/HNYJWMiP

I should also note that JSON would be superior to xml for all sorts of reasons (file size, parsing speed / ease) (yes I know its less human readable, but who's actually reading these files vs. parsing them?)

Posted By: Larry
Date Posted: 12 Jul 2010 at 06:47

Larry wrote:

I feel rather strongly that the format the xml file ought to be restructured. As it is there is an immense amount of duplication of data, particularly among players with multiple towns. Each and every one of their towns duplicates not only all of the player data but also (as a consequence) all of the alliance data. The downsides to this are rather obvious from both a resource usage and parsing speed perspective.

http://pastebin.com/HNYJWMiP - http://pastebin.com/HNYJWMiP

Any thoughts at all? Anything from "This idea sucks here's why" to "great idea, but we're too busy right now" to "we'll do this tomorrow" would be awesome :)

Posted By: GM Stormcrow
Date Posted: 13 Jul 2010 at 08:57

Larry wrote:

Any thoughts at all? Anything from "This idea sucks here's why" to "great idea, but we're too busy right now" to "we'll do this tomorrow" would be awesome :)

I certainly agree we should separate the data into separate, smaller files.

I'm not really sure about json, but that's largely because I personally have no experience with the structure/format; so that'd be a question for ThunderCat.

However, your middle answer is pretty much the right one!

Best,

SC

Posted By: HonoredMule
Date Posted: 14 Jul 2010 at 18:32

File size is getting quite unwieldy, but there are many use cases where a gzipped file would be just as handy and faster to fetch besides. The raw XML is approaching 8MB and still climbing steadily while a gzipped version runs around 600KB.

Think we could get a datafile_towns.xml.gz?

Format amendments (data normalization and JSON) would also be welcome--preferrably keeping the old version around too, at least for a transition period--but (optional) file compression can do the most to improve download time and bandwidth usage.

Posted By: Dogbert
Date Posted: 15 Jul 2010 at 02:19

it would also be awesome and impossible an illyriad map... now with sov...

-------------
(.)

Posted By: Shrapnel
Date Posted: 26 Aug 2010 at 14:46

How can I download this file and look at it? I tried using my browser to go to the given address, but it keeps freezing up my browser.

Posted By: GM ThunderCat
Date Posted: 27 Aug 2010 at 11:11

Shrapnel wrote:

How can I download this file and look at it? I tried using my browser to go to the given address, but it keeps freezing up my browser.

Right-click an choose save as; otherwise your browser will try to open and display the file and it is rather large...

Posted By: Shrapnel
Date Posted: 27 Aug 2010 at 13:33

Unfortunately it wasn't given as a hyperlink, which is why I didn't think of that, but I thought of a way to get around it. Thanks.

http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml -

Posted By: Larry
Date Posted: 27 Aug 2010 at 17:28

Shrapnel wrote:

Unfortunately it wasn't given as a hyperlink, which is why I didn't think of that, but I thought of a way to get around it. Thanks.

curl http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml -o datafile_towns.xml

Posted By: fluffy
Date Posted: 28 Aug 2010 at 05:14

Dogbert wrote:

it would also be awesome and impossible an illyriad map... now with sov...

tehee :D

Posted By: Shrapnel
Date Posted: 06 Sep 2010 at 02:19

I'm trying to use the xml data provided by our great GMs, but I'm having difficulty. I'd appreciate any help. Here's my code:

<?php
        // create curl resource
        $ch = curl_init();

        // set url
        curl_setopt($ch, CURLOPT_URL, "http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml");

        //return the transfer as a string
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

        // $output contains the output string
        $output = curl_exec($ch);

        // close curl resource to free up system resources
        curl_close($ch);

$p = xml_parser_create();
xml_parse_into_struct($p, $output, $vals, $index);
xml_parser_free($p);
echo "Index array\n";
print_r($index);
echo "\nVals array\n";
print_r($vals);

?>

It doesn't work though. Instead I get the following error:
Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 40 bytes) in /home/cased/public_html/IllyriadTownData.php on line 19

Can anyone help?

Posted By: HonoredMule
Date Posted: 06 Sep 2010 at 16:39

This:

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

is a no-go. You're trying to load the entire file into memory, and it's way too large for that. You need to instead set a file location to download it like so:

curl_setopt($ch, CURLOPT_FILE, '/path/to/file.xml');

After that you'll have to adjust your xml parsing code to open a file handle and parse from that instead of a string. For that I have no examples as currently I'm only downloading and gzipping the files for historical archives and smaller downloads by gzip-capable client-side tools.

Posted By: Shrapnel
Date Posted: 08 Sep 2010 at 15:58

I revised my script based on new information found and HonoredMule's suggestion (thanks HM). It's been a pain in the butt, but I finally got it to print out town names and player names, however it's not 100% correct. It's assigning towns to the wrong players. Here's the new code:


<?php
     
  
$ch = curl_init(" http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml - http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml ");
$fp = fopen("towndata.xml", "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
  $reader = new XMLReader();
  $reader->open("towndata.xml");
  echo("<HTML><BODY>");
  
  
 
   while ($reader->read()) {
     
   FindElement($reader,"mapx");
   $mapx = $reader->value;
   FindElement($reader,"mapy");
   $mapy = $reader->value;
   FindElement($reader,"terraintype");
   $terraintype = $reader->value;
   FindElement($reader,"playername");
   $playername = $reader->value;
   FindElement($reader,"playerrace");
   $playerrace = $reader->value;
   FindElement($reader,"alliancename");
   $alliancename = $reader->value;
   FindElement($reader,"allianceticker");
   $allianceticker = $reader->value;
   FindElement($reader,"alliancetaxrate");
   $alliancetaxrate = $reader->value;
   
   FindElement($reader,"townname");
   $townname = $reader->value;
   
   FindElement($reader,"population");
   $population = $reader->value;
   FindElement($reader,"iscapitalcity");
   $iscapitalcity = $reader->value;
   FindElement($reader,"isalliancecapitalcity");
   $isalliancecapitalcity = $reader->value;
      
   if($allianceticker == "FDU")
   {
    echo "Town Name:  ", $townname, "<BR>";
    echo "Player Name:  ", $playername, "<BR>";
    
   }
   
   }
   
   echo("</BODY></HTML>");
   
        
function FindElement(&$readerobject, $name)
{
    while($readerobject->name != $name)
 {
  if(!$readerobject->read()) break;
 }
 if($readerobject->name == $name) 
 {$readerobject->read();
  return 1;
 }
 else return 0;
}   
?>

FindElement is a function I wrote to skip to the appropriate tag that contains the information I'm seeking. I'm not familiar enough with XMLReader and had a heck of a time finding any examples or tutorials on how to use it. The problem may be there.

Posted By: Larry
Date Posted: 08 Sep 2010 at 16:56

For reference, the entire code to the parser used to populate www.illyriad.heroku.com can be found at http://github.com/benwilson512/illyriad/blob/master/app/models/parser.rb . I realize its in ruby, but the rough concept will be roughly the same in other languages as well (at least, I've seen a similar parser in Python and it ran nearly the exact same way).

I should note that that parser DOES in fact load the whole file into memory (the process consumes about 240mb, including the ruby interpreter and so on), although streaming it could work without really changing any of the core methods, just how the file is opened.

Posted By: Shrapnel
Date Posted: 09 Sep 2010 at 16:03

Problem solved. Towns unaffiliated with any alliance do not contain the alliancename and allianceticker tags. I did not account for that.

Posted By: HonoredMule
Date Posted: 09 Sep 2010 at 23:22

It sounds like you've written your own custom parser, which is considerably more work than necessary. If I've guessed this correctly, you might want to look into SimpleXML. It's a library built into PHP that parses XML and provides easy access to all this data. You can extract all the information to a data structure of your own design (or better yet a database) with just a few lines of code, or even just find the information you want in-place (though it's not as clean and easy--or efficient--as it should be due to how this particular XML is structured).

It's even cleaner, shorter, and more structured/organized than the Ruby example above. For example in towndata.xml you can just loop over the set of top-level <town> nodes and within that loop directly access $currentTown->location->mapx, $currentTown->townname, $currentTown->townname['id'], etc. If the data doesn't exist for that town, you just get back a null.

http://php.net/manual/en/simplexml.examples-basic.php

Posted By: Shrapnel
Date Posted: 10 Sep 2010 at 04:20

I tried SimpleXML first. I kept getting an error and a few people told me php has a memory limit and the towndata xml is too big for it, so I had to move onto XMLReader. I finally got a working tool. It's not pretty to look at, and I don't have and data validation yet, but it successfully returns searches based on several criteria. You can see it at ohiotech.elementfx.com/Illyriad.html.

Posted By: fluffy
Date Posted: 21 Apr 2011 at 23:47

anyone know whether theres a new link for the data? the link doesnt seem to work anymore:(

Posted By: HonoredMule
Date Posted: 21 Apr 2011 at 23:56

change the url to old.illyriad.co.uk/...

Posted By: Kenjifujima
Date Posted: 28 Apr 2011 at 04:28

Problem solved. not affiliated with any alliance cities contain labels and alliancename allianceticker. I would not mind him.

-------------
http://www.wly.com/Runescape.gold - Buy Runescape Gold

Posted By: Albatross
Date Posted: 19 Oct 2011 at 14:36

I'm finding that town data is out-of-date, even though the file is internally tagged as being generated within the last 24 hours.

e.g. In the XML file I see a town at a location, but going to it in the live map shows a different player on it.

Am I missing something here?

Posted By: HonoredMule
Date Posted: 19 Oct 2011 at 19:40

Unless the new player owned that city prior to the XML generation date, I don't see anything amiss.

-------------
"Apparently, quoting me is a 'thing' now."
- HonoredMule

Posted By: Durham
Date Posted: 03 Nov 2011 at 13:25

I would like to create an overview of all my armies in my 8 towns. I know next to nothing about using the available data so any advice on a good learning site or book?

Posted By: Hugie
Date Posted: 04 Nov 2011 at 21:40

Looks as though there are about 9 or 10 changes of owners of towns every 3 days, so I would expect you to find that.

I matched towns with mapx mapy - took me ages to realise that id5 contains a unique ID for each (I'm loading the file into MS Excel because I can't work computers that don't use Windows - my bad and I'm in awe of people who can Star

)

Posted By: Bonaparta
Date Posted: 24 Nov 2011 at 21:03

Durham wrote:

I would like to create an overview of all my armies in my 8 towns. I know next to nothing about using the available data so any advice on a good learning site or book?

I've created google doc table for this purpose, it is https://docs.google.com/spreadsheet/ccc?key=0Aj2ixHh9gxNmdDJjY0ZYc1lBb3ZaaHdBaEUyR0pPVlE&hl=en_US#gid=0" rel="nofollow - here . Feel free to make your own personal copy or export it to excel.

Posted By: Julius Cofen
Date Posted: 07 Dec 2011 at 07:48

A quick query... a while ago I saw somebody who had used this data to generate graphs of town growth over time for each town, but I've lost the link. Can somebody tell me where I'd find that?

Or, failing that, does anyone have a historical collection of these files they can share?

Thanks.

Posted By: Hugie
Date Posted: 07 Dec 2011 at 23:59

only a few recent ones Julius Cofen. What can you do with data since 21 Oct?

Posted By: JimJams
Date Posted: 06 Feb 2012 at 22:05

The link isn't working for me... it is just me ?