Play Now Login Create Account
illyriad
  New Posts New Posts RSS Feed - Current Towns Data
  FAQ FAQ  Forum Search   Register Register  Login Login

Topic ClosedCurrent Towns Data
 Post Reply Post Reply Page  <1 2345>
Author
  Topic Search Topic Search  Topic Options Topic Options
HonoredMule View Drop Down
Postmaster General
Postmaster General
Avatar

Joined: 05 Mar 2010
Location: Canada
Status: Offline
Points: 1650 		Direct Link To This Post Posted: 06 Sep 2010 at 16:39
This:

Quote curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);


is a no-go.  You're trying to load the entire file into memory, and it's way too large for that.  You need to instead set a file location to download it like so:

Quote curl_setopt($ch, CURLOPT_FILE, '/path/to/file.xml');


After that you'll have to adjust your xml parsing code to open a file handle and parse from that instead of a string.  For that I have no examples as currently I'm only downloading and gzipping the files for historical archives and smaller downloads by gzip-capable client-side tools.


Edited by HonoredMule - 06 Sep 2010 at 16:42
Back to Top
Shrapnel View Drop Down
Wordsmith
Wordsmith
Avatar

Joined: 01 Jun 2010
Status: Offline
Points: 180 		Direct Link To This Post Posted: 08 Sep 2010 at 15:58

I revised my script based on new information found and HonoredMule's suggestion (thanks HM).  It's been a pain in the butt, but I finally got it to print out town names and player names, however it's not 100% correct.  It's assigning towns to the wrong players.  Here's the new code:



<?php
     
  
$ch = curl_init("http://uk1.illyriad.co.uk/data_downloads/datafile_towns.xml");
$fp = fopen("towndata.xml", "w");


curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);


curl_exec($ch);
curl_close($ch);
fclose($fp);


  $reader = new XMLReader();
  $reader->open("towndata.xml");
  echo("<HTML><BODY>");
  
  
 
   while ($reader->read()) {
     
   FindElement($reader,"mapx");
   $mapx = $reader->value;
   FindElement($reader,"mapy");
   $mapy = $reader->value;
   FindElement($reader,"terraintype");
   $terraintype = $reader->value;
   FindElement($reader,"playername");
   $playername = $reader->value;
   FindElement($reader,"playerrace");
   $playerrace = $reader->value;
   FindElement($reader,"alliancename");
   $alliancename = $reader->value;
   FindElement($reader,"allianceticker");
   $allianceticker = $reader->value;
   FindElement($reader,"alliancetaxrate");
   $alliancetaxrate = $reader->value;
   
   FindElement($reader,"townname");
   $townname = $reader->value;
   
   FindElement($reader,"population");
   $population = $reader->value;
   FindElement($reader,"iscapitalcity");
   $iscapitalcity = $reader->value;
   FindElement($reader,"isalliancecapitalcity");
   $isalliancecapitalcity = $reader->value;
      
   if($allianceticker == "FDU")
   {
    echo "Town Name:  ", $townname, "<BR>";
    echo "Player Name:  ", $playername, "<BR>";
    
   }
   
   }
   
   echo("</BODY></HTML>");
   
        
function FindElement(&$readerobject, $name)
{
    while($readerobject->name != $name)
 {
  if(!$readerobject->read()) break;
 }
 if($readerobject->name == $name) 
 {$readerobject->read();
  return 1;
 }
 else return 0;
}   
?>

 

FindElement is a function I wrote to skip to the appropriate tag that contains the information I'm seeking.  I'm not familiar enough with XMLReader and had a heck of a time finding any examples or tutorials on how to use it.  The problem may be there.
Back to Top
Larry View Drop Down
Wordsmith
Wordsmith


Joined: 10 Mar 2010
Status: Offline
Points: 114 		Direct Link To This Post Posted: 08 Sep 2010 at 16:56
For reference, the entire code to the parser used to populate www.illyriad.heroku.com can be found at http://github.com/benwilson512/illyriad/blob/master/app/models/parser.rb . I realize its in ruby, but the rough concept will be roughly the same in other languages as well (at least, I've seen a similar parser in Python and it ran nearly the exact same way).

I should note that that parser DOES in fact load the whole file into memory (the process consumes about 240mb, including the ruby interpreter and so on), although streaming it could work without really changing any of the core methods, just how the file is opened.


Edited by Larry - 08 Sep 2010 at 17:01
Back to Top
Shrapnel View Drop Down
Wordsmith
Wordsmith
Avatar

Joined: 01 Jun 2010
Status: Offline
Points: 180 		Direct Link To This Post Posted: 09 Sep 2010 at 16:03
Problem solved.  Towns unaffiliated with any alliance do not contain the alliancename and allianceticker tags.  I did not account for that. 
Back to Top
HonoredMule View Drop Down
Postmaster General
Postmaster General
Avatar

Joined: 05 Mar 2010
Location: Canada
Status: Offline
Points: 1650 		Direct Link To This Post Posted: 09 Sep 2010 at 23:22
It sounds like you've written your own custom parser, which is considerably more work than necessary.  If I've guessed this correctly, you might want to look into SimpleXML.  It's a library built into PHP that parses XML and provides easy access to all this data.  You can extract all the information to a data structure of your own design (or better yet a database) with just a few lines of code, or even just find the information you want in-place (though it's not as clean and easy--or efficient--as it should be due to how this particular XML is structured).

It's even cleaner, shorter, and more structured/organized than the Ruby example above.  For example in towndata.xml you can just loop over the set of top-level <town> nodes and within that loop directly access $currentTown->location->mapx, $currentTown->townname, $currentTown->townname['id'], etc.  If the data doesn't exist for that town, you just get back a null.

http://php.net/manual/en/simplexml.examples-basic.php
Back to Top
Shrapnel View Drop Down
Wordsmith
Wordsmith
Avatar

Joined: 01 Jun 2010
Status: Offline
Points: 180 		Direct Link To This Post Posted: 10 Sep 2010 at 04:20
I tried SimpleXML first.  I kept getting an error and a few people told me php has a memory limit and the towndata xml is too big for it, so I had to move onto XMLReader.  I finally got a working tool.  It's not pretty to look at, and I don't have and data validation yet, but it successfully returns searches based on several criteria.  You can see it at ohiotech.elementfx.com/Illyriad.html.
Back to Top
fluffy View Drop Down
Forum Warrior
Forum Warrior
Avatar

Joined: 02 Mar 2010
Status: Offline
Points: 335 		Direct Link To This Post Posted: 21 Apr 2011 at 23:47
anyone know whether theres a new link for the data?  the link doesnt seem to work anymore:(

Back to Top
HonoredMule View Drop Down
Postmaster General
Postmaster General
Avatar

Joined: 05 Mar 2010
Location: Canada
Status: Offline
Points: 1650 		Direct Link To This Post Posted: 21 Apr 2011 at 23:56
change the url to old.illyriad.co.uk/...
Back to Top
Kenjifujima View Drop Down
New Poster
New Poster


Joined: 28 Apr 2011
Status: Offline
Points: 3 		Direct Link To This Post Posted: 28 Apr 2011 at 04:28
Problem solved. not affiliated with any alliance cities contain labels and alliancename allianceticker. I would not mind him.
Buy Runescape Gold
Back to Top
Albatross View Drop Down
Postmaster General
Postmaster General


Joined: 11 May 2011
Status: Offline
Points: 1118 		Direct Link To This Post Posted: 19 Oct 2011 at 14:36
I'm finding that town data is out-of-date, even though the file is internally tagged as being generated within the last 24 hours.

e.g. In the XML file I see a town at a location, but going to it in the live map shows a different player on it.

Am I missing something here?
Back to Top
 Post Reply Post Reply Page  <1 2345>
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.03
Copyright ©2001-2019 Web Wiz Ltd.