Print Page | Close Window

Daily XML dumps down

Printed From: Illyriad
Category: Miscellaneous
Forum Name: Technology & data
Forum Description: Discussions on data dumps, downloads, and third party applications.
URL: http://forum.illyriad.co.uk/forum_posts.asp?TID=5414
Printed Date: 16 Apr 2022 at 21:12
Software Version: Web Wiz Forums 12.03 - http://www.webwizforums.com


Topic: Daily XML dumps down
Posted By: HonoredMule
Subject: Daily XML dumps down
Date Posted: 30 Jan 2014 at 04:51
My webserver alerted me that today's XML dumps cannot be reached - server returns 503.  Retried a few times over the next 20 minutes but no luck.  I'm hoping this can be fixed soon enough to avoid ending up with a one (or more) day gap in my archives.

Geek


-------------
"Apparently, quoting me is a 'thing' now."
- HonoredMule


Replies:
Posted By: ubluntu
Date Posted: 30 Jan 2014 at 05:11
Im getting 503 for all datafiles as well.

$ curl -I http://elgea.illyriad.co.uk/data_downloads/datafile_{towns.xml,players.xml,alliances.xml,factions.xml,worldmap.txt,terrain.xml,terrain_combat.xml}
HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

HTTP/1.1 503 Service Unavailable
Content-Length: 326
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Jan 2014 05:08:28 GMT
Connection: close

Posted By: Malek
Date Posted: 30 Jan 2014 at 09:21
Possibly a merging of the two maps, I know that SC hinted at that end of last year.  No clue if it would affect the xml's though. 

Posted By: GM ThunderCat
Date Posted: 30 Jan 2014 at 13:31
We have moved the data to a different server to reduce the bandwidth requirements for this data on the game server; and improve the compression level for download, if you use http://data-root.illyriad.co.uk/ plus file name you should be able to access the data.

e.g. datafile_towns_new.xml was 45MB, now is served compressed at 3.9MB, though is the same data.

Posted By: GM ThunderCat
Date Posted: 30 Jan 2014 at 14:13
Originally posted by GM ThunderCat GM ThunderCat wrote:

We have moved the data to a different server to reduce the bandwidth requirements for this data on the game server; and improve the compression level for download, if you use http://data-root.illyriad.co.uk/ plus file name you should be able to access the data.

e.g. datafile_towns_new.xml was 45MB, now is served compressed at 3.9MB, though is the same data.
Files are indexed via  http://data-root.illyriad.co.uk/data.html" rel="nofollow - http://data-root.illyriad.co.uk/data.html

Posted By: Albatross
Date Posted: 30 Jan 2014 at 14:23
Thanks TC.

It should be noted (in my setup at least) that the file is downloaded as the requested name, e.g. datafile_towns.xml, but it's actually a compressed file (rather than just being transported using protocol compression), so needs to be uncompressed prior to parsing, if your downloader or parser don't already do that.


-------------

Posted By: GM ThunderCat
Date Posted: 30 Jan 2014 at 14:32
Originally posted by Albatross Albatross wrote:

Thanks TC.

It should be noted (in my setup at least) that the file is downloaded as the requested name, e.g. datafile_towns.xml, but it's actually a compressed file (rather than just being transported using protocol compression), so needs to be uncompressed prior to parsing, if your downloader or parser don't already do that.
It should have the gzip content-encoding http header on it; but if your downloader doesn't understand gzip compressed http streams, yes you will need to decompress it.

Add a .gz extension if this is the case.

Posted By: Albatross
Date Posted: 30 Jan 2014 at 14:43
Originally posted by GM ThunderCat GM ThunderCat wrote:

Add a .gz extension if this is the case.
*nods*
I'm using PowerShell's $webclient.DownloadFile() function, which seems to ignore that header info.
Yes, I told 7-zip it was a .gz file, and that worked.


-------------

Posted By: GM ThunderCat
Date Posted: 30 Jan 2014 at 14:53
Originally posted by Albatross Albatross wrote:

Originally posted by GM ThunderCat GM ThunderCat wrote:

Add a .gz extension if this is the case.
*nods*
I'm using PowerShell's $webclient.DownloadFile() function, which seems to ignore that header info.
Yes, I told 7-zip it was a .gz file, and that worked.
My PowerShell is a bit rusty; but can you set?

$webclient.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

Posted By: HonoredMule
Date Posted: 30 Jan 2014 at 17:20
Thanks TC, all good now.

-------------
"Apparently, quoting me is a 'thing' now."
- HonoredMule

Posted By: Albatross
Date Posted: 30 Jan 2014 at 17:41
Originally posted by GM ThunderCat GM ThunderCat wrote:

My PowerShell is a bit rusty; but can you set?
$webclient.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
That might work in C# or other platform, but Powershell is not playing ball using that API.

BITS transfer service also seems to ignore the header, and cannot be set to explicitly unpack gzip on transfer. It does have a nice progress bar though :o]
I think it's going to have to be decompressed inline at my end. It's no bother; just less tidy. No further action required.

FYI, some issues are discussed here:
> http://jon.netdork.net/2012/07/08/downloading-files-with-powershell/


-------------

Posted By: Malek
Date Posted: 31 Jan 2014 at 07:29
Thanks for the Update TC. Now I better see if i can get it to work.

Posted By: MoguI
Date Posted: 31 Jan 2014 at 07:59
I am using php cron job and reading it as data stream to XMLReader.
Adding 'compress.zlib://' before url makes it possible to read now.

$xmlReader = new XMLReader;
$ret = $xmlReader->open('compress.zlib://http://data-root.illyriad.co.uk/datafile_towns.xml');

PS:
I missed this info yesterday and so my "intelligent" script deleted all abandoned towns from my database table as no more existent and so I lost info when those towns where abandoned.
Therefore list of abandoned towns (http://www.puzzleslogic.com/illy/suspended_towns.php) now shows all as abandoned yesterday... and I can't fix that because I lost data, but it will fix itself during next 2 weeks...

Posted By: Albatross
Date Posted: 31 Jan 2014 at 10:38
Originally posted by MoguI MoguI wrote:

Adding 'compress.zlib://' before url makes it possible to read now.
...my "intelligent" script deleted all abandoned towns from my database table as no more existent and so I lost info
(thanks for hint)
Try having 'abandoned' and 'removed' as different data: 'abandoned' by checking town names for that text, and set Removed (Boolean: true when the town id is not present in the latest info).


-------------

Posted By: HonoredMule
Date Posted: 01 Feb 2014 at 19:01
Originally posted by MoguI MoguI wrote:

I am using php cron job and reading it as data stream to XMLReader.
Adding 'compress.zlib://' before url makes it possible to read now.

$xmlReader = new XMLReader;
$ret = $xmlReader->open('compress.zlib://http://data-root.illyriad.co.uk/datafile_towns.xml');

PS:
I missed this info yesterday and so my "intelligent" script deleted all abandoned towns from my database table as no more existent and so I lost info when those towns where abandoned.
Therefore list of abandoned towns (http://www.puzzleslogic.com/illy/suspended_towns.php) now shows all as abandoned yesterday... and I can't fix that because I lost data, but it will fix itself during next 2 weeks...



Hopefully this serves as a good lesson in error handling.  First get file, then verify file legit/download successful.  Then and only then process data.  PHP makes it very easy to do a lot with just a little code and that's a feature I well appreciate.  But it's up to the developer to identify when code is doing too much at once and/or with too little oversight.

Lesson 2 involves data retention and archival/backup, both for source data and running state. Wink


-------------
"Apparently, quoting me is a 'thing' now."
- HonoredMule

Posted By: Samalander
Date Posted: 07 Feb 2014 at 01:43
  Does anyone know if this why Illytools is not updating city locations.   Exodused 1 and alt  made new  1 neither shows up.                        Confused       

Posted By: ubluntu
Date Posted: 07 Feb 2014 at 04:41
Originally posted by Samalander Samalander wrote:

  Does anyone know if this why Illytools is not updating city locations.   Exodused 1 and alt  made new  1 neither shows up.                        Confused       

I presume illytools will not get any new town data until Eaque updates the download url in his scripts as myself, HonoredMule, & Mogul had to.

Posted By: Albatross
Date Posted: 07 Feb 2014 at 13:01
*nod* IllyTools has not updating its data since the end of last month, assumedly for exactly that reason.

-------------


Print Page | Close Window

Forum Software by Web Wiz Forums® version 12.03 - http://www.webwizforums.com
Copyright ©2001-2019 Web Wiz Ltd. - https://www.webwiz.net