Play Now Login Create Account
illyriad
  New Posts New Posts RSS Feed - Daily XML dumps down
  FAQ FAQ  Forum Search   Register Register  Login Login

Daily XML dumps down

 Post Reply Post Reply Page  <12
Author
Albatross View Drop Down
Postmaster General
Postmaster General


Joined: 11 May 2011
Status: Offline
Points: 1118
Post Options Post Options   Thanks (0) Thanks(0)   Quote Albatross Quote  Post ReplyReply Direct Link To This Post Posted: 30 Jan 2014 at 17:41
Originally posted by GM ThunderCat GM ThunderCat wrote:

My PowerShell is a bit rusty; but can you set?
$webclient.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
That might work in C# or other platform, but Powershell is not playing ball using that API.

BITS transfer service also seems to ignore the header, and cannot be set to explicitly unpack gzip on transfer. It does have a nice progress bar though :o]
I think it's going to have to be decompressed inline at my end. It's no bother; just less tidy. No further action required.

FYI, some issues are discussed here:
> http://jon.netdork.net/2012/07/08/downloading-files-with-powershell/
Back to Top
Malek View Drop Down
Forum Warrior
Forum Warrior
Avatar

Joined: 09 Jul 2012
Location: Australia
Status: Offline
Points: 252
Post Options Post Options   Thanks (0) Thanks(0)   Quote Malek Quote  Post ReplyReply Direct Link To This Post Posted: 31 Jan 2014 at 07:29
Thanks for the Update TC. Now I better see if i can get it to work.
Back to Top
MoguI View Drop Down
New Poster
New Poster


Joined: 07 Feb 2013
Status: Offline
Points: 28
Post Options Post Options   Thanks (0) Thanks(0)   Quote MoguI Quote  Post ReplyReply Direct Link To This Post Posted: 31 Jan 2014 at 07:59
I am using php cron job and reading it as data stream to XMLReader.
Adding 'compress.zlib://' before url makes it possible to read now.

$xmlReader = new XMLReader;
$ret = $xmlReader->open('compress.zlib://http://data-root.illyriad.co.uk/datafile_towns.xml');

PS:
I missed this info yesterday and so my "intelligent" script deleted all abandoned towns from my database table as no more existent and so I lost info when those towns where abandoned.
Therefore list of abandoned towns (http://www.puzzleslogic.com/illy/suspended_towns.php) now shows all as abandoned yesterday... and I can't fix that because I lost data, but it will fix itself during next 2 weeks...

Back to Top
Albatross View Drop Down
Postmaster General
Postmaster General


Joined: 11 May 2011
Status: Offline
Points: 1118
Post Options Post Options   Thanks (0) Thanks(0)   Quote Albatross Quote  Post ReplyReply Direct Link To This Post Posted: 31 Jan 2014 at 10:38
Originally posted by MoguI MoguI wrote:

Adding 'compress.zlib://' before url makes it possible to read now.
...my "intelligent" script deleted all abandoned towns from my database table as no more existent and so I lost info
(thanks for hint)
Try having 'abandoned' and 'removed' as different data: 'abandoned' by checking town names for that text, and set Removed (Boolean: true when the town id is not present in the latest info).


Edited by Albatross - 31 Jan 2014 at 10:38
Back to Top
HonoredMule View Drop Down
Postmaster General
Postmaster General
Avatar

Joined: 05 Mar 2010
Location: Canada
Status: Offline
Points: 1650
Post Options Post Options   Thanks (0) Thanks(0)   Quote HonoredMule Quote  Post ReplyReply Direct Link To This Post Posted: 01 Feb 2014 at 19:01
Originally posted by MoguI MoguI wrote:

I am using php cron job and reading it as data stream to XMLReader.
Adding 'compress.zlib://' before url makes it possible to read now.

$xmlReader = new XMLReader;
$ret = $xmlReader->open('compress.zlib://http://data-root.illyriad.co.uk/datafile_towns.xml');

PS:
I missed this info yesterday and so my "intelligent" script deleted all abandoned towns from my database table as no more existent and so I lost info when those towns where abandoned.
Therefore list of abandoned towns (http://www.puzzleslogic.com/illy/suspended_towns.php) now shows all as abandoned yesterday... and I can't fix that because I lost data, but it will fix itself during next 2 weeks...



Hopefully this serves as a good lesson in error handling.  First get file, then verify file legit/download successful.  Then and only then process data.  PHP makes it very easy to do a lot with just a little code and that's a feature I well appreciate.  But it's up to the developer to identify when code is doing too much at once and/or with too little oversight.

Lesson 2 involves data retention and archival/backup, both for source data and running state. Wink


Edited by HonoredMule - 01 Feb 2014 at 19:15
"Apparently, quoting me is a 'thing' now."
- HonoredMule
Back to Top
Samalander View Drop Down
New Poster
New Poster
Avatar

Joined: 24 Dec 2012
Location: oregon
Status: Offline
Points: 39
Post Options Post Options   Thanks (0) Thanks(0)   Quote Samalander Quote  Post ReplyReply Direct Link To This Post Posted: 07 Feb 2014 at 01:43
  Does anyone know if this why Illytools is not updating city locations.   Exodused 1 and alt  made new  1 neither shows up.                        Confused       
Back to Top
ubluntu View Drop Down
Forum Warrior
Forum Warrior


Joined: 24 Aug 2013
Status: Offline
Points: 324
Post Options Post Options   Thanks (0) Thanks(0)   Quote ubluntu Quote  Post ReplyReply Direct Link To This Post Posted: 07 Feb 2014 at 04:41
Originally posted by Samalander Samalander wrote:

  Does anyone know if this why Illytools is not updating city locations.   Exodused 1 and alt  made new  1 neither shows up.                        Confused       

I presume illytools will not get any new town data until Eaque updates the download url in his scripts as myself, HonoredMule, & Mogul had to.
Back to Top
Albatross View Drop Down
Postmaster General
Postmaster General


Joined: 11 May 2011
Status: Offline
Points: 1118
Post Options Post Options   Thanks (0) Thanks(0)   Quote Albatross Quote  Post ReplyReply Direct Link To This Post Posted: 07 Feb 2014 at 13:01
*nod* IllyTools has not updating its data since the end of last month, assumedly for exactly that reason.
Back to Top
 Post Reply Post Reply Page  <12
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.03
Copyright ©2001-2019 Web Wiz Ltd.