Play Now Login Create Account
illyriad
  New Posts New Posts RSS Feed - Data provision
  FAQ FAQ  Forum Search   Register Register  Login Login

Topic ClosedData provision

 Post Reply Post Reply
Author
GM Stormcrow View Drop Down
Moderator Group
Moderator Group
Avatar
GM

Joined: 23 Feb 2010
Location: Illyria
Status: Offline
Points: 3926
Direct Link To This Post Topic: Data provision
    Posted: 06 Aug 2010 at 12:56
Alrighty.

We're doing some work on providing XML data out for a variety of things.

The broad picture is that:
  • There's not enough data available
  • The current copy'n'pasta system from combat reports is terrible
  • The current towns xml is desperately bloated and should be slimmed
  • We want to improve this
So...

We know that many players might be able to use (eg) Excel to import a specific combat report, but also that they may not want to write a full set of import routines for a bunch of lookup tables, so we have a requirement for a hybrid/dual solution.

We know a JSON dump would keep the files smaller, but - frankly - I know nothing about JSON and so things are as they are for now, however it's distinctly possible that in the future GM ThunderCat might have some time to do JSON versions of the feeds.

Taking the items one-by-one:


Not enough data available
We're adding more feeds (and factoring other feeds).

For a start, check out:
http://uk1.illyriad.co.uk/data_downloads/datafile_alliances.xml

and

http://uk1.illyriad.co.uk/data_downloads/datafile_players.xml

The first (alliances) is an expanded dataset.

The second (players) you extrapolate already from the towns file, but we want to provide these separately in the future.

Both generate overnight at approx midnight, as per usual.

The current copy'n'pasta system from combat reports is terrible
We agree. 

So, from later on today, ingame mail combat reports will have an "XML" link to a page that specifically shows the data from that combat report in a nicely machine-readable format.

You do not need to be logged in to query the XML file, so long as you have the key, which is specific to the involved player's combat report.

We intend on accelerating this out fairly quickly to a player-by-player issued Combat API key which can be globally queried (for the whole player's history) by whoever has the key. 

This Combat API key query will return a list of combat reports that this player was involved in, and then further queries can be made to the individual reports for that player. 

This makes Killboard applications entirely possible.

We're going to have to rely on the writers of these killboard apps to not spam the system, however: the system won't accept more than one query per player key per hour, and we'll set out some easily followable rules for KB writers to know what specific combat reports they should query for "new data" vs ones they've already queried for old (perhaps from a different player who was involved in the combat as well).

Regarding historic data - we have some problems with issuing old data in the new XML format. 

The long and short of it is simply that much of the "content" data pertaining to older combat items has been destroyed (well, in a tabular sense anyway). 

For example, we still have a full copy of a combat report from 5 months ago as it was emailed (back then) to a player. 

And we have a full copy of the combat outcome - troops involved, casualties etc. 

But if their army, commander or division was subsequently destroyed  (or renamed etc) we don't have a record of what the name was (back then), unless we text parse through the old emails - which I'm really not willing to do!

And so, I'm afraid to say, we don't intend to issue these XML combat reports for combats prior to the last few days, and your older text-parsing systems will need to suffice.

PLEASE NOTE that the new XML combat log has not been tested in anger, and so I strongly recommend that people compare the new XML log to their current "text parsing" applications to make sure that everything adds up the same.  If they don't, please open Petitions ingame with specifics.

Please also note that the data visibility requirements (eg Raid reports where < 20% of atk force vs def force means that the raiders don't get a "free scout report") are being applied for these logs.

The current towns xml is desperately bloated and should be slimmed
Indeed. 

We'll be adding in some extra (new) feeds as required for terrain and for towns - and then we expect data users to use these individual feeds (terrain, players, alliances, towns) to make up the composite picture themselves. 

We will retire the current towns XML feed at some point in the future, but for the moment it'll run overnight as always.
General Notes
We will expand on the data in each and every XML feed as and when we feel like it.

We will endeavour to keep you informed of the updates in advance, but cannot guarantee this.

We will not, however, change the names of existing xml descriptors without warning, as we know this will break things.

So - when you code against the XML feeds, please bear in mind that we may be adding additional datafields to this feed without warning, and you need to ignore them if your routine doesn't understand them - but we won't break what you are already looking for.
Any thoughts / comments / suggestions?

Best,

SC



Edited by GM Stormcrow - 06 Aug 2010 at 12:59
Back to Top
GM Stormcrow View Drop Down
Moderator Group
Moderator Group
Avatar
GM

Joined: 23 Feb 2010
Location: Illyria
Status: Offline
Points: 3926
Direct Link To This Post Posted: 06 Aug 2010 at 13:06
IMPORTANT

I forgot to mention...

Combat log XML data will NOT be available if the Covert Ops stratagem is used, either individually to the player or via the API key.

This decision helps return Covert Ops to its intended function - keeping player-personal things hidden from the alliance - rather than its current function of being the default that everyone uses to prevent metagame spies from gaining access to data.


Edited by GM Stormcrow - 06 Aug 2010 at 13:08
Back to Top
GM Stormcrow View Drop Down
Moderator Group
Moderator Group
Avatar
GM

Joined: 23 Feb 2010
Location: Illyria
Status: Offline
Points: 3926
Direct Link To This Post Posted: 06 Aug 2010 at 17:07
Specific information on the Combat Reports XML is available here
Back to Top
Dogbert View Drop Down
Greenhorn
Greenhorn
Avatar

Joined: 09 May 2010
Status: Offline
Points: 72
Direct Link To This Post Posted: 07 Aug 2010 at 03:56
congrats... i really love participating in a game that is continuously growing and becoming better and better. i think you do really a good job, and sometimes i'd like to have more participation on the very interesting discussion forums. it's great to be a part of this, really.

anyhoo, as for rss feeds i'd find very usefull all town data; resources in store and in production, level of facilities, storage... all the drill. i know it'd be a resource drag, but i'm ok with it Tongue

as a collaborator, the only help i can offer are my programming skills in Excel (VBA). i know my way quite well there. but unfortunatelly i have 0 experience in other languages, least of all communicating with a web site.

i'know, it's lame. but, anything anybody needs to be done in excel, i'm your man.


(.)
Back to Top
Larry View Drop Down
Wordsmith
Wordsmith


Joined: 10 Mar 2010
Status: Offline
Points: 114
Direct Link To This Post Posted: 11 Aug 2010 at 04:02
Awesome work, looking forward to further advances.
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.03
Copyright ©2001-2019 Web Wiz Ltd.