Technology & data - Index/landing page for data downloads

Print Page | Close Window

Index/landing page for data downloads

Printed From: Illyriad
Category: Miscellaneous
Forum Name: Technology & data
Forum Description: Discussions on data dumps, downloads, and third party applications.
URL: http://forum.illyriad.co.uk/forum_posts.asp?TID=3493
Printed Date: 16 Apr 2022 at 21:07
Software Version: Web Wiz Forums 12.03 - http://www.webwizforums.com

Topic: Index/landing page for data downloads

Posted By: Blargh
Subject: Index/landing page for data downloads
Date Posted: 26 Apr 2012 at 06:47

Would it be possible to add an index/landing page to http://elgea.illyriad.co.uk/data_downloads/datafile_terrain.xml" rel="nofollow - http://elgea.illyriad.co.uk/data_downloads/ such that it displays a summary of the available feeds?

Something like this:

Name	URL	Description	Size (KB)	Refresh Rate (hours)	Updated
Terrain type	file:///C:%5CDocuments%20and%20Settings%5CDJ%5CDesktop%5C" rel="nofollow - http://elgea.illyriad.co.uk/data_downloads/datafile_terrain.xml	...	13,000	24	2012-04-26T00:21:13Z
Combat Terrain type	file:///C:%5CDocuments%20and%20Settings%5CDJ%5CDesktop%5C" rel="nofollow - http://elgea.illyriad.co.uk/data_downloads/datafile_terrain_combat.xml	...	???	24	2012-04-26T00:21:13Z

The fields would be as such:

Name: Friendly name

URL: Fully-specified hyperlink to teh goods. J

Description: Format-specific notes, i.e.:

It is in a txt flat-file format to save space.
The first row contains column headers.
It is column-delimited by a Vertical Bar, like this: |
It is row-delimited by <CR><LF>
TerrainTypeID in the flat-file links to the terraintype id in the datafile_terrain.xml file
TerrainCombatTypeID in the flat-file links to the terraincombattype id in the datafile_terrain_combat.xml file

Size: Size of the (current) file in Kilobytes.

Refresh Rate: How often users should expect changes, i.e. “If it says 24 hours, don’t keep hitting this URL every 10 minutes…”

Updated: A timestamp of the last refresh in ISO 8601 format (at Zulu time in case of server migration to other timezones) J

This would consolidate the list of available feeds into a single location (rather than buried in the forum) and provide additional useful metadata (size, last updated, etc.).

Replies:

Posted By: dunnoob
Date Posted: 08 Jul 2012 at 03:56

Bump.

Posted By: Machete
Date Posted: 09 Jul 2012 at 03:42

Here are the 6 I have found. Would be nice to know if there are more.

http://elgea.illyriad.co.uk/data_downloads/datafile_towns.xml

http://elgea.illyriad.co.uk/data_downloads/datafile_players.xml

http://elgea.illyriad.co.uk/data_downloads/datafile_alliances.xml

http://elgea.illyriad.co.uk/data_downloads/datafile_worldmap.txt

http://elgea.illyriad.co.uk/data_downloads/datafile_terrain.xml

http://elgea.illyriad.co.uk/data_downloads/datafile_terrain_combat.xml

Posted By: dunnoob
Date Posted: 09 Jul 2012 at 06:48

Great, I didn't know alliances, thanks.

Posted By: allamagoosa
Date Posted: 30 Jul 2012 at 23:24

I would like to know if there are others as well.

Posted By: Prakarp
Date Posted: 14 Aug 2012 at 22:22

Now that Trade v2 is out, is there an additional XML file detailing Trade v2 locations?

Posted By: Salararius
Date Posted: 22 Oct 2012 at 02:57

I was looking at the datafiles and trying to decide if there is anything that hasn't already been done with the information. While doing so I pulled out the following guesses at their structure. Here is that information in DTD format for the XML files and the delimited fields for the delimited file if it's helpful to anyone.

*************************************************************************

************************** datafile_worldmapl.txt ***********************

*************************************************************************

delimited text file with '|' delimiters.

The fields are:

GeoID

Wood

Clay

Iron

Stone

Food

TerrainSpecificTypeID

TerrainCombatTypeID

RegionID

*************************************************************************

**************************** datafile_towns.xml *************************

*************************************************************************

<!ELEMENT towns (server, town*)>

<!ELEMENT server (name, servername, servercountrycode, serverlanguagecode, serverlivedate, datagenerationdatetime)>

<!ELEMENT name (#CDATA)>

<!ELEMENT servername (#CDATA)>

<!ELEMENT servercountrycode (#CDATA)>

<!ELEMENT serverlanguagecode (#CDATA)>

<!ELEMENT serverlivedate (#CDATA)>

<!ELEMENT datagenerationdatetime (#CDATA)>

<!ELEMENT town (location, player, towndata)>

<!ELEMENT location (mapx, mapy, terraintype, terrainoveralltype)>

<!ELEMENT mapx (#CDATA)>

<!ELEMENT mapy (#CDATA)>

<!ELEMENT terraintype (#CDATA)>

<!ATTLIST terraintype id CDATA #REQUIRED>

<!ELEMENT terrainoveralltype (#CDATA)>

<!ATTLIST terrainoveralltype id CDATA #REQUIRED>

<!ELEMENT player (playername, playerrace, playeralliance?)>

<!ELEMENT playername (#CDATA)>

<!ATTLIST playername id CDATA #REQUIRED>

<!ELEMENT playerrace (#CDATA)>

<!ELEMENT playeralliance (alliancename, allianceticker, alliancetaxrate)>

<!ELEMENT alliancename (#CDATA)>

<!ATTLIST alliancename id CDATA #REQUIRED>

<!ELEMENT allianceticker (#CDATA)>

<!ELEMENT alliancetaxrate (#CDATA)>

<!ELEMENT towndata (townname, foundeddatetime, population, iscapitalcity, isalliancecapitalcity)>

<!ELEMENT townname (#CDATA)>

<!ATTLIST townname id #CDATA #REQUIRED>

<!ELEMENT foundeddatetime (#CDATA)>

<!ELEMENT population (#CDATA)>

<!ELEMENT iscapitalcity (#CDATA)>

<!ELEMENT isalliancecapitalcity (#CDATA)>

*************************************************************************

*********************** datafile_alliances.xml **************************

*************************************************************************

<!ELEMENT alliancedata (server, alliances)>

<!ELEMENT server (name, servercountrycode, serverlanguagecode, serverlivedate, datagenerationdatetime)>

<!ELEMENT name (#CDATA)>

<!ELEMENT servercountrycode (#CDATA)>

<!ELEMENT serverlanguagecode (#CDATA)>

<!ELEMENT serverlivedate (#CDATA)>

<!ELEMENT datagenerationdatetime (#CDATA)>

<!ELEMENT alliances (alliance*)>

<!ELEMENT alliance (alliance, foundedbyplayerid, alliancecapitaltownid, allianceticker, foundeddatetime, alliancecapitallastmoved, alliancetaxrate, alliancetaxratelastchanged, membercount, totalpopulation, roles, relationships)>

<!ELEMENT alliance (#CDATA)>

<!ATTLIST alliance id #CDATA #REQUIRED>

<!ELEMENT foundedbyplayerid EMPTY>

<!ATTLIST foundedbyplayerid id #CDATA #REQUIRED>

<!ELEMENT alliancecapitaltownid EMPTY>

<!ATTLIST alliancecapitaltownid id #CDATA #REQUIRED>

<!ELEMENT allianceticker (#CDATA)>

<!ELEMENT foundeddatetime (#CDATA)>

<!ELEMENT alliancecapitallastmoved (#CDATA)>

<!ELEMENT alliancetaxrate (#CDATA)>

<!ELEMENT alliancetaxratelastchanged (#CDATA)>

<!ELEMENT membercount (#CDATA)>

<!ELEMENT totalpopulation (#CDATA)>

<!ELEMENT roles (role*)>

<!ELEMENT relationships (relationship*)>

<!ELEMENT role (role, heirarchy)>

<!ELEMENT role (#CDATA>

<!ATTLIST role id #CDATA #REQUIRED>

<!ELEMENT heirarchy EMPTY>

<!ATTLIST heirarchy id #CDATA #REQUIRED>

<!ELEMENT relationship (proposedbyalliance, acceptedbyalliance, relationshiptype, establishedsince)>

<!ELEMENT proposedbyalliance EMPTY>

<!ATTLIST proposedbyalliance id #CDATA #REQUIRED>

<!ELEMENT acceptedbyalliance EMPTY>

<!ATTLIST acceptedbyalliance id #CDATA #REQUIRED>

<!ELEMENT relationshiptype (#CDATA)>

<!ATTLIST relationshiptype id #CDATA #REQUIRED>

<!ELEMENT establishedsince (#CDATA)>

*************************************************************************

*************************** datafile_players.xml ************************

*************************************************************************

<!ELEMENT playerdata (server, players)>

<!ELEMENT server (name, servercountrycode, serverlanguagecode, serverlivedate, datagenerationdatetime)>

<!ELEMENT name (#CDATA)>

<!ELEMENT servercountrycode (#CDATA)>

<!ELEMENT serverlanguagecode (#CDATA)>

<!ELEMENT serverlivedate (#CDATA)>

<!ELEMENT datagenerationdatetime (#CDATA)>

<!ELEMENT players (player*)>

<!ELEMENT player (playername, allianceid, allianceroleid, race)>

<!ELEMENT playername (#CDATA)>

<!ATTLIST playername id #CDATA #REQUIRED>

<!ELEMENT allianceid EMPTY>

<!ATTLIST allianceid id #CDATA #REQUIRED>

<!ELEMENT allianceroleid EMPTY>

<!ATTLIST allianceroleid id #CDATA #REQUIRED>

<!ELEMENT race EMPTY>

<!ATTLIST race id #CDATA #REQUIRED>

*************************************************************************

**************************** datafile_terrain.xml ***********************

*************************************************************************

<!ELEMENT terraindata (server, terraintypes)>

<!ELEMENT server (name, servercountrycode, serverlanguagecode, serverlivedate, datagenerationdatetime)>

<!ELEMENT name (#CDATA)>

<!ELEMENT servercountrycode (#CDATA)>

<!ELEMENT serverlanguagecode (#CDATA)>

<!ELEMENT serverlivedate (#CDATA)>

<!ELEMENT datagenerationdatetime (#CDATA)>

<!ELEMENT terraintypes (terrain*)>

<!ELEMENT terrain (terrainspecifictype, impassableterrain, npcterrain)>

<!ELEMENT teterrainspecifictype rrainspecifictype (#CDATA)>

<!ATTLIST race id #CDATA #REQUIRED>

<!ELEMENT impassableterrain (#CDATA)>

<!ELEMENT npcterrain (#CDATA)>

*************************************************************************

************************** datafile_terrain_combat.xml ******************

*************************************************************************

<!ELEMENT terraincombatdata (server, terraincombattypes)>

<!ELEMENT server (name, servercountrycode, serverlanguagecode, serverlivedate, datagenerationdatetime)>

<!ELEMENT name (#CDATA)>

<!ELEMENT servercountrycode (#CDATA)>

<!ELEMENT serverlanguagecode (#CDATA)>

<!ELEMENT serverlivedate (#CDATA)>

<!ELEMENT datagenerationdatetime (#CDATA)>

<!ELEMENT terraincombattypes (terraincombat*)>

<!ELEMENT terraincombat (terraincombattype)>

<!ELEMENT terraincombattype (#CDATA)>

<!ATTLIST terraincombattype id #CDATA #REQUIRED>

Posted By: dunnoob
Date Posted: 22 Oct 2012 at 10:40

The W3 validator does not hate the http://validator.w3.org/check?uri=http%3A%2F%2Felgea.illyriad.co.uk%2Fdata_downloads%2Fdatafile_alliances.xml&charset=utf-8&doctype=Inline&group=0&user-agent=W3C_Validator%2F1.3" rel="nofollow - alliances , but maybe it's lost without a DTD. The files are rather long, adding DTDs at the top to get standalone XML files would be nice. Or they could offer them also as less verbose CSV, JSON, record-JAR, whatever... <g,d,&r>

Posted By: Salararius
Date Posted: 23 Oct 2012 at 04:19

dunnoob wrote:

The W3 validator does not hate the http://validator.w3.org/check?uri=http%3A%2F%2Felgea.illyriad.co.uk%2Fdata_downloads%2Fdatafile_alliances.xml&charset=utf-8&doctype=Inline&group=0&user-agent=W3C_Validator%2F1.3" rel="nofollow - alliances , but maybe it's lost without a DTD. The files are rather long, adding DTDs at the top to get standalone XML files would be nice. Or they could offer them also as less verbose CSV, JSON, record-JAR, whatever... <g,d,&r>

I really can't explain myself, it's seems to be a fatal flaw :( I shouldn't have said it wouldn't parse, you are right, of course it will. I don't think the XML (or rather the DTD inferred from the XML) will enforce the structure the developers want because it allows an alliance element to be nested inside an alliance element (same for roles). It only matters if someone is looking at my inferred DTD and trying to understand the data and doesn't realize that isn't possible. If it is possible, then I don't know what that would mean in terms of the data and game. If it isn't possible to nest alliance elements inside alliance elements then I haven't a clue how to represent that with a DTD structure to help people understand the data hierarchy (which was all I was trying to do for myself).

Posted By: Rosenfield
Date Posted: 23 Jan 2013 at 00:29

Has there been any progress with this issue? Any new data files or tools?

Posted By: Rosenfield
Date Posted: 23 Jan 2013 at 00:41

A landing page would be great. Assuming you know a little PHP, you can get the file's size and last-modified date without too much hassle:

$mtime = date("r",filemtime ($infile));

$size = filesize($infile);

where $infile is the url of the file you're thinking of reading in.

You could run such a script at intervals to see how often it's modified... not exactly what you're looking for, but it approaches such.

Posted By: Rosenfield
Date Posted: 06 Apr 2013 at 05:03

Is there an official data file somewhere with the chat logs, or is that just something Illytools does?

Posted By: Nesse
Date Posted: 08 Sep 2013 at 20:13

http://elgea.illyriad.co.uk/data_downloads/datafile_factions.xml

-------------
Nesse(Dwarven Druids) and Odd (Fairy Road Authority)

Posted By: ubluntu
Date Posted: 20 Sep 2013 at 05:06

Observed update frequency and current last modified time.

alliances	9/20/2013 00:11:08	datafile_alliances.xml	Daily
players	9/20/2013 00:11:08	datafile_players.xml	Daily
terrain	9/20/2013 00:11:08	datafile_terrain.xml	Daily
towns	9/20/2013 00:11:06	datafile_towns.xml	Daily
factions	9/1/2012 17:55:13	datafile_factions.xml	N/A
terrain_combat	4/20/2012 19:47:48	datafile_terrain_combat.xml	N/A
worldmap	4/22/2012 02:53:13	datafile_worldmap.txt	N/A

Some bash for headers only:

curl -I http://elgea.illyriad.co.uk/data_downloads/datafile_worldmap.txt

Edit:

Datafiles are now listed http://data-root.illyriad.co.uk/data.html" rel="nofollow - here , the files are now gzipped, and the new base url for the datafiles is 'http://data-root.illyriad.co.uk/' instead of 'http://elgea.illyriad.co.uk/data_downloads/'.

In case anyone missed http://forum.illyriad.co.uk/03feb14-data-feeds_topic5428.html" rel="nofollow - this .