Play Now Login Create Account
illyriad
  New Posts New Posts RSS Feed - Accessing Pages Programatically
  FAQ FAQ  Forum Search   Register Register  Login Login

Topic ClosedAccessing Pages Programatically

 Post Reply Post Reply Page  12>
Author
matiez View Drop Down
New Poster
New Poster


Joined: 31 Oct 2011
Status: Offline
Points: 30
Direct Link To This Post Topic: Accessing Pages Programatically
    Posted: 20 Jan 2012 at 18:08
I am developing a personal application that requires pulling data from the HTML pages. When the user presses a button the application will log in, parse for information, and report back. I have successfully logged in programatically. I am having issues actually reaching the data pages, though.

The URL fragments (i.e. #/Towns/Castle) are whats tripping me up. Can anyone help me with how to access these fragment pages from within Java code? The following code example returns a "Resource not found" response.

        String url2 = URLEncoder.encode("#");
        HttpPost httppost = new HttpPost("http://elgea.illyriad.co.uk/"+url2+"/Town/Castle");
        CookieSpecBase cookieSpecBase = new BrowserCompatSpec();
        List<Header> cookieHeader = cookieSpecBase.formatCookies(cookies);
        httppost.setHeader(cookieHeader.get(0));
        HttpResponse response;

        try {
            response = client.execute(httppost);
        } catch( Exception e) {}

Edit: The personal application will be made public after a potential future release from the developers, since it is against policy to request for Uname and Pwd.


Edited by matiez - 20 Jan 2012 at 18:10
Back to Top
Albatross View Drop Down
Postmaster General
Postmaster General


Joined: 11 May 2011
Status: Offline
Points: 1118
Direct Link To This Post Posted: 20 Jan 2012 at 21:18
Watch out for the Terms of Use, which explicitly prohibits automated requests for the purposes of gathering information or triggering actions. If you think it doesn't trigger actions, here's a counter-argument: it could be used to keep-alive multiple accounts.

I'm sure someone else could more eloquently encourage you towards the terms of use, and advise you on how this is (or is not) allowed.
Back to Top
GM Stormcrow View Drop Down
Moderator Group
Moderator Group
Avatar
GM

Joined: 23 Feb 2010
Location: Illyria
Status: Offline
Points: 3926
Direct Link To This Post Posted: 21 Jan 2012 at 00:01
Perhaps I can do so!

We encourage the development of third party tools - however, we want the data they access to be made available programatically by corp-defined feeds such as XML.

If people want an XML feed (via a player-revokable API key) that provides data on what's in the queues for that player/town... or the ingame mail inbox... or whatever - then please simply ask for it, preferably with a definition of the parameters you're looking for it to return.

What we don't want are tools that spider over the website, changing towns in rapid succession and pulling pages as fast as possible, as these will generally gum up the works and would also definitely be in breach of the Ts and Cs.  

However, a well-defined request for a player-revokable-API-key-based XML feed for X, Y or Z dataset from the public interface will be regarded with fondness, and pretty quick to put together as well.

Regards,

SC
Back to Top
GM Stormcrow View Drop Down
Moderator Group
Moderator Group
Avatar
GM

Joined: 23 Feb 2010
Location: Illyria
Status: Offline
Points: 3926
Direct Link To This Post Posted: 21 Jan 2012 at 04:29
Actually, I'm not sure I was clear enough above, so I want to be explicit here (ooh-err).

matiez - What you're trying to do is very much a no-no.

There's a huge difference between "passive" tools that read data from the public interface when a user clicks on something (ie plug-in browser scripts like HarmlessButler or Illytools)... and active tools that automatically do stuff, which are classified as "bots" and are a total no-no, like the tool you're suggesting.

If you want an "active" tool of some sort for data reading, XML feeds are the only way to go.  Give us the precise details of what you want and we'll put it together in a sensible way that suits both you and us.

We've put in some "flood-control" rules on the server recently after last Sunday's 2,000+ bot-driven new player sign-ups in a matter of hours debacle, and we're now actively watching for unusual patterns in the entire request set.

Anyone in breach of the "humanly possible" rules will be on the receiving end of a very, very pissed-off-50-megaton-thermonuclear Stormcrow who will be very unamused in so many ways like there's no tomorrow.  And when I talk about myself in the third person, you know it's serious.

Regards,

SC
Back to Top
matiez View Drop Down
New Poster
New Poster


Joined: 31 Oct 2011
Status: Offline
Points: 30
Direct Link To This Post Posted: 24 Jan 2012 at 14:10
Thank you for the clarification, GM Stormcrow. I will cease attempting to access information in this manner. I am highly interested in the player-revokable XML feed. I will prepare a sample XML data file that will fit my needs.

Edit: GM Stormcrow: I have an XML schema and example file for your review. Where would you like me to send them?


Edited by matiez - 24 Jan 2012 at 17:12
Back to Top
Muykul View Drop Down
New Poster
New Poster


Joined: 23 Mar 2012
Status: Offline
Points: 4
Direct Link To This Post Posted: 23 Mar 2012 at 15:32

I had an idea similar to Maitez's, so I'm looking for clarification here / playing a bit of devil's advocate.

I think the T&C condition in question is the one here:

You must not conduct any systematic or automated data collection activities (including without limitation scraping, data mining, data extraction and data harvesting) on or in relation to our website without our express written consent.

Which part of the proposed application violates this condition (or is there another?)

"Non-browser" (IE, Samba, Firefox, Opera etc..) application making HTTP requests to the Illyriad website?
User not authenticating / saving of username/password?
Pushing a button that automatically retrieves and consolidates information?

Please understand that I am not looking to violate the T&C in any way. I just thought this would be a fun side-project that would enhance my gameplay. I am simply trying to understand which part of the T&C is being violated and how that can be worked around as to not violate the T&C.

Perhaps a better way of asking this question would be - How might I go about creating an Android App that would allow me to view my cities' status without violating the T&C?

Thank you for your consideration,

-Muykul


Edited by Muykul - 23 Mar 2012 at 15:41
Back to Top
Rill View Drop Down
Postmaster General
Postmaster General
Avatar
Player Council - Geographer

Joined: 17 Jun 2011
Location: California
Status: Offline
Points: 7078
Direct Link To This Post Posted: 23 Mar 2012 at 17:27
sounds like the XML feed might do something like that, muykul.  But I'm not an expert.
Back to Top
Muykul View Drop Down
New Poster
New Poster


Joined: 23 Mar 2012
Status: Offline
Points: 4
Direct Link To This Post Posted: 23 Mar 2012 at 18:03
I agree that an XML feed would be the way to go as it wouldn't include the "overhead" (graphics, css, javascript etc...) of the full rendered HTML version. I wonder if matiez every got his schema to the devs? Half the battle is just finding an acceptable way to get the information from the webserver to the device without violating the T&C. After that is done, writing the app should be relatively trivial / easily upgraded later.

My thinking is along the lines of:
Verify writing an app is possible under the T&C
Determine which data should be downloaded.
Determine the authentication mechanism.
Determine the exchange method (XMl, API etc...)
Verify data exchange functionality
Write/test the application

I'm encouraged because the Devs have stated in other posts that they are willing to do this sort of work, however I understand they are fighting a battle against unscrupulous individuals who would try and "farm" the game for money.

-Muykul


Back to Top
GM Stormcrow View Drop Down
Moderator Group
Moderator Group
Avatar
GM

Joined: 23 Feb 2010
Location: Illyria
Status: Offline
Points: 3926
Direct Link To This Post Posted: 24 Mar 2012 at 03:46
Originally posted by Muykul Muykul wrote:

I agree that an XML feed would be the way to go as it wouldn't include the "overhead" (graphics, css, javascript etc...) of the full rendered HTML version. I wonder if matiez every got his schema to the devs? Half the battle is just finding an acceptable way to get the information from the webserver to the device without violating the T&C. After that is done, writing the app should be relatively trivial / easily upgraded later.

My thinking is along the lines of:
Verify writing an app is possible under the T&C
Determine which data should be downloaded.
Determine the authentication mechanism.
Determine the exchange method (XMl, API etc...)
Verify data exchange functionality
Write/test the application

I'm encouraged because the Devs have stated in other posts that they are willing to do this sort of work, however I understand they are fighting a battle against unscrupulous individuals who would try and "farm" the game for money.

-Muykul



Hi Muykul,

Thanks for this post; it's a useful checklist for clarification.

We have in fact already built an XML/API-key system.  

It was live once upon a time, but didn't make it over to UIv2 (largely because it wasn't being used, essentially, so wasn't a priority to move back over).  

It would, however, be quite easy to rekindle as all the server end code is written.

At present, the (non-live) system consists of:
  • A player interface ingame (account holder only, not sitters) from the account management page to an API key page, where:
  • Players can create and change on demand a highly-specific, encrypted API key that allows any application (that has access to the key) to make 2 different http combat report requests that respond with XML
  • These 2 combat report http requests are:
    • "Give me a list of the last combat reports that the players was involved in since the last combat report ID I know about for that player"
    • "Give me access to the specific combat XML report (based on the list data) as was supplied to the player on the ingame mail behind the XML link"
The server-side code for:
  • player creation and revoking of both highly-specific (eg. combat data only) and/or "full access" (everything available) API keys 
  • authentication
  • handling the http request and floodgate control
  • presenting back XML data on a web interface to an app
... that's all there already, and wouldn't require much work to expand out to other datasets.

Matiez, to my knowledge, didn't put together a full schema - at least not one I saw.

We are, however, delighted to work with players who want access to their (and also authorised third-party) data via provided and revokable API keys.

I can see great potential for apps that present (eg) ingame mail, ingame notifications, resource summaries, rankings etc.  

We're very keen on third-party data-display-only apps (as these are reasonably easy to create at our end, as well as to "police").  

We're also more than willing to consider pre-authorised usage of "action" apps, especially in non-gameplay-contentious areas such as sending ingame mails (or chat) etc.

Under well-considered conditions, we're also cautiously willing to consider authorising "action" requests that go beyond data display and into other areas.  Whilst we're reasonably confident in our ability to detect botting, we don't want to accidentally 'encourage' it.

What I strongly suggest is that people interested in writing apps for Illy let us know what kind of app they want to write (platform, public/private etc), what datasets they want access to (and in what priority order), and a rough suggested format for our data return.

It's absolutely fine if that request is in plain English - we can't expect you to know the whole database schemas we run, and we can put the request and return schema together on your behalf, add in fields as requested (and add some more that we may think other people may want), put in the API key security handling, and then present the interface that'll respond with XML to a properly formatted and authenticated http request.

I hope that's of interest?

Best,

SC
Back to Top
Muykul View Drop Down
New Poster
New Poster


Joined: 23 Mar 2012
Status: Offline
Points: 4
Direct Link To This Post Posted: 27 Mar 2012 at 18:10
Thank you for the information Stormcrow.

I would like to propose a proof of concept application. It would be a little something to get my feet wet with your system and will not require much by way of additional contribution from yourself.

Application: Illyriad Battlefield Report
Platform: Android
Purpose: Proof of Concept / Personal Use
Functions: Login to Illyriad
Receive and store API key
Use API Key to retrieve battle information
Request battle details via XML
Display / browse details of multiple battles

That should get me firmly grounded, and give me an opportunity to get Android under my belt. (I am familiar with Java, but I've never written an Android application.) Assuming that goes well, I can make a more educated request for other functionality.

-Muykul


Edited by Muykul - 27 Mar 2012 at 19:53
Back to Top
 Post Reply Post Reply Page  12>
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.03
Copyright ©2001-2019 Web Wiz Ltd.