Print Page | Close Window

Connectivity Issues - broken images, buttons etc

Printed From: Illyriad
Category: News & Announcements
Forum Name: News & Announcements
Forum Description: Changes, patch release dates, server launch dates, downtime notifications etc.
URL: http://forum.illyriad.co.uk/forum_posts.asp?TID=4450
Printed Date: 16 Apr 2022 at 21:04
Software Version: Web Wiz Forums 12.03 - http://www.webwizforums.com


Topic: Connectivity Issues - broken images, buttons etc
Posted By: GM Stormcrow
Subject: Connectivity Issues - broken images, buttons etc
Date Posted: 08 Nov 2012 at 18:23
Hi everyone,

tldr Summary
We're aware that some players are experiencing connectivity issues to Illy at the moment.

These issues manifest themselves as broken images, missing buttons, and as broken scripts that can prevent much of the illy functionality from initialising.

Only a small subset of users are affected by this... and it will resolve itself shortly.

Problem 1 - the CDN
We use a Content Delivery Network (CDN) to deliver our "most loaded/greatest bandwidth" files to each player's web browser. These files almost entirely consist of images and script files. 

The reason why we use a CDN is that they generally can deliver these larger files to each player more reliably and speedily than our servers can, as the CDNs use servers located geographically close to each individual player.

The problem is that our (now old) CDN (Rackspace/Akamai) has had a repeat of the same issue we experienced about a month ago ( http://forum.illyriad.co.uk/display-problem-on-chrome_topic4286_post54887.html#54887" rel="nofollow - as noted here ) and apparently are unable to rectify the issue with any sense of urgency.

We've thrown our bill-paying toys out of the corporate perambulator, and have switched CDN providers so as to (hopefully) avoid this kind of problem in the future.  

This has required us to re-point our CDN content domain names - in the form of cdnX.illyriad.co.uk - from the faulty CDN provider to our new CDN provider.

Problem 2 - a corrupt upstream routing table
Slightly exacerbating the CDN issue for some people is that, close upstream from the elgea server (based in Tampa, Florida), one of our 4 backbone internet providers (Level3 in this case) experienced a corrupt routing table at their Miami hub.

This meant that players on certain parts of the internet (notably Verizon users in the south and south-west) may have potentially experienced a brief double-whammy of technical issues.  Our ISP quickly rerouted traffic bound in that direction onto the other backbone providers, and Level3 quickly offlined the corrupted router.  But this second issue won't have helped anything.

Who is affected
Anyone who had the existing images and scripts in their local browser cache would not be experiencing this issue, and anyone whose Internet Service Providers' DNS boxes have caught up with the change will similarly be fine.  

You'll know if this issue is affecting you, as your User Interface may be partially or completely broken, with missing or broken images being the most obviously visible symptom.

Actual ongoing gameplay (completion of ongoing construction, research or military orders etc) is entirely unaffected, for all users.

The Solution
The downside of this issue is that it takes the internet's Domain Name Servers (DNS) a while to catch up with the propagation of the amended record of the domain names we use for content delivery. 

Our primary, authoritative nameservers are based in the United Kingdom at the Telehouse in London's Docklands (sort of the European equivalent of Carrier Hotel in New York), so the propagation out from us has been fairly swift.

The speed of this propagation is, however, something we don't have much control over - but can take up to 48 hours, depending on how "mainstream" or "backwoods" your Internet Service Provider is. 

Entirely anecdotally... 3 hours ago I was experiencing some broken images, but my ISP (in the southern USA where I currently am located) has now caught up and everything is working again for me - so the DNS changes are definitely propagating around.

------

We're very sorry if you're one of those players who have experienced these issues.  

Unfortunately, neither issue was of our own making, and there's nothing we could have done (or can do now) to rectify either issue more speedily, as they're both entirely out of our control.  We can but hope that our new CDN will be more reliable.

Best wishes,

SC



Replies:
Posted By: Elmindra
Date Posted: 08 Nov 2012 at 18:36
Thanks for the quick update and we understand that the interwebs is one giant maze of tubes where sometimes the routing fairies decide to go on strike.


Posted By: dunnoob
Date Posted: 08 Nov 2012 at 18:43
Originally posted by GM Stormcrow GM Stormcrow wrote:

The speed of this propagation is, however, something we don't have much control over - but can take up to 48 hours, depending on how "mainstream" or "backwoods" your Internet Service Provider is. 
For planned changes you could reduce the TTL (time to live, suggested DNS cache time).  Against all odds my ISP refused to be a part of the backwoods today, and everything works with msecnd.net.


Posted By: belargyle
Date Posted: 08 Nov 2012 at 18:52
Not sure if what I am experiencing is the same thing. But I did get a whole bunch of broken images, then I refreshed the page an all the icons were no longer there. So I logged out, and now I cannot log back in. The login page uploads fine, and I can place my name and password in but it will not respond to my trying to login via mouse, enter key or anything.. even hitting the computer didn't work :)  I can't even login under my alt.

Is this part of it?


Posted By: Aeonflux
Date Posted: 08 Nov 2012 at 18:54
thanks SC! quick response as well, and LOL @ Elmindra. Not sure what we would could expect differently from routing fairies, since they only work for dust and such. LOL


Posted By: GM Stormcrow
Date Posted: 08 Nov 2012 at 18:55
Originally posted by dunnoob dunnoob wrote:

 For planned changes you could reduce the TTL (time to live, suggested DNS cache time). 

Yup.  

Unfortunately the DNS A/CNAME amend wasn't a planned change this time (or rather, we had to do it pretty much immediately when our CDN borked itself) and - unbeknownst to us - our old crossdomain TTL appears to have been increased to 86400 by our nameserver hosts when they moved over to IPv6, grrr.  But I decided not to mention all that in my original post...

As Elmindra says, the intertube fairies are fickle indeed.





Posted By: GM Luna
Date Posted: 08 Nov 2012 at 18:59
Intertube fairies? And here I thought servers were powered by hamsters. Shows how much I know about tech support. ;)







-------------
GM Luna | Illyriad Community Manager | community@illyriad.co.uk



Posted By: GM Stormcrow
Date Posted: 08 Nov 2012 at 19:04
Originally posted by belargyle belargyle wrote:

Not sure if what I am experiencing is the same thing. But I did get a whole bunch of broken images, then I refreshed the page an all the icons were no longer there. So I logged out, and now I cannot log back in. The login page uploads fine, and I can place my name and password in but it will not respond to my trying to login via mouse, enter key or anything.. even hitting the computer didn't work :)  I can't even login under my alt.

Is this part of it?

Yes, very much so belargyle - many apologies.  It will all work again soon on it's own.

It's possible, if you're highly-network-and-OS-literate that you could force your computer to work by looking up the new authoritative records, flushing your revolver DNS cache and by putting manual DNS and IP entries for the new records into your local system's hosts file... 

But I'm certainly not recommending anyone does any of that, or even describing how to do it, as this could be very dangerous advice in inexperienced hands!  Basically, if you know how to do it, you have probably already done it.  

If you don't know how to do it or don't know what I'm talking about, I strongly recommend you don't start googling this procedure as you will inevitably end up on a 36 hours phonecall with a technical support department in Bangladesh before throwing your now entirely useless computer out of the nearest window - and blaming me for showing you how to force your computer to do things it shouldn't be doing.  Dead


Posted By: Aeonflux
Date Posted: 08 Nov 2012 at 19:05
I think I found the problem at my ISP. Thanks to this dude, I can't log into Illy.



Posted By: belargyle
Date Posted: 08 Nov 2012 at 19:11
Already did.. I'm back on :)

Thanks for the heads up though.


Posted By: GM Stormcrow
Date Posted: 08 Nov 2012 at 19:17
Originally posted by GM Luna GM Luna wrote:

Intertube fairies? And here I thought servers were powered by hamsters. Shows how much I know about tech support. ;)






You're part right, Luna... 

Hamsters power the servers, the intertubes are controlled by fairies.  Gremlins hide in the code.




Posted By: GM Stormcrow
Date Posted: 08 Nov 2012 at 19:22
Originally posted by belargyle belargyle wrote:

Already did.. I'm back on :)

Thanks for the heads up though.

Heh.  And you will remember, of course, to nslookup the cdnX domains over the next couple of days, and when they match the authoritative records, to remove the hosts file entries, won't you...?

EDIT: That's if, of course, you did the hosts file thing...  To those people who think I'm speaking even more gobbledigook than usual, please ignore everything except my first post in this thread.


Posted By: Aeonflux
Date Posted: 08 Nov 2012 at 19:46
back on illy! :) 


Posted By: Nikon
Date Posted: 09 Nov 2012 at 00:07
ok, i can understand the not logging in bit, im fine with that. but for the last 2 weeks or so ive tried to get onto illy via the illyriad.co.uk website on chrome and come up with this
 

This webpage is not available

The webpage at http://www.illyriad.co.uk/ might be temporarily down or it may have moved permanently to a new web address.
Error 330 (net::ERR_CONTENT_DECODING_FAILED): Unknown error.
is my chrome broken? or is the website broken?


Posted By: GM ThunderCat
Date Posted: 09 Nov 2012 at 10:29
Originally posted by Nikon Nikon wrote:

The webpage at http://www.illyriad.co.uk/ might be temporarily down or it may have moved permanently to a new web address.
Error 330 (net::ERR_CONTENT_DECODING_FAILED): Unknown error.
is my chrome broken? or is the website broken?
Are you using a proxy, virus checker, chrome extension, ad blocker etc that would be modifying the response for the server incorrectly?

e.g.

http://productforums.google.com/forum/#!topic/chrome/vxIgSHj3CGg" rel="nofollow - http://productforums.google.com/forum/#!topic/chrome/vxIgSHj3CGg

http://code.google.com/p/chromium/issues/detail?id=24365" rel="nofollow - http://code.google.com/p/chromium/issues/detail?id=24365


Posted By: Crozno
Date Posted: 13 Nov 2012 at 01:44
Nevermind, routing changed and I am again to login :)

Problem 2 - a corrupt upstream routing table
Slightly exacerbating the CDN issue for some people is that, close upstream from the elgea server (based in Tampa, Florida), one of our 4 backbone internet providers (Level3 in this case) experienced a corrupt routing table at their Miami hub.

Is there repeat of this issue?  I cannot get past this on traceroute:

ae-3-5.bar1.tampa1.level3.net (4.69.148.214)

...so unable to reach Illy


Posted By: Sir Bradly
Date Posted: 13 Nov 2012 at 04:40
My world map has been missing sov or posted partial sov since we've experienced this problem.  Any idea when it will be fixed?

SB



Print Page | Close Window

Forum Software by Web Wiz Forums® version 12.03 - http://www.webwizforums.com
Copyright ©2001-2019 Web Wiz Ltd. - https://www.webwiz.net