more armies in limbo...? urgent action needed
Printed From: Illyriad
Category: Strategies, Guides & Help
Forum Name: Technical Support
Forum Description: Post your technical support related questions here.
URL: http://forum.illyriad.co.uk/forum_posts.asp?TID=5456
Printed Date: 17 Apr 2022 at 10:38
Software Version: Web Wiz Forums 12.03 - http://www.webwizforums.com
Topic: more armies in limbo...? urgent action needed
Posted By: dittobite
Subject: more armies in limbo...? urgent action needed
Date Posted: 21 Feb 2014 at 15:28
|
vCrow and allies have placed a siege on plains on Electrok's city wormwood. Although our seige was initially hit by troops now ALL NC troops and their allies seem to be stuck in limbo around the siege site and are NOT hitting the siege. Although vCrow and allies has sufficient forces there to take out anything sent at us I believe, it still is a huge malfunction and the devs should get on this right away. Petitions have been filed but this needs to be solved yesterday since its not fair for NC and their allies to have troops tied up in a glitch like this. http://elgea.illyriad.co.uk/#/World/Map/207/-504/5%20 " rel="nofollow - http://elgea.illyriad.co.uk/#/World/Map/207/-504/5 If you examine the siege site below you will see tons of red armies just circling about the siege site. Ditto
 |
Replies:
Posted By: Rill
Date Posted: 21 Feb 2014 at 17:01
|
dittobite has created Illyriad's first battlefield singularity! I was monitoring the square when it occurred, shortly after 4:37 server time on Feb 21. What appeared to happen is that two red (to me) armies hit virtually simultaneously. No battle reports were generated as a result of those armies and they seemed to just "hang" over the square. No further reports of battles have been generated, so far as I know.
|
Posted By: Myr
Date Posted: 21 Feb 2014 at 18:12
|
I don't think that's the same issue I am experiencing. On my screen that city is green and the stuck armies are as well. On the bright side, at least my troops are still alive!  |
Posted By: Aurordan
Date Posted: 21 Feb 2014 at 18:22
| Have you seen that square? I wouldn't attack it either. |
Posted By: Eddy
Date Posted: 21 Feb 2014 at 21:29
|
/me begins to wonder whether the devs lack a "once you start drinking on a Friday, you should stop pushing changes to the revision control system ..." policy. ------------- Eddy. |
Posted By: Mr Damage
Date Posted: 21 Feb 2014 at 22:33
| These are the reasons why people will leave the game. C'mon Devs sort it out!!!!! |
Posted By: Tatharion
Date Posted: 21 Feb 2014 at 23:35
|
I hope the Devs will consider resolving this malfunction a top priority. I find it odd it hasn't been dealt with yet. ------------- Be without fear in the face of your enemies. Be brave and upright, that God may love thee. Speak the truth always, even if it leads to your death. Safeguard the helpless and do no wrong. |
Posted By: Deranzin
Date Posted: 21 Feb 2014 at 23:37
An obvious problem does not necessarily have an obvious solution ...  |
Tatharion wrote:Posted By: Daefis
Date Posted: 22 Feb 2014 at 00:09
|
Wow... Ignoring minor issues is one thing but this is something else. Not picking sides but if this was my army in limbo i'd be pissed....
------------- http://elgea.illyriad.co.uk/a/p/37796" rel="nofollow"> 
|
Posted By: The Electrocutioner
Date Posted: 22 Feb 2014 at 00:21
|
Can we at least get an acknowledgement that the problem is being investigated with high priority? Any chance we had to further defend my city is being denied to us by this bug, while my city's population is dropping every hour and we have other cities with incoming sieges to defend against. This situation is grossly intolerable. Everyone on both sides has invested a lot of time and energy to arrive at a battle of this magnitude. We all deserve to have a fair fight. Thanks dittobite for starting this thread.
|
Posted By: GM Luna
Date Posted: 22 Feb 2014 at 01:53
|
Could I get the petition numbers please? If there are multiples I'd like to know which are linked. Thank you. Luna ------------- GM Luna | Illyriad Community Manager | community@illyriad.co.uk 
|
Posted By: asasama
Date Posted: 22 Feb 2014 at 02:02
| petition 9684 |
Posted By: Rill
Date Posted: 22 Feb 2014 at 02:11
| You mentioned in chat that player names would help. I believe Wild Monk filed a petition. |
Posted By: Cactus
Date Posted: 22 Feb 2014 at 02:27
| I am having the same problem. One of my armies has been tied up since the first week of February and they keep ignoring my pleas to fix the glitch. They still have not assigned anyone to it and I have posted several additional requests to fix it. No reply form them at all. It seems they want our money but dont want to fix major problems when they happen. Hmmmm. Doesn't make sense. You would think they would want to keep their patrons happy! |
Posted By: Myr
Date Posted: 22 Feb 2014 at 05:12
| Wild Monk's petition # is Petition 9683. I have also lost troops from at least one stuck army and no, I was not out of gold. |
Posted By: Chaos Armor
Date Posted: 22 Feb 2014 at 06:25
|
Players should realize that the developers have to read through thousands of lines of code to FIND the problem. Then attempt to figure out what has gone wonky. It could take a while. Patience young grasshoppers.
|
Posted By: Deranzin
Date Posted: 22 Feb 2014 at 08:31
I didn't file a petition, but it seems that the armies got unstuck and the commanders are back ... the armies though disappeared and they did NOT hit the siege square since the commander only gained 40 XP.
|
|||||||||||||||
Posted By: Rill
Date Posted: 22 Feb 2014 at 08:45
| the reports on the defending side show no troops and no commanders. There are also a lot fewer of them than might be expected if the system had actually processed the hits. |
Posted By: Myr
Date Posted: 22 Feb 2014 at 20:06
|
Thank you for releasing what is left of my troops. Sadly, I lost a lot of troops that aren't included in that battle report I got when they were finally set free. Some of my troops seem to have vaporized.
|
Posted By: The Electrocutioner
Date Posted: 23 Feb 2014 at 21:20
|
So armies were freed, but tens of thousands of vanished troops were not replenished. I never expected you to roll back the clock, but we certainly deserve to have those troops back in our armies for the next onslaught that is currently underway. We feel cheated, but vCrow and their allies are equally cheated. They deserve a clean, untainted victory, just as we deserve a fair opportunity to defend ourselves. And most importantly, we've received no explanation, no indication that the issue is fixed and will not happen again. Are we to carry on like nothing happened and just expect better results next time? The wheels came off the game this weekend. Even peaceful harvesting players had their game sabotaged by a bug that was just as serious for them as this one is for us. We deserve an explanation of what the problem is and what is being done to prevent it in the future. Confidence in the playability of this game is at an all-time low. I question whether it's worth playing a game that is going to break on me at a critical moment, and I know I'm not alone.
|
Posted By: Corwin
Date Posted: 23 Feb 2014 at 23:42
I agree with every word. If no explanation comes about these issues I think I'm gonna leave the game. There's no fun in a game when rules suddenly change without warning and without explanation.
|
Posted By: Tatharion
Date Posted: 24 Feb 2014 at 00:09
|
Devs, a little effort on global communication when such critical bugs occur in times of war couldn't hurt going forward. ------------- Be without fear in the face of your enemies. Be brave and upright, that God may love thee. Speak the truth always, even if it leads to your death. Safeguard the helpless and do no wrong. |
Posted By: GM Luna
Date Posted: 24 Feb 2014 at 00:33
|
Still investigating the issue. We will update when able. Sorry for the delay. GM Luna ------------- GM Luna | Illyriad Community Manager | community@illyriad.co.uk
|
Posted By: Tatharion
Date Posted: 24 Feb 2014 at 01:39
|
Devs, I am sorry in advance if this message will sound a bit too confrontational. But as much as I hate to admit it, your last one-liner answer is coming in way too short of what we expect from you. Best, Tatharion Consul Invictus ------------- Be without fear in the face of your enemies. Be brave and upright, that God may love thee. Speak the truth always, even if it leads to your death. Safeguard the helpless and do no wrong. |
Posted By: Elmindra
Date Posted: 24 Feb 2014 at 02:13
|
I agree 100% The herb regen issue happens constantly, and all we ever get is a "it's getting looked into" There are still multiple outstanding petitions with missing and lost troops, some years old with no answers. Now there is a massive glitch involving hundreds of thousands of troops missing, and all we get it a "please let me know the petition numbers" followed by a "still looking into it" message a few days later? I do believe I have spent my last dime on Illyriad, over a year of complete silence and little to no support does not warrant any more money wasted in such a way. ------------- 
|
Posted By: Le Roux
Date Posted: 24 Feb 2014 at 06:12
|
Seems very akin to the episode I had with a wonky siege that left 700 pages of null attacks in both my and my siege targets mail boxes, in the end the only thing that broke the infinite loop was a secondary siege that razed the town in question. In the end the only result was the Dev's removing the 700 pages of empty attack spam, never an explanation or indication something was fixed.... ------------- 
|
Posted By: GM Stormcrow
Date Posted: 24 Feb 2014 at 14:13
|
Hi everyone, I've repatriated 14 armies that appear to have been stuck in limbo. I believe this is all of the stuck armies, but if you still have an army stuck that's not now back at home, please make sure we know about it by opening a new petition (so that it doesn't get lost in the petition queue), please feel free to reference your earlier petition if you have one! And please, please make sure the petition clearly contains both the name of the army and the town it belongs to. This bug is extremely weird. And yes, it was also the root cause of the herb respawn issue. If it seems extremely strange that an as-yet-unseen error in the military outcome algorithms could affect the rate at which herbs respawn (or don't) then yes - that's because it is very peculiar indeed. But the two were linked, and we don't yet know why. There's currently a piece of sticky tape and a bit of chewing gum holding a very temporary Macgyver-esque fix together on top of two systems that should be entirely unrelated to one another, but clearly aren't. We have a complete copy of the full server including all the past transaction logs and we're stepping through a complete replay of each event that we're aware of - to find out what happened, step-by-step and millisecond by millisecond. This is going to take some time. Once we've found the root cause, we'll work out our next steps to a more permanent fix, and we'll keep you informed. Thank you for your patience, and please accept our apologies. SC
|
Posted By: Corwin
Date Posted: 24 Feb 2014 at 16:03
| thanks for the response Stormcrow. It's not the bug itself that bothered me (heard about it in time to keep my harvesters away from the herbs and didn't lose any troops), it was the lack of a satisfying reaction. Never thought of the possibillty that the bugs could be linked. |
Posted By: Rill
Date Posted: 24 Feb 2014 at 16:48
|
Thanks for the response Stormcrow. Color me amazed that the bugs were linked. Anyone ever get the feeling that Illy is bigger than all of us ... even the almighty devs? Glad folks have their armies back so we can fight fair and square again.
|
Posted By: Tatharion
Date Posted: 24 Feb 2014 at 17:57
|
Thank you for your prompt answer GM Stormcrow. Military impacting/impacted by herbs non re-spawning ???? It feels like your algorithm is alive and seriously kicking. Best of luck in deciphering that odd bug. Best, Tath
------------- Be without fear in the face of your enemies. Be brave and upright, that God may love thee. Speak the truth always, even if it leads to your death. Safeguard the helpless and do no wrong. |
Posted By: GM Stormcrow
Date Posted: 24 Feb 2014 at 18:02
No, us neither! XD
The possibility has dawned on me that we're currently witnessing the system's first faltering steps towards achieving sentience.
|
Posted By: Praetor Nistiner
Date Posted: 24 Feb 2014 at 18:52
 ------------- Tutela 111-Sir Bradly is surprisingly nice. SSH :D |
Posted By: The Electrocutioner
Date Posted: 24 Feb 2014 at 21:04
|
Thank you GM Stormcrow for the update. Best of luck in your troubleshooting, and we will eagerly await good news. Cheers,
ELECTROK |
Posted By: Myr
Date Posted: 24 Feb 2014 at 21:07
Not quite yet Rill, he released what was left of my troops there. However the number of my troops were falling while they were stuck. I got no kills but I lost a majority of my troops so I'm still not happy. We have enough enemies without losing troops to bugs. |
Posted By: Rill
Date Posted: 24 Feb 2014 at 21:12
|
I assume you've filed a new petition regarding the number of troops you thought should be in your army? We did throw fliers out of the siege camp offering amnesty for any troops that wanted to make individual peace, so that could be another explanation.
|
Posted By: Myr
Date Posted: 25 Feb 2014 at 01:17
| Fliers! I'm going to try that tactic next war. |
Posted By: HATHALDIR
Date Posted: 25 Feb 2014 at 02:37
|
The fliers have also solved the Great Toilet Paper Shortage at various latrines involvedin the siege. An army may march on its stomach, but it definately sits on its butt! ------------- There's worse blokes than me!! |
Posted By: Rill
Date Posted: 25 Feb 2014 at 02:48
| Indeed, vCrow is responsible for all sorts of advances in siege technology. |
Posted By: Miklabjarnir
Date Posted: 25 Feb 2014 at 17:04
That kind of thing happens in complicated systems that need to respond to lots of external input. Sometimes a thread is stuck. It may be a race condition or a deadlock because of a missed event in another thread. It is a true hell to debug these things.
|
Posted By: Rill
Date Posted: 25 Feb 2014 at 17:25
| I watched the bug develop and one thing I noticed is that 2-3 armies impacted on the square at the same time. I imagine that there is coding that is supposed to deal with this, but maybe it is not quite robust enough? Or doesn't have enough "space" in the code for the number of 0s of the troops involved? |
Posted By: GM Stormcrow
Date Posted: 26 Feb 2014 at 10:33
^^ qft, and yes - exactly one (or more) of those things. And producing a replication (outside of a live environment under stress) isn't happening as yet. But we have, at least, found the precise moment that a thread deadlock/collision occurred whilst one or more threads were under pressure. However, stepping through without dumps in a hyperthreaded dual hexacore (24-core total) environment isn't a walk in the park :) Still working on it - will keep you posted. SC
|
Posted By: Le Roux
Date Posted: 04 Mar 2014 at 19:10
|
. . . and ? -------------
|
Posted By: darpansah
Date Posted: 04 Mar 2014 at 19:17
|
Made some attcks on red cities report came in saying mission aborted cant attack friendly city...lol....is this a glitch or ?????? |
Posted By: Rill
Date Posted: 04 Mar 2014 at 19:24
| darpansah, there might be reinforcements in the city from an alliance with which you are NAP'd or confed. Have your alliance leader check your diplomacy. |
Posted By: GM Stormcrow
Date Posted: 04 Mar 2014 at 23:03
EDIT: ty Le Roux for nudging me to give an answer. It's still not a complete answer, but I don't communicate enough atm.
Step through is complete, and it appears to have been caused by a broad combination of multiple things happening at the same time:
All of these things put pressure on the database. However, the database is built so that it should work through these things linearly (and it's been under a lot more pressure than this in the past). However, basically, a deadlock that should have been broadly impossible (before I get jumped on, I do understand there's no such word in code) in a nolock read /rowlock write environment produced a NULL in a (db-constrained) NOT NULL column. And other not-directly related things had fits thereafter, based on reading a single NULL data entry that really shouldn't have been there if there was fair and just code deity. This behaviour was unexpected enough for us to have raised a ticket with our db vendors - with whom we have a fairly close relationship - and we've been rapidly through the "are you patched up qs" and it's been escalated into realms beyond our understanding, but will hopefully descend again in a SQL update patch in the near future. We'll keep on applying pressure for a satisfactory resolution, but there is only so much we can do. Somehow we manage to find these issues that have everyone shaking their heads. On a different issue recently we found a (reproducible) bug in .NET that has entailed us actually running our very own, unique version of the core .NET framework to enable us to continue development. I do sometimes wonder if we're not a bit too far on the "extreme" side, but I gather a "fix for all" will make it into the next update. On a more practical level, I can't say it can't and won't happen again, but the areas that exhibited obvious symptoms (such as herb respawning and some armies getting stuck) have been instructed how to bypass such an event so they now understand how to handle NULLs, even in columns that are apparently constrained to be un-NULLable. We've also written some monitoring procedures to actively look for brief glimpses of these creatures that live at the bottom of the Mandelbrot set, and to alert us immediately should they happen in the future. I can't guarantee we've got them all or are looking in all the right places, but it's a stab at it anyway. In terms of a longer term fix, beyond the hopefully forthcoming vendor patch, there are various options we're exploring in terms of re-architecting various core components. Many thanks for your patience, and I apologise if it got a bit technical in some areas above - I know some of you are techies, and many aren't, so I tried to balance decent answers with explanatory ones. SC
|
Posted By: Angrim
Date Posted: 04 Mar 2014 at 23:12
|
so what i'm reading is: sieging ELECTROK broke the database. getting a NULL into a NOT NULL column sounds medal-worthy. |
Posted By: Rill
Date Posted: 04 Mar 2014 at 23:13
|
Stormcrow, have you ever considered repositioning Illyriad Ltd. as a bug-finding enterprise? Where you create value by testing the limits of software and hardware architecture and identifying bugs for organizations like Google? You could put us players on salaries and pay us to break stuff!
|
Posted By: Merlinus
Date Posted: 04 Mar 2014 at 23:39
|
I used to write qa code, but I'm glad I'm not writing THIS code. :) Just a question: Has anybody looked into the .round function--32 v. 64 bit for time lag crashes? Just wondering. ------------- In Peace we reign. In War we RULE! Long live the Royal House of Merlinus! |
Posted By: Miklabjarnir
Date Posted: 05 Mar 2014 at 01:03
Rather not, Rill. I have been a professional breaker of code (AKA QA manager) long enough, and am currently retired  |
Posted By: GM Stormcrow
Date Posted: 05 Mar 2014 at 19:37
We think so! Not sure our database vendor agrees :)
One of the most perfect technical descriptions of the current illyriad corp I've ever read 
tbh Merlinus, that question is probably above my pay grade. I'll ask TC. Best, SC
|
Posted By: Albatross
Date Posted: 06 Mar 2014 at 10:48
|
Given that you use transactional processing, is it possible to extend the fail condition to also include a NULL test, either in the db engine or in the db-calling block? It's nice that you're getting the support you are from software providers.
I feel very different when coding in environments that I suspect do not work as specified. All kinds of doubts creep in, and coding (or making changes) unnecessarily defensively is difficult. -------------  |
Posted By: GM Stormcrow
Date Posted: 07 Mar 2014 at 10:54
Hi albatross, Yup, that's basically what we've put in the Stored Procs that call in, those that were definitely affected by this single instance. As mentioned, I do broadly regard this as temporary sticky-plaster and bubblegum. But as you rightly say, the concern is that if it's happened here and we don't yet know why... then it may also happen elsewhere in the future. And if I have to write specific isnull handlers for every select and where statement in the codebase that looks at a NOT NULLable column, I can probably kiss goodbye to the next X months  ; which is why it's important the vendor is involved!SC
|
Posted By: scaramouche
Date Posted: 07 Mar 2014 at 18:11
hmm...what with all these glitches and bugs SC...do I get a rebate on my illy membership payments? ------------- NO..I dont do the Fandango! |
Posted By: Myr
Date Posted: 20 Mar 2014 at 22:45
Hey Stormcrow, I still have missing troops that haven't been returned from this whole fiasco on my alt. Since we're kind of in a heated war still it would be nice to get them back some day.
|
|||||||||||||||
Posted By: Myr
Date Posted: 26 Mar 2014 at 22:44
Still unassigned.  |
Posted By: Myr
Date Posted: 11 Apr 2014 at 00:42
| Thanks for letting me know you're looking into it SC. This is just a reminder that they're still missing. It would be nice to have them home before all our cities are dust. Thanks! |
Posted By: Ander
Date Posted: 04 May 2014 at 12:46
|
Armies in limbo on the tournament square, gaining tournament time. (Petition number 9910) Happened after this particular attack where there was no casualties despite a failed defence. From: System Subject: FW: Failed defense by Mariabronne's forces at [400|-504] in Turalia under attack by h2Oxy's forces from Lovely Received: 04 May 2014 10:55 Original Message: Two opposing forces clash against each other. Cavalry comes into its own when able to strike hostile forces at will, and from unexpected directions - and nowhere is this more feasible than on open plains. Lightly armoured spear units, however, prefer terrain where there's some cover available. Fighting defensively on open plains, cavalry draws strength from the ability to form and reform their lines of engagement depending on the direction of battle, and it is here where cavalry excels.
|
|||||||||||||||||||||||||||||||||||
Posted By: Ander
Date Posted: 04 May 2014 at 14:02
|
All the limbo armies were destroyed in the last attack. Your army was driven back and enemy units have taken control of this location. Your defending army was entirely destroyed at [400|-504] in Turalia by h2Oxy's forces from AquaFire
|
||||||||||||||||||||||||||||||||||||||||