Terrain modifier adjustment seems pretty reasonable. Yes, on plains cavalry should be 2-3 times more effective than other troop types. However, given that their base attack is 60-65 and infantry cav defense is only 17-20, odds are that a cavalry attack on infantry entrenched in a forest is still going to inflict 2:1 losses and that is preposterous.
Ideally, each troop type should be specialized for terrain. Let cavalry be absolutely dominating on plains and small hills for example, but conversely almost useless in mountains, forests and buildings. Let Infantry be the master of large hills and forests but helpless in open ground or mountains. Let archers rule the mountains and buildings, but weak in plains and forests. Let spearmen be the default, reasonably good everywhere but always weaker than a specialist troop on its preferred ground. Every unit type should be significantly stronger than the others in choice terrain and significantly weaker in unfavorable terrain. As it stands now the terrain modifiers are worth taking into account (you might need to send an extra 15-25% troops) but some units are so strong as to be the default choice and some so weak as to be pointless.
Cavalry has always been the primary culprit here. Yes, cavalry has always been the king of the battlefield -- field being the operative word. On uneven ground they never have been terribly effective. The Saxons on Senlac Hill at Hastings turned back countless Norman cavalry charges, and we all know how archers at Agincourt obliterated the French cavalry who charged them across a sea of mud. The whole point of the pike squares ubiquitous in 17th century warfare was their immunity to cavalry charge. You don't see much cavalry action in the mountains, and I have never heard of a cavalry army successfully storming a walled city either... horses are lousy climbers.
Cities should also be their own terrain, preferably something like buildings. Do the defenders all just line up on main street and wait for cavalry to ride them down after opening the gates for them? A cavalry attack on a walled plains city defended by T2 spearmen is still going to give a better than 1:1 kill ratio and that is totally absurd. Even the Huns and Mongols had to dismount to fight inside a city. Historically the only effective way to capture a city was to force a surrender through starvation or to break down the walls with siege engines and then storm with infantry in a house-to-house bloodbath. In Illyriad it basically boils down to whoever has the most cavalry wins.
If you are human there is almost no reason to build anything but cavalry: you get a 10% bonus to cav's already bloated attack numbers and you produce them more cheaply than any other race. Even dwarves, with the worst cavalry and best infantry in the game, are better off building cavalry as often as not. Unless an opponent is considerate enough to build his siege camp in a forest and protect it with pikemen, I'm better off siccing the mules on them. For defense almost everyone is better off building archers because they have good all around scores and you can attack with them as well, particularly if you are an elf. Spearmen are cheaper, but they are lousy on the attack and only good against cavalry, which is so overpowered that it will still wipe out your spear army easily.
TLDR:
As I see it, the heart of the problem is that military effectiveness of many troop types is grossly unrealistic, as is having a walled city carry no special terrain modifier.