Do you like conspiracy theories?

I'm having problems with a neighbouring player about harvesting rights.

It has happened a couple of times that my harvesters were killed while gathering close to one of his cities.

I understand that some players claim an exclusive right on  whatever resources they see around their cities (see this topic, for example: http://forum.illyriad.co.uk/harvesting-hides-and-animal-parts_topic5109.html?KW=gathering+resources). I do not agree with this view, but I am not here to dispute it now.

Since that, I have not been attacked by him again but here's where the conspiracy theory starts: I have just been attacked by a different player that recently appeared right next to my town.

He killed a bunch of my harvesters and then attacked my town, only this time he did it openly, using a large army.

This village is still very small (pop. 124) so I think it's impossible he could have raised an army  of ~800 units without receiving supplies from somewhere else.

The MO is more or less the same, so I am almost sure this is the author of the previous attacks who is now using an alt account just to keep harassing me.

Now that you know the facts, I would like to ask you a few questions:

First of all, as anyone else been harassed with similar methods? Did you identify the attacker?

And then, how should I handle the situation? Retaliate with force and risk an escalation of the conflict? Use the same sneaky tactics and just kill his harvesters whenever they cross my path?

Please note I would prefer not to join an alliance at this time because I think receiving too much help would somewhat defeat the purpose of the game and spoil the fun.

Cheers

Edit:

@ Rill: sorry, I guess I had to clarify that this happened to my alt account. I do not want to drag my main account or my alliance into this dispute

Re-edit:

@ Rill: I do indeed suspect that my "nemesis" tenariled next to me (he has no other city). That's the whole point of the conspiracy theory ;-)

@ Darkwords: I don't want to occupy the square where I was attacked as it is often harvested by other players. I don't want to kill their caravans or deny anyone access to the resource. That would be against my "liberal" views on resources gathering.

@ Epidemic: I can easily attack the small village (and maybe I will) but the other player is definitely out of my league. Even if his alliance did not back him up, we're talking three burgeoning cities against my small town and large village.