I think that there should be one small change to the rules and here it is:
a fast cavalry unit or a skirmisher unit may declare "flee!" as a charge reaction WITHOUT CAUSING PANIC in your fellow troops, AND they can auto rally the next turn.

WHY NOT? they are skirmishing pester units, they screen the army and when the enemy gets close they get out of range! thats like the whole point of skirmishers in my opinion but the thought that you have to risk running off the board or causing panic down the whole line just to do it seems utterly lame.

With this change, skirmishers could do the job that they were intended to do instead of being pathetic chaff.