Like the title says, what's the thing about Fantasy's background that annoys you the most?
For me the big problem is how insanely dangerous the world is! If we go by a standard battlefield, nearly every tree, river, pond, hill, wall, building, etc WILL KILL YOU!
I find it difficult to take this seriously. How are you going to feed a city like Altdorf when cabbages come to life overnight or every third calf turns out to be a warped beast? If the Empire is a series of "islands" amongst endless malevolent forests inhabited by bloodthirsty creatures and spiders the size of a house, how do you maintain trade and contact between cities? How is an economy maintained if there's a major invasion/civil war/marauding monster ravaging any given part of the nation every ten years or so?
Maybe this is part of the reason why Fantasy's background isn't as popular as 40K - sure, they're both as OTT as each other, but 40K's variety and scope saves it from the ultimate silliness that is the Old World.