Good thread. My thoughts, which I've tried to keep as short as possible but lol overspill:

I don't think I'd be far wrong in saying that the 40k background, art, model design, troop choices, and all the rest of it, actually tell us two contradictory things at once about our individual game-stories (I think calling them game-stories is a useful descriptor) that we play out on the table:

1) The game on your tabletop is a mighty clash of arms in which the souls of millions hang in the balance. (On one side we have a great Space Marine Captain, on the other a Greater Daemon of Khorne, etc. We see all the different wings of a combined forces army present in the same place - land, air, infantry, armor, big guns, etc. Ancient artefacts and relics of awesome power are trotted out onto the battlefield!) And yet also,

2) The game on your tabletop is a drop in the ocean against a huge background, a more or less infinite universe where millions of other battles churn and churn. ('In the grim darkness of the far future there is only war', the phrase 'The universe is a big place' turning up everywhere, and so on.)

Meanwhile, if we look strictly at the rules and number of actual models on the table, they seem to be telling us yet a third thing:

3) In the absence of absolute information about scale, or about the ratio of models to real soldiers, and given that the rules treat the figures as if that ratio were 1:1, we are looking usually at a battle where the army of either side is what, six or seven squads, a couple of vehicles? In real-life military terms (I know I know but work with me here) that's just a couple of particularly large patrols encountering one another.

Hold that thought, I'll come back to it.

Now going back to the first two things, that contradiction between the Grim-dark Importance of your game-story that you're playing out, and yet the Irrelevance of it in the face of the Grim-dark; that contradiction makes me see why people could call it 'stagnant' or ask for the story-line to move forward in some appreciable way. It's a concession I don't want to make, because I agree with OP and the blog that was quoted upthread, but it's there for all to see.

The significance of all the insane characters and huge vehicles (Knights!) and all the rest of it is conceptually undermined, by the series of games that get played out every weekend round at Bob's house or whatever; and by the what, ten, twenty real-life years of more or less stalemate between the factions.

Now, consider a game-story played during the Rogue Trader period, where - now harking back to my comment about a couple of patrols encountering one another - two or three squads on either side kick off. If a Land Raider turns up it's a very big deal. There will be a couple of heavy or special weapons knocking around, that also constitute a big event. A character of some sort - less a great hero, more just an officer or stand-out warrior - will be involved, and sure, the weapons will all be incredibly nasty, but it's very much a story about who gets to claim a particular asteroid. Or vague 'hill' made out of polystyrene. It less obviously contradicts the idea of a huge dystopian stalemate in the background.

Or if my mention of Rogue Trader sounds a bit rose-tinted, consider a game of Necromunda or Inquisitor or one of the RPGs. In such a game-story I'm worried about my shotgun running out of ammo; whether I can get through a bulkhead; the overall fate of the entire Hive City can be left stagnating awesomely in the background.

So the gradual over-powering of 40k through 2nd and onwards eventually gets us to a point where, though I strongly agree with OP's concerns about meta-narrative, contradictions start to become overwhelmingly evident to anyone involved with 40k beyond a superficial level. (But be aware I haven't yet said whether I approve of what GW are actually doing! Read on ...)

As well as the trend for over-powering everything, we've also seen a trend away from a loose to a tight background and a trend toward specificity in miniatures. At the loose end you've got Rogue Trader, early WHFB, things like ye olde Realms of Chaos books; you've got a load of miniatures which, if you look at how early RT minis were sold, were never that closely intertwined with the background story as sketched out (in happily incoherent suggestions) in the books. It was often two different teams covering each job in those days. All very different by the time we get to the 90s and mid-2000s.

This is clearly what led to the demise of the Old World setting! I'm not saying this was a good thing. But like, 80s-era Eavy Metal photos of WFB battles in progress are really evocative and full of variety; compare them to the 2000s or even 90s equivalent and the images are much more routine. Yes, even though the quality of the figures has gone up by at least 100%.

So to re-iterate there's that well-known GW trend to over-power everything over the years, and another trend to tighten up and so limit the fluff and model ranges. All of which leads to a situation where, conceptually at least, the stagnation is evident.

But having acknowledged all this on the conceptual level, this is where I now turn around and say I dislike the WFB End Times thing, especially the AoS thing, and also now this whatever they're calling it project with 40k.

Very simply, it's one thing to advance the plot of some novels/TV show/film series. I may not like the new direction, but don't mind going to watch the latest Alien or Blade Runner movie, even if it's bad, because that conceptual shuffling and shaking up they've done hasn't cost me anything more than the price of a DVD or cinema ticket. The shuffling and shaking is being done with fictional characters that have no more bearing on my life than I want.

Whereas with a wargame setting – ah, it's very different. People have bought lots and lots of expensive figures, with which primarily to take part in a game with others, and the conceptual shuffling and shaking impacts on the usability of those figures.

Whether these people are old guards or new to it, the principle that you could invest a lump of money into some figures for a game, and then the company changes the game to such an extent that those figures are no longer usable, is ... well, if I say it's 'wrong', people will come back at me by talking about the free hand of the market. So perhaps 'unprofessional' would be a better term, or 'ignorant of the point of their product as their consumers see it'.

N.B. I know part of the use-value of said items is to paint or convert or look at, not play with. I know you can use unsupported figures as proxies or counts-as. I know you can look the other way at now-illegal weapons. You can play old editions. You can house-rule. You can write your own army list! You can play entirely other game systems. I know that even if a faction is killed off in game background, and the next rules edition trundles round, and they no longer exist legally, like Squats, you can probably use some imagination to still play with those figures you bought.

Sure, it isn't 100% absolute, but then let's not fall into the fallacy of grey: just because you can find a way to use your 80s Squats - for which read, any other physical model no longer supported - in the current 40k hobby, doesn't mean you wouldn't be doing so at a huge disadvantage (no new figures, no tournaments, loads of work coming up with rules, loads of work in getting other players to agree with your rules patch, etc).

Obviously people who like Squats rarely bother to complain seriously about them not being supported in current 40k. And yeah, it's been a very long time since anyone bought an Imperial Guard Landspeeder or Space Marine Jetbike new from GW.

But the general principle remains: advancing the plot is understandable - arguably necessary - for a host of reasons, but overall bad if done in a way that impacts in real life, on people who want to play a game with some expensive figures they've bought, and may in potentia find difficult if not impossible to use.
Obviously if we follow the law of caveat emptor, we have an object lesson in why not to assume anything about the future use-value of a product you purchase which is part of an implicit system. I would append that to invoke the law of caveat emptor as the last word to everything is not too far removed from the tendency to building a beardy list so as to win every game with unpainted figures ...