So, I have been wondering lately, what was going on in Mordor between Sauron's defeat by Isildur and his return prior to the War of the Ring?

I know that for at least a few centuries prior to the Hobbit he had sent the Witch King to start rebuilding Barad Dur and begin gathering new armies, but this a a 3000 year time period. What was going on in the black land while Sauron was basically an impotent shadow?

Did Tolkien ever say anything on the subject?

What do you guys think?