This is entirely supposition on my part, but maybe it's the will of the banisher that somehow puts roadblocks in the daemon's ability to reappear, maybe they can't actually be summoned back to that area as long as the banisher remains (i.e., their soul is teethered to a life anchor) that acts as a stopgap in addition to any other restrictions (such as realspace not being 'thin' etc, etc)