The books of Judges and Ruth and things associated with them. Yep, angels are meant to be part of this. And Rahab. You'll see.
Aside from their literal purpose (a safe haven for those wanted for manslaughter), they also have a symbolic meaning as a bellwether.