The former prince was evicted following revelations about his friendship with Jeffrey Epstein