Residents have said they do not feel safe and are starting to avoid the city centre due to the rise in visits from asylum seekers.