The singer said he would do ‘long-lasting damage’ if the gigs went ahead