Canadian visitors, irked by talk of "annexation," are staying away from the city and United States altogether.