PM refusing to budge on India travel pause