More then that, if there's no such thing as borders then people can move freely wherever they want. That means nobody can complain about the European settlers coming to the United States and colonizing it. In fact, you couldn't complain about colonization anywhere.