Taking a person's property without their consent is immoral, and being the government doesn't change that.
Now, the government restricting property-owner's rights of association and exclusion means that both open and closed border policies are fundamentally immoral, making a debate between the two a debate on practicality and lesser evils.
That depends on whether or not the way you build it is moral. If you build it on stolen land (eminent domain) or using stolen money (taxes), then you have an issue.