It's an Undertale AU if it's essentially the same game with altered characters. Examples include UnderSwap, UnderFell, and other AUs I can't name. A fangame would be something like Deltatraveler. A game that's takes inspiration from another game, but is still its own standalone game isn't a fangame or AU