Well, not necessarily, as changes to the past resulting from time travel can create loops, alter memory of both present and future people affected, or even generate alternate timelines. Thus, travelers often must avoid directly affecting events that impact their own existence. So, in other words, we can't know exactly if they've visited us, as it's impossible for us to find out without creating an inconsistency in the fabric of space-time continuum.
Caption this Meme 
