What are your thoughts that gravity would enter the equation when time traveling? And it could be gravity from the Earth, sun, or even the center of the galaxy. Maybe it would act as an anchor. There is no data to suggest that the effects of gravity would change while traveling through time, or even if gravity would work differently if our entropy were reversed like in Tenet.
For all we know, gravity could hold you where you are whether you are traveling forward at 1 sec per 1 sec (normal speed), or -24 hours per sec.