You don't have to be attracted to everyone to not be racist. As long as you accept everyone and think that every person should have equal rights, you're fine. Would you call a gay man sexist because he just isn't attracted to women?
Also, I used to think the same. I just had never seen a dark skinned person I was attracted to and I thought the same way about it as you are now. But I just hadn't met a lot of people with dark skin overall so my sample size was pretty small. Turns out I have since met or seen people with really dark skin that I found very attractive.
And last but not least, the fact that you're asking yourself this question is a good sign already. It's not enough to not be racist these days. We have to actively be against racism. And part of that is questioning everything we do to find out if it could be seen as racist. By thinking about it you've already done more than most people would. And from what you've written I think you're fine. 😉
Caption this Meme