The first part of what you wrote was true.
The sun and moon were created on the fourth day. And yes, they were given as signs and seasons. Since part of the "signs and seasons" description presumably means helping mark the time, I can see why you would say that the concept of days was created after 4 days had passed.
But there's 2 reasons why it seems reasonable to me to assume that concept was in place before the fourth day: firstly, the language of Genesis chapter 1 repeatedly uses the phrase "and there was evening and there was morning, the first day, the second day, the third day" etc, before the fourth day happens. If evening and morning are occurring, the passing of a day is being measured.
Second, just the fact that they're calling it a "fourth" day means that the concept was there in order to name it that. If there had been no concept of day before the fourth day, there would be no meaning to calling them day one, two, and three.