Huh, after a bit of digging it turns out the color was named after the fruit, so in an etymologic sense the fruit came first.

As a wavelength of the electromagnetic spectrum, the color we call orange has existed for at least as long as there have been sensory structures capable of detecting it. I'd wager that goes back considerably farther than the crossbreeding of citrus.