It's not racially motivated but the term could be taken as an insult. Brownies are tiny humanoids like leprechauns and faries. It can be seen in the movie Willow. The brownies are the people who could fit inside normal people's pockets. But it is meant to be an endearing term.