He didn't say "the BLACK inmates running the prison"

If some people take what he said as racist, then guess what...they're the racists. Is "inmate" dogwhistle code word for African-American? Are all inmates people of color? Only people of color go to prison and no white people ever do? I don't get it, it's a legitimate phrase.