those people turned against Trump during and after his tenure as their supervisor. they came to that decision by direct experience under his management. and their voices do matter because a lot of people look up to the joint chiefs, and to Pence, I'm sure. whenever i would go about selecting a new workplace in my younger days, i would first talk to people already working there to get their impressions of the management.
the opinions of these people might not matter to you or your friends, but they matter to some people. one of Trump's biggest mistakes, in my opinion, is his reliable willingness to piss people of, and to cast associates aside if they don't toe the line. every time he does that, he loses some votes, and they all add up.
it looks like the election could be closer this time. if Trump loses by a small number of votes in a few swing states, it will be largely his own fault.