Gender regards sociology and psychology. Sex regards biology. Learn your science.
Gender terms = man, woman, non-binary, etc.
Sex terms = male, female, intersex, etc.
Using them interchangeably is only evidence that you yourself don't understand the science. Sex is broken into three distinct groups: Male, female, and intersex (intersex being the group for anyone who does not align with the chromosomal or physical traits we typically assign to one of the other groups).
The male group isn't "static" either - it's a spectrum. For instance, not all males have the same physiology. Some have bigger and smaller genital areas. Some have more or less hair. Some have more or less testosterone, etc. The same is true for the female group and the intersex group.
In sociology and psychology, the terms man, woman, nonbinary, fluid, etc - they all regard identity of self and how that identity relates to roles within one's personal life and that of society.
These ideas are correlated but they are NOT interchangeable.
The problem you're having is that your common sense is fueled by ignorance.