But how bad does it have to be to be considered abuse

Idk
Enough that it has long term effects I think?
Im not an expert in this and emotional abuse can be harder to identify than other types of abuse however if you suspect your being emotionally abused id recommend doing research.
Also , to my understanding, at any point if you feel unsafe around them you are allowed to call the cops even if it isnt necessarily abuse.