One way to know is that comments are sorted by upvote count first. So if you have 5 upvotes on a comment, but a comment *above* yours has 4 or less upvotes showing, you know you've been downvoted.
I've added a screenshot above of a few comments lower down as an example.
This also applies to nested comments, but only within their own nests and at the same reply stage.
Another way to know you've received a certain number of downvotes is obviously when your comment gets collapsed to hidden.