Jimmy Carter's domestic policies were VERY popular. What got him was the Iran hostage crisis, which we know today - proven - that the Reagan campaign cut a deal with them saying that they'd get a better deal if they kept the hostages until after the election. Which is just plain treason.
Other than that, people remember Carter fondly. You don't, but it's not difficult to get on your bad side.