I looked up the historical ranking of best president ( https://www.c-span.org/presidentsurvey2017/?page=overall ) (all judges were historians. Trump is not on this list because he's still president. and this list only contains FORMER presidents).
I guess the scores for each department are 0 minimal and 100 max: https://www.c-span.org/presidentsurvey2017/?personid=2484 and when I look at Carter I see he scored most of all high on equal justice for all (Trump is loved for doing bad here... so does that explain something?) and the rest of the scores were pretty average...
Of course, these are just ratings (in case you wanna know, Lincoln was ranked 1, and Buchanan was ranked last), since there is no motivation just a rating, but as the ratings are specific... Quick study on Carter pointed out that Carter was just cursed with the heritage of his predecessor. Most notably the national debt caused by the war in Vietnam. And since the current president is always to blame regardless of the things his predecessor did being to blame for that or not (and for that reason I wouldn't wanna be Trump's successor... Now I wouldn't wanna be president (of any nation) anyway, as I think it's a terrible job).
What I also see (and I guess people are always angry when that happens) Carter was a very active when it came to human rights. They often conflict with "our own interest", hence Trump's popularity, and I see it in other nations too that there is a lot of support for politicians who openly declare human rights as rubbish, and therefore play on how selfish people are (and who take the fact that their own human rights were never violated for granted)... I wonder if that were the reasons why Carter didn't make it into 2nd term... Just a quick guess...