If Carter has a copy of a working memo, addressed to or signed by him, from his administration, it was "Classified," at one time. What are the odds that any one of them had over a hundred documents, some of them classified as "Top Secret," that they denied they had?
The operative word is "shouldn't." If they had a "need to know," at the time the documents were issued and the originator or National Archives did not request their return, returning them now is just a formality.