Madison’s notes are useful historical context, but they don’t change the text’s scope. Article I, Section 4 names elections for Senators and Representatives; Article II separately governs appointment of presidential electors.
If it ain't in the constitution, it ain't the law.