And "shall have" and "Is the only government entity that shall have" are two different things. And other legal precedent, such as with the establishment clause has previously been ruled such that "congress" ends up basically equating to the government as a whole.
The Congress shall have the power to enforce, by appropriate legislation, the provisions of this article.
One scholar notes that the drafting history of Section 3 of the Fourteenth Amendment suggests that the office of the President is covered:
During the debate on Section Three, one Senator asked why ex-Confederates “may be elected President or Vice President of the United States, and why did you all omit to exclude them? I do not understand them to be excluded from the privilege of holding the two highest offices in the gift of the nation.” Another Senator replied that the lack of specific language on the Presidency and Vice-
Presidency was irrelevant: “Let me call the Senator’s attention to the words ‘or hold any office, civil or military, under the United States.’”
I'll highlight that last bit again:
Another Senator replied that the lack of specific language on the Presidency and Vice-
Presidency was irrelevant: “Let me call the Senator’s attention to the words ‘or hold any office, civil or military, under the United States.’”