Only, that's not what Jesus told Peter. Catholics have been purposely misinterpreting that since the 4th century, using it as justification for their man-made version of Christianity. Contextually, what Jesus was talking about was Peter's confession of Him - "He was asking His disciples, “Who do people say that the Son of Man is?” And they said, “Some say John the Baptist; and others, Elijah; but still others, Jeremiah, or one of the prophets.” He *said to them, “But who do you say that I am?” Simon Peter answered, “You are the Christ, the Son of the living God.” Jesus was referring to Peter's answer "You are the Christ..." as the rock upon which His church is built.
Similarly, they have misinterpreted the phrase you bolded as justification for their blasphemous usurpation of Christ's intermediary power. The pope has no such power as you suggest. Indeed, he is just a man, no more no less.