"In a passage in this book, I said about the Apostle Peter: ‘On him as on a rock the Church was built’...But I know that very frequently at a later time, I so explained what the Lord said: ‘Thou art Peter, and upon this rock I will build my Church,’ that it be understood as built upon Him whom Peter confessed saying: ‘Thou art the Christ, the Son of the living God,’ and so Peter, called after this rock, represented the person of the Church which is built upon this rock, and has received ‘the keys of the kingdom of heaven.’ For, ‘Thou art Peter’ and not ‘Thou art the rock’ was said to him. But ‘the rock was Christ,’ in confessing whom, as also the whole Church confesses, Simon was called Peter. But let the reader decide which of these two opinions is the more probable." - Augustine "Retractions"

""If, because the Lord has said to Peter, ‘Upon this rock I will build My Church,’ ‘to thee have I given the keys of the heavenly kingdom;’ or, ‘Whatsoever thou shalt have bound or loosed in earth, shall be bound or loosed in the heavens,’ you therefore presume that the power of binding and loosing has derived to you, that is, to every Church akin to Peter, what sort of man are you, subverting and wholly changing the manifest intention of the Lord, conferring (as that intention did) this (gift) personally upon Peter? ‘On thee,’ He says, ‘will I build My church;’ and, ‘I will give thee the keys’...and, ‘Whatsoever thou shalt have loosed or bound’...In (Peter) himself the Church was reared; that is, through (Peter) himself; (Peter) himself essayed the key; you see what key: ‘Men of Israel, let what I say sink into your ears: Jesus the Nazarene, a man destined by God for you,’ and so forth. (Peter) himself, therefore, was the first to unbar, in Christ’s baptism, the entrance to the heavenly kingdom, in which kingdom are ‘loosed’ the sins that were beforetime ‘bound;’ and those which have not been ‘loosed’ are ‘bound,’ in accordance with true salvation.." - Tertullian, "On Modesty" 21

"And if we too have said like Peter, ‘Thou art the Christ, the Son of the living God,’ not as if flesh and blood had revealed it unto us, but by the light from the Father in heaven having shone in our heart, we become a Peter, and to us there might be said by the Word, ‘Thou art Peter,’ etc. For a rock is every disciple of Christ of whom those drank who drank of the spiritual rock which followed them, and upon every such rock is built every word of the Church, and the polity in accordance with it; for in each of the perfect, who have the combination of words and deeds and thoughts which fill up the blessedness, is the church built by God.
But if you suppose that upon the one Peter only the whole church is built by God, what would you say about John the son of thunder or each one of the Apostles? Shall we otherwise dare to say, that against Peter in particular the gates of Hades shall not prevail, but that they shall prevail against the other Apostles and the perfect? Does not the saying previously made, ‘The gates of Hades shall not prevail against it,’ hold in regard to all and in the case of each of them? And also the saying, ‘Upon this rock I will build My Church?’ Are the keys of the kingdom of heaven given by the Lord to Peter only, and will no other of the blessed receive them? But if this promise, ‘I will give unto thee the keys of the kingdom of heaven,’ be common to others, how shall not all things previously spoken of, and the things which are subjoined as having been addressed to Peter, be common to them?
‘Thou art the Christ, the Son of the living God.’ If any one says this to Him...he will obtain the things that were spoken according to the letter of the Gospel to that Peter, but, as the spirit of the Gospel teaches to every one who becomes such as that Peter was. For all bear the surname ‘rock’ who are the imitators of Christ, that is, of the spiritual rock which followed those who are being saved, that they may drink from it the spiritual draught. But these bear the surname of rock just as Christ does. But also as members of Christ deriving their surname from Him they are called Christians, and from the rock, Peters...And to all such the saying of the Savior might be spoken, ‘Thou art Peter’ etc., down to the words, ‘prevail against it.’ But what is the it? Is it the rock upon which Christ builds the Church, or is it the Church? For the phrase is ambiguous. Or is it as if the rock and the Church were one and the same? This I think to be true; for neither against the rock on which Christ builds His Church, nor against the Church will the gates of Hades prevail. Now, if the gates of Hades prevail against any one, such an one cannot be a rock upon which the Christ builds the Church, nor the Church built by Jesus upon the rock" Origen, Commentary on Matthew, Chapters 10-11).

These last three I have to re-check the sources again for.
""Upon this rock," not upon Peter. For He built His Church not upon man, but upon the faith of Peter. But what was his faith? "Thou art the Christ, the Son of the living God." - John Chrysostom
"To Peter the Father revealed that he should say, "Thou art the Son of the living God." Therefore the building of the Church is upon this rock of confession; this faith is the foundation of the Church." - Hilary
""Faith, then, is the foundation of the Church, for it was not said of Peter’s flesh, but of his faith, that ‘the gates of hell shall not prevail against it.’ But his confession of faith conquered hell." - Ambrose of Milan
If I can go through the Fathers and find a consensus against it, not only can I prove Traditionally speaking that Rome is not the Rock on which the Church is built, but also that faith is what the Church is built on. That's a big if though, and only one of my plans.