Look at the 4th amendment, protection against unlawful search and seizures.
"The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."
Let me break this down. First, its the right to be "...secure in [our] persons, houses, papers and effects..." Metadata is like me watching you talk to somebody, and taking note of that. It isn't intruding physically into our houses, persons, nor is it reading our papers or going through our effects.
"...Searches and seizures..." Metadata is neither taking anything from us, nor is it searching anything that belongs to anyone. It's observing interactions taking place. It is not a search because its intent is not to check what we're saying.
"...Particularly describing the places to be searched..." Imagine that all our electronic interactions are moved into a physical realm. Our personal data would be stuff we carry on us. Our text messages and so forth (the words in the conversations) are like two people talking face to face.
And metadata is like us walking up to someone and shaking their hand, waving at them, or hugging them. But even less so, because hugging vs. shaking hands tells us how close the two people are. Metadata doesn't even do that. It does nothing more than show us who is talking to whom. Not what the contents of the conversation are, nor does it provide us with any information about the people themselves.
I don't think that the entire process is fully constitutional, however. If there is one link in the chain I'm worried about, it is the process by which the DOJ gives the NSA a warrant to probe into the individuals they've observed. I don't know much about this process. I don't know what the checks and balances are, what the requirements are, all I know is that over 95% of 500 requests have been granted.
If everyone wants to pick something about PRISM that's shady, this is the thing to go after and demand accountability for. Sure, it's creepy to think that the government is watching to see who we all talk to, but there's nothing unconstitutional about it. The DOJ giving out warrants to basically every person the NSA is remotely interested in is the worrying thing about all this, and I hope you all realize that.