"I do not believe in a personal God and I have never denied this but have expressed it clearly. If something is in me which can be called religious then it is the unbounded admiration for the structure of the world so far as our science can reveal it."
To be honest, this quote is not taken out of context, but just because he did not believe in a personal God does not mean that he does not believe in God. He also said,
"I believe in Spinoza's God who reveals himself in the orderly harmony of what exists, not in a God who concerns himself with the fates and actions of human beings."
Of the way his quotes are typically used to portray him as an atheist, he had this to say,
"In view of such harmony in the cosmos which I, with my limited human mind, am able to recognize, there are yet people who say there is no God. But what really makes me angry is that they quote me for the support of such views." The Expanded Quotable Einstein, Princeton University, page 214
An example of this is a quote by Richard Dawkins,
"Einstein sometimes invoked the name of God (and he is not the only atheistic scientist to do so), inviting misunderstanding by supernaturalists eager to misunderstand and claim so illustrious a thinker as their own." The God Delusion, Richard Dawkins
This is exactly what bothered Albert Einstein. Dawkins either purposefully misleads readers or he has not done his homework. It is fairly common knowledge in academia that Einstein followed Spinoza's conception of God, but he did believe in a God. In fairness, he did not believe in the personal God of Christianity. In fact, he was rather critical of the idea, but he did believe in an intelligent force behind the universe. He was not an atheist.
A few more quotes from Albert Einstein about God.
"I'm not an atheist and I don't think I can call myself a pantheist. We are in the position of a little child entering a huge library filled with books in many languages. The child knows someone must have written those books. It does not know how. It does not understand the languages in which they are written. The child dimly suspects a mysterious order in the arrangements of the books, but doesn't know what it is. That, it seems to me, is the attitude of even the most intelligent human being toward God. We see a universe marvelously arranged and obeying certain laws, but only dimly understand these laws. Our limited minds cannot grasp the mysterious force that moves the constellations."
"My religion consists of a humble admiration of the illimitable superior spirit who reveals himself in the slight details we are able to perceive with our frail and feeble mind."
"Science without religion is lame. Religion without science is blind."
"When the solution is simple, God is answering."
"God does not play dice with the universe."
"God is subtle but he is not malicious."