Is Mitt Romney correct? Do non-religious people lack freedom?

As an agnostic, I was offended by Mitt Romney's ertion that only people with a religion have freedom. Am I wrong to ume that I have freedom without a religion?