Google has announced that it will change its search indexing algorithms to favor mobile-friendly sites starting on April 21, 2015. This means that responsive sites will see an increase in traffic referred from Google, and non-responsive sites will see a reduction.
In Google's own words, they describe the impact as "significant":
"Starting April 21, we will be expanding our use of mobile-friendliness as a ranking signal. This change will affect mobile searches in all languages worldwide and will have a significant impact in our search results. Consequently, users will find it easier to get relevant, high quality search results that are optimized for their devices."