New Google algorithm for page ranking will include your page load time.
The faster your page loads, the better Google ranking you will have.
Matt Cutts from Google commented on page loading speed in November.
Page loading speed will affect your Quality Score for AdWords and your Ranking in Google's index.
So quality of your hosting will be important, optimization of your page code to have high-performance website and cache system used.
Advanced cache system includes at least 5 different components and requires good VPS server. You need to have enabled on your server mem-cached for static cache, for gzip cache you need to have on you server mod_deflate and mod_headers, in addition your WP cache plug-in should use object cache, query cache and asset cache, as Semiologic Pro does.
To help accomplish better results, Google just added Site Performance to webmasters tools. The key component is Page Speed add-on for Firefox. It adds Page Speed tool to Firebug, takes few seconds to install and you can analyze any page, find your score and suggestions for improvements. Really very cool and useful open source tool.
Below are some examples how to reduce page loading time with help of Page Speed.
Here is a useful WP plugin to display Page Load Speed.
It will show your page load time automatically in the footer of your WordPress website. You can keep it always activated, or just for load speed tests to optimize your website for high performance and better SE rankings.
You should have your home page loading in under 1 sec, and other pages (posts) containing more content or product listings, in less than 5 seconds for better user experience and higher organic rankings.