Making your site as cache-able as possible is vital to ensure a smooth browsing experience. WordPress in it's basic form is quite efficient, when you compare it to code-bases such as Magento. Adding functionality such as plug-ins, media, themes and widgets all have a negative effect on performance. As part of the process of making this domain as efficient as possible, a number of caching techniques have been considered and employed.
The first, easiest and probably most important aspect of caching is setting up your web server to serve sensible Expires and Cache-control headers on a file-type basis. This allows your visitor to have their browser cache large files upon subsequent page requests, and decreases the number of requests per visitor. It also cuts down on load times; if you set a lifetime for a resource, a browser (with a standard configuration) won't request the resource at all.
You could use something like this for configuring Apache; the answer provides insight into why these headers need to be set.
As with anything WordPress related, there are many tools to perform a single task. The first is WP Super Cache, which I have configured to serve static HTML files. Load times are improved after the first visitor, as subsequent visitors will be served a pre-generated file from cache storage.
Static caching is a trade-off though. Any dynamically generated content and widgets will also be cached. I use a relatively low cache lifetime, so that widgets still update quite often (such as the Last.fm, Lifestream and Twitter widgets). You could also have the widgets update via AJAX calls to a non-cached script, though this depends on the Widget supporting this. Its also worth noting that the first user hitting the page will wait longer than subsequent visitors; the cache must be generated and served, which has a negative effect on load times on the first visit. WP Super Cache also invalidates a cache file when a post is updated or a comment posted, which causes a cache refresh.
There are downsides to optimising output, as it is quite easy to over-optimise and impact on user experience. For example; Enabling conversion of background images in CSS to data URIs caused load times to spike to around 20-30 seconds (for non-cached page requests). Static caching fixes the issue on subsequent visits, but each invalidation causes an unacceptable load time for the visitor. This could possibly be resolved in the plug-in, though for the time being I have the feature disabled.
I encountered yet another issue with data URI images; malformed style-sheet parsing in Firefox Desktop and Mobile releases. This caused the stylesheet processing to end prematurely, as not only were background images missing but also basic style rules. This could be caused by the CSS implemented in my theme causing an incompatibility, though with the load time problem I haven't seen much point in debugging.
I'm looking to integrate a caching server in the future, and it's looking like Varnish is a likely candidate. Varnish will act as a proxy to the web server, and there will be some in-depth configuration required for optimal operation. Varnish may replace WP Super Cache, as it's cache will be in memory via memcached. I have also found a suitable Varnish WordPress integration, which should ease the process of migration.
The problem with integrating Varnish is that it acts as a proxy to the application (web) server. The application server would also have to be reconfigured with this in mind. It also introduces additional administration, as you may need to alter the ruleset to support specific plug-ins, widgets and core WordPress updates.