We took one virtual machine from our fleet and crafted it so it proxies HN via Varnish and Nginx with the intention to see how much we can speed things up for read-only visitors. The results are interesting although not so surprising (TL;DR at end of post). The caching logic from this experiment can be integrated with the our previously suggested auto throttling of Varnish.
The VM is constructed to:
a) Run Varnish in front of Nginx, which pulls objects from news.ycombinator.com for caching
b) Create a shared cache object for each resource (non-cookied, anonymous, etc)
c) Replace ycombinator.com instances with hn.unixy.net so http_sub_module was compiled in.
d) Inject Non binary objects with header X-Robots-Tag “noindex, nofollow” to prevent SEO penalties
This is the score of HN without Varnish proxying:
This is the score of HN with Varnish proxying: