Latency and response time are often used synonymously, but they are not the same. The response time is what the client sees:
The real thing you care about is perceived response time. Dazzle your user with the quick stuff while the slow stuff slowly fills in.
Speeding up the entire page might not be worth it if it slows down the fastest bits.

· Flag
Brian