A cache is a net benefit in performance if the time saved during cache hits exceeds the time lost from the additional overhead. We can estimate this using weighted averages. If the typical time for a regular lookup is L, a cache hit is H, a cache miss is M, and the cache hit ratio is R, then using the cache is more effective if H × R + M × (1 − R) < L.
This is just looking at averages though. If your caching has a big benefit to your p90 latency but holds average latency the same or even degrades just a little, you're probably coming out well ahead overall. Need to look at the shape of the performance distribution, not just mean response times.
