Caching is a technique used to store frequently accessed or expensive-to-compute data in a faster storage layer so that future requests can be served more quickly. It helps improve system performance by reducing repeated access to slower data sources such as databases, APIs, or storage systems.
- Cached data is usually stored closer to the application or user for faster retrieval.
- Applications can use local in-memory caches or external caching systems such as Redis.
Example: When a tweet becomes viral, millions of users may request the same content. Instead of querying the database for every request, the application can serve the tweet from a cache, reducing database load and improving response time.
Working of Cache
A web application usually stores persistent data in a database. Accessing the database may require network communication, disk I/O, and query processing, which can be slower than retrieving data from a cache.
A common cache flow works as follows:
- The application receives a request.
- It first checks whether the required data exists in the cache.
- If the data is available, it is returned directly. This is called a cache hit.
- If the data is not available, it is called a cache miss.
- On a cache miss, the application retrieves the required data from the database or another underlying data source.
- The application stores the retrieved data in the cache.
- The response is then returned to the user.
- Future requests for the same data can be served directly from the cache until the cached entry expires or is invalidated.
Challenges of Storing All Data in Cache
Although caching improves performance, storing all application data in a cache is usually not practical.
- Limited and Expensive Memory: High-speed cache storage, particularly RAM-based caching, is generally more expensive than persistent storage.
- Limited Capacity: Cache capacity is usually much smaller than the total amount of application data.
- Volatile Storage: Many in-memory caches may lose cached data after a restart or failure unless persistence is specifically configured.
- Stale Data: Cached copies can become outdated when the underlying data changes.
- Cache Management: Systems need appropriate expiration, invalidation, and eviction policies to determine which data should remain cached.
Types of Cache
Caching can be implemented at different levels of a system depending on where the cached data is stored and how it is shared.
1. Application Server Cache
An application server cache stores frequently accessed data locally within or alongside an application server. Because the data is available close to the application, access can be very fast and does not require a network request to an external cache for every lookup.
Example: If an application repeatedly needs product configuration data, each application server can keep a local cached copy and serve it without repeatedly querying the database.
Limitations:
- Local application-server caching becomes more complicated when an application runs on multiple servers.
- Each application server may maintain its own independent cache.
- A load balancer may send different requests from a user to different servers.
- One server may have an item cached while another server does not.
- Multiple servers may store duplicate copies of the same data.
- Maintaining consistency between local caches can become difficult when underlying data changes.
Note: To fix this, there are two main options: Distributed Cache and Global Cache.
2. Distributed Cache
A distributed cache stores cached data across multiple cache nodes instead of keeping all cached data on a single machine.
- Cached entries are partitioned across multiple cache servers.
- Applications can locate the cache node responsible for a particular key using an appropriate partitioning mechanism.
- Consistent hashing is one commonly used technique for partitioning keys, but it is not required for every distributed cache implementation.
- Other systems may use fixed partitions, hash slots, directories, or other routing mechanisms.
- Adding more cache nodes can increase capacity and throughput.
3. Global or Shared Cache
A global cache is a shared caching layer that can be accessed by multiple application servers. Instead of every application server maintaining only its own local copy, all application servers can access a common cache.
When a requested value is missing from the shared cache, the system can handle the cache miss in different ways.
- Cache-Managed Loading: The caching layer or an associated cache service retrieves missing data from the underlying data source and stores it in the cache.
- Application-Managed Loading: The application detects the cache miss, reads the required data from the database or another source, stores the result in the cache, and then returns it to the user.
4. CDN (Content Delivery Network)/ Edge Cache
A Content Delivery Network (CDN) is a geographically distributed network of edge servers that caches content closer to users.
- Static Content: Delivers HTML, CSS, JavaScript, images, and downloadable files.
- Media Content: Distributes videos and other large media files efficiently.
- Cacheable Dynamic Content: Caches eligible dynamic responses based on configured caching policies.
A typical CDN request flow is:
- A user sends a request for content.
- The request reaches a nearby CDN edge server.
- The edge checks whether the requested content exists in its cache.
- On a cache hit, the CDN returns the cached content directly to the user.
- On a cache miss, the CDN requests the content from the origin server or backend.
- The origin returns the content to the CDN.
- The CDN may store the content in its cache according to its caching rules.
- The CDN returns the content to the user.
Cache Invalidation Strategies
Cache invalidation ensures that outdated cached data is removed or updated when the authoritative data changes.
Without appropriate invalidation, users may receive stale information.
Common cache invalidation approaches include:
- Time-Based Expiration: Cached entries expire after a configured Time-to-Live (TTL).
- Event-Driven Invalidation: Cached entries are invalidated when the underlying data changes.
- Explicit Invalidation: The application manually deletes or updates cached values after write operations.
Proper cache invalidation helps balance performance with data freshness.
Eviction Policies
A cache has limited capacity, so it cannot keep every cached item indefinitely. When the cache becomes full, an eviction policy determines which cached entry should be removed to make space for new data.
Common policies include:
- Least Recently Used (LRU): Removes the entry that has not been accessed for the longest time.
- Least Frequently Used (LFU): Removes entries that have been accessed least frequently.
- First-In-First-Out (FIFO): Removes the oldest cached entry first.
Note: Cache eviction normally removes only the cached copy of the data. The authoritative data remains in the database or primary data store. Poor eviction policies generally lead to additional cache misses and reduced performance rather than permanent data loss.
Advantages of Caching
Caching provides several benefits in system design.
- Reduced Latency: Frequently accessed data can be retrieved from a faster storage layer.
- Reduced Database Load: Repeated requests can be served without querying the database every time.
- Improved Throughput: Faster data access can allow the system to handle more requests.
- Better Scalability: Caching can reduce pressure on backend services as traffic grows.
- Reduced Network Traffic: Cached content may avoid unnecessary communication with remote services.
- Cost Optimization: Reducing database queries and backend processing can lower infrastructure requirements in some workloads.
Limitations of Caching
Caching also introduces additional challenges.
- Data Consistency: Cached data may become stale when the authoritative data changes.
- Cache Invalidation Complexity: Determining when cached entries should be refreshed or deleted can be difficult.
- Cache Miss Overhead: A cache miss may require accessing the slower underlying data source.
- Additional Infrastructure: Distributed caching may require extra servers, monitoring, replication, and maintenance.
- Cold Start: After a cache restart or deployment, the cache may initially be empty, resulting in increased requests to backend systems.
- Hot Keys: Frequently requested keys can overload individual cache nodes in some distributed architectures.
