Skip to main content

Caching in System Design

Caching is a technique that stores frequently accessed data in fast memory to improve system performance. It takes advantage of the locality of reference principle: recently requested data is likely to be requested again.

Types of Caching​

1. Application Server Cache​

  • Cache placed directly on the request layer node
  • Each node has its own cache

Challenges with Multiple Nodes:

  • Load balancer randomly distributes requests
  • Same request can hit different nodes
  • Increased cache misses
  • Reduced cache effectiveness

Solutions:

  • Global caches
  • Distributed caches
  • Consistent hashing

2. Distributed Cache​

A caching system spread across multiple nodes.

Characteristics:

  • Each node owns part of the cached data
  • Uses consistent hashing for data distribution
  • Scales horizontally

Advantages:

  • Easy to scale by adding nodes
  • Better resource utilization
  • Improved resilience

Disadvantages:

  • Node failure causes cache loss
  • Network overhead
  • Complexity in management

3. Global Cache​

A single cache space accessible by all application nodes.

Implementation Types:

  1. Cache Server Handles Cache Miss

    • Most common approach
    • Simplified application logic
    • Centralized cache management

  2. Request Nodes Handle Cache Miss

    • Better for static data
    • Application controls eviction
    • Useful for specialized caching needs

4. Content Delivery Network (CDN)​

Specialized cache for static media content.

Process Flow:

  1. User requests static content
  2. CDN serves if content exists locally
  3. If missing, CDN fetches from origin
  4. Content cached for future requests

When to Use:

  • Large amounts of static media
  • Geographically distributed users
  • High availability requirements

Cache Invalidation​

Keeping cache and source of truth synchronized.

1. Write-Through Cache​

Process:

  • Write to cache and storage simultaneously
  • Ensures complete data consistency

Advantages:

  • Data safety and consistency
  • Reliable recovery
  • Simple to implement

Disadvantages:

  • Higher write latency
  • Some writes may never be read

2. Write-Around Cache​

Process:

  • Write only to permanent storage
  • Cache bypassed for writes

Advantages:

  • Reduced cache pollution
  • Good for write-heavy workloads
  • Optimized for read patterns

Disadvantages:

  • Cache miss on recent writes
  • Higher read latency for new data
  • Potential data inconsistency

3. Write-Back Cache​

Process:

  • Write only to cache
  • Asynchronous write to storage
  • Periodic cache flush

Advantages:

  • Low write latency
  • High write throughput
  • Reduced load on storage

Disadvantages:

  • Risk of data loss
  • More complex implementation
  • Requires careful monitoring

Cache Eviction Policies​

Methods to remove items when cache is full:

1. First In First Out (FIFO)​

  • Evicts oldest entries first
  • Simple to implement
  • Good for static content

2. Last In First Out (LIFO)​

  • Evicts newest entries first
  • Rarely used in practice
  • Special use cases only

3. Least Recently Used (LRU)​

  • Evicts least recently accessed items
  • Most common approach
  • Good general-purpose policy

4. Most Recently Used (MRU)​

  • Evicts most recently used items
  • Good for cyclic access patterns
  • Specialized use cases

5. Least Frequently Used (LFU)​

  • Tracks access frequency
  • Keeps popular items longer
  • Memory intensive

6. Random Replacement (RR)​

  • Randomly selects items to evict
  • Simple to implement
  • Surprisingly effective

Best Practices​

  1. Choose the Right Type

    • Consider access patterns
    • Evaluate consistency needs
    • Account for scale requirements

  2. Monitor Cache Performance

    • Track hit/miss ratios
    • Monitor eviction rates
    • Measure latency impact

  3. Set Appropriate Sizes

    • Balance memory usage
    • Consider working set size
    • Plan for growth

  4. Handle Edge Cases

    • Cache stampede prevention
    • Error handling
    • Fallback mechanisms

Remember​

  • Cache for performance, not functionality
  • Consider data consistency requirements
  • Monitor and adjust cache settings
  • Plan for cache failures
  • Keep it simple when possible

Caching is a powerful optimization technique, but requires careful consideration of trade-offs between consistency, complexity, and performance.