Understanding Little’s Law

Little’s Law determines the average number of items in a stationary queuing system, based on the average waiting time of an item within a system and the average number of items arriving at the system per unit of time.

In API context it means:

If an API endpoint takes on average 100ms and the API endpoint is receiving 100 requests per second then the average number of concurrent requests in the system is 10.

L = 100 * (100/1000) = 10

If rate remains constant then latency(W) is directly proportional to concurrent requests (L).

Little Law in action:

Simple Go code to test above hypothesis:

package main
import (
    "github.com/gin-gonic/gin"
    "net/http"
    "time"
)
func main() {
    router := gin.Default()
    router.GET("/", runLittleLawTest)
    router.Run("localhost:8080")
}
// getAlbums responds with the list of all albums as JSON.
func runLittleLawTest(c *gin.Context) {
    // Calling Sleep method
    time.Sleep(100 * time.Millisecond)
    c.JSON(http.StatusOK, gin.H{
        "code" : http.StatusOK,
    })
}

Lets load test above with some tool and see the response

Summary:
Total: 10.4566 secs
Slowest: 0.1124 secs
Fastest: 0.1003 secs
Average: 0.1035 secs
Requests/sec: 95.6334

we get 95.6334 requests per second.

L = 95.6334 * 0.1035
L = 9.90

Finding response time

Imagine an application that had no easy way to measure response time. If the mean number in the system and the throughput are known, the average response time can be found using Little’s Law:

mean response time = mean number in system / mean throughput

For example: A queue depth meter shows an average of nine jobs waiting to be serviced. Add one for the job being serviced, so there is an average of ten jobs in the system. Another meter shows a mean throughput of 50 per second. The mean response time is calculated as 0.2 seconds = 10 / 50 per seconds

Applications

1. Little’s law is widely used in manufacturing to predict lead time based on the production rate and the amount of work-in-process.
2. Software-performance testers have used Little’s law to ensure that the observed performance results are not due to bottlenecks imposed by the testing apparatus.
3. Other applications include staffing emergency departments in hospitals