# Lorbic - Full Content

> Decoding the mechanics of high-performance systems.

---

## How Go Channels Actually Work

**Date:** 2026-08-13
**URL:** http://localhost:1313/go-channels-deep-dive/
**Description:** Go channels look simple, but they rely on heap-allocated ring buffers and runtime scheduler parks. Learn how the hchan structure, stack-to-stack copies, and lock ordering work to prevent leaks and panics.


A single unbuffered channel send without an active receiver is enough to silently lock up a production worker pool, leaking memory until your service gets killed by the OS kernel.

On the surface, Go channels look simple: they synchronize concurrent execution without manual mutex management. But under heavy load, a minor oversight in channel sizing or worker coordination blocks the scheduler, leaks memory, or crashes processes with runtime panics.

![The Go scheduler coordinating 10,000 goroutines on a single-core CPU](go-scheduler-meme.webp)

Writing concurrent systems that stay stable under production load requires understanding the underlying machinery: how channels sit in memory, how the scheduler coordinates parked goroutines, and how to structure production pipelines.

---

## Why Channels Exist

Go's concurrency model comes from Tony Hoare's 1978 paper, *Communicating Sequential Processes* (CSP).

In traditional multi-threaded environments, threads communicate by sharing memory. To prevent race conditions, you lock that memory using mutexes. This approach has well-known failure modes: it is easy to introduce deadlocks, lock contention degrades throughput, and tracking ownership of shared data structures becomes difficult as codebases grow.

CSP flips this pattern. Concurrent processes communicate by passing data through explicit, synchronized input and output ports:

> Do not communicate by sharing memory; instead, share memory by communicating.

Channels are first-class values that connect independent execution units. They decouple the sender from the receiver. The sender does not need to know which goroutine receives the data, and the receiver does not need to know who sent it. The channel manages both the data transfer and the synchronization.

---

## Inside the hchan Struct

In Go, a channel is a heap-allocated structure called `hchan`, defined in `src/runtime/chan.go`.

When you call `ch := make(chan int, 10)`, the compiler allocates an `hchan` struct on the heap and returns a pointer to it. Channels are passed by reference because passing a channel copies the pointer to this underlying struct.

{{< details title="Stack vs Heap Allocation" label="Glossary //" >}}
When Go runs your code, it places variables in either stack or heap memory.

* **The Stack**: Fast, per-goroutine memory allocated in stack frames. When a function returns, its frame is reclaimed immediately. It only holds data whose lifetime is strictly bound to that function call.
* **The Heap**: Global storage for variables that outlive the function that created them. Heap allocations require the runtime to locate free memory and later track and clean it up via the Garbage Collector (GC).

Because channels are shared across independent goroutines with different lifecycles, Go allocates the `hchan` struct on the heap.
{{< /details >}}

The `hchan` memory layout in `src/runtime/chan.go`:

```go
type hchan struct {
	qcount   uint           // Total data currently in the queue
	dataqsiz uint           // Size of the circular queue buffer
	buf      unsafe.Pointer // Points to an array of dataqsiz elements (nil if unbuffered)
	elemsize uint16
	closed   uint32
	elemtype *_type         // Element type descriptor for GC and runtime checks
	sendx    uint           // Buffer index where the next send will write
	recvx    uint           // Buffer index where the next receive will read
	recvq    waitq          // Linked list of blocked receivers (waiting goroutines)
	sendq    waitq          // Linked list of blocked senders (waiting goroutines)

	lock mutex              // Protects all fields in hchan and blocked sudogs
}
```

### The Ring Buffer

For buffered channels, `buf` points to a contiguous block of memory allocated for the queue, managed as a circular ring buffer:
* `dataqsiz` is the total capacity passed to `make`.
* `qcount` is the number of active items in the buffer.
* `sendx` is the array index for the next write.
* `recvx` is the array index for the next read.

When `sendx` or `recvx` reaches `dataqsiz`, it wraps back to index `0`.

### The Wait Queues

When a goroutine sends to a full channel or receives from an empty channel, it cannot proceed. The runtime parks the goroutine and registers it in `sendq` or `recvq`.

These queues are doubly-linked lists of `sudog` structures. A `sudog` represents a parked goroutine:
* `g`: Pointer to the parked goroutine.
* `elem`: Pointer to the memory address for data transfer.
* `next` and `prev`: Pointers for wait queue linking.

### The Channel Lock

Every channel operation (sending, receiving, closing, querying length) acquires `hchan.lock` first. This is a low-level runtime spinlock.

Channels are not lock-free data structures. The runtime uses standard locking internally, but keeps critical sections short and optimized to maintain low contention.

{{< details title="What is a sudog?" label="Glossary //" >}}
The Go scheduler represents every goroutine as a `g` struct. But a single goroutine can wait on multiple channels simultaneously in a `select` statement.

Because a `g` struct cannot be linked into multiple wait queues at the same time, the runtime creates a `sudog` as an intermediary ticket. If a goroutine blocks on a `select` with three channels, the runtime creates three `sudog` structs, each pointing back to the same `g`, and enqueues one into each channel's wait list.
{{< /details >}}

{{< details title="How gopark and goready coordinate sleep" label="Glossary //" >}}
When a goroutine blocks on a channel, it cannot spin in a busy loop without wasting CPU cycles. The runtime calls `gopark()` to detach the goroutine from its operating system thread (M) and changes its state from running to waiting.

When another goroutine completes the transfer, it finds the waiting `sudog` in the queue and calls `goready()`. This moves the parked goroutine back to runnable and pushes it to the scheduler's run queue.
{{< /details >}}

{{< details title="What is a spinlock?" label="Glossary //" >}}
A spinlock repeatedly checks a lock status in a tight loop instead of putting the operating system thread to sleep.

Go uses a spinlock for `hchan.lock` because channel queue updates take only a few nanoseconds. Spinning for a fraction of a microsecond avoids the expensive context switch of sleeping and waking an OS thread.
{{< /details >}}

Memory layout of a buffered channel (`dataqsiz: 8`, `qcount: 3`) with active wait queues:

```mermaid
graph TD
    subgraph hchan ["hchan Struct in Heap"]
        hchan_lock["lock (mutex)"]
        qcount["qcount: 3"]
        dataqsiz["dataqsiz: 8"]
        sendx["sendx: 5"]
        recvx["recvx: 2"]
        
        subgraph RingBuffer ["Circular Buffer (buf)"]
            buf0["[0] Data"]
            buf1["[1] Data"]
            buf2["[2] Data (recvx)"]
            buf3["[3] Data"]
            buf4["[4] Data"]
            buf5["[5] Empty (sendx)"]
            buf6["[6] Empty"]
            buf7["[7] Empty"]
        end
        
        subgraph wait_queues ["Wait Queues"]
            recvq["recvq (waitq)"] --> recv_head["sudog (G1)"]
            sendq["sendq (waitq)"] --> send_head["sudog (G2)"] --> send_next["sudog (G3)"]
        end
    end

    G1["Goroutine 1 (Blocked Receiver)"]
    G2["Goroutine 2 (Blocked Sender)"]
    G3["Goroutine 3 (Blocked Sender)"]

    recv_head -.->|points to| G1
    send_head -.->|points to| G2
    send_next -.->|points to| G3
```

---

## Channel Types and Memory Behavior

Go provides unbuffered and buffered channels. Both share the `hchan` layout, but their scheduling and memory mechanics differ.

### Unbuffered Channels

An unbuffered channel has zero buffer capacity (`dataqsiz == 0` and `buf == nil`).

```go
ch := make(chan int)
```

A send cannot complete until a receive starts, and vice versa. The two goroutines synchronize at a shared rendezvous point.

#### Unbuffered Send Execution Flow:
1. Goroutine A attempts to send `x` to `ch`.
2. Goroutine A acquires `ch.lock`.
3. It inspects `recvq`.
4. If a receiver (Goroutine B) is waiting in `recvq`:
   * The runtime executes a direct stack-to-stack copy: it copies `x` from Goroutine A's stack directly to the memory address stored in Goroutine B's `sudog.elem`.
   * This direct copy bypasses intermediate buffer allocations entirely.
   * Goroutine A releases `ch.lock`.
   * The runtime calls `goready(gB)` to transition Goroutine B back to a runnable state on the scheduler's run queue.
5. If no receiver is waiting:
   * Goroutine A allocates a `sudog` on its stack.
   * It writes the address of `x` into `sudog.elem`.
   * It pushes the `sudog` into `ch.sendq`.
   * It releases `ch.lock`.
   * It calls `gopark()`, which instructs the Go scheduler to detach Goroutine A from its OS thread (M) and put it into a waiting state. The OS thread is then free to run other runnable goroutines.

```mermaid
sequenceDiagram
    autonumber
    actor G_Sender as Sender Goroutine (G1)
    participant Runtime as Go Runtime Scheduler
    participant Chan as Channel (hchan)
    actor G_Recv as Receiver Goroutine (G2)

    Note over G_Sender, G_Recv: Scenario: Sender arrives first on unbuffered channel

    G_Sender->>Chan: Send value (ch <- val)
    Chan->>Chan: Lock channel
    Chan->>Chan: Check recvq (empty)
    Note over G_Sender: Create sudog on stack,<br/>set elem = &val
    G_Sender->>Chan: Enqueue sudog to sendq
    Chan->>Chan: Unlock channel
    G_Sender->>Runtime: gopark() (Transition G1 to waiting)
    Runtime->>Runtime: Run other goroutines on OS thread

    Note over G_Recv: Receiver arrives later
    G_Recv->>Chan: Receive value (<-ch)
    Chan->>Chan: Lock channel
    Chan->>Chan: Check sendq (finds G1 sudog)
    Note over Chan: Direct Memory Copy:<br/>G1 Stack (&val) to G2 Stack
    Chan->>G_Sender: Dequeue sudog
    Chan->>Chan: Unlock channel
    G_Recv->>G_Recv: Proceed with copied value
    Chan->>Runtime: goready(G1) (Transition G1 to runnable)
```

### Buffered Channels

A buffered channel maintains an in-memory queue (`dataqsiz > 0` and `buf != nil`).

```go
ch := make(chan int, 3)
```

Senders proceed as long as empty slots remain. Receivers proceed as long as data remains in the buffer.

#### Buffered Send Execution Flow:
1. Goroutine A acquires `ch.lock`.
2. It checks buffer capacity (`qcount < dataqsiz`).
3. If buffer space is available:
   * It copies `x` to `buf[sendx]`.
   * It updates `sendx = (sendx + 1) % dataqsiz`.
   * It increments `qcount`.
   * It releases `ch.lock` and returns.
4. If the buffer is full:
   * It allocates a `sudog` with `elem = &x`.
   * It appends the `sudog` to `ch.sendq`.
   * It releases `ch.lock`.
   * It calls `gopark()`, which detaches Goroutine A from its OS thread (M) and puts it into a waiting state until buffer space becomes available.

#### Buffered Receive Execution Flow:
1. Goroutine B acquires `ch.lock`.
2. It checks buffer occupancy (`qcount > 0`).
3. If data is in the buffer:
   * **Case A: `sendq` is empty.**
     * Copies value from `buf[recvx]` to destination.
     * Clears memory at `buf[recvx]` for GC safety.
     * Updates `recvx = (recvx + 1) % dataqsiz`.
     * Decrements `qcount`.
     * Releases `ch.lock`.
   * **Case B: A sender (Goroutine A) is blocked in `sendq`.**
     * The buffer is full. To maintain FIFO order, Goroutine B copies the item at `buf[recvx]` to its destination.
     * It copies Goroutine A's value from `sudog.elem` directly into `buf[recvx]`.
     * Advances `recvx` and `sendx`.
     * Pops Goroutine A from `sendq`.
     * Releases `ch.lock` and calls `goready(gA)` to mark Goroutine A runnable, moving it back to the scheduler's run queue.

### Directional Channels

Directional constraints are compile-time type restrictions that do not alter runtime memory layout:

```go
func worker(jobs <-chan int, results chan<- int) {
    for job := range jobs {
        results <- job * 2
    }
}
```

* `<-chan int`: Receive-only. The compiler forbids sends and `close()`.
* `chan<- int`: Send-only. The compiler forbids receives.

Directional types enforce ownership boundaries and prevent workers from closing shared input channels.

---

## Select Statement Mechanics

The `select` statement multiplexes multiple channel operations. The runtime implements this in `src/runtime/select.go`.

```go
select {
case val := <-ch1:
    fmt.Println("Received from ch1:", val)
case ch2 <- 42:
    fmt.Println("Sent to ch2")
default:
    fmt.Println("No channel was ready")
}
```

### Lock Ordering

Evaluating a `select` requires locking all participating channels. To prevent deadlocks when concurrent goroutines execute `select` blocks with overlapping channels in different orders, the runtime sorts all channels by their memory addresses:

1. Creates an array of `scase` records.
2. Sorts the array by `hchan` pointer addresses.
3. Acquires locks sequentially in sorted address order.

Sorting locks by address eliminates lock-inversion deadlocks.

### Randomized Case Evaluation

If multiple cases are ready simultaneously, scanning top-to-bottom would starve lower cases.

To ensure fairness, the runtime generates a pseudo-random permutation of case indices and evaluates them in that randomized order. The first channel found ready executes its transfer, unlocks all channels, and returns.

### Multiplexed Parking

When no channel is ready and there is no `default` case:
1. The runtime allocates a `sudog` for each case.
2. It registers the goroutine in the wait queue (`sendq` or `recvq`) of every channel in the `select`.
3. It calls `gopark()` to detach the goroutine from its OS thread and park it across all selected channels.
4. When any channel wakes the goroutine via `goready()`, it locks all channels in address order, removes its `sudog` from the remaining wait queues, unlocks all channels, and executes the selected case block.

---

## Production Concurrency Patterns

### Worker Pool with Graceful Shutdown

Worker pools require explicit lifecycle handling to prevent goroutines from hanging when job dispatch finishes:

```go
package pool

import (
	"context"
	"sync"
)

type Job struct {
	ID    int
	Data  string
	Error error
}

type Result struct {
	JobID  int
	Output string
	Err    error
}

func Worker(ctx context.Context, id int, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done()
	for {
		select {
		case <-ctx.Done():
			return
		case job, ok := <-jobs:
			if !ok {
				return
			}
			
			res, err := processJob(ctx, job)
			
			select {
			case <-ctx.Done():
				return
			case results <- Result{JobID: job.ID, Output: res, Err: err}:
			}
		}
	}
}

func processJob(ctx context.Context, job Job) (string, error) {
	return "done: " + job.Data, nil
}
```

### Fan-In Multiplexing

Merging multiple inbound channels into a single consolidated output stream:

```go
package fanin

import (
	"context"
	"sync"
)

func Merge[T any](ctx context.Context, channels ...<-chan T) <-chan T {
	out := make(chan T)
	var wg sync.WaitGroup

	multiplex := func(c <-chan T) {
		defer wg.Done()
		for {
			select {
			case <-ctx.Done():
				return
			case val, ok := <-c:
				if !ok {
					return
				}
				select {
				case <-ctx.Done():
					return
				case out <- val:
				}
			}
		}
	}

	wg.Add(len(channels))
	for _, c := range channels {
		go multiplex(c)
	}

	go func() {
		wg.Wait()
		close(out)
	}()

	return out
}
```

### Token Bucket Rate Limiting

A buffered channel pre-filled with empty structs controls throughput by tying execution to token availability:

```go
package rate

import (
	"context"
	"time"
)

type Limiter struct {
	tokens chan struct{}
	ticker *time.Ticker
}

func NewLimiter(rate int, burst int) *Limiter {
	l := &Limiter{
		tokens: make(chan struct{}, burst),
		ticker: time.NewTicker(time.Second / time.Duration(rate)),
	}

	for i := 0; i < burst; i++ {
		l.tokens <- struct{}{}
	}

	go func() {
		for range l.ticker.C {
			select {
			case l.tokens <- struct{}{}:
			default:
			}
		}
	}()

	return l
}

func (l *Limiter) Wait(ctx context.Context) error {
	select {
	case <-ctx.Done():
		return ctx.Err()
	case <-l.tokens:
		return nil
	}
}

func (l *Limiter) Stop() {
	l.ticker.Stop()
}
```

---

## Channel State Matrix

Runtime behavior for every channel state and operation:

| Channel State | Send (`ch <- val`) | Receive (`<-ch`) | Close (`close(ch)`) |
| :--- | :--- | :--- | :--- |
| **`nil`** | Blocks forever (goroutine leak) | Blocks forever (goroutine leak) | **Panic** (`panic: close of nil channel`) |
| **Open & Empty** | Writes to buffer or direct copy | Blocks until sent or closed | Closes channel, wakes all receivers |
| **Open & Full** | Blocks until slot freed | Reads from buffer head | Closes channel, wakes all receivers |
| **Closed** | **Panic** (`panic: send on closed channel`) | Returns zero value (`ok == false`) | **Panic** (`panic: close of closed channel`) |

### Blocking on Nil Channels

A nil channel is declared without initialization:

```go
var ch chan int // nil
```

Sends and receives on a nil channel bypass the `hchan` lock and call `gopark()` directly, putting the calling goroutine into a permanent waiting state with no wake-up signal.

While setting a channel variable to `nil` is a valid technique to disable a branch in a dynamic `select` loop, doing so unintentionally permanently parks the calling goroutine.

### Sending to Closed Channels

Sending to a closed channel panics immediately:

```go
ch := make(chan int)
close(ch)
ch <- 42 // panic: send on closed channel
```

The sender that produces data must own and close the channel. Receivers should never close channels. When multiple senders exist, coordinate closure using `sync.WaitGroup` or a dedicated coordinator goroutine.

### Leaking Goroutines on Abandoned Channels

When a producer goroutine sends on an unbuffered channel, but the receiver abandons the read due to a timeout or early exit, the producer remains blocked forever:

```go
func QueryDatabase() string {
    ch := make(chan string) // Unbuffered
    
    go func() {
        res := runHeavyQuery()
        ch <- res // Leaks if QueryDatabase exits early
    }()
    
    select {
    case res := <-ch:
        return res
    case <-time.After(100 * time.Millisecond):
        return "timeout"
    }
}
```

Remedies:
1. Use a buffered channel of size 1 (`make(chan string, 1)`) so the producer can write and exit without waiting.
2. Bind the worker to a `context.Context` cancellation check.

---

## Frequently Asked Questions

### Why does receiving from a closed channel return data, but sending panics?

This behavior supports pipeline drain semantics. When an upstream stage closes its output channel, downstream consumers must read all buffered items before terminating.

Once the buffer is drained, `val, ok := <-ch` returns the zero value with `ok == false`, signaling EOF. Permitting sends on closed channels would invalidate this termination guarantee by injecting data after the close signal.

### Can unclosed channels be garbage collected?

Yes. Channels do not require explicit closing to be garbage collected.

If a channel is unreachable and no goroutines are parked in `sendq` or `recvq`, the GC reclaims the `hchan` memory. However, if goroutines remain blocked on the channel, they retain references to the `hchan` struct, preventing garbage collection and leaking memory.

### How does the runtime enforce channel type safety?

The compiler stores a `*_type` descriptor in `hchan.elemtype` and the byte size in `hchan.elemsize`.

During memory transfers (stack-to-stack copies or ring buffer updates), the runtime copies `elemsize` bytes directly. The garbage collector reads `elemtype` to identify and trace heap pointers stored within the buffer.

### Why is there no `IsClosed(ch)` function?

An `IsClosed(ch)` check creates a Time-of-Check to Time-of-Use (TOCTOU) race condition: a channel checked as open could be closed by another goroutine an instant before the subsequent send executes, triggering a panic. Concurrency designs must manage channel closure through single-sender ownership or synchronization primitives like `sync.Once`.

### What is the performance overhead of channels versus mutexes?

`sync.Mutex` operations execute in user space using atomic CPU instructions when uncontended.

Channels involve acquiring an internal spinlock, copying memory payloads, and interacting with the scheduler (`gopark`/`goready`) when blocking. Use mutexes for protecting internal state within a single struct. Use channels for orchestrating concurrent data flow and process lifecycles across independent components.

---

## Recommended Reading

* **The Go Programming Language** by Alan A. A. Donovan and Brian W. Kernighan: Chapter 8 covers goroutines and channel synchronization.
* **Concurrency in Go** by Katherine Cox-Buday: Chapters 3 and 4 detail concurrency building blocks and pipeline patterns.
* **Scheduling In Go** by William Kennedy: Deep-dive series on the Go scheduler, OS thread transitions, and context switching.
* **Go Concurrency Patterns** by Rob Pike: The foundational Google I/O talk on channel composition and multiplexing.
* **Go Runtime Source Code (`src/runtime/chan.go`)**: The authoritative implementation of `hchan`, `chansend`, and `chanrecv`.
* **Google I/O 2012 - Go Concurrency Patterns**: https://youtu.be/f6kdp27TYZsc


---

## Part 6: Simulating a Living Facility: Room Derivation, Sunlight Spillover, and Positional SFX

**Date:** 2026-08-10
**URL:** http://localhost:1313/modular-missions-floodfill-audio-raylib/
**Description:** Building structural algorithms to identify rooms procedurally from coordinate maps, propagate sunlight through doorways, parse modular JSON missions, and render positional sound.


Generating a raw matrix of integer tiles gives you a map layout, but it doesn't give you a living environment. When I loaded my first level into *Derelict Facility*, the engine had no concept of what a "Laboratory" or a "Reactor Room" was. It just saw a flat array of walls and floors.

If a player flipped a power terminal inside a room, I had no clean way to know which room lights or doors to toggle without scanning the whole map grid on every frame. If an alarm fired down the hall, it played at full volume regardless of where the player was standing.

To make the environment feel reactive, I built spatial abstractions on top of the raw map: extracting room boundaries via flood fill, spilling sunlight through open doorways, loading modular JSON missions, and attenuating positional audio.

{{< youtube p4MoXKkQl4A >}}

## Engine Simulation Architecture

The diagram below shows how the raw tile map flows into spatial systems during initialization and runtime updates:

```mermaid
graph TD
    RawMap["Raw Tile Array (1D Slice)"] --> BFS["BFS Flood Fill (deriveRooms)"]
    BFS --> Rooms["Logical Room Bounding Boxes (Rect)"]
    
    GlassTiles["Skylight Glass Tiles"] --> LightProp["Recursive Light Decay (PropagateSunlight)"]
    LightProp --> DaylightFactors["Tile Daylight Factors (0.0 to 1.0)"]
    
    AudioEvent["Spatial Sound Event (x, y)"] --> AudioAtten["Positional Attenuation (PlayPositionalSound)"]
    PlayerPos["Player Coordinates (px, py)"] --> AudioAtten
    AudioAtten --> RaylibAudio["Raylib CGO Sound Engine"]
```

---

## Procedural Room Derivation via BFS Flood Fill

Map data loaded from disk only marks individual coordinates as `TileTypeFloor` or `TileTypeWall`. To group contiguous floor regions into logical rooms, we run a Breadth-First Search (BFS) flood fill (`internal/world/json_loader.go`).

I chose BFS over recursion (DFS) here to guarantee bounded memory consumption on large maps and avoid stack overflow risk during level initialization.

### Extraction Pipeline
1. Iterate over map coordinates to locate unvisited floor tiles that are not doors.
2. Initialize a queue seeded with the starting coordinate and mark it as visited.
3. Traverse cardinal neighbors (Up, Down, Left, Right). Stop expanding when encountering walls or door tiles.
4. Track the minimum and maximum bounds (`minX`, `minY`, `maxX`, `maxY`) to construct a bounding box (`Rect`) representing the room.

{{< code lang="go" file="internal/world/json_loader.go" >}}
func (l *JSONMapLoader) deriveRooms(m *Map) {
	visited := make([]bool, m.Width*m.Height)
	isDoor := make([]bool, m.Width*m.Height)
	for _, d := range m.Doors { 
		isDoor[d.Y*m.Width+d.X] = true 
	}

	for y := 0; y < m.Height; y++ {
		for x := 0; x < m.Width; x++ {
			idx := y*m.Width + x
			tile := m.GetTile(x, y)
			if tile != nil && tile.Type == TileTypeFloor && !visited[idx] && !isDoor[idx] {
				minX, minY := x, y
				maxX, maxY := x, y
				queue := []entity.Point{{X: x, Y: y}}
				visited[idx] = true

				for len(queue) > 0 {
					p := queue[0]
					queue = queue[1:]
					
					if p.X < minX { minX = p.X }
					if p.Y < minY { minY = p.Y }
					if p.X > maxX { maxX = p.X }
					if p.Y > maxY { maxY = p.Y }

					neighbors := []entity.Point{
						{X: p.X, Y: p.Y - 1}, {X: p.X, Y: p.Y + 1},
						{X: p.X - 1, Y: p.Y}, {X: p.X + 1, Y: p.Y},
					}
					for _, n := range neighbors {
						if n.X < 0 || n.X >= m.Width || n.Y < 0 || n.Y >= m.Height { 
							continue 
						}
						nIdx := n.Y*m.Width + n.X
						nTile := m.GetTile(n.X, n.Y)
						if nTile != nil && nTile.Type == TileTypeFloor && !visited[nIdx] && !isDoor[nIdx] {
							visited[nIdx] = true
							queue = append(queue, n)
						}
					}
				}
				if (maxX-minX) >= 1 && (maxY-minY) >= 1 {
					m.Rooms = append(m.Rooms, Rect{X1: minX, Y1: minY, X2: maxX, Y2: maxY})
				}
			}
		}
	}
}
{{< /code >}}

### Architectural Trade-off
Storing rooms as bounding boxes (`Rect`) enables fast `O(1)` point-in-rectangle collision checks. However, this assumption breaks for concave or L-shaped rooms, where a bounding box might overlap wall coordinates. For non-rectangular layouts, storing an explicit bitset of tile coordinates per room is necessary.

---

## Dynamic Sunlight Propagation

Skylight glass tiles (`IsSunlit`) act as primary light sources, receiving exterior ambient light calculated from the engine's time clock (`internal/world/clock.go`). To bleed light into dark corridors, we propagate sunlight into adjacent non-wall tiles using multiplicative decay.

{{< code lang="go" file="internal/world/map.go" >}}
// PropagateSunlight recursively bleeds daylight from sunlit tiles into adjacent spaces.
func (m *Map) PropagateSunlight(x, y int, currentLight float32) {
    if currentLight < 0.1 {
        return
    }
    
    tile := m.GetTile(x, y)
    if tile == nil || tile.Type == TileTypeWall {
        return
    }
    
    // Only propagate if incoming light exceeds existing tile light value
    if currentLight > tile.DaylightFactor {
        tile.DaylightFactor = currentLight
        
        // 25% attenuation per grid step
        decay := currentLight * 0.75
        m.PropagateSunlight(x+1, y, decay)
        m.PropagateSunlight(x-1, y, decay)
        m.PropagateSunlight(x, y+1, decay)
        m.PropagateSunlight(x, y-1, decay)
    }
}
{{< /code >}}

### Termination Guard & Performance
The check `if currentLight > tile.DaylightFactor` serves as the recursion boundary guard. Because `decay` reduces intensity by 25% each step, `currentLight` decreases monotonically until it drops below `0.1` or hits a tile already lit by a brighter path.

If a door is closed (`TileTypeWall`), recursion halts immediately. Opening a door re-runs light propagation, allowing sunlight to flood dark hallways without recalculating global scene lighting.

---

## Modular Campaigns via JSON Manifests

Hardcoding map loading scripts makes content creation brittle. I decoupled narrative data and level ordering into structured JSON mission manifests (`internal/mission/mission.go`).

{{< code lang="json" file="assets/missions/sector_4_incident/mission.json" >}}
{
  "id": "sector_4_incident",
  "title": "Sector 4 Incident",
  "author": "Vikash Patel",
  "synopsis": "Investigate structural power failure in Sector 4 Research Labs.",
  "start_level": "level_01_surface",
  "levels": [
    { "id": "level_01_surface", "name": "Surface Entry", "file": "maps/level_01_surface.json" },
    { "id": "level_02_labs", "name": "Research Laboratories", "file": "maps/level_02_labs.json" }
  ]
}
{{< /code >}}

The `JSONMapLoader` reads map definitions, instantiating tiles, door entities, terminals (`T`), room generators (`g`), and stairs (`<` / `>`) directly into the ECS registry.

---

## 2D Positional Audio Attenuation

Visual feedback alone is insufficient for spatial awareness. Sound effects need volume falloff and stereo panning based on their relative distance to the player.

{{< details title="What is Audio Attenuation?" label="Glossary //" >}}
**Attenuation** just means volume drops off with distance. In code, it means scaling a sound's volume based on how far the noise is from the player. A terminal exploding next to you plays at 100% volume, while an alarm 20 tiles away in another corridor plays at 10% volume or gets muted entirely.
{{< /details >}}

`PlayPositionalSound` (`internal/audio/audio.go`) wraps Raylib's CGO audio bindings to calculate Euclidean distance and horizontal panning:

{{< code lang="go" file="internal/audio/audio.go" >}}
func PlayPositionalSound(sound rl.Sound, playerX, playerY, sourceX, sourceY int, maxRange float32) {
    dx := float32(sourceX - playerX)
    dy := float32(sourceY - playerY)
    distance := float32(math.Sqrt(float64(dx*dx + dy*dy)))
    
    if distance > maxRange {
        return
    }
    
    // Linear volume attenuation
    volume := 1.0 - (distance / maxRange)
    rl.SetSoundVolume(sound, volume)
    
    // Stereo panning based on relative horizontal offset (-1.0 left, 1.0 right)
    pan := 0.5 + (dx / (maxRange * 2.0))
    if pan < 0.0 { pan = 0.0 }
    if pan > 1.0 { pan = 1.0 }
    rl.SetSoundPan(sound, pan)
    
    rl.PlaySound(sound)
}
{{< /code >}}

### Audio Design Choices
For a 2D tile-based engine, linear volume falloff (`1.0 - distance/maxRange`) provides predictable player feedback compared to inverse-square log attenuation, which can drop off too abruptly in constrained indoor hallways.

---

## Summary

Instead of relying on heavy third-party scene graphs, spatial logic in *Derelict Facility* relies on lightweight grid operations:

- **BFS Flood Fill**: Clusters raw floor coordinates into room bounding boxes for fast spatial queries.
- **Recursive Attenuation**: Bleeds sunlight dynamically through open doorways until light intensity drops below `0.1`.
- **JSON Level Pipelines**: Decouples campaign structure from core engine code.
- **Euclidean Audio Panning**: Maps 2D relative tile distances directly to Raylib stereo volume channels.


---

## Designing a Distributed Job Scheduler in Go: Partitioning, Locking, and Backpressure

**Date:** 2026-08-09
**URL:** http://localhost:1313/designing-a-distributed-job-scheduler/
**Description:** A complete system design of a fault-tolerant distributed job scheduler built for scale. Covers PostgreSQL row locking with SKIP LOCKED, hash ring shard distribution, cron parsing, worker backpressure, and missed job recovery.


Linux `crontab` is one of the most elegant pieces of software ever written for single-host automation. It is simple, clear, and has kept Unix systems running reliably since 1975.

The problem starts when we take a single-host tool and deploy it across a multi-node cloud setup.

In [Relay](/how-multi-tenant-saas-works/), a multi-tenant AI API gateway system design, background jobs power core operations: every top of the hour, a job rolls up raw API usage tokens into tenant billing metrics; every 15 minutes, another job scans for expired API keys and purges them from cache; every 30 seconds, a health checker pings upstream LLM provider endpoints.

When an application runs on a single server, `crontab` works perfectly. But when running a distributed gateway across multiple application nodes and availability zones, running uncoordinated local cron loops causes immediate failures:

1. **Duplicate Execution**: If three nodes run the same cron schedule, all three wake up at 00:00:00 UTC, pull the same billing job, and bill tenants three times for the exact same usage.
2. **Thundering Herds**: Dozens of nodes running `SELECT * FROM jobs WHERE next_run_at <= NOW()` at the exact same millisecond exhaust database connection pools and cause lock wait timeouts.

### Why This Post Exists (The Central Idea)

When teams face this challenge, engineering advice usually splits into two extremes:

- **The Overkill Path**: Deploy heavy external distributed orchestrators (like Temporal, Airflow, or a ZooKeeper/Etcd cluster). This adds complex setup, new infrastructure dependencies, and steep learning curves for teams that simply need reliable scheduled tasks.
- **The Naïve Path**: Write custom Redis key polling or uncoordinated database loops that suffer from thundering herds, race conditions, and lost jobs when nodes crash.

**The thesis of this post is that there is a pragmatic middle ground**: you can build a zero-dependency, fault-tolerant, horizontally scalable distributed scheduler using two building blocks already in your stack: **PostgreSQL (`SKIP LOCKED` + partial indexes)** and **Go (`chan struct{}` counting semaphores)**.

We will design **Pulse**, a distributed job scheduling engine built for Relay. This post breaks down Pulse step by step: defining what a job scheduler is, what "distributed" means in this context, how PostgreSQL `SKIP LOCKED` guarantees atomic locks, how hash ring sharding avoids central bottlenecks, and how to write production Go code that survives node crashes mid-execution.

---

## What Is a Distributed Job Scheduler?

Before looking at database queries or Go code, we need to clearly define the problem space.

### 1. Task Queue vs. Job Scheduler

Engineers often mix up task queues and job schedulers. They are two different tools:

- **Task Queue (Reactive / Push-Driven)**: Pushes work as fast as possible when triggered by an application event. An API endpoint receives a request, enqueues a message to RabbitMQ or SQS, and a worker consumes it immediately. The driver is an **external event**.
- **Job Scheduler (Temporal / Time-Driven)**: Evaluates schedules over time. It maintains schedule rules (such as cron expressions like `0 * * * *`), continuously monitors wall-clock time, computes target execution deadlines (`next_run_at`), and triggers work when current time reaches or exceeds `next_run_at`. The driver is **time itself**.

```
Task Queue:    [User Request] ---> (Enqueue) ---> [Queue] ---> (Execute Now)
Job Scheduler: [Cron Rule]     ---> (Clock Poll: current_time >= next_run_at) ---> [Dispatch] ---> (Execute)
```

### 2. What Does "Distributed" Mean for a Scheduler?

Running a single-node scheduler (like `robfig/cron` in a Go binary or a Linux `crontab`) is simple: one process owns the clock, the schedule list, and the worker threads in local memory.

A **distributed job scheduler** runs across `N` independent nodes (for example, 5 Go worker containers across 3 AWS availability zones). This creates three hard problems to solve:

1. **Clock Drift Across Nodes**: Physical server clocks are never perfectly in sync. Even with NTP (Network Time Protocol), clock skew between Node A and Node B typically ranges from 5ms to 50ms. If Node A's clock is 20ms ahead of Node B, Node A will see `current_time >= next_run_at` first. The system must work correctly no matter which node's clock triggers first.
2. **Preventing Double Work Without a Single Master Node**: If 10 scheduler nodes poll for runnable jobs, how do we guarantee that *exactly one node* claims job `J` for execution timestamp `T`, without routing every single request through a single master server?
3. **Handling Node Crashes and Lost Networks**: If Node A claims job `J`, sets its state to `running`, and immediately suffers an OOM crash, kernel panic, or network partition, the system must detect Node A's death, release the lock, and re-schedule job `J` on Node B without causing double execution.

---

## The Core Architecture of Pulse

To solve these problems, Pulse splits work across three main components:

```mermaid
flowchart TD
    subgraph Storage ["Database Layer (Shared State)"]
        DB[("PostgreSQL\n(Job Metadata & State)")]
    end

    subgraph Scheduler ["Pulse Scheduler Service (Cluster)"]
        S1["Scheduler Node 1\n(Shard 0 & 1)"]
        S2["Scheduler Node 2\n(Shard 2 & 3)"]
    end

    subgraph Workers ["Execution Worker Pool"]
        W1["Worker 1 (Billing)"]
        W2["Worker 2 (Cache Pruning)"]
        W3["Worker 3 (Health Check)"]
    end

    DB <-->|"SELECT FOR UPDATE\nSKIP LOCKED"| S1
    DB <-->|"SELECT FOR UPDATE\nSKIP LOCKED"| S2

    S1 -->|"Buffered Channel"| W1
    S1 -->|"Buffered Channel"| W2
    S2 -->|"Buffered Channel"| W3
```

1. **Job Store**: A relational database table storing schedule definitions, next execution timestamps, locks, and status flags. This serves as the single source of truth for time state.
2. **Poller / Scheduler Loop**: A cluster of stateless Go nodes that poll owned database shards, claim due jobs atomically, recalculate `next_run_at`, and dispatch payloads.
3. **Execution Workers**: Worker pools that run the actual business logic (calling webhooks, querying ClickHouse, purging caches) with strict context timeouts.

---

## Database Schema: Designing for Concurrent State

A distributed scheduler needs database storage that stays safe even if every server crashes. If all Pulse nodes crash at the same time, no scheduled jobs can be lost.

Here is the schema we will use for Pulse in PostgreSQL:

```sql
CREATE TYPE job_status AS ENUM ('idle', 'running', 'failed', 'disabled');

CREATE TABLE scheduled_jobs (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id       TEXT NOT NULL,
    name            TEXT NOT NULL,
    cron_expr       TEXT NOT NULL, -- e.g., '0 * * * *'
    payload         JSONB NOT NULL DEFAULT '{}'::jsonb,
    
    -- Execution State
    status          job_status NOT NULL DEFAULT 'idle',
    shard_id        INT NOT NULL DEFAULT 0,
    
    -- Scheduling Timestamps
    last_run_at     TIMESTAMPTZ,
    next_run_at     TIMESTAMPTZ NOT NULL,
    
    -- Lease & Lock Control
    locked_at       TIMESTAMPTZ,
    locked_by       TEXT,
    lease_timeout   INTERVAL NOT NULL DEFAULT INTERVAL '5 minutes',
    
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Critical Index for the Hot Polling Path
CREATE INDEX idx_scheduled_jobs_poll 
ON scheduled_jobs (shard_id, next_run_at) 
WHERE status = 'idle';
```

Notice the partial index definition: `WHERE status = 'idle'`. 

In a system with 500,000 scheduled jobs, only a tiny fraction (perhaps 50 to 500) will be in the `idle` status with `next_run_at <= NOW()` at any given second. By filtering the index specifically to `idle` rows, the index tree stays minimal in memory, keeping index lookups and fetch latency under 1ms.

---

## High-Throughput Polling with `SKIP LOCKED`

The biggest mistake developers make with a database scheduler is simple row locking:

```sql
-- DO NOT DO THIS IN PRODUCTION
SELECT * FROM scheduled_jobs 
WHERE next_run_at <= NOW() AND status = 'idle' 
FOR UPDATE;
```

When multiple scheduler nodes run this query at once, Node 1 locks matching rows. Node 2 and Node 3 do not skip those rows; instead, they **block and wait** for Node 1 to finish its transaction. This causes lock contention, context switching, and query timeouts.

PostgreSQL 9.5 introduced `FOR UPDATE SKIP LOCKED`. When a node executes a query with `SKIP LOCKED`, PostgreSQL instantly skips any rows currently locked by another transaction and returns only available rows.

Here is how Pulse claims a batch of executable jobs atomically:

```sql
WITH runnable_jobs AS (
    SELECT id 
    FROM scheduled_jobs
    WHERE shard_id = ANY($1) 
      AND status = 'idle' 
      AND next_run_at <= NOW()
    ORDER BY next_run_at ASC
    LIMIT $2
    FOR UPDATE SKIP LOCKED
)
UPDATE scheduled_jobs j
SET 
    status = 'running',
    locked_at = NOW(),
    locked_by = $3,
    updated_at = NOW()
FROM runnable_jobs r
WHERE j.id = r.id
RETURNING j.id, j.tenant_id, j.name, j.cron_expr, j.payload, j.next_run_at, j.shard_id;
```

When PostgreSQL evaluates a standard `SELECT ... FOR UPDATE` query:

1. It scans the table or index for matching tuples.
2. For every matching row, it inspects the tuple header (`xmin`/`xmax`) and the lock manager memory structure to see if an active transaction holds an exclusive row-level lock.
3. **Without `SKIP LOCKED`**: If Row 1 is locked by Worker Node A, Worker Node B is placed into a lock wait queue. Node B **blocks execution** until Node A commits or rolls back its transaction. If 10 scheduler nodes poll simultaneously, 9 nodes freeze until the leader finishes.
4. **With `SKIP LOCKED`**: When Node B sees Row 1 is locked by Node A, it **bypasses Row 1 immediately with zero wait time** and moves to Row 2. If Row 2 is unlocked, Node B claims Row 2 and returns it.

If all candidate rows in a batch are currently locked by other workers, `SKIP LOCKED` returns an **empty result set instantly (0ms latency)** rather than blocking. The poller cleanly finishes its tick and sleeps until the next interval.

This single atomic query gives us three important guarantees:
- **Zero Double-Execution**: Two pollers will never receive the same job ID.
- **Non-Blocking Operation**: Pollers do not wait on each other. If Node 1 claims jobs 1 through 10, Node 2 instantly receives jobs 11 through 20.
- **Minimal Transaction Window**: The lock is held only for the microsecond duration of the batch `UPDATE` statement.

---

## Shard Partitioning Across Scheduler Replica Nodes

While `SKIP LOCKED` stops lock waiting, having 20 nodes scan the full table over and over still wastes database CPU.

We can partition jobs across nodes using a **Shard Hash Ring**.

```mermaid
graph LR
    J1[Job: Usage Rollup<br/>Tenant A] -->|Hash ID % 4| S0[Shard 0]
    J2[Job: Key Cleanup<br/>Tenant B] -->|Hash ID % 4| S1[Shard 1]
    J3[Job: Health Check<br/>Tenant C] -->|Hash ID % 4| S2[Shard 2]
    J4[Job: Invoice Sync<br/>Tenant D] -->|Hash ID % 4| S3[Shard 3]

    S0 --> NodeA[Scheduler Node A]
    S1 --> NodeA
    S2 --> NodeB[Scheduler Node B]
    S3 --> NodeB
```

Every job is assigned a `shard_id` upon creation (`hash(job_id) % total_shards`).

When a Pulse node boots up, it registers itself with a coordinator (or claims a set of integer shard IDs via advisory locks in PostgreSQL). Node A polls shards `[0, 1]`, while Node B polls shards `[2, 3]`.

This design reduces database query overhead from `O(N)` pollers scanning everything to `O(1)` owned shard lookups per node.

---

## Building the Scheduler Engine in Go

Let us translate these principles into a clean, complete Go implementation.

### The Cron Parser & Next Run Resolver

Pulse relies on cron parsing to update a job's `next_run_at` timestamp immediately after claiming it.

```go
package scheduler

import (
	"context"
	"database/sql"
	"encoding/json"
	"fmt"
	"log"
	"sync"
	"time"

	"github.com/robfig/cron/v3"
)

// Job represents a scheduled task stored in the database.
type Job struct {
	ID          string          `json:"id"`
	TenantID    string          `json:"tenant_id"`
	Name        string          `json:"name"`
	CronExpr    string          `json:"cron_expr"`
	Payload     json.RawMessage `json:"payload"`
	NextRunAt   time.Time       `json:"next_run_at"`
	ShardID     int             `json:"shard_id"`
}

// Scheduler Engine orchestrates polling and dispatching.
type Engine struct {
	db         *sql.DB
	nodeID     string
	cronParser cron.Parser
	workerSem  chan struct{} // Semaphore for concurrency backpressure
	wg         sync.WaitGroup
	quit       chan struct{}
}

func NewEngine(db *sql.DB, nodeID string, maxConcurrentWorkers int) *Engine {
	return &Engine{
		db:         db,
		nodeID:     nodeID,
		cronParser: cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow),
		workerSem:  make(chan struct{}, maxConcurrentWorkers),
		quit:       make(chan struct{}),
	}
}
```

### The Atomic Polling and Claim Loop

Here is how the engine polls owned shards and dispatches jobs to workers with backpressure:

```go
// ClaimBatch fetches and locks pending jobs for owned shards using SKIP LOCKED.
func (e *Engine) ClaimBatch(ctx context.Context, shardIDs []int, batchSize int) ([]Job, error) {
	tx, err := e.db.BeginTx(ctx, nil)
	if err != nil {
		return nil, fmt.Errorf("begin tx: %w", err)
	}
	defer tx.Rollback()

	query := `
		WITH runnable AS (
			SELECT id 
			FROM scheduled_jobs
			WHERE shard_id = ANY($1) 
			  AND status = 'idle' 
			  AND next_run_at <= NOW()
			ORDER BY next_run_at ASC
			LIMIT $2
			FOR UPDATE SKIP LOCKED
		)
		UPDATE scheduled_jobs j
		SET 
			status = 'running',
			locked_at = NOW(),
			locked_by = $3,
			updated_at = NOW()
		FROM runnable r
		WHERE j.id = r.id
		RETURNING j.id, j.tenant_id, j.name, j.cron_expr, j.payload, j.next_run_at, j.shard_id;
	`

	rows, err := tx.QueryContext(ctx, query, shardIDs, batchSize, e.nodeID)
	if err != nil {
		return nil, fmt.Errorf("query claim batch: %w", err)
	}
	defer rows.Close()

	var jobs []Job
	for rows.Next() {
		var j Job
		if err := rows.Scan(&j.ID, &j.TenantID, &j.Name, &j.CronExpr, &j.Payload, &j.NextRunAt, &j.ShardID); err != nil {
			return nil, fmt.Errorf("scan job: %w", err)
		}
		jobs = append(jobs, j)
	}

	if err := tx.Commit(); err != nil {
		return nil, fmt.Errorf("commit tx: %w", err)
	}

	return jobs, nil
}
```

### Worker Dispatch with Backpressure

When a node claims 50 jobs, sending all 50 to memory simultaneously can crash the node if memory or CPU is tight. We enforce worker concurrency using a Go channel semaphore:

```go
func (e *Engine) Start(ctx context.Context, shardIDs []int, pollInterval time.Duration) {
	ticker := time.NewTicker(pollInterval)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			log.Println("[Pulse] Shutdown signal received. Waiting for active workers...")
			e.wg.Wait()
			return
		case <-ticker.C:
			jobs, err := e.ClaimBatch(ctx, shardIDs, 25)
			if err != nil {
				log.Printf("[Pulse] Error claiming batch: %v", err)
				continue
			}

			for _, job := range jobs {
				// Acquire semaphore slot with context cancellation support (Backpressure)
				select {
				case e.workerSem <- struct{}{}:
				case <-ctx.Done():
					log.Println("[Pulse] Context cancelled while acquiring worker semaphore slot")
					e.wg.Wait()
					return
				}

				e.wg.Add(1)
				go func(j Job) {
					defer func() {
						<-e.workerSem
						e.wg.Done()
					}()
					e.executeJob(ctx, j)
				}(job)
			}
		}
	}
}
```

In Go, `chan struct{}` with a fixed buffer capacity (e.g., `make(chan struct{}, 10)`) acts as a zero-allocation counting semaphore:

1. **Acquiring a slot**: `select { case workerSem <- struct{}{}: default: ... }` sends an empty struct into the channel buffer. If the buffer is full (10 active goroutines), this operation **blocks the polling loop**, preventing new goroutines from being spawned until a slot is freed.
2. **Releasing a slot**: Inside `defer func() { <-workerSem }()`, reading from the channel frees a slot in the buffer, allowing the main loop to proceed and process the next job.

This pattern enforces **backpressure**: the scheduler automatically slows down batch claiming if existing jobs take longer to process than expected, preventing OOM (Out Of Memory) crashes.

### Updating Next Execution & Releasing Locks

Once execution succeeds, Pulse schedules the next run based on its cron expression and transitions status back to `idle`:

```go
func (e *Engine) executeJob(parentCtx context.Context, j Job) {
	ctx, cancel := context.WithTimeout(parentCtx, 2*time.Minute)
	defer cancel()

	// 1. Perform actual business work
	execErr := e.runHandler(ctx, j)

	// 2. Calculate next schedule run timestamp relative to completion time
	sched, err := e.cronParser.Parse(j.CronExpr)
	if err != nil {
		log.Printf("[Pulse] Invalid cron expression for job %s: %v", j.ID, err)
		e.markFailed(parentCtx, j.ID, err.Error())
		return
	}
	nextRun := sched.Next(time.Now())

	// 3. Update job record state atomically
	if execErr != nil {
		log.Printf("[Pulse] Job %s failed: %v", j.ID, execErr)
		e.markFailed(parentCtx, j.ID, execErr.Error())
		return
	}

	query := `
		UPDATE scheduled_jobs
		SET 
			status = 'idle',
			last_run_at = NOW(),
			next_run_at = $1,
			locked_at = NULL,
			locked_by = NULL,
			updated_at = NOW()
		WHERE id = $2;
	`
	if _, err := e.db.ExecContext(parentCtx, query, nextRun, j.ID); err != nil {
		log.Printf("[Pulse] Error completing job %s: %v", j.ID, err)
	}
}

func (e *Engine) runHandler(ctx context.Context, j Job) error {
	// Dispatch based on job payload name (e.g. usage-rollup, key-cleanup)
	timer := time.NewTimer(150 * time.Millisecond)
	defer timer.Stop()

	select {
	case <-ctx.Done():
		return ctx.Err()
	case <-timer.C:
		return nil
	}
}

func (e *Engine) markFailed(ctx context.Context, jobID string, reason string) {
	query := `
		UPDATE scheduled_jobs
		SET 
			status = 'failed',
			locked_at = NULL,
			locked_by = NULL,
			updated_at = NOW()
		WHERE id = $1;
	`
	if _, err := e.db.ExecContext(ctx, query, jobID); err != nil {
		log.Printf("[Pulse] Error marking job %s as failed: %v", jobID, err)
	}
}
```

---

## Surviving Edge Cases and Disasters

Systems break in unexpected ways. Here is how to handle the most common failures:

### Case 1: The Scheduler Node Crashes Mid-Job

Suppose Node A claims a billing aggregation job, sets `status = 'running'`, and starts execution. Mid-way through, Node A suffers an OOM (Out Of Memory) crash. 

The job is left stuck in `status = 'running'` forever unless cleaned up.

**Solution: Dead-Man Lease Sweeper**
We can run a lightweight background sweeper query every 60 seconds on the leader node:

```sql
-- Recover jobs whose lease expired (node died mid-execution)
UPDATE scheduled_jobs
SET 
    status = 'idle',
    locked_at = NULL,
    locked_by = NULL,
    updated_at = NOW()
WHERE status = 'running' 
  AND locked_at < NOW() - lease_timeout;
```

If Node A dies, its held jobs are reset back to `idle` after `lease_timeout` (e.g., 5 minutes) and automatically picked up by surviving nodes.

### Case 2: What Happens During System Downtime? (Missed Runs)

If the scheduling cluster goes down for 2 hours during a database upgrade, 120 minute-interval jobs will be past their `next_run_at` time when nodes boot back up.

Do you execute all 120 missed runs sequentially (Backfill), or do you skip straight to the current time?

```mermaid
stateDiagram-v2
    [*] --> CheckMissed: Scheduler Boots Up
    CheckMissed --> EvaluatePolicy: next_run_at < NOW() - Threshold
    
    state EvaluatePolicy {
        [*] --> SkipPolicy: Catch-Up Policy = SKIP
        [*] --> BackfillPolicy: Catch-Up Policy = BACKFILL
        
        SkipPolicy --> ResetCurrent: Update next_run_at = NextCron(NOW())
        BackfillPolicy --> QueueAll: Fire missed execution loop
    }
```

In this system design, we configure this per job type:
- **Billing Aggregations**: Must **BACKFILL**. Missing a run means unbilled customer revenue.
- **Cache Pruning / Health Checks**: Must **SKIP**. Running 120 consecutive cache purges provides zero value and overloads your Redis cluster.

In Go, skip logic is simple:

```go
func CalculateNextRun(cronExpr string, lastNextRun time.Time, catchUpPolicy string) (time.Time, error) {
	sched, err := cron.ParseStandard(cronExpr)
	if err != nil {
		return time.Time{}, fmt.Errorf("parse cron expression: %w", err)
	}
	now := time.Now()

	if catchUpPolicy == "SKIP" && lastNextRun.Before(now) {
		// Advance next_run to the upcoming valid time slot relative to NOW
		return sched.Next(now), nil
	}

	// Advance relative to previous scheduled time (Backfill step-by-step)
	return sched.Next(lastNextRun), nil
}
```

---

## Summary & Architectural Checklist

Building a reliable distributed job scheduler means moving away from local timers and sharing state safely.

When building or reviewing your scheduler architecture, verify these core properties:

| Component | Bad Practice | Production Best Practice |
|---|---|---|
| **Concurrency** | `SELECT ... FOR UPDATE` (Blocks nodes) | `SELECT ... FOR UPDATE SKIP LOCKED` |
| **Node Partitioning** | All nodes poll the full DB table | Partition jobs into shards via hash ring |
| **Backpressure** | Unbounded goroutine spawning | Channel semaphore / Worker pools |
| **Crash Recovery** | Manual intervention on stuck jobs | Automatic lease timeouts (`locked_at < NOW() - lease`) |
| **Catch-Up Logic** | Re-executing all missed runs indiscriminately | Explicit per-job policy (`SKIP` vs `BACKFILL`) |

By pairing PostgreSQL's concurrency primitives with Go's lightweight channel mechanics, this architecture can execute millions of scheduled tasks across a multi-tenant gateway with zero duplicate runs and sub-millisecond polling latency.


---

## Part 5: From Terminal Cells to Sprite Maps: Font Fallbacks and Auto-Tiling

**Date:** 2026-08-09
**URL:** http://localhost:1313/autotiling-sprites-font-fallbacks-raylib/
**Description:** Building a 2D Raylib rendering pipeline capable of 4-bit cardinal autotiling and multi-font fallback chains for unicode glyphs and emojis.


Building a terminal-based engine in pure ASCII looks cool for about five minutes. Then you try rendering a complex facility map with corners, T-junctions, and status icons, and raw character cells start feeling incredibly limiting.

Two specific problems hit me immediately when I tried switching to Raylib for graphics. First, drawing wall tiles manually by hand in level files meant placing 16 different corner variations by hand. Second, when I tried rendering a 🚨 warning emoji alongside FiraCode monospace text, Raylib just rendered a missing glyph box (`?` or `□`).

To solve this, *Derelict Facility* uses a 4-bit cardinal autotiling algorithm alongside a cascading font fallback resolver in the Raylib display pipeline (`internal/display/raylib.go`).

---

## 4-Bit Cardinal Autotiling Mathematics

Drawing walls as uniform square blocks (`#`) breaks visual continuity in corridors. We want corners, T-junctions, and straight hallways to select their matching sprite tile automatically.

Instead of manual texture assignment, we evaluate each wall tile's 4 cardinal neighbors (North, East, South, West) and compile a 4-bit integer bitmask.

```mermaid
graph TD
    subgraph NeighborBitmask ["4-Bit Cardinal Bitmask Rules"]
        N["North: Bit 0 (Value 1)"]
        E["East: Bit 1 (Value 2)"]
        S["South: Bit 2 (Value 4)"]
        W["West: Bit 3 (Value 8)"]
    end

    NeighborBitmask --> Sum["Bitwise OR Sum (0 to 15)"]
    Sum --> SpriteMap["Tileset Source Rect (Bitmask * 16px, 0)"]
```

### Cardinal Neighbor Values
Each direction is mapped to a power-of-two bit index:
- **North (N)**: `1 << 0 = 1`
- **East (E)**: `1 << 1 = 2`
- **South (S)**: `1 << 2 = 4`
- **West (W)**: `1 << 3 = 8`

During level loading (`internal/world/json_loader.go`), we inspect adjacent tiles and construct the bitmask:

{{< code lang="go" file="internal/world/json_loader.go" >}}
func (l *JSONMapLoader) calculateWallBitmasks(m *Map) {
	for y := 0; y < m.Height; y++ {
		for x := 0; x < m.Width; x++ {
			tile := m.GetTile(x, y)
			if tile == nil || tile.Type != TileTypeWall {
				continue
			}

			var mask uint8 = 0
			if tN := m.GetTile(x, y-1); tN != nil && tN.Type == TileTypeWall { mask |= 1 }
			if tE := m.GetTile(x+1, y); tE != nil && tE.Type == TileTypeWall { mask |= 2 }
			if tS := m.GetTile(x, y+1); tS != nil && tS.Type == TileTypeWall { mask |= 4 }
			if tW := m.GetTile(x-1, y); tW != nil && tW.Type == TileTypeWall { mask |= 8 }

			tile.Bitmask = mask
		}
	}
}
{{< /code >}}

This produces a single byte value between `0` (an isolated wall column) and `15` (a 4-way intersection).

### Tileset Coordinate Mapping
The computed `Bitmask` maps directly to sprite sheet source X offsets:

```
Bitmask Value -> Visual Connection -> Sprite Sheet Offset X
0             -> Isolated Pillar    -> 0px
1             -> North Cap          -> 16px
3 (1+2)       -> N-E Corner         -> 48px
15 (1+2+4+8)  -> 4-Way Cross        -> 240px
```

### Autotiling Trade-offs (4-Bit vs. 8-Bit)
4-bit cardinal autotiling requires only 16 sprite variations, keeping tileset textures small. However, cardinal autotiling does not evaluate diagonal neighbors (NW, NE, SE, SW). If your game requires outer vs. inner corner transitions on thick 2D terrain, an 8-bit (47-tile Blob autotiling) lookup table is required instead.

---

## Cascading Font Fallback Chain

Standard font loading functions in graphics libraries bind a single TTF/OTF font file. When rendering terminal UI text mixed with unicode indicators or status emojis, missing glyphs trigger placeholder box rendering.

To fix this, we implement a fallback resolver (`internal/display/raylib.go`) that queries a prioritized chain of loaded font instances:

```mermaid
graph LR
    Rune["Target Unicode Rune"] --> CheckPrimary{"IsGlyphAvailable(PrimaryFont)?"}
    CheckPrimary -- Yes --> RenderPrimary["Render with FiraCode"]
    CheckPrimary -- No --> CheckEmoji{"IsGlyphAvailable(EmojiFont)?"}
    CheckEmoji -- Yes --> RenderEmoji["Render with Noto Emoji"]
    CheckEmoji -- No --> RenderSymbol["Render with Symbola"]
```

{{< code lang="go" file="internal/display/raylib.go" >}}
func (r *RaylibDisplay) getFontForGlyph(char rune) rl.Font {
    if rl.IsGlyphAvailable(r.PrimaryFont, char) {
        return r.PrimaryFont
    }
    
    if rl.IsGlyphAvailable(r.EmojiFont, char) {
        return r.EmojiFont
    }
    
    return r.SymbolaFont
}
{{< /code >}}

### Render Loop Integration
During draw passes, the display driver uses the calculated bitmask for sprite sheet clipping while routing text glyphs through the font fallback chain:

{{< code lang="go" file="internal/display/raylib.go" >}}
func (r *RaylibDisplay) DrawMapTile(gridX, gridY int, tile world.Tile) {
    pixelX := int32(gridX) * r.CellWidth
    pixelY := int32(gridY) * r.CellHeight
    
    if tile.Type == world.TileTypeWall {
        srcRect := rl.NewRectangle(float32(tile.Bitmask)*16, 0, 16, 16)
        destRect := rl.NewRectangle(float32(pixelX), float32(pixelY), float32(r.CellWidth), float32(r.CellHeight))
        rl.DrawTexturePro(r.Tileset, srcRect, destRect, rl.NewVector2(0,0), 0, rl.White)
    } else {
        rl.DrawRectangle(pixelX, pixelY, r.CellWidth, r.CellHeight, rl.GetColor(0x222222FF))
    }
}
{{< /code >}}

---

## Summary

Combining 4-bit cardinal autotiling bitmasks with cascading font fallback chains resolves two core 2D rendering bottlenecks:

- **Automated Autotiling**: Eliminates manual wall placement errors by compiling neighbor connections into a single `0 to 15` byte index.
- **Robust Text & Emoji Rendering**: Prevents missing glyph artifacts when mixing vector fonts with unicode emojis in game UI layers.


---

## Part 4: Refactoring to SoA ECS: Bitmasks and Flat Component Arrays

**Date:** 2026-08-08
**URL:** http://localhost:1313/structure-of-arrays-ecs-bitmasks-golang/
**Description:** Refactoring entity storage from pointer-heavy OOP structs to a Structure of Arrays (SoA) ECS using bitwise masks for entity identity and zero-allocation system loops.


When I started building *Derelict Facility*, my initial instinct for game actors was standard Object-Oriented design: create an `Entity` struct, add pointers for position, sprite, and stats, and store them in a slice (`[]*Entity`).

It worked fine for five entities. But as soon as I added automated doors, save terminals, and active power grids across a 1000-tile map, keeping track of separate heap pointers became a headache. My Go profiler showed GC pauses spiking while the CPU spent more time chasing heap pointers across non-contiguous memory than actually updating game state.

To fix this, I refactored the engine to a data-oriented **Structure of Arrays (SoA) Entity-Component System (ECS)** driven by bitmask indexing.

---

## AoS vs. SoA Memory Layout

The diagram below compares how CPU cache lines load data in an Array of Structures (AoS) layout versus a Structure of Arrays (SoA) layout:

```mermaid
graph TD
    subgraph AoS ["Array of Structures (AoS) - Pointer Chasing"]
        E0["Entity 0: [ID | Position | Sprite | Walkable]"] --> E1["Entity 1: [ID | Position | Sprite | Walkable]"]
        E1 --> E2["Entity 2: [ID | Position | Sprite | Walkable]"]
    end

    subgraph SoA ["Structure of Arrays (SoA) - Cache Line Friendly"]
        Masks["Masks Array: [M0, M1, M2, ...]"]
        Positions["Positions Array: [P0, P1, P2, ...] (Contiguous)"]
        Sprites["Sprites Array: [S0, S1, S2, ...]"]
    end
```

In the classic Array of Structures (AoS) approach, each entity holds all its fields:

{{< code lang="go" >}}
// Array of Structures (AoS)
type Entity struct {
    ID       uint32
    Position Point
    Render   Sprite
    Walkable bool
}

var Entities []Entity
{{< /code >}}

When the movement system loops through `Entities` to update coordinates, the CPU loads the entire `Entity` struct (including rendering and collision flags) into 64-byte L1 cache lines. You end up wasting up to 75% of your cache line bandwidth on data the movement loop never reads.

In a Structure of Arrays (SoA) layout, an entity is just an integer index (`uint32`). The actual component data lives in separate, flat, parallel slices:

{{< code lang="go" >}}
// Structure of Arrays (SoA)
type World struct {
    Masks     [1000]uint32
    Positions [1000]Position
    Sprites   [1000]Sprite
}
{{< /code >}}

Now, when the movement system runs, it iterates exclusively over the flat `Positions` slice. The CPU hardware prefetcher streams contiguous `Position` memory sequentially into cache without loading unused render state.

---

## Entity Identity: Component Bitmasks

How do we know which components are attached to entity index 5 without doing expensive map lookups? We use bitwise masks (`internal/components/components.go`).

{{< code lang="go" file="internal/components/components.go" >}}
type ComponentMask uint32

const (
	MaskNone         ComponentMask = 0
	MaskPosition     ComponentMask = 1 << iota
	MaskSprite
	MaskPlayerControl
	MaskGlyph
	MaskSolid
	MaskInteractable
)
{{< /code >}}

An entity's active composition is set using bitwise `OR`:

```go
// Entity 5 has Position and Sprite components
Masks[5] = MaskPosition | MaskSprite // binary: 00000011
```

When a system wants to process controllable entities with a position, it queries the mask with a single bitwise `AND` check:

```go
required := MaskPosition | MaskPlayerControl

// Returns true if entity 5 has both required components
hasComponents := (Masks[e] & required) == required
```

This translates to a single CPU bitwise instruction.

---

## World Registry Implementation

The registry (`internal/ecs/world.go`) manages these pre-allocated component arrays. Instead of dynamic slice reallocations on the fly, I capped entity capacity to a fixed array limit (1000 entities) initialized on boot.

{{< code lang="go" file="internal/ecs/world.go" >}}
type Entity uint32

type World struct {
	NextEntityID Entity
	FreeEntities []Entity
	Masks        [1000]components.ComponentMask
	Positions    [1000]components.Position
	Sprites      [1000]components.Sprite
}

func (w *World) CreateEntity() Entity {
	var id Entity
	if len(w.FreeEntities) > 0 {
		id = w.FreeEntities[len(w.FreeEntities)-1]
		w.FreeEntities = w.FreeEntities[:len(w.FreeEntities)-1]
	} else {
		id = w.NextEntityID
		w.NextEntityID++
	}
	w.Masks[id] = components.MaskNone
	return id
}

func (w *World) AddPosition(e Entity, pos components.Position) {
	w.Positions[e] = pos
	w.Masks[e] |= components.MaskPosition
}

func (w *World) DestroyEntity(e Entity) {
	w.Masks[e] = components.MaskNone
	w.FreeEntities = append(w.FreeEntities, e)
}
{{< /code >}}

When `DestroyEntity` is called, we don't zero out or reallocate array data. Unsetting `w.Masks[e] = MaskNone` tells systems to skip that index immediately. The ID is pushed onto `FreeEntities` for `O(1)` reuse on the next `CreateEntity` call.

---

## Zero-Allocation System Loops

In standard Go code, returning slices of matching entity IDs inside a system loop creates garbage collection pressure on every single frame.

Operating directly on `World` memory avoids allocations entirely:

{{< code lang="go" file="internal/systems/input.go" >}}
func UpdateMovement(w *ecs.World, dx, dy int) {
    required := components.MaskPosition | components.MaskPlayerControl
    
    for i := 0; i < int(w.NextEntityID); i++ {
        mask := w.Masks[i]
        if (mask & required) == required {
            w.Positions[i].X += dx
            w.Positions[i].Y += dy
        }
    }
}
{{< /code >}}

Because this loop reads directly from fixed memory slices without allocating temporary objects, runtime allocations remain at **0 B/op**.

---

## The Practical Trade-offs

Choosing a simple pre-allocated flat array ECS comes with explicit trade-offs:

- **Upfront RAM Footprint**: Pre-allocating `[1000]Position` reserves memory up front even when only 10 entities are active. At ~120 KB total overhead for 1000 entities, this is a tiny price to pay to eliminate GC pauses.
- **Hard Entity Limits**: If entity count exceeds 1000, the array bounds check will panic. For a grid simulation with known level bounds, this cap keeps the design clean and predictable.


---

## Designing a Usage-Based Billing Pipeline for SaaS

**Date:** 2026-08-01
**URL:** http://localhost:1313/designing-a-usage-based-billing-pipeline-for-saas/
**Description:** A deep technical guide to designing a robust, provider-agnostic billing pipeline for usage-based SaaS. Covers subscription state machines, idempotent event processing, automated dunning logic, credit notes, and metered invoice generation with full SQL schemas, Mermaid diagrams, and Go snippets.


In [How Multi-Tenant SaaS Actually Works](/posts/2026-07-01-how-multi-tenant-saas-works/), I left one major promise unfulfilled at the end: *"Billing is an entire system design on its own. The billing pipeline gets its own post."*

This is that post.

Building a billing pipeline is different from building almost any other backend component. If your caching layer has a bug, latencies increase by 50ms. If your log aggregator drops a packet, a dashboard chart dips for a minute. But if your billing pipeline has a bug, it operates silently in the dark. It either undercharges your customers (and your business bleeds cash without throwing a single 500 status code) or overcharges them (and you wake up to customer support tickets, disputes, and ruined trust).

Relay pays upstream LLM providers per token, marks up the cost by 20%, and bills tenants monthly based on their total usage. In this post, we will design the complete billing system that makes this work reliably.

Whether you use Stripe, Razorpay, Paddle, PayU, or PayPal, the fundamental engineering problems are identical. They all emit webhooks, they all encounter payment failures, and they all model subscriptions using the same underlying mechanics. We are going to design this system using clean, provider-agnostic architecture.

---

## How Relay's Billing Model Works

Before diving into webhooks and database schemas, let us lay out how data moves through Relay's billing pipeline across three distinct layers:

1. **Real-Time Cost Tracking**: Every API request passing through Relay logs token usage and cost calculation into the `requests` table.
2. **Asynchronous Aggregation**: A background worker continuously aggregates raw request data into the `usage_daily` rollup table.
3. **Monthly Metered Invoicing**: At the end of a billing cycle, Relay's billing worker queries the daily usage, compiles invoice line items, and submits them to the payment gateway.

```mermaid
graph TB
    RELAY[Relay API Gateway] --> CUST["Gateway Customer\n(Created per tenant at signup)"]
    CUST --> SUB["Subscription\n(Monthly cycle)"]
    SUB --> ITEM1["Item: OpenAI Token Usage"]
    SUB --> ITEM2["Item: Anthropic Token Usage"]
    ITEM1 --> UR["Usage Reports\n(Flushed hourly by worker)"]
    ITEM2 --> UR
    CUST --> INV["Invoice\n(Generated at period end)"]
    INV --> PAY["Payment Attempt\n(Automatic charge)"]
```

### Local Database State vs Gateway State

The payment gateway is always the source of truth for payment status, customer card details, and invoice settlements. Relay maintains a local replica in Postgres to serve API authorization checks and dashboard views without making remote HTTP calls on every incoming API request.

```sql
CREATE TABLE tenant_billing (
    tenant_id               TEXT PRIMARY KEY REFERENCES tenants(id),
    gateway                 TEXT NOT NULL,              -- stripe | razorpay | paddle | payu
    gateway_customer_id     TEXT NOT NULL UNIQUE,       -- cus_... | cust_... | ctm_...
    gateway_subscription_id TEXT,                       -- sub_... | null until subscribed
    subscription_status     TEXT,                       -- active | past_due | canceled | ...
    current_period_start    TIMESTAMPTZ,
    current_period_end      TIMESTAMPTZ,
    created_at              TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at              TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE tenant_subscription_items (
    tenant_id               TEXT NOT NULL REFERENCES tenants(id),
    model_family            TEXT NOT NULL,              -- openai_gpt4 | anthropic_claude | ...
    gateway_item_id         TEXT NOT NULL,              -- subscription item ID from gateway
    gateway_price_id        TEXT NOT NULL,
    updated_at              TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, model_family)
);
```

Always create the gateway customer object eagerly during tenant provisioning rather than lazily when they add a card. Every downstream flow (adding payment methods, applying credits, emitting metered usage) assumes a valid `gateway_customer_id` exists. Handling lazy customer creation forces conditional logic across dozens of handlers, creating unnecessary surface area for bugs.

---

## Subscription Lifecycle Management

Payment gateways model recurring billing using finite state machines. While vendor documentation uses different terms for these states, their operational behavior is identical.

```mermaid
stateDiagram-v2
    [*] --> pending : Subscription created (payment pending)
    pending --> active : First payment succeeded
    pending --> expired : Payment not completed within gateway window
    active --> past_due : Renewal payment fails
    past_due --> active : Retry payment succeeds
    past_due --> suspended : All retries exhausted
    active --> canceled : Tenant cancels
    suspended --> active : Payment method updated + retried
    suspended --> canceled : Grace period expires
    expired --> [*]
    canceled --> [*]
```

Here is how common gateway states map to Relay's internal subscription model:

| Operational State | Stripe | Razorpay | Paddle | PayU | Relay Action |
|---|---|---|---|---|---|
| **Pending** | `incomplete` | `created` | `trialing` | `PENDING` | Allow trial limits; prompt for card |
| **Active** | `active` | `authenticated` | `active` | `ACTIVE` | Normal API access |
| **Renewal Failing** | `past_due` | `halted` | `past_due` | `PAUSED` | Grace period; send dunning notices |
| **Exhausted** | `unpaid` | `cancelled` | `canceled` | `FAILED` | Suspend access (HTTP 402 on API calls) |

### Subscription Creation and Initial Payment

When a customer signs up for a paid plan, Relay creates a subscription object at the gateway.

1. **Synchronous Card Verification**: For traditional card payments without extra verification, the initial charge processes immediately. The subscription transitions directly from `pending` to `active`.
2. **Asynchronous 3D Secure / SCA**: Under regulations like PSD2 in Europe or RBI guidelines in India, payments frequently require multi-factor authentication (3D Secure). The gateway returns a pending status alongside a redirect URL. Relay keeps the subscription state as `pending` until the customer completes authentication on their bank's portal.

### Renewal Cycles and Invoice Draft Windows

At the end of a billing cycle, the gateway automatically generates a renewal invoice. 

Most payment gateways provide a temporary draft window (typically 1 hour) between invoice creation and charge execution. This draft window acts as an operational buffer. If Relay's hourly usage worker experienced a transient delay near midnight, the system uses the `invoice.created` webhook event to perform a final reconciliation pass, injecting any missing line items before the gateway finalizes the bill.

---

## Idempotent Webhook Processing

Webhooks are delivered over public networks using **at-least-once delivery guarantees**. This means your system *will* receive identical events multiple times, and events *will* occasionally arrive out of sequence.

A webhook processing pipeline must obey three strict engineering rules:

1. **Verify signature and enqueue instantly**: Validate signatures against raw payload bytes, insert the event ID into a storage table, enqueue a worker task, and return HTTP 200 immediately.
2. **Deduplicate atomically**: Use database constraints to prevent double-processing.
3. **Re-fetch live state**: Do not rely on event payload snapshots. Fetch the latest state directly from the gateway API before applying changes.

```mermaid
sequenceDiagram
    participant Gateway as Payment Gateway
    participant Endpoint as API Endpoint (/webhooks)
    participant DB as Postgres DB
    participant Queue as Worker Queue
    participant Worker as Background Worker

    Gateway->>Endpoint: POST /v1/webhooks/billing (Raw Body + Signature)
    Endpoint->>Endpoint: Verify HMAC Signature on Raw Bytes
    Endpoint->>Queue: Enqueue Event Processing Job
    Endpoint-->>Gateway: HTTP 200 OK (Fast Ack < 100ms)

    Queue->>Worker: Dequeue Job (Event ID)
    Worker->>DB: INSERT INTO webhook_events ON CONFLICT DO NOTHING
    alt Event Already Exists (RowsAffected == 0)
        Worker-->>Queue: Acknowledge Job (Skip duplicate)
    else New Event
        Worker->>Gateway: GET /v1/subscriptions/{id} (Fetch Live State)
        Gateway-->>Worker: Return Current Subscription Object
        Worker->>DB: Update Local Tenant State in Transaction
        Worker->>DB: UPDATE webhook_events SET status = 'done'
        Worker-->>Queue: Acknowledge Job Complete
    end
```

### Raw Signature Verification

Always verify HMAC signatures on the raw byte stream from the HTTP request body. Never parse JSON first and verify second; JSON parsers do not preserve key ordering or whitespace formatting, which invalidates the computed signature digest.

```go
package billing

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"net/http"
)

type GatewayEvent struct {
	ID         string
	Type       string
	ObjectType string
	ObjectID   string
	RawPayload []byte
}

type PaymentGateway interface {
	Name() string
	VerifyWebhook(body []byte, headers http.Header) (GatewayEvent, error)
	FetchEvent(ctx context.Context, eventID string) (GatewayEvent, error)
	GetSubscription(ctx context.Context, subID string) (Subscription, error)
	GetInvoice(ctx context.Context, invoiceID string) (Invoice, error)
	ReportUsage(ctx context.Context, record UsageRecord) error
	CreateCreditNote(ctx context.Context, req CreditNoteRequest) (string, error)
}

func VerifyHMACSignature(rawBody []byte, signatureHeader, secret string) bool {
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write(rawBody)
	expectedMAC := hex.EncodeToString(mac.Sum(nil))
	return hmac.Equal([]byte(signatureHeader), []byte(expectedMAC))
}
```

### Atomic Database Deduplication

We track processed events using a dedicated database table. Using PostgreSQL's `ON CONFLICT (event_id) DO NOTHING`, we guard against concurrent or repeated event processing inside a single database transaction.

```sql
CREATE TABLE webhook_events (
    event_id     TEXT PRIMARY KEY,
    gateway      TEXT NOT NULL,
    event_type   TEXT NOT NULL,
    status       TEXT NOT NULL DEFAULT 'pending', -- pending | done | failed
    payload      JSONB,
    processed_at TIMESTAMPTZ
);
```

```go
func (w *BillingWorker) ProcessEvent(ctx context.Context, eventID, eventType string, payload []byte) error {
	return w.db.WithTx(ctx, func(tx *sql.Tx) error {
		res, err := tx.ExecContext(ctx, `
			INSERT INTO webhook_events (event_id, gateway, event_type, status, payload)
			VALUES ($1, $2, $3, 'pending', $4)
			ON CONFLICT (event_id) DO NOTHING`,
			eventID, w.gateway.Name(), eventType, payload)
		if err != nil {
			return err
		}

		rows, err := res.RowsAffected()
		if err != nil {
			return err
		}
		if rows == 0 {
			// Event already processed by another worker; exit safely.
			return nil
		}

		// Re-fetch the live object from the gateway API to handle out-of-order events.
		event, err := w.gateway.FetchEvent(ctx, eventID)
		if err != nil {
			return err
		}

		if err := w.applyEvent(ctx, tx, event); err != nil {
			return err // Rollback transaction so retry can attempt later
		}

		_, err = tx.ExecContext(ctx,
			`UPDATE webhook_events SET status = 'done', processed_at = now() WHERE event_id = $1`,
			eventID)
		return err
	})
}
```

If your business logic fails, the database transaction rolls back, reverting both the tenant state update and the `webhook_events` insert. When the gateway retries the webhook delivery later, the worker can safely try processing it again.

---

## Dunning Logic and Failed Payment Handling

Dunning is the automated process of managing failed payments. Involuntary churn (failures caused by expired cards, temporary bank holds, or insufficient funds) accounts for a large percentage of revenue loss in SaaS.

Dunning consists of two synchronized layers:

1. **Technical Dunning**: Automated retry schedules managed by the payment gateway.
2. **Communication & Access Dunning**: In-app banners, email notifications, feature restrictions, and eventual account suspension managed by Relay.

```mermaid
flowchart TD
    A[Webhook: payment.failed] --> B{Decline Classification}
    B -->|Soft Decline: Insufficient Funds| C[Keep API Active\nSchedule Retry]
    B -->|Hard Decline: Stolen / Invalid Card| D[Immediate Access Restriction]
    
    C --> E{Attempt Count}
    E -->|Attempt 1| F[Send Email: Card Retry Notice]
    E -->|Attempt 2| G[In-App Dashboard Warning Banner]
    E -->|Attempt 3| H[Restrict New API Key Generation]
    E -->|Final Attempt Failed| I[Update Tenant Status to SUSPENDED\nReturn HTTP 402 on Gateway Requests]

    D --> I
```

### Soft vs Hard Declines

Payment declines fall into two broad categories:

- **Soft Declines** (`insufficient_funds`, `network_timeout`, `processing_error`): These indicate temporary issues. The gateway retries the charge automatically over a multi-day schedule. Relay grants a grace period during soft declines.
- **Hard Declines** (`stolen_card`, `lost_card`, `invalid_account`): These failures are permanent. Retrying will not succeed. Relay immediately prompts the customer for a new payment method and restricts administrative operations.

### Graduated Grace Period Implementation

Relay handles dunning progressively without immediately shutting down a customer's production traffic on the first payment failure:

```go
func (w *BillingWorker) OnPaymentFailed(ctx context.Context, tx *sql.Tx, payment FailedPayment) error {
	tenant, err := w.tenants.GetByGatewayCustomer(ctx, payment.CustomerID)
	if err != nil {
		return err
	}

	mrr, err := w.billing.GetMRR(ctx, tenant.ID)
	if err != nil {
		return err
	}

	if isHardDecline(payment.DeclineCode) {
		w.notifyOps(ctx, tenant.ID, payment.DeclineCode, "Hard decline encountered")
	}

	switch payment.AttemptCount {
	case 1:
		portalURL := w.gateway.BillingPortalURL(ctx, tenant.GatewayCustomerID)
		w.mailer.Send(ctx, tenant.BillingEmail, "payment_failed_initial", map[string]string{
			"portal_url": portalURL,
		})
	case 2:
		w.mailer.Send(ctx, tenant.BillingEmail, "payment_failed_reminder", nil)
		if mrr > 50000 { // High value customer (> $500 MRR)
			w.notifySales(ctx, tenant.ID, "medium_priority")
		}
	default:
		w.mailer.Send(ctx, tenant.BillingEmail, "payment_failed_final", nil)
		if mrr > 50000 {
			w.notifySales(ctx, tenant.ID, "high_priority")
		}
		// Restrict administrative setup while keeping runtime proxy endpoints working
		w.features.Restrict(ctx, tenant.ID, "create_api_keys", "upgrade_models")
	}

	return w.tenants.UpdateBillingStatus(ctx, tx, tenant.ID, "PAYMENT_FAILING", map[string]any{
		"attempt_count": payment.AttemptCount,
	})
}
```

If payment retries fail completely and the gateway marks the subscription as uncollectible, Relay transitions the tenant status to `SUSPENDED`. The API gateway middleware immediately begins rejecting runtime API requests with HTTP status `402 Payment Required`.

---

## Credit Notes and Accounting Adjustments

A credit note is a formal accounting document that offsets a previously issued invoice. You should never edit or delete an existing finalized invoice in your database. Instead, issue a credit note to maintain a clean financial audit trail.

Common scenarios requiring credit notes include:

1. **Outage Credits**: SLA compensation issued after an upstream provider outage.
2. **Partial Overcharge Refunds**: Correcting token calculation overcharges.
3. **Bad Debt Write-Offs**: Clearing unpaid invoices for canceled accounts.

```sql
CREATE TABLE billing_adjustments (
    id                      TEXT PRIMARY KEY,
    tenant_id               TEXT NOT NULL REFERENCES tenants(id),
    gateway_credit_note_id  TEXT NOT NULL,
    type                    TEXT NOT NULL, -- service_credit | refund | write_off
    amount_cents            BIGINT NOT NULL,
    currency                TEXT NOT NULL DEFAULT 'USD',
    memo                    TEXT,
    issued_at               TIMESTAMPTZ NOT NULL DEFAULT now(),
    issued_by               TEXT NOT NULL
);
```

### The Paid Invoice Allocation Rule

When issuing a credit note against a paid invoice, the sum of all credit allocations must equal the total invoice value being adjusted:

```
Refund Amount + Customer Balance Credit + Write-off Amount = Invoice Amount Adjusted
```

```go
type CreditNoteRequest struct {
	InvoiceID      string
	AmountCents    int64
	Type           string // "balance_credit" | "refund" | "write_off"
	Memo           string
	IdempotencyKey string
}

func (s *BillingService) IssueServiceCredit(ctx context.Context, tenantID string, amountCents int64, memo string) error {
	billing, err := s.db.GetTenantBilling(ctx, tenantID)
	if err != nil {
		return err
	}

	invoice, err := s.gateway.GetLastPaidInvoice(ctx, billing.GatewayCustomerID)
	if err != nil {
		return err
	}

	idempotencyKey := fmt.Sprintf("credit-%s-%d", tenantID, time.Now().UnixMilli())
	creditNoteID, err := s.gateway.CreateCreditNote(ctx, CreditNoteRequest{
		InvoiceID:      invoice.ID,
		AmountCents:    amountCents,
		Type:           "balance_credit",
		Memo:           memo,
		IdempotencyKey: idempotencyKey,
	})
	if err != nil {
		return err
	}

	return s.db.InsertBillingAdjustment(ctx, BillingAdjustment{
		TenantID:            tenantID,
		GatewayCreditNoteID: creditNoteID,
		Type:                "service_credit",
		AmountCents:         amountCents,
		Memo:                memo,
		IssuedBy:            "system",
	})
}
```

---

## Usage-Based Invoice Generation

Unlike fixed monthly software subscriptions, metered billing requires reporting cumulative consumption metrics to the gateway throughout the billing cycle.

```mermaid
sequenceDiagram
    participant Proxy as Relay API Proxy
    participant Log as Request Logger
    participant DB as Postgres DB
    participant Worker as Hourly Billing Worker
    participant GW as Payment Gateway

    Proxy->>Log: Request Complete (1,500 Tokens)
    Log->>DB: INSERT INTO requests
    Log->>DB: UPSERT INTO usage_daily

    Note over Worker: Runs Hourly via Cron Job
    Worker->>DB: SELECT SUM(total_tokens) WHERE hour = LAST_HOUR
    Worker->>GW: ReportUsage(item_id, quantity, idempotency_key)
    GW-->>Worker: Usage Record Acknowledged

    Note over GW: Billing Period Closes
    GW->>GW: Compile Invoice Line Items (Quantity * Tier Rate)
    GW->>Proxy: Webhook: invoice.created (Draft Status)
    Proxy->>DB: Run Final Reconciliation Check
    Proxy->>GW: Inject Any Unreported Residual Usage Items
    GW->>GW: Finalize & Charge Customer Card
```

### Reporting Hourly Usage Safely

To report consumption without double-counting usage during retries, construct a deterministic **idempotency key** for every usage payload.

Format: `usage-{tenant_id}-{model_family}-{hour_timestamp}`

```go
type UsageRecord struct {
	ItemID         string
	Quantity       int64
	At             time.Time
	IdempotencyKey string
}

func (w *BillingWorker) ReportHourlyUsage(ctx context.Context) error {
	hourBucket := time.Now().UTC().Truncate(time.Hour).Add(-time.Hour)

	rows, err := w.db.QueryHourlyUsage(ctx, hourBucket)
	if err != nil {
		return err
	}

	for _, row := range rows {
		itemID, err := w.db.GetGatewayItemID(ctx, row.TenantID, row.ModelFamily)
		if err != nil {
			if errors.Is(err, ErrItemNotFound) {
				if refreshErr := w.refreshSubscriptionItems(ctx, row.TenantID); refreshErr != nil {
					return refreshErr
				}
				itemID, err = w.db.GetGatewayItemID(ctx, row.TenantID, row.ModelFamily)
			}
			if err != nil {
				return err
			}
		}

		idempotencyKey := fmt.Sprintf("usage-%s-%s-%d",
			row.TenantID, row.ModelFamily, hourBucket.Unix())

		err = w.gateway.ReportUsage(ctx, UsageRecord{
			ItemID:         itemID,
			Quantity:       row.TotalTokens,
			At:             hourBucket,
			IdempotencyKey: idempotencyKey,
		})
		if err != nil {
			w.log.Error("Failed to report usage", "tenant", row.TenantID, "error", err)
			// Continue processing other tenants
		}
	}
	return nil
}
```

If the hourly job fails midway through processing 1,000 tenants, re-running the job is completely safe. The gateway uses the deterministic idempotency key to ignore duplicate reports.

---

## End-to-End Billing Architecture

Here is the complete blueprint showing how real-time request tracking, hourly aggregation, webhook processing, and dunning workflows fit together:

```mermaid
graph TB
    subgraph HotPath["1. Real-Time Hot Path"]
        API[API Request] --> LOG[Request Logger]
        LOG --> REQS[(requests Table)]
        LOG --> DAILY[(usage_daily Rollup)]
    end

    subgraph HourlyWorker["2. Metered Usage Pipeline"]
        HW[Hourly Cron Worker] --> DAILY
        HW -->|ReportUsage with Idempotency Key| GW_USAGE[Gateway API]
    end

    subgraph WebhookPipeline["3. Asynchronous Webhook Processor"]
        WH[POST /webhooks/billing] --> SIG[HMAC Signature Check]
        SIG --> Q[Worker Queue]
        Q --> WW[Webhook Worker]
        WW --> IDEM[(webhook_events Table)]
        WW -->|Fetch Live State| GW_API[Gateway API]
        GW_API --> STATE[Update Postgres Tenant Status]
    end

    subgraph DunningPipeline["4. Dunning Engine"]
        FAIL_EVENT[Event: payment.failed] --> DW[Dunning Worker]
        DW --> EMAIL[Email Provider API]
        DW --> RESTRICT[Restrict Tenant Permissions]
    end

    subgraph Reconciliation["5. Period-End Reconciliation"]
        INV_EVENT[Event: invoice.created] --> RECON[Reconciliation Worker]
        RECON --> REQS
        RECON -->|Inject Missing Line Items| GW_API
    end
```

---

## FAQ

**Why create gateway customer objects eagerly at signup instead of lazily when adding a payment method?**  
Creating customer objects at signup guarantees that every tenant record in your database has a valid `gateway_customer_id`. This eliminates conditional checks across your codebase when attaching payment methods, processing trials, issuing promotional credits, or fetching invoices.

**What happens if our webhook processor drops offline for 12 hours?**  
Payment gateways store undelivered webhooks and retry delivery for 3 to 7 days using exponential backoff schedules. When your processor comes back online, it processes accumulated events. Because your worker re-fetches live state from the gateway API before making database updates, processing stale events remains safe and correct.

**How do we handle mid-cycle plan upgrades?**  
When a tenant upgrades their plan, the payment gateway replaces existing subscription items with new ones. Your system must listen for `subscription.updated` webhooks and update local `tenant_subscription_items` mappings immediately. Otherwise, hourly usage reports will target deleted item IDs and fail.

**Can daily usage rollups fall behind during traffic spikes?**  
Yes. Aggregation jobs can lag during heavy traffic surges. For this reason, period-end invoice reconciliation queries the primary `requests` log table directly rather than relying solely on rollups.


---

## Couchbase Index Best Practices and Query Performance Tuning

**Date:** 2026-07-31
**URL:** http://localhost:1313/couchbase-index-best-practices-performance-tuning/
**Description:** A practical field guide to Couchbase indexing. Covers primary vs GSI indexes, composite key ordering, index-order sorting, covering and partial indexes, array indexes (including replacing OR and LIKE), IntersectScan and UnionScan avoidance, replication and partitioning, projection selectivity, the ADVISE and INFER tools, scan consistency, and pagination, all with runnable SQL++ (N1QL) examples against the travel-sample dataset.


Most Couchbase performance tickets I have seen end the same way: someone adds more nodes, the dashboard looks a little better for a week, and the same query shows up in the slow log a month later. The node count was never the problem. The index was.

This is a working reference, not an essay. I already wrote [the reflective version](/what-couchbase-taught-me-about-system-thinking/) of what indexing in Couchbase teaches you about systems in general. This post skips the reflection and gets straight to the decisions: which index type to reach for, how to order composite keys, how to tell if a query is actually using what you built, and which four or five mistakes account for most of the slow queries you will ever debug. Couchbase's own documentation is the primary source for the recommendations below. Every example runs against `travel-sample`, the sample bucket Couchbase ships with every install, so you can paste these into your own cluster and see the plan yourself.

---

## The Three Services, Just Enough to Matter Here

Couchbase splits into three services, and indexing only makes sense once you know which one does what.

```mermaid
flowchart LR
    Q["Query Service\nparses SQL++, plans execution"] --> I{"Index Service\ncan this index cover the query?"}
    I -->|yes, covering| R["Return rows directly\nfrom the index"]
    I -->|no| D["Data Service\nfetch full document by key"]
    D --> R
```

The **Query Service** parses SQL++ (the language Couchbase used to call N1QL) and decides which index, if any, can answer the query. The **Index Service** maintains Global Secondary Indexes (GSIs), which are separate structures built from document fields, not the documents themselves. The **Data Service** holds the actual documents and answers key-value fetches.

Every index decision in this post is really a decision about how much work the Query Service can finish inside the Index Service before it has to go ask the Data Service for the rest. The less it has to ask, the faster the query.

---

## Primary Index: The One You Should Almost Never Use in Production

```sql
CREATE PRIMARY INDEX ON `travel-sample`;
```

A primary index maps every document key in a keyspace to itself. It exists so `SELECT * FROM travel-sample` works with zero setup, which makes it perfect for a five-minute demo and a liability everywhere else. A primary index scan is a full keyspace scan, every document, every time, the same category of operation as `SELECT * FROM a_million_row_table` with no `WHERE` clause on any relational database.

If `EXPLAIN` on a production query shows `"index": "#primary"`, that is not the database being slow. That is a missing secondary index, and the fix is always the same: build one that actually matches the query's predicates, then drop the primary index, or at least make sure the query planner never reaches for it by mistake.

---

## Composite Indexes: Key Order Is Not Arbitrary

A composite (compound) index combines multiple fields into one index, and the order of those fields determines which queries it can actually serve efficiently. Get the order wrong and the index still exists, it just does far less work than it looks like it should.

The ordering rule, in priority order:

1. **Equality predicates** (`WHERE city = "Paris"`)
2. **IN predicates** (`WHERE city IN ["Paris", "London"]`)
3. **LESS THAN / LESS THAN OR EQUAL**
4. **BETWEEN**
5. **GREATER THAN / GREATER THAN OR EQUAL**
6. **Array predicates**
7. **Any remaining fields needed only for covering** (fields the query selects or filters on with something other than the above)

Within the same predicate type, order by cardinality, the field with more distinct values first. A `country` field with 5 values does far less work eliminating rows than a `city` field with 500.

One relief: you don't have to write your query's `WHERE` clause in this exact order. This rule governs how you declare the `CREATE INDEX` statement, not how you phrase the query, the planner reorders predicates internally to match whatever index order is available.

```sql
-- Query: find vacant hotels in a given city, cheapest first
SELECT name, price
FROM `travel-sample`
WHERE type = 'hotel' AND city = $city AND vacancy = true
ORDER BY price
LIMIT 20;

-- Bad: vacancy has lower cardinality than city, and it's listed first
CREATE INDEX idx_hotels_bad ON `travel-sample` (vacancy, city, price)
WHERE type = 'hotel';

-- Better: equality fields ordered by cardinality, then the ORDER BY field
CREATE INDEX idx_hotels_good ON `travel-sample` (city, vacancy, price)
WHERE type = 'hotel';
```

Both indexes technically "work." The second one eliminates far more rows per key comparison before it ever touches `price`, because `city` (hundreds of distinct values) is far more selective than `vacancy` (only `true` or `false`). On a small sample bucket the difference is invisible. On a keyspace with a few million documents, it is the difference between milliseconds and seconds.

---

## Indexing to Avoid Sorting

A GSI doesn't just store field values, it stores them **pre-sorted** by the index's key order. When a query's `ORDER BY` lines up with that same order, positioned right after the predicate keys, the Query Service reads rows straight off the index in the order it already needs and skips a separate sort step entirely.

```sql
-- ORDER BY price matches the index's key order (after the equality predicate on city)
SELECT name, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris'
ORDER BY price;

CREATE INDEX idx_hotel_city_price ON `travel-sample` (city, price)
WHERE type = 'hotel';
```

If the query needs descending order, declare that direction in the index itself rather than leaving the Query Service to reverse a whole result set afterward:

```sql
SELECT name, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris'
ORDER BY price DESC;

CREATE INDEX idx_hotel_city_price_desc ON `travel-sample` (city, price DESC)
WHERE type = 'hotel';
```

`EXPLAIN` shows whether this actually worked. An explicit `Order` operator in the plan means the Query Service sorted in memory after the scan. No `Order` operator means the index handled it for free. This is worth checking on any query with an `ORDER BY`: sorting a small result set is invisible, sorting a large one right before a `LIMIT` is a common source of latency that looks, from the outside, like the index isn't helping at all.

---

## Favor Equality Over Ranges

Given a choice, design the query and the index around equality predicates instead of ranges. An equality match is a direct lookup inside the index's sorted structure. A range (`<`, `>`, `BETWEEN`) has to walk every entry between the two bounds, and the cost scales with how wide that range is, not with how many rows eventually match.

Timestamps are the field type this bites most often, and `travel-sample`'s own documents don't carry one worth ranging over, so here's the pattern as you'd hit it against any timestamped collection instead, orders, events, sessions. "Give me everything from a particular day" usually gets written as a range over an ISO-8601 timestamp:

```sql
-- Range scan: walks every index entry between the two timestamps
SELECT * FROM `travel-sample`
WHERE type = 'order'
  AND orderDate >= '2026-07-28T00:00:00' AND orderDate < '2026-07-29T00:00:00';
```

A functional index on just the date portion turns the same query into an equality lookup:

```sql
CREATE INDEX idx_order_date ON `travel-sample` (SPLIT(orderDate, "T")[0])
WHERE type = 'order';

SELECT * FROM `travel-sample`
WHERE type = 'order' AND SPLIT(orderDate, "T")[0] = '2026-07-28';
```

This isn't only about scan cost. Equality predicates are also what enables partition elimination on a partitioned index ([more on that below](#index-replication-and-partitioning-for-scale)), a range predicate generally can't tell which partitions to skip, so it ends up scanning all of them.

---

## Covering Indexes: Skip the Fetch Entirely

An index **covers** a query when every field the query needs, filters, selects, and sorts on, already lives in the index. When that's true, the Query Service never has to ask the Data Service for the full document. It answers straight from the Index Service.

The `idx_hotel_city_price` index from the previous section is already doing double duty here, the same index that avoids a sort also covers any query that only touches `city`, `price`, and the implicit `type` filter:

```sql
-- Covered: city, price, and type all come from idx_hotel_city_price
SELECT city, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris';

-- Not covered: name isn't in the index, so this triggers a Data Service fetch per row
SELECT name, city, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris';
```

`EXPLAIN` tells you directly. Look for a `covers` array in the `IndexScan3` operator, if the fields you selected aren't listed there, you're paying for a fetch.

```
EXPLAIN SELECT city, price FROM `travel-sample` WHERE type = 'hotel' AND city = 'Paris';
```

```json
{
  "#operator": "IndexScan3",
  "index": "idx_hotel_city_price",
  "covers": [
    "cover ((`travel-sample`.`city`))",
    "cover ((`travel-sample`.`price`))",
    "cover ((`travel-sample`.`type`))"
  ]
}
```

`SELECT *` breaks this every single time, by definition. `*` means "give me the whole document," and no index (short of duplicating every field in the collection) can cover that. If a query plan needs to stay covering, name the fields it needs. This alone is one of the highest-leverage changes you can make to an existing codebase full of `SELECT *`.

Covering isn't limited to document fields, either. Couchbase's `META()` function, `META().id` for the document key, `META().cas` for its CAS value, can be indexed and selected the same way:

```sql
CREATE INDEX idx_hotel_cas ON `travel-sample` (city, META().cas)
WHERE type = 'hotel';

SELECT META().id, META().cas FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris';
```

That's useful for an optimistic-locking check, confirm a document's CAS hasn't changed since it was last read, without a full document fetch, or for any query that only ever needed the key back, `META().id` in the `SELECT` list instead of a real field.

---

## Partial (Filtered) Indexes: Index a Subset, Not Everything

Every `CREATE INDEX` above already has a `WHERE type = 'hotel'` clause. That's deliberate, and it should be the default, not the exception. A partial index only indexes documents matching its filter, which means a smaller index, a faster build, and a faster scan, because the Index Service never has to store or skip over documents the query would never match anyway.

```sql
-- Index only hotel documents, not the whole travel-sample bucket
CREATE INDEX idx_cx ON `travel-sample` (state, city)
WHERE type = 'hotel';
```

One easy mistake to avoid here: don't repeat the partial filter's equality field as an index key too. If every document the index covers already has `type = 'hotel'` by definition of the `WHERE` clause, indexing `type` again wastes space and adds nothing:

```sql
-- Wasteful: type is already guaranteed by the WHERE clause
CREATE INDEX idx_bad ON `travel-sample` (type, city)
WHERE type = 'hotel';

-- Correct: the WHERE clause already filters on type, don't index it again
CREATE INDEX idx_good ON `travel-sample` (city)
WHERE type = 'hotel';
```

A consistent `type` (or `docType`) field on every document is what makes this pattern possible in the first place. If your data model doesn't tag documents with a stable type field, partial indexes have nothing to filter on, and you're back to indexing the whole keyspace.

That `type` field is one of three habits worth building into the document model itself, before a collection has millions of documents in it and these become much harder to retrofit.

**Give every document a consistent `type` field.** Covered just above, and worth repeating: without it, there's no cheap way to filter an index down to "just hotels" or "just routes", every index falls back to scanning the entire keyspace.

**Store computed values, don't recompute them at query time.** If a hotel document has a `reviews` array, and queries frequently need the average rating or review count, store `average_rating` and `review_count` as fields on the document itself, updated when a review is added, rather than making every query aggregate the array on the fly. A stored field can be indexed and read straight from a covering index. An aggregate computed inside the query cannot.

**Avoid dynamic or unpredictable key names.** A document shaped like `{"amenityDetails": {"wifi": true, "pool": true}}` can't be indexed by key, because `CREATE INDEX` needs to name a field, and `wifi` and `pool` are values here, not a fixed field name, some other document in the same collection might have `{"spa": true, "gym": true}` instead. Restructure it as an array, `{"amenities": ["wifi", "pool"]}`, and it becomes exactly the kind of array index the next section covers.

---

## Array Indexes: Indexing Inside a Document

Couchbase documents routinely nest arrays, `travel-sample`'s route documents each carry a `stops` array of intermediate airport codes. Querying inside those arrays needs its own index shape.

```sql
-- Document shape: { "type": "route", "sourceairport": "SFO", "destinationairport": "JFK", "stops": ["ORD"] }

CREATE INDEX idx_route_stops ON `travel-sample` (DISTINCT ARRAY s FOR s IN stops END)
WHERE type = 'route';

SELECT sourceairport, destinationairport FROM `travel-sample`
WHERE type = 'route' AND ANY s IN stops SATISFIES s = 'ORD' END;
```

For partial text matching inside an array (autocomplete-style search on names, tags, or amenities), reach for `SUFFIXES()` or `TOKENS()` instead of a leading-wildcard `LIKE`:

```sql
-- Bad: a leading wildcard forces a full index scan, wildcard or not
SELECT name FROM `travel-sample`
WHERE type = 'hotel' AND name LIKE '%luteti%';

-- Good: index the suffixes, then the same search becomes a targeted lookup
CREATE INDEX idx_name_suffixes ON `travel-sample` (ALL ARRAY s FOR s IN SUFFIXES(LOWER(name)) END)
WHERE type = 'hotel';

SELECT name FROM `travel-sample`
WHERE type = 'hotel'
  AND ANY s IN SUFFIXES(LOWER(name)) SATISFIES s LIKE 'luteti%' END;
```

One more distinction that's easy to get backwards: `IN` checks the current array level, `WITHIN` recurses into every nested array and object underneath it. `WITHIN` is more powerful and noticeably more expensive. Reach for `IN` unless you specifically need to search nested structures you can't flatten.

### Replacing `OR` With an Array Index

An `OR` across two different fields forces Couchbase into a `UnionScan`: one index scan per side of the `OR`, then a merge. It's the same problem `IntersectScan` causes for `AND` across narrow indexes, just for the opposite operator, and it has the same fix: give the planner one index instead of two scans to reconcile. Searching a hotel by name, where the guest might type either its formal name or its shorter alias, is a real example of this shape in `travel-sample` itself:

```sql
-- OR across two fields: one scan per branch, then a union
SELECT name FROM `travel-sample`
WHERE type = 'hotel' AND (name = $query OR alias = $query);

-- A functional array index turns the same lookup into a single DistinctScan
CREATE INDEX idx_hotel_name_alias ON `travel-sample` (DISTINCT ARRAY v FOR v IN [name, alias] END)
WHERE type = 'hotel';

SELECT name FROM `travel-sample`
WHERE type = 'hotel' AND ANY v IN [name, alias] SATISFIES v = $query END;
```

Add more fields to this index and it stays covering the same way any other array index does. The `DistinctScan` replaces the `UnionScan`, nothing else about how the index behaves changes.

---

## Stop the IntersectScan: One Wide Index Beats Three Narrow Ones

If a query filters on three fields and three separate single-field indexes each match one of them, Couchbase can technically use all three at once and intersect the results. It works. It is also close to the worst way to answer that query.

```mermaid
flowchart TB
    subgraph narrow["Three narrow indexes"]
      Q1["Query: city, type, price"] --> S1["Scan idx_city"]
      Q1 --> S2["Scan idx_type"]
      Q1 --> S3["Scan idx_price"]
      S1 --> X["IntersectScan\nmerge three result sets"]
      S2 --> X
      S3 --> X
    end
    subgraph wide["One composite index"]
      Q2["Query: city, type, price"] --> S4["Scan idx_city_type_price\nsingle ordered scan"]
    end
```

Each narrow index scan returns its own candidate set, and the Query Service has to merge all three before it can return anything, an `IntersectScan` operator in the plan is the tell. A single composite index ordered correctly for the query (equality fields, then ranges, per the ordering rule above) answers the same query in one ordered scan with no merge step. If you find yourself adding index after index chasing individual query clauses, stop and check whether one wider composite index, matching the field order rule, replaces all of them.

The same logic applies to indexes built purely on a low-cardinality `type` or `docType` field with nothing else. An index like that matches almost every document in the keyspace, which means the query planner treats it as barely better than the primary index, an expensive fallback rather than a real optimization.

The same consolidation instinct applies across separate queries, not just within one. If three different endpoints each query `travel-sample` filtered by `city`, plus one other field apiece (`state` for one, `price` for another, `country` for a third), a single index on `(city, state, price, country)` can often serve all three, instead of maintaining three separate indexes that each duplicate the `city` prefix. Whether that consolidation is worth it depends on whether the combined index still meets each query's own latency budget, check `EXPLAIN` against each of the three queries before committing to one wide index over three narrow ones.

---

## Avoid `USE INDEX` Hints

N1QL lets you force a specific index by name:

```sql
SELECT name FROM `travel-sample` USE INDEX (idx_hotel_city_price)
WHERE type = 'hotel' AND city = 'Paris';
```

Resist reaching for it as a default habit. The query planner is rule-based and, given the indexes covered in this post, usually makes a reasonable choice on its own. Naming an index in application code creates a dependency the planner can't see past: that index can't be dropped, renamed, or replaced without also changing and redeploying whatever code hints at it, and adding a better index later does nothing for a query that's still hardcoded to the old one.

The narrow, legitimate use is diagnostic: temporarily forcing a specific index while tracking down why the planner picked a worse plan, or manually working around an `IntersectScan` you haven't consolidated yet. Treat it as a debugging tool you remove once the underlying index shape is fixed, not a permanent fixture in production code.

---

## Index Replication and Partitioning for Scale

Two settings on `CREATE INDEX` matter once a single index node stops being enough.

**Replicas** protect against losing an index node, and unlike data replicas they're active: Couchbase load-balances queries across all of them, so `num_replica` buys query throughput, not just failover. It defaults to 0 unless you set a cluster-wide default via the `indexer.settings.num_replica` setting, and the value always has to be less than the number of index nodes you actually have. These are separate `CREATE INDEX` statements from `idx_hotel_city_price` above, not settings bolted onto it after the fact, Couchbase can't add replicas or partitioning to an existing index without recreating it:

```sql
CREATE INDEX idx_hotel_city_price_r2 ON `travel-sample` (city, price)
WHERE type = 'hotel'
WITH { "num_replica": 2 };

-- Pin the original and its replicas to specific nodes instead of letting
-- Couchbase place them: the "nodes" array length should equal num_replica plus one.
CREATE INDEX idx_hotel_city_price_r2_pinned ON `travel-sample` (city, price)
WHERE type = 'hotel'
WITH { "num_replica": 2, "nodes": ["index1.internal", "index2.internal", "index3.internal"] };
```

**Partitioning** splits one large index across multiple nodes by a hash of an expression, useful once a single index no longer fits comfortably in one node's memory. Before Couchbase Server 5.5, this meant manually building separate range-partitioned indexes yourself, one index per key range (`LOWER(username) >= 'a' AND < 'n'`, another for `'n'` to `'z'`, and so on) and gluing the results together in the application. `PARTITION BY HASH` replaces all of that: Couchbase decides which of `num_partitions` partitions (16 by default) each document lands in, and spreads those partitions across the available index nodes automatically.

```sql
CREATE INDEX idx_hotel_city_price_partitioned ON `travel-sample` (city, price)
WHERE type = 'hotel'
PARTITION BY HASH(city)
WITH { "num_replica": 1 };
```

The payoff shows up as **partition elimination**: a query with an equality predicate on the partition key only has to scan the one partition that document could possibly be in, not all sixteen. This is one more reason equality predicates (the previous section) matter more than they look, a range predicate on a partitioned field generally can't tell which partitions to skip.

When you're creating several indexes on a bucket that already has data (a new feature launch, a migration, a new query pattern), build them deferred and trigger the build once instead of once per index:

```sql
CREATE INDEX idx_a ON `travel-sample` (city) WHERE type = 'hotel' WITH { "defer_build": true };
CREATE INDEX idx_b ON `travel-sample` (country) WHERE type = 'hotel' WITH { "defer_build": true };
CREATE INDEX idx_c ON `travel-sample` (price) WHERE type = 'hotel' WITH { "defer_build": true };

BUILD INDEX ON `travel-sample` (idx_a, idx_b, idx_c);
```

Deferred indexes share a single DCP stream (Couchbase's internal change stream) when built together, so the Index Service reads every document once and feeds all three indexes from that one pass, instead of re-reading the entire keyspace for each index in turn.

---

## Projection Selectivity: What Every Mutation Costs

Every index has an upkeep cost that's easy to forget, because it doesn't show up when you run a query, it shows up on every write. The Data Service's Projector inspects each document mutation and checks it against every index definition on that keyspace; anything that matches gets routed to the Index Service to update. **Projection selectivity** is the fraction of mutations that actually qualify: qualifying documents divided by total documents mutated.

```mermaid
flowchart LR
    M["Document mutation"] --> P["Projector\nchecks against every index filter"]
    P -->|matches WHERE clause| RT["Router\nsends to Index Service"]
    P -->|doesn't match| SKIP["Skipped\nno index update"]
```

An index filtered down to a narrow subset, `WHERE type = 'hotel' AND country = 'US'` against a keyspace that's mostly something else, might see a selectivity of a fraction of a percent: almost every mutation gets checked and immediately skipped, cheaply. An index with no filter, or one whose filter matches nearly everything in the keyspace, sits at or near 100% selectivity: every single mutation in the keyspace triggers a real update to that index.

100% selectivity isn't automatically wrong, an index genuinely meant to cover most of a keyspace pays that cost on purpose. What's worth catching is the accidental version: a partial index whose `WHERE` clause is broader than it needs to be, or a composite index that quietly lost its filter during a rewrite. This is the concrete cost partial indexes (covered earlier) are actually buying you: keeping this number low so most mutations never touch the Index Service at all.

---

## Query-Side Tuning

Indexes only get you half the win. The query itself needs to actually use them well.

**`USE KEYS` when you already have the document key.** If you know the key, skip the Index Service entirely and go straight to the Data Service, the same cost as a key-value get. It accepts more than one key:

```sql
SELECT * FROM `travel-sample` USE KEYS ['hotel_123', 'hotel_456'];
```

It works the same way inside a `JOIN`'s `ON KEYS` clause, whenever one document embeds another's key directly, instead of needing its own index lookup to find the match. `travel-sample`'s hotels embed their reviews as an array rather than storing them as separate documents, so this doesn't come up inside the sample data itself, but it's a common shape elsewhere: an order document that stores its customer's key directly, for example.

```sql
-- General pattern, not travel-sample: orders store the customer's key directly
SELECT o.total, c.name AS customer_name
FROM orders o
JOIN customers c ON KEYS o.customerId;
```

And if the query doesn't need SQL++ at all, just the full document by a key you already have, skip the Query Service entirely too. A plain key-value `GET()` through the SDK is cheaper than any N1QL statement, `USE KEYS` included, because it never touches the Query Service's parsing and planning path in the first place.

**Prepared statements, not ad hoc SQL++ on every call.** A prepared statement parses and plans once, then reuses that plan on every execution. From the SDK, this usually means setting `adhoc: false` on the query options. From the shell or a raw HTTP call:

```sql
PREPARE hotel_by_city AS
  SELECT name, price FROM `travel-sample`
  WHERE type = 'hotel' AND city = $city;

EXECUTE hotel_by_city USING { "city": "Paris" };
```

**Parameters, not string concatenation.** `$city` above, not `'` + city + `'` spliced into the query text. This is a SQL-injection defense first, but it also means the Query Service sees the same query shape every time regardless of the value, which is what makes the prepared plan reusable in the first place.

**One query with `CASE`, not several queries for several aggregates.** If a dashboard needs several different counts filtered several different ways over the same keyspace, it's faster to compute all of them in one pass than to scan the same data repeatedly:

```sql
SELECT
  COUNT(CASE WHEN free_internet THEN 1 END) AS free_wifi_count,
  COUNT(CASE WHEN free_breakfast THEN 1 END) AS free_breakfast_count,
  COUNT(CASE WHEN free_parking THEN 1 END) AS free_parking_count
FROM `travel-sample`
WHERE type = 'hotel';
```

---

## Pagination: Pushdown vs Keyset

`LIMIT` and `OFFSET` only get pushed down into the Index Service, meaning the index itself stops early instead of the Query Service fetching everything and truncating afterward, when four conditions all hold: the query's predicates fit a single index (no `IntersectScan`), there's no join, and `ORDER BY` matches the index's own key order. Break any one of those and `LIMIT 20` still means "scan everything, then keep 20."

Confirm this actually happened rather than assuming it did: `EXPLAIN` the query and check the `IndexScan3` operator for explicit `limit` and `offset` properties. If they're missing, the Query Service is applying `LIMIT`/`OFFSET` itself after assembling the full result set, meaning one of the four conditions above wasn't met.

Even with pushdown working, `OFFSET` on a deep page is the same problem [the API design post](/posts/api-design-for-backend-systems/#pagination-offset-vs-keyset-cursor) covers for any database: the index still has to walk past every skipped row before it can return anything. For a paginated feed backed by Couchbase, a keyset-style cursor, "give me rows after this specific `(price, id)` pair," not "skip this many rows", avoids the walk entirely and stays fast at any depth:

```sql
-- Offset pagination: gets slower every page, deep pages walk thousands of skipped rows
SELECT name, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris'
ORDER BY price LIMIT 20 OFFSET 200;

-- Keyset pagination: constant cost regardless of depth
SELECT name, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris' AND price > $last_seen_price
ORDER BY price LIMIT 20;
```

---

## Scan Consistency: The Tradeoff Nobody Reads the Docs For

Couchbase's indexes update asynchronously from writes. A document written a moment ago might not be reflected in an index yet. Every query picks a consistency level, and the default is not always the right default for the query in front of you.

| Level | Behavior | Cost |
|---|---|---|
| `NOT_BOUNDED` (default) | Returns whatever the index currently has, may miss very recent writes | Fastest, no wait |
| `AT_PLUS` | Waits only for the specific mutations your application already tracked and passed in as scan vectors (mutation tokens), nothing else | Small, targeted wait, but your code has to track the tokens |
| `REQUEST_PLUS` / `STATEMENT_PLUS` | Waits for the index to catch up to the latest state of the whole keyspace before scanning, no manual tracking needed | Slowest, unbounded wait under write load |

These aren't three tiers of the same thing, `AT_PLUS` and `REQUEST_PLUS` solve the consistency problem in genuinely different ways. `AT_PLUS` is cheaper precisely because it only waits for the mutations you tell it about, the tradeoff is that your application has to capture and pass along the mutation token from the write it cares about. `REQUEST_PLUS` needs nothing from the caller, it just waits for everything, which is simpler to use and more expensive to run.

`NOT_BOUNDED` is correct for the overwhelming majority of reads, a dashboard, a search page, a list view. The other two earn their cost in a narrow case: you just wrote a document and the very next request in the same flow needs to see it, a "create and immediately redirect to the detail page" pattern. If your SDK already hands you the mutation token from the write, `AT_PLUS` is the cheaper way to guarantee that. Reaching for `REQUEST_PLUS` everywhere "to be safe" quietly adds latency to every query in the system for a consistency guarantee most of them never needed.

---

## Let the Tools Do the First Pass

### `ADVISE`: A Concrete Recommendation, Not a Guess

Couchbase ships an actual index advisor, not a linter guessing from syntax, it reads the query and the keyspace's existing indexes and recommends what to build.

```sql
ADVISE SELECT name, price FROM `travel-sample`
WHERE type = 'hotel' AND city = 'Paris'
ORDER BY price;
```

`ADVISE` returns a concrete `CREATE INDEX` recommendation, correctly ordered keys included, for that specific query. It's a starting point, not a replacement for understanding why an index is shaped the way it is, but it's a fast way to catch an obviously missing index before a query ships. It also only ever looks at one query at a time, so it won't tell you when three of its recommendations across three different queries should really be one composite index, that consolidation judgment (covered above) is still yours to make.

Underneath, Couchbase's cost-based optimizer relies on statistics gathered by `UPDATE STATISTICS`. As of Couchbase Server 7.6, the Query Service gathers these statistics automatically whenever an index is created or built, so the optimizer generally has real cardinality data to plan against rather than guessing. On older clusters, or when you've made a large data shape change, running it manually keeps the optimizer's assumptions from going stale:

```sql
UPDATE STATISTICS FOR `travel-sample` (type, city, price);
```

### `INFER`: See What's Actually in the Bucket

Before deciding what to index at all, `INFER` profiles a keyspace by sampling its documents and returning a JSON Schema-shaped summary of what it actually found, field names, types, and how common each field is.

```sql
INFER `travel-sample` WITH { "sample_size": 1000, "num_sample_values": 3 };
```

```json
{
  "properties": {
    "city": {
      "%docs": 87.5,
      "#docs": 875,
      "samples": ["Paris", "London", "Tokyo"],
      "type": "string"
    },
    "public_likes": {
      "%docs": 62.1,
      "#docs": 621,
      "minitems": 1,
      "maxitems": 12,
      "type": "array"
    }
  }
}
```

`%docs` and `#docs` tell you how common a field actually is, useful for catching the assumption that every document has a field when only two-thirds of them do. `minitems`/`maxitems` on an array field gives a rough sense of how big an array index on it will be. `sample_size` and `num_sample_values` control how much of the keyspace gets sampled and how many example values come back per field; `similarity_metric` controls how aggressively `INFER` merges documents with slightly different shapes into one schema entry instead of reporting them as separate variants. On a keyspace you didn't design yourself, or one that's evolved for a few years, this is a faster way to find out what's really there than reading old code or asking whoever wrote it originally.

One more habit worth adopting from the start: design and iterate on a query and its index against an empty or near-empty keyspace first. `EXPLAIN` output, key ordering, and covering behavior are all things you can verify structurally before a single document is loaded, and iterating there is fast. Load real data once the shape is right, then re-check the same `EXPLAIN` output against real cardinality, not before.

---

## Finding the Query That's Actually Slow

Two system keyspaces cover almost every "why is this slow right now" investigation.

```sql
-- Currently running queries, useful for catching a runaway query live
SELECT * FROM system:active_requests;

-- Cancel one by request ID
DELETE FROM system:active_requests WHERE requestId = 'abc-123';

-- Recently completed queries, filter to anything that took more than a second
SELECT statement, elapsedTime, resultCount
FROM system:completed_requests
WHERE elapsedTime > "1s"
ORDER BY elapsedTime DESC
LIMIT 20;
```

That `elapsedTime > "1s"` comparison is a string comparison against a formatted duration (`"250ms"`, `"1.2s"`, and so on), not a numeric one. It works for the common case of picking a single threshold like "over a second," but don't trust it to sort or compare cleanly across mixed units, `"500ms"` sorts after `"2s"` as a string even though it's the faster query. Fine for a threshold filter, worth converting to a numeric duration first if you need to compare or sort a wider range of values.

`system:completed_requests` is the single best place to start any performance investigation. It's the actual, historical record of what ran slow, not a guess about what might be slow. Once you've pulled the offending statement and either fixed the index or the query, delete the entry so the next investigation isn't wading through queries you already resolved.

One field makes both of these keyspaces far more useful once more than one service is calling Couchbase: `clientContextID`. Couchbase never looks at this value itself, it exists purely for your application to set, from the SDK, so a query shows up in `system:active_requests` and `system:completed_requests` tagged with whatever request or trace ID your own code already uses.

```sql
SELECT statement, elapsedTime, clientContextID
FROM system:completed_requests
WHERE clientContextID LIKE 'checkout-svc:%'
ORDER BY elapsedTime DESC
LIMIT 20;
```

Set it to the same request ID your logs and traces already carry, and a slow-query investigation stops being "reverse-engineer which request this was from the raw statement text" and starts being a direct join against your own application logs.

---

## The Anti-Pattern Checklist

| Anti-pattern | Why it hurts | Fix |
|---|---|---|
| Primary index in production | Full keyspace scan on every query that falls back to it | Build a matching secondary index, remove the fallback path |
| `SELECT *` | Disables covering indexes, forces a fetch every row, every time | Name the fields the query actually needs |
| Composite index keys in the wrong order | Index does far less filtering per comparison than it should | Equality, then IN, then ranges, then array, ordered by cardinality within each group |
| Several narrow indexes on one query | Triggers `IntersectScan`, merging result sets instead of one ordered scan | One composite index matching the query's full predicate set |
| `OR` across different fields | Triggers `UnionScan`, one scan per branch plus a merge | Functional `DISTINCT ARRAY` index with `ANY`/`SATISFIES` |
| Leading-wildcard `LIKE '%x%'` | Forces a full index scan regardless of the index | `SUFFIXES()` or `TOKENS()` array index |
| `USE INDEX` hints in application code | Couples code to one index by name, blocks the optimizer from improving | Let the planner choose, reserve `USE INDEX` for temporary diagnosis |
| Dynamic or unpredictable JSON key names | Can't be named in a `CREATE INDEX`, so can't be indexed at all | Restructure as an array of consistent values |
| `REQUEST_PLUS` everywhere | Adds unbounded wait to every query for a guarantee most don't need | `NOT_BOUNDED` by default, `AT_PLUS` for the specific read-your-write cases |
| Deep `OFFSET` pagination | Walks every skipped row before returning anything | Keyset pagination on an indexed sort key |
| Indexes nobody monitors | Accumulate write cost and memory long after the query that needed them is gone | Track usage, drop indexes with no recorded scans |

---

## FAQ

**Do I need a primary index at all?**

Only in development, for ad hoc exploration, or briefly while building out real secondary indexes for a new keyspace. Drop it once the keyspace has proper indexes, or the query planner will occasionally fall back to it for queries that don't match any secondary index, and you won't notice until that fallback is scanning millions of documents in production.

**How many indexes is too many?**

Every index costs write throughput (each mutation updates every index on that document's fields) and memory. The practical ceiling isn't a fixed number, it's whichever comes first: write latency degrading under your actual mutation rate, or index memory pressure. Consolidating narrow, overlapping indexes into fewer wide composite ones (the `IntersectScan` fix above) is usually how you buy back headroom, not simply deleting indexes until something breaks.

**Does `ADVISE` replace understanding key ordering myself?**

No, and treating it as a replacement is where advisor-generated indexes go wrong. `ADVISE` is excellent at catching "you're missing an index for this query" fast. It is analyzing one query at a time, so it won't tell you that three of its recommendations across three queries should really be one composite index. That consolidation judgment is still yours to make.

**What's the difference between `INFER` and `ADVISE`?**

`INFER` answers "what does my data actually look like", field names, types, how common each one is, before you've written a query. `ADVISE` answers "what index does this specific query need", after you've written a query. Reach for `INFER` when exploring an unfamiliar or evolving keyspace, `ADVISE` once you already have a slow query in hand.

**Is a 100% projection selectivity index ever acceptable?**

Yes, when it's deliberate. An index meant to cover most of a keyspace (a primary-key-style lookup, or a field genuinely present on every document) will legitimately see every mutation. What's worth catching is the accidental version: a partial index whose `WHERE` clause is wider than it needs to be, or a composite index that quietly lost its filter during a rewrite. Check the intent behind the number, not just the number itself.

**Is `WITHIN` ever the right choice over `IN`?**

Yes, specifically when you genuinely need to search nested arrays or objects at unknown depth and can't restructure the data to flatten them. That's a real but narrow case. Reach for `IN` by default and only step up to `WITHIN` when a document's nesting is truly variable depth.

**Why does the same query get a different plan on two different clusters?**

Cardinality. The cost-based optimizer's choices depend on `UPDATE STATISTICS` output, which reflects the actual data distribution in that specific keyspace. A staging cluster with a thousand documents and a production cluster with fifty million can reasonably produce different plans for the same query and the same indexes, because the actual selectivity of each field is different in each place. Don't assume a plan verified in staging holds in production without checking `EXPLAIN` against real production statistics.

---

## Things Worth Their Own Post

- **Full-Text Search (FTS) indexes vs GSI**: when relevance ranking, fuzzy matching, and language-aware tokenization outgrow what `SUFFIXES()` and array indexes can reasonably do.
- **The cost-based optimizer in depth**: how `UPDATE STATISTICS` sampling actually works, what it stores, and how to read a plan's cost estimates against real cardinality.
- **Vector search indexes**: Couchbase's vector index support for embedding similarity search, and how it composes with GSI on the same bucket.
- **Capella-specific tuning**: autoscaling behavior, multi-region XDCR replication lag, and what changes about index design when the ops layer is managed for you.


---

## API Design for Backend Systems

**Date:** 2026-07-26
**URL:** http://localhost:1313/api-design-for-backend-systems/
**Description:** A practical, opinionated guide to designing backend APIs. Covers request and response schema, offset and keyset pagination, BFF, server-driven UI, caching, versioning, naming, the new HTTP QUERY verb, WebSocket vs SSE, API security (auth headers, server-to-server auth, webhook receivers), and bulk data load APIs. Built around Relay, the same AI proxy system from the multi-tenant SaaS post, with REST and JSON as the running example and Go code throughout.


A few weeks ago I designed [Relay](/how-multi-tenant-saas-works/), a multi-tenant AI API gateway, as a system design exercise. That post covered the database, the auth, the billing. It did not cover the thing every one of those systems is sitting behind: the API itself.

Here is the question that post left open. When someone builds `GET /v1/requests` to list a tenant's API call history, what should that endpoint actually look like? Offset or cursor pagination? What does the response envelope contain? What happens when Relay needs to add a filter that does not fit in a query string? What does a webhook payload look like when Relay tells a tenant "your batch job finished"? None of that is specific to AI gateways. It is the same decisions every backend API makes, made badly often enough that "REST API" has become a phrase that means "JSON over HTTP, structure unspecified."

So this post is about those decisions. We will use Relay as the running example, specifically its dashboard and management API (tenants managing API keys, viewing usage, receiving webhooks, exporting data), not the OpenAI-compatible hot path, which already has its shape fixed by the providers it mirrors. Everything here is REST with JSON. At the end there is a pointer to what changes if you reach for GraphQL, RPC, or SOAP instead, but that is its own post.

Examples are in Go. The principles are not Go-specific, or even REST-specific. If you write APIs in any language, the same decisions will apply.

---

## What Makes an API Good

Before naming conventions and pagination schemes, it helps to state the actual goal. An API is good when a developer who has never seen your system can read one endpoint, guess the shape of the next ten, and be right most of the time.

That is a narrower goal than "clean" or "RESTful." It comes down to four properties.

**Predictability.** If `GET /v1/api-keys/{id}` returns an object with `created_at` as an RFC 3339 string, every other endpoint in the API returns timestamps the same way. If list endpoints wrap results in `{"data": [...], "pagination": {...}}`, every list endpoint does, not just the ones someone remembered to be consistent on.

**One way to do a thing.** If there are two ways to filter a list (a query parameter and a request body), developers will use both inconsistently across your own codebase, and support will have to debug both. Pick one path per capability.

**Evolvability without breakage.** You will need to add fields, deprecate endpoints, and fix mistakes. The API needs a built-in way to do all three without an existing client waking up broken. This is what versioning and additive schema changes are for, covered later.

**Explicit contracts, not implied ones.** "The list is usually sorted by creation date" is not a contract. `"sort": "created_at desc"` in the response, or documented and fixed behavior, is. Anything a client might reasonably depend on should either be guaranteed or explicitly called out as unstable.

Everything below is really just these four properties applied to a specific decision: URLs, schemas, pagination, caching, and so on.

---

## Naming and Resource Structure

Relay's dashboard API exposes tenants, API keys, usage, and webhooks. All of them are resources, and resources get nouns, not verbs.

```
GET    /v1/api-keys              list API keys for the authenticated tenant
POST   /v1/api-keys              create an API key
GET    /v1/api-keys/{id}         fetch one API key
DELETE /v1/api-keys/{id}         revoke an API key
```

Not `POST /v1/createApiKey` or `GET /v1/getApiKeys`. The HTTP method already carries the verb. Repeating it in the path is redundant, and once you have fifty endpoints named as functions instead of nouns, you no longer have a resource model. You have RPC that happens to run over HTTP.

A few rules that keep this consistent as the API grows:

- **Plural nouns for collections.** `/api-keys`, not `/api-key`. The singular form is only used when the ID follows it: `/api-keys/{id}`.
- **Nest only one level deep, and only for true ownership.** `/v1/api-keys/{id}/rotate` is fine because rotation belongs to a specific key. `/v1/tenants/{id}/api-keys/{id}/rotate` is not, because the tenant is already implied by the auth token. If you find yourself nesting three levels, it usually means the child resource should be its own top-level collection with a filter, like `/v1/requests?api_key_id={id}`.
- **Actions that are not CRUD get a verb, deliberately.** `rotate`, `revoke`, `cancel`, `retry` are real operations that do not map to PUT or DELETE cleanly. `POST /v1/api-keys/{id}/rotate` is a controlled escape hatch, not a violation of the noun rule. The rule is "resources are nouns," not "there shall be no verbs anywhere."
- **snake_case in JSON bodies, kebab-case in URLs.** `key_hint` and `created_at` in the payload, `/api-keys` in the path. This is a convention fight with no universally right answer (Stripe uses snake_case bodies and hyphenated paths; some Microsoft APIs use camelCase throughout). Pick one and never mix both cases inside the same field set. Mixed casing inside one JSON object is the single most common thing that makes an API feel unpolished.

---

## Request and Response Schema

### Envelope or bare object

A single resource response should be the object itself, not wrapped:

```json
{
  "id": "key_9fL2xQ",
  "name": "Production",
  "key_prefix": "rly_live_",
  "key_hint": "...1eA",
  "status": "active",
  "created_at": "2026-07-20T09:14:00Z"
}
```

A list response needs an envelope, because a list carries more than just the items: pagination cursors, total counts, sort order.

```json
{
  "data": [
    { "id": "key_9fL2xQ", "name": "Production", "status": "active" },
    { "id": "key_2vM8pR", "name": "CI", "status": "active" }
  ],
  "pagination": {
    "next_cursor": "eyJpZCI6ImtleV8ydk04cFIifQ",
    "has_more": true
  }
}
```

Do not wrap the single-resource response in a `{"data": {...}}` envelope too, just to be consistent with the list response. It buys nothing and forces every client to unwrap one extra layer on every single call. Consistency should apply to shapes that need the same thing, not blindly to everything.

### Field conventions

A few small decisions, made once, save a hundred future arguments in code review:

- **IDs are prefixed strings, not raw UUIDs or integers.** `key_9fL2xQ`, `req_7yN3kM`, `ten_xyz789`. The prefix tells you what kind of object you are looking at from the ID alone, in a log line, in a support ticket, in a stack trace. This is the same pattern the multi-tenant post used for `tenants`, `api_keys`, and `requests`, and it should be consistent across every resource in the API, not just some of them.
- **Timestamps are RFC 3339 strings in UTC, always.** `"2026-07-20T09:14:00Z"`. Never Unix epoch integers in a public API. Epoch integers are ambiguous about seconds versus milliseconds, and unreadable in a raw log dump. If you need epoch millis for a specific high-frequency internal API, that is a different contract than your public REST API.
- **Money is a string or an integer in minor units, never a float.** `"amount": "9.00"` or `"amount_cents": 900`. A `float64` for currency will eventually produce `8.999999999` somewhere and someone will spend an afternoon on it.
- **Null means "explicitly no value." Omitted means "not applicable to this resource state."** A revoked API key might have `revoked_at: "2026-07-21T10:00:00Z"` and an active one has `revoked_at: null`, present but null, because the field is always meaningful for that resource type. A `next_cursor` field is omitted entirely (not `null`) once there is nothing left to fetch, because at that point the pagination concept does not apply anymore. Be deliberate about which of these two you mean, and do not let the serializer decide for you.
- **Enums are lowercase strings, not integers.** `"status": "active"`, not `"status": 1`. An integer enum forces every client to keep a lookup table in sync with your source code. A string enum is self-documenting in every request log you will ever grep.

### Partial updates

Relay's dashboard lets a tenant rename an API key or change its `expires_at` without resending the whole object. Two real options exist for this.

**JSON Merge Patch (RFC 7396)**: send only the fields you want to change, and a JSON body with `null` means "set this field to null."

```json
PATCH /v1/api-keys/key_9fL2xQ
{ "name": "Production (rotated July)" }
```

**JSON Patch (RFC 6902)**: send an array of explicit operations (`add`, `remove`, `replace`).

```json
PATCH /v1/api-keys/key_9fL2xQ
[{ "op": "replace", "path": "/name", "value": "Production (rotated July)" }]
```

Merge Patch is simpler and covers almost every real case. Reach for JSON Patch only when you genuinely need array element operations (insert at index 2, remove a specific item from a list) that Merge Patch cannot express. For Relay's dashboard API, Merge Patch is the right call everywhere. Most APIs never need JSON Patch's complexity, and adding it "for completeness" is exactly the kind of premature flexibility that makes a schema harder to reason about later.

---

## Pagination: Offset vs Keyset (Cursor)

Relay's `requests` table is the one from the multi-tenant post: every API call the tenant makes gets a row, and at 1,000 tenants making 100 calls a day, that is 3 million rows a month. `GET /v1/requests` has to paginate through that, and the pagination strategy you pick here is the single decision most likely to bite you in production six months from now.

### Offset pagination

```
GET /v1/requests?page=3&limit=50
```

```sql
SELECT * FROM requests
WHERE tenant_id = $1
ORDER BY created_at DESC
LIMIT 50 OFFSET 100;
```

Trivial to implement, and it gives you random access: jump straight to page 40. This is the right choice for small, mostly static collections where users expect page numbers, like a list of API keys (a tenant has a handful, not millions).

It falls apart on large, frequently changing tables for two separate reasons:

1. **It gets slow at depth.** `OFFSET 100000` still has to scan and discard 100,000 rows before returning anything. The query gets linearly slower the deeper a client pages, even though the page size never changes.
2. **It is unstable under writes.** If a new row is inserted while a tenant is paginating page 2 of `requests`, every row after it shifts by one. The client can see the same row twice, or skip one entirely, depending on sort direction. For a table that gets a new row every time the tenant makes an API call, this is not a rare edge case, it happens on every active tenant's very first page-through.

### Keyset (cursor) pagination

```
GET /v1/requests?cursor=eyJjcmVhdGVkX2F0IjoiMjAyNi0wNy0yMFQwOTowMDowMFoifQ&limit=50
```

Instead of an offset, the cursor encodes the last row's sort key. The next page is "give me rows after this specific point," not "skip this many rows."

```sql
SELECT * FROM requests
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)   -- values decoded from the cursor
ORDER BY created_at DESC, id DESC
LIMIT 50;
```

Note the compound key: `(created_at, id)`, not `created_at` alone. Two requests can share the same millisecond timestamp under load, and a cursor on `created_at` alone would either skip or repeat rows in that case. `id` breaks the tie and makes the ordering total and deterministic. This is the detail that makes keyset pagination correct instead of "usually correct."

```go
type Cursor struct {
    CreatedAt time.Time `json:"created_at"`
    ID        string    `json:"id"`
}

func EncodeCursor(c Cursor) string {
    b, _ := json.Marshal(c)
    return base64.URLEncoding.EncodeToString(b)
}

func DecodeCursor(s string) (Cursor, error) {
    var c Cursor
    b, err := base64.URLEncoding.DecodeString(s)
    if err != nil {
        return c, ErrInvalidCursor
    }
    if err := json.Unmarshal(b, &c); err != nil {
        return c, ErrInvalidCursor
    }
    return c, nil
}

func (h *Handler) ListRequests(w http.ResponseWriter, r *http.Request) {
    tenantID := TenantFromContext(r.Context())
    limit := clampLimit(r.URL.Query().Get("limit"), 50, 200)

    var after *Cursor
    if raw := r.URL.Query().Get("cursor"); raw != "" {
        c, err := DecodeCursor(raw)
        if err != nil {
            writeError(w, 400, "invalid_cursor")
            return
        }
        after = &c
    }

    rows, err := h.store.ListRequests(r.Context(), tenantID, after, limit+1)
    if err != nil {
        writeError(w, 500, "internal_error")
        return
    }

    hasMore := len(rows) > limit
    if hasMore {
        rows = rows[:limit]
    }

    resp := ListResponse{Data: rows, Pagination: Pagination{HasMore: hasMore}}
    if hasMore {
        last := rows[len(rows)-1]
        resp.Pagination.NextCursor = EncodeCursor(Cursor{CreatedAt: last.CreatedAt, ID: last.ID})
    }
    writeJSON(w, 200, resp)
}
```

The `limit+1` trick (fetch one extra row) is how you compute `has_more` without a separate `COUNT(*)` query. If you get back 51 rows for a limit of 50, there is more data, and you trim the 51st before returning.

```mermaid
flowchart LR
    A["Client: GET /v1/requests?cursor=..."] --> B["Decode cursor\n{created_at, id}"]
    B --> C["Query: WHERE (created_at, id) < cursor\nORDER BY created_at DESC, id DESC\nLIMIT 51"]
    C --> D{"51 rows\nreturned?"}
    D -->|yes| E["Trim to 50\nhas_more = true\nencode row 50 as next_cursor"]
    D -->|no| F["Return all rows\nhas_more = false\nomit next_cursor"]
    E --> G[Response]
    F --> G
```

The cursor is opaque to the client on purpose. It is base64-encoded JSON here for simplicity, but treat it as an implementation detail you are free to change, not a contract. If you switch the underlying sort key next year, old cursors from bookmarked URLs should fail with a clear `invalid_cursor` error, not silently return wrong data.

### Which one, when

| | Offset | Keyset / cursor |
|---|---|---|
| Random access (jump to page 40) | Yes | No, forward/backward only |
| Stable under concurrent writes | No | Yes |
| Performance at depth | Degrades (O(n)) | Constant (O(log n) via index) |
| Good fit | API keys list, admin tables, small collections | `requests`, usage logs, any high-write table, infinite scroll, bulk export |

Relay's dashboard uses offset for `/v1/api-keys` (a tenant has maybe a dozen) and keyset for `/v1/requests` and `/v1/usage` (thousands to millions of rows, growing every second). Do not pick one pagination style for the whole API. Pick it per resource, based on how that resource actually grows and gets queried.

---

## The New Verb: QUERY

Every API eventually needs to filter on more than a URL can comfortably hold. Relay's usage dashboard wants to filter `requests` by a combination of model, status, date range, cost threshold, and cache hit state, at once. That does not fit cleanly in query parameters:

```
GET /v1/requests?model=gpt-5.5&status=success&cache_hit=true&min_cost=0.01&created_after=2026-07-01&created_before=2026-07-20
```

It technically works, until the filter needs an array of models, a nested boolean condition, or hits a URL length limit some proxy enforces. The traditional workaround is `POST /v1/requests/search` with the filter in the body. It works, but it is semantically wrong: a search is a read, and POST tells every cache, proxy, and retry mechanism between the client and your server "this is a write, do not cache it, do not safely retry it, it might not be idempotent." That mismatch is a real cost, and it is why Stripe, GitHub, and most search-heavy APIs use `POST /search` anyway: it is the wrong method, and it is still the pragmatic choice, because nothing better existed until recently.

As of June 2026, HTTP has an actual answer: **QUERY**, standardized as [RFC 10008](https://datatracker.ietf.org/doc/rfc10008/). It is the first new HTTP method since PATCH in 2010. QUERY is what the name suggests: a read, like GET, that carries a request body, like POST.

```
QUERY /v1/requests
Content-Type: application/json

{
  "filter": {
    "model": ["gpt-5.5", "claude-3-5-sonnet"],
    "status": "success",
    "cache_hit": true,
    "cost": { "gte": 0.01 },
    "created_at": { "after": "2026-07-01", "before": "2026-07-20" }
  },
  "sort": "created_at desc",
  "limit": 50
}
```

The important part is not the syntax, it is the contract. QUERY is defined as safe and idempotent, the same guarantees GET carries. That means a proxy or gateway is allowed to cache a QUERY response and safely retry it on a network failure, the same way it would a GET, something it can never safely assume about a POST body. For an endpoint like Relay's usage search, that is a real operational win: it means Relay's own gateway can add caching in front of `QUERY /v1/requests` without special-casing it, because HTTP already tells it that this method is safe to cache.

The honest caveat: QUERY took eleven years to standardize (it spent six of those years named SEARCH before the rename), and as of mid-2026 support is still filling in. Node.js parses it natively, ASP.NET Core 10 recognizes it, Jetty and Tomcat support is in flight. Older reverse proxies, some corporate firewalls, and plenty of client libraries do not know what to do with a request body on a method they have never heard of. The pragmatic default for a production API in 2026 is still `POST /v1/requests/query` (name the endpoint `query`, not `search`, to signal the intent even while using POST as the transport), with a plan to add QUERY as the primary method once your infrastructure and clients catch up, keeping POST around as a fallback. Relay's API does exactly this: `POST /v1/requests/query` today, documented as "QUERY-semantics over POST," with QUERY itself supported as soon as the load balancer in front of it does.

---

## Verbs, Actions, and Idempotency

Standard CRUD covers create, read, update, delete. Real APIs need a fifth category: actions that change state without being any of the four. Revoking a key. Retrying a failed batch job. Canceling a subscription.

```
POST /v1/api-keys/{id}/rotate
POST /v1/batches/{id}/cancel
POST /v1/webhooks/{id}/test
```

These are POST because they are not safe (they change state) and not idempotent by default (calling `rotate` twice generates two different new keys). That second property is the actual problem, because networks fail mid-request. A client calls `POST /v1/api-keys/{id}/rotate`, the connection drops before the response arrives, and the client cannot tell whether the rotation happened or not. Retrying blindly might rotate the key twice, invalidating a key the client thinks is still valid.

The fix is the `Idempotency-Key` header, the same pattern Stripe popularized:

```
POST /v1/api-keys/key_9fL2xQ/rotate
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7
```

```go
func IdempotencyMiddleware(store IdempotencyStore) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := r.Header.Get("Idempotency-Key")
            if key == "" || r.Method == http.MethodGet {
                next.ServeHTTP(w, r)
                return
            }

            cached, found := store.Get(r.Context(), key)
            if found {
                w.Header().Set("Content-Type", "application/json")
                w.WriteHeader(cached.StatusCode)
                w.Write(cached.Body)
                return
            }

            rec := httptest.NewRecorder()
            next.ServeHTTP(rec, r)

            // Only successful responses are safe to replay verbatim.
            // A 5xx might mean the write never committed, so let a retry try again for real.
            if rec.Code < 500 {
                store.Set(r.Context(), key, rec.Code, rec.Body.Bytes(), 24*time.Hour)
            }

            w.WriteHeader(rec.Code)
            w.Write(rec.Body.Bytes())
        })
    }
}
```

The store is keyed on the idempotency key, scoped to the tenant (two different tenants can reuse the same UUID by coincidence), and holds the response for 24 hours. The first request with a given key executes and its response is cached. Every retry with the same key, whether it is the client retrying after a timeout or an accidental double-click, gets the original response replayed, not a second execution.

```mermaid
sequenceDiagram
    actor Client
    participant Relay
    participant Store as Idempotency Store

    Client->>Relay: POST /v1/api-keys/key_9fL2xQ/rotate (Idempotency-Key: 7c9e...)
    Relay->>Store: lookup key
    Store-->>Relay: not found
    Relay->>Relay: execute rotation
    Relay->>Store: cache response (24h)
    Relay-->>Client: 200 OK, new key

    Note over Client,Relay: connection drops before client sees the response

    Client->>Relay: POST /v1/api-keys/key_9fL2xQ/rotate (Idempotency-Key: 7c9e..., retry)
    Relay->>Store: lookup key
    Store-->>Relay: found, cached response
    Relay-->>Client: 200 OK, same response (no second rotation)
```

```sql
CREATE TABLE idempotency_keys (
    tenant_id     TEXT        NOT NULL REFERENCES tenants(id),
    key           TEXT        NOT NULL,
    status_code   INT         NOT NULL,
    response_body JSONB       NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at    TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (tenant_id, key)
);
```

Apply this to every non-idempotent POST, not just the "dangerous" ones. `POST /v1/api-keys` (create) needs it too. Without it, a client retrying a timed-out create request ends up with two API keys instead of one, and neither the client nor the tenant has any way to know that happened until they see two keys in the dashboard.

---

## Versioning

Relay's hot path is already `/v1/chat/completions`. The version lives in the URL, and that choice was made for a reason worth stating explicitly: it is the simplest option to get right, and simple is what you want for a contract other people's production systems depend on.

The alternative, header-based versioning (`Accept: application/vnd.relay.v2+json` or a custom `X-API-Version` header), keeps URLs clean and lets you version at a finer grain than "the whole API." It also means every debugging session starts with "what version is this request even using," because it is invisible in the URL, invisible in a browser address bar, invisible in a curl command someone pastes into a support ticket. For an API whose primary consumers are backend developers pasting example requests into their own code (which is every API in this post), that invisibility is a real cost that outweighs the cleanliness argument. Use URL versioning unless you have a specific reason not to.

Whichever you pick, the harder discipline is not breaking `v1` while `v2` doesn't exist yet. Two rules cover almost every case:

**Additive changes do not need a new version.** Adding a new optional field to a response, adding a new endpoint, adding a new optional request parameter: none of these break an existing client, because existing clients simply ignore fields they do not recognize. This is why a JSON API should never validate against a schema that rejects unknown fields on the way in from the server to the client.

**Breaking changes need a new version, full stop.** Removing a field, renaming a field, changing a field's type (a string ID becoming an integer), changing the meaning of an existing status code: all of these break something, somewhere, for someone. There is no clever way around this. You cut `v2`.

When `v2` ships, `v1` does not disappear immediately. It gets a deprecation window, communicated in the response headers of the old version so that anyone still calling it finds out from their own logs, not from a changelog they never read:

```
Deprecation: true
Sunset: Sat, 1 May 2027 00:00:00 GMT
Link: <https://relay.lorbic.com/docs/v2/migration>; rel="successor-version"
```

`Deprecation` and `Sunset` are real, registered HTTP headers (RFC 8594 and the corresponding Deprecation header draft), not something Relay invented. A well-behaved client library can check for the `Deprecation` header and log a warning automatically, which is a much more reliable distribution channel for "please migrate" than an email that goes to whoever signed up eighteen months ago and has since left the company.

---

## Caching

The multi-tenant post covered caching the actual LLM responses, exact-match and semantic. This is a different layer: standard HTTP caching for the dashboard's REST resources, `GET /v1/api-keys`, `GET /v1/requests/{id}`, the things a tenant's own dashboard fetches on every page load.

The two headers that matter are `ETag` and `Cache-Control`.

```go
func (h *Handler) GetAPIKey(w http.ResponseWriter, r *http.Request) {
    key, err := h.store.GetAPIKey(r.Context(), chi.URLParam(r, "id"))
    if err != nil {
        writeError(w, 404, "not_found")
        return
    }

    etag := fmt.Sprintf(`"%x"`, sha256.Sum256(mustJSON(key)))
    if inm := r.Header.Get("If-None-Match"); inm == etag {
        w.WriteHeader(http.StatusNotModified)
        return
    }

    w.Header().Set("ETag", etag)
    w.Header().Set("Cache-Control", "private, max-age=30")
    writeJSON(w, 200, key)
}
```

`ETag` is a hash of the current representation. A client that already has a copy sends it back as `If-None-Match` on the next request, and if nothing changed, the server returns `304 Not Modified` with an empty body instead of resending the whole object. For a dashboard polling `GET /v1/api-keys` every few seconds to keep its UI fresh, this turns most of those requests into a few hundred bytes of headers instead of a full JSON payload.

`Cache-Control: private, max-age=30` tells any cache (browser, CDN, intermediate proxy) that this response belongs to one tenant (`private`, never shared across tenants) and can be reused for 30 seconds without even asking the server. `no-store` on anything containing a secret, like the one-time display of a freshly created API key, which must never be cached anywhere, by any layer, for any amount of time.

```mermaid
sequenceDiagram
    actor Client
    participant Relay

    Client->>Relay: GET /v1/api-keys/key_9fL2xQ
    Relay-->>Client: 200 OK (ETag: "a1b2c3", Cache-Control: private, max-age=30)

    Note over Client: within max-age, client serves from local cache, no request sent

    Note over Client: past max-age, client revalidates

    Client->>Relay: GET /v1/api-keys/key_9fL2xQ (If-None-Match: "a1b2c3")
    alt nothing changed
        Relay-->>Client: 304 Not Modified (empty body)
    else key was renamed
        Relay-->>Client: 200 OK (ETag: "d4e5f6", full body)
    end
```

Cache invalidation on writes is the part people forget. When a tenant renames an API key, the write handler needs to either bump a version number embedded in the ETag computation (so the next read naturally produces a different hash) or explicitly purge the cached entry if you are running a CDN or reverse proxy cache in front of the API. An ETag that is a hash of the current row's content, computed fresh on every read as in the example above, sidesteps this entirely: it is always correct because it is never stored, only recomputed. The tradeoff is a hash computation on every request instead of a cheap lookup, which is a fine trade for a dashboard's read volume and a bad one for a hot path serving millions of requests a second, where you would maintain a version column instead and hash nothing.

---

## BFF: Backend for Frontend

Relay's dashboard home screen needs, in one page load: the tenant's plan and status, their last 5 API keys, this month's usage rollup, and any active billing alerts. That is four different internal queries (`tenants`, `api_keys`, `usage_daily`, `billing_periods`), and if the dashboard frontend calls Relay's general-purpose API directly, it makes four round trips, each carrying its own auth check, its own network latency, and its own over-fetched fields the dashboard does not render.

A **Backend for Frontend** is a thin API layer that exists for exactly one client (the dashboard, in this case) and shapes its response around what that one screen needs, not around the general resource model underneath.

```mermaid
graph TB
    APP[Dashboard Frontend] -->|GET /v1/dashboard/home| BFF[Dashboard BFF]

    BFF --> T[Tenants Service]
    BFF --> K[API Keys Service]
    BFF --> U[Usage Service]
    BFF --> BILL[Billing Service]

    T -.-> BFF
    K -.-> BFF
    U -.-> BFF
    BILL -.-> BFF

    BFF -->|one shaped response| APP
```

```go
type DashboardHomeResponse struct {
    Tenant       TenantSummary   `json:"tenant"`
    RecentKeys   []APIKeySummary `json:"recent_keys"`
    UsageThisMonth UsageSummary  `json:"usage_this_month"`
    BillingAlert  *BillingAlert  `json:"billing_alert,omitempty"`
}

func (bff *DashboardBFF) Home(w http.ResponseWriter, r *http.Request) {
    tenantID := TenantFromContext(r.Context())

    var tenant Tenant
    var keys []APIKey
    var usage UsageSummary
    var billing *BillingAlert

    g, ctx := errgroup.WithContext(r.Context())
    g.Go(func() (err error) { tenant, err = bff.tenants.Get(ctx, tenantID); return })
    g.Go(func() (err error) { keys, err = bff.keys.ListRecent(ctx, tenantID, 5); return })
    g.Go(func() (err error) { usage, err = bff.usage.ThisMonth(ctx, tenantID); return })
    g.Go(func() (err error) { billing, err = bff.billing.ActiveAlert(ctx, tenantID); return })

    if err := g.Wait(); err != nil {
        writeError(w, 500, "internal_error")
        return
    }

    writeJSON(w, 200, DashboardHomeResponse{
        Tenant:         toTenantSummary(tenant),
        RecentKeys:     toAPIKeySummaries(keys),
        UsageThisMonth: usage,
        BillingAlert:   billing,
    })
}
```

The four internal calls still happen, but they happen concurrently, server-side, over a fast internal network, and the dashboard gets one response shaped exactly like the screen it renders. No over-fetching (the dashboard never sees the full `api_keys` row, `key_hash`, `created_by`, none of that, just the five fields the summary card shows), and no round trips from the browser, which is the network path that actually has latency worth worrying about.

The tradeoff is real, not free: a BFF is another service to deploy, monitor, and keep in sync with the resources it aggregates. It is worth it when a specific client (a mobile app, a dashboard, a public-facing website) has a shape of need that is meaningfully different from your general resource API, and painful when teams reach for it reflexively for every client, producing a pile of near-identical BFFs that all drift independently. Relay has exactly one BFF, for the dashboard. The mobile SDK and CLI tooling call the general API directly, because their needs match the resource model closely enough that a BFF would just be a pass-through layer adding latency for no reshaping benefit.

---

## SDUI: Server-Driven UI

The BFF above still assumes the dashboard's frontend code knows how to render a "billing alert" card, because that layout is baked into the frontend's own code. Server-driven UI goes one step further: the server sends not just the data, but a description of what to render and how to lay it out, and the client is a generic renderer that interprets that description.

Relay's onboarding wizard is the natural candidate. It has steps that change: today it is "add a payment method, create your first API key, pick a default model." Next quarter product wants to add a step, reorder them, or A/B test two different copies of step 2, all without shipping a new frontend build or waiting on a mobile app store review for the same change on iOS.

```json
GET /v1/dashboard/onboarding-ui

{
  "version": "2026-07-20",
  "screen": {
    "type": "wizard",
    "steps": [
      {
        "id": "payment_method",
        "title": "Add a payment method",
        "components": [
          { "type": "text", "content": "You get $5 in free credits during trial." },
          { "type": "button", "action": "open_stripe_portal", "label": "Add card" }
        ]
      },
      {
        "id": "first_key",
        "title": "Create your first API key",
        "components": [
          { "type": "button", "action": "create_api_key", "label": "Generate key" },
          { "type": "code_block", "bind": "generated_key" }
        ]
      }
    ]
  }
}
```

The client renderer knows how to draw a `text`, a `button`, and a `code_block`, and how to execute a fixed, small set of named actions (`open_stripe_portal`, `create_api_key`). It does not know anything about onboarding specifically. Add a third step, reorder the two existing ones, or change the copy, and every client picks it up on next load, no release required, on web, iOS, and Android at once.

```mermaid
flowchart LR
    A["Server: GET /v1/dashboard/onboarding-ui"] --> B["JSON schema:\nscreen + steps + components"]
    B --> C["Generic client renderer"]
    C --> D{"Component type\nrecognized?"}
    D -->|yes| E["Render text / button / code_block"]
    D -->|no, older client| F["Render placeholder\nnever crash"]
    E --> G["User taps button"]
    G --> H["Execute named action\ne.g. create_api_key"]
```

The real system has a few more pieces than the JSON above suggests: a small, versioned component vocabulary the client renderer supports (this is the part that has to ship in a client release, so keep it deliberately small and stable), a schema-validated payload so a malformed server response cannot crash the renderer, and a fallback screen for when the client's renderer version is older than the payload it received (an unrecognized component type should render as nothing, or a generic placeholder, never a crash).

SDUI is worth the investment for exactly the kind of surface that changes often and is annoying to gate behind app store review: onboarding flows, promotional banners, plan comparison screens, empty states. It is a bad fit for anything with complex client-side interaction or heavy platform-specific behavior (a rich text editor, a data table with column resizing), where forcing the interaction model through a generic JSON schema costs more in renderer complexity than it saves in release cycles. Relay uses it for onboarding and dashboard banners. The actual usage charts and the API key management table are hand-built in the dashboard frontend, because a chart's interaction model does not compress into a generic component vocabulary without losing the thing that makes it a good chart.

---

## WebSocket and Server-Sent Events

The multi-tenant post already covers SSE for streaming chat completions: the provider sends chunks, Relay pipes them straight through as `data:` events, one direction, client to open connection and read. That pattern generalizes past LLM streaming to anything that is a server pushing a sequence of updates: a live token counter on the dashboard, a tail of the tenant's most recent requests as they land.

```go
func (h *Handler) StreamUsage(w http.ResponseWriter, r *http.Request) {
    tenantID := TenantFromContext(r.Context())
    flusher, ok := w.(http.Flusher)
    if !ok {
        writeError(w, 500, "streaming_unsupported")
        return
    }

    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")

    updates := h.usage.Subscribe(r.Context(), tenantID)
    defer h.usage.Unsubscribe(tenantID, updates)

    for {
        select {
        case <-r.Context().Done():
            return
        case u := <-updates:
            fmt.Fprintf(w, "data: %s\n\n", mustJSON(u))
            flusher.Flush()
        }
    }
}
```

SSE runs over plain HTTP, reconnects automatically in every browser's built-in `EventSource` client, and passes through ordinary HTTP infrastructure (load balancers, proxies, corporate firewalls) without special handling. Its one real limitation is that it is one-way. The server pushes, the client only ever reads.

WebSocket is the right tool when the client needs to talk back on the same connection, not just make a new HTTP request. Relay's dashboard has exactly one feature that needs this: a live request log viewer where a tenant can pause the stream, adjust a filter (show only errors, show only one model), and have that filter applied server-side without reopening the connection.

```mermaid
sequenceDiagram
    actor Tenant as Dashboard
    participant Relay as Relay (WebSocket)

    Tenant->>Relay: Upgrade: websocket
    Relay-->>Tenant: 101 Switching Protocols

    Relay-->>Tenant: {"type":"request_log","data":{...}}
    Relay-->>Tenant: {"type":"request_log","data":{...}}

    Tenant->>Relay: {"action":"filter","status":"error"}
    Relay-->>Tenant: {"type":"request_log","data":{...error only...}}

    Tenant->>Relay: {"action":"pause"}
    Note over Relay,Tenant: server stops pushing until resumed
```

| | SSE | WebSocket |
|---|---|---|
| Direction | Server to client only | Bidirectional |
| Transport | Plain HTTP, works over HTTP/2 | Its own protocol upgrade (`Upgrade: websocket`) |
| Reconnect | Automatic, built into `EventSource` | Manual, you write the reconnect logic |
| Proxy/firewall friendliness | High, looks like a normal long HTTP response | Lower, some corporate proxies block the upgrade |
| Right fit | Streaming completions, usage counters, notifications | Live filtering, chat-style back-and-forth, collaborative editing |

Default to SSE. It solves the actual problem (server pushes updates) with less protocol surface, less client code, and fewer things that can go wrong at the network layer. Reach for WebSocket only when the client genuinely needs to send messages back on the same open connection, not just make occasional separate HTTP calls alongside a stream it is reading.

---

## API Security

### Auth headers

Relay's hot path already established the pattern: `Authorization: Bearer rly_live_sk_...`. The dashboard API's JWT-based session auth uses the same header, same scheme, different token. This consistency matters more than it looks: one auth header format across the entire API means one code path for every client library to implement, instead of "use this header for these endpoints, that header for those."

Never accept credentials in the query string (`?api_key=...`). Query strings end up in access logs, browser history, and `Referer` headers sent to third-party resources on the same page. A credential that only ever appears in the `Authorization` header stays out of all three by default.

### Server-to-server communication

Bearer tokens work when a human or a client SDK is calling Relay. A different problem shows up when two backend services need to trust each other directly, with no human in the loop: Relay's billing worker calling Stripe, or an enterprise tenant's own internal service calling Relay's BYOK credential endpoint.

Two patterns cover almost every case:

**mTLS (mutual TLS)** where both sides present a certificate, is the right choice for a small, fixed set of trusted internal services that rarely change, like Relay's own microservices talking to each other inside a VPC. It is strong (the private key never leaves the holding service) but operationally heavy: certificate issuance, rotation, and revocation all need their own infrastructure.

**HMAC request signing** is the more common choice for API-to-API integrations across organizational boundaries, because it needs no certificate infrastructure on the caller's side, just a shared secret.

```go
func SignRequest(secret []byte, method, path string, body []byte, timestamp string) string {
    mac := hmac.New(sha256.New, secret)
    mac.Write([]byte(method + "\n" + path + "\n" + timestamp + "\n"))
    mac.Write(body)
    return hex.EncodeToString(mac.Sum(nil))
}

func VerifySignedRequest(secret []byte, r *http.Request, body []byte) error {
    ts := r.Header.Get("X-Relay-Timestamp")
    sig := r.Header.Get("X-Relay-Signature")

    t, err := strconv.ParseInt(ts, 10, 64)
    if err != nil || time.Since(time.Unix(t, 0)) > 5*time.Minute {
        return ErrStaleRequest
    }

    expected := SignRequest(secret, r.Method, r.URL.Path, body, ts)
    if !hmac.Equal([]byte(expected), []byte(sig)) {
        return ErrInvalidSignature
    }
    return nil
}
```

The timestamp check matters as much as the signature itself. Without it, a captured request (from a compromised log, a network dump, a misconfigured proxy that echoes requests) can be replayed indefinitely with a valid signature, because the signature alone only proves the request was not tampered with, not that it is fresh. Rejecting anything older than a few minutes closes that window without needing a stateful nonce store for most use cases. If you need to guard against replay within that window too (the same signed request sent twice, both within 5 minutes), track seen `(timestamp, signature)` pairs in Redis with a TTL matching the window, and reject duplicates.

### Webhook receiver APIs

A webhook receiver is unusual among "APIs you build" because you are not the one designing the request format, the sender is (Stripe, a provider, another tenant's system), and you have to accept it anyway. Relay has two separate webhook relationships to get right, in opposite directions.

**Receiving**: Stripe calls Relay's `POST /v1/webhooks/stripe` when a payment succeeds or fails. Every webhook receiver needs the same three defenses, regardless of who is sending it:

1. **Verify the signature before doing anything else.** Stripe signs every webhook with a secret only Relay and Stripe know. Skipping this check means anyone who finds the URL can POST a fake "payment succeeded" event and reactivate a suspended tenant's account without paying anything.

```go
func StripeWebhookHandler(secret string) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        body, _ := io.ReadAll(r.Body)
        sig := r.Header.Get("Stripe-Signature")

        event, err := webhook.ConstructEvent(body, sig, secret)
        if err != nil {
            writeError(w, 400, "invalid_signature")
            return
        }

        // Ack immediately, process async. Stripe retries on anything but 2xx,
        // and a slow handler here is Relay's problem, not Stripe's to wait on.
        if err := enqueueWebhookEvent(r.Context(), event); err != nil {
            writeError(w, 500, "enqueue_failed")
            return
        }
        w.WriteHeader(http.StatusOK)
    }
}
```

```mermaid
sequenceDiagram
    participant Stripe
    participant Relay as Relay /v1/webhooks/stripe
    participant Queue
    participant Worker

    Stripe->>Relay: POST event + Stripe-Signature
    Relay->>Relay: verify signature
    alt invalid signature
        Relay-->>Stripe: 400 Invalid signature
    else valid
        Relay->>Queue: enqueue event
        Relay-->>Stripe: 200 OK (fast ack)
        Queue->>Worker: dequeue
        Worker->>Worker: check processed_webhook_events (source, event_id)
        alt already processed
            Worker->>Worker: skip (duplicate delivery)
        else new event
            Worker->>Worker: update tenant billing status
        end
    end
```

2. **Acknowledge fast, process async.** The handler's only job is to verify the signature and enqueue the event. Anything slower (updating the tenant's billing status, sending a follow-up email) happens in a background worker reading off that queue. Stripe retries a webhook it does not get a `2xx` for within its timeout, and a synchronous handler that occasionally takes 8 seconds under load will get duplicate deliveries for the exact same event, on top of whatever else went wrong.

3. **Deduplicate by event ID, because "at least once" delivery is the norm, not the exception.** Every webhook sender you integrate with, Stripe included, documents that a given event might be delivered more than once. Store the event ID with a unique constraint and skip processing on a duplicate.

```sql
CREATE TABLE processed_webhook_events (
    source     TEXT NOT NULL,        -- stripe | provider_x
    event_id   TEXT NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (source, event_id)
);
```

**Sending**: Relay also sends webhooks, to tenants, when their batch job finishes or a provider outage affects their traffic. This is the same problem from the other side, and the same three defenses apply in reverse: Relay signs every outbound webhook so the tenant's receiver can verify it actually came from Relay, retries with backoff on anything but a `2xx` from the tenant's endpoint, and includes an `event_id` so the tenant's own receiver can deduplicate.

```sql
CREATE TABLE webhook_endpoints (
    id            TEXT        PRIMARY KEY,
    tenant_id     TEXT        NOT NULL REFERENCES tenants(id),
    url           TEXT        NOT NULL,
    secret        TEXT        NOT NULL,        -- shared secret, shown once at creation
    status        TEXT        NOT NULL DEFAULT 'active',
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE webhook_deliveries (
    id            TEXT        PRIMARY KEY,
    endpoint_id   TEXT        NOT NULL REFERENCES webhook_endpoints(id),
    event_id      TEXT        NOT NULL,
    event_type    TEXT        NOT NULL,        -- batch.completed | provider.degraded
    payload       JSONB       NOT NULL,
    status        TEXT        NOT NULL DEFAULT 'pending', -- pending | delivered | failed
    attempt_count INT         NOT NULL DEFAULT 0,
    last_attempt_at TIMESTAMPTZ,
    next_attempt_at TIMESTAMPTZ,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(endpoint_id, event_id)
);
```

`webhook_deliveries` is the audit trail tenants need when they ask "why didn't I get notified my batch finished." It records every attempt, every failure, and lets the dashboard show a delivery log, plus a manual "resend" button that replays the same `event_id` and payload rather than generating a new event, keeping the tenant's own deduplication intact.

---

## Bulk Data Load APIs

Everything so far has been synchronous: request in, response out, within one HTTP round trip. Some operations do not fit that shape. A tenant wants to submit 50,000 chat completion requests at once for an overnight batch job, or export a year of `requests` history for their own analytics warehouse. Neither belongs on the request-response path.

### Bulk submission: the async job pattern

```
POST /v1/batches
Content-Type: application/jsonl

{"custom_id": "req-1", "model": "gpt-5.5", "messages": [...]}
{"custom_id": "req-2", "model": "gpt-5.5", "messages": [...]}
... (50,000 lines)
```

```json
201 Created
{
  "id": "batch_4mK9pL",
  "status": "validating",
  "request_count": 50000,
  "created_at": "2026-07-26T10:00:00Z"
}
```

The client gets back a job ID immediately, not 50,000 responses. Relay validates and processes the batch asynchronously (typically at a lower priority and lower cost than the synchronous API, since there is no client waiting on the other end of an open connection), and the tenant finds out it is done one of two ways: polling `GET /v1/batches/{id}` for a `status` field, or, better, registering a webhook for `batch.completed` and finding out the moment it happens instead of however often they happen to poll.

```sql
CREATE TABLE batches (
    id              TEXT        PRIMARY KEY,
    tenant_id       TEXT        NOT NULL REFERENCES tenants(id),
    status          TEXT        NOT NULL DEFAULT 'validating', -- validating | in_progress | completed | failed | expired
    request_count   INT         NOT NULL,
    completed_count INT         NOT NULL DEFAULT 0,
    failed_count    INT         NOT NULL DEFAULT 0,
    input_file_id   TEXT        NOT NULL,
    output_file_id  TEXT,
    error_file_id   TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    completed_at    TIMESTAMPTZ,
    expires_at      TIMESTAMPTZ NOT NULL
);
```

Each line in the input file is independent and carries its own `custom_id`, so a failure on line 30,000 does not invalidate the other 49,999. The output file matches each result back to its `custom_id`, and a separate error file lists exactly which lines failed and why, so the tenant does not have to diff the whole input against the whole output to find the two that broke.

```mermaid
stateDiagram-v2
    [*] --> validating : POST /v1/batches
    validating --> failed : malformed input file
    validating --> in_progress : validation passed
    in_progress --> completed : all lines processed
    in_progress --> failed : unrecoverable error
    completed --> expired : output not downloaded within retention window
    failed --> [*]
    expired --> [*]
    completed --> [*]
```

### Bulk export: the other direction

Exporting `requests` history works the same way in reverse: `POST /v1/exports` with a date range and format (CSV or JSONL), returns a job ID, and a webhook or poll tells the tenant when a download URL is ready. Never generate a large export synchronously inside a request handler. A "small" export today is a multi-gigabyte one in a year, and an HTTP request handler holding a connection open for minutes while it streams a database cursor to a response is exactly the kind of resource that starves every other tenant sharing that gateway pod, which is the noisy neighbor problem the multi-tenant post covered from the database side. The async job pattern sidesteps it entirely: the actual export work runs in a background worker with its own resource budget, not inside the request-handling pool.

The one rule that makes both directions safe: **never make a bulk operation block a request handler.** If it takes longer than a user is willing to stare at a spinner (a rough rule of thumb: a few hundred milliseconds), it is a job, not a request.

---

## The Full Surface, at a Glance

Putting the whole dashboard and management API together:

| Method | Path | Pagination | Notes |
|---|---|---|---|
| GET | `/v1/api-keys` | offset | small collection per tenant |
| POST | `/v1/api-keys` | | idempotency key required |
| POST | `/v1/api-keys/{id}/rotate` | | idempotency key required |
| DELETE | `/v1/api-keys/{id}` | | |
| GET | `/v1/requests` | keyset | high-write table |
| POST | `/v1/requests/query` | keyset | complex filters, QUERY-semantics over POST today |
| GET | `/v1/requests/stream` | | SSE, live tail |
| GET | `/v1/dashboard/home` | | BFF, aggregates 4 resources |
| GET | `/v1/dashboard/onboarding-ui` | | SDUI payload |
| POST | `/v1/batches` | | async job, JSONL input |
| GET | `/v1/batches/{id}` | | poll for status |
| POST | `/v1/exports` | | async job |
| POST | `/v1/webhooks/stripe` | | inbound receiver, signature verified |
| POST/PATCH/DELETE | `/v1/webhook-endpoints` | offset | tenant-configured outbound targets |

Every row on this table made an explicit choice: which pagination style, whether it needs an idempotency key, whether it is sync or async. None of those are free defaults. Each one is a deliberate answer to a question raised earlier in this post, applied to one specific resource.

---

## FAQ

**Why not use JSON Patch everywhere instead of JSON Merge Patch, since it is more powerful?**

Power you do not need is complexity you pay for anyway. Merge Patch covers "change these fields" with a plain JSON object anyone can write by hand while testing with curl. JSON Patch requires an array of `op`/`path`/`value` triples for even a single field change, and its real advantage (precise array operations) rarely comes up outside of collaborative editing tools. Default to Merge Patch and only add JSON Patch support if a specific endpoint genuinely needs array-level operations Merge Patch cannot express.

**Should the dashboard's list endpoints (like `/v1/api-keys`) use keyset pagination too, just to be consistent with `/v1/requests`?**

No. Consistency should apply within a category of decision (all list responses share an envelope shape), not force the same pagination strategy onto every resource regardless of how it actually grows. A tenant has a handful of API keys, ever. Offset pagination there is simpler to implement, gives useful random page access, and never encounters the depth or write-instability problems keyset solves, because the table it is querying never gets big enough for those problems to exist.

**Does GraphQL solve the over-fetching problem better than a BFF?**

It solves a similar problem differently: instead of a purpose-built endpoint per client, GraphQL lets each client specify exactly the fields and nested resources it wants in one query, against one general schema. That is a genuine strength for clients with very different, evolving data needs (a mobile app and a web dashboard wanting different subsets of the same resources). It comes with its own costs: caching gets harder because every query is potentially unique, and query complexity needs its own guardrails so a client cannot accidentally request a deeply nested, expensive-to-resolve tree. Whether that tradeoff beats a small number of purpose-built BFFs depends on how many genuinely different clients you have and how often their needs diverge. This post's promised GraphQL follow-up will design Relay's dashboard API both ways and compare them directly.

**Is QUERY actually going to replace `POST /search` any time soon?**

Not in the next year or two, no. Standardization is necessary but not sufficient. Every intermediary between a client and your server (corporate proxies, older load balancers, some client HTTP libraries) needs to know what to do with a request carrying a body on an unfamiliar method, and that kind of infrastructure upgrades slowly. The realistic path is: design your search endpoints today as if they were QUERY (safe, idempotent, filter in the body, name the endpoint `query` not `search`), implement them as POST for now, and switch the method once your gateway and client libraries support it. The design decision and the transport are separable, and getting the design right now costs nothing extra.

**Why sign outbound webhooks if the connection is already over HTTPS?**

HTTPS proves the payload was not tampered with in transit and that the tenant is talking to the right server. It says nothing about whether the payload the tenant's server receives on `/hooks/relay` actually came from Relay and not from anyone else who discovered that URL and decided to POST a fake `batch.completed` event to it. TLS secures the pipe. A signature over the payload proves who put something into the pipe. Both are needed, and neither substitutes for the other.

**What should a webhook receiver return if it cannot process the event right away?**

A `2xx` anyway, as soon as the signature is verified and the event is safely enqueued, not after it is fully processed. The sender's retry logic exists to handle real failures (the receiver was down, the network dropped the request), not to serve as your queue. If your handler holds the connection open until processing finishes, you have turned every sender's retry timeout into your own processing deadline, and a legitimately slow-but-successful process now looks like a failure and triggers a duplicate delivery on top of the one already in flight.

---

## Things Worth Their Own Post

This post stayed inside REST with JSON on purpose, because that combination alone has enough unresolved decisions to fill it. A few things named along the way that deserve their own full treatment:

- **GraphQL version of this API**: the same Relay dashboard, redesigned around a single schema and client-specified queries instead of purpose-built REST endpoints and a BFF. Where it genuinely wins, where it just relocates the same complexity, and what N+1 resolver problems look like once `requests` and `usage_daily` are both graph edges.
- **RPC and gRPC for internal services**: Relay's own microservices (the router talking to the health monitor, the billing worker talking to the usage aggregator) do not need REST's discoverability or HTTP semantics. Protobuf contracts, streaming RPCs, and what changes when the client is always your own code, not a third-party developer.
- **SOAP and XML for legacy enterprise integration**: some of Relay's enterprise prospects still run integration layers built for a SOAP world: WSDL contracts, XML schemas, and the reasons this stack still shows up in real RFPs from large, regulated companies, whether anyone likes it or not.


---

## Cache-Driven Development: Saving Your Database From Itself

**Date:** 2026-07-22
**URL:** http://localhost:1313/cache-driven-development/
**Description:** How to use cache to prevent your database from getting hammered. Understanding cache mechanics, types, strategies, and when each one wins.


There is a moment in every backend engineer's life when their database starts refusing connections.

Picture 2 a.m. on a Tuesday. The app is operating under normal traffic, nothing unusual. But the database connection pool is saturated. Queries are timing out. The monitoring dashboard shows 50,000 database operations per second, far beyond what the system should be handling.

When someone pulls the slow-query logs, the pattern is immediately clear. The same query appears thousands of times:

```sql
SELECT * FROM user_profiles WHERE user_id = ?
```

The individual query is fast and simple. The problem is scale. Thousands of concurrent requests are running this exact same query every few seconds. Each request makes the call, gets back the same data, discards it, and repeats the process.

The fix is straightforward: cache the result with a five-minute TTL. One line of code. The database load drops from saturated to 500 QPS and the incident is over.

This illustrates a larger point. This post is not about the sophisticated architecture of building distributed cache systems. It is about something more fundamental: cache is not a performance optimization you add later. It is a load shield built into the system from the start. Your database without proper caching is not a resilient system. It is a ticking timer.

---

## The Scale Problem

Let's do the math.

Say you have 10,000 users online. Each user's request does one database call that it doesn't have to do: maybe a lookup that gets cached, or a precomputation that could be stored. It's a small inefficiency. Your database can handle it.

Now imagine that query is slow. It takes 100 milliseconds. So 10,000 users, each doing that query once, generates 10,000 queries per second. At 100ms each, you've used up 1,000 seconds of database CPU per second. You don't have 1,000 CPUs.

Or worse: imagine that query is cached, but cached poorly. The TTL is short. Every 10 seconds, all 10,000 users' caches expire at once, and they all hit the database simultaneously. This **thundering herd** causes a spike of 10,000 queries in 1 second. The connection pool fills. Queries queue. Latency spikes. Users retry. The system fails.

This is not hypothetical. It happens in production at 3 a.m. And it happens because someone thought "we can fetch this on demand, it's fine."

The rule is simple: **every database call at request time that could have been cached multiplies by your concurrent user count**. If your users are making 10 requests per second, and each request does 1 uncached lookup, you are generating 10 uncached lookups per second per user. At 10,000 users, that is 100,000 uncached lookups per second.

Add a cache with a 5-minute TTL, and it drops to the cost of refreshing 10,000 users' worth of cached data across 5 minutes. That is 33 lookups per second. A reduction of three orders of magnitude.

```mermaid
graph LR
    A["1 user"] -->|1 DB call/sec| B["1 DB call/sec"]
    C["1,000 users"] -->|1 DB call each/sec| D["1,000 DB calls/sec"]
    E["10,000 users"] -->|1 DB call each/sec| F["10,000 DB calls/sec\nDatabase dies"]
    
    G["10,000 users\nwith 5min cache"] -->|33 cache refreshes/sec| H["33 DB calls/sec\nDatabase fine"]
    
    style F fill:#ffcccc
    style H fill:#ccffcc
```

Cache is not optional. It is infrastructure.

---

## Cache Strategies

Before diving into what to cache, you need to understand how to cache. There are two main strategies, and they work in opposite directions.

```mermaid
graph TD
    A["Data Changes"] 
    A -->|Write-Through| B["Update DB\nUpdate Cache\nBoth in sync"]
    A -->|Read-Through| C["Update DB\nCache still has old data\nNext read refreshes it"]
    
    style B fill:#e1f5ff
    style C fill:#fff3e0
```

### Write-Through Cache

In a write-through cache, whenever data changes, you update the cache at the same time as the database. The application code looks like:

```go
// Update the database
err := db.Update("user_profiles", userID, newData)
if err != nil {
  return err
}

// Simultaneously update the cache
cache.Set("user:"+userID, newData, 24*time.Hour)
return nil
```

The cache is always correct. If data changes, the cache changes immediately. Reads are fast because they hit the cache.

**Trade-off**: Write operations are slower. You are doing two writes instead of one. If the cache write fails but the database write succeeds, they become inconsistent: the database has the new data but the cache still has the old data. You have to handle that scenario.

**When it wins**: When writes are rare and reads are frequent. A user profile is written once, read hundreds of times. Update it once, serve it from cache for days. The cost of the extra write is paid back instantly.

### Read-Through Cache

In a read-through cache, you don't pre-populate. When a request needs data, it checks the cache first. On a miss, it fetches from the database and then stores the result in the cache for the next request.

```go
// Try cache first
if cached := cache.Get("user:" + userID); cached != nil {
  return cached, nil
}

// Cache miss, fetch from DB
data, err := db.Get("user_profiles", userID)
if err != nil {
  return nil, err
}

// Store for next time
cache.Set("user:"+userID, data, 5*time.Minute)
return data, nil
```

The cache fills up as requests come in. You don't pre-populate anything.

**Trade-off**: The first read is slow. Subsequent reads are fast. If the cache TTL expires and nobody has read the data in that window, you are right back to a database query on the next read.

**When it wins**: When you don't know what data will be accessed, or when the dataset is too large to precompute. A user searches for something. The first search query is slow. But if 10 users search for the same thing in the next 5 minutes, the next 9 are fast.

### Hybrid: Write-Through Cache, Read-Through Fallback

In practice, most systems use both. Write-through for things you control and write frequently. Read-through for everything else. Precompute the data you access constantly (the hot path), and for less frequently accessed data, compute it on the first request and cache the result for subsequent requests.

---

## Types of Caches

Now that you know how to cache, you need to know what to cache. There are many kinds. They solve different problems.

```mermaid
graph TD
    A["Cache Types"]
    A --> B["In-Memory"]
    A --> C["Database"]
    A --> D["HTTP/CDN"]
    A --> E["Application-Level"]
    
    B --> B1["Precomputed\nTables"]
    B --> B2["Document\nCache"]
    
    E --> E1["Function\nCache"]
    E --> E2["Route\nCache"]
    E --> E3["TTL\nCache"]
    E --> E4["Context\nCache"]
```

### 1. Precomputed Tables

The simplest cache is a table. You compute something expensive once, store the result, and point reads to that table instead of recomputing.

Example: a user's follower count. Computing it from scratch means counting rows in a followers table. At scale, that is slow. Instead, maintain a `user_stats` table that stores a copy of the follower count alongside other user statistics. Update this copy whenever someone follows or unfollows. Now reads are instant (no counting needed).

```sql
SELECT follower_count FROM user_stats WHERE user_id = ?
```

This is write-through cache in database form. You update it when data changes, and reads are instant.

**Cost**: Storage and maintenance overhead. You are duplicating data. If the count gets out of sync with the source, you have to repair it.

**When to use**: When you read frequently and writes are predictable. Aggregations, counts, rankings. Things that are expensive to compute and updated only when specific events happen (someone follows, a post gets a like).

### 2. Function-Result Cache

Cache the output of expensive functions. If a function is deterministic (same input always gives same output), cache the result.

Example: converting markdown to HTML. You have a blog post in markdown. You could render it every time someone views the page. Or cache the HTML result. When the post is edited, invalidate the cache.

```go
// Hash the function input and use it as the cache key
inputHash := md5.Sum([]byte(markdownContent))
cacheKey := "md_render:" + hex.EncodeToString(inputHash[:])

if cached := cache.Get(cacheKey); cached != nil {
  return cached.(string)
}

html := renderMarkdownToHTML(markdownContent)
cache.Set(cacheKey, html, 0) // No expiration, invalidate on write
return html
```

This is common with image processing, PDF generation, or any computation that is deterministic but expensive.

**Cost**: Cache invalidation. If you change how the function works, old cached results are wrong. You need a versioning strategy.

**When to use**: Any function you call frequently with the same inputs. Expensive computations that don't change unless the input changes.

### 3. Route Cache

Cache entire HTTP responses. If a route is expensive (multiple database queries, rendering), cache the HTML result.

```go
// Check if we have a cached response
if cached := cache.Get("route:/blog/" + postID); cached != nil {
  w.Header().Set("Content-Type", "text/html")
  w.Write(cached.([]byte))
  return
}

// Generate the response
data, err := db.GetPost(postID)
// ... render HTML ...
responseBody := renderTemplate(data)

cache.Set("route:/blog/"+postID, responseBody, 24*time.Hour)
w.Write(responseBody)
```

Popular on static or semi-static pages. Blog posts, documentation, product pages. Cache the entire rendered output.

**Cost**: Cache invalidation. When a post is edited, you have to invalidate the cached response.

**When to use**: Routes that are read much more than they are written. Static content, public pages, anything that does not change per-user.

### 4. CDN Cache

A CDN (Content Delivery Network) caches static files and HTTP responses at the edge, close to users. Requests never hit your origin server.

Your infrastructure looks like:

```
User in London
    ↓
CDN edge server in London (has cached copy of your JS/CSS/images)
    ↓
User in Singapore
    ↓
CDN edge server in Singapore (also has cached copy)
    ↓
Origin server (only used on cache misses or refreshes)
```

This is handled by HTTP headers. You tell the CDN "this asset can be cached for 30 days" using the `Cache-Control` header.

```
Cache-Control: public, max-age=2592000
```

**Cost**: You are not in control of cache invalidation. If you set `max-age=30d` on a JavaScript file and it has a bug, users will serve the old version for days.

**When to use**: Versioned assets (JS/CSS with content hashes in the filename), images, and anything that doesn't need to change instantly.

### 5. Database Document Cache (Result Cache)

Some queries are expensive but produce large, well-defined result sets. Instead of running the query every time, compute it once, store it as a document, and serve from there.

Example: a user's feed. Computing a feed involves joining posts from 100 followed accounts, ordering by relevance, and filtering. That is three or four queries. Cache the entire result as a document.

```go
// Try to get the cached feed document
if cached := cache.Get("feed:user:" + userID); cached != nil {
  return json.Unmarshal(cached, &feed)
}

// Expensive query: get all relevant posts, join, order, filter
feed, err := db.GetUserFeed(userID)
if err != nil {
  return nil, err
}

// Cache the entire result set
feedJSON, _ := json.Marshal(feed)
cache.Set("feed:user:"+userID, feedJSON, 5*time.Minute)
return feed, nil
```

This is read-through caching. The cache fills as users request their feeds.

**Cost**: Memory usage can be high. You are storing entire result sets. If the feed is 500 posts per user and you have a million users, that is gigabytes of memory.

**When to use**: Results that are expensive to compute but referenced frequently. User-specific data like feeds, recommendations, or personalized lists.

### 6. Temp Expiring Cache (TTL-Based Cache)

The simplest cache. Store data with a short expiration time. When it expires, the next request recomputes it.

```go
// Set with a 5-minute TTL
cache.Set("expensive_stat", computeExpensiveValue(), 5*time.Minute)

// Later, when you need it:
value := cache.Get("expensive_stat")
if value == nil {
  value = computeExpensiveValue()
  cache.Set("expensive_stat", value, 5*time.Minute)
}
return value
```

No invalidation logic. No coordination. Data expires automatically.

**Cost**: Stale data. Between the time the cache expires and the next request, the value is stale. If data changes and the cache has 4 minutes left, a user might see the old value for up to 4 minutes.

**When to use**: When staleness is acceptable. Metrics, statistics, things that don't need to be fresh to the second. The 5-minute user count on your dashboard doesn't need to be exact.

### 7. Context-Aware Cache (In-Request Cache)

In Go, the `context` package is often used to pass values through a request chain. You can use it as a cache to avoid repeated database queries within a single request.

```go
func (s *Service) GetUser(ctx context.Context, userID string) (*User, error) {
  // Check if we already fetched this user in this request
  if cached := ctx.Value("user:" + userID); cached != nil {
    return cached.(*User), nil
  }

  // First fetch, hit the database
  user, err := s.db.GetUser(userID)
  if err != nil {
    return nil, err
  }

  // Store in context for this request chain
  ctx = context.WithValue(ctx, "user:"+userID, user)
  return user, nil
}
```

This prevents the same request from hitting the database twice for the same user, even if the request logic is complex and makes multiple calls to `GetUser()`.

**Cost**: Memory per request. If a request creates a million users in context, that is a million users in memory for that request. Also, the cache is thrown away after the request ends.

**When to use**: Within a single request, when you might fetch the same thing multiple times. API requests that do bulk operations. Queries inside loops.

---

## Understanding the Scale Multiplier

The math is simple, and brutal.

Say you have 1000 concurrent users. Each makes one request per second. And each request does one query it shouldn't do: maybe a count, maybe a lookup that should have been cached.

```
1000 users × 1 request/sec × 1 extra query/request = 1000 extra queries/sec
```

If that query takes 50ms, your database needs 50 seconds of CPU per second just to handle those. You don't have 50 CPU cores.

But here's the real problem: most systems don't start with 1000 users. They scale up. When you hit 10,000 users, the same code suddenly does 10,000 extra queries per second. Your database goes from stressed to dead.

This is not hypothetical. It happens because one engineer, one time, wrote a query that was "fast enough" without thinking about the multiplier.

With caching, the math flips:

```
Uncached version: 1000 users × 1 uncached query/request = 1000 queries/sec
Cached version (5 min TTL): 1 cache refresh per user per 5 minutes = 3.3 queries/sec
```

That is three orders of magnitude. The difference between your database staying alive and your pager going off at 3 a.m.

Here's what it looks like in practice:

```
Without cache:
10,000 users → 10,000 queries/sec → database CPU maxed → timeouts → cascading failures

With cache (5 min TTL):
10,000 users → 33 queries/sec → database barely notices → users happy
```

The rule: **Before deploying, ask yourself: does every request hit the database for something it could have cached?** If yes, you will regret it at scale.

---

## Cache Technologies

Different tools solve caching at different layers. Pick the right one.

### In-Memory Cache (Application)

Store data in memory within your application process. When memory fills up, the cache removes the least recently used items to make room for new ones.

```go
import "github.com/patrickmn/go-cache"

// Create a cache that expires items after 5 minutes
c := cache.New(5*time.Minute, 10*time.Minute)

// Store data
c.Set("user:123", userData, cache.DefaultExpiration)

// Retrieve data
if data, found := c.Get("user:123"); found {
  user := data.(User)
  // use user
}

// Manual invalidation
c.Delete("user:123")
```

**Pros**: Extremely fast. No network latency. Simple to debug.

**Cons**: Not shared across servers. If you have 10 application instances, each has its own cache, 10x the memory overhead. Cache invalidation has to be coordinated across all instances.

**Use for**: Request-level caching (context cache), warm-up data that doesn't change often, local state.

### Redis

In-memory key-value store accessible over the network. Supports expiration, lists, sets, sorted sets, and more.

```go
import "github.com/redis/go-redis/v9"

client := redis.NewClient(&redis.Options{
  Addr: "localhost:6379",
})

// Set with 5-minute expiration
err := client.Set(ctx, "user:123", jsonMarshal(userData), 5*time.Minute).Err()

// Get
val, err := client.Get(ctx, "user:123").Result()
if err == redis.Nil {
  // Cache miss, fetch from DB
} else if err != nil {
  // Redis error
} else {
  // Cache hit, unmarshal val
  var user User
  json.Unmarshal([]byte(val), &user)
}

// Increment (useful for counters)
client.Incr(ctx, "post:view-count:456")

// Sorted set (useful for leaderboards)
client.ZAdd(ctx, "leaderboard", &redis.Z{Score: 100, Member: "user:123"})

// Manual invalidation
client.Del(ctx, "user:123")
```

**Pros**: Shared cache across all servers. Rich data structures (lists, sets, sorted sets for leaderboards). Built-in TTL. Persistent snapshots to disk.

**Cons**: Extra network latency (~1-5ms). Single point of failure. If the Redis instance goes down, all cached data is lost and every request hits the database simultaneously (you can use Redis Cluster for redundancy, but that adds complexity). Memory cost grows with data size.

**Use for**: User sessions, rate limiting, feature flags, leaderboards, any cached data you want visible across servers.

### Memcached

Similar to Redis but simpler. Key-value store, expiration, done. No lists or sorted sets.

```go
import "github.com/bradfitz/gomemcache/memcache"

mc := memcache.New("localhost:11211")

// Set with 5-minute expiration (300 seconds)
err := mc.Set(&memcache.Item{
  Key: "user:123",
  Value: jsonMarshal(userData),
  Expiration: 300,
})

// Get
item, err := mc.Get("user:123")
if err == memcache.ErrCacheMiss {
  // Cache miss, fetch from DB
} else if err != nil {
  // Memcached error
} else {
  // Cache hit
  var user User
  json.Unmarshal(item.Value, &user)
}

// Delete
mc.Delete("user:123")
```

**Pros**: Very fast, very simple. Lower overhead than Redis. Excellent for pure key-value access patterns.

**Cons**: No persistence, no complex data types, no TTL per key granularity (uses seconds globally).

**Use for**: High-throughput caching where you just need basic key-value, and you don't need persistence or rich data structures.

### Nginx / Varnish (HTTP Cache)

Sit in front of your application and cache HTTP responses.

**Nginx example:**

```nginx
http {
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m;

  server {
    listen 80;

    location /api/posts {
      proxy_pass http://backend;
      proxy_cache my_cache;
      proxy_cache_valid 200 5m;
      proxy_cache_bypass $http_pragma $http_authorization;
      add_header X-Cache-Status $upstream_cache_status;
    }

    location /api/user/ {
      proxy_pass http://backend;
      # Don't cache authenticated requests
      proxy_cache_bypass $http_authorization;
    }
  }
}
```

**Varnish example:**

```vcl
backend default {
  .host = "localhost";
  .port = "8080";
}

sub vcl_backend_response {
  if (bereq.url ~ "^/api/posts") {
    set beresp.ttl = 5m;
  } else if (bereq.url ~ "^/api/user/") {
    set beresp.ttl = 0s; // Don't cache
  }
}
```

Requests for cached routes go through Nginx/Varnish. If a response is cached, it returns immediately without hitting your app. If not, it forwards to your backend, caches the response, and returns it.

**Pros**: Transparent caching. Your application doesn't need to know it's happening. Scales extremely well (thousands of requests per second). Can handle spike traffic without hitting backend.

**Cons**: Only works for HTTP. Hard to invalidate (you have to purge the cache explicitly). Not suitable for dynamic, user-specific, or authenticated content.

**Use for**: Public APIs, semi-static endpoints, content that doesn't change per-user, protecting your origin from traffic spikes.

### Database Built-In Caching

Modern databases like PostgreSQL have built-in caching (shared_buffers, OS page cache). You don't control it directly, but you can tune it.

```sql
-- In postgresql.conf
shared_buffers = 256MB
```

This is the first line of defense. Your OS also caches disk pages automatically (page cache). For most applications, this is enough for small datasets.

**Pros**: Zero application code. Automatic.

**Cons**: Limited to the database machine. If you add a second database for backup and failover, it has its own separate cache. Each database instance caches independently, so you're not sharing the benefit across servers. Not useful at massive scale.

**Use for**: Small datasets (< 10GB), when you don't want to deploy separate infrastructure.

---

## Putting It Together: A Real Example

A common pattern: an analytics endpoint that runs three expensive aggregates. Without caching, at 1000 concurrent users refreshing every 10 seconds, this is 100+ complex queries per second against a database designed for 50 QPS.

Here's the uncached approach:

```go
func GetDailyStats(w http.ResponseWriter, r *http.Request) {
  userID := r.URL.Query().Get("user_id")
  
  // Three expensive queries
  count := db.Query(`SELECT COUNT(*) FROM events WHERE user_id = ? AND date(created_at) = ?`, userID, today)
  revenue := db.Query(`SELECT SUM(amount) FROM events WHERE user_id = ? AND date(created_at) = ?`, userID, today)
  topEvents := db.Query(`SELECT event_type, COUNT(*) FROM events WHERE user_id = ? AND date(created_at) = ? GROUP BY event_type ORDER BY COUNT(*) DESC LIMIT 10`, userID, today)
  
  w.Header().Set("Content-Type", "application/json")
  json.NewEncoder(w).Encode(map[string]interface{}{
    "count": count,
    "revenue": revenue,
    "topEvents": topEvents,
  })
}
```

With layered caching:

```go
func GetDailyStats(w http.ResponseWriter, r *http.Request) {
  userID := r.URL.Query().Get("user_id")
  ctx := r.Context()
  
  // 1. Check context cache first (avoid refetching in a single request)
  cacheKey := "stats:" + userID + ":" + today
  if cached := ctx.Value(cacheKey); cached != nil {
    w.Header().Set("X-Cache", "context")
    return writeStats(w, cached)
  }
  
  // 2. Check Redis (shared across all servers, 5-minute TTL)
  if cached, err := redis.Get(ctx, cacheKey).Result(); err == nil {
    w.Header().Set("X-Cache", "redis")
    ctx = context.WithValue(ctx, cacheKey, cached)
    return writeStats(w, cached)
  }
  
  // 3. Cache miss, but before hitting DB, check if another request is computing
  // Use a lock to ensure only one request hits the DB for this key
  once, _ := computeOnce.LoadOrStore(cacheKey, &sync.Once{})
  once.(*sync.Once).Do(func() {
    // This block runs once, other requests wait for it to finish
    stats := computeStats(userID, today) // Hits DB
    statJSON, _ := json.Marshal(stats)
    
    // Store in Redis for next 5 minutes
    redis.Set(ctx, cacheKey, statJSON, 5*time.Minute)
    
    // Store in context for this request
    ctx = context.WithValue(ctx, cacheKey, stats)
  })
  
  // 4. At the HTTP layer, tell CDN to cache public dashboards
  if r.Header.Get("Authorization") == "" { // Public request
    w.Header().Set("Cache-Control", "public, max-age=60")
  }
  w.Header().Set("X-Cache", "db")
  
  // Get stats from context (now populated by the once.Do block)
  stats := ctx.Value(cacheKey)
  writeStats(w, stats)
}
```

```mermaid
graph TB
    User["User Request"]
    User -->|Check| L1["Context Cache\nwithin request"]
    L1 -->|Hit| Return1["Return cached\nzero latency"]
    L1 -->|Miss| L2["Redis Cache\n5 min TTL"]
    
    L2 -->|Hit| Return2["Return from Redis\n1-5ms latency"]
    L2 -->|Miss| Lock["sync.Once Lock\nonly 1 DB call"]
    
    Lock -->|Hits DB| Compute["Compute Stats\n3 queries"]
    Compute -->|Store| L2
    Lock -->|Others Wait| L2
    
    Return1 --> Success["✓ Fast Response"]
    Return2 --> Success
    
    style L1 fill:#e1f5ff
    style L2 fill:#f3e5f5
    style Lock fill:#fff3e0
    style Compute fill:#ffcccc
    style Success fill:#c8e6c9
```

What each layer does:

1. **Context cache**: Within a single request, avoid refetching the same stats if called multiple times. Zero network latency.
2. **Redis cache**: Shared across all servers with a 5-minute TTL. Most requests hit this.
3. **sync.Once lock**: When Redis expires, only one request recomputes. Others wait for it to finish and grab the result. Prevents thundering herd.
4. **CDN headers**: For public dashboards, tell the CDN to cache the response. Removes traffic before it reaches your servers.

The math:
- **Without cache**: 100 queries/sec hitting the database.
- **With cache**: ~3 queries/sec (one per 5 minutes per user across 10,000 users).

This pattern scales. The caching layers handle load at different points: within the request, across servers, and at the edge.

---

## Cache in Distributed Systems

When you move from one server to many, caching changes.

### Local Cache Problems

In-memory caches don't synchronize across servers. If you cache data locally on server A, server B doesn't know about it. Users get inconsistent views depending on which server handles their request.

```
User A makes request to server 1 → cache hit, sees data version 1
User A makes request to server 2 → cache miss, sees data version 2
User is confused.
```

Solution 1: Use a shared cache (Redis) instead of local cache.

Solution 2: Use versioning. Include a version number in the cache key, and bump the version when data changes.

```go
// Use version in the key
versionedKey := "user:" + userID + ":v" + user.Version
cache.Set(versionedKey, userData, 24*time.Hour)

// All servers see the same version
// No coordination needed
```

### Cache Invalidation Across Servers

When data changes on one server, how do you tell all the other servers to invalidate their cache?

```mermaid
graph LR
    A["Data Change\nServer A"]
    A -->|Event| Q["Message Queue"]
    Q -->|Subscribe| S1["Server 1\nInvalidate"]
    Q -->|Subscribe| S2["Server 2\nInvalidate"]
    Q -->|Subscribe| S3["Server 3\nInvalidate"]
    
    style A fill:#ffcccc
    style S1 fill:#ccffcc
    style S2 fill:#ccffcc
    style S3 fill:#ccffcc
```

Option 1: **Event-driven invalidation**. When a user updates their profile, publish an event to a message queue. All servers listen to that queue and invalidate the relevant cache key.

```go
// When profile is updated
profileUpdated := ProfileUpdatedEvent{UserID: userID, NewVersion: newVersion}
eventBus.Publish("profile.updated", profileUpdated)

// On all servers
eventBus.Subscribe("profile.updated", func(e ProfileUpdatedEvent) {
  cache.Delete("user:" + e.UserID + ":v*") // Delete old versions
})
```

Option 2: **Time-based invalidation (TTL)**. Don't try to coordinate. Just set a short TTL and let caches expire naturally.

```go
cache.Set("user:"+userID, data, 5*time.Minute)
// After 5 minutes, all servers refresh independently
```

Option 3: **Hybrid**. Set a medium TTL, but also support explicit invalidation when data changes.

```go
// Set with 1-hour TTL as a safety net
cache.Set("user:"+userID, data, 1*time.Hour)

// But also listen for updates and invalidate immediately
eventBus.Subscribe("profile.updated", func(e ProfileUpdatedEvent) {
  cache.Delete("user:" + e.UserID)
})
```

### Cache Warming

On startup, servers have empty caches. The first requests are slow (cache misses, database hits). To prevent this, you can "warm" the cache by preloading hot data.

```go
func (s *Server) Start() error {
  // On startup, preload the top 1000 users from the database
  topUsers, err := s.db.GetTopUsers(1000)
  if err != nil {
    return err
  }
  
  for _, user := range topUsers {
    key := "user:" + user.ID
    s.cache.Set(key, user, 24*time.Hour)
  }
  
  return s.ListenAndServe()
}
```

This ensures that as soon as the server comes online, the hot data is ready. Users get cache hits immediately instead of waiting for cache misses to populate it.

### Cascading Cache Failures

In a distributed system, if a shared cache (like Redis) goes down, all servers lose their cache simultaneously. If you don't handle this gracefully, your database gets hammered.

```go
// Bad: crash if Redis is unavailable
value := redis.Get("user:" + userID) // panics if Redis down

// Good: fall back to database on cache error
value, err := redis.Get("user:" + userID)
if err != nil {
  // Cache error, fall back to DB but don't cache the result
  // to avoid hitting the DB repeatedly
  return db.GetUser(userID), nil
}
return value, nil
```

Better: use a circuit breaker. A circuit breaker watches for repeated failures (cache is down, timeouts, etc.). Once failures exceed a threshold, it "opens" and stops attempting to use the cache, letting requests go straight to the database. Once the cache recovers, it slowly re-enables it.

```go
if cacheCircuitBreaker.IsOpen() {
  // Circuit is open because cache has been failing
  // Skip the cache and hit the database directly
  return db.GetUser(userID), nil
}

value, err := redis.Get("user:" + userID)
if err != nil {
  cacheCircuitBreaker.RecordFailure()
  // If failures pile up, the circuit breaker will open
  return db.GetUser(userID), nil // Fallback
}
cacheCircuitBreaker.RecordSuccess()
// Successful read means the cache is recovering
return value, nil
```

---

## The Gotchas

Cache is powerful, but easy to get wrong.

### Cache Invalidation Is Hard

The old saying: "There are only two hard things in Computer Science: cache invalidation and naming things."

When data changes, you have to invalidate the cache. Miss this, and users see stale data. Example:

```go
// Update user profile
db.UpdateUser(userID, newData)

// Oops, forgot to invalidate the cache
// Users still see the old profile for 24 hours
```

Solution: write invalidation at the same time as the write.

```go
db.UpdateUser(userID, newData)
cache.Delete("user:" + userID) // Invalidate cache
```

Or, use a pattern where cache keys change with data versions:

```go
// Include a version in the key
cacheKey := "user:" + userID + ":v" + user.Version
cache.Set(cacheKey, newData, 24*time.Hour)
```

### Thundering Herd

When a cache entry expires and multiple requests try to refresh it simultaneously, they all hit the database at once.

Solution 1: Use leases. Only one request recomputes the value, others wait.

```go
lease, err := cache.Lease("expensive:" + key)
if lease != nil {
  // This request won the lease and recomputes
  result := expensiveComputation()
  cache.Set(key, result, 5*time.Minute)
  lease.Release()
} else {
  // Other requests wait for the winner to populate the cache
  result := cache.WaitFor(key, 1*time.Second)
  return result
}
```

Solution 2: Extend TTL before expiry. Refresh the cache in the background before it expires.

### Hot Keys

Some keys are accessed far more than others. A celebrity's profile, a viral post, a feature flag that controls your entire app. If that key is cached on one Redis instance, that instance gets hammered while others are idle.

Solution: replicate hot keys across multiple cache instances. Or use a local cache in front of Redis.

### Storage Bloat

Caching everything leads to memory exhaustion. You cache the entire feed for every user, suddenly you need 10TB of Redis memory.

Solution: be selective. Cache only what is expensive and frequently accessed. Use smaller TTLs for less critical data.

---

## Rules of Thumb

1. **Profile data**: Write-through cache (update cache when profile is updated). 24-hour TTL.
2. **Counts and aggregates**: Precomputed table. Update when the underlying data changes (a user follows, a post gets a like).
3. **User-specific data (feed, recommendations)**: Read-through cache. 5-10 minute TTL.
4. **Static routes (public pages)**: Route cache or CDN. 24+ hour TTL.
5. **Expensive computations**: Function cache with input-based keys. No expiration, invalidate on code change.
6. **Hot data within requests**: Context cache. Automatic cleanup after request.

---

## Why This Matters

At 2 a.m., when your database is on fire and your monitoring is screaming, you will not be thinking about the theoretical purity of your system. You will be thinking about what is hitting the database and why.

The engineer who understands cache sees the problem immediately: "Which queries are running that don't need to? What can I cache?"

The engineer who doesn't understand cache sees chaos. They add more database replicas. They tune connection pools. They blame the database. Hours pass. The problem is still there.

Cache is not something you add as a performance optimization. Cache is something you build into the architecture from day one, because without it, your database doesn't survive.

---

## Further Reading

- [Scaling Memcache at Facebook](https://research.facebook.com/publications/scaling-memcache-at-facebook/). How Facebook keeps a billion QPS alive with intelligent caching.
- [Redis documentation](https://redis.io/documentation). Reference and tutorials.
- [Nginx caching guide](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache). HTTP caching at the reverse proxy layer.
- [HTTP caching (MDN)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching). Browser and CDN cache headers.


---

## Designing a Distributed Rate Limiter with Redis

**Date:** 2026-07-07
**URL:** http://localhost:1313/designing-a-rate-limiter/
**Description:** A full system design of a production rate limiter. Covers every major algorithm (token bucket, sliding window, leaky bucket, fixed window), their tradeoffs, how they break in a distributed system, how Redis fixes them, and how Stripe, Cloudflare, and API gateways actually deploy this in production.


There is a class of bugs that only happen at midnight.

Your API has a rate limit: 1000 requests per minute. A client hits 999 at 11:59:59 PM, then fires another 999 at 12:00:00 AM. Two windows, two clean counters, all 1998 requests allowed. Your database gets a spike it was never designed to handle, and you spend the next hour wondering how your rate limiter let this through.

It did let it through. You just did not know it.

This is the boundary spike problem, and it is one of about ten things that will go wrong when you design a rate limiter from scratch. The algorithm is the easy part. The hard parts are what happens when two servers see the same request at the same time, what happens when Redis goes down, and what happens when a client figures out they can route around your limits by spreading requests across IP addresses.

This post designs **Throttle**, a rate limiting service you can embed as middleware or run as a standalone sidecar, the same way I built Relay in the [previous post](/how-multi-tenant-saas-works/). We will go through every major algorithm, how each one breaks in a distributed system, how to fix those breaks, and how to deploy it.

---

## What Rate Limiting Is Actually For

Most people think of rate limiting as one thing. It is not. It is four different problems wearing the same name.

The first is infrastructure protection. A single client with a buggy retry loop can take down your API. The rate limiter is the wall between that client and your database.

The second is fair use. On a multi-tenant system like Relay, one tenant having a bad day should not become everyone else's problem. This is resource isolation at the API layer.

The third is monetization. Free tier gets 100 requests per minute. Pro gets 1000. Rate limits are what make pricing tiers real rather than theoretical.

The fourth is security. Slowing down brute-force login attempts, stopping credential stuffing, limiting the blast radius of a stolen API key.

These four problems lead to different design decisions. A security rate limiter needs to stay effective even when Redis is down. A billing rate limiter is fine with a few seconds of lag. If you mix them up, you either overbuild the wrong thing or build the right thing in a way that fails in the wrong direction.

Throttle handles all four. That means multiple limit scopes (per IP, per key, per tenant), multiple time windows (per second, per minute, per day, per month), and a design that fails gracefully instead of all at once.

---

## The Four Algorithms

Before picking an algorithm, understand that every rate limiting algorithm is making a bet about what "fair" means. Some bet on time windows. Some bet on token accumulation. The bet determines what kind of abuse gets through and what kind of legitimate traffic gets punished.

There are four algorithms that actually matter in production. They are not interchangeable.

### Fixed Window Counter

The simplest one. Divide time into buckets of fixed length W (say, 60 seconds). Keep a counter per bucket. Add one on every request. Reject when the counter goes over the limit.

```
window:  [0s ----------- 60s][60s ----------- 120s]
counter:  897                  0
limit:    1000
```

**Implementation in Redis:**

```go
func (r *RedisLimiter) AllowFixed(ctx context.Context, key string, limit int, window time.Duration) (bool, error) {
    now := time.Now().Unix()
    windowStart := now - (now % int64(window.Seconds()))
    bucketKey := fmt.Sprintf("rl:fixed:%s:%d", key, windowStart)

    pipe := r.client.Pipeline()
    incr := pipe.Incr(ctx, bucketKey)
    pipe.Expire(ctx, bucketKey, window*2) // 2x window so key expires naturally
    _, err := pipe.Exec(ctx)
    if err != nil {
        return false, err
    }

    return incr.Val() <= int64(limit), nil
}
```

The window start is calculated by rounding `now` down to the nearest window boundary. Every request in that 60-second bucket hits the same counter.

**The boundary spike problem.** Fixed windows have one well-known flaw. A client can send `limit` requests at 23:59:59.999 and `limit` more at 00:00:00.001, and you just let `2 * limit` requests through in 2 milliseconds.

```
limit = 1000

[      window 1      ][      window 2      ]
...            999 req|1000 req ...
                      ^
                2 ms, 1999 requests allowed
```

This is not a hypothetical. It is how you get database crashes even when you think you have rate limiting in place.

**When to use it anyway.** Fixed window is fine when the boundary burst does not matter much. Monthly quota enforcement is the classic example. If a user burns 1000 API calls in the last second of their billing month and 1000 in the first second of the next, that is a real problem for strict quota. But for most monthly caps, simple is better.

---

### Sliding Window Log

Instead of counting requests per window, you store the timestamp of every request in the last W seconds and count those.

```go
func (r *RedisLimiter) AllowSlidingLog(ctx context.Context, key string, limit int, window time.Duration) (bool, error) {
    now := time.Now()
    windowStart := now.Add(-window).UnixMilli()
    logKey := fmt.Sprintf("rl:log:%s", key)

    pipe := r.client.Pipeline()
    // Remove timestamps older than the window
    pipe.ZRemRangeByScore(ctx, logKey, "0", strconv.FormatInt(windowStart, 10))
    // Count remaining timestamps (requests in the current window)
    count := pipe.ZCard(ctx, logKey)
    // Add this request's timestamp
    pipe.ZAdd(ctx, logKey, redis.Z{Score: float64(now.UnixMilli()), Member: now.UnixNano()})
    pipe.Expire(ctx, logKey, window)
    _, err := pipe.Exec(ctx)
    if err != nil {
        return false, err
    }

    return count.Val() < int64(limit), nil
}
```

We use a Redis sorted set. The score is the timestamp in milliseconds. On every request: remove old entries, count what is left, add the current timestamp. The window always looks backward exactly W seconds from right now.

This is the most accurate algorithm. No boundary spike. Every point in time sees the correct count. A client who sends 1000 requests spread across the last 60 seconds will be rejected on request 1001 no matter when it arrives.

The problem is memory. You store one entry per request per user. If a client is allowed 10,000 requests per minute, you store up to 10,000 sorted set members. At 1 million users each with a 1000 req/min limit, that is up to 1 billion sorted set entries. This is why production systems at high volume almost never use sliding log.

Use it for low-volume, high-precision cases: login attempts, password resets, admin APIs where correctness matters more than memory.

---

### Sliding Window Counter

A hybrid. It approximates the sliding window using two fixed-window counters: the current window and the previous one, weighted by how far into the current window you are.

```
window = 60s
current window started 15s ago (we are 25% into it)

estimated count = prev_count * (1 - 0.25) + curr_count * 1.0
                = prev_count * 0.75 + curr_count
```

```go
func (r *RedisLimiter) AllowSlidingCounter(ctx context.Context, key string, limit int, window time.Duration) (bool, error) {
    now := time.Now()
    windowSecs := int64(window.Seconds())
    currentWindowStart := now.Unix() - (now.Unix() % windowSecs)
    prevWindowStart := currentWindowStart - windowSecs

    currentKey := fmt.Sprintf("rl:sw:%s:%d", key, currentWindowStart)
    prevKey := fmt.Sprintf("rl:sw:%s:%d", key, prevWindowStart)

    pipe := r.client.Pipeline()
    curr := pipe.Get(ctx, currentKey)
    prev := pipe.Get(ctx, prevKey)
    _, _ = pipe.Exec(ctx)

    currentCount, _ := strconv.ParseFloat(curr.Val(), 64)
    prevCount, _ := strconv.ParseFloat(prev.Val(), 64)

    // How far are we into the current window? (0.0 = start, 1.0 = end)
    elapsed := float64(now.Unix()-currentWindowStart) / float64(windowSecs)
    estimated := prevCount*(1-elapsed) + currentCount

    if estimated >= float64(limit) {
        return false, nil
    }

    // Increment current window counter
    r.client.Incr(ctx, currentKey)
    r.client.Expire(ctx, currentKey, window*2)
    return true, nil
}
```

This is what Cloudflare uses. Their blog post on rate limiting describes exactly this approach. You get O(1) memory per user (just two counters), near-accurate sliding window behavior, and no meaningful boundary spikes. The approximation error is a few percent at most.

The formula assumes requests were spread evenly through the previous window. If they were all at the very end (a burst), the estimate will be a bit too low. In practice this error is small enough that this is the right default for most production systems.

---

### Token Bucket

Token bucket works differently from the counter-based algorithms. Instead of counting past requests, you maintain a bucket that fills with tokens at a fixed rate. Each request uses one token. If the bucket is empty, the request is rejected.

```
bucket capacity = 100 tokens
refill rate     = 10 tokens/second

t=0:   bucket = 100
t=5:   100 requests arrive, bucket = 0, all allowed
t=5.1: 1 request arrives, bucket = 1 (0.1s * 10), allowed
t=5.1: 1 more request, bucket = 0, rejected
t=6:   bucket = 9, next request allowed
```

Bucket capacity controls the maximum burst. Refill rate controls sustained throughput. If someone has not touched the API in 100 seconds and the refill rate is 10 tokens/second, they can burst 1000 requests when they come back, then trickle at 10 per second.

**Redis implementation using timestamps (no background thread needed):**

```go
type TokenBucketState struct {
    Tokens     float64
    LastRefill int64 // Unix timestamp in milliseconds
}

func (r *RedisLimiter) AllowTokenBucket(ctx context.Context, key string, capacity float64, refillRate float64) (bool, error) {
    // refillRate = tokens per millisecond
    bucketKey := fmt.Sprintf("rl:tb:%s", key)

    // Lua script for atomicity - read-modify-write in one round trip
    script := redis.NewScript(`
        local key = KEYS[1]
        local capacity = tonumber(ARGV[1])
        local refill_rate = tonumber(ARGV[2])  -- tokens per ms
        local now = tonumber(ARGV[3])

        local state = redis.call("HMGET", key, "tokens", "last_refill")
        local tokens = tonumber(state[1]) or capacity
        local last_refill = tonumber(state[2]) or now

        -- Compute how many tokens have been added since last request
        local elapsed = now - last_refill
        tokens = math.min(capacity, tokens + elapsed * refill_rate)

        if tokens < 1 then
            -- Update last_refill even on rejection so we track time correctly
            redis.call("HMSET", key, "tokens", tokens, "last_refill", now)
            redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate))
            return 0
        end

        tokens = tokens - 1
        redis.call("HMSET", key, "tokens", tokens, "last_refill", now)
        redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate))
        return 1
    `)

    now := time.Now().UnixMilli()
    result, err := script.Run(ctx, r.client, []string{bucketKey}, capacity, refillRate/1000.0, now).Int()
    if err != nil {
        return false, err
    }
    return result == 1, nil
}
```

The Lua script is the key part. Redis runs Lua scripts atomically, so the read, compute, and write happen as a single operation. Nothing else can interleave. This kills the race condition you would get with a WATCH/MULTI/EXEC approach.

Token bucket is what most production systems use for per-key limiting. It is why API clients can send a short burst if they have been idle. The burst is intentional. It handles legitimate traffic spikes without punishing users.

The tradeoff is state. Each key stores two fields: tokens and last_refill. Cheap in Redis, but more than a simple counter. And you have to decide: what does a brand-new user start with? Full bucket. Otherwise new users get rejected before they even get started.

---

### Leaky Bucket

Leaky bucket is the reverse of token bucket. Requests flow in and the bucket drains at a fixed rate.

```
requests go in -> [queue] -> processed at 1 req/10ms -> response
                     |
                  if full: reject
```

The bucket is a FIFO queue. Requests enter, a worker drains it at a constant rate. If the queue is full, new requests are dropped. Output is perfectly smooth. You never send more than 1 request per 10ms no matter how bursty the input is.

You use this for shaping traffic toward a dependency, not for user-facing limits. If you are calling an external API that handles exactly 100 req/s and has no burst tolerance, a leaky bucket in front of it smooths your traffic and stops you from getting throttled. Users do not see it because it is between your code and theirs.

For user-facing rate limiting, leaky bucket is usually wrong. Users experience queuing delay instead of a clean rejection, which is confusing. And you have to cap the queue size, which means deciding whether to reject immediately or wait when it fills up. Waiting adds unpredictable latency. This is more of a traffic shaper than a rate limiter.

---

## Choosing an Algorithm

| Algorithm | Memory | Burst handling | Accuracy | Complexity |
|---|---|---|---|---|
| Fixed window | O(1) per key | Poor (boundary spike) | Low | Low |
| Sliding log | O(n) per key | Exact | Highest | Medium |
| Sliding counter | O(1) per key | Good (approximated) | High | Medium |
| Token bucket | O(1) per key | Configurable | High | Medium |
| Leaky bucket | O(queue) | Smooths it away | High | High |

For Throttle:

- **Token bucket** for per-key and per-IP short-window limits. Configurable burst, O(1) memory, accurate enough.
- **Sliding counter** for per-tenant limits over longer windows (hourly, daily). Low memory, near-exact, no thundering herd at window boundaries.
- **Fixed window** for monthly quota only. When the window is 30 days, simplicity wins.

---

## The Distributed Problem

So far the algorithms look reasonable on paper. Pick token bucket, write the Lua script, done. But all of that assumes one Redis instance and one application server. Production is neither, and that changes everything.

When you run multiple API servers behind a load balancer, rate limit state needs to be consistent across all of them. If it is not, clients can bypass the limit just by having their requests spread across servers.

```mermaid
graph TD
    Client -->|1000 requests| LB[Load Balancer]
    LB -->|500 requests| Server1
    LB -->|500 requests| Server2
    Server1 -->|counter=500, under limit| OK1[Allow]
    Server2 -->|counter=500, under limit| OK2[Allow]

    style OK1 fill:#ef4444
    style OK2 fill:#ef4444
```

Both servers see 500 requests. Both think the user is under the limit. The user sent 1000 and all went through. If each server has its own local counter with no sync, your "1000 req/min" limit is really `1000 * num_servers` in practice.

The fix is shared state. This is why Redis is the standard for distributed rate limiting. Every server reads and writes to the same Redis instance.

```mermaid
graph TD
    Client -->|1000 requests| LB[Load Balancer]
    LB -->|500 requests| Server1
    LB -->|500 requests| Server2
    Server1 -->|INCR counter| Redis[(Redis)]
    Server2 -->|INCR counter| Redis
    Redis -->|counter=1000, over limit| Both[Reject]
```

Now the counter is shared. But you just added a network hop on every request for the rate limit check. That is on the critical path. Redis is fast (under a millisecond), but it is not free.

### The Race Condition

Even with shared Redis, there is a race condition in the naive implementation.

```
Server 1: GET counter -> 999
Server 2: GET counter -> 999    (reads before Server 1 writes)
Server 1: SET counter = 1000 -> allow
Server 2: SET counter = 1000 -> allow  (wrong, should be 1001)
```

GET followed by SET is not atomic. Between the read and the write, another server can jump in. The fix is atomic operations.

For simple counters, `INCR` and `INCRBY` are atomic in Redis. A single INCR atomically reads, increments, and returns the new value. No race.

For complex state like token bucket, use a Lua script. Redis runs Lua atomically with no interleaving. This is exactly why the token bucket implementation above uses a script.

There is also WATCH/MULTI/EXEC, Redis's optimistic transaction mode. You watch a key, start a transaction, read, then exec. If someone else modified the key between WATCH and EXEC, the transaction aborts and you retry. It works, but you need retry logic and it fails badly under high contention.

Lua scripts are simpler and more reliable. Use them.

---

## The Redis Failure Problem

Redis is on the critical path now. If it goes down, every request fails the rate limit check. You have to decide: **fail open** or **fail closed**.

**Fail open** means if Redis is down, let all requests through. The system runs unprotected. You take the traffic risk to stay available. Right choice for non-security limits like billing tiers and usage dashboards.

**Fail closed** means if Redis is down, reject everything. You stay protected at the cost of total unavailability. Right choice for security limits like login brute force and account lockout.

**The hybrid** is local fallback. Each server keeps a local in-process rate limiter, just a simple atomic counter or token bucket in memory. When Redis is unreachable, fall back to local limits. Local limits are per-server, so the effective limit becomes `local_limit * num_servers`, but that beats either zero protection or a total outage.

```go
type TieredLimiter struct {
    redis  *RedisLimiter
    local  *LocalLimiter // in-process token bucket per key
    policy FailurePolicy // FailOpen or FailClosed or LocalFallback
}

func (t *TieredLimiter) Allow(ctx context.Context, key string, limit Limit) (Decision, error) {
    decision, err := t.redis.Allow(ctx, key, limit)
    if err == nil {
        return decision, nil
    }

    // Redis is down
    switch t.policy {
    case FailOpen:
        return Allow, nil
    case FailClosed:
        return Deny, nil
    case LocalFallback:
        return t.local.Allow(key, limit.ScaledBy(1.0/float64(t.serverCount)))
    }
    return Allow, nil
}
```

The `ScaledBy` call matters. If you run 10 servers and the real limit is 1000 req/min, the local fallback limit should be 100 req/min. Across all 10 servers, the aggregate stays at 1000.

---

## The Sliding Window Race

The sliding counter has its own distributed race. The read-modify-write across the current and previous window counters is not atomic.

```go
// This is NOT safe without additional atomicity
curr := pipe.Get(ctx, currentKey)  // read
prev := pipe.Get(ctx, prevKey)     // read
// ... compute estimated ...
pipe.Incr(ctx, currentKey)         // write
```

Another server can increment between your read and your write. Wrap it in Lua.

```go
var slidingScript = redis.NewScript(`
    local curr_key = KEYS[1]
    local prev_key = KEYS[2]
    local limit = tonumber(ARGV[1])
    local window_secs = tonumber(ARGV[2])
    local elapsed_ratio = tonumber(ARGV[3])  -- 0.0 to 1.0

    local curr = tonumber(redis.call("GET", curr_key)) or 0
    local prev = tonumber(redis.call("GET", prev_key)) or 0

    local estimated = prev * (1 - elapsed_ratio) + curr
    if estimated >= limit then
        return 0
    end

    redis.call("INCR", curr_key)
    redis.call("EXPIRE", curr_key, window_secs * 2)
    return 1
`)
```

One round trip, atomic. This is the rule: any time you have a read-compute-write on rate limit state, put it in Lua.

---

## Who to Limit

A rate limiter is only as good as the identity it limits on. The key you choose determines how the limit behaves.

**By IP address.** Simple. Works against naive scrapers. Breaks behind NAT (a company office might have 500 employees sharing one IP), breaks against distributed attackers who rotate IPs, and it punishes shared infrastructure like CI runners.

**By API key.** The right choice for authenticated APIs. Each key gets its own limit. One bad key does not affect anyone else. This is the main identity for Throttle.

**By tenant.** Useful for aggregate limits across an organization. An enterprise tenant with 50 API keys should not get 50 times the limit just by spreading requests across keys.

**By user.** For consumer products with session-based auth.

In production you apply all of these at once, in layers:

```mermaid
flowchart TD
    Req[Request] --> IP["IP limit (1000/min)"]
    IP -->|allowed| Key["API key limit (5000/min)"]
    Key -->|allowed| Tenant["Tenant aggregate (50000/min)"]
    Tenant -->|allowed| Global["Global system (10M/min)"]
    Global -->|allowed| Handler

    IP -->|denied| R1[429]
    Key -->|denied| R2[429]
    Tenant -->|denied| R3[429]
    Global -->|denied| R4[429]
```

Each layer catches a different kind of abuse. IP limits stop anonymous scrapers. Key limits handle a compromised key. Tenant limits stop one big customer from saturating the system. Global limits are the last line of defense.

The layered approach has a nice property: you only check deeper layers if the outer ones pass. If the IP is already over the limit, you never touch the API key counter. This saves Redis reads on already-rejected traffic.

---

## The Full Data Model

Throttle stores limit configurations in Postgres and live state in Redis.

```sql
-- Limit policies, referenced by API keys and tenants
CREATE TABLE rate_limit_policies (
    id              TEXT PRIMARY KEY,
    name            TEXT NOT NULL,          -- "free_tier", "pro_tier", "enterprise"
    
    -- Per-key limits
    req_per_second  INT  NOT NULL DEFAULT 10,
    req_per_minute  INT  NOT NULL DEFAULT 500,
    req_per_hour    INT  NOT NULL DEFAULT 10000,
    req_per_day     INT  NOT NULL DEFAULT 100000,
    req_per_month   INT  NOT NULL DEFAULT 1000000,
    
    -- Burst headroom (token bucket capacity multiplier)
    burst_multiplier FLOAT NOT NULL DEFAULT 1.5,
    
    -- Algorithm override (null = system default)
    algorithm       TEXT,
    
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Per-key overrides (when a key needs a non-default policy)
CREATE TABLE api_key_rate_limits (
    api_key_id      TEXT PRIMARY KEY REFERENCES api_keys(id),
    policy_id       TEXT NOT NULL REFERENCES rate_limit_policies(id),
    custom_override JSONB           -- ad-hoc overrides without a new policy row
);

-- Blocking rules (IPs, tenants, keys that are permanently blocked)
CREATE TABLE rate_limit_blocks (
    id          TEXT PRIMARY KEY,
    scope       TEXT NOT NULL,      -- 'ip', 'key', 'tenant'
    value       TEXT NOT NULL,      -- the IP or key ID or tenant ID
    reason      TEXT,
    blocked_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at  TIMESTAMPTZ,        -- null = permanent
    UNIQUE (scope, value)
);

CREATE INDEX idx_blocks_scope_value ON rate_limit_blocks(scope, value);
CREATE INDEX idx_blocks_expires_at  ON rate_limit_blocks(expires_at) WHERE expires_at IS NOT NULL;

-- Persistent rate limit violations log (for billing, audit, anomaly detection)
CREATE TABLE rate_limit_events (
    id          TEXT PRIMARY KEY,
    tenant_id   TEXT,
    api_key_id  TEXT,
    ip          INET,
    scope       TEXT NOT NULL,      -- 'ip', 'key', 'tenant', 'global'
    limit_key   TEXT NOT NULL,      -- the Redis key that was hit
    limit_value INT NOT NULL,       -- the limit that was exceeded
    count       INT NOT NULL,       -- the count at time of rejection
    window      INTERVAL NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
) PARTITION BY RANGE (created_at);
```

The `rate_limit_events` table is append-only and partitioned by month. It feeds dashboards ("you are at 90% of your limit"), anomaly detection ("this IP hit rate limits 500 times in the last hour"), and the upsell pipeline ("this tenant exceeded their plan limit 3 times this week").

---

## The Middleware

Here is how Throttle wires into an HTTP server. It resolves context from the request, makes a decision, and either passes through or rejects.

```go
type Limiter interface {
    Allow(ctx context.Context, req *LimitRequest) (*Decision, error)
}

type LimitRequest struct {
    IP       string
    APIKeyID string
    TenantID string
    Policy   *Policy
}

type Decision struct {
    Allowed    bool
    Limit      int
    Remaining  int
    ResetAt    time.Time
    RetryAfter time.Duration // set when Allowed=false
    Scope      string        // which scope denied: "ip", "key", "tenant"
}

func RateLimitMiddleware(limiter Limiter) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            ctx := r.Context()
            tc := TenantFromContext(ctx) // set by auth middleware upstream

            req := &LimitRequest{
                IP:       realIP(r),
                APIKeyID: tc.APIKeyID,
                TenantID: tc.TenantID,
                Policy:   tc.RateLimitPolicy,
            }

            decision, err := limiter.Allow(ctx, req)
            if err != nil {
                // Redis is unavailable - policy decision (see failure handling)
                if isRedisDown(err) {
                    slog.Warn("rate limiter degraded", "err", err)
                    // For this example: fail open, continue to handler
                    next.ServeHTTP(w, r)
                    return
                }
                http.Error(w, "internal error", http.StatusInternalServerError)
                return
            }

            // Always set rate limit headers, even on success
            w.Header().Set("X-RateLimit-Limit",     strconv.Itoa(decision.Limit))
            w.Header().Set("X-RateLimit-Remaining", strconv.Itoa(decision.Remaining))
            w.Header().Set("X-RateLimit-Reset",     strconv.FormatInt(decision.ResetAt.Unix(), 10))

            if !decision.Allowed {
                w.Header().Set("Retry-After", strconv.Itoa(int(decision.RetryAfter.Seconds())))
                w.WriteHeader(http.StatusTooManyRequests)
                json.NewEncoder(w).Encode(map[string]any{
                    "error":       "rate_limit_exceeded",
                    "scope":       decision.Scope,
                    "retry_after": decision.RetryAfter.Seconds(),
                    "message":     fmt.Sprintf("Rate limit exceeded. Retry in %.1fs.", decision.RetryAfter.Seconds()),
                })
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

// realIP handles X-Forwarded-For and X-Real-IP headers from load balancers.
// Never trust the first IP in XFF blindly. Take the last IP added
// by a trusted proxy if you control the proxy chain.
func realIP(r *http.Request) string {
    if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
        parts := strings.Split(xff, ",")
        return strings.TrimSpace(parts[len(parts)-1])
    }
    if xri := r.Header.Get("X-Real-IP"); xri != "" {
        return xri
    }
    host, _, _ := net.SplitHostPort(r.RemoteAddr)
    return host
}
```

The response headers matter. Clients that read `X-RateLimit-*` headers can back off on their own. `Retry-After` tells them exactly when to retry. Stripe's client libraries implement exponential backoff using these headers. A rate limiter that only returns 429 with no headers forces clients to guess, and guessing usually means retrying immediately, which makes the problem worse.

---

## The Full Request Flow

```mermaid
flowchart TD
    Req["Incoming request + Authorization header"] --> Auth["Auth middleware, resolve tenant + policy"]
    Auth -->|401| Rej1[Reject]
    Auth -->|context set| BlockCheck["Block list check, IP + key + tenant"]
    BlockCheck -->|blocked| Rej2["429 or 403 for permanent blocks"]
    BlockCheck -->|clean| IPLimit["IP rate limit"]
    IPLimit -->|denied| Rej3["429 + Retry-After"]
    IPLimit -->|allowed| KeyLimit["API key rate limit"]
    KeyLimit -->|denied| Rej4["429 + Retry-After"]
    KeyLimit -->|allowed| TenantLimit["Tenant aggregate limit"]
    TenantLimit -->|denied| Rej5["429 + Retry-After"]
    TenantLimit -->|allowed| Handler["Route handler"]
    Handler --> Resp["200 + X-RateLimit headers"]
```

Each check is one Redis read. A clean request with three limit scopes costs three Redis round trips. In practice you run them concurrently:

```go
func (l *TieredRedisLimiter) Allow(ctx context.Context, req *LimitRequest) (*Decision, error) {
    blocked, err := l.checkBlockList(ctx, req)
    if err != nil || blocked {
        return &Decision{Allowed: false, Scope: "block"}, err
    }

    // Run limit checks concurrently using errgroup
    var (
        ipDec     *Decision
        keyDec    *Decision
        tenantDec *Decision
    )
    g, ctx := errgroup.WithContext(ctx)
    g.Go(func() error {
        var err error
        ipDec, err = l.ipLimiter.Allow(ctx, req.IP, req.Policy.IPLimit)
        return err
    })
    g.Go(func() error {
        var err error
        keyDec, err = l.keyLimiter.Allow(ctx, req.APIKeyID, req.Policy.KeyLimit)
        return err
    })
    g.Go(func() error {
        var err error
        tenantDec, err = l.tenantLimiter.Allow(ctx, req.TenantID, req.Policy.TenantLimit)
        return err
    })
    if err := g.Wait(); err != nil {
        return nil, err
    }

    // Return the most restrictive decision
    for _, d := range []*Decision{ipDec, keyDec, tenantDec} {
        if !d.Allowed {
            return d, nil
        }
    }

    // All passed - return the tightest remaining quota (for response headers)
    return mostRestrictive(ipDec, keyDec, tenantDec), nil
}
```

Three Lua scripts run in parallel. In practice the three checks take about the same time as one.

---

## Returning Useful Rate Limit Information

The 429 response is a user-facing feature. It is what developers see when their integration breaks. Getting it right cuts support tickets.

```
X-RateLimit-Limit: 1000          # The limit that applies
X-RateLimit-Remaining: 847       # Requests left in the current window
X-RateLimit-Reset: 1720425600    # Unix timestamp when the window resets
Retry-After: 14                  # Seconds until they can retry (on 429 only)
```

GitHub and Stripe both use this format. It is close enough to a standard that most HTTP clients handle it.

The response body on a 429 should give the developer exactly what they need to understand and fix the problem:

```json
{
  "error": "rate_limit_exceeded",
  "scope": "tenant",
  "limit": 50000,
  "current": 50001,
  "window": "1h",
  "retry_after": 847.3,
  "message": "Tenant aggregate limit exceeded. Your plan allows 50,000 requests/hour. Retry in 847 seconds or upgrade to Pro."
}
```

The `scope` field tells the developer which limit they hit. Without it, they cannot tell if one bad client is the problem or the entire organization is over quota. The `window` tells them whether waiting 1 second or 15 minutes is appropriate. The upsell nudge in `message` also happens to reduce support tickets.

---

## Scalability

A single Redis instance handles roughly 100,000 commands per second. Each rate limit check runs 1 to 3 Redis commands depending on the Lua script. At 100,000 req/s of API traffic with three limit scopes per request, you need around 300,000 Redis commands per second. That means Redis Cluster.

Redis Cluster shards keys across multiple primary nodes. Rate limit keys shard naturally by hash. Each user's counter lives on one shard and stays consistent within that shard. Cross-shard operations are rare in rate limiting because each scope's key is self-contained.

```mermaid
graph TD
    Server1 --> RC[Redis Cluster]
    Server2 --> RC
    Server3 --> RC
    RC --> Shard1["Shard 1, IP limits: slots 0-5460"]
    RC --> Shard2["Shard 2, Key limits: slots 5461-10922"]
    RC --> Shard3["Shard 3, Tenant limits: slots 10923-16383"]
```

**Memory.** With token bucket (two fields per key per scope) and sliding counter (two counters per key per window), memory per active user is roughly:

- Token bucket state: about 200 bytes per scope
- Sliding counter: about 200 bytes per active window
- 3 scopes times 2 windows each = about 1.2 KB per active user

At 1 million concurrent active users, that is about 1.2 GB of Redis memory. Fine for a dedicated cluster.

**Local admission control.** At very high traffic, even one Redis round trip per request starts to hurt. Twitter's rate limiter uses a two-layer approach: a local approximate counter per server tracks recent requests. If clearly under the limit, skip Redis. If near the limit, do the authoritative Redis check. This cuts Redis load by around 80% under normal traffic.

```go
type LocalApproximateLimiter struct {
    local     *LocalTokenBucket
    remote    *RedisLimiter
    threshold float64 // 0.8 = check Redis when local is at 80% of limit
}

func (l *LocalApproximateLimiter) Allow(ctx context.Context, key string, limit int) (bool, error) {
    localDecision := l.local.Estimate(key, limit)
    if localDecision.RatioUsed < l.threshold {
        // Clearly under limit, skip Redis
        l.local.Increment(key)
        return true, nil
    }
    // Near or over limit - do authoritative check
    return l.remote.Allow(ctx, key, limit)
}
```

The local estimate is wrong in a multi-server world because it only sees its own server's traffic. But "clearly under limit" is almost always correct. If this server has seen 10% of the limit, the other servers would have to account for 90% for the real limit to be close. In practice traffic distributes roughly evenly.

---

## Availability

**Redis Sentinel** provides automatic failover for a single Redis primary. Sentinel watches the primary and promotes a replica if it goes down. Failover takes 10 to 30 seconds. During that window your rate limiter is unavailable, which is why the local fallback from earlier matters.

**Redis Cluster** gives you both sharding and HA. Each shard has a primary and replicas. If a shard primary fails, a replica is promoted. A cluster with 3 primaries and 3 replicas survives any single-node failure without meaningful disruption.

Rate limit state must be written to the primary. But block list checks and policy reads can go to replicas. Routing read-only operations to replicas reduces primary load.

**What happens during a network partition?** Some servers can reach Redis, some cannot. The ones that cannot fall back to local limiting. The ones that can behave normally. The two groups now have inconsistent views of the rate limit. At most `limit * (num_isolated_servers / total_servers)` extra requests leak through during the partition. This is the CAP tradeoff. To stay available, you accept some inconsistency. For rate limiting, that is the right call. A small violation of rate limits during a partition beats 100% request failure.

---

## Deployment Patterns

### Embedded Middleware

Throttle runs as a Go package imported into your service. No extra network hop for the rate limit check beyond Redis. The simplest deployment.

```
Your service
  HTTP server
    Auth middleware
    RateLimitMiddleware (Throttle, embedded)
    Route handlers
```

The downside is that every service needs to import and configure Throttle separately. Configuration drifts across services. Centralized changes require redeployments.

### Sidecar

Throttle runs as a sidecar process alongside each service instance. The service calls Throttle over a local unix socket or loopback. The network hop is under a millisecond.

```
Pod (Kubernetes)
  Service container  ->  POST localhost:8081/check
  Throttle sidecar   ->  Redis Cluster
```

This is how Envoy's rate limiting works. The sidecar model gives you centralized configuration while keeping latency low.

### API Gateway / Reverse Proxy

Rate limiting happens at the reverse proxy before traffic reaches your service. Nginx, HAProxy, Kong, Envoy, and cloud load balancers all support this. The service never sees over-limit requests.

```
Internet -> Cloudflare (IP and DDoS limits) -> ALB -> API Gateway (key and tenant limits) -> Your service
```

This is the model for public APIs at scale. Cloudflare handles volumetric attacks. The API gateway handles per-key business logic limits. Your service handles nothing.

The limitation is context. Rate limiting at the proxy means less information about the request. You cannot limit on application-level concepts like "number of tokens consumed" for an LLM API without custom logic. Proxies are good at "N requests per minute". They are not good at "N output tokens per minute".

### Centralized Rate Limit Service

Run Throttle as a standalone service. Every other service calls it before processing a request.

```
Service A  ->  Throttle service  ->  Redis
Service B  ->  Throttle service
Service C  ->  Throttle service
```

Centralized configuration, no drift, easy to update policies. The cost: every request has a synchronous network call to Throttle on the critical path. Throttle becomes a hard dependency. If it goes down, all services fail unless each one has its own fallback logic. At high scale, Throttle itself needs to be distributed.

Lyft's Ratelimit service (open source) uses this model. It is the right choice for large organizations with many independent services that need consistent rate limiting policy.

---

## Trade-offs and Costs

**Accuracy vs. memory.** Sliding window log is perfectly accurate. Sliding counter approximates it. Token bucket is accurate for burst but cannot give you an exact count of total requests in a window. The more accurate the algorithm, the more memory or complexity it needs.

**Consistency vs. latency.** Authoritative Redis checks give you global consistency. Local approximate checks give you low latency. Under high load, the local check should be the fast path, and you accept that a few extra requests might slip through.

**Fairness across users.** A per-key limit of 1000 req/min is fair in isolation. But if 10% of your users are on enterprise plans with 100,000 req/min each, one enterprise customer can still saturate your infrastructure while every free-tier user is comfortably under their limit. You need per-scope limits at every layer, not just per-key.

**The header trust problem.** `X-Forwarded-For` is set by load balancers and proxies, but it can also be set by attackers. A client can send `X-Forwarded-For: 1.2.3.4` and trick a naive rate limiter into thinking they are a different IP on every request. Always use the IP added by your own infrastructure, not the one claimed by the client.

**Clock skew.** Sliding window counters depend on the current time. Across servers with clock skew, window boundaries can differ slightly. NTP usually keeps skew under 1ms, which does not matter for second-level windows. For millisecond-precision limits, be careful comparing timestamps across machines.

**The thundering herd at window reset.** At midnight, when daily limits reset, every user who was at the limit can send requests at the same moment. This creates a traffic spike. Sliding windows help compared to fixed windows. You can also add random jitter to reset times per user so not everyone resets at once.

---

## What Production Systems Actually Do

**Stripe** uses token bucket per API key with per-endpoint weights. Some endpoints cost more tokens than others. They have written publicly about decrementing tokens based on actual compute cost, not request count. A heavy request costs more tokens than a lightweight one. This stops "request inflation," where clients make many cheap requests to stay under count-based limits while consuming far more resources than the limit intended.

**GitHub** uses sliding window with separate limits per authenticated user and per IP for unauthenticated users. Their secondary rate limit targets repeated failed requests, which signals brute-forcing or scraping rather than legitimate volume.

**Cloudflare** uses the sliding window counter from this post. Their rules run at edge nodes in every data center globally, evaluated in-process with no Redis round trip. They use a custom in-memory structure per edge PoP that gossips state with nearby PoPs. That is how they rate-limit at millions of requests per second.

**Twilio** applies limits at the product level. Some Twilio APIs are limited by concurrent connections, not requests per second. That requires a semaphore (a Redis counter with a max value) rather than a windowed counter.

The common thread: production rate limiters are layered, they make explicit choices about consistency vs. availability, and they put real work into the developer experience of the 429 response.

---

## One Improvement Worth Building

Most teams treat rate limiting as a static policy: the limit is 1000 req/min, and that is it forever. But the thing rate limiting is protecting (your database, your inference cluster, your third-party budget) does not have a fixed capacity. It changes with load, with time of day, with deploys.

**Adaptive rate limiting** is the idea that the limit itself should respond to system health. If your database is at 90% CPU, tighten rate limits by 20% across all keys. When load drops back to normal, loosen them. The rate limiter becomes a feedback loop rather than a fixed gate.

```go
type AdaptiveLimiter struct {
    base    Limiter
    metrics SystemMetrics // CPU, latency p99, error rate
}

func (a *AdaptiveLimiter) scaleFactor() float64 {
    cpu := a.metrics.DatabaseCPU()
    switch {
    case cpu > 0.90:
        return 0.5  // cut limits in half
    case cpu > 0.75:
        return 0.75
    default:
        return 1.0
    }
}

func (a *AdaptiveLimiter) Allow(ctx context.Context, req *LimitRequest) (*Decision, error) {
    scaled := req.Policy.ScaledBy(a.scaleFactor())
    return a.base.Allow(ctx, &LimitRequest{
        IP:       req.IP,
        APIKeyID: req.APIKeyID,
        TenantID: req.TenantID,
        Policy:   scaled,
    })
}
```

This is simple to build on top of the existing structure. The base limiter does not change. You just scale the policy before passing it in.

The tricky part is choosing the right signal. CPU is a lagging indicator. A better signal is p99 latency of your slowest dependency, because it responds faster to the start of a problem. If your database p99 goes from 20ms to 200ms, something is wrong and you want the rate limiter to react before CPU follows.

The other tricky part is communicating the change to clients. If you tighten limits dynamically, `X-RateLimit-Limit` in the response headers now lies. The number the client stored from the last response is no longer accurate. You can mitigate this by returning the effective limit (already scaled) in the header, not the policy limit, so clients always see the limit that actually applies to their next request.

---

## Further Reading

- [Cloudflare blog: How we built rate limiting](https://blog.cloudflare.com/counting-things-a-lot-of-different-things/) - the source for the sliding window counter approach.
- [Stripe rate limiting](https://stripe.com/blog/rate-limiters) - the canonical write-up on layered rate limiting in production.
- [Lyft's Ratelimit service](https://github.com/envoyproxy/ratelimit) - open source centralized rate limiting service used with Envoy.
- [Redis documentation: INCR](https://redis.io/commands/incr/), [Lua scripting](https://redis.io/docs/latest/develop/programmability/eval-intro/) - the primitives everything here is built on.
- [GitHub REST API rate limiting](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api) - a real-world example of multi-tier rate limiting with decent documentation.

---

The thing I keep coming back to is that rate limiting is not a feature you add. It is a set of decisions you make about who you trust, how much, and what you do when that trust runs out. Every choice in this post, the algorithm, the identity key, the failure policy, flows from that.

Most teams add a Redis counter, call it done, and discover the boundary spike on a bad Tuesday. The engineers who designed Cloudflare's edge rate limiting and Stripe's API limits thought about these tradeoffs deliberately. They chose approximation over exactness, local over global, graceful degradation over hard failure. Those choices let their systems handle traffic that would have broken a naive implementation.


---

## How Multi-Tenant SaaS Actually Works

**Date:** 2026-07-01
**URL:** http://localhost:1313/how-multi-tenant-saas-works/
**Description:** A complete system design of Relay, a multi-tenant AI API gateway. Covers multi-tenant database architecture (silo vs pool vs bridge), API key authentication, provider routing with failover, dual-layer rate limiting, token-based billing, response caching, row-level security, tenant provisioning, and observability. Full Postgres schemas, mermaid diagrams, and Go snippets included.


For the past month I have been reading about multi-tenant SaaS architecture. Not as an academic exercise. I kept running into the same questions on every project I looked at: where exactly does tenant data go, how do you stop one customer's bug from becoming every customer's problem, how does billing actually work at the database level. The blog posts I found were either too abstract or skipped the hard parts entirely.

So I designed one from scratch as a system design exercise.

We are going to design a multi-tenant AI API gateway called **Relay** (`relay.lorbic.com`).

Relay does one thing: it sits between your application and every major LLM provider. You send a single API call to Relay. Relay figures out which provider to use, forwards the request, streams the response back, counts the tokens, logs the cost, and bills you at the end of the month. You never manage provider API keys in your application. You never write a retry loop for provider outages. You just call Relay.

```
your app
  → POST relay.lorbic.com/v1/chat/completions
      { model: "gpt-5.5", messages: [...] }
  → Relay validates your key, checks limits, routes to OpenAI
  → OpenAI responds
  → Relay streams the response back, logs the tokens, records the cost
```

That is the product. Now let us design the system that makes it work.

---

## What is Multi-Tenant SaaS

Multi-tenancy means one running instance of your software serves many separate customers. Each customer is a **tenant**. Every tenant has their own data, their own users, their own configuration. But they all share the same servers, the same codebase, and usually the same database.

The word that matters here is **isolation**. Tenant A must not see tenant B's data. Tenant A must not slow down tenant B's requests. Tenant A must not exhaust resources that tenant B is counting on. Isolation is not just a nice property. It is the entire contract with your customers.

In B2B SaaS, a tenant is a company. In B2C, it can be an individual user. For Relay, tenants are engineering teams: a startup that signs up, creates API keys, and calls the Relay API inside their own product. Their end users never know Relay exists.

Three concepts come up everywhere in this design. It is worth defining them upfront before the schemas start:

- **Tenant**: the customer unit. One company = one tenant. Has a `tenant_id` that appears on every row of data they own.
- **User**: a person inside a tenant. One tenant can have many users. A user belongs to exactly one tenant at a time (though the same email address can have accounts in multiple tenants).
- **Plan**: the subscription tier (free, starter, pro, enterprise). Determines what features are enabled and what rate limits apply.

---

## API Compatibility

Relay uses the OpenAI API format. The same request shape, the same response shape, the same streaming protocol. This is not an accident. It means a developer can point their existing OpenAI SDK at Relay by changing one line.

{{< code lang="python" >}}
# Before: calling OpenAI directly
client = OpenAI(api_key="sk-...")

# After: calling Relay, zero other changes
client = OpenAI(
    api_key="rly_live_sk_4xKp9mN2qR8vL3wB7tJ6cD0hF5sG1eA",
    base_url="https://relay.lorbic.com/v1"
)

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Hello"}]
)
{{< /code >}}

Relay exposes three endpoints:

```
POST /v1/chat/completions     — chat completions (streaming and non-streaming)
POST /v1/completions          — legacy text completions
POST /v1/embeddings           — text embeddings
```

The request body and response body match the OpenAI spec exactly. Relay adds a few optional headers for its own features: `x-relay-cache-ttl` to control caching TTL, `x-relay-fallback-models` to specify per-request fallback preferences, and `x-relay-metadata` to attach arbitrary key-value pairs to the request log.

Internally, Relay translates requests to each provider's native format. Anthropic's API uses a different message structure than OpenAI's. Mistral's streaming format has small differences. The provider router handles all of that. The tenant never sees it.

---

## Choosing a Multi-Tenant Database Architecture

Every multi-tenant SaaS starts with the same decision: how do you separate one customer's data from another's?

There are three approaches.

**Silo** gives each tenant their own database. Complete isolation. Very expensive at scale. Used by regulated industries or large enterprise contracts.

**Bridge** gives each tenant their own schema inside a shared database. `acme.requests`, `beta.requests`. Medium isolation, medium cost.

**Pool** puts everyone in the same tables with a `tenant_id` column on every row. Cheapest, hardest to get right, works for thousands of tenants.

Relay uses the pool model. Here is why: Relay's customers are developers and startups. There will be thousands of them. The data shape is uniform across all tenants (requests, tokens, costs). There is no per-tenant schema customization. And the operational cost of running a database per tenant would make the pricing unworkable.

When I was reading about this, the thing that surprised me was how many production SaaS companies started with silo because it felt "safer", then spent months migrating to pool because the operational cost was unsustainable at scale. The right model depends almost entirely on who your customer is and what you are charging them.

For the very few large enterprise tenants who need data residency or dedicated infrastructure, Relay offers silo as an upgrade. The architecture supports routing them to a separate database via a tenant config flag. But that is the exception, not the default.

```mermaid
graph TD
    subgraph pool["Pool Model"]
        A[acme] --> DB[(relay_db)]
        B[beta] --> DB
        C[gamma] --> DB
        DB --> R1[tenant_id = acme rows]
        DB --> R2[tenant_id = beta rows]
        DB --> R3[tenant_id = gamma rows]
    end

    subgraph silo["Silo - Enterprise"]
        D[bigcorp] --> DB2[(bigcorp_db)]
    end
```

In the pool model, all tenants share one database. Rows are separated by `tenant_id`. In the silo model, a tenant gets a dedicated database. Relay defaults to pool. Enterprise tenants that need dedicated infrastructure are routed to their own database via a flag in `tenant_config`.

---

## System Overview

Before going into each component, here is the full picture of what Relay looks like.

```mermaid
graph TB
    subgraph client["Client"]
        APP[Your Application]
    end

    subgraph relayedge["Relay Edge"]
        GW[API Gateway]
        AUTH[Auth Middleware]
        RL[Rate Limiter]
        CACHE[Cache Layer]
    end

    subgraph relaycore["Relay Core"]
        ROUTER[Provider Router]
        STREAM[Stream Handler]
        LOGGER[Request Logger]
    end

    subgraph providers["Providers"]
        OAI[OpenAI]
        ANT[Anthropic]
        MIS[Mistral]
        GRQ[Groq]
    end

    subgraph data["Data"]
        PG[(Postgres)]
        RD[(Redis)]
    end

    subgraph background["Background"]
        AGG[Usage Aggregator]
        BILL[Billing Worker]
    end

    APP -->|POST /v1/chat/completions| GW
    GW --> AUTH
    AUTH --> RL
    RL --> CACHE
    CACHE -->|miss| ROUTER
    ROUTER --> OAI
    ROUTER --> ANT
    ROUTER --> MIS
    ROUTER --> GRQ
    ROUTER --> STREAM
    STREAM -->|SSE chunks| APP
    STREAM --> LOGGER
    LOGGER --> PG
    LOGGER --> RD
    RD --> AGG
    AGG --> PG
    PG --> BILL
```

Every request passes through auth, rate limiting, and cache check before reaching the provider router. If the cache misses, the router picks a provider, forwards the request, and streams the response back. Background workers handle everything async: logging, usage aggregation, billing, and provider health monitoring.

---

## API Key Authentication and Tenant Resolution

Relay is an API product. There are no browser sessions, no subdomains per tenant, no cookie-based auth for the hot path. Authentication is an API key in the `Authorization` header.

```
Authorization: Bearer rly_live_sk_abc123xyz...
```

The key itself encodes enough information to skip a lookup in most cases, but the real resolution always hits the database (or Redis cache) once per request.

### API key design

Relay uses prefixed, hashed API keys. The key the user sees is never stored. Only its SHA-256 hash is stored.

```
Full key:    rly_live_sk_4xKp9mN2qR8vL3wB7tJ6cD0hF5sG1eA
Prefix:      rly_live_
Environment: live (vs test)
Random part: 4xKp9mN2qR8vL3wB7tJ6cD0hF5sG1eA (256 bits, URL-safe base64)
```

The prefix makes it easy to detect keys in source code scanning, error logs, and GitHub secret scanning. Stripe does this. Anthropic does this. It is a good pattern.

```sql
CREATE TABLE api_keys (
    id              TEXT        PRIMARY KEY,
    tenant_id       TEXT        NOT NULL REFERENCES tenants(id),
    name            TEXT        NOT NULL,           -- "Production", "Dev", "CI"
    key_prefix      TEXT        NOT NULL,           -- rly_live_ or rly_test_
    key_hash        TEXT        NOT NULL UNIQUE,    -- SHA-256 of the full key
    key_hint        TEXT        NOT NULL,           -- last 4 chars: "...1eA"
    environment     TEXT        NOT NULL DEFAULT 'live',
    status          TEXT        NOT NULL DEFAULT 'active',
    last_used_at    TIMESTAMPTZ,
    expires_at      TIMESTAMPTZ,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    created_by      TEXT        NOT NULL REFERENCES users(id)
);

CREATE INDEX idx_api_keys_tenant_id ON api_keys(tenant_id);
CREATE INDEX idx_api_keys_key_hash  ON api_keys(key_hash);
```

You never store the full key. When you create a key, you show it to the user once and tell them to copy it. After that, if they lose it, they rotate it. The `key_hint` column (last 4 characters) helps them identify which key is which in the dashboard.

### Test mode

`rly_test_sk_` keys run in test mode. Relay does not forward test requests to real providers. It returns deterministic mocked responses from a fixed response library. This means no provider costs during development, predictable responses in CI/CD pipelines, and no test traffic polluting the live usage dashboard.

Test mode behaves identically to live mode in every other way: rate limits apply, tokens are counted, requests are logged, errors are returned in the same format. The only difference is the backend. When Relay sees a test key, the router skips provider selection entirely and calls a local mock handler instead.

```sql
-- Requests table distinguishes test from live via the api_key's environment
-- api_keys.environment = 'live' | 'test'
-- usage_daily is partitioned by environment so dashboards stay clean
```

Test responses are deterministic based on the request hash. The same input always produces the same mock output. This makes test assertions reliable.

### Auth flow

```mermaid
flowchart LR
    A["Request\nAuthorization: Bearer rly_live_sk_..."] --> B["Extract key\nfrom header"]
    B --> C["SHA-256 hash\nthe key"]
    C --> D{"Redis cache\nkey_hash lookup?"}
    D -->|hit| E["Return cached\ntenant + key metadata"]
    D -->|miss| F["Postgres lookup\nWHERE key_hash = ?"]
    F --> G{"Found and \nstatus = active?"}
    G -->|no| H[401 Unauthorized]
    G -->|yes| I["Cache in Redis\nTTL 5 minutes"]
    I --> J["Inject TenantContext\ninto request"]
    J --> K[Next middleware]
```

The key is hashed on every request and looked up in Redis first. A cache hit means zero database reads for auth. A miss hits Postgres and populates Redis for the next 5 minutes. The resolved tenant context flows through every subsequent middleware and handler via the request context.

{{< code lang="go" >}}
func APIKeyMiddleware(keys KeyStore, cache Cache) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            raw := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
            if raw == "" {
                writeError(w, 401, "missing api key")
                return
            }

            hash := sha256Hex(raw)

            key, err := cache.GetAPIKey(r.Context(), hash)
            if err != nil {
                key, err = keys.GetByHash(r.Context(), hash)
                if err != nil || key.Status != "active" {
                    writeError(w, 401, "invalid api key")
                    return
                }
                cache.SetAPIKey(r.Context(), hash, key, 5*time.Minute)
            }

            ctx := WithAPIKey(r.Context(), key)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
}
{{< /code >}}

---

## Database Schema Design

### Tenants and users

```sql
CREATE TABLE tenants (
    id          TEXT        PRIMARY KEY,             -- ten_abc123
    slug        TEXT        UNIQUE NOT NULL,         -- acme
    name        TEXT        NOT NULL,                -- Acme Inc.
    plan        TEXT        NOT NULL DEFAULT 'free', -- free | starter | pro | enterprise
    status      TEXT        NOT NULL DEFAULT 'trial',-- trial | active | suspended | deleted
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    deleted_at  TIMESTAMPTZ,
    metadata    JSONB
);

CREATE INDEX idx_tenants_slug   ON tenants(slug);
CREATE INDEX idx_tenants_status ON tenants(status);

CREATE TABLE users (
    id          TEXT        PRIMARY KEY,
    tenant_id   TEXT        NOT NULL REFERENCES tenants(id),
    email       TEXT        NOT NULL,
    role        TEXT        NOT NULL DEFAULT 'member', -- owner | admin | member | viewer
    status      TEXT        NOT NULL DEFAULT 'active',
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    deleted_at  TIMESTAMPTZ,
    UNIQUE(tenant_id, email)
);

CREATE INDEX idx_users_tenant_id ON users(tenant_id);
CREATE INDEX idx_users_email     ON users(email);
```

### Model catalog

Relay maintains a catalog of all supported models with their pricing. This drives cost calculation and provider routing.

```sql
CREATE TABLE providers (
    id          TEXT    PRIMARY KEY,         -- openai | anthropic | mistral | groq
    name        TEXT    NOT NULL,            -- OpenAI
    base_url    TEXT    NOT NULL,            -- https://api.openai.com/v1
    status      TEXT    NOT NULL DEFAULT 'active', -- active | degraded | down
    priority    INT     NOT NULL DEFAULT 100,      -- lower = more preferred
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE models (
    id                  TEXT    PRIMARY KEY,        -- gpt-5.5 | claude-3-5-sonnet-20241022
    provider_id         TEXT    NOT NULL REFERENCES providers(id),
    relay_name          TEXT    UNIQUE NOT NULL,    -- the name tenants use: "gpt-5.5"
    provider_name       TEXT    NOT NULL,           -- name the provider actually accepts
    display_name        TEXT    NOT NULL,           -- GPT-4o
    input_cost_per_1m   NUMERIC NOT NULL,           -- USD per 1M input tokens
    output_cost_per_1m  NUMERIC NOT NULL,           -- USD per 1M output tokens
    context_window      INT     NOT NULL,           -- max tokens in context
    supports_streaming  BOOLEAN NOT NULL DEFAULT true,
    supports_functions  BOOLEAN NOT NULL DEFAULT false,
    supports_vision     BOOLEAN NOT NULL DEFAULT false,
    status              TEXT    NOT NULL DEFAULT 'active',
    created_at          TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_models_provider_id ON models(provider_id);
```

A few example rows of how the table looks:

```
id                            provider_id  relay_name           input_cost   output_cost  context_window
gpt-5.5                       openai       gpt-5.5               2.50         10.00        128000
claude-3-5-sonnet-20241022    anthropic    claude-3-5-sonnet    3.00         15.00        200000
mistral-large-2411            mistral      mistral-large        2.00         6.00         131072
llama-3.3-70b-versatile       groq         llama-3.3-70b        0.59         0.79         128000
```

Costs are in USD per 1 million tokens.

### Provider credentials

Relay uses its own API keys to call providers. These are pooled: Relay pays the providers, marks up the cost, and charges tenants.

For enterprise tenants who want to bring their own keys (BYOK) and pay providers directly, Relay supports that too. The `tenant_id` being NULL means the credential belongs to Relay.

Provider credentials are among the most sensitive data Relay stores. A leaked provider key means an attacker can run LLM calls billed to that key. The `key_encrypted` column stores the key encrypted with AES-256-GCM using a data encryption key (DEK) that is itself encrypted by a master key stored in a KMS (AWS KMS or HashiCorp Vault). The plaintext key never touches disk. Relay calls the KMS API to decrypt the DEK at runtime, decrypts the credential in memory, uses it for the provider request, and discards it.

Key rotation works in two steps: the KMS master key is rotated on a schedule (annually at minimum), then a background job re-encrypts every `key_encrypted` value under the new master key. The job runs row by row with checkpointing so it can resume if interrupted.

```sql
CREATE TABLE provider_credentials (
    id              TEXT        PRIMARY KEY,
    tenant_id       TEXT        REFERENCES tenants(id),  -- NULL = Relay's pooled key
    provider_id     TEXT        NOT NULL REFERENCES providers(id),
    key_encrypted   BYTEA       NOT NULL,    -- encrypted with Relay's KMS key
    key_hint        TEXT        NOT NULL,    -- last 4 chars for display
    is_relay_owned  BOOLEAN     NOT NULL DEFAULT true,
    status          TEXT        NOT NULL DEFAULT 'active',
    rpm_limit       INT,                     -- provider's RPM cap for this key
    tpm_limit       INT,                     -- provider's TPM cap for this key
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(tenant_id, provider_id)
);
```

### Requests

Every API call Relay processes gets a row here. This is the source of truth for billing, debugging, and usage dashboards.

```sql
CREATE TABLE requests (
    id              TEXT        PRIMARY KEY,            -- req_abc123
    tenant_id       TEXT        NOT NULL REFERENCES tenants(id),
    api_key_id      TEXT        NOT NULL REFERENCES api_keys(id),
    model_id        TEXT        NOT NULL REFERENCES models(id),
    provider_id     TEXT        NOT NULL REFERENCES providers(id),
    credential_id   TEXT        NOT NULL REFERENCES provider_credentials(id),

    -- What kind of request
    request_type    TEXT        NOT NULL,               -- chat | completion | embedding
    stream          BOOLEAN     NOT NULL DEFAULT false,

    -- Token counts (filled in after response)
    input_tokens    INT,
    output_tokens   INT,
    total_tokens    INT,

    -- Cost in USD
    input_cost      NUMERIC(12,8),
    output_cost     NUMERIC(12,8),
    total_cost      NUMERIC(12,8),

    -- Relay's markup (stored separately for accounting)
    markup_rate     NUMERIC(5,4) NOT NULL DEFAULT 0.20, -- 20% markup by default
    relay_revenue   NUMERIC(12,8),

    -- Timing
    started_at      TIMESTAMPTZ NOT NULL,
    first_token_at  TIMESTAMPTZ,       -- for streaming: TTFT
    completed_at    TIMESTAMPTZ,
    latency_ms      INT,
    ttft_ms         INT,               -- time to first token

    -- Outcome
    status          TEXT        NOT NULL, -- success | error | cached | timeout
    error_code      TEXT,
    error_message   TEXT,
    http_status     INT,

    -- Cache
    cache_hit       BOOLEAN     NOT NULL DEFAULT false,
    cache_key       TEXT,

    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_requests_tenant_created  ON requests(tenant_id, created_at DESC);
CREATE INDEX idx_requests_model_created   ON requests(model_id, created_at DESC);
CREATE INDEX idx_requests_api_key_created ON requests(api_key_id, created_at DESC);
CREATE INDEX idx_requests_status          ON requests(status, created_at DESC);
```

The `requests` table grows fast. At 1,000 tenants each making 100 requests per day, that is 100,000 rows per day, 3 million per month. After 90 days you want to start archiving cold rows to cheaper storage (S3 + Parquet). Keep hot rows (last 30 days) in Postgres. Query historical data via a warehouse.

### Daily usage rollup

Querying `requests` for billing is expensive at scale. Aggregate into a daily rollup table that background workers update continuously.

```sql
CREATE TABLE usage_daily (
    tenant_id       TEXT    NOT NULL REFERENCES tenants(id),
    model_id        TEXT    NOT NULL REFERENCES models(id),
    date            DATE    NOT NULL,
    request_count   BIGINT  NOT NULL DEFAULT 0,
    input_tokens    BIGINT  NOT NULL DEFAULT 0,
    output_tokens   BIGINT  NOT NULL DEFAULT 0,
    total_tokens    BIGINT  NOT NULL DEFAULT 0,
    total_cost      NUMERIC(12,4) NOT NULL DEFAULT 0,
    relay_revenue   NUMERIC(12,4) NOT NULL DEFAULT 0,
    cache_hits      BIGINT  NOT NULL DEFAULT 0,
    error_count     BIGINT  NOT NULL DEFAULT 0,
    avg_latency_ms  INT,
    PRIMARY KEY (tenant_id, model_id, date)
);
```

---

## Request Flow: From API Call to Provider and Back

This is the core of Relay. Every API call goes through this pipeline.

```mermaid
sequenceDiagram
    actor App as Your App
    participant GW as Relay Gateway
    participant Auth as Auth Middleware
    participant RL as Rate Limiter
    participant Cache as Cache Layer
    participant Router as Provider Router
    participant OAI as OpenAI

    App->>GW: POST /v1/chat/completions<br/>Authorization: Bearer rly_live_sk_...
    GW->>Auth: validate API key
    Auth->>Auth: SHA-256 hash key<br/>lookup in Redis / Postgres
    Auth-->>GW: TenantContext injected

    GW->>RL: check RPM + TPM limits
    RL-->>GW: ok

    GW->>Cache: check cache<br/>hash(model + messages + params)
    Cache-->>GW: miss

    GW->>Router: route request<br/>model=gpt-5.5
    Router->>Router: lookup model in catalog<br/>select provider + credential
    Router->>OAI: POST api.openai.com/v1/chat/completions<br/>Authorization: Bearer sk-relay-pooled-...

    OAI-->>Router: 200 OK (streaming SSE)
    Router-->>App: pipe SSE chunks

    Router->>Router: count tokens from response<br/>calculate cost

    Router->>GW: log request async
    GW->>GW: INSERT into requests<br/>INCR Redis counters
```

### Model routing and failover

The router resolves which provider to use for a given model name. If the primary provider is down or over its rate limit, it falls back to an alternative.

```mermaid
flowchart TD
    A["Request<br/>model = gpt-5.5"] --> B["Lookup model<br/>in catalog"]
    B --> C["Get providers<br/>ordered by priority"]
    C --> D{"Tenant has<br/>BYOK key?"}
    D -->|yes| E[Use tenant's key]
    D -->|no| F[Use Relay pooled key]
    E --> G{"Provider<br/>healthy?"}
    F --> G
    G -->|yes| H[Forward request]
    G -->|no| I["Next provider<br/>by priority"]
    I --> J{Any left?}
    J -->|yes| G
    J -->|no| K[503 No providers available]
    H -->|5xx from provider| I
    H -->|success| L[Return response]
```

{{< code lang="go" >}}
func (r *Router) Route(ctx context.Context, req *RelayRequest) (*RelayResponse, error) {
    model, err := r.catalog.GetModel(req.Model)
    if err != nil {
        return nil, ErrModelNotFound
    }

    providers := r.getOrderedProviders(ctx, model)
    for _, p := range providers {
        cred, err := r.getCredential(ctx, req.TenantID, p.ID)
        if err != nil {
            continue
        }

        resp, err := r.forward(ctx, p, cred, req)
        if err == nil {
            return resp, nil
        }

        // 5xx from provider: try next
        if isProviderError(err) {
            r.health.MarkDegraded(p.ID)
            continue
        }

        // 4xx from provider: the request itself is bad, do not retry
        return nil, err
    }

    return nil, ErrNoProvidersAvailable
}
{{< /code >}}

### Handling streaming (SSE)

When the tenant sends `"stream": true`, the provider returns a Server-Sent Events stream. Relay proxies it chunk by chunk. This is not a buffered response, it is a live pipe.

```mermaid
sequenceDiagram
    participant App
    participant Relay
    participant Provider

    App->>Relay: POST /v1/chat/completions<br/>stream: true
    Relay->>Provider: POST (stream: true)
    Provider-->>Relay: data: {"delta": {"content": "Hello"}}
    Relay-->>App: data: {"delta": {"content": "Hello"}}
    Provider-->>Relay: data: {"delta": {"content": " world"}}
    Relay-->>App: data: {"delta": {"content": " world"}}
    Provider-->>Relay: data: [DONE]
    Relay-->>App: data: [DONE]
    Relay->>Relay: count tokens from chunks<br/>log request async
```

Relay counts tokens as chunks arrive by tracking the `usage` field that providers send in the final chunk. For providers that do not send usage in streaming mode, Relay uses a local tokenizer (tiktoken for OpenAI models, SentencePiece for others) to estimate counts. The estimate is reconciled against the actual count when available.

Token counting is done on a goroutine that reads from the same SSE stream, separate from the goroutine that proxies chunks to the client. The client sees no added latency.

### Retry logic

Relay retries failed provider requests automatically, but only for the right kind of failures.

**Retryable**: 5xx errors from the provider (server errors), network timeouts, connection resets. These indicate the provider had a problem, not that the request was wrong.

**Not retryable**: 4xx errors from the provider (except 429). A 400 means the request was malformed. A 401 means the credential is wrong. Retrying will not help and wastes time.

**Provider rate limit (429 from provider)**: Relay does not wait and retry on the same key. It immediately switches to the next available provider key or provider. Waiting for a rate limit window to reset would add seconds of latency for the tenant.

Retry policy: up to 3 attempts, 100ms exponential backoff between attempts, each attempt on a different provider or key if possible. If all attempts fail, Relay returns the last error to the tenant.

Relay does not retry on behalf of the tenant after returning an error. If Relay returns a 502 or 503, the tenant's code decides whether to retry at their layer.

### Error response format

Every error Relay returns uses the same JSON structure. Developers should be able to write one error handler and handle all Relay errors with it.

```json
{
  "error": {
    "code": "rate_limit_exceeded",
    "message": "You have exceeded 100 requests per minute on the free plan.",
    "type": "rate_limit_error",
    "details": {
      "limit": 100,
      "window": "per_minute",
      "reset_at": "2026-07-01T12:05:00Z",
      "upgrade_url": "https://relay.lorbic.com/billing"
    }
  }
}
```

The full set of error codes:

| HTTP status | error.code | Meaning |
|---|---|---|
| 401 | `invalid_api_key` | Key not found or revoked |
| 402 | `tenant_suspended` | Account suspended, check billing |
| 402 | `quota_exceeded` | Hard quota limit hit (seats, storage) |
| 404 | `model_not_found` | Requested model does not exist in catalog |
| 422 | `invalid_request` | Request body failed validation |
| 429 | `rate_limit_exceeded` | RPM or TPM limit hit |
| 502 | `provider_error` | Provider returned an error after retries |
| 503 | `no_provider_available` | All providers for this model are degraded |
| 504 | `request_timeout` | Provider did not respond within the timeout |

The `details` field is present on `rate_limit_exceeded` and `quota_exceeded` errors so tenants can surface actionable information to their own users. If a tenant's app is hitting Relay's rate limit, they should know the reset time and the upgrade path, not just a generic "too many requests."

---

## Rate Limiting

Relay has to manage rate limits at two levels.

**Inbound**: protecting Relay's own infrastructure from tenants who send too many requests.

**Outbound**: protecting Relay's provider API keys from hitting the provider's limits.

Both use Redis counters with sliding windows.

The dual-layer rate limiting was the part I found most interesting while designing this. Most rate limiting content covers the inbound side. The outbound side is specific to API gateway products but just as important. If Relay's OpenAI key hits OpenAI's TPM limit because of one large tenant, every other tenant on that key starts getting 429s from the provider. The router has to detect this and re-route before it becomes the tenant's problem.

### Inbound tenant limits

```sql
-- Plan-level defaults
CREATE TABLE quota_definitions (
    plan         TEXT    NOT NULL,
    resource     TEXT    NOT NULL,   -- rpm | tpm | requests_per_day
    limit_value  BIGINT  NOT NULL,   -- -1 = unlimited
    window       TEXT    NOT NULL,   -- per_minute | per_day
    UNIQUE(plan, resource, window)
);

-- Sample values:
-- plan      resource          limit     window
-- free      rpm               60        per_minute
-- free      tpm               40000     per_minute
-- free      requests_per_day  500       per_day
-- starter   rpm               500       per_minute
-- starter   tpm               400000    per_minute
-- pro       rpm               3000      per_minute
-- pro       tpm               2000000   per_minute
-- pro       requests_per_day  -1        per_day (unlimited)

-- Per-tenant overrides (for enterprise deals or support credits)
CREATE TABLE quota_overrides (
    id           TEXT        PRIMARY KEY,
    tenant_id    TEXT        NOT NULL REFERENCES tenants(id),
    resource     TEXT        NOT NULL,
    limit_value  BIGINT      NOT NULL,
    window       TEXT        NOT NULL,
    reason       TEXT,
    expires_at   TIMESTAMPTZ,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(tenant_id, resource, window)
);
```

Redis key structure for inbound limits:

```
rl:in:ten_abc123:rpm   → sliding window counter, 60s window
rl:in:ten_abc123:tpm   → sliding window counter, 60s window
```

### Outbound provider limits

Each of Relay's provider API keys has its own RPM and TPM cap set by the provider. Relay tracks these in Redis too.

```
rl:out:openai:key_xyz:rpm   → counter, 60s window
rl:out:openai:key_xyz:tpm   → counter, 60s window
```

When a pooled key is close to its provider limit, the router skips it and picks the next available key or provider.

### The rate limit pipeline

```mermaid
flowchart LR
    A[Request] --> B{"Inbound tenant<br/>RPM check"}
    B -->|over| C["429 Too Many Requests<br/>retry-after header"]
    B -->|ok| D{"Inbound tenant<br/>TPM check"}
    D -->|over| C
    D -->|ok| E[Route to provider]
    E --> F{"Outbound key<br/>RPM check"}
    F -->|over| G["Pick next key<br/>or provider"]
    F -->|ok| H[Forward to provider]
    G --> F
```

A request fails fast at inbound RPM or TPM before any provider call is made. The outbound check happens inside the router, per-key. If a key is saturated, the router picks the next key or the next provider, not the next retry window.

The TPM check on inbound traffic uses an estimated token count from the request body (input tokens only). The actual token count (input + output) is only known after the response. Relay does a conservative pre-check against the estimated input, then reconciles and updates the TPM counter after the response.

---

## Token Cost Tracking and Billing

Relay's pricing model: Relay pays the provider at published rates, marks up by 20%, and charges tenants at the marked-up rate. The markup funds Relay's infrastructure and margin.

```
Tenant sends:  1,000 input tokens, 500 output tokens on gpt-5.5
Provider cost: (1000 / 1M) × $2.50 + (500 / 1M) × $10.00 = $0.00250 + $0.00500 = $0.00750
Relay markup:  $0.00750 × 1.20 = $0.00900
Tenant billed: $0.00900
Relay revenue: $0.00150
```

Every request row in the `requests` table stores both the raw provider cost and the Relay revenue. This makes accounting and audits straightforward.

### Cost calculation

```mermaid
flowchart LR
    A["Response received<br/>input_tokens=1000<br/>output_tokens=500"] --> B["Lookup model pricing<br/>input_cost_per_1m<br/>output_cost_per_1m"]
    B --> C["input_cost = tokens / 1M x rate<br/>output_cost = tokens / 1M x rate"]
    C --> D[total_cost = input + output]
    D --> E[relay_revenue = total_cost x markup_rate]
    E --> F[INSERT into requests]
    F --> G[INCR usage_daily rollup]
```

### Billing cycle

Relay charges tenants monthly via Stripe. The billing worker runs at the start of each month, reads the `usage_daily` table for the prior month, and creates a Stripe invoice line item per model per tenant.

```sql
CREATE TABLE billing_periods (
    id          TEXT        PRIMARY KEY,
    tenant_id   TEXT        NOT NULL REFERENCES tenants(id),
    period_start DATE       NOT NULL,
    period_end   DATE       NOT NULL,
    status      TEXT        NOT NULL DEFAULT 'open',  -- open | finalized | paid | failed
    total_cost  NUMERIC(12,4),
    invoice_id  TEXT,              -- Stripe invoice ID
    finalized_at TIMESTAMPTZ,
    paid_at     TIMESTAMPTZ,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(tenant_id, period_start)
);

CREATE TABLE billing_line_items (
    id              TEXT        PRIMARY KEY,
    billing_period_id TEXT      NOT NULL REFERENCES billing_periods(id),
    tenant_id       TEXT        NOT NULL REFERENCES tenants(id),
    model_id        TEXT        NOT NULL REFERENCES models(id),
    request_count   BIGINT      NOT NULL,
    input_tokens    BIGINT      NOT NULL,
    output_tokens   BIGINT      NOT NULL,
    total_cost      NUMERIC(12,4) NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(billing_period_id, model_id)
);
```

> Billing is an entire system design on its own: Stripe subscription lifecycle, idempotent webhook handling, retry logic for failed payments, proration, credits, and tax calculation. This post covers the data model. The billing pipeline gets its own post.

---

## Response Caching: Exact Match and Semantic

Caching completions saves money. If 1,000 tenants call the same model with the same system prompt and similar questions, Relay can serve many of those from cache instead of paying the provider.

Relay supports two cache strategies.

### Exact match caching

The simplest form. Hash the full request (model + messages + temperature + all params). If you have seen this exact request before, return the cached response.

```
cache_key = SHA-256(
    model_id +
    JSON.stringify(messages) +
    temperature + top_p + max_tokens + ...
)
```

Stored in Redis with a configurable TTL. Default 1 hour. Tenants can set their own TTL per API call via a `x-relay-cache-ttl` header.

### Semantic caching

Exact match only works for identical requests. Semantic caching uses embeddings to find requests that are similar enough to return the same response.

```mermaid
flowchart TD
    A[New request] --> B["Embed the user message<br/>using a small model"]
    B --> C["Vector similarity search<br/>against cached embeddings"]
    C --> D{"Similarity score<br/>> threshold?"}
    D -->|yes| E["Return cached response<br/>cache_hit = semantic"]
    D -->|no| F["Forward to provider<br/>cache the response + embedding"]
```

The embedding model for cache lookup should be cheap and fast (text-embedding-3-small or similar). The cost of generating the lookup embedding is tiny compared to the LLM call it might save.

```sql
CREATE TABLE response_cache (
    id              TEXT        PRIMARY KEY,
    tenant_id       TEXT        NOT NULL REFERENCES tenants(id),
    cache_key       TEXT        NOT NULL,         -- exact match hash
    model_id        TEXT        NOT NULL REFERENCES models(id),
    request_hash    TEXT        NOT NULL,
    response_body   JSONB       NOT NULL,
    input_tokens    INT         NOT NULL,
    output_tokens   INT         NOT NULL,
    embedding       vector(1536),                 -- pgvector, for semantic cache
    hit_count       INT         NOT NULL DEFAULT 0,
    expires_at      TIMESTAMPTZ NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_response_cache_tenant_key ON response_cache(tenant_id, cache_key);
CREATE INDEX idx_response_cache_embedding  ON response_cache
    USING hnsw (embedding vector_cosine_ops)
    WHERE embedding IS NOT NULL;
```

---

## Observability

You cannot operate a multi-tenant API platform without per-tenant observability. When a tenant files a support ticket saying "your API is slow," you need to be able to pull up exactly their requests, their latencies, and their errors in under a minute.

### What to track per request

Every request log entry should have:

- `tenant_id` and `api_key_id`
- `model_id` and `provider_id`
- `latency_ms` and `ttft_ms` (time to first token, the metric tenants care about for streaming)
- `input_tokens`, `output_tokens`, `total_cost`
- `status` and `error_code`
- `cache_hit`

These are already in the `requests` table. Make sure every log line, trace span, and metric tag also carries `tenant_id`.

### Structured logging

{{< code lang="go" >}}
func RequestLoggingMiddleware(logger *slog.Logger) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := APIKeyFromContext(r.Context())
            reqLogger := logger.With(
                slog.String("tenant_id", key.TenantID),
                slog.String("api_key_id", key.ID),
                slog.String("request_id", newRequestID()),
            )
            ctx := WithLogger(r.Context(), reqLogger)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
}
{{< /code >}}

### Health status per provider

Relay's router uses a health tracker to avoid sending requests to degraded providers. The tracker works on a circuit breaker pattern: after N consecutive errors from a provider, mark it degraded and stop routing to it until it recovers.

```sql
CREATE TABLE provider_health_events (
    id          TEXT        PRIMARY KEY,
    provider_id TEXT        NOT NULL REFERENCES providers(id),
    status      TEXT        NOT NULL,       -- healthy | degraded | down
    error_rate  NUMERIC,                   -- 5-minute error rate at time of event
    p95_ms      INT,                       -- p95 latency at time of event
    message     TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_provider_health_events ON provider_health_events(provider_id, created_at DESC);
```

The provider status page (status.relay.lorbic.com) reads from this table. Tenants can subscribe to status updates.

### Noisy neighbor detection

In the pool model, one tenant running an aggressive batch job can saturate Relay's provider keys and cause latency for everyone else. Detect this by tracking per-tenant request volume in Redis with a 1-minute rolling window. If a single tenant exceeds a percentage threshold of total traffic, flag them and apply stricter rate limits automatically.

---

## Role-Based Access Control

API calls use API key auth. The Relay dashboard (where tenants manage keys, view usage, configure settings) uses JWT auth.

```json
{
  "iss": "relay.lorbic.com",
  "sub": "usr_abc123",
  "tenant_id": "ten_xyz789",
  "tenant_slug": "acme",
  "role": "admin",
  "permissions": ["api_keys:read", "api_keys:write", "billing:read", "usage:read"],
  "iat": 1751400000,
  "exp": 1751500000
}
```

### RBAC tables

```sql
CREATE TABLE roles (
    id          TEXT        PRIMARY KEY,
    tenant_id   TEXT        NOT NULL REFERENCES tenants(id),
    name        TEXT        NOT NULL,       -- owner | admin | developer | viewer
    is_system   BOOLEAN     NOT NULL DEFAULT false,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(tenant_id, name)
);

CREATE TABLE permissions (
    id          TEXT PRIMARY KEY,
    resource    TEXT NOT NULL,              -- api_keys | usage | billing | members | settings
    action      TEXT NOT NULL,             -- create | read | update | delete
    UNIQUE(resource, action)
);

CREATE TABLE role_permissions (
    role_id       TEXT NOT NULL REFERENCES roles(id),
    permission_id TEXT NOT NULL REFERENCES permissions(id),
    PRIMARY KEY (role_id, permission_id)
);

CREATE TABLE user_roles (
    user_id   TEXT NOT NULL REFERENCES users(id),
    role_id   TEXT NOT NULL REFERENCES roles(id),
    PRIMARY KEY (user_id, role_id)
);
```

### Audit log

Every sensitive action in the dashboard gets recorded: key creation, key revocation, plan changes, member removal, billing configuration.

```sql
CREATE TABLE audit_log (
    id            TEXT        PRIMARY KEY,
    tenant_id     TEXT        REFERENCES tenants(id),
    actor_id      TEXT        NOT NULL,
    actor_type    TEXT        NOT NULL,    -- user | platform_admin | system
    action        TEXT        NOT NULL,    -- api_key.created | api_key.revoked | member.removed
    resource_id   TEXT,
    resource_type TEXT,
    metadata      JSONB,
    ip_address    INET,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_audit_log_tenant   ON audit_log(tenant_id, created_at DESC);
CREATE INDEX idx_audit_log_actor    ON audit_log(actor_id, created_at DESC);
```

---

## Tenant Provisioning: From Signup to First API Call

### What happens when someone signs up

When a user submits the signup form, Relay does the minimum work synchronously: insert a tenant row with `status=PROVISIONING`, create the owner user record, and return a 201. The browser redirects immediately. Everything else runs in the background.

The background worker picks up the job and does the slower parts: seeding system roles, creating a Stripe customer record, and flipping the tenant to `TRIAL`. The Stripe customer is created at signup, not when the user adds a payment method. This is intentional. You need a Stripe customer ID to attach payment methods, subscriptions, and invoices later. Creating it lazily (at payment time) means every billing code path has to handle the case where the ID does not exist yet, which causes bugs.

The welcome email includes the tenant's first API key. The key is generated by the worker, not the API handler, because key generation requires the tenant record to be fully set up.

If the worker fails partway through, the tenant stays in `PROVISIONING`. A monitoring job scans for tenants stuck in `PROVISIONING` for more than 5 minutes and either retries the job or pages on-call.

```mermaid
sequenceDiagram
    actor User
    participant API
    participant Queue
    participant Worker
    participant DB
    participant Stripe

    User->>API: POST /auth/signup<br/>{ email, password, org_name }
    API->>DB: INSERT tenant (status=PROVISIONING)
    API->>DB: INSERT user (role=owner)
    API->>Queue: enqueue ProvisionTenant job
    API-->>User: 201 Created<br/>{ tenant_id, redirect: /onboarding }

    Worker->>Queue: dequeue job
    Worker->>DB: seed default roles (owner, admin, developer, viewer)
    Worker->>DB: seed plan quota definitions
    Worker->>Stripe: create Stripe customer
    Worker->>DB: UPDATE tenant SET status=TRIAL, stripe_customer_id=cus_...
    Worker-->>User: send welcome email with first API key
```

### Tenant status transitions

The `status` column on the `tenants` table is the single source of truth for what a tenant can do. Every API request, every dashboard login, and every background job checks this column before doing anything.

**PROVISIONING**: The tenant record exists but setup is not complete. API calls are rejected with a 503. The dashboard shows a loading state.

**TRIAL**: Setup is complete. The tenant has full API access up to the trial plan's rate limits. A trial expiry timestamp lives in `tenant_config`. When it passes without a payment method being added, a background job moves the tenant to `EXPIRED`.

**ACTIVE**: A payment method is on file and billing is working. Full access based on the tenant's plan.

**SUSPENDED**: Payment failed, or a Relay admin manually suspended the account. API calls return a 402 with a clear message and a link to the billing page. The tenant's data is untouched. If payment resolves within the grace period, the account returns to `ACTIVE` with no data loss. This is intentional: you want tenants to fix their billing, not lose access to everything they built.

**EXPIRED**: The trial ended without a payment method being added. Same behavior as `SUSPENDED` for API calls. After N days with no action, the tenant moves to `DELETED`.

**DELETED**: Soft delete. The row stays in the database with `deleted_at` set, but all API calls are rejected. Data is retained for 30 days for recovery or dispute resolution, then a GDPR-compliant hard deletion job removes all tenant rows across every table.

Status transitions are intentionally one-directional except for `SUSPENDED` back to `ACTIVE`. You cannot un-delete a tenant. You cannot go from `ACTIVE` back to `TRIAL`. These constraints are enforced at the application layer via explicit state machine logic, not just database constraints.

```mermaid
stateDiagram-v2
    [*] --> PROVISIONING : signup
    PROVISIONING --> TRIAL : provisioning complete
    TRIAL --> ACTIVE : payment method added
    TRIAL --> EXPIRED : trial period ends
    ACTIVE --> SUSPENDED : payment failed or admin action
    SUSPENDED --> ACTIVE : payment resolved
    SUSPENDED --> DELETED : grace period expires
    EXPIRED --> DELETED : no conversion
    ACTIVE --> DELETED : user cancels
    DELETED --> [*]
```

Suspended tenants get a 429 response with a billing error message on every API call. This is important: the error message should tell them what is wrong and link them to the billing page. A silent 401 causes confusion.

---

## Feature Flags and Per-Tenant Configuration

```sql
CREATE TABLE plan_features (
    plan     TEXT    NOT NULL,
    feature  TEXT    NOT NULL,    -- semantic_cache | byok | sso | audit_log | higher_rpm | webhooks
    enabled  BOOLEAN NOT NULL DEFAULT true,
    PRIMARY KEY (plan, feature)
);

CREATE TABLE tenant_feature_overrides (
    tenant_id   TEXT        NOT NULL REFERENCES tenants(id),
    feature     TEXT        NOT NULL,
    enabled     BOOLEAN     NOT NULL,
    reason      TEXT,
    expires_at  TIMESTAMPTZ,
    PRIMARY KEY (tenant_id, feature)
);

CREATE TABLE tenant_config (
    tenant_id   TEXT        NOT NULL REFERENCES tenants(id),
    key         TEXT        NOT NULL,
    value       TEXT        NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, key)
);

-- Example config rows:
-- tenant_id   key                      value
-- ten_abc     default_cache_ttl_sec    3600
-- ten_abc     markup_rate              0.15     (negotiated lower markup for volume)
-- ten_abc     allowed_models           ["gpt-5.5","claude-3-5-sonnet"]
-- ten_abc     webhook_url              https://acme.com/hooks/relay
```

Cache the feature flag state per tenant in Redis (60s TTL). Do not query Postgres on every API call.

---

## Full Architecture

```mermaid
graph TB
    subgraph tenants["Tenants"]
        APP1[App: acme]
        APP2[App: beta]
        DASH[Dashboard]
    end

    subgraph edge["Edge"]
        DNS[Wildcard DNS]
        LB[Load Balancer]
    end

    subgraph gateway["Gateway"]
        GW[API Gateway]
        AUTH[API Key Auth]
        RL_IN[Rate Limiter]
        CACHE[Cache Layer]
        ROUTER[Provider Router]
        STREAM[SSE Handler]
    end

    subgraph dashapi["Dashboard API"]
        DASH_API[Dashboard API]
    end

    subgraph providers["Providers"]
        OAI[OpenAI]
        ANT[Anthropic]
        MIS[Mistral]
        GRQ[Groq]
    end

    subgraph data["Data"]
        PG[(Postgres)]
        RD[(Redis)]
        S3[(S3)]
    end

    subgraph workers["Workers"]
        LOGGER[Request Logger]
        AGG[Usage Aggregator]
        BILL[Billing Worker]
        HEALTH[Health Monitor]
        ARCHIVE[Request Archiver]
    end

    APP1 --> DNS
    APP2 --> DNS
    DASH --> DNS
    DNS --> LB
    LB --> GW
    LB --> DASH_API

    GW --> AUTH
    AUTH --> RL_IN
    RL_IN --> CACHE
    CACHE -->|miss| ROUTER
    ROUTER --> OAI
    ROUTER --> ANT
    ROUTER --> MIS
    ROUTER --> GRQ
    ROUTER --> STREAM
    STREAM --> LOGGER

    DASH_API --> PG
    DASH_API --> RD

    LOGGER --> PG
    LOGGER --> RD

    RD --> AGG
    AGG --> PG
    PG --> BILL
    PG --> ARCHIVE
    ARCHIVE --> S3
    HEALTH --> PG
    HEALTH --> RD
```

The gateway layer is stateless. Tenant apps hit the load balancer, which routes to any gateway pod. All shared state (rate limits, cache, feature flags) lives in Redis. All durable state (requests, usage, billing, config) lives in Postgres. Workers run separately and never block the hot path.

---

## Scaling a Multi-Tenant API Gateway

Relay is stateless at the gateway layer. You can add more gateway pods behind the load balancer without coordination. The state lives in Postgres and Redis.

**Redis** is the first bottleneck. Rate limit counters and cache lookups hit Redis on every request. Run Redis in cluster mode with at least 3 shards. Shard by `tenant_id` so one tenant's traffic does not create hot keys for others.

**Postgres** becomes the bottleneck as the `requests` table grows. Partition it by `created_at` (monthly partitions). Drop or archive partitions older than your retention window. Use PgBouncer in transaction mode for the gateway (short, fast INSERTs) and session mode for the background workers (longer transactions, aggregation queries).

**Provider key rotation**: as Relay grows, a single OpenAI API key will hit TPM limits. Relay maintains a pool of provider keys per provider and distributes load across them using the outbound rate limit counters in Redis. The router picks the key with the most remaining capacity in the current window.

---

## FAQ

**Why store the hash of an API key instead of the key itself?**

If your database is compromised, the attacker gets hashes, not keys. A raw API key in the database is a plaintext credential. SHA-256 is one-way: you verify a key by hashing the incoming value and comparing it against the stored hash, but you cannot reverse a hash back into the original key. Treat API keys the same way you treat passwords.

**How do you prevent a missing `WHERE tenant_id` clause from leaking data across tenants?**

Row-Level Security in Postgres. You attach a policy to every tenant-scoped table that restricts all queries to rows matching `app.current_tenant_id`, a session variable the application sets at the start of every request. A bare `SELECT * FROM requests` with no WHERE clause still returns only that tenant's rows. The policy runs inside the database engine, below the application layer, so a bug in application code cannot bypass it.

**What happens if OpenAI goes down?**

The router tries providers in priority order. If OpenAI returns a 5xx, it marks OpenAI as degraded and tries the next provider that supports the requested model. If no provider is available, the request returns 503. The health monitor runs on a short interval and flips providers back to healthy once they start returning 2xx responses again. Tenants see a brief error window during a full provider outage, not an extended one.

**How do you count tokens accurately for billing when streaming does not always return usage data?**

Providers that support `stream_options: { include_usage: true }` (OpenAI does) include a `usage` object in the final SSE chunk. Relay reads those counts directly. For providers that do not, Relay runs a local tokenizer (tiktoken for OpenAI-compatible models, SentencePiece for others) to estimate counts. The estimate is within 1-2% of the actual count in practice. When there is a discrepancy, Relay bills the lower number rather than over-billing the tenant.

**How do you handle rate limiting if Redis goes down?**

Redis Cluster with at least 3 nodes and automatic failover handles most availability scenarios. If the entire cluster becomes unreachable, Relay fails open: it allows requests through rather than rejecting all traffic. Losing rate limit enforcement for a short period during a Redis failure is an acceptable trade-off. Taking down the entire API is not.

**What is the noisy neighbor problem and how does it show up in a pool model SaaS?**

One tenant runs a large batch job and consumes a disproportionate share of shared infrastructure: database connections, query time, provider key capacity. This increases latency for every other tenant on the same cluster without them doing anything wrong. Relay tracks per-tenant request volume in Redis with a 1-minute rolling window. Tenants that exceed a share threshold get their rate limits tightened automatically. If the pattern persists, they are a candidate for migration to a silo.

**How do you migrate a tenant from the pool model to a silo without downtime?**

Postgres logical replication. Set up a replication slot on the shared database filtered to the tenant's rows. Let the silo database catch up to near-real-time lag. Then pause writes for that tenant briefly (a few seconds), apply the final delta, update the routing config to point new requests at the silo, and release the write pause. The tenant sees a brief stall, not a maintenance window.

**Why keep billing records in separate tables instead of querying the `requests` table at invoice time?**

The `requests` table gets very large and is archived after 30-90 days to keep query performance acceptable. Billing records need to be retained for years for tax, legal, and dispute reasons. Separating `billing_line_items` and `usage_daily` from raw request logs means you can archive or drop old request partitions without losing billing history. The rollup also makes monthly invoice generation a fast aggregation query rather than a multi-billion-row scan.

**What does BYOK mean and why would an enterprise tenant want it?**

Bring Your Own Keys. The tenant provides their own OpenAI or Anthropic API keys instead of Relay's pooled ones. They pay the provider directly at published rates, and Relay charges a flat platform fee instead of a per-token markup. Enterprises want BYOK for three reasons: cost control at high volume, data privacy (their traffic flows under their own API key, not Relay's), and compliance requirements that restrict which credentials can touch their data.

**Why use prefixed API keys like `rly_live_sk_`?**

Three reasons. First, the prefix makes keys identifiable in logs, error messages, and accidental source code commits. Second, secret scanning tools (GitHub, trufflehog, Doppler) can be configured to detect your specific prefix and alert on leaks before damage is done. Third, the environment segment (`live` vs `test`) lets Relay reject live keys on test endpoints and test keys on live endpoints at the routing layer, before any business logic runs.

**How does semantic caching differ from exact match caching, and what can go wrong?**

Exact match caching hashes the full request and returns a cached response only if the exact same request was made before. Hit rates are low. Semantic caching embeds the user message into a vector and finds cached responses from requests that are semantically similar. Hit rates are higher, but there is a risk: two questions that are phrased similarly but have meaningfully different answers could get the same cached response. The similarity threshold controls this trade-off. Set it too high and you get wrong answers. Set it too low and cache hits are rare. Semantic caching should never be enabled for requests where correctness is critical (medical, legal, financial).

**What happens to a tenant's data when their account is deleted?**

Relay uses a soft delete: the tenant row gets `deleted_at` set and `status` set to `DELETED`. All API access is immediately rejected. The underlying data stays in the database for 30 days. During that window, a tenant can contact support to recover their account. After 30 days, a scheduled deletion job hard-deletes all rows across every table where `tenant_id` matches, in dependency order (child tables first, then the tenant row). This satisfies GDPR Article 17 (right to erasure) and gives you an audit trail during the retention window.

---

## Things Worth Their Own Post

- **Billing pipeline in depth**: Stripe subscription webhooks, idempotent event processing, dunning logic, credit notes, usage-based invoice generation.
- **Semantic caching**: embedding model selection, similarity thresholds, cache invalidation strategy, cost-benefit analysis.
- **BYOK (Bring Your Own Keys)**: encrypting and storing tenant provider credentials, key rotation, audit trail for credential usage.
- **Webhooks**: delivering request completion events to tenants, retry logic, failure handling, signing payloads.
- **Provider health monitoring**: circuit breaker implementation, alerting on degradation, automatic failover testing.
- **Request archival**: streaming from Postgres to Parquet on S3, querying historical data with DuckDB or Athena.
- **Multiregion**: running Relay in US, EU, AP with tenant routing to nearest region, cross-region cost aggregation.


---

## Nobody Will Read This

**Date:** 2026-06-22
**URL:** http://localhost:1313/nobody-will-read-this/
**Description:** Writing documentation, comments, or even blog posts into the void, and doing it anyway.


There is a funny truth in software engineering: almost nobody reads the docs.

We spend hours writing careful comments in our code. We build personal websites. We write plain text files and push them to live servers. But if we check the stats, we know the truth. The audience is basically zero.

So why do we do it?

My day is full of loud noise. It is filled with heavy backend logic, endless tasks, and the crushing weight of non-stop engineering. The pressure is always on. The systems never sleep. It is exhausting.

But writing is different.

When I open a blank text file, the world gets quiet. There are no urgent alerts. Nobody is shouting. It is just me, a blinking cursor, and plain text.

I do not write these posts for an audience. I write them for me. Writing takes the messy, stressful noise in my head and turns it into clean, simple lines. Like this one.

It is a small thing I can actually control when the rest of the work feels too big and moves too fast. 
I have decided to take a break from reading or writing anything tech deepdive.

Nobody will read this. And honestly, that is exactly why it makes me happy.


---

## What is DRY? (And Other Things We Say at 3 AM)

**Date:** 2026-06-22
**URL:** http://localhost:1313/what-is-dry/
**Description:** DRY is a rule we all know. But at 3 AM with a crashing server, the copy-paste starts looking pretty reasonable.


There is a famous programmer joke that always gets me:

"What is DRY? Well, at the risk of repeating myself..."

It is funny, but it also hurts a little because it is so true. We all know the rule: Don't Repeat Yourself (DRY). We learn it from day one. You write a piece of code once, put it in a clean function, and never type it again.

But let's be honest. It is the middle of the night. You are staring at your screen, your eyes hurt from the light, and you just need the server to stop crashing. What really happens? You copy that block of code from another file. You paste it right where you need it. You change one variable, save the file, and quietly hope nobody looks too closely.

We talk about DRY online, but then we go to work and type the exact same error check fifty times a day just to keep the backend running. We try to be perfect and make everything reusable, but sometimes that just makes the code harder to read. Sometimes, the most practical thing you can do is just let the code repeat.

Building a perfectly clean system is a nice idea. But servers have real limits, and honestly, our brains do too when we are just trying to finish the job so we can finally sleep.

Sometimes you just have to accept the copy-paste.

Anyway, I will end this here... at the risk of repeating myself.


---

## So, We Are Writing Efficient Software Again

**Date:** 2026-06-17
**URL:** http://localhost:1313/so-we-are-writing-efficient-software-again/
**Description:** RAM is up 400%. CPUs cost 15% more. The free hardware lunch is finally over. Welcome back to the 80s, where every byte counts.


I wanted to upgrade the RAM in my PC.

Nothing fancy. I bought 32GB in 2024 and I wanted to double it. I opened a tab, checked the price, and closed the tab. Then I sat quietly for a moment.

The same 32GB kit I bought in 2024 now costs more than double. DDR5 prices have gone up roughly 400% since mid-2025 [1]. DDR4 is not much better. A kit that cost $60–$90 in late 2025 now sells for $150–$180 [2].

I did not upgrade my RAM. I opened my code editor instead.

---

## What happened

Two things happened at the same time.

**AI ate the memory market.** You know the old meme: "why is my PC slow?" "the cat ate my RAM." Funny because cats do not eat RAM. Except now the cat is a data center, and the RAM is real, and it is not funny anymore. Data centers need enormous amounts of memory to run inference workloads. Manufacturers are happy to sell high-margin server chips to hyperscalers. The consumer market gets whatever is left. Memory shortages are expected to last until at least Q4 2027 [3].

**Tariffs made everything worse.** On top of that, there are now 25% tariffs on many semiconductor imports [4]. At some point ASUS warned that laptops could cost 30% more [5]. I read that headline, closed it, and stared at my ceiling for a while.

The best joke in this whole situation is that the people responsible for the price increase are the same companies whose tools we use to write more code, faster, with less thought. We used AI to produce bloated software at scale, and now AI is pricing us out of the hardware to run it.

---

## The villains

Let us talk about what we built during the cheap hardware era.

Electron. The framework that takes a web app and wraps it in its own copy of Chrome. Discord, Slack, VS Code: all Electron apps. Each one ships with a browser runtime that uses hundreds of megabytes before you open a single document. Slack on startup can use 400MB or more. You are running a browser to run a chat app inside a browser.

`node_modules`. The folder that contains the internet. A fresh React project can have 300MB of dependencies before you write one line of application code. The `is-odd` package has 70 million weekly downloads. It is 6 lines long. It checks if a number is odd.

Spring Boot. A Java framework where the "Hello World" service takes 45 seconds to start and uses 512MB of heap. There are enterprise teams who have been waiting for their local dev server to start since 2019.

Nobody blamed the developers, because there was not much to blame them for. Hardware was cheap, abstraction genuinely helps, and shipping faster is a real goal. The incentives just did not reward caring about memory.

---

## Back to the future

Here is the funny part.

We are now back to thinking like programmers from the 1980s.

In 1984, you had 64KB of RAM. You counted every byte. You decided which data to keep in memory and which to load from disk because you had no real choice. Carmack's DOOM ran on a 386 with 4MB of RAM. The Apollo Guidance Computer had 4KB of RAM and landed humans on the moon. The demoscene still fits entire animated 3D worlds into 4KB executables, just because they enjoy the constraint.

Then RAM got cheap, CPUs got fast, and cloud servers became easy to rent by the hour. Writing inefficient software stopped having real consequences, so people mostly stopped thinking about it.

---

## A memo from 1984

**FROM:** A programmer who wrote a ray caster in 640KB  
**TO:** Modern software engineers  
**RE:** Your allocator

We have reviewed your code. A few notes.

Your struct has 7 bytes of padding between the boolean and the integer. This happens on every element of a slice with ten million entries. You are wasting 70MB of RAM for no reason. We could have fit a spreadsheet application in that space.

Your dependency tree has a package that checks if a string is empty. It has its own dependency. That dependency has a dependency. We checked: none of them check if the string is empty correctly.

Your Docker image is 2.1GB. Our entire operating system was 160KB.

We are not angry. We are just disappointed.

Regards,  
1984

---

So, we are writing efficient software again.

---

## What this means for code

If RAM is expensive, your code's memory usage is a cost.

Not a performance metric. An actual money cost. A machine that needs 64GB instead of 16GB costs significantly more to buy or rent. A service that allocates 200MB instead of 2GB runs on a smaller, cheaper instance. When you multiply that across a fleet, the difference is real budget.

A few things worth thinking about:

**Struct layout.** I wrote about this in [A Love Letter to the L1 Cache](../love-letter-to-the-l1-cache/). Compiler padding from bad field ordering can make your structs 50% larger than they need to be. On a tight memory budget, that inflates your working set directly. Run [`betteralign`](https://github.com/dkorunic/betteralign) on your Go codebase and see what it finds.

**Allocations.** Every allocation that is not necessary is now more expensive than it was. In Go, this means thinking about whether you should preallocate a slice, whether that intermediate `[]byte` is avoidable, and whether a sync.Pool makes sense for objects you create at high frequency.

**Dependencies.** Every library has a memory footprint. Pulling in a 10,000-line package for one function is harder to justify when RAM costs three times what it did in 2024.

---

## A small confession

I am not old enough to have written code under real memory constraints. I did not count bytes in 1984. I did not have to. By the time I started programming, RAM was cheap and getting cheaper, and nobody expected you to care.

But I have always liked writing code that does not waste things. Not because I had to, just because it feels better. A [loop that fits in cache](../love-letter-to-the-l1-cache/), a struct that has no padding it does not need, a service that uses 80MB instead of 800MB: these feel like better solutions, the same way a tight paragraph feels better than a loose one.

After I decided not to upgrade my RAM, I ran `betteralign` on my current project. It found 14 structs with suboptimal field ordering. I felt a little embarrassed, fixed them, and watched the working set of the main loop drop by about 18%. I did not buy any new hardware. I just stopped wasting the hardware I already have.

The programmer in the memo above would say "took you long enough". They would be right. But at least I got there.

---

## References

[1] [G.SKILL Addresses Sharp DDR5 RAM Price Increases](https://www.neowin.net/news/gskill-addresses-sharp-ddr5-ram-price-increases-for-2025-2026/) — Neowin

[2] [RAM Price Index 2026](https://www.tomshardware.com/pc-components/ram/ram-price-index-2026-lowest-price-on-ddr5-and-ddr4-memory-of-all-capacities) — Tom's Hardware

[3] [Memory Shortages To Last Till Q4 2027](https://wccftech.com/memory-ddr5-ddr4-shortages-last-till-q4-2027-higher-prices-throughout-2026/) — WCCFtech

[4] [Trump Tariffs to Hike PC Costs at Least 20%](https://www.techpowerup.com/335054/trump-tariffs-to-hike-pc-costs-at-least-20-system-integrators-take-the-biggest-blow) — TechPowerUp

[5] [Server CPU Prices Up as Much as 20% Since March](https://www.trendforce.com/news/2026/04/22/news-server-cpu-prices-up-as-much-as-20-since-march-intel-may-raise-prices-another-8-10-in-2h26/) — TrendForce

---

## Constructing Concurrent Inverted Indexes in Go

**Date:** 2026-06-16
**URL:** http://localhost:1313/constructing-concurrent-inverted-indexes-in-go/
**Description:** Building a thread-safe inverted index from scratch in Go. Covers sharded mutexes, lock contention profiling, slice pooling to avoid GC pressure, and benchmark comparisons against a naive sync.RWMutex approach under varying read/write ratios.


I spent a Saturday afternoon benchmarking a concurrent inverted index and discovered that a single `sync.RWMutex` starts to break down at roughly 4 concurrent readers. The degradation is not linear. It is not graceful. It is a cliff.

The inverted index is one of the oldest data structures in information retrieval. It maps terms to the documents that contain them, forming the backbone of every search engine from Elasticsearch to Lucene to Google's earliest prototypes. The data structure itself is simple. Making it fast under concurrent load is not.

This article walks through building a concurrent inverted index in Go. I start with a naive implementation, profile its lock contention, and progressively refine it with sharded mutexes, compressed posting lists, and allocation-aware index structures.

If you have ever wondered what happens under the hood when you type a query into a search box, or why some databases can handle 50,000 index writes per second while others stall at 5,000, this is for you.

---

## The inverted index: a five-minute primer

An inverted index takes a collection of documents and builds a mapping from each term to a list of document identifiers where that term appears. Given a query like "concurrent Go", the engine looks up "concurrent" and "Go" in the index, retrieves their posting lists, and intersects them to find documents containing both terms.

{{< code lang="go" >}}
// The core data structure, stripped to its essence
type InvertedIndex struct {
    mu    sync.RWMutex
    index map[string][]int // term → sorted list of document IDs
}

func (idx *InvertedIndex) Index(docID int, terms []string) {
    idx.mu.Lock()
    defer idx.mu.Unlock()
    for _, term := range terms {
        idx.index[term] = append(idx.index[term], docID)
    }
}

func (idx *InvertedIndex) Search(term string) []int {
    idx.mu.RLock()
    defer idx.mu.RUnlock()
    return idx.index[term]
}
{{< /code >}}

This works. It is correct. It is also catastrophically slow under any real workload.

The problem is the single `sync.RWMutex`. Every reader acquires a read lock. Every writer acquires a write lock. When a writer arrives, it blocks all new readers. When even a modest number of readers are active, writers starve. The mutex becomes a serialization point, and your concurrent index degrades into a sequential one.

---

## Measuring the damage: lock contention profiling

Before optimizing, we need to quantify the problem. I wrote a benchmark harness that simulates different read/write ratios against the naive index. The workload: 1 million terms across 100,000 documents, with goroutine counts varying from 1 to 64.

{{< code lang="go" >}}
func BenchmarkNaiveIndex(b *testing.B) {
    scenarios := []struct {
        name       string
        readRatio  float64 // fraction of operations that are reads
        goroutines int
    }{
        {"read-heavy-4", 0.95, 4},
        {"read-heavy-16", 0.95, 16},
        {"balanced-4", 0.50, 4},
        {"balanced-16", 0.50, 16},
        {"write-heavy-4", 0.05, 4},
        {"write-heavy-16", 0.05, 16},
    }

    for _, sc := range scenarios {
        b.Run(sc.name, func(b *testing.B) {
            idx := NewNaiveIndex()
            // pre-populate with 100k documents, 10 terms each
            for i := 0; i < 100_000; i++ {
                idx.Index(i, generateTerms(10))
            }

            b.ResetTimer()
            b.SetParallelism(sc.goroutines)
            b.RunParallel(func(pb *testing.PB) {
                rng := rand.New(rand.NewSource(time.Now().UnixNano()))
                for pb.Next() {
                    if rng.Float64() < sc.readRatio {
                        idx.Search(randomTerm(rng))
                    } else {
                        idx.Index(randomDocID(rng), generateTerms(5))
                    }
                }
            })
        })
    }
}
{{< /code >}}

The results, measured in operations per second on an M2 Pro with 12 cores:

| Scenario | Naive RWMutex (ops/sec) | Contention % |
| :--- | :--- | :--- |
| read-heavy-4 | 2,847,000 | 12% |
| read-heavy-16 | 1,210,000 | 58% |
| balanced-4 | 892,000 | 43% |
| balanced-16 | 412,000 | 76% |
| write-heavy-4 | 310,000 | 65% |
| write-heavy-16 | 94,000 | 91% |

At 16 goroutines with a balanced workload, 76% of CPU time is spent waiting on the mutex. The goroutines are not doing useful work. They are standing in line.

{{< details title="Reading Go Mutex Profiles" label="Diagnostics //" >}}
The Go runtime exposes mutex contention profiling through `runtime.SetMutexProfileFraction`. Enable it with a non-zero rate, then visualize with `go tool pprof`:

```bash
go test -bench=BenchmarkNaiveIndex -mutexprofile=mutex.out
go tool pprof -http=:8080 mutex.out
```

The flame graph will show `sync.(*RWMutex).Lock` dominating your CPU samples. This is the "contention %" column above. If this number exceeds 20%, your mutex is your bottleneck.
{{< /details >}}

---

## Sharded mutexes: divide and conquer

The core insight is that contention arises because all operations—reads and writes—funnel through a single lock, even when they target different terms. A query for "database" should not block an index update for "garbage-collector".

A sharded mutex distributes the lock across N independent partitions. Each partition guards a subset of the term space. The term is hashed to a shard, and only that shard's mutex is acquired.

{{< code lang="go" >}}
const numShards = 256

type ShardedIndex struct {
    shards [numShards]indexShard
}

type indexShard struct {
    mu    sync.RWMutex
    terms map[string]*PostingList
}

func (idx *ShardedIndex) shard(term string) *indexShard {
    h := fnv.New32a()
    h.Write([]byte(term))
    return &idx.shards[h.Sum32()%numShards]
}

func (idx *ShardedIndex) Index(docID int, terms []string) {
    // Group terms by shard to minimize lock acquisitions
    grouped := make(map[uint32][]string)
    for _, term := range terms {
        h := fnv.New32a()
        h.Write([]byte(term))
        shardID := h.Sum32() % numShards
        grouped[shardID] = append(grouped[shardID], term)
    }

    for shardID, shardTerms := range grouped {
        shard := &idx.shards[shardID]
        shard.mu.Lock()
        for _, term := range shardTerms {
            shard.terms[term].Add(docID)
        }
        shard.mu.Unlock()
    }
}
{{< /code >}}

Two details matter here. First, terms are grouped by shard before acquiring locks. Without this grouping, the function would acquire and release a new lock for every term in the document, doubling the lock/unlock overhead. Second, the hash function is FNV-32a, not a cryptographic hash. We need speed, not collision resistance. A handful of collisions across 256 shards is statistically irrelevant.

The results:

| Scenario | Naive RWMutex (ops/sec) | Sharded 256-way (ops/sec) | Improvement |
| :--- | :--- | :--- | :--- |
| read-heavy-4 | 2,847,000 | 4,120,000 | 1.45× |
| read-heavy-16 | 1,210,000 | 3,890,000 | 3.21× |
| balanced-4 | 892,000 | 1,740,000 | 1.95× |
| balanced-16 | 412,000 | 1,610,000 | 3.91× |
| write-heavy-4 | 310,000 | 520,000 | 1.68× |
| write-heavy-16 | 94,000 | 420,000 | 4.47× |

The improvement is most dramatic at high goroutine counts because sharding reduces the probability of any two operations landing on the same mutex. With 256 shards, the expected collision rate for 16 concurrent operations is roughly 6%, compared to 100% with a single mutex.

---

## How many shards? The Goldilocks problem

The number of shards is a tuning parameter with real trade-offs. Too few, and you retain contention. Too many, and you waste memory on mutex overhead and increase the cost of the hash-and-dispatch step.

I ran the balanced-16 benchmark against shard counts from 1 to 2048:

| Shards | Throughput (ops/sec) | Index Memory (MB) | Useful Work % |
| :--- | :--- | :--- | :--- |
| 1 | 412,000 | 48 | 24% |
| 8 | 1,140,000 | 49 | 52% |
| 32 | 1,480,000 | 50 | 68% |
| 128 | 1,620,000 | 52 | 82% |
| 256 | 1,610,000 | 54 | 86% |
| 512 | 1,590,000 | 58 | 87% |
| 1024 | 1,550,000 | 66 | 88% |
| 2048 | 1,490,000 | 82 | 89% |

Beyond 256 shards, throughput plateaus and then declines. The mutex overhead per shard (each `sync.RWMutex` is 24 bytes plus the map header) starts consuming measurable memory, and the hash-compute cost per operation overtakes the contention savings. 256 is the empirical sweet spot for this workload.

---

## The garbage collector is watching

Lock contention is not the only threat to throughput. The Go garbage collector is a concurrent, tri-color mark-and-sweep collector. It runs alongside your goroutines. Every heap allocation you make during indexing adds to its workload.

In our naive implementation, every call to `append` on a posting list potentially triggers a slice growth, which allocates a new backing array and copies the old one. Over millions of documents, this is a constant, grinding allocation pressure.

The fix is a posting list backed by a pre-allocated, growable buffer with amortized constant-time append. But even better: we can pool the individual posting entries.

{{< code lang="go" >}}
var postingPool = sync.Pool{
    New: func() interface{} {
        return &Posting{}
    },
}

type Posting struct {
    DocID     int
    TermFreq  int
    Positions []int
}

type PostingList struct {
    mu       sync.RWMutex
    postings []*Posting
}

func (pl *PostingList) Add(docID int, positions []int) {
    pl.mu.Lock()
    defer pl.mu.Unlock()

    p := postingPool.Get().(*Posting)
    p.DocID = docID
    p.TermFreq = len(positions)
    p.Positions = append(p.Positions[:0], positions...)
    pl.postings = append(pl.postings, p)
}
{{< /code >}}

{{< details title="sync.Pool Mechanics" label="Diagnostics //" >}}
`sync.Pool` is a per-P (logical processor) cache of objects. When you call `Get()`, the runtime checks the local P's pool first, falling back to other P's pools and finally to `New()`. When you call `Put()`, the object is returned to the local pool.

Critical caveat: objects in a `sync.Pool` can be garbage collected at any time. The pool is a cache, not a guarantee. Do not store state in pooled objects that cannot be reconstructed.

For the posting pool, this is fine. Every `Get()` call is followed by a full re-initialization of all fields.
{{< /details >}}

With `sync.Pool` in place, the allocation profile shifted:

| Metric | Without Pooling | With Pooling |
| :--- | :--- | :--- |
| Allocations per index op | 47 | 3 |
| Bytes allocated per op | 2,840 | 312 |
| GC pause p99 (μs) | 1,240 | 180 |
| GC CPU overhead | 18% | 4% |

The GC CPU overhead dropped from 18% to 4%. That is 14% of your CPU budget returned to actual indexing work. Pooling is not a micro-optimization. It is a structural decision that changes how your program interacts with the runtime.

---

## Compressed posting lists: Elias-Fano encoding

A posting list for a common term like "the" might contain millions of document IDs. Storing these as a raw `[]int` consumes 8 bytes per ID. On a corpus of 10 million documents, "the" alone would occupy 80 MB.

Compression matters. The standard approach in information retrieval is delta encoding followed by variable-length integer coding. Delta encoding stores the difference between consecutive document IDs (which are sorted). Since document IDs are monotonically increasing, the deltas are small positive integers.

Elias-Fano encoding is a quasi-succinct representation that stores a monotone sequence of N integers from a universe of size U in roughly N * (2 + log₂(U/N)) bits. For a posting list with an average gap of 64 between document IDs, Elias-Fano uses approximately 5-6 bits per integer, compared to 64 bits for raw `int`.

{{< code lang="go" >}}
type EliasFanoPostingList struct {
    lowerBits    uint8
    lowerBitsLen int
    lowBits      []uint64
    highBits     *roaring.Bitmap // from github.com/RoaringBitmap/roaring
    lastDocID    int
    length       int
}

func (pl *EliasFanoPostingList) Add(docID int) {
    delta := docID - pl.lastDocID

    // Lower bits: store the lower 'lowerBitsLen' bits of delta
    lowMask := (1 << pl.lowerBitsLen) - 1
    low := delta & lowMask
    pl.appendLowBits(low)

    // High bits: store (delta >> lowerBitsLen) as unary in a bitmap
    high := delta >> pl.lowerBitsLen
    pos := pl.highBits.GetCardinality()
    pl.highBits.Add(pos + high)

    pl.lastDocID = docID
    pl.length++
}
{{< /code >}}

The decoding path reconstructs the original document ID by reading the high-bit bitmap to find the position, extracting the lower bits at the corresponding index, and adding the previous document ID.

The space savings are substantial:

| Corpus Size | Raw int[] (MB) | Elias-Fano (MB) | Compression Ratio |
| :--- | :--- | :--- | :--- |
| 1M docs, avg 200 terms/doc | 1,600 | 380 | 4.2× |
| 10M docs, avg 200 terms/doc | 16,000 | 3,600 | 4.4× |

The trade-off is decode latency. Random access into an Elias-Fano list requires a rank operation on the high-bit bitmap, which is O(log N) with a rank-indexed roaring bitmap. For intersection-heavy query workloads, this cost is amortized across thousands of list accesses. For single-term lookups, a variable-byte encoded list (with O(1) sequential decode) may be faster.

---

## Putting it together: the production index

The final index combines sharded mutexes, pooled postings, and compressed posting lists into a single structure:

{{< code lang="go" >}}
type Index struct {
    shards     [256]indexShard
    docStore   *DocumentStore
    termDict   *TermDictionary
    stats      *IndexStats
    closeCh    chan struct{}
}

type indexShard struct {
    mu    sync.RWMutex
    terms map[string]*EliasFanoPostingList
}

func NewIndex() *Index {
    idx := &Index{
        docStore: NewDocumentStore(),
        termDict: NewTermDictionary(),
        stats:    NewIndexStats(),
        closeCh:  make(chan struct{}),
    }
    for i := range idx.shards {
        idx.shards[i].terms = make(map[string]*EliasFanoPostingList)
    }
    return idx
}

func (idx *Index) Index(doc *Document) error {
    idx.stats.IncrDocuments()

    // Tokenize with pipeline: lowercase → tokenize → filter stopwords → stem
    tokens := idx.tokenize(doc.Content)

    // Store document
    idx.docStore.Put(doc.ID, doc)

    // Group tokens by shard
    grouped := idx.groupByShard(tokens)

    // Index each shard's batch under that shard's lock
    for shardID, shardTokens := range grouped {
        shard := &idx.shards[shardID]
        shard.mu.Lock()
        for _, token := range shardTokens {
            pl, ok := shard.terms[token]
            if !ok {
                pl = NewEliasFanoPostingList()
                shard.terms[token] = pl
                idx.termDict.Add(token)
            }
            pl.Add(doc.ID)
        }
        shard.mu.Unlock()
    }

    return nil
}
{{< /code >}}

This is the shape of a production inverted index. It is correct under concurrent reads and writes. It compresses its posting lists. It pools its allocations. It can be extended with a background merge goroutine to periodically compact posting lists and a WAL (write-ahead log) for crash recovery.

---

## What we did not cover

This article focuses on the index structure and concurrency model. Several adjacent concerns are intentionally deferred:

- **Tokenization and analysis**: Lowercasing, stemming (Porter2), stop-word removal, and n-gram generation are critical for recall but are pipeline stages that plug into the index, not properties of the index itself.
- **Query execution**: Boolean AND/OR/NOT, phrase queries with position information, and relevance scoring (TF-IDF, BM25) operate on top of the index. The index provides raw posting lists. The query executor does the rest.
- **Persistence**: An in-memory index is fast but ephemeral. A production system needs a write-ahead log and periodic snapshots to disk.
- **Distributed indexing**: Sharding the index across multiple machines introduces consensus, replication, and partition-tolerant query routing.

Each of these will be the subject of a dedicated article in this series.

---

## The benchmark table

| Approach | Balanced-16 (ops/sec) | Memory (MB) | GC Overhead | Compression |
| :--- | :--- | :--- | :--- | :--- |
| Naive RWMutex | 412,000 | 48 | 18% | None |
| Sharded (256-way) | 1,610,000 | 54 | 14% | None |
| Sharded + Pooling | 1,720,000 | 52 | 4% | None |
| Sharded + Pooling + Elias-Fano | 1,680,000 | 18 | 4% | 4.2× |

The fully optimized index achieves a 4× throughput improvement over the naive implementation while using 63% less memory and 78% less GC CPU time. The slight throughput regression from Elias-Fano compression (1,720k → 1,680k) is the decode penalty. In practice, this is offset by the memory savings, which allow larger working sets to fit in the CPU cache.

---

## Further reading

- Manning, Raghavan, and Schütze. *Introduction to Information Retrieval*. Chapter 1 (Boolean Retrieval) and Chapter 5 (Index Compression).
- Sebastiano Vigna. "Quasi-succinct indices". *Proceedings of WSDM 2013*. The canonical paper on Elias-Fano for inverted indexes.
- Go Blog. "Go's http/2 server: Priority and flow control for a concurrent world". Demonstrates sharded synchronization patterns in the standard library.
- Daniel Lemire's blog: [Roaring Bitmaps](https://roaringbitmap.org/). The compressed bitmap library used in the high-bit component of Elias-Fano.

---

*The next article in this series is [Implementing BM25 Scoring From First Principles](/implementing-bm25-scoring-from-first-principles/). It builds on the index here, deriving BM25's term frequency saturation and length normalization from scratch, with benchmarks against TF-IDF.*


---

## A Love Letter to the L1 Cache

**Date:** 2026-06-15
**URL:** http://localhost:1313/love-letter-to-the-l1-cache/
**Description:** Why reordering two struct fields made my loop 30% faster, and what it taught me about how CPUs actually work.


I recently spent four hours staring at a benchmark that didn't make sense.

It started while working on [Derelict Facility](https://github.com/vikash-paf/derelict-facility), my grid-based game engine in Go. I was trying to tighten the main update loop, specifically the part that iterates over every entity on the map each frame. I pulled out a small benchmark to isolate the cost, and something looked wrong.

I had two Go structs. They held the exact same data: two booleans and a 64-bit integer. I was iterating over a slice of 10 million of these structs, doing a simple addition. They should have been identical in performance.

They weren't.

Struct A was consistently 30% slower than Struct B. Struct A was also 50% larger in memory.

In high-level languages, we are taught to think of memory as a simple, uniform resource. You ask for some bytes, you get them. But when you start caring about nanoseconds, you find out that memory is not uniform at all. Getting a value from RAM takes real time, and your CPU has no choice but to sit and wait for it.

This post is about that wait.

---

## The code

Here is what started it.

{{< code lang="go" >}}
type BadStruct struct {
    A bool  // 1 byte
    B int64 // 8 bytes
    C bool  // 1 byte
}

type GoodStruct struct {
    B int64 // 8 bytes
    A bool  // 1 byte
    C bool  // 1 byte
}
{{< /code >}}

If you use `unsafe.Sizeof()`, you will find that `BadStruct` occupies **24 bytes**, while `GoodStruct` only takes **16 bytes**.

Why? Because the CPU reads memory in fixed-size chunks called "words". On a 64-bit machine, a word is 8 bytes. To make reading fast, the compiler adds "padding" (empty bytes) to make sure that 8-byte values like `int64` start at an address that is a multiple of 8.

{{< details title="Padding and Alignment" label="Glossary //" >}}
**Alignment:** CPUs fetch data in fixed-size chunks. If an 8-byte integer starts at byte 3 instead of byte 8, the CPU may need two fetches to read it instead of one.

**Padding:** To avoid misalignment, the compiler inserts empty bytes between fields. In `BadStruct`, it adds 7 bytes after the first `bool` so the `int64` can start at byte 8. It then adds 7 more bytes at the end to keep the struct aligned for the next element in a slice.
{{< /details >}}

But the size difference is only half the story. The real reason `BadStruct` is slower is that it wastes cache space.

---

## Why RAM is slow

To understand why 8 bytes of padding matter, you need to know how the memory hierarchy works.

RAM is not fast. Relative to a CPU running at 4GHz, it is very slow. Here is a way to think about it: if one CPU cycle took one second, a round trip to main memory would take about four minutes.

To deal with this, CPUs have a hierarchy of caches. Each layer is smaller, faster, and closer to the processor.

{{< details title="SRAM vs DRAM" label="Glossary //" >}}
**SRAM (Static RAM):** Used for CPU caches (L1, L2, L3). It is fast because each bit stays set without any refreshing. It uses more transistors per bit, so it is expensive and takes more space on the chip.

**DRAM (Dynamic RAM):** Used for main memory (your 16GB or 32GB sticks). It is cheaper and denser, but each bit has to be constantly refreshed. This is what makes it slower.
{{< /details >}}

### The latency ladder
- **L1 Cache:** ~1 nanosecond. Sits directly on the CPU die. About 32–64KB per core.
- **L2 Cache:** ~4 nanoseconds. Larger, slightly further away.
- **L3 Cache:** ~10–15 nanoseconds. Shared across cores.
- **Main Memory (DRAM):** ~100 nanoseconds. A completely separate chip.

This is the only pyramid scheme where the returns are real.

When your CPU needs a value, it checks L1 first. If it is not there (a "cache miss"), it checks L2, then L3, then finally goes to main memory.

Each miss costs time your CPU spends doing nothing. It just waits. This is why the L1 cache matters so much. It is the only memory fast enough to keep the CPU busy.

---

## The secret life of a cache line

Here is the important part: **the CPU never fetches a single byte**.

When you ask for 1 byte, the CPU assumes you will probably need the bytes next to it soon. This is called **spatial locality**. So instead of fetching 1 byte, it fetches a **64-byte block** called a **cache line**.

{{< details title="Cache Line" label="Glossary //" >}}
**Cache Line:** The smallest unit of data transfer between the cache and main memory. Even if you only need 1 byte, the hardware fetches 64 bytes and stores the whole block in the cache.
{{< /details >}}

Here is what that looks like in practice with our two structs:

```
One 64-byte cache line:

BadStruct (24 bytes) — 2 fit, 16 bytes wasted:
┌────────────────────────┬────────────────────────┬─ ─ ─
│       entity 0         │       entity 1         │
└────────────────────────┴────────────────────────┴─ ─ ─
 0                      23 24                     47

GoodStruct (16 bytes) — 4 fit, 0 bytes wasted:
┌────────────────┬────────────────┬────────────────┬────────────────┐
│    entity 0    │    entity 1    │    entity 2    │    entity 3    │
└────────────────┴────────────────┴────────────────┴────────────────┘
 0              15 16            31 32            47 48            63
```

If your structs are small and tightly packed, one 64-byte cache line holds four of them. The CPU hits main memory once and has data for the next four loop iterations.

If your structs are bloated with padding, that same cache line holds only two. You hit main memory twice as often for the same amount of work.

---

## Why struct field order matters

Let's look at how our two structs actually sit in memory.

### BadStruct (24 bytes)
| Byte Offset | Field | Content |
| :--- | :--- | :--- |
| 0 | `A` | `bool` (1 byte) |
| 1–7 | – | **Padding** (7 bytes) |
| 8–15 | `B` | `int64` (8 bytes) |
| 16 | `C` | `bool` (1 byte) |
| 17–23 | – | **Padding** (7 bytes) |

### GoodStruct (16 bytes)
| Byte Offset | Field | Content |
| :--- | :--- | :--- |
| 0–7 | `B` | `int64` (8 bytes) |
| 8 | `A` | `bool` (1 byte) |
| 9 | `C` | `bool` (1 byte) |
| 10–15 | – | **Padding** (6 bytes) |

By moving the `int64` to the top, the two booleans sit together at the end. We cut the struct size by 33%.

In a tight loop over millions of elements, this means fitting more data into each cache line, which means fewer trips to main memory.

---

## The benchmark

I wrote a Go benchmark to confirm this. Two slices of 10,000,000 structs. Iterate over each and sum the integer field.

{{< code lang="go" >}}
func BenchmarkBadStruct(b *testing.B) {
    data := make([]BadStruct, 10_000_000)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        var sum int64
        for j := 0; j < len(data); j++ {
            sum += data[j].B
        }
    }
}

func BenchmarkGoodStruct(b *testing.B) {
    data := make([]GoodStruct, 10_000_000)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        var sum int64
        for j := 0; j < len(data); j++ {
            sum += data[j].B
        }
    }
}
{{< /code >}}

### Result (`benchstat`)
```text
name           time/op
BadStruct-16   12.4ms ± 2%
GoodStruct-16   8.7ms ± 1%
```

30% faster. Just from reordering fields.

The reason: `GoodStruct` fits roughly 4,000,000 elements into 64MB. `BadStruct` fits only 2,666,666. The bad version keeps evicting cache lines and waiting for main memory. The CPU has no work to do. It just waits.

If you want to catch this automatically, [`betteralign`](https://github.com/dkorunic/betteralign) is a Go linter that finds structs with suboptimal field ordering and suggests the fix. One pass over your codebase and it will find every case like this one.

---

## Hardware sympathy

Languages like Go, Java, and Python hide most of the hardware from you. That is a good thing. You can focus on the actual problem.

But the hiding breaks down at scale.

The CPU does not care about your abstractions. It only sees cache lines, bus cycles, and latency. When your working set stops fitting in cache, you pay the full 100ns penalty for every miss, millions of times per second.

"Hardware sympathy" is a term from the LMAX Disruptor project. The idea is simple: understand the physical machine well enough to write code that works with it, not against it. It does not mean writing assembly. It means knowing that a cache line is 64 bytes, and thinking about whether your data layout respects that.

Back in Derelict Facility, I went through the entity structs with fresh eyes. Most of them had the same problem: small boolean flags scattered between larger fields. After reordering, the update loop got noticeably faster without touching a single line of game logic.

It took roughly four hours to understand the problem and five minutes to fix it. Run `betteralign` on your own codebase and see what it finds.

---


---

## Why Explaining Technical Difficulty is Hard

**Date:** 2026-05-14
**URL:** http://localhost:1313/why-explaining-technical-difficulty-is-hard/
**Description:** "It's just a simple query". Why the distance between a logical requirement and its infrastructure cost is the most expensive gap in engineering.


"Can we just add a real-time visitor counter to the homepage? It's just a query, right"?

Every engineer has heard some variation of this. On the surface, the logic is sound. You have data, you have a UI, and you want to connect them. In the world of business requirements, this is a solved problem. You write a line of code, and the feature exists.

But your database doesn't care about business logic. It cares about **Infrastructure Constraints**.

While the request sounds like a simple query, the gap between the requirement and the execution is where the hidden costs live. You aren't just "showing a number"; you are introducing global lock contention on a high-traffic table, triggering `COUNT(*)` scans on 100M rows, and potentially pinning your database CPU at 100% during peak hours.

> "For every complex problem, there is a solution that is simple, neat, and wrong". - H.L. Mencken

![The Implementation Iceberg](implementation-iceberg.svg)

This is the **Implementation Gap**: the distance between the logical intent and the physical cost of execution.

## The logic vs. the stack

We often operate in two different modes:

### 1. The feature mode
This is where requirements are born. The goal is to move fast, reduce uncertainty, and get feedback. Here, code is just a means to an end. If it "works" on a local dev machine with 100 rows, the requirement is considered met.

### 2. The system mode
This is where the stack lives. This mode is governed by the physics of computation: memory latency, network overhead, and concurrency limits. In this mode, every line of code is an allocation, a network round-trip, or a lock that must be managed.

{{< details title="The Divergent Perspectives" label="Deep Dive //" >}}
**Feature mode** asks: "Does this fulfill the user story"?

**System mode** asks: "What does this do to the p99 latency of the entire gateway"?

The friction in engineering teams isn't about personality; it's about whether you're looking at the logic or the underlying hardware.
{{< /details >}}

## The model

After a few years of building and breaking things, you start to develop an understanding of your stack. This isn't about job titles; it's about **Tacit Knowledge**. It's the intuition you develop after watching a "quick fix" turn into a 2 AM outage.

You start to see the infrastructure behind the syntax. When you hear "real-time counter", you aren't thinking about the SQL; you're thinking about the lock contention on the index. You aren't being a "blocker"; you are observing the mechanical limits of your environment.

{{< alert type="important" >}}
**The Simulation Gap:** Modern tools like AI are excellent at **Syntax**. They can generate the logic for a feature in seconds. But they don't know your **Infrastructure**. They can't tell you that your query will fail because of your specific index strategy or your Redis eviction policy.
{{< /alert >}}

## Bridging the gap: the API of options

The most expensive mistake an engineer can make is trying to explain **Infrastructure Complexity** to someone who is focused on **Product Feedback**. 

If you explain "why it's hard" in technical terms, you sound like you're making excuses. It reminds me of the scene in *The Big Bang Theory* where Penny is trying to explain social conventions to Sheldon, and after listening to his pedantic reasoning, she realizes: "Now I know what I sound like to you when I say stupid stuff".

When we lecture a product manager on database internal locks, we are Penny. We are just saying "stupid stuff" that doesn't map to their reality. Instead, you should treat your expertise as a system interface that provides **Validated Options**.

Instead of a "No", offer a path that separates the experiment from the infrastructure.

- **The Request:** "We need a real-time visitor counter".
- **The Feature View:** "Let's just query the `visits` table directly".
- **The System View:** "That will kill the DB. We need a pre-aggregated counter service".
- **The Bridge:** "If we want to see if users care about visitor counts, let's hardcode a '5,000+ users today' badge for 48 hours. If it moves the needle, we'll invest the time to build the counter service properly next week".

{{< alert type="tip" >}}
This approach satisfies the need for speed without polluting your core architecture. It allows you to validate the business value before you pay the infrastructure price.
{{< /alert >}}

---

Engineering is the management of trade-offs between what the business wants and what the infrastructure can sustain. When a request feels like friction, it is usually just your model flagging a resource contention before it happens. Use that intuition to offer a low-fidelity path that validates the idea without pinning your database.



---

## Just About Go Time

**Date:** 2026-05-10
**URL:** http://localhost:1313/just-about-go-time/
**Description:** A breakdown of the absolute absurdity of human time, monotonic clocks in Go, and the one true way to store time across PostgreSQL, Couchbase, and mobile clients.


Time is an illusion. Or more accurately, time is a political consensus poorly masquerading as physics. 

![Time is relative, absolute, and a scam](time-theory-meme.jpg)

If you've ever seen Dylan Beattie's "Plain Text" [talk](https://www.youtube.com/watch?v=gd5uJ7Nlvvo), you know that humans have spent centuries making data storage as complicated as possible. But text encoding has nothing on time zones.

As engineers, we like to pretend that `time.Now()` returns an objective truth. It doesn't. It returns a snapshot of a highly contested, historically unstable set of political boundaries. In 2011, the island nation of Samoa decided they wanted to align their workweek with Australia rather than the United States. To do this, they didn't just change their clocks; they completely skipped Friday, December 30th. At 11:59 PM on Thursday, the clock ticked over, and it was suddenly Saturday.

A whole day, erased from existence because a prime minister signed a piece of paper. If your backend job ran a cron on December 30th in Samoa, it just... didn't happen.

Here is what I've learned about handling time without losing sleep over it.

## Go Time Package Mechanics

Go's `time` package is notorious for being incredibly robust, but deeply weird to newcomers.

### The 1 2 3 4 5 6 7 Reference Date

If you want to parse or format a date in Python or Java, you use a format string like `YYYY-MM-DD HH:mm:ss`. 

Go looked at that and said, "No, that's too abstract". Instead, Go uses a reference date. To format a date, you have to write out what the reference date would look like in your desired format.

The reference date is: **January 2nd, 3:04:05 PM, 2006, timezone -0700 (MST)**.

Why? Because if you write it out in the American format, it counts up sequentially:
`01/02 03:04:05 '06 -0700`

It is a dry engineering joke that has cursed millions of developers to Google "golang time format string" every single day of their careers.

```go
// Parsing an ISO 8601 string
t, err := time.Parse("2006-01-02T15:04:05Z07:00", "2026-05-15T14:30:00Z")

// Formatting to a custom layout
fmt.Println(t.Format("2006-01-02 15:04")) 
```

### Monotonic Clocks in Go

Before Go 1.9, measuring execution time was dangerous.

```go
start := time.Now()
doWork()
elapsed := time.Since(start)
```

In older versions of Go, `elapsed` could theoretically be a negative number. 

Why? Because `time.Now()` used the operating system's "wall clock". The wall clock is tied to NTP (Network Time Protocol). If the OS synched with an NTP server during `doWork()` and realized its clock was 2 seconds fast, it would violently jump the clock backward. Your start time would suddenly be in the future.

In 2017, Russ Cox proposed a transparent fix. Rather than creating a separate `MonotonicTime` type, Go simply jammed a monotonic clock reading inside the existing `time.Time` struct alongside the wall clock. 

If a `time.Time` struct has a monotonic clock reading, operations like `time.Since()` or `t.Sub(u)` automatically use the monotonic (always forward-moving) clock to measure duration, completely ignoring any NTP timezone jumps. It fixed a massive class of distributed system bugs without breaking a single line of backward compatibility.

### Testing Temporal Logic

Writing tests for code that depends on the current time is a common source of flaky CI/CD pipelines. If your logic checks if a token is expired using `time.Now()`, your test is coupled to the execution clock of the runner.

To solve this, treat time as a dependency. Never call `time.Now()` inside deep business logic. Instead, pass the time in as an argument or use a provider function.

```go
type Now func() time.Time

func IsExpired(expiry time.Time, now Now) bool {
    return now().After(expiry)
}
```

By passing a `Now` function, you can inject a fixed timestamp in your tests, ensuring your logic is deterministic and reproducible.

## Database Persistence Patterns

How you store time depends on your database. Here is what works.

### PostgreSQL timestamptz Storage

PostgreSQL provides the `TIMESTAMP WITH TIME ZONE` (or `timestamptz`) type. It is a common misconception that this type stores the timezone. 

When you send a timestamp with an offset to a `timestamptz` column, PostgreSQL performs a three-step operation:
1. It parses the incoming string and offset.
2. It converts the value to UTC.
3. It stores the UTC value as an 8-byte integer.

The offset is discarded. This conversion happens on the fly. While it is a session-level CPU cost rather than a storage bottleneck, it causes panic when a developer queries the database from a GUI tool set to local time, sees a different hour than what is on the server, and assumes the database is corrupted. The data is fine. Your DB tool is just "helping" you.

```sql
-- Checking the current session timezone
SHOW timezone;

-- Retrieving UTC data formatted for a specific zone
SELECT created_at AT TIME ZONE 'America/New_York' FROM transactions;
```

### Couchbase and NoSQL UTCMilli Persistence

In a NoSQL JSON document store like Couchbase, you should prioritize **Millisecond Unix Epochs (UTCMilli)** for database storage.

```json
{
  "created_at": 1778857200000
}
```

Storing time as a 64-bit integer is the most accurate way to persist temporal data. It is inherently normalized to UTC, avoids the overhead of string parsing, and prevents timezone ambiguity. Because it is a numeric type, it supports high-performance range scans and sorting in SQL++ without the need for complex casting.

Treat ISO 8601 strings as a serialization format for transit, not storage. If you need human readability during a 3 AM incident response, use the `MILLIS_TO_STR()` function in your query. The minor benefit of raw readability in a JSON file is never worth the loss of numeric precision.

## The Handshake Boundary

While the database wants integers, the network wants strings. The absolute rule for the client-server boundary is a two-step handshake:

1. **Transport:** The backend sends an **ISO 8601 String** ending in "Z".
2. **Presentation:** The client converts that UTC string into **Local Time**.

Never perform localization on the server. A mobile device can change timezones while a request is in flight. The backend should provide the raw UTC data, and the client application must use the local operating system APIs to format the string for the user.

### Browser JavaScript Formatting

JavaScript's `Date` object is notoriously flawed, but modern browsers have finally stabilized `Intl.DateTimeFormat`. You pass it the UTC string, and the browser automatically detects the user's OS timezone and formats it correctly.

```javascript
const utcString = "2026-05-15T14:30:00Z";
const date = new Date(utcString);

const formatter = new Intl.DateTimeFormat('default', {
  year: 'numeric', month: 'short', day: 'numeric',
  hour: 'numeric', minute: '2-digit'
});

console.log(formatter.format(date)); 
```

### Mobile OS Localization

Mobile OS libraries are specifically designed to handle timezone changes on the fly. 

On iOS (Swift), parse the string using `ISO8601DateFormatter` and format it with `DateFormatter`. The OS will automatically pick up the user's current locale and timezone.

```swift
let formatter = ISO8601DateFormatter()
let date = formatter.date(from: "2026-05-15T15:00:00Z")!

let localFormatter = DateFormatter()
localFormatter.dateStyle = .medium
localFormatter.timeStyle = .short
// Let the iPhone figure out if they just flew to Samoa
print(localFormatter.string(from: date)) 
```

Android (Kotlin) uses the `java.time` package (`Instant`). You parse the UTC moment and convert it to the system's default zone for display.

```kotlin
val instant = Instant.parse("2026-05-15T15:00:00Z")
val localDateTime = ZonedDateTime.ofInstant(instant, ZoneId.systemDefault())

val formatter = DateTimeFormatter.ofLocalizedDateTime(FormatStyle.MEDIUM)
println(localDateTime.format(formatter))
```

These libraries are updated by the OS vendors whenever political boundaries or DST rules change, shielding your application from maintenance overhead.

## The short version

Measure duration with monotonic clocks. Store time as UTCMilli integers in NoSQL, `timestamptz` in Postgres. Send ISO 8601 strings over the wire. Let the client handle formatting.

Do that consistently and your system stays correct even when governments decide to delete a Friday.

## Further Reading
- [The Amazing "Plain Text" Talk by Dylan Beattie](https://www.youtube.com/watch?v=gd5uJ7Nlvvo)
- [Go Official: Time Package Documentation](https://pkg.go.dev/time)
- [Russ Cox: Monotonic Elapsed Time Measurements in Go](https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md)
- [PostgreSQL Official: Date/Time Types (timestamptz)](https://www.postgresql.org/docs/current/datatype-datetime.html)
- [Couchbase Official: SQL++ Date Functions](https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/datefun.html)


---

## Building a Poor Document Store inside PostgreSQL

**Date:** 2026-05-09
**URL:** http://localhost:1313/building-a-poor-document-store-inside-postgresql/
**Description:** A forensic analysis of why abusing JSONB for schemaless architecture leads to write amplification, TOAST bloat, and catastrophic query planner failures.


The marketing for "schemaless" architecture was brilliant. It promised speed, agility, and a life free from the tyranny of `ALTER TABLE` migrations. When PostgreSQL introduced `JSONB` in version 9.4, many developers saw it as a green light to treat Postgres like MongoDB. 

I’ve seen this pattern in dozens of codebases. It starts with a single `metadata` column, but within months, the entire business logic is buried inside a 2MB JSON blob. 

This isn't agility; it’s an architectural debt trap. By bypassing the relational schema, you aren't just losing type safety; you are fighting the physical storage engine of PostgreSQL.

![PostgreSQL JSONB Performance Failure](jsonb-anti-pattern-header.svg)

## The Cost of Write Amplification

PostgreSQL was designed to handle rows. When you update a row, Postgres doesn't change the data in place. Instead, it uses a mechanism called MVCC to create a new version of that row.

{{< details "What is MVCC?" >}}
**Multi-Version Concurrency Control (MVCC)** is how Postgres handles multiple people reading and writing at the same time without locking the whole table. Instead of overwriting data, Postgres keeps the old version (for people still reading it) and writes a brand new version of the row for the new data. Later, a background process called **VACUUM** cleans up the old "dead" rows.
{{< /details >}}

If you have a normalized table with 50 columns, and you update a single `boolean` flag, Postgres writes a few bytes to a new row version. 

But if you store your data in a massive `JSONB` document, Postgres has no idea how to "partially" update it. If your document is 1MB and you update a single key, Postgres must write the entire 1MB document to a new location on disk. 

## Disk Bloat from TOAST

Postgres stores data in 8KB pages. If a row exceeds that size, Postgres moves the large data into a separate storage area called TOAST.

{{< details "What is TOAST?" >}}
**TOAST (The Oversized-Attribute Storage Technique)** is the "trunk" of the database. When a row is too big to fit in a standard 8KB "closet" (a page), Postgres breaks the large parts into chunks and puts them in the TOAST table. It leaves a small pointer in the original row so it can find the data later. 
{{< /details >}}

When you abuse `JSONB` to store large documents, almost every row ends up in TOAST. Every time you update a small field inside that JSON, Postgres has to rewrite the entire TOAST entry. 

I ran a quick test to see the impact. I updated a single flag in a 2MB JSON document 1,000 times.

```sql
-- Checking the physical size of the TOAST table
SELECT relname, relpages 
FROM pg_class 
WHERE oid = (SELECT reltoastrelid FROM pg_class WHERE relname = 'users_jsonb');
```

The result was 2GB of disk bloat for a single row’s updates. In a normalized table, that same operation would have cost less than 100KB. You are literally burning disk I/O to maintain a "flexible" schema. Quite literally, your disk is getting TOASTed.

## Query Planner Blindness

The PostgreSQL query planner is an incredible piece of engineering. It uses statistics to decide whether to use an index or scan the whole table. It knows how many `NULLs` are in a column and the distribution of values.

But the planner is blind to what is inside your `JSONB` blob. 

If you query a standard column, the planner knows exactly how many rows to expect. If you query a key inside JSON, it has to guess. Usually, it guesses a very small number (like 10 rows). 

If the planner thinks only 10 rows match, it might choose a "Nested Loop" join, which is fast for 10 rows but catastrophic for 100,000 rows. I’ve seen queries that should take 5ms balloon to 30 seconds because the planner made a bad guess based on a `JSONB` blind spot.

## The Cost of GIN Indexes

To make JSON queries fast, you usually end up using a **GIN (Generalized Inverted Index)**. 

GIN indexes are powerful, but they are heavy. Every time you insert a JSON document, Postgres has to break the entire document apart, find every key and value, and update the index. 

In my benchmarks, inserting into a table with a GIN index is **5x to 10x slower** than a standard B-Tree index. You've traded your write performance for the convenience of not defining a schema.

## Valid Use Cases for JSONB

After heavily trashing it, I should clarify: `JSONB` is an incredible tool when used for its intended purpose. You should absolutely use it for:

1. **Third-Party Webhooks:** When a service like Stripe or GitHub sends you a webhook, you don't control the schema. Storing the raw payload in a `JSONB` column ensures you never drop data if they add a new field.
2. **User-Defined Custom Fields:** If you are building a SaaS where users can define their own attributes for a CRM contact, a sparse `JSONB` column is vastly superior to creating a table with 100 `custom_field_x` columns.
3. **Historical Snapshots:** Storing a frozen snapshot of an order receipt or a configuration state at a specific point in time, where you need to guarantee the shape of the data won't change even if your active relational tables evolve.

## Fixing Slow JSON Queries

If you have a valid `JSONB` payload but find yourself frequently filtering or joining on a specific key inside of it, you don't have to choose between flexibility and performance. You can use **Generated Columns** to bridge the gap:

```sql
ALTER TABLE users 
ADD COLUMN external_id TEXT 
GENERATED ALWAYS AS (payload->>'id') STORED;

CREATE INDEX idx_external_id ON users(external_id);
```

This gives you the best of both worlds: the raw payload stays safely in your JSON blob, but the query planner gets a physical column with real statistics to work with. 

Don't let the allure of "agility" trick you into building a second-rate document store inside the world's most advanced relational database. Define your core schema, and use `JSONB` only for the margins. Your disk, your CPU, and your future self will thank you.

## Further Reading
- [PostgreSQL Official: JSON Types](https://www.postgresql.org/docs/current/datatype-json.html)
- [PostgreSQL Official: JSONB Indexing](https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING)
- [PostgreSQL Official: Generated Columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html)
- [PostgreSQL Official: TOAST Storage](https://www.postgresql.org/docs/current/storage-toast.html)
- [PostgreSQL Official: Concurrency Control (MVCC)](https://www.postgresql.org/docs/current/mvcc.html)


---

## PostgreSQL Migrations in Go: Production Schema Patterns with Goose

**Date:** 2026-05-08
**URL:** http://localhost:1313/postgresql-migrations-in-go-production-schema-patterns-with-goose/
**Description:** Execute transactional SQL migrations in Go using Goose. Covers lock avoidance, rollback safety, version tracking, and embedding migration SQL in Go binaries.


Application code is stateless. You can tear down a container and spin up a new one in milliseconds without losing data. Databases are stateful. When you deploy new application logic that requires a new column, an index, or a table, you must transition the physical storage schema from state A to state B without destroying the underlying data or locking the system.

This process is a database migration. Working with databases is the kind of thing I will gladly skip dinner for (:

```mermaid
graph LR
    A[State A: Initial Schema] -->|Migration 001| B[State B: Tables Created]
    B -->|Migration 002| C[State C: Indexes Added]
    C -->|Rollback 002| B
    class A state-initial
    class C state-final
```

![Goose Migration Terminal Status](goose-migration-header.svg)

## The Road to Raw SQL

My obsession with relational databases started in 2018 while building my first Django app in college. Django’s ORM is a masterpiece of abstraction, and it was through its lens that I first learned about ACID properties, Normal Forms, and the uncompromising logic of foreign keys. 

In the years since, I’ve worked across the spectrum, including production environments and toy projects alike, using .NET, PHP, Python, and Node. Each ecosystem has its own way of shielding the developer from the database. Each tries to pretend that the disk doesn't exist.

But as I’ve moved deeper into Go and PostgreSQL, my preference has shifted. I no longer want the abstraction; I want the raw SQL. I want to see the DDL. I want to own the migration. This isn't just about the language. I just want to know exactly what is happening to the data.

Executing migrations manually via SQL clients is an operational liability. It lacks version control, it cannot be integrated into CI/CD pipelines, and it is prone to human error. Migrations must be treated as code: versioned, reviewed, and executed deterministically.

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant Git as Version Control
    participant CI as CI/CD Pipeline
    participant DB as Production Database

    Dev->>Git: Push SQL Migration
    Git->>CI: Trigger Build
    CI->>CI: Run Tests/Lint
    CI->>DB: Apply Migration (Goose)
    DB-->>CI: Success/Version Updated
    CI-->>Dev: Deployment Successful
```

This blog details the mechanics of database migrations and provides a strict implementation guide using PostgreSQL for storage, Go for execution, and Goose as the migration engine.

## The Mechanics of Schema State

A migration engine requires three components to operate safely:
1. **Immutable Migration Files**: Ordered scripts containing the Data Definition Language (DDL) required to transition the schema.
2. **Directionality**: Explicit `Up` (apply) and `Down` (rollback) logic.
3. **State Tracking**: An internal table within the target database recording which migration files have been successfully executed.

When the migration engine runs, it compares the local directory of migration files against the internal state table. It computes the delta and applies the pending `Up` scripts sequentially.

```mermaid
flowchart TD
    subgraph Local Files
        F1[001_init.sql]
        F2[002_add_index.sql]
        F3[003_add_column.sql]
    end
    
    subgraph DB State Table
        S1[001_init]
        S2[002_add_index]
    end

    F1 --- S1
    F2 --- S2
    F3 -->|Delta Found| Engine{Goose Engine}
    Engine -->|Apply| DB[(PostgreSQL)]
    DB -->|Update| S3[003_add_column]
```

## The Stack: PostgreSQL, Go, and Goose

The Go ecosystem has several migration tools. Goose is my favorite choice because it treats SQL as the primary interface. Unlike ORM-based migration tools that abstract DDL into application code, Goose uses raw `.sql` files. This lets you use PostgreSQL-specific features (like concurrent index creation or custom data types) that abstraction layers often block.

Goose also supports `go:embed`, allowing you to compile your SQL migration files directly into your Go binary. This produces a single, self-contained executable that can self-migrate before starting the HTTP server.

```mermaid
graph TD
    subgraph Compile Time
        SQL[.sql files] --> Embed[go:embed]
        Embed --> Bin[Binary Executable]
    end
    
    subgraph Deployment
        Bin -->|1. Migrate| DB[(Postgres)]
        Bin -->|2. Start| Srv[HTTP Server]
    end
```

## How to Implement

We will follow a strict implementation workflow to build the migration binary.

### 1. Project Architecture

Create the directory structure for the migrations. We keep the SQL files isolated from the application logic.

```bash
mkdir -p db/migrations
```

### 2. Defining the SQL Migrations

Goose uses a specific comment syntax to separate `Up` and `Down` migrations within a single file. 

Create the initial schema in `db/migrations/001_initialize_users.sql`.

```sql
-- +goose Up
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- +goose Down
DROP TABLE users;
```

### 3. The Zero-Downtime Migration Pattern

Standard DDL commands in PostgreSQL, such as `ALTER TABLE` or `CREATE INDEX`, acquire an `AccessExclusiveLock`. This lock blocks all read and write operations on the table until the migration completes. On a table with millions of rows, this causes an immediate production outage.

```mermaid
stateDiagram-v2
    [*] --> Idle
    Idle --> MigrationStarted: Run ALTER TABLE
    state MigrationStarted {
        [*] --> LockAcquired: AccessExclusiveLock
        LockAcquired --> WriteBlocked: App Writes Waiting...
        WriteBlocked --> ReadBlocked: App Reads Waiting...
        ReadBlocked --> Outage: Queues Full / Timeouts
    }
    Outage --> Idle: Migration Finished / Lock Released
```

To bypass this, PostgreSQL provides `CREATE INDEX CONCURRENTLY`. However, this command cannot be run inside a transaction block. Goose wraps migrations in transactions by default. We must use the `-- +goose NO TRANSACTION` directive to execute zero-downtime migrations safely.

Create `db/migrations/002_add_status_index.sql`.

```sql
-- +goose NO TRANSACTION
-- +goose Up
CREATE INDEX CONCURRENTLY idx_users_created_at ON users(created_at);

-- +goose Down
DROP INDEX CONCURRENTLY IF EXISTS idx_users_created_at;
```

### 4. Go Implementation

We will now implement the Go code to execute these migrations. Following the staged implementation workflow, we isolate the migration logic into its own package. 

```go
package database

import (
	"database/sql"
	"embed"
	"fmt"

	"github.com/pressly/goose/v3"
	_ "github.com/jackc/pgx/v5/stdlib"
)

//go:embed migrations/*.sql
var embedMigrations embed.FS

func MigrateDatabase(dbUrl string) error {
	db, err := sql.Open("pgx", dbUrl)
	if err != nil {
		return fmt.Errorf("failed to open db: %w", err)
	}
	defer db.Close()

	goose.SetBaseFS(embedMigrations)

	if err := goose.SetDialect("postgres"); err != nil {
		return fmt.Errorf("failed to set dialect: %w", err)
	}

	if err := goose.Up(db, "migrations"); err != nil {
		return fmt.Errorf("migration failed: %w", err)
	}

	return nil
}
```

### 5. Execution and State Verification

When `MigrateDatabase` is called from your `main.go`, Goose will connect to PostgreSQL and execute the pending `.sql` files.

You can verify the state by querying the internal tracking table created by Goose.

```sql
SELECT id, version_id, is_applied, tstamp 
FROM goose_db_version 
ORDER BY id DESC;
```

The output will confirm the exact timestamp each migration was applied. The database state is now deterministic, version-controlled, and mechanically linked to the application binary.

### 6. Local Development with the Goose CLI

While embedding migrations in the Go binary is perfect for production deployments, you need the Goose CLI for day-to-day local development. 

Install the CLI via Go:

```bash
go install github.com/pressly/goose/v3/cmd/goose@latest
```

To avoid typing the driver and connection string every time, I export them in my `.env` or `.envrc` file:

```bash
export GOOSE_DRIVER=postgres
export GOOSE_DBSTRING="postgres://user:pass@localhost:5432/app?sslmode=disable"
export GOOSE_MIGRATION_DIR="db/migrations"
```

Once those are set, these are the core commands you will use:

**Creating a new migration file:**
This generates a timestamped `.sql` file with the correct `Up` and `Down` annotations.
```bash
goose create add_users_table sql
```

**Checking migration status:**
See exactly which migrations are pending and which have been applied to your local database.
```bash
goose status
```

**Applying pending migrations:**
Run all unapplied `Up` scripts sequentially.
```bash
goose up
```

**Rolling back a migration:**
Run the `Down` script for the most recently applied migration. If you break your local schema, this is how you step backwards safely.
```bash
goose down
```

### 7. Automating in CI/CD

Running `goose up` from your laptop against the production database is a rite of passage, but it is also how you take down the company. Migrations belong in your deployment pipeline.

Because Goose relies on standard SQL files and a single binary, integrating it into CI/CD is just a matter of running the CLI against your production database *before* you deploy the new application code.

Here is what that looks like in a standard GitHub Actions workflow:

```yaml
name: Production Migrations
on:
  push:
    branches:
      - main
jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        
      - name: Install Goose
        run: curl -fsSL https://raw.githubusercontent.com/pressly/goose/master/install.sh | sh
        
      - name: Execute Migrations
        env:
          GOOSE_DRIVER: postgres
          GOOSE_DBSTRING: ${{ secrets.PROD_DB_URL }}
        run: goose -dir db/migrations up
```

For the legacy enterprise folks still riding the Jenkins train, the mechanics are exactly the same. You just drop the `curl` install and the `goose up` command into a `sh` block inside your `Jenkinsfile` deployment stage. 

By keeping this in CI/CD, the pipeline becomes the only entity with the credentials and authority to alter the production schema. It is deterministic, auditable, and entirely mechanical.


---

## Migrating Cloudflare to Terraform

**Date:** 2026-04-28
**URL:** http://localhost:1313/cloudflare-management-with-terraform/
**Description:** A technical reference on migrating existing Cloudflare infrastructure to Terraform. Includes an automated bootstrapping script to bypass manual state imports and handle API edge cases.


A while back, I was tinkering in the Cloudflare dashboard and accidentally fat-fingered a DNS configuration. I didn't realize the impact immediately, but I ended up taking down `lorbic.com` for an hour. 

When I scrambled to fix it, I hit a wall: there was no "undo" button. There was no Git history to tell me what the record *used* to point to, and no review to catch the mistake before I blew it. 

That was the day I realized that managing infrastructure by clicking around a UI, "Click-Ops" is a massive liability. And why everyone screams IaC, Terraform, Cloudformation etc. Moving to Infrastructure as Code (IaC) with Terraform was the obvious fix. But migrating an *existing* Cloudflare account with dozens of records and rules isn't as simple as just writing the code. 

![Terraform Apply Cloudflare DNS](terraform-cloudflare-header.svg)

## The 60-Second Terraform Crash Course

If you are new to DevOps, you only need to understand three core concepts to see why migrating existing infrastructure is notoriously difficult:

1. **The Code (`.tf` files):** This is where you declare your desired infrastructure (e.g., "I want a CNAME record pointing to my server").
2. **The Provider:** The plugin that translates your code into Cloudflare API requests. They are available for almost every cloud provider on planet Earth and beyond (maybe!).
3. **The State (`.tfstate`):** A JSON file where Terraform maps your code to the actual resources that *already exist* in the real world.

That third point is the bottleneck. If your `.tfstate` file is empty, Terraform assumes your Cloudflare account is empty. If you write your configuration and run `terraform apply`, Terraform tries to create brand new records. The Cloudflare API will immediately reject the request because those records already exist.

To fix this, you must explicitly link your code to your live infrastructure using `terraform import`. For a few dozen records across multiple domains, doing this by hand is a soul-crushing exercise.

Here is how I automated the entire generation and state import process.

## The Target Architecture

Dumping hundreds of DNS records into a single `main.tf` file creates an unreadable configuration. A better approach is to use Terraform modules—one for the account-level resources (like Zero Trust), and one for each specific domain (zone).

The goal is a directory structure that looks like this:

```text
cloudflare/
├── main.tf                # Provider config and module definitions
├── terraform.tfstate      # The single state file
├── account/               # Account-level config
│   ├── access_apps.tf
│   └── access_policies.tf
└── zones/                 
    ├── lorbic.com/        # Zone-specific config
    │   ├── dns.tf
    │   └── rules.tf
    └── vikashpatel.net/
        ├── dns.tf
        └── rules.tf
```

The root `main.tf` stays clean, simply tying the modules together:

```hcl
# cloudflare/main.tf
module "lorbic_com" {
  source = "./zones/lorbic.com"
  providers = { cloudflare = cloudflare }
}

module "vikashpatel_net" {
  source = "./zones/vikashpatel.net"
  providers = { cloudflare = cloudflare }
}
```

## The Automation Script

Cloudflare maintains a CLI tool called [`cf-terraforming`](https://github.com/cloudflare/cf-terraforming). It queries your account and generates both the `.tf` configuration files and the shell commands required to import the state.

However, `cf-terraforming import` only *outputs* the raw `terraform import` commands. It does not execute them, and more importantly, it assumes a flat directory structure. Because we are using a modular architecture, the resource addresses must be prefixed with the module name.

Instead of running these manually, I wrote a bash script to loop through an array of my domains, generate the `.tf` files, parse the import output, remap the addresses to the correct modules, and execute the imports automatically.

Here is the core parsing logic from the script:

```bash
# 1. Generate the Terraform config
cf-terraforming generate \
  --zone $ZONE_ID \
  --resource-type cloudflare_record > "zones/${DOMAIN}/dns.tf"

# 2. Generate the import commands, parse them, and execute
cf-terraforming import \
  --zone $ZONE_ID \
  --resource-type cloudflare_record | while IFS= read -r line; do
    
  if [[ "$line" == terraform\ import\ * ]]; then
    # Parse the raw output: "terraform import cloudflare_record.name <id>"
    resource_addr=$(echo "$line" | awk '{print $3}')
    import_id=$(echo "$line" | awk '{print $4}')

    # Prefix with the module name (e.g., module.vikashpatel_net.cloudflare_record.name)
    module_addr="module.${MODULE_NAME}.${resource_addr}"

    # Execute the import
    echo "Importing: $module_addr"
    terraform import "$module_addr" "$import_id"
  fi
done
```

By wrapping this in a loop, you can bootstrap an entire account in seconds. Once the script finishes, running `terraform plan` should cleanly report that your infrastructure matches your configuration.

## Edge Cases and API Limitations

Automated generation gets you 95% of the way there. During the migration, you will likely hit API quirks that require manual reconciliation.

### 1. The Read-Only Record Trap
Cloudflare manages certain DNS records automatically, such as Email Routing MX records or Zero Trust Access IPv6 CNAMEs. 

If `cf-terraforming` exports these into your `.tf` files, your next `terraform apply` will fail with **API Error 1046** or **1043**. Terraform cannot assume ownership of records that Cloudflare provisions internally. 

To fix it, you must manually delete these specific resource blocks from your generated `.tf` files.

### 2. Ruleset Index Conflicts
Cloudflare restricts accounts to a single "URL Normalization" ruleset per zone. Occasionally, the generation tool gets confused by naming conflicts, generating `cloudflare_ruleset.transform_1` alongside `cloudflare_ruleset.transform_2`. 

Attempting to apply this will result in an API rejection. You will need to manually reconcile these conflicts by deleting the duplicate block and using `terraform state rm <resource_address>` to clear it from the state file.

### 3. String Escaping and Normalization
Terraform is strict regarding state representation. Cloudflare's API might return a TXT record with explicit quotes (`"v=spf1 -all"`), but the generated HCL might omit them or escape them differently. 

During your first `terraform plan`, you may see dozens of `~ update in-place` changes. This is Terraform normalizing the formatting. Review the diff carefully to ensure the actual values aren't mutating, then apply it to sync the API with your local state format.

## The Day-to-Day Workflow

Post-migration, the Cloudflare dashboard becomes strictly read-only. Manual UI changes will be overwritten the next time Terraform runs.

The new workflow relies entirely on the CLI:

```bash
# Verify changes against live infrastructure
terraform -chdir=cloudflare plan

# Apply changes defined in .tf files
terraform -chdir=cloudflare apply
```

Migrating requires some upfront effort to handle API errors and organize the module structure. In exchange, the infrastructure becomes version-controlled, auditable, and predictably reproducible. You'll never have to guess what a deleted DNS record used to look like again. 

And honestly, having your entire DNS infrastructure version-controlled is incredibly satisfying to work with. 



---

## Internet Graveyard

**Date:** 2026-04-27
**URL:** http://localhost:1313/internet-graveyard/
**Description:** An archive of sunsetted technologies, derelict architectures, and dead protocols. We learn as much from failure as we do from success.


{{< graveyard_archive >}}

---

## Critical Rendering Path Optimization: 8 Proven Strategies to Boost Web Performance

**Date:** 2026-04-11
**URL:** http://localhost:1313/critical-rendering-path-optimization-8-proven-strategies-to-boost-web-performance/
**Description:** Master critical rendering path optimization. Learn 8 proven strategies to reduce First Contentful Paint, improve Lighthouse scores, and boost Core Web Vitals.


**TL;DR:** The Critical Rendering Path (CRP) is how browsers convert code into pixels. Optimizing it means reducing bottlenecks at five stages: Network, Parsing, Tree Building, Layout, and Paint. Use this guide to reduce First Contentful Paint by 40-60%, improve Lighthouse scores, and master the eight optimization strategies that separate fast sites from slow ones.

To master web performance, stop guessing and start engineering. View the browser as a single-threaded virtual machine that must convert raw text into pixels 60 times per second. Every performance delay is a bottleneck in the [**Critical Rendering Path (CRP)**](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path).

This is the forensic anatomy of how browsers render pages—and how to optimize it.

---

## The Macro Pipeline

The journey from URL to pixels happens in five stages:

```text
 1. Bytes ──> 2. Parsing ──> 3. Tree Building ──> 4. Render Engine ──> 5. Pixels
[Network]      [HTML/CSS]     [DOM & CSSOM]         [Layout/Paint]       [GPU]
```

---

## Fetching Bytes

The browser must fetch data before rendering pixels. Standard TCP/TLS handshakes occur, but the primary metric is [**TTFB (Time to First Byte)**](https://web.dev/articles/ttfb). The browser sits idle until the first HTML bytes arrive. Once the stream begins, the **Main Thread** wakes up.

## Building Object Models

Browsers do not understand HTML or CSS. They understand object models. The engine must parse raw bytes into formats it can manipulate.

### Document Object Model

As HTML streams in, the parser converts bytes into tokens, then nodes, and finally the [**DOM tree**](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path#dom).

```text
Bytes: 3C 62 6F 64 79 3E 48 69...
           ↓
Characters: <body>Hi...
           ↓
Nodes:    HTMLBodyElement, TextNode
           ↓
DOM TREE:
          html
         /    \
      head    body
       |       |
     title   p ("Hi")
```

The browser builds the DOM **incrementally**, processing nodes as they arrive.

### CSS Object Model

Encountering a `<link rel="stylesheet">` triggers a CSS request. Unlike HTML, CSS **cannot** be parsed incrementally. A rule at the bottom can override one at the top, so the engine must parse the entire file before proceeding.

This makes CSS **render-blocking**. The browser refuses to paint a half-styled page to prevent [**Flash of Unstyled Content (FOUC)**](https://en.wikipedia.org/wiki/Flash_of_unstyled_content).

### The JavaScript Blockade

Scripts stop the parser. JavaScript can alter the DOM or CSSOM, so the browser halts parsing to download, compile, and execute JS. High-performance sites use `defer` or `async` to download scripts in the background while the tree continues building.

## The Render Tree

The engine combines content (DOM) and styles (CSSOM) into the [**Render Tree**](https://web.dev/articles/critical-rendering-path-render-tree-construction).

```text
[DOM Tree]       [CSSOM Tree]
    |                 |
    +-----> ⨁ <-------+
            |
    [ Render Tree ]
```

The browser traverses the DOM and maps CSSOM styles to it. Invisible elements like `<head>`, `<script>`, or `display: none` are dropped. If it does not need pixels, it stays out of the tree.

## Layout and Reflow

The Render Tree knows _what_ to show, but not _where_. [**Layout**](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path#layout) calculates the geometry of every node based on the viewport.

The browser converts relative units (`vw`, `%`) into absolute pixels. This is **CPU-intensive**. Changing an element's width later forces a recalculation (often called [**Reflow**](https://developer.mozilla.org/en-US/docs/Glossary/Reflow)) for that element and its neighbors.

## Paint and Compositing

With geometry calculated, the browser enters [**Paint**](https://developer.mozilla.org/en-US/docs/Web/Performance/Critical_rendering_path#paint). It rasterizes text, colors, shadows, and images into pixels.

To maintain speed, modern browsers use [**Compositing**](https://developer.mozilla.org/en-US/docs/Glossary/Compositing):

```text
[ Layer 1: Background & Text ]  <-- CPU
[ Layer 2: Sticky Header ]      <-- CPU
[ Layer 3: GPU Animated Div ]   <-- GPU
             ↓
      [ COMPOSITOR ]
   (Draws layers to screen)
```

The Main Thread hands painted layers to the **Compositor Thread**, which communicates with the GPU. This separation makes `transform` and `opacity` animations smooth; they skip Layout and Paint to happen directly on the GPU.

---

## 8 Optimization Strategies

### 1. Minify and Compress CSS & JavaScript

CSS and JavaScript are render-blocking. Every KB matters.

```bash
# Production build: minify and gzip
npx terser app.js -o app.min.js
gzip -9 app.min.js
```

**Impact:** Reduce TTFB and parsing time by 30-50%.

### 2. Defer Non-Critical Scripts

Scripts block parsing. Move them to the end of the body or use `defer`:

```html
<!-- ❌ Blocks parsing -->
<script src="analytics.js"></script>

<!-- ✅ Downloads in background, executes after parsing -->
<script src="analytics.js" defer></script>

<!-- ✅ For third-party (tracking, ads) -->
<script src="ga.js" async></script>
```

**Impact:** Reduce DOM interactive time by 20-40%.

### 3. Inline Critical CSS

Inline the above-the-fold styles to eliminate render-blocking requests:

```html
<head>
  <style>
    /* Critical path styles only: header, hero, viewport content */
    body { margin: 0; font-family: sans-serif; }
    .hero { background: url(...); }
  </style>
  <link rel="stylesheet" href="full-styles.css" media="print" onload="this.media='all'">
</head>
```

**Impact:** Reduce First Contentful Paint (FCP) by 40-60%.

### 4. Lazy Load Images and Below-the-Fold Content

Don't load what users can't see yet:

```html
<!-- Native lazy loading (no JS required) -->
<img src="hero.jpg" alt="Hero" loading="lazy">

<!-- Or with IntersectionObserver for older browsers -->
<img data-src="image.jpg" class="lazy-img">

<script>
  const observer = new IntersectionObserver((entries) => {
    entries.forEach(entry => {
      if (entry.isIntersecting) {
        entry.target.src = entry.target.dataset.src;
        observer.unobserve(entry.target);
      }
    });
  });
  document.querySelectorAll('.lazy-img').forEach(img => observer.observe(img));
</script>
```

**Impact:** Reduce initial paint time by 30-50%, lower bandwidth by 40-60%.

### 5. Optimize Web Fonts (or Avoid Them)

Web fonts are render-blocking. Use `font-display: swap` or host locally:

```css
@font-face {
  font-family: 'Custom';
  src: url('font.woff2') format('woff2');
  font-display: swap; /* Show fallback immediately, swap when ready */
}
```

**Impact:** Reduce text rendering delay by 200-400ms.

### 6. Reduce Layout Thrashing

Avoid reading and writing the DOM in tight loops:

```javascript
// ❌ Bad: Forces layout recalculation 1000 times
for (let i = 0; i < 1000; i++) {
  element.style.width = element.offsetWidth + 1 + 'px'; // Read + Write
}

// ✅ Good: Read once, write once
let width = element.offsetWidth;
for (let i = 0; i < 1000; i++) {
  width += 1;
}
element.style.width = width + 'px';
```

**Impact:** Reduce Layout time by 50-80% for dynamic content.

### 7. Use CSS Containment for Isolated Layouts

Prevent layout recalculation of unrelated elements:

```css
.card {
  contain: layout style paint; /* Browser: don't recalc siblings when this changes */
}
```

**Impact:** Reduce reflow cost by 30-70% when updating isolated components.

### 8. Prioritize Network Requests with Resource Hints

Tell the browser what matters:

```html
<!-- Preconnect to critical domains -->
<link rel="preconnect" href="https://cdn.example.com">
<link rel="preconnect" href="https://fonts.googleapis.com">

<!-- Prefetch non-critical resources (next page) -->
<link rel="prefetch" href="/next-page.js">

<!-- Preload critical assets immediately -->
<link rel="preload" href="critical.js" as="script">
<link rel="preload" href="hero.jpg" as="image">
```

**Impact:** Reduce TTFB for critical resources by 100-300ms.

---

## Real-World Impact: Before & After

Here's what optimizing CRP looks like on a real site:

| Metric | Before | After | Improvement |
|--------|--------|-------|------------|
| First Contentful Paint (FCP) | 3.2s | 1.4s | -56% |
| Largest Contentful Paint (LCP) | 4.8s | 2.1s | -56% |
| Cumulative Layout Shift (CLS) | 0.18 | 0.05 | -72% |
| Lighthouse Score | 42 | 87 | +45 points |

These sites applied 5-6 of the above strategies. Start with #1 (minify), #2 (defer scripts), and #3 (inline critical CSS) for the fastest ROI.

---

## Why This Matters

Understanding this lifecycle replaces guessing with engineering. Knowing that CSS blocks rendering or that 3000px images penalize the Paint phase on mobile allows you to write better code instead of chasing plugins.

To see this theory applied to a real site, read [A Tale of Web Vitals](/a-tale-of-web-vitals/).


---

## A Tale of Web Vitals

**Date:** 2026-04-11
**URL:** http://localhost:1313/a-tale-of-web-vitals/
**Description:** A technical guide to Web Vitals. Solve LCP, TBT, CLS, and Accessibility issues using automated build-time pipelines and vanilla JS patches.


**TL;DR:** I achieved 95+ Lighthouse scores by removing abstractions and automating browser fundamentals. This guide details how to solve **LCP network discovery**, **main thread congestion**, and **third-party accessibility traps** using Hugo pipelines and vanilla JavaScript.

---

"Why is the LCP 4.2 seconds? It's just a static site".

I was staring at a Lighthouse report that felt like an insult. **Lorbic.com is a zero-dependency Hugo site.** No React, no heavy frameworks, just vanilla CSS and minimal JS. Yet, the mobile performance was tanking.

Performance is not a byproduct of your stack; it is a byproduct of engineering discipline. Lighthouse is an [**emulator of user frustration**](https://developer.chrome.com/docs/lighthouse/overview/), not a game to be tricked. Green scores come from understanding **browser threads**, not performance plugins.

Here is the before and after of my recent performance audit.

**The Baseline (Before):**
![Lighthouse Audit Before](before.png)

**The Result (After):**
![Lighthouse Audit Mobile After](after.png)
_Mobile: Significant improvement in Performance and Accessibility._

![Lighthouse Audit Desktop After](desktop-after.png)
_Desktop: Near-perfect scores across all categories._

This guide explains what these metrics actually mean, how developers break them, and the exact code I used to solve them.

_Note: To understand the deep mechanics behind these optimizations, read my companion guide on [The Critical Rendering Path](/til/2026-04-11-the-critical-rendering-path/)._

---

## Largest Contentful Paint

[**Largest Contentful Paint (LCP)**](https://web.dev/articles/lcp) measures when the largest visual element renders. It is the primary metric for perceived speed. If LCP exceeds 2.5 seconds, the user feels the site is stuck.

Standard advice says to lazy-load everything. But **blanket lazy loading kills LCP.** When you tag a hero image as `lazy`, you force the [**browser's preload scanner**](https://web.dev/articles/preload-scanner) to ignore it. The engine must parse CSS and build the render tree before discovering the image exists. This single oversight costs over a second in network delay.

I automated the distinction between **critical and deferred assets** in the build pipeline. My Hugo `render-image.html` hook now tracks state during the render cycle. The first image rendered on any post automatically receives [`fetchpriority="high"`](https://web.dev/articles/fetch-priority) and [`loading="eager"`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img#loading).

```go
{{- /* Smart Auto-Eager Loading for the First Image (LCP) */ -}}
{{- $isFirstImage := false -}}
{{- if not (.Page.Scratch.Get "hasRenderedFirstImage") -}}
  {{- $isFirstImage = true -}}
  {{- .Page.Scratch.Set "hasRenderedFirstImage" true -}}
{{- end -}}

{{- $loading := "lazy" -}}
{{- $fetchpriority := "auto" -}}

{{- if $isFirstImage -}}
  {{- $loading = "eager" -}}
  {{- $fetchpriority = "high" -}}
{{- end -}}
```

By forcing the browser to start the network request the millisecond the HTML parses, my LCP dropped from 4.2s to 1.1s.

## Total Blocking Time

[**Total Blocking Time (TBT)**](https://web.dev/articles/tbt) tracks main thread congestion. If the thread locks for over 50ms, the page freezes. This happens when we ship desktop assets to mobile devices. Shipping a 3000px wide image to an iPhone wastes data and chokes the CPU during decoding.

I implemented an **automated WebP pipeline** that generates a multi-size `<picture>` element at build time. Mobile users now download a ~45KB WebP asset instead of a multi-megabyte original. This reduces CPU strain.

```go
{{- if $image -}}
    {{- $widths := slice 480 800 1200 -}}
    <picture>
      <source srcset="{{ delimit $srcset ", " }}" sizes="(max-width: 800px) 100vw, 800px" type="image/webp">
      <img src="{{ $originalWebp.RelPermalink }}" width="{{ $image.Width }}" ... />
    </picture>
{{- end -}}
```

**CSS is also render-blocking.** The browser won't draw a single pixel until it parses the entire CSS file. I use [**PurgeCSS**](https://purgecss.com/) to drop my payload from 850KB to 87KB. The trick is cleanup the css. By running PurgeCSS and using `hugo_stats.json` as the source of truth, I ensure dynamically generated classes never get remove by using the following rule.

```javascript
// postcss.config.js
const purgecss = require("@fullhuman/postcss-purgecss")({
  content: ["./hugo_stats.json", "./layouts/**/*.html", "./content/**/*.md"],
  defaultExtractor: (content) => {
    if (content.trim().startsWith("{")) {
      const els = JSON.parse(content).htmlElements;
      return (els.tags || []).concat(els.classes || [], els.ids || []);
    }
    return content.match(/[A-Za-z0-9-_:\/]+/g) || [];
  },
});
```

## Cumulative Layout Shift

[**Cumulative Layout Shift (CLS)**](https://web.dev/articles/cls) measures visual instability. This is caused by images without dimensions, custom fonts that cause a **Flash of Unstyled Text (FOUT)**, or late theme scripts.

My image pipeline automatically injects `width` and `height` attributes. This reserves physical space before the image downloads. I also added [`<link rel="preload">`](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/preload) to the `baseof.html` for critical font files.

I updated the js with an inline theme script at the top of the `<head>`. By setting the `data-theme` attribute before the browser reaches the `<body>`, I apply colors instantly. This eliminates the "white flash" typical of dark mode sites.

## Accessibility

Accessibility is about **keyboard flow**, not just contrast. I was failing audits because of an anti-spam widget. It injected a link marked as `aria-hidden="true"` that was still focusable via keyboard. This created a [**"focus trap"**](https://developer.mozilla.org/en-US/docs/Web/Accessibility/Keyboard-navigable_JavaScript_widgets#using_tabindex): a keyboard user lands on an element the screen reader claims does not exist.

Since I do not control the third-party script, I used a [**`MutationObserver`**](https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver) in vanilla JS to patch the DOM the moment the widget renders.

```javascript
const mutObserver = new MutationObserver(() => {
  const logo = widget.querySelector(".altcha-logo");
  if (logo && logo.hasAttribute("aria-hidden")) {
    logo.removeAttribute("aria-hidden");
    mutObserver.disconnect();
  }
});
mutObserver.observe(widget, { childList: true, subtree: true });
```

This tiny patch was the difference between a 90 and a perfect 100 on my accessibility score.

## Icon Latency

Interactive elements like **"Copy Code"** buttons should not wait for an external font file. Relying on icon fonts adds unnecessary network requests and causes a **"Flash of Unstyled Icon"**.

I replaced FontAwesome with **optimized SVG strings** embedded directly in the JavaScript bundles. UI feedback is now instantaneous. The main thread does not wait for external files to resolve before rendering button states.

## Live Observability

Performance is a state of being, not a one-time event. To keep myself honest, I built a **Performance Telemetry HUD** into the footer. It uses the [**Performance API**](https://developer.mozilla.org/en-US/docs/Web/API/Performance_API) to show real-time load metrics, transfer size, and request counts.

Having these numbers visible during development makes it impossible to ignore heavy images or bloated dependencies.

## Dealing with Phantom Warnings

Even perfect code faces noise. While auditing, I noticed intermittent warnings for `SharedStorage` and `Fledge` cookies. These came from **Cloudflare Zaraz and Google Analytics**, not my source code. High-performance engineering means knowing which bytes you own and which bytes are forced upon your users by third-party services.

## Trade-offs and Costs

Every architectural victory has a cost. Choosing a **zero-dependency static architecture** means:

- **Build Complexity:** My `render-image.html` hook is more complex than a standard `<img>` tag. I have to maintain a stateful build-time pipeline for LCP detection.
- **The Caching Trade-off:** I explicitly set `inline_css = false`. Lighthouse recommends "inlining critical CSS" to save a request, but this destroys long-term caching. I trade a fraction of a second on the first load for massive gains on the next ten visits.
- **Maintenance:** Since I do not use a framework, I am responsible for the "patch work" like the `MutationObserver` for accessibility. There is no library to update; there is only code to maintain.

## The Power of Hugo

None of these optimizations would be sustainable without Hugo. I love it for its **uncompromising speed** and engineering elegance.

Hugo builds 400+ pages and my entire asset pipeline in under 700ms. This speed is a productivity multiplier. The template engine is the real power: using `.Page.Scratch` to build a **build-time state machine** for LCP detection is a level of flexibility rarely found elsewhere. Hugo allows me to solve complex browser problems during the build so the user never has to solve them at runtime.

## The Cost of Complexity

Reflecting on the last [ten-year journey](/10-years-of-lorbic.com-architecture/), I have gone through every phase of the developer lifecycle. I started on managed platforms with no control. I moved to dynamic backends with Django because I thought "real" sites needed databases. I finally landed on a zero-dependency static architecture.

The irony is that the more code I removed, the better the experience became. Performance is not about how much logic you add; it is about **technical sovereignty**. In 2018, I wanted a complex platform. In 2026, I want to engineer a fast, accessible, and honest piece of the web.

## Respect the Browser

The browser is fast if you give it the right instructions in the right order. Performance is an **engineering discipline** you maintain from the first line of HTML. Respect the browser, and it will reward you with speed.


---

## Why Your Goroutines Need a Speed Limit: Bounded Concurrency in Go

**Date:** 2026-04-10
**URL:** http://localhost:1313/bounded-concurrency-in-go/
**Description:** Unbounded concurrency is a reliability nightmare. Learn how to protect your system from OOM kills and database exhaustion by implementing Semaphores and Worker Pools in Go.


**TL;DR:** Spawning `go func()` without a limiter is a recipe for system collapse. This guide details how to use **Semaphores** and **Worker Pools** to prioritize predictable stability over absolute speed, protecting downstream dependencies from the thundering herd.

---

It's a rite of passage for every Go developer. You receive a list of 10,000 URLs to fetch or 50,000 rows to process. You wrap the workload in an unbounded `go func()` loop, achieving maximum throughput in milliseconds.

```go
// Initial approach: unbounded concurrency
func ProcessItems(items []Item) {
    var wg sync.WaitGroup
    for _, item := range items {
        wg.Add(1)
        go func(i Item) {
            defer wg.Done()
            process(i) // HTTP request, DB write, etc.
        }(item)
    }
    wg.Wait()
}
```

This code runs flawlessly on a local machine. However, in production, it triggers **OOM (Out of Memory)** kills and database connection failures. You achieved speed, but you sacrificed **reliability**.

---

## Why cheap goroutines are expensive

Go makes concurrency easy because goroutines are lightweight. They start with a [**2KB stack**](https://go.dev/doc/faq#goroutines). While your CPU can manage 10,000 threads, your downstream dependencies cannot.

Unbounded `go func()` loops create a **Thundering Herd**. Instantaneous execution forces your system to simultaneously demand:

1.  **Memory:** 10,000 goroutines require 20MB of stack space just to initialize. This excludes the heap allocations required to process JSON or hold HTTP buffers.
2.  **File Descriptors:** Every outbound request requires a socket. Standard Linux environments cap file descriptors at 1,024 per process.
3.  **Connection Pools:** Your database pool is a finite resource. If it is capped at 50 connections, 9,950 goroutines will block while holding onto their allocated memory, causing massive **GC pressure**.

To build resilient systems, you must impose a **Speed Limit**.

---

## Use Semaphores for minimal refactoring

A [**Semaphore**](<https://en.wikipedia.org/wiki/Semaphore_(programming)>) restricts the number of threads accessing a shared resource. In Go, you can implement this pattern using a **buffered channel**.

A buffered channel blocks when full. By using it as a [**token bucket**](https://github.com/vikash-paf/goutils/tree/main/rate#tokenbucket), you control exactly how many goroutines execute their critical path at once. For production workloads requiring burst handling and precise refills, I use the [TokenBucket](https://pkg.go.dev/github.com/vikash-paf/goutils@v1.0.0/rate#TokenBucket) implementation from my `goutils` library.

### How to implement a channel-based Semaphore

1.  Create a buffered channel with a capacity equal to your limit.
2.  Push an empty struct into the channel before starting work (acquire token).
3.  Read from the channel when the work finishes (release token).

```go
func ProcessItemsWithSemaphore(items []Item, maxConcurrency int) {
    var wg sync.WaitGroup
    sem := make(chan struct{}, maxConcurrency) // The token bucket

    for _, item := range items {
        wg.Add(1)
        go func(i Item) {
            defer wg.Done()
            sem <- struct{}{}        // Acquire token (blocks if full)
            defer func() { <-sem }() // Release token
            process(i)
        }(item)
    }
    wg.Wait()
}
```

This pattern is ideal for quick scripts because it requires minimal code changes. However, it still spawns 10,000 goroutines. The loop completes instantly, leaving thousands of blocked goroutines parked in memory. For massive workloads, you need a more robust architecture.

---

## Use Worker Pools for sustained processing

While a Semaphore is a traffic light, a [**Worker Pool**](https://en.wikipedia.org/wiki/Thread_pool) is an assembly line. You spawn a fixed number of long-lived "Worker" goroutines that pull jobs from a shared channel.

### How to architect an assembly line

1.  Initialize a `jobs` channel.
2.  Spawn `N` workers that `range` over that channel.
3.  Feed items into the channel.
4.  Close the channel to signal workers to exit.

```go
func worker(id int, jobs <-chan Item, wg *sync.WaitGroup) {
    defer wg.Done()
    for item := range jobs {
        process(item)
    }
}

func ProcessItemsWithPool(items []Item, numWorkers int) {
    jobs := make(chan Item, len(items))
    var wg sync.WaitGroup

    for w := 1; w <= numWorkers; w++ {
        wg.Add(1)
        go worker(w, jobs, &wg)
    }

    for _, item := range items {
        jobs <- item
    }
    close(jobs) // Signal shutdown
    wg.Wait()
}
```

Worker Pools decouple the **volume of work** from **resource consumption**. Whether processing 100 items or 100,000, your system only spawns `numWorkers` goroutines. Memory profiles remain flat, and GC pressure is minimized.

---

## Trade-offs and Costs

Engineering is a series of trade-offs. Bounding your concurrency introduces specific costs:

- **Latency vs. Stability:** Limiting concurrency increases the total execution time for a batch. You trade absolute speed for a predictable, non-collapsing system.
- **Complexity:** Worker Pools require more boilerplate than a simple `go func()`. You must manage channel lifecycles and [**context-based cancellation**](https://go.dev/blog/context) properly.
- **Deadlock Risk:** Improper channel management in pools can lead to deadlocks if workers are blocked while the feeder loop waits for capacity.

---

Optimizing for speed is the first phase of implementation. The second phase; and the most critical for production reliability, is identifying how safely to constrain that speed.

Unbounded concurrency is a bug that waits for a traffic spike to trigger. Establish a habit: every time you type `go func()`, identify its upper bound. If the loop is tied to user input or database rows, you need a speed limit.

Grab a **Semaphore** for scripts and a **Worker Pool** for production pipelines. Respect the hardware, and your database will remain stable.


---

## Part 3: Casting Shadows Without Trigonometry: The Beauty of Integer Math

**Date:** 2026-04-07
**URL:** http://localhost:1313/casting-shadows-bresenham-integer-math/
**Description:** To calculate Field of View in a grid-based game engine, you have to cast rays. The naive approach uses heavy floating-point trigonometry. This post explores Bresenham's Line Algorithm: a 60-year-old technique from the era of hardware plotters that draws perfect lines using only integer addition and comparison.


At the end of generating a procedural map for the _[Derelict Facility](https://github.com/vikash-paf/derelict-facility)_ engine, I had a sprawling interconnected maze of rooms and corridors. But there was a devastating problem: I could see everything.

The entire map was rendered at once. It looked like a top-down blueprint, not a dark, atmospheric facility. I needed to implement Fog of War. I needed to treat the player character like a lighthouse in the dark.

## The Engineering Problem: The Grid vs. The Real World

In the real world, light travels in perfectly straight lines. If you look at a wall, photons travel from the wall to your eye.

In a game engine, you don't have a "real world". You have a 2D grid of memory (`[]Tile`). If the player is at `X: 10, Y: 10` and a wall is at `X: 15, Y: 12`, how do you mathematically determine if the player can see the wall?

You need to draw a line between them. But grids don't allow smooth lines. They only allow stepping from one integer coordinate to the next. You can't stand at `X: 10.5`.

## The Naive Approach: Floating Point Math

The modern instinct for drawing a line from point A to point B is trigonometric. You calculate the angle (`math.Atan2`), find the sine and cosine, and step along the floating-point line, rounding to the nearest integer grid tile at each step.

This works, but it is deeply flawed for systems engineering:

1.  **It is slow.** Transcendental functions (`sin`, `cos`) are expensive CPU operations.
2.  **It is imprecise.** Floating-point rounding errors create "holes" where a ray might accidentally skip a wall tile.
3.  **It is overkill.** You don't need a floating-point unit to navigate a 2D integer grid.

## Bresenham's Line Algorithm (1962)

In 1962, Jack Bresenham at IBM solved this problem for hardware plotters and early CRT displays. His algorithm determines which pixels to illuminate to form a straight line using **only integer addition, subtraction, and bit-shifting**. No division. No floating point.

It was designed for hardware that literally lacked decimal math processors, and it remains the industry standard today.

Here is the exact implementation used in the _Derelict Facility_ engine to chart the path of a photon across the map grid:

{{< code lang="go" file="internal/world/algo.go" >}}
func getLine(x0, y0, x1, y1 int) []entity.Point {
var points []entity.Point

    // dx is the absolute horizontal distance
    dx := math.Abs(x1 - x0)

    // dy is the absolute vertical distance. We make it negative as a
    // math trick so we can ADD it to our error tracker later.
    dy := -math.Abs(y1 - y0)

    // Step direction: are we going left or right? Up or down?
    sx := -1
    if x0 < x1 { sx = 1 }

    sy := -1
    if y0 < y1 { sy = 1 }

    // The Error Tracker: how far we've drifted from the true mathematical line
    err := dx + dy

    for {
        points = append(points, entity.Point{X: x0, Y: y0})

        if x0 == x1 && y0 == y1 {
            break // Destination reached
        }

        // Double the error for comparison.
        // This prevents us from ever needing fractions (like 0.5).
        e2 := 2 * err

        // Adjust X if our drift tolerance allows it
        if e2 >= dy {
            err += dy
            x0 += sx
        }

        // Adjust Y if our drift tolerance allows it
        if e2 <= dx {
            err += dx
            y0 += sy
        }
    }

    return points

}
{{< /code >}}

### How the Error Tracker Works

The genius of Bresenham is the `err` accumulator. It tracks how far our jagged, step-by-step integer path has drifted from the mathematically perfect straight line.

When the drift exceeds a specific threshold, the algorithm corrects itself by stepping diagonally. By using `e2 := 2 * err`, we eliminate the need to compare against `0.5`, allowing the entire calculation to remain strictly within integer bounds.

## Perimeter Raycasting

To implement the "Lighthouse", we define a vision radius (e.g., 8 tiles). We want to illuminate everything within that radius, unless a wall blocks the light.

The obvious instinct is to just cast 360 rays in a circle based on angles (using sine/cosine). However, as these rays get further away from the player, the physical gap between the rays gets larger. Eventually, the gap becomes wider than a single 1x1 grid tile. This results in missing tiles—annoying "stripes of darkness" appearing inside a lit room.

Instead, we use a technique called **Perimeter Raycasting**.

The algorithm is strictly grid-aligned:

1.  Define a square box around the player based on the radius (Top, Bottom, Left, and Right edges).
2.  Iterate through every single tile on that square's perimeter.
3.  Calculate a line from the Player to that Target Tile (using Bresenham's algorithm).
4.  Walk along that line, tile by tile, setting `Visible = true` until it hits a wall.

{{< code lang="go" file="internal/world/map.go" >}}
func (m \*Map) castRay(x1, y1, x2, y2 int) {
line := getLine(x1, y1, x2, y2)

    for _, p := range line {
        // Stop if the ray leaves the map
        if p.X < 0 || p.X >= m.Width || p.Y < 0 || p.Y >= m.Height {
            break
        }

        idx := m.GetIndex(p.X, p.Y)

        // Mark the tile as currently visible
        m.Tiles[idx].Visible = true

        // Permanently mark it as explored (for fog of war memory)
        m.Tiles[idx].Explored = true

        // If the tile is solid matter, light cannot pass through it.
        // Break the loop to stop casting this specific ray.
        if !m.Tiles[idx].Walkable {
            break
        }
    }

}
{{< /code >}}

Because `getLine` is so highly optimized, I can comfortably cast hundreds of rays every frame without dropping below 60 FPS.

When you strip away heavy abstractions and write algorithms that respect the physical capabilities of the CPU (in this case, basic integer arithmetic), performance stops being a goal you have to fight for and becomes the natural default state of the code.

## Further Reading

- [What the Hero Sees (Bob Nystrom)](https://journal.stuffwithstuff.com/2015/09/07/what-the-hero-sees/) - A phenomenal visual breakdown of grid-based FOV algorithms.

---

_This is Part 3 of the [Building a Game Engine in Pure Go](/series/building-a-game-engine-in-pure-go/) series. The full source code for Derelict Facility is available on [GitHub](https://github.com/vikash-paf/derelict-facility)._


---

## Part 2: Decoupling the Renderer: Terminal to Raylib in One Interface

**Date:** 2026-04-06
**URL:** http://localhost:1313/decoupling-game-engine-renderer-raylib/
**Description:** If your game logic knows about OpenGL, your architecture has failed. This post dissects the Interface Segregation Principle in Go, demonstrating how the Derelict Facility engine swapped an ANSI terminal renderer for hardware-accelerated Raylib without changing a single line of game simulation code.


The biggest architectural mistake you can make when building a game engine is letting the game know how it is being drawn.

When I started building the _Derelict Facility_ engine, the output target was a raw ANSI terminal. The engine calculated A\* paths, resolved line-of-sight, and then spewed escape codes (`\033[31m`) to `os.Stdout`.

Eventually, I hit the physical limits of terminal emulators: inconsistent character widths (especially for emojis), slow double-buffering, and a hard cap on frame rates. I needed to move to a real, hardware-accelerated graphics library like Raylib.

Because I had architected the engine around Go interfaces, I replaced the entire visual and input layer of the game without modifying a single line of the core simulation.

## The Display Boundary

If your game logic calls `raylib.DrawRectangle()`, your game logic is permanently bound to Raylib.

To prevent this, I defined a strict boundary. The `Engine` struct knows absolutely nothing about ANSI, terminals, OS windows, or GPUs. It only knows about an interface called `Display`:

{{< code lang="go" file="internal/display/display.go" >}}
type Display interface {
Init(gridWidth, gridHeight int, title string) error
Close()
ShouldClose() bool

    BeginFrame()
    EndFrame()
    Clear(colorHex uint32)
    DrawText(gridX, gridY int, text string, colorHex uint32)

    PollInput() []core.InputEvent

}
{{< /code >}}

This interface enforces two critical constraints:

1.  **Coordinate Abstraction:** The engine asks to draw text at `(gridX: 10, gridY: 5)`. It does not know or care if that translates to `Row 5, Column 10` in a terminal or `Pixel X: 120, Pixel Y: 120` on a 4K monitor.
2.  **Color Abstraction:** The engine asks for a standard `uint32` hex color (`0xFF0000FF`). It is the Display's job to figure out how to render red.

## Implementing the Raylib Concrete

With the boundary defined, implementing the Raylib backend was an isolated exercise. The concrete `RaylibDisplay` struct simply translates the abstract grid commands into Raylib-specific hardware calls.

{{< code lang="go" file="internal/display/raylib.go" >}}
type RaylibDisplay struct {
CellWidth int32
CellHeight int32
FontSize int32
FontPath string
Font rl.Font
}

func (r _RaylibDisplay) DrawText(gridX, gridY int, text string, colorHex uint32) {
// 1. Translate Grid coordinates to Pixel coordinates
pixelY := int32(gridY) _ r.CellHeight

    colOffset := 0
    for _, char := range text {
        charStr := string(char)

        // Calculate precise centering for the font glyph
        vecSize := rl.MeasureTextEx(r.Font, charStr, float32(r.FontSize), 0)
        charWidth := int32(vecSize.X)
        pixelX := int32(gridX+colOffset)*r.CellWidth + (r.CellWidth-charWidth)/2

        // 2. Execute the hardware-accelerated draw call
        position := rl.NewVector2(float32(pixelX), float32(pixelY))

        // Notice we cast our abstract uint32 color directly into a Raylib Color struct
        rl.DrawTextEx(r.Font, charStr, position, float32(r.FontSize), 0, rl.GetColor(uint(colorHex)))
        colOffset++
    }

}
{{< /code >}}

This implementation hides massive complexity from the game engine. The engine doesn't need to load the `.ttf` font file, generate a texture atlas, calculate bilinear filtering, or measure bounding boxes. It just says "Draw an `@` at 5,5 in Red".

## Input Mechanism

Output is only half the battle. If your game loop calls `raylib.IsKeyPressed()`, it is once again permanently bound to the framework.

The `Display` interface demands a slice of `core.InputEvent`. This completely abstracts away the physical hardware capturing the keystrokes.

{{< code lang="go" file="internal/display/raylib.go" >}}
func (r \*RaylibDisplay) PollInput() []core.InputEvent {
var events []core.InputEvent

    if rl.IsKeyPressed(rl.KeyW) || rl.IsKeyPressedRepeat(rl.KeyW) {
        events = append(events, core.InputEvent{Key: core.KeyW})
    }
    if rl.IsKeyPressed(rl.KeyS) || rl.IsKeyPressedRepeat(rl.KeyS) {
        events = append(events, core.InputEvent{Key: core.KeyS})
    }
    // ...

    return events

}
{{< /code >}}

Because the engine only consumes `core.InputEvent`, I could swap the Raylib keyboard hook for an Xbox controller hook, a network socket (for multiplayer), or a pre-recorded slice of inputs (for deterministic unit testing), all without touching the core game loop.

## Dependency Injection

When the program starts, the `main` function decides which reality the engine lives in. The `Engine` struct accepts the `Display` interface via Dependency Injection.

{{< code lang="go" file="cmd/derelict/main.go" >}}
func main() {
// 1. Instantiate the concrete hardware layer
disp := display.NewRaylibDisplay(12, 24, 24, "assets/fonts/FiraCode-Bold.ttf")

    // 2. Instantiate the game logic
    generatedMap, _, _ := world.NewFacilityGenerator(1234).Generate(120, 30)
    ecsWorld := ecs.NewWorld()

    // 3. Inject the hardware layer into the agnostic engine
    gameEngine := engine.NewEngine(disp, generatedMap, ecsWorld, world.TileVariantCold)

    // 4. Start the simulation
    gameEngine.Run()

}
{{< /code >}}

If I want to run the game headless on a CI server to benchmark the pathfinding algorithm, I write a `NullDisplay` struct that implements the interface but does nothing.

The Interface Segregation Principle is rarely used correctly in modern microservice architectures, where interfaces are often just 1:1 mocks for database calls.

But in systems engineering and game development, interfaces are the blast doors that protect your complex, expensive business logic (procedural generation, AI, physics) from the volatile, ever-changing demands of I/O hardware.

---

_This is Part 2 of the [Building a Game Engine in Pure Go](/series/building-a-game-engine-in-pure-go/) series. The full source code for Derelict Facility is available on [GitHub](https://github.com/vikash-paf/derelict-facility)._


---

## Part 1: Data-Oriented Design in Go: Why [][]Tile Destroyed My Game Engine

**Date:** 2026-04-05
**URL:** http://localhost:1313/data-oriented-design-go-contiguous-memory/
**Description:** The textbook answer for a 2D grid in Go is a slice of slices. In a systems-level game engine running at 60 FPS, this innocent data structure becomes a performance landmine. This post explores pointer chasing, CPU cache lines, and how flattening a 2D map into contiguous memory creates massive performance gains through Data-Oriented Design.


Most game development stories start the same way: install Unity, drag some sprites onto a canvas, and press Play.

I wanted to understand the metal. I set out to build _[Derelict Facility](https://github.com/vikash-paf/derelict-facility)_, a systems-level game engine from scratch in pure Go. No SDL, no OpenGL wrappers, no Ebiten. The goal wasn't just to ship a game; the goal was to learn the memory layouts and I/O pipelines that modern engines hide behind friendly APIs.

The very first decision you face when building a grid-based simulation is how to represent the map in memory. The textbook answer looks obvious:

{{< code lang="go" >}}
// The obvious approach : a slice of slices
type Map struct {
Grid [][]Tile
}
{{< /code >}}

In standard application code, this is perfectly fine. In systems programming, where you are rendering a 120x30 grid 60 times a second and running A\* pathfinding algorithms across thousands of nodes, this data structure is a performance landmine.

## The Problem: Pointer Chasing

To understand why `[][]Tile` fails at scale, you have to look at how Go allocates memory on the heap.

A slice in Go is a small struct containing a pointer to an underlying array, a length, and a capacity. Therefore, a "slice of slices" is actually a list of pointers. Each inner slice (`Grid[y]`) is a separate heap allocation pointing to a block of memory located _somewhere_ else.

But "somewhere" is the problem. These inner arrays are scattered randomly across the heap.

When the CPU tries to iterate over the grid row by row to render the map or calculate Field of View (FOV), it has to chase a pointer to a completely new memory address for every single row. Each jump is a potential **cache miss**. The CPU stalls while it waits to fetch data from main RAM instead of the blazing-fast L1/L2 cache sitting next to the core.

On a tight 16ms frame budget, those memory stalls compound into visible frame drops.

## The Solution: Contiguous Memory

In Data-Oriented Design, the primary rule is: **respect the CPU cache**. CPUs do not read memory one byte at a time; they read chunks of memory (cache lines, usually 64 bytes) at once. If your data is laid out sequentially, the CPU will automatically prefetch the next items before your code even asks for them.

I re-architected the `Derelict Facility` map as a single, flat, contiguous 1D block of memory:

{{< code lang="go" file="internal/world/map.go" >}}
ttype Map struct {
Tiles []Tile // Size = Width \* Height (one contiguous block)
Width int
Height int
Rooms []Rect
}

func NewMap(width, height int) *Map {
return &Map{
Width: width,
Height: height,
// One single allocation for the entire world
Tiles: make([]Tile, width*height),
}
}

{{< /code >}}

Instead of allocating memory `Height` times, we allocate exactly once. The entire map lives in one unbroken block of RAM.

### Stride Math (O(1) Access)

If the map is a flat 1D array, how do we access a specific `(x, y)` coordinate? We use **stride math**: a simple formula you'll find at the core of every framebuffer, texture map, and video decoder on Earth:

`Index = X + (Y * Width)`

{{< code lang="go" file="internal/world/map.go" >}}
func (m *Map) GetTile(x, y int) *Tile {
// Bounds checking
if x < 0 || x >= m.Width || y < 0 || y >= m.Height {
return nil
}

    // O(1) mathematical lookup
    return &m.Tiles[x + y*m.Width]

}
{{< /code >}}

There is zero pointer chasing. The CPU calculates the exact memory offset instantly. When our Raylib renderer scans across the map from left to right, the CPU aggressively prefetches the `Tile` structs because it knows exactly where they are.

## Shrinking the Struct

Contiguous memory is only half the battle; the other half is data density. The smaller the struct, the more of them fit into a single 64-byte L1 cache line.

If I had defined my `Tile` struct with strings for colors or complex interfaces, the memory footprint would bloat. Instead, I kept it as small as physically possible:

{{< code lang="go" file="internal/world/tile.go" >}}
type TileType uint8

const (
TileTypeEmpty TileType = iota
TileTypeWall
TileTypeFloor
)

type Tile struct {
Type TileType // 1 byte
Walkable bool // 1 byte
Visible bool // 1 byte
Explored bool // 1 byte
}
{{< /code >}}

This struct packs perfectly into **4 bytes**.

Do the math: A 120x30 map contains 3,600 tiles. At 4 bytes per tile, the entire map of the _Derelict Facility_ fits into exactly **14.4 KB** of RAM.

A modern CPU L1 data cache is typically 32 KB or 64 KB. This means the engine can load the _entire physical game world_ into the absolute fastest layer of CPU memory simultaneously.

## The Trade-offs

Is a contiguous 1D array always the right answer? No.

1.  **Immutability of Size:** A flat slice is brilliant for a fixed-size grid (like a game map or an image matrix). If your map needs to dynamically grow or shrink in unpredictable directions during runtime, managing a massive contiguous reallocation is expensive. You would need to implement a chunking system (like Minecraft).
2.  **Cognitive Overhead:** `Grid[y][x]` is undeniably easier to read and write than `m.Tiles[x + y*m.Width]`. You must abstract the math behind reliable helper functions (like `GetTile`) to prevent developers from messing up the index calculations.

Before you write a single line of business logic or rendering code, your data layout is already dictating your system's performance ceiling.

By flattening our 2D structures, leaning on simple mathematical strides, and ruthlessly stripping our structs of heap-allocated pointers, we stop fighting the Go garbage collector and start working in harmony with the CPU cache. In systems engineering, mechanical sympathy is everything.

## Further Reading

If you want to dive deeper into Data-Oriented Design and Engine Architecture, these are the foundational resources that shaped my approach to _Derelict Facility_:

**Books:**

- _"Game Engine Architecture"_ by Jason Gregory
- _"Game Programming Patterns"_ by Robert Nystrom (specifically the [Data Locality](https://gameprogrammingpatterns.com/data-locality.html) chapter)
- _"Data-Oriented Design"_ by Richard Fabian

**Lectures & Talks:**

- _"Data-Oriented Design and C++"_ by Mike Acton (GDC 2014)
- _"A Practical Guide to Applying Data-Oriented Design"_ by Andrew Kelley

**Online Resources:**

- [The `dbartolini/data-oriented-design` GitHub Repository](https://github.com/dbartolini/data-oriented-design)
- Unity DOTS (Data-Oriented Technology Stack) Documentation

---

_This is Part 1 of the [Building a Game Engine in Pure Go](/series/building-a-game-engine-in-pure-go/) series. The full source code for Derelict Facility is available on [GitHub](https://github.com/vikash-paf/derelict-facility)._


---

## 10 years of lorbic.com architecture

**Date:** 2026-04-02
**URL:** http://localhost:1313/10-years-of-lorbic.com-architecture/
**Description:** Reviewing the technical evolution of this blog over a decade: from managed platforms and dynamic backends to a custom, zero-dependency Hugo architecture.


![Lorbic Evolution: 2017 vs 2026](evolution.webp)

This post tracks four architectural rewrites of my personal site over ten years: moving from managed platforms (Blogger) to dynamic backends (Django), static site generators (Jekyll/Hugo), and finally to a custom, zero-dependency architecture built for absolute control and performance.

---

## Technical Context

For an engineer, a personal site is the only project where you have absolute authority over the stack. There are no product managers, no legacy constraints, and no enterprise technical debt: you own the metal.

Over the last decade, my approach to this site has evolved from simple content publishing to a pursuit of complete technical control. This is the timeline of that evolution.

## 2015–2018: Managed Platforms (WordPress & Blogger)

**Stack:** WordPress.com -> Blogger.com  
**Links:** [villageprogrammer.blogspot.com](https://villageprogrammer.blogspot.com)

The initial goal was low-friction publishing. I started on WordPress in 2015 because it was the default, then moved to Blogger in late 2017 to create "Village Programmer" and get slightly better access to the underlying HTML/XML templates.

![The 2017 villageprogrammer.blogspot.com design](2017.webp)

**Technical Trade-off:**
Control was completely sacrificed for convenience. The rendering pipeline was a black box. The UI was composed of unmanaged CSS and proprietary widgets injected by the platform. Portability was non-existent; the content was locked in a managed ecosystem. I could write, but I couldn't engineer.

## 2018–2020: Dynamic Architecture (Django & PostgreSQL)

**Stack:** Django + PostgreSQL + Nginx  
**Links:** Self-hosted Linux instance

By college, I had learned MVC patterns and relational databases. Naturally, I assumed my blog needed a backend. I migrated everything to a custom Django application.

**The Trade-off:**
Operational overhead was the primary failure point. Managing a relational database and a Python runtime for a read-heavy, low-write blog is an architectural mismatch. The maintenance burden, including security patches, database backups, and server monitoring, became a distraction from actually writing or building new features. I learned that "Full-Stack" often just means "more things to break".

## 2020–2024: Static Site Generators (Jekyll & Hugo)

**Stack:** Jekyll -> Hugo (Even Theme)  
**Links:** [vk4s.github.io](https://vk4s.github.io)

I eventually recognized that a blog is an immutable dataset that should be pre-compiled. I migrated to the Jamstack: first using Jekyll on GitHub Pages, then porting to Hugo for sub-second build times. My content was finally decoupled from the infrastructure, living purely as Markdown in a Git repository.

![The 2021 vk4s.github.io static site using the Even theme](2021.webp)

**The Trade-off:**
Performance and portability were solved, but I was still using an off-the-shelf theme ("Even"). Modifying the site's behavior meant fighting against foreign CSS conventions and non-essential JavaScript dependencies. It was fast, but it wasn't mine.

## 2025–2026: Custom Architecture (Zero Dependencies)

**Stack:** Hugo + Vanilla Web Technologies  
**Links:** [lorbic.com](https://lorbic.com) (Engineering Blog) / [vikashpatel.net](https://vikashpatel.net) (Portfolio)

The current iteration of `lorbic.com` is a complete rewrite. Beginning in early 2025, I started stripping away the layers of accumulated technical debt: removing heavy CSS frameworks like Bootstrap, and deleting bloated JavaScript dependencies like jQuery and Zoom JS.

The mandate was strict: zero external dependencies, zero CSS frameworks, and maximum performance through native primitives. I separated my portfolio (`vikashpatel.net`) from my engineering writing (`lorbic.com`).

![The current 2026 Lorbic dark-mode architecture](2026.webp)

### Engineered Features

Instead of importing "black box" plugins, I implemented focused, vanilla solutions for specific needs.

**1. Command Palette (Terminal-inspired UI)**  
Navigation is handled via a native terminal overlay (`Cmd+K`). It utilizes a pre-compiled JSON search index and a custom routing engine. This allows for instant theme toggling and site-wide search without the overhead of heavy third-party search libraries.

![The Cmd+K Command Palette](command-pallet.webp)

**2. Contextual Selection Tooltip**  
Standard anchor links are static. I engineered a tooltip that hooks into the browser's Selection API. Highlighting text creates a [Text Fragment](https://developer.mozilla.org/en-US/docs/Web/Text_fragments) URL (`#:~:text=...`), allowing readers to share exact paragraph references.

![Deep Linking Context Menu](select-share-context-menu.webp)

**3. Interactive Data Tables**  
All Markdown tables are wrapped in a custom JS observer. This adds an interactive action bar to every dataset, enabling one-click "Copy to Markdown" and "Download as CSV" functionality. It relies purely on the DOM; zero external CSV-parsing libraries are required.

**4. Performance Telemetry HUD**  
To maintain performance hygiene, a real-time HUD tracks metrics via the `Navigation Timing API`. It displays DOM load times (typically < 40ms) and network payload sizes directly in the viewport.

![Live Performance Telemetry HUD](live-perf.webp)

## The Backstage Workflow

While the site is static at the edge, the management layer is automated.

I manage the publication pipeline using custom Python tooling via Poetry. Scripts residing in the `scripts/` directory handle automated backups of my self-hosted Listmonk newsletter instance and schedule content distributions.

The deployment pipeline is a standard Git-based CI/CD flow via GitHub Actions, which builds the static site via Hugo and pushes the optimized assets to the CDN.

---

Building your own blog engine is not about necessity. It's about forcing yourself to understand the boundaries of the systems you use.

By stripping away managed platforms and heavy themes, I had to solve core problems in CSS containment, the critical rendering path, and DOM performance manually. The result is an infrastructure that behaves exactly as designed, served as fast as physics allows.

![Terminal 404 Page Design](not-found.webp)


---

## My Reading List

**Date:** 2026-04-02
**URL:** http://localhost:1313/reading-list/
**Description:** A living collection of the best engineering blogs, essays, and deep dives I've read and want to remember.


{{< resource_archive >}}


---

## Kubernetes on WSL2 and the macOS tunnel

**Date:** 2026-04-01
**URL:** http://localhost:1313/kubernetes-on-wsl2-macos-access/
**Description:** A pragmatic guide to running k3s inside WSL2 on a Windows gaming PC and accessing it securely from a MacBook. No LAN exposure, no fragile port proxies: just SSH and control-plane tunneling.


**TL;DR:** I run **k3s** on a Windows gaming PC via **WSL2** to master Kubernetes networking without cloud costs. This guide details a secure access model using **SSH** and **kubectl port-forward** to bypass NAT boundaries and maintain technical sovereignty.

---

This guide omits Kubernetes feature tutorials. Instead, it details how to architect an environment where Kubernetes can exist without auxiliary hardware, idle cloud bills, or hidden networking realities.

I wanted to learn Kubernetes properly, which meant understanding how **scheduling**, **access paths**, and **network boundaries** behave under real constraints. The problem was simple: I lacked a reasonable execution environment.

## Why my cloud server is full

I maintain a crowded cloud server running a **Grafana observability stack**, **ClickHouse**, and **Couchbase**. While this setup is robust, the machine is effectively full. Mixing orchestration models on a single host would create unnecessary **debugging overhead** and risk the stability of existing services. Consequently, adding Kubernetes there would violate my goal of learning the system in isolation.

## Why a dedicated server was overkill

I evaluated several alternatives, but most provided poor value. A **Mac Mini** or **Intel NUC** is expensive for the performance offered, while **Raspberry Pi clusters** are underpowered and diverge too far from production hardware. Furthermore, used datacenter servers are impractical due to noise and power consumption.

My gaming PC, however, has a modern CPU, high-density RAM, and fast NVMe storage. Although it runs Windows, its raw capacity makes it an ideal Kubernetes host.

## Why WSL2 was the least-bad option

On Windows, the choice is between a full **Linux VM** or **WSL2**. I chose WSL2 because a full VM adds a redundant virtualization layer and higher overhead, whereas WSL2 provides a **real Linux kernel** with near‑native performance.

Crucially, WSL2 is a Linux VM with a specific networking model. This detail is critical for cross-device access.

{{< details title="The 9P Boundary" label="Deep Dive //" >}}
As I wrote in [The WSL2 performance tax](/posts/wsl2-performance-tax-go-windows/), the **9P protocol** bridges the Linux guest and Windows host. While it is efficient for file I/O, it does not resolve the **NAT boundary**. Because WSL2 sits in its own virtual network, its IP is not reachable from the LAN.
{{< /details >}}

## Why k3s was the correct engine

I avoided **Docker Desktop**, **Minikube**, and **MicroK8s** because these tools often hide technical complexity or add unnecessary layers. Instead, I chose **k3s** because it is **CNCF‑certified**, uses upstream components, and runs efficiently in constrained environments. It behaves like a production cluster without the resource tax.

## Installing k3s in a single command

Installing k3s inside WSL2 is trivial:

```bash
curl -sfL https://get.k3s.io | sh -
```

This command starts the **control plane** and **kubelet**, then generates a valid kubeconfig at `/etc/rancher/k3s/k3s.yaml`.

## WSL2 is trapped behind Windows NAT

My primary machine is a **MacBook Air**. It is where I write manifests and run `kubectl`, yet the cluster lives inside WSL2, which sits behind Windows NAT. This architecture creates three friction points:
1. **WSL2 IPs** are not reachable from the LAN.
2. **NodePorts** fail to resolve because the node is not LAN-addressable.
3. **Firewall rules** on Windows manage permission but do not establish routing into the WSL2 subsystem.

## Tunneling through the control plane

Rather than fighting WSL2 networking, I settled on a model that mirrors private production clusters. Nothing is exposed to the LAN; instead, we secure the **Kubernetes API** and tunnel application traffic through the control plane using **SSH** and `kubectl port-forward`.

## Setting up the Windows SSH bridge

The first step is enabling **OpenSSH Server** on Windows. Run these commands in PowerShell as Administrator:

```powershell
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Start-Service sshd
Set-Service sshd -StartupType Automatic
```

Then, add a firewall rule to restrict SSH to **Private** networks:

```powershell
New-NetFirewallRule \
  -Name "OpenSSH-Server-In-TCP" \
  -DisplayName "OpenSSH Server (Private)" \
  -Enabled True \
  -Direction Inbound \
  -Protocol TCP \
  -Action Allow \
  -LocalPort 22 \
  -Profile Private
```

At this stage, the only exposed port is **22**. Kubernetes remains private.

## Mapping the cluster to macOS

From WSL2, retrieve the kubeconfig: `sudo cat /etc/rancher/k3s/k3s.yaml`. Copy this file to the Mac at `~/.kube/config`, then update the **server address** to use localhost:

```yaml
server: https://127.0.0.1:6443
```

Keep all other certificates and keys unchanged.

## Securing the API endpoint

From macOS, create an **SSH tunnel**:

```bash
ssh -N -L 6443:127.0.0.1:6443 <windows-user>@<windows-ip>
```

This forwards the **Kubernetes API** securely from WSL2 to the Mac's localhost. With the tunnel active, `kubectl get nodes` and `kubectl get pods -A` function normally.

## Why NodePorts fail on WSL2

**NodePort** services assume nodes have routable IPs reachable by clients, but WSL2 violates this assumption. The node lives behind NAT, and Windows does not forward arbitrary ports into the Linux guest. Opening firewall ports only grants **permission**; it does not establish **routing**. This behavior is not a Kubernetes bug, but rather a mismatch between environment reality and service expectations.

## Forwarding application traffic

Instead of NodePorts, use `kubectl port-forward` for service access:

```bash
kubectl port-forward svc/app1 8081:8081
```

The service is now available at `http://localhost:8081` on the Mac. **Important:** Even if the Service defines `nodePort: 30081`, `port-forward` ignores it because it tunnels directly to the **Service port** and the **Pod targetPort**. This is why attempting to forward 30081 fails.

## How traffic flows through the tunnel

When `port-forward` is active, traffic flows from the Mac to `kubectl`, through the **Kubernetes API** over **mTLS**, to the `kubelet`, and finally to the Pod port. This is an **application-layer tunnel**, meaning no node-level networking is involved.

## Why this setup is actually secure

The Kubernetes API is not exposed to the LAN. **SSH** is the only entry point, and it can be restricted to a single client IP. Because Kubernetes authentication remains the primary gatekeeper and application access is bound strictly to localhost, this is more secure than exposing NodePorts or using fragile Windows port‑proxy rules.

## The maintenance tax

Every architectural decision has a cost:
*   **Persistent Tunneling:** You must maintain an active SSH session to interact with the cluster.
*   **DNS Resolution:** Localhost access bypasses standard Kubernetes DNS, meaning you cannot use internal cluster URLs from the Mac browser.
*   **Gaming PC Availability:** The cluster is only reachable when the host PC is awake and the WSL2 subsystem is running.

## Kubernetes is a system of constraints

This setup allowed me to learn Kubernetes without auxiliary hardware or cloud overhead. It forced me to understand **networking boundaries** instead of bypassing them with convenience tools. Ultimately, it made Kubernetes less like "Docker with YAML" and more like the **distributed system** it actually is.

---

{{< newsletter >}}


---

## ClickHouse data masking with regex

**Date:** 2026-03-31
**URL:** http://localhost:1313/clickhouse-regex-data-masking/
**Description:** Ingesting PII is easy; cleaning it up is hard. Here is how to use ClickHouse's data masking policies and regex to redact sensitive logs in flight without killing your query performance.


If you're running a production observability stack, you've already leaked PII. An engineer forgot to redact an email in a log line, or a JWT token ended up in a stack trace. 

In most databases, the only fix is to `DELETE` the data—killing your metrics along with the sensitive info. But ClickHouse has a more elegant approach: **Data Masking Policies**.

By defining a policy at the role level, you can use regex to swap sensitive patterns with `[REDACTED]` in real-time, before the query results even leave the server.

---

{{< details title="How Regex Masking works" label="Glossary //" >}}
ClickHouse uses the **re2** engine for regex. When a masking policy is applied, the query planner injects a `regexpReplace` function into the `SELECT` path. 

This means your raw data remains on disk untouched (preserving its checksums and history), but the "View" layer ensures unauthorized users only see the masked version.
{{< /details >}}

---

## Defining the policy

The core of the "Clean Room" strategy is identifying the patterns that shouldn't be there. Let's look at a policy for redacting emails and IPv4 addresses using `multiRegexpReplace`.

{{< code lang="sql" title="Masking Policy with Regex" >}}
-- 1. Create the policy
CREATE MASKING POLICY email_masking ON logs.payload
USING (
    multiRegexpReplace(
        payload,
        ['[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}', '\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}'],
        ['[EMAIL_REDACTED]', '[IP_REDACTED]']
    )
);

-- 2. Apply it to an analyst role
GRANT SELECT ON logs.payload MASKING POLICY email_masking TO analyst_role;
{{< /code >}}

{{< alert type="important" >}}
Notice the use of `multiRegexpReplace`. It is significantly faster than chaining multiple `regexpReplace` calls because it traverses the string once for all patterns, rather than rescanning the entire string for every individual regex.
{{< /alert >}}

---

## The Throughput Penalty: A Billion Row Tax

Regex is computationally expensive. When you apply a masking policy, you are effectively adding a "CPU tax" to every `SELECT` query. If your query is scanning a billion rows, the regex engine has to run a billion times. 

In a high-throughput OLAP system like ClickHouse, where we expect sub-second aggregations, a "busy" regex can turn a 100ms query into a 10-second wait.

Here is how to optimize your patterns to keep your "beast" status.

### 1. Anchors are your friend

If you know the sensitive data always appears after a specific key in a JSON log, anchor your regex. Without anchors, the engine has to attempt a match at every single character position in the string.

{{< code lang="text" title="Regex Anchoring" >}}
# BAD: Scans every character for a potential email
[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

# GOOD: Only looks for patterns following the "user_email" key
(?i)"user_email"\s*:\s*"([^"]+)"
{{< /code >}}

By using the key as an anchor, the RE2 engine can quickly skip over irrelevant parts of the log line using fast substring matching (like Boyer-Moore) before even engaging the complex regex state machine.

### 2. Avoid back-tracking and "The Greediness Trap"

ClickHouse uses **RE2**, which is designed to prevent "catastrophic backtracking" (a common vulnerability in other engines like PCRE). However, RE2 still pays a price for ambiguity.

The most common performance killer is the `.*` operator when used inside a capture group.

{{< code lang="text" title="Avoiding Ambiguity" >}}
# DANGEROUS: Highly ambiguous, forces the engine to explore many paths
ID:.*Email:.*

# FAST: Uses negated character classes to limit the search space
ID:[^,]+,Email:[^,\s]+
{{< /code >}}

By being explicit about what *cannot* be in the match (e.g., `[^,]+` means "anything that isn't a comma"), you provide the engine with a clear "stop" signal, reducing the number of states the machine has to track.

---

## Benchmarking the Tax

If you are deciding whether to mask at the database level or the application level, consider these numbers (improvised based on typical RE2 behavior):

| Strategy | Rows/sec | Latency (p99) |
| :--- | :--- | :--- |
| Raw SELECT (No Masking) | 500M | 80ms |
| Anchored Regex Masking | 120M | 450ms |
| Unanchored/Naive Regex | 15M | 4200ms |

**The takeaway:** A poorly written regex is 30x slower than an optimized one. If your masking policy is poorly designed, it doesn't just protect data; it effectively DOSes your own database.

---

## Closing thoughts

Data masking isn't a replacement for proper sanitization at the edge. But in the real world, things leak. ClickHouse's regex masking gives you a safety net that protects your data without requiring a full database wipe. 

The goal of a senior engineer isn't just to "make it work", but to "make it work within the constraints of the hardware". 

Stay slow and steady. Secure your logs, but don't kill your throughput.

---

{{< newsletter >}}


---

## Understanding CPU Caches in Go

**Date:** 2026-03-31
**URL:** http://localhost:1313/cpu-caches-in-go/
**Description:** A practical guide to understanding how CPU caches (L1/L2/L3) impact Go service performance, with benchmarks on modern hardware.


When you're building Go services that handle millions of operations per second, the hardware beneath your abstractions starts to matter. Specifically, the CPU cache hierarchy, and whether your data fits in it.

---

#### The Hardware Context: It's Not Just "RAM"

Your server has 32GB or 64GB of RAM, but the CPU avoids touching it whenever possible. Instead, it works through a chain of caches:

- **L1 Cache (Instruction & Data)**: Tiny (~64KB data per core), near-instant access (1–4 cycles).
- **L2 Cache**: Larger (1–4MB per core), still extremely fast, your true private workspace.
- **L3 / Last Level Cache (LLC)**: Shared across cores, much larger (32MB+), but noticeably slower.

On modern ARM server chips (Ampere Altra, AWS Graviton) and Apple M-series silicon, **L2 is your most valuable per-core resource**. If your Go service's active working set fits in L2, it runs fast. Spill into L3 or RAM, and you hit a latency cliff.

The practical question isn't "do you know what a cache is?" It's: **what does modern hardware actually penalize you for?**

---

#### What Modern Hardware Gets Right (And What It Doesn't)

I ran benchmarks on an Apple M4 to find out. The results changed some assumptions.

##### Sequential vs. Stride Access: The Prefetcher Wins

The classic cache benchmark is matrix traversal: iterating a 1000×1000 integer matrix by row (sequential, cache-friendly) vs. by column (jumping 8KB every step, cache-hostile). On older CPUs, column traversal is reliably ~10× slower.

On the M4:

```text
BenchmarkMatrixRowTraversal     236,982 ns/op
BenchmarkMatrixColTraversal     239,203 ns/op
```

Identical. The hardware prefetcher detected the stride pattern and pre-fetched the next chunk before the CPU needed it, hiding the latency entirely.

This holds for stride-walk benchmarks too, even 64KB jumps don't fool it:

```text
BenchmarkStrideWalk/Stride_64         0.30 ns/op
BenchmarkStrideWalk/Stride_65536      0.28 ns/op
```

**Takeaway**: Modern CPUs are remarkably good at hiding predictable access patterns. Simple, linear data structures (slices, arrays) are still best, but you probably don't need to hand-tune memory layout unless profiling shows a real bottleneck. If you want to benchmark true cache misses, use **pointer-chasing** (random linked-list traversal), where the CPU cannot predict the next address until it loads the current one.

---

#### What Still Kills Performance: Shared Mutable State

This is where the latency cliff is real, and where Go developers get burned.

When multiple cores read and write the same memory location, even through atomic operations, they trigger the CPU's cache coherence protocol (MESI). Every write on one core must invalidate copies on every other core. The more cores fighting over the same cache line, the worse it gets.

I measured this directly with adjacent atomic counters under concurrent load:

| Scenario | Latency |
|---|---|
| Single core, no contention | ~2.2 ns/op |
| 8 cores, high contention | ~28.4 ns/op |

**A 12× slowdown**, not from algorithmic complexity, but from cores arguing over a 64-byte chunk of memory.

This is the real enemy in high-throughput Go services, a shared `var RequestCount int64` or a single global mutex protecting a hot map.

##### Fix 1: Shard your counters

Don't use a single shared counter. Map goroutines or connections to buckets:

```go
type ShardedCounter struct {
    counts [128]struct {
        n   int64
        _   [56]byte // pad to 64-byte cache line
    }
}

func (c *ShardedCounter) Inc(shard int) {
    atomic.AddInt64(&c.counts[shard%128].n, 1)
}
```

Each shard lives on its own cache line. Cores no longer fight.

##### Fix 2: Pad hot struct fields

If two fields in a struct are written by different goroutines, put padding between them so they land on separate cache lines:

```go
type HotStruct struct {
    FieldA int64
    _      [56]byte // force onto separate cache line
    FieldB int64
}
```

Without the padding, `FieldA` and `FieldB` share a line. A write to one invalidates the other on every other core.

##### Fix 3: Prefer channels over shared memory

Go's channel semantics transfer *ownership*, not just a value. The sending goroutine gives up its reference; the receiver takes over. This is exactly what cache coherence wants, one writer at a time, and it's why idiomatic Go with channels often outperforms lock-heavy designs even when the channel has overhead.

---

#### Keep Your Working Set in L2

The 12× contention penalty is dramatic, but the more insidious problem in real services is simply having a working set that's too large.

If a request handler touches 10MB of data, a large in-process cache, a bloated context struct, a deep call chain with big allocations, that data spills out of L2 and into L3 or RAM on every request. You won't see a single expensive operation; you'll see uniformly elevated tail latency under load.

Practical rules:

- **Keep per-connection and per-request structs small.** Prefer slices of IDs over slices of full objects when you don't need all the fields.
- **Avoid large in-process caches for hot-path data.** A 50MB LRU cache sounds helpful, but if it exceeds your LLC, you're paying RAM latency for every miss.
- **Use `sync.Pool` for short-lived allocations** to reduce GC pressure and improve locality, reused objects tend to stay in cache.

---

#### Measuring It Yourself

To see cache effects in your own service, the most useful tools are:

**`go test -bench`** for microbenchmarks. Run with `-cpu=1,4,8` to see how contention scales:

```bash
go test -bench=. -benchmem -cpu=1,4,8 .
```

**Linux `perf`** for hardware counters on production workloads:

```bash
perf stat -e cache-misses,cache-references,LLC-load-misses ./your-binary
```

High LLC (Last Level Cache) miss rates, especially above 5–10%, are the clearest signal that your working set is exceeding the cache. That's where to optimize first.

I've added a benchmark suite to experiment with these patterns yourself.
{{< download file="cache_test.go" label="Download cache_test.go" >}}

---

#### Summary

Modern hardware prefetchers have largely eliminated the penalties for predictable access patterns. The naive "row vs. column traversal" benchmarks from ten years ago often don't apply anymore.

What *does* still matter, and significantly:

1. **Shared mutable state across cores**, cache coherence contention is a 10× problem, not a 2× problem. Shard counters, pad structs, prefer channels.
2. **Working set size**, if your per-request data footprint exceeds L2 (~1–4MB per core), you're paying LLC or RAM latency on every operation. Keep hot structures lean.
3. **Measure before optimizing**, use `perf` to look for LLC misses. A high miss rate is the smoking gun; a low one means your bottleneck is somewhere else entirely.

---

## ClickHouse vs. Postgres: When to Move Your Logs Out of a Relational DB

**Date:** 2026-03-30
**URL:** http://localhost:1313/clickhouse-vs-postgres-log-storage/
**Description:** Postgres is the swiss-army knife of databases, but when you hit the 10-million row mark for write-heavy logs, the relational wall becomes real. Here is why we moved our observability stack to ClickHouse.


Postgres is the most reliable tool in my stack. It handles users, configurations, and complex relations without breaking a sweat. But databases, like any physical system, have a "design limit". For Postgres, that limit usually appears when you try to use it as a dumping ground for high-velocity logs and metrics.

When your `ANALYZE` commands start taking minutes and your indexes consume more RAM than your data, you aren't facing a Postgres bug; you're facing an architectural mismatch.

## The Write-Heavy Wall

Relational databases are designed for **OLTP** (Online Transactional Processing). They prioritize consistency, ACID compliance, and point-lookups. Logs, however, are **OLAP** (Online Analytical Processing) data. They are immutable, sequential, and usually queried in broad sweeps rather than by individual IDs.

In Postgres, every new log entry must be written to the Write-Ahead Log (WAL) and then inserted into a B-Tree index. As the table grows, the cost of maintaining those B-Trees during high-concurrency writes creates a "latency floor" that you simply cannot drop below.

{{< details title="B-Tree vs. LSM-Tree" label="Deep Dive //" >}}
**B-Trees (Postgres):** Optimized for reading small amounts of data quickly. Every write requires finding the correct spot in a tree, which can lead to random disk I/O as the tree grows larger than memory.

**LSM-Trees (ClickHouse):** Optimized for high-volume writes. It appends data to sorted "parts" in memory and flushes them to disk sequentially. Background "merges" combine these parts later. This turns random I/O into sequential I/O, which is where modern SSDs (and even HDDs) truly shine.
{{< /details >}}

### The "Write Amplification" Problem

In Postgres, if you have a table with 5 indexes, every single `INSERT` must update those 5 separate B-Trees. This is **Write Amplification**. For a log-heavy system processing 5,000 events/sec, your disk is working 5x harder just to keep the indexes "ready", even if you only query those logs once a week.

---

## Why ClickHouse?

Moving logs to ClickHouse isn't just about "faster writes". It's about changing how you think about data storage.

### 1. Columnar Compression

In Postgres, data is stored in rows. If you have 50 columns but only query 2 (`timestamp` and `error_code`), Postgres still has to read the whole row from disk. ClickHouse stores each column in its own file.

{{< alert type="tip" >}}
Because similar data is stored together in columns (e.g., thousands of "200 OK" strings), ClickHouse can use specialized algorithms like **ZSTD** or **Delta encoding** to achieve 10x to 100x compression ratios.
{{< /alert >}}

### 2. The Power of Materialized Views

One of ClickHouse's "superpowers" is the ability to aggregate data _on ingestion_. Instead of running a heavy `COUNT(*)` query every time you refresh a dashboard, ClickHouse can maintain a running total in a background table as the data arrives using the `SummingMergeTree` or `AggregatingMergeTree` engines.

{{< code lang="sql" title="Example: Continuous Aggregation" >}}
CREATE MATERIALIZED VIEW hourly_errors_mv
ENGINE = SummingMergeTree
AS SELECT
toStartOfHour(timestamp) as hour,
count() as error_count
FROM logs
WHERE status >= 400
GROUP BY hour;
{{< /code >}}

---

I don't want to make this sound like a free lunch. Moving to ClickHouse introduces "Operational Tax":

1.  **No Real Updates:** ClickHouse is for immutable data. `UPDATE` and `DELETE` commands exist (as `ALTER TABLE ... UPDATE`), but they are heavy, asynchronous background operations. If you need to frequently edit your data, stay with Postgres.
2.  **Join Performance:** While ClickHouse _can_ do joins, it isn't its primary strength. It prefers large, "denormalized" tables.
3.  **Consistency Models:** Postgres gives you strict ACID. ClickHouse is "eventually consistent" during background merges. For logs, this is fine. For a bank balance, it is a disaster.

If your logs table is under 5 million rows and your write rate is low, **stay with Postgres**. The operational simplicity of having one database outweighs the performance gains of ClickHouse.

However, if you're seeing any of the following, it's time to move:

- `VACUUM` processes are struggling to keep up with dead tuples (bloat).
- Your log indexes are larger than the actual log data.
- Simple aggregations (like "average response time per hour") are timing out.
- You are spending more on RDS storage than on actual compute.

The goal is to use the right tool for the job. Postgres remains my "Source of Truth" for application state: users, sessions, and relationships. But for the firehose of observability data, ClickHouse has reclaimed our performance headroom and slashed our storage bills.

---

{{< newsletter >}}


---

## WSL2 Is Slow? Fix /mnt/c/ File System Performance & Go Latency

**Date:** 2026-03-29
**URL:** http://localhost:1313/wsl2-performance-tax-go-windows/
**Description:** Why is WSL2 so slow? Understand the 9P filesystem boundary, why /mnt/c/ degrades Go build times, and how to fix WSL2 performance bottlenecks on Windows.


WSL2 (Windows Subsystem for Linux) has been a godsend for developers who love Linux tools but need/have a Windows environment. But for Go developers, WSL2 isn't just a "transparent layer". If configured incorrectly, it becomes the bottleneck that can slow down builds by 3x and introduce mysterious latency in networked services.

This isn't a failure of WSL2; it's a failure of understanding the **9p boundary**.

{{< details title="What is the 9P Boundary?" label="Glossary //" >}}
WSL2 is a Virtual Machine. To let that VM see your Windows files, Microsoft uses the **9P protocol** (Plan 9).

When you access `/mnt/c/`, you aren't talking to the disk directly; you're acting as a network client. Every file read requires a round-trip across the Hyper-V socket to the Windows host. For Go, which reads thousands of small files during compilation, this latency is catastrophic.
{{< /details >}}

## The Performance Cliff: /mnt/c/

The most common mistake I see is developers (including me) keeping their Go projects in `C:\Users\Name\Projects` and accessing them via `/mnt/c/Users/...` inside WSL.

When you run `go build` on a project located in the Windows filesystem:

1.  Go makes thousands of syscalls to read source files.
2.  WSL2 must translate these Linux syscalls into Windows I/O via the **9p protocol**.
3.  Each translation adds microseconds of latency.

For a project with 50 dependencies, those microseconds compound into seconds.

### The Benchmarkc

| Filesystem Location | `go build` Time | I/O Throughput |
| :------------------ | :-------------- | :------------- |
| Windows (/mnt/c/)   | 12.4s           | ~40 MB/s       |
| Linux Native (~)    | 3.2s            | ~800 MB/s      |

**The takeaway:** You are paying a 4x build-time tax just for the convenience of using Windows Explorer.

---

## Tweak 1: Move to the Native Filesystem

The solution is simple but often ignored: **Clone your code into the Linux home directory (`~/`)**.

{{< terminal >}}

# BAD

cd /mnt/c/Users/vikash/projects/myapp

# GOOD

cd ~/projects/myapp
{{< /terminal >}}

If you need to access these files from Windows, use the built-in WSL share: `\\wsl$\Ubuntu\home\user\projects`.

## Tweak 2: The .wslconfig Memory Buffer

By default, WSL2 can be greedy with memory but slow to release it. For Go's memory-intensive compilation, you should constrain WSL to prevent Windows from swapping to disk.

Create or edit `C:\Users\<You>\.wslconfig`:

{{< code lang="ini" file=".wslconfig" >}}
[wsl2]
memory=8GB # Limits VM memory to 8GB
processors=4 # Limits VM to 4 cores
{{< /code >}}

## Tweak 3: Windows Defender Exclusion

Windows Defender scans every file access. When Go builds, it creates thousands of small temporary files in `/tmp`. Defender sees this as suspicious activity and hooks into every process.

**The Fix:** Add an exclusion in Windows Security for the `vmmem` process and your WSL storage location (usually `%USERPROFILE%\AppData\Local\Packages\...\LocalState\ext4.vhdx`).

---

## Conclusion

WSL2 is a high-performance environment, but it respects the laws of physics. If you force it to cross the boundary between two disparate filesystems on every build, it will slow down.

Move your code to Linux, tune your config, and get back to writing Go.

---

{{< newsletter >}}


---

## AI Coding in 2026: Productivity Multipliers vs. Skill Replacements

**Date:** 2026-03-28
**URL:** http://localhost:1313/ai-coding-productivity-vs-skill/
**Description:** Two years of coding with AI every day: what actually saves time, what creates hidden debt, and why your fundamentals are your only real defense against hallucinated correctness.


"Write a REST API with authentication".

I hit enter. Thirty seconds later, Claude spat out 400 lines of perfectly formatted Go code. JWT middleware, password hashing, error handling, even rate limiting. It looked professional. It looked production-ready.

It took me three hours to figure out why the token refresh logic had a race condition.

This is AI-assisted coding in 2026. It's not magic. It's not a replacement. It's a very fast junior developer that never gets tired, never complains, and confidently makes mistakes you won't catch unless you actually understand the **architectural nuances** of what you're building.

I've spent two years using AI coding tools daily. From Aider and Antigravity to Claude Code. I've used them for everything from refactoring legacy monoliths to building new microservices from scratch. I've saved hundreds of hours, and I've lost dozens more debugging "hallucinated correctness".

This isn't a tutorial on transformers. This is a forensic report on what actually works, what fails, and how to use these tools without increasing the **cognitive entropy** of your codebase.


{{< details title="Probabilistic vs. Deterministic" label="Glossary //" >}}
**Probabilistic (AI):** LLMs predict the "most likely" next token based on statistical patterns. They don't "know" if code works; they know it "looks like" code that works.

**Deterministic (Engineering):** Compilers and tests follow strict logical rules. A program either satisfies its constraints or it doesn't. 

Engineering with AI is the art of using a probabilistic tool to generate deterministic results.
{{< /details >}}

---

## The mirage of understanding

Before we talk about tools, we must address the fundamental truth: **AI coding tools lack grounded understanding.** They are sophisticated pattern-matching engines. They excel at well-trodden paths (CRUD apps, standard algorithms) but struggle with novelty or deep domain logic.

When you ask an AI to "write a function that does X", you are essentially performing a high-speed search across billions of lines of code. 

**The Amazing:** Need boilerplate database migrations? Test scaffolding? API endpoints that follow standard patterns? AI crushes these. It has seen the "correct" way ten thousand times.

**The Dangerous:** Need to debug a distributed race condition? Optimize a slow query in a complex join? AI is mediocre at best. It often suggests "plausible but wrong" solutions that require a senior engineer to debunk.

{{< alert type="important" >}}
**The "Autocomplete" Fallacy:** I stopped thinking of AI as "intelligence" and started thinking of it as "autocomplete on steroids". It suggests the most likely implementation, not the most correct one.
{{< /alert >}}

---

## The forensic workflow

I've tried most major tools. Here is the stack that actually survived the transition from hype to production.

### The Stack
- **Aider + Claude Opus 4.5 API:** My primary tool for surgical edits. It is git-aware and allows me to control the context window manually.
- **Gemini CLI:** For high-speed reasoning and massive context handling when bridging multiple architectural layers.
- **Cline:** For end-to-end task delegation and autonomous research.
- **Antigravity:** For exploratory work where I need the agent to find relevant files itself.
- **Requirement.md:** I write down the architecture, goals, and constraints in a Markdown file *before* prompting. This forces me to solve the problem in my head first.

### The Process: A Real-World Example
**Task:** Add rate limiting middleware to a Go service.

1.  **Specification:** I write a `ratelimit.md` defining the 100 req/min limit, the 429 response, and the Redis backend.
2.  **Context Priming:** I add existing middleware to the chat so the AI learns our specific error handling patterns.
3.  **Generation:** Aider produces the code in ~45 seconds.
4.  **Forensic Review:** I read every single line. I'm not looking for syntax errors (the compiler finds those); I'm looking for **logical drift**.

{{< alert type="tip" >}}
**The Review Ratio:** AI didn't save me 5 hours. It saved me 4 hours of typing and cost me 1 hour of intense review. The net win is 3 hours, but only if that 1 hour of review is higher quality than the original typing.
{{< /alert >}}

---

## Where AI nails it

Two years in, I've mapped the "Safe Zones" for AI assistance:

1.  **Boilerplate & Patterns:** "Generate CRUD handlers for this model". (95% accuracy)
2.  **Code Translation:** "Convert this Python utility to idiomatic Go". (80% accuracy)
3.  **Test Scaffolding:** "Generate table-driven tests for this parser". (90% accuracy)
4.  **Documentation:** "Add docstrings explaining the concurrency model here". (100% accuracy for description)

{{< details title="The Context Boundary" label="Deep Dive //" >}}
In physics, the **9P protocol** (used by WSL2) adds a latency tax when crossing the Windows/Linux boundary. 

In AI development, there is a similar **Context Tax**. The larger your codebase, the more "noise" enters the context window. High-performance AI coding requires **Surgical Context Management**: only showing the AI the exact files it needs to solve the immediate problem.
{{< /details >}}

---

## The hallucination minefield

This is where the "Senior" in Senior Engineer becomes non-negotiable.

### 1. Confident Misinformation
AI confidently invents APIs that don't exist.
```go
// AI suggested this for a Redis client:
client.IncrementWithExpiry("key", 60) 
// Reality: This method doesn't exist in 'go-redis'. 
// You need an atomic Lua script or multi-exec.
```

### 2. Concurrency Blindness
AI writes code that works in a single-threaded test but fails in a high-concurrency production environment. It often "forgets" that maps aren't thread-safe or that `atomic.Add` is required for global counters.

{{< alert type="warning" >}}
**The "Looks Good, Doesn't Work" Bug:** Syntactically perfect code with logical flaws is the hardest bug to find. AI specializes in this.
{{< /alert >}}

---

## Skills that matter now

The developer skill set has shifted. If you memorize syntax, you are competing with a machine that has already won. If you understand **Systems Thinking**, you are the machine's pilot.

### What Matters More (Way More)
1.  **Code Reading:** You are now a professional reviewer. Can you spot a race condition in 500 lines of generated code?
2.  **System Design:** AI can implement a feature, but it can't decide if you need a microservice or a monolith.
3.  **Deterministic Validation:** Writing benchmarks and stress tests to prove the AI's probabilistic output is correct.
4.  **Precision Communication:** The quality of your output is a direct function of the precision of your prompt.

---

## The force multiplier

AI is not replacing developers. Not at least for half a decade, but it is raising the bar for what a developer is expected to deliver.

If you understand the underlying mechanics, including memory management, the cache hierarchy, and database trade-offs, AI becomes a high-speed tool that lets you build faster than ever. If you use it to skip learning those things, you are building a house on shifting sand.

**The Rule:** Use AI cautiously, verify everything, own what you ship, and never stop learning the fundamentals.

---

{{< newsletter >}}


---

## Fast Docker CI: Stop Rebuilding Container Images on Every Commit

**Date:** 2026-02-15
**URL:** http://localhost:1313/stop-rebuilding-docker-images-runtime-containers/
**Description:** Rebuilding multi-stage Docker images on every code change wastes minutes and cache. How to decouple build dependencies from runtime containers for instant feedback.


I used to rebuild my Docker image every time I fixed a typo. A one-line change meant waiting 2-3 minutes for Docker to rebuild layers, reinstall dependencies, and restart the container. I thought this was just the cost of containerized development.

Then I discovered runtime containers. Same isolated environment, same reproducibility, but code changes reflect instantly. No rebuilds. No waiting. The runtime lives in the container, the code lives on the host.

This pattern isn't new, but it's criminally underused. Most developers either suffer through slow rebuild cycles or abandon containers for local development entirely. Both approaches miss the point: you can have containerized consistency AND development speed.

This guide covers the what, why, and how of runtime containers. When they're perfect, when they're terrible, and how to build production-quality containerized development workflows.

## Why most containerized development is painfully slow

Consider a typical Python web application. Your Dockerfile looks like this:

{{< code lang="dockerfile" file="Dockerfile" >}}
FROM python:3.11-slim

WORKDIR /app

# Install dependencies

COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy application code

COPY . .

# Run the application

CMD ["python", "app.py"]
{{< /code >}}

This works. But every code change requires:

1. Rebuild the image (`docker build -t myapp .`)
2. Stop the old container
3. Start a new container from the new image
4. Wait for the application to initialize

Even with layer caching, this takes 30 seconds to 2 minutes. Fix a typo, wait 90 seconds. Adjust a log statement, wait 90 seconds. You're spending more time waiting than coding.

The standard "solution" is to use bind mounts in development:

{{< code lang="bash" >}}
docker run -v $(pwd):/app myapp
{{< /code >}}

But developers rarely do this correctly. They mount code over an image that already has code baked in. The image is still bloated with stale code. Layer caching is still broken by code changes. The workflow is still hybrid and confusing.

{{< alert type="important" >}}
Runtime containers solve this properly. Build an image with _only_ the runtime and dependencies. Mount your code at runtime. Separate concerns cleanly.
{{< /alert >}}

## What is a runtime container?

A runtime container is an image that contains everything needed to execute your code, except the code itself.

**A runtime container contains:**

- The language runtime (Python interpreter, Node.js, JVM, etc.)
- System dependencies (libraries, build tools)
- Application dependencies (pip packages, npm modules)

**A runtime container does NOT contain:**

- Your application source code
- Configuration files that change frequently
- Data or state

Your code lives on the host machine. You mount it into the container when you run it.

```
+----------------------------------------------------------+
|                    Docker Container                      |
|                                                          |
|  +------------------+        +-----------------------+   |
|  | Runtime & Deps   |  +---> | Mounted: /app/src     |   |
|  | - Python 3.11    |  |     | (from host filesystem)|   |
|  | - pip packages   |  |     +-----------------------+   |
|  | - system libs    |  |                                 |
|  +------------------+  |     +-----------------------+   |
|                        +---> | Mounted: /app/config  |   |
|                              | (from host filesystem)|   |
|                              +-----------------------+   |
+----------------------------------------------------------+
           ^
           |
    Code lives on host, mounted into container at runtime
```

<small>Created with [asciiflow](https://asciiflow.com?ref=lorbic.com)</small>  
<br>

This architecture gives you:

- **Instant code changes**: Edit files on your host, changes reflect immediately in the container
- **Fast rebuilds**: Only rebuild the runtime image when dependencies change, not on every code change
- **Familiar workflow**: Use your local IDE, debugger, and tools
- **True isolation**: Runtime is containerized, ensuring consistency across team members

## Level 1: Your first runtime container

Let's build a runtime container for a Python Flask application.

**Traditional approach (what you're probably doing, what I was doing):**

{{< code lang="dockerfile" file="Dockerfile" >}}
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . . # This bakes your code into the image
CMD ["python", "app.py"]
{{< /code >}}

Every time you change `app.py`, you rebuild the entire image.

**Runtime container approach:**

{{< code lang="dockerfile" file="Dockerfile.runtime" >}}
FROM python:3.11-slim

WORKDIR /app

# Install dependencies (this rarely changes)

COPY requirements.txt .
RUN pip install -r requirements.txt

# DO NOT COPY SOURCE CODE

# Code will be mounted at runtime

CMD ["python", "src/app.py"]
{{< /code >}}

Notice what's missing? No `COPY . .`. Your code isn't baked in.

Build the runtime image once:

{{< code lang="bash" >}}
docker build -f Dockerfile.runtime -t myapp-runtime .
{{< /code >}}

Run it with your code mounted:

{{< code lang="bash" >}}
docker run -v $(pwd)/src:/app/src myapp-runtime
{{< /code >}}

Now edit `src/app.py` on your host machine. If you're using a framework with auto-reload (Flask's debug mode, FastAPI with `--reload`), changes appear instantly. Zero rebuild time.

### Node.js example

The same pattern works for any language runtime.

{{< code lang="dockerfile" file="Dockerfile.runtime" >}}
FROM node:18-slim

WORKDIR /app

# Install dependencies

COPY package.json package-lock.json ./
RUN npm ci --only=production

# DO NOT COPY SOURCE CODE

CMD ["node", "src/server.js"]
{{< /code >}}

Run with mounted code and nodemon for hot-reloading:

{{< code lang="bash" >}}
docker run \
 -v $(pwd)/src:/app/src \
 -e NODE_ENV=development \
 myapp-runtime \
 npx nodemon src/server.js
{{< /code >}}

Edit `src/server.js`, nodemon detects the change, restarts the process. Entire cycle takes milliseconds.

## Level 2: Docker Compose workflows

Managing mount paths and environment variables in `docker run` commands gets tedious with more than 2-3 containers. Docker Compose makes managing multiple containers and their configurations easy for development.

{{< code lang="yaml" file="docker compose.yml" >}}
version: '3.8'

services:
app:
build:
context: .
dockerfile: Dockerfile.runtime
volumes: # Mount source code - ./src:/app/src:ro - ./config:/app/config:ro
environment: - FLASK_ENV=development - FLASK_DEBUG=1
ports: - "5000:5000"
command: flask run --host=0.0.0.0

db:
image: postgres:15-alpine
environment:
POSTGRES_PASSWORD: devpassword
POSTGRES_DB: myapp
volumes: - postgres_data:/var/lib/postgresql/data

volumes:
postgres_data:
{{< /code >}}

Start the entire stack:

{{< code lang="bash" >}}
docker compose up
{{< /code >}}

Your application runs in a container with the Python runtime and dependencies. Your database runs in its own container. Your code is mounted from the host. Edit code, refresh browser. That's it.

### Read-only mounts for safety

Notice `:ro` on the volume mounts? This makes them read-only from inside the container. Your application can't accidentally modify source files. For development this isn't strictly necessary, but it's good practice.

If your app needs to write files (logs, uploads, generated assets), mount those directories separately as read-write:

{{< code lang="yaml" >}}
volumes:

- ./src:/app/src:ro
- ./config:/app/config:ro
- ./logs:/app/logs # Read-write for log output
- ./uploads:/app/uploads # Read-write for user uploads
  {{< /code >}}

### Environment-specific configuration

Development and production often need different settings. Use environment variables to control behavior:

{{< code lang="yaml" file="docker compose.yml" >}}
services:
app:
environment: - FLASK_ENV=development - DATABASE_URL=postgresql://postgres:devpassword@db:5432/myapp - LOG_LEVEL=DEBUG - ENABLE_DEBUGGER=1
{{< /code >}}

Or load from an `.env` file:

{{< code lang="yaml" >}}
services:
app:
env_file: - .env.development
{{< /code >}}

## Level 3: Advanced patterns

### Handling compiled dependencies

Some languages need to compile dependencies (C extensions in Python, native modules in Node.js, cli-tools like ffmpeg). These dependencies are platform-specific. If you're on macOS but deploying to Linux, the compiled cli-tools won't work as is. E.g. on ubuntu 22.04, you can't use the latest ffmpeg (>v4) from apt. macOS brew has the latest version..

**Solution:** Compile dependencies inside the container, store them in a named volume.

{{< code lang="dockerfile" file="Dockerfile.runtime" >}}
FROM python:3.11-slim

WORKDIR /app

# Install system dependencies for compilation

RUN apt-get update && apt-get install -y \
 gcc \
 && rm -rf /var/lib/apt/lists/\*

COPY requirements.txt .
RUN pip install -r requirements.txt

CMD ["python", "src/app.py"]
{{< /code >}}

{{< code lang="yaml" file="docker compose.yml" >}}
services:
app:
volumes: - ./src:/app/src:ro # Anonymous volume to prevent host node_modules from overriding container's - /app/node_modules
{{< /code >}}

The anonymous volume `/app/node_modules` ensures the container uses its own compiled modules, not your host's.

### Multi-stage builds for dev and production

You can use a single Dockerfile for both development (runtime container) and production (baked-in code).

{{< code lang="dockerfile" file="Dockerfile" >}}
FROM python:3.11-slim AS base

WORKDIR /app

# Install dependencies

COPY requirements.txt .
RUN pip install -r requirements.txt

# Development stage: no code, expects it to be mounted

FROM base AS development
CMD ["flask", "run", "--host=0.0.0.0", "--reload"]

# Production stage: code baked in

FROM base AS production
COPY . .
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]
{{< /code >}}

{{< code lang="yaml" file="docker compose.yml" >}}
services:
app:
build:
context: .
target: development # Use the development stage
volumes: - ./src:/app/src:ro
{{< /code >}}

Development: `docker compose up` (uses mounted code from development stage)  
Production: `docker build --target production -t myapp .` (bakes code into production stage)

One Dockerfile, two workflows. Clean separation of concerns.

### Debugging inside containers

Runtime containers make debugging easier because you're using your local IDE and tools. But sometimes you need to debug inside the container itself.

**Python with debugpy:**

{{< code lang="dockerfile" file="Dockerfile.runtime" >}}
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

# Install debugger in development

RUN pip install debugpy

CMD ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678", "--wait-for-client", "src/app.py"]
{{< /code >}}

{{< code lang="yaml" file="docker compose.yml" >}}
services:
app:
ports: - "5000:5000" - "5678:5678" # Debugger port
{{< /code >}}

Connect your IDE's debugger to `localhost:5678`. Set breakpoints in your local files. The debugger works because the code is mounted from the host.

### Watching for dependency changes

You update `requirements.txt` but forget to rebuild the runtime image. Your app crashes with "ModuleNotFoundError". This is frustrating.

**Solution 1: Document the workflow**

Add clear instructions in your README:

```markdown
## Development Workflow

- Edit code in `src/` -> changes reflect instantly
- Change dependencies in `requirements.txt` -> run `docker compose build app`
- Change system packages in Dockerfile -> run `docker compose build app`
```

**Solution 2: Automate with Make**

{{< code lang="makefile" file="Makefile" >}}
.PHONY: dev rebuild deps

dev:
docker compose up

rebuild:
docker compose build
docker compose up

deps: rebuild # Alias for clarity when you add dependencies
{{< /code >}}

Usage:

- `make dev` - start development
- `make deps` - rebuild and restart after changing dependencies

**Solution 3: File watchers**

Use a file watcher to automatically rebuild when dependency files change:

{{< code lang="bash" file="watch-deps.sh" >}}
#!/bin/bash

# Requires entr: brew install entr (macOS) or apt-get install entr (Linux)

ls requirements.txt package.json | entr -r docker compose build
{{< /code >}}

Run this in a separate terminal. When `requirements.txt` or `package.json` changes, it rebuilds automatically.

## Common mistakes and how to avoid them

### Mistake 1: Mounting too much

Don't mount your entire project directory:

{{< code lang="yaml" >}}

# BAD: Mounts everything, including .git, node_modules, **pycache**

volumes:

- .:/app
  {{< /code >}}

This clutters the container with development artifacts, breaks package management, and causes permission issues.

**Solution:** Mount only what you need.

{{< code lang="yaml" >}}

# GOOD: Mount only source and config

volumes:

- ./src:/app/src:ro
- ./config:/app/config:ro
  {{< /code >}}

Add a `.dockerignore` file to prevent accidental copies during builds:

{{< code lang="dockerignore" file=".dockerignore" >}}
**/**pycache**
**/_.pyc
\*\*/.pytest_cache
node_modules
.git
.env
.vscode
.idea
_.log
{{< /code >}}

### Mistake 2: Ignoring file permissions

On Linux, files created by the container might be owned by `root`, making them uneditable on the host.

**Solution:** Run the container as your user.

{{< code lang="bash" >}}
docker run -u $(id -u):$(id -g) -v $(pwd)/src:/app/src myapp
{{< /code >}}

Or set the user in `docker compose.yml`:

{{< code lang="yaml" >}}
services:
app:
user: "${UID:-1000}:${GID:-1000}"
volumes: - ./src:/app/src
{{< /code >}}

### Mistake 3: Forgetting to rebuild when dependencies change

You add a new package to `requirements.txt` but forget to rebuild the runtime image. Your app crashes with "ModuleNotFoundError".

This is the single most common mistake with runtime containers. The container has an old snapshot of your dependencies.

**Solution:** Make rebuilds obvious. Use clear naming conventions, add checks, or automate.

Add a check script that compares dependency files:

{{< code lang="bash" file="check-deps.sh" >}}
#!/bin/bash

if [ requirements.txt -nt .docker-built ] || [ ! -f .docker-built ]; then
echo "requirements.txt changed since last build"
echo "Run: docker compose build"
exit 1
fi
{{< /code >}}

Update your Makefile to track builds:

{{< code lang="makefile" >}}
.PHONY: dev rebuild

dev: .docker-built
docker compose up

rebuild:
docker compose build
touch .docker-built
docker compose up

.docker-built: requirements.txt
docker compose build
touch .docker-built
{{< /code >}}

### Mistake 4: Using runtime containers in CI/CD

Your CI pipeline should NOT mount code into containers. It should build complete, testable images that mirror production.

{{< code lang="yaml" file=".github/workflows/ci.yml" >}}

# BAD: Mounting code in CI

- name: Test
  run: docker run -v $(pwd):/app myapp-runtime pytest

# GOOD: Build a complete image for testing

- name: Build test image
  run: docker build --target production -t myapp:test .
- name: Test
  run: docker run myapp:test pytest
  {{< /code >}}

CI should mimic production as closely as possible. Runtime containers are for development velocity, not deployment.

### Mistake 5: Mixing development and production configurations

Developers sometimes accidentally deploy development images or use development settings in production.

**Solution:** Use clear naming and separate files.

```
Dockerfile              # Production image (code baked in)
Dockerfile.runtime      # Development runtime (no code)
docker compose.yml      # Development environment
docker compose.prod.yml # Production environment
```

Never deploy anything with `.runtime` or `development` in the name.

## When NOT to use runtime containers

Runtime containers aren't universally better. They have specific use cases.

**Use runtime containers when:**

- You're actively developing and iterating on code
- You want instant feedback without rebuilds
- Your team uses different operating systems
- You need consistent development environments

**Don't use runtime containers when:**

- Building for production (bake code in)
- Running CI/CD tests (build complete images)
- Code and dependencies change together frequently
- You're working on a compiled language with slow compilation (Go, Rust, C++)

For compiled languages, the compilation step is the bottleneck, not Docker builds. Runtime containers don't help much there.

## Real-world example: Polyglot microservices

Here's a complete setup for a Python API backend + Node.js frontend, both using runtime containers.

{{< code lang="yaml" file="docker compose.yml" >}}
version: '3.8'

services:

# Python API backend

api:
build:
context: ./backend
dockerfile: Dockerfile.runtime
volumes: - ./backend/src:/app/src:ro - ./backend/tests:/app/tests:ro
environment: - DATABASE_URL=postgresql://postgres:password@db:5432/myapp - FLASK_ENV=development - FLASK_DEBUG=1
ports: - "5000:5000"
depends_on: - db
command: flask run --host=0.0.0.0

# Node.js frontend

frontend:
build:
context: ./frontend
dockerfile: Dockerfile.runtime
volumes: - ./frontend/src:/app/src:ro - ./frontend/public:/app/public:ro - /app/node_modules # Anonymous volume for dependencies
environment: - NODE_ENV=development - REACT_APP_API_URL=http://localhost:5000
ports: - "3000:3000"
command: npm start

# PostgreSQL database

db:
image: postgres:15-alpine
environment:
POSTGRES_PASSWORD: password
POSTGRES_DB: myapp
volumes: - postgres_data:/var/lib/postgresql/data
ports: - "5432:5432"

volumes:
postgres_data:
{{< /code >}}

**backend/Dockerfile.runtime:**

{{< code lang="dockerfile" >}}
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

CMD ["flask", "run", "--host=0.0.0.0"]
{{< /code >}}

**frontend/Dockerfile.runtime:**

{{< code lang="dockerfile" >}}
FROM node:18-slim

WORKDIR /app

COPY package\*.json ./
RUN npm ci

CMD ["npm", "start"]
{{< /code >}}

Start everything:

{{< code lang="bash" >}}
docker compose up
{{< /code >}}

The workflow:

- Edit Python code -> Flask auto-reloads
- Edit React code -> Webpack auto-reloads
- Add Python dependency -> `docker compose build api && docker compose up`
- Add Node dependency -> `docker compose build frontend && docker compose up`

Both services have isolated runtimes. Both reload instantly on code changes. Both share the same database. The entire stack runs with one command.

## The mental model

The key to using runtime containers effectively is changing how you think about containers.

**Traditional Docker thinking:**  
Container = snapshot of entire application (code + runtime + dependencies)

**Runtime container thinking:**  
Container = runtime environment, code is dynamic

This mental changes your development approach:

- **Don't rebuild for code changes**: Rebuilds are only for runtime/dependency changes
- **Separate concerns**: Runtime stability vs code iteration are different problems
- **Development ≠ Production**: Use different strategies for different environments
- **Mount, don't copy**: During development, mount everything you're actively changing

The goal is to maintain containerized consistency without sacrificing development velocity. You want the isolation and reproducibility of containers with the speed of local development.

---

Runtime containers solve a specific problem: containerized development is slow. By separating the runtime from the code, you get the best of both worlds: isolated, reproducible environments and instant feedback loops.

This isn't a replacement for production container builds. It's a development accelerator. Use runtime containers to iterate quickly during active development. Build complete, immutable images for testing and deployment.

The workflows in this guide are battle-tested patterns from real projects: Python APIs processing thousands of requests, Node.js frontends with hot module replacement, polyglot microservices running in Docker Compose. They work because they respect the separation of concerns: runtime in the container, code on the host, clear boundaries between development and production.

If you're rebuilding your Docker image every time you change a line of code, you're doing it wrong. Adopt runtime containers. Your feedback loop will thank you.

## Further Reading

- [Docker Volumes vs Bind Mounts](https://docs.docker.com/engine/storage/volumes/?ref=lorbic.com) - Official Docker documentation on storage options
- [Multi-Stage Builds](https://docs.docker.com/build/building/multi-stage/?ref=lorbic.com) - How to build development and production images from one Dockerfile
- [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers?ref=lorbic.com) - VS Code DevContainers guide
- [Best practices for writing Dockerfiles](https://docs.docker.com/build/building/best-practices/?ref=lorbic.com) - Production Dockerfile optimizations
- [Docker Compose for Development](https://docs.docker.com/compose/gettingstarted/?ref=lorbic.com) - Multi-container development workflows


---

## Python Background Workers: Architecture, Queues, and Retry Strategies

**Date:** 2026-02-01
**URL:** http://localhost:1313/building-production-ready-background-workers-in-python/
**Description:** Learn how to build production-ready Python background workers that scale under load. Covers worker pools, queue architecture, retry strategies, idempotency, and graceful shutdown.


I thought processing audio in the background was simple: spawn a thread, run the script, save the file. Then I hit 200 concurrent requests, and it failed epically.

The CPU spiked to full usage because of pydub's processing. The TTS API didn't rate-limit me but, it was horribly slow. Then the jobs started failing. Half the jobs died silently. The other half wrote corrupted files because of race conditions I didn't know existed. And when I deployed a little fix? The deployment killed in-flight jobs, leaving orphaned audio segments scattered across cloud storage directory.

This is not a guide about how to spawn a background task. This is about what happens after that, when the simple script becomes production infrastructure, when "it works on my machine" becomes "why did it fail in production and how can I fix it?"

Referencing existing production systems was a great way to learn about the tradeoffs and patterns that exist in the space. I found a lot of cool articles. So, I started looking for architectural keywords, patterns, and tradeoffs. And started reading about their implementation in production.

Then I built them into the batch audio processing system. Python, FastAPI, asyncio. It takes text, sends it to Google Cloud Text-to-Speech, processes the audio, and stitches hundreds of segments into final output. I wrote down every pattern, every decision, every tradeoff. This is that document (with the important parts).

## Why background processing is harder than it looks

Consider a client request to generate audio for 200 text segments. Each segment requires a TTS API call (300-1000ms), post-processing (adding silence, normalizing), upload to cloud storage, progress tracking, final stitching, and webhook notification.

Sequentially? That's 4-5 minutes. You can't hold an HTTP connection open that long.

The answer is obvious: background processing. Accept the request, return a job ID, process asynchronously. But this creates a cascade of new problems.

What happens if a worker crashes mid-job? What if two workers grab the same job? What about rate limits? Retries? What if the database locks under concurrent writes? What if a deployment kills a running job? How do you even debug failures that happen while you're asleep?

{{< alert type="important" >}}
Background processing isn't hard because the happy path is hard. It's hard because everything else is hard.
{{< /alert >}}

This blog covers the patterns that solve these problems.

## Level 1: The simple approach

Here's how most people start:

{{< code lang="python" file="api.py" >}}
@app.post("/generate")
async def generate_audio(request: AudioRequest): # Bad idea: processing in the request handler
for segment in request.segments:
audio = await tts_service.generate(segment.text)
await storage.upload(audio)
return {"status": "done"}
{{< /code >}}

This fails immediately. The client times out. The request gets killed. Half the segments are processed, half aren't. There's no way to recover.

The first fix is obvious: move to background tasks.

{{< code lang="python" file="api.py" >}}
@app.post("/generate")
async def generate_audio(request: AudioRequest):
job_id = create_job(request) # Stores job in database with PENDING status
asyncio.create_task(process_job(job_id)) # Fire and forget
return {"job_id": job_id, "status": "accepted"}
{{< /code >}}

Better. The client gets a response immediately. But now you have new problems.

If the process restarts, all running jobs are lost. There's no persistence. If `process_job` throws an exception, it vanishes silently. If you scale to multiple instances, you have no coordination. If a job takes too long, there's no timeout.

This is where I learned and used the real architectural patterns and techniques.

## Level 2: Using a worker pool

The fundamental unit of background processing is the **worker pool**: a fixed set of workers pulling jobs from a queue.

```
+-------------------------------------------------------+
|                         Job Queue                     |
|    [Job1] [Job2] [Job3] [Job4] [Job5] [Job6] ...      |
+--------------------------+----------------------------+
                           |
        +------------------+------------------+
        v                  v                  v
   +---------+        +---------+        +---------+
   | Worker1 |        | Worker2 |        | Worker3 |
   +---------+        +---------+        +---------+
```

<small>Created with [asciiflow](https://asciiflow.com?ref=lorbic.com)</small>  
<br>

Unlike fire-and-forget tasks, a worker pool provides bounded concurrency, crash recovery through job persistence, and visibility into what's running.

The queue can be in-memory (fast, but loses jobs on crash), database-backed (durable, simple), or a dedicated message broker (Kafka, RabbitMQ, SQS). I used SQLite with WAL mode. It's surprisingly capable for moderate workloads, and deployment is trivial. One file, no infrastructure.

<br>

### The Worker Loop

Here's the skeleton of a worker in Python:

{{< code lang="python" file="worker.py" >}}
class Worker:
def **init**(self, db: Database, semaphore: asyncio.Semaphore):
self.db = db
self.semaphore = semaphore
self.shutdown_event = asyncio.Event()

    async def run(self):
        while not self.shutdown_event.is_set():
            job = await self.db.claim_next_pending_job()
            if job is None:
                await asyncio.sleep(1)  # No work, poll again
                continue

            async with self.semaphore:  # Bound concurrent operations
                try:
                    await self.process_job(job)
                    await self.db.mark_completed(job.id)
                except Exception as e:
                    await self.db.mark_failed(job.id, str(e))

    async def process_job(self, job):
        for segment in job.segments:
            if segment.status == "COMPLETED":
                continue  # Skip already-done segments (resumption)
            audio = await self.tts_service.generate(segment.text)
            await self.storage.upload(audio)
            await self.db.mark_segment_completed(segment.id)

{{< /code >}}

A few critical details here:

The `claim_next_pending_job` must be **atomic**. Two workers should never claim the same job. In SQL, this looks like:

{{< code lang="sql" file="claim_job.sql" >}}
UPDATE jobs
SET status = 'IN_PROGRESS', worker_id = ?
WHERE id = (
SELECT id FROM jobs
WHERE status = 'PENDING'
LIMIT 1
)
RETURNING \*;
{{< /code >}}

The `semaphore` bounds concurrency. If you fire 1000 API calls at once, you'll exhaust rate limits, connection pools, and most importantly, system resources like CPU and memory. Start with 5-10 concurrent operations and increase based on monitoring.

The loop checks for already-completed segments before processing. This enables **resumption**. If the job failed at segment 150, the retry skips segments 1-149.

## Level 3: Making it resilient

Failures are not exceptional. They're normal in distributed systems. There are many reasons why things can fail: networks drop, APIs rate-limit, databases lock, servers restart etc. Your system needs to survive all of this.

### Retry with exponential backoff

The naive approach to retries: retry immediately. This is wrong. If an API is overloaded, hammering it with immediate retries makes things worse.

Exponential backoff spaces out retries. And it allows the system to cool down.

{{< code lang="python" file="retry.py" >}}
import asyncio
import random
from functools import wraps

def retry_with_backoff(max_retries=5, base_delay=1.0, max_delay=60.0):
def decorator(func):
@wraps(func)
async def wrapper(*args, \*\*kwargs):
last_exception = None
for attempt in range(max_retries):
try:
return await func(*args, \*\*kwargs)
except TransientError as e:
last_exception = e
if attempt == max_retries - 1:
raise

                    # Exponential backoff with jitter
                    delay = min(base_delay * (2 ** attempt), max_delay)
                    jitter = delay * 0.5 * random.random()
                    await asyncio.sleep(delay + jitter)
            raise last_exception
        return wrapper
    return decorator

@retry_with_backoff(max_retries=5)
async def call_tts_api(text: str) -> bytes:
response = await http_client.post("/synthesize", json={"text": text})
if response.status_code == 429:
raise TransientError("Rate limited")
if response.status_code >= 500:
raise TransientError("Server error")
if response.status_code == 400:
raise PermanentError("Invalid input") # Don't retry this
return response.content
{{< /code >}}

The **jitter** is crucial. Without it, all clients retry at the same instant after an outage, creating thundering herd spikes. Random jitter spreads the load and prevent repeated failures. For example if 1000 clients all retry at the same instant, they'll all fail and retry at the same instant, creating a cascade of failures.

{{< alert type="tip" >}}
Retry transient failures. Fail fast on permanent errors.
{{< /alert >}}

A 400 Bad Request won't magically succeed on retry. A 429 or 503 might. Classify your errors and act accordingly.

### Making operations safe to repeat (Idempotency)

What happens if a worker crashes after uploading an audio file but before marking the segment as complete? The retry will upload the same file again.

If your upload uses a deterministic key (e.g., `job_123/segment_045.mp3`), this is fine. Uploading the same content to the same key is a no-op. That's **idempotency**.

For database updates, use conditional writes:

{{< code lang="sql" file="idempotent_update.sql" >}}
UPDATE segments
SET status = 'COMPLETED'
WHERE id = ? AND status = 'IN_PROGRESS';
{{< /code >}}

This won't double-complete a segment. If something else already completed it, the update affects zero rows.

For external APIs that aren't idempotent, use client-generated request IDs:

{{< code lang="python" >}}
request*id = f"job*{job*id}\_segment*{segment_id}\_v{attempt}"
response = await api.synthesize(text=text, request_id=request_id)
{{< /code >}}

Many APIs deduplicate by request ID. Check your provider's documentation to implement what's best for your use case.

## Level 4: Production Readiness

Your system works. Jobs process. Retries happen. But you're not done. Production means handling deployments, shutdowns, and failures you haven't anticipated yet while developing.

### Graceful Shutdown

When you deploy new code, what happens to running jobs?

The naive answer: they die. SIGTERM kills the process. Jobs are left in `IN_PROGRESS` forever, or worse, with corrupted output.

The correct answer is to implement graceful shutdown.

{{< code lang="python" file="shutdown.py" >}}
import signal
import asyncio

class WorkerManager:
def **init**(self):
self.workers = []
self.active_jobs = set()
self.shutdown_requested = False

    def setup_signal_handlers(self):
        for sig in (signal.SIGTERM, signal.SIGINT):
            signal.signal(sig, self._handle_shutdown)

    def _handle_shutdown(self, signum, frame):
        self.shutdown_requested = True
        for worker in self.workers:
            worker.shutdown_event.set()

    async def wait_for_shutdown(self, timeout=30):
        start = asyncio.get_event_loop().time()
        while self.active_jobs:
            if asyncio.get_event_loop().time() - start > timeout:
                # Hard deadline: force exit
                break
            await asyncio.sleep(0.1)

{{< /code >}}

Here the sequence matters:

1. Receive SIGTERM
2. Stop accepting new work
3. Signal workers to stop pulling new jobs
4. Wait for in-progress jobs to finish (with timeout)
5. Flush any pending writes
6. Exit

If a job doesn't finish within the timeout, it should be **resumable**. When the new instance starts, it should pick up where the old one left off.

### Structured Logging

When something fails, your logs are your only debugging tool.

Unstructured logs are useless:

{{< terminal title="Bad Logs" >}}
Processing job job_123 segment 45 with voice en-US-Standard-A
{{< /terminal >}}

Structured logs are useful and queryable:

{{< code lang="python" file="logging.py" >}}
import structlog

log = structlog.get_logger()

log.info(
"segment_processing_start",
job_id=job.id,
segment_id=segment.id,
voice=segment.voice,
attempt=attempt_number
)

{{< /code >}}

This outputs JSON that you can search in your logging platform:
{{< terminal title="Good Logs">}}
{
"event": "segment_processing_start",
"job_id": "job_123",
"segment_id": "45",
"voice": "en-US-Standard-A",
"attempt": 1,
"timestamp": "2026-02-01T10:30:00Z"
}
{{< /terminal >}}

Log job state transitions, external API calls (with durations), retries, and errors. Don't log sensitive data. Use log levels appropriately (DEBUG for verbose tracing, INFO for milestones, ERROR for failures).

### Health Checks and Startup Validation

A service that starts with invalid configuration will fail eventually, usually at the worst time.

Validate everything at startup:

{{< code lang="python" file="validation.py" >}}
async def startup_validation(): # Check secrets exist
if not os.getenv("TTS_API_KEY"):
raise ConfigurationError("TTS_API_KEY not set")

    # Check external connectivity
    try:
        await tts_client.ping()
    except Exception as e:
        raise ConfigurationError(f"Cannot reach TTS API: {e}")

    # Check database
    try:
        await db.execute("SELECT 1")
    except Exception as e:
        raise ConfigurationError(f"Database unavailable: {e}")

    # Check storage
    try:
        await storage.check_permissions()
    except Exception as e:
        raise ConfigurationError(f"Storage not writable: {e}")

{{< /code >}}

If validation fails, exit immediately with a clear error message. Don't try to "work around" missing configuration. It will cause confusing failures later.

## The reusable patterns

No single pattern is sufficient. It's their combination that produces reliability.

**Statelessness** means any worker can pick up any job. State lives in the database, not in memory.

**Idempotency** means operations are safe to repeat. Retries can't corrupt data.

**Bounded concurrency** means you don't overwhelm downstream services or your own resources. Semaphores control parallelism.

**Granular progress tracking** means jobs can resume from where they failed. You don't re-process 149 segments because segment 150 failed.

**Graceful shutdown** means deployments don't kill work. In-flight jobs complete or become resumable.

**Structured logging** means you can debug failures after the fact. Every significant event is recorded with context.

**Startup validation** means configuration errors are caught early. You don't discover missing credentials during processing job, if they were never added in the first place.

## Common mistakes to avoid

**Unbounded queues.** Memory grows without limit as queue depth increases. Bound your queues. Apply backpressure (return 503) when full. For internal systems, persist all submitted jobs to the database and process them in the background. Rate limiting and capacity planning can handle excessive request volumes.

**Missing timeouts.** Operations hang forever, blocking workers. Timeout everything: API calls, database queries, entire jobs.

**Ignoring partial failures.** One failed segment causes the entire job to fail. Track granular status. Report partial success. Enable resumption.

**State in memory.** Worker crash loses all in-progress state. Persisting state externally is a must.

**Retry storms.** All clients retry simultaneously after an outage (you restart the service). Adding jitter to backoff and using circuit breakers can help.

**No graceful shutdown.** Deployments kill jobs, causing failures and orphaned state. Handle SIGTERM, wait for active jobs to finish and drain queues.

## Do you need all this?

- **< 10 jobs/hour, quick tasks**: FastAPI BackgroundTasks is fine
- **100s of jobs/hour, mission-critical**: Use these patterns
- **1000s+ jobs/hour**: Consider dedicated systems (Celery, etc)

---

Building production-ready background processing systems requires attention to many details. Worker pools with bounded concurrency prevent resource exhaustion. Retry strategies with exponential backoff and jitter handle transient failures gracefully. Idempotent operations enable safe retries and resumption. Graceful shutdown protects in-flight work during deployments.

The audio processing system I built handles thousands of segments daily (tested with 20rps, with 10 segments each and 5 workers). The principles behind it would work equally well for video transcoding, document processing, email sending, or any other batch workload.
I learned a lot from this experience and I will be using these principles in my future projects and learn more design patterns.

My understanding is, start simple and add complexity only when needed. Measure everything you can. And always design as if your system will fail, because eventually, it will. The question is whether it fails gracefully or just plain fails.

## Further Reading

These books and articles helped me a lot to build this system.

- For queue design: Chapter 11 on Queues and Streams of [Designing Data-Intensive Applications](https://dataintensive.net/?ref=lorbic.com) by Martin Kleppmann
- For retry patterns: Chapter 5 on Stability Patterns of [Release It!](https://pragprog.com/titles/mnee2/release-it-second-edition/?ref=lorbic.com) by Michael Nygard
- [FastAPI Background Tasks](https://fastapi.tiangolo.com/tutorial/background-tasks/?ref=lorbic.com)
- [structlog for Python](https://www.structlog.org/en/stable/?ref=lorbic.com)
- [SQLite WAL Mode Explained](https://www.sqlite.org/wal.html?ref=lorbic.com)


---

## Go Struct Field Alignment: How Memory Padding Wastes Your RAM

**Date:** 2026-01-24
**URL:** http://localhost:1313/go-struct-field-alignment/
**Description:** How CPU word alignment and struct field ordering silently inject memory padding in Go. Learn field reordering to cut heap memory allocations and cache misses.


You write a struct to represent a database entity. Maybe 10 fields, maybe 20. What could possibly go wrong?

Nothing, according to your tests. But somewhere in production, your heap is 30% larger than it should be, your Garbage Collector is working overtime, and your L1 cache is not used properly. The reason? **Invisible padding bytes** silently inflating every instance of your struct.

This is the story of struct field alignment: a memory optimization that costs nothing to implement but can significantly improve performance.



## Why Alignment Exists

Modern CPUs don't load memory one byte at a time. They operate on **aligned words**, typically 8 bytes on 64-bit systems. When a data type's memory address is not divisible by its alignment requirement, one of two things happens:

1. **Performance penalty**: The CPU performs multiple memory accesses to load the value
2. **Hardware fault**: On some architectures (historically common, now rare), unaligned access causes a crash

To prevent this, the Go compiler automatically inserts **padding bytes** between struct fields. The padding ensures each field starts at a memory address that satisfies its alignment requirement.

### The Go Spec on Alignment

According to the [Go Language Specification](https://go.dev/ref/spec#Size_and_alignment_guarantees):

> For a variable `x` of any type: `unsafe.Alignof(x)` is at least 1.
> For a variable `x` of struct type: `unsafe.Alignof(x)` is the largest of all the values `unsafe.Alignof(x.f)` for each field `f` of `x`, but at least 1.

This means a struct's alignment is determined by its largest-aligned field. A struct containing an `int64` must be 8-byte aligned, even if its first field is a `bool`.



## Demonstrating the Silent Bloat

Consider this innocent-looking struct with fields ordered "logically" (booleans together, then data):

```go
type BadStruct struct {
    IsActive   bool              // 1 byte  + 7 padding
    ID         uint64            // 8 bytes
    IsVerified bool              // 1 byte  + 7 padding
    Name       string            // 16 bytes (ptr + len)
    IsAdmin    bool              // 1 byte  + 7 padding
    Score      float64           // 8 bytes
    // ... more fields
}
```

Each 1-byte `bool` followed by an 8-byte type wastes 7 bytes of padding.

Let's measure real structs with all common Go types:

```go
// BadStruct: Fields interleaved to maximize padding
type BadStruct struct {
    IsActive   bool              // 1 byte  + 7 padding
    ID         uint64            // 8 bytes
    IsVerified bool              // 1 byte  + 7 padding
    Name       string            // 16 bytes
    IsAdmin    bool              // 1 byte  + 7 padding
    Score      float64           // 8 bytes
    IsPremium  bool              // 1 byte  + 3 padding
    ParentID   uint32            // 4 bytes
    TinyVal    int8              // 1 byte  + 1 padding
    SmallVal   int16             // 2 bytes + 4 padding
    Email      string            // 16 bytes
    IsDeleted  bool              // 1 byte  + 7 padding
    Count      int64             // 8 bytes
    IsArchived bool              // 1 byte  + 3 padding
    Rating     float32           // 4 bytes
    Status     int8              // 1 byte  + 7 padding
    Tags       []string          // 24 bytes
    Enabled    bool              // 1 byte  + 7 padding
    Metadata   map[string]string // 8 bytes
    Ready      bool              // 1 byte  + 7 padding
    CreatedAt  int64             // 8 bytes
    Done       bool              // 1 byte  + 7 padding
    UpdatedAt  int64             // 8 bytes
    Callback   func()            // 8 bytes
    Flag       bool              // 1 byte  + 7 padding
    Description string           // 16 bytes
}
```

Measuring with `unsafe.Sizeof`:

```go
fmt.Println(unsafe.Sizeof(BadStruct{}))  // Output: 224 bytes
```

**224 bytes.** Let's see how much is wasted.



## The Alignment Rules

Go's alignment requirements are architecture-dependent but follow predictable rules on 64-bit systems:

| Type | Size | Alignment | Notes |
|:-----|:----:|:---------:|:------|
| `bool` | 1 | 1 | |
| `int8`, `uint8`, `byte` | 1 | 1 | |
| `int16`, `uint16` | 2 | 2 | |
| `int32`, `uint32`, `float32` | 4 | 4 | |
| `int64`, `uint64`, `float64` | 8 | 8 | |
| `int`, `uint`, `uintptr` | 8 | 8 | On 64-bit systems |
| `string` | 16 | 8 | Header: `{ptr, len}` |
| `slice` (`[]T`) | 24 | 8 | Header: `{ptr, len, cap}` |
| `map` | 8 | 8 | Pointer to `hmap` |
| `func` | 8 | 8 | Pointer |
| `interface{}` | 16 | 8 | `{type, data}` |
| `pointer` (`*T`) | 8 | 8 | |
| `chan` | 8 | 8 | Pointer to `hchan` |

{{< alert type="important" >}}
Strings and slices are **header types**. They contain pointers to the actual data, not the data itself. A `string` is 16 bytes regardless of its content length.
{{< /alert >}}

### The Optimization Rule

**Order fields from largest alignment requirement to smallest:**

1. **24-byte types**: Slices (`[]T`)
2. **16-byte types**: Strings, interfaces
3. **8-byte types**: `int64`, `uint64`, `float64`, pointers, maps, funcs, chans
4. **4-byte types**: `int32`, `uint32`, `float32`
5. **2-byte types**: `int16`, `uint16`
6. **1-byte types**: `int8`, `uint8`, `bool`, `byte`

Applying this to our struct:

{{< code lang="go" wrap="true" >}}
// GoodStruct: Fields ordered by alignment (largest to smallest)
type GoodStruct struct {
    // 24-byte: Slices
    Tags []string // 24 bytes

    // 16-byte: Strings
    Name        string // 16 bytes
    Email       string // 16 bytes
    Description string // 16 bytes

    // 8-byte: int64, uint64, float64, pointers
    ID        uint64            // 8 bytes
    Count     int64             // 8 bytes
    Score     float64           // 8 bytes
    CreatedAt int64             // 8 bytes
    UpdatedAt int64             // 8 bytes
    Metadata  map[string]string // 8 bytes (pointer)
    Callback  func()            // 8 bytes (pointer)

    // 4-byte: int32, uint32, float32
    ParentID uint32  // 4 bytes
    Rating   float32 // 4 bytes

    // 2-byte: int16, uint16
    SmallVal int16 // 2 bytes

    // 1-byte: int8, bool (packed together)
    TinyVal    int8 // 1 byte
    Status     int8 // 1 byte
    IsActive   bool // 1 byte
    IsVerified bool // 1 byte
    IsAdmin    bool // 1 byte
    IsPremium  bool // 1 byte
    IsDeleted  bool // 1 byte
    IsArchived bool // 1 byte
    Enabled    bool // 1 byte
    Ready      bool // 1 byte
    Done       bool // 1 byte
    Flag       bool // 1 byte + 2 padding (struct alignment)
}
{{< /code >}}

```go
fmt.Println(unsafe.Sizeof(GoodStruct{}))  // Output: 152 bytes
```

**152 bytes.** We saved **72 bytes** (32% reduction) with zero code changes, just by reordering fields.



## Measuring the Impact with Benchmarks

Here are real benchmarks run on an Apple M2 (darwin/arm64):

### Struct Sizes

```text
=== TestEntitySizes ===
BadStruct size:  224 bytes
GoodStruct size: 152 bytes
Memory saved:    72 bytes (32.1% reduction)
```

### Allocation Benchmarks

```text
goos: darwin
goarch: arm64
cpu: Apple M2

BenchmarkBadStruct_Alloc-8       17997783    61.11 ns/op    224 B/op    1 allocs/op
BenchmarkGoodStruct_Alloc-8      33894393    31.85 ns/op    160 B/op    1 allocs/op

BenchmarkBadStruct_Slice1k-8        95834    12318 ns/op    229377 B/op    1 allocs/op
BenchmarkGoodStruct_Slice1k-8      136102     8421 ns/op    155649 B/op    1 allocs/op

BenchmarkBadStruct_Slice10k-8       16683    71874 ns/op    2244613 B/op    1 allocs/op
BenchmarkGoodStruct_Slice10k-8      20770    60512 ns/op    1523715 B/op    1 allocs/op
```

### Analysis

| Benchmark | Bad | Good | Improvement |
|:----------|:----|:-----|:------------|
| **Single Alloc** | 61.1 ns, 224 B | 31.9 ns, 160 B | **48% faster** |
| **1k Slice** | 12.3 µs, 224 KB | 8.4 µs, 152 KB | **32% faster, 32% less memory** |
| **10k Slice** | 71.9 µs, 2.19 MB | 60.5 µs, 1.49 MB | **16% faster, 32% less memory** |

{{< alert type="tip" >}}
The 32% memory reduction is consistent because it directly reflects the struct size difference. The time improvement comes from less memory to zero-initialize and better cache utilization.
{{< /alert >}}

### Run the Benchmarks Yourself

{{< download file="struct_alignment_benchmark_test.go" label="Download struct_alignment_benchmark_test.go" >}}

{{< terminal title="Running the benchmarks" >}}
$ go mod init alignment_test
$ go test -v -run TestEntitySizes
$ go test -bench=. -benchmem
{{< /terminal >}}



## How Padding Works Behind the Scenes

Let's trace through the field offsets to understand exactly where padding is inserted:

```text
BadStruct Field Offsets:
```

| Field | Offset | Size | Explanation |
|:------|:------:|:----:|:------------|
| IsActive | 0 | 1 | Starts at 0 |
| (padding) | 1 | 7 | ID needs 8-byte alignment |
| ID | 8 | 8 | Starts at 8 (divisible by 8) |
| IsVerified | 16 | 1 | Starts at 16 |
| (padding) | 17 | 7 | Name needs 8-byte alignment |
| Name | 24 | 16 | Starts at 24 (divisible by 8) |
| IsAdmin | 40 | 1 | |
| (padding) | 41 | 7 | Score needs 8-byte alignment |
| Score | 48 | 8 | |
| IsPremium | 56 | 1 | |
| (padding) | 57 | 3 | ParentID needs 4-byte alignment |
| ParentID | 60 | 4 | 60 is divisible by 4 |
| TinyVal | 64 | 1 | |
| (padding) | 65 | 1 | SmallVal needs 2-byte alignment |
| SmallVal | 66 | 2 | 66 is divisible by 2 |
| (padding) | 68 | 4 | Email needs 8-byte alignment |
| Email | 72 | 16 | 72 is divisible by 8 |
| ... | | | |

You can inspect these offsets programmatically:

```go
var s BadStruct
fmt.Printf("IsActive offset: %d\n", unsafe.Offsetof(s.IsActive))   // 0
fmt.Printf("ID offset: %d\n", unsafe.Offsetof(s.ID))               // 8
fmt.Printf("IsVerified offset: %d\n", unsafe.Offsetof(s.IsVerified)) // 16
fmt.Printf("Name offset: %d\n", unsafe.Offsetof(s.Name))           // 24
```

## Why This Matters Beyond Memory

### 1. Reduced GC Pressure

Smaller structs mean:
- **Smaller heap** -> Less memory to scan during garbage collection
- **Better cache locality** -> GC mark phase runs faster
- **Lower allocation churn** -> GC triggers less frequently

{{< alert type="tip" >}}
**Want to go deeper on GC?** Read my other deep dive: [Don't Take Out the Garbage: Go GC Deep Dive](/posts/dont-take-out-the-garbage-go-gc-deep-dive/). It explains how the mark-and-sweep algorithm works and why large heaps crush performance.
{{< /alert >}}

If `runtime.scanobject` appears in your CPU profiles, you have GC pressure. Shrinking struct sizes directly reduces this cost.

### 2. CPU Cache Efficiency

The L1 data cache line is typically 64 bytes. Compact structs fit more instances per cache line:

```text
64-byte cache line capacity:
- BadStruct (224 bytes):  0.28 structs per line
- GoodStruct (152 bytes): 0.42 structs per line
```

When iterating over a slice of structs, compact layouts mean fewer cache misses and better hardware prefetching.

### 3. Network/Disk Efficiency

If you use binary serialization (protobuf, msgpack, gob), struct layout can affect:
- Network payload sizes
- Memory-mapped file efficiency
- Serialization/deserialization speed



## Detecting Alignment Issues

### Tool: `fieldalignment`

The Go team provides an official analyzer:

{{< terminal title="Using fieldalignment" wrap="true" >}}
$ go install golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment@latest
$ fieldalignment ./...
$ fieldalignment -fix ./...
{{< /terminal >}}

Example output:

```text
entity/user.go:15:6: struct of size 224 could be 152 (order fields by alignment)
entity/post.go:42:6: struct of size 120 could be 104 (order fields by alignment)
```

### Manual Inspection

Use `unsafe.Sizeof` and `unsafe.Offsetof` to audit critical structs:

```go
import "unsafe"

func auditStruct() {
    var e Entity
    fmt.Printf("Total size: %d bytes\n", unsafe.Sizeof(e))
    fmt.Printf("Field1 offset: %d\n", unsafe.Offsetof(e.Field1))
    fmt.Printf("Field2 offset: %d\n", unsafe.Offsetof(e.Field2))
    // Check for gaps between (offset + size) and next offset
}
```



## Pitfalls and Caveats

### 1. Auto-fix Can Break Binary Compatibility

{{< alert type="caution" >}}
The `fieldalignment -fix` tool reorders fields. If your code depends on specific field ordering for binary serialization, memory-mapped files, or CGO interop, this will break things silently.
{{< /alert >}}

Always review changes before applying them. For CGO structs, field order must often match C struct definitions exactly.

### 2. JSON/YAML Unmarshalling Is Unaffected

The common fear that reordering fields breaks JSON parsing is **unfounded**. The `encoding/json` package uses reflection and struct tags to map JSON keys to fields:

```go
type User struct {
    Age  int    `json:"age"`   // Field order in memory
    Name string `json:"name"`  // ≠ key order in JSON
}

// JSON: {"name":"Vikash","age":40}
// Works identically regardless of struct field order
```

### 3. Don't Micro-optimize Everything

Alignment optimization matters when:
- The struct is instantiated **thousands or millions of times**
- The struct appears in **hot paths** (request handlers, tight loops)
- You're seeing **GC pressure** (`runtime.scanobject` > 5% CPU)
- Memory is a **constraint** (embedded systems, large caches)

For structs used sparingly, readability may be more valuable than a few bytes.

### 4. Struct Tail Padding

Even optimally ordered structs may have tail padding. The struct's total size must be a multiple of its alignment:

```go
type Example struct {
    A int64 // 8 bytes
    B bool  // 1 byte + 7 padding = 16 total
}
```

This ensures arrays of structs maintain proper alignment for each element.



## Quick Reference Checklist

When designing or refactoring structs:

1. **Order by alignment**: 24-byte -> 16-byte -> 8-byte -> 4-byte -> 2-byte -> 1-byte
2. **Pack bools together**: Place all `bool` fields at the end
3. **Verify with `unsafe.Sizeof`**: Confirm your optimizations work
4. **Run `fieldalignment`**: Catch what you miss, but review before applying
5. **Profile first**: Only optimize structs that matter

```go
// Quick verification
import "unsafe"

func checkSize[T any]() {
    var x T
    fmt.Printf("Type: %T, Size: %d bytes, Align: %d\n", 
        x, unsafe.Sizeof(x), unsafe.Alignof(x))
}
```



## Summary

| Concept | Key Point |
|:--------|:----------|
| **Problem** | CPU alignment requirements force compilers to insert padding |
| **Symptom** | Structs consume more memory than the sum of their fields |
| **Solution** | Order fields from largest to smallest alignment requirement |
| **Benefit** | 20-40% typical memory reduction, better cache usage, lower GC pressure |
| **Caveat** | Don't auto-fix binary-serialized or CGO structs without proper review and testing |

Field alignment is the kind of optimization that costs nothing to implement correctly. Once you internalize the ordering rules, writing compact structs becomes second nature. Your heap will be smaller, your GC will run faster, and your CPU cache will thank you.



## Further Reading

1. **[Go Language Specification: Size and Alignment](https://go.dev/ref/spec#Size_and_alignment_guarantees)** - Go spec
2. **[fieldalignment analyzer source](https://cs.opensource.google/go/x/tools/+/master:go/analysis/passes/fieldalignment/)** - Understand how the tool works
3. **[A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide)** - Why heap size matters
4. **[unsafe package documentation](https://pkg.go.dev/unsafe)** - `Sizeof`, `Offsetof`, `Alignof`


---

## I Added Session Management to Aider

**Date:** 2026-01-15
**URL:** http://localhost:1313/aider-session-management/
**Description:** Aider is my favorite AI coding assistant. It lives in the terminal, commits directly to Git, and actually understands my codebase. But it was missing one thing: the ability to save and restore chat sessions. So I built it. The PR never got merged, but I've been running my fork for months now. Here's the story.


I've been using [aider](https://aider.chat) as my primary AI coding assistant for a while now. It's the one tool that actually fits my workflow: terminal-based, Git-native, and it works directly on my local files. No copy-pasting into web forms. No context windows that forget everything. And not waiting for agents to keep thinking.

But it was missing one thing that drove me crazy.

## The Problem: Context Evaporates

Here's the scenario. I'm deep into a complex refactor. I've added 15 files to the chat, built up tons of context with the model, and we're making real progress.

Then one of these happens:
- I need to step away for a meeting
- I merge a big feature branch (need to change branches, aider detects repo changes and resets, loosing all context)
- My terminal session dies
- I accidentally hit `/clear`

Poof. All that context is gone. The conversation history, the files I'd carefully curated, everything. Starting over every time was killing my momentum.

I kept thinking: *tmux has sessions. Why doesn't aider?*

So I built it.

## The `/session` Command

I added a full session management system to aider. Five simple commands:

```bash
# Save your current session
/session save my-feature-work

# List all saved sessions
/session list

# Peek inside a session without loading it
/session view my-feature-work

# Load a session (clears current chat and restores the saved one)
/session load my-feature-work

# Delete a session you don't need anymore
/session delete my-feature-work
```

That's it. Now I can save context before stepping away, after hitting a good milestone, or just because I'm paranoid about losing progress.

## How It Works

When you run `/session save`, here's what happens:

1. A `.aider/sessions/` directory gets created in your repo root
2. The current state is captured: chat history, editable files, read-only files
3. Everything gets bundled into a JSON file with metadata (timestamp, aider version)

Loading is the reverse, read the JSON, reset state, restore everything.

The sessions are project-specific (stored in each repo), so you don't get cross-contamination between projects. And since it's just JSON files, you can even version control them if you want.

I put the implementation in `aider/commands.py` and added a full test suite in `tests/test_session.py`. The tests spin up a temporary Git repo and run all the subcommands.

## The PR Situation

Here's the honest part: **my PR never got merged into upstream aider**.

I'm not entirely sure why. Maybe the maintainers are busy with other things, or maybe upstream is not in active development. These things happen in open source. Thanks to it being open source, I can still fork and build my own version to use it daily.

But here's the thing: **I've been running my fork daily for months now**.

Check it out: [github.com/vk4s/aider](https://github.com/vk4s/aider)

The feature works great. I use it constantly. Every time I'm about to do something risky or need to step away, I hit `/session save`. It's muscle memory now.

## Why I'm Still Happy About It

Even without the merge, this contribution taught me a lot:

1. **I got to dive deep into aider's codebase** - Understanding how it manages state, handles commands, and integrates with Git was fascinating.

2. **I scratched my own itch** - The best open source contributions come from actual pain points. This wasn't a theoretical improvement; it was something I really needed.

3. **I use it every day** - The feature exists. It works. I benefit from it. That's a win.

4. **Forking is fine** - There's sometimes stigma around maintaining a fork, but if the upstream isn't taking your changes and the feature is valuable to you, why not? Keep your fork in sync with upstream, apply your patches, and move on. I'm leaving my pr open in the upstream and when it's merged I'll use the upstream version. (:

## If You Want Session Management

If you're an aider user who wants this feature:

1. Clone my fork: `git clone https://github.com/vk4s/aider`
2. Install it: `pip install -e .`
3. Use `/session save|load|list|view|delete`

Or just grab the relevant commits and cherry-pick them into your own setup.

---

Contributing to open source isn't always about getting merged. Sometimes it's about solving your own problems, learning a codebase, and sharing the solution for others who might want it too.

The code is there. The feature works. And I'm still waiting for the day I can delete my fork because it landed upstream. 

Until then, `/session save peace-of-mind` is my new best friend.

---

PS: I wrote this about 4 months ago, and a lot has changed, we have claude-code, antigravity. But aider is still my favorite AI coding assistant for working with monoliths. 



---

## Can ClickHouse Replace Vector Databases? HNSW, Benchmarks & SQL Setup

**Date:** 2026-01-14
**URL:** http://localhost:1313/clickhouse-as-a-vector-database/
**Description:** Can ClickHouse replace dedicated vector databases like Pinecone, Weaviate, or Milvus? Explore HNSW indexing, vector similarity search performance, and how to store embeddings with SQL in ClickHouse.


Everyone's building AI apps now. And every AI app needs a place to stash embeddings. The instant your data grows beyond "fits in memory", you need a vector database. Or do you?

If you're already running ClickHouse for analytics, here's some good news: you might not need another database. ClickHouse can hold its own as a vector store. Not because it was built for it, but because it's built to be fast at everything.

Let's dig in.

## What Even Is a Vector Database?

To understand what a vector database is, we need to start with what vectors are. In the context of machine learning and AI, a vector is a numerical representation of data - often used to represent text, images, audio, or even user behavior. These are not arbitrary numbers; they are structured in such a way that their relative distances capture some idea of *similarity*.

For instance, in the world of natural language processing (NLP), embeddings are generated from text using models like BERT or OpenAI's Ada. These embeddings transform language into dense vectors of floating point numbers, often 384 or 768 dimensions long. Words or phrases with similar meanings will have embeddings that are closer together in vector space, while dissimilar ones will be far apart.

Here's what that looks like in practice, if we plot words in a simplified 2D vector space, semantically similar words cluster together:

> *Vector space: cat and kitten are neighbors, toaster is not invited.*

![Vector space: cat and kitten are neighbors, toaster is not invited](vector-space.png)

A vector database, then, is a database that is optimized to store these high-dimensional vectors and perform efficient similarity searches on them. The core operation is typically the k-nearest neighbors (KNN) search: given a query vector, return the most similar vectors (and hence documents, images, etc.) from the database.

Unlike traditional databases (e.g. MySQL) which rely on B-trees and hash indexes for exact matches or range queries, vector databases use specialized indexing structures like HNSW (Hierarchical Navigable Small World graphs), IVF (Inverted File Index - cool stuff), and PQ (Product Quantization). These indexes allow for approximate nearest neighbor (ANN) search, which trades a bit of accuracy for massive speed-ups in high-dimensional spaces.

Some popular purpose-built vector databases are Pinecone, Weaviate, Milvus, and Faiss (from Meta). However, the recent evolution in OLAP systems like ClickHouse has opened the door to hybrid approaches, combining their analytical power with vector search at big-scale.

## Why ClickHouse?

ClickHouse isn't a vector database per se. It's a lightning-fast OLAP column-store database originally developed at Yandex for real-time analytics. It's built for high-throughput aggregations over massive datasets, with features like compression, parallel processing, and late materialization.

### So why use ClickHouse for vectors?

**First**, it supports arrays and fixed-size numeric types, including `Array(Float32)`, which makes it possible to store embedding vectors natively.

**Second**, ClickHouse has recently introduced native support for vector indexes, including HNSW and flat indexes. This means you can create an ANN index directly on an embedding column and query it using standard SQL syntax. This native support makes it viable to unify analytical and semantic queries in a single system, no need to juggle a separate vector store and an analytics database.

**Third**, ClickHouse's scalability and cost-efficiency make it attractive for production workloads. If you're already using ClickHouse for logs, metrics, or analytics, extending it for semantic search can be both convenient and economical.

In short: fewer moving parts, same fast queries, one less thing to sync.

## Example with ClickHouse

The foundation of most ClickHouse tables is the MergeTree engine. It provides partitioning, ordering, indexing, and background merges, ideal for write-heavy and analytical use cases.

To store embeddings, you create a table with a column of type `Array(Float32)`. Here's a simplified schema:

```sql
CREATE TABLE content_transcript_segment (
    segment_id String,
    text String,
    start_time Float64,
    end_time Float64,
    embeddings Array(Float32),
    embedding_dimensions UInt64
) ENGINE = MergeTree
ORDER BY segment_id;
```

Starting from ClickHouse v24.6, you can add a vector index like this:

```sql
ALTER TABLE content_transcript_segment 
ADD INDEX vec_idx embeddings TYPE HNSW('metric_type=cosine') GRANULARITY 1;
```

This creates a cosine similarity-based HNSW index over the embedding vectors. The `GRANULARITY` defines how frequently the index is updated in data parts, a tradeoff between index precision and performance.

After creating the index, you must materialize it to populate it:

```sql
ALTER TABLE content_transcript_segment MATERIALIZE INDEX vec_idx;
```

From there, queries are straightforward.

## Semantic Search in SQL

Once your embeddings and index are in place, querying becomes remarkably ergonomic. Let's say you have a query embedding from an external model, a 384-dimensional `Array(Float32)` value. You can use SQL to find the most similar segments:

```sql
SELECT 
    segment_id, 
    text, 
    cosineDistance(embeddings, [your_query_vector]) AS score
FROM content_transcript_segment
ORDER BY score ASC
LIMIT 5;
```

You can even wrap this in a view or a UDF to expose semantic search as a high-level API.

This simplicity is what makes ClickHouse compelling. You stay in SQL. You use familiar tooling. You don't need to manage and sync a separate vector database. It's just another analytical query, but instead of filtering on keywords, you're querying on meaning.

## Pain Points and Learnings

Nothing's free. Here's what to watch out for:

1. **Indexing Costs**: HNSW indexes aren't free. Adding or materializing the index can be resource-intensive. Batch writes are better than high-frequency single inserts.

2. **Embedding Compatibility**: Your application must ensure all embeddings are of the same dimensionality, dtype (`Float32`), and normalization (e.g., unit length for cosine distance).

3. **Debugging Vectors**: It's easy to make mistakes in preprocessing or vector normalization. Simple metrics like `L2Norm`, `dotProduct`, or histograms can help debug anomalies in your embeddings.

4. **Query Limits**: For now, only certain similarity metrics are supported (`cosineDistance`, `l2Distance`, etc.), and the query vector has to be a literal, not dynamically fetched from a `JOIN` or subquery. Workarounds include injecting the vector at query time from the application layer.

5. **Index Evolution**: As the schema evolves, you may need to rebuild or rematerialize indexes. Something to plan for in CI/CD pipelines.

## What's Next

This journey is far from over. There are open questions around live updates, incremental index maintenance, hybrid filtering (semantic + keyword), and ranking models.

I'm exploring whether hybrid scoring functions can be created, for example, combining vector similarity with popularity scores or recency.

Another area of interest is applying semantic clustering to group text segments automatically, then using these groups to power navigation in long-form content.

I'm also experimenting with embedding compression techniques and hashing for storage optimization. High-dimensional vectors can get expensive, and every byte matters at scale.

---

ClickHouse isn't a plug-and-play vector database, but it gives you some pretty sharp tools. If you understand the mechanics, how indexes work, how embeddings behave, how OLAP engines think, you can build a system that handles millions of vectors, performs blazing-fast searches, and answers questions that once required a fleet of infrastructure.

ClickHouse is the AMD of OLAP databases. Best price-performance ratio.

The real value of these tools lie in the kind of questions we can now ask. 

Go ahead, [sign up for their trial account](https://clickhouse.cloud/?ref=lorbic.com) and build something cool.


## Further Reading

- **HNSW Indexes** (many sources because one is never enough):
    - [Supabase - HNSW Indexes](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes?ref=lorbic.com)
    - [Pinecone - HNSW](https://www.pinecone.io/learn/series/faiss/hnsw/?ref=lorbic.com)
    - [Neon - Understanding Vector Search and HNSW](https://neon.com/blog/understanding-vector-search-and-hnsw-index-with-pgvector?ref=lorbic.com)
    - [ClickHouse - HackerNews Vector Search (Cool story)](https://clickhouse.com/docs/getting-started/example-datasets/hackernews-vector-search-dataset?ref=lorbic.com)

- **ClickHouse Vector Search**:
    - [ClickHouse - Vector Search Documentation](https://clickhouse.com/docs/engines/table-engines/mergetree-family/annindexes?ref=lorbic.com)
    - [ClickHouse Blog - Vector Search with ClickHouse](https://clickhouse.com/blog/vector-search-clickhouse-p1?ref=lorbic.com)
    - [ClickHouse - Distance Functions](https://clickhouse.com/docs/sql-reference/functions/distance-functions?ref=lorbic.com)

- **Embeddings and Vector Fundamentals**:
    - [OpenAI - Embeddings Guide (Use this to generate embeddings)](https://platform.openai.com/docs/guides/embeddings?ref=lorbic.com)
    - [Hugging Face - Sentence Transformers](https://huggingface.co/docs/hub/en/sentence-transformers?ref=lorbic.com)
    - [Jay Alammar - The Illustrated Word2Vec (Great intro)](https://jalammar.github.io/illustrated-word2vec/?ref=lorbic.com)

- **ANN Algorithms Deep Dive**:
    - [Faiss Wiki - Index Structures](https://github.com/facebookresearch/faiss/wiki?ref=lorbic.com)
    - [ANN Benchmarks - Compare Algorithm Performance](https://ann-benchmarks.com/index.html?ref=lorbic.com)



---

## Memory Mechanics In Go - Stack vs Heap

**Date:** 2026-01-12
**URL:** http://localhost:1313/memory-mechanics-stack-vs-heap-in-go/
**Description:** When thinking about performance, it's easy to focus on Big O notation. But in Go, the difference between the Stack and the Heap is often the difference between a service that scales and one that chokes on GC pauses. This post explores escape analysis, the "Pointer Myth", and why passing by value is often 40x faster than passing by pointer.


We often talk about "fast" code in terms of Big O notation or algorithmic complexity. But in systems programming languages like Go, "fast" is often a function of _where_ your data lives in memory.

When optimizing for high throughput, efficient loops and database indexes are only part of the story. Eventually, you have to talk about the Stack and the Heap.

Understanding the difference isn't just trivia. It is the difference between a service that hums along at 100k OPS with flat latency, and one that chokes on Garbage Collection (GC) pauses every few seconds.

This post explores the mechanics of Go's memory management, why the Stack is your friend, and why "using pointers for performance" is often a lie we tell ourselves.

## The Two Worlds: Stack vs Heap

Memory in your Go program is effectively divided into two zones. They aren't physically different RAM chips, but they are managed with completely different strategies.

### The Stack: The Fast Lane

Every Goroutine in Go gets its own stack. It starts small (2KB) and grows/shrinks dynamically.

Think of the stack like a scratchpad. When a function is called, it gets a slice of this scratchpad (a stack frame) to write its variables. When the function returns, that slice is simply marked as "free" by moving a pointer.

- **Allocation Cost:** One CPU instruction (subtracting from the stack pointer).
- **Cleanup Cost:** Zero (adding to the stack pointer).
- **Access:** Extremely fast (L1/L2 cache locality).

Technically, Go uses continuous stacks. When a goroutine runs out of stack space, the runtime allocates a new, larger segment and copies the existing stack to it. This "stack copying" is the only time stack memory incurs a significant cost, but it happens rarely compared to function calls.

### The Heap: The Shared Chaos

Anything that cannot fit on the stack, or needs to live longer than the function that created it, goes to the Heap. The Heap is a massive pool of shared memory.

- **Allocation Cost:** Expensive. The runtime (`runtime.mallocgc`) must find a free block of the right size, potentially hold locks, and update metadata.
- **Cleanup Cost:** Very Expensive. This is the domain of the Garbage Collector. It has to scan, mark, and eventually sweep this memory.
- **Access:** Slower (pointer chasing, less cache friendly).

The golden rule of Go performance: **If it's on the Stack, you don't pay for it. If it's on the Heap, you pay tax on every GC cycle.**

## Escape Analysis: The Compiler's Decision

So, how does Go decide? You don't have `malloc` or `free`. You have the compiler.

Go uses a technique called **Escape Analysis** to determine the lifetime of a variable. If the compiler can prove that a variable _never_ leaves the function scope, it allocates it on the stack. If the variable "escapes" (e.g., is returned to a caller, stored in a global variable, or sent to a channel), it generally must be moved to the Heap.

### Seeing It In Action

You can see this decision-making process by using the `-gcflags` flag.

```go
package main

type User struct {
    ID   int
    Name string
}

func createUserV1() User {
    u := User{ID: 1, Name: "Vikash"}
    return u // Returns a VALUE
}

func createUserV2() *User {
    u := User{ID: 2, Name: "Vikash"}
    return &u // Returns a POINTER
}

func main() {
    _ = createUserV1()
    _ = createUserV2()
}
```

Compile with analysis enabled:

```bash
$ go build -gcflags="-m" main.go

./main.go:14:2: moved to heap: u
```

- `createUserV1`: Returns `User` by value. The data is copied to the caller's stack frame. `u` stays on the stack. **Fast.**
- `createUserV2`: Returns `&u`. The caller needs to access variables created inside `createUserV2` _after_ the function returns. If `u` were on the stack, it would be overwritten by the next function call. The compiler _must_ move `u` to the heap. **Slow.**

## Hidden Escape Routes

Returning a pointer is the obvious escape route. But there are subtle ones that catch even experienced engineers.

### 1. Interfaces (Dynamic Dispatch)

When you assign a concrete value to an interface, the runtime often needs to store the type information along with the data. If the compiler cannot deduce the type at compile time, or if the method call involves dynamic dispatch, the value often escapes.

```go
func Log(v interface{}) {
    fmt.Println(v) // 'v' escapes to heap because fmt.Println takes interface{}
}
```

This is why passing huge structs to `log.Println` or `json.Marshal` kills performance, it forces them onto the heap.

### 2. Slices with Dynamic Size

A slice on the stack must have a known size at compile time. If the size is determined by a variable, it often escapes.

```go
func makeSpace(n int) {
    x := make([]byte, n) // Escapes to heap
}

func makeFixed() {
    x := make([]byte, 64) // Stays on stack
}
```

### 3. Closure Capture

If a closure (anonymous function) references a variable from the outer scope, and that closure is passed around, the closed-over variable escapes.

## Benchmarking the Cost

Let's prove the "Pointer Myth" wrong with data. We will compare passing a small struct (64 bytes) by value vs. by reference.

```go
// Benchmark Code
type Config struct {
    ID   int
    Name string
    Data [50]byte // Padding to make it ~64 bytes
}

//go:noinline
func byValue(c Config) int {
    return c.ID
}

//go:noinline
func byPointer(c *Config) int {
    return c.ID
}
```

**Results:**

```text
BenchmarkByValue-10     1000000000           0.30 ns/op        0 B/op       0 allocs/op
BenchmarkByPointer-10    100000000          12.50 ns/op       64 B/op       1 allocs/op
```

- **Pass by Value:** 0.3ns. Zero allocations. The CPU simply copies the registers.
- **Pass by Pointer:** 12.5ns. One allocation.
- **Difference:** 40x slower.

Why? Because `byPointer` creates a heap allocation (in this specific microbenchmark setup where the pointer effectively escapes or the compiler decides to alloc). In real-world code, even if it doesn't always alloc, the _pressure_ on the GC adds up.

## Practical Advice for Optimization

When optimizing:

1.  **Default to "Pass by Value".** Current CPUs can copy 64 bytes faster than you can blink. Do not use pointers just to avoid copying small structs.
2.  **Watch your Interfaces.** Heavy use of `interface{}` in hot paths (like middleware or data ingest loops) is a common source of hidden allocations.
3.  **Pre-allocate Slices.** Use `make([]T, 0, capacity)` where `capacity` is a constant if possible, or at least a calculated max, to avoid resizing allocations.
4.  **Profile `mallocgc`.** If `runtime.mallocgc` is more than 5% of your CPU profile, you have a churn problem. Use `go build -gcflags="-m"` to find the leaks.

## Summary

- **Stack** = Fast, local, free.
- **Heap** = Slow, shared, taxed by GC.
- **Pointers** = Cause heap allocations. Use them for _semantics_ (I need to modify this), not for _performance_ (unless the struct is > 2KB).

Memory management in Go is about empathy for the runtime. If you treat the Garbage Collector like a colleague who is already overworked, you'll naturally write code that stays on the stack, keeps the heap clean, and scales effortlessly.

## Further Reading

1.  **[Language Mechanics On Escape Analysis](https://www.ardanlabs.com/blog/2017/05/language-mechanics-on-escape-analysis.html)** – Ardan Labs (William Kennedy)
    - _The definitive 4-part series on how the Go compiler makes allocation decisions._
2.  **[A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide)** – Official Go Documentation
    - _Deep dive into the tri-color mark-and-sweep algorithm and tuning `GOGC`._
3.  **[Go SliceTricks](https://go.dev/wiki/SliceTricks)** – Go Wiki
    - _Efficiency patterns for manipulating slices without unnecessary allocations._


---

## OLTP vs OLAP - Why You Need Two Databases

**Date:** 2026-01-11
**URL:** http://localhost:1313/oltp-vs-olap-why-you-need-two-databases/
**Description:** "The database that runs your app cannot be the database that analyzes your app". It's a hard lesson learned at scale. Early on, Postgres does it all. But as you hit massive scale, your analytics queries start killing your login APIs. This post breaks down the physics of Row-oriented (Couchbase) vs Column-oriented (ClickHouse) databases, and how to bridge them using Change Data Capture (CDC) for a robust, lag-free architecture.


At a recent ClickHouse conference in New Delhi, I attended this Saturday. There were many interesting sessions, but one that stood out was a talk on multi-tenant analytics at scale. I was reminded of a fundamental truth in backend engineering: "The database that runs your app cannot be the database that analyzes your app".

Early in a startup's life, we shove everything into one database. User profiles, session data, logs, and analytics events all live in the same Postgres, Mongo, or Couchbase instance. It works, until it throttles.

Eventually, your analytics queries start creating backpressure on your login API. That is when you realize you have two distinct problems pretending to be one. You need to split your world into OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing).

In my stack, this split is represented by two heavyweights: **Couchbase** and **ClickHouse**.

> **Note:** While this post focuses on Couchbase and ClickHouse because they are part of my stack, these principles are universal.
> *   **OLTP Examples:** Postgres, MySQL, MongoDB, DynamoDB.
> *   **OLAP Examples:** Snowflake, BigQuery, Redshift, DuckDB.
>
> The physics of Row vs Column remains the same regardless of the brand.

## The fundamental Divergence: Row vs Column

The difference isn't just branding. It is physics.

### Couchbase (The OLTP Workhorse)

Couchbase is designed for **Transactions**. It is a document store that excels at "Key-Value" lookups.

Imagine you are fetching a User Profile. You want all the data for *one specific user* (Name, Email, Preferences, LastLogin).
In a Row-Oriented database (and for the sake of simplicity, a JSON Document is just a fancy Row), this data is stored contiguously on disk.
*   **The "Row" Concept:** Whether it is a PostgreSQL Tuple or a Couchbase Document, the principle is the same: all fields for one ID are stored together.
*   **Read Pattern:** Fetch row `user:123`. The disk head jumps to one spot and reads the whole block. **Fast.**
*   **Write Pattern:** Update `user:123`. The DB locks that specific row/document, updates it, and moves on. **Fast.**

### ClickHouse (The OLAP Beast)

ClickHouse is designed for **Analytics**. It is a Column-Oriented store.

Imagine you need to calculate the average age of 50 million users. You don't care about their names, emails, or session IDs.

*   **The "Column" Concept:** ClickHouse stores every `age` value for every user together on disk. All `names` are in a separate physical location.
*   **Read Pattern:** To calculate an average, the system only reads the `age` file. It skips 90% of the data it doesn't need. **Blazing Fast for Aggregations.**
*   **Write Pattern:** Updating a single record is "expensive" because it involves touching many different files. ClickHouse is optimized for appending massive batches of immutable data. **High Throughput.**


### See It In Action

Let's say you have `users.json` (1TB of Data).

**Query:** `SELECT AVG(age) FROM users`

*   **OLTP (Row/Doc):**
    1.  Load Block 1 (User A).
    2.  Parse JSON/Row.
    3.  Extract `age`.
    4.  Discard rest.
    5.  Repeat for 1 billion users.
    *   **Result:** Massive I/O waste. Your disk is reading "Name","Email","Bio" just to throw it away.

*   **OLAP (Column):**
    1.  Load `age.bin` (Just the integers).
    2.  Send array to CPU.
    3.  **Vectorized Processing (SIMD):** The CPU adds 4 or 8 integers *in a single instruction* (AVX-512). I'll write about SIMD in a future blog.
    4.  Zero branching, zero parsing overhead.
    *   **Result:** 100x faster execution. This is why ClickHouse can scan billions of rows per second on a modern laptop.

## The Architecture of the Split

So, how do you use both without going crazy? You need a bridge.

At this scale of our example, we use Couchbase for the "Live" path and ClickHouse for the "Retrospective" path.

1.  **The Live Path (Couchbase):**
    *   User logs in? Hit Couchbase.
    *   User posts a comment? Write to Couchbase.
    *   **Requirement:** Sub-millisecond latency, high concurrency, strong consistency (for the user).

2.  **The Bridge (CDC - The "Real" Engineering):**
    *   **The Amateur Move (Dual Writes):**
        ```go
        // Don't do this
        db.SaveUser(user)
        clickhouse.InsertUser(user) // What if this fails? Now your analytics is drifting.
        ```
    *   **The Pro Move (Change Data Capture or CDC):**
        We treat the database log (WAL or <span title="Database Change Protocol of couchbase">DCP</span> Stream) as the source of truth.
        *   User updates profile -> Couchbase commits to disk -> Couchbase emits event.
        *   **Kafka/Redpanda** captures this event.
        *   **ClickHouse Connector** consumes the batch and inserts it.
    *   **Why this matters:** If ClickHouse goes down for maintenance, Kafka buffers the events. When ClickHouse comes back up, it drains the buffer. **Zero data loss. Zero impact on the Login API.**

3.  **The Retrospective Path (ClickHouse):**
    *   Product Manager wants to know: "How many users logged in from iOS vs Android last month?"
    *   Dashboard query hits ClickHouse.
    *   **Requirement:** Throughput. Scanning millions of rows in milliseconds. Latency can be 200ms-500ms, but it must handle massive volume.

## Why Not Just Use Postgres for Everything?

"But Postgres has columnar indexes now!" or "What about TimescaleDB?"

Yes, tools are converging. Postgres is the swiss-army knife of databases, and for 90% of use cases, it is enough. But at high scale, general-purpose tools hit hard limits where **specialization wins**.

### 1. The WAL Bottleneck (Ingestion)
Postgres guarantees ACID compliance for every transaction. This means every insert must be written to the **Write Ahead Log (WAL)** before it is visible.
*   **Postgres:** High write volume = High WAL contention. If you try to insert 100k events/sec into Postgres, your WAL becomes the choke point.
*   **ClickHouse:** Uses `MergeTree` engines. Data is written to parts in memory and flushed to disk as immutable files. There is no traditional "WAL" for every row. It is designed to swallow firehoses of log data.

### 2. The Maintenance Tax (Vacuum vs Merges)
*   **Postgres:** Updates are "Copy on Write". Old versions of rows are kept around (dead tuples). You rely on `AUTOVACUUM` to clean them up. At terabyte scale, Vacuuming is a heavy background process that eats I/O.
*   **ClickHouse:** Updates are rare. It relies on **background merges** to combine data parts. This is efficient for write-once-read-many (WORM) patterns typical of analytics.

### 3. Resource Isolation
If you run a heavy analytical query (`GROUP BY` over 1 billion rows) on your primary Postgres instance:
*   It poisons the page cache (evicting hot OLTP usage data).
*   It maxes out CPU.
*   **Result:** Your user login times out because the DB is busy calculating "Average Age".

By splitting them, you ensure that your **Analysts' dashboard queries never bring down the Checkout page.**

---

The key to a scalable architecture is knowing the tool's purpose.

*   **OLTP (Couchbase):** Optimized for the **individual**. "Who is User X?"
*   **OLAP (ClickHouse):** Optimized for the **aggregate**. "What are users doing?"

Don't force one to do the other's job. Your on-call rotation (and your latency SLOs) will thank you.

---

## Further Reading

1.  **[OLTP vs. OLAP](https://clickhouse.com/resources/engineering/oltp-vs-olap)** – Clickhouse Engineering
    *   *An overview of the fundamental differences between transaction processing and analytical processing.*
2.  **[ClickHouse MergeTree Family](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree)** – ClickHouse Docs
    *   *Deep dive into the LSM-tree inspired architecture that powers high-speed analytics.*
3.  **[PostgreSQL and ClickHouse for Analytics](https://clickhouse.com/comparison/postgresql)** – ClickHouse
    *   *A benevolent comparison helping you decide when to stick with Postgres and when to introduce ClickHouse.*
4.  **[Couchbase Architecture Overview](https://docs.couchbase.com/server/current/learn/architecture-overview.html)** – Couchbase Docs
    *   *Understanding the memory-first, shared-nothing architecture of a modern document store.*


---

## Go Garbage Collector Mechanics: Pacer, GOGC, and Allocation Latency

**Date:** 2026-01-07
**URL:** http://localhost:1313/dont-take-out-the-garbage-go-gc-deep-dive/
**Description:** How Go's concurrent tri-color mark-sweep collector works under the hood. Learn pacer triggers, GOGC tuning, and zero-allocation memory pool patterns.


In the world of high-throughput backend services, we often obsess over the usual suspects of performance: database indexing, network latency, and algorithmic complexity. But recently, while debugging our core gateway service (`backend-gw`), I encountered a bottleneck that defied standard logic.

The service was **CPU-bound**, yet active heap usage was surprisingly low (~200MB). P99 latency was spiking at random intervals, but database queries were returning in milliseconds.

The culprit was not the business logic. It was **memory management**. I was effectively running a Denial-of-Service attack on my own runtime.

This post is a detailed breakdown of the Go Garbage Collector (GC). I'll explain how the GC *actually* works, dissect its specific phases (including the dreaded "Stop The World"), and show you how to use the Go Trace tool to identify when your application is losing the battle against allocation churn.



## Part 1: Finding the Smoking Gun

My investigation began with a standard CPU profile. The service processes JSON documents from Couchbase and serves them via HTTP. I expected the CPU time to be dominated by business rules or I/O handling.

Instead, the profile (`cpu_top_cumulative.txt`) showed this:

```text
Showing nodes accounting for 29.17s, 72.56% of 40.20s total
      flat  flat%   sum%        cum   cum%
     0.02s  0.05%  ...     19.04s 47.36%  couchbaseDB.GetWithCAS
         0     0%  ...     18.40s 45.77%  encoding/json.Unmarshal
     0.64s  1.59%  ...     13.89s 34.55%  encoding/json.(*decodeState).object
     ...
     0.48s  1.19%  ...      1.94s  4.83%  runtime.scanobject
     0.02s  0.05%  ...      1.81s  4.50%  runtime.gcDrain
```

I had **two major bottlenecks** consuming roughly equal CPU time:
1. **Database operations** (47.36% cumulative) - Couchbase fetch and I/O
2. **JSON deserialization** (45.77% cumulative) - Standard library unmarshalling

But crucially, functions like `runtime.scanobject` and `runtime.gcDrain` were consuming significant cycles. These are internal GC methods. While only 4.83% of CPU was directly in GC functions, this was a sign of a deeper problem.

The allocation profile (`allocs_top_cumulative.txt`) confirmed my suspicion. We weren't leaking memory, we were "churning" it. Allocation profile shows cumulative allocations since last deployment (4 hours in this case). 

```text
Showing nodes accounting for 11054705.39MB, 89.25% of 12386152.87MB total

2981421.22MB  24.07%  github.com/go-kit/log.With
 216546.74MB   1.75%  encoding/json.Unmarshal
```

### Understanding the Gap: Heap vs. Allocations

Here's the critical insight that I had missed. The heap profile showed:

```text
Type: inuse_space
Showing nodes accounting for 188.98MB, 92.83% of 203.58MB total
```

Let that sink in:
- **Active Heap:** 203 MB (memory in use at snapshot time)
- **Total Allocations:** 12,386 GB (cumulative garbage created since last deployment)
- **Churn Ratio:** 62,000:1 (cumulative allocations / active heap)

We were generating **twelve terabytes** of cumulative garbage while maintaining only 200MB of live data. This meant that for every byte we needed, we were creating 62,000 bytes of garbage that the GC had to clean up.

**This is allocation churn**, and it's the silent killer. The GC must scan and mark every reachable object during collection, and when you're creating millions of temporary objects, the GC spends more time cleaning than the application spends doing actual work.

We were generating this garbage using standard library JSON parsers and an inefficient logging strategy. To understand why this crushed our CPU, I had to look under the hood of the Go Runtime.

Then I started reading about the Go GC. I had some familiarity with Python's GC. So, I started reading countless articles, reading Go docs and reading the source code. After about 25 hours, I finally got some clarity on how the GC works and why the backend was so slow.



## Part 2: The Architecture of the Go GC

The Go GC is a **Non-Generational, Concurrent, Tri-Color Mark-and-Sweep** collector.

That is a mouthful. Let's break down the implications:
1.  **Non-Generational:** Unlike Java or Python, Go does not separate objects into "Young" and "Old" generations. It treats the heap as one singular space. This simplifies the runtime but means the GC must scan *everything* that is live during every cycle.
2.  **Concurrent:** The GC runs *alongside* your application code (except during brief Stop-The-World phases). It does not pause your app for the entire duration of the collection.
3.  **Mark-and-Sweep:** It traverses the object graph to find live objects ("Mark") and reclaims the rest ("Sweep").

### The Object Graph
You must visualize your application's memory as a directed graph.
*   **Nodes:** Objects (structs, strings, slices).
*   **Edges:** Pointers (references).

The GC starts at the **Roots** (global variables, stack frames of active goroutines) and follows the pointers. Anything it can reach is "Live". Anything it cannot reach is "Garbage".



## Part 3: The Lifecycle of a GC Cycle (The 4 Phases)

A GC cycle is a carefully choreographed dance between your code (the "Mutator") and the GC background workers. It consists of four specific phases.

### Phase 1: Sweep Termination (Stop-The-World)
The cycle begins here. The GC must ensure that the *previous* cycle's cleanup is 100% complete before starting a new one.

*   **What happens:** The runtime pauses **all** application goroutines (Stop-The-World, or STW). It finishes any remaining sweeping of memory spans.
*   **Latency Impact:** In modern Go versions (1.18+), this is typically sub-millisecond. However, if your heap is highly fragmented or massive, this pause can stretch, causing "jitter" in your p99 latency.

### Phase 2: Concurrent Mark (The Heavy Lifter)
This is the longest phase, accounting for the bulk of the GC-related CPU usage I saw in my profiles. The world resumes, and the GC runs in the background (consuming ~25% of CPU capacity by default).

This phase uses the **Tri-Color Invariant**:
1.  **White:** Candidate for collection (not yet visited).
2.  **Grey:** Alive, but its children (pointers) have not been scanned.
3.  **Black:** Alive, and fully scanned.

The GC picks a Grey object, scans its memory for pointers (`runtime.scanobject`), marks the referenced objects Grey, and turns the original object Black.

**The Bottleneck:** The cost of this phase is determined by **Pointer Density**, not just object size.

```text
Cost = (BytesScanned * C_1) + (PointerCount * C_2)
```

In my profiling data, I saw `runtime.scanobject` consuming 4.83% of CPU. This is because `encoding/json` creates a graph full of `interface{}` and pointer wrappers. I was forcing the GC to traverse millions of tiny edges.

### Phase 3: Mark Termination (Stop-The-World)
Once the Concurrent Mark is done, the GC must ensure no Grey objects remain.

*   **What happens:** A second Stop-The-World (STW) pause. The GC performs final housekeeping, flushes caches, and calculates the target heap size for the *next* cycle.
*   **Latency Impact:** This pause is generally longer than Sweep Termination. Its duration scales with the number of active goroutines and the complexity of the root set.

### Phase 4: Concurrent Sweep
The STW pause ends. The GC now knows that any object still marked "White" is garbage.
*   **What happens:** The GC does not explicitly "delete" every object immediately. Instead, as your application requests *new* memory (via `mallocgc`), the runtime lazily reclaims the White space.
*   **Resource Usage:** This happens largely off the critical path, but high churn can cause fragmentation here.



## Part 4: Analyzing the Trace – The "G Block" and Mark Assists

This is where the theory meets the reality of debugging. When you run `go tool trace trace.out`, you are presented with a timeline view.

### What is a "G Block"?
In the trace tool, you will see rows labeled **Proc 0, Proc 1, ...** (representing logical Processors/Cores). The colorful bars inside these rows are **G Blocks** (Goroutine execution blocks).

A "G Block" represents a specific Goroutine running on a specific Processor for a duration of time.
*   **Green:** User code running normally.
*   **Blue:** Networking/IO.
*   **Yellow/Red:** GC related delays or synchronization waiting.

### The Latency Killer: Mark Assists
The Go GC has a "Pacing" algorithm. It aims to prevent the heap from growing out of control before the GC cycle finishes.

**The Concept:**
Imagine your application is allocating memory (creating garbage) faster than the background GC workers can clean it up. The runtime realizes it is losing the race. To prevent an Out-Of-Memory (OOM) crash, it begins to "tax" your application.

**The Mechanism:**
When a Goroutine attempts to allocate memory (`runtime.mallocgc`), the runtime checks the current GC credit. If the Goroutine is in "debt" (it has allocated too much relative to the scan progress), the runtime **hijacks** the Goroutine.

Instead of running your business logic, your Goroutine is forced to switch into **Mark Assist** mode. It must scan a portion of the heap (doing the GC's job) to "pay off its debt" before it is allowed to perform the allocation.

### Interpreting the Trace
If you see this in your trace, you have an allocation problem:

1.  Look at the **"G" row** (Goroutine analysis).
2.  Look for the state labels. You want to see **"Running"**.
3.  If you see large chunks labeled **"Mark Assist"** or time spent in **"GCWaiting"** states, your CPU is not serving users; it is fighting the heap.

In my specific case, my API handlers were spending significant time in Mark Assist because `log.With` and `json.Unmarshal` were flooding the heap with pointers.



## Part 5: The Dual Problem - JSON and Logging

### The JSON Unmarshalling Problem

While I can't optimize the database (network and disk I/O are physical constraints), I *can* optimize how I process the data once it arrives. The standard library's `encoding/json` package is general-purpose and safe, but it's not optimized for high-throughput scenarios.

The issue is **how** it allocates memory:
- Creates many small allocations for each field
- Uses `interface{}` extensively, forcing heap allocations
- Generates intermediate objects during parsing
- Creates pointer-heavy object graphs

### The Logging Problem (The Silent Memory Hog)

Looking deeper at the allocation profile:

```text
2981421.22MB  24.07%  github.com/go-kit/log.With
 464404.86MB   3.75%  backend-gw/routes/rctx.LoggerWithFn
```

The go-kit logger was allocating **2.9 TB of memory** - nearly 24% of all allocations! This is because `log.With()` creates a new logger instance every time it's called, and we are calling it in HTTP middleware and every function for every single request.

Every log line with context creates:
- A new logger instance
- Copies of all context key-value pairs
- String conversions for values
- Interface boxing for non-string types

When you're handling thousands of requests per second, this adds up catastrophically.

**Why This Compounds the GC Problem:**
These allocations don't just cost memory, they create millions of objects that the GC must scan during the mark phase. Even though each logger instance is short-lived, the GC must still prove it's garbage by scanning the entire object graph.



## Part 6: The Fix and The Benchmarks

I replaced `encoding/json` with `goccy/go-json`, a drop-in replacement that's heavily optimized for performance. The impact was visible in micro-benchmarks immediately.

### Benchmark Setup

```go
func BenchmarkUnmarshal_Std_UserSE(b *testing.B) {
    data := []byte(`{"userId":"..".,"sessions":[...],...}`) // Representative data
    b.ReportAllocs()
    b.ResetTimer()
    
    for i := 0; i < b.N; i++ {
        var result UserSE
        if err := json.Unmarshal(data, &result); err != nil {
            b.Fatal(err)
        }
    }
}

func BenchmarkUnmarshal_Goccy_UserSE(b *testing.B) {
    data := []byte(`{"userId":"..".,"sessions":[...],...}`)
    b.ReportAllocs()
    b.ResetTimer()
    
    for i := 0; i < b.N; i++ {
        var result UserSE
        if err := goccyjson.Unmarshal(data, &result); err != nil {
            b.Fatal(err)
        }
    }
}
```

### Benchmark Results: The "Pointer Chasing" Proof

I benchmarked the unmarshalling of several of our critical database document types on Apple M2:

| Struct Type | Library | Time (ns/op) | Allocs/op | Bytes/op | Speedup | Alloc Reduction |
|:---|:---|---:|---:|---:|---:|---:|
| **UserSE** | encoding/json | 127,135 | **589** | 33,625 | - | - |
| | goccy/go-json | 27,118 | **273** | 32,637 | 4.7x | 53.7% |
| **UserEE** | encoding/json | 133,181 | **347** | 29,745 | - | - |
| | goccy/go-json | 25,736 | **154** | 36,283 | 5.2x | 55.6% |
| **User** | encoding/json | 4,819 | **22** | 2,392 | - | - |
| | goccy/go-json | 1,095 | **9** | 2,545 | 4.4x | 59.1% |
| **Post** | encoding/json | 6,685 | **18** | 1,184 | - | - |
| | goccy/go-json | 1,621 | **11** | 1,689 | 4.1x | 38.9% |
| **UserENR** | encoding/json | 10,345 | **44** | 1,704 | - | - |
| | goccy/go-json | 2,286 | **16** | 2,533 | 4.5x | 63.6% |

**The Analysis:**

Across all document types, I saw:
- **Average speedup:** 4.6x faster unmarshalling
- **Average allocation reduction:** 54% fewer allocations
- **Bytes per operation:** Roughly similar (slight increase in some cases)

The byte count (`B/op`) remained similar or slightly higher in some cases. **This is the key insight.** The GC does not care about the total bytes allocated as much as it cares about the **number of objects** (allocations) it must track.

When unmarshalling `UserSE`:
- Standard lib: 589 separate allocations = 589 objects to scan
- goccy/go-json: 273 allocations = 273 objects to scan

This change effectively **cut the work of `runtime.scanobject` by more than half** for these hot paths.

### Why goccy/go-json Is Faster

The performance difference comes from:
1. **Reduced intermediate allocations** - Directly writes to destination structs
2. **Optimized pointer handling** - Minimizes pointer chains in the object graph  
3. **Better memory layout** - Fewer separate heap allocations
4. **SIMD optimizations** - Uses CPU vector instructions where available

The slight increase in bytes/op for some structs is acceptable because it reduces allocation count, the metric that actually drives GC pressure.

### Next Steps: Logging Optimization

My next phase of optimization is to replace the go-kit logger with a zero-allocation alternative like zerolog or zap, and to restructure the logging middleware to reduce the quantity and improve the quality of logs.

Based on my profiling data showing 2.9TB allocated by `log.With()`, I anticipate this will:
- Eliminate ~600GB per hour of memory churn
- Free up an additional 15-20% of CPU cycles
- Further reduce GC pressure



## Architectural takeaways

### For Senior Engineers: Rethink "Clean Code"
We often prefer interfaces, wrappers, and layers of abstraction for "clean architecture". In Go, every interface wrapper that escapes to the heap is a node in the graph.
*   **The Trade-off:** Be wary of libraries that accept `interface{}` for everything (like standard `log` or `json`).
*   **Structure:** Design structs to keep pointers contiguous. The GC scans memory faster when pointers are grouped at the start or end of a struct, rather than interleaved with scalars (int/bool), due to how bitmap marking works.

### For Junior Engineers: Value vs. Pointer
*   **Pointers are not free.** Passing a pointer to a function might save a memory copy, but it might burden the GC if that pointer escapes to the heap.
*   **Pre-allocate Slices.** Use `make([]T, 0, capacity)` if you know the size. Appending to a slice causes resizing, which creates a *new* backing array and leaves the old one as garbage.

### Red Flags: Is Your Application GC-Bound?

Watch for these warning signs in your profiles:

1. **Allocation/Heap Ratio > 1000:1** - You're churning far more than you're using
2. **`runtime.scanobject` > 5% of CPU** - GC is spending significant time scanning
3. **Many small allocations** - Objects < 1KB in hot paths create pointer-chasing
4. **`interface{}` in unmarshal targets** - Forces heap allocation for every value
5. **Logger allocations in middleware** - Creating context on every request

### General Optimization Strategies

1. **Use `sync.Pool` for frequently allocated objects** - Reuse instead of allocating
2. **Prefer value types over pointers** - When objects are small (<100 bytes)
3. **Consider binary protocols** - protobuf or msgpack have less allocation overhead
4. **Batch allocations** - Pre-allocate slices to known capacity
5. **Profile before and after** - Always measure; intuition fails at scale



## Conclusion

The Go Garbage Collector is a marvel of engineering, prioritizing low latency over high throughput. However, it obeys the laws of physics. If you create millions of objects, the GC must touch millions of objects.

By analyzing my profiles and traces, I realized the service wasn't slow, it was just distracted. The CPU was spending its time:
- Fetching data from Couchbase (necessary I/O)
- Deserializing JSON with inefficient allocations (optimizable)
- Creating logger instances for every request (optimizable)
- Running the GC to clean up the resulting garbage (consequence)

By addressing the two optimizable factors, switching to `goccy/go-json` and planning to fix the logging, I'm reclaiming significant CPU capacity. I reduced allocation counts by 50%+, which directly reduces the work that `runtime.scanobject` must perform during each GC cycle.

**The lesson:** In high-performance Go services, allocation count matters more than allocation size. The GC must scan the object graph, and every object is a node. Minimize the nodes, and you minimize the GC's work.

Understanding your memory allocation patterns through profiling is not optional, it's essential for building services that scale.



**Reference Documentation:**
*   [A Guide to the Go Garbage Collector (Official)](https://tip.golang.org/doc/gc-guide)
*   [Go Tool Trace User Guide](https://go.dev/doc/diagnostics#trace)
*   [Go Memory Model](https://go.dev/ref/mem)
*   [goccy/go-json on GitHub](https://github.com/goccy/go-json)



**Profiling Commands Used:**
```bash
# CPU profiling
go test -cpuprofile=cpu.prof -bench=. ./...
go tool pprof -top cpu.prof

# Memory allocation profiling  
go test -memprofile=mem.prof -bench=. ./...
go tool pprof -alloc_space -top mem.prof

# Heap profiling
go tool pprof -inuse_space -top mem.prof

# Generating trace
go test -trace=trace.out -bench=. ./...
go tool trace trace.out
```

**Collect profiling data from production systems:**
```bash
# 1. Collect the raw data
{
    # 0. CPU Profile: Identifies exactly which line is creating the most garbage.
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/profile?seconds=30" -o cpu.pprof &
    # 3. Execution Trace: Captures a 10-second timeline of every event (GC, Network, Scheduler).
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/trace?seconds=15" -o trace.out &

    sleep 2
    # 1. Heap Profile: Identifies exactly which line is creating the most garbage.
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/heap" -o heap.pprof

    # 2. Goroutine Profile: Shows what all concurrent workers are doing at this exact moment.
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/goroutine" -o goroutine.pprof

    # Block Profile: Shows where code is waiting for Channels or Network.
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/block" -o block.pprof

    # Mutex Profile: Shows which locks are causing contention.
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/mutex" -o mutex.pprof

    # Allocation profiles
    curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/allocs" -o allocs.pprof

    wait
    echo "All 30s/15s profiles collected."
}

# 2. Convert the data into readable format (you can use web version as well)
{
    go tool pprof -top -cum cpu.pprof > cpu_top_cumulative.txt
    go tool pprof -tree cpu.pprof >> cpu_tree.txt

    go tool pprof -top -cum heap.pprof > heap_top_cumulative.txt
    go tool pprof -tree heap.pprof >> heap_tree.txt

    go tool pprof -top -cum allocs.pprof > allocs_top_cumulative.txt
    go tool pprof -tree allocs.pprof >> allocs_tree.txt

    go tool pprof -top -cum block.pprof > block_top_cumulative.txt
    go tool pprof -tree block.pprof >> block_tree.txt

    go tool pprof -top -cum mutex.pprof > mutex_top_cumulative.txt
    go tool pprof -tree mutex.pprof >> mutex_tree.txt

    go tool pprof -top goroutine.pprof >> goroutines.txt

    go tool trace -pprof=sync trace.out > trace_sync.pprof
    go tool pprof -top trace_sync.pprof > trace_summary.txt
}
```

Then you can open these text files in any text editor to view the profiles. You can also upload these to an LLM to get insights. I used Claude Opus 4.5 for correlating these prifiles with my p90 and p99 metrics and keeping them open on the side to read and cross-reference. While you are reading the profiles you should also look at allocation patterns in your code and make notes of them to see if you can optimize them.

**Note:** If you use goccy/go-json, test your code extensively. It is a drop-in replacement for encoding/json as claimed, but it has some issues parsing multibyte characters in some languages like Hindi.

PS: Just deployed to prod, and p90 and p99 are looking much better. 

Next: I will be testing the bytedance/sonic library and see how it's JIT and SIMD features compare to goccy/go-json. And maybe I will try to build a simple transcoder to learn more about GC, SIMD, JIT and systems programming. I am liking this a lot.

Until next time.


---

## Introducing CouchLens: A Query Analysis Tool for Couchbase

**Date:** 2025-12-28
**URL:** http://localhost:1313/couchlens-couchbase-query-analysis-tool/
**Description:** CouchLens is a browser-based tool for analyzing Couchbase N1QL query performance. It parses system tables, extracts execution plans, and generates insights to help database administrators find performance bottlenecks without sending data to external servers. This post explains what it does, how to use it, and what features are coming.


CouchLens is a client-side web application for analyzing Couchbase N1QL query performance. You feed it JSON exports from `system:completed_requests`, `system:indexes`, and schema inference results. It parses execution plans, computes metrics, detects inefficiencies, and generates a report showing where your queries are slow.

Everything runs in the browser. No data leaves your machine. The tool is a Progressive Web App, so you can install it and use it offline. The goal is to give database administrators and developers a way to understand query behavior without writing custom scripts or debugging raw JSON.

### Why I Built This

I work for an organization that relies heavily on Couchbase. Although my background was primarily in traditional SQL databases (MySQL, MariaDB, PostgreSQL) and occasionally MongoDB, being introduced to Couchbase here was a revelation. I fell in love with the capability of running SQL on JSON via SQL++ (formerly N1QL).

This enthusiasm led me to dive deep into Couchbase's internals. I constantly read about how the query planner, indexing strategies, cost-based optimizer, and storage engines function. Concepts I explored in {{< link href="/what-couchbase-taught-me-about-system-thinking" text="What Couchbase Taught Me About System Thinking" target="_blank" >}}
I built this tool as a way to understand these concepts better and to share my learnings.

**You can give it a try at {{< link href="https://couchlens.lorbic.com/#input?ref=lorbic.com" text="couchlens.lorbic.com" target="_blank" >}}.**

Looking ahead, I am working on an index generation engine. This engine will look at your document schema and query patterns to suggest covering indexes or identify inefficiencies in existing ones. These features are currently in development and will eventually be integrated into CouchLens.

This post describes what CouchLens does now (version 0.3 alpha), how it works, and what features are planned.

---

### What It Does

CouchLens provides eight main views organized into tabs:

#### Dashboard

Charts showing query throughput over time, average durations, and statement type distribution. Use this to identify periods of high load or performance degradation.

![CouchLens Dashboard displaying query throughput charts and duration metrics](couchlens.lorbic.com_dashboard.webp)

#### Insights

Automated detection of performance issues. The insight engine scans execution plans and flags queries that match 11 predefined patterns:

**Index Issues:**

- Inefficient index scans (scanned 50k+ rows but returned less than 10%)
- Pagination over-fetch (ORDER BY + LIMIT + OFFSET scanning 10k+ items)
- Primary index usage (full bucket scans)

**Pattern Analysis:**

- SELECT \* usage
- LIKE with leading wildcards
- Missing WHERE clauses
- Slow USE KEYS operations (taking more than 20ms)

**Resource Issues:**

- High memory usage (more than 10MB per query)
- High kernel time (more than 10ms, indicating OS scheduling pressure)
- Slow parse or plan phases (more than 5ms)

**Performance:**

- Long-running queries (more than 1 second)
- Large result sets (more than 20MB)
- Concurrent query conflicts (heuristic based on fetch latency, index scan throughput, and kernel time ratio)

Each insight includes affected query counts, severity (critical/warning/info), and recommendations.

![CouchLens Insights tab highlighting performance issues and recommendations](couchlens.lorbic.com_insights.webp)

#### Analysis

Statistical breakdown of query performance: p50, p95, p99 latencies, min/max/average durations, and performance grouped by statement type (SELECT, INSERT, UPDATE, etc.). Lists the top 10 slowest queries.

![CouchLens Analysis tab showing query performance statistics including latency percentiles and slowest queries](couchlens.lorbic.com_analysis.webp)

#### Queries

A sortable, searchable table of every query in the dataset. Sort by duration, result count, or payload size. Full-text search to find specific SQL patterns. Case-insensitive filtering.

#### Flow

A Sankey diagram showing which queries use which indexes. Visualizes index usage patterns. Helps identify underutilized indexes or queries hitting primary indexes. Includes text filters to include or exclude specific index names.

![CouchLens Flow diagram visualizing the relationship between queries and indexes](couchlens.lorbic.com_flow.webp)

#### Indexes

A filterable table of index definitions with search by name or definition, status badges (online/building/offline), and copy-to-clipboard support.

#### Schema

Hierarchical view of buckets, scopes, and collections with type analysis for each field. Detects mixed-type fields (where a field is a number in some documents and a string in others) and highlights date/time fields and unique identifiers.

#### Report

Selectively generate a printable report combining multiple sections (Dashboard, Insights, Analysis, etc.) into a single view. Export to PDF via browser print.

![CouchLens Report view for generating printable summaries](couchlens.lorbic.com_report.webp)

The tool also includes integrated documentation (Getting Started, SQL Queries reference, User Guide, and Changelog) accessible from the Help tab.

---

### How to Use It

**Collect Data**

Run these queries in Couchbase Query Workbench:

```sql
-- Completed Requests
SELECT *, meta().plan
FROM system:completed_requests;
```

```sql
-- Indexes
SELECT
 s.name,
 s.id,
 s.metadata,
 s.state,
 s.num_replica,
CONCAT("CREATE INDEX ", s.name, " ON ", k, ks, p, w, ";") AS indexString
FROM system:indexes AS s
LET bid = CONCAT("", s.bucket_id, ""),
    sid = CONCAT("", s.scope_id, ""),
    kid = CONCAT("", s.keyspace_id, ""),
    k = NVL2(bid, CONCAT2(".", bid, sid, kid), kid),
    ks = CASE WHEN s.is_primary THEN "" ELSE "(" || CONCAT2(",", s.index_key) || ")" END,
    w = CASE WHEN s.condition IS NOT NULL THEN " WHERE " || REPLACE(s.condition, '"', "'") ELSE "" END,
    p = CASE WHEN s.`partition` IS NOT NULL THEN " PARTITION BY " || s.`partition` ELSE "" END;
```

```sql
-- Schema (per collection)
INFER `bucket`.`scope`.`collection` WITH {"sample_size": 10000};
```

Export each result as JSON.

**Load Data**

Open CouchLens. Navigate to the Input tab. Upload the JSON files or paste JSON directly. Click Analyze. Parsing typically completes in under a second for 10,000 queries.

![CouchLens Input tab for uploading JSON data files](couchlens.lorbic.com_input.webp)

**Explore**

Navigate through tabs to review metrics, insights, schema, and individual queries. Use insights to prioritize optimization work. Generate reports for documentation or sharing.

---

### Design Decisions

CouchLens uses Svelte 5 and TypeScript. Svelte provides reactive state management with a small runtime. TypeScript ensures type safety during data transformations. All parsing and analysis runs client-side in the browser's JavaScript runtime.

The tool does not connect to Couchbase directly. This avoids security concerns and deployment complexity. You export data once and analyze locally. For continuous monitoring, collect new exports periodically.

Charts use Chart.js with zooming and panning enabled. The flow diagram uses a custom Sankey implementation. The insight engine is rule-based. Each insight category has thresholds and pattern matchers defined in `src/lib/insights.ts`. Thresholds are currently hardcoded but tunable by editing the source.

The app is a Progressive Web App. Once installed, it works offline. The service worker caches assets for fast loading.

---

### Current Limitations

CouchLens is version 0.3 alpha. Known limitations:

- **No Historical Tracking**: Each analysis session is independent. Tracking performance over time requires manual export and comparison.
- **No Auto-Refresh**: You must manually re-export and re-upload data to see new queries.
- **Fixed Thresholds**: Insight thresholds are hardcoded. Different workloads may need different definitions of "slow" or "inefficient".
- **Limited Correlation**: The tool does not correlate queries with external events (deployments, traffic spikes, configuration changes).
- **No Background Monitoring**: There is no alerting when new issues appear.

---

### Roadmap

Future versions will focus on making CouchLens more useful for ongoing performance work. These plans are not committed and may change based on my availability and requirement.

#### Planned Features

**Index Recommendations**  
A rule-based recommendation engine that suggests specific indexes for queries scanning too many rows or using primary indexes. The engine will parse query predicates and suggest covering indexes or compound indexes based on filter patterns.

**Custom Thresholds**  
User-configurable thresholds for each insight category. Allow users to define what "slow" or "inefficient" means for their workload.

**Query Profiling Wizard**  
A guided workflow that takes a slow query, generates explanations of execution plan phases, and suggests specific optimizations (add index, rewrite WHERE clause, use covering index, etc.).

**Historical Comparison**  
Store past analysis results (in browser local storage or exported files) and compare performance across time periods. Show which queries degraded, improved, or appeared/disappeared.

**Direct Couchbase Integration (Optional)**  
An optional mode that connects directly to Couchbase and pulls fresh data on demand. For users who prefer live data over manual exports. This would be opt-in.

**Alerting**  
Background checks that notify when p99 latency crosses thresholds or when new inefficient queries appear in fresh datasets.

**Export Reports to File**  
Generate standalone HTML or PDF reports summarizing insights and metrics for sharing or archiving.

Priority will be given to index recommendations and custom thresholds as these directly address current pain points.

---

### Operational Cost

Right now CouchLens runs entirely in the browser. There is no backend to maintain, no database to configure, no infrastructure to deploy.

The queries you run against `system:completed_requests` and `system:indexes` do have a cost on the Couchbase cluster. `system:completed_requests` is bounded by the `completed-threshold` setting (default 1000 queries). Fetching this data is a single query. `system:indexes` typically contains hundreds of indexes, not thousands, so fetching is cheap. Schema inference with `INFER` samples documents and can be expensive on large collections. Use `sample_size` to control cost.

Once you have JSON exports, analysis completes in under a second for 10,000 queries on a modern browser.

---

### Comparison to Other Tools

Couchbase provides the Query Workbench, which shows `EXPLAIN` and `PROFILE` for individual queries. The Couchbase Server UI shows cluster-level metrics. CouchLens fits between these tools. It provides batch analysis across many queries and automates pattern detection that would otherwise require manual inspection of each query.

For real-time monitoring with dashboards and alerting, use Prometheus, Grafana, and the Couchbase exporter. CouchLens is for ad-hoc analysis and optimization work, not 24/7 monitoring. Although it can become one in future.

For application-level tracing (tracking queries from application code through to database execution), integrate OpenTelemetry or a similar distributed tracing framework. CouchLens focuses only on the database layer.

---

### Technical Details

The codebase structure:

```
src/
├── components/       # Svelte UI components (dashboard, insights, etc.)
├── lib/
│   ├── parsing.ts    # Query log parsing, execution plan extraction
│   ├── insights.ts   # Insight detection rules and thresholds
│   ├── schema.ts     # Schema type analysis
│   ├── aggregation.ts # Data aggregation for charts
│   ├── types.ts      # TypeScript type definitions
│   └── stores.ts     # Svelte reactive state stores
└── App.svelte        # Main application with routing
```

Parsing logic in `lib/parsing.ts` extracts fields from `system:completed_requests`, computes derived metrics (elapsed time in milliseconds, statement type, indexes used), and normalizes timestamps. Insight detection in `lib/insights.ts` scans the parsed dataset and returns structured results with affected query IDs, severity, and recommendations.

Chart aggregation in `lib/aggregation.ts` groups queries by time intervals (minute, hour, day) and computes throughput and average durations per bucket. Schema analysis in `lib/schema.ts` recursively walks the `INFER` output and detects type inconsistencies and special field types (dates, UUIDs, etc.).

---

### Conclusion

CouchLens is a tool for parsing Couchbase query logs and finding inefficiencies. It runs in the browser, requires no backend, and works offline. It automates detection of common performance issues and provides a structured way to explore query behavior across large datasets.

The tool is not a replacement for `EXPLAIN` or `PROFILE`. It is a complement. Use CouchLens to find which queries need investigation, then use `EXPLAIN` and `PROFILE` to diagnose the specific problem and validate fixes.

Future versions will add index recommendations, customizable thresholds, and historical tracking.

If you manage Couchbase clusters and spend time optimizing N1QL queries, try CouchLens. Load your query logs and see what it finds.

Further readings:

- https://docs.couchbase.com/server/current/n1ql/n1ql-manage/monitoring-n1ql-query.html
- https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/explain.html
- https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/query-hints.html


---

## What Couchbase Taught Me About System Thinking

**Date:** 2025-11-18
**URL:** http://localhost:1313/what-couchbase-taught-me-about-system-thinking/
**Description:** Working with Couchbase for the past year and a half has been more than just learning a database. It has been an exercise in system thinking: seeing how indexes, queries, consistency, and durability interact, and how small design choices ripple through performance and reliability.
This essay is my personal notebook from that journey. It is technically dense because Couchbase demands precision, but it is also reflective because the lessons extend beyond one database. Understanding how array indexes multiply entries, why compound index order matters, or how query consistency flags change latency is not just about Couchbase, it is about learning to think in terms of systems, trade‑offs, and consequences.


## Couchbase Internals: Indexes, Queries, Consistency, and Performance

### Introduction

Over the last few years I've lived deep inside backend systems, and for the past year and a half Couchbase has been my daily companion. Working with Go services that depend on Couchbase taught me that the real lessons aren't in the marketing slides or quick‑start guides. They're in the internals: how indexes are built, how queries are planned, how consistency flags change the story, and how durability levels quietly decide whether your system survives a failure or not.

This post is my attempt to capture those lessons in one place. It's not a tutorial or a sales pitch. It's a personal forensic walk through the parts of Couchbase that I had to understand to keep my systems stable. I'll show you the indexes I built, the queries I ran, and the mistakes I made. Think of it as a long journal entry written for other engineers who want to see what Couchbase looks like when you peel back the surface.

This post is a detailed, technical exploration of the internal mechanics that determine Couchbase behavior in production. It assumes you have practical familiarity with JSON document databases, basic distributed systems concepts, and a working knowledge of Go and N1QL.

The goal is to explain cause and effect: why an index behaves the way it does, why a query plan chooses a document fetch instead of an index seek, and how durability choices translate into latency. This post is built on practical examples. Each index example below is followed by what the index stores, how queries use it, and the operational cost you must pay to maintain it.

---

### Architecture and Data Model

Couchbase is a set of cooperating services.

- The **Data Service** implements the key-value interface, persistence, and replica mechanics.
- The **Index Service** builds and serves Global Secondary Indexes (GSIs).
- The **Query Service** parses N1QL, plans execution, and orchestrates distributed operations.

Documents are assigned to logical shards called **vBuckets**. A hashing function maps document keys to vBuckets, and the cluster maintains a vBucket map that SDKs use for direct routing of key-value operations. Rebalancing moves vBuckets across nodes, streaming data and per-vBucket sequence metadata.

Two storage engines are important: **Couchstore** and **Magma**.

- **Couchstore** behaves like a compacting, append-oriented structure with B-tree-like lookup characteristics.
- **Magma** behaves like a Log-Structured Merge-Tree (LSM) store, optimized for large datasets and high write throughput.

This choice affects **write amplification** - a term describing how a single database write (like an update) can cause multiple I/O operations on disk as data is compacted or moved. In production, you must watch compaction metrics and index backfill durations, as these operations reveal whether the chosen engine fits your workload.

Document shape matters. Couchbase stores JSON documents with metadata such as CAS, expiration, and optional extended attributes. Updates use a **copy-on-write** model from the client's perspective, meaning the entire document is typically sent for any change.

- **Large or "hot" documents** (frequently updated) increase write amplification and amplify index maintenance cost when indexed fields change.
- **Embedding** many logically independent items in a single document gives read locality (fast reads) at the cost of heavier writes and larger fetch units.
- **Splitting** independent items across documents reduces write amplification and allows finer-grained sharding by vBucket.

---

### Indexing Deep Dive with Examples

A **Global Secondary Index (GSI)** maps keys derived from document fields to document identifiers. Indexes significantly change read cost and raise write cost proportionally to the number and size of indexed fields.

- **Array elements** create multiple index entries per document.
- **Compound indexes** create keys ordered alphabetically or numerically (a "lexographical" order).
- **Partial indexes** restrict entries to only those documents that match a specific predicate (a `WHERE` clause).
- **Covering indexes** store all fields required by a query _inside_ the index itself, so a query can be answered without fetching the full document. This is achieved by making all filtered and projected fields part of the index key.

Below are detailed, concrete index examples, each followed by its semantics, storage, query use, and operational trade-offs.

#### Example 1 - Primary Index, Full Bucket Scans

```sql
CREATE PRIMARY INDEX ON `bucket`;
```

- **Semantics and Use:** The primary index provides a mapping of every document ID, allowing N1QL to perform full bucket scans for ad-hoc queries. This is useful for development but is rarely acceptable for production queries on large datasets.
- **Operational Cost:** Full scans scale linearly with document count and cause heavy I/O. Keep this index only if you explicitly need ad-hoc scanning capabilities.

#### Example 2 - Single Field Secondary Index

```sql
CREATE INDEX idx_user_age ON `users`(age);
```

- **What it Stores:** For each document containing `age`, the index stores the `age` value and a pointer to the document ID.
- **Query Usage:** A predicate `WHERE age = 30` becomes an index seek on `idx_user_age`. If the query returns only `age` and the document ID, the index can serve the request without a document fetch. If the query projects other fields (e.g., `SELECT name...`), a Fetch is required.
- **Operational Cost:** Each mutation that changes `age` must update the index. The cost is proportional to the write rate and the index size.

#### Example 3 - Compound Index and Left-Prefix Semantics

```sql
CREATE INDEX idx_user_age_name ON `users`(age, name);
```

- **What it Stores:** Tuples of `(age, name)` mapped to document IDs. Keys are ordered by `age` first, then `name`.
- **Query Implications:** This demonstrates **left-prefix semantics**. Queries must use the indexed fields from left to right.
  - `WHERE age = 30` **can** use the index (a left-prefix seek).
  - `WHERE age = 30 AND name = "Alice"` **can** use the index (a full compound seek).
  - `WHERE name = "Alice"` **cannot** use this index for a seek, because `age` is the leading component and is not included in the query.
- **Operational Guidance:** Place the most selective or most commonly filtered field first, especially when using equality predicates.

#### Example 4 - Compound Index with Range Predicates

```sql
CREATE INDEX idx_user_country_age ON `users`(country, age);
```

- **Query Patterns:**
  - `WHERE country = "IN" AND age BETWEEN 25 AND 35` benefits from this index. It performs an efficient seek on `country = "IN"` and then a contiguous range scan for the `age`.
- **Design Rule:** For compound indexes mixing equality and range predicates, place the equality columns _before_ the range columns.

#### Example 5 - Array Index, Distinct Semantics

```sql
CREATE INDEX idx_posts_tags_distinct ON `posts`(DISTINCT ARRAY t FOR t IN tags END);
```

- **What it Stores:** For each document, the index stores an entry for each _unique_ element `t` in the `tags` array. If a document has `tags: ["go", "go"]`, `DISTINCT` ensures only one index entry for "go" is created for that document.
- **Query Usage:**
  ```sql
  SELECT META().id FROM posts WHERE ANY t IN tags SATISFIES t = "go" END;
  ```
  The query becomes an index seek for the key "go". The engine returns one entry per matching document.
- **Operational Cost:** Cost is proportional to the total number of _unique_ array elements across all documents.

#### Example 6 - Array Index without DISTINCT

```sql
CREATE INDEX idx_posts_tags ON `posts`(ARRAY t FOR t IN tags END);
```

- **What it Stores:** An entry for _every_ array element, including duplicates. A document with `tags: ["go", "go"]` will produce two index entries for "go".
- **Query Semantics:** Index scans return multiple entries per document if multiple elements match. The query engine must then deduplicate these results, which adds CPU cost.

#### Example 7 - Compound Index that Includes an Array Element

```sql
CREATE INDEX idx_user_age_tag ON `users`(age, DISTINCT ARRAY t FOR t IN tags END);
```

- **What it Stores:** Tuples such as `(age, tag)` where `tag` iterates over the distinct tags in each document.
- **Example Query:**

  ```sql
  SELECT u.id, t
  FROM users u
  UNNEST u.tags t
  WHERE u.age BETWEEN 25 AND 35 AND t = "k8s";
  ```

- **How the Planner Can Use It:** The index supports an `age` range seek and a specific `tag` match simultaneously.
- **Operational Impact:** This index is more selective but also larger, as it stores a tuple for every `(age, tag)` pair.

#### Example 8 - Partial Index to Reduce Index Size

```sql
CREATE INDEX idx_active_users ON `users`(signup_date)
WHERE status = "active";
```

- **What it Stores:** Only documents with `status = "active"` appear in the index.
- **Use Case:** If the majority of queries target active users, this index saves significant space and increases selectivity.
- **Operational Cost:** Updates that change `status` (e.g., from "pending" to "active") require an index entry insertion or deletion. Partial indexes work best when the predicate defines a stable, frequently queried subset.

#### Example 9 - Covering Index

```sql
CREATE INDEX idx_cover_profile ON `users`(last_login, email, name);
```

- **What it Stores:** The index stores a composite key of `(last_login, email, name)` mapped to document IDs.
- **Query Example:**
  ```sql
  SELECT name, email FROM users WHERE last_login > "2025-01-01";
  ```
- **Execution:** The query can be satisfied entirely by the index. It seeks on `last_login` (the leading key) and then projects the `name` and `email` values directly from the index keys, avoiding a full document Fetch.
- **Trade-off:** Adding fields to the index key increases index size and write cost, but it can dramatically reduce query latency for high-frequency, read-heavy queries.

#### Example 10 - Indexes for JOIN Probes

- **Index Definitions:**
  ```sql
  CREATE INDEX idx_orders_userid ON `orders`(user_id);
  CREATE INDEX idx_users_id ON `users`(id);
  ```
- **Query:**
  ```sql
  SELECT o.*, u.*
  FROM orders o
  JOIN users u ON o.user_id = u.id
  WHERE o.date > "2025-01-01";
  ```
- **Planner Choices:**
  1.  **Nested Loop:** If the planner expects few orders for the date range, it will scan `orders` and then "probe" the `users` table using `idx_users_id` for each `user_id`. Each probe is an efficient index lookup.
  2.  **Hash Join:** If the planner expects many orders, it may choose a hash join and build an in-memory hash table of one side, provided memory permits.
- **Diagnostic Action:** Use `EXPLAIN` and `PROFILE` to compare estimated sizes against real row counts. If estimates are wrong, consider improving statistics or rewriting the query.

#### Example 11 - Composite Index with Function-Based Keys

```sql
CREATE INDEX idx_lower_email ON `users`(LOWER(email));
```

- **Use Case:** When queries request case-insensitive email lookups (e.g., `WHERE LOWER(email) = "..."`), indexing the computed value removes the need to fetch and compute during query time.
- **Operational Caveat:** Index maintenance requires computing the function on every mutation that might change the input field.

#### Example 12 - Index Including Geospatial or Numeric Projections

```sql
CREATE INDEX idx_locations ON `places`(dist_lat, dist_lon, name);
```

- **Index Use:** When queries perform bounding box or distance filters (e.g., on `dist_lat` and `dist_lon`) and also need to return the `name`, placing all fields in the index key allows the index to cover the query.
- **Design Considerations:** Be mindful of precision and range bucket sizes, which affect selectivity.

---

### Query Planning, Cardinality Estimation, and Execution

N1QL queries are transformed into physical plans formed from operators such as `IndexScan`, `Fetch`, `Filter`, `NestedLoopJoin`, and `HashJoin`.

The planner relies heavily on statistics. The most fragile part of this process is **cardinality estimation** - the planner's _guess_ at how many documents will match a filter. It often assumes columns are independent (e..g., `city` and `job_title` are unrelated). When columns _are_ correlated (e.g., "San Francisco" and "Software Engineer"), the planner's guesses can be wildly wrong, leading to poor join choices.

#### Statistics: Collection and Consequences

Statistics (distinct counts, histograms) are collected during index builds and via the `COLLECT STATS` or `ANALYZE` operations.

**Stale statistics** (out-of-date information) create wrong cost estimates. If your data changes often (**data drift**) or has **heavy skew** (an uneven distribution of values), you must refresh statistics more frequently or design targeted indexes to reduce dependence on inaccurate estimates.

#### Recognizing Bad Plans

A common diagnostic pattern is to run `EXPLAIN` to see the _estimated_ operator row counts, then run `PROFILE` to run the query and see the _actual_ row counts and operator timing.

When you see a large discrepancy between estimated and actual rows at a particular operator, the logical fix is one or more of these actions:

- Tune or refresh statistics for the indexes involved.
- Create smaller, targeted indexes that expose correlation between fields.
- Rewrite the query so its selectivity is more explicit (e.g., move predicates into a sub-select).
- Add covering index projections to avoid expensive Fetch operations.
- Use hints to force an index or join strategy while you diagnose the root cause.

#### Join Strategies: Pick the Right Tool

- A **Nested Loop Join** is efficient when the "outer" side of the join is small; the join then probes the "inner" side by its index.
- A **Hash Join** suits large datasets where both sides can be materialized within memory constraints.

Wrong planner estimates often cause the planner to pick a nested loop join against a large outer set. This results in millions of index probes and thousands of Fetches, revealing the need for better statistics or query refactoring.

---

### Consistency, Durability, and Mutation Ordering

Couchbase exposes query consistency and durability options. Consistency controls whether a query must reflect recent mutations. Durability controls whether a mutation is acknowledged only after being replicated or persisted.

#### Query Consistency Options

The common modes are `NOT_BOUNDED`, `REQUEST_PLUS`, and `STATEMENT_PLUS`.

- **`NOT_BOUNDED`:** This is the fastest. It does no index synchronization, and queries may return slightly stale results (i.e., not reflecting writes made milliseconds ago).
- **`REQUEST_PLUS`:** Guarantees that a query observes mutations that _that specific client_ performed prior to the query. It uses mutation tokens to achieve this.
- **`STATEMENT_PLUS`:** Similar to `REQUEST_PLUS`, but applies when coordinating multiple statements in a single logical sequence.

#### Mechanics Behind REQUEST_PLUS

When an SDK performs a mutation, it can ask for a **mutation token**. This token contains the vBucket ID and a sequence number for that write. The client then attaches this token to a subsequent query.

The index service checks its local sequence numbers for that vBucket and will pause (or block) the query until its index has processed mutations up to the requested sequence. This targeted synchronization is far more efficient than forcing a global index refresh.

#### Durability Levels and Costs

Durability levels such as `NONE`, `MAJORITY`, and `MAJORITY_AND_PERSIST` control how many replicas must acknowledge a mutation.

- Use `MAJORITY` when you need to survive single-node failures without data loss.
- Use `MAJORITY_AND_PERSIST` when you require on-disk durability across a majority of nodes.
- Each level increases write latency and affects throughput. For high-throughput write paths, prefer `NONE` or asynchronous replication if your application can tolerate it.

#### CAS and Optimistic Concurrency

**CAS (Compare-And-Swap)** tokens permit optimistic concurrency control.

1.  Read a document, and get its unique CAS value.
2.  Perform a conditional write _with_ that CAS value.
3.  If the CAS has changed (meaning someone else modified the document in the meantime), the write fails.
4.  Your application must then retry the read-modify-write cycle or reconcile the conflict.

CAS works well in low-contention patterns but induces retry cycles under "hot key" contention.

#### XDCR and Conflict Resolution

**XDCR (Cross-Datacenter Replication)** uses metadata to resolve conflicts according to a set policy (e.g., last-writer-wins). Active-active designs require application-level awareness of conflict semantics, because a simple metadata-based resolution may not match your domain-level business rules.

---

### Performance Engineering, Operations, and Experiments

Performance engineering must be based on data. Observability is the first necessity. Monitor indexer memory, index build progress, index disk usage, KV get and mutate latencies, **resident ratio** (the percentage of active data held in RAM), compaction throughput, disk IOPS, and CPU per service.

#### Compaction and Storage Engine Effects

Compaction is a background process. Couchstore compaction rewrites files to remove **tombstones** (markers for deleted data) and obsolete versions. Magma compaction (LSM-based) produces different I/O patterns. On large clusters, compaction or index backfills can saturate I/O and cause tail latency spikes. Schedule heavy maintenance windows and test compaction settings under load.

#### Hot Keys and Sharding

A **hot key** (a single document or small set of keys receiving disproportionate writes) produces tail latency and write contention. To mitigate this, shard the logical state across multiple documents and use deterministic sharding schemes or application-level throttles for heavy producers.

#### Go SDK Tuning and Client Behavior

The Go SDK offers connection pooling, retry behavior, and Durability options. Tune connection pool sizes, pipeline depth, and maximum in-flight requests to match cluster capacity. Use mutation tokens to apply `REQUEST_PLUS` only on paths that require it. Avoid aggressive client-side retries, which amplify server load.

#### Reproducible Experiments

Below are experiment outlines that isolate variables to reveal index behavior.

**Experiment A - Array Cardinality Effect**

- **Data:** 100k documents with `tags` arrays of sizes 1, 10, 100, and 1000 (in separate runs).
- **Index:**
  ```sql
  CREATE INDEX idx_posts_tags_distinct ON `posts`(DISTINCT ARRAY t FOR t IN tags END);
  ```
- **Measure:** Index disk size, build time, per-mutation index update latency, and query latency for `ANY t IN tags SATISFIES t = "X" END`.
- **Expected Outcome:** Index size scales with the total number of unique array elements. Per-mutation cost increases with array size. Queries for rare tags remain efficient, but index build and maintenance cost become the bottleneck.

**Experiment B - Cardinality Skew and Join Planning**

- **Data:** Two 1-million-document datasets. Dataset 1 has a uniform distribution on field `x`. Dataset 2 has 99% of documents with `x = "A"`.
- **Indexes:**
  ```sql
  CREATE INDEX idx_items_x ON `items`(x);
  CREATE INDEX idx_other_itemkey ON `other`(item_key);
  ```
- **Query:**
  ```sql
  SELECT i.*, o.*
  FROM items i
  JOIN other o ON i.key = o.item_key
  WHERE i.x = "somevalue";
  ```
- **Measure:** `EXPLAIN` and `PROFILE` outputs, operator row counts, join choice, and end-to-end latency.
- **Expected Outcome:** The skewed dataset will reveal planner misestimation, likely causing bad nested loop joins with many probes.

**Experiment C - Durability Latency Trade-off**

- **Operation:** Upsert a document with Durability `NONE`, `MAJORITY`, and `MAJORITY_AND_PERSIST`.
- **Measure:** p50, p95, and p99 write latency and throughput under identical concurrency.
- **Expected Outcome:** Write latency and throughput degrade as the durability level increases.

#### Minimal Go Harness Snippet for Mutation Latency

Collect Couchbase server metrics from the REST API and combine them with client timings from a harness like this.

```go
package main

import (
  "context"
  "fmt"
  "time"

  "github.com/couchbase/gocb/v2"
)

func main() {
  cluster, err := gocb.Connect("couchbase://localhost", gocb.ClusterOptions{
    Username: "Administrator",
    Password: "password",
  })
  if err != nil {
    panic(err)
  }
  bucket := cluster.Bucket("test")
  coll := bucket.DefaultCollection()
  ctx := context.Background()

  start := time.Now()
  _, err = coll.Upsert("docKey", map[string]interface{}{"name": "test"}, &gocb.UpsertOptions{
    DurabilityLevel: gocb.DurabilityLevelMajority,
    Timeout:         5 * time.Second,
  })
  elapsed := time.Since(start)
  if err != nil {
    fmt.Println("upsert error", err)
  } else {
    fmt.Println("upsert ok", elapsed)
  }
  _ = ctx
}
```

---

### Remediation Patterns for Common Failures

When `PROFILE` shows that an `IndexScan` returns far more rows than estimated and a subsequent `Fetch` consumes large resources, use this ordered approach:

1.  **Check covering potential.** Add the projected fields to the end of the index key list (e.g., `CREATE INDEX... ON \`bucket\`(filter_field, projected_field1, projected_field2)\`) if the index size increase is acceptable. This is often the biggest win.
2.  **Refresh index statistics** so that the optimizer uses recent histograms.
3.  **Create partial indexes** for hot predicates (e.g., `WHERE type="widget"`) to improve selectivity.
4.  **Rewrite the query** to make selectivity explicit or to push predicates earlier in the plan.
5.  **Apply hints** to force an index or join strategy while you iterate on diagnostics.

---

### Conclusion and Practical Checklist

Index design and data modeling are not separate tasks; they are deeply connected. Each index you add is a measured trade-off between read latency and write cost. Indexes are concrete structures that occupy memory and disk and must be updated on every mutation that touches indexed fields.

#### Checklist

- Design compound indexes with equality fields first, then range fields.
- Avoid indexing very large arrays. If arrays are needed, consider splitting high-cardinality members into separate documents and indexing the link key.
- Use `DISTINCT` in array indexes when duplicate elements are possible and you prefer deduplicated index entries.
- Favor partial indexes when queries target a narrow and stable subset of documents.
- Create covering indexes by adding projected fields to the index key list to avoid Fetches for frequent read paths, after evaluating the index size impact.
- Use `REQUEST_PLUS` with mutation tokens only where correctness demands immediate visibility of recent writes.
- Refresh statistics when data distribution changes, especially when you see poor plan choices.
- Monitor indexer memory, index build times, compaction load, and resident ratio continuously.
- Reproduce issues with controlled experiments that vary a single factor at a time.

Understanding Couchbase internals changes how you model data and design queries. Indexes are costly tools that, when used precisely, produce dramatic latency improvements. When misused, they are a primary source of unpredictable resource usage. Read plans, analyze `EXPLAIN` and `PROFILE`, design targeted indexes, and measure constantly.

I should admit that I've skipped over some details here and fast‑forwarded through others. Couchbase internals are too broad to cover in one sitting, and I wanted this essay to stay readable rather than drown in every operator and statistic.

In particular, I haven't gone step by step through query plans or shown how to interpret every line of EXPLAIN and PROFILE. That deserves its own dedicated post. My plan is to write another piece where I take one index and one query, dissect the query plan in detail, and show how to improve the index design based on what the planner is telling you.

So treat this essay as the map. The microscope will come later.

{{< highlight >}}
If you want a practical tool to help with this analysis, I built {{< link href="/couchlens-couchbase-query-analysis-tool" text="CouchLens" target="_blank" >}}, a browser-based query analysis tool for Couchbase. It parses `system:completed_requests` and `system:indexes`, detects inefficient index scans, primary index usage, and other performance issues, and generates insights to help you find which queries need attention. Everything runs locally in your browser, no data leaves your machine.
{{< /highlight >}}

Further readings:

- https://docs.couchbase.com/server/current/indexes/indexing-overview.html
- https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/vbuckets.html
- https://docs.couchbase.com/server/current/learn/data/durability.html
- https://support.couchbase.com/hc/en-us/articles/23629925229851-Completed-Requests-basics
- https://github.com/aakash-advait/couchbase-sync-tool - if you need to sync couchbase nodes. For local setup etc.


---

## Understanding Private-Public Key Encryption

**Date:** 2025-11-06
**URL:** http://localhost:1313/private-public-key-encryption/
**Description:** Understanding Private-Public Key Encryption - The Backbone of Modern Security.


In today's digital world, where secure communication, authentication, and data integrity are non-negotiable, **private-public key encryption** (also called *asymmetric cryptography*) plays a foundational role. From HTTPS in your browser to SSH logins, cryptocurrency wallets, and email encryption, this elegant cryptographic system enables trust without prior shared secrets.

In this blog, we'll dive into:
- The core concepts behind asymmetric encryption
- How it differs from symmetric encryption
- The mathematics (intuitively explained) behind popular algorithms like RSA and ECC
- Real-world examples and code snippets
- Common use cases and pitfalls

Whether you're a curious beginner or a seasoned developer brushing up on fundamentals, you'll find something valuable here.

## A Quick Comparison of Symmetric vs. Asymmetric Encryption

Before we dive into public-key crypto, let's contrast it with its older sibling: **symmetric encryption**.

| Feature                | Symmetric Encryption          | Asymmetric Encryption             |
|------------------------|-------------------------------|-----------------------------------|
| Keys                   | One shared secret key         | Two keys: public + private        |
| Speed                  | Fast                          | Slower (computationally heavy)    |
| Key Distribution       | Hard (needs secure channel)   | Easy (public key can be shared)   |
| Use Cases              | Bulk data encryption (e.g., AES) | Key exchange, digital signatures |

**Symmetric encryption** (like AES) is great for encrypting large amounts of data quickly, but you need a way to **securely share the secret key** beforehand. That's where **asymmetric encryption** shines: it solves the *key distribution problem*.

## Two Keys, One Mathematical Bond

In asymmetric cryptography:
- The **public key** is shared openly.
- The **private key** is kept secret.
- Data encrypted with one key can **only** be decrypted with the other.

This creates two powerful capabilities:
1. **Confidentiality**: Encrypt with someone's *public key* → only they can decrypt with their *private key*.
2. **Authentication & Integrity**: Sign data with your *private key* → anyone can verify it using your *public key*.

> **Crucial Insight**: The public key *cannot* be used to derive the private key, even though they're mathematically linked. This is the "hard problem" that makes the system secure.


## How It Works

### Example 1: Sending a Secure Message

Alice wants to send a confidential message to Bob.

1. Bob generates a key pair:  
   - Public key: `Bob_pub`  
   - Private key: `Bob_priv` (kept secret)
2. Bob shares `Bob_pub` with Alice (e.g., via email, website, key server).
3. Alice encrypts her message using `Bob_pub`.
4. Bob receives the ciphertext and decrypts it with `Bob_priv`.

Even if Eve intercepts the message and has `Bob_pub`, she **cannot** decrypt it, because the private key is required.

### Example 2: Digital Signatures

Now, Alice wants to prove a message truly came from her.

1. Alice hashes her message → `H = SHA256("Hello Bob!")`
2. She encrypts the hash with her *private key*: `Sig = Encrypt(H, Alice_priv)`
3. She sends both the message and `Sig` to Bob.
4. Bob:
   - Hashes the received message → `H'`
   - Decrypts `Sig` using `Alice_pub` → gets `H`
   - Checks if `H == H'`

If they match, the message is **authentic** and **unchanged**.

> Note: In practice, we don't encrypt the whole message, just its hash (for efficiency and security).

## Under the Hood: RSA and ECC

Two major algorithms power most public-key cryptography today:

### 1. RSA (Rivest–Shamir–Adleman)

**Based on**: The difficulty of factoring large prime numbers.

#### Key Generation (Simplified):
1. Choose two large primes: `p` and `q`
2. Compute `n = p * q`
3. Compute Euler's totient: `φ(n) = (p-1)(q-1)`
4. Choose public exponent `e` (commonly 65537) such that `1 < e < φ(n)` and `gcd(e, φ(n)) = 1`
5. Compute private exponent `d` such that `(d * e) mod φ(n) = 1`

- Public key: `(n, e)`
- Private key: `(n, d)`

**Encryption**:  
`ciphertext = plaintext^e mod n`

**Decryption**:  
`plaintext = ciphertext^d mod n`

> RSA requires large keys (2048+ bits) for security today.

#### Python Example (using `cryptography` library):

```python
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization


# Generate keys
private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
public_key = private_key.public_key()

# We are writing the keys to files for demonstration purposes
with open("public_key.pem", "wb") as f:
    f.write(public_key.public_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PublicFormat.SubjectPublicKeyInfo
    ))

with open("private_key.pem", "wb") as f:
    f.write(private_key.private_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PrivateFormat.PKCS8,
        encryption_algorithm=serialization.NoEncryption()
    ))

# Encrypt
message = b"Secret message"
ciphertext = public_key.encrypt(
    message,
    padding.OAEP(
        mgf=padding.MGF1(algorithm=hashes.SHA256()),
        algorithm=hashes.SHA256(),
        label=None
    )
)

# Decrypt
plaintext = private_key.decrypt(
    ciphertext,
    padding.OAEP(
        mgf=padding.MGF1(algorithm=hashes.SHA256()),
        algorithm=hashes.SHA256(),
        label=None
    )
)

print(plaintext)  # b"Secret message"
```

### 2. ECC (Elliptic Curve Cryptography)

**Based on**: The algebraic structure of elliptic curves over finite fields. A type of math involving curved lines and numbers that reset after a certain point, designed so that going forward is easy, but going backward without a secret key is practically impossible. 

- Offers **same security as RSA** with **much smaller keys** (e.g., 256-bit ECC ≈ 3072-bit RSA).
- Faster and more efficient, ideal for mobile and embedded systems.

Common curves: `secp256k1` (used in Bitcoin), `Curve25519` (used in Signal, SSH).

#### ECC Key Generation (Conceptual):
1. Choose a standard elliptic curve (e.g., `secp256r1`)
2. Pick a random private key `d` (a large integer)
3. Compute public key `Q = d * G`, where `G` is a known base point on the curve

The security relies on the **Elliptic Curve Discrete Logarithm Problem (ECDLP)**: given `Q` and `G`, it's computationally infeasible to find `d`.

> ECC is now the **preferred choice** for new systems (TLS 1.3, blockchain, etc.).


## Real-World Use Cases

### 1. **TLS/SSL (HTTPS)**
- Your browser uses the server's public key to establish a secure session.
- Asymmetric crypto exchanges a symmetric key (e.g., AES) for efficient data transfer.

### 2. **SSH Authentication**
- Your SSH client proves identity using a private key; the server verifies with your public key.

### 3. **PGP/GPG Email Encryption**
- Encrypt emails with recipient's public key; sign with your private key.

### 4. **Cryptocurrencies**
- Bitcoin wallet addresses are derived from public keys.
- Transactions are signed with private keys to prove ownership.


## Common Pitfalls & Best Practices

| Mistake | Why It's Bad | Best Practice |
|--------|-------------|---------------|
| Reusing RSA keys for encryption + signing | Increases attack surface | Use separate keys or follow PKCS standards |
| Using small key sizes (e.g., 1024-bit RSA) | Vulnerable to factorization | Use ≥2048-bit RSA or 256-bit ECC |
| Not using padding (e.g., textbook RSA) | Vulnerable to chosen-ciphertext attacks | Always use OAEP (encryption) or PSS (signing) |
| Hardcoding private keys in code | Massive security risk | Store in secure vaults (e.g., HashiCorp Vault, AWS KMS) |


---

Private-public key encryption is not just a cryptographic curiosity, it's the **trust engine** of the internet. By elegantly solving the key distribution problem, it enables secure communication between strangers across an insecure network.

While the math behind RSA and ECC can seem daunting, modern libraries abstract away the complexity. As a developer, your job isn't to implement these algorithms from scratch, but to **use them correctly**, respect key hygiene, and understand their limits.

As you explore steganography or other low-level projects (like embedding data in media), remember: **encryption protects content, steganography hides its existence**, and the two can even work together!


## Further Reading

- [PKCS Standards](https://en.wikipedia.org/wiki/PKCS?ref=lorbic.com)
- [NIST Guidelines on Key Management](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final?ref=lorbic.com)
- [Serious Cryptography by Jean-Philippe Aumasson](https://nostarch.com/serious-cryptography-2nd-edition?ref=lorbic.com)
- [Cloudflare's ECC explanation](https://blog.cloudflare.com/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/?ref=lorbic.com)

---

**Got questions or want a deep dive into a specific algorithm?** Leave a comment below!

Happy encrypting!


---

## The 5-Minute Refactor

**Date:** 2025-10-28
**URL:** http://localhost:1313/the-5-minute-refactor/
**Description:** Learn how to refactor your code in 5 minutes.


## The 5-Minute Refactoring Guide

As experienced software engineers, we often face a dilemma: our codebases, like all physical systems, trend toward entropy. The gap between "getting it done" and "getting it right" grows, leading to sluggish feature delivery and inevitable technical debt. The solution isn't a massive, heroic rewrite; it's the disciplined, humble practice of **Kaizen**, or continuous improvement.

For Golang engineers, this translates to the **5-Minute Refactor**: a daily commitment to making **one tiny, tangible, quality improvement** to any code you touch. This practice leverages Go's philosophy of simplicity to prevent decay and sharpen your engineering judgment, all in less time than it takes to make coffee.

## Why 5 Minutes Works (The Deep Engineering Principles)

This micro-habit is effective because it adheres to core principles that govern high-performing systems:

1.  **Combats Entropy:** The 5-Minute Refactor is your consistent, small-energy input to counteract the natural disorder that makes codebases harder to maintain over time.
2.  **Improves Flow (The First Way of DevOps):** Every small refactor removes a cognitive obstacle (like a confusing variable name or a nested block). Removing friction speeds up feature delivery.
3.  **Enhances Judgment:** By constantly pausing to ask, "How can I make this single line better?" you train your eye to spot code smells, which is the definition of true engineering expertise.
4.  **Fosters Humility:** It instills a sense of craftsmanship and shared responsibility for the codebase, moving the culture away from merely "shipping code" to "owning quality".

## The Practical "How-To" in Golang

The goal is to choose **ONE** of these improvements when you open a file, then commit the change. The focus is on embracing Go idioms for simplicity, readability, and explicit design.

### 1\. Simplify Error Handling: The Early Return

Go code famously suffers from deeply nested error checks. The Kaizen approach is to flatten the logic using the **Guard Clause** or **Early Return** pattern, ensuring the "happy path" (the successful outcome) is easy to follow.

##### Before (Deep Nesting)

```go
func loadUser(id int) (*User, error) {
	if id > 0 {
		user, err := db.fetch(id)
		if err == nil {
			if user.Active {
				return user, nil
			} else {
				return nil, errors.New("user is inactive")
			}
		} else {
			return nil, err
		}
	}
	return nil, errors.New("invalid ID provided")
}
```

##### After (Flat Flow)

```go
func loadUser(id int) (*User, error) {
	// Guard 1: Input validation returns immediately
	if id <= 0 {
		return nil, errors.New("invalid ID provided")
	}

	// Guard 2: I/O error check returns immediately
	user, err := db.fetch(id)
	if err != nil {
		return nil, fmt.Errorf("fetch failed: %w", err)
	}

	// Guard 3: Business logic check returns immediately
	if !user.Active {
		return nil, errors.New("user is inactive")
	}

	// The clear 'Happy Path' follows after all guards
	return user, nil
}
```

**Why it helps:** Reduces cognitive load by keeping the core logic clear of error branches, making it significantly easier to read and reason about.

### 2\. Extract Logic: The Pure Function Refactor

This improvement separates pure business logic (calculations) from side-effecting I/O logic (HTTP, DB calls), improving testability and clarity.

##### Before (Mixed Concerns in a Handler)

```go
func handler(w http.ResponseWriter, r *http.Request) {
	// ... reading request, validation ...

	total := 0
	for _, item := range items {
		// Business logic (calculation) mixed with I/O concerns
		total += item.Price + (item.Price * config.TaxRate) 
	}
	
	// ... writing total to response ...
}
```

##### After (Extracted Pure Logic)

```go
// Extracted Pure Function: easily unit testable, no side effects
func calculateTotal(items []Item, taxRate float64) int {
	total := 0
	for _, item := range items {
		total += item.Price + int(float64(item.Price) * taxRate)
	}
	return total
}

// The handler now only focuses on I/O and orchestration
func handler(w http.ResponseWriter, r *http.Request) {
	// ... reading request, validation ...

	total := calculateTotal(items, config.TaxRate) // Call the pure function
	
	// ... writing total to response ...
}
```

**Why it helps:** The core business logic is now isolated and **unit-testable**, improving quality and maintainability by adhering to the single responsibility principle.

### 3\. Improve Clarity by Using Clearer Names

This simple refactor replaces an overly abbreviated receiver name with one that clearly conveys the context within the function body, especially in more complex methods.

##### Before (Too Generic Receiver)

```go
type CacheService struct { 
    data map[string]string 
    metrics *MetricsCollector 
}

// What is 'c'? It forces the reader to pause and remember the type.
func (c *CacheService) Get(key string) (string, error) {
    // ... complex logic using c.data, c.metrics, logging ...
    val, ok := c.data[key]
    c.metrics.Increment("cache_hit")
    // ...
    return val, nil
}
```

##### After (Clearer Role)

```go
type CacheService struct { 
    data map[string]string 
    metrics *MetricsCollector 
}

// 'cache' clearly refers to the cache service instance, improving readability.
func (cache *CacheService) Get(key string) (string, error) {
    // The scope of 'cache' is immediately clear throughout the method body
    val, ok := cache.data[key]
    cache.metrics.Increment("cache_hit")
    // ...
    return val, nil
}
```

**Why it helps:** You communicate intent, reduce ambiguity, and make the method body easier to parse by clearly referencing the service instance.

### 4\. Enhance Type Safety: Custom Domain Types

This refactor uses Go's type system to embed domain meaning into primitive types (`int`, `string`), preventing accidental misassignment of IDs or values and leading to compile-time checks for logical errors.

##### Before (Ambiguous Primitives)

```go
// Both User IDs and Product IDs might be just 'int'
func deleteRecord(id int) error { 
	// ... logic to delete a record ...
}

// A call might accidentally pass the wrong type of ID, leading to a silent bug
userID := 1001
productID := 2005 

deleteRecord(userID)    // Correct intent
deleteRecord(productID) // Potential bug if deleteRecord expects a UserID!
```

##### After (Type Safety Kaizen)

```go
// Define custom types for domain clarity
type UserID int
type ProductID int

// The function signature now clearly dictates the required input type
func deleteUser(id UserID) error { 
	// ... logic to delete a user ...
	fmt.Printf("Deleting user with ID: %d\n", id)
	return nil
}

// The compiler now prevents mistakes: deleteUser(ProductID(2005)) would be a compile-time error!
userID := UserID(1001)
productID := ProductID(2005)

deleteUser(userID)
// deleteUser(productID) // This line would cause a compiler error, catching a bug early!
```

**Why it helps:** This is Kaizen for **type safety**. The compiler now enforces domain rules, preventing an entire class of runtime errors by catching them at compile-time.

### 5. Tidy Up: Remove Dead Code & Redundant Variables

Clutter, such as unused code, commented-out sections, or unnecessary intermediate variables, adds cognitive load. Removing it makes the actual working code stand out.


##### Before (Unnecessary Variable and Comment)

```go
func HashData(data string) string {
    // The data needs to be converted to a byte slice
    // This comment just restates what the code does, adding no value.
    dataBytes := []byte(data)
    
    // Return the hashed value
    return util.hash(dataBytes)
}
```

##### After (Concise and Direct)
The intermediate variable is inlined; the redundant comment is removed.
The code is now more direct and less noisy.
```go
func HashData(data string) string {
    return util.hash([]byte(data))
}
```

**Why it helps:** Go values conciseness. Eliminating code noise ensures engineers focus their attention only on lines that contain actual logic or important business context.



### The Kaizen Mindset

The 5-Minute Refactor is about **consistency, not intensity**. Your goal is to make a tiny deposit into the quality bank every day. Don't wait for permission or a dedicated task. When you open a file for any reason, ask yourself:

> **What is the smallest, safest, most impactful quality improvement I can make in this file in the next five minutes?**

By adopting this mindset, you turn every code session into a micro-learning experience, steadily transforming your codebase and your engineering capabilities. This humble, daily discipline is how you truly become a humble engineer. Start today. Your future self (and your teammates) will thank you.


---

## I Built My Own Google Drive

**Date:** 2025-10-01
**URL:** http://localhost:1313/i-built-my-own-google-drive/
**Description:** My journey to build my own cloud storage from scratch with nextcloudflare and the lessons I learned along the way


## Why Bother?

Whenever I want to download a folder from Google Drive, it starts zipping it for what feels like minutes, and then it downloads the whole zip. There's no incremental download, and I can't add it as a network drive for obvious reasons. These limitations are frustrating, but they also got me thinking: what if I could have a solution that's more flexible, more customizable, and truly mine?

The natural instinct for a systems engineer is: "Fine, I'll build my own NAS".

But then you remember that a NAS at home means a mini data center: constant uptime, power backup, static IPs, UPS, maybe even an inverter if you live in India. Suddenly, the "simple" idea of owning your storage becomes a logistics project.

I already had a cloud VM sitting idle. Something I'd spun up in 2022 for experiments. So I thought: what if I just turn that into my "own Google Drive"?

Attach a 100–200 GB volume, mount it, run Nextcloud, and I'd have a private, scalable "Google Drive" that exists entirely under my control.

The only catch: I didn't want to just make it work.

I wanted to learn something new in the process, specifically about Cloudflare's networking stack. So instead of exposing ports and managing SSL myself, I decided to route everything through Cloudflare Tunnel. That meant I could have a nice domain like `lorbic.com` sitting in front of my VM, with Cloudflare handling SSL, DDoS protection, and tunneling, no public IP exposure at all.

There are obviously easier paths. Hetzner's Storage Box. Dropbox. Even a hosted Nextcloud provider. But that's not ownership; that's delegation. I wanted to see my own storage service boot up from scratch, because there's a quiet kind of satisfaction in watching the pieces come alive and realizing, this is mine.

## Setup

Everything revolved around Docker.

I wrote a clean little docker-compose.yml with four containers: MariaDB, Redis, Nextcloud, and a Cron worker. No external dependencies, no magic scripts. Just containers and mounted volumes.

Here's the core of it:

```yaml
services:
  db:
    image: mariadb:12.0.2
    restart: always
    command: >
      --transaction-isolation=READ-COMMITTED
      --binlog-format=ROW
      --innodb-buffer-pool-size=1024M
    volumes:
      - /srv/nextcloud/db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=...
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=...
  redis:
    image: redis:7-alpine
    restart: always
    command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
    volumes:
      - /srv/nextcloud/redis:/data
  app:
    image: nextcloud:32.0.0-apache
    restart: always
    depends_on: [db, redis]
    ports:
      - 8080:80
    environment:
      - MYSQL_HOST=db
      - REDIS_HOST=redis
      - PHP_MEMORY_LIMIT=1024M
    volumes:
      - /srv/nextcloud/app:/var/www/html
      - /srv/nextcloud/data:/var/www/html/data
  cron:
    image: nextcloud:32.0.0-apache
    depends_on: [db, redis]
    entrypoint: /cron.sh
    volumes:
      - /srv/nextcloud/app:/var/www/html
      - /srv/nextcloud/data:/var/www/html/data
```

Simple, clean and repeatable.

Except it didn't work.

The Nextcloud container couldn't reach the internet. Not even a curl google.com from inside the container worked. I went down every rabbit hole: DNS settings, IPTables, VCN routes, even Cloudflare Tunnel's network binding. Nothing changed. Three hours gone.

Then I noticed something: Docker was installed via Snap.

This server was old, probably an artifact of some experiment from two years ago. Snap had quietly sandboxed Docker in a way that broke outbound connectivity for containers. Once I ripped out Snap and reinstalled Docker from apt. Everything started working like magic. One of those moments where you simultaneously feel relieved and slightly betrayed by your past self.

## The Cloudflare Tunnel

Next step: make it accessible to the outside world.

Instead of opening port 8080 (or reverse proxying it) and managing SSL certificates (although it can be done with certbot), I ran a Cloudflare Tunnel using cloudflared. My config.yml looked like this:

```yaml
tunnel: <tunnel-id>
ingress:
  - hostname: lorbic.com
    service: http://localhost:8080
  - service: http_status:404
```

Newer version of cloudflared does not rely on the `credential-file` based config. It needs you to directly use the run command. So, be mindful of the version you're using when setting up the tunnel.

That's it.

Cloudflare handles DNS, SSL, and traffic routing. My VM stays hidden behind the tunnel.

No open ports. No certbot. No firewall headache.

From my perspective, it's almost unfair how simple it is. The tunnel connected within seconds, and I could access https://lorbic.com from anywhere in the world.

The Nextcloud setup wizard appeared. I connected it to the MariaDB container, configured Redis, and within minutes had my own private cloud running securely behind Cloudflare.

### Some Issues
Everything looked good until I tried to log in from the Android and Mac Nextcloud apps.

It threw a cryptic error:

"The polling URL does not start with HTTPS despite the login URL starting with HTTPS".

Basically, the app was paranoid, and rightfully so. If the polling endpoint (used during login) isn't HTTPS, it refuses to proceed for security reasons.

After some investigation, I realized I had misconfigured the hostname in the Nextcloud setup. I was using the http version in my domain. So, while my login started with HTTPS (https://lorbic.com), the polling went to the tunnel URL, which was HTTP.

Fixing the Nextcloud config to use the HTTPS version of the domain and adding some header related config solved it.
Change it here: 
file: `nextcloud/app/config/config.php`
```php
'trusted_domains' =>
  array (
    0 => 'lorbic.com',
    1 => 'localhost',
    2 => '<tunnel-id>.cfargotunnel.com',
  ),
  'overwrite.cli.url' => 'https://lorbic.com',
  'overwriteprotocol' => 'https',
  'trusted_proxies' => ['127.0.0.1', '::1'],
  'forwarded_for_headers' => ['HTTP_X_FORWARDED_FOR'],
```

### The Warp Problem

Then came Cloudflare Warp, the VPN.

I like Warp because it's secure and integrates beautifully with the Cloudflare network. But here's the irony: when I connected Warp on my phone, https://lorbic.com stopped working.

Warp, being part of the same Cloudflare ecosystem, was actually routing my requests differently, through Cloudflare's internal network rather than over the open internet. And since I had configured the tunnel for external access only, Warp essentially short-circuited the route.

The workaround was simple: add a local override or exclude that domain from Warp's routing. That way, Warp would stop being too clever for its own good. _There are proper ways to setup Cloudflare Tunnel to work with Warp, but that's a topic for another post_.

## What I Learned

Beyond the technical triumph, this little project taught me how powerful modern abstractions have become.

Cloudflare Tunnel abstracts away networking in the same way Docker abstracts away system configuration. Together, they turn the once-daunting process of self-hosting into something playful, something you can do in an evening with coffee and a terminal.

I also ended up writing a simple cron job to sync some important directories from the cloud volume to an S3 bucket. It's not elegant, but it's effective, a redundancy layer for my "private Google Drive".

More importantly, I now have a personal cloud that I control.

No ads. No quota nags. No "Your storage is almost full". Just a small, efficient, self-contained service that does exactly what I want, and nothing I didn't agree to.

There's something oddly satisfying about this kind of engineering.

Not because it saves money, but because it gives back a sense of ownership, that feeling of knowing your system, line by line, container by container.

When I log into lorbic.com, it doesn't feel like another SaaS dashboard.

It feels like home.

**PS:** This post is a work in progress. I'll add more details as I go. And there are some simpler and much cheaper ways to get cloud storage than self-hosting Nextcloud. For example, you can use Hetzner Storage Box, which is a great option for a small, private cloud storage solution.  
Note: The domain `lorbic.com` is just a placeholder.

---


---

## Introducing Quark

**Date:** 2025-09-23
**URL:** http://localhost:1313/introducing-quark/
**Description:** A Minimal Note-Taking System for Thoughts, Ideas, Questions, and Actions


## Quark: A Minimal Note-Taking System

A few months ago, I was in a meeting where the conversation was moving at lightning speed. Updates, critical decisions, all flying around the room. I opened my laptop, ready to take notes in Notion, but within minutes I was lost. Too many clicks, too much formatting, and many annoying boxes. By the time I found the right template. I had already missed half of the discussion.

That was the moment I realized: most note-taking systems are built for after the fact for polishing, arranging, and linking. But when you're in the middle of the storm of thoughts. You don't need polish; you need to write.

## Why I Like Quark

I've tried the big tools like Notion, Obsidian, Roam. They're powerful, no doubt. For people who enjoy configuring, linking, and fiddling with templates, they can be great. But I don't like them that much. For me, they get in the way. Instead of writing, I spend a lot of time formatting, arranging, and configuring the tools. Instead of writing.

What I wanted was something simpler: a method that lets me write immediately. Whether I'm in a meeting, reading a book, or listening to a lecture, I need to capture what's happening **now**. Refinement can be done later.

With Quark, I don't lose important points. I don't pad my notes with fluff. I just write what matters, in plain text. That's the beauty: Quark works in a terminal, in Sublime, or on a piece of paper.

That's also how I write all my blogs and notes. With pen and paper first, then simple markdown files. No databases, no custom dashboards, no unnecessary friction. Just thinking, written down.

## What is Quark?

Quark is a minimal, universal note-taking system based on three atomic symbols:

- `-` → Idea / Statement
- `!` → Action / Task
- `?` → Question / Doubt

That's it. Three prefixes, plus a bit of indentation, and you have a system that works across meetings, research notes, books, and even journaling.

Why the name? In physics, _quarks_ are the fundamental particles that make up matter. Small, essential and universal. Quark notes are the same: tiny, fast to write, and the building blocks of larger understanding.

## The Core Syntax

Here's all you need to remember:

| Symbol | Meaning           | Example                           |
| ------ | ----------------- | --------------------------------- |
| -      | Statement / Idea  | `- Revenue grew 12% this quarter` |
| !      | Action / Decision | `! Send the draft by Friday`      |
| ?      | Question / Doubt  | `? Why did conversions drop?`     |

Indentation creates hierarchy. Nothing else required.

## Lists and Structure

Quark supports both unordered and ordered notes.

**Unordered (default):**

```
- Item one
- Item two
  - Sub-item A
  - Sub-item B
```

**Ordered (when sequence matters):**

```
1. Step one
2. Step two
   1. Sub-step
```

**Quick flow (sequence without numbering):**

```
> Start
> Middle
> End
```

## Modifiers (Optional Power-ups)

You don't need these to use Quark. But they add expressive punch without complexity.

| Symbol | Use Case             | Example                                 |
| ------ | -------------------- | --------------------------------------- |
| \*     | Highlight / Emphasis | `- * Critical concept: entropy`         |
| →      | Link / Cause-effect  | `- Pressure ↑ → Volume ↓ (Boyle's Law)` |
| >      | Quote                | `> "Knowledge is power." - Bacon`       |
| #tag   | Topic / Label        | `- Distributed systems #scalability`    |
| %      | Personal note        | `% This connects to my project`         |

## Real-World Examples

**Meeting Notes**

```
- Marketing update
  - Campaign CTR up 15%
  ? Why drop in conversions despite CTR?
  ! Check landing page load speed
```

**Reading Notes**

```
- Plato: Allegory of the Cave
  - Prisoners see shadows = illusion
  - Sun = truth
  * Core idea: reality ≠ appearances
  ? Compare with Advaita concept of maya
```

**Learning (Programming)**

```
- Goroutines are lightweight threads in Go
  - Managed by Go scheduler
  - Not 1:1 with OS threads
  ? What is max concurrency limit?
  ! Write demo with 1000 goroutines
```

## Why Quark Works

1. **Speed**
   You're never stuck choosing a format. Just `- ! ?`. Everything else is optional.

2. **Universality**
   Works on paper, in a terminal, or in your favorite editor. No special software needed.

3. **Scalability**
   Hierarchy comes from indentation. Tags and symbols keep meaning intact even across hundreds of notes.

4. **Searchability**
   In text editors, you can instantly jump to all actions (`/^!`) or all questions (`/^?`). In a notebook, the symbols stand out visually.

## The Philosophy of Quark

Quark is deliberately kept small. It resists the temptation of sprawling frameworks.

The Quark philosophy is simple:

- Notes are atoms of thought.
- Keep them fast to capture and easy to scan.
- Don't let the medium (software, notebook, template) become the bottleneck.

You can always refine Quark notes later into essays, reports, or flashcards. But the raw capture stays lightweight and universal.

### Quick Workflow Summary

1. **Capture fast**: `- ! ?` with optional `* → # % >`
2. **Stream by context**: Reading, meetings, learning
3. **Tag and link lightly**: Only when meaningful
4. **Review regularly**: Refine, consolidate, prioritize
5. **Act on `!` and investigate `?`**: Close the loop on actions and knowledge

## How Quark Compares to Other Note-Taking Systems

You might wonder: why not just use an existing method like Cornell, Zettelkasten, or Bullet Journal? Each has its strengths, but here's why Quark stays leaner:

- **Cornell Notes**
  Great for structured lecture review, but the rigid split (cues, notes, summary) slows you down in real time. Quark keeps capture fast and lets you organize later.

- **Zettelkasten**
  Powerful for long-term knowledge building, but it requires discipline to use effectively. IDs, cross-links, atomic notes. Quark skips the overhead. You can always turn Quark notes into Zettels later if you want.

- **Bullet Journal**
  A creative, flexible paper system, but often drifts into over-customization and decoration. Quark removes the artistry by using just symbols and indentation.

In short: those systems are great for refinement. Quark is built for capture. The fastest way to not lose ideas in the moment.

## Conclusion

Note-taking doesn't need to be complicated. With Quark, you only need three prefixes and a bit of indentation. The rest is mental freedom.

**Quark Rule:** If you can type `- ! ?`, you can capture the world.

**Download:** {{< link href="/media/uploads/quark_cheatcheet_by_lorbic.com.pdf" text="The Quark Cheat Sheet" target="_blank" >}}

---


---

## Why Your Password Habits Will Get You Hacked

**Date:** 2025-08-13
**URL:** http://localhost:1313/password-security-habits/
**Description:** Practical, unsentimental steps to lock down your passwords, devices, and online identity before someone else owns them.


# The Real Cost of Digital Laziness

We've built our entire lives on top of fragile passwords.  
Most people treat them like they treat dental checkups, ignore it until something hurts. By then, it's too late.

The numbers are not on your side. Every password you've ever created has probably been leaked in some breach you've never heard about. It doesn't matter that the breach was from a forgotten forum you joined in 2012; attackers don't forget. They run automated "credential stuffing" attacks across every major platform, testing your old passwords against your bank, your email, your cloud storage. One lazy reuse, and they're in.

The weak link is always human behavior.  
We think we're clever with "Password123!" or by swapping 'E' for '3', patterns a brute-force script can guess in milliseconds. We reuse passwords because we tell ourselves, "Nobody would bother hacking me". That's wrong. Nobody is hacking *you*. They're hacking *everybody*, at scale, without even knowing who you are.

Security is not paranoia; it's hygiene.  
You wouldn't leave your apartment door wide open just because you've never been robbed. So why leave your digital life exposed? A compromised email account isn't just a privacy leak, it's a skeleton key to reset every other account you own.

**What works?**
- Unique passwords for every account. Non-negotiable.
- Password managers to store them. The "I don't trust password managers" crowd is already trusting their brain, which is worse.
- Two-factor authentication on critical accounts (email, bank, primary cloud). And no, SMS OTP is not enough.
- Regular breach checks using services like [HaveIBeenPwned](https://haveibeenpwned.com) to see if your credentials have been exposed.

Digital security is boring, until it becomes catastrophic.  
The same way seatbelts don't make you a better driver, strong passwords won't make you invincible. But they drastically reduce the damage when the inevitable collision happens.

Most hacks aren't cinematic scenes of hoodie-wearing geniuses bypassing firewalls. They're just someone logging in with your password, because you gave it to them years ago and never thought twice.

You are either disciplined about security, or you're gambling with your entire online identity. There's no middle ground.


## A Checklist to Lock Down Your Digital Life
*(No motivation speeches. Just do it.)*

**1. Secure Your Email.**  
- Your email is the control tower of your digital identity.  
- If it's weak, every "secure" account you own is just one password reset away from theft.
- Set a strong password, enable 2FA, and set recovery settings. 

**2. Get a Good Password Manager.**  
- Bitwarden, 1Password, or KeePass.  
- Stop using your brain as storage; it's a terrible database.
- Never write down your passwords in your diary.

**3. Enable 2FA.**  
- Authenticator apps (Aegis, Authy, Google Authenticator) or hardware keys (YubiKey).  
- SMS OTP should be the last resort.
- Enable passkeys if the website supports it. Passkeys replace passwords with cryptographic keys tied to your device, making phishing almost impossible.

**4. Use Unique Passwords.**  
- If one account is compromised, it shouldn't unlock the rest of your life.
- Use password generators like Bitwarden's password generator.
- Prefer passphrases over passwords.

**5. Check for Breaches.**  
- Use [haveibeenpwned.com](https://haveibeenpwned.com)  
- If your password is in a breach, assume it's public and change it now.

**6. Update Your Devices.**  
- Phone, laptop, browser extensions, outdated software is an open window.

**7. Remove Accounts You Don't Use.**  
- Old, abandoned accounts are breach magnets. Shut them down.

**8. Prevention Best Practices.**
- Change passwords when account breached or suspected compromised.
- For high-value accounts like bank, cloud service etc; rotate passwords regularly.
- For banking and other sensitive logins, use a clean browser profile with no extensions, or incognito mode.
- Avoid logging into critical accounts on public or shared computers, they may have keyloggers or malware.

## Social Engineering & Phishing Vectors

Even the most secure device can be compromised if the attacker convinces you to willingly install or permit malicious code. Common tactics include:
1. Fake App Updates: Pop-ups claiming "Your WhatsApp is out of date, download now" that lead to malicious APKs.
2. Lookalike Apps: Apps in third-party stores with names and icons mimicking legitimate software.
3. Malicious Links: SMS, WhatsApp, or email links that lead to credential phishing sites or drive-by downloads.
4. Fake System Alerts: In-browser alerts saying "Your phone is infected" to prompt a bogus cleaner app download.
5. Compromised QR Codes: Publicly posted QR codes (cafés, events) that link to APKs or phishing pages instead of expected URLs.
6. Public Wi-Fi: The public wi-fi could be setup to decrypt your traffic. It could be an evil-twin access point.

Mitigation:
- Install only from Google Play or trusted app stores.
- Avoid clicking links from unsolicited messages.
- Verify app publisher names and permissions before installing.
- Use a scanner like VirusTotal Mobile to check suspicious APKs before opening.
- Use a trusted VPN (ProtonVPN, MullvadVPN, IVPN). Avoid the VPNs advertised by YouTubers. 
- Know that VPNs don't make you anonymous, they only encrypt your traffic.
- You can host your own VPN with Wireguard, OpenVPN or PI-VPN.
- Avoid all free VPNs at all times. If you're not paying for the service, your data is the product. The only widely recommended exception is ProtonVPN's free tier, which is subsidized by paid users and has a transparent privacy policy.

A flowchart of a social engineering attack:
![Flowchart of a social engineering attack. Picture by Vikash from Lorbic.com](/images/uploads/android_malware_kill_chain.png)

## Trusted Tools for Digital Security
*(No sponsorships. Just what works.)*

**Password Managers**  
- [Bitwarden](https://bitwarden.com) (Free + open-source, cloud sync, self-hostable)  
- [1Password](https://1password.com) (Paid, polished UI, family plans)  
- [KeePassXC](https://keepassxc.org) (Free, open-source, offline storage)

**2FA / MFA Apps**  
- [Aegis Authenticator](https://getaegis.app) (Free, encrypted backups, Android)  
- [Authy](https://authy.com) (Cross-device sync, encrypted, consider disabling cloud-sync if you want maximum control.)  
- [YubiKey](https://www.yubico.com) (Hardware key, phishing-resistant)

**Breach Checkers**  
- [Have I Been Pwned](https://haveibeenpwned.com) (Email & password leak lookup)  
- [Firefox Monitor](https://monitor.firefox.com) (Automated breach alerts)

**Teach yourself**
- [Techlore's resources page](https://techlore.tech/resources/?ref=lorbic.com) (Learning Resource)

---

If you skip these steps, you're not being "laid-back about security". You're just volunteering to be the easy target in a very crowded shooting gallery.


---

## Stealth VPN with Outline over HTTPS

**Date:** 2025-08-12
**URL:** http://localhost:1313/outline-vpn-over-https/
**Description:** Running Outline VPN over HTTPS (port 443) to slip past censorship. Learn how to blend in with HTTPS traffic and what it actually takes to stay invisible.


## What We Are Doing

If you are somewhere that blocks VPNs through deep packet inspection (DPI), SNI filtering, or crude port blocking, the goal is to make your VPN traffic look as boring as possible.  
The quick win is to put it on port 443, the same port used by HTTPS. That is the lifeline of the modern web, so most firewalls are reluctant to block it.

But think of it as changing your outfit but not your walk: just putting Shadowsocks (which Outline runs under the hood) on port 443 does not make it indistinguishable from real HTTPS. To a modern DPI system, it still has a unique fingerprint. If you want to disappear in plain sight, you need to wrap it in actual TLS or another disguise layer.

## Why Port 443 Matters and Its Limits

Port 443 is the default for encrypted web traffic. Blocking it would break Gmail, YouTube, Facebook, banking apps, and much more.  
Using 443 for your VPN helps against simple port-based blocks. It does not stop advanced DPI, which can still detect Shadowsocks' handshake patterns even inside encryption.

Think of it as changing your outfit but not your walk. Observers may not see your face, but they still recognize the way you move.

## How Outline Works

Outline is a friendly manager and server wrapper for Shadowsocks. It runs a SOCKS5 proxy with strong AEAD encryption and offers an API for generating and revoking access keys.  
It does **not** speak TLS natively. To mimic HTTPS convincingly, you must add a plugin or a front-end reverse proxy that handles TLS and passes traffic inside a valid protocol like WebSocket.

## 1. Provision Your Server

Choose a VPS provider that is not hostile to privacy. Good options include Hetzner, DigitalOcean, and Vultr. The Oracle free tier works if you enjoy tinkering, but it can be unreliable.  
Stick to Ubuntu 20.04 or 22.04 for compatibility.

Open only these inbound ports for now:

```
443/tcp   # VPN traffic disguised as HTTPS
53/udp    # DNS (for fallback only, see DNS notes below)
```

## 2. Install Outline Server

Run the official installer:

```bash
curl -sS https://getoutline.org/install.sh | sudo bash
```

This will:

- Install Docker if needed
- Set up the Outline Manager service
- Give you an API URL and fingerprint for connecting

Manage the server using the [Outline Manager](https://getoutline.org) desktop app.

## 3. Put It on Port 443

By default, Outline listens on a high random port. To change it to 443, you need to modify the Docker Compose configuration directly, as editing `access.txt` won't change the listening port of the Shadowsocks server managed by Outline. The `access.txt` file is for access keys, not server configuration.

Edit the Docker Compose mapping to expose port 443 and ensure the Outline service inside the container listens on port 443.

First, access your server via SSH and open the `docker-compose.yml` file:

```bash
sudo nano /opt/outline/docker-compose.yml
```

Locate the `ports` section for the `shadowsocks` service and ensure it maps the external port 443 to the internal container port 443. It might look something like this initially, with a high random port:

```yaml
ports:
  - "8080:8080" # Example, your random port will be different
```

Change it to:

```yaml
ports:
  - "443:443"
```

Additionally, you might need to ensure the Shadowsocks configuration within the Docker container is also set to listen on port 443. While Outline typically handles this when it sets up the service, if you encounter issues, you may need to check Outline's internal configuration files (though this is less common for simple port changes). For most setups, just changing the `docker-compose.yml` ports mapping is sufficient.

Restart:

```bash
docker-compose -f /opt/outline/docker-compose.yml down
docker-compose -f /opt/outline/docker-compose.yml up -d
```

This gets you the "minimal camouflage" setup. It helps against basic censorship but still looks like Shadowsocks to advanced DPI.

## 4. HTTPS (Recommended)

Running Outline over HTTPS significantly enhances security and can help bypass certain network restrictions. This involves setting up a reverse proxy (like Nginx) to handle SSL/TLS encryption and forward traffic to your Outline server.

First, ensure you have a domain name (e.g., `vpn.example.com`) pointed to your server's IP address.

### Install Nginx and Certbot

```bash
sudo apt update
sudo apt install nginx certbot python3-certbot-nginx -y
```

### Obtain an SSL/TLS Certificate

Use Certbot to get a free SSL/TLS certificate from Let's Encrypt for your domain. Replace `vpn.example.com` with your actual domain.

```bash
sudo certbot --nginx -d vpn.example.com
```

Follow the prompts. Certbot will automatically configure Nginx to use the certificate and set up automatic renewals.

### Configure Nginx as a Reverse Proxy

Create a new Nginx configuration file for your domain:

```bash
sudo nano /etc/nginx/sites-available/vpn.example.com
```

Add the following configuration. This assumes your Outline server is listening on `localhost:443` (as configured in the previous step). If you changed Outline's port, adjust `proxy_pass` accordingly.

```nginx
server {
    listen 80;
    server_name vpn.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name vpn.example.com;

    ssl_certificate /etc/letsencrypt/live/vpn.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/vpn.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers off;

    location / {
        proxy_pass http://127.0.0.1:443; # Or your Outline's listening address and port
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        proxy_max_temp_file_size 0;
    }
}
```

### Enable the Nginx Configuration

Create a symbolic link to enable the new configuration and then test Nginx syntax and restart.

```bash
sudo ln -s /etc/nginx/sites-available/vpn.example.com /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx
```

With this setup, Nginx handles the HTTPS connection, decrypts the traffic, and forwards it to your Outline server, which can now listen on its internal port (e.g., 443) without directly managing SSL/TLS certificates.

## 5. Cloudflare Routing

Cloudflare does not proxy raw TCP over 443 unless you pay for [Spectrum](https://www.cloudflare.com/products/cloudflare-spectrum/). The free plan works for HTTP, HTTPS, and WebSocket. Therefore, to route your Outline VPN traffic through Cloudflare's free tier, your Outline client must use a WebSocket plugin.

Configure your Nginx (as described in the HTTPS section) to proxy WebSocket traffic to your Outline Shadowsocks server. This involves adding a `location` block in your Nginx configuration to handle a specific WebSocket path (e.g., `/your_websocket_path`) and proxy it to your Outline server's internal address and port.

On the client side, ensure your Shadowsocks client is configured to use a WebSocket plugin (like [v2ray-plugin](https://github.com/shadowsocks/v2ray-plugin)) and the specified WebSocket path. For example, with Shadowsocks-Rust, your client configuration might look like this:

```json
{
  "server": "your_domain.com",
  "server_port": 443,
  "password": "your_password",
  "method": "chacha20-ietf-poly1305",
  "plugin": "v2ray-plugin",
  "plugin_opts": "tls;host=your_domain.com;path=/your_websocket_path"
}
```

This setup makes your VPN traffic appear as regular web traffic to Cloudflare, further enhancing obfuscation and bypassing advanced deep packet inspection (DPI).

## 6. Connect Your Client

Download the Outline client for your platform:

- [Android](https://play.google.com/store/apps/details?id=org.outline.android.client)
- [iOS](https://apps.apple.com/us/app/outline-app/id1356177741)
- [Windows, Mac, Linux](https://getoutline.org)

Paste in your access key from the Outline Manager.  
If you switch from raw IP to a domain for TLS, you must regenerate or edit the key to match the new hostname.

## 7. Test Your Stealth

Testing is more than "it connects". Use these:

- [gfw.report](https://gfw.report) to check if the handshake is blocked in censorship regions
- Wireshark or [mitmproxy](https://mitmproxy.org) to inspect packet patterns

Minimal camouflage will still show a Shadowsocks signature. Strong camouflage should be indistinguishable from normal HTTPS.

> **DNS warning:** Many censorship regimes hijack or block UDP 53. For better stealth, configure DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT) on your device or in the Outline client.

## Active Probing Defense

In some places, censors will connect to your IP on 443 and try to speak Shadowsocks. If your server replies, they blacklist you.  
Plugins like [Cloak](https://github.com/cbeuw/Cloak) or nginx fronting can drop these connections unless they match a known pattern.

## Summary

- Port 443 is a good first step, but on its own it only beats basic port blocking.
- For serious stealth, add a TLS or WebSocket transport with a valid certificate.
- Cloudflare routing only works if traffic is inside HTTP or WebSocket.
- Protect against active probing in high-risk environments.

## References and Further Reading

- [Outline Server GitHub](https://github.com/Jigsaw-Code/outline-server)
- [Shadowsocks Plugins](https://shadowsocks.org/doc/plugins.html)
- [v2ray-plugin](https://github.com/shadowsocks/v2ray-plugin)
- [Cloak](https://github.com/cbeuw/Cloak)
- [Let's Encrypt](https://letsencrypt.org/)
- [gfw.report](https://gfw.report)

If you only need to get around crude blocks, changing Outline to port 443 might be enough. If you are under active, sophisticated censorship, you will need to dress it up in TLS and maybe hide it behind something bigger like Cloudflare. I will try this idea and write another blog detailing the journey.

The more it looks and acts like normal HTTPS, the longer it will survive.


---

## How to Resolve Huge Git Merge Conflicts Without Losing Your Mind

**Date:** 2025-07-17
**URL:** http://localhost:1313/resolving-huge-merge-conflicts/
**Description:** A no-fluff, step-by-step guide for software engineers dealing with massive Git merge conflicts during migrations or long-lived branches. Includes real anecdotes and ASCII diagrams.


**How to Resolve Huge Git Merge Conflicts Without Losing Your Mind**

If you've ever been knee-deep in a long-lived branch merge and thought, _"this can't be what version control was meant for"_, you're not alone. Git gives us powerful tools, but resolving large merge conflicts, especially during major migrations or rewrites can feel like surgery without anesthesia.

In this post, we'll walk through how to approach and resolve massive merge conflicts systematically. Not with hand-wavy advice, but with actual steps, real mental models, and clear visual understanding.

Let's say you're working on a legacy codebase, maybe something ancient like Django 1.10, and you're helping move it to Django 4.2. That's what I had to do once. The problem? Two long-running branches with divergent histories. One branch added features while the other refactored the core framework. Two months of drift meant the merge base was ancient history. When we finally attempted the merge, Git didn't just report conflicts; it presented a battlefield. Time to toss a coin to your engineer.

Here's what we saw:

<pre><code>
*--*--*--*--*--*-- Feature Branch
 \
  *--*--*--*--*--*--*--*-- Main (Updated Django)
</code></pre>

Trying to merge led to a wall of conflicts across dozens of files. Looks like rain. Here's how to approach this situation methodically.

### 1. Stop. Don't Start Fixing Right Away.

Before typing `git merge`, take a moment. Understand the nature and scope of the divergence.

- **See commits exclusive to each branch:**
  - `git log main..feature-branch --oneline` (commits in `feature-branch` but not `main`)
  - `git log feature-branch..main --oneline` (commits in `main` but not `feature-branch`)
- **See the total diff:**
  - `git diff main...feature-branch` (three dots) shows all changes on `feature-branch` since the common ancestor. This is the work you are trying to merge in.

This preliminary investigation tells you whether you're dealing with a skirmish or a war. Hmm.

### 2. Visualize the Conflict

Use:

```bash
git log --graph --oneline --all --decorate
```

Or:

```bash
gitk --all
```

Or use a GUI tool like **gitk**, **Sublime Merge**, or your IDE's Git history viewer. These help you find the _merge base_: the common ancestor commit from which the two branches diverged.

You can find it on the command line too:

```bash
git merge-base main feature-branch
```

Every conflict is a story about how two different developers modified the same starting point (`BASE`). Understanding that `BASE` commit is key.

### 3. Create a Temporary Merge Playground

Never fight the monster directly on the main branch.

```bash
git checkout -b conflict-playground feature-branch
git merge main
```

Now you're in a temporary space to resolve without pressure.

### 4. Know What You're Resolving

Use a 3-way diff tool. For example, `meld`, `vscode`, or `sublime-merge` diff view.

A 3-way diff tool is non-negotiable. Configure it first:

```bash
git config --global merge.tool meld # Or vscode, p4merge, etc.
```

Then run `git mergetool` to open the first conflicted file. It will show you three versions:

- **`LOCAL`**: Your version (`HEAD`, i.e., `main`).
- **`REMOTE`**: Their version (the one you're merging in, i.e., `feature-branch`).
- **`BASE`**: The common ancestor. The state of the file before either branch made changes.

Your job is not to choose between `LOCAL` and `REMOTE`. It's to integrate the _intent_ of both changes into the `BASE` to create the final, `MERGED` result.

### 5. Triage: Divide and Conquer

List all conflicts to see what you're up against: `git status --short | grep "^U"`
Then, group them into categories:

1. **File-level conflicts**: One branch deleted a file that the other modified (`DU` or `UD`), or both added the same file (`AA`). Resolve these first with `git rm` or `git add`.
2. **Trivial content conflicts** (`UU`): Whitespace, comments. Easy wins.
3. **Structural conflicts** (`UU`): A class was renamed in `main` while a method was added to it in the feature. The tool will show a huge conflict, but the resolution is often straightforward: apply the feature's logic to the newly renamed code.
4. **Logical conflicts** (`UU`): Both branches modified the same piece of logic in incompatible ways. These require the most thought and careful testing.

Tackle them in that order. Steel for humans, silver for monsters and planning for complex merge-conflict. Gaining momentum is half the battle.

### 6. Use `git merge --abort` When Overwhelmed

Made a mess? Damn it. No problem.

```bash
git merge --abort
```

Back to a clean slate.

You can retry as many times as needed.

### 7. Use Commands to Resolve Files Quickly

For some files, you know one side is definitively correct. Don't open the mergetool; use a command.

- To accept your version (from `main`): `git checkout --ours path/to/file.js`
- To accept their version (from `feature-branch`): `git checkout --theirs path/to/file.js`

This is extremely efficient for structural refactors where you know the `main` branch's changes (e.g., a file move or rename) are the correct "canvas" to paint the feature changes onto. After checking out the correct base version, you can manually re-apply the other side's changes.

For manual resolution, open the file and look for the conflict markers. Edit the code to combine the sections, remove the `<<<<<<<`, `=======`, and `>>>>>>>` markers, and then `git add path/to/file.js`.

### 8. Run Tests Aggressively After Each Batch

After every 5–10 files, run your tests. Don't wait till you've resolved 100 conflicts to see what broke. Think of it as taking a potion before the fight, not after you're already bleeding out.

### 9. Use `git rerere` to Avoid Redoing Work

If you abort and retry a merge, `rerere` (Reuse Recorded Resolution) is your best friend.

```bash
git config --global rerere.enabled true
```

Once enabled, the first time you resolve a hunk of a conflict and `git add` the file, Git records the "before" (the conflict state) and "after" (your resolution) in the `.git/rr-cache` directory. If you abort and later encounter the exact same conflict, Git will automatically apply your recorded resolution. This is a massive time-saver. You can even commit the `rr-cache` directory to share conflict resolutions with your team.

*⚠️ Note: `rerere` works best when the conflicting hunks are *identical* to the previous conflict. If the context or line numbers shift slightly, Git might not auto-resolve them. It's useful, but not magic.*

### 10. Distinguish a Strategy from an Option

Sometimes you want to favor one branch's changes during a merge. There's a sharp tool for this, but you must know the difference between a strategy and an option.

- **The `ours` strategy**: `git merge -s ours feature-branch`
  This is a blunt instrument. It creates a merge commit that includes `feature-branch` as a parent, but **discards all of its changes entirely**. The resulting code is identical to `main`. This is useful for marking a dead-end feature as "merged" to clean up branch history, but it doesn't integrate any code.

- **The `-X ours` or `-X theirs` option**: `git merge -X ours feature-branch`
  This is far more useful for complex merges. It tells the standard `recursive` merge strategy to attempt a normal merge, but for any hunk that has a conflict, it automatically resolves it by choosing _your_ side (`-X ours`) or _their_ side (`-X theirs`). This is powerful for refactoring merges where you know one side's changes are globally more important.

### 11. ASCII Summary: Merging Chaos into Order

<pre><code>
Before:
Feature:   A -- B -- C -- D
                  \
Main:              X -- Y -- Z
</code></pre>

Conflict explosion: merging D and Z

Resolution Strategy:

1. Visualize
2. Playground Branch
3. Triage by File Type
4. Resolve in Order (Trivial → Structural → Logical)
5. Test Often

---

That old Django migration? It took me three solid days to resolve the merge. I didn't use any fancy tools, just `git mergetool`, `sublime-merge` diff viewers, and manual review. What saved me wasn't skill; it was method.

Large merge conflicts are not bugs. They're symptoms of drift. And you don't fight drift by panic-merging. You fight it with discipline, snapshots, and composure.

Git won't make it easy. But it gives you enough rope to build a bridge or hang yourself. Your call.

### Resources Worth Reading

1. [Git Branching - Basic Branching and Merging](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging)
2. [Advanced Merging](https://git-scm.com/book/en/v2/Git-Tools-Advanced-Merging)
3. [git-mergetool Documentation](https://git-scm.com/docs/git-mergetool)

### Git Conflict Glossary (Quick Reference)

- **BASE**: The common ancestor of the two branches being merged.
- **LOCAL**: Your current branch (the one you're merging _into_).
- **REMOTE**: The branch you're merging _from_.
- **ours**: Refers to LOCAL in conflict resolution.
- **theirs**: Refers to REMOTE in conflict resolution.
- **Merge strategy -s recursive**: Git's default merge strategy that tries to automatically resolve conflicts using a three-way merge.

**If you've got a merge war story, I'd like to hear it. Or better yet, avoid one, by rebasing early and often. Don't let your codebase become a cautionary tale.**

---


---

## What Facebook's Memcache Taught Me About Systems Thinking

**Date:** 2025-07-07
**URL:** http://localhost:1313/scaling-memcache-facebook/
**Description:** A deep dive into Facebook's memcache architecture and the hard lessons it teaches about real-world system design, performance, and failure.


What Facebook's Memcache Taught Me About Systems Thinking

> "The probability of reading transient stale data is a tunable parameter". - Scaling Memcache at Facebook (NSDI, 2013)

There's a moment in every engineer's life when a seemingly simple component like a cache, suddenly becomes the most complex piece in the stack. For me, that moment arrived reading Facebook's paper on scaling Memcache.
I didn't expect a key-value store to challenge my understanding of systems architecture. But this paper wasn't about cache keys or TTLs. It was about what happens when infrastructure hits the limits of scale, physics, and human reliability.  
What follows isn't a summary. It's a set of systems insights that stayed with me, principles that go deeper than code and that I now see everywhere.



#### 1. Keep the Core Dumb. Push Complexity Outward.

Facebook's memcached servers are intentionally brain-dead. They don't coordinate. They don't know each other exists. There's no replication, gossip protocol, or consensus layer.  
Instead, all intelligence like request routing, error handling, retries, batching is pushed into the client. The client might be an embedded library, or a proxy called mcrouter, which looks like a memcached server but simply forwards requests based on consistent hashing and configuration state.

At first, this seems backwards. Wouldn't smart servers reduce coordination overhead? Wouldn't a central cache service simplify consistency?  

But at their scale, the opposite holds true.  
By keeping memcached dumb and stateless:
- They can deploy and upgrade it independently.
- Failure recovery becomes local: just restart or replace the box.
- There's no server-side coordination failure to debug at 2 a.m.

The lesson here is architectural, not technical:
> Complexity doesn't vanish. But if you push it to the edge into clients it becomes easier to version, test, and evolve.

Dumb, fast, and replaceable beats smart and brittle.

#### 2. Systems Thinking Begins Where Feature Thinking Ends

Most of us reach for a cache to "improve performance". That's a feature mindset. But Facebook engineers were solving something else: survival at scale.

Example: they used UDP for GET requests. Not TCP. UDP doesn't guarantee delivery or order. Why use it?  
Because:
- It has less overhead per connection.
- It avoids memory-hungry TCP state on the server.
- It allows each thread to talk directly to memcached with minimal coordination.

Yes, they dropped 0.25% of packets. They didn't retry. They just treated those as cache misses and moved on. At a billion QPS, a few dropped packets don't matter. A congested server does.
Another example: leases. A lease is a token that a client gets on a cache miss. Only that client is allowed to re-populate the value. Everyone else waits briefly.

Why? Because without this, when a hot key is evicted, thousands of clients might simultaneously hit the database to fetch and repopulate it. That's called a thundering herd. You don't want that during peak load.  
Leases rate-limit writes back into cache. They also prevent stale sets, cases where two clients race to set different values for the same key, and the loser overwrites the winner.

These are not elegant solutions. They are pragmatic design decisions. They're what you build when you're no longer solving "how do I cache this" but "how do I keep the backend alive".

> Systems thinking asks not "what's ideal", but "what fails gracefully, what scales, and what I can fix under pressure".


#### 3. Consistency Is a Dial

We're taught to think of consistency as sacred: the database is right, and the cache must reflect it. But Facebook intentionally treats consistency as a performance lever.

Their memcache infrastructure accepts that cached data can be stale, duplicated, or missing, as long as the user experience doesn't suffer.  

How do they do it?

They treat the cache as discardable. The database is the source of truth. Cache can be evicted, expired, or wrong, as long as the database eventually corrects it.

They even allow slightly stale values to serve requests after deletes. Instead of immediately purging deleted keys, they move them to a short-lived "recently deleted" buffer. If a client requests that key again soon, it might get a _marked-as-stale_ value and proceed anyway, especially if the data is not user-critical.

In globally distributed systems, they use remote markers. If a region is lagging in replication, a client can mark a key as _recently deleted_ in a remote region. Other clients, on seeing the marker, are redirected to the master region for fresh data.

This isn't strong consistency. It's controlled inconsistency. And it works.  
Once you realize that consistency can be tuned, like latency or throughput, you start seeing your cache as a system of probabilities, not guarantees.

#### 4. Don't Just Cache for Speed, Cache for Economics

Facebook's memcache isn't a single cache. It's a collection of memory pools, each tuned to a different access pattern.

Why?  
Because not all keys are equal. Some are:
- Accessed frequently but cheap to miss (e.g., user online status).
- Rarely accessed but expensive to compute (e.g., machine learning model outputs).
- Frequently updated and high-churn (e.g., news feed orderings).

If you put these into one common pool, the result is negative interference:
- Hot, volatile keys evict long-lived but valuable data.
- Rare items get flushed before they're reused.
- Hit rates drop, and backend load increases.

So they created segregated pools:
- A wildcard pool for general use.
- An app-specific pool for temporary data.
- A replicated pool for hot data served from multiple servers.
- A regional pool for large, rarely accessed items shared across clusters.

Each pool can be tuned, size, eviction policy, replication strategy, without hurting the others.  
In other words, they cache not just for speed, but for cost-efficiency.  
The question isn't "how do I cache this?" It's "what is the cost of a miss, and is that cost worth the memory?"

#### 5. Failure Is Not the Exception

At Facebook's scale, servers fail constantly. Network partitions happen. Machines go dark. It's not an incident, it's Tuesday.

The naive approach on cache failure is to rehash keys onto the remaining servers. But in their system, key access is skewed. A single key might account for 20% of a server's load. Rehashing it elsewhere could overload the new server and create a cascade failure.

Their solution? *Gutter pools*.  
Gutter servers are a small fraction (~1%) of memcached boxes. When a regular server fails, clients don't retry the same key elsewhere. They redirect the request to the Gutter pool.

If the Gutter doesn't have the value, it fetches from the DB and inserts it, temporarily. These entries expire quickly.  
The point isn't to serve 100% correct data. The point is to absorb the shock. To give the system a soft place to land while the real servers recover.  
This is a lesson in system realism. Don't try to prevent all failure. Design for graceful degradation.

#### 6. Performance Lives

Most performance advice stops at averages: reduce latency, increase QPS. But real systems hurt in the tail latencies, the 95th, 99th, 99.9th percentiles.

Facebook found:
- A popular page triggers hundreds of memcache GETs.
- A slow server or delayed response creates fan-out latency.
- Incast congestion can flood switches when too many responses arrive simultaneously.

To fix this, they:
- Used batching to reduce round-trips.
- Employed sliding windows to limit outstanding requests.
- Tuned window size to balance between underutilization and congestion.

They also studied memory fragmentation and introduced:
- Adaptive slab rebalancing: reclaim slabs from underused classes and give them to those under memory pressure.
- Transient item cache: proactively purge expired items to prevent wasted memory.

All of this tuned not just throughput, but predictability. Not just average speed, but worst-case resilience.  
Tail latency isn't a technical problem. It's an experience problem. It's what the user actually feels.

#### 7. Systems Thinking as Infrastructure Literacy

Reading this paper made me realize that systems thinking is not optional. It's the difference between building something that works and something that survives.

Facebook's memcache is full of trade-offs I wouldn't have guessed:
- Serving stale data to save a DB.
- Using UDP with dropped packets on purpose.
- Letting cache invalidations lag, on purpose, to keep systems flowing.

This is the kind of engineering that comes from pressure, not planning. From watching things break and learning what actually matters.  
I came away not just with design ideas, but with a new lens on how to build systems. It questions purity, tolerates imperfection, and optimizes for the system, not the module.

I don't need a billion QPS to apply that.

----
References:  
[1] [Scaling Memcache at Facebook
](https://research.facebook.com/publications/scaling-memcache-at-facebook/)


---

## Bitmasking In Go

**Date:** 2025-04-17
**URL:** http://localhost:1313/bitmasking/
**Description:** Bitmasking is one of those computer science tricks that feels like wizardry, until you realize it's just some clever shifting and binary math. This blog explores the idea, shows how we use it in Go, and why it's surprisingly useful when working with databases like Couchbase.


# Bitmasking in Go

Bitmasking is one of those computer science tricks that feels like wizardry, until you realize it's just some clever shifting and binary math. This blog explores the idea, shows how we use it in Go, and why it's surprisingly useful when working with databases like Couchbase.

---

## What's Bitmasking?

A **bitmask** is just an integer where each **bit** (0 or 1) represents a flag or state. Instead of storing multiple booleans in a slice or map, you cram them into a single `int`. Fast to compute, fast to store, and great for indexing.

Let's get into what actually happens.

---

## How Bitmasking Actually Works

Let's say you want to represent a few features:

| Feature Name | Bit Index |
| ------------ | --------- |
| Feature A    | 0         |
| Feature B    | 1         |
| Feature C    | 2         |
| Feature D    | 3         |

We don't store `[true, false, true, false]`. We store a single number.

### Encoding

To encode which bits are **on**, we use the left-shift operator `1 << n`. This is how we turn a bit position into a power of two.

For example:

```go
1 << 0  // = 1     = 00000001
1 << 1  // = 2     = 00000010
1 << 2  // = 4     = 00000100
1 << 3  // = 8     = 00001000
```

To combine them, we use **bitwise OR** (`|=`):

```go
mask := 0
mask |= 1 << 1  // enable Feature B → mask = 2
mask |= 1 << 3  // enable Feature D → mask = 10
```

So:

```go
Encode([]int{1, 3}) = 10
// Binary: 00001010
```

### Decoding

To get the list of enabled features back, we use **bitwise AND** (`&`) to check each bit one by one:

```go
for i := 0; i < 32; i++ {
    if mask & (1 << i) != 0 {
        fmt.Println("Bit", i, "is set")
    }
}
```

Why does this work?

Because `(mask & (1 << i))` will only be non-zero **if** bit `i` is set in `mask`. For example:

```go
mask := 10        // 00001010
1 << 3 = 8        // 00001000
10 & 8 = 8        // Bit 3 is set

1 << 1 = 2        // 00000010
10 & 2 = 2        // Bit 1 is set

1 << 2 = 4        // 00000100
10 & 4 = 0        // Bit 2 is NOT set
```

### Toggle a Bit

To flip a bit (on → off, or off → on), use **bitwise XOR** (`^`):

```go
mask := 10 // 00001010
mask ^= 1 << 3 // Bit 3 was 1 → now 0 → mask becomes 2
```

### Summary of Operators

| Operator | Meaning             | Use Case            |
| -------- | ------------------- | ------------------- |
| `1 << n` | Bit at position `n` | Build mask          |
| `\|=`    | Bitwise OR          | Set a bit           |
| `&`      | Bitwise AND         | Check if bit is set |
| `^=`     | Bitwise XOR         | Toggle bit (flip)   |

---

## The Go Bitmask Package

Here's the package I wrote with these concepts built in:

```go
// Encode returns a bitmask with bits set for each ID in the input slice.
// Each ID is converted using: 1 << id = 2^id
func Encode(ids []int) int {
	var mask int
	for _, id := range ids {
		mask |= 1 << id
	}
	return mask
}

// Decode returns which bits are set in the mask.
func Decode(mask int) []int {
	var ids []int
	for bit := 0; mask != 0; bit++ {
		if mask&1 == 1 {
			ids = append(ids, bit)
		}
		mask >>= 1
	}
	return ids
}

// HasBit returns true if the bit at position id is set.
func HasBit(mask int, id int) bool {
	return (mask & (1 << id)) != 0
}

// ToggleBit flips the bit at position id.
func ToggleBit(mask int, id int) int {
	return mask ^ (1 << id)
}
```

---

## Why Use Bitmasking in Couchbase?

Let's take the example of **feature flags** or **user roles** stored in Couchbase.

### Option A: Store as Arrays (Bad)

```json
{
  "user_id": 123,
  "roles": [1, 3, 5]
}
```

Problems:

- Indexing arrays is expensive. Each value creates a separate index entry.
- Couchbase will use `DISTINCT SCAN`, which can be heavy.
- Filtering with `WHERE ANY r IN roles SATISFIES r = 3 END` is slow under load.

### Option B: Store as Bitmask (Good)

```json
{
  "user_id": 123,
  "role_mask": 42
}
```

- `42 = 00101010` → roles 1, 3, and 5 are set.
- Use a simple numeric index:
  ```sql
  CREATE INDEX idx_role_mask ON users(role_mask);
  ```
- Filter with:
  ```sql
  SELECT meta().id FROM users WHERE BITAND(role_mask, 8) = 8;
  ```
  This checks if role 3 is enabled (since `1 << 3 = 8`).

Fast, lean, and no complex joins or array scans.

---

## When to Use Bitmasking

Good for:

- Feature flags
- User roles/permissions
- Categorical flags with a known upper limit (< 64 ideally)
- Couchbase performance under large load

Not great for:

- Huge dynamic lists
- Highly descriptive data (bitmasks are not self-documenting)
- Frequent schema changes (bit positions are hard-coded)

---

Bitmasking isn't magic; it's just integers in disguise. When used well, it's a powerful way to compress and manipulate data for faster storage and filtering, especially in database systems like Couchbase.

If you're tracking a small set of flags or categories, think in powers of two and let your bits do the work.


---

## Docker Volume Backup: The 1-Line Command You Should Be Using (Instead of tar)

**Date:** 2025-04-13
**URL:** http://localhost:1313/how-to-backup-and-restore-docker-volumes/
**Description:** When working with Docker, persistent data is often stored in volumes. Unlike container filesystems, volumes survive restarts and recreations. But what if you need to move this data from one machine to another?  Here's a quick and reliable way to back up a Docker volume on one computer and restore it on another.

# How to Backup and Restore Docker Volumes Between Machines

When working with Docker, persistent data is often stored in volumes. Unlike container filesystems, volumes survive restarts and recreations. But what if you need to move this data from one machine to another?  
Here's a quick and reliable way to back up a Docker volume on one computer and restore it on another.

I find this use case very often when I'm working on a new project that changes the database schema and I need to maintain existing code with old schema.   
To keep both the new project and the existing code working, I create a new volume from my backup, attach it to my docker-compose database service and work on the new project. When I need to go back to the existing code for a bug fix or enhancement, it is a one line change in the docker-compose.

## Step 1: Back Up the Docker Volume

Let's say you have a volume named db_data that you want to back up.
Run this command from the terminal in your project directory:
```sh
docker run --rm \
    -v db_data:/source \ 
    -v $(pwd):/backup \ 
    busybox \ 
    tar czf /backup/db_data.tar.gz -C /source .
```

What it does?  
- `--rm`: Automatically removes the container after it finishes.  
- `-v db_data:/source`: Mounts the source Docker volume.
- `-v $(pwd):/backup`: Mounts your current directory to write the backup file. 
- `busybox`: A tiny Linux container that includes tar.  
- `tar czf`: Creates a compressed tarball of the volume contents.

This will create a file called `db_data.tar.gz` in your current directory.


## Step 2: Check the Backup Contents

Want to double-check what's inside the archive?  
Use:
```sh
tar -tzf db_data.tar.gz | head
```
This lists the first few files inside the backup, so you know it's not empty or corrupt.

## Step 3: Move the Backup to the New Machine
Transfer `db_data.tar.gz` to the new machine. You can use scp, ftp, a USB drive, cloud storage, AirDrop, whatever works for you.

Example:
```sh
scp db_data.tar.gz v@mac2:\~/Downloads
```

## Step 4: Restore the Docker Volume
On the new machine, create an empty volume with the same name:
```sh
docker volume create db_data
```
Now restore the backup into that volume:
```sh
docker run --rm \  
    -v db_data:/restore \  
    -v \~/Downloads:/backup \  
    busybox \  
    tar xzf /backup/db_data.tar.gz -C /restore
```

What it does?  
- `-v db_data:/restore`: Mounts the target volume to write data into.  
- `-v ~/Downloads:/backup`: Mounts the directory where the tarball lives.  
- `tar xzf`: Extracts the archive into the volume.


## Step 5: Verify the Restore

Option 1: Use Docker Desktop, Go to Volumes and explore `db_data` visually.  
Option 2: Inspect from the terminal:
```sh
docker run --rm -v db_data:/data busybox ls /data
```

This lets you see the contents directly from the CLI.

## Bonus: Use the `dvbackup` Script

Source code: [github.com/vk4s/docker-volume-backup](https://github.com/vk4s/docker-volume-backup)

If you find yourself doing this often, you might want to use the `dvbackup` script (I have created it), which simplifies the entire process:

1. Install the script: (always check the contents of any script before running it in your system).
```sh
curl -O https://raw.githubusercontent.com/vk4s/docker-volume-backup/main/dvbackup
chmod +x dvbackup
./dvbackup install
```

2. Backup volumes:
```sh
dvbackup backup ./backups db_data postgres_data
```

3. Restore volumes (with same names):
```sh
dvbackup restore ./backups db_data postgres_data
```

4. Restore with new names:
```sh
dvbackup restore ./backups new_db=db_data new_pg=postgres_data
```

The script provides following features:
- Automatic validation of volume names
- Safe handling (won't overwrite existing volumes)
- Support for multiple volumes in one command
- Cross-platform compatibility (macOS, Linux, Windows Git Bash)

It's a great tool for developers who frequently need to move Docker volumes between machines or create backups of their data.


## Docker Volume Cheatsheet

### Basic Volume Operations

```bash
# Create a new volume
docker volume create my_volume

# List all volumes
docker volume ls

# Inspect a volume
docker volume inspect my_volume

# Remove a volume
docker volume rm my_volume

# Remove all unused volumes
docker volume prune
```

### Using Volumes with Containers

```bash
# Mount a volume to a container
docker run -v my_volume:/data my_image

# Mount a volume with read-only access
docker run -v my_volume:/data:ro my_image

# Mount a specific directory as a volume
docker run -v /host/path:/container/path my_image

# Use a named volume with docker-compose
# volumes:
#   my_volume:
#     name: my_volume
```

### Volume Backup and Restore

Checkout `dvbackup` above.

### Volume Management

```bash
# Check volume disk usage
docker system df -v

# Backup all volumes in a project
docker-compose down
dvbackup backup ./backups $(docker volume ls -q --filter name=project_name)

# Migrate volumes between hosts
dvbackup backup ./backups volume_name
# Transfer backup file to new host
dvbackup restore ./backups volume_name
```

### Common Volume Patterns

```bash
# Database volumes
docker volume create postgres_data
docker run -v postgres_data:/var/lib/postgresql/data postgres

# Application data
docker volume create app_data
docker run -v app_data:/app/data my_app

# Configuration volumes
docker volume create app_config
docker run -v app_config:/app/config my_app

# Cache volumes
docker volume create app_cache
docker run -v app_cache:/app/cache my_app
```




---

## Testing Types

**Date:** 2025-03-11
**URL:** http://localhost:1313/testing-types/
**Description:** Understanding different types of software testing is crucial for delivering a reliable application. Smoke Testing checks basic stability after a new build, while Sanity Testing verifies specific bug fixes or minor updates. Functional Testing ensures that features work as expected based on business requirements. Regression Testing prevents new changes from breaking existing functionality. End-to-End (E2E) Testing simulates real-world user workflows, and Performance Testing checks system speed, load handling capacity, and responsiveness. Implementing these (some or all) test types helps maintain software quality and prevent critical failures.


Understanding different types of software testing is crucial for delivering a reliable application. Smoke Testing checks basic stability after a new build, while Sanity Testing verifies specific bug fixes or minor updates. Functional Testing ensures that features work as expected based on business requirements. Regression Testing prevents new changes from breaking existing functionality. End-to-End (E2E) Testing simulates real-world user workflows, and Performance Testing checks system speed, load handling capacity, and responsiveness. Implementing these (some or all) test types helps maintain software quality and prevent critical failures.

Here's a basic explanation of each test type with simple examples:

## 1. Smoke Testing

Purpose: Ensures that the core functionality of the application is working before deeper testing. It acts as a basic stability check after a build or deployment.

Example:  
• Checking if the application launches successfully.  
• Verifying if a user can log in with valid credentials.  
• Ensuring the homepage loads without errors.

💡 Analogy: Like checking if a car starts before going on a long drive.

## 2. Sanity Testing

Purpose: A quick check of specific functionality after minor changes or bug fixes to ensure they work as expected. It is narrower than smoke testing.

Example:  
• Verifying if a fixed issue (e.g., password reset functionality) works correctly.  
• Ensuring a new button added to the UI works as expected.  
• Checking if a database update reflects correctly in the UI.

💡 Analogy: Like fixing a broken switch in a house and ensuring only that switch works, without checking all the lights.

## 3. Functional Testing

Purpose: Tests whether the application behaves as expected according to business requirements. It focuses on what the system does.

Example:  
• Testing a shopping cart to see if adding/removing items works correctly.  
• Checking if an email is sent when a user registers.  
• Ensuring a user can apply a discount coupon and see the correct price.

💡 Analogy: Like testing if a vending machine dispenses the correct snack when the right button is pressed.

## 4. Regression Testing

Purpose: Ensures that recent code changes do not break existing functionality. It is done after bug fixes, updates, or new feature additions.

Example:  
• After updating the login process, testing if previous login methods (Google, Facebook) still work.  
• Ensuring a new feature in a banking app does not break existing transactions.  
• After fixing a checkout bug, verifying if other payment options still work.

💡 Analogy: Like fixing a broken pipe in a house and checking if all other plumbing still works.

## 5. End-to-End (E2E) Testing

Purpose: Tests the entire user journey across multiple integrated systems, mimicking real-world usage.

Example:  
• Verifying if a user can register, search for a product, add it to the cart, make a payment, and receive an order confirmation email.  
• Testing if a banking app allows fund transfers, sends notifications, and updates transaction history correctly.
Testing.

💡 Analogy: Like checking if a restaurant experience flows smoothly: from ordering food to payment and receiving a bill.

## 6. Performance Testing

Purpose: Checks how the system performs under different loads, ensuring speed, scalability, and responsiveness.

Example:  
• Measuring if a website loads within 2 seconds under normal traffic.  
• Checking if a mobile app can handle 10,000 users at the same time.  
• Verifying if a database query retrieves results within an acceptable time limit.

💡 Analogy: Like stress-testing a bridge to see how much weight it can handle before it collapses.


---

## Tools I Use on My Mac

**Date:** 2025-03-10
**URL:** http://localhost:1313/tools-i-use-on-my-mac/
**Description:** 


# Tools I Use on My Mac

As a new Mac user, it has been a tough journey getting used to limitations of macOS compare to linux based os. I rely on a set of carefully chosen tools to improve productivity, streamline workflows, and make my overall experience less frustrating. Here's a look at top 20 essential apps I use daily:

## System & UI Enhancements

### 1. Scroll Reverser

Allows me to reverse the scrolling direction for trackpads and mice independently, making navigation more intuitive.

### 2. BeardedSpice

A must-have for controlling media playback on websites and apps using my keyboard's media keys. It can control Spotify.

### 3. KDE Connect

Seamlessly integrates my Mac with my Android phone, enabling file transfers, notifications, and remote input control.

### 4. Flux

Automatically adjusts my screen's color temperature based on the time of day, reducing eye strain and improving sleep.

### 5. Shottr

A lightweight and fast screenshot tool that includes built-in annotation and OCR capabilities.

### 6. Maccy

A clipboard manager that keeps track of copied text and allows quick retrieval of previous copies.

### 7. GPG Keychain

Essential for managing GPG encryption keys for secure email communication and file encryption.

### 8. Music Decoy

Helps me keep block apple music from launching whenever I connect my earphones/headphones. What a stupid idea to auto launch it.

### 9. AltTab

Brings a Windows-style Alt+Tab app switcher to macOS, making app switching faster and more convenient.

### 10. Rectangle

A powerful window management tool that lets me quickly arrange and snap windows using keyboard shortcuts.

### 11. Monitor Control

Gives me precise control over external monitor brightness and volume directly from macOS.

## Development & Productivity

### 12. Docker

Enables me to run containerized applications, making development and deployment more efficient.

### 13. iTerm2

A feature-packed terminal replacement that enhances productivity with split panes, custom profiles, and better performance.

### 14. Obsidian

My go-to knowledge management tool for organizing notes, ideas, and projects in a Markdown-based system.

### 15. Postman

A great API testing tool that helps me debug and develop RESTful services with ease.

### 16. JetBrains Fleet/GoLand/PyCharm

A suit of powerful and modern IDEs from JetBrains that provide world's best coding experience.

### 17. VS Code

Well, we all know what it is. Anyways, it's a highly customizable and extensible code editor that I use for various programming tasks.

### 18. Sublime Merge

A fast and powerful Git GUI that makes managing repositories much smoother.

### 19. Scrcpy

A fantastic open-source tool for mirroring and controlling my Android phone from my Mac.

### 20. ChatGPT

I use ChatGPT for brainstorming, coding assistance, and getting quick explanations on technical topics.

These tools have significantly improved my daily workflow, making my Mac experience more efficient and enjoyable. If you use any of these or have other recommendations, do let me know!


---

## Understanding T and *T method receivers in Go

**Date:** 2025-02-23
**URL:** http://localhost:1313/method-receivers-in-go/
**Description:** In Go, method receivers determine whether a method acts on a copy of a value or a reference to it. This choice isn't just about performance; it affects correctness and behavior, especially when dealing with synchronization primitives (mutex, wait group, etc), slices, and embedded types.


In Go, method receivers determine whether a method acts on a copy of a value or a reference to it. This choice isn't just about performance; it affects correctness and behavior, especially when dealing with synchronization primitives (mutex, wait group, etc), slices, and embedded types.

In this post I explore when to use T (a value receiver) vs. \*T (a pointer receiver) and why, in most cases, pointer receivers should be the default and preferred.

---

#### Understanding T and \*T

In Go, every type T has an associated pointer type \*T. Taking the address of a T variable results in a \*T:

```go
type T struct {
    a int
    b bool
}

var t T     // t is of type T
var p = &t  // p is of type *T
```

T and \*T are distinct types. You can't substitute \*T where T is expected and vice versa.

#### Declaring Methods on Types

Go allows you to declare methods on any type you define in your package. That means you can attach methods to both T and \*T:

```go
func (t T) ValueMethod() { /* operates on a copy */ }
func (t *T) PointerMethod() { /* operates on the original */ }
```

Now the billion dollar question: When should a method use a pointer receiver?

#### When to Use a Pointer Receiver

##### 1. The method modifies the receiver.

If your method changes the state of T, you must use \*T, or modifications won't persist.

```go
func (p *Person) Rename(name string) {
    p.Name = name
}
```

##### 2. The type is large.

Passing large structs by value incurs unnecessary copying.

```go
type BigStruct struct {
    Data [1024]byte
}

func (b BigStruct) BadMethod() { /* Copies 1024 bytes */ }

func (b *BigStruct) GoodMethod() { /* Uses reference of the 1024 bytes */ }
```

##### 3. The type contains synchronization primitives (e.g., sync.Mutex).

Copying a sync.Mutex defeats it purpose of maintaining the state.

```go
type Counter struct {
    mu  sync.Mutex
    val int
}

func (c *Counter) Increment() {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.val++
}
```

Declaring Increment on Counter (not \*Counter) would copy mu, making it useless.

##### 4. The type is meant to be used via pointers.

Some types naturally work as pointers, such as linked list nodes.

##### 5. The type is embedded in another struct.

If a type is meant to be embedded, methods must use pointer receivers to avoid unintended copies. Copies introduce performance penalty.

```go
type Stats struct {
    a, b, c Counter
}

func (s Stats) Sum() int {
    return s.a.Increment() + s.b.Increment() + s.c.Increment() // Copies `Counter`, breaks mutex
}
```

#### When To Use a Value Receiver?

A value receiver is fine if:

- The method does not modify the receiver.
- The type is small and simple (like int, bool, or struct{X, Y int}).
- The type is inherently immutable.

For example:

```go
type Point struct {
    X, Y int
}

func (p Point) Distance(q Point) float64 {
    dx, dy := float64(q.X-p.X), float64(q.Y-p.Y)
    return math.Sqrt(dx*dx + dy*dy)
}
```

Here, Distance doesn't modify Point, and copying two ints is negligible.

#### Should Methods That Don't Mutate Use a Pointer Receiver?

Yes, in most cases. Even if a method doesn't modify the receiver, using \*T avoids unintended copies and maintains consistency across your codebase.
The exceptions:

- Small, immutable types (e.g., Point).
- Types that should be copied intentionally, like function options or temporary configurations.

Basically:

- Default to using \*T (reference receiver) unless you have a good reason not to.
- Use T (value receiver) only when your type is small and meant to be copied.
- Never use T (value receiver) if the type has a sync.Mutex, a slice, or is embedded in another struct.

By following these guidelines, you avoid subtle bugs and unintended performance penalties in your Go code.

Cheers!!
Happy coding.


---

## Analysis Paralysis in Engineering Teams

**Date:** 2025-02-10
**URL:** http://localhost:1313/analysis-paralysis/
**Description:** In engineering, progress is key. However, sometimes teams get stuck in endless discussions, over-planning, and constant changes, delaying actual work. This situation is called "Analysis Paralysis." It happens when people focus too much on making perfect decisions instead of moving forward with practical solutions.


Analysis Paralysis in Engineering

![An image representing "Analysis Paralysis" in engineering. It shows a team stuck in decision-making and hesitant to take action.](/images/uploads/image.webp)

In engineering, progress is key. However, sometimes teams get stuck in endless discussions, over-planning, and constant changes, delaying actual work. This situation is called "Analysis Paralysis." It happens when people focus too much on making perfect decisions instead of moving forward with practical solutions.

One common cause of analysis paralysis is fear of failure. Engineers and managers want to avoid mistakes, so they keep analyzing every possible outcome. While planning is important, overthinking can waste time and resources. Another reason is complex decision-making. If too many people are involved, or if there are too many options, teams struggle to decide and keep delaying action.

For example, imagine a software development team working on a new app. They spend weeks debating the best database technology: SQL or NoSQL. They read articles, compare benchmarks, and discuss performance differences. Meanwhile, no actual coding happens. Instead of choosing one and adjusting later if needed, they waste valuable time in discussions.

The solution to analysis paralysis is to set clear deadlines for decisions and follow the "good enough" principle. Instead of chasing perfection, engineers should pick a reasonable option and refine it along the way. Agile development, for instance, focuses on building a minimum viable product (MVP) and improving it step by step rather than perfecting everything upfront.

In short, analysis paralysis slows down innovation. While careful planning is necessary, taking action is just as important. Engineers must find the right balance: analyzing enough to make informed decisions but not so much that progress stops.


---

## Go Clean Code Guidelines

**Date:** 2024-09-22
**URL:** http://localhost:1313/go-clean-code-guidelines/
**Description:** When it comes to writing clean, maintainable code, there are a few fundamental rules that can help improve the overall structure and quality of your codebase. As engineers, our goal should be to keep things simple, clear, and scalable. With this in mind, here are some guidelines which prioritize code readability, functional clarity, and the overall maintainability of a project.  These guidelines are based on principles from clean code, SOLID, and functional programming while emphasizing simplicity over unnecessary complexity.


### Go Clean Code Guidelines for Code Quality and Maintainability.

When it comes to writing clean, maintainable code, there are a few fundamental rules that can help improve the overall structure and quality of your codebase. As engineers, our goal should be to keep things simple, clear, and scalable. With this in mind, here are some guidelines which prioritize code readability, functional clarity, and the overall maintainability of a project.
These guidelines are based on principles from clean code, SOLID, and functional programming while emphasizing simplicity over unnecessary complexity.

---

### Rule 1: Function Naming Clarity

Your function names must clearly indicate what they do. Avoid vague names or overly complex abstractions. Each function should serve a single purpose, and its name should reflect this responsibility.

#### Example:

Instead of a vague name like `ProcessData`:

```go
// Bad
func ProcessData() {
    // Complex logic for processing some data
}
```

Use something more descriptive like `ConvertOrderToInvoice`:

```go
// Good
func ConvertOrderToInvoice(order Order) Invoice {
    // Convert an Order object into an Invoice
}
```

Here, the function name precisely conveys what the function does, making it easier for developers to follow the code.

### Rule 2: No Magic Numbers or Strings

Avoid using magic numbers or strings directly in your code. Instead, define constants with meaningful names to improve readability and reduce errors.

#### Example:

```go
// Bad
if user.Role == "admin" {
    // Grant access
}
```

Replace magic strings with constants:

```go
// Good
const AdminRole = "admin"
if user.Role == AdminRole {
    // Grant access
}
```

This makes the code more readable and easier to maintain. If the role name changes in the future, you only need to update it in one place.

### Rule 3: Use Clear and Consistent Variable Names

Variable names should be descriptive enough to indicate their purpose. Do not abbreviate in a way that makes it hard for others to understand the meaning of the variable.

#### Example:

```go
// Bad
v := 10 // What is 'v'?
```

Instead, use meaningful names:

```go
// Good
maxAllowedItems := 10
```

With clear variable names, other developers can quickly understand what the variable represents.

### Rule 4: Keep Functions Small

Each function should perform one task and do it well. If a function is doing multiple things, break it down into smaller functions. This improves code readability and makes testing easier.

#### Example:

```go
// Bad
func HandleOrder(order Order) Invoice {
    // Generating invoice, many lines of code

    // Save invoice to database, many lines of code

    // Send notification to customer, many lines of code
    return invoice
}
```

The function is doing too many things. Instead, split it into smaller functions:

```go
// Good
func HandleOrder(order Order) Invoice {
    invoice := generateInvoice(order)
    persistInvoice(invoice)
    notifyCustomer(invoice)
    return invoice
}
func persistInvoice(invoice Invoice) {
    // Save invoice to the database
}
func notifyCustomer(invoice Invoice) {
    // Send a notification to the customer
}
```

This makes each function's responsibility clearer, and it is easier to test and maintain.

### Rule 5: Error Handling Should Be Explicit

Always check for errors and handle them explicitly. Do not assume the absence of an error. The code should either handle the error gracefully or propagate it back to the caller.

#### Example:

```go
// Bad
result, err := getUserDetails(userID)
process(result) // No error handling!
```

Instead, always handle or propagate errors:

```go
// Good
result, err := getUserDetails(userID)
if err != nil {
    logError(err)
    return nil, err
}
process(result)
```

Proper error handling ensures that bugs and edge cases are caught early and makes the code more reliable.

### Rule 6: Use Structs for Complex Data

If a function accepts multiple parameters of related data, bundle them into a struct. This makes the function signature cleaner and more flexible.

#### Example:

```go
// Bad
func CreateUser(name string, age int, address string, email string) User {
    // Create a new user
}
```

Instead, use a struct for the user's details:

```go
// Good
type UserDetails struct {
    Name    string
    Age     int
    Address string
    Email   string
}
func CreateUser(details UserDetails) User {
    // Create a new user
}
```

This approach makes the code easier to read and modify in the future, especially if more fields are added to the user details.

### Rule 7: Use Consistent Struct Field Order

Maintain consistency in the order of struct fields. Group similar fields together to enhance readability.

#### Example:

```go
// Bad
type User struct {
    Address   string
    Age       int
    Email     string
    Name      string
}
```

Instead, group fields logically:

```go
// Good
type User struct {
    Name      string
    Age       int
    Address   string
    Email     string
}
```

This way, when you or others revisit the code, it's easier to understand the structure of the object.

### Rule 8: Inline Commenting for Complex Logic

Where necessary, provide inline comments explaining the purpose of complex logic or algorithms. Avoid obvious comments, but make sure to clarify non-obvious code.

#### Example:

```go
// Bad
// Add two numbers
total := a + b
```

Instead, explain complex logic:

```go
// Good
// Apply discount if the customer is a premium user and their purchase exceeds $100
if customer.IsPremium && order.Total > 100 {
    applyDiscount(order)
}
```

The inline comment explains the logic behind applying the discount, which may not be immediately apparent from the code itself.

### Rule 9: Group Related Code into Logical Blocks

Separate blocks of code that serve different purposes with blank lines. This improves readability and helps group related logic together.

#### Example:

```go
// Bad
func ProcessOrder(order Order) {
    validateOrder(order)
    applyDiscount(order)
    finalizePayment(order)
    sendReceipt(order)
}
```

Instead, separate the code into logical sections:

```go
// Good
func ProcessOrder(order Order) {
    validateOrder(order)
    applyDiscount(order)

    finalizePayment(order)
    sendReceipt(order)
}
```

This makes the code visually easier to follow and highlights the different stages of processing an order.

### Rule 10: Function Arguments vs. Return Types

Make sure function names clearly reflect their return types and arguments. This prevents confusion and makes the function's purpose obvious to the caller.

#### Example:

```go
// Bad
func FetchData(input string) bool {
    // Returning a boolean from a 'Fetch' function is misleading
}
```

Instead, update the name to reflect the return type:

```go
// Good
func IsDataAvailable(input string) bool {
    // Now it's clear that the function returns a boolean
}
```

#### This helps maintain consistency in your codebase and makes the function's role immediately clear.

By following these guidelines, you can ensure that your codebase remains clean, efficient, and easier to navigate for you and your team. Keep function names clear, avoid magic numbers, handle errors explicitly, and group your code logically for the best results.
Happy coding!


---

## SOLID and Functional Programming Principles in Go

**Date:** 2024-09-22
**URL:** http://localhost:1313/solid-and-functional-programming-principles-in-go/
**Description:** SOLID and functional programming principles explained and implemented in Go


Let's go through the **SOLID principles** and **functional programming principles** that can apply to your Go codebase. I'll provide simple Go examples along with brief explanations to show how each principle can improve code quality.

---

### **SOLID Principles**

1. **S**ingle Responsibility Principle (SRP)
   - **Description:** A class or function should have only one reason to change, meaning it should only have one job or responsibility.

   **Example:**

   ```go
   // Good: Separate responsibilities into two functions
   type EmailSender struct{}


   func (e *EmailSender) SendEmail(to, subject, body string) {
       // logic for sending email
   }


   func formatEmailBody(template string, data map[string]string) string {
       // logic for formatting the email body
       return ""
   }


   // Avoid: Single function doing multiple jobs
   func sendFormattedEmail(to, subject, template string, data map[string]string) {
       body := formatEmailBody(template, data)
       // email sending logic mixed with formatting
   }
   ```

---

2. **O**pen/Closed Principle (OCP)
   - **Description:** Software entities (classes, modules, functions) should be open for extension but closed for modification.

   **Example:**

   ```go
   // Good: Extend functionality via interfaces
   type PaymentProcessor interface {
       ProcessPayment(amount float64) error
   }


   type PayPalProcessor struct{}


   func (p PayPalProcessor) ProcessPayment(amount float64) error {
       // PayPal payment processing logic
       return nil
   }


   type CreditCardProcessor struct{}


   func (c CreditCardProcessor) ProcessPayment(amount float64) error {
       // Credit card processing logic
       return nil
   }


   // Avoid: Modifying existing functions to add new payment methods
   func processPayment(method string, amount float64) error {
       if method == "paypal" {
           // PayPal logic
       } else if method == "creditcard" {
           // Credit card logic
       }
       return nil
   }
   ```

---

3. **L**iskov Substitution Principle (LSP)
   - **Description:** Objects should be replaceable by their subtypes without altering the correctness of the program.

   **Example:**

   ```go
   // Good: Subtypes maintain expected behavior
   type Animal interface {
       Speak() string
   }


   type Dog struct{}


   func (d Dog) Speak() string {
       return "Woof!"
   }


   type Cat struct{}


   func (c Cat) Speak() string {
       return "Meow!"
   }


   func makeAnimalSpeak(a Animal) {
       fmt.Println(a.Speak())
   }


   // Avoid: Subtypes behaving unpredictably
   type SilentDog struct{}


   func (s SilentDog) Speak() string {
       return "" // Violates expectations of the Speak method
   }
   ```

---

4. **I**nterface Segregation Principle (ISP)
   - **Description:** No client should be forced to depend on methods it doesn't use. Create small, specific interfaces instead of one large interface.

   **Example:**

   ```go
   // Good: Segregated interfaces
   type Reader interface {
       Read() string
   }


   type Writer interface {
       Write(data string) error
   }


   // Avoid: Fat interface forcing implementations to use unused methods
   type ReadWriteCloser interface {
       Read() string
       Write(data string) error
       Close() error
   }


   // If a class only needs reading, it should not be forced to implement Write and Close.
   ```

---

5. **D**ependency Inversion Principle (DIP)
   - **Description:** High-level modules should not depend on low-level modules. Both should depend on abstractions (interfaces).

   **Example:**

   ```go
   // Good: High-level module depends on abstraction
   type NotificationService struct {
       sender EmailSender
   }


   func NewNotificationService(sender EmailSender) *NotificationService {
       return &NotificationService{sender: sender}
   }


   func (n *NotificationService) Notify(message string) {
       n.sender.SendEmail("user@example.com", "Notification", message)
   }


   // Avoid: High-level module directly depending on a low-level module
   func sendNotification() {
       sender := &EmailSender{} // tight coupling with EmailSender
       sender.SendEmail("user@example.com", "Notification", "Hello!")
   }
   ```

---

### **Functional Programming Principles**

1. **First-Class and Higher-Order Functions**
   - **Description:** Functions are treated as first-class citizens and can be passed around as arguments, returned from other functions, or assigned to variables.

   **Example:**

   ```go
   // Higher-order function that accepts another function as a parameter
   func applyDiscount(price float64, discountFunc func(float64) float64) float64 {
       return discountFunc(price)
   }


   func tenPercentDiscount(price float64) float64 {
       return price * 0.9
   }


   func main() {
       price := 100.0
       discountedPrice := applyDiscount(price, tenPercentDiscount)
       fmt.Println(discountedPrice) // 90.0
   }
   ```

---

2. **Pure Functions**
   - **Description:** A pure function's output is determined only by its input values, without observable side effects.

   **Example:**

   ```go
   // Good: Pure function (no side effects)
   func add(a, b int) int {
       return a + b
   }


   // Avoid: Impure function (with side effects)
   var counter int


   func addWithSideEffect(a, b int) int {
       counter++ // changes external state
       return a + b
   }
   ```

---

3. **Immutability**
   - **Description:** Data should not be modified after it is created. Instead, create a new copy of the data when changes are needed.

   **Example:**

   ```go
   // Good: Immutability by creating a new slice
   func addToSlice(slice []int, value int) []int {
       newSlice := append(slice, value)
       return newSlice
   }


   // Avoid: Modifying the original slice
   func modifySlice(slice []int, value int) {
       slice = append(slice, value)
   }
   ```

---

4. **Recursion Over Loops**
   - **Description:** Functional programming often favors recursion over iterative loops, especially when solving problems like list processing.

   **Example:**

   ```go
   // Good: Recursion
   func factorial(n int) int {
       if n == 0 {
           return 1
       }
       return n * factorial(n-1)
   }


   // Avoid: Iterative approach (loops)
   func iterativeFactorial(n int) int {
       result := 1
       for i := n; i > 0; i-- {
           result *= i
       }
       return result
   }
   ```

---

5. **Function Composition**
   - **Description:** Combine simple functions to build more complex ones, allowing for clean and reusable logic.

   **Example:**

   ```go
   func double(x int) int {
       return x * 2
   }


   func square(x int) int {
       return x * x
   }


   func compose(f, g func(int) int) func(int) int {
       return func(x int) int {
           return f(g(x)) // Apply g, then f
       }
   }


   func main() {
       doubleSquare := compose(double, square)
       fmt.Println(doubleSquare(3)) // Output: 18 (3^2 * 2)
   }
   ```

---

These principles can help keep your code clean, modular, and testable while using both object-oriented and functional paradigms.


---

## 10 Essential Tips for Beginners Starting in IT

**Date:** 2024-02-23
**URL:** http://localhost:1313/essential-tips-for-beginners-starting-in-it/
**Description:** In the ever-evolving world of IT, prioritize mastering core concepts over chasing trends. Embrace hands-on learning through projects, stay curious, seek guidance when needed, and remember that problem-solving and persistence are key to success in this dynamic field.

![A girl sitting in a room studying and thinking. Generated with AI by Vikash from Lorbic.com](/images/uploads/10-essential-tips-for-beginners-starting-in-it.jpg)

Are you new to IT? Feeling a bit overwhelmed? Don't worry, I've got you covered! In this article, I'll share 10 essential tips to help you get started on your journey in Information Technology. From mastering the basics to gaining hands-on experience and staying curious, these tips will set you on the right path. Let's dive in and start the IT adventure!

1. **Focus on Fundamentals:** Instead of trying to understand every buzzword out there or the latest tech trend, focus on building a strong foundation in core concepts such as networking, operating systems, databases, and programming languages.


2. **Hands-on Practice:** Don't just rely on theory. Get your hands dirty by working on projects, setting up environments, and troubleshooting issues. Practical experience is invaluable in IT. I can tell you from personal experience, it doesn't matter what you know or which course you have completed, only thing that matters is if you can convert that Jira ticket into some sweet .py files 😉. 


3. **Learn by Doing:** Regularly work on projects and real-world scenarios. Whether it's setting up a home lab, create open-source projects (Invite your internet friends to add features). Practical experience is the most effective way to understand complex stuff, eg. like how that payment is processed in the eCommerce website and how you can do it on your own.  Create imaginary scenarios like there is some XYZ that sells car parts and you are responsible for all tech things, now how will you design, build, host a website where you will sell those parts. Some projects can be simple 1 hour coding, some will take many months to complete, once you are addicted you can't go back. **YOU NEED THAT ADDICTION** in this distracted, shorts-oriented world.


4. **Stay Curious:** The field of IT is rapidly changing (new JS and CSS frameworks being born every week), so it's essential to stay curious and keep learning. Be open to exploring new technologies, tools, and methodologies to expand your knowledge and skills continuously. Like the AI things going on right now. But don't distract from your path.


5. **Need Help? ask for it:** Don't hesitate to reach out to more experienced professionals for guidance or mentorship. Not everyone you ask will help you, but if you ask 10 people at least 1 may help you. As we say in Hindi *"*जहाँ अहंकार होता है, वहाँ ज्ञान नहीं होता", which translates to "Where there is pride, there is no wisdom" in English. 


6. **Stay Updated but Don't Chase Every Trend:** Stay informed about industry trends and advancements, but don't feel pressured to chase every new technology or buzzword. Instead, focus on understanding the underlying principles and how they apply to your work. You can't master everything. Focus on the things you care and are useful.


7. **Practice Problem-Solving:** Our industry always revolves around solving complex problems and troubleshooting issues. Strengthen your problem-solving skills by tackling challenges, breaking down problems into smaller steps, and systematically finding solutions. I'm not talking about mindlessly doing DSA questions. There are other problems that need solutions. Such as, how will you modernize and deploy a decade old monolith into cloud which is so goddamn fragile, you touch it, it breaks!


8. **Document and read docs:** Keep track of your learning journey by documenting your progress, experiences, and lessons learned. Reflecting on your achievements and challenges can help remember your understanding and identify areas for improvement. For this build your own blog.


9. **Stay Patient and Persistent:** Learning IT is challenging, and you will encounter setbacks along the way. Stay patient, be persistent, and don't get discouraged by obstacles. Every challenge is an opportunity to learn and grow. Back in college, when I was learning Django and working on my first project there was an issue with my implementation which took over a week to find it and fix it.


10. **Enjoy the Journey:** Remember to enjoy the learning process and celebrate your successes, no matter how small. IT is a fascinating and rewarding field, and embracing the journey of continuous learning can lead to fulfilling and meaningful career opportunities.



Thinking something? Write it down in the comments.

Have fun. Happy learning.


---

## NGINX: Building from Source and Installation on Linux (Updated)

**Date:** 2024-02-02
**URL:** http://localhost:1313/build-nginx-from-source/
**Description:** In this blog we are going to build NGINX from source code. And we will configure NGINX's settings, paths, and add or remove modules.


[NGINX](https://en.wikipedia.org/wiki/Nginx) is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. NGINX is a free and open source software licensed under _2-clause BSD_.\
You can easily download and use NGINX in Windows, macOS, Linux and other operating systems [link](https://nginx.org/en/download.html) read NGINX installation instructions from the [official docs](https://nginx.org/en/docs/install.html) and install on your operating system for free.

But, in this blog we are going to build NGINX from source code. And we will configure NGINX's settings, paths, and add or remove modules.

So let's get started…

## Preparation

We'll use Ubuntu 20.04 LTS as our operating system, though these instructions are applicable to other Linux-based systems as well.

### 1. Download the source code:

Start by visiting the NGINX download page and choose either the stable release or the mainline version for experimental features. For example, you can download nginx-1.21.4.

Link: [NGINX download page](https://nginx.org/en/download.html). \
Note: You can use `wget` or `curl` to download.

```shell
wget https://nginx.org/download/nginx-1.21.4.tar.gz
```

After downloading, extract the tar file using:

```shell
tar -zxvf nginx-1.21.4.tar.gz
```

Now cd into `nginx-<version>` directory.

### 2. Install Required Tools:

Ensure you have the necessary build tools. On Ubuntu, use `apt` package manager:

```shell
sudo apt install build-essential
```

Additionally, install required dependencies and libraries:

```shell
sudo apt install libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev
```

### 3. Configure NGINX for build

Check all configuration options:

`sudo ./configure --help`

Refer to the NGINX documentation's module reference section for details on modules. Configure installation paths and modules as needed.\
Follow the link: [nginx docs > module reference section](https://nginx.org/en/docs/)

Configure Installation:

```shell
sudo ./configure \
  --sbin-path=/usr/bin/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --pid-path=/var/run/nginx.pid \
  --with-pcre \
  --with-http_ssl_module \
  --with-http_v2_module \
  --with-http_gunzip_module
```

> –sbin-path=/usr/bin/nginx\
> –conf-path=/etc/nginx/nginx.conf\
> –error-log-path=/var/log/nginx/error.log\
> –http-log-path=/var/log/nginx/access.log\
> –pid-path=/var/run/nginx.pid\
> –with-pcre --with-http_ssl_module\
> –with-http_v2_module\
> –with-http_gunzip_module

### 4. Compile and Install the Application:

Compile NGINX by running:

```shell
make
# Run below command to install NGINX
make install
```

### 5. Restart NGINX and Configure as a Service:

Reload NGINX to apply changes:

`nginx -s reload`

**If you encounter an error related to the nginx.pid file missing, follow these steps:**

> nginx: \[error] open() "/var/run/nginx.pid" failed (2: No such file or directory)

**5.1. Add Nginx as service**:

Add NGINX as a service by creating a systemd config file. Create or modify `/lib/systemd/system/nginx.service` with appropriate paths:\
Visit: [nginx systemd config](https://www.nginx.com/resources/wiki/start/topics/examples/systemd/) and create the file.\
_`--/lib/systemd/system/nginx.service--`_\
Change _pid_ and _sbin_ path as set in step 3.

```/lib/systemd/system/nginx.service
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/bin/nginx -t
ExecStart=/usr/bin/nginx
ExecReload=/usr/bin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target
```

**5.2. Start NGINX using systemd:**

```sh
sudo systemctl start nginx
```

**5.3. Enable NGINX to start on boot:**

```sh
sudo systemctl enable nginx
```

That's all you need to do to compile and run NGINX on Ubuntu.\
If you get any error or suggestion please leave a comment.

Happy NGINX-ing! 🤟


---

## How to Install Docker Desktop on Another Drive in Windows

**Date:** 2024-01-27
**URL:** http://localhost:1313/docker-custom-installation-location-windows/
**Description:** Is Docker filling up your C: drive? Here is how to install or move Docker Desktop and its containers to D: drive on Windows 10/11 using WSL2 flags.


Docker Desktop on Windows installs to your C: drive by default. When you pull large images (often 1.5GB each), add build layers, and stack containers, the ext4.vhdx file that holds all your container data bloats fast. This guide shows you how to route Docker to a secondary drive entirely, using CLI flags instead of the GUI.

## Quick Start

Uninstall any existing Docker first. Then open PowerShell as admin and navigate to your Downloads folder where the installer is. Run:

```powershell
start /w "" "Docker Desktop Installer.exe" install --accept-license `
  --installation-dir="D:\Containers\docker" `
  --wsl-default-data-root="D:\Containers\wsl" `
  --windows-containers-default-data-root="D:\Containers\windows-data"
```

Replace `D:\Containers` with your target path. Wait for the installer to finish (it won't show a progress window), then restart Windows.

---

## Pre-Flight Checklist

Before you start, verify your environment.

**Windows version (Build 19041+):**
```powershell
[System.Environment]::OSVersion.Version
```
Check `Settings > System > About > Windows specifications` to confirm.

**WSL2 installed:**
```powershell
wsl --list --verbose
```
If nothing shows, install WSL2 first: `wsl --install`. This installs Ubuntu by default.

**Hyper-V enabled:**
```powershell
Get-WindowsOptionalFeature -Online -FeatureName Hyper-V
```
Look for `State: Enabled`. If it says `Disabled`, enable it:
```powershell
Enable-WindowsOptionalFeature -Online -FeatureName Hyper-V -All
```
Then restart.

**Secondary drive with 80GB+ free space:**
```powershell
Get-Volume
```
Verify your target drive (D:, E:, etc.) is listed and has at least 80GB free. SSD is strongly recommended—VHDX files are I/O-heavy.

**Admin permissions:** Run PowerShell as admin. You'll see an error immediately if you lack permissions.

## Recommended Directory Structure

On your secondary drive, create this layout:

```text
D:\Containers
├───docker          (Docker application binaries, ~1GB)
├───wsl              (WSL2 ext4.vhdx file, grows with images)
└───windows-data     (Windows container data, usually empty)
```

Create these folders first:

```powershell
mkdir "D:\Containers\docker"
mkdir "D:\Containers\wsl"
mkdir "D:\Containers\windows-data"
```

## Understanding the Flags

The `--installation-dir` flag moves the Docker Desktop app binaries. These are small and don't change often.

The `--wsl-default-data-root` flag is the important one. It moves the ext4.vhdx file, which holds all your Linux container data. This file grows with every image pull, layer, and container you create. It's why your C: drive fills up. This is where the space gets consumed.

The `--windows-containers-default-data-root` flag moves the Windows container data root. You rarely need this unless you explicitly enable Windows container mode in Docker Desktop settings.

---

## Installation Steps

### Step 1: Uninstall Existing Docker (if present)

If Docker is already installed:

```powershell
# Stop the service
sc.exe stop docker

# Uninstall via Control Panel (or CLI)
"C:\Program Files\Docker\Docker\uninstall.exe" --quiet
```

Wait for the process to finish. You can verify removal:
```powershell
Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | 
  Select-Object DisplayName | 
  grep Docker
```

If nothing shows, Docker is gone.

### Step 2: Navigate to the Installer

Download Docker Desktop from [here](https://docs.docker.com/desktop/install/windows-install/). You'll get `Docker Desktop Installer.exe`. Move it to an easy location (e.g., `C:\Users\YourUsername\Downloads`) so the command below works.

Then, open PowerShell as admin and navigate to the folder:
```powershell
cd C:\Users\YourUsername\Downloads
```

### Step 3: Run the Installer with Flags

```powershell
start /w "" "Docker Desktop Installer.exe" install --accept-license `
  --installation-dir="D:\Containers\docker" `
  --wsl-default-data-root="D:\Containers\wsl" `
  --windows-containers-default-data-root="D:\Containers\windows-data"
```

The `start /w` waits for the installer to finish. You won't see a progress window, so the command will hang silently for 2-5 minutes depending on disk speed. This is normal and expected.

If it fails, skip to the Troubleshooting section below for solutions.

### Step 4: Restart Windows

```powershell
Restart-Computer
```

This is critical. Registry changes and WSL mount points won't take effect until you restart.

---

## Verification

After restart, verify the installation worked.

Run `docker info` to check the installation path:

```powershell
docker info
```

Look for `Docker Root Dir`. It should show your new path:

```
Docker Root Dir: D:\Containers\wsl\docker-desktop-data\mnt\wsl\docker-desktop\cli-tools
```

To filter just this line:

```powershell
docker info | select-string "Root Dir"
```

Pull a test image to ensure the system works:

```powershell
docker pull nginx:latest
```

Check the file size of your WSL directory to confirm data is being stored there:

```powershell
(Get-ChildItem -Path "D:\Containers\wsl" -Recurse | Measure-Object -Property Length -Sum).Sum / 1GB
```

You should see a few GB now. On subsequent image pulls, this number grows.

Finally, verify that Docker isn't secretly writing to C: anymore:

```powershell
Get-ChildItem "C:\Users\$env:USERNAME\AppData\Local\Docker" -ErrorAction SilentlyContinue
```

This folder may be empty now or contain only config files, not data.

---

## Troubleshooting

### Docker Desktop Won't Start

If Docker Desktop icon appears in the system tray but immediately crashes or disappears, you likely have a permissions issue or registry mismatch. Open PowerShell as admin and stop any lingering process first:

```powershell
Get-Process -Name "Docker Desktop" -ErrorAction SilentlyContinue | Stop-Process -Force
```

Then delete the Docker Desktop config cache:

```powershell
Remove-Item "$env:APPDATA\Docker\*" -Recurse -Force -ErrorAction SilentlyContinue
```

Restart Docker Desktop from the Start Menu. It should reinitialize cleanly.

### "Path not found" or "Directory does not exist"

If the installer complains about your path, the directories probably don't exist or there's a typo. Verify the path exists first:

```powershell
Test-Path "D:\Containers\docker"
```

Check that the drive letter is correct:

```powershell
Get-Volume -DriveLetter D
```

If the directories are missing, recreate them:

```powershell
mkdir "D:\Containers\docker"
mkdir "D:\Containers\wsl"
mkdir "D:\Containers\windows-data"
```

Then re-run the installer.

### "Access Denied" or Permission Errors

If the installer fails with a permission error even though you're running as admin, Windows Defender or antivirus is locking files. Temporarily disable real-time scanning:

```powershell
Set-MpPreference -DisableRealtimeMonitoring $true
```

Re-run the installer, then re-enable scanning:

```powershell
Set-MpPreference -DisableRealtimeMonitoring $false
```

Alternatively, add your Docker path to Defender exclusions permanently:

```powershell
Add-MpPreference -ExclusionPath "D:\Containers"
```

This prevents future scanning slowdowns.

### "WSL 2 installation is incomplete" or WSL Errors

If Docker complains about WSL2 not being ready, the WSL2 kernel or distro is missing. Run these commands:

```powershell
wsl --install
wsl --update
```

Then restart Windows.

### Installation Succeeds but Docker Uses Old C: Drive Path

If `docker info` still shows `C:\ProgramData\Docker`, the registry settings weren't applied or Docker has stale config. Uninstall completely first:

```powershell
wsl --unregister docker-desktop
wsl --unregister docker-desktop-data
```

Delete the installation folder:

```powershell
Remove-Item "C:\Program Files\Docker" -Recurse -Force -ErrorAction SilentlyContinue
```

Then reinstall using the CLI command from the Quick Start section.

### "Docker daemon is not running"

If `docker` commands fail with a daemon not running error, the Docker Desktop service didn't start. Manually start Docker Desktop from the Start Menu. Wait 10 seconds, then try a docker command:

```powershell
docker ps
```

If it still fails, check the Docker Desktop logs:

```powershell
Get-Content "$env:APPDATA\Docker\log.txt" -Tail 50
```

The last 50 lines usually show what's preventing startup.

---

## Performance Optimization

Once Docker is running from your secondary drive, a few adjustments improve performance.

Real-time scanning on the VHDX file during builds kills performance. Add the WSL directory to Windows Defender exclusions:

```powershell
Add-MpPreference -ExclusionPath "D:\Containers\wsl"
```

Verify it was added:

```powershell
Get-MpPreference | Select-Object -ExpandProperty ExclusionPath | grep Containers
```

Open Docker Desktop settings (right-click the icon, then Settings, then Resources) and adjust memory and CPU. Set "Memory" to 50-75% of your available RAM. Set "CPU" to 50% of your available cores. The disk image size auto-allocates to your secondary drive's available space.

If images pull slowly, your secondary drive itself might be the bottleneck. Test its write speed:

```powershell
$testFile = "D:\Containers\speed-test.bin"
$testSize = 100MB
$buffer = new-object byte[] 1MB
$file = [io.file]::Create($testFile)
$sw = [diagnostics.stopwatch]::StartNew()
for ($i = 0; $i -lt ($testSize / 1MB); $i++) {
  $file.Write($buffer, 0, $buffer.Length)
}
$file.Close()
$sw.Stop()
$speed = ($testSize / 1MB) / $sw.ElapsedMilliseconds * 1000
Write-Host "Write speed: $speed MB/s"
Remove-Item $testFile
```

If you're seeing below 50 MB/s, you either need an SSD or you have disk errors. HDDs are too slow for Docker's VHDX workload.

---

## FAQ

**Can I move Docker after installation?**

Yes. In Docker Desktop settings (right-click the icon, then Settings, then Resources, then Disk image location), you can change the data path. Docker will re-initialize the VHDX.

**What if I only move the WSL data, not the application?**

That's fine. The `--installation-dir` flag is optional. If you only want to free space, use only `--wsl-default-data-root`.

**Does this work with Docker Compose?**

Yes. Docker Compose uses the same Docker daemon, so it inherits the new paths automatically.

**Can I symlink the directory instead?**

Not recommended. Windows VHDX files don't play well with symlinks. The CLI flags are the proper way.

**How do I migrate containers from C: to D:?**

Export containers with `docker export`, or save images with `docker save`. Remove the old C: installation and reinstall to D:. When you restart, pull your images again.

**Is this reversible?**

Yes. Reinstall Docker to C: using the GUI installer or change the path back in Docker Desktop settings.

---

## Why This Matters

Docker pulls large base images (Alpine 370MB, Ubuntu 1.3GB, Node 1.5GB, etc.) and layers them up. A real project easily accumulates 10-20GB. Add multiple projects, build caches, and temporary containers, and your C: drive fills in weeks.

When your C: drive fills, Windows degrades. It can't write swap files, temp logs, or system caches. The whole machine gets slower. Moving Docker to a secondary drive is one of the highest-impact optimizations you can do as a developer because it's both simple and immediately tangible.

{{< newsletter >}}


---

## Build a URL redirector application with Cloudflare Workers and Cloudflare D1

**Date:** 2024-01-07
**URL:** http://localhost:1313/build-a-url-redirector-application-with-cloudflare-workers-and-cloudflare-d1/
**Description:** Discover the power of Cloudflare Workers and D1 database as we guide you through building a robust URL redirector application on LorbiC.com. Shorten and customize your links effortlessly, optimizing user experience and website performance. Learn step-by-step with prerequisites, database setup, and code implementation. Unlock the potential of Cloudflare's edge computing and revolutionize URL handling for seamless redirections. Elevate your web application's performance with this comprehensive guide at Lorbic.com.

![Lorbic.com - Cloudflare Workers and D1 database used to build a simple url redirector app.](/images/uploads/lorbic.com-cloudflare-workers-d1-featured-image-2x.jpg "Clopudflare Workers and D1 database - lorbic.com")

In today's fast-paced digital world, where attention spans are short, optimizing website performance and user experience is crucial. Imagine asking someone to visit a lengthy URL (<https://lorbic.com/setting-up-llama-2-with-docker-on-your-gaming-laptop-using-ollama/>) – their reaction might be a disappointed face (🙁). Typing such URLs can lead to errors, resulting in users landing on a dreaded 404 page.

To address this, we'll build a simple and powerful solution – a URL redirector application using Cloudflare Workers and D1 database. This application will not only shorten your links but also provide a seamless and error-free redirection feature.\
\
For the sake of simplicity, we'll keep the set of features to a minimum and add whatever necessary at some later time.

### Main features:

* Create a short code for a URL.
* Redirect users to the original link.

### Understanding Cloudflare Workers:

Cloudflare Workers allow developers to run code at the edge of the Cloudflare network, enabling the execution of custom logic for HTTP requests. This allows for enhanced performance and reduced latency, as the code runs closer to the end-users. 

That's what Workers are, basically intercepting requests before they reach your origin server. This allows for lightning-fast redirects based on URL paths, query parameters, headers, and even cookies. You can put workers between **USER** and **ORIGIN SERVER.** Then handle the request accordingly.

#### **Here's how Workers handle redirects:**

* **Interception:** The Worker intercepts incoming requests based on your chosen route configuration.
* **Rule Matching:** The Worker analyzes the request against your predefined rules, stored in JavaScript code, or you can connect Cloudflare KV, Cloudflare D1 or other database via Cloudflare Hyperdrive and get the configuration.
* **Redirection Magic:** If a rule matches, the Worker constructs a redirect response, specifying the new target URL and status code (e.g., 301 for permanent move, 302 for temporary).
* **Speedy Delivery:** The redirect response is instantly delivered back to the user, bypassing the origin server for optimal performance.

- - -

### Prerequisites:

Before diving into the implementation, ensure you have the following:

* Node JS
* A Cloudflare account.
* A domain configured with Cloudflare.
* Cloudflare Wrangler CLI (check [here](https://developers.cloudflare.com/workers/wrangler/install-and-update/))
* Familiarity with JavaScript, Node.js, and ExpressJS (we'll use HonoJS, similar to ExpressJS).
* Install Hono js.

### Create Cloudflare D1 Database

Login to Cloudflare Wrangler CLI by running the following command and allow Cloudflare permissions.

```javascript
wrangler login
```

##### Create a D1 database

Run the command below to create a database. Change <my-db-name> with your actual database name.

```shell
wrangler d1 create <my-db-name>
```

This returns following response, with details of your database. \
Paste these into your `wrangler.toml`

```toml
[[d1_databases]]
binding = "DB" # i.e. available in your Worker on env.DB
database_name = "<my-db-name>"
database_id = "5283redacted23-4a04-456b-af3f-redacted"
```

##### Create database schema

We will keep it very simple with just 1 table and 3 columns in the table. Here is the command to create the database table. **Note:** you'll have to paste the database configuration in **wrangler.toml** and have to be inside the project directory. 

```shell
wrangler d1 execute DB  \
  --command "CREATE TABLE IF NOT EXISTS urls (short_code TEXT PRIMARY KEY, created_at INTEGER, url TEXT);"
```

**Columns:**

* short_code: A string column to store the short code associated with the URL.
* url: A string column to store the full URL.
* created_at: An integer column to store the Unix timestamp representing when the URL was created.

Check your database by running the following command:

```shell
wrangler d1 execute DB \
 --command "select * from urls"
```

### Create Cloudflare Worker

Go to [this page](https://developers.cloudflare.com/workers/get-started/guide/) and create the Cloudflare worker. Or simple install Wrangler CLI and run following command to initialize a project. 

Change **`<project-name>`** with actual name. (I've chosen **lrbc).**

```shell
npx wrangler init <project-name>
```

Select, "hello world" example in the list and chose other options accordingly.

Now go to the **lrbc** directory (your project directory).

```shell
cd lrbc
```

Now open the `wrangler.toml` file in the root of your directory. Add the following code to connect your Cloudflare D1 database. If you do not have it set-up check the **Create Cloudflare D1 Database** of this blog.

Replace the name of your database and database id with actual values. The binding is how you will refer to this configuration in your worker code.

```toml
[[d1_databases]]
binding = "DB"
database_name = "<YOUR_DATABASE_NAME>"
database_id = "<YOUR_DATABASE_ID>"
```

##### Install Honojs:

Install hono js by running following code `npm install --save hono`.

### The Code:

Next, open the `src/index.js` file and add the following code. It uses Hono JS. Hono is a tiny and super fast framework for creating web application and it is very easy to integrate with Cloudflare Workers. It's feels native.

```javascript
import { Hono } from 'hono';

const app = new Hono();

app.get('/:code', async (c) => {
    try {
        const code = c.req.param('code');
        console.log(code);
        const { results } = await c.env.DB.prepare('SELECT url FROM urls WHERE short_code = ?').bind(code).all();

        if (results.length > 0) {
            const websiteUrl = results[0].url;
            console.log('Redirecting to ' + websiteUrl);
            return c.redirect(websiteUrl);
        } else {
            console.log('Short code not found');
            return c.json({ status: 404,  error: 'Short code not found' });
        }
    } 
    catch (e) {
        console.log('Error fetching URL:', e);
        return c.json({ status: 500, error: 'Failed to fetch URL' });
    }
});


export default app;
```

The provided code is written in JavaScript and uses the Hono library. Let's break down its key components:

1. ##### **Route Handling:**

   The code defines a route using the 'app.get' method, which responds to incoming HTTP GET requests with a specific parameter in the URL. 

   ```javascript
   app.get('/:code', async (c) => {
     // Code logic goes here
   });
   ```
2. ##### **Database Query:**

   Inside the route handler, the code attempts to fetch the full URL associated with the provided short code from a database. Notice, the **DB** in the code **c.env.DB** . This is the binding we've defined in **wrangler.toml** file. Here **c** is the context of Hono. Everything related request and response is stored in this object. 

   ```javascript
   const code = c.req.param('code');
   const { results } = await c.env.DB.prepare('SELECT url FROM urls WHERE short_code = ?').bind(code).all();
   ```
3. ##### URL Redirection or Error Handling:

   Based on the query results, the code decides whether to redirect the user to the corresponding website URL or return a JSON response with an error message.

   ```javascript
   if (results.length > 0) {
       // Redirect to the full website URL
       return c.redirect(websiteUrl);
   } else {
       // Short code not found, return a JSON response
       return c.json({ status: 404, error: 'Short code not found' });
   }
   ```
4. ##### Error Handling:

   The code includes basic error handling, logging any encountered errors and returning an appropriate JSON response. You can check these logs in the Workers dashboard. And by running `wrangler tail`

   ```javascript
   } catch (e) {
       console.log('Error fetching URL:', e);
       return c.json({ status: 500, error: 'Failed to fetch URL' });
   }
   ```

### Deploy your worker:

You can deploy your worker by running just one command `wrangler deploy`. It will create a ***.workers.dev*** URL.

```shell
 wrangler deploy
```

### Create Short URLs:

You can either create the URLs by running a following code and directly inserting them into your D1 database.

```javascript
app.post('/store', async (c) => {
	try {
		// get the long url from json payload in request body
		const { url } = await c.req.json();

		// Generate a random id of length = NANOID_SIZE
		const shortCode = nanoid();
		const created_at = Date.now();

		console.log(shortCode);
    // Validation
    if (!shortCode || !url) {
      return c.json({status: 400,  error: 'Missing short code or URL' });
		}

		// Save the url and
		const { results } = await c.env.DB.prepare("INSERT INTO urls (short_code, url, created_at) VALUES (?, ?, ?)").bind(shortCode, url, created_at).run();

		return c.json({ message: `URL stored successfully. short: ${shortCode}, url: ${url}` });
  } catch (e) {
    console.error('Error storing URL:', e);
    return c.json({ status: 500, error: 'Failed to store URL. There is some error or the code already exists.' });
  }
});
```

### What's Next

* Create a GitHub repository and store this project there.
* Extend the functionality by adding the short URL creation feature.
* Integrate this worker (project) into your existing website and create a share functionality, so when someone clicks the share button they get a short URL.
* Extend the database schema and add a hash of URL, so creating existing URLs will return the short existing code instead of creating new code, reducing the number of duplicate URLs.

If you have any question, write it down in the comments below. Your response really matters 😅.

Share it with your friends. Have a nice Sunday. Cheers ✌

---

## Setting Up Llama 2 with Docker on Your Gaming Laptop Using Ollama

**Date:** 2023-12-02
**URL:** http://localhost:1313/setting-up-llama-2-with-docker-on-your-gaming-laptop-using-ollama/
**Description:** Effortlessly set up Llama 2 on your gaming laptop using Docker and Ollama, with a user-friendly interface and local data storage for privacy.

This article is simple guide on setting up Llama 2, a powerful large language model, on your gaming laptop using Docker and the Ollama platform. Ollama makes it very easy to run many LLMs locally. In this detailed article, I'll cover each step, such as Ollama installation and ease of interaction through a Chat GPT-like UI. Before we go into the technical details, let's understand what a large language model is.

![Llama working on a laptop. ](/images/uploads/llama-computer.webp "Llama working on a laptop")

## What is a Large Language Model?

Large language models, like Llama 2, Code Llama are advanced artificial intelligence algorithms designed to understand and generate human-like language. These models are trained on massive datasets (like thousands of books), learning the fundamentals of language structure, context, and syntax. Llama 2, in particular, is a versatile language model capable of various natural language processing tasks, making it a valuable tool for developers, researchers, and enthusiasts. And it is open source. So, we can use it for free.

## Prerequisites:

Before diving into the installation process, let's ensure your system meets the necessary prerequisites:

* **Minimum RAM:** 16 GB (8 GB is fine but may result in slower performance)
* **Docker:** Install the latest version of Docker on your machine. If you don't have it, visit Docker's official website for detailed installation instructions.
* **For Windows Users:** Install the latest version of Windows Subsystem for Linux (WSL 2). (I couldn't get the GPU to work, I'll learn more and keep this blog updated).
* **For Linux Users:** Ensure you have an Nvidia GPU. Note that you may need the Nvidia Container Toolkit<sup>\[3]</sup> for GPU support.

## Installing Docker:

If you haven't installed Docker yet, follow these steps:

Download and Install Docker:

1. Visit [Docker's official website](https://www.docker.com/get-started).
2. Choose the right version for your operating system.
3. Follow the installation instructions provided.

## Setting Up Ollama with Docker:

Now, let's set up the Ollama Docker container for Llama 2:

#### 1. Install the Ollama Docker Container:

```shell
docker run -d --gpus=all -v ${PWD}:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

* This command downloads the Ollama Docker image and creates a container named "ollama", exposing Ollama's APIs on port 11434.
* For Linux users, the option --gpus=all enables GPU support.

#### 2. Run the Llama 2 Model:

```shell
docker exec -it ollama ollama-run Llama2
```

* Execute this command to run Llama 2 inside the Ollama Docker container.

## Installing Chat GPT-like UI:

Enhance your user experience by installing the Ollama-webui container:

```shell
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main
```

* This command creates a container named "ollama-webui" and exports it to localhost:3000.

## Accessing Your Chat GPT:

With everything set up, access your **Chat GPT** from your browser at http://localhost:3000. Enjoy the chatting with your AI in a user-friendly interface with all your data stored locally for maximum privacy.

![Llama 2 running locally on Ollama](/images/uploads/ollama.jpg "Llama 2 running locally on Ollama | Lorbic.com")

## Additional Tips and Considerations:

Disclaimer: Keep in mind that the response time of Llama 2 may vary depending on your machine's capabilities.

You can also install other models. Find a model from [Ollama Library](https://ollama.ai/library) and run it inside the a container just like we did with Llama 2.

## Reference Links:

* [Stackoverflow - Mount a drive](https://stackoverflow.com/a/41489151/11175853)
* [Docker Official Website](https://docs.docker.com/engine/install/)
* [Ollama Official Website](https://ollama.ai/)
* [](https://ollama.ai/)[Ollama Models Library](https://ollama.ai/library)
* [Nvidia Container Toolkit Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation)
* [Ollama Webui GitHub Repository](https://github.com/ollama-webui/ollama-webui)

This complete-ish guide provides a step-by-step approach to setting up and running Llama 2 (and other LLMs) on your gaming laptop using Docker and the Ollama platform. If you encounter any issues or have suggestions, feel free to leave a comment. \
\
Have an awesome day!

---

## AWS - Host Static Website With S3 + Cloudfront + Route53  

**Date:** 2023-05-07
**URL:** http://localhost:1313/aws-host-static-website-with-s3--cloudfront--route53/
**Description:** In this blog learn how to host a static website on AWS with S3, Cloudfront and Route53. Enable HTTP/2, HTTP/3 and reditect insecure http to secure https. Set the bucket to private.

In this blog I'll discuss how to host a static website on AWS with S3, CloudFront and Route 53. We will secure our website with SSL certificate generated from Certificate Manger, enable HTTP/2 and HTTP/3 and redirect insecure http traffic to https version of the website. We will also make our bucket private and add policy to allow access from CloudFront.

![Picture of a Motor Buggy](/images/uploads/museums-victoria-zg0if6dcns0-unsplash_2.webp "Picture of a Motor Buggy")

In a later blog I will also setup GitHub actions to automatically deploy the website when we create a new post. 

### GOALS:

1. Create S3 bucket.
2. Create CloudFront distribution.
3. Point CloudFront to s3 bucket and restrict bucket access to CloudFront only.
4. Add your domain (optional).

***NOTE: Replace \`example.org\` with your actual domain name.***

# Create S3 bucket

Open AWS console and got to S3 dashboard. Then create a new bucket with name as \`example.org\`.\
Go to **Properties** tab and enable **Bucket Versioning.** By enabling bucket versioning you will not lose previous version of your assets (static files).\
Go to **Permissions** tab and enable **Block public access (bucket settings)**, this will prevent access to the bucket from the public. Remember, goal #2 we will use CloudFront to deliver the assets.

# Create CloudFront distribution

Now we will create the CloudFront distribution and point it to our S3 bucket.\
In *CloudFront Dashboard* create a new CloudFront distribution.\
**Origin > Origin Domain,** select your S3 bucket.\
**Origin > Origin Access,** Select **origin access control settings** and select **Create Control Settings** and fill the following details.

> *Name:* leave it default or update\
> *Description (Optional):* add a description\
> *Signing Behavior:* Sign requests\
> *Origin Type:* S3

Then click create and select this newly created settings from the **origin access control** drop-down.\
(We will need to update our bucket's policy to allow CloudFront access to the bucket. We will do this later.)

In the **Web Application Firewall (WAF),** select `Do not enable security protections`\
At the very bottom, in the *Settings section* set **Default root object** to `index.html`. If we do not this we will see a sweet *access denied error.*  Because we are allowing access to all the files inside our bucket **YOUR_BUCKET_NAME/** but not the bucket itself. So when we visit CloudFront distribution URL (e.g. `https://dfi9s7i5typ2c.cloudfront.net/`) CloudFront will not know what does `/` means. And when we set the default root object, CloudFront will know `/` means `/index.html` and it will load the index.html file.

Leave everything else as default, scroll down to bottom and click **Create Distribution.**

**Note: It will take 5-10 minutes to fully deploy the CloudFront distribution.**

# Configure bucket access

Open S3 Dashboard, go to **Permissions** tab. Select **Edit** on the *bucket policy* section and paste the following policy in the text block. This will allow CloudFront to access the objects while keeping the bucket private.

Replace the **YOUR_BUCKET_NAME**, **YOUR_ACCOUNT_ID** and **YOUR_CLOUDFRONT_DISTRIBUTION_ID** placeholders with correct values and save the policy.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontServicePrincipalReadOnly",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudfront::YOUR_ACCOUNT_ID:distribution/YOUR_CLOUDFRONT_DISTRIBUTION_ID"
        }
      }
    }
  ]
}
```

That's it, your website is live at your CloudFront distribution URL. You can find the URL on the home page of your distribution.\
If you are seeing some DNS error, then wait for some time.\
**Note: It will take 5-10 minutes to fully deploy the CloudFront distribution.**

Now, we will add a custom domain in our distribution.

# Custom Domain (Optional)

## Add Domain to Route 53

To complete this step you will need an existing domain or need to purchase a new one.\
Go to Route 53 dashboard and create a hosted zone (it will incur charges) and add your custom domain. Then go to your domain registrar and update the nameserver records, this will make Route 53 your DNS manager.\
It will take up to 24 hours to fully propagate the DNS. You can check the status on <https://dnschecker.org/>.

## Create SSL certificate

Open **Certificate Manager**, request a new certificate. Add your domain name and update your DNS records to validate the ownership. 

Keep in mind to add both the root domain (example.org) and the wildcard domain (*.example.org).

## Add the domain to CloudFront

To add the domain, go to CloudFront Dashboard and select **Edit** under **General > Settings.** In the *Settings* section, update `Alternate domain name (CNAME)` and add your domain name. In `Custom SSL certificate` select your newly created SSL certificate. \
To point this domain to CloudFront. Open to your domain settings (*in R*oute *53*), click on **Create Record** and create an `A` record. Toggle the **Alias** button and in the *Route Traffic To* select your CloudFront distribution.\
In the same section enable HTTP/2 and HTTP3.

After some time (5-10 minutes) your website will be available at your domain address.\
\
Thanks for reading.\
If you have any questions, leave a comment below or drop an email at vikash\[at]lorbic\[dot]com.\
\
Have a nice day.

---

## Simplifying Search Activation: A Quick Guide with JavaScript

**Date:** 2023-04-01
**URL:** http://localhost:1313/simplifying-search-activation-a-quick-guide-with-javascript/
**Description:** Learn how to enhance user experience by activating the search input field with a simple key press on your website. This quick guide provides a code snippet and easy-to-follow steps.

Learn how to enhance user experience by activating the search input field with a simple key press on your website. This quick guide provides a code snippet and easy-to-follow steps.

### 1. HTML Markup for Search Input:

```html
<form>
  <input type="text" id="search-input" placeholder="$ grep...">
</form>

<!-- If you have a button to activate the search field like in this website -->
<li class="search-icon">
  <a href="javascript:void(0)" id="search-button">
    <i class="fa fa-search"></i>
  </a>
</li>
```

### 2. JavaScript Functionality:

Activate the search input field by pressing a key (in this example, the '/' key). Modify the key binding according to your preferences.

```javascript
// app.js (app.min.js)
function handleSearchActivation(event, searchInput, searchButton) {
    let KEY_TO_ACTIVATE_SEARCH = '/';
    if (event.key === KEY_TO_ACTIVATE_SEARCH) {
        searchInput.focus();
        searchInput.value = "";
        searchButton.click();
        console.log("Pressed '/' to activate search.");
    }
}

window.onload = function () {
    console.log("Document loaded...");
    let searchInput = document.querySelector("#search-input");
    let searchButton = document.querySelector("#search-button");

    document.addEventListener("keyup", (event) => {
        handleSearchActivation(event, searchInput, searchButton);
    });
};
```

### 3. Link the Script File in Your HTML:

Include the JavaScript file in the head or footer of your HTML. Alternatively, copy and paste the code into an existing JS file to reduce file requests.

```html
<head>
  ...
  <script defer src="/js/app.min.js" type="text/javascript"></script>
  ...
</head>
```

### 4. Customize UI:

You can also tailor the UI of the search input to match your preferences.\
\
Now, users can effortlessly activate the search input by pressing a key and start typing. You can also add a submit button and press 'Enter' to initiate the search.\
\
Thanks for Simplifying Your User's Experience!

---

## Setting Up a PostgreSQL Container: A Quick Guide

**Date:** 2023-04-01
**URL:** http://localhost:1313/setting-up-a-postgresql-container-a-quick-guide/
**Description:** Learn the simple and fast process of creating a PostgreSQL container using Docker. This guide provides both the command and a detailed explanation of the container and PostgreSQL database configuration.

Learn the simple and fast process of creating a PostgreSQL container using Docker. This guide provides both the command and a detailed explanation of the container and PostgreSQL database configuration.

## 1. Create the PostgreSQL Container:

```shell
docker run -d --name postgres \
  -e POSTGRES_PASSWORD=RunningFlying2 \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v postgres_data:/var/lib/postgresql/data \
  -p 5432:5432 \
  postgres:latest
  
```

## **2. Explanation:**

The above command initiates the creation of a container with the latest version of PostgreSQL. Here's a breakdown of the parameters used:

`-d: Runs the container in detached mode.`

`--name postgres: Assigns the name "postgres" to the container.`

`-e POSTGRES_PASSWORD=RunningFlying2: Sets the PostgreSQL user password to "RunningFlying2".`

`-e PGDATA=/var/lib/postgresql/data/pgdata: Specifies the PostgreSQL data directory.`

`-v postgres_data:/var/lib/postgresql/data: Creates a Docker volume named "postgres_data" for persistent data storage.`

`-p 5432:5432: Maps the container's 5432 port to the host machine's 5432 port for external access.`

`postgres:latest: Pulls and uses the latest version of the PostgreSQL image.`

## 3. Technical Details:

```
Container Name: postgres
Docker Volume: postgres_data
Host: 127.0.0.1
Port: 5432
User: postgres
Password: RunningFlying2
```

You have successfully set up a PostgreSQL container, ready to handle your database needs. The provided technical details ensure you can seamlessly connect to this container from your development or production environment.

**Explore More:**

Feel free to explore additional PostgreSQL configurations and features to tailor the container to your specific requirements.

> **Note:** For enhanced security, consider changing default passwords in production environments.

Have an amazing day.

---

## Learn Dart Programming

**Date:** 2021-05-12
**URL:** http://localhost:1313/learn-dart-programming/
**Description:** Learn basics of Dart programming language with code examples and explainations.


Good Evening,
Here are the Hands-on notes for Dart Programming Language.
I created these notes when I was learning Dart Language. So, these notes might help you too.
You can use these notes as a quick reference to Syntax and Demo.
I've covered almost every fundamental concept of Dart in this article.

![](https://images.unsplash.com/photo-1576504164753-8fd338cd450b?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=727&q=80)

Photo by Birmingham Museums Trust
  ---

# 1. Basics

## 1.1 The Main Function

Every Dart program starts execution from `main()` functions.

### The syntax of main function

_We use this version when we want to use command line arguments._

```dart
void main(List<String> args){
    // do your stuff here
}
```

Or simple version
_Generally we use this version of main._

```dart
void main(){
    // do your stuff here
}
```

### Demo

```dart
void main() {
  // say hello world
  print("Hello World!");
  // print some numbers, integer expression
  print(1 * 2 + 34 - 12 / 5);
  // print double
  print(2.53);
  // boolean
  print(true);
}
```

  ---

## 1.2 Data Types

Literals in Dart.

```dart
// literals in Dart
"Dan"; // String literal
23; // int literal
45.6; // double literal
true; // bool literal
```

There are 4 built-in data types in Dart.

| SN | Data Type | Description |
| --- | --- | --- |
| 1. | int | Integer value |
| 2. | double | Floating point value |
| 3. | String | String value |
| 4. | bool | `true` , `false` value |

### Syntax creating variables in Dart

  **Use explicit way to declare the variable.**

`typename variable_name;`

Or initialize while creating.
`typename variable_name = initial_value;`

```dart
int intVariable;
int anotherIntVariable = 111;
String name = "Vikas";
double pi = 3.14;
bool isdeveloper = true;
```

  **Use implicit way to declare the variable.**

Use `var` to create any type of variable.
`var variable_name;`

Or initialize while creating.
`var variable_name = intial_value`;\`

### Demo

  **Note: _In Dart all the data types are objects and their initial value is `null`._**

Create the variables

```dart
 // numbers: int
  int score = 90;
  // store hexadecimal value
  int hexValue = 0xEAEAEA;

  // numbers: double
  double length = 24.6;
  double exponentValue = 1.4e5;

  // String
  String name = "Dave";

  // boolean
  bool going = false;

  // use var
  var name2 = "Vikas"; // String
  var age = 20;        // int
  var score2 = 45.6;   // double
  var isHere = true;   // bool
  var exp = 2.5e6;     // exponential (double)
  var hex2 = 0xEC43AB; // hex (int)
  var data;            // will have null
```

Print the values

```dart
print(score);        //int
print(hexValue);     //int
print(length);       //double
print(exponentValue);//double
print(going);        //bool
print(name);         //String
print(name2);        //String
print(age);          //int
print(score2);       //int
print(isHere);       //bool
print(exp);          //double
print(hex2);         //int
print(data);         //null value
```

  ---

## 1.3 String and String Interpolation

Ways to define a String variable

```dart
// ways to define string
  String s1 = 'Vikas'; // with single quote
  String s2 = "Dave"; // with double quote
  String s3 = "It's Easy"; // can use both
  String s4 = 'It\'s Easy'; // excape sequence
  String s5 = 'This is a very very long string '
      'and exceeding the line.'; // long string
  // auto concatenation of strings
```

String Interpolation

```dart
// Normal Concatenation
String name = "Vikas";
String msg = name + ", We are heading to the west";

// Interpolation
String msg2 = "$name We are heading to the west";
String msglen = "Length of msg = ${msg2.length}";

// Print
print(msg2);
print(msglen);
```

We can interpolate any data type with String in `print(...)` using `$varname`.

```dart
int BoxLength = 123;
double BoxWeight = 4522.4545;
print("Length of box = $BoxLength \nWeight of Box = $BoxWeight");
```

  ---

## 1.4 Constant and Final Variables

In dart we can create constants using
-> final
-> const

1. **final vs const:**

- `final` variable can only be set once and it is initialized when accessed
- `const` is internally final in nature but it is a `compile-time constant`
      -> i.e. it is initialized during compilation

2. Instance variable can be final but cannot be constant
3. If you want a Constant class level then make it
   `static const` .

### Demo

```dart
void main() {
  // final
  final city = "Toronto";
  // or
  // final String city = "Toronto";
  // city = "NY"; // Error: Can't assign to the final variable 'city'.

  const PI = 3.1415;
  const double gravity = 9.8;
}
```

Class level constant

```dart
class Circle {
  final color = "RED"; // OK
  // const PI = 3.1415; // Not OK
  static const PI = 3.1415; // OK
}
```

  **Only static fields can be declared as `const`.**
**Try declaring the field as `final`, or adding the keyword `static`.**


  ---

# 2. Control Statements

These statements are use to control the execution flow of the program and add various logics to the program.

## 2.1 If - Else Statements

Simple condition checking.
**Syntax**

#### IF alone

```dart
if(condition){
    // block 1
    // execute the code
}
```

When the condition is `true` the code inside `block 1` gets executed.
Otherwise nothing happens to the code inside `block 1`.

#### IF and ELSE together

```dart
if(condition){
    // block 1
    // execute the code
} else{
    // block 2
    // execute the code
}
```

#### IF-ELSE ladder

```dart
if(condition){
    // block 1
    // execute the code
} else if(condition2){
    // block 2
    // execute the code
} else if(condition3){
    // block 3
    // execute the code
}
.
.
.
else if(condition n){
    // block n
    // execute the code
} else{
    // block last
    // execute the code
}
```

When the condition is `true` the code inside the respective `block` gets executed. Otherwise `block last` is executed.

### Demo

```dart
void main() {
  var salary = 250000;
  if (salary > 200000) {
    print("You, can get the Credit Card");
  } else if (salary > 150000) {
    print("You can apply for the Credit Card");
  } else {
    print("You cannot get the Credit Card");
  }
}
```

Check code inside `01 basics/05if_else.dart`.


  ---

## 2.2 Conditional Expression

Conditional expression is a simple alternative to `if-else`.
There are two conditional expressions

1. **`condition ? expr1 : expr2;`**
   returns `expr1` if condition is true else returns `expr2`

```dart
  int a = 12, b = 45;
   print("Small Number = ${(a < b) ? a : b}");
   int c = (a < b) ? a : b;
   print("Small Number = $c);
```

2. **`expr1 ?? expr2`**
   if `expr1` is not-null, returns its value; otherwise, evaluates and returns the value of `expr2`

```dart
  int x, y = 345; // x = null;
   print("Value = ${x ?? y}");
   int z = x ?? y;
   print(z);
```

  ---

## 2.3 Switch Case

If we have constant conditions like ids as `1, 2, 3, 4, 5`
Instead using many `if-else` statements we can use `switch-case` statement.

The case can be a constant `int` or a constant `String`.

#### Syntax

```dart
switch(variable){
    case 1:
    // code
    break; // this is must

    case 2:
    // another code
    break;

    default:
    // will execute if no case matches
}
```

`break` is must to stop execution after every `case`.

### Demo

With integer cases.

```dart
// Integer (int)
var option = 2;
switch (option) {
case 1:
    print("Option 1");
    break;
case 2:
    print("Option 2");
    break;
default:
    print("Invalid option");
}
```

With String cases.

```dart
String name = "Andy";
switch (name) {
case "Andy":
    print("Hello, Andy");
    break;
case "Dave":
    print("Hello, Dave");
    break;
default:
    print("No name");
}
```

  ---

# 3. Looping

We can repeat the statements using looping.
There are 3 loops in Dart :

- For
- While
- Do ...While

## 3.1 For loop

1. The loop first initializes the `iteration_var`
2. then checks `breaking_condition`
3. then executes the `Statements` and
4. increments or decrements the `iteration_var` and
5. 2,3,4, are repeated until `breaking_condition` remains `true`.

### Syntax

```dart
for(iteration_var; breaking_condition;  increment/decrement)
{
    // Statements
}
```

### Demo

```dart
// print even nums
for (var i = 1; i <= 10; i++) {
    if (i % 2 == 0){
        print(i);
    }
}
```

### 3.1.1 For ... in Loop

#### Syntax

`for(var varname in IterableList){....}`

#### Demo

```dart
// for ..in loop
List names = ["Dave", "Andy", "Vikas", "Mark"];
for (var name in names) {
    print(name);
}
```

This loop is very useful when using List and other data containers.

### 3.1.2 Variation in for

we can omit the `initial`, `breaking` and `incr/decr` statements.

```dart
for (; x < 20; x++) {
    print(x);
}

for(;;x++){
    print(x);
    if(x==10){
        break;
    }
}

for(var x = 0;;){
    x++;
    print(x);
    if(x==10){
        break;
    }
}
```

  ---

## 3.2 While Loop

While loop is an `entry controlled` loop.
The code inside `{...}` of while loop will execute only when the condition is `true`.

### Syntax

`while(condition){...}`

Here we declare a condition or breaking condition for the loop.
See the demo

### Demo

```dart
// while loop
// print 0...9
var k = 0; // this is must
while (k < 10) {
    print(k);
    k++;
}
```

  ---

## 3.3 Do ...While Loop

Do ...while Loop is an `exit controlled` loop.
Here the `Statement` will run for the first time regardless the `condition` is `true` or `false`.
Then it will check the `condition` if it is `true` the the loop will continue execution otherwise it will jump out of the loop.

### Syntax

`do{...} while(conditon);`
Here the semicolon(`;`) is must.

### Demo

```dart
// print 0...9
// do ..while loop
int x = 0;
do {
    print(x);
    x++;
} while (x < 10);
```

### While vs Do...While

Both work in similar manner.
But, `do...while` will execute at least once and `while` won't.


  ---

## 3.4 Break and Continue

These are two loop control statements.
Which are used to _forcefully control_ the flow of loop execution.

### Break

`break` is used to stop the execution, and jump out of the loop.

#### Demo

```dart
// break after printing 5
for(int i = 0; i < 10; i++){
    print(i);
    if(i==5){
        break;
    }
}

// run and observe
// break -> stop execution of loop
for (int i = 0; i < 5; i++) {
    for (int j = 0; j < 5; j++) {
        print("$i $j");
        if (i == 2 && j == 2) {
            break; // will break inner loop
        }
    }
}
```

  **We have already used `break` in `switch...case` program above.**

### Continue

`continue` is used to skip current iteration.

#### Demo

```dart
// continue -> skip an iteraion in a loop
for (var i = 0; i < 10; i++) {
    if (i == 5) {
        // skip when i==5
        continue;
    }
    print(i);
}
```

  ---

## 3.5 Label

Label is a new concept in Dart (we have seen it in `C` language with its buddy `goto`).
This is not a magical spell from _Hogwarts_ that will do amazing magical things.
It is just a way to make loop more controlled than before.
We can name a block of code using label.
Take a look at below code snippet.

### Demo

```dart
print("i j");

outerLoop:
for (int i = 0; i < 5; i++) {
  // ignore: unused_label
  innerloop:
  for (int j = 0; j < 5; j++) {
    print("$i $j");
    if (i == 2 && j == 2) {
      break outerLoop; // will break outerloop
    }
  }
}
```

Try doing this and observe the result of `i j`.

```dart
print("i j");

outerLoop:
for (int i = 0; i < 5; i++) {
  // ignore: unused_label
  innerloop:
  for (int j = 0; j < 5; j++) {
    print("$i $j");
    if (i == 2 && j == 2) {
      break innerLoop; // will break innerloop
    }
  }
}
```

  ---

# 4. Functions

Functions are used to group the code statements that do something.
e.g. You want to calculate the area of a rectangle then the formula will be `area = length * width`.
You can do it where you want to use the `area`. _But, if you want to double the area and add 20 to the result then what ?  _
_Will you write all the code again and again or find some convenient solution._
So, the solution is using functions.
Your code will be

```dart
var area = length * width;
area = area * 2;
area = area + 20;
```

Now you are able to use the final value of `area`.
So, let's declare a simple function that says `Hello World!` to the user (after learning the simple function we will continue with our `area` calculation)

```dart
void sayHello(){
  print("Hello World!");
  // That's common saying hello to the world
  // let's say 'Hello Dart, Welcome to Earth'
  print("Hello Dart, Welcome to Earth");
}
```

## 4.1 Breakdown of function (Syntax)

  **`void`** : It is the return type as you know Dart is a statically typed language like Java, C++ and C. This tells the compiler that we will return a value of the given type `(int, double, String, bool, other user defined types)`.
Here `void` tells that the function will not return any value.
  **`sayHello`** : It is the name of the function. It can be anything but, should be relevant to the work of the function like: `multiplyBy2(), makeHalf(), doubleThevalue(), reduceThree(), calculateSI() etc.`
  **`()`** : It is called parameter. we can pass values that we want to work on. Like a `name` that should be printed instead of `Dart` or `World` in the above example.
Let's assume we want to say Hello to you and we don't know your name then we will pass a `String` called `name` and interpolate it with the word `Hello` and print it.
Look at the code below..

```dart
void sayHello(String name){
  print("Hello, $name");

}

void main(){
  sayHello("Vikas"); // Hello, Vikas
  sayHello("Dart"); // Hello, Dart
  sayHello("World"); // Hello, World
}
```

Copy the above code and run it.

You can also return the string.

```dart
String sayHello(String name){
  return "Hello, $name";
  // don't forget to change the return type
}

void main(){
  print(sayHello("Vikas")); // Hello, Vikas
  print(sayHello("Dart")); // Hello, Dart
  print(sayHello("World")); // Hello, World
}
```

Now, let's work on our `area` example:

```dart
int smartAreaOfRectange(int length, int width) {
  int area = length * width;
  area = area * 2;
  area = area + 20;
  return area;
}
```

Put this code in your file and call it in the main function as we've done in `sayHello(...)` function.

Another Example:

```dart
int doubleIt(int x) {
  return x * 2 + 17;
}

void main() {
  for (int i = 1; i <= 10; i++) {
    print(doubleIt(i));
  }
}
```

  ---

## 4.2 Function Expression (Fat Arrow `=>`)

The function expression or _Fat Arrow `=>`_ is used to make the function definition shorter and simpler.
**It can only be used when there is only one statement in the function.**
Like , double the value

```dart
int doubleIt(int x) => x*2;
```

we don't use the `return` keyword here.
So ,

```dart
int doubleIt(int x) => return x*2;
```

will be wrong.
  ---

## 4.3 Types of function parameters

PARAMETERS:

- Required
- Optional:
    - Positional
    - Named
    - Default:
        - Positional
        - Named

### 4.3.1 Required Parameters

These are the required parameters that cannot be omitted.

```dart
// required parameters
void requiredParameters(String c1, String c2, String c3) {
  print("Required Parameters");
  print("$c1, $c2, $c3");
}
void main() {
  // required parameters
  requiredParameters("NY", "Toronto", "San Diego");
}
```

### 4.3.2 Optional Parameters

We can omit passing these parameters.
Blank value is considered as `null`.

#### 4.3.2.1 Optional Positional Parameters

We pass the value based on the portion of the parameter in the function.
These parameters are surrounded by brackets `[ ... ]`.

```dart
// optional positional parameters
void optionalPositionalParameters(String c1, String c2, [String c3]) {
  print("Optional Positional Parameters");
  print("$c1, $c2, $c3");
}
void main() {
  // optional positional parameters
  optionalPositionalParameters("NY", "Toronto"); // 2 values
  optionalPositionalParameters("NY", "Toronto", "San Diego"); // 3 values
}
```

#### 4.3.2.2 Optional Named Parameters

we can give the name of parameter while passing the value.
These parameters are surrounded by braces `{ ... }`.

```dart
// optional named parameters
void optionalNamedParameters(String c1, String c2, {String c3}) {
  print("Optional Named Parameters");
  print("$c1, $c2, $c3");
}
void main() {
  // optional named parameters
  optionalNamedParameters("NY", "Toronto");
  optionalNamedParameters("NY", "Toronto", c3: "San Diego");
}
```

#### 4.3.2.3 Default Parameters

These are the `positional and named` parameters with default value.
  **Default Positional**
We pass the default value of the Positional Parameter while defining the function `[String name = "Dart", int id = 1111]`.

```dart
// default positional parameters
void defaultPositionalParameters(String c1, String c2, [String c3 = "Mumbai"]) {
  print("deafault positional Parameters");
  print("$c1, $c2, $c3");
}
void main() {
  // default positional
  defaultPositionalParameters("NY", "Toronto");
  defaultPositionalParameters("NY", "Toronto", "San Diego");
}
```

  **Default Named**
We pass the default value of the Named Parameter while defining the function `{String name = "Dart", int id = 1111}`.

```dart
// default named parameters
void defaultNamedParameters(String c1, String c2, {String c3 = "Mumbai"}) {
  print("default Named Parameters");
  print("$c1, $c2, $c3");
}

void main() {
  // default named
  defaultNamedParameters("NY", "Toronto");
  defaultNamedParameters("NY", "Toronto", c3: "San Diego");
}
```

  ---

# 5. Exception Handling

_What are exceptions ?_
An **Exception** is a runtime error which occurs when the program is running due to some error that the compiler was unable to detect.
If an exception occurs the program crashes.
If not handled may cause serious issues.

Below table showing some exceptions in Dart

<div class="scrollme" markdown="block">

| S.N. | Exception | Description |
| --- | --- | --- |
| 1. | DeferredLoadException | Thrown when a deferred library fails to load. |
| 2. | FormatException | Exception thrown when a string or some other data does not have an expected format and cannot be parsed or processed. |
| 3. | IntegerDivisionByZeroException | Thrown when a number is divided by zero. |
| 4. | IOException | Base class for all Input-Output related exceptions. |
| 5. | IsolateSpawnException | Thrown when an isolate cannot be created. |
| 6. | Timeout | Thrown when a scheduled timeout happens while waiting for an async result. |

</div>

## 5.1 Example of Exception

Simplest example will be dividing a number by _Zero_.

```dart
 void main(){
    // integer division
    int result = 12 ~/ 4; // OK
    int badResult = 15 ~/ 0; // NOT OK

  }
```

So, the line `double badResult = 15 / 0;` raises an `IntegerDivisionByZeroException` exception.

## 5.2 Handling of Exception

We can handle the exceptions by surrounding the exception occurring code inside a `try` block.
And handle the exception using `on`, `catch`, and `finally` blocks.

### 5.2.1 Try Block

This is the place where we write an exception occurring statement.

#### Demo

```dart
try {
    int answer = 12 ~/ 0;
    print("Division = $answer");
}
```

If the statement inside `try` block raises an exception the program will not crash this time instead it will throw the exception out of the `try` block which we'll have to catch.

### 5.2.2 On Block

We can use `on` to catch the exception _if we know the thrown exception._

```dart
// CASE1: Handle with "on"
// use on when we know the exception
try {
  int answer = 12 ~/ 0;
  print("Division = $answer");
} on IntegerDivisionByZeroException {
  print("Cannot divide by zero");
}
```

Here we know the thrown exception `IntegerDivisionByZeroException`.

### 5.2.3 Catch Block

If we don't know the exception we can simply catch it using `catch` block.

```dart
// CASE2: Handle with "catch"
// use catch(e) when we don't know the exception
try {
  int answer = 12 ~/ 0;
  print("Division = $answer");
} catch (e) {
  print("The exception : $e");
}
```

Here the unknown exception is successfully handled.

### 5.2.4 Stack Trace

We can use the Stack Trace to find the events that threw the exception.

```dart
// CASE3: using STACK TRACE to know the events occured
// before exception was thrown
try {
  int answer = 12 ~/ 0;
  print("Division = $answer");
} catch (e, stackTraceObject) {
  print("The exception : $e");
  print("Stack Trace:\n$stackTraceObject");
}
```

### 5.2.5 Finally Block

This Block is always executed regardless the occurance of the exception.
**No exception:**

```dart
// CASE4: finally clause
try {
  int answer = 12 ~/ 4;
  print("Division = $answer");
} catch (e, stackTraceObject) {
  print("The exception : $e");
  print("Stack Trace:\n$stackTraceObject");
} finally {
  print("This will always execute No exception)");
}
```

  **With Exception:**

```dart
// CASE4: finally clause
try {
  int answer = 12 ~/ 0;
  print("Division = $answer");
} catch (e, stackTraceObject) {
  print("The exception : $e");
  print("Stack Trace:\n$stackTraceObject");
} finally {
  print("This will always execute (With exception)");
}
```

## 5.3 Custom Exception

  **Q. Can we have our iwn exceptions ?**
**A. Yes, we can define our own exception class using the following method.**

First create a class with `YourCustomExceptionName` and _implement_\* the built-in `Exception` class.
Then _override_ the `errorMessage()` method to display "Your custom error message".

\*_We will learn about Object Oriented Programming later in the article._

#### Demo

```dart
// custom exception class
class DepositAmountLessThanZeroException implements Exception {
  String errorMessage() {
    return "Ammount cannot be less than 0";
  }
}
```

Let's use the custom exception class.
Create a function which throws the exception.

```dart
void depositMoney(int amount) {
  if (amount < 0) {
    throw new DepositAmountLessThanZeroException();
  } else {
    print("Successful, Deposited $amount");
  }
}
```

Now call the function in the try block.

```dart
void main() {
  // OK
  try {
    depositMoney(1500);
  } catch (e, s) {
    print(e.errorMessage());
    print(s);
  }

  // OK
  try {
    depositMoney(0);
  } catch (e, s) {
    print(e.errorMessage());
    print(s);
  }

  // Exception
  try {
    depositMoney(-1233);
  } catch (e, s) {
    print(e.errorMessage());
    print(s);
  }
}
```

That's all from my side on basic exception handling.
We will discuss the topic in detail later in another blog.

Thanks for reading. 🙏
If you liked it feel free to share with your dear ones.💗
And tell me what do you think in the comment box.💬

Stay tuned.


---

## Types of Pointers in C/C++

**Date:** 2021-05-12
**URL:** http://localhost:1313/types-of-pointers-in-c/c-/
**Description:** 


![Photo of Hyacinth macaw on Unsplash - Village Programmer](https://images.unsplash.com/photo-1509116453669-0baa67ef9073?ixid=MnwxMjA3fDB8MHxzZWFyY2h8Mnx8aHlhY2ludGglMjBtYWNhd3xlbnwwfHwwfHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=500&q=60)  
Photo of _Hyacinth macaw by_ Roi Dimor on [Unsplash](https://unsplash.com/photos/AEQ-166CWY8)

---

# Types of Pointers in C / C++

## 1. Null Pointer

It is a pointer pointing to nothing. **NULL** pointer points to the base address of the segment.

`-EXAMPLE-`

```c
int * ptr = (int) * 0;
float * fptr = (float) * 0;
double * dptr = (double) * 0;
char * chptr = (char) * 0;
```

##### Other ways of initializing NULL pointer

```c
int * ptr = NULL;
char * chptr = '\0';
```

> #### NULL also means 0 in macro
>
> ```c
> #define NULL 0
> ```

---

## 2. Dangling Pointer

A pointer pointing to the memory address of any variable (or object) which has been deleted from memory.  
When a pointer points to a deleted memory address, the pointer is called as a _dangling pointer_.

`-EXAMPLE-`

```cpp
Person p = new Person();
Person * pptr = &p;
delete p;
/* here pointer pptr (* pptr) is still pointing to
the Person object which has been deleted.
*/
```

---

## 3. Generic Pointer

_void_ pointer is known as generic pointer. It can point to **_any type_** of data.

`-EXAMPLE-`

```cpp
void * ptr;
```

#### Points to remember

- We can't de-reference a generic pointer.
- We can find the size of pointer using `sizeof()` operator.
- These pointers can hold any type of pointer such as char, int, float, structure, array, object etc without any typecasting.
- Any type of pointer can hold generic pointer without typecasting.
- Generic pointer can be used to implement dynamic datatype.

---

## 4. Wild Pointer

A pointer which has not been initialized is called as _wild pointer_.

`-EXAMPLE-`

```cpp
int * ptr;
```

---

## 5. Complex Pointers

These are pointers pointing to derived data-types.

- Pointer to array
- Pointer to function
- Pointer to structure / union
- Pointer to object
- Multilevel pointers

---

## 6. near / far / huge Pointers

These pointer modifiers were used in 1980's-90's before 32 bit architecture.  
**_near_** : a 16 bit pointer that can address any byte in a 64k segment.  
**_far_** : a 32 bit pointer that contains a segment and an offset.  
**_huge_** : a 32 bit pointer in which the segment is "normalized" so that no two far pointers point to the same address unless they have the same value.

You can read <a href="https://stackoverflow.com/questions/3575592/what-are-near-far-and-huge-pointers" target="_BLANK">this</a> stack overflow answer for more information.

---

Thanks for reading. 🙏  
If you liked it feel free to share with your dear ones.💗  
And tell me what do you think in the comment box.💬

Stay tuned.

---


---

## Avoid Memory Leaks In C++

**Date:** 2021-05-04
**URL:** http://localhost:1313/avoid-memory-leaks-in-c-/
**Description:** 


**Instructions**

Things You'll Need

- Proficiency in C++
- C++ compiler
- Debugger and other investigative software tools

# Part 1

Understand the operator basics. The C++ operator `new` allocates heap memory. The `delete` operator frees heap memory. For every `new`, you should use a `delete` so that you free the same memory you allocated:

```cpp
char* str = new char [30]; // Allocate 30 bytes to house a string.
delete [] str; // Clear those 30 bytes and make str point nowhere.

```

# Part 2

Reallocate memory only if you've deleted. In the code below, `str` acquires a new address with the second allocation. The first address is lost irretrievably, and so are the 30 bytes that it pointed to. Now they're impossible to free, and you have a memory leak:

```cpp
char* str = new char [30]; // Give str a memory address.
// delete [] str; // Remove the first comment marking in this line to correct.
str = new char [60]; /* Give str another memory address with the first one gone forever.*/
delete [] str; // This deletes the 60 bytes, not just the first 30.

```

# Part 3

Watch those pointer assignments. Every dynamic variable (allocated memory on the heap) needs to be associated with a pointer. When a dynamic variable becomes disassociated from its pointer(s), it becomes impossible to erase. Again, this results in a memory leak:

```cpp
char* str1 = new char [30];
char* str2 = new char [40];
strcpy(str1, "Memory leak");
str2 = str1; // Bad! Now the 40 bytes are impossible to free.
delete [] str2; // This deletes the 30 bytes.
delete [] str1; // Possible access violation. What a disaster!

```

# Part 4

Be careful with local pointers. A pointer you declare in a function is allocated on the stack, but the dynamic variable it points to is allocated on the heap. If you don't delete it, it will persist after the program exits from the function:

```cpp
void Leak(int x){
char* p = new char [x];
// delete [] p; // Remove the first comment marking to correct.
}

```

# Part 5

Pay attention to the square braces after "delete". Use `delete` by itself to free a single object. Use `delete []` with square brackets to free a heap array. Don't do something like this:

```cpp
char* one = new char;
delete [] one; // Wrong
char* many = new char [30];
delete many; // Wrong!

```

---

_**Keep Learining, Keep Practicing. :) :)**_

---

That's All. 😅

Thanks for reading. 🙏  
If you liked it feel free to share with your dear ones.💗  
And tell me what do you think in the comment box.💬

Stay tuned with CS111.


---

## Operators in Bash Shell Programming

**Date:** 2021-05-04
**URL:** http://localhost:1313/operators-in-bash-shell-programming/
**Description:** 


![A Picture of Pluto by NASA featured on Village Programmer](https://images.unsplash.com/photo-1614314107768-6018061b5b72?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1334&q=80)
Photo by [NASA](https://unsplash.com/photos/-5V6VZxSQRo) on Unsplash

---

If you want to learn bash bash scripting please read this article.  
 [Learn complete Linux bash (shell) scripting in one article](/post/learn-complete-linux-bash-scripting-in-one-article/)

Operators help us perform various types of operations such as addition, multiplication etc.  
There are following types of operators present in bash:

- Arithmetic,
- Relational,
- Boolean,
- File Test,
- String Test

## 1. Arithmetic Operators

All the arithmetic operators present in bash are discussed below with examples.  
These operators work with integers.

#### 1.1 Addition Operator

It's a **`+`** that adds operands together.  
Shown in the below example:

```bash
$x=20
$y=30
a=`expr $x + $y`
echo $a
```

This will display:

> 50

##### NOTE:

> 1. The **`expr `** command evaluates the expression.
> 2. There is `<space>` before and after the **`+`** operator .

#### 1.2 Subtraction Operator

It's a **`-`** that subtracts second operand from first operand.  
Shown in the below example:

```bash
$x=40
$y=30
a=`expr $x - $y`
echo $a
```

This will display:

> 10

#### 1.3 Multiplication Operator

It's a **`*`** that multplies operands together.  
Shown in the below example:

```bash
$x=2
$y=3
a=`expr $x * $y`
echo $a
```

This will display:

> 6

#### 1.4 Division Operator

It's a **`/`** that performs division operation.  
Shown in the below example:  
 -Example 1-

```bash
$x=200
$y=50
a=`expr $x / $y`
echo $a
```

This will display:

> 4

-Example 2-

```bash
$x=20
$y=3
a=`expr $x / $y`
echo $a
```

This will display:

> 6

#### 1.5 Modulus Operator

It's a **`%`** that performs modulus operation and returns the remainder.  
Shown in the below example:

```bash
$x=10
$y=3
a=`expr $x % $y`
echo $a
```

This will display:

> 1

#### 1.6 Equality Check Operator

It's a **`==`** that performs equality check operation and returns 1 if both sides are equal otherwise returns 0.  
Shown in the below example:

-Example 1-

```bash
$x=10
$y=10
a=`expr $x == $y`
echo $a
```

This will display:

> 1

-Example 2-

```bash
$x=10
$y=5
a=`expr $x == $y`
echo $a
```

This will display:

> 0

#### 1.7 Not Equal Check Operator

It's a **`!=`** that performs equality check operation and returns 0 if both sides are equal otherwise returns 1.  
Shown in the below example:  
-Example 1-

```bash
$x=10
$y=10
a=`expr $x == $y`
echo $a
```

This will display:

> 0

-Example 2-

```bash
$x=10
$y=5
a=`expr $x == $y`
echo $a
```

This will display:

> 1

## 2. Relational Operators

Relational operators are used to perform comparison among given operands.

### -eq (Equals to operator)

This checks whether the operands are equal.  
This operator is used to perform comparison in the `if` condition.

```bash
a=10
b=10
if [ $a -eq $b ]
	then
	echo "Equal"
else
	echo "Not Equal"
fi
```

> Equal

### -ne (Not equals to operator)

This checks whether the operands are equal.  
This operator is used to perform comparison in the `if` condition.

```bash
a=10
b=20
if [ $a -ne $b ]
	then
	echo "Not Equal"
else
	echo "Equal"
fi
```

> Not Equal

### -gt (Greater than operator)

This checks whether the first operand is greater than the second operand.  
This operator is used to perform comparison in the `if` condition.

```bash
a=40
b=20
if [ $a -gt $b ]
	then
	echo "$a is greater than $b"
else
	echo "$a is not greater than $b"
fi
```

> 40 is greater than 20

### -lt (Less than operator)

This checks whether the first operand is less than the second operand.  
This operator is used to perform comparison in the `if` condition.

```bash
a=20
b=40
if [ $a -lt $b ]
	then
	echo "$a is less than $b"
else
	echo "$a is not less than $b"
fi
```

> 20 is greater than 40

### -ge (Greater than or equals to operator)

This checks whether the first operand is greater than or equals to the second operand.  
This operator is used to perform comparison in the `if` condition.

```bash
a=40
b=20
if [ $a -ge $b ]
	then
	echo "$a is greater than or equals to $b"
else
	echo "$a is not greater than $b"
fi
```

> 40 is greater than or equals to 20

### -le (Less than or equals to operator)

This checks whether the first operand is greater than or equals to the second operand.  
This operator is used to perform comparison in the `if` condition.

```bash
a=40
b=200
if [ $a -le $b ]
	then
	echo "$a is less than or equals to $b"
else
	echo "$a is not less than $b"
fi
```

> 40 is less than or equals to 200

## 3. Boolean Operators

### ! (Not operator)

This changes `true` into `false` and vice versa.  
We have used this operator in _Not equal operator_ under arithmetic operators.
Shown in the below example:  
-Example 1-

```bash
$x=10
$y=10
a=`expr $x == $y`
echo $a
```

This will display:

> 0

-Example 2-

```bash
$x=10
$y=5
a=`expr $x == $y`
echo $a
```

This will display:

> 1

### -o (OR operator)

Checks whether one of the operands is true and executes the statements.

```bash
a=40
b=20
c=10
if [ $a -gt $b -o $c -gt $b]
	then
	echo "$a or $c is greater than $b"
else
	echo "$a is not less than $b"
fi
```

> 40 or 10 is greater than 20

### -a (AND operator)

Checks whether all of the operands are true and executes the statements.

```bash
a=40
b=20
c=100
if [ $a -gt $b -a $c -gt $b]
	then
	echo "$a and $c are greater than $b"
else
	echo "$a is not less than $b"
fi
```

> 40 and 100 are greater than 20

## 4. String Test Operators

These operators are used to perform string related comparisons.

### = (Equality Operator)

Check whether two strings are equal.

```bash
user="VIKAS"
defaultUser="VIKAS"
if [ $user = $defaultUser ]
then
	echo "User is allowed"
else
	echo "User is not allowed"
fi
```

> User is allowed

### != (Not equal operator)

Check whether two strings are not equal.

```bash
user="VIKAS"
defaultUser="DAVE"
if [ $user != $defaultUser ]
then
	echo "User is not allowed"
else
	echo "User is allowed"
fi
```

> User is not allowed

### -z (Zero length string)

Checks whether the string is zero length.

```bash
s=""
if [ -z $s ]
then
	echo "String is zero length"
else
	echo "String is non zero length"
fi
```

> String is zero length

### -n (Non zero length string)

Checks whether the string is empty.

```bash
s="Village Programmer"
if [ -n $s ]
then
	echo "String is non zero length"
else
	echo "String is zero length"
fi
```

> String is non zero length

### Empty string check

```bash
s=""
if [ $s ]
then
	echo "String is not empty"
else
	echo "String is empty"
fi
```

> String is not empty

---

_**Keep Learining, Keep Practicing. :) :)**_

---

That's All. 😅

Thanks for reading. 🙏  
If you liked it feel free to share with your dear ones.💗  
And tell me what do you think in the comment box.💬

Stay tuned with CS111.

This post was originally posted on [Village Programmer](https://villageprogrammer.blogspot.com)


---

## Learn Complete Linux Bash Scripting in One Article

**Date:** 2021-05-04
**URL:** http://localhost:1313/learn-complete-linux-bash-scripting-in-one-article/
**Description:** 


![](https://images.pexels.com/photos/745266/pexels-photo-745266.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=650&w=940)
-- Photo by **[Iván Rivero](https://www.pexels.com/@osho?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)** from **[Pexels](https://www.pexels.com/photo/tractor-with-trailer-under-cloudy-skies-during-day-745266/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)**

In this article I'll be discussing about linux bash scripting (shell scripting) and I will cover every fundamental concept that you need to get started with bash scripting under linux environment.  
Bash scripting is a critical skill for every programmer.

First of all you will need a **linux machine** to execute these scripts.  
You can use any of the below techniques to get a linux machine:

- Install a linux distribution
- Install WSL
- Install Cygwin or MSYS
- Installing _git for bash_ will provide a bash prompt but this prompt is not a complete linux environment

I'll suggest you to install **WSL2** with **Ubuntu-2020.04** (_same is installed on my machine_).

This one article will cover all the basics and some advanced concepts about bash (shell) scripting, so read it completely.

## Very Basics

### 1. Hello World

Every programmer start by writing a program to print _'Hello World'_ on the console.  
Create a file named `hello.sh` and open it with your favourite text editor (notepad, sublime, atom, vscode, nano, vim, etc).  
Write the following code into the file and save.

```bash
 echo "Hello World :)"
```

Now run the script by using following command

```sh
$ bash ./hello.sh
```

This will print

> Hello World :)

### 2. Comments

We can write comments by using `#` symbol.

```bash
# This is a comment and will not be executed
# The below command will print "Hello World" to the console
echo "Hello World"
```

> Hello World

### 3. What is shebang (#! /bin/bash)

The **shebang is used** if you run the script directly as an executable file (for example with the command ./script.sh ). In this case it tells the operating system which executable to run. It's not required and has no effect if you for example write `bash ./script.sh` or source the script.  
We write the below line at the top of every script
`#!/bin/bash`
or
`#!/usr/bin/env bash`

##### Hello World with shebang

```bash
#!/usr/bin/env bash
# hello.sh
echo "Hello World :)"
```

> Hello World :)

### 4. Variables

Creating variables in bash script is as simple as

```bash
name="Vikash"
```

**NOTE:** Remember there is no space around _=_ operator.

##### Hello World with variables

-- `hello2.sh` --

```bash
#!/usr/bin/env bash
msg="Hello World from a variable"
echo $msg
```

`$ ./hello2.sh`

> Hello World from a variable

### 5. Command line arguments

We can pass arguments while executing the script and use them in the script.  
A simple example is below.

-- `params.sh`--

```bash
#!/usr/bin/env bash
name=$1
website=$2
echo "Hello I am $name"
echo "My website is $website"
```

`$ ./params.sh "Vikash" "Villageprogrammer.blogspot.com"`

> Hello I am Vikash
> My website is Villageprogrammer

| Parameter Name | Description              |
| -------------- | ------------------------ |
| $0             | Script name and path     |
| $1             | First argument           |
| $2 - $9        | Second to ninth argument |
| ${10} - ${...} | More than tenth argument |

You can execute a standard linux command and store the result in the variable.
-- `params2.sh`--

```bash
#!/usr/bin/env bash
today=$(date)
directory=$(pwd)
echo "Today is : $today"
echo "Directory : $directory"
```

`$ ./params2.sh`

> Today is : Thu Feb 18 17:05:56 IST 202
> Directory : /mnt/d/Xplore-Training/UNIX/bash-scripting/scripting

## Controls

### 1. If statement

The syntax of if statement is similar to other programming languages like _C, C++_.  
Syntax

```bash
if [ condition ]
then
	# write your code here
fi
```

Example:

--`if.sh `--

```bash
#!/usr/bin/env bash
num=10
if [ $num -lt 20 ]
then
	echo "$num is less than 20"
fi
```

`$ if.sh`

> 10 is less than 20

You can combine parameter and variable logic with if to write complex scripts.

### 2. if-else

Syntax

```bash
if [ condition ]
then
	# write your code here
else
	# write your code here
fi
```

Example:

--`if-else.sh `--

```bash
#!/usr/bin/env bash
num=10
if [ $num -eq 50 ]
then
	echo "$num is equal to 50"
else
	echo "$num is not equal to 50"
fi
```

`$ ./if-else.sh`

> 10 is not equal to 50

### 3. if-elif-else

Syntax

```bash
if [ condition ]
then
	# write your code here
elif [ condition ]
then
	# write your code here
fi
```

Example:

--`if-elif-else.sh `--

```bash
#!/usr/bin/env bash
num=10
if [ $num -eq 20 ]
then
	echo "$num is equal to 20"
elif [ $num -lt 20 ]
then
	echo "$num is less than 20"
else
	echo "$num is greater than 20"
fi
```

`$ ./if-elif-else.sh`

> 10 is less than 20

## Looping

Repetition of a code block can be done using while loop and for loop.

### 1. while loop

Syntax

```bash
while [ condition ]
do
	# write your code here
done
```

Example

-- `while.sh`--

```bash
count=1
while [ $count -lt 5 ]
do
	echo $count
	((count++))
done
```

`$ ./while.sh`

> 1
> 2
> 3
> 4
> 5

### 2. for loop

We can pass an array of elements from command line and use them in the script by creating an array
**`array=$@`**
Syntax:

```bash
for num in $container
do
	echo $num
done
```

Example:

--`for.sh`--

```bash
#!/usr/bin/env bash
nums=$@
for num in $nums
do
	echo $num
done
```

`$ ./for.sh 1 2 3 `

> 1
> 2
> 3

### 3. break and continue

--`break-continue.sh`--

```bash
#!/usr/bin/env bash
# use '@' for entering more than one inputs (simply an array)
# Pass the values from command line arguments
names=$@
# use of break
for name in $names
do
    if [ $name = "Andy" ]
    then
        echo "End found"
        break
    else
        echo "Hello, $name"
    fi
done

echo "for <break> loop terminated"
# use of continue
for name in $names
do
    if [ $name = "Andy" ]
    then
        echo "Skip found"
        continue
    else
        echo "Hello, $name"
    fi
done

echo "for <continue> loop terminated"
exit 0
```

`$ break-continue.sh "Vikas" "Andy" "Mark"`

> Hello, Vikas
> End Found
> for \<break\> loop terminated
> Hello, Vikas
> Skip found
> Hello, Mark
> for \<continue\> loop terminated

## Working with environment variables

We can access any of the environment variable from the bash script.

--`vars.sh`--

```bash
#!/usr/bin/env bash
# environment variables

echo "The path is : $PATH"
echo "The terminal is : $TERM"
echo "The editor is : $EDITOR"

# -z checks for empty
if [ -z $EDITOR ]
then
    echo "The Editor variable is empty"
fi
exit 0
```

`$ ./vars.sh`

> The path is : \<very long list of all paths will be printed\>
> The terminal is : xterm-256color
> The editor is :
> The Editor variable is empty

##### Environment variables and their meaning

HOME : user's home directory
PATH : executable binary's path
HOSTNAME : hostname of the machine
SHELL : shell that is being used
USER : user of this session
TERM : type of command line terminal that is being used

##### write a program to print user's name, computer name, and home directory

```bash
echo "$USER@$HOSTNAME in [$HOME]"
```

> vikas@DAVE in [/home/vikas]

## Functions in bash

There are two ways to create functions in _bash_ :

#### 1. Function declaration syntax with _function_ keyword

-`function1.sh`-

```bash
#!/usr/bin/env bash
function greeting(){
	echo "Hello"
}
```

#### 2. Function declaration without keyword

-`function2.sh`-

```bash
#!/usr/bin/env bash
greeting(){
	echo "Hello"
}
```

#### 3. Calling the function

To call the function we just need to use the name of the function, we do not use parentheses as we use in other programming languages (e.g. C, C++, Java, Python, etc.).  
-`function1.sh`-

```bash
#!/usr/bin/env bash
# declare the function
function greeting(){
	echo "Hello"
}
# calling the function
greeting
```

**`$ bash ./function1.sh`**

> Hello

#### 4. What about passing parameters ?

It's a two step process

1. **Access the parameter value**
   To access the parameter use :  
   _`$1`_: for first parameter,  
   _`$2`_: for second parameter,  
    ... ,  
    _`$n`_: for nth parameter.  
   You can either create a local variable and save the value by _`local variable=$n`_ or directly access the value by _`$n`_ (n is the parameter number 1, 2, 3, etc).
2. **Pass the parameter value**
   To pass the value write the value after the function name while calling the function (seperate by space).
   `add 2 3`

#### 5. Complete example

-`function.sh`-

```bash
#!/usr/bin/env bash
function add(){
	local num1=$1
	local num2=$2
	sum=`expr $num1 + $num2`
	echo "Sum=$sum"
}

# call the function
add 2 3 # 5
add 4 5 # 9
add -10 30 # 20
# or use command line arguments (uncomment if used)
# add $1 $2
```

**`$ bash ./function.sh`**

> 5
> 9
> 20

## Working with prompt message

You can write a message when asking the user for input.  
It can be done by using **`read`** command.

-`user-prompt.sh`-

```bash
#!/usr/bin/env bash
read "Hey, What's your name ?" name
echo "Hello, $name"
```

**`$ bash ./user-prompt.sh`**

> Hey, What's your name ?
> Vikash
> Hello, Vikash

If you want the input to be in the same line use **`-p`**..

-`user-prompt2.sh`-

```bash
#!/usr/bin/env bash
read -p "Hey, What's your name ?" name
echo "Hello, $name"
```

**`$ bash ./user-prompt2.sh`**

> Hey, What's your name ? Vikash
> Hello, Vikash

---

_Now you have enough understanding to get started with bash (shell) scripting. Go ahead an read a book or two on bash scripting and you will be good at it._

**_Keep Learining, Keep Practicing. :) :)_**

---

That's All. 😅

Thanks for reading. 🙏  
If you liked it feel free to share with your dear ones.💗  
And tell me what do you think in the comment box.💬

Stay tuned with CS111.

This post was originally posted on [Village Programmer](https://villageprogrammer.blogspot.com)


---

## Access Wsl Filesystem From Windows 10 File Explorer

**Date:** 2021-05-04
**URL:** http://localhost:1313/access-wsl-filesystem-from-windows-10-file-explorer/
**Description:** 


![](https://images.unsplash.com/photo-1442120108414-42e7ea50d0b5?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw=&ixlib=rb-1.2.1&auto=format&fit=crop&w=818&q=80)

If you want to get Linux environment in your Windows system there are some solutions that provide the opportunity.\
For example MYSYS, CYGWIN and a Virtual Machine (If you want complete linux machine) or Multiple Boot.\
But some of them aren't much powerful and some are resources hungry like Virtual Machine.\
So, the solution is **WSL (Windows Subsystem for Linux)**.\
You can learn how to setup WSL from [here](https://docs.microsoft.com/en-us/windows/wsl/install-win10#manual-installation-steps).

In this article I'll be sharing how to access WSL file system from Windows 10 file explorer.\
I am assuming you have completely setup WSL and is able to access it through command prompt/powershell.

### Step 1

Open Windows Explorer\
Then type `\\wsl$` in the path location as shown in the below picture and hut enter. ![](https://i.imgur.com/sHbW2MZ.png)

### Step 2

Now a new window will open having your WSL linux machine name.![](https://i.imgur.com/9czE5vN.png)

### Step 3

Now right click on the icon shown in the above image (Name may not be the same)![](https://i.imgur.com/qAexGU9.png)

### Step 4

Now Select `Map Network Drive` option.\
A new diaglog box will open, As shown in the below picture.![](https://i.imgur.com/PXs7nwn.png)

Now choose a drive letter and click **Finish**.

### Step 5

You will have the Linux machine file system listed under Network Storage.![](https://i.imgur.com/ASl6mIy.png)

### Conclusion

You can view, rename, delete, copy/paste the files as you do with normal windows files.\
But do not delete any of the files you don't know. If you do that it may break the WSL System.

That's all from my side.

Thanks for reading. 🙏\
If you liked it feel free to share with your dear ones.💗\
And tell me what do you think in the comment box.💬

Stay tuned with CS111.

This post was originally posted on [Village Programmer](https://villageprogrammer.blogspot.com)