Prateek Codes - Building Scalable Backend Systems

Ruby::Box Practical Guide: Use Cases and Integration Patterns (Part 2)

Thu, 15 Jan 2026 00:00:00 +0000

In Part 1, we covered what Ruby::Box is and how it provides namespace isolation. Now let’s explore practical patterns for integrating it into real applications.

Use Case: Plugin Systems

Plugin systems benefit significantly from Ruby::Box. Each plugin runs in its own isolated environment, preventing plugins from interfering with each other or the host application.

class PluginManager
  def initialize
    @plugins = {}
  end

  def load_plugin(name, path)
    box = Ruby::Box.new
    box.require(path)

    # Access the plugin class from within the box
    plugin_class = box.eval('Plugin')
    @plugins[name] = {
      box: box,
      instance: plugin_class.new
    }
  end

  def run(name, method, *args)
    plugin = @plugins[name]
    plugin[:instance].public_send(method, *args)
  end

  def unload(name)
    @plugins.delete(name)
    # Box becomes eligible for garbage collection
  end
end

# Usage
manager = PluginManager.new
manager.load_plugin(:markdown, './plugins/markdown_plugin')
manager.load_plugin(:syntax_highlight, './plugins/syntax_plugin')

# Each plugin has its own isolated environment
# If markdown_plugin patches String, syntax_plugin won't see it
manager.run(:markdown, :process, content)

This pattern ensures that a misbehaving plugin cannot corrupt the global namespace or break other plugins.

Use Case: Multi-Tenant Configuration

Applications serving multiple tenants often need per-tenant configurations. Ruby::Box provides clean isolation without complex scoping logic.

class TenantContext
  def initialize(tenant_id, config_path)
    @tenant_id = tenant_id
    @box = Ruby::Box.new
    @box.require(config_path)
  end

  def config
    @box.eval('TenantConfig')
  end

  def execute(code)
    @box.eval(code)
  end
end

# Each tenant gets isolated configuration
tenant_a = TenantContext.new('acme', './tenants/acme/config')
tenant_b = TenantContext.new('globex', './tenants/globex/config')

tenant_a.config.theme      # => "dark"
tenant_b.config.theme      # => "light"

# Global variables are isolated too
tenant_a.execute('$rate_limit = 100')
tenant_b.execute('$rate_limit = 500')

tenant_a.execute('$rate_limit')  # => 100
tenant_b.execute('$rate_limit')  # => 500

Use Case: Running Multiple Gem Versions

During migrations, you might need to run two versions of the same gem simultaneously. Ruby::Box makes this possible without separate processes.

# Load v1 API client in one box
v1_box = Ruby::Box.new
v1_box.eval <<~RUBY
  $LOAD_PATH.unshift('./vendor/api_client_v1/lib')
  require 'api_client'
RUBY

# Load v2 API client in another box
v2_box = Ruby::Box.new
v2_box.eval <<~RUBY
  $LOAD_PATH.unshift('./vendor/api_client_v2/lib')
  require 'api_client'
RUBY

# Compare behavior during migration
def compare_responses(endpoint, params)
  code = "ApiClient.get('#{endpoint}', #{params.inspect})"
  v1_response = v1_box.eval(code)
  v2_response = v2_box.eval(code)

  if v1_response != v2_response
    log_difference(endpoint, v1_response, v2_response)
  end

  v1_response  # Return v1 for now, switch to v2 when ready
end

Use Case: Isolated Monkey Patches for Testing

Some tests require monkey patches that would pollute the global namespace. Ruby::Box keeps these contained.

# test_helper.rb
def create_time_frozen_box(frozen_time)
  box = Ruby::Box.new
  box.eval <<~RUBY
    class Time
      def self.now
        Time.new(#{frozen_time.year}, #{frozen_time.month}, #{frozen_time.day})
      end
    end
  RUBY
  box
end

# In your test
def test_subscription_expiry
  box = create_time_frozen_box(Time.new(2026, 1, 1))

  # Load and test code within the frozen-time box
  box.eval <<~RUBY
    expiry_date = Time.new(2025, 12, 31)
    subscription = Subscription.new(expires_at: expiry_date)
    raise "Expected expired" unless subscription.expired?
  RUBY

  # Time.now is unchanged outside the box
  Time.now  # => Current actual time
end

Use Case: Shadow Testing

Run new code paths alongside production code to compare results without affecting users. This pattern is useful for validating refactors or new implementations.

class ShadowRunner
  def initialize(production_box, shadow_box)
    @production = production_box
    @shadow = shadow_box
  end

  def run(method, *args)
    code = "#{method}(#{args.map(&:inspect).join(', ')})"

    # Production path returns the result
    production_result = @production.eval(code)

    # Shadow path runs asynchronously, logs differences
    Thread.new do
      shadow_result = @shadow.eval(code)

      unless production_result == shadow_result
        Logger.warn("Shadow mismatch for #{method}",
          production: production_result,
          shadow: shadow_result
        )
      end
    end

    production_result
  end
end

Working Around Native Extension Issues

Native extensions may fail to install with RUBY_BOX=1 enabled. The solution is to separate installation from execution:

# Gemfile installation without Boxing
bundle install

# Application execution with Boxing
RUBY_BOX=1 bundle exec ruby app.rb

For CI/CD pipelines:

# .github/workflows/test.yml
jobs:
  test:
    steps:
      - name: Install dependencies
        run: bundle install

      - name: Run tests with Ruby::Box
        run: RUBY_BOX=1 bundle exec rspec
        env:
          RUBY_BOX: "1"

Working Around ActiveSupport Issues

Some ActiveSupport core extensions have compatibility issues. Load them in your main context before creating boxes:

# At application startup, before creating any boxes
require 'active_support/core_ext/string/inflections'
require 'active_support/core_ext/hash/keys'

# Now create boxes for isolated code
plugin_box = Ruby::Box.new
# Plugins can use the already-loaded extensions

Alternatively, selectively load only what you need inside boxes:

box = Ruby::Box.new
box.eval <<~RUBY
  # Load specific extensions that are known to work
  require 'active_support/core_ext/object/blank'
RUBY

Performance Considerations

Ruby::Box adds minimal overhead for most operations:

Method dispatch: Slightly more indirection through separate method tables
Object creation: Unaffected, objects pass freely between boxes
Memory: Each box maintains its own class/module definitions

For performance-critical paths, cache class references:

class OptimizedPluginRunner
  def initialize(box)
    @box = box
    # Cache the class reference once
    @processor_class = box.eval('DataProcessor')
  end

  def process(data)
    # Use cached reference instead of evaluating each time
    @processor_class.new.process(data)
  end
end

When to Use `Ruby::Box`

Good candidates:

Plugin or extension systems where isolation is critical
Multi-tenant applications with per-tenant customizations
Testing scenarios requiring invasive monkey patches
Gradual migration between gem versions
Applications loading third-party code that might conflict

Poor candidates:

Running untrusted or potentially malicious code (use OS-level sandboxing)
Production systems until the feature stabilizes
Applications heavily dependent on native extensions
Simple applications without isolation requirements

Migration Strategy

If you’re considering Ruby::Box for an existing application:

Step 1: Test compatibility

# Run your test suite with Boxing enabled
RUBY_BOX=1 bundle exec rspec

Step 2: Identify issues

Look for failures related to:

Shared global state across files
Assumptions about class modifications being visible everywhere
Native extension loading errors

Step 3: Refactor incrementally

Start with isolated subsystems that don’t share state with the rest of your application. Move more code into boxes as you gain confidence.

Step 4: Monitor in staging

Run your staging environment with RUBY_BOX=1 before considering production deployment.

What’s Next for `Ruby::Box`

The Ruby core team has discussed building a higher-level “packages” API on top of Ruby::Box. This would provide more ergonomic ways to manage gem isolation without manual box management. Track progress in Ruby Issue #21681.

Ruby::Box solves real problems around namespace pollution and gem conflicts. While still experimental, it’s worth exploring for applications where isolation matters. Start with non-critical paths, understand the limitations, and provide feedback to the Ruby core team as you experiment.

References

Ruby 4.0 Introduces Ruby::Box for In-Process Isolation (Part 1)

Wed, 14 Jan 2026 00:00:00 +0000

Ruby 4.0 introduces Ruby::Box, a feature that provides isolated namespaces within a single Ruby process. This solves a long-standing problem: monkey patches and global modifications from one gem affecting all other code in your application.

The Problem with Shared Namespaces

When you load a gem that modifies core classes, those changes affect everything in your Ruby process:

# Gem A adds a titleize method to String
class String
  def titleize
    split.map(&:capitalize).join(' ')
  end
end

# Now EVERY piece of code in your process sees this method
# Including Gem B, which might have its own expectations

"hello world".titleize  # => "Hello World"

This becomes problematic when:

Two gems define conflicting methods on the same class
A gem’s monkey patch breaks another library’s assumptions
You want to test code in isolation from invasive patches
You need to run multiple versions of a gem simultaneously

Before Ruby 4.0, the only solutions were separate Ruby processes (with IPC overhead) or containers (with even more overhead).

Ruby 4.0: Enter Ruby::Box

Ruby::Box creates isolated spaces where code runs with its own class definitions, constants, and global variables. Changes made inside a box stay inside that box.

# Enable with environment variable at startup
# RUBY_BOX=1 ruby my_script.rb

# Check if Boxing is available
Ruby::Box.enabled?  # => true

# Create an isolated box
box = Ruby::Box.new

# Load code that patches String
box.eval <<~RUBY
  class String
    def shout
      upcase + "!!!"
    end
  end
RUBY

# The patch exists only inside the box
box.eval('"hello".shout')  # => "HELLO!!!"

# Outside the box, String is unchanged
"hello".shout  # => NoMethodError: undefined method `shout'

Understanding Box Types

Ruby::Box operates with three types of boxes:

Root Box: Contains all built-in Ruby classes and modules. This is established before any user code runs and serves as the template for other boxes.

Main Box: Your application’s default execution context. It’s automatically created from the root box when the process starts. This is where your main script runs.

User Boxes: Custom boxes you create with Ruby::Box.new. Each is copied from the root box, giving it a clean slate of built-in classes without any modifications from the main box or other user boxes.

# Your script runs in the "main" box
Ruby::Box.current  # => #<Ruby::Box main>

# Create isolated boxes
plugin_box = Ruby::Box.new
another_box = Ruby::Box.new

# Each box is independent
plugin_box.object_id != another_box.object_id  # => true

The Ruby::Box API

The API is straightforward with just a few methods:

# Creation
box = Ruby::Box.new

# Loading code
box.require('some_library')        # Respects box's $LOAD_PATH
box.require_relative('./my_file')  # Relative to current file
box.load('script.rb')              # Direct file execution

# Executing code
box.eval('1 + 1')                  # Execute Ruby code as string

# Inspection
Ruby::Box.current    # Returns the currently executing box
Ruby::Box.enabled?   # Check if Boxing is active

What Gets Isolated

Ruby::Box isolates several aspects of the Ruby runtime:

Classes and Constants: Reopening a built-in class in one box doesn’t affect other boxes.

box = Ruby::Box.new
box.eval <<~RUBY
  class Array
    def sum_squares
      map { |n| n ** 2 }.sum
    end
  end
RUBY

box.eval('[1, 2, 3].sum_squares')  # => 14
[1, 2, 3].sum_squares              # => NoMethodError

Global Variables: Changes to globals stay within the box.

box = Ruby::Box.new
box.eval('$my_config = { debug: true }')

box.eval('$my_config')  # => { debug: true }
$my_config              # => nil

Top-Level Methods: Methods defined at the top level become private instance methods of Object within that box only.

box = Ruby::Box.new
box.eval <<~RUBY
  def helper_method
    "I'm only available in this box"
  end
RUBY

box.eval('helper_method')  # => "I'm only available in this box"
helper_method              # => NoMethodError

Enabling Ruby::Box

Ruby::Box is disabled by default. Enable it by setting the RUBY_BOX environment variable before the Ruby process starts:

RUBY_BOX=1 ruby my_application.rb

Important: Setting RUBY_BOX after the process has started has no effect. The boxing infrastructure must be initialized during Ruby’s boot sequence, so the variable must be set before the Ruby process starts.

# This check should be at the top of your application
unless Ruby::Box.enabled?
  warn "Ruby::Box is not enabled. Start with RUBY_BOX=1"
  exit 1
end

Important Limitations

Before adopting Ruby::Box, be aware of these constraints:

Not a Security Sandbox: Ruby::Box provides namespace isolation, not security isolation. Code in a box can still access the filesystem, network, and system resources. Do not use it to run untrusted code.

Native Extensions: Installing gems with native extensions may fail when RUBY_BOX=1 is set. The workaround is to install gems without the flag, then run your application with it enabled.

# Install gems normally
bundle install

# Run with Boxing enabled
RUBY_BOX=1 bundle exec ruby app.rb

ActiveSupport Compatibility: Some parts of active_support/core_ext have compatibility issues with Ruby::Box. Load ActiveSupport in your main context before creating boxes if needed.

Experimental Status: This feature is experimental in Ruby 4.0. Behavior may change in future versions. The Ruby core team recommends experimentation but advises caution in production environments.

File Scope Execution

One important detail: Ruby::Box operates on a file-scope basis. Each .rb file executes entirely within a single box. Once loaded, all methods and procs defined in that file operate within their originating box, regardless of where they’re called from.

# helper.rb
def process(data)
  # This method always runs in the box where helper.rb was loaded
  data.transform
end

# main.rb
box = Ruby::Box.new
box.require_relative('helper')

# Even when called from main, process() runs in box's context
box.eval('process(my_data)')

Ruby::Box brings a long-requested capability to Ruby: proper namespace isolation without process boundaries. In Part 2, we’ll explore practical use cases including plugin systems, multi-tenant configurations, and strategies for gradual adoption.

References

Rails 8.2 makes enqueue_after_transaction_commit the default

Wed, 31 Dec 2025 00:00:00 +0000

Rails 7.2 introduced enqueue_after_transaction_commit to prevent race conditions when jobs are enqueued inside database transactions. However, it required explicit opt-in. Rails 8.2 flips the default. Jobs are now automatically deferred until after the transaction commits.

The Problem with Opt-In

With the opt-in approach in Rails 7.2, teams had to remember to enable the feature:

config.active_job.enqueue_after_transaction_commit = :default

Or configure it per-job:

class WelcomeEmailJob < ApplicationJob
  self.enqueue_after_transaction_commit = :always
end

This created inconsistency. Some jobs would be transaction-aware, others would not. The safer behavior required explicit action.

Rails 8.2 Changes the Default

PR #55788 changes this. When you upgrade to Rails 8.2 and run load_defaults "8.2", jobs are automatically deferred until after the transaction commits.

def create
  User.transaction do
    user = User.create!(params)
    WelcomeEmailJob.perform_later(user)  # Deferred until commit
  end
end

No configuration needed. The job waits for the transaction to complete before being dispatched to the queue.

Opting Out

If you need immediate enqueueing for backward compatibility or specific use cases, you have two options.

Global configuration:

config.active_job.enqueue_after_transaction_commit = false

Per-job configuration:

class TimeStampedJob < ApplicationJob
  self.enqueue_after_transaction_commit = false
end

Why the Global Config Was Restored

The global configuration option has an interesting history. It was deprecated and removed in Rails 8.1. The team initially wanted each job to declare its own preference. However, changing the default behavior without a global opt-out would break existing applications.

The PR restored the global configuration specifically to allow apps upgrading to Rails 8.2 to maintain their existing behavior without modifying every job class.

When This Matters

The new default primarily affects jobs enqueued to external queues like Redis (Sidekiq, Resque). If you use a database-backed queue like Solid Queue or GoodJob with the same database, your jobs are already part of the same transaction.

Jobs that do not depend on transaction data can still be configured for immediate enqueueing if needed.

Conclusion

Rails 8.2 makes the safer behavior the default. Jobs enqueued inside transactions automatically wait for the commit, eliminating a common source of race conditions without requiring explicit configuration.

References

Pull Request #55788 making this the default
Pull Request #51426 introducing the feature in Rails 7.2
Rails 7.2 enqueue_after_transaction_commit - detailed explanation of the feature

Rails 7.2 adds enqueue_after_transaction_commit to prevent job race conditions

Tue, 30 Dec 2025 00:00:00 +0000

Scheduling background jobs inside database transactions is a common anti-pattern which is a source of several production bugs in Rails applications. The job can execute before the transaction commits, leading to RecordNotFound or ActiveJob::DeserializationError because the data it needs does not exist yet. Or worse, the job could run assuming the txn would commit, but it rolls back at a later stage. We don’t need that kind of optimism.

Rails 7.2 addresses this with enqueue_after_transaction_commit, which automatically defers job enqueueing until the transaction completes.

Before

Consider a typical pattern where you create a user and send a welcome email:

class UsersController < ApplicationController
  def create
    User.transaction do
      @user = User.create!(user_params)
      WelcomeEmailJob.perform_later(@user)
    end
  end
end

This code works fine in development where your job queue is slow and transactions commit quickly. In production, with a fast Redis-backed queue like Sidekiq and a busy database, the job can start executing before the transaction commits:

Timeline:
1. Transaction begins
2. User INSERT executes (not committed yet)
3. Job enqueued to Redis
4. Sidekiq picks up job immediately
5. Job tries to find User -> RecordNotFound!
6. Transaction commits (too late)

The same problem occurs with after_create callbacks in models:

class Project < ApplicationRecord
  after_create -> { NotifyParticipantsJob.perform_later(self) }
end

The Workaround

The standard fix was to use after_commit callbacks instead:

class Project < ApplicationRecord
  after_create_commit -> { NotifyParticipantsJob.perform_later(self) }
end

Or wrap job scheduling in explicit after_commit blocks:

class UsersController < ApplicationController
  def create
    User.transaction do
      @user = User.create!(user_params)

      ActiveRecord::Base.connection.after_transaction_commit do
        WelcomeEmailJob.perform_later(@user)
      end
    end
  end
end

This worked but had problems:

Easy to forget: Using after_create instead of after_create_commit is a common mistake
Scattered logic: Job scheduling gets coupled to model callbacks instead of staying in controllers or service objects
Verbose: Wrapping every perform_later call in after_commit blocks adds boilerplate
Testing friction: Transaction callbacks behave differently in test environments using database cleaner with transactions

The after_commit_everywhere gem became popular specifically to address this problem. It lets you use after_commit callbacks anywhere in your application, not just in ActiveRecord models:

class UserRegistrationService
  include AfterCommitEverywhere

  def call(params)
    User.transaction do
      user = User.create!(params)

      after_commit do
        WelcomeEmailJob.perform_later(user)
      end
    end
  end
end

The gem hooks into ActiveRecord’s transaction lifecycle and ensures callbacks only fire after the outermost transaction commits. It handled nested transactions correctly and became a go-to solution for service objects that needed transaction-safe job scheduling.

Some teams built their own lightweight wrappers instead:

# Custom AsyncRecord class that hooks into transaction callbacks
class AsyncRecord
  def initialize(&block)
    @callback = block
  end

  def has_transactional_callbacks?
    true
  end

  def committed!(*)
    @callback.call
  end

  def rolledback!(*)
    # Do nothing if transaction rolled back
  end
end

# Usage
User.transaction do
  user = User.create!(params)
  record = AsyncRecord.new { WelcomeEmailJob.perform_later(user) }
  user.class.connection.add_transaction_record(record)
end

Both approaches worked, but required teams to remember to use them consistently.

Rails 7.2

Rails 7.2 makes Active Job transaction-aware. Jobs are automatically deferred until the transaction commits, and dropped if it rolls back.

Enable it globally in your application:

# config/application.rb
config.active_job.enqueue_after_transaction_commit = :default

Now the original code just works:

class UsersController < ApplicationController
  def create
    User.transaction do
      @user = User.create!(user_params)
      WelcomeEmailJob.perform_later(@user)  # Deferred until commit
    end
  end
end

The job only gets enqueued after the transaction successfully commits. If the transaction rolls back, the job is never enqueued.

Configuration Options

You can control this behavior at three levels:

Global configuration:

# config/application.rb
config.active_job.enqueue_after_transaction_commit = :default

Per-job configuration:

class WelcomeEmailJob < ApplicationJob
  self.enqueue_after_transaction_commit = :always
end

class AuditLogJob < ApplicationJob
  self.enqueue_after_transaction_commit = :never  # Queue immediately
end

The available values are:

:default - Let the queue adapter decide the behavior
:always - Always defer until transaction commits
:never - Queue immediately (pre-7.2 behavior)

Checking Enqueue Status

Since perform_later returns immediately even when the job is deferred, you can check if it was actually enqueued:

User.transaction do
  user = User.create!(user_params)
  job = WelcomeEmailJob.perform_later(user)

  # job.successfully_enqueued? returns false here (still deferred)
end

# After transaction commits, job.successfully_enqueued? returns true

Model Callbacks Simplified

You can now safely use after_create for job scheduling without worrying about transaction timing:

class Project < ApplicationRecord
  # This is now safe with enqueue_after_transaction_commit enabled
  after_create -> { NotifyParticipantsJob.perform_later(self) }
end

The job automatically waits for any enclosing transaction to complete.

When to Disable

Some scenarios require immediate enqueueing:

Database-backed queues: If you use Solid Queue, GoodJob, or Delayed Job with the same database, jobs are part of the same transaction and this deferral is unnecessary
Fire-and-forget jobs: Jobs that do not depend on the transaction data can run immediately
Time-sensitive operations: If you need the job queued at a specific moment regardless of transaction state

class TimeStampedJob < ApplicationJob
  self.enqueue_after_transaction_commit = :never

  def perform
    # This job needs to capture the exact enqueue time
  end
end

Update: Rails 8.2 Makes This the Default

Rails 8.2 makes enqueue_after_transaction_commit the default behavior. Jobs are now automatically deferred until after the transaction commits without requiring explicit configuration.

See Rails 8.2 makes enqueue_after_transaction_commit the default for details on the change, opting out, and the deprecation history.

Conclusion

enqueue_after_transaction_commit eliminates a common source of race conditions in Rails applications. Instead of remembering to use after_commit callbacks or building custom workarounds, jobs are automatically deferred until transactions complete.

References

Pull Request #51426 introducing the feature
Pull Request #55788 making this the default in Rails 8.2
Original Issue #26045 by DHH describing the problem
Active Job Basics Guide

Rails 8.2 introduces Rails.app.creds for unified credential management

Mon, 29 Dec 2025 00:00:00 +0000

Applications often store secrets in both environment variables and encrypted credential files. Migrating between these storage methods or using both simultaneously has traditionally required code changes. Rails 8.2 solves this with Rails.app.creds, a unified API that checks ENV first, then falls back to encrypted credentials.

Before

Managing credentials from multiple sources meant mixing different APIs:

class StripeService
  def initialize
    # Check ENV first, fallback to credentials
    @api_key = ENV["STRIPE_API_KEY"] || Rails.application.credentials.dig(:stripe, :api_key)
    @webhook_secret = ENV.fetch("STRIPE_WEBHOOK_SECRET") {
      Rails.application.credentials.stripe&.webhook_secret
    }

    raise "Missing Stripe API key!" unless @api_key
  end
end

class DatabaseConfig
  def connection_url
    # Different syntax for each source
    ENV["DATABASE_URL"] || Rails.application.credentials.database_url
  end

  def redis_url
    ENV.fetch("REDIS_URL", Rails.application.credentials.dig(:redis, :url) || "redis://localhost:6379")
  end
end

This approach has several problems:

Inconsistent APIs between ENV.fetch() and credentials.dig()
Manual fallback logic scattered throughout the codebase
Code changes required when moving secrets between storage methods
Easy to forget nil checks on nested credentials

Rails 8.2

The new Rails.app.creds provides a consistent interface:

class StripeService
  def initialize
    @api_key = Rails.app.creds.require(:stripe_api_key)
    @webhook_secret = Rails.app.creds.require(:stripe_webhook_secret)
  end
end

class DatabaseConfig
  def connection_url
    Rails.app.creds.require(:database_url)
  end

  def redis_url
    Rails.app.creds.option(:redis_url, default: "redis://localhost:6379")
  end
end

The require method mandates a value exists and raises KeyError if missing from both ENV and encrypted credentials. The option method returns nil or a default value gracefully.

Nested Keys

For nested credentials, pass multiple keys. Rails automatically converts them to the appropriate format for each source:

# Checks ENV["AWS__ACCESS_KEY_ID"] first, then credentials.dig(:aws, :access_key_id)
Rails.app.creds.require(:aws, :access_key_id)

# Multi-level nesting
# ENV["REDIS__CACHE__TTL"] || credentials.dig(:redis, :cache, :ttl)
Rails.app.creds.option(:redis, :cache, :ttl, default: 3600)

The ENV lookup uses double underscores (__) as separators for nested keys:

:database_url → ENV["DATABASE_URL"]
[:aws, :region] → ENV["AWS__REGION"]
[:redis, :cache, :ttl] → ENV["REDIS__CACHE__TTL"]

Dynamic Defaults

The option method accepts callable defaults, evaluated only when needed:

Rails.app.creds.option(:cache_ttl, default: -> { 1.hour })
Rails.app.creds.option(:max_connections, default: -> { calculate_pool_size })

ENV-Only Access

Access environment variables directly using the same API via Rails.app.envs:

# Only checks ENV, no encrypted credentials fallback
Rails.app.envs.require(:port)
Rails.app.envs.option(:log_level, default: "info")

Custom Credential Sources

Under the hood, Rails.app.creds is powered by ActiveSupport::CombinedConfiguration, which checks multiple credential sources (called backends) in order. By default, it checks ENV first, then encrypted credentials. You can customize this chain to include external secret managers:

# config/initializers/credentials.rb
Rails.app.creds = ActiveSupport::CombinedConfiguration.new(
  Rails.app.envs,                   # Check ENV first
  VaultConfiguration.new,           # Then HashiCorp Vault
  OnePasswordConfiguration.new,     # Then 1Password
  Rails.app.credentials             # Finally, encrypted credentials
)

Each credential source needs to implement require and option methods matching the API.

Rails.app Alias

This feature comes alongside a new Rails.app alias for Rails.application:

# Before
Rails.application.credentials.aws.access_key_id

# After
Rails.app.credentials.aws.access_key_id

The shorter alias makes chained method calls more pleasant to read and write.

Conclusion

Rails.app.creds eliminates the friction of managing credentials across multiple sources. Secrets can move between ENV and encrypted files without touching application code.

References

PR #56404 - Add Rails.app.creds for combined credentials lookup
PR #56403 - Add Rails.app alias for Rails.application

Understanding PostgreSQL Checkpoints: From WAL to Disk

Fri, 17 Oct 2025 00:00:00 +0000

PostgreSQL relies on checkpoints to ensure data durability while maintaining performance. Understanding how checkpoints work and their relationship with Write-Ahead Logging is essential for database performance tuning and troubleshooting.

This post builds on fundamental concepts covered in our PostgreSQL internals series:

Write-Ahead Logging: The Foundation

Checkpoints work hand-in-hand with Write-Ahead Logging (WAL). When PostgreSQL modifies data, changes are written to WAL files first (sequential, fast) before updating data pages in memory. Modified pages (called “dirty pages”) accumulate in shared_buffers, and eventually these changes need to be written to the actual data files. That’s where checkpoints come in.

What Happens During a Checkpoint

A checkpoint is PostgreSQL’s process of writing all dirty pages from shared buffers to disk. It creates a known recovery point and ensures data durability.

The Checkpoint Process

When a checkpoint occurs:

Checkpoint starts
- PostgreSQL marks the current WAL position as the checkpoint location
- This position is the recovery starting point if a crash occurs
Dirty pages are written
- All modified data pages in shared_buffers are flushed to disk
- This happens gradually to avoid I/O spikes (controlled by checkpoint_completion_target)
- Pages are written in order to minimize random I/O
Checkpoint completes
- A checkpoint record is written to WAL
- The pg_control file is updated with the new checkpoint location
- Old WAL files (before the checkpoint) can now be recycled or archived

What Triggers a Checkpoint?

PostgreSQL creates checkpoints based on:

Time: checkpoint_timeout parameter (default: 5 minutes)
WAL volume: max_wal_size parameter (default: 1GB)
Manual trigger: CHECKPOINT command
Shutdown: Always creates a checkpoint during clean shutdown

-- Force an immediate checkpoint
CHECKPOINT;

Checkpoint Impact on Performance

Checkpoints involve heavy I/O (writing potentially gigabytes of dirty pages), which can cause temporary performance degradation. Understanding performance trade-offs helps you balance durability with speed. PostgreSQL spreads checkpoint writes over time using checkpoint_completion_target (default: 0.9) to minimize I/O spikes.

Monitoring Checkpoint Activity

For detailed monitoring techniques, see Part 6: Monitoring and Administration. Key metrics to track:

Check Checkpoint Statistics

-- PostgreSQL 17+
SELECT * FROM pg_stat_checkpointer;

-- PostgreSQL 16 and earlier
SELECT * FROM pg_stat_bgwriter;

What to look for:

High checkpoints_req (or num_requested in v17+) means checkpoints are happening too frequently
Large checkpoint_write_time (or write_time in v17+) indicates heavy I/O load

Monitor WAL Generation Rate

High WAL generation can trigger frequent checkpoints. See Part 6 for detailed WAL monitoring queries and interpretation.

Tuning Checkpoint Behavior

Key parameters to adjust (see Part 4: Performance Patterns for trade-offs):

checkpoint_timeout = 15min          # Default: 5min
max_wal_size = 4GB                  # Default: 1GB
checkpoint_completion_target = 0.9  # Default: 0.9
log_checkpoints = on

Guidelines:

Increase max_wal_size if checkpoints_req is high
Increase checkpoint_timeout for write-heavy workloads
Keep checkpoint_completion_target at 0.9 to avoid I/O spikes

For broader PostgreSQL performance optimization, see PostgreSQL query optimization guide.

Conclusion

Checkpoints are PostgreSQL’s mechanism for persisting in-memory changes to disk, creating recovery points, and managing WAL files. They balance performance with durability by batching writes and spreading I/O over time. Watch for frequent requested checkpoints and long write times as signals for tuning opportunities.

For deeper understanding, explore the PostgreSQL internals series, or dive into PostgreSQL EXPLAIN ANALYZE for query optimization.

References

PostgreSQL Fundamentals: Monitoring and Administration Tools (Part 6)

Thu, 16 Oct 2025 00:00:00 +0000

In Part 5, we learned how Write-Ahead Logging works internally. Now let’s explore the tools PostgreSQL provides for monitoring and administering these systems.

This is Part 6 (final part) of a series on PostgreSQL internals.

System Views: Your Window Into PostgreSQL

PostgreSQL exposes extensive information through system views. These views are your primary tool for understanding what’s happening inside the database.

pg_stat_checkpointer: Checkpoint Statistics (PostgreSQL 17+)

The most important view for checkpoint monitoring:

SELECT * FROM pg_stat_checkpointer;

Key columns:

SELECT
    num_timed,              -- Scheduled checkpoints (time-based)
    num_requested,          -- Requested checkpoints (WAL-based or manual)
    write_time,             -- Milliseconds spent writing files
    sync_time,              -- Milliseconds spent syncing files
    buffers_written,        -- Buffers written during checkpoints
    stats_reset            -- When stats were last reset
FROM pg_stat_checkpointer;

Note: Before PostgreSQL 17, checkpoint statistics were in pg_stat_bgwriter with column names checkpoints_timed, checkpoints_req, checkpoint_write_time, checkpoint_sync_time, and buffers_checkpoint.

Interpreting pg_stat_checkpointer

-- Example output:
num_timed:       1250    -- Mostly time-based (good)
num_requested:   45      -- Few requested (good)
write_time:      450000  -- 450 seconds total writing
sync_time:       2500    -- 2.5 seconds total syncing
buffers_written: 500000  -- 500k buffers written at checkpoints

What to look for:

High num_requested: Checkpoints happening too frequently
- Solution: Increase max_wal_size
High write_time: Checkpoint I/O is slow
- Solution: Increase checkpoint_completion_target
- Or: Improve disk I/O performance
High buffers_written relative to checkpoint frequency: Large checkpoints
- Solution: More frequent checkpoints or increase shared_buffers

pg_stat_wal: WAL Activity

Monitor WAL generation and flush activity:

SELECT
    wal_records,        -- Total WAL records generated
    wal_fpi,           -- Full page images written
    wal_bytes,         -- Total bytes written to WAL
    wal_buffers_full,  -- Times WAL buffer was full
    wal_write,         -- Number of WAL writes
    wal_sync,          -- Number of WAL syncs (fsync)
    wal_write_time,    -- Time spent writing WAL (ms)
    wal_sync_time,     -- Time spent syncing WAL (ms)
    stats_reset
FROM pg_stat_wal;

Calculating WAL Generation Rate

-- Record current WAL stats
CREATE TEMP TABLE wal_baseline AS
SELECT
    now() AS measured_at,
    pg_current_wal_lsn() AS wal_lsn,
    wal_bytes
FROM pg_stat_wal;

-- Wait 60 seconds...
SELECT pg_sleep(60);

-- Calculate rate
SELECT
    pg_size_pretty(
        w.wal_bytes - b.wal_bytes
    ) AS wal_generated,
    EXTRACT(EPOCH FROM (now() - b.measured_at)) AS seconds,
    pg_size_pretty(
        (w.wal_bytes - b.wal_bytes) /
        EXTRACT(EPOCH FROM (now() - b.measured_at))
    ) || '/s' AS wal_rate
FROM pg_stat_wal w, wal_baseline b;

-- Example result:
-- wal_generated: 25 MB
-- seconds: 60
-- wal_rate: 427 kB/s

pg_stat_database: Database-Level Stats

SELECT
    datname,
    xact_commit,           -- Transactions committed
    xact_rollback,         -- Transactions rolled back
    blks_read,            -- Disk blocks read
    blks_hit,             -- Disk blocks found in cache
    tup_inserted,         -- Rows inserted
    tup_updated,          -- Rows updated
    tup_deleted,          -- Rows deleted
    temp_files,           -- Temp files created
    temp_bytes            -- Temp file bytes
FROM pg_stat_database
WHERE datname = current_database();

Calculate cache hit ratio:

SELECT
    datname,
    round(
        100.0 * blks_hit / nullif(blks_hit + blks_read, 0),
        2
    ) AS cache_hit_ratio
FROM pg_stat_database
WHERE datname = current_database();

-- Healthy databases: 95%+
-- Low ratio: Need more shared_buffers or working set too large

pg_stat_statements: Query-Level WAL Generation

Track which queries generate the most WAL:

-- Enable extension
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Top WAL generators (PostgreSQL 13+)
SELECT
    substring(query, 1, 60) AS query_preview,
    calls,
    pg_size_pretty(wal_bytes) AS wal_generated,
    pg_size_pretty(wal_bytes / calls) AS wal_per_call,
    round(100.0 * wal_bytes / sum(wal_bytes) OVER (), 2) AS wal_percent
FROM pg_stat_statements
ORDER BY wal_bytes DESC
LIMIT 10;

Once you identify high WAL-generating queries, you can optimize write operations: batch INSERT/UPDATE operations, use COPY for bulk loads, and consider whether all indexes are necessary.

pg_stat_activity: Live Connections

SELECT
    pid,
    usename,
    application_name,
    state,
    query_start,
    state_change,
    wait_event_type,
    wait_event,
    substring(query, 1, 50) AS query_preview
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;

Find long-running queries:

SELECT
    pid,
    now() - query_start AS duration,
    query
FROM pg_stat_activity
WHERE state = 'active'
  AND now() - query_start > interval '5 minutes'
ORDER BY duration DESC;

For long-running queries, you can:

Analyze execution plans: Use EXPLAIN ANALYZE to understand why queries are slow. See PostgreSQL EXPLAIN ANALYZE Deep Dive for detailed analysis techniques.
Offload reads to replicas: Move long-running SELECT queries to read replicas to reduce contention on the primary database. See Rails Read Replicas Part 1 for implementation patterns.

PostgreSQL Logs: Checkpoint and WAL Messages

Enable detailed logging in postgresql.conf:

# Log checkpoints
log_checkpoints = on

# Log long-running statements
log_min_duration_statement = 1000  # Log queries > 1 second

# Log connections and disconnections
log_connections = on
log_disconnections = on

# Set log destination
logging_collector = on
log_directory = 'log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

Reading Checkpoint Logs

With log_checkpoints = on, you’ll see:

2025-10-17 10:15:42.123 UTC [12345] LOG: checkpoint starting: time
2025-10-17 10:16:11.456 UTC [12345] LOG: checkpoint complete: wrote 2435 buffers (14.9%); 0 WAL file(s) added, 0 removed, 3 recycled; write=29.725 s, sync=0.004 s, total=29.780 s; sync files=7, longest=0.003 s, average=0.001 s; distance=49142 kB, estimate=49142 kB

Breakdown:

wrote 2435 buffers (14.9%)    # Dirty pages written (14.9% of shared_buffers)
0 WAL file(s) added           # New WAL segments created
0 removed                     # Old WAL segments deleted
3 recycled                    # WAL segments renamed for reuse
write=29.725 s                # Time spent writing buffers
sync=0.004 s                  # Time spent fsync'ing
total=29.780 s                # Total checkpoint duration
distance=49142 kB             # WAL generated since last checkpoint
estimate=49142 kB             # Estimated WAL to next checkpoint

What to watch:

High write time: Checkpoint taking too long
- Check disk I/O performance
- Consider spreading checkpoint over more time
Frequent checkpoints: If you see many “checkpoint starting: xlog” instead of “checkpoint starting: time”
- Increase max_wal_size
Large distance: Generating lots of WAL
- Normal for write-heavy workloads
- Ensure max_wal_size is adequate

pg_waldump: Inspecting WAL Records

pg_waldump lets you read WAL files directly:

# Find WAL files
ls $PGDATA/pg_wal/

# If this doesn't work, replace PGDATA with output from SHOW data_directory;

# Dump a WAL segment
pg_waldump $PGDATA/pg_wal/000000010000000000000001

# Output shows each WAL record:
rmgr: Heap        len: 54   rec: INSERT off 1 flags 0x00
rmgr: Btree       len: 72   rec: INSERT_LEAF off 5
rmgr: Transaction len: 34   rec: COMMIT 2025-10-17 10:15:42.123456 UTC

Filtering WAL Records

# Show only specific resource manager (Heap = table data)
pg_waldump -r Heap $PGDATA/pg_wal/000000010000000000000001

# Show records from specific LSN range
pg_waldump -s 0/1500000 -e 0/1600000 $PGDATA/pg_wal/000000010000000000000001

# Show statistics summary
pg_waldump --stats $PGDATA/pg_wal/000000010000000000000001

Understanding WAL Record Output

rmgr: Heap        len: 54   rec: INSERT off 1
    lsn: 0/01500028, prev: 0/01500000, desc: INSERT off 1 flags 0x00
    blkref #0: rel 1663/16384/16385 blk 0

Breaking this down:

rmgr: Heap              # Resource manager (table data)
len: 54                 # Record length in bytes
rec: INSERT             # Operation type
lsn: 0/01500028        # Log Sequence Number
prev: 0/01500000       # Previous record LSN
blkref #0:             # Block reference
  rel 1663/16384/16385 # Relation OID (tablespace/database/relation)
  blk 0                # Block number

pg_control: Database Cluster State

View control file (requires command-line tool):

pg_controldata $PGDATA

Key information:

pg_control version number:            1300
Catalog version number:               202107181
Database system identifier:           7012345678901234567
Database cluster state:               in production
pg_control last modified:             Thu 17 Oct 2025 10:15:42 AM UTC
Latest checkpoint location:           0/1500000
Latest checkpoint's REDO location:    0/1480000
Latest checkpoint's TimeLineID:       1
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID:          0:1000
Latest checkpoint's NextOID:          24576

This shows the last checkpoint LSN, which is crucial for crash recovery.

Monitoring Checkpoint Health

Create a monitoring query (PostgreSQL 17+):

CREATE OR REPLACE VIEW checkpoint_health AS
SELECT
    num_timed,
    num_requested,
    round(100.0 * num_requested /
          nullif(num_timed + num_requested, 0), 2
    ) AS req_checkpoint_pct,
    pg_size_pretty(
        buffers_written * 8192::bigint
    ) AS checkpoint_write_size,
    round(
        write_time::numeric /
        nullif(num_timed + num_requested, 0),
        2
    ) AS avg_checkpoint_write_ms,
    round(
        sync_time::numeric /
        nullif(num_timed + num_requested, 0),
        2
    ) AS avg_checkpoint_sync_ms
FROM pg_stat_checkpointer;

-- Check health
SELECT * FROM checkpoint_health;

-- Example output:
-- num_timed: 1200
-- num_requested: 50
-- req_checkpoint_pct: 4.00           ← Good (< 10%)
-- checkpoint_write_size: 4000 MB
-- avg_checkpoint_write_ms: 375.21
-- avg_checkpoint_sync_ms: 2.08

For PostgreSQL 16 and earlier, use pg_stat_bgwriter with column names checkpoints_timed, checkpoints_req, checkpoint_write_time, checkpoint_sync_time, and buffers_checkpoint.

Healthy checkpoint system:

req_checkpoint_pct < 10%: Most checkpoints are scheduled
Reasonable write times: Not overwhelming the I/O system
Consistent checkpoint sizes

Resetting Statistics

Statistics accumulate since the last reset:

-- Reset all statistics
SELECT pg_stat_reset();

-- Reset bgwriter stats
SELECT pg_stat_reset_shared('bgwriter');

-- Reset WAL stats
SELECT pg_stat_reset_shared('wal');

-- Check when stats were last reset
SELECT stats_reset FROM pg_stat_bgwriter;

Reset stats to measure recent behavior or after configuration changes.

Putting It All Together

A complete checkpoint monitoring query:

WITH wal_rate AS (
    SELECT
        pg_size_pretty(wal_bytes) AS total_wal,
        wal_records AS total_records,
        wal_fpi AS full_page_images
    FROM pg_stat_wal
),
checkpoint_stats AS (
    SELECT
        checkpoints_timed + checkpoints_req AS total_checkpoints,
        checkpoints_req,
        round(100.0 * checkpoints_req /
              nullif(checkpoints_timed + checkpoints_req, 0), 2
        ) AS req_pct,
        pg_size_pretty(buffers_checkpoint * 8192::bigint) AS data_written,
        round(checkpoint_write_time::numeric /
              nullif(checkpoints_timed + checkpoints_req, 0), 2
        ) AS avg_write_ms
    FROM pg_stat_bgwriter
)
SELECT
    c.total_checkpoints,
    c.checkpoints_req,
    c.req_pct || '%' AS req_checkpoint_pct,
    w.total_wal,
    w.total_records,
    w.full_page_images,
    c.data_written AS checkpoint_data_written,
    c.avg_write_ms || ' ms' AS avg_checkpoint_write_time
FROM checkpoint_stats c, wal_rate w;

Conclusion

You now have the foundational knowledge of PostgreSQL internals:

Memory vs disk performance (Part 1)
How data is stored in pages (Part 2)
Transactions and ACID (Part 3)
Performance trade-offs (Part 4)
Write-Ahead Logging (Part 5)
Monitoring tools (Part 6)

Think I missed out on a key topic? Please reach out to me.

Previous: Part 5 - Write-Ahead Logging Deep Dive

Next: Understanding PostgreSQL Checkpoints

References

PostgreSQL Fundamentals: Write-Ahead Logging Deep Dive (Part 5)

Wed, 15 Oct 2025 00:00:00 +0000

In Part 4, we learned about database performance trade-offs and why sequential I/O is preferred. Now let’s dive deep into Write-Ahead Logging (WAL), which is PostgreSQL’s solution to the durability problem.

This is Part 5 of a series on PostgreSQL internals.

What Problem Does WAL Solve?

Remember the durability dilemma from Part 3:

UPDATE users SET balance = 500 WHERE id = 1;
COMMIT;  -- Must survive a crash from this point forward

The naive approach would be:

Modify the data page in memory
Write the page to disk (random I/O, slow)
Acknowledge COMMIT

But this has problems:

Slow: Random writes to data files (10-100ms)
Fragile: Partial page writes during crash
No batching: Can’t combine multiple updates

WAL solves all three problems.

How WAL Works: The Big Picture

Instead of writing data pages immediately, PostgreSQL writes changes to a sequential log first:

1. Transaction modifies data
   ↓
2. Write change to WAL (sequential, fast)
   ↓
3. Acknowledge COMMIT (user sees success)
   ↓
4. Later: Apply changes to data files (background)

If PostgreSQL crashes before step 4, the WAL contains everything needed to reconstruct the changes.

The Write-Ahead Principle

Write-Ahead: Changes must be logged in WAL before the data page is written to disk.

This ensures crash recovery always works.

WAL File Structure

WAL is stored in 16 MB segment files:

# Find your data directory first
psql -c "SHOW data_directory;"

# WAL directory (use the path from above)
ls -lh /path/to/data/pg_wal/

# Example output:
-rw------- 1 postgres postgres 16M Oct 17 10:00 000000010000000000000001
-rw------- 1 postgres postgres 16M Oct 17 10:05 000000010000000000000002
-rw------- 1 postgres postgres 16M Oct 17 10:10 000000010000000000000003

Each file is exactly 16 MB and contains many WAL records.

WAL Segment Naming

000000010000000000000001
│      ││              │
│      ││              └─ Segment number (hex)
│      │└──────────────── High 32 bits of LSN
│      └───────────────── Timeline ID
└──────────────────────── Always starts with 00000001

As the database runs, PostgreSQL creates new segments sequentially.

LSN: Log Sequence Number

Every position in WAL has a unique identifier called an LSN (Log Sequence Number):

LSN format: 0/16A4B80
            │ │
            │ └─ Offset within segment (hex)
            └─── Segment file number (hex)

LSN is essentially a 64-bit offset into the infinite WAL stream.

-- Get current WAL write position
SELECT pg_current_wal_lsn();
-- Result: 0/16A4B80

-- Get current WAL insert position
SELECT pg_current_wal_insert_lsn();
-- Result: 0/16A4C00

LSNs increase monotonically. A higher LSN means a later point in time.

WAL Record Structure

Each change to the database generates a WAL record:

WAL Record:
┌─────────────────────────────┐
│ Record Header               │  ← Metadata (24 bytes)
│  - Total length             │
│  - Transaction ID           │
│  - Previous record pointer  │
│  - CRC checksum             │
├─────────────────────────────┤
│ Resource Manager Info       │  ← What kind of change?
│  - Heap, Btree, Sequence... │
├─────────────────────────────┤
│ Block References            │  ← Which pages changed?
│  - Page number              │
│  - Fork (main/FSM/VM)       │
├─────────────────────────────┤
│ Main Data                   │  ← The actual change
│  - Old values (for undo)    │
│  - New values (for redo)    │
└─────────────────────────────┘

Example WAL Record

When you run:

UPDATE users SET balance = 500 WHERE id = 1;

PostgreSQL generates a WAL record like:

Resource Manager: Heap (table data)
Block Reference: Relation 16385, Block 0
Old Tuple: (id=1, balance=400, name='Alice')
New Tuple: (id=1, balance=500, name='Alice')
Transaction ID: 1234
LSN: 0/16A4B80

This record contains everything needed to:

Redo: Apply the change (crash recovery)
Undo: Reverse the change (rollback, though PostgreSQL uses MVCC instead)

Types of WAL Records

Different operations generate different WAL record types:

Heap Records (Table Data)

INSERT INTO users VALUES (1, 'Alice');
-- WAL: HEAP_INSERT, block 0, tuple data [1, 'Alice']

UPDATE users SET name = 'Alice Smith' WHERE id = 1;
-- WAL: HEAP_UPDATE, block 0, old tuple, new tuple

DELETE FROM users WHERE id = 1;
-- WAL: HEAP_DELETE, block 0, tuple offset

Btree Records (Index Data)

CREATE INDEX idx_users_email ON users(email);
-- WAL: BTREE_INSERT for each index entry

UPDATE users SET email = 'newemail@example.com' WHERE id = 1;
-- WAL: BTREE_DELETE (old entry), BTREE_INSERT (new entry)

Transaction Records

COMMIT;
-- WAL: TRANSACTION_COMMIT, transaction ID, timestamp

ROLLBACK;
-- WAL: TRANSACTION_ABORT, transaction ID

Checkpoint Records

-- Checkpoint happens
-- WAL: CHECKPOINT, LSN, redo point, next XID, database state

How Crash Recovery Works

When PostgreSQL starts after a crash:

Step 1: Read Last Checkpoint

PostgreSQL reads pg_control to find the last completed checkpoint:

-- View control file info (requires pg_controldata tool)
-- $ pg_controldata $PGDATA

-- Shows:
Latest checkpoint location: 0/1500000
Prior checkpoint location:  0/1000000

Step 2: Find Redo Point

The checkpoint record contains the “redo point”, which is the LSN where recovery should start:

Checkpoint at LSN 0/1500000
Redo point: 0/1480000

This means:
- All changes before 0/1480000 are on disk
- Changes from 0/1480000 to crash point need replay

Step 3: Replay WAL

PostgreSQL reads WAL records from the redo point forward:

Read WAL record at 0/1480000:
  HEAP_UPDATE on block 5
  Apply: Load page 5, apply update

Read WAL record at 0/1480100:
  BTREE_INSERT on index block 10
  Apply: Load page 10, insert index entry

... continue until end of WAL ...

Each record is applied to reconstruct the database state.

Step 4: Reach Consistent State

When replay completes, the database is consistent up to the crash point. All committed transactions are present, all uncommitted transactions are absent (because COMMIT records weren’t written).

Handling Corrupted Pages

Crash recovery through WAL replay sounds straightforward, but there’s a subtle problem: what if the data pages themselves become corrupted during a crash? PostgreSQL needs a way to handle partial writes that leave pages in an inconsistent state.

The Problem

PostgreSQL writes 8 KB page to disk
Crash happens after 4 KB written
Page is corrupted (half old, half new)

Even with WAL, you can’t replay changes onto a corrupted page.

The Solution: Full Page Writes

After each checkpoint, the first modification to a page writes the entire page to WAL:

-- First update to a page after checkpoint
UPDATE users SET balance = 500 WHERE id = 1;

-- WAL record contains:
-- 1. Full 8 KB page image (FPW)
-- 2. The change record

Subsequent updates to the same page only log the change (until next checkpoint).

This ensures:

If page is partially written during crash, WAL contains a good copy
Recovery can restore the page from WAL, then apply changes

-- Check FPW setting
SHOW wal_log_hints;
-- or
SHOW full_page_writes;  -- Should be 'on'

Disabling full page writes improves performance but risks data corruption during crashes.

Managing WAL in Memory

WAL records need to reach disk to guarantee durability, but writing to disk on every change would be too slow. PostgreSQL uses an in-memory buffer to batch WAL writes efficiently.

WAL records aren’t written directly to disk. They go through WAL buffers:

Transaction generates WAL record
    ↓
WAL buffer (in memory, 16 MB)
    ↓
fsync to disk (when needed)

When WAL is Flushed

WAL is flushed to disk when:

Transaction commits:

COMMIT;  -- Forces fsync of WAL up to this point

WAL buffer fills:

SHOW wal_buffers;  -- Default: 16 MB
-- When buffer is full, flush to disk

Background writer:
```
Every 200ms, flush any unwritten WAL
```

-- Check WAL flush stats
SELECT * FROM pg_stat_wal;

-- Key metrics:
-- wal_write: Number of times WAL was written
-- wal_sync:  Number of times WAL was synced (fsync)
-- wal_bytes: Total bytes written to WAL

Beyond Crash Recovery

While WAL’s primary purpose is crash recovery, it also enables several advanced PostgreSQL features. By preserving a complete history of changes, WAL becomes the foundation for backup strategies and replication.

For point-in-time recovery and replication, you can archive completed WAL segments:

-- Enable archiving in postgresql.conf
archive_mode = on
archive_command = 'cp %p /mnt/wal_archive/%f'

-- PostgreSQL will archive each 16 MB segment when full

Archived WAL lets you:

Restore to any point in time
Set up streaming replication
Build read replicas

WAL Generation Rate

Different workloads generate WAL at different rates:

-- Read-only query (no WAL generated)
SELECT * FROM users;

-- Small insert (~100 bytes of WAL)
INSERT INTO users VALUES (1, 'Alice');

-- Large update with indexes (~1 KB of WAL)
UPDATE users SET email = 'newemail@example.com' WHERE id = 1;

-- Create index (MB of WAL)
CREATE INDEX idx_users_email ON users(email);

Monitoring WAL Generation

-- Current WAL position
SELECT pg_current_wal_lsn();
-- Result: 0/16A4B80

-- Wait 1 minute, check again
SELECT pg_current_wal_lsn();
-- Result: 0/18C7000

-- Calculate WAL generated
SELECT pg_size_pretty(
    pg_wal_lsn_diff('0/18C7000', '0/16A4B80')
);
-- Result: 2.1 MB generated in 1 minute

High WAL generation rate = more frequent checkpoints.

WAL and Performance

Understanding how WAL affects performance helps you make informed trade-offs between durability and speed. WAL introduces overhead in multiple areas, from write latency to disk space usage. Let’s look at some of the ways:

1. Write Performance

Every transaction writes to WAL:

Fast (sequential I/O)
But still I/O (slower than memory)

-- Synchronous commit (default, safe)
SET synchronous_commit = on;
-- Every COMMIT waits for WAL fsync

-- Asynchronous commit (faster, less safe)
SET synchronous_commit = off;
-- COMMIT returns immediately, fsync happens later
-- Risk: Lose last ~200ms of commits if crash

2. Disk Space

WAL segments accumulate:

Each 16 MB
Kept until checkpoint completes
Then recycled or archived

-- Check WAL directory size
SELECT pg_size_pretty(
    pg_stat_file('pg_wal', true).size
);

3. Checkpoint Frequency

More WAL generation → more frequent checkpoints:

-- If you generate 1 GB WAL per minute
-- And max_wal_size = 1GB
-- Checkpoints happen every minute (frequent!)

-- Solution: Increase max_wal_size
max_wal_size = 4GB

With the recommended solution, can you guess the trade-offs now? If you’re not sure, make sure to read the previous parts and revisit this post.

What’s Next?

Now that you understand how WAL works internally, we can explore the monitoring and administration tools PostgreSQL provides. In Part 6, we’ll learn about system views, log analysis, and how to use pg_waldump to inspect WAL records.

Previous: Part 4 - Performance Patterns and Trade-offs

References

PostgreSQL Fundamentals: Performance Patterns and Trade-offs (Part 4)

Tue, 14 Oct 2025 00:00:00 +0000

In Part 3, we learned about transactions and ACID properties. Now let’s explore the performance implications of building a durable database and the trade-offs PostgreSQL makes.

This is Part 4 of a series on PostgreSQL internals.

The Durability vs Performance Dilemma

Remember from Part 3: durability means committed data survives crashes. But guaranteeing durability is expensive.

-- When you commit
COMMIT;

-- PostgreSQL must ensure data is on disk
-- But disk writes are slow (See Part 1)
-- This creates a fundamental tension

Every database faces this challenge. The solution lies in clever I/O patterns.

Write Amplification: The Overhead

Write amplification is when you write more data to disk than the actual change requires.

Example: Updating One Field

-- You change one field
UPDATE users SET last_login = NOW() WHERE id = 123;

What actually happens:

Logical change: 8 bytes (a timestamp)
Physical write: Entire 8 KB page must be written

That’s 1,000x amplification (8 KB ÷ 8 bytes).

Why Write the Whole Page?

Disks don’t write 8 bytes at a time. The smallest writable unit is typically 512 bytes or 4 KB (disk sector/block size). PostgreSQL’s page size matches this reality.

You want to change: [X]
Must write: [XXXXXXXX] ← Entire page

Write Amplification with Indexes

It gets worse with indexes:

UPDATE users SET email = 'newemail@example.com' WHERE id = 123;

PostgreSQL must update:

The table page (8 KB)
The old index entry (mark as deleted)
The new index entry (insert)
Any other indexes on updated columns

One logical change = multiple page writes.

I/O Batching: This is the way

Instead of writing each change immediately, batch them:

-- Transaction 1
UPDATE users SET last_login = NOW() WHERE id = 1;
COMMIT;

-- Transaction 2
UPDATE users SET last_login = NOW() WHERE id = 2;
COMMIT;

-- Transaction 3
UPDATE users SET last_login = NOW() WHERE id = 3;
COMMIT;

If users 1, 2, and 3 are on the same page:

Without batching: Write the page 3 times
With batching: Write the page once with all changes

PostgreSQL batches writes in two ways:

WAL buffering: Buffer multiple WAL records before flushing
Checkpoint batching: Accumulate dirty pages, flush together

Introducing fsync

fsync is the system call that forces data to physical disk (not just OS cache). It’s slow but necessary for durability.

// Simplified PostgreSQL commit
write(wal_fd, wal_record);   // Write to OS buffer (fast)
fsync(wal_fd);               // Force to physical disk (slow!)
return COMMIT_SUCCESS;       // Now safe to acknowledge

fsync Performance

Without fsync:  ~1-2 microseconds (OS buffer)
With fsync:     ~1-2 milliseconds (physical disk)

That's a 1,000x difference!

If you commit 1,000 transactions per second and fsync each one:

1,000 commits × 2ms = 2,000ms = 2 seconds
You can only do 500 commits/second, not 1,000

Group Commit: Batching fsyncs

PostgreSQL uses group commit to amortize fsync cost:

Transaction 1 commits → Write WAL record → Wait for fsync
Transaction 2 commits → Write WAL record → Wait for fsync
Transaction 3 commits → Write WAL record → Wait for fsync
                                             ↓
                                    Single fsync for all three!

Multiple transactions share the same fsync operation:

Transaction 1 arrives, starts fsync
Transactions 2 and 3 arrive while fsync is happening
All three complete when fsync finishes

-- See group commit in action
SELECT
    (SELECT sum(xact_commit) FROM pg_stat_database) AS total_commits,
    wal_sync,
    round((SELECT sum(xact_commit) FROM pg_stat_database)::numeric / wal_sync, 2) AS commits_per_sync
FROM pg_stat_wal;

-- Example output:
-- total_commits | wal_sync  | commits_per_sync
-- --------------+-----------+-----------------
-- 68245891203   | 204512847 | 33.37

-- commits_per_sync of 33.37 means ~33 commits share each fsync
-- Group commit is batching effectively

Sequential vs Random I/O Revisited

From Part 1, we know sequential I/O is faster. Let’s see why this matters for database design.

Random Write Pattern (Slow)

Updating scattered rows:

UPDATE users SET last_login = NOW() WHERE id IN (1, 5000, 10000, 50000);

-- These rows are likely on different pages
-- PostgreSQL must write multiple pages scattered across disk

Disk head (or SSD controller) jumps around:

Write page 1    → seek →
Write page 5000 → seek →
Write page 10000 → seek →
Write page 50000

Each seek adds latency (5-10ms on HDD).

Sequential Write Pattern (Fast)

WAL writes sequentially:

-- Same updates, but WAL records are written sequentially
[Record 1][Record 2][Record 3][Record 4]...

-- No seeking, just append

This is why PostgreSQL uses WAL:

Changes go to WAL (sequential, fast)
Data pages updated later (random, slow but batched)

The Write-Back Cache Pattern

PostgreSQL uses a write-back cache pattern:

1. Change happens → Write to WAL (fast, sequential)
2. Change happens → Update page in memory (fastest)
3. Later → Flush dirty pages to disk (slow, random, but batched)

This is the same pattern CPUs use with L1/L2 cache and main memory. For more on CPU cache hierarchies, see What Every Programmer Should Know About Memory.

Dirty Pages Accumulate

-- Transaction 1
UPDATE users SET name = 'Alice' WHERE id = 1;
COMMIT;
-- Page 1 is now "dirty" (in memory, not yet written to data file)

-- Transaction 2
UPDATE users SET email = 'bob@example.com' WHERE id = 2;
COMMIT;
-- Page 1 is STILL dirty (not written yet)

-- Checkpoint happens
-- Now page 1 with BOTH changes is written once

Dirty pages stay in shared_buffers until a checkpoint.

Checkpoint (Trade-off alert)

Checkpoints must balance (for a complete understanding of checkpoints, see Understanding PostgreSQL Checkpoints):

Frequency: How often to flush dirty pages?
- Too frequent: Excessive I/O overhead
- Too infrequent: Long crash recovery time
Duration: How fast to write dirty pages?
- Too fast: I/O spike, queries slow down
- Too slow: Checkpoint takes too long

PostgreSQL’s solution:

-- Checkpoint configuration
checkpoint_timeout = 5min          -- Max time between checkpoints
max_wal_size = 1GB                 -- Max WAL before forcing checkpoint
checkpoint_completion_target = 0.9 -- Spread writes over 90% of interval

checkpoint_completion_target = 0.9 means:

Checkpoint interval is 5 minutes
Spread writes over 4.5 minutes (90%)
Write ~100 MB per minute instead of 1 GB in one burst

Visualizing Checkpoint I/O

Without spreading (checkpoint_completion_target = 0):
I/O │     ┌────┐
    │     │    │
    │─────┘    └───────────────
    Time →

With spreading (checkpoint_completion_target = 0.9):
I/O │     ┌────────────┐
    │    ╱              ╲
    │───┘                └─────
    Time →

Spreading reduces I/O spikes but increases checkpoint duration.

Write Amplification in PostgreSQL

Let’s calculate actual write amplification:

-- Simple update
UPDATE users SET status = 'active' WHERE id = 123;

Writes required:

WAL record: ~100 bytes (change record + metadata)
Data page: 8 KB (entire page)
Index page: 8 KB (if updating indexed column)

Total: ~16 KB written for a ~10 byte logical change.

Amplification factor: 1,600x

But this gets batched:

100 updates to same page → Write page once at checkpoint
Effective amplification: ~160x instead of 1,600x

Understanding the Trade-offs

PostgreSQL’s design creates fundamental trade-offs between durability, performance, and resource usage. Let’s explore the key decisions database administrators must make.

The Memory Trade-off

More memory (shared_buffers) = better batching:

-- Small shared_buffers (128 MB)
-- Pages evicted quickly
-- Less batching opportunity
-- More checkpoints needed

-- Large shared_buffers (8 GB)
-- Pages stay in memory longer
-- More batching opportunity
-- Fewer checkpoints needed

But memory is expensive and limited.

Crash Recovery Time Trade-off

The longer between checkpoints, the more WAL to replay after a crash:

Checkpoint every 5 min → ~5 min of WAL to replay
Checkpoint every 30 min → ~30 min of WAL to replay

Larger checkpoint interval = longer recovery time

PostgreSQL balances:

Performance (less frequent checkpoints)
Recovery time (more frequent checkpoints)

Default checkpoint_timeout = 5min is a reasonable middle ground.

Practical Example: Measuring Write Amplification

-- Create test table
CREATE TABLE write_test (id SERIAL PRIMARY KEY, data TEXT);

-- Record starting WAL position
SELECT pg_current_wal_lsn() AS start_lsn;
-- Result: 0/1000000

-- Insert 1000 rows
INSERT INTO write_test (data)
SELECT repeat('x', 100)
FROM generate_series(1, 1000);

-- Record ending WAL position
SELECT pg_current_wal_lsn() AS end_lsn;
-- Result: 0/1500000

-- Calculate WAL generated
SELECT pg_wal_lsn_diff('0/1500000', '0/1000000') AS wal_bytes;
-- Result: 5242880 (5 MB)

-- Logical data size
SELECT pg_size_pretty(pg_relation_size('write_test'));
-- Result: 128 KB

-- Write amplification: 5 MB WAL / 128 KB data = ~40x

The WAL is much larger than the data due to:

Transaction metadata
Index updates
MVCC version information

What’s Next?

Now that you understand the performance trade-offs databases face, we can dive deep into Write-Ahead Logging. In Part 5, we’ll explore exactly how WAL works, what’s inside WAL records, and how it enables both durability and crash recovery.

Previous: Part 3 - Transactions and ACID

References

PostgreSQL Fundamentals: Transactions and ACID (Part 3)

Mon, 13 Oct 2025 00:00:00 +0000

In Part 2, we learned how PostgreSQL stores data in pages. Now let’s explore transactions, which are the mechanism that keeps your data consistent even when multiple users access it simultaneously or the system crashes.

This is Part 3 of a series on PostgreSQL internals.

What Is a Transaction?

A transaction is a sequence of database operations that are treated as a single unit of work. Either all operations succeed, or none do.

-- Start a transaction
BEGIN;

-- Multiple operations
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;

-- Commit: Make changes permanent
COMMIT;

If anything goes wrong between BEGIN and COMMIT, you can ROLLBACK to undo everything:

BEGIN;

UPDATE accounts SET balance = balance - 100 WHERE id = 1;
-- Oops, error or change your mind
ROLLBACK;  -- First UPDATE is undone

The ACID Properties

ACID is an acronym for the four properties that guarantee database transactions are processed reliably:

1. Atomicity: All or Nothing

A transaction either completes fully or has no effect at all.

BEGIN;

-- Transfer $100 from Alice to Bob
UPDATE accounts SET balance = balance - 100 WHERE user_id = 1;  -- Alice
UPDATE accounts SET balance = balance + 100 WHERE user_id = 2;  -- Bob

COMMIT;

If the database crashes after the first UPDATE but before COMMIT:

Both updates are undone
No money is lost or created
The system is as if the transaction never happened

This is atomicity. The transaction is atomic (indivisible).

2. Consistency: Rules Are Always Enforced

The database moves from one valid state to another valid state. Constraints are always respected.

CREATE TABLE accounts (
    user_id INT PRIMARY KEY,
    balance DECIMAL CHECK (balance >= 0)  -- Constraint: no negative balances
);

BEGIN;
UPDATE accounts SET balance = balance - 200 WHERE user_id = 1;
-- If this would make balance negative, transaction fails
COMMIT;
-- ERROR: new row violates check constraint "accounts_balance_check"

The database prevents you from violating constraints. This is consistency.

3. Isolation: Transactions Don’t Interfere

When multiple transactions run concurrently, each transaction sees a consistent snapshot of the database, as if it’s running alone.

-- Transaction 1
BEGIN;
SELECT balance FROM accounts WHERE user_id = 1;  -- Returns 500
-- ... doing some work ...

-- Transaction 2 (running at the same time)
BEGIN;
UPDATE accounts SET balance = 1000 WHERE user_id = 1;
COMMIT;

-- Back to Transaction 1
SELECT balance FROM accounts WHERE user_id = 1;  -- Still returns 500!
COMMIT;

Transaction 1 sees a consistent view even though Transaction 2 modified the data. This is isolation.

PostgreSQL provides multiple isolation levels:

-- Set isolation level
BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;
-- or
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
-- or
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;

Each level provides different guarantees about what changes from other transactions you can see.

4. Durability: Committed Data Survives

Once a transaction commits, the changes are permanent, even if the system crashes immediately after.

BEGIN;
INSERT INTO orders (user_id, total) VALUES (1, 99.99);
COMMIT;  -- From this point, the data MUST survive

-- Even if PostgreSQL crashes here, the order exists when it restarts

This is durability, which is the hardest property to implement and the reason Write-Ahead Logging exists.

How PostgreSQL Implements Durability

When you commit a transaction, PostgreSQL must ensure the data survives a crash. But writing to disk is slow (remember Part 1?).

PostgreSQL’s solution: Write-Ahead Logging (WAL)

The Durability Problem

COMMIT;  -- User expects this to be permanent

-- But writing all changed pages to disk takes time:
-- - Multiple random disk writes
-- - Could be many pages scattered across disk
-- - Takes 10-100ms

-- What if the system crashes during these writes?

The WAL Solution

Instead of writing data pages directly:

Write changes to WAL (sequential, fast)
Acknowledge COMMIT to user
Write data pages later (in background)

Transaction commits:
    ↓
Write to WAL (sequential, fast: 1-2ms)
    ↓
COMMIT acknowledged ✅
    ↓
Later: Write dirty pages to data files

The WAL is a sequential log of all changes. Sequential writes are fast (see Part 1).

WAL Example

BEGIN;
UPDATE accounts SET balance = 500 WHERE user_id = 1;
COMMIT;

-- Behind the scenes:
-- 1. Change recorded in WAL:
--    "Transaction 12345: Change page 42, offset 10, old value 400, new value 500"
-- 2. WAL flushed to disk (fsync)
-- 3. COMMIT returns to user
-- 4. Dirty page stays in memory (shared_buffers)
-- 5. Background writer eventually writes page 42 to data file

If PostgreSQL crashes after step 3:

WAL has the change recorded
On restart, PostgreSQL replays the WAL
The change is reconstructed
Durability preserved ✌🏽

Isolation Levels in Detail

PostgreSQL offers four isolation levels, each with different trade-offs. Each level provides different guarantees about what changes from other transactions you can see. For a complete reference, see the PostgreSQL Documentation: Transaction Isolation.

Read Uncommitted (Not Really Implemented)

BEGIN TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;

In PostgreSQL, this behaves the same as Read Committed. PostgreSQL doesn’t allow reading uncommitted data.

Read Committed (Default)

BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;

Each query sees data committed before the query started:

-- Transaction 1
BEGIN;
SELECT COUNT(*) FROM orders;  -- Returns 100

-- Transaction 2 (different session)
BEGIN;
INSERT INTO orders VALUES (...);
COMMIT;

-- Back to Transaction 1
SELECT COUNT(*) FROM orders;  -- Returns 101 (sees new commit)
COMMIT;

You might see different data in each query within the same transaction.

Repeatable Read

BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;

Sees a consistent snapshot of data from when the transaction started:

-- Transaction 1
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SELECT COUNT(*) FROM orders;  -- Returns 100

-- Transaction 2
BEGIN;
INSERT INTO orders VALUES (...);
COMMIT;

-- Back to Transaction 1
SELECT COUNT(*) FROM orders;  -- Still returns 100 (snapshot isolation)
COMMIT;

The entire transaction sees data as it was at the start.

Serializable

BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;

Strongest isolation. Guarantees transactions behave as if executed one at a time.

If concurrent transactions would produce inconsistent results, PostgreSQL aborts one:

-- Transaction 1: Serializable
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SELECT SUM(balance) FROM accounts;  -- 1000

-- Transaction 2: Serializable
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
INSERT INTO accounts (user_id, balance) VALUES (3, 100);
COMMIT;

-- Back to Transaction 1
INSERT INTO audit_log (total) VALUES (1000);  -- Uses old sum
COMMIT;
-- ERROR: could not serialize access due to read/write dependencies

PostgreSQL detects that Transaction 1’s view is stale and aborts it.

MVCC: How PostgreSQL Implements Isolation

PostgreSQL uses Multi-Version Concurrency Control (MVCC). Instead of locking, it keeps multiple versions of rows.

-- Original data
SELECT * FROM accounts WHERE user_id = 1;
-- balance: 500, xmin: 100, xmax: 0

-- Transaction 1 updates
BEGIN;  -- Transaction ID: 101
UPDATE accounts SET balance = 600 WHERE user_id = 1;
COMMIT;

-- Physical storage now has TWO versions:
-- Old: balance 500, xmin: 100, xmax: 101
-- New: balance 600, xmin: 101, xmax: 0

When you query, PostgreSQL uses transaction IDs to determine which version is visible:

Transactions started before 101: See balance 500
Transactions started after 101: See balance 600

No locks needed for reads and writes on different versions.

Practical Example: Transaction Behavior

-- Create test table
CREATE TABLE transfers (
    id SERIAL PRIMARY KEY,
    from_account INT,
    to_account INT,
    amount DECIMAL
);

-- Transaction that demonstrates atomicity
BEGIN;
INSERT INTO transfers (from_account, to_account, amount) VALUES (1, 2, 100);
SELECT * FROM transfers WHERE from_account = 1;  -- Shows the new row
ROLLBACK;
SELECT * FROM transfers WHERE from_account = 1;  -- Row is gone (atomicity)

-- Transaction that demonstrates isolation
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SELECT COUNT(*) FROM transfers;  -- Let's say it's 0

-- In another session, insert a row and commit
-- (Open another terminal)
-- BEGIN; INSERT INTO transfers VALUES (DEFAULT, 1, 2, 50); COMMIT;

-- Back in first session
SELECT COUNT(*) FROM transfers;  -- Still 0 (isolation)
COMMIT;
SELECT COUNT(*) FROM transfers;  -- Now 1 (after commit, you see changes)

What’s Next?

Now that you understand transactions and why durability requires persisting data to disk, we can explore the performance implications of different I/O patterns. In Part 4, we’ll learn about write amplification, I/O batching, and why PostgreSQL makes the design choices it does.

Previous: Part 2 - How Databases Store Data

Prateek Codes - Building Scalable Backend Systems

Ruby::Box Practical Guide: Use Cases and Integration Patterns (Part 2)

Use Case: Plugin Systems

Use Case: Multi-Tenant Configuration

Use Case: Running Multiple Gem Versions

Use Case: Isolated Monkey Patches for Testing

Use Case: Shadow Testing

Working Around Native Extension Issues

Working Around ActiveSupport Issues

Performance Considerations

When to Use Ruby::Box

Migration Strategy

What’s Next for Ruby::Box

References

Ruby 4.0 Introduces Ruby::Box for In-Process Isolation (Part 1)

The Problem with Shared Namespaces

Ruby 4.0: Enter Ruby::Box

Understanding Box Types

The Ruby::Box API

What Gets Isolated

Enabling Ruby::Box

Important Limitations

File Scope Execution

References

Rails 8.2 makes enqueue_after_transaction_commit the default

The Problem with Opt-In

Rails 8.2 Changes the Default

Opting Out

Why the Global Config Was Restored

When This Matters

Conclusion

References

Rails 7.2 adds enqueue_after_transaction_commit to prevent job race conditions

Before

The Workaround

Rails 7.2

Configuration Options

Checking Enqueue Status

Model Callbacks Simplified

When to Disable

Update: Rails 8.2 Makes This the Default

Conclusion

References

Rails 8.2 introduces Rails.app.creds for unified credential management

Before

Rails 8.2

Nested Keys

Dynamic Defaults

ENV-Only Access

Custom Credential Sources

Rails.app Alias

Conclusion

References

Understanding PostgreSQL Checkpoints: From WAL to Disk

Write-Ahead Logging: The Foundation

What Happens During a Checkpoint

The Checkpoint Process

What Triggers a Checkpoint?

Checkpoint Impact on Performance

Monitoring Checkpoint Activity

Check Checkpoint Statistics

Monitor WAL Generation Rate

Tuning Checkpoint Behavior

Conclusion

References

PostgreSQL Fundamentals: Monitoring and Administration Tools (Part 6)

System Views: Your Window Into PostgreSQL

pg_stat_checkpointer: Checkpoint Statistics (PostgreSQL 17+)

Interpreting pg_stat_checkpointer

pg_stat_wal: WAL Activity

Calculating WAL Generation Rate

pg_stat_database: Database-Level Stats

pg_stat_statements: Query-Level WAL Generation

pg_stat_activity: Live Connections

PostgreSQL Logs: Checkpoint and WAL Messages

Reading Checkpoint Logs

pg_waldump: Inspecting WAL Records

Filtering WAL Records

Understanding WAL Record Output

pg_control: Database Cluster State

When to Use `Ruby::Box`

What’s Next for `Ruby::Box`