Linux 7.0 Preemption Mechanism Change Triggers PostgreSQL Performance Collapse

Hidden Risks Behind the Kernel Version Leap

Recently, the Linux kernel officially entered the 7.0 major version. However, this seemingly routine version number bump has triggered a far-reaching technical incident — PostgreSQL databases exhibited significant performance regression on the new kernel. The root cause was traced to a critical change in the kernel's preemption mechanism, sparking widespread discussion across the community.

For infrastructure supporting AI training data management, vector databases, and large model inference services, the impact of this regression is not to be underestimated.

What Is Preemption and Why Does It Matter?

Preemption is one of the core concepts in OS kernel scheduling. In simple terms, it determines whether the kernel can "interrupt" a task mid-execution to switch to another higher-priority task.

Historically, the Linux kernel has offered several preemption models:

• PREEMPT_NONE: No active preemption; throughput-first, suitable for server scenarios
• PREEMPT_VOLUNTARY: Explicit preemption points set in kernel code, balancing latency and throughput
• PREEMPT_FULL: Allows preemption of kernel code almost at any time; lowest latency but potentially reduced throughput

Database systems like PostgreSQL are extremely sensitive to scheduling behavior. They typically run under PREEMPT_NONE or PREEMPT_VOLUNTARY modes, relying on stable, predictable CPU time slices to complete transaction processing. Any subtle change in scheduling policy can intensify lock contention, cause context switch frequency to spike, and ultimately manifest as increased query latency and decreased overall throughput.

Technical Details of the Regression

In Linux 7.0, the kernel scheduling subsystem introduced a "lazy preemption" (PREEMPT_LAZY) mechanism, designed to unify the previously fragmented preemption models with a more elegant dynamic approach replacing compile-time static selection. This design philosophy evolved from the earlier PREEMPT_DYNAMIC concept — allowing the kernel to adaptively adjust preemption behavior at runtime based on workload characteristics.

However, community developers quickly discovered that this change exhibited severe performance degradation under PostgreSQL's typical workloads. Specific symptoms included:

• Transaction throughput decline: In high-concurrency OLTP scenarios, TPS (transactions per second) dropped by double-digit percentages
• Tail latency spikes: P99 latency deteriorated significantly, even doubling in some scenarios
• Context switch surge: The kernel preempted database worker threads at unnecessary moments with high frequency

The core issue was that the new lazy preemption mechanism's heuristic strategy for determining "when to preempt" made incorrect judgments for workloads like databases that hold short-lived spinlocks and perform frequent memory operations. The kernel incorrectly assumed these threads could be safely interrupted, but in reality, frequent preemption led to extended lock hold times and cache invalidation, creating a vicious cycle.

Heated Community Discussion

This incident simultaneously sparked intense discussion in both the Linux kernel and PostgreSQL communities.

Several senior developers pointed out that this was not the first time a kernel scheduling change had affected database performance. Historically, multiple adjustments to the CFS (Completely Fair Scheduler) had triggered similar regressions. One community member commented: "Databases are the most demanding litmus test for kernel scheduling policies. Any theoretical optimization that hasn't been thoroughly validated under database workloads could be a ticking time bomb."

Other developers defended the kernel team, arguing that the direction of unifying preemption models was correct but needed further refinement in implementation details. The core idea behind lazy preemption — deferring preemption decisions until truly necessary — is sound in itself; the problem lies in the criteria for "truly necessary" needing calibration for different workloads.

Notably, multiple commenters mentioned that this issue affects not only PostgreSQL but could similarly impact other database systems such as MySQL and MariaDB. PostgreSQL, due to its multi-process architecture (as opposed to multi-threaded), is more sensitive to scheduling changes and therefore exposed the problem first.

Potential Impact on AI Infrastructure

In the context of today's booming AI technology, the impact of this regression extends far beyond traditional database applications:

• Vector databases: PostgreSQL-based vector retrieval extensions like Pgvector are widely used in RAG (Retrieval-Augmented Generation) systems, and performance regression directly affects the response speed of large model applications
• Training data management: Numerous AI training pipelines rely on PostgreSQL for metadata and data lineage management
• Model serving platforms: MLOps platforms such as MLflow and Kubeflow use PostgreSQL as backend storage, and scheduling performance degradation affects model deployment efficiency
• GPU server environments: If Linux servers running AI training tasks upgrade to the 7.0 kernel, not only databases but also CUDA task scheduling may exhibit subtle behavioral changes

Fix Progress and Temporary Workarounds

The Linux kernel team has confirmed this regression and is actively developing patches. Temporary mitigation measures include:

1. Kernel boot parameter rollback: Force the old preemption mode by setting the preempt=voluntary boot parameter
2. Delay upgrades: For production database servers, it is recommended to remain on 6.x kernels for the time being
3. Scheduler parameter tuning: Adjust parameters such as sched_min_granularity_ns to reduce unnecessary preemption

The community expects an official fix to be released in version 7.1 or a subsequent patch release for 7.0.

Deeper Lessons

This incident once again reminds us that in the AI era, every layer of the software stack matters. While the industry focuses heavily on model architecture innovation and the compute power race, the operating system kernel — the "invisible foundation" — still profoundly influences the stability and efficiency of the entire technology ecosystem.

As one community developer summarized: "While chasing trillion-parameter models, let's not forget to check whether the ground beneath our feet remains solid."

For operations teams and infrastructure engineers, this is also an important reminder — before major kernel version upgrades, be sure to conduct thorough performance benchmarking under production-equivalent workloads, especially for scheduling-sensitive workloads such as databases and AI inference services.