Reading Notes

Personal reading notes. Original paper: arXiv:2606.14415v1.

Paper Reading Notes: CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Paper ID: 2606.14415v1
Title: CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
Authors: Ayoub Belouadah, Sylvain Kubler, Yves Le Traon
Conference: ICML 2026 (Proceedings of the 43rd International Conference on Machine Learning)

1. Research Background and Motivation

Safe Reinforcement Learning (Safe RL) aims to maximize expected returns while satisfying safety constraints, typically modeled as a Constrained Markov Decision Process (CMDP). Although existing Primal-Dual methods exhibit good scalability in deep reinforcement learning, the paper points out that they often suffer from the “delayed constraint correction” problem.

The authors argue that current mainstream methods have the following limitations:

• Primal-Dual Methods: While computationally efficient, they suffer from the well-known “dual-lag effect”. When Lagrange multipliers are small, unsafe behavior may occur; when multipliers are too large, it leads to excessive conservatism. This dynamic often results in oscillatory behavior during learning and prolonged safety violations near constraint boundaries.
• Penalty Methods: These require careful tuning of penalty factors. If the factor is too small, it leads to persistent constraint violations; if too large, it results in an overly conservative policy.
• Ignoring Local Constraint Sensitivity: Existing methods typically penalize constraint violations uniformly, ignoring the local sensitivity and steepness of the constraint function. The paper notes that flat regions may require stronger correction updates to restore feasibility, while steep regions require more cautious steps to avoid overshooting.

2. Core Method: CSPO

To address the above issues, the authors propose a first-order primal-dual safe reinforcement learning algorithm named Constraint-Sensitive Policy Optimization (CSPO).

2.1 Core Algorithm Idea

The core of CSPO lies in incorporating local constraint sensitivity into the policy update. The method adds a constraint-sensitive correction term to the primal objective function, based on the shortest signed distance to the safety boundary.

The paper argues that by scaling the correction term using the norm of the constraint gradient, CSPO can adaptively adjust the intensity of constraint enforcement according to local constraint sensitivity. Specifically:

• In regions where the constraint surface is steep (large gradient norm), the correction strength is weakened to prevent overshooting and oscillation.
• In regions where the constraint surface is flat (small gradient norm), the correction strength is enhanced to achieve efficient feasibility restoration.

2.2 Mathematical Derivation and Update Rules

Within the framework of constrained policy optimization, the paper defines a surrogate cost constraint $g(\theta)$. When a constraint violation occurs, the algorithm seeks the minimal parameter update $\Delta\theta$ to return to the safety boundary.

By solving the problem of minimizing the update norm, the authors derive a relationship where the update magnitude is inversely proportional to the constraint gradient norm. To accommodate the instability of stochastic gradient estimation, CSPO does not force a return to the boundary in one step but enforces a proportional reduction (fractional decrease) in the violation degree. The final update magnitude formula reflects the utilization of local geometric structure, where the update step size depends on the ratio of the current violation degree $g(\theta_k)$ to the constraint gradient norm $\|\nabla_\theta g(\theta_k)\|$.

3. Theoretical Properties and Geometric Intuition

The paper explains the advantages of CSPO over traditional primal-dual methods through geometric intuition.

3.1 Geometric Intuition (Corresponding to Figure 1)

Although the image cannot be directly displayed, the paper describes the geometric meaning of Figure 1 in detail:

• Dashed lines in the figure represent the contours of the objective function, dotted lines represent the contours of the constraint function, the purple curve represents the safety boundary $g(\theta)=0$, and the shaded region represents the infeasible region.
• Starting from an infeasible iteration point $\pi_0$, standard Lagrangian updates (Primal-dual) may overshoot and cause oscillations in highly sensitive regions (steep boundaries).
• In contrast, CSPO scales the constraint correction by $|\nabla g(\theta)|$, enabling stable recovery towards feasibility while minimizing return degradation.

3.2 Theoretical Guarantees

The authors emphasize that CSPO remains equivalent to the original constrained problem in terms of the feasible set and optimal solution. This means that introducing the constraint-sensitive correction does not alter the essential solution space of the problem but optimizes the convergence path. The method preserves the KKT solutions of the original constrained problem, ensuring theoretical correctness.

4. Experimental Evaluation and New Metrics

To more comprehensively evaluate the dynamics of safety recovery, the paper introduces three complementary evaluation metrics:

1. Time-To-Safety (TTS): Measures the time required to restore feasibility, i.e., the speed from violation to satisfying constraints again.
2. Reward Preservation (RP): Quantifies the degree of return preserved during the recovery process.
3. Violation Frequency (VF): Captures the frequency with which safety constraints are violated throughout the training process.

4.1 Experimental Results

Experiments were conducted on navigation and locomotion benchmarks. The results show:

• CSPO achieves faster safety recovery compared to state-of-the-art (SOTA) primal-dual methods and penalty methods.
• During the recovery process, CSPO maintains higher returns better.
• Ultimately, CSPO achieves higher constrained returns, demonstrating its superiority in balancing safety and performance.

5. Summary

The CSPO algorithm proposed in this paper effectively solves the delayed correction and oscillation problems in traditional safe reinforcement learning algorithms by introducing local constraint sensitivity.

• Main Contribution: Proposed the first-order primal-dual safe RL algorithm CSPO, which uses the constraint gradient norm to adaptively adjust correction strength.
• Theoretical Value: Proved that the algorithm preserves the feasible set and optimal solution of the original problem.
• Evaluation Innovation: Proposed three new metrics (TTS, RP, VF) to quantify safety recovery dynamics.
• Practical Effect: Demonstrated faster and more stable safety convergence performance in continuous control benchmarks.

The authors believe that CSPO, while maintaining the simplicity of primal-dual methods, explicitly utilizes first-order constraint information, providing a more robust optimization path for safe reinforcement learning.