Reading Notes

Personal reading notes. Original paper: arXiv:2606.13604v1.

arxiv_id: 2606.13604v1

arXiv Reading Note: Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch

1. Paper Overview

This paper introduces a reinforcement learning (RL) system deployed in DoorDash’s large-scale food delivery marketplace. The system utilizes delayed marketplace feedback signals to adaptively adjust dispatch objective weights. The authors propose a method different from directly replacing the combinatorial assignment optimizer: through a learned store-level policy, a discrete multiplier is selected to adjust the trade-off between “delivery quality” and “batching efficiency” in the optimizer.

The core contribution of the paper lies in demonstrating how to utilize world feedback from real economic and logistics systems to safely adapt decision policies online while maintaining production feasibility and operational safety. Through switchback experiments in the production environment, the offline-trained policy increased the batching rate and reduced courier-side time costs without compromising customer-facing delivery quality.

2. Core Problem and Motivation

In the three-sided market of food delivery (customers, merchants, riders), dispatching is a sequential decision problem embedded in a complex environment. Every decision affects customer delivery quality, merchant congestion, rider availability, batching opportunities, and cost-effectiveness.

Main Challenges:

1. Delayed Feedback: The effect of decisions is not evaluated immediately but through delayed operational outcomes (e.g., delivery speed, rider utilization, merchant congestion).
2. Multi-Objective Trade-off: Batching can improve rider efficiency, but rapid execution can reduce latency and improve customer experience. In production environments, this trade-off is typically controlled by static heuristic weights, which are usually globally adjusted and manually updated, making them fragile when facing local, time-varying conditions.
3. Coupled Feedback: Dispatch decisions jointly affect customer quality, merchant congestion, and rider utilization; the feedback is coupled.

Solution Approach: The authors did not attempt to replace the existing combinatorial assignment optimizer directly with RL. Instead, they let the RL agent control a narrow objective weight interface. In each dispatch cycle, the store-level RL agent observes the local market state and selects a discrete multiplier applied to the optimizer objectives. This constrained action space preserves existing optimization and operational safeguards while enabling local adaptation.

3. Method Design

3.1 System Architecture

The system proposed in the paper contains two nested decision layers. The inner layer is the optimizer, responsible for mapping orders, riders, constraints, and objective weights to rider-order assignments. The outer layer is the Objective Weight Adaptation agent (OWA-RL), which learns from market outcomes and selects store-level objective weight multipliers before optimization.

As shown in Figure 1, the agent objective weight adaptation loop for production dispatch is illustrated. During online service, the policy agent observes local states and selects an objective weight multiplier, which parameterizes the assignment optimizer. The optimizer remains responsible for feasible rider-order assignment decisions. During offline learning, logged runs are combined with delayed market outcomes to construct transition data tuples for policy training and deployment.

3.2 Problem Formalization

The authors formalize the outer adaptation problem as a centralized offline-trained decentralized multi-agent Markov Decision Process (MDP).

• Action: The action space $A = \{0.8, 0.9, 1.0, 1.1, 1.2\}$. This is a discrete multiplier for the baseline ASAP weight. Store $i$ selects $a_t^i \in A$, producing an adapted weight $\lambda_t^i = a_t^i \lambda_0$.
  • Lower weights make batching or more efficient routes more attractive in optimization.
  • Higher weights emphasize faster order completion, potentially leading the optimizer to assign nearby orders to different riders rather than batching them.
  • The neutral action $a_t^i = 1.0$ restores the static production baseline.
• State: Store-level state $s_t^i = [d_t^i, sup_t^i, cwt_t^i]$, refreshed once per assignment run.
  • $d_t^i$: Number of pending deliveries.
  • $cwt_t^i$: Median rider wait time.
  • $sup_t^i$: Localized supply pressure features. To expose store-level variations masked by regional supply signals, the authors rescale regional features by effective rider supply.
• Reward: Rewards are calculated from delayed delivery outcomes and aggregated to the regional level to capture network effects between nearby stores and riders.

3.3 Reward Design and Delayed Attribution

Reward construction relies on logistics time components. As shown in Figure 2, ASAP measures customer-facing delivery duration, while XCAT measures excess rider active time beyond the direct route (CAT).

Specific definitions are as follows:

• $ASAP_j = t_{drop}^j - t_{create}^j$ (Delivery completion time - Order creation time)
• $CAT_j = t_{drop}^j - t_{acc}^j$ (Delivery completion time - Rider acceptance time)
• $XCAT_j = CAT_j - T_{direct}^j$ (Excess rider time - Direct pickup and drop-off time for single service)

The regional reward $r_{g(i)}^t$ is defined as the average negative weighted sum of the set of deliveries attributed to the region in decision cycle $t$: $r_{g(i)}^t = -\frac{1}{|D_{g(i)}^t|} \sum_{j \in D_{g(i)}^t} (\alpha \cdot ASAP_j + \beta \cdot XCAT_j)$ where ASAP captures customer-side latency, XCAT captures extra effort on the rider side, and regional aggregation captures network effects.

3.4 Offline Training Algorithm

The policy $\pi_\theta(a|s)$ is a shared store-level policy. Stores independently execute the policy using local states, while the training pool aggregates experiences across stores and uses regional rewards to capture cross-store network effects.

The authors use offline Double DQN targets to train the discrete action value function $Q_\theta(s, a)$. To reduce overestimation of out-of-distribution values, the authors add a discrete Conservative Q-Learning (CQL) regularization term.

The final training objective is: $L(\theta) = L_{DDQN}(\theta) + \eta L_{CQL}(\theta)$ where Double DQN reduces maximization bias, and the conservative penalty prevents unsupported actions from obtaining high values, thereby improving offline training stability before deployment.

4. Experiments and Results

4.1 Experimental Setup

Before deployment, the authors used offline reward reweighting diagnostics to test whether the learned policy responds directionally to differently defined market feedback. $\alpha = 0.9$ was selected to produce a more balanced action distribution, avoiding collapse towards aggressive batching or speed priority.

Evaluation employed a global switchback experiment, comparing OWA-RL against the DoorDash production baseline. The randomized units were approximately 4,000 geographic regions.

4.2 Policy Behavior Analysis

The paper shows the predicted action distribution of the policy under different reward weight settings. As shown in Figure 5, changing reward weights alters the distribution of selected actions, verifying the policy’s sensitivity to feedback signals.

Additionally, Figure 3 shows how the policy shifts probability mass under different conditions. This indicates that the policy can dynamically adjust strategies based on market states rather than fixedly selecting a specific action.

4.3 Production Deployment Results

In the production environment, the system performs hundreds of millions of inferences daily with a cycle of 20 seconds. Figure 6 shows the daily distribution of selected ASAP weight multipliers. Results show that the policy can select multipliers different from the baseline (1.0) during actual operation to dynamically adjust dispatch behavior based on real-time conditions.

Main Results:

• Increased Batching: The offline-trained policy increased batching rates.
• Reduced Costs: Courier-side time costs were reduced.
• Quality Maintained: Customer-facing delivery quality was not compromised.

5. Summary

This paper proposes a method for multi-agent reinforcement learning using delayed feedback in a real three-sided market environment. The authors’ main contributions include:

1. Introducing an RL architecture that adapts dispatch objective weights through a low-dimensional control interface rather than replacing the production assignment optimizer.
2. Formalizing objective weight adaptation as an offline multi-agent decision problem with store-level decentralized execution and delayed regional rewards.
3. Providing production switchback evidence proving that the method can improve operational efficiency without reducing customer experience.

This work demonstrates how to safely utilize world feedback from active economic and logistics systems to adapt decision policies online, providing valuable reference for large-scale industrial deployment of reinforcement learning.