Independent Reading · 2023

Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks

Siyu Chen, Mengdi Wang, and Zhuoran Yang · arXiv preprint

Stackelberg GamesReinforcement LearningQuantal Response

Full text

Read the Paper

Use View PDF or Download PDF to read the full paper on this device.

Paper Information

Authors
Siyu Chen, Mengdi Wang, and Zhuoran Yang
Venue
arXiv preprint · 2023
Open official paper

Summary

This paper considers learning a quantal Stackelberg equilibrium in episodic leader-follower Markov games from follower actions.

Core Idea

A leader can infer an unobserved follower response model from strategic feedback and use its uncertainty when optimizing a policy.

Method

The proposed online and offline algorithms combine maximum-likelihood estimation of quantal response with model-free or model-based reinforcement learning.

Key Takeaways

  • Follower actions can provide enough feedback to learn strategic response models.
  • Uncertainty-aware optimism and pessimism support online and offline learning guarantees.
  • The framework handles both myopic and farsighted followers.

Questions

  • How robust are the guarantees when follower behavior changes across episodes?
  • Can the algorithms scale to large policy classes without strong function-approximation assumptions?