Independent Reading · 2023
Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks
Siyu Chen, Mengdi Wang, and Zhuoran Yang · arXiv preprint
Stackelberg GamesReinforcement LearningQuantal Response
Full text
Read the Paper
Use View PDF or Download PDF to read the full paper on this device.
Paper Information
- Authors
- Siyu Chen, Mengdi Wang, and Zhuoran Yang
- Venue
- arXiv preprint · 2023
Summary
This paper considers learning a quantal Stackelberg equilibrium in episodic leader-follower Markov games from follower actions.
Core Idea
A leader can infer an unobserved follower response model from strategic feedback and use its uncertainty when optimizing a policy.
Method
The proposed online and offline algorithms combine maximum-likelihood estimation of quantal response with model-free or model-based reinforcement learning.
Key Takeaways
- Follower actions can provide enough feedback to learn strategic response models.
- Uncertainty-aware optimism and pessimism support online and offline learning guarantees.
- The framework handles both myopic and farsighted followers.
Questions
- How robust are the guarantees when follower behavior changes across episodes?
- Can the algorithms scale to large policy classes without strong function-approximation assumptions?