Risk and uncertainty

Talk outline

• Distorted riskmetrics (DRMs)

• DRM-sensitive bandit problem

• Regret-efficient algorithms

Part 0: Introduction to Distortion Riskmetrics (DRMs)

• Distortion Riskmetrics: a rich class that includes

Distortion Riskmetric: definition

• For a r.v. \(X\ge 0\) with CDF \(F\),

  \[ U(F)= \int_{0}^{\infty} h(1-F(x)) \, dx \]

• Distortion function \(h\) s.t. \(h(0)=0\)
• For \(h(x)=x\), \(U(F)\) is the expected value

• More generally,

  \[U(F)=\int_{-\infty}^{0} \left(h(1- F(x)) - h(1)\right) \, dx + \int_{0}^{\infty} h(1-F(x)) \, dx \]

• For \(h(x)=x\):
  \[\begin{aligned} U(F)&=\int_{-\infty}^{0} \left(1- F(x) - 1\right) \, dx + \int_{0}^{\infty} 1-F(x) \, dx\\ &= \int_{-\infty}^{0} F(x)\, dx + \int_{0}^{\infty} 1-F(x) \, dx, \end{aligned}\]

• we recover the expected value

Risk maximization

Quote from Cover (1991), “Universal portfolios”

In general, volatile uncorrelated stocks lead to great gains for the rate at which a portfolio grows...

Quote from V. Anantharam and V. S. Borkar (2017), “A variational formula for risk-sensitive reward.”

Work on *risk-sensitive reward maximization* has been relatively uncommon; see, e.g., [24]. Unlike in the case of the classical discounted or ergodic costs, the two risk-sensitive control problems are not trivially equivalent by treating cost as a negative reward. In fact, risk-sensitive reward maximization is the natural set-up in portfolio optimization...

Part I: Distortion riskmetrics + bandits

Risk-sensitive Bandits: Arm Mixture Optimality and Regret-efficient Algorithms
M. Tatli, A. Mukherjee, P. L.A., K. Shanmugam and A. Tajer
AISTATS 2025 (To appear)

Summary

• DRM-sensitive bandit problem
  • Many DRMs, solitary arm is not optimal. Instead, it is optimal to play an arm-mixture
• Learning optimal mixtures

• Regret bounds for ETC-type and UCB-type algorithms

Multi-armed Bandits: A Sequential Experimental Design Framework

Risk-neutral bandit problem setup

• \(K\) arms / experiments

• Expected return of arms: \(\mathbf{\mu} = [\mu_1,\cdots,\mu_K]^\top\)

• Goal: Arm with largest expected return:

  \[ a^* \in \arg\max_{i\in[K]} \mu_i \]

Risk-neutral Objective: Regret minimization (Exploration-Exploitation trade-off)

Minimize cumulative regret:

\[R_T\triangleq T\mu_{a^\star} - \sum\limits_{s=1}^T \mathbb{E}[X_{A_s}]\]

Bandit Settings and Applications

:::

Bandit Settings and Applications – Focus

Risk-Sensitive Decision Making

• Option A: Larger average reward (mean), larger risk (variance)!

• Option B: Smaller average reward (mean), smaller risk (variance)!

Another Motivating Example

• Option A: Win $100,000 with probability \(0.0001\), lose nothing

• Option B: Win $10 with certainty

Which one would you go for?

• Option A is usually preferred over Option B

Main message

How to distort the probabilities? Distortion riskmetrics (DR)

Probabilistic distortions: basis for Nobel-prize winning Prospect Theory work of Tversky and Kahnemann

Risk Nomenclature

:::

Gini Deviation

Linking Risk-Sensitivity and Experimental Design

:::

Risk-Sensitive Bandits: Existing Literature

Sporadic investigations on monotone distortion functions:

A Unified Framework for Risk Measures (Cassel et. al. 2018)

• Let \(a^\star\) denote the risk-maximizing arm, i.e.,

  \[a^\star\;\triangleq\;\arg\max\limits_{i\in[K]}\;U\big ( \mathbb{F}_i\big )\]

• Goal: Minimize the average regret

  \[ \mathfrak{R}_{\nu}^\pi(T)\;\triangleq\; U\left (\mathbb{F}_{a^\star}\right) - \mathbb{E}_{\nu}^\pi\Bigg [ U\Bigg(\sum\limits_{i\in[K]}\frac{\tau^\pi_T(i)}{T}\mathbb{F}_i\Bigg )\Bigg]\]

• Assumptions:
• The utility is convex \(\implies\) solitary best arm
• The utility is stable in an abstract semi-normed space – CDF estimates admit exponential convergence to the ground truth
• Utility is Lipschitz

Gaps in the Literature...

:::

Revised Objective: Regret w.r.t. Infinite Horizon Oracle Policy

• Mixtures may be optimal as opposed to solitary arms

• Oracle Policy: Policy that attains the maximum utility over an infinite horizon, i.e.,

  \[\mathbf{\alpha}^\star_{\mathbf{\nu}}\;\in\;\arg\sup\limits_{\mathbf{\alpha}\in\Delta^{K-1}}\;U\Big ( \sum\limits_{i\in[K]}\alpha(i) \; \mathbb{F}_i\Big )\]

• Goal: Define regret w.r.t. the oracle policy

• Assumption: Hölder continuous utility, Hölder exponent \(q\)

Algorithm Design – Challenges

Algorithm Design Components

Risk-Sensitive Explore Then Commit for Mixtures (RS-ETC-M)

• Step 1 (Explore): Estimate CDFs, draw each arm \(\lceil N(\varepsilon)/K\rceil\) times (\(N(\varepsilon)\) is instance-dependent)

• Step 2 (Estimate): Using CDF estimates \(\mathbb{F}_{t,i}^{\rm E}\) of each arm, estimate mixing coefficients through discretization

  \[\mathbf{\alpha}_{N(\varepsilon)}\;\in\;\arg\max\limits_{\mathbf{\alpha}\in\Delta_{\varepsilon}^{K-1}} U\Big( \sum\limits_{i\in[K]}\alpha(i)\mathbb{F}_{t,i}^{\rm E}\Big) \]

• Why discretize?

  1. Computational tractability

  2. Transforms the problem into a finite-armed bandit instance in terms of discrete mixing coefficients

Risk-Sensitive Explore Then Commit for Mixtures (RS-ETC-M)

• Step 2 (Commit): Sample arms in a way that best matches the allocation fractions to the estimated mixing coefficient

• Define \(S \triangleq [K-1]\) as the first \(K-1\) arms

  \[\begin{aligned} &\tau^{\rm E}_T(i)\triangleq \left\{ \begin{array}{ll} \max\Big\{\Big\lceil\frac{N(\varepsilon)}{K}\Big\rceil,\lfloor T\widehat\alpha_{N(\varepsilon)}(i) \rfloor\Big\}, & \mbox{if} \;\; i \in S\\ \\ T - \sum\limits_{i\in S}\tau^{\rm E}_T(i) , & \mbox{otherwise} \end{array}\right. \end{aligned}\]

Risk-Sensitive Upper Confidence Bound for Mixture (RS-UCB-M)

• Step 1 (Forced exploration): Form reliable estimates of arm CDFs, draw each arm \(\zeta T\) times

  • Forced exploration is absent in canonical UCB

  • Reason: sub-optimal arms should not be sampled over \(O(\log T)\) times

  • In our setting, mixtures may necessitate a linear order of exploration for every arm!

Risk-Sensitive Upper Confidence Bound for Mixture (RS-UCB-M)

• Step 2 (Estimating optimal mixtures): Using CDF estimates \(\mathbb{F}_{t,i}^{\rm U}\) of each arm:

  • Optimistic estimate: For any mixture \(\mathbf{\alpha}\in\Delta^{K-1}\), define the upper confidence bound (UCB):
    \[\begin{aligned} {\rm UCB}_t(\mathbf{\alpha}) \;\triangleq\; U\Big( \sum\limits_{i\in[K]}\alpha(i)\mathbb{F}_{t,i}^{\rm U}\Big) + L \sum\limits_{i\in[K]} \bigg( \alpha(i) \cdot 16\; \frac{\sqrt{2 {\rm e} \log T } + 32}{\sqrt{\tau^{\rm U}_t(i)}} \bigg)^q \end{aligned}\]

• Term 1: Estimated utility, Term 2: Upper confidence bound

  • Estimate mixture through discretization:

    \({\mathbf{\alpha}}_t \in \arg\max_{\mathbf{\alpha}\in\Delta_{\varepsilon}^{K-1}}\; {\rm UCB}_t(\mathbf{\alpha})\)

Risk-Sensitive Upper Confidence Bound for Mixture (RS-UCB-M)

• Step 3 (Tracking): Undersample according to the estimated mixing coefficients, i.e., for all \(t>KT\zeta\),

  \[A_{t+1}\;\triangleq\;\arg\max_{i\in[K]}\; \{T\alpha_{t}(i) - \tau_t^{\rm U}(i)\}\]

• No instance dependence

• Empirically performs better than randomly sampling according to the estimated mixtures

How I learned to stop regretting…

Performance Guarantees (Takeaways)

• RS-ETC-M has better regret guarantees for solitary arms (known gap information)

• For mixtures, RS-UCB-M better for some measures, e.g., Gini deviation

Risk and happiness: A matter of perspective