Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics
There is a wide range of sequential decision problems in transportation and logistics that require dealing with uncertainty. There are four classes of policies that we can draw on for different types of decisions, but many problems in transportation and logistics will ultimately require some form of...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Open Journal of Intelligent Transportation Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9702124/ |
| _version_ | 1797974439115096064 |
|---|---|
| author | Warren B. Powell |
| author_facet | Warren B. Powell |
| author_sort | Warren B. Powell |
| collection | DOAJ |
| description | There is a wide range of sequential decision problems in transportation and logistics that require dealing with uncertainty. There are four classes of policies that we can draw on for different types of decisions, but many problems in transportation and logistics will ultimately require some form of direct lookahead policy (DLA) where we optimize decisions over some horizon to make a decision now. The most common strategy is to use a deterministic lookahead (think Google maps), but what if you want to handle uncertainty? In this paper, we identify two major strategies for designing practical, implementable lookahead policies which handle uncertainty in fundamentally different ways. The first is a suitably parameterized deterministic lookahead, where the parameterization is tuned in a stochastic simulator. The second uses an approximate stochastic lookahead, where we identify six classes of approximations, one of which involves designing a “policy-within-a-policy,” for which we turn to all four classes of policies. We claim that our approximate lookahead model spans all the classical stochastic optimization tools for lookahead policies, while opening up pathways for new policies. But we also insist that the idea of a parameterized deterministic lookahead is a powerful new idea that offers features that, for some problems, can outperform the more familiar stochastic lookahead policies. |
| first_indexed | 2024-04-11T04:19:59Z |
| format | Article |
| id | doaj.art-d6d790db22d94705975507ba8bc1b774 |
| institution | Directory Open Access Journal |
| issn | 2687-7813 |
| language | English |
| last_indexed | 2024-04-11T04:19:59Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Intelligent Transportation Systems |
| spelling | doaj.art-d6d790db22d94705975507ba8bc1b7742022-12-31T00:01:50ZengIEEEIEEE Open Journal of Intelligent Transportation Systems2687-78132022-01-01331332710.1109/OJITS.2022.31485749702124Designing Lookahead Policies for Sequential Decision Problems in Transportation and LogisticsWarren B. Powell0https://orcid.org/0000-0002-4364-7555Operations Research and Financial Engineering Department, Princeton University, Princeton, NJ, USAThere is a wide range of sequential decision problems in transportation and logistics that require dealing with uncertainty. There are four classes of policies that we can draw on for different types of decisions, but many problems in transportation and logistics will ultimately require some form of direct lookahead policy (DLA) where we optimize decisions over some horizon to make a decision now. The most common strategy is to use a deterministic lookahead (think Google maps), but what if you want to handle uncertainty? In this paper, we identify two major strategies for designing practical, implementable lookahead policies which handle uncertainty in fundamentally different ways. The first is a suitably parameterized deterministic lookahead, where the parameterization is tuned in a stochastic simulator. The second uses an approximate stochastic lookahead, where we identify six classes of approximations, one of which involves designing a “policy-within-a-policy,” for which we turn to all four classes of policies. We claim that our approximate lookahead model spans all the classical stochastic optimization tools for lookahead policies, while opening up pathways for new policies. But we also insist that the idea of a parameterized deterministic lookahead is a powerful new idea that offers features that, for some problems, can outperform the more familiar stochastic lookahead policies.https://ieeexplore.ieee.org/document/9702124/Direct lookahead approximationsmodel predictive controlparametric cost function approximationpolicy searchreinforcement learningsequential decisions |
| spellingShingle | Warren B. Powell Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics IEEE Open Journal of Intelligent Transportation Systems Direct lookahead approximations model predictive control parametric cost function approximation policy search reinforcement learning sequential decisions |
| title | Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics |
| title_full | Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics |
| title_fullStr | Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics |
| title_full_unstemmed | Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics |
| title_short | Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics |
| title_sort | designing lookahead policies for sequential decision problems in transportation and logistics |
| topic | Direct lookahead approximations model predictive control parametric cost function approximation policy search reinforcement learning sequential decisions |
| url | https://ieeexplore.ieee.org/document/9702124/ |
| work_keys_str_mv | AT warrenbpowell designinglookaheadpoliciesforsequentialdecisionproblemsintransportationandlogistics |