An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis
An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable a...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-12-01
|
| Series: | Philosophies |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2409-9287/5/4/40 |
| _version_ | 1797226372205117440 |
|---|---|
| author | James D. Miller Roman Yampolskiy Olle Häggström |
| author_facet | James D. Miller Roman Yampolskiy Olle Häggström |
| author_sort | James D. Miller |
| collection | DOAJ |
| description | An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI. |
| first_indexed | 2024-03-10T14:25:27Z |
| format | Article |
| id | doaj.art-9d84e717e19746a89815903d63473e61 |
| institution | Directory Open Access Journal |
| issn | 2409-9287 |
| language | English |
| last_indexed | 2024-04-24T14:23:51Z |
| publishDate | 2020-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Philosophies |
| spelling | doaj.art-9d84e717e19746a89815903d63473e612024-04-03T05:44:19ZengMDPI AGPhilosophies2409-92872020-12-015440010.3390/philosophies5040040An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality ThesisJames D. Miller0Roman Yampolskiy1Olle Häggström2Economics Department, Smith College, Northampton, MA 01063, USASchool of Engineering, University of Louisville, Louisville, KY 40292, USAMathematical Sciences Department, Chalmers University of Technology, 405 30 Gothenburg, SwedenAn artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI.https://www.mdpi.com/2409-9287/5/4/40artificial general intelligenceorthogonality thesisinstrumental drives |
| spellingShingle | James D. Miller Roman Yampolskiy Olle Häggström An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis Philosophies artificial general intelligence orthogonality thesis instrumental drives |
| title | An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis |
| title_full | An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis |
| title_fullStr | An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis |
| title_full_unstemmed | An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis |
| title_short | An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis |
| title_sort | agi modifying its utility function in violation of the strong orthogonality thesis |
| topic | artificial general intelligence orthogonality thesis instrumental drives |
| url | https://www.mdpi.com/2409-9287/5/4/40 |
| work_keys_str_mv | AT jamesdmiller anagimodifyingitsutilityfunctioninviolationofthestrongorthogonalitythesis AT romanyampolskiy anagimodifyingitsutilityfunctioninviolationofthestrongorthogonalitythesis AT ollehaggstrom anagimodifyingitsutilityfunctioninviolationofthestrongorthogonalitythesis AT jamesdmiller agimodifyingitsutilityfunctioninviolationofthestrongorthogonalitythesis AT romanyampolskiy agimodifyingitsutilityfunctioninviolationofthestrongorthogonalitythesis AT ollehaggstrom agimodifyingitsutilityfunctioninviolationofthestrongorthogonalitythesis |