Language models as a service: overview of a new paradigm and its challenges
Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-aService (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case o...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Association for Computing Machinery
2024
|
| _version_ | 1826316035078225920 |
|---|---|
| author | La Malfa, E Petrov, A Frieder, S Weinhuber, C Burnell, R Nazar, R Cohn, AG Shadbolt, N Wooldridge, M |
| author_facet | La Malfa, E Petrov, A Frieder, S Weinhuber, C Burnell, R Nazar, R Cohn, AG Shadbolt, N Wooldridge, M |
| author_sort | La Malfa, E |
| collection | OXFORD |
| description | Some of the most powerful language models currently are proprietary systems, accessible only via
(typically restrictive) web or software programming interfaces. This is the Language-Models-as-aService (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the
case of open-source models, such closed-off language models present specific challenges for evaluating,
benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how
the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and
trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information
about language models for each of these four aspects. We conduct a detailed analysis of existing
solutions and put forth a number of considered recommendations, and highlight the directions for
future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge
on current, major Language-Models-as-a-Service, offering a synthesized overview of the licences and
capabilities their interfaces offer. |
| first_indexed | 2024-12-09T03:36:36Z |
| format | Journal article |
| id | oxford-uuid:14aa74ca-9cdc-4091-86b4-f3b4c41a32f2 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-12-09T03:36:36Z |
| publishDate | 2024 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | oxford-uuid:14aa74ca-9cdc-4091-86b4-f3b4c41a32f22024-12-02T15:28:27ZLanguage models as a service: overview of a new paradigm and its challengesJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:14aa74ca-9cdc-4091-86b4-f3b4c41a32f2EnglishSymplectic ElementsAssociation for Computing Machinery2024La Malfa, EPetrov, AFrieder, SWeinhuber, CBurnell, RNazar, RCohn, AGShadbolt, NWooldridge, MSome of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-aService (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major Language-Models-as-a-Service, offering a synthesized overview of the licences and capabilities their interfaces offer. |
| spellingShingle | La Malfa, E Petrov, A Frieder, S Weinhuber, C Burnell, R Nazar, R Cohn, AG Shadbolt, N Wooldridge, M Language models as a service: overview of a new paradigm and its challenges |
| title | Language models as a service: overview of a new paradigm and its challenges |
| title_full | Language models as a service: overview of a new paradigm and its challenges |
| title_fullStr | Language models as a service: overview of a new paradigm and its challenges |
| title_full_unstemmed | Language models as a service: overview of a new paradigm and its challenges |
| title_short | Language models as a service: overview of a new paradigm and its challenges |
| title_sort | language models as a service overview of a new paradigm and its challenges |
| work_keys_str_mv | AT lamalfae languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT petrova languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT frieders languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT weinhuberc languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT burnellr languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT nazarr languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT cohnag languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT shadboltn languagemodelsasaserviceoverviewofanewparadigmanditschallenges AT wooldridgem languagemodelsasaserviceoverviewofanewparadigmanditschallenges |