Non-parametric deep learning with applications in active learning - Band, N (2022)
We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture---Non-Parametric Transformers (NPTs)---that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, NPTs solve complex reasoning tasks unsolvable by traditional deep learning models and make use of interactions between datapoints in achieving highly competitive results on tabular data.