Anfonwch hwn fel neges destun: The limited role of galaxy mergers in driving stellar mass growth over cosmic time