Seol mar théacs é seo: The limited role of galaxy mergers in driving stellar mass growth over cosmic time