Zaslat SMS: The limited role of galaxy mergers in driving stellar mass growth over cosmic time