Seol mar théacs é seo: Limitations of gyrokinetics on transport time scales