Text this: Learning environment simulators from sparse signals