Yes, but due to it being derived from the same underlying source dataset, it is effectively evaluating on the training dataset, not an independent validation/ test dataset.
The difference is subtle but important. If we expect the model to truly outperform a general model, it should generalize to a completely independent set.
Hm, no.
They trained on a part of their synthetic set and tested on another part of the set. Or at least that's what they said they did:
> from which 1,000 were held out as a benchmark test set.
Emphasis mine.