Hi gwern, asking earnestly and without snark about this:
"And if you're wondering how it can look so good when 'everyone knows GANs don't work because they're too unstable', a widespread myth, repeated by many DL researchers who ought to know better, GANs can scale to high-quality realistic images on billion-image scale datasets, and become more, not less, stable with scale, like many things in deep reinforcement learning."
My interpretation from how this is written is that you are saying: "Researchers say that GANs are unstable at inference time." Is that the correct reading? If so, where have you seen this sentiment? More commonly I've heard that people criticize GANs at inference time for mode collapse and monotony (which comes from training), not instability.
Or are you saying that researchers say GANs are unstable during training, which is a common criticism. Don't you feel this is the case? A lot of different tricks are used for getting generators that are roughly balanced with more powerful discriminators so that the adversarial game is balanced, like TTUR and encodec's weight balancer etc. In this case, are you saying that GAN training is as straightforward as diffusion training?
In my experience, GAN training involves an unholy number of heuristics and best-practices are still quite murky.
I am eager to hear your response.
[edit: my experience with GANs and diffusion models isn't about taking image problems with existing working models and scaling or refining them, but about applying GANs and diffusion models to different domains (audio) and on novel problems with different kinds of conditioning. I would love to see more controlled experiments comparing weak and sophisticated generative backbones as the experimental control and varying the training regime across different GAN + diffusion flavors.]
"And if you're wondering how it can look so good when 'everyone knows GANs don't work because they're too unstable', a widespread myth, repeated by many DL researchers who ought to know better, GANs can scale to high-quality realistic images on billion-image scale datasets, and become more, not less, stable with scale, like many things in deep reinforcement learning."
My interpretation from how this is written is that you are saying: "Researchers say that GANs are unstable at inference time." Is that the correct reading? If so, where have you seen this sentiment? More commonly I've heard that people criticize GANs at inference time for mode collapse and monotony (which comes from training), not instability.
Or are you saying that researchers say GANs are unstable during training, which is a common criticism. Don't you feel this is the case? A lot of different tricks are used for getting generators that are roughly balanced with more powerful discriminators so that the adversarial game is balanced, like TTUR and encodec's weight balancer etc. In this case, are you saying that GAN training is as straightforward as diffusion training?
In my experience, GAN training involves an unholy number of heuristics and best-practices are still quite murky.
I am eager to hear your response.
[edit: my experience with GANs and diffusion models isn't about taking image problems with existing working models and scaling or refining them, but about applying GANs and diffusion models to different domains (audio) and on novel problems with different kinds of conditioning. I would love to see more controlled experiments comparing weak and sophisticated generative backbones as the experimental control and varying the training regime across different GAN + diffusion flavors.]