Self Supervised Learning: The Final Frontier of AI - Panel Discussion (April 29, 2025)

uh so maybe I will just start with a very high level quick question . So during all those talks today or most of them we saw a lot of uh figures showing what happens when you scale data and scale parameters . So my first question is uh do you think SSL now is at a stage where most of the details have been figured out and we just need to scale in term of data and parameters ? Uh and if not what do you think is the main bottleneck ? Whoever wants to start uh the answer is no . Um I don't think we have a perfect recipe yet . um in the sense that um the stuff that seems to work uh the best or the most efficiently for images and video to train uh jetpas . So first of all it's got to be a that works by reconstruction . I can make that point really clear . I'm sure a lot of people here disagree . I have a lot of colleagues who disagree but like um this is something I feel extremely strongly about and we have lots of empirical evidence for that but um so so it's easy to train by reconstructions it's not that easy to train jaming because it can collapse so the question is what is the best anti-olapse method which is why I talked about to some extent and um as I said contrasting methods don't don't scale very well with dimension so I'm a little skeptic about them Um the EMA type distillation methods uh work well but EMA is pain ass um and I think it comes with a lot of limitations . Um I'm not so worried about the fact that we don't understand why they work . Um although probably we should be a little worried about it . uh conceptually I think the infomax method are have all but we haven't really gone through like a completely um you know perfect recipe