this is such a beautiful essay. thank you op for posting. made my day :)

this comment needs to be at the top

I joined the startup I work for because it’s exciting and not because it will be valuable ( statistically it’s a lottery ticket ). We have a small team and the problems we solve are very complex and challenging. the kind of things I can do here, exposed me to learnings and knowledge I would never get from a bigger company. Big tech derisks most problems they solve because of specialized people helping solve specific problems but your opportunities to learn things across the spectrum are lower.

But milage may vary. I work for a startup where everyone is really experienced/wise and have worked in big tech too. But choosing and finding a good startup is a lot more work then cracking LC and working at a big tech company.

That's a fair assessment but it was also 6 years ago. Back then transformers had recently come out. Tricks to do DL at scale were still brewing. Even achieving what they did for 10 heroes showed DL could work in "non-deterministic-ish" problem settings.

The more interesting question is: can we train a Dota model that plays with all 124 heroes today?

how can I pip install on this?

I have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.

The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.

Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.

Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow

It lost me when the specification was in YAML. I hate YAML for ML.

Then lucky you: continue to use JSON since it's a subset of YAML

If we consider a frame rate of 60 FPS then a 5 year old would have seen about ~ 6.3 billion images [60 (frames) * 60 (seconds) * 60 (minutes) * 16 (waking hours) * 365 (days) * 5 (years)]. Even with 30 FPS you can halve the number and it's still a huge number.

A cool fact is that this model fit ~5B images in 900M parameter model which is tiny compared to the size of the data.

Yes. Zerodha is hands down the largest broker in India. They provide really beautiful suite of products that play very well with each other.

I changed countries on the apple store and in transition I lost my entire apple music library which I curated for over 6 years (thousands of songs). All vanished. It was so painful.

I have moved to spotify but I am still overcoming the grief :(

The one good feature Apple Music has is the "export" feature. Spotify makes it a pain, Apple just lets you make an archive of your whole library instantly.

Sorry about your old library. Don't trust Spotify either! You can download a listing on your account page.