I think you just need to build more of everything. They're building luxury condos because there is demand for them. So let them have their fun building luxury condos (which are likely higher profit) but at the same time let them also build affordable housing, which they can also make a profit off of.
Building becomes more expensive across the board (when things like Impact Fees are used to disincentive new construction).
Is it surprising that only large projects that benefit the well-to-do are feasible to pull through this tangle of regulation and fees? If we streamline construction and incentivize new projects, we will get not only new luxury housing, but more affordable housing for everyone.