In practice, there really are several dozen gotras (at least among the Brahmins), if not hundreds.
The traditional practice was for Brahmins to declare their exact line of descent alongside the gotra, by naming five noteworthy ancestors (pravara rishis), starting from the top level saptarishi ancestor gotra(one of seven). Some of the saptarishis famously didn't get along, and their descendants are considered incompatible matches for marriage. When considering a marriage alliance from a family with the same top level saptarishi gotra, it was considered acceptable if at least three of the five pravara rishis were different, ie, a distant branch of the family. As families lost their traditions, people identified their nodal pravara rishi as the gotra, which is how most people today know them. In addition, in ancient times the custom was for non-brahmins to state the gotra & lineage of their family priest when performing religious rituals, and ended up adopting them.
I suppose this was an ancient way to ensure genetic diversity and prevent pregnancy losses. There is already quite a bit of evidence that consanguinity leads to miscarriages [0].
Funnily enough, the strict endogamy among Indian caste groups actually leads to far less diversity. There have been many genetic studies on this from Dr. Lal's and Dr. Thangaraj's group at Center for Cellular and Molecular Biology.
This was true in the southern clans as well. The practice of lineage splitting is well documented. As for the concept of gotra incompatibility, its something your astrologer might bring up, and not something well known among the general public.
The traditional practice was for Brahmins to declare their exact line of descent alongside the gotra, by naming five noteworthy ancestors (pravara rishis), starting from the top level saptarishi ancestor gotra(one of seven). Some of the saptarishis famously didn't get along, and their descendants are considered incompatible matches for marriage. When considering a marriage alliance from a family with the same top level saptarishi gotra, it was considered acceptable if at least three of the five pravara rishis were different, ie, a distant branch of the family. As families lost their traditions, people identified their nodal pravara rishi as the gotra, which is how most people today know them. In addition, in ancient times the custom was for non-brahmins to state the gotra & lineage of their family priest when performing religious rituals, and ended up adopting them.