Salman Rushdie started as a Sci-fi author. Then he moved into "Magic realism", which is Fantasy without the barbarians and buxom wenches (or more eloquently written barbarians and buxom wenches).
"Literature" is the label for works that are good enough that everyone should read them, not just genre fans. "Romance", "Sci fi", "Travel", "Crime", "Comedy", "Detective", "Drama" and so on are works that aren't good enough for everyone to read.
Of course, there's writers who hack their way into the "Literature" section by imitating the quirks of a genuine pieces of literature, or just avoiding all the other categories.
No, "Literature" is the label for works of middle-class introspection, whereas "Genre" fictions are books that are about any other subject matter.
If you move from actually having a subject to making your subject a metaphor for middle-class internal conflicts, you have made the move from Genre to Lit.
"Literature" is the label for works that are good enough that everyone should read them, not just genre fans. "Romance", "Sci fi", "Travel", "Crime", "Comedy", "Detective", "Drama" and so on are works that aren't good enough for everyone to read.
Of course, there's writers who hack their way into the "Literature" section by imitating the quirks of a genuine pieces of literature, or just avoiding all the other categories.