There's a really neat gender system that some fungi use where they have dozens, hundreds or thousands of biological "mating types" (that sometimes get describes as "genders".) The mating types are functionally identical without a genetic test, but they are only cross-fertile with individuals who are of sufficiently different mating type. The result is that they can't breed with (most) of their siblings, and (some) of their non-siblings, with likelihood of compatibility increasing as they are less closely related, but never at 100%.

I've always wanted to see that worked into an SF system with intelligent creatures. Overlaying it on a pre-existing, visible m/f system would be interesting, too.

(Here's the article that explains the fungi biology if you're interested.)