I have no good answer and suffer from similar problems! I'm starting to get down two phrases that I pull out depending on the question. One is something like "I write fiction, a lot of it tends to range somewhere between fantasy and magical realism" and "I'm working on a magical realist murder mystery." Both of those are quasi-inaccurate, but they contain enough of the truth for hey-we-just-met conversation. :-)

answer decisively in favour of the genre your hart is in most, or the genre of the latest book you're writing/trying to sell.
If someone asks me what I write, I answer: hard SF
Even though I've just written a story about vampires (gawd, can't believe this myself), and I have written fantasy, as well as MG and space opera.
Giving half an answer decisively is going to sound more professional to them than indistinct mumbling of the entire answer.

All fiction is fantasy, even if it isn't Fantasy. It isn't real, even if it is realistic. It isn't reportage, even if some 'reporters' are the best paid writers of fiction around. I writre left handed fantasy in what I hope is a world recognisable as our own with just a small 'what if' change. I write about what happens just beyond the edge of my peripheral vision.

Except when I don't.

Either "stories", or "stuff", if they're that interested. They can always Google me. XD

It usually goes like this:
Person: You're a writer? Cool! What do you write?
Me: Primarily adult fantasy, but sometimes mainstream.
Person: Adult fantasy? You mean like...(whispers)...porn?
Me: No. More like Lord of the Rings*.
Person: Oh. I don't really read.

Or it goes like this:
Person: You're a writer? Cool! What do you write?
Me: Primarily adult fantasy, but sometimes mainstream.
Person: Oh, like Kay or Tolkein or McKillip?
Me: I wouldn't put myself on that level, but yes, that sort of fantasy.
Person: Awesome! Where can I get it?

(*Because LOTR and Harry Potter are the only fantasy most will know.)

A tremendous advantage of writing historical fiction
is that it's so easy to say.

-What sort of things do you write?
-Mostly historical fiction.

If it's obvious the person talking to me
is inclined toward reading about transsexuals and homosexuals,
I can mention that I've got collections of stories pertaining to such.

If they might like cop stories,
well, I've got a collection of those, too.
If that catches their interest, I may or may not mention that
the main character/narrator is a stealth transsexual.

I just say speculative fiction--but then I usually have to explain what that is.

I usually say Young Adult books, with fantasy elements. Of course, most childless adults have NO CLUE what YA is. So I have to explain. If I have to explain my stories...I just trip over my tongue.

See, this is why it's easier for me to just stay mum on the whole subject. It's always embarrassing (to me, and I don't know why, other than a natural reticence to speak about myself) to say I write fantasy, because most people either have a disdain for the genre and think it's "kid stuff" or think the word "fantasy" refers to porn or erotica. *sigh* I do like the term speculative fiction, but agree, most people don't know what it means. But it does sound more "grownup" than fantasy. I don't think SF writers have that problem. Science Fiction also sounds more grownup.

If I have a feeling the person asking me is simply making conversation, I say: Fantasy and Science Fiction. Yes, those are labels, but people generally understand them. Only if they ask deeper questions do I go into the subgenres of what I write, and why.

No, you've never heard of me.
No, I don't have anything published yet. I'm working on it.
I write fiction; nonfiction seldom holds my interest when I read, so I don't write it.
Oh, all sorts of fiction; it helps to be well-rounded, you know. What about you?

There's mine, Jon. You have more positive things to say than I, but that's mine.

I keep it pretty basic. I usually answer, "Science fiction and dark fantasy. A little mystery here and there." Then if they want specifics I give it to them.

But I understand what you mean. I think many writers write across a lot of genres, so it's sometimes difficult to answer that question without getting lost in the weeds.

I think maybe you should say you write genre fiction. And if they don't know what that is, tell them it's fantasy, science fiction, westerns and mysteries and that's what you write.

I try to explode their brains... Maybe tell them that you write scripts for hardcore movies. They will look at you wide-eyed, try to think of the dialogue, then be too ambarassed to ask you why anyone would need a writer on those things. Brilliant fun.

I have the same approach and experience you do. I write in a variety of genres, and I also determine my answer by who is asking. Some people get, "oh literary and some *mumbling* darker fantasy." Others get "psychological horror and vampires! and some literary." Vague seems better unless I know the person's interests a bit. We probably should practice a response. Usually when I say I prefer short story writing to novel, they just dismiss the whole thing--I should find a way to make that more ... acceptable.