It's the lead-in that ruins him, and she's already leaning back and watching him unhappily by the time he confesses to her needing to take a night shift.

Man, this is not how she wanted to spend her night. Not after Mary Margaret had been so unhappy and so conflicted, and someone ought to be around in case David just happens to drop by again, so she doesn't bother hiding the petulance in her voice or expression.

"Why?"

So she's complaining: who cares? It's not like she has to be anything other than herself around Graham, there's nobody here to sugarcoat it for.