From what I understand, he is frustrated by your former allies and ideals. However, given how much he bothers you, I would guess that he does indeed care for you in some way.

Is that all, Kaaree? Or is it that she reminds both of you of young women you were unable to save? Why open yourself up to that kind of pain again?

Silence, you. [He waves away the lurking black hole]

And anyways, youth has licence to be annoying. How else will you learn who you are and what you are meant to be?

That implies that people cannot change their fate, of course, which you know to be a lie...