Does he want to hang out or was he just being polite? Should I stop trying?

I think you should just wait and see if he responds to you. From the way you explained it, I could see how he might think you were blowing him off repeatedly. I'm sure you weren't, but look at it from his point of view. He may have been thinking the same thing you are now, then decided he didn't have time to play games. Just wait it out, I think.