Guys, how would you feel in this situation?

No he doesn't think you are a pervert.He is obviously attracted to you also or he wouldn't have turned around and caught you.He was most likely turning around to check out your backside and you caught him! Things will probably get more intense between him and you now that you both know there is a mutual attraction.