I certainly wouldn't let him in, under any circumstances, to do any sort of inspection. They can get someone official, or the person who did it last time. Mom lived in a building once, where they had a woman who was only supposed to be cleaning the entryways. She somehow got ahold of a master key by sweet talking someone, and would slip into people's apartments when they weren't home. She ended up fired and jailed after sneaking into the town marshal's apartment while he was home.
I'd also call the agency and police and report each and every incident. They will see it is a repeat problem. Also, as it has been mentioned, keep a record, date and time of each incident.
You should not be afraid either in your home, or coming and going from your home, and no one has the right to stalk. Even if he hasn't 'done' anything, you are frightened and that is wrong.
Big hugs to you!