It may be that he's just that bad at taking hints. If he does show up the next weekend you're at work and continues with this, I'd just come right out and say (as gently as you feel is necessary) "Not interested." Hold up a big old sign if need be.

You were absolutely right to mention it to your manager. Even though you can kick creepy dude's ass if need be, it's still better to have someone else keeping an eye out, just in case.