Hey I've been coding a little protector for PE files.
Let me explain you in short what it does:
- Password protect some EXE (optional though) - Encrypt the EXE except IAT/RSRC sections - Encrypt the EXE with password (if user wanted password) - Ask for password on startup if EXE protected with password
Check the link below for both binary and source, I got the idea of a program called 'vodet' and I've certainly looked at the code to get some idea of how to lay-out the DialogBox in memory for asking the user for a password.
Also I've used Yoda's code to get the kernel base and API addresses when the encrypted EXE starts up.
If you enter some password the EXE will decrypt using that password, then check if some bytes of the OEP still match, if not -> exit.