Steps to reproduce:
1) Login and go to My Home.
2) Go to the Profile tab.
3) Go to Edit Profile
There is no option to delete the portrait in the Edit Profile page.
Note: This option does exist in My Account in the Control Panel, however, only admins have the Control Panel in SO.