To reproduce this:
- Create a new site (Site1) from Community Site template
- Navigate to Go to >> Site1
- Click "Please configure this portlet and select at least one ranking criteria"
- Flag "Rank by Contribution" in Configuration window
- Flag "Rank by Participation" in Configuration window
- Configurations are not saved and "Please configure this portlet and select at least one ranking criteria" displays still
When you save the configuration it gets overwritten with the default one because the original site template layout is merged back all the time.