According to your description, I understand that you want to set My Site permission in SharePoint 2010.
We can do as follows:
1. Navigate directly to any My Site where you already have administration access and go to "Site Settings" via the "Site Actions" button.
2. From "Site Settings" select "Site collection administrators" from the "Users and Permissions" group on the left-hand side.
3. The "Site Collection Administrators" link allows you to create additional users or groups with Site Collection Administration permissions on this My Site. Simply add the
"Colleagues group" groups and remove other group.
More information about colleagues for your reference:
Microsoft is conducting an online survey to understand your opinion of the Technet Web site. If you choose to participate, the online survey will be presented to you when you leave the Technet Web site.