Only show componentGroup based on user group

We have a bunch of template authors who can add "component policies" which allows specific components to be used on the page.

During the configuration of the  "component policy", the dropdown list composes of a lot of componentGroup. We allow template authors to select any componentGroup of their choice...

However, we have a requirement to hide a specific componentGroup based on user group.

My question is, can we hide a specific componentGroup from template authors but show it to admins?