My CRM for: limit users displayed?

SUGGESTED

Is it possible to limit the users displayed on the My CRM for: list?  If so, can someone please shove me in the right direction on how to do this.

In user setup, we have a user with My CRM Lists set to All Users.  For the particular list that we are displaying, not all of the users will have records.  Therefore, I would prefer to only show those users where the record list applies to the user as it makes for a cleaner presentation when there are a lot of users.

I noticed in the user setup, the choices are No Users, User Only, All Users.

Any assistance would be greatly appreciated.  Thank you!!