memberOf (in AD) is stored as a list of distinguishedNames. Your filter needs to be something like:
(&(objectCategory=user)(memberOf=cn=MyCustomGroup,ou=ouOfGroup,dc=subdomain,dc=domain,dc=com))
If you don’t yet have the distinguished name, you can search for it with:
(&(objectCategory=group)(cn=myCustomGroup))
and return the attribute distinguishedName
. Case may matter.