I have no answer, but was able to reproduce this.
SharePoint 2010 Server with a site based on a team site template (not a publishing site; viewformpageslockdown feature is NOT enabled)
Created a document library and enabled document sets. Anonymous user can view content of the document library, but can not browse the document set. Fiddler shows the 401 occurring on _layouts/docsethomepage.aspx
Any suggestions? Or is this another feature that has difficulty with anonymous:
http://mikesnotebook.wordpress.com/2010/11/17/sharepoint-2010-anonymous-sites-for-mobile-devices/
Mike G.