Code that called the CryptDecryptMessage function successfully under previous versions of Windows may fail with a CRYPT_E_ASN1_MEMORY error under Windows 7.
There is a counter in the ASN structure that is increased every time an octet block is processed when the ASN sequence that wraps the octets is BER encoded. It adds the number of bytes that have already been processed plus the new block size; as the number of processed bytes grows that is added to the counter and the rate at which the counter grows increases. Since the counter is represented by an integer it can overflow. On Windows 7, the ASN code checks for overflow so the function fails. On pre-Windows 7 platforms, the overflow still occurs but it wasn't checked. That counter is not critical for operation and did not cause issues in the code for the previous versions of the OS.
Download the hotfix described in Microsoft Knowledge Base article 2480994, "ASN1 out of memory" error when the CryptMsgUpdate function decodes a message that is larger than 5 MB in Windows 7 or in Windows Server 2008 R2 (http://support.microsoft.com/kb/2480994).
You can also call the lower level message functions (CryptMsgOpenToDecode, CryptMsgUpdate, CryptMsgControl) to process BER encoded content.
Hi, do you know if Microsoft will release a fix for this? I have been struggling with the error for quite some time, and I have not been able to successfully implement a solution using the lower level message functions.