DESCRIPTION
This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Computer Associates ARCserver Backup. Authentication is not required exploit this vulnerability and both the client and server are affected.

The problem specifically exists within the handling of long messages received over the Mailslot named 'CheyenneDS'. As no explicit MaxMessageSize is supplied in the call to CreateMailslot, an attacker can cause an exploitable stack-based buffer overflow.

The vulnerable Mailslot creation occurs:

   casdscsvc.exe -> Asbrdcst.dll
   20C14E8C push 0                  ; lpSecurityAttributes
   20C14E8E push 0                  ; lReadTimeout
   20C14E90 push 0                  ; nMaxMessageSize
   20C14E92 push offset Name        ; "\\\\.\\mailslot\\CheyenneDS"
   20C14E97 stosb
   20C14E98 call ds:CreateMailslotA
   20C14E9E cmp eax, INVALID_HANDLE_VALUE
   20C14EA1 mov mailslot_handle, eax

Note there is no explicit MaxMessageSize specified. Later the mailslot handle is read from into a 4k buffer. The read data is also passed to a routine which calls vsprintf into a 1k buffer.

   casdscsvc.exe -> Asbrdcst.dll
   20C15024 mov eax, mailslot_handle
   20C15029 lea edx, [esp+1044h+Buffer_4k]
   20C1502D push ecx                          ; nNumberOfBytesToRead
   20C1502E push edx                          ; lpBuffer
   20C1502F push eax                          ; hFile
   20C15030 call edi ; ReadFile
   20C15032 test eax, eax
   20C15034 jz  short read_failed
   20C15036 lea ecx, [esp+3Dh]
   20C1503A push ecx                          ; char
   20C1503B push offset str_ReadmailslotS     ; "ReadMailSlot: %s\n"
   20C15040 call not_interesting_call_to_vsnprtinf
   20C15045 add esp, 8
   20C15048 lea edx, [esp+3Dh]
   20C1504C push edx                          ; va_list
   20C1504D push offset str_ReadmailslotS_0   ; "ReadMailSlot: %s"
   20C15052 push 0                            ; for_debug_log
   20C15054 call vsprintf_into_1024_stack_buf_and_debug_log

As mentioned in TSRT-06-02, exploitation of this vulnerability is possible due to the ability to exceeding the second-class Mailslot message size limitation.

TippingPoint Advisory