Wednesday, August 6, 2008

Memory leak in FAXCOMEXLib

Yuan has been working on a fax / e-mail / SMS etc server we use internally and also give to our clients, and has come across some strange behavior in Microsoft's COM wrapper to the fax console.

Recently I have been working on our communications toolset – which manages our SMTP, FTP, SMS and Fax sending..

The application has 2 major components: An ASP.NET application to view, search and manage messages and message batches being sent; And a windows service which periodically queries the database to send any queued messages.

Each of the message types is run in its own thread with SMTP, FTP and SMS working for years with no problem.

However, a few weeks after deploying the new Fax component, we began to receive “Out of memory” log traces.

After a little investigation, I narrowed down the problem to a FAXCOMEXLib memory leak. Read below fo the details:

To queue a fax, we use FAXCOMEXLib - which is a Microsoft COM wrapper to the windows fax console. Once the server sends a fax to the console, the fax thread polls to check whether the fax job terminated successfully. The code we use to poll the fax queue is shown below:



            iterator = faxServer.Folders.OutgoingArchive.GetMessages(FAX_QUEUE_DEFAULT_PREFETCH) ' This routine exhibits the memory leak


            For i As Integer = 1 To filesCount


               If Not iterator.Message Is Nothing Then

                  Dim FaxRecipient As Recipient = GetRecipientForFax(iterator.Message.Id)


                  '     Recipient will be null if the fax job was added to the fax server

                  '     by anything other than this service.


                  If Not FaxRecipient Is Nothing AndAlso FaxRecipient.Status <> FaxRecipient.Statuses.sent.ToString() Then

                     ' Acknowledge fax has been sent



                     Dim LogMessage As String = String.Format("CR_ID: {0} Fax sent successfully", FaxRecipient.ID)

                     Log.write(MaxSoft.Common.Log.LevelEnum.INFO, Me, LOG_AREA, LogMessage)

                  End If


               End If






            If Not iterator Is Nothing Then


            End If

            iterator = Nothing

            faxServer = Nothing

         End Try

As you can see in the code above, we iterate through the files in the OutgoingArchive folder of the fax console and match the id with the id stored in our message queue. Unfortunately, every time we call the faxServer.Folders.OutgoingArchive.GetMessages the Microsoft COM dll leaks memory. With a little more research, we determined that if the ArchiveFolder was empty, there was no memory leak, and that the size of the memory leak was directly proportional to the number of files in the FAXCOMEXLib folder.

Once tracked down, we tried a number of options to dispose the object correctly including using Marshal.REleaseComObject, but with no luck. It appears that the problem is not in the disposal of the FAXCOMEXLib object, but in one of the underlying private objects or code used by FAXCOMEXLib. Since these objects are not exposed via COM, we cannot code around the memory leak.

After briefly considering restarting the windows service periodically I went and had a cup of coffee, and the solution came to me. Don't use FAXCOMEXLib.

Since I know that FAXCOMEXLib persists the completed faxes as a physical file in a known location, and that the filenames are the same as the id, we can get to the completed faxes by going through the physical filesystem rather then through the FAXCOMEXLib queue.

Here is the refactored code with no leaking:


            Dim unsentFaxRecipientTrackings As New RecipientDeliveryTrackingList

            unsentFaxRecipientTrackings.loadFromSQL("SELECT tr.* FROM tblCommRecipientDeliveryTracking tr JOIN tblCommRecipient cr ON tr.CR_Id = cr.CR_Id WHERE cr.CR_Status = 'sending'"Nothing, CommandType.Text, -1)

            If unsentFaxRecipientTrackings.Count > 0 Then

               Dim archiveFolder As String = faxServer.Folders.OutgoingArchive.ArchiveFolder

               Dim allFiles As New Hashtable

               ' Load files into hashtable

               For Each fileName As String In System.IO.Directory.GetFiles(archiveFolder)

                  ' The file name format is LoginUserID$MessageID.tif

                  If fileName.Split("$").Length > 1 Then

                     fileName = fileName.Split("$"c)(1).Replace(".tif""")

                     allFiles.Add(fileName, fileName)

                  End If


               For Each FaxTracking As RecipientDeliveryTracking In unsentFaxRecipientTrackings

                  If allFiles.ContainsKey(FaxTracking.FaxDocId) Then

                     Dim recipient As New recipient

                     recipient.loadFromSQL("SELECT * FROM tblCommRecipient WHERE CR_Id = " & FaxTracking.CR_Id.ToString(), Nothing, CommandType.Text, -1)

                     ' Might be a chance that someone remove the record from database manually

                     If Not recipient.isNew Then



                        Dim LogMessage As String = String.Format("CR_ID: {0} Fax sent successfully", FaxTracking.CR_Id)

                        Log.write(MaxSoft.Common.Log.LevelEnum.INFO, Me, LOG_AREA, LogMessage)

                     End If

                  End If


            End If




         End Try

So, rather than ask FAXCOMEXLib to return a list of messages, I go to the file system and get them myself. Happily FAXCOMEXLib aids with this approach, and will happily provide the physical path to Folders.OutgoingArchive.ArchiveFolder.

Since failed messages do not go into the OutgoingArchive.ArchiveFolder, we do not need any additional fax metadata.

Result: After running for 1 week, no memory leak found.

Tools: .NET Memory Profiler


Anonymous said...

Weird. Thanks for the tip.

Be Meo said...

i visited your site, it's very nice. Wellcome to my blog:
thank you so much!