Memory management when using MailMerge in Spire.Doc

Technical support for Spire.Doc

Moderator: iceblue support

Memory management when using MailMerge in Spire.Doc

Postby Olaf » Thu Jul 14, 2011 3:12 pm

We are evaluating Spire.doc because we need a tool to mail merge documents online. Everything works fine and fast enough if we use 5.000 records or less.

However, we need to merge 50.000 documents, and even on a 8 GB RAM dedicted server, all the memory is used and the process takes very long and ususally something crashes.

That's the code we are using:

Code: Select all
            Document document = new Document();
            document.LoadFromFile(template);

            document.MailMerge.ClearFields = true;
            document.MailMerge.RemoveEmptyParagraphs = true;

            document.MailMerge.Execute(recipients);

            document.ViewSetup.DocumentViewType = DocumentViewType.PrintLayout;
            document.SaveToFile(saveAs, FileFormat.Doc);

Is there any way to improve this? Does it help to use a DataReader?

Any hints are highly appreciated.
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Justin Weng » Fri Jul 15, 2011 2:01 am

Dear Olaf,
Thanks for your inquiry.

#1 I noticed when you mailmerge file you used recipients. What's the data struct about it?
#2 Did you try multithreading?
#3 About crash, is it memory crash or result file?
#4 What kind of your manipulation cause performance bottleneck? Please have a test.
#5 We need your some file and Data to reproduce your problem.You can upload to us or to support@e-iceblue.com
Justin
Technical Support / Developer,
e-iceblue Support Team
User avatar
Justin Weng
 
Posts: 110
Joined: Mon Mar 28, 2011 5:54 am

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Fri Jul 15, 2011 8:52 am

Dear Justin,

thanks for your reply and sorry for not being more specific.

Here's the details:

#1 recipients is of type DataTable (10.000 DataRows, 5 Columns)
#2 Didn't know that was possible with the Document object of Spire.doc, so: no
#3 Out of memory exception (hitting 16 GB usage rather quickly)
#4 I think the script is very simple and straightforward - nothing special. The bottleneck is just the sheer amount of data. So the question is: can this be done in chunks, or is there another way of creating such a huge mailmerge document?
#5 Here's the whole code. It will automatically create a DataTable with 10000 rows, and to merge you can use any mail merge Word document with some text and 4 placeholder fields named Name,Street,City,Country.

Code: Select all
        public static void Main(string[] args)
        {
            string[] fields = "Name,Street,City,Country".Split(',');
            DataTable dataTable = new DataTable();
            dataTable.Columns.Add(new DataColumn("id"));
            foreach (string field in fields) dataTable.Columns.Add(new DataColumn(field));

            int numberOfRecords = 10000;

            for (int i = 1; i <= numberOfRecords; i++)
            {
                DataRow newRow = dataTable.NewRow();
                newRow["id"] = i;
                foreach (string field in fields) newRow[field] = field + i;
                dataTable.Rows.Add(newRow);
            }

            string saveAs = "result.doc";
            string template = "any_word_file_that_contains_the_fields.doc";
            Merge(dataTable, template, saveAs);
           
        }

        private static void Merge(DataTable recipients, string template, string saveAs)
        {
            Document document = new Document();
            document.LoadFromFile(template);

            document.MailMerge.ClearFields = true;
            document.MailMerge.RemoveEmptyParagraphs = true;

            document.MailMerge.Execute(recipients);

            document.ViewSetup.DocumentViewType = DocumentViewType.PrintLayout;
            document.SaveToFile(saveAs, FileFormat.Doc);
        }


Everything works very well unless you have a large number of rows (depending on workstation or server memory).

It is ok to say that Spire.doc is not intended for so many rows. We might also try to create them in chunks and then combine the documents into one.

Thanks
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Fri Jul 15, 2011 9:15 am

Dear Justin,

another idea: is there a 64 bit version of Spire.doc? This might enable us to use more virtual memory if we compile our application in 64 bit.

Thanks, regards
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Justin Weng » Tue Jul 19, 2011 3:19 am

Dear Olaf,
Thanks for your inquiry.

We reviewed your code and find out a little problem.
After SaveToFile() method, you should call Close() method.
We used your code to merge about 50000 file, it's ok.
The memory is used about 2G.
So we think you should call Close().
Code: Select all
document.SaveToFile(saveAs, FileFormat.Doc);
document.Close();
Justin
Technical Support / Developer,
e-iceblue Support Team
User avatar
Justin Weng
 
Posts: 110
Joined: Mon Mar 28, 2011 5:54 am

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Tue Jul 19, 2011 9:06 am

Dear Justin,

thanks for your answer. It's hard for me to see, though, why closing the document should help. The memory is used up during the merge process - neither the SaveToFile nor subsequently the Close methods are ever reached.

If you really did get the code I posted to run I'd like to know in which environment. In all our tests, 8 GB physical and 8 GB virtual memory were hit and the process broke down.

Looking forward to your reply
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Justin Weng » Wed Jul 20, 2011 1:47 am

Dear Olaf,
Thanks for your inquiry.
#1 The reason why you should Close() after SaveToFile() is to free lastest used memory which is used to merge.
#2 Do you have to merge so many files at on time? If not, could you please merge files at sevral times? For instance, every one time merge 10000 files.
Justin
Technical Support / Developer,
e-iceblue Support Team
User avatar
Justin Weng
 
Posts: 110
Joined: Mon Mar 28, 2011 5:54 am

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Wed Jul 20, 2011 8:01 am

Dear Justin,

I know what a Close() usually does, but in this case it has no meaning because we are talking about only one file all the time.

And yes, that's what we will do, build chunks. I was just hoping there was another way. Spire.doc memory management is certainly not optimized.

You haven't answered the question about your environment. We were running this on a new 64 bit Windows Server, and as I described, Spire.doc uses exactly 8 GB physical and 8 GB virtual memory and then crashes. Always. We could not create one single file with 10.000 records. I really wonder how you did it.

Thanks
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Justin Weng » Wed Jul 20, 2011 8:52 am

Dear Olaf,
Thanks for your inquiry.
My eviroment is Win7 32bit 4GB memory. I upload my demo. You can have a check whether it fit your problem.
If not, please upload your demo to us so that we could know the reason why used so much memory.
You do not have the required permissions to view the files attached to this post.
Justin
Technical Support / Developer,
e-iceblue Support Team
User avatar
Justin Weng
 
Posts: 110
Joined: Mon Mar 28, 2011 5:54 am

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Thu Jul 21, 2011 8:14 am

Dear Justin,

oh come on.... what you uploaded just opens and saves a file. You have NOT performed any MailMerge.Execute(). Sorry to be that blunt, but this is no reply that has ANYTHING to do with the problem.

My code has already been posted (see above).

Sorry, Justin, but we really need a higher level of support here. Please try to run my code or write any other code that really merges 50.000 records.

Thanks
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Justin Weng » Thu Jul 21, 2011 8:35 am

Dear Olaf,
Thanks for your inquiry.
It's my fault to comment the code....Very Sorry!
I upload the code to you again. Please check it.
My enviroment is Win7 32bit 4GB. This sample do excute MailMerge.Execute().
And during the excuting, this demo used about 2GB memory.
If this can not help. I will request for a higher level of support.
Sorry for inconvenience.
You do not have the required permissions to view the files attached to this post.
Justin
Technical Support / Developer,
e-iceblue Support Team
User avatar
Justin Weng
 
Posts: 110
Joined: Mon Mar 28, 2011 5:54 am

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Mon Aug 01, 2011 2:06 pm

Dear Justin,

thanks again for your answer (and sorry this reply took some time).

The code you posted is exactly the code I posted. Good to know it worked for you, using less than 2 GB. But there is no change in the code.

The reason for the MUST be that your sample doc was so small. I can't imaging anything else.

Can I ask you to try again, using the document you find here as template: http://www.raecke-schreiber.de/download ... nonym.docx ?

And confirm you still need only 2 GB?

Thanks very much for your support
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm

Re: Memory management when using MailMerge in Spire.Doc

Postby Justin Weng » Tue Aug 02, 2011 6:34 am

Dear Olaf,
Thanks for your inquiry.
I reproduced your problem with your new document file.
And then, I asked my leader about this question.
In Spire.Doc a complete document is kept in memory, so if you have a large document, it will take more memory. that a document size on disk is usually smaller than document size in memory. There is no fixed relationship between disk size and memory size, it depends on the complexity of the document, formatting etc,
So we recommend you to merge for more times/more files.
We also try to optimize memory management.
Justin
Technical Support / Developer,
e-iceblue Support Team
User avatar
Justin Weng
 
Posts: 110
Joined: Mon Mar 28, 2011 5:54 am

Re: Memory management when using MailMerge in Spire.Doc

Postby Olaf » Thu Aug 11, 2011 9:01 am

Dear Justin,

that makes sense. Thanks for all your support. Basically, Spire.doc does all we want, so we will live with the memory issue and hope for improvements in future version (and maybe also for a 64bit version).

Best regards
Olaf
Olaf
 
Posts: 9
Joined: Wed Jul 13, 2011 2:35 pm


Return to Spire.Doc

Who is online

Users browsing this forum: No registered users and 1 guest