XML Binding, COM/ActiveX, Threads, Oh my..

Using XE7, 64 bit, win32.

After doing searching, it seems this kinda an old "issue" with a memory leak type situation when using IXMLNODE and the such in threads.

Each thread runs CoInitializeEx(nil, COINIT_MULTITHREADED) and CoUninitialize...

I have and old program with a problem where I have to process over 10000+ XML files in one blast. All the same layout. Ranging as small as 1k.. upto 8mb. I need EVERY BIT of data out of these.

The memory usage climbs up very high. And when program is done running the threads. Its about 40mb higher. Take it back to single thread setup works great, mem is all ok.

All the code currently generated via the XML mapper, which works great and been in use for a long time. 

The current use program was single thread, so decided hey, finally  move to this to threading since we got good multi core systems now. Well that is when I found out about all this mess.

Is there any replacement packages to use my existing code that the XML mapper generated? Only reason I ask there is nearly 100 of these XML mapped sources to deal with and same problems :)

Don't really have time to rewrite all these by hand if gotta use some other DOM/SAX/whatever. Be nice to have semi-drop in solution to use the existing code.
0
steven
4/21/2015 2:49:53 AM
embarcadero.delphi.xml 291 articles. 0 followers. Follow

8 Replies
610 Views

Similar Articles

[PageSpeed] 9

I followed this

http://community.embarcadero.com/index.php/blogs/entry/setting-the-default-xml-dom-in-delphi-xe7-1

OMNI takes ~14x longer to run and just as bad .. actually worse, when it comes to memory. 

ADOM couldn't open anything.
0
steven
4/21/2015 4:37:22 AM
steven chesser wrote:

> Using XE7, 64 bit, win32.
> 
> After doing searching, it seems this kinda an old "issue" with a
> memory leak type situation when using IXMLNODE and the such in
> threads.
> 
> Each thread runs CoInitializeEx(nil, COINIT_MULTITHREADED) and
> CoUninitialize...

In such a setup you have to make sure you create *every single* COM
object, including the DOM document you load the file into, in the
context of the secondary thread. And of course the COM library you are
using (MSXML in this case) has to actually support free threading. I'm
not sure it does...

> The memory usage climbs up very high. And when program is done
> running the threads. Its about 40mb higher. Take it back to single
> thread setup works great, mem is all ok.

If you use MSXML the memory for the nodes is allocated by it, not by
the Delphi memory manager...



-- 
Peter Below (TeamB)
0
Peter
4/21/2015 4:56:55 PM
> {quote:title=Peter Below wrote:}{quote}
> steven chesser wrote:
> 
> > Using XE7, 64 bit, win32.
> > 
> > After doing searching, it seems this kinda an old "issue" with a
> > memory leak type situation when using IXMLNODE and the such in
> > threads.
> > 
> > Each thread runs CoInitializeEx(nil, COINIT_MULTITHREADED) and
> > CoUninitialize...
> 
> In such a setup you have to make sure you create *every single* COM
> object, including the DOM document you load the file into, in the
> context of the secondary thread. And of course the COM library you are
> using (MSXML in this case) has to actually support free threading. I'm
> not sure it does...
> 
> > The memory usage climbs up very high. And when program is done
> > running the threads. Its about 40mb higher. Take it back to single
> > thread setup works great, mem is all ok.
> 
> If you use MSXML the memory for the nodes is allocated by it, not by
> the Delphi memory manager...
> 
> 

I am not 100% understanding? .. This is my current code in a demo program I did last night playing around.

{code}

procedure tLoadXML.Execute;
var i : IXMLACES; // XML binded code Delphi created
    x : integer;
begin
   CoInitializeEx(nil, COINIT_MULTITHREADED);
   xlist := tstringlist.Create;
   xlist.LoadFromFile('xml\list.txt');
   for x := 0 to xlist.Count-1 do
    begin
      sleep(1);
      i := LoadACES('xml\'+xlist[x]);
      i := nil;
    end;
  xlist.Free;
  CoUninitialize;
end;
{code}

To run the code I have this on main form.
{code}
 ..

 t1 := tloadxml.Create(true);
 t1.FreeOnTerminate := true;
...
...
t1.start;
{code}

As for the MSXML memory manager. With that above running.. each time it runs, memory consumed goes up about 10% with every run of it.
Peak memory usage is always roughly the same. Which is fine. If I run this enough times, it will eventually error out with out of memory.
0
steven
4/21/2015 5:18:56 PM
steven chesser wrote:

> > {quote:title=Peter Below wrote:}{quote}
> > steven chesser wrote:
> > 
> > > Using XE7, 64 bit, win32.
> > > 
> > > After doing searching, it seems this kinda an old "issue" with a
> > > memory leak type situation when using IXMLNODE and the such in
> > > threads.
> > > 
> > > Each thread runs CoInitializeEx(nil, COINIT_MULTITHREADED) and
> > > CoUninitialize...
> > 
> > In such a setup you have to make sure you create *every single* COM
> > object, including the DOM document you load the file into, in the
> > context of the secondary thread. And of course the COM library you
> > are using (MSXML in this case) has to actually support free
> > threading. I'm not sure it does...
> > 
> > > The memory usage climbs up very high. And when program is done
> > > running the threads. Its about 40mb higher. Take it back to single
> > > thread setup works great, mem is all ok.
> > 
> > If you use MSXML the memory for the nodes is allocated by it, not by
> > the Delphi memory manager...
> > 
> > 
> 
> I am not 100% understanding? .. This is my current code in a demo
> program I did last night playing around.
> 
> {code}
> 
> procedure tLoadXML.Execute;
> var i : IXMLACES; // XML binded code Delphi created
>     x : integer;
> begin
>    CoInitializeEx(nil, COINIT_MULTITHREADED);

Have you tried to use  COINIT_APARTMENTTHREADED as a test to see
whether it changes the outcome?


>    xlist := tstringlist.Create;
>    xlist.LoadFromFile('xml\list.txt');
>    for x := 0 to xlist.Count-1 do
>     begin
>       sleep(1);
>       i := LoadACES('xml\'+xlist[x]);
>       i := nil;
>     end;
>   xlist.Free;
>   CoUninitialize;
> end;
> {code}
> 
> To run the code I have this on main form.
> {code}
>  ..
> 
>  t1 := tloadxml.Create(true);
>  t1.FreeOnTerminate := true;
> ..
> ..
> t1.start;
> {code}
> 
> As for the MSXML memory manager. With that above running.. each time
> it runs, memory consumed goes up about 10% with every run of it.

And if you run this loop in the main thread context that does not
happen? Weird, but if COM is involved weirdness is the norm sometimes.
I remember one particular problem I ran into with using a COM server in
a secondary thread were I found no other solution than to move the
processing to an external program that was launched by the main one (a
service) to perform the operation I had previously tried in a secondary
thread in the main thread context....

> Peak memory usage is always roughly the same. Which is fine. If I run
> this enough times, it will eventually error out with out of memory.



-- 
Peter Below (TeamB)
0
Peter
4/22/2015 6:38:04 PM
> {quote:title=Peter Below wrote:}{quote}

> And if you run this loop in the main thread context that does not
> happen? Weird, but if COM is involved weirdness is the norm sometimes.
> I remember one particular problem I ran into with using a COM server in
> a secondary thread were I found no other solution than to move the
> processing to an external program that was launched by the main one (a
> service) to perform the operation I had previously tried in a secondary
> thread in the main thread context....

Correct.. move it to main thread, seems to act more behaved. Which is kinda
what I found in searching around for this problem. I guess sloppy COM design and implementation by
languages or something.

So with that said.. I wonder if any other drop in solutions I can use with the existing code, that is
fast and memory wise... Omni option worked, but its stupid slow, and memory usage was even worse.

Kinda boggles my mind that something like XML is still a problem to deal with now a days.
0
steven
4/23/2015 6:17:10 AM
steven chesser wrote:

> > {quote:title=Peter Below wrote:}{quote}
> 
> > And if you run this loop in the main thread context that does not
> > happen? Weird, but if COM is involved weirdness is the norm
> > sometimes.  I remember one particular problem I ran into with using
> > a COM server in a secondary thread were I found no other solution
> > than to move the processing to an external program that was
> > launched by the main one (a service) to perform the operation I had
> > previously tried in a secondary thread in the main thread
> > context....
> 
> Correct.. move it to main thread, seems to act more behaved. Which is
> kinda what I found in searching around for this problem. I guess
> sloppy COM design and implementation by languages or something.
> 
> So with that said.. I wonder if any other drop in solutions I can use
> with the existing code, that is fast and memory wise... Omni option
> worked, but its stupid slow, and memory usage was even worse.
> 
> Kinda boggles my mind that something like XML is still a problem to
> deal with now a days.

It all depends on how you need to process the file. If you can do a
sequential read then simply use a SAX parser (I have had good success
with SAX for Pascal http://saxforpascal.sourceforge.net/ in the past)
instead of a DOM parser. This way *you* are in complete control of how
you store the data from the file.

-- 
Peter Below (TeamB)
0
Peter
4/23/2015 8:46:21 AM
> {quote:title=Peter Below wrote:}{quote}

> It all depends on how you need to process the file. If you can do a
> sequential read then simply use a SAX parser (I have had good success
> with SAX for Pascal http://saxforpascal.sourceforge.net/ in the past)
> instead of a DOM parser. This way *you* are in complete control of how
> you store the data from the file.
> 

All the data in XML files are stored.. All of it. The DOM approach works.. a bit memory hoggy but does work and the
memory leak is kind of a pain, but appears no way around it. Design flaw of it and MS isn't gonna do anything about
it anytime soon.

SAX i've looked into a few times before, but for what I need, its just way more complicated then really desired. I think when
we last looked into it and wrote some samples with it, for way we need to deal with stuff, it ended up being WAAAAY more 
code and work on our end and speed wise, didn't work any faster in the end. Plus side was memory use was lower. But
this runs on one machine in our data center and crunch it all, so its not making it to any customers to use or anything. 

So I think for now, will just keep the years worth of work with the already wrote code with the XML binding stuff done and
just deal with the memory leaks and adjust our thread count of processes to run depending on the source files.

If the majority are "smallish" we can run 8 threads fine... if they are big freaking files, run only two threads. Still faster than
doing 1 by 1 at  a time like before.
0
steven
4/24/2015 3:26:24 AM
Dear Sir,

This a bit far of the issue, but with all those DOM problems i just wonder why(?) Embarcadero encourage the use of Dom instad of the use of this better tools such that Sax. After investing in Delphi XE7 Enterprise, i've been very concerned about the way that Embarcadero products are loosing market... The policy with Univerty Students vs Microsoft pruducts like #C for free for those users,  while they have to pay for the Embarcadero etc...
Later, after they finish their degree and they go to work and recommend #C, not Delphi neither C++Builder!!!
After all it's hard  investing  much more money in a tool like Enterprise wich is suposed be a RAD generating XML, but at at the end  there is a "big problem"... 

Mário Reis

> {quote:title=steven chesser wrote:}{quote}
> Using XE7, 64 bit, win32.
> 
> After doing searching, it seems this kinda an old "issue" with a memory leak type situation when using IXMLNODE and the such in threads.
> 
> Each thread runs CoInitializeEx(nil, COINIT_MULTITHREADED) and CoUninitialize...
> 
> I have and old program with a problem where I have to process over 10000+ XML files in one blast. All the same layout. Ranging as small as 1k.. upto 8mb. I need EVERY BIT of data out of these.
> 
> The memory usage climbs up very high. And when program is done running the threads. Its about 40mb higher. Take it back to single thread setup works great, mem is all ok.
> 
> All the code currently generated via the XML mapper, which works great and been in use for a long time. 
> 
> The current use program was single thread, so decided hey, finally  move to this to threading since we got good multi core systems now. Well that is when I found out about all this mess.
> 
> Is there any replacement packages to use my existing code that the XML mapper generated? Only reason I ask there is nearly 100 of these XML mapped sources to deal with and same problems :)
> 
> Don't really have time to rewrite all these by hand if gotta use some other DOM/SAX/whatever. Be nice to have semi-drop in solution to use the existing code.
0
Utf
4/25/2015 1:45:06 AM
Reply:

Similar Artilces:

XMl to XML
Hello,   I retrieved an XML from the dataset which retrieves data from the table in the database. I need to present the data  in a different structure. Is XSLT the way or are there any other options. please let me know. Thanks!! Yes XSLT is a good choice to convert xml file to other forms. You can take a look at XSLT Tutorial.Sincerely,Young Fang...

XML in, XML out
Hi All (and a happy holiday to those that will get a break), I am trying to read in an XML file of addresses. I need to remove all the address data from the file where code =~ /^000/ (there are none in the example data below). I need to reproduce that data 'as is', so I need to honour the tag structure, although the order of the tags doesn't need to be honoured. I have been trying to use XML::Simple and I had a go with XML::Smart but I haven't been able to get the results I want. My best effort is below (with XML::Simple). There are a couple of differences ...

XML::Parser XML::DOM XML::XSLT
------_=_NextPart_001_01C6E0D0.3E27D77A Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Hi Craig and fellow VMS Perlers, You last wrote about this Fri 3/16/2001 11:24 AM, not counting your very helpful comments on Eisner/Decuserve more recently. One of my colleagues sent me an email requesting that I make XML::XSLT=20 available on VMS. perl 5.8.6, VMS 7.3-2, DECC 7.1 latest ACRTL (version 3) on Alpha XML::XSLT (version 0.48) depends on XML::Parser (version 2.34). This depends on expat (version 2.0.0). I built expat a...

sort xml in xml
Hi, How I can sort xml file in xml with VB.NetThanks,<?xml version="1.0" encoding="ISO-8859-1" ?>  <Liste>   <Form Famille="Administration" idXhtml="" idOffice="31574" nom="ADM" desc="" />   <Form Famille="Admini" idXhtml="" idOffice="31708" nom="ADM- 54 (2000)" desc="" />   <Form Famille="communication" idXhtml="" idOffice="31709" nom="ADM- 133 (2000)" desc="" />   <Form Famille="Admini" idXhtml="" idOffice="31710" nom="ADM- 9 (2000)" desc="" />   <Form Famille="Admini" idXhtml="" idOffice="31711" no...

XML to XML transform
Hi all, I have an XML file in which I just want to change a couple of values.  So, the end result is a new file with exactly the same XML structure.  I am new to XML and XSLT, so I thought I'd start off with something simple; I created an xsl file that would just copy the source directly across without modifying the values (I can do this later, once I have the first bit figured) - <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:con="http://eviware.com/soapui/config"><xsl:template match="/"...

Transform XML to XML
I've been trying to transform an XML document to another XML document. The original document is very large (6megs) and the application that needs to use it only needs a fraction of the data. I'm trying to take this mother-of-all XML documents and filter out unwanted nodes. I could probably do this very easily with code by running a few XPath Queries against the document, and placing the results in a new document...however...I have an interest in learning how to do this with XSL. In the past, I've done some pretty basic XML to HTML transformation ans that worked out well. This ti...

Delphi Threads XML slow
Hello, why does Button1Click takes ~ 1 sec. and Button2Click ~ 2 sec? See code below. I have a 5 MB XML file. If I load/parse this file in main VCL thread it takes about 1 sec. (Button1Click) If I load/parse this file (a copy of this file for each thread) in two different threads at the same time it takes about 2 sec. (Button2Click) Theoretically it should take the same time (with a little thread creation overhead). I have tried other XML Parsers (OXML, JvSimpleXML, MSXML6) which are faster than TXMLDocument but with multithreading it takes really more time. The same with t...

How to select Xml as Xml?
If I select Xml nodes like this from sql server; select xmlData.query('//Field[@F = "True"]') as extendedInfo from.... ..how to loop through the items in C#? I can't cast it to a System.Xml.XmlNodeList..? You can use the XmlDocument class.  There's a Load method and a LoadXml method.  One of these should get you what you want, after which you can select nodes as you need.C# <---> VB.Net Translator mm8: If I select Xml nodes like this from sql server; select xmlData.query('//Field[@F = "True"]') as extendedInfo from.... ...

bind xml doc to datagrid,how to update the changes to xml doc
when i use DataSet.writexml(filename), i want to write the DataSet to the file "filename", but the new data is add to the end of the "filename",but i want to update the file. DataSet ds = new DataSet(); ds.readxml("d:\\aa.xml"); and then i binddata the ds to DataGrid,i edit the DataGrid i want to update the file,so i use ds.writexml("d:\\aa.xml"); but the new data add to the end of aa.xml how can i update the aa.xml but not add to the end of file if anyone advise other sln ,i will very appreciated. help, thank...

XML Transform to 1 XML file to another XML File
Hello All  I am a novice programmer. I want to transform 1 xml file to another xml file. I am looking for a example code where user will select source xml file and create another xml file in different file with different fields with some default validation of parsing date and default values.  Can anyone help me or let me know any available source code in C#/ASP.Net?  Ms Disha I am assuming you know xslt, if not, you would have to go look at that.  But if you do, then this document will help you out http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=...

Exception on: com.sybase.xml.xql.Xql.query('/', xml)
Has anyone encountered this problem? I am using Jconnect 5. I want to read out all data from a parsed XML Image type column. select com.sybase.xml.xql.Xql.query('/', xml) from xmltest where recid = 101 The above works fine via ISQL. But using JDBC, I get the following exception when I attempt to execute resultSet.getString(1); java.io.UTFDataFormatException at java.lang.Throwable.fillInStackTrace(Throwable.java) at java.lang.Throwable.<init>(Throwable.java:82) at java.lang.Exception.<init>(Exception.java:32) at java.io.IOException.<init>(...

Creating a xml file with just field names and then binding the xml fields to a datasource???
I have an xml file I created called ReportFields.xml which looks like this: <?xml version="1.0" encoding="utf-8" ?> <ReportFields><AccountNumber></AccountNumber <FirstName></FirstName><LastName></LastName> </ReportFields> I then create a blank crystal report and go into the Database Expert and choose ADO.NET DataSets, I browse to the xml file I just created, in this case ReportFields and give it a Class Name of ReportFields.  Now under Database Fields, I have a ReportFields table with the following columns: ...

XML Data Binding in Delphi 2010 Professional?
I have Delphi 2010 Pro, and I have access to the XML Data Binding wizard. I tested it against an XML file and it worked great. So my question is: Has this feature been officially moved to the Professional SKU? I'm wondering if this was an oversight and it might be removed from Professional in a future update. The feature matrix indicates that it's Enterprise only. Thanks, Troy > {quote:title=Troy Wolbrink wrote:}{quote} > I have Delphi 2010 Pro, and I have access to the XML Data Binding wizard. I tested it against an XML file and it worked great. So my question is: ...

Problem loading an xml string containing xml data into XML document
 Hi all,I have a problem loading an xml string containing xml response data returned by other server(spatial server) into XML document some times becoz the returned xml response contains special chars like <, >, ', etc.Is there any way to parse this XML response string before loading into XML document. Hi Bharath, if you can't find a ready made one you can write one yourself. Parse the string that you get for the elements that it would contain and inject the CDATA element between them to allow special character. example if you have a string <childNode>This con...

Web resources about - XML Binding, COM/ActiveX, Threads, Oh my.. - embarcadero.delphi.xml

Tight binding - Wikipedia, the free encyclopedia
In solid-state physics , the tight-binding model (or TB model ) is an approach to the calculation of electronic band structure using an approximate ...

Facebook To Hold Non-Binding Shareholder Advisory Votes On Executives’ Compensation Every Three Years ...
... filing with the Securities and Exchange Commission Wednesday that, as voted on at its annual meeting in June, the company will hold non-binding ...

Molecule World DNA Binding Lab
Get Molecule World DNA Binding Lab on the App Store. See screenshots and ratings, and read customer reviews.

"Big binding crab"s on the plate - Flickr - Photo Sharing!
... this crab. But they were nice (^_^) Thank you raemin for ID !! from Wiki "The Chinese mitten crab (Eriocheir sinensis), also known as big binding ...

The Binding of Isaac: Rebirth Release Date Trailer - YouTube
PRE-ORDER NOW! http://store.steampowered.com/app/250900 IT'S HAPPENING! The Binding of Isaac: Rebirth coming to PlayStation 4, PS Vita and Steam! ...

Bill Shorten, Anthony Albanese reject call for binding vote on same-sex marriage
Union leader Paul Howes' call for the ALP to move to a binding vote in support of same-sex marriage has been rejected by Labor's potential leaders. ...

NSW seeks binding bids in Delta Coastal sale
Participants view AGL’s progress in the MacGen deal as a good omen for regulatory consent.

Abigail claims victory with a hushed, spell-binding display
As Abigail Carungay stepped up to the microphone in the final rounds of the Premier's Spelling Bee, the audience leaned forward in their chairs ...

Canada backs U.S., saying carbon-reduction targets shouldn't be legally binding
... approach to major climate change talks in Paris, saying any carbon reduction targets agreed to at the negotiations should not be legally binding. ...

Paris climate talks shift from binding targets to ‘name and shame’
Even if the international climate accord being negotiated in Paris becomes legally binding, it won't include punitive measures like trade sanctions ...

Resources last updated: 1/4/2016 7:01:35 AM