XML and performance

I am struggling with the performance of an xml application and would like somebody more experience than myself to give a view/perspective etc.

I have been asked to help in moving an application from our dev area onto our production servers. The app was written outside my control and done via an agency. It is a jscript/asp/xml app - webpages posting to asp-scripts and data being recorded on local .xml files. - no probs so far.

At this point we couldnt the app to our prod servers. We have rules that the webserver have no writeable directories. The app breaking at this point as it updates the local .xml files.

(we did some work using virtual directories but this is dis-allowed as well.)

So here's where I jumped in - I did some hacking into the app - I decided that I would move the data off the server on our main oracle DB. I do not (politically I cannot) want to re-write the app - I therefore elected to create a table in the following format

tab-key - 20 chr
tab-sequence - number
tab-data - 1024 chr

when the xml structure is to be saved, I want to cut it into blocks 1K and dump it

<bigdata><firststuff>........morexml.......</laststuff></bigdata>

would be saved as
database -

key - seq - data

bigdata - 0001 - <bigdata><firststuff>..
bigdata - 0002 - ....morexml............
bigdata - 0001 - ..</laststuff></bigdata>
.
.

and on reading the reverse happens.

what I did is intercept all the "doc.loadXML(strxml)" and run my load from db. On the saves, intercept all the "doc.save(Server.MapPath("data/users.xml")".

Got it all working and all ok - all except the performance.

Its a dog - my code seems to be fast enough but the speed across the network/dbase seems to be the main part of the response.

My question is this - if you are writing updating xml application, do most people have the .xml file local on the server.
(from a security position - are letting the iuser_ account update your server? - isnt this bad practice - my organisation dis-allows it fullstop...)

My solution of lobbing in over in a db is too slow as the whole xml message has to be recalled and rebuilt then given to the ms-xml parser etc (microsoft.xmldom).

I would like to limit the size of the xml data to speed things up but understand that whole structures are loaded, processed then saved.

Eg - if using local xml files - these holding employee data - the following would happen

- user enters the update employee function (say - emppost.asp runs....)
this script does

1- doc.loadXML("data/emp.xml")
2- uses the object to add/change the relevant employee structure
3- doc.save("data/emp.xml")

ok - works - what happens if the file holds hundreds of employees - we are loading & saving now a very large file just to do a small change. For me this is ok if local to the server and the file is updateable. Xml over to the db is just to slow.

My only thought is *not* to use xml structures and create a set of properly structured tables where I can just add/change the employee data as per emp key (one by one).

does xml scale (at least when using the ms objects)? - im not sure!

It seems a retrograde step to move from xml or am I reading it wrong.

any input would be greatly appreciate (hopefully I am missing something obvious and my performance prob easily fixed??? xml isnt a strong area for me - my background is s.a.p)


cheers A.J.P.
Senior AP