To run a style sheet against a large XML file (large meaning about 4-5MB in size), one can reference the file on disk instead of loading the file into the editor. This frees the memory that would have been used to show the document in the Ent, do the document highlighting, etc. When you load an XML file ('File' then 'Load XML') the load dialog has a check box (in English it reads 'Reference Only Do Not Load'), if you click the check box the file will not be loaded into the editor; However, since the Tree View uses SAX, you can still use the tree view by clicking the button. Note 'reference only' only works with files on the local file system.
You can also increase the amount of memory given to Treebeard at startup. This setting depends on what JVM you are using - see your JVM documentation for details (the Treebeard web site FAQ explain how to do it with the Sun JVM).