I would like to add some information to this. We have a website that unfortunately we have to keep in quirks mode for the time, but are starting to add some jQuery/jQueryUI. Everything has worked great except when we use IE, then dialog boxes just don't work quite right. After some investigation, I discovered that the _size function in the dialog code sets the height of the contained node to 0 to check the height of the dialog wrapper and subtract it from the requested height. I've found that IE in quirks mode will treat a height of 0 as auto, causing the problem I've been experiencing. I have not been able to find any documentation of this quirk on the internet.
I am attaching a test html page that will demonstrate the error and a possible fix. Each time you click on the open link, it will add a random number to the end of the div. In IE, every time you click on it, the window will get smaller and smaller until it gets too big that it will then become bigger and bigger. I assume this oddity happens because of the min-height setting. Another oddity is if you open a dialog and attempt to drag it, it will attempt to correct the size by making the button area much much larger. Clicking on the fix link will install a modified 1.8a2 _size function that uses a height of 1px to calculate the height of the wrapper, and then compensates for that 1px. After that, the dialog in IE will work as expected.
I apologize that the html is not completely self contained, it does require being pointed at a version of jQueryUI 1.8b2. I currently have it pointed to my local copy at "/static/js/jquery-ui.min.js".
I hope this helps.