In the application i work on, we have custom styled iconized (size changes based on dialog type) buttons, and they have corresponding classes (JS Classes) to control their behavior. jQuery UI Buttons does not suffice for us. However jQuery UI Dialog with some custom styling has been suiting our needs well (if not all). We require to place buttons on left as well on right edge of the dialog. The button widget itself has so many styling (esp. V-align ) bugs due to the inherent nature of HTML elements it uses to structure itself.
What I think:
Dialog should give a choice to use jQuery UI Buttons OR custom markup (for the entire button bar). In the latter case the Dialog algo should just care about the sizing logic regarding the button bar (i.e during dialog resizes).
The current logic makes Dialog's good and simple but not customizable to work with components that have many more behaviors and style posiibilities than jQueryUI buttons.