There are 2 problems I have found in the current implementation.
1. Blurring tab focus (by clicking in empty space) resets the tab index to the document and pressing tab again means you are now focussed behind the modal dialog.
I solved this myself by adding a focus event to the first tabbable thing in the page
2. the :tabbable pseudo selector still looks at all parents that are hidden. But this includes elements that take up no space, so if you have a parent element that takes no space it is possible to have something tabbable that is not being picked up by the tabbable class. In my case our designer had an inline div containing a floated block element - though I'm sure its possible to create with floats without it being invalid.