CFileDialog is retaining handles to directory under Windows XP
I have been battling this issue for a day and a half and cannot find an answer. I have reduced the code down to this simple bit of code for opening a save dialog:
CFileDialog dlg( FALSE, _T("xml"), _T("zzz.xml"), OFN_PATHMUSTEXIST | OFN_NOCHANGEDIR, _T("XML Files|*.xml||"), this); dlg.m_ofn.lpstrInitialDir = ... a string for the initial directory dlg.DoModal();
The CFileDialog class (also verified just using GetSaveFileName directly) is retaining handles to the directory that is chosen, but apparently only under Windows XP (which is our deployment platform). This prevents renaming the directory while the application is still running.
Before anyone starts talking about CFileDialog setting the current directory, I know about that and I know about OFN_NOCHANGEDIR. This is not that issue.
Running this code in a loop causes increasing numbers of handles to be retained which can be verified using Process Explorer to see that the application has Handles to that directory. Repeatedly running the dialog causes additional handles, but not always. If I change the name every time then it seems to add a new one each time although not 100% consistently.
Every time I think I have something that works around it, it doesn't pan out. How can such a basic thing like the file dialogs be this broken?
I don't know what else to do. This is a problem discovered just before we are about to release a new version.
Given that MFC's CFileDialog as well as the underlying Windows API have been around for a long time, my first reaction is to doubt they have a bug that's leaking handles so obviously. So I would guess the issue you're seeing is coming from something else.
One thing about the file dialog is that it loads shell extensions. For example, the various third-party right-click features that you would see in Windows Explorer will also be available in the file dialog you invoke in your own program. That's ok if the shell extensions are well behaved, but some aren't. So there's a chance the system(s) you have tested this on have a poorly written shell extension installed which ends up leaking handles when used by the file dialog.
I've seen cases before where these extra components loaded by the file dialog caused issues for other parts of a piece of software, but only on certain people's computers because not everyone has the same stuff installed.
Of course, I don't have enough information to say this is actually the problem for you. I can only offer the idea and suggest that you might try testing for this issue on different Windows XP computers which weren't setup similarly. You could also try testing for it on a completely fresh installation of Windows XP before anything else is installed. If the problem doesn't show up on some computers, then that's a good indication that some third-party component is the problem.
I have discovered what the culprit was. The application in question is not very well written and frequently does long running tasks on the main UI thread. There is also other code that runs on the same machine that controls a physical device. One of the problems encountered wit the other code was that it was getting held off by the application in question which caused it to stop. Unfortunately, they had to turn off hyperthreading to fix another problem, whcih would allow the two to coexist.
So in order to allow the two applications to coexist and to give the machine control program priority over this app the application's thread priority was lowered to the lowest priority.
Turns out this lowering of thread priority is what causes it. So to put another band-aid on top of the band-aid, on top of the band-aid. I will create a CFileDialog subclass that bumps the thread priority back up for the file dialogs and bumps it back down when it returns.