Solution
The FileSelectPopupEx function use a specific threading technique, which if combined with other functions which require multi-threading, such as some ActiveX functions, this error will appear.
To ensure that these functions are using the correct type of threading you must include the following line of code at the start of your function, specifying the threading method before any other function does.
CA_InitActiveXThreadStyleForCurrentThread (0, COINIT_APARTMENTTHREADED)
If you are using TestStand, you need to add
CA_InitActiveXThreadStyleForCurrentThread (0, COINIT_APARTMENTTHREADED)
before calling
TS_NewEngine()
Once this line is included, continue the code as normal and you should not see this error.