Why does COM CoInitializeSecurity fail in my DLL?

Question

I'm currently studying VSHADOW.EXE 3.0 from the MS Windows SDK 6.1. I have made a version which can be compiled into a DLL that only exports one newly written function which expects the commandline as a string, tokenizes it, and then calls the old wmain. The DLL is not a COM server.

It works exactly as the old one when compiled as an EXE but doesn't quite work when compiled as a DLL because this call fails:

 CoInitializeSecurity(NULL, -1, NULL, NULL, 
                      RPC_C_AUTHN_LEVEL_PKT_PRIVACY, 
                      RPC_C_IMP_LEVEL_IDENTIFY, 
                      NULL, EOAC_NONE, NULL);

which fails with HRESULT error 0x80010119 (RPC_E_TOO_LATE, Security must be initialized before any interfaces are marshalled or unmarshalled. It cannot be changed once initialized.)

I run the exported function from a VB6 program where the function is imported with Declare Function vss Lib vshadow.dll ....

Does the error mean that the VB6 program already called CoInitializeSecurity? What can I do against the error?

Also, I have another question: why were exactly the security values RPC_C_AUTHN_LEVEL_PKT_PRIVACY and RPC_C_IMP_LEVEL_IDENTIFY chosen? What impact would other settings have?

Solution

There are a couple of standard COM calls that do not belong in a DLL. Like CoInitializeEx(), the call that initializes COM for a thread. The DLL doesn't own the thread, it is powerless to override the apartment state that the EXE selected.

CoInitializeSecurity() is another one, it is the job of the EXE to call it. Only it knows the proper values to pass, it's the one that determines the security policy. A DLL can't, it doesn't know anything about the client process.

Comments

[Felix Dombek]

OK, I need some clarification here ... (1) what proper values does the EXE know and the DLL doesn't? If the project compiled as an EXE uses exactly the same values as the DLL, why are they not correct? There isn't even a unique ID in the CoInitializeSecurity parameters ... (2) Do you have any ideas how I could adress this problem and initiate volume shadow copies from a DLL? The current state of the DLL is only intermediate; I'm only practicing to write a general VSS DLL.

[user1703401]

It's pretty simple, the DLL might be used by multiple processes. It is the process that determines the security context it needs. I would strongly recommend you forget about this, COM security is ugly to get right and largely irrelevant in non-DCOM scenarios.

[Felix Dombek]

@Hans: Well, my boss told me to "write a DLL which can create volume shadow copies" ... and I've heard of DLLs which can do this, there must be a way!

[user1703401]

Well, make your boss happy and actually write a DLL that does that. Leave it up to the system administrator to run whatever process that uses your server to have sufficient privileges.

[user1703401]

You are just looking at boilerplate code, MSFT includes it in all their sample code. The passed arguments don't actually change anything from the default. Again, as long as it is your job to only provide a DLL then security is not your concern. I'm signing off, good luck with it.

[Felix Dombek]

OK well, I'm kind of slow and inexperienced, but what you probably wanted to tell me was: "Just remove BOTH the CoInitialize AND CoInitializeSecurity calls from your DLL and it will work fine." which I found out after some trial and error and talking to my boss. Well then, thanks ...