I'm building a game where each player must program his bot. The key idea is that the player will program in C (or C++, or whatever compatible language), build a DLL and send this DLL to the server, so that no one can get his code. The problem is: how to make sure that he is not calling any illegal function? Like creating files or opening a socket. The DLL will be loaded with LoadLibrary and a function will be called. All interaction will happen with callback functions. A possible solution would be placing a empty kernel32.dll (and others) so that all winapi calls will fail. Is this safe and works on every case? Is there a better way to do it?
Please note that the player thread (the one how called the dll) must still be able to comunicate with the game, maybe with an open socket. On Linux this can be easily done with seccomp.
Your best bet is to create a user with reduced privileges, which will allow you to control file access quite easily, and run the bot code in a sub-process running as that user.
If you also want to restrict network connections, it is also easy to setup a firewall so that the aforementionned process does not have the right to connect to external hosts.
If you need more control over which API calls you allow or not, there is a technique called 'API Interception via DLL Redirection' which is explained for example here:
(found via google)
This will not work without either fully analyzing all DLLs prior to loading them (which is basically impossible), or creating some sort of magic on the same level as seccomp. Given any measures you take to restrict their access, such as creating a dummy kernel32.dll, the user-submitted DLL can take countermeasures, such as loading DLLs you didn't consider, calling DLLs which were loaded by the host process (possibly via functions in the host application!), or submitting Windows system calls directly.
One option may be making use of the Windows security token functionality which Google Chrome uses to "sandbox" web renderer processes. This is incredibly arcane stuff, though, and it involves running the DLL in a separate process anyway. If you're still interested, you may want to read the Windows Chrome sandbox design documents. Keep in mind, though, that Google has spent an lot of development time getting the sandbox "right", and yet there have still been a couple of high-profile compromises -- the chances that you will be able to do the same on your own are slim, especially without extensive previous experience with Windows security.