How to forbid a .NET DLL class library to be referenced
How can I forbid dll class library to be referenced in other solutions?
Alternatively, you could explore using InternalsVisibleToAttribute, although it may require some design changes in your library code. This should work as long as neither assembly is signed, or both are signed with a strong name. The argument specified on the attribute should match the public key and the name of the assembly that you want to be able to access its internal members.
But really, this will only stop someone who isn't trying very hard to use your library. They won't be able to add a reference, but it doesn't prevent someone from bypassing it through Reflection or disassembling your code. There are always ways around almost any security measure that you implement.