You are barking up the right tree.
Basic Teachings say that our own defiled mind, emotions, and actions are what keep us in the cycle of dissatisfaction and suffering. So what you are taking refuge from is your own mind. You can put it in the terms of refuge from your own b.s., or your own limited understanding, or any of a number of parts of your internal makeup you choose. But chief among them is "taking refuge" from your own criticisms, preconditions, and bad behavior. But most of all, we take refuge from our fear(s)
Since this website is Mahayana & Vajrayana oriented, let us say that it is a given that underneath it all, that there is a "rightness" (self-perfection) to all things. It is we that have made things all wrong by how we view life, feel about it, and behave (thus creating karmically negative situations). The idea of "refuge" contains within it the idea of safety. So we stop relying on our fears to keep us safe, and find safety in the truly sane. Relying on our fears to keep us safe is saying, "I don't know how to do this, and it could be painful, so I must avoid it." Our fears are the initial experience of our problem.
Don't worry, you won't loose anything that you would want to keep. A sick patient wants to loose his illness, unless he is a hypochondriac that is afraid of being well. A prison inmate wants to lose captivity, unless he is frightened by the idea of freedom. So when you are doing your practice, don't struggle with your fears, just recognize them as being confronted by the limitation of the moment.
I`m focusing on Chenrezig
There is absolutely nothing to fear in Chenrezig.