Below Custom Audience, it asks you to choose demographics & interests for your audience. I don't want it to send to random people from that type of audience.
Once you've built your Custom Audience, you'll have to select it at Ad Set level in the targeting section - it's right above location, demographics and interests. As long as you've selected your specific custom audience it will only target that audience.
If you add other criteria to your targeting, say you add a 20-55 year bracket and men only on top of using your custom audience, then it will target people in your custom audience within that age group and only men. Does that make sense?