Drain the olives in a colander, rinse under cold water, and pat dry with paper towels.
Warm the olive oil in a medium saucepan over medium-low heat. Add the garlic, orange peel, rosemary, red pepper flakes, fennel seeds, and peppercorns. Cook, stirring, until fragrant and the garlic is pale gold at the edges, 2 to 3 minutes; do not let it brown.
Add the olives and stir to coat. Cook until heated through, 3 to 4 minutes.
Take the pan off the heat and stir in the orange juice and vinegar, then return it to low heat for 1 minute. Adding the juice off the heat keeps it from spitting in the hot oil.
Serve warm with toothpicks, or cool completely and refrigerate in a covered jar with the oil and aromatics for at least 1 hour and up to 1 week for a deeper flavor.
Let refrigerated olives stand at room temperature for 15 minutes before serving, since the oil solidifies when cold.