Altruistic punishment of defectors is critical for the evolution of cooperation in large groups of nonrelatives. Laboratory and ethnographic evidence show people’s willingness to punish defectors at a cost to themselves, and the effectiveness of such altruistic punishment in sustaining cooperation. However, an evolutionary puzzle remains: how can altruistic punishment evolve despite the payoff disadvantage of punishers? Group selection models illuminate that the selection of cooperative groups can increase the evolution of altruistic punishment. These models generally assume that participation in a public goods project is obligatory, i.e., individuals lack alternate options (or “exit options”) to the project. However, this assumption does not reflect real-world conditions where individuals often exercise the exit options. I address this gap by developing an agent-based model (ABM) in which group selection occurs, and group members can opt out of the public goods project. The model results show that voluntary participation has negative effects on the evolution of cooperation in small groups, regardless of changes in exit payoff, whereas it has positive effects in large groups, given a certain range of exit payoffs. Such asymmetric effects can provide insights into human response to social changes that lead individuals to opt out of traditional collective-action arenas.