The wind farm layout optimization problem is concerned with the optimal location of turbines within a fixed geographical area to maximize profit under stochastic wind conditions. Previously, it has been modeled as a maximum diversity (or $$p$$ -dispersion-sum) problem, but such a formulation cannot capture the nonlinearity of aerodynamic interactions among multiple wind turbines. We present the first constraint programming (CP) and mixed integer linear programming (MIP) models that incorporate such nonlinearity. Our empirical results indicate that the relative performance between these two models reverses when the wind scenario changes from a simple to a more complex one. We then extend these models to include landowner participation and noise constraints. With the additional constraints, the MIP-based decomposition outperforms CP in almost all cases. We also propose an improvement to the previous maximum diversity model and demonstrate that the improved model solves more problem instances.