This study aims at providing a better understanding of the effects of both microstructure and defect on the high cycle fatigue behavior of metallic alloys using finite element simulations of polycrystalline aggregates. It is well known that the microstructure strongly affects the average fatigue strength and when the cyclic stress level is close to the fatigue limit, it is often seen as the main source of the huge scatter generally observed in this fatigue regime. The presence of geometrical defects in a material can also strongly alter the fatigue behavior. Nonetheless, when the defect size is small enough, i.e. under a critical value, the fatigue strength is no more affected by the defect. The so-called Kitagawa effect can be interpreted as a competition between the crack initiation mechanisms governed either by the microstructure or by the defect. Surprisingly, only few studies have been done to date to explain the Kitagawa effect from the point of view of this competition, even though this effect has been extensively investigated in the literature. The primary focus of this paper is hence on the use of both FE simulations and explicit descriptions of the microstructure to get insight into how the competition between defect and microstructure operates in HCF. In order to account for the variability of the microstructure in the predictions of the macroscopic fatigue limits, several configurations of crystalline orientations, crystal aggregates and defects are studied. The results of each individual FE simulation are used to assess the response at the macroscopic scale thanks to a probabilistic fatigue criterion proposed by the authors in previous works. The ability of this criterion to predict the influence of defects on the average and the scatter of macroscopic fatigue limits is evaluated. In this paper, particular emphasis is also placed on the effect of different loading modes (pure tension, pure torsion and combined tension and torsion) on the experimental and predicted fatigue strength of a 316 stainless steel containing artificial defect.< Réduire