Phase transitions in anisotropic superconducting and magnetic systems with vector order parameters: Three-loop renormalization-group analysis.

The critical behavior of a model with N-vector complex order parameters and three quartic coupling constants that describes phase transitions in unconventional superconductors, helical magnets, stacked triangular antiferromagnets, superfluid $^{3}\mathrm{He}$, and zero-temperature transitions in fully frustrated Josephson-junction arrays is studied within the field-theoretical renormalization-group (RG) approach in three dimensions. To obtain qualitatively and quantitatively correct results perturbative expansions for \ensuremath{\beta} functions and critical exponents are caluculated up to three-loop order and resummed by means of the generalized Pad\'e-Borel procedure. Fixed-point coordinates, critical-exponent values, RG flows, etc., are found for the physically interesting cases of N=2 and 3. Critical (marginal) values of N at which the topology of the flow diagram changes are determined as well. It is argued, on the basis of several independent criteria, that the accuracy of the numerical results obtained is about 0.01, an order of magnitude better than that given by resummed two-loop RG expansions. In most case the systems mentioned are shown to undergo fluctuation-driven first-order phase transitions. Continuous transitions are allowed in hexagonal d-wave superconductors, in planar helical magnets (into sinusoidal linearly polarized phase), and in triangular antiferromagnets (into simple unfrustrated order states) with critical exponents \ensuremath{\gamma}=1.336, \ensuremath{\nu}=0.677, \ensuremath{\alpha}=-0.030, \ensuremath{\beta}=0.347, and \ensuremath{\eta}=0.026, which are hardly believed to be experimentally distinguishable from those of the three-dimensional XY model. The chiral fixed point of RG equations is found to exist and possess some domain of attraction provided N\ensuremath{\ge}4. Thus, magnets with Heisenberg (N=3) and XY-like (N=2) spins should not demonstrate chiral critical behavior with unusual values of critical exponents; they can approach the chiral state only via first-order phase transitions.