Fractional Chern Insulators vs. Non-Magnetic States in Twisted Bilayer MoTe$_2$

Jiabin Yu, Jonah Herzog-Arbeitman, Minxuan Wang, Oskar Vafek, B. Andrei Bernevig, Nicolas Regnault

Published 2023-09-25Version 1

Fractionally filled Chern bands with strong interactions may give rise to fractional Chern insulator (FCI) states, the zero-field analogue of the fractional quantum Hall effect. Recent experiments have demonstrated the existence of FCIs in twisted bilayer MoTe$_2$ without external magnetic fields -- most robust at $\nu=-2/3$ -- as well as Chern insulators (CIs) at $\nu=-1$. Although the appearance of both of these states is theoretically natural in an interacting topological system, experiments repeatedly observe nonmagnetic states (lacking FCIs) at $\nu=-1/3$ and $-4/3$, a puzzling result which has not been fully theoretically explained. In this work, we perform Hartree-Fock and exact diagonalization calculations to test whether the standard MoTe$_2$ moir\'e model with the (greatly varying) parameter values available in the literature can reproduce the non-magnetic states at $\nu=-1/3$ and $-4/3$ in unison with the FCI at $\nu=-2/3$ and CI state at $\nu = -1$. We focus on the experimentally relevant twist angles and, crucially, include remote bands. We find that the parameters proposed in [Wang et al. (2023)] can nearly capture the experimental phenomena at $\nu=-1/3,-2/3,-1,-4/3$ simultaneously, though the predicted ground states at $\nu=-1/3$ are still mostly fully-spin-polarized and a larger dielectric constant $\epsilon>10$ than is typical of hexagonal boron nitride (h-BN) substrate $\epsilon\sim 6$ is required. Our results show the importance of remote bands in identifying the competing magnetic orders and lay the groundwork for further study of the realistic phase diagram.