Segregation hinders inclusive cities and persists in everyday activities outside homes, yet the contributions of pivotal factors remain insufficiently quantified. Here, we elucidate how different factors influence outside-home segregation, looking at the distinctive segregation experienced by native and foreign-born segregated individuals, using data from ~320,000 smartphone devices collected in Sweden. We find that while day-to-day activities lead to mixing for native-born majorities, foreign-born minorities remain segregated in their out-of-home activities. Our results highlight two factors for foreign-born segregated individuals, who (i) tend to visit same-group destinations (homophily), and (ii) have limited mobility ranges (limited travel). Counterfactual simulations further reveal that homophily alone, as represented by destination preference, plays a minor role, while combined with limited mobility, restricts foreign-born minorities' exposure to diverse groups, which is linked to limited public transport access. Enhancing transport accessibility for foreign-born minorities could potentially reduce social segregation.