Prediction of the glass-forming ability (GFA) of alloys remains a major challenge. We are not yet able to predict the composition dependence of the GFA of even binary alloys. To investigate the effect of each element's propensity to form particular crystal structures on glass formation, we focus on binary alloys composed of elements with the same size but different atomic symmetries using the patchy-particle model. For mixtures with atomic symmetries that promote different crystal structures, the minimum critical cooling rate R-c is only a factor of 5 lower than that for the pure substances. For mixtures with different atomic symmetries that promote local crystalline and icosahedral order, the minimum R-c is more than three orders of magnitude lower than that for pure substances. Results for R-c for the patchy-particle model are consistent with those from embedded atom method simulations and sputtering experiments of NiCu, TiAl, and high entropy alloys.