We estimate the conversion factor relating CO emission to H2 mass, alpha_CO, in five Local Group galaxies that span approximately an order of magnitude in metallicity - M31, M 33, the Large Magellanic Cloud (LMC), NGC 6822, and the Small Magellanic Cloud (SMC). We model the dust mass along the line of sight from infrared (IR) emission and then solve for the alpha_CO that best allows a single gas-to-dust ratio (delta_GDR) to describe each system. This approach remains sensitive to CO-dark envelopes of H2 surrounding molecular clouds. In M 31, M 33, and the LMC we find alpha_CO \approx 3-9 M_sun pc^-2 (K km s^-1)^-1, consistent with the Milky Way value within the uncertainties. The two lowest metallicity galaxies in our sample, NGC 6822 and the SMC (12 + log(O/H) \approx 8.2 and 8.0), exhibit a much higher alpha_CO. Our best estimates are \alpha_NGC6822 \approx 30 M_sun/pc^-2 (K km s^-1)^-1 and \alpha_SMC \approx 70 M_sun/pc^-2 (K km s-1)-1. These results are consistent with the conversion factor becoming CO a strong function of metallicity around 12 + log(O/H) \sim 8.4 - 8.2. We favor an interpretation where decreased dust-shielding leads to the dominance of CO-free envelopes around molecular clouds below this metallicity.Read less <