2024 workshop program
SpectraFM: Tuning into Stellar Foundation Models
The work
Abstract
Neural networks are increasingly crucial for predicting stellar properties in large spectroscopic surveys but struggle when applied to scenarios outside their training distribution, such as with new instruments, different wavelength ranges, or limited data. To address this, we developed SpectraFM, a ‘Foundation Model’ based on a Transformer neural network, a machine learning technique that can transfer knowledge from data-rich tasks to new data-scarce situations through pre-training. Our model is pre-trained on half the wavelength range of the APOGEE spectroscopic survey with ~100,000 stars to predict elemental abundances (Fe, Mg, O) and stellar properties. We then fine-tune the model using only 100 stars to predict abundances that were not in pre-training (Si, Ti, Ni) and surpass the performance of a traditional neural network with the same data and size. Additionally, we fine-tune the model to predict [Fe/H] from a wavelength range unseen during pre-training with only 100 stars, demonstrating its capability to generalize its knowledge to any wavelength range. We train the model to be uniquely flexible, so it has the ability to predict any output from any combination of inputs, making it a powerful tool for stellar astrophysics research. We plan to further train our model on a wider range of surveys, wavelengths, and stellar properties, and to include simulated spectra. This will improve its ability to generalize and its performance when dealing with new instruments and limited data.