Quantitative Assessment of Energy Consumption and Carbon Emissions in Transformer Based NLP Models

Abstract

The expansion of transformer-based neural networks has led to a rise in their environmental footprint and computational expenses. This research investigates the energy consumption and carbon emissions associated with fine-tuning transformer models, focusing specifically on the DistilBERT-base-uncased model trained on the GLUE SST-2 dataset. Using the CodeCarbon tracker, we recorded approximately 0.00733 kg of CO2 equivalent emissions from a single fine-tuning session. Although these results are modest in scale, they align with broader evidence that training large-scale models significantly contributes to carbon emissions, particularly when powered by non-renewable energy sources. Beyond environmental metrics, this study also addresses practical challenges in model implementation, such as the uninitialized weights of classifiers in DistilBERT and the lack of padding tokens in GPT-2, highlighting common issues in transfer learning and tokenizer configurations. Additionally, we discuss how hardware limitations, specifically CPU-only training, impact efficiency and emission estimates. This work contributes to the conversation on green AI tools by presenting measured environmental data and identifying reproducibility challenges, which aid in promoting transparency and sustainability in transformer model training.