However, the growing problem of the size of deep learning models has brought the issue of energy use, in which a single large transformer model can produce over 500 metric tons of CO? equivalent when training. We, in this work, propose the first green awareness framework, named GreenAI-Framework, that alters the precision level of a given model, making it sparse, and scheduling its computations on low-carbon energy sources using the carbon intensity signal. There are three proposed algorithms in our proposed framework, and these are as follows: (1) Adaptive Precision Scaling (APS) with the use of reinforcement learning to decrease the number of FLOPS between 40% to 60% with no accuracy cost, (2) Energy Aware Early Exiting (EAEE) to exit from low confidence inference requests, and (3) Carbon-Aware Task Scheduling (CATS) for executing non-urgent tasks in low-carbon energy slots. Experimental analysis demonstrates that our framework helps reduce energy use by 47.3%, having only 0.9% loss in accuracy for ResNet-50, BERT, and GPT-2 on GPU clusters.