Prediction models in healthcare are being utilized for many tasks. However, the use of these models for medical decision-making warrants special considerations that are less critical when prediction models are used in other domains. Two of these considerations, which we will discuss in the talk, are fairness and explainability. We will discuss these considerations from the viewpoint of a large healthcare organization that uses prediction models ubiquity on a daily basis. We will also describe how academic collaborations can expand our toolbox for handling these issues in practice.