Real-time detection of internal (occupancy, equipment, lighting) and solar (window) gains is valuable for improved HVAC and lighting operation in order to meet the actual dynamic energy demand of different zones and optimize building operation. Commonly assumed fixed HVAC and lighting schedules cannot consider dynamic changes in indoor conditions and can lead to reduced comfort and increased energy use. This paper presents a method for real-time monitoring of dynamic internal and solar gains using programmable low-cost cameras and deep learning techniques. A convolutional neural network (CNN) multi-head classification model was developed, trained, and deployed to low-cost fisheye cameras, which capture High Dynamic Range (HDR) images of the space, for real-time detection of changes in occupancy, equipment, lighting, and window status. The classification heads of the model were trained using pre-defined areas of interest from collected fisheye HDR images in a private office first; then they were fine-tuned with another collected HDR image set in an open-plan office to input target object information to the model. The pre-trained weights of the original Resnet18 architecture were partially applied to transfer necessary feature extraction attributes, while a self-attention mechanism was implemented to increase the receptive field area of feature pixels and account for the distorted objects in fisheye images. The detection model performance was evaluated via mean classification precision and recall, and the results show that the developed model could classify the status of objects in the predefined areas of the scene with great accuracy. The detailed heat gain information can be easily communicated to the Building Management System for accurate estimation of real-time energy demand and efficient HVAC and lighting/window control, also useful in demand-response applications.