At this early stage in the COVID-19 epidemic, researchers are looking for all possible insights into the new corona virus SARS-CoV-2. One of the possibilities is an in-depth analysis of X-ray images from COVID-19 patients. We first developed a new adapted classification method that is able to identify COVID-19 patients based on a chest X-ray, and then adopted a local interpretable model-agnostic explanations approach to provide the insights. The classification method uses a grey wolf optimizer algorithm for the purpose of optimizing hyper-parameter values within the transfer learning tuning of a CNN. The trained model is then used to classify a set of X-ray images, upon which the qualitative explanations are performed. The presented approach was tested on a dataset of 842 X-ray images, with the overall accuracy of 94.76%, outperforming both conventional CNN method as well as the compared baseline transfer learning method. The achieved high classification accuracy enabled us to perform a qualitative in-depth analysis, which revealed that there are some regions of greater importance when identifying COVID-19 cases, like aortic arch or carina and right main bronchus. The proposed classification method proved to be very competitive, enabling one to perform an in-depth analysis, necessary to gain qualitative insights into the characteristics of COVID-19 disease.