Keras with Tensorflow - How to flush CUDA Memory

As a young researcher working in the field of machine learning, I am on a regular basis utilizing the GPUs to train different kinds of neural networks such as convolutional neural networks to tackle various problems. In the majority of cases my tool-of-choice Keras library with Tensorflow backend, which has lovely support for multi-GPU training. A lot of my work consists of experimenting with various un-conventional approaches to train deep neural networks, and with a huge amount of experiments comes along quite a lot of failures.