Colorizing Old B&W Photos and Videos with the Help of AI

This project is based on a research work developed at the University of California, Berkeley by Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful Image Colorization.

The idea behind this tutorial is to develop a fully automatic approach that will generate realistic colorizations of Black & White (B&W) photos and by extension, videos. As explained in the original paper, the authors, embraced the underlying uncertainty of the problem by posing it as a classification task using class-rebalancing at training time to increase the diversity of colors in the result. The Artificial Intelligent (AI) approach is implemented as a feed-forward pass in a CNN (" Convolutional Neural Network") at test time and is trained on over a million color images.

Here a photo shoot on 1906, showing one of the first tests with Santos Dumont's plane "14-bis" in Paris:

And its colorized version using the models developed using these AI technics:

The same technics could be applied to old videos. Here a 1932 B&W footage of the city of Rio de Janeiro, Brazil:

And the colorized version:

1. Lab Color Space

Usually, we are used to coding a color photo using the RGB model. The RGB color model is an additive color model in which red, green and blue light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green, and blue.

But, the model that will be used on this project is the "Lab".

The CIELAB color space (also known as CIE L*a*b* or sometimes abbreviated as simply "Lab" color space) is a color space defined by the International Commission on Illumination (CIE) in 1976. It expresses color as three numerical values, L* for the lightness and a* and b* for the green–red and blue-yellow color components.

The color space L * a * b * was created after the theory of opposing colors, where two colors cannot be green and red at the same time, or yellow and blue at the same time. CIELAB was designed to be perceptually uniform with respect to human color vision, meaning that the same amount of numerical change in these values corresponds to about the same amount of visually perceived change.

Unlike the RGB color model, Lab color is designed to approximate human vision. It aspires to perceptual uniformity, and its L component closely matches human perception of lightness. The L component is exactly what is used as input of the AI model, that was train to estimate the remained components, "a" and "b".

2. The AI (Deep Learning) Process

As commented on the introduction, the Artificial Intelligent (AI) approach is implemented as a feed-forward pass in a CNN (" Convolutional Neural Network") at test time and is trained on over a million color images. In other words, millions of color photos were decomposed using Lab model and used as an input feature ("L") and classification labels ("a" and "b"). For simplicity let's split in two: "L" and "a+b" as shown in the block diagram:

Having the trained model (that is available publically), we can use it to colorize a new B&W foto, where this photo will be the input of the model or the component "L". The output of the model will be the other components "a" and "b", that once added to the original "L", will return a full colorized photo as shown here:

In short, using a broad and diverse set of objects and scenes dataset of 1.3 Millon photos from ImageNet and applying a Deep Learning algorithm (Feed-Forward CNN), final models were generated and are available at:

Zhang et al - Colorful Image Colorization - models

3. The Working Envirolment

The first thing to do is to organize an environment where we will work. Let's create a folder and name it:

• Photo_Video_Colorization

Under this main created directory, let's create sub-folders:

• model
• input_images
• input_videos
• colorized_images
• colorized_frames
• colorized_videos

Go to Zhang et al - Colorful Image Colorization - models and download the 3 files and load them at the created subfolder "/model". The files are:

• colorization_release_v2.caffemodel
• colorization_release_v2_norebal.caffemodel
• colorization_release_v1.caffemodel

I will assume that you have Python (version 3.6) and OpenCV (4.0) installed in your machine. We will describe step by step all the process of colorization using Jupyter Notebooks. I recommend that you follow my explanation, but if you want, you can go and download the notebooks and test photos from my GitHub:

Photo_Video_Colorization

I also recommend that you review the great tutorial " Black and white image colorization with OpenCV and Deep Learning" by Dr. Adrian Rosebrok, that was the inspiration and guide for this project.

4. Black & White Photo Colorization

Let's follow the steps to colorize a BT&W photo:

Import important Libraries:

import numpy as np
import matplotlib.pyplot as plt
import cv2

Define Image to be colorized:

IMAGE = "soldiers_1941"

Note: You can use any photo here. In this case, I use a 1941 B&W photo of soldiers at WWII. The photo is available on my GitHub.

Define Model Paths:

prototxt = "./model/colorization_deploy_v2.prototxt"
model = "./model/colorization_release_v2.caffemodel"
points = "./model/pts_in_hull.npy"
image =  "./input_images/"+IMAGE

Load serialized black and white colorizer model and cluster:

net = cv2.dnn.readNetFromCaffe(prototxt, model)
pts = np.load(points)

Add the cluster centers as 1x1 convolutions to the model:

class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2, 313, 1, 1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")]

Load the input image, scale it and convert it to Lab:

Note that we will first convert the image to grayscale. This step is not really necessary, but I realize that some B&W photos, especially the old ones, could have some treatment during the years, so, better to clean them a little.

image = cv2.imread(image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
image = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)

At this point, we have our original image, but in order to show it straight on a Jupyter cell, we should use a pyplot library :

plt.imshow(image)
plt.axis('off');

Extracting "L":

Now, let's take our " image" and proceed with the colorization process, where first should be rescaled, converted to Lab in order to extract the component "L" and center it:

scaled = image.astype("float32") / 255.0
lab = cv2.cvtColor(scaled, cv2.COLOR_RGB2LAB)
resized = cv2.resize(lab, (224, 224))
L = cv2.split(resized)[0]
L -= 50

Predicting "a" and "b":

net.setInput(cv2.dnn.blobFromImage(L))
ab = net.forward()[0, :, :, :].transpose((1, 2, 0))
ab = cv2.resize(ab, (image.shape[1], image.shape[0]))

Creating a colorized Lab photo (L + a + b):

L = cv2.split(lab)[0]
colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)

Same as we did with the grayscaled image, let's check how the colorized image looks like:

plt.imshow(colorized)
plt.axis('off');
plt.title('colorized LAB image');

Ops, seems that Lab image cannot tell us much, let's convert it to RGB and see the result:

Converting to RGB:

colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2RGB)
colorized = np.clip(colorized, 0, 1)
colorized = (255 * colorized).astype("uint8")
plt.imshow(colorized)
plt.axis('off');

Uau! Pretty amazing!!!!! This is a 1941 photo that seems to really be shot in full colors! Let's save it:

Saving the final RGB photo:

cv2.imwrite("./colorized_images/Color_"+IMAGE, cv2.cvtColor(colorized, cv2.COLOR_RGB2BGR))

One more: Charles Darwin visiting Rio de Janeiro, Brazil in 1832:

Great!!!! Try with other B&W photos and see how amazing Artificial Intelligence can be!

5. Colorizing Videos

Once we have colorized photos, colorize videos is not a complicated task. We must follow the following general steps:

• Get B&W footage and load it on input_video subdirectory
• Read the video, one frame at a time
• Having a single frame, apply what we did for a photo
• Having a colorized frame, save it on another subfolder: colorized_video_frames
• Close the OpenCv windows.

Let's do a real case:

You can download the notebook B_W_Video_Colorization.ipynb from my GitHub, or follow the steps.

The first thing that I did, was to download from Youtube a B&W movie, in this case:

For that, I used a free tool: VidPaw to download it. I named it: "rio_32.mp4" and save it on input_video subfolder.

Applying the steps described above, at the end we will have all colorized frames stored on the subfolder colorized_video_frames. Let's do it:

1. Start defining the file that should be colorized:

VIDEO = "rio_32.mp4"

2. Define Paths, constants, and video variables:

prototxt = "./model/colorization_deploy_v2.prototxt"
model = "./model/colorization_release_v2.caffemodel"
points = "./model/pts_in_hull.npy"
video =  "./input_video/"+VIDEO
width = 500
vs = cv2.VideoCapture(video)

3. Load and prepare models:

net = cv2.dnn.readNetFromCaffe(prototxt,model)
pts = np.load(points)
class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2, 313, 1, 1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")] 

4. Split video, frame by frame and apply model:

count = 0
success = True
while success:
	success, frame = vs.read()
	if frame is None:
		break
	frame = imutils.resize(frame, 500)
	frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
	frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2RGB)
	scaled = frame.astype("float32") / 255.0
	lab = cv2.cvtColor(scaled, cv2.COLOR_RGB2LAB)
	resized = cv2.resize(lab, (224, 224))
	L = cv2.split(resized)[0]
	L -= 50
    
	net.setInput(cv2.dnn.blobFromImage(L))
	ab = net.forward()[0, :, :, :].transpose((1, 2, 0))
	ab = cv2.resize(ab, (frame.shape[1], frame.shape[0]))
	L = cv2.split(lab)[0]
	colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)
	colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2BGR)
	colorized = np.clip(colorized, 0, 1)
	colorized = (255 * colorized).astype("uint8")
	cv2.imshow("Original", frame)
	cv2.imshow("Colorized", colorized)
    
	cv2.imwrite("./colorized_video_frames/frame%d.jpg" % count, colorized)
	count += 1
	key = cv2.waitKey(1) & 0xFF
	if key == ord("q"):
		break
vs.release()
cv2.destroyAllWindows()

The above loop process, usually takes a while. For example, the process of colorization of this video (8 minutes) having around of 14, 000 frames took me around 3 hours on a MacBook Pro - 2.9GHz Core i7 with 16GB 2133MHz of RAM.

5. Once you have the file with the frames, you must "re-assemble" it to create a video. The bellow function can do it:

def convert_frames_to_video(pathIn, pathOut, fps):
    frame_array = []
    files = [f for f in os.listdir(pathIn) if isfile(join(pathIn, f))]
    #for sorting the file names properly
    files.sort(key = lambda x: int(x[5:-4]))
    for i in range(len(files)):
        filename=pathIn + files[i]
        #reading each files
        img = cv2.imread(filename)
        height, width, layers = img.shape
        size = (width,height)
        print(filename)
        #inserting the frames into an image array
        frame_array.append(img)
    out = cv2.VideoWriter(pathOut,cv2.VideoWriter_fourcc(*'MJPG'), fps, size)
    for i in range(len(frame_array)):
        # writing to a image array
        out.write(frame_array[i])
    out.release()

Note that depending on what Video Controler you have installed on your machine, the codec (*'MJPG'), should be changed. Please check with OpenCV documentation. At end would be a "try&error" experience.

Now, you should apply the above function on the colorized frames:

pathIn= './colorized_video_frames/'
pathOut = './colorized_videos/color_rio_32.avi'
fps = 30.0
convert_frames_to_video(pathIn, pathOut, fps)

The resulting "raw" video can be seen here:

Note that the video obviously has no sound. What I did was to strip the original sound from the B&W movie and add it to the colorized one, using iMovie. The final result is here:

That's it! Very cool, isn't? ;-)

6. Homage to Santos Dumont

Taking the opportunity to colorize old photos and videos, I decided to honor one of the great inventors of last century, Alberto Santos-Dumont.

Santos-Dumont (1873 - 1932) was a Brazilian inventor and aviation pioneer, one of the very few people to have contributed significantly to the development of both lighter-than-air and heavier-than-air aircraft.

The heir of a wealthy family of coffee producers, Santos-Dumont dedicated himself to aeronautical study and experimentation in Paris, where he spent most of his adult life. In his early career, he designed, built, and flew hot air balloons and early dirigibles, culminating in his winning the Deutsch de la Meurthe prize on 19 October 1901 for a flight that rounded the Eiffel Tower. He then turned to heavier-than-air machines and on 23 October 1906, his 14-is made the first powered heavier-than-air flight in Europe to be certified by the Aéro-Club de France and the Fédération Aéronautique Internationale. His conviction that aviation would usher in an era of worldwide peace and prosperity led him to freely publish his designs and forego patenting his various innovations.

Santos-Dumont was really an early adept of Open Science, his goal was to make the good for mankind. One example was that all plans of his Demoiselle airplane were openly published on "Popular Mechanics" magazine, on its December 1909 edition, where was stated: in order to encourage aviation, and anyone may use his plans to construct a similar machine.

All the photos that appears on this work are from the first decade of XX century and were fully colorized by computer, using the technics described in this tutorial. And below, a video collage is done. I hope you appreciate it.

7. Conclusion

As always, I hope this project can help others find their way into the exciting world of technology!

For details and final code, please visit my GitHub repository: MJRoBot-Python4DS

Saludos from the south of the world!

See you in my next tutorial!

Thank you,

Marcelo