Created November 28, 2023

Machine Learning Video Player

A lightweight video player that allows you to see the result of your ML algorithm run on a video. Includes frame stepping and frame saving.

Things used in this project

Hardware components

Camera (generic)

Story

Inspiration

Many machine learning projects include a video analysis component. Whether you are attempting to identify and classify objects or track the movements of a person, you need to view the output of your model when used on different videos or different parts of videos. And even if you are just using an existing model, you will probably want to see how effective it is for your data, and whether or not you need to change some parameters. I was working on one such project and found myself in need of a good way to view the results when my code was used on a video. OpenCV has an existing VideoPlayer, but it didn't have all of the functionality that I wanted. So naturally, I decided to make my own.

Functionality

This project uses the OpenCV VideoPlayer and its properties to create an interactive video player that allows more control when playing a video. The controls are triggered by key presses for ease of use.

H : Toggle Controls - This simply toggles whether the other controls are displayed or not
Q : Quit - Closes the window
Space : Pause/Play - also starts the video over if at the end
F : Fast Forward
D : Step Forward - Pauses the video and displays the next frame
A : Step Backward - Pauses the video and displays the previous frame
W : Jump Forward - Pauses the video and jumps forward 1 second
S : Jump Backward - Pauses the video and jumps backward 1 second
Z : Jump to Beginning
C : Jump to End
P : Save Frame - Saves the currently displayed frame as a jpg with the same name as the video with the time in milliseconds appended

Usage

Save the videoplayer.py file in the same folder and import it.

import videoplayer

Call the run_player function, the only required argument is the videopath

videoplayer.run_player(videopath)

There are several optional parameters:

display_size_modifier - a value between 0 and 1 that alters the height and width of the video before it is displayed
text_color - a BGR tuple that changes the color of the state and control text
passed_function - a function that uses each frame for computation/alteration and then returns the altered frame
any other arguments necessary for the passed function

videoplayer.run_player(videopath, display_size_modifier = 0.5, text_color = (0, 0, 0), passed_function = my_func, my_func_arg1, my_func_arg2, ...)

passed_function

In order for the video player to display any changes made to each frame, the run_player function need to be passed a frame-altering function. For example, if you wanted to find and draw a circle of a particular color in each frame, you might have a function such as the following:

def find_circle(frame):
    blurred = cv2.GaussianBlur(frame, (3,3), 0)
    hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV)
    color_bound_lower = (7, 100, 20)
    color_bound_upper = (25, 200, 200)
    mask = cv2.inRange(hsv, color_bound_lower, color_bound_upper)

    contours = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    c = max(contours, key=cv2.contourArea)
    ((x, y), radius = minEnclosingCircle(c)
    cv2.circle(frame, (int(x), int(y)), int(radius), (0, 255, 255), 2)

    return frame

To use this function with the player, simply pass it to the run_player function.

videoplayer.run(videopath, passed_function=find_circle)

The only requirement for the passed function is that it must take a frame as the first argument and return the frame after it has been altered (returning an unaltered frame is fine too). The passed function is allowed to take other arguments and those arguments must be passed to the run_player function in order, after all of the optional arguments. For example, the find_circle function could take arguments for the lower and upper color bounds.

def find_circle(frame, lower_bound, upper_bound)

This is perfectly fine, as long as values for these arguments are passed to the run_player function as well, after any other arguments.

videoplayer.run(videopath, display_size_modifier = 0.7, passed_function = find_circle, lower_bound=(10, 10, 10), upper_bound=(20, 20, 20))

The run_player function was designed this way in order to be more versatile and so that it could easily be used in many different projects.

Code

videoplayer.py

import cv2
import mediapipe as mp
import os
import time as tm


# handles displaying the image with the state and state of help display,
# checks if at the end of the video and returns that status
def displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame):

	# ending check
	ended = False
	if video_time >= video_length - msec_per_frame:
		ended = True
		cv2.putText(imS, "End", (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, text_color, 2)
	else:
		cv2.putText(imS, state, (25, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, text_color, 2)

	if show_controls:
		cv2.putText(imS, "H: Toggle Controls", (25, 75), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "Q: Quit", (25, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "Space: Play/Pause", (25, 125), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "F: Fast Forward", (25, 150), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "D: Step Forward", (25, 175), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "A: Step Backward", (25, 200), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "W: Jump Forward", (25, 225), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "S: Jump Backward", (25, 250), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "Z: Jump to Beginning", (25, 275), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "C: Jump to End", (25, 300), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		cv2.putText(imS, "P: Save Frame", (25, 325), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
	else:
		cv2.putText(imS, "H: Toggle Controls", (25, 75), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color, 2)
		pass

	cv2.imshow("player", imS)

	return ended


# main function of module, takes a videopath a display modifier between 0 and 1, an rgb tuple to change the color
# of the controls and the state, a function that is applied to every frame before display, and a dictionary of other
# arguments that are passed to the passed_function
def run_player(videopath, display_size_modifier = 1.0, text_color = (0, 0, 0), passed_function = lambda frame: frame, **kwargs):

	# check video path is a readable .mp4 or .mov file
	if not os.path.exists(videopath):
		print("videopath does not exist")
		return

	if not os.path.isfile(videopath):
		print("videopath is not a file")
		return

	if not os.access(videopath, os.R_OK):
		print("videopath cannot be read")
		return

	file_extension = os.path.splitext(videopath)[1]
	if file_extension.lower() != ".mp4" and file_extension.lower() != ".mov":
		print("videopath is not a video file")
		return

	# check text color
	if type(text_color) not in [list, tuple] \
		or len(text_color) != 3 \
		or text_color[0] < 0 or text_color[0] > 255 \
		or text_color[1] < 0 or text_color[1] > 255 \
		or text_color[2] < 0 or text_color[2] > 255:
		print("text color is invalid")
		return

	print("Opening VideoCapture")
	cap = cv2.VideoCapture(videopath)

	# property values
	frame_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
	frame_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
	frames_per_second = cap.get(cv2.CAP_PROP_FPS)
	time_between_frames = frames_per_second/1000
	total_frames = cap.get(cv2.CAP_PROP_FRAME_COUNT)
	current_time_position = cap.get(cv2.CAP_PROP_POS_MSEC)

	# make sure the ratio is between 0 and 1
	if display_size_modifier < 0.0 or display_size_modifier > 1.0:
		print("display_size_modifier must be between 0.0 and 1.0")
		return

	# apply size modifier
	display_width = int(frame_width*display_size_modifier)
	display_height = int(frame_height*display_size_modifier)

	# get milliseconds per frame and total length of video
	msec_per_frame = int(1000 / frames_per_second)
	video_length = total_frames*msec_per_frame

	cv2.namedWindow('player', cv2.WINDOW_AUTOSIZE)

	# booleans related to the state of the player
	playing = True
	fast_forward = False
	ended = False
	show_controls = True

	state = ""

	# a copy of the frame, needed for saving purposes
	display = None

	## actual run
	while cap.isOpened():
		if playing and not ended:
			if fast_forward:
				video_time += 10*msec_per_frame
				# end check
				if video_time >= video_length - msec_per_frame:
					video_time = video_length - msec_per_frame
					fast_forward = False
				cap.set(cv2.CAP_PROP_POS_MSEC, video_time)

			start_time = tm.time()
	
	
			ret, frame = cap.read()
			video_time = cap.get(cv2.CAP_PROP_POS_MSEC)
			#print("current_time_position:", video_time)
			
			if ret == True:
				imS = cv2.resize(frame, (display_width, display_height))

				if fast_forward:
					state = "Fast Forward"
				else:
					state = ""

				#apply passed function
				imS = passed_function(imS, **kwargs)
				display = imS.copy()
				ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)
			else:
				break

			# we want the time each frame takes to be the fps so in case the execution of the passed function
			# takes less than the desired time between frames smay have to sleep
			time_passed = tm.time() - start_time
			if time_passed < time_between_frames:
				print("sleeping for %f" % (time_between_frames - time_passed))
				tm.sleep(time_between_frames - time_passed)


		# look for a key press
		key = cv2.waitKey(1)

		# q key quits the player
		if key == ord('q'):
			break

		# space key toggles pausing
		elif key == ord(' '):
			playing = not playing
			fast_forward = False
			# if ended and press space, want to play from beginning
			if ended:
				ended = False
				playing = True
				fast_forward = False
				video_time = 0
				cap.set(cv2.CAP_PROP_POS_MSEC, video_time)
			# if not ended display current frame with "paused"
			else:
				imS = display.copy()
				state = "Paused"
				ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)


		# d key is how you step, works if paused or playing
		elif key == ord('d'):
			
			#frame, video_time = get_altered_frame(cap)
			# end check, if at end don't do anything
			if not ended:
				playing = False
				ret, frame = cap.read()
				video_time = cap.get(cv2.CAP_PROP_POS_MSEC)
				if ret == True:
					imS = cv2.resize(frame, (display_width, display_height))
					state = "Stepping Forward"
					# apply passed function
					imS = passed_function(imS, **kwargs)
					display = imS.copy()
					ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)

				else:
					break

		# a key steps backwards
		elif key == ord('a'):
			
			# beginning check, if at beginning, don't do anything
			if video_time >= msec_per_frame:
				# ending check
				if ended:
					ended = False
				playing = False
				video_time -= msec_per_frame
				# edge adjustment
				if video_time < 0:
					video_time = 0
				cap.set(cv2.CAP_PROP_POS_MSEC, video_time)

				ret, frame = cap.read()
				video_time = cap.get(cv2.CAP_PROP_POS_MSEC)
				#print("current_time_position:", video_time)
				if ret == True:
					imS = cv2.resize(frame, (display_width, display_height))
					state = "Stepping Backward"
					# apply passed function
					imS = passed_function(imS, **kwargs)
					display = imS.copy()
					ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)

				else:
					break
		
		# f key fast forwards
		elif key == ord('f'):
			playing = True
			fast_forward = True

		# w skips ahead a second
		elif key == ord('w'):
			
			# end check, if at end, do nothing
			if not ended:
				playing = False
				video_time += 1000
				# edge adjustment
				if video_time > video_length:
					video_time = video_length - msec_per_frame
				cap.set(cv2.CAP_PROP_POS_MSEC, video_time)

				ret, frame = cap.read()
				video_time = cap.get(cv2.CAP_PROP_POS_MSEC)
				if ret == True:
					imS = cv2.resize(frame, (display_width, display_height))
					state = "Paused"
					# apply passed function
					imS = passed_function(imS, **kwargs)
					display = imS.copy()
					ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)

				else:
					break

		# s steps back a second
		elif key == ord('s'):
			
			# beginning check, if at the beginnning, don't do anything
			if video_time >= msec_per_frame:
				playing = False
				video_time -= 1000
				# ending check
				if ended:
					ended = False
				# edge check
				if video_time < 0:
					video_time = 0
				cap.set(cv2.CAP_PROP_POS_MSEC, video_time)

				ret, frame = cap.read()
				video_time = cap.get(cv2.CAP_PROP_POS_MSEC)
				if ret == True:
					imS = cv2.resize(frame, (display_width, display_height))
					state = "Paused"
					# apply passed function
					imS = passed_function(imS, **kwargs)
					display = imS.copy()
					ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)

				else:
					break

		# z goes back to the beginning
		elif key == ord('z'):
			# beginning check, if at beginning don't do anything
			if video_time >= msec_per_frame:
				# ending check
				if ended:
					ended = False
				playing = False
				cap.set(cv2.CAP_PROP_POS_MSEC, 0)

				ret, frame = cap.read()
				video_time = cap.get(cv2.CAP_PROP_POS_MSEC)
				imS = cv2.resize(frame, (display_width, display_height))
				state = "To Beginning"
				# apply passed function
				imS = passed_function(imS, **kwargs)
				display = imS.copy()
				ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)


		# c goes to the end
		elif key == ord('c'):
			# ending check, if ended, don't do anything
			if not ended:
				playing = False
				cap.set(cv2.CAP_PROP_POS_MSEC, video_length - msec_per_frame)
				ret, frame = cap.read()
				imS = cv2.resize(frame, (display_width, display_height))
				state = "To Ending"
				# apply passed function
				imS = passed_function(imS, **kwargs)
				display = imS.copy()
				ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)



		# h toggles the control display
		elif key == ord('h'):
			show_controls = not show_controls
			imS = display.copy()
			ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)

		# p saves an image of display
		elif key == ord('p'):
			saved_video_path = videopath + "_" + str(video_time - msec_per_frame) + "msec.jpg"
			imS = display.copy()
			state = "Saved Frame"
			ended = displayFrame(imS, state, show_controls, text_color, video_time, video_length, msec_per_frame)
			cv2.imwrite(saved_video_path, display)
			print("saved image at " + saved_video_path)



	cap.release()
	cv2.destroyAllWindows()

Credits

Brian Koga

1 project • 0 followers

Machine Learning Video Player

Things used in this project

Hardware components

Story

Schematics

State Diagram

Code

videoplayer.py

videoplayer

Credits

Brian Koga

Comments

Embed the widget on your own site

Machine Learning Video Player

Machine Learning Video Player

Things used in this project

Hardware components

Story

Schematics

State Diagram

Code

videoplayer.py

videoplayer

Credits

Brian Koga

Comments