正所谓做工作要做好记录&＃xff0c;现在&＃xff0c;我要开始记录啦。
1、每个视频都是一个图像序列&＃xff0c;其内容比一张图像丰富很多,表现力强&＃xff0c;信息量大。对视频的分析通常是基于视频帧&＃xff0c;但视频帧通常存在大量冗余&＃xff0c;对视频帧的提取也存在漏帧、冗余的现象。视频关键帧提取则主要体现视频中各个镜头的显著特征&＃xff0c;通过视频关键帧提取能够有效减少视频检索所需要花费的时间&＃xff0c;并能够增强视频检索的精确度。
2、关键帧定义&＃xff1a;把图像坐标系中每个“视频帧”都叠加在一起&＃xff0c;这时镜头中视频帧的特征矢量会在空间中呈现出一个轨迹的状态&＃xff0c;而与轨迹中特征值进行对应的“帧”即可称之为关键帧[1]
3、视频具有层次化结构&＃xff0c;由场景、镜头和帧三个逻辑单元组成。视频检索常基于帧进行&＃xff0c;因此&＃xff0c;提取视频的关键帧至关重要[2]
1、关键帧提取思想&＃xff1a;对视频序列采用镜头分割的方式&＃xff0c;然后在镜头当中获得内容关键帧提取&＃xff0c;接着利用“关键帧”来获得底层的形状、纹理和颜色等特征。
2、关键帧提取方法&＃xff1a;
&＃xff08;1&＃xff09;全图像序列
镜头边界方法是将镜头中的第一帧和最后一帧&＃xff08;或中间帧&＃xff09;作为关键帧。该方法简单易行&＃xff0c;适于内容活动性小或内容保持不变的镜头&＃xff0c;但未考虑镜头视觉内容的复杂性&＃xff0c;限制了镜头关键帧的个数&＃xff0c;提取的关键帧代表性不强&＃xff0c;效果不够稳定。
&＃xff08;2&＃xff09;压缩视频
(3) 自定义k值聚类和内容分析的关键帧提取方法[2]
&＃xff08;4&＃xff09;a 基于抽样的关键帧提取[3]
基于抽样的方法是通过随机抽取或在规定的时间间隔内随机抽取视频帧。这种方法简单不实用。
b 基于颜色特征的关键帧提取
c 基于运动分析的关键帧提取
d 基于镜头边界的关键帧提取(*)
e 基于视频内容的关键帧提取(*)
f 基于聚类的关键帧提取 (可能是最合适的一个了)&＃xff0c;我要识别的类别已经确定。
&＃xff08;5&＃xff09;使用3D-CNN提取关键帧[4]
他提出了一种基于语义的视频关键帧提取算法&＃xff0c;该算法首先使用层次聚类算法对视频关键帧进行初步提取&＃xff1b;然后结合语义相关算法对初步提取的关键帧进行直方图对比去掉冗余帧&＃xff0c;确定视频的关键帧&＃xff1b;最后通过与其他算法比较&＃xff0c;本文提出的算法提取的关键帧冗余度相对小。
&＃xff08;6&＃xff09;首先利用卷积自编码器提取视频帧的深度特征&＃xff0c;对其进行 K-means 聚类&＃xff0c;在每类视频帧中采用清晰度筛选取出最清晰的视频帧作为初次提取的关键帧&＃xff1b;然后利用点密度方法对初次提取的关键帧进行二次优化&＃xff0c;得到最终提取的关键帧进行手语识别.[5]
(7) 视频关键帧提取方法一般可分为四大类&＃xff1a;
第一类&＃xff1a;基于图像内容的方法
第二类&＃xff1a;基于运动分析的方法
第三类&＃xff1a;基于轨迹曲线点密度特征的关键帧检测算法
第四类&＃xff1a;目前主流方法&＃xff1a;基于聚类的方法
&＃xff08;8&＃xff09;帧间差法
源自: 以下代码出自zyb_as的github
# -*- coding: utf-8 -*-
"""
帧间最大值法
Created on Tue Dec 4 16:48:57 2018
keyframes extract tool
this key frame extract algorithm is based on interframe difference.
The principle is very simple
First, we load the video and compute the interframe difference between each frames
Then, we can choose one of these three methods to extract keyframes, which are
all based on the difference method:1. use the difference orderThe first few frames with the largest average interframe difference are considered to be key frames.
2. use the difference thresholdThe frames which the average interframe difference are large than the threshold are considered to be key frames.
3. use local maximumThe frames which the average interframe difference are local maximum are considered to be key frames.It should be noted that smoothing the average difference value before calculating the local maximum can effectively remove noise to avoid repeated extraction of frames of similar scenes.
After a few experiment, the third method has a better key frame extraction effect.
The original code comes from the link below, I optimized the code to reduce
unnecessary memory consumption.
https://blog.csdn.net/qq_21997625/article/details/81285096
&＃64;author: zyb_as
"""
import cv2
import operator
import numpy as np
import matplotlib.pyplot as plt
import sys
from scipy.signal import argrelextremadef smooth(x, window_len&＃61;13, window&＃61;&＃39;hanning&＃39;):"""smooth the data using a window with requested size.This method is based on the convolution of a scaled window with the signal.The signal is prepared by introducing reflected copies of the signal (with the window size) in both ends so that transient parts are minimizedin the begining and end part of the output signal.input:x: the input signal window_len: the dimension of the smoothing windowwindow: the type of window from &＃39;flat&＃39;, &＃39;hanning&＃39;, &＃39;hamming&＃39;, &＃39;bartlett&＃39;, &＃39;blackman&＃39;flat window will produce a moving average smoothing.output:the smoothed signalexample:import numpy as np t &＃61; np.linspace(-2,2,0.1)x &＃61; np.sin(t)&＃43;np.random.randn(len(t))*0.1y &＃61; smooth(x)see also: numpy.hanning, numpy.hamming, numpy.bartlett, numpy.blackman, numpy.convolvescipy.signal.lfilterTODO: the window parameter could be the window itself if an array instead of a string """print(len(x), window_len)# if x.ndim !&＃61; 1:# raise ValueError, "smooth only accepts 1 dimension arrays."## if x.size
运动分析流光法进行关键帧提取
源自&＃xff1a;AillenAnthony的github
# Scripts to try and detect key frames that represent scene transitions
# in a video. Has only been tried out on video of slides, so is likely not
# robust for other types of video.# 1. 基于图像信息
# 2. 基于运动分析(光流分析)import cv2
import argparse
import json
import os
import numpy as np
import errnodef getInfo(sourcePath):cap &＃61; cv2.VideoCapture(sourcePath)info &＃61; {"framecount": cap.get(cv2.CAP_PROP_FRAME_COUNT),"fps": cap.get(cv2.CAP_PROP_FPS),"width": int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),"heigth": int(cap.get(cv2.CAP_PROP_FRAME_Heigth)),"codec": int(cap.get(cv2.CAP_PROP_FOURCC))}cap.release()return infodef scale(img, xScale, yScale):res &＃61; cv2.resize(img, None,fx&＃61;xScale, fy&＃61;yScale, interpolation &＃61; cv2.INTER_AREA)return resdef resize(img, width, heigth):res &＃61; cv2.resize(img, (width, heigth), interpolation &＃61; cv2.INTER_AREA)return res#
# Extract [numCols] domninant colors from an image
# Uses KMeans on the pixels and then returns the centriods
# of the colors
#
def extract_cols(image, numCols):# convert to np.float32 matrix that can be clusteredZ &＃61; image.reshape((-1,3))Z &＃61; np.float32(Z)# Set parameters for the clusteringmax_iter &＃61; 20epsilon &＃61; 1.0K &＃61; numColscriteria &＃61; (cv2.TERM_CRITERIA_EPS &＃43; cv2.TERM_CRITERIA_MAX_ITER, max_iter, epsilon)labels &＃61; np.array([])# clustercompactness, labels, centers &＃61; cv2.kmeans(Z, K, labels, criteria, 10, cv2.KMEANS_RANDOM_CENTERS)clusterCounts &＃61; []for idx in range(K):count &＃61; len(Z[labels &＃61;&＃61; idx])clusterCounts.append(count)#Reverse the cols stored in centers because cols are stored in BGR#in opencv.rgbCenters &＃61; []for center in centers:bgr &＃61; center.tolist()bgr.reverse()rgbCenters.append(bgr)cols &＃61; []for i in range(K):iCol &＃61; {"count": clusterCounts[i],"col": rgbCenters[i]}cols.append(iCol)return cols#
# Calculates change data one one frame to the next one.
#
def calculateFrameStats(sourcePath, verbose&＃61;False, after_frame&＃61;0): # 提取相邻帧的差别cap &＃61; cv2.VideoCapture(sourcePath)#提取视频data &＃61; {"frame_info": []}lastFrame &＃61; Nonewhile(cap.isOpened()):ret, frame &＃61; cap.read()if frame &＃61;&＃61; None:breakframe_number &＃61; cap.get(cv2.CAP_PROP_POS_FRAMES) - 1# Convert to grayscale, scale down and blur to make# calculate image differences more robust to noisegray &＃61; cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) # 提取灰度信息gray &＃61; scale(gray, 0.25, 0.25) # 缩放为原来的四分之一gray &＃61; cv2.GaussianBlur(gray, (9,9), 0.0) # 做高斯模糊if frame_number
# Take an image and write it out at various sizes.
#
# TODO: Create output directories if they do not exist.
#
def writeImagePyramid(destPath, name, seqNumber, image):fullPath &＃61; os.path.join(destPath, "full", name &＃43; "-" &＃43; str(seqNumber) &＃43; ".png")halfPath &＃61; os.path.join(destPath, "half", name &＃43; "-" &＃43; str(seqNumber) &＃43; ".png")quarterPath &＃61; os.path.join(destPath, "quarter", name &＃43; "-" &＃43; str(seqNumber) &＃43; ".png")eigthPath &＃61; os.path.join(destPath, "eigth", name &＃43; "-" &＃43; str(seqNumber) &＃43; ".png")sixteenthPath &＃61; os.path.join(destPath, "sixteenth", name &＃43; "-" &＃43; str(seqNumber) &＃43; ".png")hImage &＃61; scale(image, 0.5, 0.5)qImage &＃61; scale(image, 0.25, 0.25)eImage &＃61; scale(image, 0.125, 0.125)sImage &＃61; scale(image, 0.0625, 0.0625)cv2.imwrite(fullPath, image)cv2.imwrite(halfPath, hImage)cv2.imwrite(quarterPath, qImage)cv2.imwrite(eigthPath, eImage)cv2.imwrite(sixteenthPath, sImage)#
# Selects a set of frames as key frames (frames that represent a significant difference in
# the video i.e. potential scene chnges). Key frames are selected as those frames where the
# number of pixels that changed from the previous frame are more than 1.85 standard deviations
# times from the mean number of changed pixels across all interframe changes.
#
def detectScenes(sourcePath, destPath, data, name, verbose&＃61;False):destDir &＃61; os.path.join(destPath, "images")# TODO make sd multiplier externally configurable#diff_threshold &＃61; (data["stats"]["sd"] * 1.85) &＃43; data["stats"]["mean"]diff_threshold &＃61; (data["stats"]["sd"] * 2.05) &＃43; (data["stats"]["mean"])cap &＃61; cv2.VideoCapture(sourcePath)for index, fi in enumerate(data["frame_info"]):if fi["diff_count"]
(9) 使用ffmpeg进行关键帧提取
代码见于此处
(10) 使用K-means聚类法
源代码见于此处
filenames&＃61;dir(&＃39;images/*.jpg&＃39;);
%file_name &＃61; fly-1;
num&＃61;size(filenames,1); %输出filenames中文件图片的个数
key&＃61;zeros(1,num); %一行&＃xff0c;num列都是0 [0,0,0,0,0,0,0,0,0,0,0,0,....0,0,0,0]
cluster&＃61;zeros(1,num); %[0,0,0,0,0,0,0,0,0,0,0,0,....0,0,0,0]
clusterCount&＃61;zeros(1,num); %各聚类有的帧数 [0,0,0,0,0,0,0,0,0,0,0,0,....0,0,0,0]
count&＃61;0; %聚类的个数 %threshold&＃61;0.75; %阈值越大帧越多
%airplane这个视频阈值设为0.93比较合适 0.95更好
%********************************************************阈值**************************************************************%
threshold&＃61;0.91; %阈值
centrodR&＃61;zeros(num,256); %聚类质心R的直方图 第一帧图片256个初始化全部为0&＃xff0c;第二帧也是&＃xff0c;其余帧都是 %%%后面相似度大加入一帧后会对其进行调整
centrodG&＃61;zeros(num,256); %聚类质心G的直方图
centrodB&＃61;zeros(num,256); %聚类质心B的直方图if num&＃61;&＃61;0error(&＃39;Sorry, there is no pictures in images folder!&＃39;);
else%令首帧形成第一个聚类img&＃61;imread(strcat(&＃39;images/&＃39;,filenames(1).name));count&＃61;count&＃43;1; %产生第一个聚类[preCountR,x]&＃61;imhist(img(:,:,1)); %red histogram 得到红色的直方图一共256个数值&＃xff0c;每个数值有多少作为直方图的高度[preCountG,x]&＃61;imhist(img(:,:,2)); %green histogram[preCountB,x]&＃61;imhist(img(:,:,3)); %blue histogramcluster(1)&＃61;1; %设定第一个聚类选取的关键帧初始为首帧 cluster变为了&＃xff08;1,0,0,0,0,......,0,0,0&＃xff09;cluster(1)是改变了第一个元素clusterCount(1)&＃61;clusterCount(1)&＃43;1;%clusterCount(1)为0&＃xff0c;加1&＃xff0c;变为1,最终 clusterCount(1)为[1,0,0,0,.....,0,0,0]centrodR(1,:)&＃61;preCountR; % centrodR本来是num&＃xff08;帧个数&＃xff09;行&＃xff0c;256列&＃xff0c;全部为0.。现在第一行为第一帧的红色直方图各个数值的高度centrodG(1,:)&＃61;preCountG;centrodB(1,:)&＃61;preCountB;visit &＃61; 1;for k&＃61;2:numimg&＃61;imread(strcat(&＃39;images/&＃39;,filenames(k).name)); %循环读取每一帧&＃xff0c;首先是第2帧[tmpCountR,x]&＃61;imhist(img(:,:,1)); %red histogram 得到红色分量直方图 第二幅图片的红色直方图[tmpCountG,x]&＃61;imhist(img(:,:,2)); %green histogram[tmpCountB,x]&＃61;imhist(img(:,:,3)); %blue histogramclusterGroupId&＃61;1; %新定义的一个变量clusterGroupId为1maxSimilar&＃61;0; %新定义&＃xff0c;相似度for clusterCountI&＃61; visit:count %目前 count为1 定义新变量clusterCountI I来确定这一帧归属于第一个聚类还是第二个聚类sR&＃61;0;sG&＃61;0;sB&＃61;0;%运用颜色直方图法的差别函数for j&＃61;1:256sR&＃61;min(centrodR(clusterCountI,j),tmpCountR(j))&＃43;sR;%&＃xff0c;j从1到256&＃xff0c;第一帧中R的所有值256个亮度 以及第二帧的红色直方图所有高度值 进行比较选最小的sG&＃61;min(centrodG(clusterCountI,j),tmpCountG(j))&＃43;sG;sB&＃61;min(centrodB(clusterCountI,j),tmpCountB(j))&＃43;sB;enddR&＃61;sR/sum(tmpCountR);dG&＃61;sG/sum(tmpCountG);dB&＃61;sB/sum(tmpCountB);%YUV,persons are sensitive to Yd&＃61;0.30*dR&＃43;0.59*dG&＃43;0.11*dB; %运用颜色直方图法的差别函数 定义了d 差别函数if d>maxSimilarclusterGroupId&＃61;clusterCountI;maxSimilar&＃61;d;endendif maxSimilar>threshold%相似度大&＃xff0c;与该聚类质心距离小%加入该聚类&＃xff0c;并调整质心for ii&＃61;1:256 centrodR(clusterGroupId,ii)&＃61;centrodR(clusterGroupId,ii)*clusterCount(clusterGroupId)/(clusterCount(clusterGroupId)&＃43;1)&＃43;tmpCountR(ii)*1.0/(clusterCount(clusterGroupId)&＃43;1);centrodG(clusterGroupId,ii)&＃61;centrodG(clusterGroupId,ii)*clusterCount(clusterGroupId)/(clusterCount(clusterGroupId)&＃43;1)&＃43;tmpCountG(ii)*1.0/(clusterCount(clusterGroupId)&＃43;1);centrodB(clusterGroupId,ii)&＃61;centrodB(clusterGroupId,ii)*clusterCount(clusterGroupId)/(clusterCount(clusterGroupId)&＃43;1)&＃43;tmpCountB(ii)*1.0/(clusterCount(clusterGroupId)&＃43;1);endclusterCount(clusterGroupId)&＃61;clusterCount(clusterGroupId)&＃43;1;cluster(k)&＃61;clusterGroupId; %第k帧在第clusterGroupId个聚类里面 cluster(3)等于1或者2&＃xff0c;&＃xff0c;也就是属于第一个聚类或者第二个聚类else%形成新的聚类&＃xff0c;增加一个聚类质心count&＃61;count&＃43;1;visit &＃61; visit&＃43;1;clusterCount(count)&＃61;clusterCount(count)&＃43;1;centrodR(count,:)&＃61;tmpCountR;centrodG(count,:)&＃61;tmpCountG;centrodB(count,:)&＃61;tmpCountB;cluster(k)&＃61;count; %第k帧在第count个聚类里面 否则 cluster(k)就在新建的聚类中endend%至此&＃xff0c;所有帧都划进相应的聚类&＃xff0c;一共有count个聚类&＃xff0c;第k帧在第cluster(k)聚类中%现欲取出每个聚类中离质心距离最近&＃xff0c;即相似度最大的作为该聚类的关键帧maxSimilarity&＃61;zeros(1,count);frame&＃61;zeros(1,count);for i&＃61;1:numsR&＃61;0;sG&＃61;0;sB&＃61;0;%运用颜色直方图法的差别函数for j&＃61;1:256sR&＃61;min(centrodR(cluster(i),j),tmpCountR(j))&＃43;sR;%每一帧和聚类质心进行比较&＃xff0c;取最小值 sG&＃61;min(centrodG(cluster(i),j),tmpCountG(j))&＃43;sG;sB&＃61;min(centrodB(cluster(i),j),tmpCountB(j))&＃43;sB;enddR&＃61;sR/sum(tmpCountR);dG&＃61;sG/sum(tmpCountG);dB&＃61;sB/sum(tmpCountB);%YUV,persons are sensitive to Yd&＃61;0.30*dR&＃43;0.59*dG&＃43;0.11*dB;if d>maxSimilarity(cluster(i))maxSimilarity(cluster(i))&＃61;d;frame(cluster(i))&＃61;i;endendfor j&＃61;1:countkey(frame(j))&＃61;1;figure(j);imshow(strcat(&＃39;images/&＃39;,filenames(frame(j)).name));end
endkeyFrameIndexes&＃61;find(key)
这种方法在878帧图片中提取出198帧&＃xff0c;冗余度还是比较高。
(11) 使用CNN来保存图片在使用聚类法提取关键帧&＃xff0c;这种方法由于TensorFlow环境搭配的有问题&＃xff0c;没有实际运行&＃xff0c;在这给出链接
和这个
博主使用一分十五秒00000.MP4视频&＃xff0c; 共1898帧&＃xff0c;分为七类&＃xff0c;帧差最大值帧间差法提取的效果很好。k-means聚类法效果不好&＃xff0c;原因在于&＃xff08;1&＃xff09;代码是copy 别人的&＃xff0c;没有进行优化&＃xff0c;&＃xff08;2&＃xff09;博主对帧间聚类方法理论研究浅薄&＃xff0c;指导不了实践。
k-means聚类 提出484帧
帧差最大值帧间差法提取出35帧
[1]苏筱涵.深度学习视角下视频关键帧提取与视频检索研究[J].网络安全技术与应用,2020(05):65-66.
[2]王红霞,王磊,晏杉杉.视频检索中的关键帧提取方法研究[J].沈阳理工大学学报,2019,38(03):78-82.
[3]王俊玲,卢新明.基于语义相关的视频关键帧提取算法[J/OL].计算机工程与应用:1-10[2020-11-04].http://kns.cnki.net/kcms/detail/11.2127.TP.20200319.1706.018.html.
[4] 张晓宇,张云华.基于融合特征的视频关键帧提取方法.计算机系统应用,2019,28(11):176–181. http://www.c-s-a.org.cn/1003-3254/7163.html
[5] [1]周舟,韩芳,王直杰.面向手语识别的视频关键帧提取和优化算法[J/OL].华东理工大学学报(自然科学版):1-8[2020-11-05].https://doi.org/10.14135/j.cnki.1006-3080.20191201002.
附录&＃xff1a;
1、数字视音频处理知识点小结
[2020-11-04].http://kns.cnki.net/kcms/detail/11.2127.TP.20200319.1706.018.html.
[4] 张晓宇,张云华.基于融合特征的视频关键帧提取方法.计算机系统应用,2019,28(11):176–181. http://www.c-s-a.org.cn/1003-3254/7163.html
[5] [1]周舟,韩芳,王直杰.面向手语识别的视频关键帧提取和优化算法[J/OL].华东理工大学学报(自然科学版):1-8[2020-11-05].https://doi.org/10.14135/j.cnki.1006-3080.20191201002.
[6] https://me.csdn.net/cungudafa这位小姐姐的博客
附录&＃xff1a;
1、数字视音频处理知识点小结
京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有