FFmpeg入门：最简单的音视频播放器（Plus优化版）

今天我们继续学习FFmpeg的入门，咱们主要是从上一期的音频播放器的基础上进行了部分优化。没有看过上期讲解的朋友可以去回顾一下，链接放下面了。
FFmpeg入门：最简单的音视频播放器

本期我们对上次遗留的问题进行优化，从而丰富音视频播放器的性能；主要解决下面两个问题，一个一个的进行解决。

1. 时钟同步怎么做
2. 如何边读出packet，边解码frame并播放

音视频时钟同步

其实音视频同步是我学ffmpeg的时候最早困惑我的一个问题，啥叫做音视频同步啊，怎么同步啊，依据什么进行同步啊？这些问题都让我充满了大大的问号，因此为了让一些和我一样是初学者的朋友能够一起入个门，我尽量讲的通俗易懂一些。

首先什么是音视频同步？
实际上对于常见的多媒体文件来说，通常分为音频流和视频流（偶尔还会有个字幕流），具象的来看就是我们经常在视频剪辑软件下面看到的那几个轨道，其实每个轨道都有对应的音频流或者视频流。

居然是不同的轨道，那么他们自己就有自己的运行速度，如果说我们不对音频和视频的播放速度进行同步和调整，那么就会出现我们常说的音画不同步的现象，观感非常难受。

如何实现音视频同步？
那么我们要如何实现音视频同步呢？其实这就是一个简单的你追我赶的事情。好比现在音频和视频是两位100米选手，要尽可能让二者在奔跑的过程中处于同一个位置。那么我们就要不断的调整。

比如现在选手A跑快了，选手B就要加快速度赶上他；选手A跑慢了，选手B就要减慢速度等待他。反过来依然如此。同理到音视频同步，二者之间会有一个作为参照着，比如音频为参照物，那么视频流就会不断检查当前的位置和音频的位置是否有差别，如果快了就延迟，慢了就丢帧。

同步相关参数概念
介绍几个用于音视频同步的核心参数，帮助大家在使用ffpmeg实际运用

time_base：时间基，定义时间戳的单位，和pts息息相关，广泛应用于音视频流处理的各个方面。
pts：即呈现时间戳，表示每个视频或音频帧在播放时应该出现在时间轴上的具体时间。
dts：即解码时间戳。DTS 指示每个视频或音频帧应该在什么时候被送到解码器进行解码。

实现
具体实现也非常简单，我们以音频时间戳作为参照（人对于音频的抖动更加敏感，通过都会选择按照音频时间戳为准），分为下面几步。

1. 在音频和时间的解码线程中，增加更新当前时间戳的步骤。即没解码一帧，都会获取当前帧的时间戳
2. 在视频解码线程中，额外增加和音频时间戳对比的过程。
3. 如果视频时间戳大于音频时间戳，丢帧处理；如果小于音频时间戳，延迟处理。
   

相关代码如下：

更新视频和音频的时间戳方法

/**更新当前的视频时钟*/
void get_current_video_time(VideoState* video_state, AVFrame* frame) {AVFormatContext* formatCtx = video_state->formatCtx;AVRational time_base = formatCtx->streams[video_state->videoStream]->time_base;if (frame->pts != AV_NOPTS_VALUE) {video_state->videoParam->timestamp = (Uint32) ((av_q2d(time_base)*frame->pts) * 1000);} else {video_state->videoParam->timestamp = 0;}
}/**更新当前的音频时钟*/
void get_current_audio_time(VideoState* video_state, AVFrame* frame) {AVFormatContext* formatCtx = video_state->formatCtx;AVRational time_base = formatCtx->streams[video_state->audioStream]->time_base;if (frame->pts != AV_NOPTS_VALUE) {video_state->audioParam->timestamp = (Uint32) ((av_q2d(time_base) * frame->pts) * 1000);} else {video_state->audioParam->timestamp = 0;}
}

音频解码过程中的处理

// 更新音频时钟
get_current_audio_time(video_state, pFrame);

视频解码过程中的处理

	// 更新时钟get_current_video_time(video_state, pFrame);// 视频同步音频时钟Uint32 audio_timestamp = video_state->audioParam->timestamp;Uint32 video_timestamp = video_state->videoParam->timestamp;if (video_timestamp > audio_timestamp) {SDL_Delay(video_timestamp-audio_timestamp);		// 延迟视频戳和音频戳的差值}if (video_timestamp < audio_timestamp) {continue;}

优化packet读和解码线程

第二部分就是提高效率，上一期我们做的音视频播放器中，有一个明显的效率问题。我们是将所有的packet读入到队列之后再进行解码和显示。这就导致整个预加载过程非常慢，尤其是对于帧数较多的视频来说。

为了解决这个问题，我们对线程进行进一步优化。拆出单独的read_thread线程专门用来读取packet。如图所示

代码实现

主要看main函数就好，其他几个c文件和上期保持一致

//
//  main.c
//  sample_player
//
//  Created by chenhuaiyi on 2025/2/26.
//#include "utils.h"
#include "manager.h"AudioInfo audio_info;/* udata: 传入的参数* stream: SDL音频缓冲区* len: SDL音频缓冲区大小* 回调函数*/
void fill_audio(void *udata, Uint8 *stream, int len){SDL_memset(stream, 0, len);			// 必须重置，不然全是电音!!!if(audio_info.audio_len==0){					// 有音频数据时才调用return;}len = (len>audio_info.audio_len ? audio_info.audio_len : len);	// 最多填充缓冲区大小的数据SDL_MixAudio(stream, audio_info.audio_pos, len, SDL_MIX_MAXVOLUME);audio_info.audio_pos += len;audio_info.audio_len -= len;
}/**音频解码线程*/
int audio_thread(void *arg) {/**1. 从packet_queue队列中取出packet2. 将packet进行解码3. 写入到sdl的缓冲区中*/VideoState* video_state = (VideoState*) arg;AudioParam* audio_param = video_state->audioParam;PacketQueue* queue = video_state->aQueue;audio_param->index = 0;AVPacket 	packet;int 		ret;AVFrame* 	pFrame = av_frame_alloc();for(;;) {if (!video_state->isReadEnd || queue->size > 0) {packet_queue_pop(queue, &packet);// 将packet写入编解码器ret = avcodec_send_packet(video_state->aCodecCtx, &packet);if ( ret < 0 ) {printf("send packet error\n");return -1;}// 获取解码后的帧while (!avcodec_receive_frame(video_state->aCodecCtx, pFrame)) {// 格式转化swr_convert(video_state->swrCtx, &audio_param->out_buffer, audio_param->out_buffer_size,(const uint8_t **)pFrame->data, pFrame->nb_samples);audio_param->index++;// 更新音频时钟get_current_audio_time(video_state, pFrame);printf("第%d音频帧 \t 帧大小(采样点):%d  \t pts:%lld \t 预期播放点:%.2fs\n",audio_param->index,packet.size,packet.pts,(double) audio_param->timestamp / 1000);
#if USE_SDL// 设置读取的音频数据audio_info.audio_len = audio_param->out_buffer_size;audio_info.audio_pos = (Uint8 *) audio_param->out_buffer;// 等待SDL播放完成while(audio_info.audio_len > 0)SDL_Delay(0.5);
#endif}av_packet_unref(&packet);}else {break;}}av_frame_free(&pFrame);// 结束video_state->isEnd = 1;return 0;
}/**视频解码线程*/
int video_thread(void *arg) {/**1. 从视频pkt队列中读出packet2. 送入解码器解码并取出3. 使用SDL进行渲染4. 根据pts计算延迟SDL_DELAY*/VideoState* 	video_state = (VideoState*) arg;PacketQueue* 	video_queue = video_state->vQueue;AVCodecContext* pCodecCtx = video_state->vCodecCtx;AVFrame* 		out_frame = video_state->videoParam->out_frame;AVPacket packet;AVFrame* pFrame = av_frame_alloc();for (;;) {if (!video_state->isReadEnd || video_queue->size > 0) {packet_queue_pop(video_queue, &packet);// 将packet写入编解码器int ret = avcodec_send_packet(pCodecCtx, &packet);if (ret < 0) {printf("packet resolve error!");break;}// 从解码器中取出原始帧while (!avcodec_receive_frame(pCodecCtx, pFrame)) {// 帧格式转化，转为YUV420Psws_scale(video_state->swsCtx,						// sws_context转换(uint8_t const * const *)pFrame->data,	// 输入 datapFrame->linesize,							// 输入 每行数据的大小（对齐）0,										// 输入 Y轴位置pCodecCtx->height,						// 输入 heightout_frame->data,							// 输出 dataout_frame->linesize);						// 输出 linesize// 帧更新video_state->videoParam->frame_update = 1;// 更新时钟get_current_video_time(video_state, pFrame);// 视频同步音频时钟Uint32 audio_timestamp = video_state->audioParam->timestamp;Uint32 video_timestamp = video_state->videoParam->timestamp;if (video_timestamp > audio_timestamp) {SDL_Delay(video_timestamp-audio_timestamp);		// 延迟视频戳和音频戳的差值}if (video_timestamp < audio_timestamp) {continue;}video_state->videoParam->index++;printf("第%i视频帧 \t 帧类型(I/P/B):%s帧 \t pts:%d \t 预期播放点:%.2fs\n",video_state->videoParam->index,get_frame_type(pFrame),(int) pFrame->pts,(double) video_timestamp / 1000);}av_packet_unref(&packet);} else {break;}}av_frame_free(&pFrame);// 结束video_state->isEnd = 1;return 1;
}/**读取packet进程*/
int read_thread(void* arg) {VideoState* video_state = (VideoState*) arg;AVPacket* packet = av_packet_alloc();	// packet初始化// 循环1: 从文件中读取packetwhile(av_read_frame(video_state->formatCtx, packet)>=0){/** 写入音频pkt队列 */if(packet->stream_index==video_state->audioStream){packet_queue_push(video_state->aQueue, packet);}/** 写入视频pkt队列 */if (packet->stream_index==video_state->videoStream) {packet_queue_push(video_state->vQueue, packet);}av_packet_unref(packet);}av_packet_free(&packet);				// 释放packet空间video_state->isReadEnd = 1;				// 读取线程已结束return 1;
}int main(int argc, char* argv[])
{VideoState* 		video_state;AudioParam*			audio_param;VideoParam*			video_param;SDL_Event 			event;SDL_Rect      		rect;if(argc < 2) {fprintf(stderr, "Usage: test <file>\n");exit(1);}/** 初始化函数 */init_video_state(&video_state);audio_param = video_state->audioParam;video_param = video_state->videoParam;avformat_network_init();// 1. 打开视频文件，获取格式上下文if(avformat_open_input(&video_state->formatCtx, argv[1], NULL, NULL)!=0){printf("Couldn't open input stream.\n");return -1;}// 2. 对文件探测流信息if(avformat_find_stream_info(video_state->formatCtx, NULL) < 0){printf("Couldn't find stream information.\n");return -1;}// 打印信息av_dump_format(video_state->formatCtx, 0, argv[1], 0);// 3. 找到对应的 音频流/视频流 索引video_state->audioStream=-1;video_state->videoStream=-1;for(int i=0; i < video_state->formatCtx->nb_streams; i++) {if(video_state->formatCtx->streams[i]->codecpar->codec_type==AVMEDIA_TYPE_AUDIO){video_state->audioStream=i;}if (video_state->formatCtx->streams[i]->codecpar->codec_type==AVMEDIA_TYPE_VIDEO) {video_state->videoStream=i;}}if(video_state->audioStream==-1){printf("Didn't find a audio stream.\n");return -1;}if (video_state->videoStream==-1) {printf("Didn't find a video stream.\n");return -1;}// 4. 将 音频流/视频流 编码参数写入上下文AVCodecParameters* aCodecParam = video_state->formatCtx->streams[video_state->audioStream]->codecpar;avcodec_parameters_to_context(video_state->aCodecCtx, aCodecParam);AVCodecParameters* vCodecParam = video_state->formatCtx->streams[video_state->videoStream]->codecpar;avcodec_parameters_to_context(video_state->vCodecCtx, vCodecParam);// 5. 查找流的编码器video_state->aCodec = avcodec_find_decoder(video_state->aCodecCtx->codec_id);if(video_state->aCodec==NULL){printf("Audio codec not found.\n");return -1;}video_state->vCodec = avcodec_find_decoder(video_state->vCodecCtx->codec_id);if(video_state->vCodec==NULL){printf("Video codec not found.\n");return -1;}// 6. 打开流的编解码器if(avcodec_open2(video_state->aCodecCtx, video_state->aCodec, NULL)<0){printf("Could not open audio codec.\n");return -1;}if(avcodec_open2(video_state->vCodecCtx, video_state->vCodec, NULL)<0){printf("Could not open video codec.\n");return -1;}/** 音频输出信息构建 */audio_output_set(video_state);/** 视频输出信息构建 */video_output_set(video_state);//	SDL 初始化
#if USE_SDLif(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO | SDL_INIT_TIMER)) {printf( "Could not initialize SDL - %s\n", SDL_GetError());return -1;}// 在 main 函数开始处添加SDL_SetHint(SDL_HINT_VIDEO_MAC_FULLSCREEN_SPACES, "0");SDL_SetHint(SDL_HINT_MAC_BACKGROUND_APP, "1");/** 初始化音频SDL设备 */SDL_AudioSpec wanted_spec;//	audio_sdl_set(video_state, &wanted_spec, fill_audio);wanted_spec.freq = audio_param->out_sample_rate;					// 采样率wanted_spec.format = AUDIO_S16SYS;									// 采样格式 16bitwanted_spec.channels = audio_param->out_channels;					// 通道数wanted_spec.silence = 0;wanted_spec.samples = audio_param->out_nb_samples;					// 单帧处理的采样点wanted_spec.callback = fill_audio;									// 回调函数wanted_spec.userdata = video_state->aCodecCtx;						// 回调函数的参数/** 初始化视频SDL设备 */SDL_Window*       window = NULL;SDL_Renderer*     renderer = NULL;SDL_Texture*      texture= NULL;//	video_sdl_set(video_state, &window, &renderer, &texture);/** 窗口 */window = SDL_CreateWindow("SDL2 window",SDL_WINDOWPOS_CENTERED,SDL_WINDOWPOS_CENTERED,video_state->vCodecCtx->width,video_state->vCodecCtx->height,SDL_WINDOW_SHOWN);if (!window) {printf("SDL_CreateWindow Error: %s\n", SDL_GetError());SDL_Quit();return 1;}/** 渲染 */renderer = SDL_CreateRenderer(window,-1,SDL_RENDERER_ACCELERATED | SDL_RENDERER_PRESENTVSYNC);if (!renderer) {printf("SDL_CreateRenderer Error: %s\n", SDL_GetError());SDL_DestroyWindow(window);SDL_Quit();return 1;}/** 纹理 */texture = SDL_CreateTexture(renderer,SDL_PIXELFORMAT_YV12,SDL_TEXTUREACCESS_STREAMING,video_state->vCodecCtx->width,video_state->vCodecCtx->height);// 打开音频播放器if (SDL_OpenAudio(&wanted_spec, NULL)<0) {printf("can't open audio.\n");return -1;}#endif// 音频上下文格式转换swr_alloc_set_opts2(&video_state->swrCtx,&audio_param->out_channel_layout,			// 输出layoutaudio_param->out_sample_fmt,				// 输出格式audio_param->out_sample_rate,				// 输出采样率&video_state->aCodecCtx->ch_layout,			// 输入layoutvideo_state->aCodecCtx->sample_fmt,			// 输入格式video_state->aCodecCtx->sample_rate,		// 输入采样率0, NULL);swr_init(video_state->swrCtx);// 视频上下文格式转换video_state->swsCtx = sws_getContext(video_state->vCodecCtx->width,			// src 宽video_state->vCodecCtx->height,		// src 高video_state->vCodecCtx->pix_fmt,		// src 格式video_param->width,					// dst 宽video_param->height,					// dst 高video_param->pix_fmt,					// dst 格式SWS_BILINEAR,NULL,NULL,NULL);// 开始播放SDL_PauseAudio(0);int64_t av_start_time = av_gettime();	// 播放开始时间戳SDL_CreateThread(read_thread, "read_thread", video_state);// 创建一个线程并启动SDL_CreateThread(audio_thread, "audio_thread", video_state);SDL_CreateThread(video_thread, "video_thread", video_state);//	video_thread(video_state);AVFrame* out_frame = NULL;while (!video_state->isEnd) {// 处理事件（必须由主线程执行）while (SDL_PollEvent(&event)) {if (event.type == SDL_QUIT) {video_state->isEnd = 1;}}if (video_state->videoParam->frame_update) {// 将AVFrame的数据写入到texture中，然后渲染后windows上rect.x = 0;rect.y = 0;rect.w = video_state->vCodecCtx->width;rect.h = video_state->vCodecCtx->height;out_frame = video_state->videoParam->out_frame;// 更新纹理SDL_UpdateYUVTexture(texture, &rect,out_frame->data[0], out_frame->linesize[0],	// 	Yout_frame->data[1], out_frame->linesize[1],	// 	Uout_frame->data[2], out_frame->linesize[2]);	//  V// 渲染页面SDL_RenderClear(renderer);SDL_RenderCopy(renderer, texture, NULL, NULL);SDL_RenderPresent(renderer);// 重置标志video_state->videoParam->frame_update = 0;}}// 打印参数printf("格式: %s\n", video_state->formatCtx->iformat->name);printf("时长: %lld us\n", video_state->formatCtx->duration);printf("音频持续时长为 %.2f，音频帧总数为 %d\n", (double)(av_gettime()-av_start_time)/AV_TIME_BASE, audio_param->index);printf("码率: %lld\n", video_state->formatCtx->bit_rate);printf("编码器: %s (%s)\n", video_state->aCodecCtx->codec->long_name, avcodec_get_name(video_state->aCodecCtx->codec_id));printf("通道数: %d\n", video_state->aCodecCtx->ch_layout.nb_channels);printf("采样率: %d \n", video_state->aCodecCtx->sample_rate);printf("单通道每帧的采样点数目: %d\n", video_state->aCodecCtx->frame_size);printf("pts单位(ms*1000): %.2f\n", av_q2d(video_state->formatCtx->streams[video_state->audioStream]->time_base) * AV_TIME_BASE);#if USE_SDLSDL_CloseAudio();SDL_DestroyTexture(texture);SDL_DestroyRenderer(renderer);SDL_DestroyWindow(window);SDL_Quit();
#endifdestory_video_state(&video_state);return 0;
}