其它过滤器文章：
day09-----1-----FFmpeg过滤器框架分析。
day09-----2-----视频过滤器实战。
day09-----3-----音频过滤器实战(ffmpeg进行混音，将两路音频pcm数据合成一路输出)。
day09-----4-----FFmpeg filter补充之使用filter添加水印。
day09-----5-----FFmpeg filter补充之使用复杂的filter过滤视频。
ffmpeg的filter⽤起来是和Gstreamer的plugin是⼀样的概念，通过avfilter_link，将各个创建好的filter按 ⾃⼰想要的次序链接到⼀起，然后avfilter_graph_config之后，就可以正常使⽤。
⽐较常⽤的滤镜有：scale、trim、overlay、rotate、movie、yadif。scale 滤镜⽤于缩放，trim 滤镜⽤ 于帧级剪切，overlay 滤镜⽤于视频叠加，rotate 滤镜实现旋转，movie 滤镜可以加载第三⽅的视频， yadif 滤镜可以去隔⾏。
1 主要结构体和API介绍
1.1 AVFilterGraph-对filters系统的整体管理
重点：
struct AVFilterGraph {
AVFilterContext **filters;
unsigned nb_filters;
};完整结构体：
1.2 AVFilter-定义filter本身的能⼒
重点：
const char *name; // overlay
const AVFilterPad *inputs;
const AVFilterPad *outputs;例如：
AVFilter ff_vf_overlay = {
.name = "overlay",
.description = NULL_IF_CONFIG_SMALL("Overlay a video source on top of the input."),
.preinit = overlay_framesync_preinit,
.init = init,
.uninit = uninit,
.priv_size = sizeof(OverlayContext),
.priv_class = &overlay_class,
.query_formats = query_formats,
.activate = activate,
.process_command = process_command,
.inputs = avfilter_vf_overlay_inputs,
.outputs = avfilter_vf_overlay_outputs,
.flags = AVFILTER_FLAG_SUPPORT_TIMELINE_INTERNAL |
AVFILTER_FLAG_SLICE_THREADS,
};定义filter本身的能⼒，拥有的pads，回调函数接⼝定义：
/**
* Filter definition. This defines the pads a filter contains, and all the
* callback functions used to interact with the filter.
*/
typedef struct AVFilter {
/**
* Filter name. Must be non-NULL and unique among filters.
*/
const char *name;
/**
* A description of the filter. May be NULL.
*
* You should use the NULL_IF_CONFIG_SMALL() macro to define it.
*/
const char *description;
/**
* List of inputs, terminated by a zeroed element.
*
* NULL if there are no (static) inputs. Instances of filters with
* AVFILTER_FLAG_DYNAMIC_INPUTS set may have more inputs than present in
* this list.
*/
const AVFilterPad *inputs;
/**
* List of outputs, terminated by a zeroed element.
*
* NULL if there are no (static) outputs. Instances of filters with
* AVFILTER_FLAG_DYNAMIC_OUTPUTS set may have more outputs than present in
* this list.
*/
const AVFilterPad *outputs;
/**
* A class for the private data, used to declare filter private AVOptions.
* This field is NULL for filters that do not declare any options.
*
* If this field is non-NULL, the first member of the filter private data
* must be a pointer to AVClass, which will be set by libavfilter generic
* code to this class.
*/
const AVClass *priv_class;
/**
* A combination of AVFILTER_FLAG_*
*/
int flags;
/*****************************************************************
* All fields below this line are not part of the public API. They
* may not be used outside of libavfilter and can be changed and
* removed at will.
* New public fields should be added right above.
*****************************************************************
*/
/**
* Filter pre-initialization function
*
* This callback will be called immediately after the filter context is
* allocated, to allow allocating and initing sub-objects.
*
* If this callback is not NULL, the uninit callback will be called on
* allocation failure.
*
* @return 0 on success,
* AVERROR code on failure (but the code will be
* dropped and treated as ENOMEM by the calling code)
*/
int (*preinit)(AVFilterContext *ctx);
/**
* Filter initialization function.
*
* This callback will be called only once during the filter lifetime, after
* all the options have been set, but before links between filters are
* established and format negotiation is done.
*
* Basic filter initialization should be done here. Filters with dynamic
* inputs and/or outputs should create those inputs/outputs here based on
* provided options. No more changes to this filter's inputs/outputs can be
* done after this callback.
*
* This callback must not assume that the filter links exist or frame
* parameters are known.
*
* @ref AVFilter.uninit "uninit" is guaranteed to be called even if
* initialization fails, so this callback does not have to clean up on
* failure.
*
* @return 0 on success, a negative AVERROR on failure
*/
int (*init)(AVFilterContext *ctx);
/**
* Should be set instead of @ref AVFilter.init "init" by the filters that
* want to pass a dictionary of AVOptions to nested contexts that are
* allocated during init.
*
* On return, the options dict should be freed and replaced with one that
* contains all the options which could not be processed by this filter (or
* with NULL if all the options were processed).
*
* Otherwise the semantics is the same as for @ref AVFilter.init "init".
*/
int (*init_dict)(AVFilterContext *ctx, AVDictionary **options);
/**
* Filter uninitialization function.
*
* Called only once right before the filter is freed. Should deallocate any
* memory held by the filter, release any buffer references, etc. It does
* not need to deallocate the AVFilterContext.priv memory itself.
*
* This callback may be called even if @ref AVFilter.init "init" was not
* called or failed, so it must be prepared to handle such a situation.
*/
void (*uninit)(AVFilterContext *ctx);
/**
* Query formats supported by the filter on its inputs and outputs.
*
* This callback is called after the filter is initialized (so the inputs
* and outputs are fixed), shortly before the format negotiation. This
* callback may be called more than once.
*
* This callback must set AVFilterLink.out_formats on every input link and
* AVFilterLink.in_formats on every output link to a list of pixel/sample
* formats that the filter supports on that link. For audio links, this
* filter must also set @ref AVFilterLink.in_samplerates "in_samplerates" /
* @ref AVFilterLink.out_samplerates "out_samplerates" and
* @ref AVFilterLink.in_channel_layouts "in_channel_layouts" /
* @ref AVFilterLink.out_channel_layouts "out_channel_layouts" analogously.
*
* This callback may be NULL for filters with one input, in which case
* libavfilter assumes that it supports all input formats and preserves
* them on output.
*
* @return zero on success, a negative value corresponding to an
* AVERROR code otherwise
*/
int (*query_formats)(AVFilterContext *);
int priv_size; ///< size of private data to allocate for the filter
int flags_internal; ///< Additional flags for avfilter internal use only.
/**
* Used by the filter registration system. Must not be touched by any other
* code.
*/
struct AVFilter *next;
/**
* Make the filter instance process a command.
*
* @param cmd the command to process, for handling simplicity all commands must be alphanumeric only
* @param arg the argument for the command
* @param res a buffer with size res_size where the filter(s) can return a response. This must not change when the command is not supported.
* @param flags if AVFILTER_CMD_FLAG_FAST is set and the command would be
* time consuming then a filter should treat it like an unsupported command
*
* @returns >=0 on success otherwise an error code.
* AVERROR(ENOSYS) on unsupported commands
*/
int (*process_command)(AVFilterContext *, const char *cmd, const char *arg, char *res, int res_len, int flags);
/**
* Filter initialization function, alternative to the init()
* callback. Args contains the user-supplied parameters, opaque is
* used for providing binary data.
*/
int (*init_opaque)(AVFilterContext *ctx, void *opaque);
/**
* Filter activation function.
*
* Called when any processing is needed from the filter, instead of any
* filter_frame and request_frame on pads.
*
* The function must examine inlinks and outlinks and perform a single
* step of processing. If there is nothing to do, the function must do
* nothing and not return an error. If more steps are or may be
* possible, it must use ff_filter_set_ready() to schedule another
* activation.
*/
int (*activate)(AVFilterContext *ctx);
} AVFilter;1.3 AVFilterContext-filter实例，管理filter与外部的联系
filter实例，管理filter与外部的联系。
重点：
struct AVFilterContext {
const AVFilter *filter;
char *name;
AVFilterPad *input_pads;
AVFilterLink **inputs;
unsigned nb_inputs AVFilterPad *output_pads;
AVFilterLink **outputs;
unsigned nb_outputs;
struct AVFilterGraph *graph; // 从属于哪个AVFilterGraph
}完整结构体：
/** An instance of a filter */
struct AVFilterContext {
const AVClass *av_class; ///< needed for av_log() and filters common options
const AVFilter *filter; ///< the AVFilter of which this is an instance
char *name; ///< name of this filter instance
AVFilterPad *input_pads; ///< array of input pads
AVFilterLink **inputs; ///< array of pointers to input links
unsigned nb_inputs; ///< number of input pads
AVFilterPad *output_pads; ///< array of output pads
AVFilterLink **outputs; ///< array of pointers to output links
unsigned nb_outputs; ///< number of output pads
void *priv; ///< private data for use by the filter
struct AVFilterGraph *graph; ///< filtergraph this filter belongs to
/**
* Type of multithreading being allowed/used. A combination of
* AVFILTER_THREAD_* flags.
*
* May be set by the caller before initializing the filter to forbid some
* or all kinds of multithreading for this filter. The default is allowing
* everything.
*
* When the filter is initialized, this field is combined using bit AND with
* AVFilterGraph.thread_type to get the final mask used for determining
* allowed threading types. I.e. a threading type needs to be set in both
* to be allowed.
*
* After the filter is initialized, libavfilter sets this field to the
* threading type that is actually used (0 for no multithreading).
*/
int thread_type;
/**
* An opaque struct for libavfilter internal use.
*/
AVFilterInternal *internal;
struct AVFilterCommand *command_queue;
char *enable_str; ///< enable expression string
void *enable; ///< parsed expression (AVExpr*)
double *var_values; ///< variable values for the enable expression
int is_disabled; ///< the enabled state from the last expression evaluation
/**
* For filters which will create hardware frames, sets the device the
* filter should create them in. All other filters will ignore this field:
* in particular, a filter which consumes or processes hardware frames will
* instead use the hw_frames_ctx field in AVFilterLink to carry the
* hardware context information.
*/
AVBufferRef *hw_device_ctx;
/**
* Max number of threads allowed in this filter instance.
* If <= 0, its value is ignored.
* Overrides global number of threads set per filter graph.
*/
int nb_threads;
/**
* Ready status of the filter.
* A non-0 value means that the filter needs activating;
* a higher value suggests a more urgent activation.
*/
unsigned ready;
/**
* Sets the number of extra hardware frames which the filter will
* allocate on its output links for use in following filters or by
* the caller.
*
* Some hardware filters require all frames that they will use for
* output to be defined in advance before filtering starts. For such
* filters, any hardware frame pools used for output must therefore be
* of fixed size. The extra frames set here are on top of any number
* that the filter needs internally in order to operate normally.
*
* This field must be set before the graph containing this filter is
* configured.
*/
int extra_hw_frames;
};1.4 AVFilterLink-定义两个filters之间的联接
重点：
struct AVFilterLink
{
AVFilterContext *src;
AVFilterPad *srcpad;
AVFilterContext *dst;
AVFilterPad *dstpad;
struct AVFilterGraph *graph;
};完整结构体：
/**
* A link between two filters. This contains pointers to the source and
* destination filters between which this link exists, and the indexes of
* the pads involved. In addition, this link also contains the parameters
* which have been negotiated and agreed upon between the filter, such as
* image dimensions, format, etc.
*
* Applications must not normally access the link structure directly.
* Use the buffersrc and buffersink API instead.
* In the future, access to the header may be reserved for filters
* implementation.
*/
struct AVFilterLink {
AVFilterContext *src; ///< source filter
AVFilterPad *srcpad; ///< output pad on the source filter
AVFilterContext *dst; ///< dest filter
AVFilterPad *dstpad; ///< input pad on the dest filter
enum AVMediaType type; ///< filter media type
/* These parameters apply only to video */
int w; ///< agreed upon image width
int h; ///< agreed upon image height
AVRational sample_aspect_ratio; ///< agreed upon sample aspect ratio
/* These parameters apply only to audio */
uint64_t channel_layout; ///< channel layout of current buffer (see libavutil/channel_layout.h)
int sample_rate; ///< samples per second
int format; ///< agreed upon media format
/**
* Define the time base used by the PTS of the frames/samples
* which will pass through this link.
* During the configuration stage, each filter is supposed to
* change only the output timebase, while the timebase of the
* input link is assumed to be an unchangeable property.
*/
AVRational time_base;
/*****************************************************************
* All fields below this line are not part of the public API. They
* may not be used outside of libavfilter and can be changed and
* removed at will.
* New public fields should be added right above.
*****************************************************************
*/
/**
* Lists of formats and channel layouts supported by the input and output
* filters respectively. These lists are used for negotiating the format
* to actually be used, which will be loaded into the format and
* channel_layout members, above, when chosen.
*
*/
AVFilterFormats *in_formats;
AVFilterFormats *out_formats;
/**
* Lists of channel layouts and sample rates used for automatic
* negotiation.
*/
AVFilterFormats *in_samplerates;
AVFilterFormats *out_samplerates;
struct AVFilterChannelLayouts *in_channel_layouts;
struct AVFilterChannelLayouts *out_channel_layouts;
/**
* Audio only, the destination filter sets this to a non-zero value to
* request that buffers with the given number of samples should be sent to
* it. AVFilterPad.needs_fifo must also be set on the corresponding input
* pad.
* Last buffer before EOF will be padded with silence.
*/
int request_samples;
/** stage of the initialization of the link properties (dimensions, etc) */
enum {
AVLINK_UNINIT = 0, ///< not started
AVLINK_STARTINIT, ///< started, but incomplete
AVLINK_INIT ///< complete
} init_state;
/**
* Graph the filter belongs to.
*/
struct AVFilterGraph *graph;
/**
* Current timestamp of the link, as defined by the most recent
* frame(s), in link time_base units.
*/
int64_t current_pts;
/**
* Current timestamp of the link, as defined by the most recent
* frame(s), in AV_TIME_BASE units.
*/
int64_t current_pts_us;
/**
* Index in the age array.
*/
int age_index;
/**
* Frame rate of the stream on the link, or 1/0 if unknown or variable;
* if left to 0/0, will be automatically copied from the first input
* of the source filter if it exists.
*
* Sources should set it to the best estimation of the real frame rate.
* If the source frame rate is unknown or variable, set this to 1/0.
* Filters should update it if necessary depending on their function.
* Sinks can use it to set a default output frame rate.
* It is similar to the r_frame_rate field in AVStream.
*/
AVRational frame_rate;
/**
* Buffer partially filled with samples to achieve a fixed/minimum size.
*/
AVFrame *partial_buf;
/**
* Size of the partial buffer to allocate.
* Must be between min_samples and max_samples.
*/
int partial_buf_size;
/**
* Minimum number of samples to filter at once. If filter_frame() is
* called with fewer samples, it will accumulate them in partial_buf.
* This field and the related ones must not be changed after filtering
* has started.
* If 0, all related fields are ignored.
*/
int min_samples;
/**
* Maximum number of samples to filter at once. If filter_frame() is
* called with more samples, it will split them.
*/
int max_samples;
/**
* Number of channels.
*/
int channels;
/**
* Link processing flags.
*/
unsigned flags;
/**
* Number of past frames sent through the link.
*/
int64_t frame_count_in, frame_count_out;
/**
* A pointer to a FFFramePool struct.
*/
void *frame_pool;
/**
* True if a frame is currently wanted on the output of this filter.
* Set when ff_request_frame() is called by the output,
* cleared when a frame is filtered.
*/
int frame_wanted_out;
/**
* For hwaccel pixel formats, this should be a reference to the
* AVHWFramesContext describing the frames.
*/
AVBufferRef *hw_frames_ctx;
#ifndef FF_INTERNAL_FIELDS
/**
* Internal structure members.
* The fields below this limit are internal for libavfilter's use
* and must in no way be accessed by applications.
*/
char reserved[0xF000];
#else /* FF_INTERNAL_FIELDS */
/**
* Queue of frames waiting to be filtered.
*/
FFFrameQueue fifo;
/**
* If set, the source filter can not generate a frame as is.
* The goal is to avoid repeatedly calling the request_frame() method on
* the same link.
*/
int frame_blocked_in;
/**
* Link input status.
* If not zero, all attempts of filter_frame will fail with the
* corresponding code.
*/
int status_in;
/**
* Timestamp of the input status change.
*/
int64_t status_in_pts;
/**
* Link output status.
* If not zero, all attempts of request_frame will fail with the
* corresponding code.
*/
int status_out;
#endif /* FF_INTERNAL_FIELDS */
};1.5 AVFilterPad-定义filter的输⼊/输出接⼝
重点：
struct AVFilterPad
{
const char *name;
AVFrame *(*get_video_buffer)(AVFilterLink *link, int w, int h);
AVFrame *(*get_audio_buffer)(AVFilterLink *link, int nb_samples);
int (*filter_frame)(AVFilterLink *link, AVFrame *frame);
int (*request_frame)(AVFilterLink *link);
};完整结构体：
/**
* A filter pad used for either input or output.
*/
struct AVFilterPad {
/**
* Pad name. The name is unique among inputs and among outputs, but an
* input may have the same name as an output. This may be NULL if this
* pad has no need to ever be referenced by name.
*/
const char *name;
/**
* AVFilterPad type.
*/
enum AVMediaType type;
/**
* Callback function to get a video buffer. If NULL, the filter system will
* use ff_default_get_video_buffer().
*
* Input video pads only.
*/
AVFrame *(*get_video_buffer)(AVFilterLink *link, int w, int h);
/**
* Callback function to get an audio buffer. If NULL, the filter system will
* use ff_default_get_audio_buffer().
*
* Input audio pads only.
*/
AVFrame *(*get_audio_buffer)(AVFilterLink *link, int nb_samples);
/**
* Filtering callback. This is where a filter receives a frame with
* audio/video data and should do its processing.
*
* Input pads only.
*
* @return >= 0 on success, a negative AVERROR on error. This function
* must ensure that frame is properly unreferenced on error if it
* hasn't been passed on to another filter.
*/
int (*filter_frame)(AVFilterLink *link, AVFrame *frame);
/**
* Frame poll callback. This returns the number of immediately available
* samples. It should return a positive value if the next request_frame()
* is guaranteed to return one frame (with no delay).
*
* Defaults to just calling the source poll_frame() method.
*
* Output pads only.
*/
int (*poll_frame)(AVFilterLink *link);
/**
* Frame request callback. A call to this should result in some progress
* towards producing output over the given link. This should return zero
* on success, and another value on error.
*
* Output pads only.
*/
int (*request_frame)(AVFilterLink *link);
/**
* Link configuration callback.
*
* For output pads, this should set the link properties such as
* width/height. This should NOT set the format property - that is
* negotiated between filters by the filter system using the
* query_formats() callback before this function is called.
*
* For input pads, this should check the properties of the link, and update
* the filter's internal state as necessary.
*
* For both input and output filters, this should return zero on success,
* and another value on error.
*/
int (*config_props)(AVFilterLink *link);
/**
* The filter expects a fifo to be inserted on its input link,
* typically because it has a delay.
*
* input pads only.
*/
int needs_fifo;
/**
* The filter expects writable frames from its input link,
* duplicating data buffers if needed.
*
* input pads only.
*/
int needs_writable;
};1.6 AVFilterInOut-过滤器链输⼊/输出的链接列表
/**
* A linked-list of the inputs/outputs of the filter chain.
*
* This is mainly useful for avfilter_graph_parse() / avfilter_graph_parse2(),
* where it is used to communicate open (unlinked) inputs and outputs from and
* to the caller.
* This struct specifies, per each not connected pad contained in the graph, the
* filter context and the pad index required for establishing a link.
*/
typedef struct AVFilterInOut {
/** unique name for this input/output in the list */
char *name;
/** filter context associated to this input/output */
AVFilterContext *filter_ctx;
/** index of the filt_ctx pad to use for linking */
int pad_idx;
/** next input/input in the list, NULL if this is the last */
struct AVFilterInOut *next;
} AVFilterInOut;在AVFilter模块中定义了AVFilter结构，很个AVFilter都是具有独⽴功能的节点，如scale filter的作⽤就是 进⾏图像尺⼨变换，overlay filter的作⽤就是进⾏图像的叠加。
这⾥需要重点提的是两个特别的filter，⼀个是buffer，⼀个是buffersink。
- 1）滤波器buffer代表filter graph中的源头，原始数据就往这个filter节点输⼊的。
- 2）⽽滤波器buffersink代表filter graph中的输出节点，处理完成的数据从这个filter节点输出。
2 函数使⽤
// 获取FFmpeg中定义的filter，调⽤该⽅法前需要先调⽤avfilter_register_all();进⾏滤波器注册
AVFilter avfilter_get_by_name(const char name);
// 往源滤波器buffer中输⼊待处理的数据
int av_buffersrc_add_frame(AVFilterContext ctx, AVFrame frame);
// 从⽬的滤波器buffersink中获取处理完的数据
int av_buffersink_get_frame(AVFilterContext ctx, AVFrame frame);
// 创建⼀个滤波器图filter graph
AVFilterGraph *avfilter_graph_alloc(void);
// 创建⼀个滤波器实例AVFilterContext，并添加到AVFilterGraph中
int avfilter_graph_create_filter(AVFilterContext **filt_ctx, const AVFilter *filt, const char name, const char args, void *opaque, AVFilterGraph *graph_ctx);
// 连接两个滤波器节点
int avfilter_link(AVFilterContext *src, unsigned srcpad, AVFilterContext *dst, unsigned dstpad);3 AVFilter主体框架流程
在利⽤AVFilter进⾏⾳视频数据处理前先将在进⾏的处理流程绘制出来，现在以FFmpeg filter官⽅⽂档中 的⼀个例⼦为例进⾏说明。
这个例⼦的处理流程如上所示，⾸先使⽤split滤波器将input流分成两路流（main和tmp），然后分别对两 路流进⾏处理。对于tmp流，先经过crop滤波器进⾏裁剪处理，再经过flip滤波器进⾏垂直⽅向上的翻转操 作，输出的结果命名为flip流。再将main流和flip流输⼊到overlay滤波器进⾏合成操作。
上图的input就是上⾯提过的buffer源滤波器，output就是上⾯的提过的buffersink滤波器。上图中每个节点都是⼀个 AVFilterContext，每个连线就是AVFliterLink。所有这些信息都统⼀由AVFilterGraph来管理。