Transform Search

Functions
int64_t	av1_pixel_diff_dist (const MACROBLOCK *x, int plane, int blk_row, int blk_col, const BLOCK_SIZE plane_bsize, const BLOCK_SIZE tx_bsize, unsigned int *block_mse_q8)
	Compute the pixel domain distortion.
int64_t	av1_uniform_txfm_yrd (const AV1_COMP *const cpi, MACROBLOCK *x, RD_STATS *rd_stats, int64_t ref_best_rd, BLOCK_SIZE bs, TX_SIZE tx_size, FAST_TX_SEARCH_MODE ftxs_mode, int skip_trellis)
	Transform type search for luma macroblock with fixed transform size.
void	av1_pick_recursive_tx_size_type_yrd (const AV1_COMP *cpi, MACROBLOCK *x, RD_STATS *rd_stats, BLOCK_SIZE bsize, int64_t ref_best_rd)
	Recursive transform size and type search.
void	av1_pick_uniform_tx_size_type_yrd (const AV1_COMP *const cpi, MACROBLOCK *x, RD_STATS *rd_stats, BLOCK_SIZE bs, int64_t ref_best_rd)
	Uniform transform size and type search.
int	av1_txfm_uvrd (const AV1_COMP *const cpi, MACROBLOCK *x, RD_STATS *rd_stats, BLOCK_SIZE bsize, int64_t ref_best_rd)
	Chroma block transform search.
void	av1_txfm_rd_in_plane (MACROBLOCK *x, const AV1_COMP *cpi, RD_STATS *rd_stats, int64_t ref_best_rd, int64_t current_rd, int plane, BLOCK_SIZE plane_bsize, TX_SIZE tx_size, FAST_TX_SEARCH_MODE ftxs_mode, int skip_trellis)
	Transform type search with fixed transform size.
int	av1_txfm_search (const AV1_COMP *cpi, MACROBLOCK *x, BLOCK_SIZE bsize, RD_STATS *rd_stats, RD_STATS *rd_stats_y, RD_STATS *rd_stats_uv, int mode_rate, int64_t ref_best_rd)
	Recursive transform size and type search.

Detailed Description

This module describes transform search algorithm in AV1.

Function Documentation

av1_pixel_diff_dist()

int64_t av1_pixel_diff_dist	(	const MACROBLOCK *	x,
		int	plane,
		int	blk_row,
		int	blk_col,
		const BLOCK_SIZE	plane_bsize,
		const BLOCK_SIZE	tx_bsize,
		unsigned int *	block_mse_q8 )

Compute the pixel domain distortion.

Compute the pixel domain distortion from diff on all visible 4x4s in the transform block.

Parameters

[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	plane	Plane index
[in]	blk_row	Block row index
[in]	blk_col	Block col index
[in]	plane_bsize	Current plane block size
[in]	tx_bsize	Transform size
[in]	block_mse_q8	Block mse

Returns

An int64_t value that is the block sse.

av1_uniform_txfm_yrd()

int64_t av1_uniform_txfm_yrd	(	const AV1_COMP *const	cpi,
		MACROBLOCK *	x,
		RD_STATS *	rd_stats,
		int64_t	ref_best_rd,
		BLOCK_SIZE	bs,
		TX_SIZE	tx_size,
		FAST_TX_SEARCH_MODE	ftxs_mode,
		int	skip_trellis )

Transform type search for luma macroblock with fixed transform size.

Search for the best transform type and return the transform coefficients RD cost of current luma macroblock with the given uniform transform size.

Parameters

[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	cpi	Top-level encoder structure
[in]	rd_stats	Pointer to struct to keep track of the RD stats
[in]	ref_best_rd	Best RD cost seen for this block so far
[in]	bs	Size of the current macroblock
[in]	tx_size	The given transform size
[in]	ftxs_mode	Transform search mode specifying desired speed and quality tradeoff
[in]	skip_trellis	Binary flag indicating if trellis optimization should be skipped

Returns

An int64_t value that is the best RD cost found.

av1_pick_recursive_tx_size_type_yrd()

void av1_pick_recursive_tx_size_type_yrd	(	const AV1_COMP *	cpi,
		MACROBLOCK *	x,
		RD_STATS *	rd_stats,
		BLOCK_SIZE	bsize,
		int64_t	ref_best_rd )

Recursive transform size and type search.

Search for best transform size and type for luma inter blocks. The transform block partitioning can be recursive resulting in non-uniform transform sizes. The best transform size and type, if found, will be saved in the MB_MODE_INFO structure, and the corresponding RD stats will be saved in rd_stats.

Parameters

[in]	cpi	Top-level encoder structure
[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	rd_stats	Pointer to struct to keep track of the RD stats
[in]	bsize	Current macroblock size
[in]	ref_best_rd	Best RD cost seen for this block so far

Remarks

Nothing is returned. The selected transform size and type will be saved in the MB_MODE_INFO structure

Referenced by av1_interpolation_filter_search().

av1_pick_uniform_tx_size_type_yrd()

void av1_pick_uniform_tx_size_type_yrd	(	const AV1_COMP *const	cpi,
		MACROBLOCK *	x,
		RD_STATS *	rd_stats,
		BLOCK_SIZE	bs,
		int64_t	ref_best_rd )

Uniform transform size and type search.

Search for the best transform size and type for current macroblock block, with the assumption that all the transform blocks have a uniform size (VP9 style). The selected transform size and type will be saved in the MB_MODE_INFO structure; the corresponding RD stats will be saved in rd_stats. This function may be used for both intra and inter predicted blocks.

Parameters

[in]	cpi	Top-level encoder structure
[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	rd_stats	Pointer to struct to keep track of the RD stats
[in]	bs	Current macroblock size
[in]	ref_best_rd	Best RD cost seen for this block so far

Remarks

Nothing is returned. The selected transform size and type will be saved in the MB_MODE_INFO structure

Referenced by av1_handle_intra_y_mode(), av1_rd_pick_intra_sby_mode(), handle_filter_intra_mode(), intra_block_yrd(), palette_rd_y(), and rd_pick_filter_intra_sby().

av1_txfm_uvrd()

int av1_txfm_uvrd	(	const AV1_COMP *const	cpi,
		MACROBLOCK *	x,
		RD_STATS *	rd_stats,
		BLOCK_SIZE	bsize,
		int64_t	ref_best_rd )

Chroma block transform search.

Calculate the transform coefficient RD cost for the given chroma macroblock If the current mode is intra, then this function will compute the predictor.

Parameters

[in]	cpi	Top-level encoder structure
[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	rd_stats	Pointer to struct to keep track of the RD stats
[in]	bsize	Current macroblock size
[in]	ref_best_rd	Best RD cost seen for this block so far

Returns

An integer value is returned. 0: early termination triggered, no valid rd cost available; 1: rd cost values are valid.

Referenced by av1_rd_pick_intra_sbuv_mode().

av1_txfm_rd_in_plane()

void av1_txfm_rd_in_plane	(	MACROBLOCK *	x,
		const AV1_COMP *	cpi,
		RD_STATS *	rd_stats,
		int64_t	ref_best_rd,
		int64_t	current_rd,
		int	plane,
		BLOCK_SIZE	plane_bsize,
		TX_SIZE	tx_size,
		FAST_TX_SEARCH_MODE	ftxs_mode,
		int	skip_trellis )

Transform type search with fixed transform size.

Search for the best transform type and calculate the transform coefficients RD cost of the current transform block with the specified (uniform) transform size and plane. The RD results will be saved in rd_stats.

Parameters

[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	cpi	Top-level encoder structure
[in]	rd_stats	Pointer to struct to keep track of the RD stats
[in]	ref_best_rd	Best RD cost seen for this block so far
[in]	current_rd	Current RD cost for this block so far
[in]	plane	Plane index
[in]	plane_bsize	Size of the current macroblock considering sup-sampling
[in]	tx_size	The given transform size
[in]	ftxs_mode	Transform search mode specifying desired speed and quality tradeoff
[in]	skip_trellis	Binary flag indicating if trellis optimization should be skipped

Remarks

Nothing is returned. The RD results will be saved in rd_stats.

av1_txfm_search()

int av1_txfm_search	(	const AV1_COMP *	cpi,
		MACROBLOCK *	x,
		BLOCK_SIZE	bsize,
		RD_STATS *	rd_stats,
		RD_STATS *	rd_stats_y,
		RD_STATS *	rd_stats_uv,
		int	mode_rate,
		int64_t	ref_best_rd )

Recursive transform size and type search.

This function combines y and uv planes' transform search processes together for inter-predicted blocks (including IntraBC), when the prediction is already generated. It first does subtraction to obtain the prediction error. Then it calls av1_pick_recursive_tx_size_type_yrd/av1_pick_uniform_tx_size_type_yrd and av1_txfm_uvrd sequentially and handles possible early terminations. The RD metrics are calculated and stored in rd_stats/_y/_uv.

Parameters

[in]	cpi	Top-level encoder structure
[in]	x	Pointer to structure holding the data for the current encoding macroblock
[in]	bsize	Current macroblock size
[in]	rd_stats	Pointer to struct to keep track of the overal RD stats
[in]	rd_stats_y	Pointer to struct to keep track of the RD stats for the luma plane
[in]	rd_stats_uv	Pointer to struct to keep track of the RD stats for the chroma planes
[in]	mode_rate	Rate cost to encode the prediction mode info. of the current macroblock
[in]	ref_best_rd	Best RD cost seen for this block so far

Returns

An integer value is returned indicating if a valid transform candidate is found (1) or not (0).

Referenced by motion_mode_rd(), and rd_pick_intrabc_mode_sb().