CodecCompTesting - Revision history

Josh: /* Planned Work (specific) */

2014-05-08T18:15:37Z

Planned Work (specific)

← Older revision		Revision as of 18:15, 8 May 2014
Line 36:		Line 36:

	* [Fixed by moving to RD Curves] The current test suite uses linear interpolation to match quality between the codec being tested and the JPEG baseline. The curve isn't linear, so there is some minor error introduced. This is resolved by switching to RD curves, which don't require quality matching or interpolation.		* [Fixed by moving to RD Curves] The current test suite uses linear interpolation to match quality between the codec being tested and the JPEG baseline. The curve isn't linear, so there is some minor error introduced. This is resolved by switching to RD curves, which don't require quality matching or interpolation.
	* Currently we run the Y-SSIM quality score on PNG images, which are using the RGB colorspace. The input and output images (after encode/decode cycle) are available in YUV image format. Evaluating Y-SSIM quality score (SSIM on the Luma channel Y) on YUV image makes more sense than computing it over RGB colorspace. Solution is to make quality metric programs take YUV input.		* [Fixed] Currently we run the Y-SSIM quality score on PNG images, which are using the RGB colorspace. The input and output images (after encode/decode cycle) are available in YUV image format. Evaluating Y-SSIM quality score (SSIM on the Luma channel Y) on YUV image makes more sense than computing it over RGB colorspace. Solution is to make quality metric programs take YUV input.
	* Currently, the matlab implementation of Y-SSIM (ssim.m) down-scales the images to a normalized dimension of 256. The downscale transformation will smooth out the image feature details and can bump up SSIM quality score evaluated on the Luma channel.		* [Fixed] Currently, the matlab implementation of Y-SSIM (ssim.m) down-scales the images to a normalized dimension of 256. The downscale transformation will smooth out the image feature details and can bump up SSIM quality score evaluated on the Luma channel.
	* Investigate the following feedback from Hacker News.		* Investigate the following feedback from Hacker News.
	** https://news.ycombinator.com/item?id=6581827		** https://news.ycombinator.com/item?id=6581827

Josh: /* Planned Work (specific) */

2014-04-10T19:32:56Z

Planned Work (specific)

← Older revision		Revision as of 19:32, 10 April 2014
Line 43:		Line 43:
	Quality Tests to Port to C		Quality Tests to Port to C

	* Y-SSIM (this currently exists and works for each plane Y, Cb, Cr: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/dump_ssim.c daala/tools/dump_ssim.c])
	* IW-SSIM		* IW-SSIM

	Data Output		Data Output

	* Output RD curve result data to format that is easily human and machine readable (e.g. CSV) (this currently exists for several still image formats: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/rd_collect.sh daala/tools/rd_collect.sh])		* [Fixed] Output RD curve result data to format that is easily human and machine readable (e.g. CSV) (this currently exists for several still image formats: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/rd_collect.sh daala/tools/rd_collect.sh])

	Graphing Capabilities		Graphing Capabilities

	* Add ability to create a graph with one or more sets of RD curve results. (this currently exists using gnuplot: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/rd_plot.sh daala/tools/rd_plot.sh])		* [Fixed] Add ability to create a graph with one or more sets of RD curve results. (this currently exists using gnuplot: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/rd_plot.sh daala/tools/rd_plot.sh])

Negge: link to daala tools

2014-03-24T15:57:15Z

link to daala tools

← Older revision		Revision as of 15:57, 24 March 2014
Line 43:		Line 43:
	Quality Tests to Port to C		Quality Tests to Port to C

	* Y-SSIM		* Y-SSIM (this currently exists and works for each plane Y, Cb, Cr: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/dump_ssim.c daala/tools/dump_ssim.c])
	* IW-SSIM		* IW-SSIM

	Data Output		Data Output

	* Output RD curve result data to format that is easily human and machine readable (e.g. CSV)		* Output RD curve result data to format that is easily human and machine readable (e.g. CSV) (this currently exists for several still image formats: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/rd_collect.sh daala/tools/rd_collect.sh])

	Graphing Capabilities		Graphing Capabilities

	* Add ability to create a graph with one or more sets of RD curve results.		* Add ability to create a graph with one or more sets of RD curve results. (this currently exists using gnuplot: [https://git.xiph.org/?p=daala.git;a=blob;f=tools/rd_plot.sh daala/tools/rd_plot.sh])

Josh at 18:20, 21 March 2014

2014-03-21T18:20:52Z

← Older revision		Revision as of 18:20, 21 March 2014
Line 28:		Line 28:
	* Switch data output to RD curve data		* Switch data output to RD curve data
	* Add graphing capabilities		* Add graphing capabilities
	* Add to quality metric options		* Port some existing quality metric options to C programs. We don't want any more MATLAB programs, they're slow and introduce frustrating dependencies.
	** Port some existing quality metric options to C programs. We don't want any more MATLAB programs, they're slow and introduce frustrating dependencies.
	* Add to collection of images we run tests on.		* Add to collection of images we run tests on.

Line 37:		Line 36:

	* [Fixed by moving to RD Curves] The current test suite uses linear interpolation to match quality between the codec being tested and the JPEG baseline. The curve isn't linear, so there is some minor error introduced. This is resolved by switching to RD curves, which don't require quality matching or interpolation.		* [Fixed by moving to RD Curves] The current test suite uses linear interpolation to match quality between the codec being tested and the JPEG baseline. The curve isn't linear, so there is some minor error introduced. This is resolved by switching to RD curves, which don't require quality matching or interpolation.
	* Currently we run the Y-SSIM quality score on PNG images, which are using the RGB colorspace. The input and output images (after encode/decode cycle) are available in YUV image format. Evaluating Y-SSIM quality score (SSIM on the Luma channel Y) on YUV image makes more sense than computing it over RGB colorspace.		* Currently we run the Y-SSIM quality score on PNG images, which are using the RGB colorspace. The input and output images (after encode/decode cycle) are available in YUV image format. Evaluating Y-SSIM quality score (SSIM on the Luma channel Y) on YUV image makes more sense than computing it over RGB colorspace. Solution is to make quality metric programs take YUV input.
	* Currently, the matlab implementation of Y-SSIM (ssim.m) down-scales the images to a normalized dimension of 256. The downscale transformation will smooth out the image feature details and can bump up SSIM quality score evaluated on the Luma channel.		* Currently, the matlab implementation of Y-SSIM (ssim.m) down-scales the images to a normalized dimension of 256. The downscale transformation will smooth out the image feature details and can bump up SSIM quality score evaluated on the Luma channel.
	* Investigate the following feedback from Hacker News.		* Investigate the following feedback from Hacker News.
	** https://news.ycombinator.com/item?id=6581827		** https://news.ycombinator.com/item?id=6581827


	~~Quality Tests to Add~~

	*

	Quality Tests to Port to C		Quality Tests to Port to C

	*		* Y-SSIM
			* IW-SSIM

	Data Output		Data Output

Josh: /* Planned Work (specific) */

2014-03-21T16:53:52Z

Planned Work (specific)

← Older revision		Revision as of 16:53, 21 March 2014
Line 34:		Line 34:
	== Planned Work (specific) ==		== Planned Work (specific) ==

	* ~~coming soon~~		Methodology

			* [Fixed by moving to RD Curves] The current test suite uses linear interpolation to match quality between the codec being tested and the JPEG baseline. The curve isn't linear, so there is some minor error introduced. This is resolved by switching to RD curves, which don't require quality matching or interpolation.
			* Currently we run the Y-SSIM quality score on PNG images, which are using the RGB colorspace. The input and output images (after encode/decode cycle) are available in YUV image format. Evaluating Y-SSIM quality score (SSIM on the Luma channel Y) on YUV image makes more sense than computing it over RGB colorspace.
			* Currently, the matlab implementation of Y-SSIM (ssim.m) down-scales the images to a normalized dimension of 256. The downscale transformation will smooth out the image feature details and can bump up SSIM quality score evaluated on the Luma channel.
			* Investigate the following feedback from Hacker News.
			** https://news.ycombinator.com/item?id=6581827


			Quality Tests to Add

			*

			Quality Tests to Port to C

			*

			Data Output

			* Output RD curve result data to format that is easily human and machine readable (e.g. CSV)

			Graphing Capabilities

			* Add ability to create a graph with one or more sets of RD curve results.

Josh: Created page with "= Code Compression Testing = == High Level Goal == The goal is to create a framework for testing image compression. We should be able to create data sets for any given codec..."

2014-03-21T16:26:26Z

Created page with "= Code Compression Testing = == High Level Goal == The goal is to create a framework for testing image compression. We should be able to create data sets for any given codec..."

New page

= Code Compression Testing =

== High Level Goal ==

The goal is to create a framework for testing image compression. We should be able to create data sets for any given codec and then compare them.

We may want to extend this system to include testing of video codecs, will keep this in mind when designing it.

== Specific Technical Goals ==

Given a codec, we want to create an RD curve which includes bits-per-pixel (BPP) values for each quality level. For example, if a codec offers quality level options 0-100, we'll create a point on an RD curve for each quality level. The Y axis will represent quality (often labelled "dB", more or less "signal to noise"), which is the result of a single or multiple quality metrics such as PSNR-HVS-M or Y-SSIM. The X axis will represent bits-per-pixel, which is the number of bits in the encoded image divided by the total number of output pixels.

We want to be able to generate graphs containing one or more RD curve from specified codecs. This will allow for comparison.

== The Code ==

Right now there is an existing test framework here:

https://github.com/bdaehlie/web_image_formats

The test harness is written in Python, encoder library wrappers are written in C. The plan is to build on this setup.

== Planned Work (higher level) ==

At a higher level, we need to:

* Make a number of fixes/improvements to test methodology based on feedback from the last study
* Switch data output to RD curve data
* Add graphing capabilities
* Add to quality metric options
** Port some existing quality metric options to C programs. We don't want any more MATLAB programs, they're slow and introduce frustrating dependencies.
* Add to collection of images we run tests on.

== Planned Work (specific) ==

* coming soon