feat: Optimize avatar size multiplier for 2 << n px avatars

[iequidoo]

Instead of 2/3 which is not optimal for 512 px avatars usually passed to Core, use the sequence 3/4,
5/8, 4/8, 3/8... to do "smaller" downscaling steps and reduce aliasing effects.

Before, it was discussed that just 3/4 can be used. However, if we repeat the reduction step, we get
3 << n as a numerator and this way increase aliasing effects on each step. Better limit the
numerator to 5.

[72374]

Note: The Android-app of Delta Chat does now use 512 * 512 for avatar-images.

Also, see: a32210d in #7822 .

---

The resampling ("resizing") is done from the original image, on each iteration, so aliasing-effects do not accumulate.

Details

Here, img is the original, and resize returns a new image:

core/src/blob.rs

Lines 450 to 454 in 5028842

	let new_img = if is_avatar {
	img.resize(target_wh, target_wh, image::imageops::FilterType::Triangle)
	} else {
	img.thumbnail(target_wh, target_wh)
	};

The encoded images are then made from the image resized from the original (new_img), on each iteration of the loop:

core/src/blob.rs

Lines 456 to 462 in 5028842

	if encoded_img_exceeds_bytes(
	context,
	&new_img,
	ofmt.clone(),
	max_bytes,
	&mut encoded,
	)? && is_avatar

encoded_img_exceeds_bytes uses encode_img on line 671 (img here, is the image given to the function; in this situation new_img):

core/src/blob.rs

Lines 664 to 684 in 5028842

	fn encoded_img_exceeds_bytes(
	context: &Context,
	img: &DynamicImage,
	fmt: ImageOutputFormat,
	max_bytes: usize,
	encoded: &mut Vec<u8>,
	) -> anyhow::Result<bool> {
	encode_img(img, fmt, encoded)?;
	if encoded.len() > max_bytes {
	info!(
	context,
	"Image size {}B ({}x{}px) exceeds {}B, need to scale down.",
	encoded.len(),
	img.width(),
	img.height(),
	max_bytes,
	);
	return Ok(true);
	}
	Ok(false)
	}

A previously encoded image (encoded) will be cleared on line 649 in encode_img, and the image, that will be encoded, will be made from the given image (which is new_img, resampled from the original on line 450, in this situation):

core/src/blob.rs

Lines 644 to 662 in 5028842

	fn encode_img(
	img: &DynamicImage,
	fmt: ImageOutputFormat,
	encoded: &mut Vec<u8>,
	) -> anyhow::Result<()> {
	encoded.clear();
	let mut buf = Cursor::new(encoded);
	match fmt {
	ImageOutputFormat::Png => img.write_to(&mut buf, ImageFormat::Png)?,
	ImageOutputFormat::Jpeg { quality } => {
	let encoder = JpegEncoder::new_with_quality(&mut buf, quality);
	// Convert image into RGB8 to avoid the error
	// "The encoder or decoder for Jpeg does not support the color type Rgba8"
	// (<https://github.com/image-rs/image/issues/2211>).
	img.clone().into_rgb8().write_with_encoder(encoder)?;
	}
	}
	Ok(())
	}

[iequidoo]

The resampling ("resizing") is done from the original image, on each iteration, so aliasing-effects do not accumulate

Ok, "accumulate" is probably a bad wording. I meant that if we need to resize a 512 px avatar, using e.g. 27/64 as a multiplier is worse than 1/2 or 3/8 in terms of aliasing effects. So the PR limits the numerator to 5.

[72374]

If i understand correctly, these would be the resolution-values used for resampling, if the file-size is too large at 512x512 (until the target-resolution is below 200 pixels in the first table, and 128 pixels in the second table; rounded down after every step; only resolutions below 512x512):

Resolution-change	Original resolution	1	2	3	4	5	6	7
* 2 / 3	>= 512	341	227	151
This PR	512	384	320	256	192
This PR	1024	384	320	256	192
This PR	1080	405	337	270	202	168
This PR	1637	511	409	306	255	204	153
This PR	1920	480	360	300	240	180
This PR	2000	500	375	312	250	187
This PR	3584	448	336	280	224	168
* 7000 / 8005	>= 512	447	390	341	298	260	227	198

Resolution-change	Original resolution	1	2	3	4	5	6
* 2 / 3	256	170	113
This PR	256	192	160	128	96
* 7000 / 8005	256	223	195	170	148	129	112

I see two issues with this change:

1. If an avatar-image with a resolution of 512x512 (all avatar-images that will be set with the next version of the Android-app of Delta Chat) does not fit within the file-size-limit at 384x384, but would fit at 341x341, it will instead be resampled to 320x320.
   With the linear resampling that is used for avatar-images, aliasing is generally not very noticeable; at least less than a resolution-difference of ~20x20 pixels, with resolutions this low.
2. The first target-resolutions below 512, can vary from 384 to 511, depending on the original resolution of the image, but not in a way that reliably results in higher resolutions for original images with a higher resolution.

Other than that, it would be a nice improvement in quality.

A simpler change, that would almost always improve quality, compared to using * 2 / 3, would be to change the resolution by * 7000 / 8005, per iteration. That would reach the previous integer-values almost exactly, so the resulting images would look at least almost as good as before, and often better.

I hope this is useful. :)

[iequidoo]

That would reach the previous integer-values almost exactly, so the resulting images would look at least almost as good as before

We don't need to try the exact previous integer values, 2 / 3 was a random choice anyway, it's not some magic constant of the world, so rare degradations are acceptable. But if you think there are many real-world images that don't fit when resized to 384 px, so they are unfortunately resized to 320, we can try harder and still use comparably low numerator values: 7/8, 6/8, 11/16, 10/16, 9/16, 8/16, 7/16... Then after 384 px, 352 px will be tried, which should help for most such images.

[iequidoo]

With the linear resampling that is used for avatar-images, aliasing is generally not very noticeable

.

.

.

I took a 512 px square piece of the test screenshot and resized it (GIMP, linear interpolation) to 320 px (i.e. 5/8 like in this PR), 288 px (i.e. 9/16 which corresponds to the geometric progression approach we currently have) and 256 px (i.e. 1/2 like in this PR). I.e. the current (geometric progression) approach is the image in the middle. So, you can compare aliasing effects. If you look at the comment ("Represents a file in the blob directory..."), you can see that the geometric progression approach leads to some parts of letters being bolder than others and some letters being sharper than others, which is almost unnoticeable for this PR's approach.

The first target-resolutions below 512, can vary from 384 to 511, depending on the original resolution of the image, but not in a way that reliably results in higher resolutions for original images with a higher resolution.

This is also not the case for the geometric progression approach. Could you please explain it better or provide an image example to avoid misunderstandings?

EDIT: Adding 257 px ("random" numerator value), 256 px (1/2 like in this PR) and 255 px images to make it clearer why the numerator value matters (i.e. smaller is better):

[72374]

Regarding aliasing:
My point was simply, that 320x320 is not better than 341x341, even if aliasing would be handled better in that case, because the resolution-difference has a larger effect on image-quality.¹

An example of that, is visible in two of the example-images in your message: in the 256x256-image, in the second line of text from the bottom, the m in "methods" looks like n, while it is still properly readable in the 288x288-image.

How noticeable the aliasing will be, can also change, depending on where one crops an image; even if it shows the same content.

The first target-resolutions below 512, can vary from 384 to 511, depending on the original resolution of the image, but not in a way that reliably results in higher resolutions for original images with a higher resolution.

This is also not the case for the geometric progression approach. Could you please explain it better or provide an image example to avoid misunderstandings?

I was arguing unnecessarily complicated there, by adding the part about the relation to the original resolution. ^^'
(Considering the option to assume that higher resolution of cropped image = more detail = higher target-resolution necessary, but that would not work well anyway.)

The actual problem is, that the target-resolution varies strongly; and additionally, in a way that is not easy to understand for the people who set their avatar-images, so that they could not even work around that easily.

When setting an avatar-image that happens to have a resolution of 960x960 (after cropping), the first target-resolution after 512x512 would then be 480x480 pixels; but if it had a resolution of 1030x1030 (after cropping), it would be 386x386 pixels, which is about ~35% lower in resolution.

Footnotes

1. This could be different, if the resampling-algorithm would not handle aliasing well, like thumbnail, but bilinear-resampling is effectively a form of anti-aliasing. ↩

[iequidoo]

My point was simply, that 320x320 is not better than 341x341, even if aliasing would be handled better in that case, because the resolution-difference has a larger effect on image-quality.

I agree that 320 px is probably worse because 341 px is quite bigger, but this PR tries 384 px first. Why assume that most images won't fit at 384 px? If it's really the case, we should consider multipliers closer to 1, but optimizing the algo so that it tries exactly 341 px doesn't make sense. Yes, some users will get slightly smaller avatars, but most users will gain.

The actual problem is, that the target-resolution varies strongly; and additionally, in a way that is not easy to understand for the people who set their avatar-images, so that they could not even work around that easily.

When setting an avatar-image that happens to have a resolution of 960x960 (after cropping), the first target-resolution after 512x512 would then be 480x480 pixels; but if it had a resolution of 1030x1030 (after cropping), it would be 386x386 pixels, which is about ~35% lower in resolution.

If this happens for most avatars, it means that 512 px are too restrictive and we should increase the limit to 1024 px. But even if we remove this limit at all, w/o this PR, passing a bigger (less cropped) image doesn't mean getting a bigger image in result because it may stop fitting into 60 kB and we may need to repeat the downscaling step.

But ok, as we still want to try 512 px (btw, i don't understand why, but i don't want to change the code too much) which doesn't care about aliasing (e.g. resizing from 520 to 512 px definitely adds aliasing), i'll remove the try to take into account the original resolution from the code.

[72374]

as we still want to try 512 px (btw, i don't understand why

Me neither, but i just found a reason why the limit should have been increased to 640 in Core, instead of reducing it to 512 in the Android-app of Delta Chat.
The highest resolution that contact-avatars can be displayed at, without opening those in the image-viewer, is 640x640 (in the Desktop-app, by increasing the size with Ctrl + +, or the option in the "View"-menu; while the screen-scaling in the system's settings is set to 100%, otherwise the resolution can be higher):

Image

Making avatar-images have a high enough resolution to properly fill the largest size at which it will be used in a GUI, seems like a good reason to try resizing it to the limit first.