Feed update compression

[jpds]

Checklist

• I am able to reproduce the bug with the latest version given here: CLICK THIS LINK.
• I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• I have read the FAQ and my problem isn't listed.
• I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• This issue contains only one bug.
• I have read and understood the contribution guidelines.
• I have read and understood the AI policy. The content of this bug report is not generated by AI.

Affected version

0.28.2

Steps to reproduce the bug

I use WireGuard on my phone and when I perform a refresh of my ~700 subscriptions, WireGuard tells me that 65MB of data has been received (tested and reset the counter three times to exclude any other apps suddenly consuming a load of data).

This feels a little heavy for a feed updates so I'm not sure if Accept-Encoding is set for the update URLs.

Expected behavior

No response

Actual behavior

No response

Screenshots/Screen recordings

No response

Logs

No response

Affected Android/Custom ROM version

No response

Affected device model

No response

Additional information

No response

We use okhttp library which by default adds Accept-Encoding gzip. I've checked and it seems to really use gzip.

One additional note. Okhttp allows to add zstd and/or brotli in several lines (adding it to the current architecture is a bit more difficult task). I quickly tested it and really youtube can answer you with brotli if you mention it in Accept-Encoding (they ignore zstd though). It is probably should be added into app. But I haven't checked yet how it really affects bandwidth consumption, cpu load, app disk usage increase, update speed(especially because we have artificial limit update to not be blocked/rate limited it probably won't have effect on update speed)

I will test it further soon and will create PR but there is no guarantee it is cost effecctive...

[dftf-stu]

Yes, tested this myself and to refresh my subs list (around 210 channels) took 23.2MB of data.

Given many-people globally may access Internet primarily on data-capped mobile-networks, this is rather a lot, given the number of times a day the feed might be refreshed, to get the latest videos, shorts and comments data.

Text comments should be highly-compressible data; and for video thumbnails and user profile images, surely these get cached, so only new (or changed) ones are downloaded each-time? 🤔

So some sort-of further compression here would be great... or, alternatively, maybe do one of the following instead:

• let people choose on a per-channel basis how frequently to get updates for it (e.g. every time; once per day; once per week, to reduce how many channels updates are obtained for, given not all channels upload as-regularly), or;
• let people choose what data gets included in a full refresh (e.g.: video thumbnails and titles, but don't download comments unless I choose to go into the comments for that particular video)

• let people choose on a per-channel basis how frequently to get updates for it (e.g. every time; once per day; once per week, to reduce how many channels updates are obtained for, given not all channels upload as-regularly), or;

You already have this opportunity with channel groups. You can also downgrade thumbs quality in settings

• let people choose what data gets included in a full refresh (e.g.: video thumbnails and titles, but don't download comments unless I choose to go into the comments for that particular video)

In comparison to original app fetching premade by youtube feed (one page) with only necessary videos, when you are external app and don't use youtube apis directly you need probably to fetch each channel page individually. Event I am not sure how concretely fetching is done, I am sure It doesn't fetch each video page not to mention fetching comments of those videos

Text comments should be highly-compressible data; and for video thumbnails and user profile images, surely these get cached, so only new (or changed) ones are downloaded each-time?

Idk but probably it fetches thumbs only when video is new in the feed. Especially given how I spot old preview in feed after refreshing for some creators who love to change their previews 100500 times for each video. I can say for sure that when you click on video, it fetches thumbs fot that video again.

[dftf-stu]

@dustdfg : I'm not-sure people would logically use the Channel Groups feature for that purpose, though... yes, it is a valid workaround, but surely the app itself should look to reduce the data somehow, or not re-import already-cached data?

I've just done a full refresh of my "All" subscriptions list, and monitored the data used during it.

23.4MB it used; so 0.2MB more than the 23.2MB when I tested originally.

So, to me, that says the app downloads ALL data for every-channel EVERY time it does a refresh (given it was 23.2MB before, then 23.4MB now -- so 0.2MB is the what should ideally be downloaded, as that represents only the new or changed data).

Updates really need to be compressed, or made incremental -- not re-download every single bit of data every time, I feel...

@dustdfg : I'm not-sure people would logically use the Channel Groups feature for that purpose, though... yes, it is a valid workaround, but surely the app itself should look to reduce the data somehow, or not re-import already-cached data?

It is mostly not reimported when possible but you can't just download part of something redistributed only fully.

I've just done a full refresh of my "All" subscriptions list, and monitored the data used during it.

23.4MB it used; so 0.2MB more than the 23.2MB when I tested originally.

So, to me, that says the app downloads ALL data for every-channel EVERY time it does a refresh (given it was 23.2MB before, then 23.4MB now -- so 0.2MB is the what should ideally be downloaded, as that represents only the new or changed data).

Can you load part of the page? Can you download for example only part of the html page? No...

Updates really need to be compressed, or made incremental -- not re-download every single bit of data every time, I feel...

They are already compressed just with gzip.

@dftf-stu If I provide you apk which enables brotli, will you test it?

[dftf-stu]

@dustdfg : I wouldn't want to install a random APK, no. I mean, I've no idea who you are so... bit risky.

But on your "can you load part of a page"? Surely when the app requests updates, for each channel it will get sent the data for each video, including the title and thumbnail, and the comments. As part of that process, is it not possible for each channel for the API or extractor (or whatever the correct term is) to say "please send me only data from after this video"?

Ideally, if I last refreshed my "All" feed say 4 hours ago, then for each channel it just needs to ask:

• What new videos, shorts or posts have been added after that point?
• Have any of the previous video or Short titles or thumbnails changed since (so look at a "last modified" property)?
• Have any videos been made private since?

Comments surely only need to be loaded if a person were to access that tab on a video, not stored offline.

Like, imagine if you used an API to obtain data for a YouTube channel, and then dumped that into a spreadsheet. You then do that a few hours later. You can then use comparison tools within the spreadsheet app to say "highlight only rows where any data has changed", so you can narrow-down on what is new. Surely this app during a data-refresh could do similar?

Some collected stats (+ apk available for testing) https://github.com/dustdfg/NewPipe/pull/1#issuecomment-3843316637

[hollownights]

As part of that process, is it not possible for each channel for the API or extractor (or whatever the correct term is) to say "please send me only data from after this video"?

@dftf-stu NewPipe doesn't have access to the official YouTube API, so is has to download the entire channel page to get data...

Can you load part of the page? Can you download for example only part of the html page? No...

@dustdfg

... BUT I think this could work:

1. For the very first feed update*, retrieve all the channels’ data using the default method (non-dedicated feeds, full channels’ pages) and save all the data gathered;

2. From now on, use the dedicated RSS feeds to check for updates. Cache all the data gathered and see if there are, for each channel, any new videos;

2.1) Use these dedicated feeds to update/replace the titles of the videos previously gathered from the full pages. This will partially solve the problem with translated titles (#3089 #13042), as the dedicated feed always has the original titles. For new videos, also save the titles from the dedicated feeds;

2.2) If there are new videos for a channel, proceed to 3. If not, just end the update process here;

3. Only for channels with new videos (seen in the dedicated feed), do a full channel’s page data retrieval. Save all the data gathered, but for the videos also seen in the dedicated feeds, keep the titles from the dedicated feeds.

*The “very first update” is the first app-wide feed update to be done after/if this change is implemented, but I also think that, when a user subscribe to a channel, NewPipe should gather all the channel's videos and add them to the “What’s new” feed, something that is not being done right now. This would reduce the need to download the entire channel again for a day or two (or even more, depending on the channel).

Also relevant: #9977 #11817