remote bookmarked playlists are sometimes sent to the top of the bookmarked playlists tab when they are reloaded

[ajgae]

Checklist

• I am able to reproduce the bug with the latest version given here: CLICK THIS LINK.
• I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• I have read the FAQ and my problem isn't listed.
• I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• This issue contains only one bug.
• I have read and understood the contribution guidelines.

Affected version

0.27.0

Steps to reproduce the bug

1. open NewPipe
2. create a local playlist
3. bookmark one user-created remote playlist
4. bookmark multiple youtube-generated remote playlists (e.g. found by looking for music abums using the Albums search filter)
5. move the local playlist and the user-created remote playlist to the top of the bookmarked playlists
6. for each youtube-generated remote playlist, open it then go back to the bookmarked playlists view

Expected behavior

the order of playlists should not change when the playlists are opened

Actual behavior

• when opened, the youtube generated remote playlists are moved up to the top of the bookmarked playlists, above the user-created remote playlist and the local playlist, where they are ordered alphabetically among themselves
• when opened, neither the local playlist nor the user-created remote playlist are sent back up to the top of the bookmarked playlists (expected behavior)

Screenshots/Screen recordings

video_2024-04-29_16-00-46.mp4

Logs

No response

Affected Android/Custom ROM version

Android 12

Affected device model

Samsung Galaxy S10

Additional information

No response

[ajgae]

reopening as a new issue from a discussion on #10993, as suggested by @opusforlife2. Here is the state of my research on the topic so far:

• the getPlaylists() queries in the RemotePlaylistDAO and local playlist DAO classes seem to be ordered by display index, as expected, which lets me think that the issue is with either merge logic or the updating of display indices, but not with the queries themselves
• the fact that only autogenerated remote playlists are affected is very strange and confuses me greatly
• the fact that remote playlists are sorted alphabetically before all other playlists may indicate that they share the same display index
• this line sets the display index to -1 for all remote playlists by default (which is strange, because newly bookmarked remote playlists do not appear at the top of the bookmarked playlists but at the bottom) (nevermind they do appear at the top)
• the only place in the entire codebase where setDisplayIndex() is called on a PlaylistRemoteEntity is in BookmarkFragment.saveImmediate(), which is called by BookmarkFragment.onPause() (this corresponds with the moment at which the playlists are moved all the way up)
  • in saveImmediate(), debounceSaver.getIsModified() returns false so the function returns immediately. The code which computes and updates the display index for playlists is not executed. it seems that the issue is with the debounceSaver in some way, but i do not understand this part of the application very well

[ajgae]

you can consider that i working on this btw, I just won't have time in the next couple of days.

Here's a new specificity i observed about the behavior (shown in the video below): the buggy reordering is only triggered when the playlist is loaded. Once it has been loaded, opening it does not trigger the reordering

video_2024-04-30_11-05-52.mp4

[opusforlife2]

@ajgae If it's a small bug fix, easy to review, sure. Otherwise it might be postponed for, or after, the rewrite.

[throwaway242685]

sorry for bumping this up, but can anyone please take a look at this bug?

it's been over 2 months already, and playlists are just unusable at this point.

can this at least be fixed in the nightly build? :(

[feeRnt]

Hey @ajgae.

I came across this issue when looking for #10993

If you/anyone's still working on this, I found some pointers that may help.

NewPipe treats local and remote playlists differently, in that you can load a local playlist's index and cached thumbnails without having internet, but the same approach on a remote playlist will return a network error. This mandates a fresh update of a remote playlist each time it is loaded.

The jumping to the top behavior of YT generated remote playlists can be reproduced for user generated remote playlists if changes are made to the playlist from YouTube's side.
Some of these changes include, adding a video to the playlist, removing a video from the playlist, changing the order of the videos in the playlist, renaming the playlist, and even changing the Youtube account display name of the owner of the playlist.

If you bookmark a user-generated remote playlist on NewPipe, it will be at the top of the tab. If you move that playlist down, then perform any of the actions above (maybe on a playlist you made on your account), restart NewPipe and then load the playlist again, the playlist will again jump to the top like in your video. If you don't perform those actions then the order of the playlist stays the same.
I figured this out with help from some older playlist I had saved and some testing.

Now on auto-generated playlists the behavior is slightly strange. I was trying to reproduce the bug, but it seemed it didn't happen every time, not with a restart, nor with a force stop. Then I noticed that it seems to happen only at each 5 minute mark. So if I load the playlist, it jumps up. If move it down, restart, reload it, nothing happens, unless the time on the clock has passed multiples of 5 minutes. So say 12:05, 12:10, 12:15 ... If I load it at 12:02, reload it at 12:04, nothing will happen. But if I reload it at 12:05 or 12:06 (anything past 12:05), then the playlist will jump to the top again (only after a NewPipe restart before each attempt of course).

The links of the playlist, and their videos always stay the same.

Now I don't know if NewPipe does something (I'm guessing unlikely) or if YouTube does something with their playlist autogeneration every 5 minutes, where it causes the videos in the playlist to be "refreshed", or their video orders "reset", (I'm not sure how NewPipe parses this ordering change either). Or maybe the autogenerated "uploader" gets a new "tag"?

I did also check the album video thumbnail and topic uploader pfp for fun and they seem to stay the same based on their checksums hah.

Either way no matter how the autogeneration causes it I think this is undesired behavior. There is a chance that that the behavior of the Playlist ordering/viewing was the same before as well with auto or user generation but it went unnoticed because they were ordered alphabetically then.

But I don't really know java or the codebase enough to fix it.

My thought was an exception for playlists (their links) that are already existent in the bookmarked playlists tab where the displayIndex = -1 ordering is not applied for their case. This might create unnecessary overhead though.

I personally prefer a solution where the ordering is switched back to alphabetical, for some of the reasons mentioned in the linked issue. I can always append a 0 to a name to bring it to the top.
I just realized the displayIndex = -1 is useful for remote playlists as you can't rename them, but it'd be good if we could rename them as well for easier sorting.

I noticed you said the remote playlists are sorted to the bottom, but I couldn't reproduce it. The remote ones have their own alphabetical sorting but they seem to follow it, autogenerated and jumping to the top or not.

Since this issue affects both user and autogenerated playlists it might be worth renaming it as well.

[ajgae]

i have looked further into this since then but i genuinely dont know how to fix this. however i agree with @throwaway26425 in that this is pretty major, its extremely annoying.

like @feeRnt pointed out, the youtube playlists are thrown up to the top when they are deemed "different enough" at load time (when opened) that they need to be reloaded (e.g. when a video has been added or removed on the youtube side), at which point they are reassigned a display index of -1. but the implementation is weird and i havent been able to figure out how to fix it, though i know precisely what and to an extent where in the code the issue is. i'd really appreciate some help.

also, i cant reproduce the behavior i had described earlier (playlists appearing at the bottom when first saved) as @feeRnt mentioned, so i have edited my comment.

[feeRnt]

I've been looking at the other related issue with playlist sorting to see if I can find the fix, since it's apparently just an sql ORDER BY clause in the code.

I probably won't be of much help, but maybe you can share your latest findings of where the problematic code is so we can take a look? @ajgae

In the meanwhile a temporary and somewhat annoying workaround for remote playlists is:

Add remote playlist -> Go back to bookmarked playlists tab -> Load playlist -> Tap on the three dots overflow menu -> Add to playlist -> New Playlist, at the top

Then you can name it whatever and sort accordingly, this new local playlist will not be updated in its index every time you open it after some state change.

[throwaway242685]

playlists are just..... unusable, can anyone please take a look at this bug?

can you please give us an option to sort playlists the way they were before?

like, sort them alphabetically? just like before? 😐