multi threaded feed fetch

[veganomy]

Checklist

• I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• I have read the FAQ and my problem isn't listed.
• I'm aware that this is a request for NewPipe itself and that requests for adding a new service need to be made at NewPipeExtractor.
• I have taken the time to fill in all the required details. I understand that the feature request will be dismissed otherwise.
• This issue contains only one feature request.
• I have read and understood the contribution guidelines.

Feature description

When the subscriptions cross beyond 30, the fetching starts to feel slow. Currently they're being fetched single threadedly. If fetched in multiple threads (for example 4), it could finish within the seconds.

Why do you want this feature?

To speed up the feed fetching

Additional information

No response

[veganomy]

Will this be triaged anytime soon ?

[TobiGr]

Hi, we are currently using three threads to fetch the feeds from the internet. However, we needed to throttle getting YouTube subscriptions to not hit their rate limits and getting your IP flagged

NewPipe/app/src/main/java/org/schabi/newpipe/local/feed/service/FeedLoadManager.kt

Lines 342 to 357 in 824fc8f

	/**
	* How many extractions will be running in parallel.
	*/
	private const val PARALLEL_EXTRACTIONS = 3
	/**
	* How many YouTube extractions to perform before waiting [DELAY_BETWEEN_BATCHES_MILLIS]
	* to avoid being rate limited
	*/
	private const val BATCH_SIZE = 50
	/**
	* Wait a random delay in this range once every [BATCH_SIZE] YouTube extractions to avoid
	* being rate limited
	*/
	private val DELAY_BETWEEN_BATCHES_MILLIS = (6000L..12000L)

NewPipe/app/src/main/java/org/schabi/newpipe/local/feed/service/FeedLoadManager.kt

Lines 98 to 133 in 824fc8f

	return outdatedSubscriptions
	.take(1)
	.doOnNext {
	currentProgress.set(0)
	maxProgress.set(it.size)
	}
	.filter { it.isNotEmpty() }
	.observeOn(AndroidSchedulers.mainThread())
	.doOnNext {
	notificationUpdater.onNext("")
	broadcastProgress()
	}
	.observeOn(Schedulers.io())
	.flatMap { Flowable.fromIterable(it) }
	.takeWhile { !cancelSignal.get() }
	.doOnNext { subscriptionEntity ->
	// throttle YouTube extractions once every BATCH_SIZE to avoid being rate limited
	if (subscriptionEntity.serviceId == ServiceList.YouTube.serviceId) {
	val previousCount = youtubeExtractionCount.getAndIncrement()
	if (previousCount != 0 && previousCount % BATCH_SIZE == 0) {
	Thread.sleep(DELAY_BETWEEN_BATCHES_MILLIS.random())
	}
	}
	}
	.parallel(PARALLEL_EXTRACTIONS, PARALLEL_EXTRACTIONS * 2)
	.runOn(Schedulers.io(), PARALLEL_EXTRACTIONS * 2)
	.filter { !cancelSignal.get() }
	.map { subscriptionEntity ->
	loadStreams(subscriptionEntity, useFeedExtractor, defaultSharedPreferences)
	}
	.sequential()
	.observeOn(AndroidSchedulers.mainThread())
	.doOnNext(NotificationConsumer())
	.observeOn(Schedulers.io())
	.buffer(BUFFER_COUNT_BEFORE_INSERT)
	.doOnNext(DatabaseConsumer())

However, we could talk about changing the thread count based on the number of CPU cores. e.g.

private val PARALLEL_EXTRACTIONS = maxOf(1, Runtime.getRuntime().availableProcessors() / 2)

[veganomy]

Aren't most devices these days are octacore with some blend of 2x/6x or 4x/4x efficient / performance cores ?

Btw, hitting rate limits is often the most frustrating thing with YouTube for me. Fortunately on a cellular network, toggling airplane mode on & off somehow resets the IP address & alleviates the rate limit.

[TobiGr]

The problem is that most people use NewPipe with wifi enabled which makes resetting the IP address more difficult or even impossible.

[veganomy]

If I'm experiencing 3 thread fetching as of now, that means it is still slow then. 4 wouldn't solve it at all. At least 8 (based on CPU thread count) would make sense in that case.

I'm not sure how to cope with rate limits on static ISP. 8 would blast the rate limit response even quicker.