Skip to content

Commit 2eeb0a3

Browse files
FireMasterKAudricV
FireMasterK
and
AudricV
committed
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`: - the desktop API is fetched. If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API. - if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent. Otherwise, the next endpoint will be fetched normally, if the content is available. If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made. We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used. If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used. Other code changes: - `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder` - `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder` - two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder` - `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper` - a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor` - `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>` - the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page - some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
1 parent 7753556 commit 2eeb0a3

10 files changed

Lines changed: 366 additions & 275 deletions

File tree

‎extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeJavaScriptExtractor.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,14 @@ public static String extractJavaScriptCode() throws ParsingException {
6060
return extractJavaScriptCode("d4IGg5dqeO8");
6161
}
6262

63+
/**
64+
* Reset the JavaScript code. It will be fetched again the next time
65+
* {@link #extractJavaScriptCode()} or {@link #extractJavaScriptCode(String)} is called.
66+
*/
67+
public static void resetJavaScriptCode() {
68+
cachedJavaScriptCode = null;
69+
}
70+
6371
private static String extractJavaScriptUrl(final String videoId) throws ParsingException {
6472
try {
6573
final String embedUrl = "https://www.youtube.com/embed/" + videoId;

‎extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java‎

Lines changed: 104 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -66,15 +66,15 @@ private YoutubeParsingHelper() {
6666

6767
public static final String YOUTUBEI_V1_URL = "https://www.youtube.com/youtubei/v1/";
6868

69-
private static final String HARDCODED_CLIENT_VERSION = "2.20210701.00.00";
69+
private static final String HARDCODED_CLIENT_VERSION = "2.20210728.00.00";
7070
private static final String HARDCODED_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8";
7171
private static final String MOBILE_YOUTUBE_KEY = "AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w";
72-
private static final String MOBILE_YOUTUBE_CLIENT_VERSION = "16.25.37";
72+
private static final String MOBILE_YOUTUBE_CLIENT_VERSION = "16.29.38";
7373
private static String clientVersion;
7474
private static String key;
7575

7676
private static final String[] HARDCODED_YOUTUBE_MUSIC_KEY =
77-
{"AIzaSyC9XL3ZjWddXya6X74dJoCTL-WEYFDNX30", "67", "1.20210628.00.00"};
77+
{"AIzaSyC9XL3ZjWddXya6X74dJoCTL-WEYFDNX30", "67", "1.20210726.00.01"};
7878
private static String[] youtubeMusicKey;
7979

8080
private static boolean keyAndVersionExtracted = false;
@@ -309,10 +309,10 @@ public static JsonObject getInitialData(final String html) throws ParsingExcepti
309309
}
310310
}
311311

312-
public static Optional<Boolean> areHardcodedClientVersionAndKeyValid()
312+
public static boolean areHardcodedClientVersionAndKeyValid()
313313
throws IOException, ExtractionException {
314314
if (hardcodedClientVersionAndKeyValid.isPresent()) {
315-
return hardcodedClientVersionAndKeyValid;
315+
return hardcodedClientVersionAndKeyValid.get();
316316
}
317317
// @formatter:off
318318
final byte[] body = JsonWriter.string()
@@ -344,8 +344,9 @@ public static Optional<Boolean> areHardcodedClientVersionAndKeyValid()
344344
final String responseBody = response.responseBody();
345345
final int responseCode = response.responseCode();
346346

347-
return hardcodedClientVersionAndKeyValid = Optional.of(responseBody.length() > 5000
347+
hardcodedClientVersionAndKeyValid = Optional.of(responseBody.length() > 5000
348348
&& responseCode == 200); // Ensure to have a valid response
349+
return hardcodedClientVersionAndKeyValid.get();
349350
}
350351

351352
private static void extractClientVersionAndKey() throws IOException, ExtractionException {
@@ -425,7 +426,7 @@ private static void extractClientVersionAndKey() throws IOException, ExtractionE
425426
*/
426427
public static String getClientVersion() throws IOException, ExtractionException {
427428
if (!isNullOrEmpty(clientVersion)) return clientVersion;
428-
if (areHardcodedClientVersionAndKeyValid().orElse(false)) {
429+
if (areHardcodedClientVersionAndKeyValid()) {
429430
return clientVersion = HARDCODED_CLIENT_VERSION;
430431
}
431432

@@ -438,7 +439,7 @@ public static String getClientVersion() throws IOException, ExtractionException
438439
*/
439440
public static String getKey() throws IOException, ExtractionException {
440441
if (!isNullOrEmpty(key)) return key;
441-
if (areHardcodedClientVersionAndKeyValid().orElse(false)) {
442+
if (areHardcodedClientVersionAndKeyValid()) {
442443
return key = HARDCODED_KEY;
443444
}
444445

@@ -799,10 +800,9 @@ public static JsonArray getJsonResponse(@Nonnull final Page page,
799800
}
800801

801802
@Nonnull
802-
public static JsonBuilder<JsonObject> prepareJsonBuilder(@Nonnull final Localization
803-
localization,
804-
@Nonnull final ContentCountry
805-
contentCountry)
803+
public static JsonBuilder<JsonObject> prepareDesktopJsonBuilder(
804+
@Nonnull final Localization localization,
805+
@Nonnull final ContentCountry contentCountry)
806806
throws IOException, ExtractionException {
807807
// @formatter:off
808808
return JsonObject.builder()
@@ -823,10 +823,9 @@ public static JsonBuilder<JsonObject> prepareJsonBuilder(@Nonnull final Localiza
823823
}
824824

825825
@Nonnull
826-
public static JsonBuilder<JsonObject> prepareMobileJsonBuilder(@Nonnull final Localization
827-
localization,
828-
@Nonnull final ContentCountry
829-
contentCountry) {
826+
public static JsonBuilder<JsonObject> prepareAndroidMobileJsonBuilder(
827+
@Nonnull final Localization localization,
828+
@Nonnull final ContentCountry contentCountry) {
830829
// @formatter:off
831830
return JsonObject.builder()
832831
.object("context")
@@ -845,6 +844,95 @@ public static JsonBuilder<JsonObject> prepareMobileJsonBuilder(@Nonnull final Lo
845844
// @formatter:on
846845
}
847846

847+
@Nonnull
848+
public static JsonBuilder<JsonObject> prepareDesktopEmbedVideoJsonBuilder(
849+
@Nonnull final Localization localization,
850+
@Nonnull final ContentCountry contentCountry,
851+
@Nonnull final String videoId) throws IOException, ExtractionException {
852+
// @formatter:off
853+
return JsonObject.builder()
854+
.object("context")
855+
.object("client")
856+
.value("hl", localization.getLocalizationCode())
857+
.value("gl", contentCountry.getCountryCode())
858+
.value("clientName", "WEB")
859+
.value("clientVersion", getClientVersion())
860+
.value("clientScreen", "EMBED")
861+
.end()
862+
.object("thirdParty")
863+
.value("embedUrl", "https://www.youtube.com/watch?v=" + videoId)
864+
.end()
865+
.object("user")
866+
// TO DO: provide a way to enable restricted mode with:
867+
// .value("enableSafetyMode", boolean)
868+
.value("lockedSafetyMode", false)
869+
.end()
870+
.end()
871+
.value("videoId", videoId);
872+
// @formatter:on
873+
}
874+
875+
@Nonnull
876+
public static JsonBuilder<JsonObject> prepareAndroidMobileEmbedVideoJsonBuilder(
877+
@Nonnull final Localization localization,
878+
@Nonnull final ContentCountry contentCountry,
879+
@Nonnull final String videoId) {
880+
// @formatter:off
881+
return JsonObject.builder()
882+
.object("context")
883+
.object("client")
884+
.value("clientName", "ANDROID")
885+
.value("clientVersion", MOBILE_YOUTUBE_CLIENT_VERSION)
886+
.value("clientScreen", "EMBED")
887+
.value("hl", localization.getLocalizationCode())
888+
.value("gl", contentCountry.getCountryCode())
889+
.end()
890+
.object("thirdParty")
891+
.value("embedUrl", "https://www.youtube.com/watch?v=" + videoId)
892+
.end()
893+
.object("user")
894+
// TO DO: provide a way to enable restricted mode with:
895+
// .value("enableSafetyMode", boolean)
896+
.value("lockedSafetyMode", false)
897+
.end()
898+
.end()
899+
.value("videoId", videoId);
900+
// @formatter:on
901+
}
902+
903+
@Nonnull
904+
public static byte[] createPlayerBodyWithSts(final Localization localization,
905+
final ContentCountry contentCountry,
906+
final String videoId,
907+
final boolean withThirdParty,
908+
@Nullable final String sts)
909+
throws IOException, ExtractionException {
910+
if (withThirdParty) {
911+
// @formatter:off
912+
return JsonWriter.string(prepareDesktopEmbedVideoJsonBuilder(localization, contentCountry, videoId)
913+
.object("playbackContext")
914+
.object("contentPlaybackContext")
915+
.value("signatureTimestamp", sts)
916+
.end()
917+
.end()
918+
.done())
919+
.getBytes(UTF_8);
920+
// @formatter:on
921+
} else {
922+
// @formatter:off
923+
return JsonWriter.string(prepareDesktopJsonBuilder(localization, contentCountry)
924+
.value("videoId", videoId)
925+
.object("playbackContext")
926+
.object("contentPlaybackContext")
927+
.value("signatureTimestamp", sts)
928+
.end()
929+
.end()
930+
.done())
931+
.getBytes(UTF_8);
932+
// @formatter:on
933+
}
934+
}
935+
848936
/**
849937
* Add required headers and cookies to an existing headers Map.
850938
* @see #addClientInfoHeaders(Map)

‎extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeChannelExtractor.java‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -88,8 +88,8 @@ public void onFetchPage(@Nonnull final Downloader downloader) throws IOException
8888
// navigation/resolve_url endpoint of the InnerTube API to get the channel id. Otherwise,
8989
// we couldn't get information about the channel associated with this URL, if there is one.
9090
if (!channelId[0].equals("channel")) {
91-
final byte[] body = JsonWriter.string(prepareJsonBuilder(getExtractorLocalization(),
92-
getExtractorContentCountry())
91+
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
92+
getExtractorLocalization(), getExtractorContentCountry())
9393
.value("url", "https://www.youtube.com/" + channelPath)
9494
.done())
9595
.getBytes(UTF_8);
@@ -135,8 +135,8 @@ public void onFetchPage(@Nonnull final Downloader downloader) throws IOException
135135

136136
int level = 0;
137137
while (level < 3) {
138-
final byte[] body = JsonWriter.string(prepareJsonBuilder(getExtractorLocalization(),
139-
getExtractorContentCountry())
138+
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
139+
getExtractorLocalization(), getExtractorContentCountry())
140140
.value("browseId", id)
141141
.value("params", "EgZ2aWRlb3M%3D") // Equal to videos
142142
.done())
@@ -384,7 +384,7 @@ private Page getNextPageFrom(final JsonObject continuations,
384384
final String continuation = continuationEndpoint.getObject("continuationCommand")
385385
.getString("token");
386386

387-
final byte[] body = JsonWriter.string(prepareJsonBuilder(getExtractorLocalization(),
387+
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(getExtractorLocalization(),
388388
getExtractorContentCountry())
389389
.value("continuation", continuation)
390390
.done())

‎extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ public void onFetchPage(@Nonnull final Downloader downloader)
6262
final String videoId = getQueryValue(url, "v");
6363
final String playlistIndexString = getQueryValue(url, "index");
6464

65-
final JsonBuilder<JsonObject> jsonBody = prepareJsonBuilder(localization,
65+
final JsonBuilder<JsonObject> jsonBody = prepareDesktopJsonBuilder(localization,
6666
getExtractorContentCountry()).value("playlistId", mixPlaylistId);
6767
if (videoId != null) {
6868
jsonBody.value("videoId", videoId);
@@ -174,7 +174,7 @@ private Page getNextPageFrom(final JsonObject playlistJson,
174174
final String videoId = watchEndpoint.getString("videoId");
175175
final int index = watchEndpoint.getInt("index");
176176
final String params = watchEndpoint.getString("params");
177-
final byte[] body = JsonWriter.string(prepareJsonBuilder(getExtractorLocalization(),
177+
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(getExtractorLocalization(),
178178
getExtractorContentCountry())
179179
.value("videoId", videoId)
180180
.value("playlistId", playlistId)

‎extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubePlaylistExtractor.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ public YoutubePlaylistExtractor(StreamingService service, ListLinkHandler linkHa
4444
public void onFetchPage(@Nonnull final Downloader downloader) throws IOException,
4545
ExtractionException {
4646
final Localization localization = getExtractorLocalization();
47-
final byte[] body = JsonWriter.string(prepareJsonBuilder(localization,
47+
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(localization,
4848
getExtractorContentCountry())
4949
.value("browseId", "VL" + getId())
5050
.value("params", "wgYCCAA%3D") // Show unavailable videos
@@ -251,8 +251,8 @@ private Page getNextPageFrom(final JsonArray contents) throws IOException,
251251
.getObject("continuationCommand")
252252
.getString("token");
253253

254-
final byte[] body = JsonWriter.string(prepareJsonBuilder(getExtractorLocalization(),
255-
getExtractorContentCountry())
254+
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
255+
getExtractorLocalization(), getExtractorContentCountry())
256256
.value("continuation", continuation)
257257
.done())
258258
.getBytes(UTF_8);

‎extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSearchExtractor.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ public void onFetchPage(@Nonnull final Downloader downloader) throws IOException
7070
params = "";
7171
}
7272

73-
final JsonBuilder<JsonObject> jsonBody = prepareJsonBuilder(localization,
73+
final JsonBuilder<JsonObject> jsonBody = prepareDesktopJsonBuilder(localization,
7474
getExtractorContentCountry())
7575
.value("query", query);
7676
if (!isNullOrEmpty(params)) {
@@ -166,7 +166,7 @@ public InfoItemsPage<InfoItem> getPage(final Page page) throws IOException,
166166
final InfoItemsSearchCollector collector = new InfoItemsSearchCollector(getServiceId());
167167

168168
// @formatter:off
169-
final byte[] json = JsonWriter.string(prepareJsonBuilder(localization,
169+
final byte[] json = JsonWriter.string(prepareDesktopJsonBuilder(localization,
170170
getExtractorContentCountry())
171171
.value("continuation", page.getId())
172172
.done())

0 commit comments

Comments
 (0)