                              YTCrack v0.50b
                              ==============
              YTCRACK.TXT - Copyright (c) 2019, Fred C. Macall
   

Overview

YTCrack is a DOS program for extracting "videoplayback" URL(s) from freshly
saved copies of YouTube Web pages.  That is, Web pages with URLs like:
http://www.youtube.com/watch?v=VideoIDCode

Beginning with version 0.11b, YTCrack also works with freshly saved copies
of video.google.com Web pages.  That is, Web pages with URLs like:
http://video.google.com/videoplay?docid=-1234567890123456789

Beginning with little published version 0.30b, YTCONFIG.EXE joined the
YTCrack package.  Together, YTConfig and YTCrack can work with YouTube Web
pages carrying invalid or scrambled signatures!  The ones that YTCrack, by
itself, wasn't able to handle during much of 2013 and 2014.  This document
covers YTCrack's side of its interface(s) with YTConfig.  Another document,
YTCONFIG.TXT, covers YTConfig's side of these interfaces.

Since some time before 2018, YTCrack also has worked with YouTube Web pages
offered via URLs like:
https://www.youtube.com/get_video_info?video_id=VideoIDCode&sts=stsnr

These Web pages are alternatives, to the watch?v= style pages given above,
which sometimes carry additional video renditions not available via the
corresponding watch?v= style URLs.  A work around was needed, with the
get_video_info? style pages, for videos burdened with invalid or scrambled
signatures.  Beginning with YTCrack version 0.49b, a new arrangement
provides support for a special FLSTSKEY= Environment value.  That gives
YTCrack a natural way to get the descrambling key that &sts=stsnr
specifies.  More about this is given in the
Installation and Operational Considerations
section of this document.

YTCrack produces its result(s) in a relatively small .HTM document or file.
This document exposes anchor(s) or link(s) to the YouTube or video.google
videoplayback URL(s) that YTCrack has extracted from a given YouTube or
video.google Web page copy.  This YTCrack result document may be used by a
Web browser, without scripts nor "Flash" support, for downloading the
YouTube or video.google video(s) its anchor(s) identify.  Beginning with
version 0.29b, the YTCrack results document also carries an anchor or link
to YouTube's latest player or base.js JavaScript file.

This file carries the signature descrambling algorithm or key information
that YTConfig extracts, for YTCrack's use.  Apparently, this file gets
updated as often as daily.  Each version of the file has a URL that
contains a distinctive <version> part.  YTCrack and YTConfig treat the
last eight characters of the URL's <version> part as the file's version
identification.  More about these things below.  Our understanding of the
player or base.js file and get_video_info? style Web pages has developed
starting from Jamie Zawinski's youtubedown perl script.  It is available
from:
http://www.jwz.org/hacks/youtubedown

The YouTube and video.google videos we've seen, so far, are .3GP, .FLV,
.MP4, and .WEB types.  (We are using .3GP for .3gpp, a video container used
in cell 'phones.  And, .WEB for .WebM, a relatively new video container
type.  We are no longer seeing any .FLV type video(s).)  Some of these
videos are talkies containing both audio and video streams.  However, since
about 1 April 2013, we have seen some .MP4 and .WEB files containing an
audio or video stream, only (AO or VO).  Some files now contain a new or
experimental audio or video stream, only (AOX or VOX).  In particular, some
.MP4 files now contain a very new or experimental audio (6 channel) or
video (AV1) stream, only (AOX or VOX).  Also, for some time now, some of
the .MP4 and .WEB offerings have appeared to contain
Side-by-Side SteroScopic 3D (3D) videos.

The video.google Web pages we've seen all carry a single videoplayback URL.
However, the YouTube Web pages we've seen, so far, all carry one, two, six,
or eight copies of each of three to thirty unique videoplayback URLs.
That is, a total of three to sixty videoplayback URLs!  (Before March 2011,
we always saw eight copies of each of three to six unique videoplayback
URLs.  For several years after, we saw only one copy of each of about three
to thirty videoplayback URLs.  Since about 14 October 2018, we've been
seeing two similar, if not identical, copies of what appear to be the same
three to thirty videoplayback URLs, again.)  So, after it has identified
them, YTCrack weeds out any exact duplicates -- to save you this effort.
The unique videoplayback URLs remaining are distinguished and identified by
their itag=nnn fields.  These itag= values appear to identify each video's
resolution and, perhaps, other characteristics.  The itag= values we've
seen so far are:

itag=  video  resolution/bitrate
value  type    ( w x h )  flags
=====  =====  ==================
   5    FLV    320 x 240
  13    3GP    176 x 144
  17    3GP    176 x 144
  18    MP4    480 x 360
  22    MP4   1280 x 720
  34    FLV    480 x 360
  35    FLV    640 x 480 
  36    3GP    320 x 240
  37    MP4   1920 x 1080
  38    MP4   2048 x 1080
  43    WEB    480 x 360
  44    WEB    640 x 480
  45    WEB   1280 x 720
  46    WEB   1920 x 1080
  59    MP4    640 x 480
  78    MP4    640 x 480
  82    MP4    480 x 360   3D
  83    MP4    640 x 480   3D
  84    MP4   1280 x 720   3D
  85    MP4   1920 x 1080  3D
 100    WEB    480 x 360   3D
 101    WEB    640 x 480   3D
 102    WEB   1280 x 720   3D
 133    MP4    320 x 240   VO
 134    MP4    480 x 360   VO
 135    MP4    640 x 480   VO
 136    MP4   1280 x 720   VO
 137    MP4   1920 x 1080  VO
 139    MP4   Low bitrate  AO
 140    MP4   Med bitrate  AO
 141    MP4   Hi  bitrate  AO
 160    MP4    256 x 144   VO
 171    WEB   Med bitrate  AO
 172    WEB   Hi  bitrate  AO
 242    WEB    320 x 240   VOX
 243    WEB    480 x 360   VOX
 244    WEB    640 x 480   VOX
 245    WEB    640 x 480   VOX
 246    WEB    640 x 480   VOX
 247    WEB   1280 x 720   VOX
 248    WEB   1920 x 1080  VOX
 249    WEB   Low bitrate  AOX
 250    WEB   Med bitrate  AOX
 251    WEB   Hi  bitrate  AOX
 256    MP4   Low bitrate  AO
 258    MP4   Hi  bitrate  AO
 264    MP4   2560 x 1440  VO
 266    MP4   3840 x 2160  VO
 271    WEB   2560 x 1440  VOX
 272    WEB   3840 x 2160  VOX
 278    WEB    256 x 144   VOX
 298    MP4   1280 x 720   VO
 299    MP4   1920 x 1080  VO
 302    WEB   1280 x 720   VOX
 303    WEB   1920 x 1080  VOX
 308    WEB   2560 x 1440  VOX
 313    WEB   3840 x 2160  VOX
 315    WEB   3840 x 2160  VOX
 325    MP4   Hi  bitrate  AOX
 394    MP4    256 x 144   VOX
 395    MP4    320 x 240   VOX
 396    MP4    480 x 360   VOX
 397    MP4    640 x 480   VOX
 398    MP4   1280 x 720   VOX

Notes for table:  Each itag= value always identifies the video type
indicated above.  The itag= values usually, but not quite always, identify
the exact video pixels heights indicated.  More often, they don't exactly
identify the pixels widths indicated.  For example:  itag=35 always
identifies an .FLV type video.  That video is quite likely to be 480 pixels
high.  Its width may be in the range of 640 to 960 pixels.  The entries
without flags are for plain talkies.  See the discussion several paragraphs
above for an introduction to the 3D, AO, VO, AOX, and VOX flags.
itag= values 244, 245, and 246 appear to be for low, medium, and high
bitrate video, respectively.  itag= values 298, 299, 302, 303, 308, and 315
might appear to duplicate itag= values 136, 137, 247, 248, 271, and 313.
Actually, the videos in that first or higher numbered set tend to have a
frame rate of 50 or 60 FPS.  While, the videos in that second or lower
numbered set tend to have a "regular" frame rate of 25 or 30 FPS.
itag= values 13 and 17 videos appear to differ mostly in their audio
sampling rates.  Of the two, itag=13 videos have the lower rate.
itag= values 59 and 78 videos appear to differ mostly in their video bit
rates.  Of the two, itag=59 videos have the higher rate.
itag= values 256 and 258 appear to be for "surround sound" or "5.1" audio.
itag= value 325 appears to be for 6 channel audio. 
itag= values 394, 395, 396, 397, and 398 might appear to duplicate itag=
values 160, 133, 134, 135, and 136.  Actually, the videos in that first or
higher numbered set specify use of the new or experimental "open" "AV1"
codec.  While, the videos in the second or lower numbered set specify the
established "AVC" codec.

A video you want may only be offered, in the resolution(s) you want, as an
AO and VO files pair.  In that case, you may first want to make a talkie
from the pair by means of, say, DOS FFMPEG.  As explained below.
Then, given suitable sound and video adapters and a fast enough processor,
all but the newest of the offered videos (all but the AOXs and VOXs) may be
played with the latest DOS MPLAYER version.  So, your whole video receiving
and watching process can be accomplished in DOS without scripts enabled and
without Flash support.  And, it yields downloaded video file(s) that don't
have to be discovered in and retrieved from a cache directory.

YTCrack uses no configuration or helper file or program, other then
YTConfig, and is intended to run on 8086/8088 based, and all later, IBM PC
compatible PCs.  It may need 300 KB to 400 KB of DOS memory, depending on
the size of the given YouTube page.  But, it requires only DOS v3.x, or
later.

In much of the rest of this document, we'll describe YTCrack's handling of
YouTube and video.google Web pages and videoplayback URLs in common.
In the few places where that isn't appropriate, we'll give details for both
types.


History

YTCrack versions 0.12b, 0.14b, and 0.15b were issued to add support for
itag= 36, 38, 4x, 8x, and 10x video properties.  We started seeing these
properties at various times from early in 2011 to the middle of 2012.

On or about 13 September 2012, YouTube made some major changes in the
videoplayback URLs offered in their /watch?v=VideoIDCode Web pages.
In particular, the URLs' established &signature= . . . field seemed to be
replaced by a new &sig= . . . field.  Also, several other new fields were
added.  And, the URLs' termination point appeared to move from the end of
the &id= . . . field to the end of the apparently new &sig= . . . field.
At the same time, YouTube started rejecting the URLs developed from these
new Web pages by YTCrack v0.15b and all previous YTCrack versions!

After a lot of effort, Glenn McCorkle discovered that &sig= , apparently,
is only a shortening or obfuscation of the old &signature= . . . field's
name.  YTCrack version 0.16b became able to develop acceptable YouTube
URLs, again, by changing &sig= to &signature= , in the URLs it extracts
from YouTube /watch?v=VideoIDCode Web pages!

In the interests of avoiding apparently unnecessary complication(s),
YTCrack v0.16b also removes the then new &type= . . . and &quality= . . .
fields from the URLs it develops.  Further, it provides support for the
itag=17 video properties we started finding in YouTube's Web pages at about
that time.

On or about 18 December 2012, YouTube made some more big changes in some of
their /watch?v=VideoIDCode Web pages.  These changes seem to be presented
in a semi-random fashion.  So that YTCrack v0.16b still can extract usable
URLs from some of the pages.  But, not from others.  If our September 2012
experience is any guide, YTCrack v0.16b may be expected to become
completely useless within a few months.

None of the December 2012 changes can be understood as being anything but
obfuscatory!  First, videoplayback URL fields are being relocated
semi-randomly.  For what purpose other than obfuscation?  Almost every
field is subject to appearing at the beginning of the URLs' parameter
fields area.  Where it will start with question mark.  So that no field may
be assumed to start with ampersand, consistently.

Next, we are seeing /watch?v=VideoIDCode Web pages bearing URLs with
apparently invalid &sig= . . . fields!  Upon deeper inspection, we find
that the &sig= . . . fields in these pages have been shuffled among the
URLs.  For what purpose other than obfuscation?  From the first URL to the
last, each &sig= . . . field has been shuffled to the URL just ahead.
So that the last URL has no &sig= . . . field.  And, what was the first
URL's &sig= . . . field now has to be found in a
url_encoded_fmt_stream_map . . . field located just ahead of the first URL!
We are using the term "URL signature fragment" to refer to these fields.

Also, we are finding duplicated itag= . . . fields in the URLs on some of
the Web pages.  For what purpose other than obfuscation?  URLs containing
these won't be accepted by YouTube until one of the duplicate itag= . . .
fields has been removed.

YTCrack version 0.21b contains accommodations and/or new provisions for
dealing with each of these new obfuscations.  So that it was working with
all of the YouTube Web pages encountered early in 2013.  Also, it provides
support for the itag=85 video properties we found in a few of YouTube's Web
pages around that time.

On or about 1 April 2013, YouTube brought us at least nine new itag= . . .
values and some more videoplayback URL structure changes.  Quite the joker.
All of the new itag= . . . values seem to identify .MP4 videos carrying
only a single stream, which may be either audio or video.
Apparently, YouTube further "fuzzed" its videoplayback URL structure by
allowing more or all of the 23 sometimes required URL fields to appear at
the very end of URL(s).  (The 23 sometimes required URL fields we've
identified don't include &fallback_host= . . . , &quality= . . . ,
nor &type= . . . .)

When present, the initial URL signature fragment, which always used to
precede all videoplayback URLs, now may be found intermingled with the
videoplayback URLs.  Apparently, reconstituting signature(s) mode doesn't
apply for any videoplayback URL(s) preceding a URL signature fragment.

YTCrack version 0.24b supports the nine new itag= . . . values and includes
new provisions for dealing with those URL structure changes.

As of June 2013, we've seen another four sometimes required videoplayback
URL fields, for a total of 27.  One of these is an &s= . . . field, which
appears to be another obfuscated &signature= . . . field.  It then appeared
that &s= . . . fields always were used in the URLs for "vevo" videos.
These still appear only randomly and infrequently in the URLs for other
videos.  The other three new fields are clen= . . . , gir= . . . , and
lmt= . . . .  We've also seen another nine new itag= . . . values, for a
total of 38!  These appear to be .WEB type videos again carrying only a
single stream, which may be either audio or video.  The newest itag= values
are 17x for audio and 24x for video.  The new .WEB type video only videos
employ what appears to be a very new video format or codec that YouTube
seems to be cultivating or developing!

Taken together, these latest developments and something we've read seem to
suggest a future obfuscation path for YouTube:  They might eventually
require a player that can combine an audio only (AO) file and a video only
(VO) file to make a talkie!  Of course, early in that game, only YouTube's
player would be likely to be very good at doing that!

That was our first thought.  However, it has developed that makes of
ffmpeg, for both linux and DOS, can combine YouTube .MP4 AO and VO file
pairs to make fine talkies.  The main difference between those two makes is
that the one for linux is amazingly fast.  While, by comparison, the one
for DOS is a competent plodder.  On a PC that is more than adequate for
playing thirty frames per second talkies, the DOS make of ffmpeg we've
found does this processing at only about fifteen frames per second, or so.

As of July 2014, we have seen some of this play out.  In particular, the
640 x 480 .FLV talkies, that used to be offered for many YouTube videos,
are no longer available.  One's recourse is to get a 640 x 480 .MP4 VO file
and an .MP4 AO file and combine the two.  More over, for many videos
offered at a top resolution of 640 x 480, there no longer is a talkie
rendition available at that top resolution!

YTCrack version 0.25b supports the latest nine new itag= . . . values.
The four new videoplayback URL field names.  And, converts &s= . . . fields
to &signature= . . . fields.

However, it has become clear that YouTube is bringing new obfuscations to
its &s= . . . style signature fields.  The first sign of this is that these
signatures often deviate from the <forty hex digits>.<forty hex digits>
(or, 40.40) signature value format that the &sig= . . . and
&signature= . . . style signatures consistently use!  For example, one
&s= . . . style signature we've seen had a 79.5.6 value format.
It seems pretty clear that signature values containing more than eighty hex
digits have extra digit(s) that will have to be removed.  But, which ones?
This isn't obvious when the extra(s) are given apparently random values.

With help from the youtube.py file in the youtube-dl package, we have
learned that the &s= . . . style signatures also are being scrambled!
That is, after the extras have been removed, the remaining eighty hex
digits still may have to be reordered to make an acceptable signature.
The youtube-dl package is updated as often as daily!  We suppose these
updates are, at least in part, for adding new signature descrambling
case(s) to youtube.py .  The youtube-dl package is available from:
http://rg3.github.io/youtube-dl/

YTCrack version 0.26b brings some corrections for signature tail handling.

YTCrack version 0.27b brought &s= . . . style signature value format
checking.  It reports when signature value format(s) other than 40.40 are
found.  And, when 40.x , where 40 < x <= 52 , format(s) are seen, it
attempts a fix by removing the last x - 40 hex digits.  This fix succeeded
on the day that YTCrack v0.27b was compiled.  However, we haven't seen it
succeed since then.

As of December 2013, we've discovered that the signatures shuffling scheme
introduced above actually is more complicated than we first thought.
The url_encoded_fmt_stream_map field introduced above is found ahead of a
group of traditional URLs for talkies.  While, another new field,
adaptive_fmts , is found ahead of a group of the newer videoplayback URLs,
for AO and VO files.  The adaptive_fmts field may or may not contain a
signature field.

YTCrack version 0.28b recognizes adaptive_fmts , as well as
url_encoded_fmt_stream_map .  When an adaptive_fmts field conveys a
signature field, that signature is taken as if introduced by a
url_encoded_fmt_stream_map field.  When an adaptive_fmts field doesn't
contain a signature field, that is taken as a signal to discontinue
reconstituting signature(s), in the subsequent URL(s).
YTCrack v0.28b also provides information about the occasional
itag=264 MP4 VO videos we've seen.  This brings the entries in our list
of itag= values to a total of 39.  Finally, it removes the apparently
unnecessary fexp= . . . field found exorbitantly lengthening many of
YouTube's videoplayback URLs.  And, it no longer includes fexp= in its
list of sometimes required videoplayback URL fields.  So, that list is
reduced to 26 entries.

Upon learning Jamie Zawinski's insights about &s= . . . style signature
descrambling, we saw that a new approach was required for this, in YTCrack.
Since the needed descrambling key appears to change no more often than
daily, we decided to place the process for obtaining it in a separate
program, YTCONFIG.EXE .

Beginning with version 0.33b, YTCrack communicates with YTConfig and its
user as follows:  While searching a YouTube Web page for videoplayback
URLs, YTCrack also looks for a URL for YouTube's player or base.js
JavaScript file.  Finding this URL enables two results:

First, an anchor or link to the file gets placed at the bottom of YTCrack's
output report file.  This keeps YTCrack's user informed of the present
player or base.js file's version.  However, it remains for the user to
decide if the latest version needs to be received, or not.

Second, if and when YTCrack has reported:
nn invalid signature(s) found.
it can probe the DOS Environment for a
<version>=<descramblingkey>
string already provided by a YTConfig run.  Finding an Environment variable
name matching the present <version> enables YTCrack to descramble the
invalid signatures and report:
nn of these corrected using version based rule.
Otherwise, it reports:
0 of these corrected using version based rule.

In the latter event, the user may choose to obtain the latest player or
base.js file and run YTConfig and its output .BAT file.  Those actions will
leave the needed
<version>=<descramblingkey>
string in the DOS Environment.  So that a YTCrack rerun for the same
YouTube Web page copy will, then, succeed.

Three other kinds of changes, for YTCrack version 0.33b, were introduced
in the little published YTCrack versions 0.29b and 0.30b:  dur= , mws= ,
initcwndbps= , keepalive= , and mime= were added to YTCrack's list of
sometimes required videoplayback URL fields.  Bringing this list to a total
of 31 entries.

&id= . . . field processing was revised to accept values that are no longer
limited to the hex digits.

Duplicated clen= . . . and lmt= . . . fields now get recognized and
removed, like duplicated itag= . . . fields.

In July 2014, YouTube's URLs for html5player.js changed, with respect to
where the file's <version> string appears.  Beginning with version 0.33b,
YTCrack and YTConfig have been extended to find the <version> string in
either of its former or present locations.

As of November 2014, we've seen four more sometimes required videoplayback
URL fields, for a total of 35.  The newest fields are mm=, pbr=, pfa=, and
pm_type=.  We've also seen three more itag= values, for a total of 42.
The newest values are itag=266, for an .MP4 3840 x 2160 VO file.
And, itag=271 and itag=272, for .WEB extra high definition VOX files.
With the possible exception of itag=38, we have yet to see any itag=
value(s) identifying extra high definition talkie file(s)!

YTCrack version 0.34b incorporates those new URL fields and itag= values.
It also includes a fix for a mistake, in a recent release, that effectively
dropped id= from the list of sometimes required videoplayback URL fields.
The version 0.34b release also includes an adjustment for an apparent
redefinition of the itag=264 value.  It now seems to designate an
.MP4 2560 x 1440 VO file.  Finally, this release includes an improvement
for videoplayback URL trimming.  After a tentative URL's final field has
been identified, both & and ",<space> , whichever comes first, are sought
for locating the URL's end point, now.

YTCrack version 0.35b includes a fix for finding &type= . . . fields to be
removed from videoplayback URLs.  Lately, these fields have contained a
quoted codec name.  So, it is important that they do get removed.
Now, such &type= . . . fields will be found even when they follow the likes
of &pm_type= . . . and &projection_type= . . . videoplayback URL field(s).

As of February 2015, we've seen four more sometimes required videoplayback
URL fields, for a total of 39.  The newest fields are cwbf=, nh=, pcm2=,
and pl=.  We've also seen six more itag= values, for a total of 48.
The newest itag= values start off with 249, 250, and 251, which are for
.WEB type AOX files.  These are the first AOX files we've seen.
The newest itag= values also include 278, 298, and 302.

We've also seen the separator sequence following url_encoded_fmt_stream_map
and adaptive_fmts change from ":<space>" (four characters) to ":" (three 
characters).  This change kept YTCrack v0.35b from recognizing URL
signature fragments, after the beginning of 2015.

YTCrack version 0.36b incorporates those new URL fields and itag= values.
And, it recognizes url_encoded_fmt_stream_map and adaptive_fmts fields when
they include any of the separator sequences:  = , ": " , or ":" .
 
Since version 0.33b, YTCrack has been checking &s= . . . style signature
values for a valid 40.40 format, as explained above.  Signatures with an
invalid format and a total of 81 or more characters have been descrambled
when the appropriate descrambling key is on hand.  This arrangement proved
to have a big limitation.  When the descrambling key in use contains no snn
field and no w40 field, signature fields can't change from an invalid
format to a valid format during descrambling.  So, they must have a valid
format before descrambling, if they are to have a valid format afterward.
In this case, YTCrack hasn't been able to detect an otherwise apparent need
for descrambling!

A review of the &s= . . . style signatures we've seen, since the beginning
of 2014, concludes that all &s= . . . style signatures now appear in need
of descrambling.  YTCrack has been changed accordingly in little published
version 0.37b and, now, in version 0.38b .  A new message and a changed
message report on this analysis.  The new message is:
nn s= . . . style signature(s) found.
It indicates finding signature(s) now thought to need descrambling.
Such signatures with a total of 81 or more characters will be descrambled
when the appropriate descrambling key is on hand.

The changed message is:
nn of these found to have an invalid format.
This reports what the
nn invalid signature(s) found.
message reported.

Of course,
nn of these corrected using version based rule.
will continue to indicate how many signature(s) have actually been
descrambled, with apparent success.

YTCrack version 0.38b also includes descriptive data for itag=13 videos.
For a total of 49 documented itag= values.  And, three more sometimes
required videoplayback URL field names.  These are hightc= , mn= , and
requiressl= .  For a total of 42 of these.

On or about 9 November 2015, YTCrack versions 0.38b and before quit
recognizing and displaying a URL for YouTube's video player, at the bottom
of their report.  That turned out to be caused by a change of the player's
name, from html5player to player or base.  Even if YTConfig had continued
working, YTCrack wouldn't have been able to use its result without having
found the applicable version string.  In any event, at the same time,
YTConfig versions 0.38b and before proved incapable of obtaining a
descrambling key from the new player files.  Also, around this time, we saw
a new itag=313 video rendition.

YTCrack version 0.40b looks for player, instead of html5player, and takes
new care to avoid being confused by other URLs also containing player !
Also, it includes descriptive data for itag=313 videos.  That makes a total
of 50 itag= values we've seen since late in 2010.  Updates for YTConfig are
discussed in YTCONFIG.TXT .

On or about 23 November 2015, another YouTube change in their video player
put YTConfig in need of immediate attention, again.  As explained in
YTCONFIG.TXT .  That provided a release opportunity for adding the recently
seen xtags= videoplayback URL field to YTCrack's list of sometimes required
fields.  That brings YTCrack version 0.41b's list to 43 entries.

On or about 18 December 2015, yet another YouTube change in their video
player put YTConfig in need of immediate attention, yet again.
As explained in YTCONFIG.TXT .  That provided a release opportunity for
incorporating descriptive data for recently seen itag=59 and itag=78
videos, in YTCrack.  That brings YTCrack version 0.42b's list of itag=
values to a total of 52, which we've seen since late in 2010.

The version and date labeling in YTCrack version 0.43b has been updated to
match the accompanying YTConfig version.

On or about 15 April 2016, the xtags= . . . videoplayback URL field, which
we first recognized during the fourth quarter of 2015, seemed to take on a
new role.  It was found to have become unhelpful for finding the end of
videoplayback URL(s).  And, an xtags= field with no value was found to be
poison.  videoplayback URL(s) containing such a field were found to be
unacceptable until it was removed!

xtags= has been removed from YTCrack's list of sometimes required fields,
in version 0.44b.  That returns the updated version's list to 42 entries.
Also, the updated version has been extended to seek and remove an
xtags= . . . field in each videoplayback URL it processes.

On or about 31 January 2017, the HostName part of the URLs for YouTube's
player or base.js files went missing.  Supplying either s.ytimg.com
(the last HostName seen in a complete URL) or www.youtube.com (the HTML
default HostName in a document obtained from
http://www.youtube.com/watch?v= . . . ) for it appears to obtain a true
copy of the desired file.  However, part of that file's JavaScript code,
for descrambling &s= . . . style signatures, also has changed in a way that
keeps YTConfig v0.44b and before from working.  Also, three more sometimes
required videoplayback URL field names and two more itag= values have been
sighted in the last nine months.

YTCrack version 0.45b includes an additional search for a URL of the form:
http<s>://<HostName>/watch?v= . . .
It uses the <HostName> from the first such URL found to complete the player
or base.js URL it has found, if/when necessary.  aol= , ei= , and id= have
been added to its list of fields sometimes required in videoplayback URLs.
Bringing that list to 45 entries.  And, descriptive information has been
added for itag=299 and itag=303 type videos.  For a total of 54 of those.

YTConfig version 0.45b has been updated as explained in YTCONFIG.TXT.

The version and date labeling in YTCrack version 0.46b has been updated to
match the accompanying YTConfig version.

On or about 28 April 2017, YouTube changed the structure of the path in the
URL for their player or base.js file.  They moved the URL's language field
(for example:  en_US) into its own directory in the path.  As a result,
YTCrack and (sometimes) YTConfig, version 0.46b, started reporting a player
or base.js file "version" different from what they had been reporting.
This change could be worked-around with some difficulty.

Both YTCrack and YTConfig, version 0.47b, have been tweaked so as to report
the player or base.js file's version as they had, before.

The version and date labeling in YTCrack version 0.48b have been updated to
match the accompanying YTConfig version.

The need to update YTConfig, that came on or about 6 November 2018, brought
a release opportunity for fixing four YTCrack issues that had accumulated.
First of all, YTCrack version 0.49b contains a fix for its handling of
sequences of repeated back slash (\) characters.  Next, aitags= has been
added to its list of sometimes required videoplayback URL fields.
That brings this list to 46 entries.  Also, descriptive information has been
added for recently sighted itag= value 256, 258, 325, 394, 395, 396, 397,
and 398 videos.  That brings the number of recognized itag= values to 62.

Finally, we had recognized a problem with the get_video_info? style URLs
given near the beginning of the Overview section.  The documents they
obtain don't identify the s= . . . style signature descrambling key that
may be needed.  However, there is no need to guess about it.
Because, the sts=stsnr field I've indicated as part of these URLs conveys
the player or base.js version to be used!  The fix for this issue,
in YTCrack version 0.49b, is to recognize a special FLSTSKEY= Environment
value when no player or base.js version has been identified.  More about
this is given in the
Installation and Operational Considerations
section of this document.

The need to update YTConfig, that came on or about 15 January 2019, brought
an opportunity to add descriptive information, for recently sighted itag=
values 308 and 315, to YTCrack version 0.50b.  With these additions,
YTCrack's list of recognized itag= values comes to 64 entries.


How It Works

YTCrack starts off by checking for a pair of command line parameters that
do not match each other.  Finding anything else, it displays its Usage
message and terminates.  For the record, this message reads:

Usage:  YTCRACK <input file name> <output file name>

Notes:  <input file name> specifies a fresh source copy of an
        http://www.youtube.com/watch?v=<videocode> Web page.
        And, <output file name> specifies an HTML page to be produced.
        The two file names must be different.

YTCrack then opens both of its given files.  And, reads through its input
file looking for URLs of the form:

http<wildtxt1>videoplayback<wildtxt2><terminator>

Where:  <wildtxt1> is any text that doesn't include space character(s),
        http, nor videoplayback .
        <wildtxt2> is any text that doesn't include http .
        <terminator> is http or the end-of-file indication.
 
Beginning with YTCrack version 0.21b, this section also looks for fields
of the form:

url_encoded_fmt_stream_map<wildtxt3><s><wildtxt4>http<wildtxt5>videoplayback

Where:  <wildtxt3> is any text that doesn't include <s>, http, nor
        videoplayback .
        <s> is sig or, beginning with YTCrack version 0.25b, s= .
        s= is sought only when sig can not be found.
        <wildtxt4> is any text that doesn't include http nor videoplayback .
        <wildtxt5> is any text that doesn't include videoplayback .

We use the term "URL signature fragment(s)" to refer to these fields.
Their value immediately follows an equal character, a four character ": "
sequence, or a three character ":" separator sequence immediately following
the url_encoded_fmt_stream_map string introduced above.  The http string in
the above definition locates their initial termination point.  Note that
the <wildtxt3> portion of these fields may be found to contain URL
field(s), in addition to the following sig= . . . or s= . . . field.
When a URL signature fragment is found, YTCrack enters a
"reconstituting signature(s)" mode.  This mode changes or extends some
YTCrack behaviors, as described below.  However, since YTCrack
version 0.24b, reconstituting signature(s) mode does not apply to any
videoplayback URL(s) preceding the URL signature fragment found.

Beginning with YTCrack version 0.28b, this section also looks for fields
of the form:

adaptive_fmts<wildtxt6>http<wildtxt7>videoplayback

Where:  <wildtxt6> is any text that doesn't include http
        nor videoplayback .
        <wildtxt7> is any text that doesn't include videoplayback .

These fields are sometimes seen to include a signature field, in the
<wildtxt6> area, as the url_encoded_fmt_stream_map fields do.
When they do, we treat them as URL signature fragment(s), as explained
above.  If not, we simply take the presence of one of these fields as an
indication that reconstituting signature(s) mode should be discontinued,
for the subsequent videoplayback URL(s).

Beginning with YTCrack version 0.45b, this section also looks for fields
of the form:

http<s>://<wildtxt8>/watch?v=

Where:  <s> is s or nul.
        <wildtxt8> is any text that doesn't include / , http , nor
        /watch?v= .

<wildtxt8> is taken to be a HostName which may be needed to complete the
player or base.js URL described below.  Once this partial URL has been
found, YTCrack quits looking for any more of these.

Beginning with YTCrack version 0.29b, this section also looks for the field
described immediately below.  This description reflects changes made for
the field in YTCrack version 0.40b:

http<wildtxt9>player<wildtxt10>.js<wildtxt11>http

Where:  <wildtxt9>  is any text that doesn't include http .
        <wildtxt10> is any text that doesn't include player , .js ,
        nor http .
        <wildtxt11> is any text that doesn't include http.

.js is taken as the exact end of this field.  <wildtxt9> is searched, from
right to left, for a quote (") character.  If a quote is found, the
character immediately to its right is taken as the exact beginning of this
field.  If no quote is found, the first http is taken as the exact
beginning of the field.  YTCrack then removes any back slash character(s)
present and prepends http: , if necessary, to make a complete URL.
This URL is provided in an anchor or link at the bottom of YTCrack's output
file report.

If <wildtext10> includes slash(es), the eight characters just before the
rightmost slash are taken as the player's version.  Otherwise, the eight
characters just before .js are taken as the player's version.

Once this URL has been found, YTCrack quits looking for any more of these.

After having scanned the whole input file for the fields described above,
YTCrack checks to see if it has found both of the URLs described
immediately above.  If so and if the player or base.js URL doesn't contain
a HostName, that is supplied from the partial watch?v= URL.

YTCrack then looks through each URL signature fragment and videoplayback
URL it has found, for %hh sequences and back slash characters.
All three character %hh sequences get replaced with the single character
that the hex digits hh represent.  And, this process gets repeated when %25
gets replaced by % .  However, any % not followed by hh gets left as-is.
Next, back slash characters are sought.  All six character \uhhhh sequences
get replaced with the single character that the least significant eight
bits of hhhh represents.  All remaining isolated back slash characters and
one back slash from each pair of them get removed.

Finally, each URL's "tail" gets trimmed at the end of the last of the 46
presently known sometimes required &fieldname= . . . videoplayback URL
fields present.  Except when a URL signature fragment is being processed.
For these, only &s= . . . , &sig= . . . , and &signature= . . . field(s)
are sought for tail trimming purposes.  The video.google videoplayback URLs
we've seen end with &key=ttn .  So, if &key= . . . is the last of the
fields found, the URL's tail gets trimmed at the first non-decimal digit
character following &key=tt .

From about 13 September 2012 until about 1 April 2013, all the YouTube
videoplayback URLs seemed to end with &sig=hhh . . . hhh.hhh . . . hhh
fields.  Beginning with version 0.16b, YTCrack changes the &sig= part of
that to &signature= .  And, then, handles the case that &signature= . . .
is the last of the 46 fields found.  Beginning with version 0.25b, YTCrack
does the same for videoplayback URLs ending with
&s=hhh . . . hhh.hhh . . . hhh fields.  In these cases, YTCrack looks for
the first non-hex digit character following &signature= .  If that is a
period, YTCrack again looks for a non-hex digit character following that.
The non-hex digit character thus found marks the spot for trimming the URL.
This kind of trimming eliminates any further field(s) present that aren't
in our list of 46 presently known sometimes required fields.

Beginning with version 0.27b, YTCrack checks &s= . . . style signatures
more thoroughly at this point.  Any found with value formats other than
<40 hex digits>.<40 hex digits> (or, 40.40) are counted and reported to be
invalid.

Beginning with version 0.29b, an attempt will be made to correct each
invalid &s= . . . style signature found as described above and containing
a total of at least 81 characters.

Beginning with version 0.37b, all &s= . . . style signatures get counted
and reported.  An attempt will be made to correct each of these
signatures, containing at least 81 characters, what ever its exact format.
In the interests of science, the number of signatures containing an
invalid format will still be reported.

Signature correction goes as follows:  First, if a player version has been
discovered, as described farther above, YTCrack requests a value for that
version name from the DOS Environment.  An exact mixed case copy of the
version gets requested, initially.  If that request gets refused, an all
upper case copy of the version gets requested.

Beginning with YTCrack version 0.49b, if a player version name hasn't been
discovered, an FLSTSKEY= Environment string's value will be sought.

The Environment string named by the player's version or FLSTSKEY= is
expected to have a value containing r , snn , and wnn field(s).
These fields represent operations to be performed on each &s= . . . style
signature's value, as follows:

     r    (as in Reverse) calls for the characters of the signature's value
          to be reversed.

     snn  (as in Slice(nn)) calls for the first nn characters of the
          signature's value to be discarded.

     wnn  (as in sWap(nn)) calls for character 0 and character nn , in the
          signature's value, to be swapped.

After an Environment string has been found and successfully applied to an
&s= . . . style signature's value, a count of corrected signature(s) gets
incremented.  Naturally, if the needed Environment string can't be
obtained, this count of corrected signature(s) will remain 0.

When
0 of these corrected using version based rule.
gets reported, the user's usual recourse is to receive the needed player or
base.js file, and save it using it's version name.  Next, run YTConfig
specifying the received file's version name for input.  And, a .BAT file
for output.  If YTConfig succeeds, run the .BAT file it has written.
That will SET the needed string into the Environment.  If DOS has to be
reBooted, or some other event leaves the needed Environment string missing,
it is sufficient to rerun the .BAT file that SETs the needed string.
Then, rerun YTCrack.

When any of the 46 sometimes required fields other than the four discussed
above are found last, the ampersand or ",<space> sequence, which ever comes
first, following that field gets taken as the termination point.
Then, when reconstituting signature(s) mode is applicable, any
videoplayback URL found lacking an &signature= . . . field is given a
temporary appended ampersand.  This provides the signature fragment tail
starting location for these URL(s), as discussed below.

With its batch of URL signature fragment(s) and videoplayback URL(s)
unencoded and clarified as explained above, YTCrack looks through them all
and eliminates any duplicate(s) present.  As stated earlier, there used to
be at least two copies of each different or distinct URL present, in
YouTube (but not video.google) Web pages.  If no duplicate(s) are present,
this check has no effect.

Next, the sequences <sep>quality=<anythingbut&>& and
<sep>type=<anythingbut&>& , where <sep> is question mark or ampersand,
are sought in each URL signature fragment and full URL processed.
For each of these sequences, the first such sequence found, if any, gets
reduced to its <sep> character.  Beginning with YTCrack version 0.28b,
the sequence <sep>fexp=<anythingbut&>& gets reduced to its <sep> character,
here, as well.  Beginning with YTCrack version 0.44b, the sequence
<sep>xtags=<anythingbut&>& also gets reduced to its <sep> character here.

When reconstituting signature(s), YTCrack then considers its remaining
full URL(s), from last to the first, following a URL signature fragment.
And, with each of these, the full URL or URL signature fragment just ahead
of it.  Signature fragment tail starting locations in each full URL are
identified by seeking the last field with a name matching the first field
named at the beginning of the applicable URL signature fragment.
For full URL(s) lacking a signature, the temporary ampersand, added as
described above, serves to locate the point where a URL signature fragment
tail is needed.  Then the earlier signature fragment tail gets copied, in
its entirety, into the later full URL.  Beginning with YTCrack v0.28b,
full URLs following a signature-less adaptive_fmts field get excluded from
this process.

Last of all, while parsing, YTCrack v0.25b and later look for fully
duplicated itag= . . . fields, within each full URL.  That is, duplicate
<sep1>itag=nnn<sep2> sequences with matching nnn values.  Where <sep1> is
question mark or ampersand.  <sep2> is ampersand or null.  And, nnn may be
any number of decimal digits.  When found, the later such duplicate,
in each URL, gets removed.  Beginning with YTCrack version 0.29b,
fully duplicated clen= . . . and lmt= . . . fields are given the same
treatment.

YTCrack then prepares a small html document exposing anchors or links
containing the distinct videoplayback URL(s) that it has found.
For all URL(s) that contain a <sep1>itag=<anythingbut&><sep2> field,
itag=<anythingbut&> gets used for the anchor's visible text.  Where <sep1>,
as usual, is question mark or ampersand.  And, <sep2> is ampersand or null.
Also, <anythingbut&> gets looked for in a list of the 64 itag= values given
in the table in the Overview section.  If this lookup succeeds, the implied
video file type, resolution, and any appropriate 3D, AO, VO, AOX, or VOX
flag get placed in the visible text, as well.  In case an
itag=<anythingbut&>& field isn't found in a URL, the visible text:
videoplayback URL number nn
will be used.  Where nn indicates the number of this videoplayback URL's
first appearance in the given document's sequence of videoplayback URLs.

Beginning with YTCrack version 0.29b, this report document ends with an
anchor or link for the current player or base.js JavaScript file, if that
has been discovered.


Installation and Operational Considerations

PKUNZIP YTCRAC50.ZIP
into an empty directory.  Copy the resulting files to where you want them.
You may as well put YTCRACK.EXE and YTCONFIG.EXE in a directory on your
Path.  There is nothing to configure, as a part of installation.
However, YTConfig is to be run whenever a new signature descrambling key is
needed for YTCrack.  This is discussed further in YTCONFIG.TXT .

YouTube and video.google seem to tailor their Web pages to their individual
users.  In particular, these Web pages seem to contain time stamps, which
cause their videoplayback URLs to quit working after a matter of day(s) or
sooner.  Also, the videoplayback URLs seem to contain information that
may have to match at least part of the downloading user's IP address.
Therefore, we suggest that you get a fresh copy of each YouTube or
video.google Web page you want to use immediately before running YTCrack
and downloading from its result document.  However, an exception to this
rule comes after running YTCrack and receiving the
0 of these corrected using version based rule.
message.  If you then receive the needed player or base.js file, process it
with YTConfig, and run its .BAT file output without much delay, a rerun of
YTCrack for the Web page already on hand, is expected to succeed.

YTCrack won't work with saved YouTube Web pages with URLs like:
http://www.youtube.com/v/VideoIDCode
or
http://www.youtube.com/embed/VideoIDCode

However, we have found that, for every valid VideoIDCode , there always
seems to be a URL available with the required form.  So, substitute:
/watch?v=
for
/v/
or
/embed/
in the URL you are attempting to use, and resave the page if necessary,
when you run into an unusable YouTube URL.

Further, YTCrack does work with saved YouTube Web pages having URLs like:
https://www.youtube.com/get_video_info?video_id=VideoIDCode&sts=stsnr

These may offer more rendition(s) of a given video than the watch?v= form
does.

With the get_video_info? variation, there is a difference in how the
descrambling key needed for s= . . . style signatures gets specified.
YTCrack can't find a URL for a player or base.js file in pages obtained
this way.  However, it doesn't need to guess at that.  Because, the
&sts=stsnr part of the URL specifies it.  In other words, the user
specifies the descrambling key to be used!

stsnr, usually a five digit number, is a serial number or sequence number
for YouTube's player versions.  On or about 29 November 2018, for example,
sts=17863 designated player or base.js file version flAhGWM_ .
As we were running YTConfig on FLAHGWM_.JS , we also commanded:
FIND "sts" FLAHGWM_.JS
to learn that.  The last two of the FINDings contained:
,sts:17863,
and
.sts="17863";

Using YTConfig's .BAT file output for FLAHGWM_.JS, we developed a short
FLSTSKEY.BAT file:
:29 Nov. 2018
:sts=17863
SET FLSTSKEY=w66rs1r
to express the result to YTCrack.

Beginning with version 0.49b, YTCrack will look for an FLSTSKEY=
Environment value when it doesn't find a URL for a player or base.js file.

sts numbers are expected to remain active for about a month before they
quit working as described here.  So, every four weeks or so, you will
want to update your FLSTSKEY.BAT file.

YouTube Web pages for a given VideoIDCode are not "deterministic" in the
full sense of that term.  In fact, YouTube is constantly "fuzzing" their
arrangement or ordering of videoplayback URL parameters so that it
appears to change with each request for a given URL!  This is true, as
well, for their fuzzed use of signature fragments along with signatures
that need shuffling or reconstitution.  When it comes to verifying an
understanding of YouTube's videoplayback URL syntax, this may be seen to
be both helpful and unhelpful.  Helpful because repetitive runs tend to
increase the chance for becoming aware of an arbitrary misunderstanding.
But, unhelpful because many misunderstanding(s) don't result in
"repeatable" errors.

When it comes to using an imperfect process to extract videoplayback
URL(s), YouTube's fuzzing, again, is both helpful and unhelpful.
We find it unhelpful when extracted videoplayback URL(s) prove to be
invalid.  At the same time, however, it is helpful.  Because, a rerun with
another copy of YouTube's Web page is quite likely to obtain valid URL(s).


Acknowledgements and Copyright Notices

YTCrack owes its existence to Glenn McCorkle and Ron Clarke, as well as to
its author.  First, Glenn informally published his technique for
discovering videoplayback URL(s) in YouTube Web pages.  Then, in his
UNTUBE.EXE program, Ron provided a sample implementation of Glenn's
technique.  Beginning with version 0.33b, YTCrack and YTConfig owe their
ability to descramble YouTube's scrambled &s= . . . style signatures to
Jamie Zawinski and his youtubedown perl script.  Jamie's script also has
informed our understanding of the get_video_info? . . . URL variation.

YTCRACK.EXE v0.50b and the libraries, materials, and tools used to make it
contain the following Copyright notices:

Borland C++ - Copyright 1991 Borland Intl.

LZEXE.EXE  Version 0.91   (c) 1989 Fabrice BELLARD


Disclaimer

This software is published in the hope that it will be useful, but without
any warranty;  without even the implied warranty of merchantability or
fitness for a particular purpose.


Fred C. Macall  K8GIV
1019 Pennfield Road
Cleveland Heights, Ohio, 44121, U.S.A.
(216) 382-3415
For e-mail contact, run YTCrack .
http://users.ohiohills.com/fmacall/

26 January 2019
