Premise

I had some extra time last weekend and decided to work on a small project that would solve my own problem. It’s suppose to be a really simple mobile PWA that only streams Youtube audio and loops repeatedly. There’s a bunch of them out there so it’s not exactly an original idea, but I wanted something simple without all the extra cruft like ads, comments, or authentication.

Feature to Implement

Right off the bat, there was a core functionality I needed to implement. A “search box” for querying the YT api for videos. The API for video search is straightforward enough: https://developers.google.com/youtube/v3/docs/search/list. Just sign up for the Youtube Data API key and we have access to all the power of Youtube search.

Now, we could just build one that the user enters in the query and then taps a button to search, but it would be much more user friendly if there was autocomplete suggest like the actual Youtube site. Auto suggest would make it so much easier to use. Here is an image of desire functionality from youtube.

No API 😞

But, alas! After a couple hours of searching on both Google and Youtube API docs, it turns out Google does not provide an official auto-suggest API. Or it seems, they provided the API previously, but decided to shut it down for some reason. Now at this point, I could just go ahead with the core search functionality and forget about the auto suggestions… but let’s have a look at Youtube first just out of curiosity.

Looking underneath the hood

Over at Youtube, if we start typing in the search bar, and open up the Chrome dev tools, we see network requests being made that points to an undocumented API endpoint: https://clients1.google.com/complete/search?client=youtube&hl=en&gl=sg&gs_rn=64&gs_ri=youtube&tok=h3yTGb1h3-yuCBwsAaQpxQ&ds=yt&cp=3&gs_id=2u&q=jaz&callback=google.sbox.p50&gs_gbg=0l0MjG05RWnWBe9WcipQbsy

After playing around with the parameters, it turns out that most of the params are not really needed. The main ones that matter to our use case is:

• client: forces json response, we want to use youtube here
• ds: google site properties, use yt to restrict to Youtube
• hl: the culture or language. use for localization. Default is usually en
• callback: this is the jsonp callback
• q: term query you want to search for

At this point, the endpoint works. If you tried it in the browser now, you would download a text. It’s strange text file with numbers and gibberish, but inside it we clearly see the data we need to implement our autocomplete search. Hooray!

// contents of autosuggest endpoint
google.sbox.p50 && google.sbox.p50(["jazz ",[["jazz music",0],["jazz piano",0],["jazz songs",0],["jazz dance",0,[131]],["jazz music best songs",0],["jazz instrumental",0],["jazz guitar",0],["jazz relaxing music",0,[131]],["jazz jennings",0],["jazz for work",0,[131]]],{"a":"FCwlE6frPjfCHAJSPzskH5xxMxJia3UhfNxNRVG6aehsz7iBn4XxJQ6ACUGMVuaAl5f1LHrO2ErGn7t4d6mIXg965Zxp3bENM4iS00nEvwhiiSe8Bi39NZsbdj2BHz3FD0C","j":"32","k":1,"q":"8KKe7s-xREtd_veunmBB7oKGghg"}])

You might recognized this as jsonp and if not, then a few google searches, and we have the answer! The google.sbox.p50 is the callback function which we’ll pass in ourselves.

In my side project, I’m using axios, and we can find a jsonp adaptor for axios here. The basic request logic looks like this:

    export const suggest = (term: string) => {
      const GOOGLE_AC_URL: string = `https://clients1.google.com/complete/search`;
      return axios({
        // A YT undocumented API for auto suggest search queries
        url: GOOGLE_AC_URL,
        adapter: jsonpAdapter,
        params: {
          client: "youtube",
          hl: "en",
          ds: "yt",
          q: term,
        }
      })
      .then((res: AxiosResponse) => {
        console.log("jsonp results >> ", res);
        if (res.status !== 200) {
          throw Error("Suggest API not 200!");
        }
        return res.data[1].map((item: any[]) => item[0]);
      })
    }

Now, we just need to connect the results to a react component input element and wire up the state and change handlers. I have used styled-components components to rename and style many of the html elements. The bare bones implementation is show here:

    /* search-box component */
    import * as React from "react";
    import styled from "styled-components";
    import * as _ from "lodash";

    import {youtubeSearch, suggest} from "./services/youtube";

    export default class SearchBox extends React.PureComponent<any, any> {
      public state = {
        suggestions: null,
      };

      constructor(props: any) {
        super(props);
        this.onTypeSuggest = _.debounce(this.onTypeSuggest, 500, { leading: true });
      }

      public render() {
        return (
          <>
            <Container>
              <SearchBox
                onChange={e => this.onTypeSuggest(e.target.value)}
                placeholder="Search for music, songs, podcasts"
                type="search"
              />
            </Container>
            <Suggestions onSearch={this.onSearch} items={this.state.suggestions} />
          </>
        )
      }

      private onTypeSuggest = async (
        queryString: string,
      ) => {
        if (queryString.length < 5) {
          // search only after 5 chars
          return null;
        }
        const list = await suggest(queryString);
        return this.setState({ suggestions: list });
      }

      // Search the term when selected using the official youtube search api:
      // https://www.googleapis.com/youtube/v3/search
      private onSearch = async (queryString: string, platform: MediaPlatforms) => {
        if (platform === MediaPlatforms.Youtube) {
          const platformSearchResults = await youtubeSearch(queryString);
          this.setState({ suggestions: null });
          this.props.update(platformSearchResults);
        }
      }
    }

We also want to show the suggestions that we get from our hand crafted youtube autosuggest api. A simple list of items will do. Each suggestion is passed the onSearch function which takes the autosuggestion selected and queries the official youtube search api above.

function Suggestions({ onSearch, items }: any) {
  if (!items || !items.length) {
    return null;
  }
  return (
    <Section>
      {items.map((item: string, key: number) => {
        return <SuggestionRow onSearch={onSearch} key={key} text={item} />;
      })}
    </Section>
  );
}

function SuggestionRow({ onSearch, text }: any) {
  return (
    <SuggestionSpan onClick={() => onSearch(text, MediaPlatforms.Youtube)}>
      {text}
    </SuggestionSpan>
  );
}

And add some styling to pretty things up a bit.

const Container = styled("div")`
  position: relative;
  width: 100%;
  display: flex;
  flex-flow: row nowrap;
  align-items: center;
  justify-content: flex-start;
`;
const SearchBox = styled("input")`
  box-sizing: border-box;
  height: 2.9rem;
  width: 100%;
  padding: 0.5rem;
  padding-left: 1rem;
  border-radius: 0.2rem;
  border: 2px solid #aaa;
  max-width: 800px;
  font-size: 1rem;
`;
const CSearchIcon = styled(SearchIcon)`
  cursor: pointer;
  margin-left: -50px;
`;
const Section = styled("section")`
  width: 95%;
  min-height: 18rem;
  height: auto;
  border: 1px solid #ddd;
  border-top: none;
  border-radius: 5px;
  margin-top: 1rem;
  padding: 0.5rem;
  box-shadow: 1px 1px 1px #ddd;
  z-index: 1000;
`;
const SuggestionSpan = styled("span")`
  display: inline-block;
  width: 100%;
  color: #9c27b0;
  font-weight: 800;
  margin-bottom: 0.5rem;
  margin-left: 0.5rem;
  cursor: pointer;
  z-index: 1000;
`;

Awesome. Looks like it works as expected!

There might be many creative uses of the autosuggest api, not just for the obvious search suggestions. Would be interesting to hear some others. But make sure to use responsibly!

Finishing up and next steps

Now that the basic functionality works. We could package it up nicely into a react hook so it’s easily usable in any component or by anyone with an npm install

If you want to reverse engineer the Youtube stream, that’s an another level of complexity and very interesting topic to explore. A good starting point would be here: https://tyrrrz.me/blog/reverse-engineering-youtube and checkout out the node-ytdl source or just use the package!

I may write follow-up post on Node.js audio streaming once I learn more about the topic. Also, I would like to write about the intricacies of audio playback for mobile browsers such as Chrome and Safari as it relates to PWAs in a future post.

Cheers.