ATLG Sidebar

But What About Errors

As always it seems like I'm running short of time! This week I wanted to put a simple retry system in place. This will allow us to make a second attempt to download and removes a few of the panic()'s we had.

Updates

I'm not going to do a full code walkthrough this time around since we aren't making too many changes. Instead, we'll look at a before and after on the bits that saw some work.

The first thing we want to do is set up a struct to hold the article IDs for the ones we want to retry.

type Retry struct {
  IDs []int32
}

Perfect! We could extend it our if we say wanted to retry 3 times or something. In fact, we don't need it at all in my experience. I have been able to pull every public article from the API without any panics at all. But, I'm sure it may not run that smooth every time.

So you don't have to swap back to the first part, here is our original main().

func main() {
  dtc := New("https://dev.to/api/", nil)
  doit := true
  c := 1

  for doit {
    req, err := dtc.FormatPagedRequest("page", fmt.Sprintf("%d", c))
    if err != nil {
      panic(err)
    }
    resp, err := dtc.Client.Do(req)
    if err != nil {
      panic(err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      panic(err)
    }

    var wg sync.WaitGroup
    var articles Articles

    json.Unmarshal(body, &articles)
    wg.Add(len(articles))

    for i := range articles {
      go getArticle(dtc, articles[i].ID, &wg)
    }
    wg.Wait()

    if string(body) != "[]" {
      c++
      continue
    }
    doit = false
  }
}

And now our updated version, with changes noted.

func main() {
  dtc := New("https://dev.to/api/", nil)
  doit := true
  c := 1

We're adding in retries and report. The first will hold the article IDs that we want to attempt to get a second time. The second will hold any IDs that failed the second time around which we'll output to the console. We didn't remove any of the panics in main() this time. I think we could extend the retry system to cover it at some point.

  retries := Retry{}
  report := Retry{}

  for doit {
    req, err := dtc.FormatPagedRequest("page", fmt.Sprintf("%d", c))
    if err != nil {
      panic(err)
    }
    resp, err := dtc.Client.Do(req)
    if err != nil {
      panic(err)
    }

As was pointed out on Twitter by VirgileMathieu we're currently using defer inside our for loop. This may not be the best idea and could lead to unintended consequences. We have a couple of options to deal with this. First, we could remove the defer and just .Close() or we could wrap the entire section inside of an anonymous function.

    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      panic(err)
    }

    var wg sync.WaitGroup
    var articles Articles
    err = json.Unmarshal(body, &articles)
    if err != nil {
      panic(err)
    }

    wg.Add(len(articles))

We're going to pass in a pointer to retries so we can keep it up to date as we try to getArticle().

    for i := range articles {
      getArticle(dtc, articles[i].ID, &retries, &wg)
    }
    wg.Wait()

    if string(body) != "[]" {
      c++
      continue
    }
    doit = false
  }

Once our main loop has ended we're going to set up a second WaitGroup. We'll then attempt to grab any articles we may have missed. It might be worth setting up a loop here to tackle them in batches of 10 or so at a time. I'll do that and update the post. We should also wrap this section in an if no point going into it if retries is empty.

  // Lets try to get the ones we couldn't before
  var wg sync.WaitGroup
  wg.Add(len(retries.IDs))

  for i := range retries.IDs {
    getArticle(dtc, retries.IDs[i], &report, &wg)
  }
  wg.Wait()

  fmt.Printf("Unable to grab the following articles: %v\n", report)
}

Get Those Articles

Now let's look at our original getArticles(). It wasn't too bad - it got the job done! We want to get rid of the panics and allow the program to continue on even if we hit an error.

func getArticle(dtc *DevtoClient, i int32, wg *sync.WaitGroup) {
  defer wg.Done()
  r, err := dtc.FormatArticleRequest(i)
  if err != nil {
    panic(err)
  }

  resp, err := dtc.Client.Do(r)
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    panic(err)
  }
  fileName := fmt.Sprintf("%d.json", i)
  ioutil.WriteFile("./out/"+fileName, body, 0666)
}

We'll need to update the signature to since we're now passing in a pointer *Retry. Second, we'll update each of our err checks to update the retries struct and return to main().

func getArticle(dtc *DevtoClient, i int32, retries *Retry, wg *sync.WaitGroup) {
  defer wg.Done()
  r, err := dtc.FormatArticleRequest(i)
  if err != nil {
    retries.IDs = append(retries.IDs, i)
    return
  }

Note that we are adding a secondary check to see if we hit a statusCode over 399. This will cause us to add that article ID for any article that returns a client or server error.

  resp, err := dtc.Client.Do(r)
  if err != nil || resp.StatusCode > 399 {
    retries.IDs = append(retries.IDs, i)
    return
  }

  defer resp.Body.Close()

  _, err = ioutil.ReadAll(resp.Body)
  if err != nil {
    retries.IDs = append(retries.IDs, i)
    return
  }
}

Next time

That's all for this time around. We've gone pretty far with this code example so I'll be looking for something else to work with. Have any ideas? Let me know in the comments!

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.