TheLazy.Dev
Portfolio GitHub

Blog

go |github |stackoverflow |terminal

How I use Stackoverflow from Terminal - Go

pr4k

Prakhar Kaushik

Posted on March 22, 2020

How I use Stackoverflow from Terminal - Go

Around a month ago I came across a project howdoi created in python. And honestly it was amazing, finding solution of basic problems without opening browser was a life saver, so I thought of creating one for myself. As one of my major goal was speed ,so I went with Go lang.

So today, let me show you how to use Stackoverflow using just a terminal.

Create Go Environment

I won't go in details about how to install Go or how the directory structure should look. You can find everything on the official page and the docs.

Official Installation Docs

What All We Need to Create

  • A scraper to get the posts using search query
  • A scraper to get the contents for each question
  • A UI to display the solution

Get Result for Search Query

A general Stackoverflow search url looks somthing like

https://stackoverflow.com/search?q=how+to+add+2+numbers

This makes it easy for us to get the result as what we need to change everytime is just the part after search?q=

After getting the content of page our next job is to get content from correct elements to get Title, Description, Link to Post and Up Votes.
For that first we need to create a Struct to store these data.

type post struct {
    title       string
    link        string
    upvotes     string
    description string
}
Enter fullscreen mode Exit fullscreen mode

for getting content of page we will be using GoQuery Library

So the code snippet for getting the data from search is

res, err := goquery.NewDocument(fmt.Sprintf("https://stackoverflow.com/search?q=%s", strings.ReplaceAll(query, " ", "+")))
    if err != nil {
        log.Fatal(err)
    }
Enter fullscreen mode Exit fullscreen mode

The question is stored in div with class question-summary So we will iterate on each element with this class and get required details from the div.

Class for

  • post link is .result-link
  • question description is .excerpt
  • upvotes count is .vote-count-post
  • title is same as link tag, all we need is to get the text for a tag

After getting all content we will store the data in list of posts struct

The final source code for getting the data will look something like

func searchPost(query string) []post {
    res, err := goquery.NewDocument(fmt.Sprintf("https://stackoverflow.com/search?q=%s", strings.ReplaceAll(query, " ", "+")))
    if err != nil {
        log.Fatal(err)
    }

    var items []post
    res.Find(".question-summary").Each(func(index int, item *goquery.Selection) {
        linkTag := item.Find(".result-link").Find("a")
        link, _ := linkTag.Attr("href")
        title := strings.TrimFunc(linkTag.Text(), func(r rune) bool {
            return !unicode.IsLetter(r) && !unicode.IsNumber(r)
        })
        description := item.Find(".excerpt").Text()
        upvotes := item.Find(".vote-count-post").Text()
        //fmt.Println(link)
        if true || (strings.HasPrefix(title, "Q:") && index < 4) {
            items = append(items, post{title, link, upvotes, description})
            index++
        }
    })
    return items
Enter fullscreen mode Exit fullscreen mode

After this we are done with getting the content of search page and storing in our array.
Next part will be getting content for each question and storing it in another array

Get Each Post Content

We have url for each post stored in the array from previous section, our next job is to go on each of these links and get content for that page and store it in another array.

So the stackoverflow page for answer has 2 parts , A question section and a list of solutions. We will get each solution and store it in another array and return the answers along with the accepted answer
This will be same as previous part.
Struct we will be using for this part will be

type solution struct {
    description string
    upvotes     string
}
Enter fullscreen mode Exit fullscreen mode

So the class for question is .question and for accepted answer it is accepted-answer

After this remaining answers will be scraped using the relative div from accepted answer div

code for getting the question content is


    var answers []solution
    question := res.Find(".question").Find(".post-layout")
    answers = append(answers, solution{strings.Trim(question.Find(".post-text").Text(), "\n"), question.Find(".js-vote-count").Text()})
Enter fullscreen mode Exit fullscreen mode

code for getting the accepted answer is


    acceptedContainer := res.Find(".accepted-answer").Find(".post-layout")
    acceptedAnswer := solution{strings.Trim(acceptedContainer.Find(".post-text").Text(), "\n"), acceptedContainer.Find(".js-vote-count").Text()}
Enter fullscreen mode Exit fullscreen mode

and finally for getting remaining answers

if (acceptedAnswer != solution{}) {
        res.Find(".accepted-answer").NextAll().Each(func(index int, item *goquery.Selection) {

            if item.Find(".post-text").Text() != "" {
                answers = append(answers, solution{strings.Trim(item.Find(".post-text").Text(), "\n"), item.Find(".js-vote-count").Text()})
            }
        })
    } else {
        res.Find(".post-layout").Each(func(index int, item *goquery.Selection) {

            if item.Find(".post-text").Text() != "" {
                answers = append(answers, solution{strings.Trim(item.Find(".post-text").Text(), "\n"), item.Find(".js-vote-count").Text()})
            }
        })
    }
Enter fullscreen mode Exit fullscreen mode

So after combining everything we will get

func getPost(node post) (solution, []solution) {
    urlString := fmt.Sprintf("https://stackoverflow.com/%s", node.link)
    res, err := goquery.NewDocument(urlString)
    if err != nil {
        log.Fatal(err)
    }

    var answers []solution
    question := res.Find(".question").Find(".post-layout")
    answers = append(answers, solution{strings.Trim(question.Find(".post-text").Text(), "\n"), question.Find(".js-vote-count").Text()})

    acceptedContainer := res.Find(".accepted-answer").Find(".post-layout")
    acceptedAnswer := solution{strings.Trim(acceptedContainer.Find(".post-text").Text(), "\n"), acceptedContainer.Find(".js-vote-count").Text()}

    if (acceptedAnswer != solution{}) {
        res.Find(".accepted-answer").NextAll().Each(func(index int, item *goquery.Selection) {

            if item.Find(".post-text").Text() != "" {
                answers = append(answers, solution{strings.Trim(item.Find(".post-text").Text(), "\n"), item.Find(".js-vote-count").Text()})
            }
        })
    } else {
        res.Find(".post-layout").Each(func(index int, item *goquery.Selection) {

            if item.Find(".post-text").Text() != "" {
                answers = append(answers, solution{strings.Trim(item.Find(".post-text").Text(), "\n"), item.Find(".js-vote-count").Text()})
            }
        })
    }

    return acceptedAnswer, answers
}
Enter fullscreen mode Exit fullscreen mode

Build the UI

So we have Everything we need, Question list from search page and then solution for each post , Now we need to display the content in a UI.
what we will be trying to obtain is 2 parts, one for question and other one for the question description.

So the library we will be using is termui.
Our UI will have 3 sections , one for question list, other for question description and the last for the possible solution and we will display 2 sections at a time
Now I won't cover the whole library as we only need some parts of it.

So for creating a box with paragraph inside we have a widget already provided in the library, but the problem with it is we can't scroll it , so for that we will create our own paragraph widget.

The source code for the widget is

For struct

type Paragraph struct {
    Block
    Text      string
    TextStyle Style
    WrapText  bool
    start     int
    end       int
}
Enter fullscreen mode Exit fullscreen mode

and the remaining code base


func NewParagraph() *Paragraph {
    return &Paragraph{
        Block:     *NewBlock(),
        TextStyle: Theme.Paragraph.Text,
        WrapText:  true,
    }
}

func (self *Paragraph) Draw(buf *Buffer) {
    self.Block.Draw(buf)

    cells := ParseStyles(self.Text, self.TextStyle)
    if self.WrapText {
        cells = WrapCells(cells, uint(self.Inner.Dx()))
    }

    rows := SplitCells(cells, '\n')
    if self.end-self.start <= len(rows) {
        if self.end > len(rows) {
            self.end = len(rows)
            self.start = self.end - 40
        }

        if self.start <= 0 {
            self.start = 0
            self.end = 40
        }
        rows = rows[self.start:self.end]
    }
    for y, row := range rows {
        if y+self.Inner.Min.Y >= self.Inner.Max.Y {
            break
        }
        row = TrimCells(row, self.Inner.Dx())
        for _, cx := range BuildCellWithXArray(row) {
            x, cell := cx.X, cx.Cell

            buf.SetCell(cell, image.Pt(x, y).Add(self.Inner.Min))
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Now explaining the whole code will be an another article, but to give you people a gist, it basically takes the style we specified and create cells, then it takes the whole text and breaks it into lines and then iterating on each line , drawing it into the cell we specified above, and placing it at the correct position.

The part which is important is

    if self.end-self.start <= len(rows) {
        if self.end > len(rows) {
            self.end = len(rows)
            self.start = self.end - 40
        }

        if self.start <= 0 {
            self.start = 0
            self.end = 40
        }
        rows = rows[self.start:self.end]
    }
Enter fullscreen mode Exit fullscreen mode

This part handles the content scrolling, what this is doing is we provide 2 values one is start and other one is end. Then it uses those values to slice the array, so that we draw only a part of the whole content on screen.This start and end value will change on each key press so the content will look like its scrolling.

So the rest of the part is pretty straight forward, we will be using this newly created widget and create 3 boxes with different content.
I won't display the code of this part here as it will make the post unnecessarily long, instead I will explain the flow.
So we will first create 3 paragraphs

Example code :

    quesBox := NewParagraph()

    quesBox.Title = "Question Description"
    quesBox.SetRect(width/2, 4, width, height-4)
    quesBox.start = 0
    quesBox.end = 44
    quesBox.WrapText = true
    quesBox.BorderStyle.Fg = ui.ColorYellow
Enter fullscreen mode Exit fullscreen mode

And then the only task would be to render correct box on correct key press.
So termui provides few functions like Render to render particular element and a whole keyboard binding to refer each key.
Example code:

uiEvents := ui.PollEvents()
    for {
        e := <-uiEvents
        switch e.ID {
        case "q", "<C-c>":
            return
Enter fullscreen mode Exit fullscreen mode

So this handles the closing of Ui if someone presses q or ctrl-c, we can add other cases to render correct Box on key press.

So the final thing will work something like

You can find the whole source code at:-

GitHub logo pr4k / howto

Terminal client for stack overflow

💖 💪 🙅 🚩
pr4k
Prakhar Kaushik

Posted on March 22, 2020

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

How I use Stackoverflow from Terminal - Go
go How I use Stackoverflow from Terminal - Go

March 22, 2020