Hobby Project: Pull All Your Dev.to Articles

jessekphillips

Jesse Phillips

Posted on February 10, 2020

Hobby Project: Pull All Your Dev.to Articles

This concludes the test drive for the api. I needed to make some final adjustments.

https://gitlab.com/jessephillips/devarticlator/-/compare/v0.0.8...v0.1.0

The purpose of this session was to navigate to the last page and store all users articles. And while this work was easy I started realizing how broken my filtering for new articles was. The returned articles are in order of publication, I knew this, the implications is that article ID has no guaranteed order. This is why I included the date pulled, I had just thought I found a shortcut.


The unpublished articles create their own set of challenges. These articles are more likely to be changing and Dev provides a different slug than what the published article will be.

At this point I'm not solving this problem. Implications

  • unpublished articles are always written
  • an unpublished article which gets published will result in a new article file

One of the more challenging parts of this was keeping the isNewerArticle function pure and nothrow. It would have been easy to just perform the string conversion on the spot, but since I marked the method nothrow this action was denied.

Remember the original statement I said was my code wasn't better for adding attributes. Well now I am paying for it and I think it is for the better.

Placing the conversation in the isNewerArticlefunction causes a delay in reporting an issue. I'm now required to validate and setup my data before beginning to process it.

auto data = res.readJson().get!(Json[]);
            data.each!(x => x["published_timestamp"] = x["published_timestamp"].get!string.empty 
            ?  SysTime.init.toISOExtString : x["published_timestamp"].get!string);
Enter fullscreen mode Exit fullscreen mode

When writing the unittests for the range got a little bit out of hand.

See nested functions actually come out as delegates, so I created an alias to a lambda so the compiler can determine context does not require a delegate. But like most writings, I had forgotten static nested functions aren't delegates.

alias fakeArticles = (uint page) @safe {
        return () @trusted {
        static iteration = 0;
        if(iteration++ < 2)
        return generate!fakeArticleMe
            .map!(x => cast(immutable(ArticleMe))x.deserializeJson!(ArticleMe))
            .take(3)
            .array;
        return generate!fakeArticleMe
            .map!(x => cast(immutable(ArticleMe))x.deserializeJson!(ArticleMe))
            .take(0)
            .array;
        }();
    };
Enter fullscreen mode Exit fullscreen mode

I also used some unsafe calls so I wrapped the body in a trusted lambda.


I've included a readme now that a have a reasonably usable application for pulling articles. I've also registered with D's code registry.

https://code.dlang.org/packages/devarticlator


My unittests still don't cover the unpublished articles.

💖 💪 🙅 🚩
jessekphillips
Jesse Phillips

Posted on February 10, 2020

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Hobby Project: Dev.to API
dlang Hobby Project: Dev.to API

January 15, 2020