My first Space service

Anyone needs to do new things from time to time. Today I decided to do two new things simultaneously. Write my first article to DEV.to and make official my first Space application.

TL;DR - go for the code: https://github.com/altavir/space-document-extractor

Motivation

JetBrains Space is a relatively new fast-growing all-in-one solution for intranet/team workspace. We've been using it in our team since the first preview announcement in 2019 (I was in KotlinConf when it was announced and got pretty excited). Right now we are moving most of our work from Slack/Confluence/GitHub stack into Space with GitHub mirror.

For our team, the knowledge base is very important and we were looking forward to using it in Space. We stopped using Confluence, but there are no full-fledged alternatives. We started using the knowledge base in YouTrack, which is not bad, but it still somehow kills the purpose of the all-in-one tool.

Recently, the knowledge base ( Documents ) in Space reach the next stage of its evolution and finally became ready for use. We started to write technical documentation and notes there. But found, that we need a way to get it from Space somehow.

Currently, Space does not provide a way to just download a directory (I think the feature is worth adding), so I decided to create an application to do that using Space API.

Implementation

The code is available in the repo. You can start with it.

The Space SDK is still pretty unstable and has some bugs, but it is quite easy to use and much more intuitive, than, say, Google Cloud. I will try to walk you through it.

Stage 1. Creating application

First, you need to create a Space application to provide access to API. Go to Administration -> Applications and create a new one. You need to set up permissions like it is shown here. It is not always obvious which permissions you need and error messages are sometimes misleading. For example, I spent some time wondering why I get a zero response for a correct request only to find out, that I need to manually add permissions to restricted projects. But I think they will clean it in the future.

The authentication is always the hardest thing, so I went with the default way (using just the client Id and a secret), but it proved to be surprisingly easy.
.

You can try the APIs in the playground in the same section and get some insight into how they work.

Stage 2. Set up the project with Space SDK.

Just take my build file as a reference.

repositories {
    mavenCentral()
    maven("https://maven.pkg.jetbrains.space/public/p/space/maven")
}

val ktorVersion = "1.6.4"

dependencies {
    implementation("io.ktor:ktor-client-core:$ktorVersion")
    implementation("io.ktor:ktor-client-cio:$ktorVersion")
    implementation("io.ktor:ktor-client-auth:$ktorVersion")
    implementation("org.jetbrains.kotlinx:kotlinx-cli:0.3.4")
    implementation("org.jetbrains:space-sdk-jvm:98244-beta")
    implementation("ch.qos.logback:logback-classic:1.2.10")
    testImplementation(kotlin("test"))
}

Be sure to use the correct Ktor version. For me 2.0.0 did not work since SDK has a dependency on it.

You don't need anything else.

Stage 3. Setting up the connection.

It is surprisingly easy:

    val space: SpaceClient = SpaceClient(
        ktorClientForSpace(CIO),
        SpaceAppInstance(
            clientId ?: System.getProperty("space.clientId"),
            clientSecret ?: System.getProperty("space.clientSecret"),
            spaceUrl
        ),
        SpaceAuth.ClientCredentials()
    )

Stage 4. Doing logic.

The whole logic is in this file. You can see there are no new classes (look, Ma, I am a functional programmer!).

The SDK is intuitive and follows the API logic. So you just take a client and see methods and properties that are available. Like this:

projects.documents.folders.subfolders.listSubfolders(projectId, folderId)

Sometimes it does not work as expected, so you have to experiment. For example I had to do this:

    val documents = projects.documents.folders.documents.listDocumentsInFolder(projectId, folderId) {
        id()
    }
    documents.data.forEach {
        val document = projects.documents.getDocument(projectId, it.id) {
            id()
            title()
            documentBody()
            bodyType()
        }
        downloadDocument(directory, document)
    }

because listDocumentsInFolder returns documents without bodies (it is rational, but not obvious from API).

Stage 5. A hacky part.

The document download went rather smoothly. But no we have markdown documents with attachment links that are:

relative,
point to the internet files that require authentication.

In order to fix that, we need to download those images and replace links by local file references.

Currently, Space does not provide API to download files (or at least I did not find it), so I had to do a little bit of reverse engineering.

I checked the requests that are used to access images and found out that they have an additional authorization header Bearer. The same one, the Space client uses to access pages:

So all I need to download images is to manually construct the request:

    val response = ktorClient.request<HttpResponse> {
        url("${server.serverUrl}/d/$imageId")
        method = HttpMethod.Get
        header(HttpHeaders.Authorization, "Bearer ${token().accessToken}")
    }

It is a hack and the actual placement of files could change in the future, but it works for now.

All that is left is to find image references and replace them with text in downloaded markdown files.

That's all.

Conclussion

I hope you will enjoy playing with Space API and I hope there will be a lot of new plugins and applications to enrich the ecosystem. I enjoyed it. Especially in comparison with older heavyweight systems. Kotlin SDK simplifies things a lot.

A bonus

Since we needed not only to download the documents but also to create a report from them, here is the simple PowerShell script to combine documents into a report, using PanDoc:

Blog