TiddlyWiki and Emacs

Since my last post on reddit asking for some help regarding Emacs and TiddlyWikis REST API I gained some elisp knowledge I’d like to share.

TiddlyWiki 5

For those of you who haven’t heard of TiddlyWiki yet:

TiddlyWiki is a personal wiki and a non-linear notebook for organising and sharing complex information. It is an open-source single page application wiki in the form of a single HTML file that includes CSS, JavaScript, and the content. It is designed to be easy to customize and re-shape depending on application. It facilitates re-use of content by dividing it into small pieces called Tiddlers. – Wikipedia

You use the wiki as a single HTML page or via nodejs. With nodejs we can
talk to Tiddlywiki via its REST API.Every single page inside the wiki is called tiddler.

On the philosophy of tiddlers: “The purpose of recording and organising information is so that it can be used again. The value of recorded information is directly proportional to the ease with which it can be re-used.”

A tiddler has following format:

Code Snippet 1: Tiddler JSON format

Next I’ll show you how to setup your TiddlyWiki instance.

Basic setup

I use node.js to run my TiddlyWiki instance. For isolation reasons I use Docker to run it. Here is my Dockerfile:

FROM mhart/alpine-node

# Create a group and user
RUN addgroup -g 984 -S appgroup
RUN adduser -h /DATA/wiki -u 1000 -S appuser -G appgroup

# Tell docker that all future commands should run as the appuser user

ENV TW_BASE=/DATA TW_NAME=wiki TW_USER="xxx" TW_PASSWORD="xxx" TW_LAZY=""
ENV TW_PATH ${TW_BASE}/${TW_NAME}

WORKDIR ${TW_BASE}

RUN npm install -g npm@8.10.0
RUN npm install -g tiddlywiki http-server

# COPY plugins/felixhayashi /usr/lib/node_modules/tiddlywiki/plugins/felixhayashi
# RUN ls -la /usr/lib/node_modules/tiddlywiki/plugins
COPY start.sh ${TW_BASE}

# Change ownership
RUN chown appuser:appgroup /DATA/start.sh

EXPOSE 8181

USER appuser

ENTRYPOINT ["/DATA/start.sh"]
CMD ["/DATA/start.sh"]

Code Snippet 2: Dockerfile for running TiddlyWiki 5 using alpine

And as for start.sh:

#!/usr/bin/env sh

# Start image server
http-server -p 82 /DATA/wiki/images &

# Start tiddlywiki server
tiddlywiki /DATA/wiki --listen port=8181 host=0.0.0.0 csrf-disable=yes

Code Snippet 3: Bash script to start a simple http-server (for uploading images) and the tiddlywiki server instance (node.js)

Now you should be able to call the API (via curl for example):

curl http://127.0.0.1:8181/recipes/default/tiddlers/Emacs | jq

Code Snippet 4: Now you should be able to call the API (via curl for example).

{
  "title": "Emacs",
  "created": "20210623082136326",
  "modified": "20210623082138258",
  "tags": "Topics",
  "type": "text/vnd.tiddlywiki",
  "revision": 0,
  "bag": "default"
}

Code Snippet 5: The REST API will send back a JSON response.

request.el

I use request.el for crafting and sending HTTP requests. So what is request.el all about?

Request.el is a HTTP request library with multiple backends. It supports url.el which is shipped with Emacs and curl command line program. User can use curl when s/he has it, as curl is more reliable than url.el. Library author can use request.el to avoid imposing external dependencies such as curl to users while giving richer experience for users who have curl. – Source

GET

Let’s have a look how a simple (GET) API call looks like:

(let*
    ((httpRequest
      (request "https://api.chucknorris.io/jokes/random"
        :parser 'json-read
        :sync t
        :success (cl-function
                  (lambda (&key data &allow-other-keys)
                    (message "I sent: %S" data)))))

     (data (request-response-data httpRequest)))

  ;; Print information
 (cl-loop for (key . value) in data
      collect (cons key value)))

Code Snippet 6: Get a random Chuck Norris joke

((categories .
             [])
 (created_at . "2020-01-05 13:42:19.576875")
 (icon_url . "https://assets.chucknorris.host/img/avatar/chuck-norris.png")
 (id . "YNmylryESKCeA5-TJKm_9g")
 (updated_at . "2020-01-05 13:42:19.576875")
 (url . "https://api.chucknorris.io/jokes/YNmylryESKCeA5-TJKm_9g")
 (value . "The descendents of Chuck Norris have divided into two widely known cultures: New Jersey and New York."))

POST

Sending a POST request is also an easy task:

(let*
    ((httpRequest
      (request "http://httpbin.org/post"
        :type "POST"
        :data '(("key" . "value") ("key2" . "value2"))
        :parser 'json-read
        :sync t
        :success (cl-function
                  (lambda (&key data &allow-other-keys)
                    (message "I sent: %S" data)))))

     (data (request-response-data httpRequest))
     (err (request-response-error-thrown httpRequest))
     (status (request-response-status-code httpRequest)))

  ;; Print information
 (cl-loop for (key . value) in data
      collect (cons key value)))

Code Snippet 7: POST request with data

And here is the result:

((args)
 (data . "")
 (files)
 (form
  (key . "value")
  (key2 . "value2"))
 (headers
  (Accept . "*/*")
  (Accept-Encoding . "deflate, gzip, br, zstd")
  (Content-Length . "21")
  (Content-Type . "application/x-www-form-urlencoded")
  (Host . "httpbin.org")
  (User-Agent . "curl/7.83.1")
  (X-Amzn-Trace-Id . "Root=1-62cdbc5c-52d3ad32436c1cb8778808e5"))
 (json)
 (origin . "127.0.0.1")
 (url . "http://httpbin.org/post"))

Code Snippet 8: POST response as list of Elisp cons cells

Emacs


;; default tiddlywiki base path
(setq tiddlywiki-base-path "http://127.0.0.1:8181/recipes/default/tiddlers/")

GET tiddler

Let’s GET a tiddler:

(let*
    ((httpRequest
      (request (concat tiddlywiki-base-path "Emacs")
        :parser 'json-read
        :sync t
        :success (cl-function
                  (lambda (&key data &allow-other-keys)
                    (message "I sent: %S" data)))))

     (data (request-response-data httpRequest))
     (err (request-response-error-thrown httpRequest))
     (status (request-response-status-code httpRequest)))

  ;; Print information
 (cl-loop for (key . value) in data
      collect (cons key value)))

Code Snippet 9: Get a tiddler by name ("Emacs")

((title . "Emacs")
(created . "20210623082136326")
(modified . "20210623082138258")
(tags . "Topics")
(type . "text/vnd.tiddlywiki")
(revision . 0)
(bag . "default"))

Code Snippet 10: Response as list of Elisp cons cells

PUT a new tiddler

Creating a new tiddler is also simple. Using ob-verb let’s add a PUT request to the API:

PUT http://127.0.0.1:8181/recipes/default/tiddlers/I%20love%20Elisp
x-requested-with: TiddlyWiki
Content-Type: application/json; charset=utf-8

{
"title": "I love Elisp",
"tags": "Emacs [[I Love]]",
"send-with": "verb",
"text": "This rocks!"
}

Code Snippet 11: Sample request for creating a new tiddler

Check if tiddler was indeed created:

GET http://127.0.0.1:8181/recipes/default/tiddlers/I%20love%20Elisp
x-requested-with: TiddlyWiki
Accept: application/json; charset=utf-8

Code Snippet 12: GET request using verb

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 13 Jul 2022 10:03:27 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

{
    "title": "I love Elisp",
    "tags": "Emacs [[I Love]]",
    "fields": {
    "send-with": "verb"
},
    "text": "This rocks!",
    "revision": 1,
    "bag": "default",
    "type": "text/vnd.tiddlywiki"
}

Code Snippet 13: A new tiddler was created

Now let’s translate that to request.el code. This I’ll some extra complexity: I’ll add a function (defun) to PUT a new tiddler for us, where name , tags and body of the tiddler are variable.

;; Define function for inserting new tiddlers
(defun insert-tiddler(name tags body)
  (let*
  (
   (tiddler-title name)
   (url-path (url-hexify-string tiddler-title))
   (tiddler-tags tags)
   (tiddler-body body)

   (httpRequest
    (request (concat tiddlywiki-base-path url-path)
      :type "PUT"
      :data (json-encode
             `(
               ("title" . ,tiddler-title)
               ("created" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
               ("modified" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
               ("tags" . ,tiddler-tags)
               ("text" . ,tiddler-body)
               ("type" . "text/vnd.tiddlywiki")))
      :headers '(
                 ("Content-Type" . "application/json")
                 ("X-Requested-With" . "Tiddlywiki")
                 ("Accept" . "application/json"))
      :encoding 'utf-8
      :sync t
      :complete
      (function*
       (lambda (&key data &allow-other-keys)
         (message "Inside function: %s" data)
         (when data
           (with-current-buffer (get-buffer-create "*request demo*")
             (erase-buffer)
             (insert (request-response-data data))
             (pop-to-buffer (current-buffer))))))
      :error
      (function* (lambda (&key error-thrown &allow-other-keys&rest _)
                   (message "Got error: %S" error-thrown)))
      )))

  (format "%s:%s"
          (request-response-headers httpRequest)
          (request-response-status-code httpRequest)
          )))

;; Insert 2 tiddlers
(insert-tiddler "I love Elisp" "Elisp [[I Love]]" "This rocks!")

Code Snippet 14: Create new function for inserting new tiddlers

"((etag . \"default/I%20love%20Elisp/61:\") (content-type . text/plain) (date . Wed, 13 Jul 2022 12:30:33 GMT) (connection . keep-alive) (keep-alive . timeout=5)):204"

Code Snippet 15: New tiddler was created

Some explanations:

in line 6 I URL encode the tiddler-title
- I love Elisp should become I%20love%20Elisp
in line 21 some headers are set
- X-Requested-With is required to be set to TiddlyWiki
- Content-Type should be json
- we also accept json as a response
in line 13 we specify the data to be sent to the API
- each field (key, value sets) is set accordingly (see 10)
- I set the created and modified fields using format-time-string

Now let’s check again if tiddler really exists:

GET http://127.0.0.1:8181/recipes/default/tiddlers/I%20love%20Elisp
x-requested-with: TiddlyWiki
Accept: application/json; charset=utf-8

Code Snippet 16: Check if new tiddler exists

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 13 Jul 2022 12:40:22 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

{
    "title": "I love Elisp",
    "created": "20220713143033566",
    "modified": "20220713143033566",
    "tags": "Elisp [[I Love]]",
    "text": "This rocks!",
    "type": "text/vnd.tiddlywiki",
    "revision": 61,
    "bag": "default"
}

Code Snippet 17: It does exist!

Use cases

Now what can you do with this little custom functions? Let me share my use cases.

Add bookmark

A bookmark in my TiddlyWiki represents a tiddler of following format:

GET http://127.0.0.1:8181/recipes/default/tiddlers/chashell
Accept: application/json; charset=utf-8

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 13 Jul 2022 12:49:58 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

{
    "title": "chashell",
    "created": "20210519103441485",
    "modified": "20210519103528982",
    "fields": {
        "name": "chashell",
        "note": "Chashell is a Go reverse shell that communicates over DNS. It can be used to bypass firewalls or tightly restricted networks.",
        "url": "https://github.com/sysdream/chashell"
    },
    "tags": "Golang Security Tool Bookmark",
    "type": "text/vnd.tiddlywiki",
    "revision": 0,
    "bag": "default"
}

Every bookmarks consists of a name , a note and an url. Every tiddler supposed to be a bookmark is tagged by Bookmark. In this chashell is a tiddler and at the same time a bookmark in my wiki. As part of my daily routine, I go through my pocket entries and decide which ones I should bookmark in Tiddlywiki. These are my keybindings for the getpocket major mode:

(map! :map pocket-reader-mode-map
:after pocket-reader
:nm "d" #'pocket-reader-delete
:nm "SD" #'dorneanu/pocket-reader-send-to-dropbox
:nm "a" #'pocket-reader-toggle-archived
:nm "B" #'pocket-reader-open-in-external-browser
:nm "e" #'pocket-reader-excerpt
:nm "G" #'pocket-reader-more
:nm "TAB" #'pocket-reader-open-url
:nm "tr" #'pocket-reader-remove-tags
:nm "tN" #'dorneanu/pocket-reader-remove-next
:nm "C-b" #'dorneanu/tiddlywiki-add-bookmark
:nm "ta" #'pocket-reader-add-tags
:nm "gr" #'pocket-reader-refresh
:nm "p" #'pocket-reader-search
:nm "U" #'pocket-reader-unmark-all
:nm "y" #'pocket-reader-copy-url
:nm "Y" #'dorneanu/pocket-reader-copy-to-scratch)

Let’s have a look at dorneanu/tiddlywiki-add-bookmark:

(defun dorneanu/tiddlywiki-add-bookmark ()
  "Adds a new bookmark to tiddlywiki. The URL is fetched from clipboard or killring"
    (require 'url-util)
    (interactive)
    (pocket-reader-copy-url)

    (setq my-url (org-web-tools--get-first-url))
    (setq url-html (org-web-tools--get-url my-url))
    (setq url-title (org-web-tools--html-title url-html))
    (setq url-title-mod (read-string "Title: " url-title))
    (setq url-path (url-hexify-string url-title-mod))
    (setq url-note (read-string (concat "Note for " my-url ":")))
    (setq url-tags (concat "Bookmark "(read-string "Additional tags: ")))

    (request (concat tiddlywiki-base-path url-path)
    :type "PUT"
    :data (json-encode `(("name" . ,url-title-mod) ("note" . ,url-note) ("url" . ,my-url) ("tags" . ,url-tags)))
    :headers '(("Content-Type" . "application/json") ("X-Requested-With" . "TiddlyWiki") ("Accept" . "application/json"))
    :parser 'json-read
    :success
    (cl-function
            (lambda (&key data &allow-other-keys)
                (message "I sent: %S" (assoc-default 'args data))))
    :complete (lambda (&rest _) (message "Added %s" (symbol-value 'url-title-mod)))
    :error (lambda (&rest _) (message "Some error"))
    :status-code '((400 . (lambda (&rest _) (message "Got 400.")))
                    (418 . (lambda (&rest _) (message "Got 418.")))
                    (204 . (lambda (&rest _) (message "Got 202."))))
    )
)

Code Snippet 18: Bookmark entries from getpocket directly into Tiddlywiki

Add quote

After reading each book I usually do some post-reading/post-processing. While I could use the Tiddlywiki web interface to add new tiddlers, I’d rather do it from Emacs directly.

Often I need to insert new quotes from book (or web articles). How to I do this:

(defun dorneanu/tiddlywiki-add-quote ()
  "Adds a new quote"
    (interactive)

    (setq quote-title (read-string "Quote title: " quote-title))
    (setq url-path (url-hexify-string quote-title))
    (setq quote-source (read-string (concat "Source for " quote-title ": ") quote-source))
    (setq quote-body (read-string (concat "Text for " quote-title ": ")))
    (setq quote-tags (concat "quote "(read-string "Additional tags: ")))

    (request (concat tiddlywiki-base-path url-path)
    :type "PUT"
    :data (json-encode `(
        ("title" . ,quote-title)
        ("created" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
        ("modified" . ,(format-time-string "%Y%m%d%H%M%S%3N"))
        ("source" . ,quote-source)
        ("tags" . ,quote-tags)
        ("text" . ,quote-body)
        ("type" . "text/vnd.tiddlywiki")))
    :headers '(("Content-Type" . "application/json") ("X-Requested-With" . "TiddlyWiki") ("Accept" . "application/json"))
    :parser 'json-read
    :success
    (cl-function
            (lambda (&key data &allow-other-keys)
                (message "I sent: %S" (assoc-default 'args data))))
    :complete (lambda (&rest _) (message "Added quote <%s>" (symbol-value 'quote-title)))
    :error (lambda (&rest _) (message "Some error"))
    :status-code '((400 . (lambda (&rest _) (message "Got 400.")))
                    (418 . (lambda (&rest _) (message "Got 418.")))
                    (204 . (lambda (&rest _) (message "Got 202."))))
    )
)

Code Snippet 19: Directly add new quotes from Emacs

I simply invoke M-x dorneanu/tiddlywiki-add-quote and read-string will ask for a quote title, some source of the quote (e.g. a book) and of course the actual text.

Hydra

I’ve recently discovered hydra and I came up with some hydra also for TiddlyWiki:

(defhydra hydra-tiddlywiki (:color blue :hint nil)
"
Tiddlywiki commands^
---------------------------------------------------------
_b_ Add new bookmark
_j_ Add new journal entry
_t_ Add new tiddler
_q_ Add new quote
"
  ("b" dorneanu/tiddlywiki-add-bookmark)
  ("j" vd/tw5-journal-file-by-date)
  ("q" dorneanu/tiddlywiki-add-quote)
  ("t" dorneanu/tiddlywiki-add-tiddler))

;; Keybindings
(my-leader-def
  :infix "m w"
  "h" '(hydra-tiddlywiki/body :which-key "Open Tiddlywiki hydra")
  "j" '(vd/tw5-journal-file-by-date :which-key "Create/Open TW5 Journal file")
  "s" '(my/rg-tiddlywiki-directory :which-key "Search in TW5 directory"))

Code Snippet 20: Hydra for Tiddlywiki

This way I press M m w h and the TiddlyWiki hydra will pop up.

Conclusion

I hope some day there will be a full (elisp) package for TiddlyWiki combining some of the functionalities/ideas mentioned here. If you have anything to add/share, please let me know.

Blog

Victor Dorneanu