Knowledge Base Management

Why you need this

Suppose you satisfy one or more of the following criteria:

  1. You work with a lot of code.
  2. You do scientific research and you read a lot of papers.
  3. You love to read and write.
  4. You are a project manager.
  5. You love to make TODO lists and nice schedules for your work.
  6. You are a musician who wants to maintain their own library of compositions/playlists.
  7. You attend a lot of meetings and you want a centralized way of managing your minutes.

All the above situations have something in common: it would be quite amazing and time-saving if you can set-up a centralized system, which is private to you, using which you can maintain your own knowledge base, i.e a nicely structured repository of ideas/work/papers/books, or really any other data you can think of, using which you can analyze your thoughts, collect them, and visualize them. This is exactly what a knowledge base manager does, and I’ll describe one of the best tools for this: obsidian.

The Zettelkasten Method

This is known as the Zettelkasten Method, wherein you create notes and connect them together to make a web of knowledge. Read more about this method here.

The tools

For this post, I’ll be using the following tools:

  1. obsidian.
  2. syncthing, for syncing directories between my linux machine and my Android tablet/phone.
  3. papis, for bibliography management and auto-fetching of papers from various providers.
  4. nvim, my editor. You can use anything you want. I also use the obsidian.nvim plugin to work with my Obsidian vault from nvim.
  5. git, for version control. You can skip this if you don’t want to backup your stuff to git. Or you can also use your favourite backup tool. I’ll be using git because I won’t backup any heavy-weight files (like videos or big PDFs).
This is just my setup

This post does not explain the workings of any of the mentioned tools. If you want to know how they work, just read the docs.

The setup

Create the vault

Install all these tools any way you want. Once you do this, create a new Obsidian vault (check the docs if you want to; they’re quite simple). One this is done, initialize a GitHub repository in your vault (with a corresponding remote, if you want). I’ll assume that the path of the vault is $HOME/obsidian-vault/. Add the following to your .gitignore:

.obsidian/*
!.obsidian/app.json
!.obsidian/appearance.json
!.obsidian/config
!.obsidian/community-plugins.json
!.obsidian/core-plugins.json
!.obsidian/graph.json
!.obsidian/hotkeys.json

Basically, ignore everything except a few configuration files which you can use to configure your Obsidian instance.

Create the papis library

The next important component of our setup will be a papis library. Since I read a lot of computer science papers, I’ll set-up a library called obsidian_research_paper_notes inside my vault, which will contain all the metadata associated to the papers which I’ve included in the library, along with any notes I need to make while reading those papers, along with (possibly hand-annotated) PDFs of the papers. For instance, here’s my papis config (both the global config and the config specific to the obsidian_research_paper_notes library):

The papis config

[settings]
add-confirm = True
browser = brave-browser
default-library = papers
editor = nvim
formatter = python
picktool = fzf
search-engine = https://www.google.com

[obsidian_research_paper_notes]
add-folder-name = {doc[title]}
dir = ~/obsidian-vault/research_paper_notes
doc-paths-lowercase = False 
multiple-authors-format = {au[family]}-{au[given]}
multiple-authors-separator = ,
notes-name = {doc[ref]}.md
notes-template = ~/obsidian-vault/research_paper_notes/notes-template.md
opentool = zathura
ref-format = '{doc[author_list][0][family]}{doc[year]}{doc[title]:.6}'

Particularly, note the add-folder-name field: instead of the default hash included in the folder name generated by papis, I use the title of the paper to name the folder in which the paper will be contained. This is useful because it makes it easier to find papers when I’m reading them on my Android tablet. I also set the doc-paths-lowercase option to False, which prevents papis from lower-casing all the path names it sees. The multiple-authors-format and multiple-authors-separator options specify how author names must be written in the metadata file (info.yaml). ref-format, which will be the ref (or citekey, as many people call it) of a particular paper in the bibliography. This is handy since I export my library in a local .bib file, which I can later use with a citation engine (eg bibtex) to automatically get citations working. Finally, for all my notes, I use a template file specified by the notes-template option of the config. Particularly, note that all notes are named using the ref of the document (we’ll see why this is useful below).

To know more about how papis configuration works, read their documentation.

A template file for markdown notes in papis

When I read papers, I also make my own notes for them (i.e papis edit --notes). I use markdown files for this, since they are the easiest to work with and render. To follow a nice template for papis notes, add a file called notes-template.md to the $HOME/obsidian-vault/research_paper_notes/ directory:

---
id: {doc[ref]} 
tags: {doc[tags]} 
authors: [{doc[author]}]
title: [{doc[title]}] 
year: {doc[year]} 
---

Essentially, whenever you’re editing notes for a particular paper, papis will automatically populate the markdown file with the title of the paper, it’s list of authors and the URL of the paper (if you’re using an importer like arxiv or doi). Moreover, this format is readily recognizable by Obsidian (works really well on the mobile application too).

Here’s the best part of using this set-up: recall from above that we named all our notes using the ref of a document. Combined with the above note template, we can now simply use the ref of any note to generate wiki-links for Obsidian! In particular, once you do this, you can use Obsidian’s graph representation to visualize the dependencies between various notes!

Ignore PDFs

I don’t want to backup any PDFs to my git repository, since I can always fetch them whenever I want. It is only the .yaml files and the notes which I want to backup. So, I add this to $HOME/obsidian-vault/research_paper_notes/.gitignore:

*.pdf

Sync across devices

I only access my vault on my linux machine, and my Android devices (a Samsung Galaxy Tab and an Android phone). For this, I use syncthing. The process is simple: just sync your Obsidian vault. Make sure that all the endpoints are set to send and receive.

However, there’s no point to sync any git specific files or other kinds of metadata to your Android device (unless you’re using termux on your device to perform git operations). So, I add a .stignore file to my vault’s root to not sync any of these files:

.gitignore
.git/
research_paper_notes/*/*.yaml

Visualize

You’re all set! Now you can freely add new papers, notes, documents, or anything else to your vault! I love to use Obsidian’s knowledge graph to visualize relationships between my ideas.

You can even annotate research paper PDFs on your Android device (I do it on my tablet using a stylus, using xodo), and the changes will reflect on your linux machine too!

results matching ""

    No results matching ""