library(pandoc)
pandoc_convert(file = tandf_tebc2014_1.bib,
from = "biblatex",
to = "markdown",
output = "index.qmd")When I was rebuilding my website in Quarto, I did what I often do and took a look at how James, my PhD advisor, built his website. In particular, I was looking at his publications page. It’s really nice. You can filter the results, it has links to the article, as well as the preprint, public code, and shared data if they are available. Looking at through the code in his website’s github repository, the only downside of the approach that I could see at the time was that it required building a quarto file with all study metadata as a YAML header. One for each publication, separately in a subfolder in the publications directory of my website files. For instance, the YAML block for our first paper together looks like this:
----
title: Four methods for analyzing partial interval recording data, with application
to single-case research
author:
- name:
given: James E.
family: Pustejovsky
- name:
given: Daniel M.
family: Swan
date: 2015-06-19T00:00:00
categories:
- alternating renewal process
- effect size
- response ratio
- single-case design
- behavioral observation
links:
- text: Supplementary materials
url: /files/4-PIR-Methods-Appendix.pdf
- text: Journal
url: http://doi.org/10.1080/00273171.2015.1014879
icon: newspaper
- icon: unlock-fill
text: Pre-Print
url: /files/4-PIR-methods-MBR.pdf
citation:
type: article-journal
container-title: Multivariate Behavioral Research
doi: 10.1080/00273171.2015.1014879
volume: 50
issue: 3
page: 365-380
---It seemed like a pretty tedious undertaking to hand-write all that YAML. My first thought was that there’s got to be a way to convert bibtex into YAML. Bibtex is a structured format for citation metadata, it’s useful for any paper-writing workflow that includes LaTeX. It makes in-text citations really simple, and when you knit your final document you end up with correctly-formatted citations and a complete bibliography in your desired style. These days, most journals have an option to download a citation for a paper, either as text or as a structured data file, including bibtex. That means there’s an easy way to get the citation information in structured format. In my imagined workflow using this structured metadata, I’d also want the option to batch-execute on a file containing multiple citations so that the code automatically creates a subfolder and populates it with the quarto file. I figured the issue of automatically generating a publication list for a quarto website is a solved problem. I started googling around and found some blog posts that sort of half get at what I was trying to do, but nothing that really solved my problem.
So I took a step back and looked for a solution to the more general problem of bibtex to YAML. That one turned out to be easy, Pandoc was my solution. Pandoc describes itself as the “swiss-army knife” of markup format conversions. I have some experience with Pandoc because just about any markup-to-PDF workflow makes use of Pandoc at some point. And there’s already a Pandoc package for R that lets you write Pandoc commands, but also has helper functions for most common commands, including pandoc_convert. So I grabbed the bibfile for one of my publications to test it out. I ended up with a file named tandf_tebc2014_1.bib, very descriptive!
Command ran fine, no problem! Excellent news. I opened my brand new index.qmd file and find that it was…empty? What? I tied running it again, same result. I opened the bibfile to look at it again, everything looks fine? Hmm, ok. I started looking at the arguments for the function again and see that it defaults standalone = FALSE. I figured it can’t hurt to try changing to true (I’m making standalone files after all!) and that works! Great! Okay, what did that do? The standalone argument’s documentation says “Should appropriate header and footer be included ?”
Savvier readers will probably immediately see what happened here. I’ll be honest: I was stumped why that worked for a while. I did some more google searches that turned up nothing. I briefly considered asking Stack Exchange, but the time it would take to come up with a minimal reproducible example did not feel worth it. I eventually ended up asking Claude about it, and it was a real “duh” moment. The YAML is metadata, it’s a header. Even though it’s the “body” of what I actually care about, it’s not the body of the document in markup terms. So if gave it a bibfile to convert and didn’t tell it I wanted metadata, it gave me a blank file because the bibfile doesn’t have a body. It’s all metadata. Great, problem solved.
Now I wanted to turn this into the described process that can convert multiple documents, placing them into appropriate subdirectories. I grabbed a bibfile from an old publication, because I want to make sure this process will work on something substantial. The first step was to create all the subdirectories, which needed unique and identifiable names. That meant I actually needed to be able to read all the titles of the papers from the bibtex, separately from the pandoc conversion process. A brief search turns up the bib2df package, which will read bibfile data in as a rectangular dataset. Using that package I could read all the large bibfile in, extract the titles, and then iterate across those titles to create subdirectories in a Publications folder.
library(tidyverse)
library(bib2df)
#Creates the subfolders where the citations are going to live
createFolders <- function(title, publication_folder){
title <- file.path(publication_folder, title)
dir.create(title)
}
publication_folder <- "Publications"
ref_dat <- bib2df(file.path(publication_folder, "pubs.bib"))
#create the individual folders
walk(ref_dat$TITLE, createFolders)This process immediately puked up errors because dir.create doesn’t like folder names with spaces. Okay, I fix that. Then I get another error. It turns out that a title had a character the process I built doesn’t like. Fine, I didn’t need the folder names to be exact matches, I could just delete any extra characters that were a problem. Okay, there are further irritating characters in some of the titles, so I have to iterate my cleaning process.
Oh, also, the folder names probably didn’t need to be arbitrarily long. Eventually I came up with a cleanTitle function that took the first 48 characters of each title and then deleted commas, question marks, colons, and brackets and replaces all the spaces with dashes. In this process I also made it so createFolders wouldn’t try and create a folder that already exists, and print out the names of the directories that it successfully created so I can track down errors more easily.
#removes a bunch of extra characters that might be lurking in bibtex titles
#for the purpose of creating clean subfolder names
cleanTitle <- function(title){
str_replace_all(str_remove_all(str_trim(str_sub(title, 1, 48)), "\\,|\\?|\\:|\\{|\\}"), " ", "-")
}
#Creates the subfolders where the citations are going to live
createFolders <- function(title, publication_folder){
title <- file.path(publication_folder, cleanTitle(title))
if(dir.exists(title)) return(NA) #doesn't create a folder if one already exists
dir.create(title)
print(paste("Created", title, "directory"))
}
ref_dat <- bib2df(file.path(publication_folder, "pubs.bib"))
#create the individual folders
walk(ref_dat$TITLE, createFolders)This eventually worked without error. Beautiful! Then I decided to complicate things further. I wanted to have more options for this process. In addition to being able to read a large citations file, I’d like to be able to scan all the subdirectories in Publications for a citations.bib file and have it create the appropriate index.qmd file if one doesn’t exist. This means that I needed to take my references.bib file that contains the big list of citations and write an individual bibfile in each folder. This is actually not too hard, the bib2df package has a function that will also write bibfiles. I just needed to iterate it across each row of the ref_dat dataframe.
createBibs <- function(df, publication_folder){
write_dr <- file.path(publication_folder,cleanTitle(df$TITLE))
if(file.exists(file.path(write_dr,"citation.bib"))) return (NA) #doesn't create a bibfile if one already exists
df2bib(df, file = file.path(write_dr, "citation.bib"))
}
ref_dat |>
rowwise() |>
group_walk(~createBibs(.x))This part was overall pretty painless, I’d already solved a lot of the problems that might have cropped up here. Now I just needed to iterate across the subfolders in Publications, and write the quarto/YAML file to any directory that has a bibfile and doesn’t already have one.
convertBibs <- function(folder){
#aborts early if the folder doesn't have any citations
if(!file.exists(file.path(folder, "citation.bib"))) return(NA)
pandoc_convert(file = file.path(folder, "citation.bib"), from = "biblatex", to = "markdown", standalone = TRUE, output = file.path(folder, "index.qmd"))
}
#Get all the top-level directory names
dirs <- list.dirs(publication_folder, recursive = FALSE)
#Get only those folders that don't presently have a qmd file
dirs <- dirs[!file.exists(file.path(dirs, "index.qmd"))]
walk(dirs, convertBibs)This worked great!! However, I didn’t actually want to create a Publications folder full of citations related to single-case designs that I didn’t author. Now that this process seemed to be working, I deleted all the test directories and compiled a references bibfile of just my publications. It is only five citations long. I haven’t been actively publishing recently, and I don’t have the kind of CV that James does. Hmm…
Anyway, after I complied my bibfile, I returned to James’ website code and figured out that I need to snag a javascript template and a few other files to make the whole thing work. I put them in all the appropriate places, hit preview, and…the Publications page was blank. I dug into some code warnings and eventually I figured out that the way the Pandoc turns the bibtex into YAML has a slightly different hierarchical structure than the javascript template I’ve borrowed from James. I was getting kind of tired of this problem, so I asked Claude for help and it offered me a very complex solution that involved reading the index.qmd files in as raw text, rearranging it, then using the yaml library to make some further edits. The solution seemed needlessly complex, so I took the revelation of the yaml library as an opportunity to do some slightly-simpler rearranging.
reorderYaml <- function(folder) {
qmd_path <- file.path(folder, "index.qmd")
if (!file.exists(qmd_path)) return(NA)
yaml_block <- yaml.load_file(qmd_path)
yaml_block <- yaml_block$references[[1]]
yaml_block$citation$type <- yaml_block$type
yaml_block$citation$`container-title` <- yaml_block$`container-title`
yaml_block$citation$url <- yaml_block$url
yaml_block$citation$doi <- yaml_block$doi
yaml_block$citation$volume <- yaml_block$volume
yaml_block$citation$page <- yaml_block$page
yaml_block$type <- NULL
yaml_block$`container-title` <- NULL
yaml_block$url <- NULL
yaml_block$doi <- NULL
yaml_block$volume <- NULL
yaml_block$page <- NULL
write_lines(c("---", as.yaml(yaml_block), "---"), qmd_path)
}
walk(dirs, reorderYaml)This worked. When I go to preview the site, the citations all show up. But the overall look of the page is, frankly, ugly as hell. It turned out that James had done some extra CSS work for his publications page to look the way it does, and I wasn’t entirely sure where to start. Plus, if I wanted the links to various things to show up nicely on my page, I’m still going to have to do a lot of tedious YAML editing. I asked Claude for some assistance again, but the answers I got were not especially helpful.
At this point I take stock. I only have five publications. I don’t have anything in the pipeline. Do I need a Publications page that I can dynamically update in a blog-like fashion? I do not. What I need is to have a finished Publications page. As it turns out, the best solution was a simple one.
Just a plain old html page created in markdown. Any future publications can be added easily. It was a little difficult to throw out hours of effort and use the approach I ended up with, but I’m familiar enough with the sunk cost fallacy to know that it wasn’t going to worth the additional effort to get the more complicated solution working for me.
That said, I don’t regret the work I did. I knew when I started out that I was interested in the problem as an exercise as much as anything else. And I learned a little more about Pandoc as well as working with bibfiles and YAML in R. However, when I started out I didn’t consider my personal scope (five publications) and whether the effort was likely to be worth it for the final outcome. It can be easy to get ahead of yourself in a technical problem if you don’t think carefully about how large or small the scope of what you’re trying to do is. Not every solution has to scale! Sometimes you’re just trying to solve a small problem, and you can use a small solution.