Paper Machines is an open-source extension for the Zotero bibliographic management software. Its purpose is to allow individual researchers to generate analyses and visualizations of user-provided corpora, without requiring extensive computational resources or technical knowledge.
This project is a collaboration between historian Jo Guldi and digital ethnomusicologist Chris Johnson-Roberson, graciously supported by Google Summer of Code, the William F. Milton Fund, and metaLAB @ Harvard.
NOTE: Paper Machines now bundles Jython 2.7b1 to ensure broader compatibility. If you encounter problems using the extension, please create an issue describing what operating system and version of Java you have installed, and the nature of the issue.
Additionally, if you have Paper Machines 0.3.6 installed and wish to upgrade to Zotero 4.0, you will need to manually install the new version from here. Paper Machines 0.4.x users will receive updates automatically.
In order to run Paper Machines, you will need the following (Java should be installed automatically on Mac OS X 10.6-10.7. If you are running Mac OS 10.8, please download it from the link below):
- Zotero with PDF indexing tools installed (see the Search pane of Zotero's Preferences)
- a corpus of documents with full text PDF/HTML and high-quality metadata (recommended: at least 1,000 for topic modeling purposes)
- Java (download page)
Paper Machines should work either in Zotero for Firefox or Zotero Standalone. To install, you must download the XPI file. If you wish to use the extension in the Standalone version, right-click on the link and save the XPI file in your Downloads folder. Then, in Zotero Standalone, go to the Tools menu -> Add-Ons. Select the gear icon at the right, then "Install Add-On From File." Navigate to your Downloads folder (or wherever you have saved the XPI file) and open it.
To begin, right-click (control-click for Mac) on the collection you wish to analyze and select "Extract Texts for Paper Machines." Once the extraction process is complete, this right-click menu will offer several different processes that may be run on a collection, each with an accompanying visualization. Once these processes have been run, selecting "Export Output of Paper Machines..." will allow you to choose which visualizations to export.
Displays words scaled according to the frequency of their occurrence. An oft-maligned, but still arguably useful way to get a quick impression of the most common words in your collection. Either a basic word cloud, a word cloud with tfidf filtering to remove unimportant words, or multiple word clouds (divided up by subcollection or time interval, specified in days) can be generated. The multiple word clouds can be filtered using tfidf, Dunning's log-likelihood, or Mann-Whitney U tests, each of which will provide different results depending on the data. By default, a basic word cloud will appear in the Tags pane of Zotero once text has been extracted.
Finds phrases that follow a certain pattern, such as "x and y," and displays the most common pairings. This method is derived from a Many Eyes visualization).
Generates a map linking texts from their places of publication to the places they mention, filtered by time.
Generates a map showing regions of relative intensity for mentions in the text. Same as the flight path visualization without the link data; may be more usable on large datasets).
Creates a CSV file with place name, latitude/longitude, the Zotero item ID number, and some context around the mention.
Annotates files using the DBpedia Spotlight service, providing a look at what named entities (people, places, organizations, etc.) are mentioned in the texts. Entities are scaled according to the frequency of their occurrence.
Shows the proportional prevalence of different "topics" (collections of words likely to co-occur) in the corpus, by time or by subcollection. This uses the MALLET package to perform latent Dirichlet allocation, and by default displays the 5 most "coherent" topics, based on a metric devised by Mimno et al. A variety of topic model hyperparameters can be specified before the model is created.
After the model is generated, clicking "Save" in display will open a new window with the graph displayed free of interactive controls; this window may be saved as an ".SVG" file or captured via screenshot. It will also, in the original window, preserve the current selection of topics, search terms, and time scale as a permalink; please bookmark this if you wish to return to a specific view with interactive controls intact.
The topic model can be supplemented with datasets from JSTOR Data For Research. You must first register for an account, after which you may search for additional articles based on keywords, years of publiation, specific journals, and so on. Once the search is to your liking, go to the Dataset Requests menu at the upper right and click "Submit New Request." Check the "Citations" and "Word Counts" boxes, select CSV output format, and enter a short job title that describes your query. Once you click "Submit Job", you will be taken to a history of your submitted requests. You will be e-mailed once the dataset is complete. Click "Download (#### docs)" in the Full Dataset column, and a zip file timestamped with the request time will be downloaded. This file (or several files with related queries) may then be incorporated into a model by selecting "By Time (With JSTOR DFR)" in the Topic Modeling submenu of Paper Machines. Multiple dataset zips will be merged and duplicates discarded before analysis begins; be warned, this may take a considerable amount of time before it begins to show progress (~15-30 minutes).
Currently, the language stoplist in use, types of data to extract, and default parameters for topic modeling may be adjusted in the preference pane. Any custom stopwords may be added to the "Stop Words" pane, one per line, to help eliminate irrelevant terms from your data.
Special thanks to Matthew Battles for providing space, guidance, and support for me at metaLAB. My gratitude also to the creators of all the open-source projects and services upon which this project relies: