Social Dynamics in the Ancient Mediterranean

sddk python package

Fri, 15 May 2020 15:00:00 +0000

Introduction

Soon after we started our SDAM project, we began to discuss what platform to choose as a data storage for our collaborative research projects. To begin with, we considered whether our needs could be fully met by a combination of Google Drive and GitHub. But it revealed that not, at least since Github is not suitable for working with large files and Google Drive is not always easily accessible programmatically and everything there tends to be too much fluid to consider it an appropriate data storage for research purposes.

Therefore, as researchers based at a Danish research institution, we were really excited by discovering sciencedata.dk, a service for storing, sharing, synchronizing and searching research data managed by DeiC. Reading the documentation, we realized that sciencedata relies on WebDAV API¹, what means that it might be also accessed programmatically via any service supporting HTTP protocol. Within our project, we work with our data mainly within R and Python, both of which support the HTTP protocol, so it was a way to go.

Realizing this, I was interested in finding the smoothiest way for exporting any data from my python environment directly to sciencedata.dk to be instantly accessible for reuse by me or anybody else within our project. With this task at hands, I started to develop the sddk python package. To explain my motivation, let me to give you a couple of examples of how I work.

Workflow

For instance, in one of our empirical research projects, we have developed a series of Jupyter Notebooks, which we used to extract, transform and clean a dataset of +80,000 Roman inscriptions. I usually run these notebooks on Google Colab online platform and its cloud servers, what means that I can easily access them regardless where I physically sit and what machine I actually use (we also share these notebooks openly on Github in this repo).

Within Python, I usually work with my data in a form of pandas DataFrame object, which is officially defined as a “two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). “ (see the docs here). Using this format, we have a table with more than 80,000 rows and 40 columns which might be easily used for any filtering, grouping or stasticial analysis.

To use this dataset in more than one Jupyter notebook and by other users as well, it has to be exported into a file. I usually export such data into .json files, since they are easily readible by other programming languages, like R.

In the example case of Roman inscriptions such a data file has ca 150 MB, but in other projects we do (like AGT) it might have even more than 1 GB. Since I work on cloud, it does not make too much sense for me to download these files into my machine, have them saved somewhere locally and then to think how to share such files with my collegues. Instead of that, I send them instatly into a team shared folder on sciencedata.dk, where anyone from our team can find it and reuse it following her or his needs. Thus, the data just go from one server (Google servers running Colab) to another (sciencedata.dk), without even touching my personal computer.

Generally speaking, sddk package is nothing more than a series of functions helping us to make this process as much easy as possible, using just one easily memorable piece of code to send the data there (sddk.write_file() function) and an analogical code to read them back (sddk.read_file() function). It deserves to be describe in more detail.

Sddk package usage demonstration

To make these functions working, you first need to install & import the package and to specify your sciencedata.dk credentials and the desired target folder for your subsequent requests using sddk.configure() function. This folder might be either your personal, or a shared one owned by some of your colleagues, as in the example below. For a shared folder called "team_folder", owned by user with username "team_folder_owner@username.inst", you would run this:

!pip install sddk # install the package
import sddk # import the packg
conf = sddk.configure("team_folder", "team_folder_owner@username.inst")

sddk.configure() is designed in a way that any project member can use exactly the same piece of code accessing the shared folder, regardless whether she or he is owner or ordinary user of the folder. Notice that goes against the internal structure of sciencedata.dk, which uses a different syntax for folders you are sharing with others (“sharingout” keyword in the HTTP request path) and folders which are shared with you by others (“sharingin” keyword in the HTTP request path). These things are treated internally in the package, since we consider more practical to have the exactly same command for both cases (the credentials are entered interatively, so the code for both cases can remain the the same) .

After successful configuration, you can write or read any number of files like in the above.

# define the object:
dataframe_object = pd.DataFrame([("a1", "b1", "c1"), ("a2", "b2", "c2")], columns=["a", "b", "c"]) 
# send the object to file:
sddk.write_file("df_file.json", dataframe_object, conf)

Later on, any other user with an access to the same folder can read the file back to python with:

dataframe_object = sddk.read_file("df_file.json", "df", conf)

The package enables you export some other types of Python objects into a couple of different formats. For instance, you can use jsons for lists or dictionaries. You can also export dataframes into csv files or feather files. You can also export strings into txt files or images into ong files. The syntax for all these cases is exactly the same as in the command above, with two minor differences:

in sddk.write_file() you have to take care for the filename extension,
in sddk.read_file() you have to always specify the python internal object you want to produce (“df”, “str”, “list”, and “dict” are currently supported python objects).

For more examples, see the documentation or watch a video demonstration.

Sciencedata.dk and R

As I have already mentioned, next to Python, our project is also using R. While we so far do not have a full-fledged R package, we already have a request() function for reading files from sciencedata and writing them back there. (see its documentation here). In combination with some other commands, the request function might be used to read a .json (exported from pandas dataframe) into R tidiverse tibble object and to export such an object back to sciencedata as a json file interpretable by Python as pandas dataframe (see the R example code). The circle is complete.

“WebDAV (RFC 4918) is an extension to HTTP, the protocol that web-browsers and webservers use to communicate with each other. The WebDAV protocol enables a webserver to behave like a fileserver too, supporting collaborative authoring of web content. WebDAV extends the set of standard HTTP methods and headers to provide the ability to create a file or folder, edit a file in place, copy or move or delete a file, etc. As an extension to HTTP, WebDAV normally uses port 80 for unencrypted access and port 443 (HTTPS) for secure access.” WebDAV: What it is, where it turns up, and its alternatives. ↩

Testing our shared infrastructure

Mon, 20 Apr 2020 12:00:00 +0000

Post-lockdown Lessons About Shared Collaboration Infrastructure

The lockdown has put tremendous pressure on most collaborative projects by pushing them to go remote. In our case too, it tested our ability to use shared collaboration infrastructures, communication channels and management skills under conditions for which these were in fact intended: to facilitate uninterrupted remote and asynchronous work. Some projects struggled more because the infrastructure they were locked into was not designed for collaboration - as Mine Cetinkaya-Rundel says on 25 March 2020 on twitter: “If this pandemic doesn’t make people completely give up on Microsoft’s collaboration tools, I don’t know what will.”

We selected our infrastructure with eye to collaboration in the first place! But just like you don’t know whether you have a good backup until you’ve had to restore from it, nothing tests your infrastructure for asynchronous remote work better than a lockdown.

During covid19 lockdown in Denmark (15 March - 15 April), the lack of childcare for half the team meant inability to focus on extended computational or other work tasks and rendered face to face time difficult to arrange before 9pm. While other team members were theoretically available to work from home, lack of home office made extended work in uncomfortable positions hazardous to future health. Voila, ideal testing conditions were born.

Before we even knew of covid, we set up the following infrastructure to facilitate our collaboration on flexible schedules and while travelling. We had months to get used to it before lockdown imposed it mercilessly and split us by distance (V. locked in Czech republic), time (A and A working under limited conditions due childcare) and other issues (P. had a festering leg injury which prevented her movement).

The project infrastructure included :

Business G Suite for collaborative project administration, communication, cowriting of research documentation etc. (1 month) after previously using Vojtech’s G Suite for previous 6 months.
sdam-au github organisation containing repos with code related to projects;
SDAM shared group on sciencedata.dk for collaborative online storage that is shareable with non-AU stakeholders yet complies with GDPR and safe data storage and is computationally accessible
sdam.au.dk website (since project start/ 7 months)
JIRA for task management (launched shortly after project start/ 6 months, but not used seriously until in February.)
Slack channel for quick and ephemeral communication (since project start/ 7 months)
Outlook email account for official business with faculty etc. (since project start/ 7 months),

Team members expressed familiarity with most of these tools or the ability to acquire it independently. Some specific tools, such as GoogleForms or JIRA or interactions with sciencedata.dk, were less well-known and different teammembers developed tools and documentation or organized instructional sessions whenever the team needed to upskill.

In addition to the infrastructure, we had also discussed the priorities for offline/remote or asynchronous work being:

Monitoring of project channels to stay in loop and be aware of fires
Prioritizing to help others
Staying on track with tasks in Jira (mapped against project targets)
Documenting own work in shared project spaces online so as to keep it as FAIR as possible

While infrastructure kept functioning, covid revealed other deficiencies in our project coordination, which I now want to write down antidotes to:

Maintain presence in your project communication channels: while working at computer was not feasible (e.g. A and A), Adela installed Slack on her phone, allowing her to respond to queries while in the playground, while Antonio could not. Phone apps can be useful! if you know how to switch the notifications off.
Manage expectations: if you cannot be in on those channels, it is better and perfectly legitimate during times of duress (such as covid) to be straightforward and say, ‘I am completely off communication channels’ or ‘I won’t be online for the next two weeks’ rather than maintain a vague semblance of work and then have the team be wondering where you’ve disappeared to.
Manage your tasks: given new circumstances, check about work priorities every time you (virtually) meet, and before you plunge into something new.
Use of project infrastructure properly and document your work to be reusable by others when you are unavailable: granted, this takes some thinking and testing ahead of time but hey, our project happens to be precisely about that ;).

First peak at multigraph

Tue, 10 Mar 2020 12:00:00 +0000

Climbing the Mountain

Some while ago, I saw this comic about the never-ending story of learning programming. The never-ending story applies to any collaborative discovery endeavour. At SDAM, we have been agonizing over what can we actually do with all the large data, once we aggregated it. It’s a question that’s been giving us anxiety, because until you harvest, streamline and wrangle your data, you cannot even start answering it. Just like a potter who collects clay from an unknown source without knowing whether it will be good for anything. You knead and levigate the cold, hard mass and only as it warms up and softens with your effort, you may gauge its qualities. Is it plastic and malleable enough to make a small sculpture with fine detail or coarse and bulky to make a big chunky dinosaur? Playing with large datasets is similar. You spend hours poking and probing its features in order to learn what it can be used for and how much effort it’ll likely take.
So far we have been following Vojtech’s python scripts in slicing and subsetting the EDH dataset and discovering what problems we need to overcome. The choice is wide; there are the incomparable diversity of chronological intervals ascribed to individual inscriptions, that we will need to refine. While working on these preliminaries, we have all been anxious about what will we do next, once we figure out a way to make chronology more tractable? Who will think of the right question and who will make it answerable with our toolkit of python and R? We saw a little bit of the possible world today after running through as of yet imperfect dataset with the multigraph package in R. We used a similarity function to assess the correspondence among inscription attributes and create a network of these similarities. The revelation to me, a newbie to networks, was that with multigraph we can choose which columns we wish to pass to R for assessment, and then display the result either as a collated network with links relating to different attributes, or filter out individual network components (for each attribute we are evaluating) and assess them in their own right. Once we implement a good (transparent and quantifiable) measure of temporal intervals and a bit of control over coordinates, we are a few steps away from analysing the diffusion of epigraphic habits over time. Maybe this sounds simple, but I learnt something today.

Lemmatization and POStagging of Ancient Greek Texts

Fri, 31 Jan 2020 12:00:00 +0000

Introduction

In the second half of January, while I was working with Vojtěch Linka on a quantitative text analysis of the concept of pain in Corpus Aristotelicum and Corpus Hippocraticum, we realized that the data which I regularly use for this type of tasks (i.e. Lemmatized Ancient Greek dataset, LAG) are substantially incomplete. We noticed this fact once we were working with Aristotelian texts (tlg0086), as we realized that the dataset completely misses such works as Ethica Nicomachea and Ethica Eudemia - works really crucial for our task at hands!

This was quite surprising finding, since from the LAG documentation I had an impression that it combines all Greek texts currently available in the Perseus Digital Library with all texts from the First1KGreek project. However, this is not the case. First1KGreek explicitly focuses upon “texts that do not already exist in the Perseus Digital Library” (see here). But it appears that in the case of Aristotle it includes only works from the First1KGreek corpus, while there are some other Aristotle’s works in the PerseusDL not included in First1KGreek .

Compiling a new dataset of all publicly available ancient Greek texts

This led me to an urgent need to develop our own comprehensive dataset of ancient Greek text combining PerseusDL and First1KGreek. Just to combine these two dataset was quite straightforward. I developed my own code being able to parse xml files for these (or any similar) datasets directly from GitHub (i.e. without locally cleaning/downloaded the repo).

My code works like this: For a chosen GitHub repository, it firstly recursively extracts paths for all files it contains (by browsing a repo folder by folder, using a loop over a PyGitHub package object). Subsequently, it uses the returned list of paths to parse all xml files.

### Lemmatization and POStagging using CLTK

But what I really need is not only a dataset of all machine-readible texts, but a lemmatized and POStagged dataset, organized in a similar way as LAG (i.e. for each wordform containing its lemmatization and part of speech parsing).

To proceed with this task, I firstly turned to CLTK and its Greek module, since I knew that it includes a number of useful functions for both lemmatization and POStagging. However, after some testing, I realized that the results are unsufficient.

For instance, once we apply the cltk function lemmatizer.lemmatize() upon all words in Aristotle’s works and calculate word frequencies, we can identify there is more than 5,000 instances of the term ζεύς. This is in striking contrast with TLG, which reports only 48 instances of the same lemma for Aristle. This discrepancy is caused by misidentification of the term διά , which in most instances means simply “through”. In this particular case, the problem might be solved by making a stopwords filtering before the lemmatization, but this is not an universal solution.

Using cltk for POStagging revealed even more complicated. After doing some preliminary cleaning of all Aristotle work, the cltk greek tagger was still able to parse only 56% of words. Perhaps I could obtain a bit better results by some parametrization of these functions, but I guess the improvement would not be substantial. But first of all, it would require to make a much more detailed inspection of the code behind these cltk functions, which is much more granular and complex than I am used to work with.

Lemmatization and POStagging with Morpheus

With these unsufficient results, I decided to develop my own functions for lemmatization, postagging and automatic translation. The main tool for this was the Morpheus Dictionary, available as a xml file for instance from here. This file contains 958,476 unique tokens. An entry for each token has the following structure:

 <t>
    <i>1631</i> 			# word-form-id
    <f>ἀγακλέᾰς</f>		# word-form
    <b>αγακλεᾰς</b>		# word-form-without-diacritics
    <l>ἀγακλεής</l>		# lemma
    <e>αγακλεης</e>		# lemma-without-diacritics
    <p>a-p---ma-</p>	# morphological-analysis
    <d>76</d>					# lemma-id
    <s>very glorious, famous</s> 	# translation
    <a>doric aeolic</a> 					# dialect 
  </t>

I parsed this structure into two python dictionaries:

The first dictionary (morpheus_dict) has as its keys all unique word forms, both with and without diacritics. It has 706,506 key-value pairs. For each key, a value is a list of all possible tokens. In most cases, there is just one element in the list. Thus:

morpheus_dict["λόγος"]
>>> [{'a': None,
  'b': 'λογος',
  'd': '19869',
  'e': 'λογος',
  'f': 'λόγος',
  'i': '542404',
  'l': 'λόγος',
  'p': 'n-s---mn-',
  's': 'the word'}]

This simple navigation is then used in a series of functions drawing upon it.

The second dictionary (morpheus_by_lemma) is much more simple, since it contains only information associated with each individual lemma (and not every wordform). It is especially useful for getting a simple translation of a lemmatized word:

morpheus_by_lemma["λόγος"]
>>> [{'a': None, 'd': 19869, 'l': 'λόγος', 's': 'the word'}]

Functions for lemmatization, POStagging and translation of ancient Greek

On the top is the function get_lemmatized_sentences(string, all_lemmata=False, filter_by_postag=None, involve_unknown=False).

As an input, it receives a raw Greek text of any kind and extent. Such input is processed by a series of subsequent functions embedded within each other:

(1) get_sentences() splits the string into sentences by common sentence separators.

(2) lemmatize_string(sentence) first calls tokenize_string(), which makes a basic cleaning and stopwords filtering for the sentence, and returns a list of words. Subsequently, each word from the tokenized sentence is sent either to return_first_lemma() or to return_all_unique_lemmata(), on the basis of the value of the parameter all_lemmata= (set to False by default).

(4) return_all_unique_lemmata()goes to the morpheus_dict values and returns all unique lemmata.

(5) Parameter filter_by_postag= (default None) enables to sub-select chosen word types from the tokens, on the basis of first character in the tag “p” . Thus, to choose only nouns, adjectives, and verbs, you can set filter_by_postag=["n", "a", "v"].

Next to the lemmatization, there is also a series of functions for translations, like return_all_unique_translations(word, filter_by_postag=None, involve_unknown=False), useful for any wordform, and lemma_translator(word), where we already have a lemma.

As a next step, I will put these functions into a package. Perhaps some of my functions could also be implemented into the cltk package. Hopefully I will soon update this post with a link to a repo containing its code and refs.

Epigraphic Database Heidelberg using R

Tue, 26 Nov 2019 12:00:00 +0000

This post is about accessing the “Epigraphic Database Heidelberg” (EDH), which is one of the longest running database projects in digital Latin epigraphy. The [EDH] database started as early as year 1986, and in 1997 the Epigraphic Database Heidelberg website was launched at https:/edh-www.adw.uni-heidelberg.de where inscriptions, images, bibliographic and geographic records can be searched and browsed online.

Despite the possibility of accessing the [EDH] database through a Web browser, it is many times convenient to get the Open Data Repository by the [EDH] through its public Application Programming Interface (API).

For inscriptions, the generic search pattern Uniform Resource Identifier (URI) is:

https://edh-www.adw.uni-heidelberg.de/data/api/inscriptions/search?par_1=value&par_2=value&par_n=value

with parameters par 1, 2, … n.

The response from a query is in a Java Script Object Notation (JSON) format such as:

{
   "total" : 61,
   "limit" : "20",
   "items" : [ ... ]
}

Accessing the EDH database using R

Accessing the [EDH] database [API] using [R] is possible with a convenient function that produces the generic search pattern [URI]. Hence, the function get.edh() allows having access to the data with the available parameters that are recorded as arguments. Then the returned [JSON] file is converted into a list data object with function fromJSON() from the [rjson] package.

Basically, the function get.edh() allows getting data with the search parameter either from "inscriptions" (the default option) or else from "geography". The other two options from the [EDH] database [API], which are "photos" and "bibliography" may be implemented in the future.

The following parameter description is from https://edh-www.adw.uni-heidelberg.de/data/api:

Search parameters for inscriptions and geography

province:

get list of valid values at https://edh-www.adw.uni-heidelberg.de/data/api/terms/province, case insensitive

country:

get list of valid values at https://edh-www.adw.uni-heidelberg.de/data/api/terms/country, case insensitive

findspot_modern:

add leading and/or trailing truncation by asterisk *, e.g. findspot_modern=köln*, case insensitive

findspot_ancient:

add leading and/or trailing truncation by asterisk *, e.g. findspot_ancient=aquae*, case insensitive

bbox:

bounding box in the format bbox=minLong, minLat, maxLong, maxLat, example: https://edh-www.adw.uni-heidelberg.de/data/api/inscriptions/search?bbox=11,47,12,48

Just make sure to quote the arguments in get.edh() for the different parameters that are not integers. This means for example that the query for the last parameter with the two search options is written as

R> get.edh(search="inscriptions", bbox="11,47,12,48")
R> get.edh(search="geography", bbox="11,47,12,48")

Search parameters for inscriptions

hd_nr:

HD-No of inscription

year_not_before:

integer, BC years are negative integers

year_not_after:

integer, BC years are negative integers

tm_nr:

integer value

transcription:

automatic leading and trailing truncation, brackets are ignored

type:

of inscription, get list of values at https://edh-www.adw.uni-heidelberg.de/data/api/terms/type, case insensitive

Search parameters for geography

findspot:

level of village, street etc.; add leading and/or trailing truncation by asterisk *, e.g. findspot_modern=köln*, case insensitive

pleiades_id:

Pleiades identifier of a place; integer value

geonames_id:

Geonames identifier of a place; integer value

Since with the "inscriptions" option the id “component” of the output list is not with a numeric format, then function get.edh() adds an ID at the beginning of the list with the identifier with a numerical format.

Hence, the query

R> get.edh(findspot_modern="madrid")

returns this truncated output:

$ID
[1] "041220"

$commentary
[1] " Verschollen. Mögliche Datierung: 99-100."

$country
[1] "Spain"

$diplomatic_text
[1] "[ ] / [ ] / [ ] / GER PO[ ]TIF / [ ] / [ ] / [ ] / ["

...

Having a numerical identifier is useful for plotting the results for example. However, it is possible to prevent this addition by disabling argument addID with FALSE.

R> get.edh(findspot_modern="madrid", addID=FALSE)

Finally, it is worth to mention that further extensions that the [EDH] database [API] may add in the future can be handled with similar arguments in function get.edh().

In search of good practices in digital research

Mon, 04 Nov 2019 12:00:00 +0000

In the world of digital scholarship, I count myself among rather late adopters. It was during my doctoral studies when I first tried to learn some programming stuff. It took me another three years until I was able to write and run my own first programming scripts in Python. However, since I reached that stage a couple of years ago, I quickly became a programming-enthusiast, fully immersed in the exciting world of web scraping, exploring countless datasets available via GitHub, and always experimenting with new Python packages and libraries.

Here I differentiate between two idealized types of behavior: computational skills and computational practices. A computational skill is, for instance, an ability to write a script implementing particular algorithm to solve a given problem; on the other hand, a computational practice is, for instance, usage of version control for managing scripts. While someone’s computational skills might be operationalized as a variable with values represented as positive numbers (e.g. “0” = “no skills”, “1” = “at least some skill”), computational practices rather differ in being worse or better, and therefore might be better operationalized by a set of real numbers (e.g. “-1” = “somehow bad practice”, “1” = “somehow good practice”). (Of course these two types of behavior often overlap, but I do not think this anyhow threatens the core of my argument.)

For now, my observation is following: while my computational skills were gradually growing over the last couple of years as I learnt new tricks almost on a daily basis, my computational practices have not been under such a vivid development and still tend to include many bad habits.

I think I have an explanation for that: I hypothesize that it is because skills and practices are learnt differently. In my case, a new good practice has been often simply transmitted by collaboration on a joint project and by imitating what other people do. This way I learnt to use GitHub, just by following up what one my colleague did once we were writing a joint article. On the other hand, new skills are typically not transmitted this way. I suggest that it is because they often require a long-term focused incremental learning. This way I learned Python, following a number of online tutorials, guiding me step-by-step from one level to another.

I now feel really lucky to be a part of our research project here in Aarhus with other like-minded scholars. I see it not only as a great opportunity to employ the computational skills I possess, but - perhaps even more importantly - as an opportunity to make collectively a substantial progress in our search for good practices in digital research. In fact, it was one of the main motivations for me for deciding to move here. It was important for me to realize that I am not the only one who struggles on this front and to discover that there already exists a whole bunch of literature on good practices in scientific computing (like this article, this article, or this article)

Short report: User Acceptance Testing

Fri, 18 Oct 2019 12:00:00 +0000

User Acceptance Test or UAT is a type of testing performed by the Client to certify the system with respect to the requirements that was agreed upon. This report is to illustrate the application of UAT in developing, testing, and releasing a new or updated R package.

User Acceptance Test for R package release

In the DTAP process (development → test → acceptance → production), which is part of a software development process, the release of a new version of an R package is at the final step where the deployment needs to meet certain requirements. The “acceptance” portion in this process comes from both humans (developer and clients) and machines (a list of rules that the software must conform). Of course, human testing involves machines as well, and this distinction is made for referring to a systematic way of testing.

Human testing

The development of a software component on a local machine has folders containing a set of functions in a structure like this

FOLDER-BUILD
FOLDER-PUB
Batch file to build

where the testing of the software occurs in the three parts.

A set of R functions are written in FOLDER-BUILD with a version control, and then the functions are tested manually in the R console or other environment. It is important at this stage that the test involves random data, different data sets, and the extreme cases that the input data may have. Once there is an acceptance from the developer, FOLDER-PUB hosts a copy of the functions to publish and then the building with the batch file.

The batch file has the instructions in a sequential list to build the package that typically are R core functions to load files, write or recreate an object to a file, and the creation of a skeleton for a new source package.

Machine testing

Once the package is built then comes the documentation of the functions with files in a Latex format that conform the manual. The machine testing, which comes afterwards, will check among other things whether there is a correspondence or not between the script codes and the documentation.

Rtools allows performing a machine testing of R packages within the MS Windows operating system. First the package is constructed as a tarball file with the command line and by typing R CMD build. Then the R CMD check performs different types of tests on this file based on pass/fail results where “fail” involves errors, warnings, and notes. The option --as-cran applies the most strict rules on the package and allows a successful submission for a publication on the CRAN repository whenever there is any fail.

An example of a successful testing is in the .log file given as appendix where the checking starts in the DESCRIPTION file with the basic information about the package. Then there is also a testing of things like whether the package can be loaded and unloaded, and consistencies both in the code and the documentation including meta data.

UAT for users in GitHub

As with machine testing, the acceptance tests among users of the R package should reveal as well a straightforward yes/no or pass/fail results. This means that the user should be able to download, install, uninstal, and run the software without any errors.

Another important component of the user acceptance is counting with a shared infrastructure that team members administer, use on a day-to-day basis and can reflect on and implement for others. GitHub is not only a git repository, but it is also a tool suitable for educational tasks such as periodical reflection and implementation for others.

GitHub storages the information and it can be used as well as backup for disaster recovery. In many cases GitHub is the place where the users report bugs, and where ideally users should become developers of the package as well.

UAT exercise

install the beta version of the package from GitHub plus dependencies into the R environment
load the package and run the scripts of the README.md file
run the program with different datasets
- if an error occurs then consult the manual to fix the input data
- if the data introduced conforms the requirements from the manual and the error remains then report the bug

A successful user acceptance test has no errors, and the user is willing to use the package in his/her own work.

Upcoming workshop on Digital Approaches

Tue, 15 Oct 2019 12:00:00 +0000

Hello, dear digital enthousiast!

We are organising a one day workshop on Digital Approaches to research in Humanities and Social Sciences which you should not miss out! You will hear about the need of building the technical infrastructure, the frustrations of venturing the digital research in the more traditional and conservative fields such as history or epigraphy. We will discuss the joys of collaboration, data standards and dataset sharing. On the top of that, you will learn more about the exciting ongoing digital projects all around the globe!

The full program with interactive links to our speakers and their projects can be accessed via this link.

Please, contact me if you need more details (petra.hermankova@cas.au.dk or @pettulda on Twitter!) or if you would like to join us for pre-workshop dinner on the 29th Oct.

30 Oct 2019 Workshop program

Learning to collaborate

Tue, 08 Oct 2019 12:00:00 +0000

Collaboration - digital infrastructure

We are one month and three meetings into the project.

Having had several discussions on the subject of digital infrastructure (understand as the research data storage, system for communicating across different google and outlook identities and the need to keep each other efficiently in the loop but also keep our workplace informed) at SDAM, our team often struggles with different opinions and different priorities when it comes to discussing and deciding on digital infrastructure. Decision making is difficult because we are facing multiple unknowns at once. We do not know local (Danish) resources well, we don’t know each other well, and in the realm we each are not certain of each others expertise in this murky digital area. We are basically still stocktaking in many aspects of our eresearch. My push towards decision making is perhaps counterproductive as we are still in the information gathering and trialling stage of the innovation adoption process. It may be too early to commit as we don’t understand all the costs and benefits of the different available infrastructures. We are circling around the basic principles (be exemplary, promote open source, collaboration-friendly online data storage systems that are modular and let us connect data and analytical scripts) but have not committed fully to the principles of openness (ie. github seems too open for the scripts/datasets) nor have we found cloud data storage in DK that would tick both the boxes of functionality (plugin tools), responsiveness (answering email queries) or sufficient data storage shareable with international collaborators. Perhaps, data storage is tricky and needs a bit more reflection and testing. Other matters, such as team management and coordination are split into several different tools, such as trello and slack. While I can live with Slack I am less enthused about Trello, mostly because it is yet another system to keep an eye on. As project director, I am happy to go along with any new app that someone installs on my computer and sets to notifications and provides instructions and support when asked. As project director, I am most anxious about getting a group email address set up so I type one address and everyone gets the memo. The same applies to calendars, I would like to know when is who available and to make my calendar visible to team members see where I am and what I am doing without me sending lots and lots of invites constantly. This makes sense as at the moment I sit on a lot of logistical and event information while feeling time-strapped. Basic project management and day to day operation sits on Google at the moment. While I had suggested we purchase membership for business apps, and have own institutional space, this suggestion was voted down at the initial meeting in favor of a better Danish cloud platform (e.g. data DeiC or sciencedata.dk). I suspect if I had purchased the business apps before asking everyone, we would be using that space now instead of private personal space. I see no problem with using a business Google apps as a temporary placeholder before we move to something else; as currently we are using all kinds of different storage (my fedarch Gdrive, MQ Gdrive, Cloudstor, Vojtech’s Gdrive, Github, et al) and it’s going to be a hell to streamline this down the line. I suppose the source of tension for me here is that we as a project should not be using personal digital resources but institutional ones (to be exemplary). Another source of tension is my preference to commit to an imperfect tool with the understanding that we switch as soon as we identify a better tool for the job. I think it poor practice to resort to a non-committal limbo stage where we sort of play with datasets from different parts without having or at least trying to build any containing structures. Maybe I exaggerate, but it sure feels more straightforward to manage the provenance and move resources from one space to another as long as they are organized to begin with. All in all, I perceive distinct lack of engagement on the digital infrastructure part of our work compared to research proper. If I put my foot down regarding GDrive three weeks ago, we’d probably be using it now despite concerns over new unnecessary emails, etc. Despite reservations, the team mostly happily delegates decision making and implementation to whoever volunteers for the task. It’s low hanging fruit for whoever is just a bit more comfortable with the tools, or knowledgable, or feels more strongly about the subject, or simply has a more discrete idea of desired outcome. I wonder whether this is because of the differences in technical expertise or because for researchers digital infrastructure is, after all, secondary. There is a reason for this situation. Most postdocs today have been trained as individual researchers who by dissertation stage have developed effective survival working habits and strategies and are justified in believing that the infrastructure they have operated within is sufficient. For individual tasks, it most probably is. For collaborative group work, however, is most probably isn’t.

Collaborative projects are more complex than individual projects as the needs and ideas of the different participants are more differentiated. Successful operation of a collaborative project requires frequent expression and juxtaposition of everyone’s needs, constant testing of these needs (to combat ‘language games’) and their prioritisation. Decision-making process is thus not a linear, informed judgement that follows directly from the information gathering and trialing stage, but a trade of resulting from the juxtaposition of individual opinions and balancing of governing principles of the project. Following the motto of The Bazaar and the Cathedral, this more complex process has the advantage of producing higher quality, better informed and less buggy solutions. The disadvantage is that it is slowed down by the complexity of decision-making amidst language games. For me, the complexity of communicating and operating in a collaborative project is precisely the reason why we need to emphasise and prioritize development of proper digital infrastructure. In contrast to individual projects, the potential for technical debt increases dramatically in collaborations. Technical debt, as defined by wikipedia, is the “implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer”. This concept, first coined by Ward Cunningham, applies both to software development and collaboration management (e.g. Stewart-Lowndes). Tolerating poor data storage now commits the team to time-consuming data transfer and documentation later. Poor/weak team coordination may cause a bit of frustration here and there today, but leads to missed opportunities in a week or month exacerbated by new pressures or additional stress. Tackling technical debt requires the refactoring of systems for more clarity and streamlined performance as these systems (be they programs or infrastructures) grow more complex. Inevitably, we are humans and therefore imperfect and so are our decisions. This is doubly true with digital infrastructure which is a constantly moving landscape, or better a constantly changing tip of an iceberg. Mindful of the imperfection, we can nonetheless mitigate it and ward off technical debt whenever possible. One such strategy is to invest considerable effort into scoping, selecting and implementing infrastructure solutions that will make the future work of the entire team easier. Settling down for a limited temporary solution or regressing into individual solutions is tantamount to shooting ourselves in the foot.

Decision making - Am I turning into a Dane?

I suspect my team think we discuss digital infrastructure too much. Wouldn’t it be better to play with data first? It might be more fun. It would certainly relieve some of the stress from deliverables. Yet we need to build the project slowly. Perhaps I should have instituted the infrastructure by fiat and started playing. However I wish the decisions to be the consensual product of joint deliberations over individual and project needs and usecases. Likewise I hope the implementation will consist of divisible parts and present a shareable load for us all.

“I wish to play with the data first”

The wish to jump into research is justified by the need to know what it is we need stored and contained. It is perhaps also good to relieve some of the pressure and novelty of the project by resorting into individual zones of comfort, with familiar data analysis, visualisation or similar. It lets each of us bring something to the table, establish value and carry on developing as a project and a team. Imagining what we as a project need now and might need in the future is a trying exercise, and it is good to have some safety valves and comfort generating activities.

Digital research and the solitary historian

This is what I actually sat down to write about today inspired by a discussion with a colleague. The topic needed such a long introduction about the pains of collaboration that I may need to finish this topic in another reflection. In respect to the language games and complex workings of team needs and priorities, it seems that digital projects should be much more straightforward if one is an individual researcher. In a team we struggle with how to work together and how to share authority/decide what technical solution to commit to /adopt. In a team, however, we have four pairs of eyes for information gathering and testing available technical solutions. This is a tremendous advantage. Unless the single researcher has gotten a degree in history after a long career in IT, then the time saved by not figuring out a team is lost in searching for the right digital tool. If we as a team of intelligent IT savvy people struggle to decide on shared digital infrastructure, how is a single researcher supposed to navigate this terrain?

Building the infrastructure

Thu, 26 Sep 2019 12:00:00 +0000

Hello, dear reader!

The project has officially started in the beginning of September 2019 and we are already fully immersed in building the digital infrastructure of the new project!

After a careful consideration we have mutually agreed on our official name “The Social Dynamics in the Ancient Mediterranean” research group. We will get back to the name in some of the future posts but it appears you can’t really start doing anything, unless you have a name!

Even Rome wasn’t built in a day… and so wasn’t the digital infrastructure of a research project. It does not only include to research the available research tools and the existing infrastructure (pun intended), but you also have to bring to the table solutions that would a) work for the entire team and that would b) survive in a long run. Suddenly you have to think about internal communication, external presentation, storage, code and data sharing, version control, publication, shared bibliography, project management and planning, shared calendar… And all that while you are keen on doing the actual research. But again, some of the decisions you make now can impact your future research in a good and in a bad way. The lesson I have learnt in life many times is: the time you invest in making the decision upfront, however tedious it may seem, will save you a great trouble in the future.

The issue with most digital tools and software is their short lifespan. It unfortunately aplies to commercial apps as well as to open-source software. And there is not an easy escape route. Maybe tune your mindset to be ready to migrate to a completely new tool every once in a while but the change is inevitable. Therefore, even before you create something, start thinking about the other end of its lifecycle - what will happen after the current structure collapses.

“It happened to all of us. You can run out of funding, change institutions or take a career break. You abandon your project and when you want to return, it is irreversibly and truly gone.”

But honestly, how many people, unless they were burnt really badly in their past experience, think about the entire lifecycle of a digital project? Reflect on your decisions and try to answer the following questions honestly:

Do you think about the format of the data that you are going to leave behind?
Are you planning to share or reuse the data in five years?
What is your exit strategy?
Do you have one?

Although it may seem pesimistic, there is no universal solution that would fit all projects. But maybe a mere thought of the inescapable end of your project is the key itself. Building the infrastructure responsibly with the end date in mind may open the door to future projects that won’t have to reinvent the wheel and will build upon your work.

What is your experience with building the infrastructure for a research project? Please, reach out to me (petra.hermankova@cas.au.dk or @pettulda on Twitter!) if you would like to ponder about the research lifecycle and our mutual building efforts.

Or simply stay tuned for more!