Introduction

SPARQL is the query language used with knowledge graphs constructed in accordance with the World Wide Web Consortium’s standards for Linked Open Data (LOD) (W3C SPARQL Working Group, 2013; DuCharme, 2013). These knowledge graphs consist of sentence-like statements (RDF triples) about entities and their properties (in the form: subject – predicate – object), defined in terms of one or more declared ontologies and vocabularies. These graphs represent a complex body of interconnected knowledge and require a relatively complex methodology for querying and exploring them, which is what SPARQL is designed to provide.

There are various other ways of structuring this kind of knowledge, ranging from relational databases to other kinds of graph databases, document databases, and key-value databases. The advantages and disadvantages of RDF triple stores, in comparison with these other approaches, have been widely discussed (Hogan et al., 2022; Groth et al., 2023). The assumption here is that RDF and SPARQL offer a sophisticated and valuable way of representing and querying data about medieval and Renaissance manuscripts, which can serve to answer complex research questions about the history and provenance of manuscripts. The Mapping Manuscript Migrations project, discussed in more detail below, found that SPARQL queries could answer a set of specific research questions considerably more effectively and fully than interfaces based on relational databases or TEI-encoded documents (Burrows et al., 2021).

SPARQL can be deployed in several different ways. It can be run against a SPARQL endpoint using a third-party Web service like Yasgui.¹ It can also be run using the native query interface to an endpoint, as is the case with Wikidata and other Wikibase implementations. Given the relative complexity and expertise required to construct these kinds of queries, other approaches have also been developed. One is to embed SPARQL queries into other forms of software, such as Sampo-UI which is used for the Mapping Manuscript Migrations service.² There have also been attempts to develop visual interfaces for constructing SPARQL queries. Some initial experiments have also been carried out in getting AI chatbots like ChatGPT to write SPARQL queries.

In recent years, several knowledge graphs of data about medieval and Renaissance manuscripts have been published to the Web which use the Linked Open Data and RDF framework. The most notable of these take the form of a standalone RDF triple store and graph database, with a SPARQL endpoint. The Mapping Manuscript Migrations project is the fullest purpose-built example of this approach and is discussed in more detail below. The Schoenberg Database of Manuscripts, which is a customized relational database, has also transformed its data to RDF and made them available through a public SPARQL endpoint.³ The Bibale database of manuscript provenance data has also transformed its data to RDF, but does not offer a SPARQL endpoint.⁴ The Biblissima service includes a standalone triple store, but its SPARQL endpoint is for internal use only and has not been made publicly available.⁵

There have also been various efforts to create Linked Open Data knowledge graphs of the content of some specific types of manuscripts, including notarial archives, Old English texts, books of hours, and music manuscripts. Projects along these lines include Searobend,⁶ Sphaera,⁷ MusicKG,⁸ and NotaryPedia.⁹ With the exception of MusicKG (Eyharabide et al., 2019), they have not made a public endpoint available for SPARQL queries.

Mapping Manuscript Migrations

The Mapping Manuscript Migrations (MMM) project aggregated manuscript provenance data exported from the Schoenberg Database, Bibale, and Medieval Manuscripts in Oxford Libraries. They were converted to RDF triples and mapped to the MMM Data Model, which was based on a combination of the FRBROO and CIDOC-CRM ontologies, with some additional entity classes and properties. The aggregated data contained more than 900,000 provenance events and over 20 million RDF triples, which could be explored through a SPARQL endpoint as well as a browsing and filtering interface (Koho et al., 2022).

This interface uses the Sampo-UI software developed at Aalto University in Finland (Ikkala et al., 2022). It leverages the relationships in the MMM Data Model to enable an extensive range of filter combinations. These might include finding manuscripts of St Augustine’s De civitate Dei produced in France before 1200 and last recorded in North America. The results can be displayed as lists and in the form of map-based visualizations of production places, last-known locations, and the movements between them.¹⁰ Sampo-UI is built on embedded SPARQL queries, which are not initially visible to the user. It is possible, however, to inspect the underlying queries as well as to amend them and re-run them directly.

The MMM project also tested the capabilities of direct queries against its SPARQL endpoint, both in comparison with the Sampo-UI interface, and in comparison with the native browsing and querying interfaces of the three source datasets (Burrows, Cleaver et al., 2021; Burrows, Cleaver et al., 2022). In particular, the SPARQL endpoint enabled more quantitative and longitudinal queries, which could not be done through the Sampo-UI interface, let alone in the original databases. These queries included tracing variations in the physical dimensions of different types of liturgical manuscripts over the centuries, which showed that the ratio between height and width varied most in Books of Hours, and that this variation increased over time. Breviaries and missals were more consistent and uniform in their layout. Another query measured rates of stock retention by manuscript dealers in 20th-century Great Britain, revealing that the antiquarian bookseller Bernard Quaritch Ltd. kept manuscripts in stock for a much longer period, on average, and advertised them less frequently, than the rival London firm owned by James Tregaskis.

Queries run against the MMM SPARQL endpoint were also extended with contexual data from Wikidata to show the occupations and birthplaces of 19th-century British manuscript collectors. Birthplaces were visualized on a map to give an indication of the number of collectors who came from North-Western industrial centres rather than London and the South-East. Visualizations of queries like these are dependent on the SPARQL query tool used; the MMM project used the Yasgui tool to produce charts, graphs, and bubble charts, as well as timelines.¹¹ The results can also be exported to interfaces which can produce map-based and network visualizations. The MMM project tested the export of SPARQL query data to software like ResearchSpace and nodegoat; the latter environment includes a time slider which can show change over time, a feature not available in Sampo-UI or Yasgui (Burrows et al., 2020).

An important aspect of the MMM project’s experience with SPARQL was the time and training required to produce these results. Taking real research queries, members of the project team worked through them at weekly online workshops over a period of 18 months, with guidance of a SPARQL expert. Related outputs include a SPARQL tutorial based on the MMM data, and published versions of the SPARQL queries themselves as well as the result sets.¹²

Wikidata

Wikidata is one of the world’s biggest public knowledge graphs, with more than 124 million items and more than 1.44 billion statements or triples (Vrandečić et al., 2023).¹³ The Wikidata Query Service is one of the most active SPARQL endpoints, with more than 10 million queries per day as of April 2021.¹⁴ Its coverage of manuscripts is still relatively limited, although some institutions—notably the Bodleian Library, the Koninklijke Bibliotheek (KB) of the Netherlands, and the National Library of Wales—have been running projects to upload manuscript records (about 1,300, 1,260 and 590 respectively). The KB has also created Wikidata records for about 490 alba amicorum (“friendship books”).¹⁵ The schema for manuscript records is somewhat confused and inconsistent, but a recent project is aiming to improve this (Poulter, 2021).¹⁶ In any case, most existing records are far from being full catalogue records; their main contents are inventory numbers (including shelf marks), external identifiers, and links to digitized versions.

The existing corpus of records supports only limited in-depth SPARQL-based semantic exploration and reasoning, especially of a quantitative kind, since the content of the records varies considerably. Most of the entries from the National Library of Wales, for example, include production (‘inception’ in Wikidata terminology) dates, languages, dimensions, number of pages, genre, and main subject. Most of the Bodleian Library entries do not give any of these, with the exception of production dates and languages. A small group of the Bodleian records shows previous owners, while only one of the Welsh manuscripts (Black Book of Carmarthen) does. The KB helpfully provides a series of sample SPARQL queries for exploring its manuscript entries, including a timeline visualization using Histropedia.¹⁷

In combination with other SPARQL endpoints, Wikidata can be queried for contextual data about the people and organizations connected with manuscripts, as demonstrated in the MMM investigations above, as well as for the works, texts, and images the manuscripts carry, and their topics, subjects, and knowledge structures. Wikidata’s ability to bring together different identifiers relating to the same manuscript could also be leveraged in the future to join up multiple descriptions of the same manuscript, representations (digital or physical) of that manuscript, and the provenance evidence and scholarship relating to that manuscript.

Wikibase

The MediaWiki software used for Wikidata has been made available as an Open Source download, as well as through the Wikibase Cloud service hosted by the German Wikimedia community.¹⁸ Each Wikibase includes a SPARQL endpoint and a query interface. Among the groups experimenting with the Wikibase Cloud service are the British Library’s Archives and Manuscripts, the Koninklijke Bibliotheek, the Medieval Mining Texts project, and the Mapping Manuscript Migrations project.¹⁹ None of these projects have gone much beyond creating lists of properties and some basic manuscript records, though most provide one or more sample SPARQL queries for testing.

The exception is the redeveloped Digital Scriptorium, a union catalogue of medieval and Renaissance manuscripts in North American libraries (Koho et al., 2023). The data model used for this project is object-centric (rather than event-centric like Mapping Manuscript Migrations) and focuses on the manuscript metadata record. As part of the evaluation of the prototype of this service, a set of 24 SPARQL queries was developed and run. These were intended to mimic searching and browsing, either by finding manuscript records with a specific value for a given metadata element (name, place, subject, and so on), or by finding all manuscript records with any value for that element. The SPARQL queries used for this prototype testing have been made available through a GitHub repository.²⁰

Visual Interfaces for SPARQL Queries

As the amount of available Linked Open Data increases, and the number of SPARQL endpoints grows, there is a growing need for advanced methods of searching and browsing across these knowledge graphs. Only in this way can the full richness and sophistication of the data models be unlocked. Two developments for the future refinement and improvement of SPARQL querying are of significance here. The first involves visual interfaces, and the second focuses on AI-based approaches to query formulation. Of these, the former is more advanced, while the latter has only really emerged since ChatGPT was publicly released in November 2022.

In this context, the term ‘visual interfaces’ refers to visual methods for constructing and conceptualizing SPARQL queries, as distinct from the visual presentation of knowledge graphs through software like Sampo-UI. Most of these interfaces work specifically with the Wikidata SPARQL endpoint. Wikidata itself offers a Query Helper, which provides a way of creating queries using drop-down menus to find and combine properties and entity classes.²¹ More recently, it also added a Query Builder for creating simple SPARQL queries by combining properties from drop-down menus.²²

In November 2023, two leading Semantic Web companies—metaphacts and Ontotext—combined to launch a free public interface to the Wikidata knowledge graph, combining their respective metaphactory and GraphDB products.²³ Starting with a keyword search for a specific entity or class, it is then possible to explore graphically the various connections step-by-step. Although there are limitations in the number of connections which can be displayed, and familiarity with the Wikidata entity and property schemas is very helpful, this service does provide a useful visual approach to researching manuscript collections (as far as they are documented in Wikidata). You can build provenance trails for individual manuscripts and collections, for example, as well as explore the overlapping connections between manuscript owners and their membership of relevant organizations. Results can also be displayed as a list or explored for direct connections using the ‘Pathfinder’ feature.

This product bears some resemblance to the ResearchSpace software, originally developed by metaphacts in association with the British Museum. Designed specifically to work with the CIDOC-CRM ontology, ResearchSpace also enables queries to be built up visually from an entity or class by prompting the user with the available properties at each step of the graph. The Mapping Manuscript Migrations project experimented with a version of ResearchSpace, importing all the MMM data, setting up some basic query paths, and producing some visualizations. The ResearchSpace community is primarily associated with museum collections, though there have been a couple of manuscript-related projects.²⁴

There have been various other efforts to develop visual query builders for SPARQL, sometimes as research projects and sometimes for specific institutions or collections. An interesting recent example from the library world is SPARNATURAL, which includes demonstrators for the Bibliothèque nationale de France and the Archives nationales de France (Francart, 2023). The look of the interface derives ultimately from ResearchSpace but it can be configured to work with various data models and endpoints. A useful feature is the ability to display the actual SPARQL query.

Using AI for SPARQL Queries

The public release of chatbots based on Large Language Models (LLMs), beginning with ChatGPT in November 2022, has prompted a great deal of discussion and experimentation around their implications for building and exploring knowledge graphs (Idehen, 2023; Petkova, 2023). Since these AI services have been, amongst many other things, promoted as a means of carrying out elaborate and complex human tasks like writing, composing music, and image-making, it might also be feasible for a chatbot to enable a non-expert user to construct SPARQL queries, without the degree of knowledge required to create such queries from scratch. One of the examples given on the ChatGPT ‘Overview’ page is ‘write an SQL query’, so we might expect AI chatbots to tackle SPARQL queries too.²⁵

GPT-4 (the most sophisticated version of the GPT family, available in a free public version in Microsoft’s Copilot)²⁶ can be prompted to construct SPARQL queries for the Wikidata Query Service. I asked GPT-4 to ‘write a Wikidata SPARQL query to find manuscripts owned by persons who became members of the Roxburghe Club in the 19th century, displaying their names and membership dates where available’. In the resulting SPARQL query (Appendix 1), the entity reference (Wikidata QID) given for the Roxburghe Club was wrong, as was one of the properties, and the query asked for dates of birth rather than dates of membership. The same request was run four times in Copilot/GPT-4, which returned a different query structure each time. None of these queries ran successfully against the Wikidata Query Service, even after the entity references and properties were corrected. This experiment showed that while GPT-4 knows enough about SPARQL and Wikidata to write a plausible query, it seems to have a problem with identifying QIDs for specific entities, as well as with the Wikidata query syntax. The resulting queries needed thorough debugging which required a significant level of SPARQL knowledge and was not something that a SPARQL beginner could simply run successfully.

Trying to extend GPT-4’s SPARQL knowledge to endpoints other than Wikidata also proved problematic. I asked Copilot to ‘write a SPARQL query to run against the ‘Mapping Manuscript Migrations’ SPARQL endpoint, to find all manuscripts (manifestation singletons) owned by Henry Yates Thompson’. I also told it to use the classes, properties, and namespace prefixes listed in the MMM schema.²⁷ The response (Appendix 2) was a SPARQL query which failed to define two of its prefixes (ecrm and mmms). When these definitions were added, the query still failed to produce any results, partly because the wrong property was being used to link owner and manuscript, and partly because the wrong method was being used to specify Thompson as the owner.

I also asked GPT-4 to write a SPARQL query to run against the Digital Scriptorium Wikibase, giving it the address of the SPARQL endpoint and asking for a query to find all manuscripts held by the University of Pennsylvania. I also referred it to the classes, properties, and namespace prefixes listed in Koho et al., 2023 for the query syntax. The response (Appendix 3) was unsatisfactory in various ways and produced no results:

• The QID for the University of Pennyslvania was incorrect;

• The property used was incorrect;

• The URL defining the 'ds:' prefix does not exist.

At present, then, while GPT-4 can provide a starting-point for constructing Wikidata SPARQL queries, these still require careful review and debugging. It appears to have difficulty formulating workable queries for other SPARQL endpoints. These were only basic, preliminary investigations, however, and better results might well be achieved using other LLMs like Mistral or Claude, through advanced prompting techniques and customization, or by fine-tuning the LLMs for specific tasks, perhaps in conjunction with a SPARQL validator. The level of technical knowledge involved in implementing these approaches is likely to be beyond the capability of the typical humanities researcher or librarian and would probably require investigation by appropriate experts.

Given that incorporating knowledge graphs into LLM training is a hot topic in generative AI research, it is highly likely that future iterations of LLMs will be able to respond more effectively to this kind of task. A more immediately promising development would seem to be the integration of LLMs into software like metaphactory, a commercial solution for building and exploring LOD knowledge graphs. In October 2023, a metaphactory app was released for beta testing, which translates natural language queries into SPARQL queries.²⁸ While it uses the public version of ChatGPT for this purpose, the translation process can be controlled from within the metaphactory software.

Issues Affecting the Usefulness of SPARQL

The Mapping Manuscript Migrations (MMM) experience demonstrated how the effectiveness and usefulness of SPARQL queries can be affected by uncertainties in data and data structures relevant to medieval and Renaissance manuscripts. There are many ambiguities inherent in manuscript descriptions: production dates are often only estimates, for example, while production places can usually only be assigned at a regional or country level, sometimes involving multiple possibilities. Other uncertainties relate to gaps in the chain of provenance evidence, such as markings in a manuscript which record only a name or family crest. Dealing with fragments of manuscripts is another challenge, as is the splitting and combining of manuscript volumes over time.

Ambiguity in data models is another critical factor. The Schoenberg Database records the appearance of manuscripts in sales and auction catalogues but does not usually include information about whether the manuscript was sold, or to whom. This inherent ambiguity meant that the CIDOC-CRM ‘Transfer of Custody’ event class (E10) could not be automatically applied to these occurrences. A generic MMM-specific entity class ‘ManuscriptActivity’ had to be created specifically for these events; this affected about two-thirds of all provenance events in the MMM data. Similarly, MMM managed to extract more than 20,000 provenance statements from the Bodleian Library’s catalogue (Burrows et al., 2021). But only the TEI <origin> element could be mapped to a specific event type in the CIDOC-CRM ontology: E12_Production. All other provenance events had to be mapped to the ‘ManuscriptActivity’ class. This significantly limited the degree of specificity with which the Oxford data could be analysed and visualized.

In Wikidata’s data model, authors should not be attached directly to manuscripts, but be linked indirectly via their works contained in a manuscript. This approach has not been adopted consistently, however. The relationship between works and authors is also problematic in the Schoenberg Database, where each of the multiple works in a single manuscript is not specifically linked to an author, rather a list of the multiple authors merely sits alongside the list of works. These approaches make it significantly more complicated to construct SPARQL queries involving authors of works carried by manuscripts.

There are a number of possible ways to improve the value of manuscript data for analysis through SPARQL queries (Burrows, 2023). Using a suitable vocabulary for different types of provenance event would be valuable, though the typology provided by the CIDOC-CRM ontology might need to be extended to cover situations such as ‘offered for sale’ (Bekiari et al., 2021:64–7). Adding roles to the occurrences of persons and organizations in provenance statements would support more granular and nuanced analyses. For CIDOC-CRM, some additions might be needed to extend its limited set of properties, which currently include ‘P51_has_former_or_current_owner’ and ‘P52_has_current_owner’.

Other improvements might include using date ranges (1300–1325) rather than text-based statements such as ‘circa 1310’, or using an uncertainty model like CIDOC CRM’s Temporal Relation Primitives (Bekiari et al., 2021:43–6). Embedding Uniform Resource Indentifiers (URIs) from widely used vocabularies for entities like persons, organizations, and places—such as VIAF, Wikidata, and GeoNames—could also enrich queries around authors, collectors and owners. Publishing Linked Open Data vocabularies for medieval and Renaissance persons, places, and works would help with this (Burrows, 2022). A universal approach to assigning manuscript identifiers (such as that proposed by the ISMI initiative) is vital for linking disparate data relating to a specific manuscript (Bougard et al., 2020).

Conclusion

Knowledge graphs constructed with RDF triples and the Linked Open Data framework have enabled the development of rich and complex datasets about medieval and Renaissance manuscripts in recent years. While these datasets may have user-friendly interfaces for searching and browsing, exploring their full potential ultimately needs a SPARQL endpoint and the SPARQL query language. SPARQL queries can explore sophisticated relationships within the data as well as enabling linked queries across multiple datasets of this kind. The complexity of the data structures is likely to require a complex query, however, and SPARQL is somewhat notorious for its apparent level of difficulty. A significant amount of training is recommended, together with access to expert advice.

Several projects—notably Mapping Manuscript Migrations and Digital Scriptorium 2.0—have used SPARQL queries systematically and successfully for testing, exploration, and diagnosis. These approaches have demonstrated SPARQL’s potential value for manuscript studies. Various initiatives are underway to improve the usability of SPARQL through the development of visual query interfaces and the embedding of queries into other software environments. While AI chatbots like GPT-4 may be able to help with writing SPARQL queries, they are currently quite limited in their ability to do this.

Wikidata and its associated Wikibase projects are the most significant area of development at present, although the relatively limited amount of manuscript data currently only supports fairly basic SPARQL queries. Once the Wikidata data model for manuscripts is improved and more libraries start contributing data, there is significant potential for sophisticated SPARQL queries through the Wikidata Query Service and its Wikibase equivalents.

Acknowledgements

The Mapping Manuscript Migrations project was funded by the Trans-Atlantic Partnership under its Digging into Data (Round 4) program. The partners in this project were the University of Oxford, the University of Pennsylvania, Aalto University in collaboration with University of Helsinki (HELDIG), and the Institut de recherche et d’histoire des textes. Funding was provided by the UK Economic and Social Research Council, the Institute of Museum and Library Services, the Academy of Finland, and the Agence nationale de la recherche.

Competing Interests

The author has no competing interests to declare.