HDTQ: Managing RDF Datasets in Compressed Space

Javier D. Fernández^{1, 2}, Miguel A. Martínez-Prieto³, Axel Polleres^{1, 2}, and Julain Reindorf¹

1. Vienna University of Economics and Business (Austria)
2. Complexity Science Hub Vienna (Austria)
3. Dept. of Computer Science, Universidad de Valladolid, Spain
firstname.lastname@wu.ac.at, migumar2@infor.uva.es, julian.reindorf@gmail.com

Go to Evaluation

This work has been submitted to ESWC 2018. In the following, we provide a brief overview of the proposal and further information regarding the empirical evaluation. Please see the “RDF/HDT” project website for a further details on HDT.

1. Introduction

HDT (Header-Dictionary-Triples) is a well-known compressed representation of RDF data that supports retrieval features without prior decompression. Yet, RDF datasets often contain additional graph information, such as the origin, version or validity time of a triple. Traditional HDT is not capable of handling this additional parameter(s).

This work introduces HDTQ (HDT Quads), an extension of HDT, which is able to represent quadruples (or quads) while still being highly compact and queryable. Two approaches of this extension, Annotated Triples and Annotated Graphs, are introduced and their performance is compared to the leading open-source RDF stores on the market. Results show that HDTQ achieves the best compression rates and is a competitive alternative to well-established systems.

2. Empirical evaluation

We evaluate the performance of HDTQ in terms of space and efficiency on quad pattern resolution. The HDTQ prototype (Get source code), built on top of the existing HDT-Java library, implements both HDT-AG and HDT-AT approaches using existing compressed bitmaps (called Roaring Bitmaps), which are optimal for sparse bitsequences.

Experiments are carried out on the following heterogeneous configuration of RDF datasets (download links are provided to get the N-Quads version of the dataset):

Note that triples are mostly present in 1 single graph in LUBM500. Repetitions are mainly due to triples specifying the type of a university. The distribution of the repetitions (number of triples ocurring with X graph) is shown below:

2.1 Space requirements

This section provides the space results (already reflected in the paper) and links to download the corresponding HDTQ files.

(*) All LUBM500 in HDTQ (in AG and AT format) are available for download.

In addition, we provide the size of the structures in HDT-AG and HDT-AT:

2.2 Indexing

2.3 Performance for Quad Pattern Resolution

In this section we provide additional information to test the performance of HDTQ and other systems (Jena and Virtuoso).

2.3.1 Get the queries

We select, for each dataset, 100 random queries for each combination of quad patterns (expect for the pattern ???? and those patterns such as ?P?? where the data distribution prevents from having 100 different queries).

Download all quad patterns.

2.3.2 Get the source code and prepare the setup

We provide a first alpha version of the HDTQ source code. While this build has been extensively tested, the current alpha state is still subject to bugs and optimizations. HDTQ content is licensed by Lesser General Public License.

We compare HDTQ against Apache Jena TDB 2.10 store (Download) and Virtuoso 7.1 Open Source (Download). For this latter, following Virtuoso instructions, we also consider a variant, named as Virtuoso+, which includes an additional index (GPOS) that may speed up quad patterns where the subject is not given. Please download and follow the Virtuoso instructions to create this additional index.

In order to alleviate the burden of executing all three systems, we have created a single jar file and a bash script to configure and run all experiments. The final lines also check that the results are the same for all systems. Please make sure transactions are disabled (by default) in all systems.

2.3.3 Raw Results

We include two JSON files that describe the results for the systems in cold and warm.

The JSON file contains the following structure:

{  
   "Dataset":{  
      "System":{  
         "Pattern":{  
            "averageTime":0,
            "Concrete measures":"[query1,query2,query100]",
            "minTime":0,
            "medianTime":0,
            "maxTime":0,
            "numResultsforEachQuery":"[query1,query2,query100]",
            "NumOfqueries":100,
            "repetitions":3
         }
      }
   }
}

2.3.4 Additional plots

In addition to the plots included in our ESWC paper, we describe here additional plots completing the experimentation. In particular, we provide the query times for LIDDI in HDT-AG: