In this post, I will share the details of my task for Week 1 of GSoC-17.
This week I ran the baseline and state-of-the-art open source codes for the existing algorithms on WN18 and FB15K dataset. The algorithms for this week included : TransE, HolE, and complex, where TransE is used as a strong baseline method for many recent research papers. WN18 and FB15K are benchmark evaluation datasets for evaluating the performance of any knowledge base embeddings algorithms. The research papers, datasets and open source codes related to our task and project can be found on our Github Wiki page here.
Motivation behind the task:
As a part of this project, I have to build a benchmark dataset for DBpedia just like WN18 and FB15K. This would be one of the key contributions of the project.
This week’s task has been essential and in line with the above goal in 3 ways:
1) Check whether code is running: It would help us to assure that the open source codes are running smoothly, and there are no existing runtime errors or bugs. Since, we would be evaluating DBpedia dataset on these codes, this becomes a mandatory task.
2) Reproducibility of the results: Before making a new DBpedia dataset straight away, this check whether the results mentioned in the research papers are reproducible for the existing benchmark datasets. This step is important because if we directly evaluate the performance of a newly created DBpedia dataset by us, and then some errors or inconsistency appears, it would be difficult to figure out if it is because of some discrepancy of code/logic or because the DBpedia dataset doesn’t follow the same trend as WN18 and FB15K datasets.
3) Finding the input data format: We would learn the format of input data for each code. This would help to build DBpedia dataset in the format which the open source code supports.
Besides the above mentioned reasons, the task has also helped to set up everything at place and understand the workings of the approaches mentioned in the papers.
i) One of the key challenges was the input format of dataset for training is not uniform for different algorithms. Even though the sample data points are same, but different algorithm use separate bin files for running their code.
For example, as mentioned by the authors in their research paper, I found their code and dataset for the baseline method TransE here. The format of the dataset given at their page doesn’t match that with what the code requires. The code requires a lot of compressed files. I had to find a compatible dataset from a different unmentioned repository here.
Similarly, for the method HolE, a separate single compressed file (different from the one used in TransE) is used for WN18 dataset. The contributor has just released the WN18 dataset in their repository, and FB15K dataset compatible with their code is not available. I tried to build a compatible bin file for FB15K for running HolE, but there were some errors in execution, I am working on it to find ways to fix it, since it would be required to run the DBpedia dataset. Only the complex method uses the standard version dataset which can also be found in our resources wiki page.
This could probably act as one of the major challenges in the project to generate a compatible version from the DBpedia dataset for evaluating these methods.