ProjectDb

A key-value storage engine library

---

Contributing to the repo

• Install clang-format-11.
• Run ./init.sh to install pre-commit hook for formatting the code.

---

Getting Started:

Requirements

• cmake version >= 3.16.*
• gcc version >= 10.2.0 (This is needed for proper c++20 support)

Usage

To use ProjectDb, follow the following steps:

• Clone this repo into some directory (<root> for example)

• Run make init_build && make projectdb

  This will build a libprojectdb.a static library, which needs to be linked against the application that uses ProjectDb.

• Set the c++ standard with -std=gnu++20

• Add <root>/include to the include path, and #include <projectdb/projectdb.h> in places where ProjectDb is used

• Add libprojectdb.a to the linker path, and add -lprojectdb -lpthread in at the end of link line

Example

The directory structure for this example is as following:

.
..
ProjectDbTest
    main.cpp .............> This is the application that's trying to use ProjectDb
ProjectDb ................> This is cloned ProjectDb repo 
    cmake-build-release
        libprojectdb.a ...> This is generated after running "make init_build && make projectdb"

main.cpp, which represents the application, is as following:

// main.cpp

#include "projectdb/projectdb.h"

int main() {
    // A config file template can be found in ./config/config.template
    // To use a customized config, create object with:
    // ProjectDb::ProjectDb db {"<config_file_path>"};
    projectdb::ProjectDb db;
    db.set("Hello", "World!");
    return 0;
}

The command used to build main.cpp is as following:

g++ -std=gnu++20 -I../ProjectDb/include main.cpp -o main \
    -L../ProjectDb/cmake-build-release/ -lprojectdb -lpthread

Current Limitations

• The current ProjectDb apis are not thread safe by itself. ProjectDb object needs to be locked by user if the apis are called in a multi-threaded fashion.

• The current implementation doesn’t provide multi-process support.

  It is important to note that only one ProjectDb should be created for one DB_FILE_PATH, otherwise, there’s a risk of data corruption.

---

Evaluating from Docker

• Build the Docker using docker build .

• Open an interactive bash with current directory mounted to projectdb: docker run -it --rm -v ${PWD}:/projectdb projectdb bash

---

Docs:

All the documents can be found in docs/, specifically:

• DesignDocument.pdf: Documents the description of the algorithm implemented, and some design decisions made while implementing.
• Document.pdf: Detailed document about the directory, files, classes and functions implemented.
• Measurements.pdf: Documentation about the benchmarks on the library.
• Presentation.pdf: The slides for the presentation for the project.
• ProjectProposal.pdf: The initial proposal for this project.
• Tutorial.pdf: Documents on how to use the library.

---

Coverage:

coverage

---

References

• pre-commit hook for clang-format: https://github.com/barisione/clang-format-hooks/
• paper for tech used by nosql: https://www.cs.utexas.edu/~rossbach/cs378h/papers/nosql-survey.pdf
• LevelDb: The database used at a single node of BigTable: https://github.com/google/leveldb
• Google big table: http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf
• Designing Data-Intensive Applications: p76 - p79

• Readings on future and promise: https://thispointer.com//c11-multithreading-part-9-stdasync-tutorial-example/

• Code coverage setup: https://github.com/codecov/codecov-action
• Codecov:
  • Supported languate: https://docs.codecov.io/docs/supported-languages

  • Supported xml format: https://docs.codecov.io/docs/supported-report-formats

Roadmaps

Features:

1. Support range query for read.
2. Add bloom filter.
3. Work on making this distributed.
4. …

Tests & Benchmark:

1. Add more unit tests.
2. Add more tests for crash recovery. (Need to investigate the concept of commit and rollback.)
3. Rewrite benchmark code and compare with leveldb.
4. …

Misc:

1. Add badge for code coverage.
2. …