Introduction

Document Graph is a flexible persistence and modeling framework for blockchain applications.

Overview

1. Document is a node in the graph. Documents have a:

   • ID (system assigned & immutable)
   • Label
   • Owner (could be owned by a DAO)
   • Content
     • Text or Primitive type data (e.g. % interest or 2.50 USD)
     • Files, photos, videos, saved into IPFS with the CID saved in the document.
     • Any content can be optionally encrypted with a password that can be shared through a side channel.

2. Edges connect nodes. Each edge has:

   • From Node
   • To Node
   • Edge Name (type: eosio::name)
   • Owner

3. Only the owner can edit a document or an edge (e.g. only dao.hypha can edit a document created by the DHO)

4. Anyone can extend an existing graph with documents that they are the owner of. This allows for multitenant control over different data in the same graph.

5. Document Graph includes a web-based Explorer and Editor that allows non-technical users to directly create, edit, erase, and extend. Screenshot shows edit screen where user has encrypted the reference field and saved the secret key to IPFS. (not recommended, this was just to illustrate the functionality)

6. Actions can be deployed anywhere on the layer 1 to interact with the graph. The code in those contracts will modify the graph contents, but will require the permission of the owner that created it. The default contract has all of the basic actions (e.g. trigger a proposal if the right DAO/DHO/owner for edits). But it is compatible with any contracts on the layer 1, so it allows for trustless deployment of business logic that scales to the bounds of the layer 1.

7. All data in the graph is accessible via GraphQL endpoints, which can be used to integrate any client or stitched to other graph data sources.

8. The documents can be certified by anyone, this allows to ensure the content of the document

Example Graph

• member is a member of two organizations
• badges are owned by the Badge Librarian
• members are assigned badges
• member has been paid multiple times

Semantic Triples

Databases typically store data in a tabular format, where a table has set of columns and rows, like a spreadsheet.

However, IRL, we tend to think in statements of Subject, Predicate (Verb), and Object. This is particularly true in the English language where word order is subject, verb, object (SVO). This information construct is known as triples, or semantic triples. The W3C Resource Description Framework is a leading specification for organizing these statements into semantic triples.

As Lenhert describes here:

"One of the fundamental benefits of these RDF-based controlled vocabulary structures are their dual nature as being both human intelligible as well as machine readable (and, therefore, portable). Like their basic constituent subject-verb-object basis, ontologies convey rich meaning in relatively simple and compact statements, making them extremely useful in modeling complex knowledge environments."

The following diagrams express semantic triples in various terms.

Using the natural English SOV terms

Using common graph theory terms

Using the Document Graph terms

Note that edges are both directional and labeled.

Simple Example of a Semantic Triple

Blockchain Change Management

At their core, blockchains are used to manage data and state changes on that data. The structure of this data is nearly always based on a custom-built struct within a pallet or smart contract. Applications that are built on blockchains are typically built for a specific smart contract’s data structure, and the application is highly coupled to that application. When a data structure changes, it typically requires a complex migration of existing data and source code within the smart contract as well as perhaps changes to other backend or service layers and the application itself.

For example, assume the overly simplified struct within a smart contract. Upon an approval, the amount would be sent to the account indicated.

#![allow(unused)]
fn main() {
pub struct Proposal {
    pub label: str,
    pub account: AccountId,
    pub amount: u32,
}
}

Then, let's say the DAO wants to add support for recurring payments. This would need to be supported within the data model by adding a time period to use as the recurring period and the total number of payments.

#![allow(unused)]
fn main() {
pub struct Proposal {
    pub title: str,
    pub account: AccountId,
    pub amount: u32,
    pub time_period_seconds: u32,
    pub payment_count: u32,
}
}

Depending on the blockchain, this change management event requires a series of gymnastics by developers to maintain a reasonable user experience. For example, on non-upgradable platforms, it may require clients to query from both structs, or if upgradable, may require erasing data from the original data store and recreating it (losing metadata in the process). Some may require maintaining both structures and adding a roll-up of sorts to maintain client backwards compatibility.

All of these complications create a significant change management burden on software developers.

Document Graph Solves This

Document Graph alleviates the need for developers to change their persistence layer when making upgrades to their data model.

This is achieved by supporting flexible variant pairs within a consistently structured document type and support for an edge type used to link the graph's documents in meaningful ways.

Use Cases

Document Graph is currently used for a number of use cases. Here are some real-world examples that we are using it for. Interestingly, all of the below use cases can be handled within the same graph where nodes can be linked to each other across contexts.

DAOs

DAOs have have descriptive attributes. They also have members, proposals, assignments, quests, bounties, badges, and many other data types that are maintained to support a fully functional platform.

Here's an actual example of how Document Graph is used to manage a few components within a DAO's information space.

Supply Chain

We use Document Graph for a supply chain traceability application. An Item and its various Checkpoints are saved as documents. When a new QR code scan event occurs, it extends the graph for that item to contain a new Checkpoint.

Since a Checkpoint is a document, it can store any arbitrary data that describes that event, such as weight, longitude, latitude, cost, and even photos or other files such as packing lists. Files are stored in IPFS and anchored into the on-chain document.

Users have the option to save data directly onto the chain (for access within smart contracts) or to save the data to IPFS. Users also have the option to encrypt the data with a password of their choice.

Accounting

We also use the Document Graph to store data required to operate a double-entry accounting system. This system allows users to generate classical accounting reports such as Balance Sheet and Income Statement from blockchain transactions.

Accountants must be able to tag transactions with additional data, references, files, etc., and they must also be able to assign a general ledger account to appropriately calculate debits and credits.

Useful Links

Features

End-user Accessibility

The Document Graph Explorer allows for any user with a blockchain account to create and edit content that they own within the graph, and they may create between any two nodes in the graph.

This level of accessibility to non-technical users is unprecedented. It allows them to collaboratively create and connect content with all of the benefits of blockchain. This level of capability was previously only accessible to highly technical engineers.

File and IPFS Integration

Document Graph has integrated support for storing data or files within IPFS and saving that files’ CID (hash) within the document.

Encryption Support

Document Graph Explorer supports encryption of a specific content item’s value. In DGE, the user is prompted to enter a password that is used for symmetric AES encryption. This secret simply encrypts the value and the ciphertext is persisted in the document.

In a future release, we may integrate with the Khala/Phala confidential blockchain. Also, we are evaluating integration with PAD as a trustless way to share the secret in a manner that alerts the owner when the secret is accessed. This is useful for interesting use cases such as one-time decrypt use cases and “in case of emergency” use cases.

GraphQL Caching

Document Graph supports easy integration with DGraph, an open source distributed graph engine. The document graph cache listens for new blocks, and upon finalization, updates the DGraph graph to reflect any updates to the on-chain graph. DGraph has excellent tooling and ergonomics for querying, custom types or schemas, data visualizer, full-text search, and much more.

Composable SDK Experience (CLI)

Document Graph is built to be highly composable and also support an ergonomic developer experience. In addition to the Document Graph Explorer web application, there is a CLI (written in Go) that supports all of the create, read, update, and delete operations directly against the blockchain.

Plugin Architecture for Custom Renderers and Editors

Developers can include special fields within their document to enable custom viewers or editors. For example, if the field “preferred_renderer” or “preferred_editor” is populated with an endpoint, this endpoint will be used to render or edit the document.

Elastic Search

Document Graph Explorer has integrated support for using a search engine to find documents without navigating the graph.

Graph

Graph composed of Nodes and Edges

A Document Graph is a set of documents that are connected with 0..n labeled, directional edges.

Specification

• A document may have many edges with the same label.
• A document may link to another document many times.
• Edges that connect the same documents in the same direction must have unique labels.

Document Data Model

Each document is comprised of the following:

• Header
  • [required] creator (account)
  • [required] graph contract (where the document is saved)
  • [optional] created date (timepoint)
  • [optional] updated date (timepoint)
  • [optional] hash of content (not including certificates or header)
• Content
  • Value (variant) = std::variant <asset, string, time_point, name, int64>
  • Content = an optionally labeled FlexValue
  • Content Group = vector
  • Content Groups = vector
  • Each document contains a single instance of ContentGroups.
  • This provides enough flexibility to support:
    • data of all supported types,
    • short clauses of annotated text,
    • longer form sequenced data, e.g. chapters.

maybe we delete certificates? a certificate could just be another document?

• Certificates [optional]
  • Each document has O..n certificates.
  • Certificate
    • certifier: the 'signer'
    • notes: string data provided by signer
    • certification_date: time_point

Most documents have adopted a system group with type and node_label. Implementing these content items enables vastly improved usability within the DGE and related tools.

Edge Data Model

* Edge
** Creator
** From Document
** Edge Label
** To Document
** Created Date

EOSIO Implementation

For the full smart contracts, see the Github repository

EOSIO (c++) implementation of the types of the Document Graph smart contract:

FlexValue type

The FlexValue type is a variant that supports all of the common data types supported in the C++/WASM SDK for EOSIO. It stores the data as an array, where the first value is a uint8 representing the index of the type in the typedef definition, followed by the actual value.

    typedef std::variant<std::monostate, 
                         eosio::name, 
                         std::string, 
                         eosio::asset, 
                         eosio::time_point,
                         std::int64_t, 
                         eosio::checksum256>
            FlexValue;

Content type

The Content type is simply a FlexValue with an optional label. The smart contract also creates an alias for ContentGroup and ContentGroups to organize the lists of Content.

It is common for ContentGroups to be labeled by using a Content item with a label of content_group_label.

    struct Content {
        std::string label;
        FlexValue value;
    };

    using ContentGroup = std::vector<Content>;
    using ContentGroups = std::vector<ContentGroup>;

Example Content Group serialized to json

    "content_groups": [
        [
            {
                "label": "content_group_name",
                "value": [
                    "string",
                    "My Salary Content Group"
                ]
            },
            {
                "label": "salary_amount",
                "value": [
                    "asset",
                    "130.00 USD"
                ]
            },
            {
                "label": "recipient",
                "value": [
                    "name",
                    "daomember"
                ]
            }
        ]
    ]

Certificate type

The Certificate type represents an account that is certifying, or attesting to, the content within that document. It includes the account name and optional notes for the signer. Of course, the signature from the certifier is required for creation of the certificate.

Certificates will likely be replaced with verifiable credentials

    struct Certificate
    {
        eosio::name certifier;
        std::string notes;
        eosio::time_point certification_date = eosio::current_time_point();
    };

Document type

The Document type stores the header information, ContentGroups, and a list of Certificates.

    struct Document 
    {
        std::uint64_t id;
        eosio::checksum256 hash;
        eosio::name creator;
        ContentGroups content_groups;
        std::vector<Certificate> certificates;
        eosio::time_point created_date;
        eosio::name contract;
    };

Example Document serialized to json

{
    "id":4965,
    "hash":"50be6cf143050a11e9db3a52ef68e10e07b07cf6cc68007ad46a14baf307c5b9",
    "creator": "bob",
    "content_groups": [
        [
            {
                "label": "content_group_name",
                "value": [
                    "string",
                    "My Content Group #1"
                ]
            },
            {
                "label": "salary_amount",
                "value": [
                    "asset",
                    "130.00 USD"
                ]
            },
            {
                "label": "referrer",
                "value": [
                    "name",
                    "friendacct"
                ]
            },
            {
                "label": "vote_count",
                "value": [
                    "int64",
                    67
                ]
            }
        ]
    ],
    "certificates":[],
    "created_date":"2021-01-12T18:21:10.000",
    "contract":"dao.hypha"
}

Substrate Implementation (coming soon)

Getting Started

Prerequisites

Readme

This apply for the following actions: Create, Edit, Delete, Extend,Certify ¹

• A Telos account
• Anchor wallet installed

¹ You do not need the prerequisite to navigate through the Graph

Features

• Read document
• Create document
• Extend document
• Delete document
• Certify document
• Navigation through URL
• Custom GraphQL endpoints
• Elastic Search Support
• Save content in IPFS
• Encrypt data

How to create a document

The Document Graph Explorer allows the creation of new documents for any user. For that, you need to click on the button New document

Then the only thing that you need to do is fill out the form and click on the Save button. The image below show the simplest document that you can create, only need a name and node label.

After the save document, the data will be updated after the finalized block and one time that the other tool catches the changes into the blockchain you will see your first document.

Tutorials

Step by step walkthroughs

- How to create a document
- How to edit a document
- How to extend the graph
- How to erase a document or edge

Advanced topics

- Using IPFS to mask content that should not be immutable
- Using the markdown editor
- Encryption
- Searching contents of linked IPFS documents

Create your first document

Once you finish the steps of getting started, you should be able to create your first document, and here we are going to explain how to do it.

Before you start

You can access the Document Graph Explorer on any of these endpoints

• Hashed System
• Telos Kitchen

Make sure you have a TELOS account and Anchor wallet up and running!

Now it's time!

When you open one of the Document Graph Explorer you'll see something like this:

What you are seeing is the general view of all documents stored at the Document Graph, feel free to play with it and navigate in all the documents and edges.

Anyway, once you played a bit with it it's time to write your own document!

To do that, you'll need to press the Sign in button in the top right corner. Once you pressed it you'll prompt to an anchor

Accept the identity authority to access all options in the Document Graph Explorer.

The steps to create a document are very straightforward, go ahead and press new the buttom that says New Document

Once you press it, you'll get a menu that looks pretty much like this

In that screen you can see the following information:

• document type
• content groups
• node label
• contents

Each concept can be described as follows

Now that is settle, let's continue with our document

Letter to Santa Claus

Lets make our first document be a letter to Santa Claus.

To do that, you should be at the create document screen,

1. Write in the document type letter and press enter.
2. Press the edit button on the nodeLabel key.
3. Change Value to Letter from and your name.

you should get something like this.

Then press the buttom New content group and name it as body

Once you made that, go ahead and add a new content for the new content group, write in the key salutation and in the value Dear santa,. Make sure the data type is String and click save.

Now lets create a new content called main.body and in the value write your letter.

At the end you should have something like this:

To conclude push the save buttom in the bottom left corner and sign the transaction with your wallet.

Congratulations! You just made your first document!

Also, you may not see your document as soon as you press the save button, its a small delay between the buttom press and the block validation of the blockchain, so be patient!

Edit a document

Oh wait! It looks like we miss some information on the letter! We forgot to write the destination!

Let's use the search bar to find our document, just type letter from and your name to find it. Once you found it, click it (log in if you have not done it yet) and press the edit buttom on the bottom of the page

It only will show up if you are the creator of the document and you are logged in.

Now lets make a new content group called inside_address

Then create a content called recipient and write Santa Claus in the value.

Create one last content called address and write North Pole.

You should have something like this

To save it, press the save buttom and sign the transaction with your anchor wallet.

We did it! Now we edited our first document!

Extend the graph

Now its time to extend the graph

Lets use the letter we made previously.

Search for it on the search bar and open the document, you shall see a buttom called extend.

Go ahead and press it. Now you should see a similar page of create a new document but with a new component that says Edge Name

What extend does is create a new document with and edge pointing to the previous document, Document Graph is based on nodes (Documents) and edges, that's why it's called Document Graph.

So, let's write our first edge, calling it as extras.

When you write an edge name, be careful of only using 12 characters or less.

Now edit the nodeLabel value to be Extra information.

Remember that nodeLabel is the title of the document.

After you completed the steps above, make a new content group called details, there you'll be storing the following information, use the data type String for all of them.

You should end with something pretty much like this:

Click save and sign the transaction - Congrats! Now you have another document and it has a relation with the previous one.

Erase a document or edge

To erase a document is straightforward.

You'll need to navigate to a document you created, then press the button erase.

A pop up should appear.

In the example it ask to write document Extra information in the white box below, it may vary depending on your document name.

Once you write the needed information you can delete the document.

THIS WILL PERMANENTLY DELETE THE DOCUMENT AND IT CANNOT BE UNDONE.

Advanced topics

Congratulations! If you are reading this part is because you are an expert creating documents on Document Graph Explorer (DGE). So now lets talk about the more advance topics that DGE offers.

Using IPFS to mask content that should not be immutable.

The very first one is using it to store the information in IPFS, that way the content stays the same as it is hashed with an unique CID and is immutable as long as the document points to the same CID.

To make that possible lets go to the old document we made (or feel free to create a new one)

If you want to implement this feature on your document, you need to edit or create a new content. Below it will appear a toggle buttom that says IPFS

Once you press that buttom, click save and now it will appear as a FILE, press it to see it's content!

Now you know how to store content as IPFS

Using the markdown editor

You may find a little plain to write your documents without a way to create content with custom views of it.

Good news! Document Graph Explorer renders contents wrote with Markdown, and its way to simple to use this feature, in your document, just create a new content and write it as you were writing any markdown document.

You can see a markdown cheat sheet if you need some examples.

PS. It may not render all of the elements and LaTex is not supported.

Encryption

Imagine you want to store something that's private and you don't want everybody to reads that. For that reason DGE offers a way to encrypt your data. With DGE you can encrypt your data with an advance encryption standard: AES-256 symmetric, that is virtually impenetrable using brute-force methods.

It sounds pretty cool, right? To do that you need to edit or create a new content. Imagine the example of extend document when we wrote the id of the card we are sending to Santa, we don't really want to show that to the world, so lets encrypt that content!

Navigate to your document using the search bar and log in with your account. Once it's done you can press the edit buttton on the bottom of the page. After that press the edit buttom next to the content you want to edit.

It should look like this:

You can see two toggle button one for IPFS and other to encrypt your data. Press the one that says Encrypt. Once you’ve selected the option, a Pop up will appear.

Next you need to provide a seed to encrypt and decrypt your data, see it as a key that would open or close the content. That key would not be stored in any place, so make sure you store it in a save place you'll remember, because once the content is encrypted it cannot be decrypted without that key!

Click Confirm. Now your content is encrypted and only you can decrypt it! You can also share the key to other people if you want them to see the content.

Searching contents of linked IPFS documents

If you are an IPFS expert or enthusiast, you may want to store a very large file. In DGE you cant point a CID in the content to point to a file stored in IPFS. Imagine you have uploaded a book or a video and you want to include it in your document.

To upload it you can use of the many IPFS node distributors that exists. Or if you have already your CID you can store it as ipfs://CID and it automatically point to that content when you press the new FILE button that will appear once you save it.

How to create a new Document

How to edit or extend a Document

How to use IPFS in a Document

How to encrypt data in a Document

dgctl is an open source, command line application for Document Graph organizations written in Go. It is also a secure wallet for managing private keys and submitted transactions.

Build

Building dgctl requires Go v15 and above.

git clone https://github.com/hashed-io/dgctl/
cd dgctl
go build
./dgctl -h

Download binary

https://github.com/hashed-io/dgctl/releases

Run help

./dgctl --help                                                                                             
Usage:
  dgctl [command]

Available Commands:
  backup      creates a local backup of current graph
  create      create an object (e.g. proposal) based on the JSON file
  edit        propose an edit to a document
  get         get objects from the on-chain smart contract
  help        Help about any command
  set         set data on the on-chain smart contract
  vault       The vault is a secure key store (wallet). Your key is stored encrypted by the passphrase.

Flags:
  -a, --active               show active objects (default true)
      --config string        config file (default is ./dgctl.yaml)
      --csv                  Output data as CSV to console - not supported on all commands yet
      --debug                Enables verbose debug messages
      --expiration int       Set time before transaction expires, in seconds. Defaults to 30 seconds. (default 30)
  -f, --file string          filename
  -h, --help                 help for dgctl
      --output-file string   Output CSV data to file - not supported on all commands yet (default "output.csv")
      --vault-file string    Wallet file that contains encrypted key material (default "./eosc-vault.json")

Use "dgctl [command] --help" for more information about a command.

Coming soon

Contributing