Implementation

Schema

Data Structures

Asset Collection

Asset

Content

A Content is an abstraction for stored data. Contents are immutable.

• Contents are stored in the bucket and their metadata is stored in the Contents Table in the database
• Contents can be of many types depending on where they are stored
  • file-content
  • sql-content
  • url-content
  • docker-content etc.

Managed-Content vs Proxy-Content

• Managed-Content is where the storage of the underlying data is in control of the asset-manager. This is the default content type that gets created when you add any file / data to an asset.
• Proxy-Content is where users actively manage where their data is stored but still want to use asset-manager for storage, retrieval, sharing and version control of data. For example, users may want to store data in google buckets and directly access those (without using asset-manager). This situation is especially useful where teams want to have minimal friction of using a data-handling tool like asset-manager. In other cases, users may want to add pre-existing cloud data without increase any cost of additional storage. For example, dna_sequence raw data can often be in size of multiple magnitudes of terabytes, any duplication of which (in creating to managed-content) can add large cost overheads.
• A key limitation of using proxy-content is that asset-manager can not guarantee availability of data since users can modify / delete the data at source. However, asset-manager can still be used to index the proxy-contents for lineage tracing in experiments.

Calculating ID for Content

• The id of a managed-content is a combination of its md5 hash of data and its storage system i.e. gs:proxy_md5_gDsxxxxxxxx5fH==. The rationale behind composite key is to allow the same content to be stored in multiple locations if required. For example: depending on the use case, we may want to have a copy of output stored in gcs bucket and in a database or bigquery.
• On the other hand, the id of proxy-content has the following form gs:proxy_md5_gDsxxxxxxxx5fH==. The proxy flag shows that it's a proxy content. However, unlike the managed-content, the md5 hash here is a hash of the string src_url:content_hash.

Object

Asset holds a collection of objects. An Object is a relation between Content and a file path inside the asset-repo dir of the user.

• Object to Content is a many-to-one relationship i.e. multiple Objects can point to the same content
• Each record in the Objects Table has a foreign Key to the Contents Table.
• Object to Asset is a many-to-many relationship i.e. an Asset holds multiple Objects and the same object can be shared by multiple Assets.
• The AssetObjects Join table maintains the relationship between Objects and Assets
• Objects are stored in the Objects Table in the database and in the asset-manifest.yaml file in the bucket

Creating Asset

Initializing an Asset

cd your_dir
ama init class_name

The following steps are performed.

Create Repo instance

• The Repo class checks if we are currently at the root or inside an already existing asset-repo
• If no asset-repos are found, the Repo class initializes a repo by creating a .assets directory at

Create Asset instance

• We create a new asset-manifest.yaml file at the root of the asset-repo, this file is a serialized representation of the Asset instance
• We create cache directory inside the .assets dir (if not already exists). This is used to cache all files added to an asset

How do we find if there are uncommitted changes.

• When users add a new file to an asset, if the asset_id is not null, we move asset_id to parent_asset_id and make asset_id as null.
• we retain the seq_id, since this asset will be a new version of the parent
• When the user removes a file, we check if the parent_id is not null, we compare the asset's hash to parent's hash, if they are same, we make parent_id as asset_id and parent.parent_id as the asset.parent_id
• How do we compare two assets, we compare the assets by their hash which is a hash of all objects in the asset
• What's a commit-hash, a commit-hash is the hash of the asset which has been successfully committed to the database

If the user calls ama init

• make asset_id as null - server will assign asset_id
• make parent as null - server will assign parent id
• make seq_id as null - the server will assign seq_id

Adding files to an asset

ama add output/logs/annotation.log

The following actions are performed to add a file to an asset.

• Verifies if the file exists at the given path
• Calculates hash of the file and checks if a content with hash is already registered.
• If content exists, use the already stored content else create a new content and add to the contents (set) variable in the asset.
• Cache the added file by creating a hard-link from the file into the path. Given the large size of files to be handled by AMA we opted for hard-links instead of copying files.
• Contents can be of many types depending on where they are stored i.e. file-content, sql-content, url-content, docker-content etc.

Ensuring atomicity of asset

Think of scenario, user A adds few files to an asset, commits the asset but there is an interruption in the commit flow. User A, then adds a few more files to the asset and commits the asset again.

Therefore, for every commit, ama should be able to detect:

• If the asset is inheriting from another asset, whether that asset was committed successfully.
• If the parent asset was committed, we can proceed with the regular flow.
• However, if the previous commit was not successful.
  • We should ask the user, if the changes are part of the previous interrupted transaction
  • If user answers - yes, then we can proceed with committing the existing asset
  • If the user answers - No, then we need to initialize a new-asset, set parent to null and commit that asset

Ensuring content integrity

A core principle of AMA is that the file-name is the md5 hash of its contents.

Data files can be very large, so in the interest of storage efficiency - when a user adds files to an asset we don't make a copy the file. Instead, we create a hardlink to the file inside the cache directory.

The hardlink is a pointer to the same inode, so whenever the user makes any changes to the files, the cached file would be changed as well. So when a user alters the file after adding it to the asset, the file-name no longer reflects the hash of its contents. This causes data integrity issues.

To avoid this, we need to make sure that all file-names are the same as their content hash before they are uploaded. We can recompute the file hashes again and make sure the filenames are same. But this presents the following problem

• computing md5 hashes is an expensive operation, especially for large files.
• The user may have deleted the file after adding, which is not handled by this approach
• The user may have renamed the file after adding

In short, we need to detect if a file has been altered, renamed or deleted before committing to the cloud. To handle this, we take the following approach.

When user adds a file, we compute the file stat and store the following information.

• md5 hash
• st_mtime : last modified time of file metadata, changes when a file is renamed
• st_nlink : number of pointers to the inode of the file.
• st_ctime : last modified time of the file content
• st_ino : inode number of the file
• st_size : file size in bytes

Detecting if a file was deleted

Detecting if the contents were modified

Detecting if a file has been renamed

Uploading the Asset

Check if the Asset can be uploaded.

In order to be eligible for upload, an Asset must have a designated storage location which it inherits from the asset collection it belongs to.

• Check if the asset has a class_id, if not - request asset_registry for id of the asset collection
• The request to asset_registry must include class_name
• asset_registry receives the request, checks if an asset collection exists for the given name. If not, the asset_registry creates an asset_class.
• client receives the class_id and top_hash from the asset_registry and updates the asset-manifest
• Asset is now eligible for upload

Removing files from an asset

cd your_dir
ama remove file

The following steps are performed.

• Normalize the user-input path to asset_repo dir
• Find all objects that match the path asset.objects.filter(lambda x: x.path == path)
• Remove the object from asset, if multiple objects, we remove the last added object
• Remove the object reference from linked content, if ref count of content is 0 - remove the cached file pointed to by the content
• If multiple Objects were found earlier, and we had removed the last object then we link the previous object to the file-path

Listing Files in Asset

ama list

The following steps are performed.

• get the list of all objects in the asset list(asset.objects)
• each object has a ref to its content i.e. object.content for each object, get the content_hash, content_type
• list in table format

Uploading Asset

Create Asset Collection

It's the asset collection that owns the storage location. So in order to be eligible for upload, an asset must have a valid class_id

• Check if the asset has a class_id, if not - request asset_registry for id of the asset collection
• The request to asset_registry must include class_name
• asset_registry receives the request, checks if an asset collection exists for the given name. If not, the asset_registry creates an asset_class.
• client receives the class_id and top_hash from the asset_registry and updates the asset-manifest
• Asset is now eligible for upload

Find Files to be Uploaded

Once we have the class_id and top_hash - the asset-contents can be uploaded to staging area. We now query the asset to get a list of files that require to be uploaded. To maintain data integrity, a file must meet the following conditions in order to be a candidate for upload.

• added by the user
• have not been deleted, modified or renamed

Since the file sizes can be very large, in AMA, we took the approach of not copying files. Therefore, in order to accomplish the above requirements when a file is added to asset, we compute and store metadata about the file

st_mtime: timestamp (last modified time of the file content) 
st_nlink: int (# of hardlinks to the file)
st_ctime: timestamp (last modified time of file meta data)
st_ino: int (inode number i.e. disk storage pointer for the file)

Before uploading, we recompute the file metadata and compare with the stored values to identify if a file has been altered by the user after adding it to asset.

Exclude content already available in cloud

In AMA, we use the md5 hash of the files for storage and indexing. Its possible, that a user might be adding a file which already exists in the cloud and is indexed. In which case, the files need not be uploaded again. Therefore, after determining the list of files to be uploaded, we check if any of the files exist in the remote storage.

If the file is already indexed it would exist in the remote repo, if another user is in the process of uploading the file and its yet to be indexed then it would exist in the staging area. Therefore, we check both remote-repo and staging-repos.

User access

Staging and remote buckets are separate by design to maintain data integrity. For staging buckets, individual user will have write acess where as for remote buckets, only the asset-server will have write-access. Individual users will have readonly-access to the remote bucket.

Upload Files to Staging

We use asyncio for fast concurrent network calls. The staging area is namespaced to the top-hash of the asset-class.

Commit Asset

{
  "class_id": 00000000-0000-0000-0000-000000000001
  "parent_asset_id": 00000000-0000-0000-0000-000000000011,
  "alias": "my_asset",
  "objects": [...],
  "patch": "...",
}

Ensuring Atomicity of the Asset-Commit process

Its possible, that the previous commit was interrupted and didn't go through. A few possible reasons could be

• Network failure with User: Content staging was successful, but a network error happened at the user-end during the commit process. In such a case the asset-client will try to first complete the previous commit before initiating a new transaction. The server should expect to receive the same payload next-time since the transaction was not completed.

• Network failure with Server: Similar issue as the above but this time, there was an error on the server-side in either gcs access or database access

• User Interruption: i.e. ctrl + c

To ensure atomicity, we need to make sure the previous transaction was committed successfully. In order to achieve this we break down the asset-commit flow into 2 different stages.

Asset Commit flow - Stage 1

In stage-1 the Asset-Client checks if the asset to be committed has an id, if not, it requests for an id to the asset-server. The request payload contains the class_name, and the parent_asset_id (if asset inherits from a different asset). The Asset-Server receives the request, creates a record in the asset-table (with seq_id and version) and returns the asset_id to the Client.

Asset Commit flow - Stage 2

In stage-2, the asset-client requests for the asset to committed. The commit process involves the following steps.

• Transfer contents from staging area in to remote-repo, do checksum validation.
• Create records in Content table, if not exist already
• Create records in Objects table, if not exist already
• Create records in Asset-Objects join table
• Update asset-record with commit-hash
• Respond to client with commit-hash

If the commit process is successful, the asset-server updates the asset-record with a commit-hash and returns the commit-hash to client. Any time, the asset-client initiates a commit, it first checks if the existing asset has a commit-hash, if it finds a commit-has this means the previous commit was successful, so the client then moves to follow the 2 stage process. If the previous commit was unsuccessful, the client will initiate stage-2 again i.e. commit the previous asset along with new updates if any.

The Previous transaction got interrupted now user wants to add more files to assets

To verify, check if the previous record created by the same user with node_type and node_name has a commit_hash. If commit_hash is missing - then we presume the current updates are also part of the same commit, and we try to recommit.

Downloading Asset

State Management

Adding references to an asset

You can add existing assets as a reference before committing a created asset. A typical flow would be

ama init <class_name: creates asset ama add refs --type input --asset <name>

It's important to note that references are allowed only between root nodes.

Storing an Asset

Asset Storage Structure in Bucket

assets is a directory inside the bucket. This directory holds a list of directories which are the ids of all the asset collections.

asset_class_id is a directory inside assets. This directory holds of list of directory, each of which are the ids of all the assets in that class

asset_id is a directory inside asset_class_id. This holds a list files

• objects.yaml: holds the list of all objects that the asset refers to
• asset.yaml: holds the root information of the asset, this is a yaml representation of the asset record from db
• version.yaml: holds the changes relevant to that version, this is a yaml representation of the version record from db