This vignette is a WIP, and is meant to describe the architecture of a FacileDataSet in more detail. This is primarily for developers to understand how it is implemented. Casual users of FacileDataSet objects have no real need to understand this level of detail.
FacileDataSet is implemented as a well-structured directory, containing the following elements:
data.sqliteSQLite database that stores feature- and sample-level metadata.
data.h5HDF5 file that stores a multitude of dense assay matrices that are generated from the assays performed on the samples in the
meta.yamlfile tha contains information about the
FacileDataSet. To better understand the structure and contents of this file, you can refer to the following:
testdata/expected-meta.yamlfile for, which is an exemplar file for the
testdata/TestFacileTcgaDataSet, which consists of data extracted from two datasets (BLCA and BRCA) from the TCGA.
eav_metadata_createfunction, which describes in greater detail how we track a dataset’s sample-level covariates (aka, “pData” in the bioconductor world). In the meantime, a short description of the entries found in the
meta.yamlfile is provided here:
name: the name of the dataset (i.e.
"Mus musculus", etc.
default_assay: the name of the assay to use by default if none is specified in calls to
fetch_assay_data(), [with_assay_data()], etc. (kind of like how
"exprs"is the default assay used when working with a [Biobase::ExpressionSet])
datasets: a section tha enumerates the datasets included internally. The datasets are further enumerated.
sample_covariates: a section that enumerates the covariates that are tracked over the samples inside the
FacileDataSet(ie. a mapping of the
pDatafor the samples). Reference
?create_eav_metadatafor more information.
custom-annotationdirectory, which stores custom
sample_covariate(aka “pData”) information that analysts can identify and describe during the course of an analysis, or even add from external sources. Although this directory is required in the directory structure of a valid
FacileDataSet()constructor can be called with a custom
anno.dirparameter so that custom annotations are stored elsewhere.
Specifying a custom
anno.dir in the
FacileDataSet() constructor enables you to define a directory external to the FacileDataSet that will be used to store custom annotatios. This can be convenient, for example, if you want update the FacileDataSet without blowing out user-level annotatations.
Sample- and feature-level metadata are stored in an SQLite database. To allow for the ragged nature of sample-level annotations across however-many datasets are internalized into a single FacileDataSet, these covariates are stored in an entity-attribute-value table (explained below).
We may refactor the feature-level metadata to use a similar approach, as well as the assay-level data (such as libsize, normfactors) since different assays require different types of metadata.
Sample covariates (aka
pData) are encoded in an [entity-attribute-value (EAV) table][EAV]. Metadata about these covariates are stored in a
meta.yaml file in the
FacileDataSet directory which enables the
FacileDataSet to cast the value stored in the EAV table to its native R type. This function generates the list-of-list structure to represent the
sample_covariates section of the
pData covariates, each column is treated independently from the rest. There are some types of covariates which require multiple columns for proper encoding, such as encoding of survival information, which requires a pair of values that indicate the “time to event” and the status of the event (death or censored). In these cases, the caller needs to provide an entry in the
covariate_def list that describes which
pData columns (
varname) goes into the single facile covariate value.
Please refer to the Encoding Survival Covariates section for a more detailed description of how to define encoding survival information into the EAV table using the
covariate_def parameter. Further examples of how to encode other complex atributes will be added as they are required, but you can reference the Encoding Arbitrarily Complex Covariates section for some more information.
UPDATE: Survival covariates can now be encoded simply as a
survival::Surv object and provided as a column in the pData data.frame. The following describes the original, and still supported, method.
Survival data in R is typically encoded by two vectors. One vector that indicates the “time to event” (tte), and a second to indicate whether or not the denoted tte is an “event” (1) or “censored” (0).
Normally these vectors appear as two columns in an experiment’s
pData, and therefore need to be encoded into the
FacileDataSet’s EAV table. To do so, the pair of vectors are turned into a signed numeric value. The absolute value of the numeric indicates the “time to event” and the sign of the value indicates its censoring status.
Let’s assume we have
event_OS column that are used to encode a patient’s overall survival (time and censor status). To store this as an “OS” covariate in the EAV table, a
covariate_def list-of-list definition that captures this encoding would look like this:
covariate_def <- list( OS=list( class="right_censored", arguments=c(time="tte_OS", event="event_OS"), label="Overall Survival", type="clinical", description="Overall survival in days"))
Note how the name of the list-entry in
covariate_def defines the name of the covariate in the
class entry for the
OS definition indicates the type of variable this is. The
varname entry lists the columns in the
pData that are combined to make this value. The
names(varnames) correspond to the parameters in the [eav_encode_right_censored()] function. The analagous
meta.yaml entry in the
sample_covariates section for the
covariate_def entry looks like so:
sample_covariates: OS: class: right_censored label: "Overall Survival" type: "clinical" description: "Overall survival in days" colnames: ["tte_OS", "event_OS"] argnames: ["time", "event"]
To encode a new type of complex covariate from a wide
pData data.frame, we need to:
"right_censored") for use within a
eav_encode_<class>(arg1, arg2, ...)function which takes the R data vectors (arg1, arg2) and converts them into a single value for the EAV table.
eav_decode_<class>(x, attrname, def, ...)function which takes the single value in the EAV table and casts it back into the R-native data vector(s).
xis the vector of (character) values from the EAV table
attrnameis the name of the covariate in the EAV table
defis the definition-list for this covariate.
...allows each decode function to be further customized. [EAV]: https://en.wikipedia.org/wiki/Entity-attribute-value_model
The HDF5 file has one directory per assay. These directories have one matrix per dataset for the given assay.
For instance, the
data.h5 file of a FacileDataSet that assembles rnaseq, cnv, and mirnaseq data from the [TCGA][tcga] data would look like this:
. data.h5 ├── rnaseq │ ├── ACC │ ├── BLCA │ ├── BRCA │ ├── CESC │ ├── ... ├── cnv_score │ ├── ACC │ ├── BLCA │ ├── BRCA │ ├── CESC │ ├── ... ├── mirnaseq │ ├── ACC │ ├── BLCA │ ├── BRCA │ ├── CESC │ ├── ...