Key concepts¶

This section explains some of the most important concepts for TRAC users. If you’re the kind of person who likes to read the IKEA instructions in full before picking up a screwdriver, this is definitely for you.

If you prefer to work it out as you go along, you may want to skip to the next section.

Metadata model¶

Layers¶

TRAC is built around a structural metadata model which catalogues and describes everything on the platform. The model consists of two layers:

OBJECTS	Objects are the model’s structural elements and each object type (e.g. models or data) has its own metadata structure
TAGS	Tags are used to index, describe and control objects. Some tags are controlled by the platform, some you set yourself

Because the metadata model records object creation and tag modification actions, it provides a comprehensive audit history for the platform and all its contents. Read actions are not recorded.

Objects¶

The most common and important object types are Data, Models, Flows and Jobs.

Object	Refers to
DATA	Collections of documents and records which have been imported into a TRAC-controlled Data Store and are therefore available for use on the platform
MODEL	Discrete units of code stored in a repository that have been made available for use on the platform via the model upload process
FLOW	The blueprint of a complex calculation involving multiple models, which TRAC can run on demand. Unlike the other object types, flows are metadata-only objects and do not refer to any external resource or asset
JOB	A calculation or process that TRAC orchestrates. The five Job types are; ImportModel, ImportData, RunModel, RunFlow and ExportData

Note

ImportData and ExportData jobs are used to move data in and out of TRAC via a back-end process and are not available in COMMUNITY TRAC, which only supports data import and export via the UI.

Tags¶

Some tags are automatically assigned by the platform for audit and control purposed (e.g. object creation timestamp) but the following tags are assigned by users through the UI, to help identify and classify objects.

Tag	Use	Type	Applies to
KEY	A unique key which is used when searching for objects that are functionally interchangeable	String	Data & models
NAME	A short meaningful label for the object	String	All
DESCRIPTION	A slightly longer description, containing enough information to differentiate it from similar objects	String	All
SHOW IN SEARCH?	Determines if the object should appears in search results (default = Y). Objects are never deleted but setting this tag to N hides them. Changing it back to Y un-hides them	Boolean	All
BUSINESS SEGMENT	An optional four-level categorical tag that can be used to capture organisation-specific object identifiers	String	All

Virtual deployment method¶

Deployment steps¶

TRAC’s approach to model orchestration is called the ‘virtual deployment’ method because complex model deployments are built using metadata rather than model code. There are three main steps involved in this approach.

IMPORT MODEL	TRAC scans the model code and creates a new metadata object. This object contains the model schema and a link to its commit ID. The code is not copied, it stays in the repository
BUILD FLOW	Models are combined to form a calculation blueprint in which the outputs of one model become the inputs to the next. Flow are built and validated using model schemas, not the models code
RUN JOB	When a calculation job is initiated, TRAC fetches all the model code from the repository. Once the job is complete, the code it not retained. It’s fetched from the repository again, the next time it’s needed

Note

This short video (under 3 mins) explains how the virtual deployment method works in detail.

Model schema¶

TRAC models must contain or be wrapped inside a custom function which declares their schema. There are three parts to a model schema.

INPUTS	The schema(s) of all the input data objects which the model needs to run
PARAMETERS	The schema of the parameters that govern how the model runs, which need to be provided via the UI
OUTPUTS	The schema(s) of all the data objects which are produced when the model runs

Note

See Modelling for more details on model schemas and building TRAC models and see TRAC data types for details of the data types that are available when defining and using schemas.

Schema validation¶

This is an example of a TRAC schema.

Schema validation occurs when pairing a data set with a model or flow when launching a job and when pairing two models together to build a flow. In both instances, TRAC applies a strict validation based on the object schemas.

Key things to remember are:

Validation occurs between object schemas not their data content
Validation considers Field name, Field Type, Business key, Categorical and Not nullable
The order in which the field names appear is not important
The source schema must contain all fields needed by the target object. It can contain extra fields.

The TRAC guarantee¶

A central feature of TRAC is the unique control environment which it creates. This is embodied by a three-part guarantee:

Every action is auditable

Every calculation is repeatable

There is no change risk on the platform

This guarantee is made possible by the interactions between the virtual deployment method, the metadata model and the fact that TRAC makes both models and data immutable. For models, immutability ia achieve via the capabilities of the repository. For data, TRAC uses an append-only data model to maintain a complete version history of all data brought onto the platform.

Note

This short video (under 2 mins) explains how the TRAC Guarantee works in detail.

System resources¶

To execute jobs TRAC uses four different types of external system resources. These resources are defined and managed at the individual tenant level, by Admins.

REPOSITORIES	The external code repositories from which models can be imported
EXECUTORS	The compute service(s) which TRAC can use to execute calculations
INTERNAL STORAGE	The data location(s) to which only TRAC has write access, that are used to store primary data
EXTERNAL STORAGE	The data location(s) which TRAC can copy data to/from when running DataImport and DataExport jobs

Note

In TRAC COMMUNITY, Internal Storage will be a local file directory and there are no mechanism to add External Storage locations.

User access¶

Tenants¶

Tenants are logically separate domains within TRAC that play a key role in user access management.

Every model, data, flow and jobs object resides in a single TRAC tenant. If you upload the same model or dataset to two different tenants, each upload creates a unique object in the metadata store, both of which reference the same external asset.

How to set up your tenants is a question of judgement. Generally, use-cases which share a common business vocabulary should share a tenant. Those which do not, should not.

Note

TRAC COMMUNITY has a single tenant. In TRAC PROFESSIONAL tenants can be created and managed by Admin users.

Roles¶

To complete any action in a tenant a user must be assigned a tenant-role, either Admin, Manager, Write or Read.

ADMIN	Admin users for the TRAC system tenant can create new tenants. Admin users for business tenants can manage tenant resources in those tenants
MANAGER	User can upload models and change object tags in their tenant
WRITE	User can import and upload data, build flows and run jobs in their tenant
READ	User can read data and metadata in their tenant

These roles are inclusive, so Write implies Read and Admin implies Manager, Read and Write.

Note

For enterprise deployments, roles would be defined in the Active Directory with users accessing TRAC via the SSO mechanism. TRAC COMMUNITY has just a single Admin user account.