11. Runtime Metadata

This tutorial is based on example code which can be found in the finTRAC Examples Repository.

Models can look up the TRAC D.A.P. metadata associated with their inputs while they run, rather than relying only on the data itself. This example builds a Markdown report describing each input dataset, using both the runtime metadata and the schema.

Accessing metadata at runtime

get_metadata() returns a RuntimeMetadata object for a named input, giving access to its TRAC object ID and any metadata attributes that were recorded when the dataset was created.

src/tutorial/runtime_metadata.py
45    def run_model(self, ctx: trac.TracContext):
46
47        customer_loans = ctx.get_pandas_table("customer_loans")
48        customer_loans_schema = ctx.get_schema("customer_loans")
49        customer_loans_metadata = ctx.get_metadata("customer_loans")
50
51        account_filter = ctx.get_pandas_table("account_filter")
52        account_filter_schema = ctx.get_schema("account_filter")
53        account_filter_metadata = ctx.get_metadata("account_filter")
54
55        with ctx.put_file_stream("data_report") as report_bytes:
56            with io.TextIOWrapper(report_bytes, encoding='utf-8') as report:
57
58                report.write(f"# Data Report\n\n")
59
60                self.write_dataset_report("customer_loans", customer_loans_metadata, customer_loans_schema, customer_loans, report)
61                self.write_dataset_report("account_filter", account_filter_metadata, account_filter_schema, account_filter, report)

Note

Metadata is only available for inputs that come from real TRAC objects. For an input that has no metadata attached, such as an intermediate result from another model in a flow, get_metadata() returns None. Calling it for a name that is not a model input or output (including a model parameter), or for an output the model has not yet saved, is an error.

Building a report

Combining get_metadata() with get_schema() gives enough information to describe a dataset without needing to know its structure in advance - the object ID and attributes come from the metadata, while the field names and types come from the schema. The report itself is written out using put_file_stream(), the same file output method used in Using Files.

63    @classmethod
64    def write_dataset_report(
65            cls, dataset_name: str, metadata: trac.RuntimeMetadata,
66            schema: trac.SchemaDefinition, dataset: pd.DataFrame,
67            report: tp.TextIO):
68
69        report.write(f"## {dataset_name}\n\n")
70
71        if metadata is not None:
72
73            report.write(f"Object ID: {metadata.objectId.objectId}  \n")
74            report.write(f"Object version: {metadata.objectId.objectVersion}  \n\n")
75
76            report.write("Attributes:\n\n")
77
78            report.write("| Key | Value |\n")
79            report.write("| --- | ----- |\n")
80
81            for attr_key, attr_value in metadata.attributes.items():
82                report.write(f"| {attr_key} | {str(attr_value)} |\n")
83
84            report.write("\n")
85
86        else:
87            report.write("Metadata not available\n\n")
88
89        report.write("Columns:\n\n")
90
91        report.write("| Column | TRAC Type | Categorical | Pandas DType |\n")
92        report.write("| ------ | --------- | ----------- | ------------ |\n")
93
94        for field in schema.table.fields:
95            report.write(f"| {field.fieldName} | {field.fieldType.name} | {field.categorical} | {dataset[field.fieldName].dtype} |\n")
96
97        report.write(f"\nRows: {len(dataset)  }\n\n")

This pattern - reading metadata and schema alongside the data itself - is useful any time a model needs to behave generically across a range of datasets, such as building data quality reports or generic monitoring tools, without hard-coding assumptions about what any particular input contains.

See also

Full source code is available for the Runtime Metadata example on GitHub.