Manage the Current Configuration
The Configuration is an extremely important concept. It is the scope for many operations, including lineage analysis, search, Version management, etc. In this way, what would otherwise be an overload of information (everything in the repository) is instead well managed according to the Configuration of metadata one is interested in analyzing or working with. The name Configuration comes from the concept of "Version & Configuration Management" where a Configurations a collocation of particular version of Models.
A valid Configuration consists of a collection of Model Versions, mapping Versions, glossaries and stitchings. The model versions relate to data stores and data processes that have been harvested into MetaKarta.
A Configuration may be understood as any of the following:
-
Repository workspace - a collection of Repository Objects to be analyzed together (search, browse, reports, etc.) as a technical scope, or business area under the same access permission scope.
-
Enterprise architecture - a collection of data store Models (ODS, data staging areas, data warehouses, data marts, etc.) and data process Models (ETL/DI, and BI) connected together through data flow stitching.
-
Design workflow - a collection of conceptual, logical and physical Models connected (semantically stitched) together through semantic mappings modeling the design process.
A Configuration will have one or more Configuration Versions. Configuration Versions may be understood each as a different collection of versions of Repository Objects. In this way, one can define several Configuration Versions, each containing various Versions of the Repository Objects. As a result, one may perform.
-
Historical analysis using Configuration Versions containing older Versions of Models which were deployed at some time in the past
-
What-if analysis using Configuration Versions containing the Versions of Models which may be deployed in the future.
When using the MetaKarta UI, as you are now, you will always be working with one configuration (version) at a time. Because of this feature of this user interface MetaKarta also provides additional MANAGE > Repository features. Creating configurations and version management of existing configurations (including publishing) may only be accomplished using the MANAGE > Repository feature.
Explore Further
Assign a group to a configuration
Groups may be assigned to a particular configuration. In this way, any users who are associated with that group are provided with the default (published) (or the latest if no version is published) configuration version.
The checkbox is part of the group definition.
Changing the Current Configuration
When using the MetaKarta UI, you will always be working with one configuration at a time. The configuration is an extremely important concept. It is the scope for many operations and analysis, including lineage, search, version management, etc. In this way, what would otherwise be an overload of information (everything in the repository including all historical and future versions of models, etc.) is instead well managed according to the configuration of metadata one is interested in analyzing or working with.
The configuration you are currently working in is displayed in the upper right in the header.
Steps
To change the configuration:
-
Click the name of the current configuration.
-
Select Change Configuration.
-
Double-click the configuration you wish to be in or select one and click on OK.
In the multi-version editions of MetaKarta, configurations can have a Published (or default) version. This is the version you are selecting above. If you wish to open a different version, click choose Version and select the specific version of the configuration you want to open.
Example
Click the configuration name in the header and select More.
Pick the Demo Enterprise Architecture configuration and go to MANAGE > Configuration.
If you have defined a home dashboard for that configuration, then you will be presented with that dashboard. Otherwise, you will see the MANAGE > Configuration screen.
Stitching Models Together for Data Flow Tracing
Some external metadata Models may contain data movement source specifications and data movement rules. These are in turn imported into MetaKarta. In many cases, these data movement source specifications may match up with another external metadata Model which was imported separately. Such data movement specification Models may then be added to a configuration and may be "stitched" together with that second Model, where one Model is the complete representation of a source that is defined in another with data movement specifications.
This is a mostly automated process which you perform once per new ETL/DI/BI model, in that MetaKarta will propose the best connection matches it can find and identify the level of confidence (completeness of the match for stitching).
Connection names, those names used within a data integration (DI/ETL) or a business intelligence (BI) tool to reference data stores, are often not the same as the names for those same data stored as harvested in the repository. Because of this difference, you will see a different presentation in the lineage overview of the model with connection names, versus what you see in a data flow trace after stitching the connections to their data stores. This data connection name resolution is performed automatically as part of the stitching process and will even present the "proper" schema names (those from the data stare harvest) in the data lineage trace view.
Steps
Ensure proper permissions
-
Sign in as a user with at least the Metadata Management capability object role assignment on the Configuration you are in.
View the configuration architecture
-
Go to the MANAGE > Configuration in the banner.
-
Click Diagram.
You can do the same steps below right in the MANAGE > Configuration page, but it is generally easier to identify the connection issues and especially the (as of yet) unconnected models that should be a part of a stitching when viewing the architecture diagram.
View the configuration list
-
Go to MANAGE > Configuration in the banner.
Identify stitching candidates
The models in the configuration require stitching when they have a warning symbol.
-
Select any model in the configuration with a warning and go to the Connections tab.
Stitch the connections
-
In the Connections tab, click the magic wand icon to Propose resolutions.
If you wish for the Propose Resolution to take effect, you must be sure to reset the connection by settings the Resolution to Ignore and ensure that it is not Approved. Propose Resolution will now try to propose resolutions for those connections edited in this manner.
-
If necessary, double-click the row with a warning to resolve any ambiguities in the harvested connection definition.
If you choose manually from the list of possible models to stitch to, MetaKarta will compare the schemas with the connection definition, table by table, and suggest a match.
-
If necessary, double-click a row and specify Select Manually to connect specific catalogs and schemas as defined in the data process model connection (model with a warning) and how that schema is defined in the data store selected.
Edits to the connection stitching are immediate. There is no need to commit them afterwards.
If you wish to simply ignore a connection so that {MIMM] will not present a warning that it is not stitched, then double-click the row and select Ignored.
-
Repeat for all other connections with a warning.
You may also use the magic wand icon in the Configuration Manager header to Propose resolutions for all the unapproved connections.
-
Click Build.
Build validates the stitching in the configuration and then builds indexes for lineage traces.
There are various options in the Build dialog.
UPDATE VERSIONS will update the configuration to ensure that they default or latest version of the contained models are the versions included in this configuration version. This insures that the newly imported model versions are included before rebuilding connections
Note, this option is grayed out in this example as there are no model versions to update.
REBUILD ALL CONNECTIONS option to rebuild all connections, even those already successfully resolved.
Example
You sign in as a user with at least the Metadata Management capability object role assignment on the Configuration you are in and go to the MANAGE
Configuration in the banner.
The configuration management feature is quite intelligent and already knows how to stitch the existing models together. Thus, we will need to begin with an clean configuration in a clean database to be sure it does not remember how to stitch and we can demonstrate the proposal and build process.
Import a Staging to Dimensional, Staging DW and Dimensional DW model.
The Staging to Dimensional model in the list on the left has a warning icon. It shows
-
Connection Dimensional is not connected.
-
Connection Staging is not connected.
These are the connections defined in the Talend DI model, the destination and the source. The goal of the configuration management process is to resolve these connections, otherwise referred to as stitching, so that lineage may be computed and presented. Just as in the actual DI tool and databases, the connections in Talend must match what is in the database exactly.
Refresh the browser to clear the caching information about what models are available and the Connections tab will then show (as the UI now understands that this is a DI model). Go to the Connections tab.
There are warnings next to the connections in the list and next to the Connections tab.
Then click on the magic wand icon for Propose Resolutions.
The configuration manager picks the dbo schemas in each of the two database as the best match. In fact, they match up at 100%.
At this point, the proposals are simply connection resolution rules, but they have not been computed. In order to do so, click BUILD.
The build action may include two steps:
-
Update the version of a model contained in the current version of the configuration you are in so that it is using latest version of that model
-
Attempt to commit the stitching or connection resolution by matching on position or name (depending upon the type of data store being connected to).
You may select multiple connections and use the right-click context menu to just Propose or Build those connections.
In this case, there is only one version of each of these models, so the UPDATE VERSIONS option is moot. However, click Yes for REBUILD ALL CONNECTIONS. Then click BUILD and refresh the browser.
The connections are resolved and lineage is indexed. However, the connection definitions are still merely proposals, even though built.
Double-click the cell for each connection under the Approved header and they will be approved.
Explore Further
View Log
View Log presents the Log Message dialog for the selected connection. You may see any connection errors documented as log messages.
Stitching Report
Stitching Report presents a complete report on what was and was not stitched between the connection and the data store. It is presented as a flat list that you may drill down in (e.g., schema > table > column)
Stitching Connection Name Resolution
Connection names, those names used within a data integration (DI/ETL) or a business intelligence (BI) tool to reference data stores, are often not the same as the names for those same data stored as harvested in the repository. Because of this difference, you will see a different presentation in the lineage overview of the model with connection names, versus what you see in a data flow trace after stitching the connections to their data stores. This data connection name resolution is performed automatically as part of the stitching process and will even present the "proper" schema names (those from the data stare harvest) in the data lineage trace view.
Example
Go to the object page for the AP to Staging ETL process.
There are two database connections. Note, their names are shortened versions (spaces missing, etc.) of the data stores.
Click on the StagingDW connection:
The schema name is not known, as it was never specified in the ETL design. This works because the database will simply use the default schema.
Now, go back up to the level of the entire ETL model and then click the Lineage tab:
Again, you see the shortened names.
Because we went to the Lineage tab with the entire ETL model open, rather than from a particular object in a database, we are presented with the Lineage Overview, rather than a Lineage Trace. With the lineage overview, you only what is in the ETL model, not the full end-to-end lineage trace.
However, since we have stitched this model to the two data stores, and have the complete (proper) names for the database and schemas, we see these in the lineage trace. Go back to the Staging DW connection and navigate to the Vendor table object page and go to the Lineage tab.
The proper names from the data store models are presented in the lineage trace because the lineage trace is not limited to only what is in the ETL model, unlike the lineage overview.
Stitching Models Together for Semantic Tracing
Some external metadata models may contain semantic information (logical names, descriptions, domains, types, etc.) that are related to a faithful picture of an underlying data store, such as a database schema (e.g., as logical/physical models from data modeling tools). These may in turn be imported into MetaKarta. If the physical database specification is a faithful (name or position matching, depending upon the implementing technology) representation, then they may be semantically stitched together with a model harvested directly from the datastore.
Steps
Ensure proper permissions
-
Sign in as a user with at least the Metadata Management capability object role assignment on the Configuration you are in.
View the configuration architecture
-
Go to the MANAGE > Configuration in the banner.
-
Click Diagram.
You can do the same steps below right in the MANAGE > Configuration page, but it is generally easier to identify the connection issues and especially the (as of yet) unconnected models that should be a part of a stitching when viewing the architecture diagram.
-
Right-click on any data model in the diagram and select Edit Connections
Stitch the connections
-
In the Connections tab, associate the Physical Semantic Connection with a model harvested directly from the datastore.
-
If necessary, double-click the row under the heading Schema/Path to resolve any ambiguities in the harvested connection definition.
Once you choose from the list of possible models to stitch to, MetaKarta will compare the schemas with the connection definition, table by table, and suggest a match.
-
If necessary, click Edit Schemas to connect specific schemas as defined in the data process model connection (model with a warning) and how that schema is defined in the data Store selected.
Edits to the connection stitching are immediate. There is no need to commit them afterwards.
If you wish to simply ignore a connection so that {MIMM] will not present a warning that it is not stitched, then click the Ignore connection checkbox.
-
Return to the architecture diagram.
You may also define the connection stitchings by clicking on the model under MANAGE > Configuration, select a model and go to the Connections tab.
-
Click Update and Build.
Update and Build validates the stitchings in the configuration and then builds indexes for lineage traces.
Example
You sign in as a user with at least the Metadata Management capability object role assignment on the Configuration you are in and go to the MANAGE
Configuration in the banner. Select the Accounts Receivable Erwin model and go to the Connections tab.
Then double-click on the Resolution for the Physical Semantic Connection, and change it to Accounting.
Select the MITI-Finance-AP in the Database Catalog pull-down and click OK.
Click Build.
Note, the connection is still unresolved as the schema names are different.
Click Open Connection Resolution Report
Architecture Diagram
Once stitched, the relationships among Repository Objects in a Configuration can be visualized producing a data flow and semantic based architecture diagram. One may edit the layout and annotate these diagrams using the architecture diagram feature to obtain a high-level configuration overview of the architecture defined by a configuration.
Overview
You may click this icon to show or hide an Overview panel of the architecture diagram. Click in the overview to quickly move to a portion of the full diagram.
Zoom In/Out and Fit to content
Click Zoom in or Zoom out icons to adjust the aspect ratio of the diagram. Also, you may click on the Fit to content icon to view the entire diagram at the best zoom that will fit.
Add Annotation
Click Annotation then click in the diagram where you want the annotation to be placed to create annotations for a diagram. You may then right-click on the resulting object to edit further, including text content, style, color, etc.
Layout diagram
You may click the Layout Diagram icon to clean up an entire diagram.
Layout will undo any editing you have performed since the last Save
Layout Selected Objects is a context menu (right-click) option that lays out only the selected objects.
You may click the Layout Selected Objects to clean up those objects in the diagram.
When editing you may use the Undo to revert back to the diagram before layout.
There are several options for presentation and layout based upon structure, methodology and other algorithms.
You may download a PNG or SVG image of the diagram.
Quick find
In the upper right, there is a search text box that will provide a quick list of object names that contain the text you type. You may click on any of the results to select that object in the diagram and moving the focus there.
Properties Panel
Click to select a object and view its properties in the Properties Panel on the right. You may show and hide this panel as needed.
Explore Further
Edit Connections
As seen in the example above, Edit Connections is available in the Architecture Diagram itself.
Edit Content Settings
Edit Model Settings will take you back to the MANAGE > Configuration page for the object that is selected (or right-clicked on).
Update and Build a Configuration
Once you have a configuration that is complete, with connection definitions resolved, you may Build the configuration. This action allows users to perform lineage and impact analysis with the configuration, as well as semantic definition and usage reporting. It is required before any of that analysis may be made accurately in that configuration versions.
Configuration Update/Build means:
-
Update the configuration with default versions (or latest if there is no default) of each model of that configuration
-
Build that version of that Configuration. Note, it does not create a new version.
Steps
-
Sign in as a user with at least the Metadata Management capability object role assignment on the Configuration you are in.
-
Go to the MANAGE > Configuration in the banner.
-
Click the Update and Build button.
Example
Sign in as a user with at least the Metadata Management capability object role assignment on the configuration.
Go to MANAGE > Configuration and click the Update and Build button.