Profiles of data assets

Requirements and restrictions

You can view the profile of assets under the following circumstances.

Ways to create a profile

Asset profiles can be created in different ways:

• In governed catalogs, profiles for individual data assets are created automatically when the data assets are added to the catalog with these exceptions:

  • You disabled automatic profiling for the catalog.
  • The asset comes from a connection that is configured to use personal credentials.
  • The asset was profiled through metadata enrichment before it was published. Such assets already have a profile that's added to the catalog along with the asset.

• In projects and in catalogs without data protection rule enforcement, you can manually create profiles for individual data assets. You can also create a profile manually in a governed catalog if the asset wasn't profiled before.

• In projects, you can create and run a metadata enrichment asset to profile large sets of data assets in one go. These asset profiles are available in the project. You can publish the enriched assets with their profiles to any type of catalog. See Managing metadata enrichment.

Within one account, profiling results are copied with the data asset when you publish an asset from a project to a catalog or add it from a catalog to a project. However, if the catalog and the project belong to different accounts, the profiles aren't copied because the set of available data classes might be different.

You can update an individual asset profile from the asset's Profile page in a project or a catalog. If you manually update a profile of a data asset that is included in a metadata enrichment, the profile and analysis information is also reflected in the respective enrichment results. Profiles are also updated when new enrichment results are published.

When you update an existing profile, you can change the data classes to include in the profile. If you exclude a data class that was previously assigned to a column, the updated profile shows Class excluded (from profile) for the respective column unless a different data class was assigned. You will also see Class excluded (from profile) for any columns where you don't have access to the assigned data class.

What is analyzed during profiling?

If you create or update an asset profile from the Profile page in a project or a catalog, columns are analyzed.

When a single asset is profiled in a project or a catalog, the profile is by default created based on the first 5,000 rows of data. If the data asset has more than 250 columns, the profile is created based on the first 1,000 rows of data. If the profile is created through metadata enrichment, sampling is determined by the metadata enrichment settings.

To identify the structure and content of your data and to classify it, analysis includes the following tasks:

• Compute statistics about the data of each analyzed column.
• Compute data types for columns and data types distribution.
• Computes data formats for columns and formats distribution.
• Classify the data and compute data class candidates for columns.
• Capture frequency distributions.

Profile information

The profile of a data asset shows information about each column in the data asset.

The Profile tab provides some general information and an overview of the analysis results:

• When was the profile created or last updated.

• How many columns and rows were analyzed.

• The inferred data class for each column and the confidence for that data class. Data classes describe the contents of the data in the column: for example, city, account number, or credit card number. Data classes can be used to mask data or to restrict access to data assets with data protection rules. The data classes appear for each column on the asset's Overview page and on the Profile page.

  The confidence of a data class is the percentage of non-null values that match the data class.

  Several data classes are more generic identifiers that are detected and assigned at a column level. These data classes are assigned when a more specific data class could not be identified at a value level. Generic identifiers always have a confidence of 100% and include the following data classes: code, date, identifier, indicator, quantity, and text.

• The percentage of matching, mismatching, or missing data for each column.

• The frequency distribution for all values identified in a column.

• Statistics about the data for each column such as the number of distinct values, the percentage of unique values, minimum, maximum, or mean, and sometimes the standard deviation in that column. The number of distinct values indicates how many different values exist in the sampled data for the column. The percentage of unique values indicates the percentage of distinct values that appear only once in the column.

  Depending on a column’s data format, the statistics vary slightly. For example, statistics for a column of data type integer have minimum, maximum, and mean values and a standard deviation value while statistics for a column of data type string have minimum length, maximum length, and mean length values.

More detailed information about column data is available when you click the column name. See Detailed profiling results.

The latest asset profile is retained and shown while the data asset exists in the catalog or in the project even if the original data in the data source is temporarily or permanently not available. To remove the profile information, you have these options:

• You can manually delete the profile on the Profile page. This option is not available if the asset is subject to any data protection rules.
• You can manually delete the data asset from the project or the catalog.
• If the asset was added through metadata import, you can rerun the metadata import with the appropriate the Delete on reimport option set.

Learn more

• Profiling an asset
• Managing metadata enrichment
• Predefined data classes
• Detailed profiling results
• Masking data

Parent topic: Asset types and properties