Skip to content

Catalog Checks: Catalog Sources#

Note

The below checks require both catalog.json and manifest.json to be present.

CheckSourceColumnsAreAllDocumented #

All columns in a source should be included in the source's properties file, i.e. .yml file.

Receives at execution time:

Name Type Description
catalog_source CatalogTable

The CatalogTable object to check.
sources List[DbtBouncerSourceBase]

List of DbtBouncerSourceBase objects parsed from catalog.json.

Other Parameters (passed via config file):

Name Type Description
exclude Optional[str]

Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include Optional[str]

Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity Optional[Literal['error', 'warn']]

Severity level of the check. Default: error.

Example(s): 

catalog_checks:
    - name: check_source_columns_are_all_documented

Source code in src/dbt_bouncer/checks/catalog/check_catalog_sources.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
class CheckSourceColumnsAreAllDocumented(BaseCheck):
    """All columns in a source should be included in the source's properties file, i.e. `.yml` file.

    Receives:
        catalog_source (CatalogTable): The CatalogTable object to check.
        sources (List[DbtBouncerSourceBase]): List of DbtBouncerSourceBase objects parsed from `catalog.json`.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        catalog_checks:
            - name: check_source_columns_are_all_documented
        ```

    """

    catalog_source: "CatalogTable" = Field(default=None)
    name: Literal["check_source_columns_are_all_documented"]
    sources: List["DbtBouncerSourceBase"] = Field(default=[])

    def execute(self) -> None:
        """Execute the check."""
        source = next(
            s for s in self.sources if s.unique_id == self.catalog_source.unique_id
        )
        undocumented_columns = [
            v.name
            for _, v in self.catalog_source.columns.items()
            if v.name not in source.columns
        ]
        assert not undocumented_columns, f"`{self.catalog_source.unique_id}` has columns that are not included in the sources properties file: {undocumented_columns}"