Manifest Checks: Sources

Note

The below checks require manifest.json to be present.

Classes:

Name	Description
CheckSourceDescriptionPopulated	Sources must have a populated description.
CheckSourceFreshnessPopulated	Sources must have a populated freshness.
CheckSourceHasMetaKeys	The meta config for sources must have the specified keys.
CheckSourceHasTags	Sources must have the specified tags.
CheckSourceLoaderPopulated	Sources must have a populated loader.
CheckSourceNames	Sources must have a name that matches the supplied regex.
CheckSourceNotOrphaned	Sources must be referenced in at least one model.
CheckSourcePropertyFileLocation	Source properties files must follow the guidance provided by dbt here.
CheckSourceUsedByModelsInSameDirectory	Sources can only be referenced by models that are located in the same directory where the source is defined.
CheckSourceUsedByOnlyOneModel	Each source can be referenced by a maximum of one model.

CheckSourceDescriptionPopulated

Sources must have a populated description.

Receives at execution time:

Name	Type	Description
source	DbtBouncerSourceBase	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_description_populated

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceDescriptionPopulated(BaseCheck):
    """Sources must have a populated description.

    Receives:
        source (DbtBouncerSourceBase): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_description_populated
        ```

    """

    name: Literal["check_source_description_populated"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        assert (
            len(self.source.description.strip()) > 4
        ), f"`{self.source.source_name}.{self.source.name}` does not have a populated description."

CheckSourceFreshnessPopulated

Sources must have a populated freshness.

Receives at execution time:

Name	Type	Description
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_freshness_populated

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceFreshnessPopulated(BaseCheck):
    """Sources must have a populated freshness.

    Receives:
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_freshness_populated
        ```

    """

    name: Literal["check_source_freshness_populated"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        error_msg = f"`{self.source.source_name}.{self.source.name}` does not have a populated freshness."
        assert self.source.freshness is not None, error_msg
        assert (
            self.source.freshness.error_after.count is not None
            and self.source.freshness.error_after.period is not None
        ) or (
            self.source.freshness.warn_after.count is not None
            and self.source.freshness.warn_after.period is not None
        ), error_msg

CheckSourceHasMetaKeys

The meta config for sources must have the specified keys.

Parameters:

Name	Type	Description	Default
keys	NestedDict	A list (that may contain sub-lists) of required keys.	required

Receives at execution time:

Name	Type	Description
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_has_meta_keys
      keys:
        - contact:
            - email
            - slack
        - owner

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceHasMetaKeys(BaseCheck):
    """The `meta` config for sources must have the specified keys.

    Parameters:
        keys (NestedDict): A list (that may contain sub-lists) of required keys.

    Receives:
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_has_meta_keys
              keys:
                - contact:
                    - email
                    - slack
                - owner
        ```

    """

    keys: "NestedDict"
    name: Literal["check_source_has_meta_keys"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        missing_keys = find_missing_meta_keys(
            meta_config=self.source.meta,
            required_keys=self.keys.model_dump(),
        )

        assert (
            missing_keys == []
        ), f"`{self.source.source_name}.{self.source.name}` is missing the following keys from the `meta` config: {[x.replace('>>', '') for x in missing_keys]}"

CheckSourceHasTags

Sources must have the specified tags.

Parameters:

Name	Type	Description	Default
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.	required
tags	List[str]	List of tags to check for.	required

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_has_tags
      tags:
        - tag_1
        - tag_2

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceHasTags(BaseCheck):
    """Sources must have the specified tags.

    Parameters:
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.
        tags (List[str]): List of tags to check for.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_has_tags
              tags:
                - tag_1
                - tag_2
        ```

    """

    name: Literal["check_source_has_tags"]
    source: "DbtBouncerSourceBase" = Field(default=None)
    tags: List[str]

    def execute(self) -> None:
        """Execute the check."""
        missing_tags = [tag for tag in self.tags if tag not in self.source.tags]
        assert not missing_tags, f"`{self.source.source_name}.{self.source.name}` is missing required tags: {missing_tags}."

CheckSourceLoaderPopulated

Sources must have a populated loader.

Parameters:

Name	Type	Description	Default
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.	required

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_loader_populated

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceLoaderPopulated(BaseCheck):
    """Sources must have a populated loader.

    Parameters:
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_loader_populated
        ```

    """

    name: Literal["check_source_loader_populated"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        assert (
            self.source.loader != ""
        ), f"`{self.source.source_name}.{self.source.name}` does not have a populated loader."

CheckSourceNames

Sources must have a name that matches the supplied regex.

Parameters:

Name	Type	Description	Default
source_name_pattern	str	Regexp the source name must match.	required

Receives at execution time:

Name	Type	Description
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_names
      source_name_pattern: >
        ^[a-z0-9_]*$

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceNames(BaseCheck):
    """Sources must have a name that matches the supplied regex.

    Parameters:
        source_name_pattern (str): Regexp the source name must match.

    Receives:
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_names
              source_name_pattern: >
                ^[a-z0-9_]*$
        ```

    """

    name: Literal["check_source_names"]
    source_name_pattern: str
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        assert (
            re.compile(self.source_name_pattern.strip()).match(self.source.name)
            is not None
        ), f"`{self.source.source_name}.{self.source.name}` does not match the supplied regex `({self.source_name_pattern.strip()})`."

CheckSourceNotOrphaned

Sources must be referenced in at least one model.

Receives at execution time:

Name	Type	Description
models	List[DbtBouncerModelBase]	List of DbtBouncerModelBase objects parsed from manifest.json.
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_not_orphaned

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceNotOrphaned(BaseCheck):
    """Sources must be referenced in at least one model.

    Receives:
        models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_not_orphaned
        ```

    """

    models: List["DbtBouncerModelBase"] = Field(default=[])
    name: Literal["check_source_not_orphaned"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        num_refs = sum(
            self.source.unique_id in model.depends_on.nodes for model in self.models
        )
        assert (
            num_refs >= 1
        ), f"Source `{self.source.source_name}.{self.source.name}` is orphaned, i.e. not referenced by any model."

CheckSourcePropertyFileLocation

Source properties files must follow the guidance provided by dbt here.

Receives at execution time:

Name	Type	Description
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_property_file_location

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourcePropertyFileLocation(BaseCheck):
    """Source properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview).

    Receives:
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_property_file_location
        ```

    """

    name: Literal["check_source_property_file_location"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        path_cleaned = clean_path_str(self.source.original_file_path).replace(
            "models/staging", ""
        )
        expected_substring = "_".join(path_cleaned.split("/")[:-1])

        assert path_cleaned.split(
            "/",
        )[
            -1
        ].startswith(
            "_",
        ), f"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not start with an underscore."
        assert (
            expected_substring in path_cleaned
        ), f"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not contain the expected substring (`{expected_substring}`)."
        assert path_cleaned.split(
            "/",
        )[
            -1
        ].endswith(
            "__sources.yml",
        ), f"The properties file for `{self.source.source_name}.{self.source.name}` (`{path_cleaned}`) does not end with `__sources.yml`."

CheckSourceUsedByModelsInSameDirectory

Sources can only be referenced by models that are located in the same directory where the source is defined.

Parameters:

Name	Type	Description	Default
models	List[DbtBouncerModelBase]	List of DbtBouncerModelBase objects parsed from manifest.json.	required
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.	required

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_used_by_models_in_same_directory

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceUsedByModelsInSameDirectory(BaseCheck):
    """Sources can only be referenced by models that are located in the same directory where the source is defined.

    Parameters:
        models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_used_by_models_in_same_directory
        ```

    """

    models: List["DbtBouncerModelBase"] = Field(default=[])
    name: Literal["check_source_used_by_models_in_same_directory"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        reffed_models_not_in_same_dir = []
        for model in self.models:
            if (
                self.source.unique_id in model.depends_on.nodes
                and model.original_file_path.split("/")[:-1]
                != self.source.original_file_path.split("/")[:-1]
            ):
                reffed_models_not_in_same_dir.append(model.unique_id.split(".")[0])

        assert (
            len(reffed_models_not_in_same_dir) == 0
        ), f"Source `{self.source.source_name}.{self.source.name}` is referenced by models defined in a different directory: {reffed_models_not_in_same_dir}"

CheckSourceUsedByOnlyOneModel

Each source can be referenced by a maximum of one model.

Receives at execution time:

Name	Type	Description
models	List[DbtBouncerModelBase]	List of DbtBouncerModelBase objects parsed from manifest.json.
source	DbtBouncerSource	The DbtBouncerSourceBase object to check.

Other Parameters (passed via config file):

Name	Type	Description
exclude	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
include	Optional[str]	Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
severity	Optional[Literal['error', 'warn']]	Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_used_by_only_one_model

Source code in src/dbt_bouncer/checks/manifest/check_sources.py

class CheckSourceUsedByOnlyOneModel(BaseCheck):
    """Each source can be referenced by a maximum of one model.

    Receives:
        models (List[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`.
        source (DbtBouncerSource): The DbtBouncerSourceBase object to check.

    Other Parameters:
        exclude (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Source paths that match the pattern will not be checked.
        include (Optional[str]): Regex pattern to match the source path (i.e the .yml file where the source is configured). Only source paths that match the pattern will be checked.
        severity (Optional[Literal["error", "warn"]]): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_used_by_only_one_model
        ```

    """

    models: List["DbtBouncerModelBase"] = Field(default=[])
    name: Literal["check_source_used_by_only_one_model"]
    source: "DbtBouncerSourceBase" = Field(default=None)

    def execute(self) -> None:
        """Execute the check."""
        num_refs = sum(
            self.source.unique_id in model.depends_on.nodes for model in self.models
        )
        assert (
            num_refs <= 1
        ), f"Source `{self.source.source_name}.{self.source.name}` is referenced by more than one model."