Skip to content

Directories#

Note

The below checks require manifest.json to be present.

Checks related to source directory and file locations.

Functions:

Name Description
check_source_file_name

Sources must have a file name that matches the supplied regex.

check_source_property_file_location

Source properties files must follow the guidance provided by dbt here.

check_source_file_name #

Sources must have a file name that matches the supplied regex.

Rationale

Consistent file naming conventions (e.g. _stripe__sources.yml) make it easy to identify a source's purpose at a glance. Enforcing naming patterns in CI prevents deviations that accumulate over time and make the project harder to navigate.

Parameters:

Name Type Description Default
file_name_pattern str

Regexp the file name must match. Please account for the .yml extension.

required

Receives at execution time:

Name Type Description
source SourceNode

The SourceNode object to check.

Other Parameters (passed via config file):

Name Type Description
description str | None

Description of what the check does and why it is implemented.

exclude str | list[str] | None

Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Source paths that match any pattern will not be checked.

include str | list[str] | None

Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Only source paths that match any pattern will be checked.

severity Literal[error, warn] | None

Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_file_name
      description: Sources must have a file name that starts with an underscore and ends with `__sources.yml`.
      file_name_pattern: ^_.*__sources\.yml$

Source code in src/dbt_bouncer/checks/manifest/sources/directories.py
@check(code="SO002")
def check_source_file_name(source, *, file_name_pattern: RegexPattern):
    r"""Sources must have a file name that matches the supplied regex.

    !!! info "Rationale"

        Consistent file naming conventions (e.g. `_stripe__sources.yml`) make it easy to identify a source's purpose at a glance. Enforcing naming patterns in CI prevents deviations that accumulate over time and make the project harder to navigate.

    Parameters:
        file_name_pattern (str): Regexp the file name must match. Please account for the `.yml` extension.

    Receives:
        source (SourceNode): The SourceNode object to check.

    Other Parameters:
        description (str | None): Description of what the check does and why it is implemented.
        exclude (str | list[str] | None): Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Source paths that match any pattern will not be checked.
        include (str | list[str] | None): Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Only source paths that match any pattern will be checked.
        severity (Literal["error", "warn"] | None): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_file_name
              description: Sources must have a file name that starts with an underscore and ends with `__sources.yml`.
              file_name_pattern: ^_.*__sources\.yml$
        ```

    """
    compiled = compile_pattern(file_name_pattern.strip())
    file_name = Path(clean_path_str(source.original_file_path)).name
    display = f"{source.source_name}.{source.name}"
    if compiled.match(file_name) is None:
        fail(
            f"`{display}` is in a file (`{file_name}`) that does not match the supplied regex `{file_name_pattern.strip()}`."
        )

check_source_property_file_location #

Source properties files must follow the guidance provided by dbt here.

The properties file must:

  • be located in a staging/ directory directly below a model path, or in one of its subdirectories (e.g. models/staging/stripe/),
  • start with an underscore,
  • contain the name of each subdirectory below staging/, each prefixed with an underscore (e.g. _stripe). A file directly in staging/ has no subdirectory, so this rule does not apply to it,
  • end with __sources.yml.

For example, models/staging/stripe/_stripe__sources.yml passes, while models/marts/stripe/_stripe__sources.yml fails because it is outside models/staging/. Any model path set in model-paths works, not only models.

Rationale

dbt's official project structure guide recommends placing source properties files in the staging layer alongside the models that consume them, named after the source subdirectory (e.g. _stripe__sources.yml). Following this convention makes it immediately obvious where to find a source's configuration, prevents properties files from being scattered across arbitrary directories, and ensures that new team members can navigate the project without guesswork. This check codifies the convention so it is enforced automatically rather than relied upon by convention alone.

Receives at execution time:

Name Type Description
source SourceNode

The SourceNode object to check.

Other Parameters (passed via config file):

Name Type Description
description str | None

Description of what the check does and why it is implemented.

exclude str | list[str] | None

Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Source paths that match any pattern will not be checked.

include str | list[str] | None

Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Only source paths that match any pattern will be checked.

severity Literal[error, warn] | None

Severity level of the check. Default: error.

Example(s):

manifest_checks:
    - name: check_source_property_file_location

Source code in src/dbt_bouncer/checks/manifest/sources/directories.py
@check(code="SO003")
def check_source_property_file_location(source):
    """Source properties files must follow the guidance provided by dbt [here](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview).

    The properties file must:

    - be located in a `staging/` directory directly below a model path, or in one of its subdirectories (e.g. `models/staging/stripe/`),
    - start with an underscore,
    - contain the name of each subdirectory below `staging/`, each prefixed with an underscore (e.g. `_stripe`). A file directly in `staging/` has no subdirectory, so this rule does not apply to it,
    - end with `__sources.yml`.

    For example, `models/staging/stripe/_stripe__sources.yml` passes, while `models/marts/stripe/_stripe__sources.yml` fails because it is outside `models/staging/`. Any model path set in `model-paths` works, not only `models`.

    !!! info "Rationale"

        [dbt's official project structure guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview) recommends placing source properties files in the staging layer alongside the models that consume them, named after the source subdirectory (e.g. `_stripe__sources.yml`). Following this convention makes it immediately obvious where to find a source's configuration, prevents properties files from being scattered across arbitrary directories, and ensures that new team members can navigate the project without guesswork. This check codifies the convention so it is enforced automatically rather than relied upon by convention alone.

    Receives:
        source (SourceNode): The SourceNode object to check.

    Other Parameters:
        description (str | None): Description of what the check does and why it is implemented.
        exclude (str | list[str] | None): Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Source paths that match any pattern will not be checked.
        include (str | list[str] | None): Regex pattern(s) to match the source path (i.e the .yml file where the source is configured). Only source paths that match any pattern will be checked.
        severity (Literal["error", "warn"] | None): Severity level of the check. Default: `error`.

    Example(s):
        ```yaml
        manifest_checks:
            - name: check_source_property_file_location
        ```

    """
    original_path = Path(clean_path_str(source.original_file_path))
    display = f"{source.source_name}.{source.name}"

    # dbt only reads source YAML from `model-paths`, so the first part is always
    # the model path. It is not checked, so any `model-paths` value works.
    if not (len(original_path.parts) > 2 and original_path.parts[1] == "staging"):
        # With no parent directory, parts[0] is the file name, not a model path.
        model_path = (
            original_path.parts[0] if len(original_path.parts) > 1 else "<model path>"
        )
        fail(
            f"The properties file for `{display}` (`{original_path.as_posix()}`) is not located in `{model_path}/staging/`."
        )

    subdir_parts = original_path.parent.parts[2:]
    expected_substring = "_" + "_".join(subdir_parts) if subdir_parts else ""
    properties_yml_name = original_path.name

    if not properties_yml_name.startswith("_"):
        fail(
            f"The properties file for `{display}` (`{properties_yml_name}`) does not start with an underscore."
        )
    if expected_substring not in properties_yml_name:
        fail(
            f"The properties file for `{display}` (`{properties_yml_name}`) does not contain the expected substring (`{expected_substring}`)."
        )
    if not properties_yml_name.endswith("__sources.yml"):
        fail(
            f"The properties file for `{display}` (`{properties_yml_name}`) does not end with `__sources.yml`."
        )