Validity Checks

Validity checks ensure that data conforms to expected formats, ranges, and patterns. These checks are essential for data type validation, format verification, and business rule compliance.

Follow these best practices for validity checks:

• Start with common patterns available in the UI before creating custom ones.

• Test regex patterns on sample data before implementing checks.

• Consider case sensitivity in pattern matching.

• Allow flexibility where appropriate (e.g., international phone formats).

• Use percentage-based thresholds for large datasets that may have some exceptions.

Available Metrics

• Invalid Count: Count of values not matching validation rules.

• Invalid Percentage: Percentage of invalid values.

• Valid Format: Validates against a regex pattern.

• Valid Values: Validates against a list of allowed values.

Configuration Examples

Check Description	Configuration Steps	Example Check
Validate email format	Select email column. Choose Invalid Count with regex pattern. Enter pattern: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}. Select = operator. Set threshold to 0.	invalid_count(email) with pattern = 0
Ensure valid status values	Select status column. Choose Valid Values check. Enter valid values: PENDING,APPROVED,REJECTED,CANCELLED. Select = operator. Set threshold to 0 for invalid count.	invalid_count(status) not in valid values = 0
Validate phone number format	Select phone column. Choose Invalid Percentage with regex pattern. Enter pattern: ^\+?[1-9]\d{1,14}. Select <= operator. Set threshold to 1.	invalid_percent(phone) with pattern <= 1

Common Validation Patterns

Data Type	Regex Pattern	Description
Email	^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}	Standard email validation
US Phone	^(?:\([0-9]{3}\))?[-. ]?([0-9]{3})[-. ]?([0-9]{4})	US phone number format
Date (YYYY-MM-DD)	^\d{4}-(0[1-9]	1[0-2])-(0[1-9]
ZIP Code	^\d{5}(-\d{4})?	US ZIP code with optional +4
URL	^(https?://)?([\\da-z\\.-]+)\\.([a-z\\.]{2,6})([/\\w \\.-]*)*/?	Web URL validation