Changelog¶

All notable changes to this project will be documented in this file.

Unreleased¶

Fixed¶

• String checks ignore null values consistently, including pandas object columns with newer Narwhals versions.

3.1.0¶

A six-angle audit of the codebase found four bugs where validation silently did not happen or crashed outside daffy's error contract. Everything here either closes one of those, or makes a failure say what went wrong.

Migrating: column specs that were quietly doing nothing now raise when the decorator is applied — a misspelled constraint ({"nullabel": False}), a non-boolean nullable/unique/ required, or a check whose dtype it cannot mean anything on. Each one was a guard that never ran, so the error is reporting a bug you already had.

Fixed¶

• Overlapping column specs no longer drop a constraint. {"r/^price/": {"checks": {"gt": 0}}, "price_eur": {"checks": {"lt": 100}}} applied only one of the two checks, and which one depended on dict insertion order. Every spec that matches a column now applies. The same overlap crashed nullable and unique with a Narwhals DuplicateError rather than an AssertionError.
• strict=True is no longer ignored when the column spec is empty. @df_in([], strict=True) permitted any frame; an empty spec now means no columns are permitted.
• Duplicate column names no longer escape as a Narwhals error. @df_in/@df_out report them as an AssertionError, and @df_log — which only logs — warns and leaves the function working instead of raising.
• A check that cannot mean anything on a column's dtype is now rejected on every backend. An ordering check on a string column, or a str_* check on a non-string column, previously raised a raw backend error on Pandas and PyArrow while Polars compared lexicographically and reported a violation that looked real. Other backend failures during a check are wrapped with the check name, dtype and value.
• Row validation no longer understates catastrophic failures. With early termination, 10,000 failing rows out of 10,000 reported "at least 6 … and 1 more row(s)". It now says counting stopped rather than inventing a remainder.
• The decorators no longer erase type information. They returned Callable[..., T], which silently disabled argument type checking at every call site of a decorated function. ColumnsDef also rejected any spec that was not an inline literal, including dict[str, ColumnConstraints].
• Errors past a *args boundary named the wrong parameter, and wrong-type errors did not name the function.

Changed¶

• Unusable column specs are rejected when the decorator is applied. Misspelled constraint names, non-boolean nullable/unique/required values, and misspelled built-in check names all used to be accepted and then do nothing. Custom checks are unaffected: a callable check value may carry any name.

Performance¶

• Another ~7% off per-call overhead (@df_in(columns=...) ~41µs → ~38µs, measured on a 100-row Pandas frame) by removing SkippableValidator, a protocol with no implementations that cost a runtime_checkable isinstance per validator per call. Total since 2.8.0 is ~212µs → ~38µs.

Removed¶

• daffy.validators.SkippableValidator and the should_skip hook. Nothing in daffy implemented it, and ValidationPipeline no longer calls it. It is removed rather than deprecated deliberately: a compatibility shim would keep from daffy.validators import SkippableValidator working while silently never skipping anything, which is the failure mode this release exists to remove. Validators already express "nothing to do" by returning [], and the builder omits validators with no work.

Documentation¶

• Corrected five quoted error/log outputs that no longer matched what daffy prints, fixed three README documentation links that 404'd, and documented strict_specs, the three config keys missing from the API reference, ColumnConstraints, and the fact that config discovery is relative to the working directory.

3.0.0¶

This release removes two pieces of hidden magic and makes validation roughly five times cheaper. Mutation testing found all three behavior changes: the test suite asserted none of them, and the documentation described none of them.

Migrating: anchor any unanchored regex you relied on (r"\d+" → r"^\d+", r/Price_\d+/ → r/^Price_\d+/), and add nullable=False to columns where you were relying on a null failing a value check.

Changed¶

• Behavior change: value checks no longer force nulls to count as failures. Daffy used to rewrite null comparison results into violations; it now leaves each backend's semantics intact, so on Polars, PyArrow and Pandas nullable dtypes a comparison against null is "unknown" and is not reported. Pandas float NaN compares as False and still fails. Use nullable=False or the notnull check to constrain nulls explicitly — those behave identically on every backend.
• Behavior change: regex column patterns ("r/pattern/") now match anywhere in the column name instead of only at the start, so r/Price_\d+/ also matches Total_Price_1. Anchor the pattern yourself (r/^Price_\d+/) to keep the old meaning. This makes column patterns and str_regex follow the same rule.
• Behavior change: str_regex now applies the pattern as written instead of silently anchoring it at the start of the value. {"str_regex": r"\d+"} used to mean "starts with digits" and now means "contains digits"; add the anchor yourself (r"^\d+", or r"^\d+$" for a full match) to keep the old meaning. Patterns that were already anchored are unaffected.

Fixed¶

• Parameterised dtypes can now be declared by base name: {"created_at": "datetime"} matches any time unit or time zone, and the same applies to duration, list, struct and enum. Previously the only accepted spelling was the full internal repr ("datetime(time_unit='ns', time_zone=none)"). Spelling the parameters out still constrains them.
• String checks (str_regex, str_startswith, str_endswith, str_contains) no longer raise TypeError: bad operand type for unary ~ on Pandas object-dtype columns containing nulls

Performance¶

• Per-call validation overhead cut by roughly 5x: @df_in(columns=...) on a 100-row Pandas frame went from ~212µs to ~41µs, @df_in(min_rows=...) from ~216µs to ~44µs. Configuration is now resolved in one lookup per call instead of one per setting, the DataFrame is converted to Narwhals once instead of three times, and column dtypes are only read when a dtype constraint needs them.

Removed¶

• daffy.config.get_strict, get_lazy, get_strict_specs and get_allow_empty — internal helpers superseded by resolve_decorator_settings. They were never exported from the daffy package.

Documentation¶

• Documented null handling per backend, str_regex anchoring, how to declare parameterised dtypes, and that regex syntax support depends on the backend's own engine

2.8.0¶

Changed¶

• Significant runtime performance improvements to minimize decorator overhead
• Eliminated repetitive Python introspection (inspect.signature) from the execution path by resolving parameters during decoration
• Vectorized validation checks (nullable and unique) across multiple columns simultaneously using Narwhals expression API (.select())
• Avoided redundant dataframe wrappers when running value checks
• Optimized the validation builder for faster pipeline setup and eliminated unused dtype resolution overhead

2.7.0¶

Added¶

• @df_in(["col1", "col2"]) shorthand: pass columns as the first positional argument
• Opt-in strict_specs mode ([tool.daffy] strict_specs = true): raises TypeError on invalid column spec keys or types instead of silently ignoring them

Changed¶

• Config discovery now searches pyproject.toml from the current directory up through parent directories
• Configuration caching is now keyed by current working directory for deterministic behavior across cwd changes
• Unnamed df_in now selects the first DataFrame-like argument instead of always taking the first positional argument

2.6.1¶

Changed¶

• Optional dependency detection now recognizes all advertised backends (Pandas, Polars, Modin, PyArrow)
• Early import failure now triggers only when none of the supported DataFrame libraries are installed
• Updated optional dependency tests and isolation docs/scripts to align with the expanded backend detection contract
• Synced API docs and README examples with current df_in/df_out/df_log signatures and built-in check names

Added¶

• Declared pydantic optional dependency extra so pip install 'daffy[pydantic]' matches runtime install guidance
• Added docs contract tests to detect signature/example drift in README.md and docs/api.md

2.6.0¶

Added¶

• Support for pandas 3.x (requires Python 3.11+)

2.5.1¶

Internal Improvements¶

• Simplified config loading by using loops over key lists instead of repetitive if-statements
• Added _get_int_config helper to deduplicate integer config getters
• Extracted _log_dataframe helper from duplicate logging functions
• Replaced repeated expand+add blocks in pipeline builder with data-driven loop

2.5.0¶

Internal Improvements¶

• Major architecture refactoring: Introduced validation pipeline with protocol-based validators
• New ValidationPipeline orchestrates all validation steps
• New ValidationContext provides single narwhals conversion point for efficiency
• Validators follow a clean protocol (Validator, SkippableValidator)
• Pipeline builder assembles validators from column specs
• Removed dead code and unused helper functions
• Removed unused type aliases (ColumnsList, ColumnsDict)
• Replaced skylos with vulture for dead code detection in CI

2.4.1¶

Internal Improvements¶

• Enabled ALL Ruff linting rules with selective ignores for comprehensive code quality checks
• Refactored validate_dataframe to reduce cyclomatic complexity
• Migrated from os.path to pathlib in configuration module
• Performance optimization: extracted try/except from validation loop (PERF203)

2.4.0¶

New Features¶

• Shape validation: enforce row count constraints on DataFrames
• min_rows: require a minimum number of rows
• max_rows: limit the maximum number of rows
• exact_rows: require an exact number of rows
• allow_empty: control whether empty DataFrames (0 rows) are allowed
• New allow_empty configuration option in pyproject.toml (default: true)

Internal Improvements¶

• Added property-based tests with Hypothesis for mathematical invariants
• Added boundary-value and default-parameter tests for improved test coverage

2.3.1¶

Bug Fixes¶

• Fixed incomplete DataFrameType union that only included Pandas and Polars, missing PyArrow and Modin. Now uses narwhals' IntoDataFrame Protocol which correctly accepts all supported DataFrame types.

2.3.0¶

Breaking Changes¶

• Dropped Python 3.9 support - Python 3.10 is now the minimum required version (Python 3.9 reached end-of-life in October 2025)

Internal Improvements¶

• Modernized type annotations using Python 3.10 syntax (X | Y instead of Union[X, Y], explicit TypeAlias)
• Updated ruff target version to Python 3.10

2.2.0¶

New Features¶

• Custom check functions: use lambdas for validation logic not covered by built-in checks
• The function receives a Narwhals Series and returns a boolean Series (True = valid)
• Example: {"price": {"checks": {"no_outliers": lambda s: s < s.mean() * 10}}}
• The dictionary key becomes the check name in error messages
• Lazy validation: collect all errors before raising with lazy=True parameter
• Use @df_in(columns=..., lazy=True) or @df_out(columns=..., lazy=True)
• Configurable via [tool.daffy] lazy = true in pyproject.toml
• Useful for debugging DataFrames with multiple issues
• Composite uniqueness: validate that column combinations are unique
• Use composite_unique=[["col1", "col2"]] to ensure column combinations are unique
• Works alongside single-column unique=True validation

Bug Fixes¶

• Fixed value checks (gt, lt, ge, le, between, eq, ne, etc.) not working with PyArrow tables

Internal Improvements¶

• Config validation: pyproject.toml settings now validated for correct types (prevents silent bugs like strict = "false" being treated as truthy)
• Regex pattern caching: compiled patterns are now cached, improving performance when validating many DataFrames with the same column patterns
• Empty regex patterns (r//) now raise a clear error instead of matching everything
• Improved error handling for custom check functions

2.1.0¶

New Features¶

• Added new value checks: notin, str_startswith, str_endswith, str_contains, str_length

2.0.2¶

Documentation¶

• Released MkDocs documentation at daffy.readthedocs.io
• Added missing attr_list, md_in_html, and pymdownx.emoji markdown extensions
• Improved SEO: updated README title/intro, added PyPI keywords, added GitHub topics

2.0.1¶

Documentation¶

• Simplified installation instructions - Daffy detects and works with whatever DataFrame library you have installed
• Updated PyPI keywords to include modin, pyarrow, and narwhals
• Added note that examples work with all supported backends (Pandas, Polars, Modin, PyArrow)

2.0.0¶

Major Refactoring¶

• Migrated to Narwhals - Major internal refactoring to use Narwhals as a unified DataFrame abstraction layer
• Narwhals is a lightweight compatibility layer used by Plotly, Altair, Bokeh, and Marimo
• Now supports Modin and PyArrow in addition to Pandas and Polars
• All existing functionality for Pandas and Polars remains unchanged
• Public API (df_in, df_out, df_log) is fully backwards compatible

Changed¶

• df_log with include_dtypes=True now shows unified Narwhals dtype representation (e.g., [String, Int64]) for both Pandas and Polars DataFrames
• Previously Pandas showed ['object', 'int64'] while Polars showed [String, Int64]
• This provides consistent logging output across all DataFrame libraries

Dependencies¶

• Added narwhals>=2.14.0 as a required dependency

Removed¶

• row_validation_convert_nans configuration option - NaN-to-None conversion is no longer needed. Pydantic handles NaN values correctly: accepts them for Optional[float] fields and correctly fails for constraints like Field(gt=0)

1.4.0¶

New Features¶

• Vectorized value checks - Fast validation of column values using vectorized DataFrame operations
• Comparison checks: gt, ge, lt, le (greater than, greater or equal, less than, less or equal)
• Range check: between for inclusive range validation
• Equality checks: eq, ne for exact value matching
• Set membership: isin for validating values against a list
• Null check: notnull for ensuring no null values
• Regex matching: str_regex for pattern validation on string columns
• Use rich column spec format: {"price": {"checks": {"gt": 0, "lt": 10000}}}
• Combine multiple checks: {"age": {"checks": {"ge": 0, "le": 120}}}
• Works with other validations: {"price": {"dtype": "float64", "nullable": False, "checks": {"gt": 0}}}
• Supports regex column patterns: {"r/score_\\d+/": {"checks": {"between": (0, 100)}}}
• Checks are skipped for missing optional columns
• Error messages include sample failing values for debugging
• Configurable via checks_max_samples in pyproject.toml

1.3.0¶

New Features¶

• Optional columns - Mark columns as optional with required=False
• Use rich column spec format: {"column": {"required": False}}
• Optional columns are not required to exist in the DataFrame
• If present, all other validations (dtype, nullable, unique) still apply
• Works with regex patterns: {"r/extra_\\d+/": {"required": False}}
• Default behavior unchanged (required=True, columns must exist)

1.2.0¶

New Features¶

• Uniqueness column validation - Validate that columns contain only unique values
• Use rich column spec format: {"column": {"unique": True}}
• Combines with dtype and nullable: {"column": {"dtype": "int64", "unique": True, "nullable": False}}
• Works with regex patterns: {"r/ID_\\d+/": {"unique": True}}
• Default behavior unchanged (unique=False, duplicates allowed)

1.1.0¶

New Features¶

• Nullable column validation - Validate that columns contain no null values
• Use rich column spec format: {"column": {"nullable": False}}
• Combines with dtype validation: {"column": {"dtype": "float64", "nullable": False}}
• Works with regex patterns: {"r/Price_\\d+/": {"nullable": False}}
• Default behavior unchanged (nullable=True, nulls allowed)

1.0.0¶

Stable Release¶

Daffy 1.0.0 marks the first stable release. The public API (df_in, df_out, df_log) is now considered stable and follows semantic versioning.

Changed¶

• Development status upgraded from Beta to Production/Stable
• Updated documentation to reflect current tooling (uv, ruff, pyrefly)

Fixed¶

• Improved error handling for invalid regex patterns in column specifications
• Better error messages when parameter extraction fails

Internal¶

• Extracted duplicate row validation logic into shared helper function
• Added docstrings to public-facing utility functions

API Stability¶

As of 1.0.0, Daffy follows semantic versioning:

• Major versions (2.0, 3.0) may contain breaking changes
• Minor versions (1.1, 1.2) add features without breaking changes
• Patch versions (1.0.1, 1.0.2) contain bug fixes only

0.19.0¶

Performance Improvements¶

• Early termination for row validation - Dramatically faster when validation errors exist
• Stops scanning after collecting max_errors (default behavior)
• 71-124x speedup when errors are present (1.2ms vs 140ms for 100k rows with errors)
• No overhead for valid data (maintains 767K rows/sec throughput)
• Can be disabled with early_termination=False parameter for exact error counts
• Error messages now indicate when scanning stopped early: "stopped scanning early (at least N more row(s) with errors)"

New Features¶

• Added early_termination parameter to validate_dataframe_rows()
• Default: True (stops after max_errors for performance)
• Set to False to scan entire DataFrame and get exact error count

0.18.0¶

Major Performance Improvements¶

• 2x faster row validation - Optimized DataFrame conversion and validation pipeline
• 767K rows/sec on simple validation (was 400K)
• 165K rows/sec on complex bioinformatics data (32 columns, 5% missing values)
• Changed from batch TypeAdapter validation to optimized row-by-row with fast DataFrame iteration
• Use itertuples() for efficient row access while preserving None values

Critical Bug Fix¶

• Fixed NaN handling for Optional fields - NaN values in Optional fields are now properly converted to None
• Previous implementation failed validation on legitimate missing data
• Now correctly handles nullable numeric fields in pandas DataFrames
• Converts numeric columns with NaN to object dtype to preserve None values

New Features¶

• Bioinformatics benchmark (scripts/benchmark_bioinformatics.py) - Realistic validation testing
• 32-column feature store schema modeling cancer research data
• Gene expression, clinical measurements, patient demographics, mutations, outcomes
• Mixed types: floats, ints, strings, bools, Optional fields, Literal enums
• Missing data patterns (~5%) typical in real-world datasets

• Cross-field validation (e.g., disease-free survival ≤ follow-up time)

• Performance benchmarking suite - Compare Daffy against competing libraries

• Test multiple scenarios (simple, medium complexity)
• Multiple dataset sizes (1k, 10k, 100k rows)
• Compare pandas vs polars implementations

Internal Improvements¶

• Removed unused internal functions (_pandas_to_records_fast, _iterate_dataframe_with_index)
• Simplified error formatting (removed dead code branches)
• Improved test coverage to 98.34%
• Better handling of edge cases in validation error reporting

0.17.1¶

• Improve package discoverability on PyPI and GitHub
• Add polars, pydantic, decorator to PyPI keywords
• Add Framework::Pydantic, Testing, and Typing classifiers
• Add changelog and issues URLs to package metadata
• Set GitHub topics for better search visibility

0.17.0¶

• Add optional row-level validation using Pydantic models (requires Pydantic >= 2.4.0)
• New row_validator parameter for @df_in and @df_out decorators
• Validates actual data values, not just column structure
• Batch validation for optimal performance (10-100x faster than row-by-row)
• Informative error messages showing which rows failed and why
• Configuration via pyproject.toml: row_validation_max_errors and row_validation_convert_nans
• Works with both Pandas and Polars DataFrames

0.16.1¶

• Internal refactoring: extracted DataFrame type handling to dedicated module for better code organization and maintainability

0.16.0¶

• Removed Pandas and Polars from required dependencies. Daffy will not pull in Polars if your project just uses Pandas and vice versa. All combinations are dynamically supported and require no changes from existing users.

Testing & CI¶

• Added comprehensive CI testing for all dependency combinations
• New test suite validates optional dependency behavior
• Manual testing script for developers (scripts/test_isolated_deps.py)
• Updated CI to test pandas-only, polars-only, both, and none scenarios

0.15.0¶

• Exception messages now include function names to improve debugging
• Input validation: "Missing columns: ['Col'] in function 'my_func' parameter 'param'. Got columns: ['Other']"
• Return value validation messages now clearly state "return value" instead of just showing function name
• Output validation: "Missing columns: ['Col'] in function 'my_func' return value. Got columns: ['Other']"

0.14.2¶

• Internal code quality improvements

0.14.1¶

• Internal code quality improvements

0.14.0¶

• Improve df_in error messages to include parameter names

0.13.2¶

• Updated urls for Pypi site compatibility

0.13.1¶

• Update documentation
• Update dependencies

0.13.0¶

• Fix type annotation issues with decorator parameters that could cause type errors in strict type checking
• Use Sequence instead of List for better type variance compatibility
• Add test case that validates type compatibility

0.12.0¶

• Add support for regex patterns used with column dtype validation

0.11.0¶

• Update function parameter types for better type safety
• Fix missing return statement in df_log decorator
• Added stricter mypy type checking settings

0.10.1¶

• Built and published with UV. No functional changes

0.10.0¶

• Add support for regex patterns in column name validation

0.9.4¶

• Fix to strict flag loading when tool config was missing

0.9.3¶

• Add configuration system to set default strict mode in pyproject.toml
• Improve logging when multiple columns are missing

0.9.2¶

• Add explicit __all__ export for functions to make Mypy happy

0.9.0¶

• Add marker (py.typed) to tell Mypy that the library has type annotations
• Fix bug when using strict parameter and no name parameter in @df_in

0.8.0¶

• Support Polars DataFrames

0.7.0¶

• Support Pandas 2.x
• Drop support for Python 3.7 and 3.8
• Build and test with Python 3.12 also

0.6.0¶

• Make checking columns of multiple function parameters work also with positional arguments (thanks @latvanii)

0.5.0¶

• Added strict parameter for @df_in and @df_out

0.4.2¶

• Added docstrings for the decorators
• Fix import of @df_log

0.4.1¶

• Add include_dtypes parameter for @df_log.
• Fix handling of empty signature with @df_in.

0.4.0¶

• Added @df_log for logging.
• Improved assertion messages.

0.3.0¶

• Added type hints.

0.2.1¶

• Added Pypi classifiers.

0.2.0¶

• Fixed decorator usage.
• Added functools wraps.

0.1.0¶

• Initial release.