Tests and numerical validation

Write a test that catches the bug you fixed

A useful test distinguishes correct behavior from a plausible wrong answer. For yaeos, that means testing mathematical contracts, units, array shapes, phase balances, and failure handling—not only checking that an iteration finished or a value is finite.

This guide starts with a small native test, then adds the independent checks needed for thermodynamic and cross-language work. Use workflow for prerequisites and build commands.

Where tests live and how they run

The native tests are standalone Fortran programs under test/, one file per program (test_<feature>.f90). The project no longer uses test-drive suites or a collector program. Every test is an independent executable, so a failure can be run and debugged on its own.

The directories under test/ mirror src/. A test goes in the directory that corresponds to the source it exercises. For example, tests of src/equilibria/flash.f90 are in test/equilibria/, and tests of the cubic mixing rules in src/models/residual_helmholtz/cubic/mixing_rules/ are in test/models/residual_helmholtz/cubic/mixing_rules/. A test that exercises several files goes in the directory of the routine it is mainly about.

Shared helpers are in the local package test/support/, which fpm.toml lists as a dev-dependency. It holds testing_aux (assert, test_title, in test/support/src/yaeos_testing.f90) and the model fixtures. It is a separate package because fpm only lets a test program use modules from its own directory or below. A helper module used by tests in a single directory can stay next to them.

auto-tests is off: every test program has a [[test]] entry in fpm.toml, generated by tools/dev/test_to_toml.py. Each test is named after its file without the test_ prefix (test_flash.f90 is the target flash), so file names must be unique across all directories. CI runs the script with --check and fails if fpm.toml is out of date.

Python tests are under python/tests/. Some compare directly with native Fortran calls by generating small programs in app/. Those tests need a checkout, fpm, and a working native build as well as the Python package.

Add a small native regression

1. Write a standalone program

Save this complete program as test_my_feature.f90, in the test/ subdirectory that mirrors the source it tests (here a residual-model calculation, so test/models/residual_helmholtz/), in a working development branch. It follows the program-based pattern used by test/models/residual_helmholtz/cubic/mixing_rules/test_huron_vidal.f90, test/models/residual_helmholtz/cubic/mixing_rules/test_sddlc.f90, and test/equilibria/test_flash_optional_iters.f90:

program test_my_feature
   use yaeos, only: pr, ArModel, PengRobinson76
   use testing_aux, only: assert, test_title
   use, intrinsic :: ieee_arithmetic, only: ieee_is_finite
   implicit none
   class(ArModel), allocatable :: model
   real(pr) :: n(2)
   real(pr) :: specified_pressure
   real(pr) :: calculated_pressure
   real(pr) :: volume
   real(pr) :: temperature

   write(*, *) test_title("Volume/pressure regression")
   model = PengRobinson76([190.564_pr, 305.32_pr], &
      [45.99_pr, 48.72_pr], [0.01142_pr, 0.0995_pr])
   n = [0.3_pr, 0.7_pr]
   specified_pressure = 10.0_pr
   temperature = 320.0_pr

   call model%volume(n, specified_pressure, temperature, volume, &
      root_type="stable")
   call assert(ieee_is_finite(volume), "Volume must be finite")
   call assert(volume > 0.0_pr, "Volume must be positive")

   call model%pressure(n, volume, temperature, calculated_pressure)
   call assert(ieee_is_finite(calculated_pressure), "Pressure must be finite")
   call assert(abs(calculated_pressure - specified_pressure) < 1.0e-7_pr, &
      "Volume reproduces the specified pressure")
end program test_my_feature

testing_aux is the repository’s test helper module, defined in test/support/src/yaeos_testing.f90, not part of the installed public yaeos API. test_title formats a heading; assert(condition, message) reports a passing check or terminates the program with error stop on failure. A failing prerequisite therefore stops before later calculations use an invalid result. Printing a warning without a nonzero exit status is not a failing test.

For a group of related checks, put private test subroutines after contains and call them explicitly from the program body. There is no collector or suite registry to update.

2. Register and run the program

From the repository root, after saving the file in the directory that mirrors the source it tests:

python tools/dev/test_to_toml.py   # adds the [[test]] entry to fpm.toml
fpm test --list
fpm test my_feature

The target name is the filename without test_, so use a descriptive and unique test_*.f90 name. The script only registers files that contain a program unit and its end program, and it refuses duplicate names. Commit the fpm.toml change with the test. Run the focused target first, then fpm test before review.

Decide what kind of evidence a test provides

Check What it establishes What it does not establish
Compare with independent data Agreement for specified states and parameters Accuracy everywhere
Numerical derivative check Local agreement of derivatives and energy evaluation That the energy equation is the intended physical model
Thermodynamic identity Internal consistency of related quantities Independence of shared implementation errors
Material balance and isofugacity A candidate phase split satisfies its equations Global stability against all other splits
Native/Python comparison Agreement across wrapper paths Independent verification of their shared native kernel
Regression reproducer A particular failure remains fixed Coverage of unrelated edge cases

Use more than one kind of evidence for a new model or solver. Two routines that share the same incorrect derivative can agree perfectly.

Derivatives: test the variables the interface actually uses

For a scalar property , a central-difference estimate is

Choose steps with both an absolute and relative scale so that a zero input still gets a nonzero perturbation. Repeat at several step sizes. Small steps amplify cancellation; large steps introduce truncation error. Neither choosing the smallest possible number nor copying another test’s step blindly is a reliable validation strategy.

Test the amount derivatives required by ArModel and GeModel, not derivatives of normalized mole fractions. For an equilibrium Jacobian in log coordinates, perturb the packed log variable and include its chain-rule factor. Avoid changing volume-root branches during a derivative check intended for a single smooth branch.

Useful native helpers include numeric_ar_derivatives, ar_consistency, numeric_ge_derivatives, ge_consistency, and individual_ar_calls under src/consistency/. Read their interfaces and the cases that call them: different outputs have different scalar, vector, and matrix shapes. individual_ar_calls helps check optional-output combinations; it does not replace a mathematical accuracy check.

For a derivative implementation, cover energy only, each derivative separately, all outputs together, mixed-output requests, and repeated calls at different states. AD implementations need these checks too: differentiation propagates the program’s operations, including mistakes in its primal expression.

Thermodynamic checks that expose amount or reference errors

Scaling and homogeneity

For residual Helmholtz energy at fixed temperature,

Pressure and fugacity coefficients should stay unchanged under that same joint scaling. Scaling amounts without volume changes the density, so it is not the same test.

For an excess-Gibbs model, , whereas activity coefficients are unchanged. These checks help catch a missing total-amount factor or an inappropriate normalization.

Heat-capacity identity

At fixed component amounts, define and . The total heat capacities satisfy . Subtracting the ideal-gas difference gives

Here and . First derivatives of residual Helmholtz energy cannot simply be multiplied to obtain this identity.

This insertable test fragment assumes an initialized model and declared n, V, and T; import pr and R from yaeos and assert from testing_aux. Place the declarations before executable statements. Choose a regular state where dPdV is nonzero:

real(pr) :: P
real(pr) :: dPdT
real(pr) :: dPdV
real(pr) :: Cp
real(pr) :: Cv
real(pr) :: expected_difference

call model%pressure(n, V, T, P, dPdT=dPdT, dPdV=dPdV)
call model%Cp_residual_vt(n, V, T, Cp=Cp)
call model%Cv_residual_vt(n, V, T, Cv=Cv)
expected_difference = -T * dPdT**2 / dPdV - sum(n) * R
call assert(abs(Cp - Cv - expected_difference) < 1.0e-8_pr, &
   "Residual heat capacities satisfy the pressure-derivative identity")

This tests consistency of property relations and derivative paths. An independent derivative check is still needed to diagnose a shared model error. Near singular states, conditioning and cancellation can make this a poor choice of numerical test even when the identity is valid.

Symmetry, Gibbs–Duhem, and limits

For a smooth excess-Gibbs model, the Hessian of is symmetric and . ge_consistency checks these relations and the link between energy and activity coefficients. The matrix of first mole derivatives of is a Jacobian, not a Hessian of .

Check dilute-gas limits where applicable: , , and residual energy tends to zero as density tends to zero at fixed amount and temperature. For activity models using pure-component reference states, check appropriate pure limits and parameter conventions. Use a sufficiently dilute finite state, not a singular zero-volume or zero-amount calculation.

Phase calculations: check physics as well as termination

For an accepted two-phase result, check:

  • Finite, nonnegative phase compositions, each summing to one.
  • Finite phase fractions in their intended domain.
  • Component balance: z = (1-beta)*x + beta*y.
  • Volume balance for a VT flash, on the routine’s specified molar basis.
  • Log-fugacity agreement on the same roots used by the solver.
  • Distinct phases when the requested calculation should not be trivial.

For multiphase output, reconstruct the feed using all phase fractions, including the reference phase. MPEquilibriumState%np counts main phases in the current representation; read the corresponding routine’s contract rather than assuming it is the total count.

Test ordinary cases, relevant boundaries, and a failure path. Keep near-critical conditioning tests distinct from ordinary accuracy tests. A converged root can be metastable, and a finite search that finds no destabilizing trial phase is not a proof of global stability.

Failure conventions differ. Some routines return a sentinel, some return an early result or status field, and some terminate with error stop. Test the specific contract; do not read uninitialized result arrays after a failed call. To test an expected terminating failure, use a subprocess and check its exit status from a harness. Do not add a program that always exits nonzero as an ordinary passing fpm test target: fpm correctly treats that exit as a test failure.

Cross-language checks: shapes before values

Python tests under python/tests/ include native-reference comparisons. Some generate Fortran programs in app/, execute them with fpm, and parse their output with helpers such as parse_vec and parse_mat. Inspect an existing test for the exact protocol instead of inventing another incompatible one.

For an interface change, check:

  1. The same model parameters, state, units, and component ordering on both sides.
  2. Scalar/vector/matrix shape before comparing values.
  3. Matrix orientation with a nonsymmetric test case where appropriate.
  4. Dtype, contiguous layout, and Fortran-order buffers where the wrapper requires them.
  5. Each optional derivative alone and in combination.
  6. The return container and unrequested-output convention, not only its numbers.
  7. Model creation/deletion and invalid-handle behavior where safely defined.

A Python test that agrees with native Fortran validates the adaptation layer, not the physical model independently. Test an installed build after changes to the compiled wrapper; a stale extension can conceal or mimic a regression.

Choose tolerances to match the claim

A practical floating-point comparison uses both scales:

Choose atol in the quantity’s units and rtol for its expected scale. Near-zero consistency residuals need an absolute criterion. Properties spanning orders of magnitude often need a relative criterion too. Do not compare log-fugacity mismatch, temperature error, and material balance with a single unexplained tolerance.

State the reason for a reference value and its tolerance in the test. Consider rounding in published parameters, conditioning, compiler differences, and finite-difference error. Do not loosen a tolerance until a failure disappears without understanding why.

CI runs on both an older and the newest Ubuntu image. The newest images use x86-64-v3 (AVX2/FMA) builds of gfortran and BLAS/LAPACK, and fused multiply-add changes the last bits of results. A limit of a few units in the last place, such as 1e-15 on a thermodynamic identity, can then fail on one machine and pass on another. Two measured examples set the scale:

  • second derivatives compared with central finite differences (step about 1e-4) agree to about 1e-6 relative in every build;
  • the Michelsen-Mollerup identities hold to about 1e-15.

The cubic EoS tests therefore use 1e-5 and 1e-13. Those limits sit well above the rounding noise, and real derivative bugs give errors of order one. To check a test against FMA rounding locally:

fpm test --profile release --flag "-g -ffree-line-length-0 -march=x86-64-v3" my_feature

Adding -ffp-contract=off to the same command disables FMA contraction. If the failure then disappears, rounding is the cause and the tolerance is too tight.

A minimal pre-review test plan

Run the affected target first, then the native suite. Rebuild the Python extension and run relevant Python tests for an interface change. Build the documentation and run or compile its complete examples separately: successful site generation does not execute the tutorial.

In the review summary, report which layers were tested, the revision and environment, and anything unavailable. Coverage is useful for finding unvisited branches, but it cannot establish that the equations, reference data, or assertions are correct.

Measure coverage

CI measures line coverage from the release-profile test run and uploads it to Codecov. The project target is 95%, and generated code such as the Tapenade PC-SAFT derivatives counts toward it. To reproduce the measurement locally (it needs gcovr):

fpm test --profile release --flag "-g -coverage -ffree-line-length-0"
bash ci/ci.sh coverage

ci/ci.sh coverage applies the same exclusions as CI: tests, examples, app/, tools/, the legacy API and the vendored hyperdual module. Use gcovr --html-details with those exclusions to see the uncovered lines. A new test should check a result, not just execute lines.

Continue with workflow, numerical implementation, or the end-to-end change walkthrough.