Changelog

​

Bug fixes

• upgrade dependencies to clear high-severity audit findings (evals) (3316b84)
  • bump effect, langchain, and socket.io in core; effect in evals
  • upgrade vitest 4 and tsup 8 across packages; turbo at root
  • add pnpm overrides for transitive advisories (axios, protobufjs, etc.)
  • align example apps and website on patched next releases
  • fix ui vitest 4 test options API in AiCursor spec

​

Bug fixes

• hotfixed ci (evals) (663083c)

​

New features

• introduction of tag filters (evals) (6f71580)

​

Bug fixes

• datasets now accept top level and filter (evals) (0c110fa)

​

Improvements

• update import paths to use .js extensions for consistency (97e481c)
• update import paths to use .js extensions for consistency (evals) (6dc73ac)

​

New features

• introduced sampling (evals) (4fd81ac)

​

New features

• introduced grouped test case export (evals) (0505039)

​

New features

• introduced triggeredAt (evals) (6258460)
• introduced trigger timestamp (evals) (e0cabeb)

​

New features

• introduced test case name in meta (evals) (a7959dc)
• introduced experiment name (evals) (c2bf0d7)
• adjustment of dataset and test case naming (evals) (f941bd7)
• introduction of ci mode (evals) (3aa1ec1)
• introduction of tags (evals) (3b82060)
• new naming convention (evals) (24f2dd9)
• new run structure with run configs (evals) (65e46db)

​

New features

• new features and project restructure (c5bdf34)

​

New features

• removed cpu concurrency defintion (evals) (189299e)
• improved concurrency for evals (evals) (50c3bcf)

​

Bug fixes

• replaced the json diff package due to invalid package hashes (evals) (7a12092)

​

New features

• integrate json-diff for enhanced diff logging and evaluation (evals) (0050228)
  • Replaced ‘diff’ dependency with ‘json-diff’ for improved diff functionality.
  • Added logging capabilities to evaluators, allowing for detailed message capture during test execution.
  • Updated evaluator interface to include a new ‘log’ method for logging messages and objects.
  • Enhanced RunDetailsView and RunView to display log messages alongside diffs, improving clarity in test case results.
  • Introduced options for customizing diff output, including sorting and key exclusion.
  • Added tests for new diff and logging functionalities to ensure reliability.
• enhance RunDetailsView and RunView for better score and metric display (evals) (5bf9416)
  • Updated RunDetailsView to conditionally render scores and metrics, improving clarity in test case results.
  • Modified RunView to aggregate and display evaluator scores and metrics, ensuring a more informative output.
  • Introduced handling for cases with no scores, enhancing user experience in the CLI.
  • Improved overall structure and readability of score and metric presentation in the CLI.
• update dependencies and enhance CLI functionality (evals) (2d29850)
  • Added ‘@pppp606/ink-chart’ and ‘diff’ as dependencies.
  • Removed ‘json-diff’ dependency in favor of ‘diff’.
  • Introduced overview row count management in the CLI for better navigation.
  • Enhanced DatasetsView to display run scores and trends using LineGraph.
  • Updated score aggregation to include standard deviation for better insights.
  • Improved overall CLI experience with refined navigation and display logic.

​

New features

• enhance test case handling with reruns and aggregation support (evals) (fb4cfb2)
• improved cli (evals) (471394a)
• improved cli (evals) (5f077da)

​

New features

• improved cli (evals) (a94da38)
• added diffing (evals) (9a9263f)

​

New features

• update version to 0.7.0, add output handling in evaluator, and enhance test case structure (#27) (evals) (eaf9d8f)
  • Bump package version to 0.7.0 in package.json
  • Introduce output handling in Evaluator and TestCase classes
  • Update README and examples to reflect new output features
  • Refactor evaluator tests to accommodate new input structure
  • Add output schema and output handling in test cases for better validation
• first version (#26) (evals) (9146b35)
  • feat(evals): first version
  • fix: for tests