                     Automated Testing Framework (atf)
                        The NetBSD Foundation, Inc.
---------------------------------------------------------------------------


Introduction
============

This document contains a *non-exhaustive* list of things that should be
addressed in ATF before it can be considered stable.  Some of the items
are planned for specific future releases, but take this with a big grain
of salt: the focus of a specific release may change completely depending
on how things develop.


Plans for 0.6
=============

* Add a binary atf-check tool that replaces atf-sh's atf_check function.
  I suspect this will speed things up considerably, and will also expose
  this extremely useful functionality to C/C++ test programs.

* Add support for ATF_REQUIRE, making ATF_CHECK non-fatal.  See Boost.Test
  for details on this idea.  Most useful for unit tests.

* Add a module to atf-c to manage dynamic memory.  Should provide a "mem
  chunk" object that can only be managed through functions (i.e. not
  directly) so that access to buffers can be safely controlled.  Dealing
  with strdup and similar functions, for example, might be complex.

  See these old revisions for a start, but these did not work correctly
  because the use of (void **) casts brought aliasing problems:

  78eeacf3b9284493e5e96b7866fc1d163a13bc02
  8e200683a2b70eadca728c2e0347d4790777b3fc
  872393ed0177cbcc30ffacede228eb3986c42ab7

* Optimize timeout handling in atf-sh: there are too many processes
  involved to run each test case, and this hurts a lot in OS X, for
  example.  One possibility is to integrate the timeout handling in
  atf-exec and then kill the auxiliary code in atf.footer.subr.


Plans for 0.7
=============

* Add a tool to collect multiple outputs of atf-run (from different
  machines) and generate a single (XML-only?) log with everything.
  Must allow easy conversion to HTML for online publishing.

* Allow grouping of test programs in tiers in an Atffile.  This is to
  permit the user specify "dependencies" between test programs: e.g. do
  not run a specific test program if its dependencies have failed, because
  it will certainly fail also.

* Provide a kernel-level unit testing API (for NetBSD only, at the moment).
  This should come in the form of an atf.ko module that provides functions
  to define and register test cases, functions for results reporting and
  an interface (a trivial file system?) that transports the
  application/X-atf-tcs output to user-space, provides information to
  user-space about the available test cases (a list) and allows user-space
  to launch the execution of test cases.

* Add expected failures and unexpected passes as test case results?  The
  former are used to indicate known failures at specific points and thus
  not raise false positives when working on some unrelated feature.  The
  latter are used to detect changes that fixed expected failures.  See
  Monotone's test suite for an example.


Plans for pre-1.0
=================

* Fix all occurrences of XXX, TODO and FIXME.

* Think of a way to properly add tests for (almost?) all error paths.
  Most of them are probably broken already.

* Improve error reporting: aside from clarifying error messages, this also
  implies adding more error cases to give them more semantic meaning at the
  source code level..

* Add an atf_check-like function to the C++ binding to allow easy testing
  of executables.

* Make the shell library work with 'set -e'?

* Shell test programs dynamically locate where the shell library is by
  calling atf-config (done by atf.init.subr).  Contrarywise, binary test
  programs are directly linked against the final location of libatf.
  It may be nice if the latter loaded the library dynamically based on
  what atf-config said, so the user could switch atf installations by
  simply changing its PATH (and effectively making atf relocatable on the
  file system).  Why could this be nice?  To painlessly run an older atf
  test suite against a more recent version of the code base to ensure
  there are no regressions even with older tests.  Just a crazy idea, as
  maybe what the shell test programs currently do is stupid.

* Allow users to customize the build of atf by defining additional meta-data
  for test cases.  At the moment this is possible because the meta-data is
  not sanity-checked, but I think it should be.  Following the previous
  item, NetBSD could add a 'netbsd.pr' variable and then use this data when
  generating reports to add direct links to the appropriate PRs.

* Make sure that the tests in tests/atf have, at the very least, the same
  coverage as the ones in tests/bootstrap.

* Document the code.

* Possibly add a way to automatically gain or drop privileges when
  require.user is set.

* Add a way to specify which bug/issue/whatever a given test case is
  stress-testing.  This information is useful when detecting regressions.


Plans for 1.0-RC1
=================

* Build libatf as a shared library and set -version-info accordingly.

* Set the DTDs' versions to 1.0.


Plans for post-1.0
==================

* Allow the parallel execution of tests.  Achieving this with a test
  program granularity is easy: only need to change atf-run.  Lowering it
  to a finer granularity (test cases) is harder and maybe not worth it.


Things that we will not do
==========================

This is a list of things that will *not* be addressed anytime soon in the
project.  Of course most of them would be nice to have in the future, but
they will not block releases nor drive development.  We can obviously
change our mind in the future and add move some of these to the above list.

* Native Win32 support: we are not talking about building atf with tools
  such as Cygwin or MinGW, which should already be possible.  Native Win32
  support means modifying the code to use Win32 library calls and easily
  build out of the box (i.e. the GNU Autotools are not useful in that
  case).

* Extreme efficiency: even though we will focus on using the most suitable
  data structures in each situation, we will not attempt to get extreme
  efficiency by adding hacks that make the code uglier.  Testing is a task
  that requires a lot of resources on its own, and some tests will be
  specially intensive, so there is no point in micro-optimizing atf if the
  global gains are negligible.

* Extreme portability: the initial goal of this project was to provide a
  testing framework for the NetBSD Operating System.  Given that this
  system has a very nice build framework, we can be sure that atf will be
  built and used with sane tools (basically a decent C++ compiler and a
  POSIX-compliant shell interpreter).  We will definitely not aim for
  portability to broken systems by tweaking our code to not follow the
  standards.


---------------------------------------------------------------------------
                              End of document
