Commit 08334c51 authored by Brooks Davis's avatar Brooks Davis
Browse files

Import the kyua testing framework for infrastructure software

Imported at 0.13 plus assumulated changes to git hash a685f91.

Obtained from:	https://github.com/jmmv/kyua
Sponsored by:	DARPA
parents
*.a
*.o
*_helpers
*_inttest
*_test
*~
.deps
.dirstamp
Doxyfile
Makefile
Makefile.in
aclocal.m4
api-docs
autom4te.cache
config.h
config.h.in
config.log
config.status
configure
kyua
local-kyua
stamp-h1
language: cpp
sudo: required
before_install:
- ./admin/travis-install-deps.sh
matrix:
include:
- os: linux
dist: xenial
compiler: clang
env: ARCH=amd64 DO=distcheck AS_ROOT=no
- os: linux
dist: xenial
compiler: gcc
env: ARCH=amd64 DO=distcheck AS_ROOT=no
- os: linux
dist: xenial
compiler: clang
env: ARCH=amd64 DO=apidocs
- os: linux
dist: xenial
compiler: clang
env: ARCH=amd64 DO=style
- os: linux
dist: xenial
compiler: clang
env: ARCH=amd64 DO=distcheck AS_ROOT=yes UNPRIVILEGED_USER=no
- os: linux
dist: xenial
compiler: clang
env: ARCH=amd64 DO=distcheck AS_ROOT=yes UNPRIVILEGED_USER=yes
# TODO(ngie): reenable i386; the libraries were not available in the
# Ubuntu Xenial x86_64 docker image.
#- os: linux
# dist: xenial
# compiler: clang
# env: ARCH=i386 DO=distcheck AS_ROOT=no
#- os: linux
# dist: xenial
# compiler: gcc
# env: ARCH=i386 DO=distcheck AS_ROOT=no
script:
- ./admin/travis-build.sh
notifications:
email:
- kyua-log@googlegroups.com
# This is the official list of Kyua authors for copyright purposes.
#
# This file is distinct from the CONTRIBUTORS files; see the latter for
# an explanation.
#
# Names are sorted alphabetically and should be added to this file as:
#
# * Name <email address>
# * Organization <optional email address>
* Google Inc.
Contributing code to Kyua
=========================
Want to contribute? Great! But first, please take a few minutes to read this
document in full. Doing so upfront will minimize the turnaround time required
to get your changes incorporated.
Legal notes
-----------
* Before we can use your code, you must sign the
[Google Individual Contributor License
Agreement](https://developers.google.com/open-source/cla/individual),
also known as the CLA, which you can easily do online. The CLA is necessary
mainly because you own the copyright to your changes, even after your
contribution becomes part of our codebase, so we need your permission to use
and distribute your code. We also need to be sure of various other
things--for instance that you will tell us if you know that your code
infringes on other people's patents. You do not have to sign the CLA until
after you have submitted your code for review and a member has approved it,
but you must do it before we can put your code into our codebase.
* Contributions made by corporations are covered by a different agreement than
the one above: the
[Google Software Grant and Corporate Contributor License
Agreement](https://developers.google.com/open-source/cla/corporate).
Please get your company to sign this agreement instead if your contribution is
on their behalf.
* Unless you have a strong reason not to, please assign copyright of your
changes to Google Inc. and use the 3-clause BSD license text included
throughout the codebase (see [LICENSE](LICENSE)). Keeping the whole project
owned by a single entity is important, particularly to avoid the problem of
having to replicate potentially hundreds of different copyright notes in
documentation materials, etc.
Communication
-------------
* Before you start working on a larger contribution, you should get in touch
with us first through the
[kyua-discuss mailing
list](https://groups.google.com/forum/#!forum/kyua-discuss)
with your idea so that we can help out and possibly guide you. Coordinating
upfront makes it much easier to avoid frustration later on.
* Subscribe to the
[kyua-log mailing list](https://groups.google.com/forum/#!forum/kyua-log) to
get notifications on new commits, Travis CI results, or changes to bugs.
Git workflow
------------
* Always work on a non-master branch.
* Make sure the history of your branch is clean. (Ab)use `git rebase -i master`
to ensure the sequence of commits you want pulled is easy to follow and that
every commit does one (and only one) thing. In particular, commits of the
form `Fix previous` or `Fix build` should never ever exist; merge those fixes
into the relevant commits so that the history is clean at pull time.
* Always trigger Travis CI builds for your changes (hence why working on a
branch is important). Push your branch to GitHub so that Travis CI picks it
up and performs a build. If you have forked the repository, you may need to
enable Travis CI builds on your end. Wait for a green result.
* It is OK and expected for you to `git push --force` on **non-master**
branches. This is required if you need to go through the commit/test cycle
more than once for any given branch after you have "fixed-up" commits to
correct problems spotted in earlier builds.
* Do not send pull requests that subsume other/older pull requests. Each major
change being submitted belongs in a different pull request, which is trivial
to achieve if you use one branch per change as requested in this workflow.
Code reviews
------------
* All changes will be subject to code reviews pre-merge time. In other words:
all pull requests will be carefully inspected before being accepted and they
will be returned to you with comments if there are issues to be fixed.
* Be careful of stylistic errors in your code (see below for style guidelines).
Style violations hinder the review process and distract from the actual code.
By keeping your code clean of style issues upfront, you will speed up the
review process and avoid frustration along the way.
* Whenever you are ready to submit a pull request, review the *combined diff*
you are requesting to be pulled and look for issues. This is the diff that
will be subject to review, not necessarily the individual commits. You can
view this diff in GitHub at the bottom of the `Open a pull request` form that
appears when you click the button to file a pull request, or you can see the
diff by typing `git diff <your-branch> master`.
Commit messages
---------------
* Follow standard Git commit message guidelines. The first line has a maximum
length of 50 characters, does not terminate in a period, and has to summarize
the whole commit. Then a blank line comes, and then multiple plain-text
paragraphs provide details on the commit if necessary with a maximum length of
72-75 characters per line. Vim has syntax highlighting for Git commit
messages and will let you know when you go above the maximum line lengths.
* Use the imperative tense. Say `Add foo-bar` or `Fix baz` instead of `Adding
blah`, `Adds bleh`, or `Added bloh`.
Handling bug tracker issues
---------------------------
* All changes pushed to `master` should cross-reference one or more issues in
the bug tracker. This is particularly important for bug fixes, but also
applies to major feature improvements.
* Unless you have a good reason to do otherwise, name your branch `issue-N`
where `N` is the number of the issue being fixed.
* If the fix to the issue can be done *in a single commit*, terminate the commit
message with `Fixes #N.` where `N` is the number of the issue being fixed and
include a note in `NEWS` about the issue in the same commit. Such fixes can
be merged onto master using fast-forward (the default behavior of `git
merge`).
* If the fix to the issue requires *more than one commit*, do **not** include
`Fixes #N.` in any of the individual commit messages of the branch nor include
any changes to the `NEWS` file in those commits. These "announcement" changes
belong in the merge commit onto `master`, which is done by `git merge --no-ff
--no-commit your-branch`, followed by an edit of `NEWS`, and terminated with a
`git commit -a` with the proper note on the bug being fixed.
Style guide
-----------
These notes are generic and certainly *non-exhaustive*:
* Respect formatting of existing files. Note where braces are placed, number of
blank lines between code chunks, how continuation lines are indented, how
docstrings are typed, etc.
* Indentation is *always* done using spaces, not tabs. The only exception is in
`Makefile`s, where any continuation line within a target must be prefixed by a
*single tab*.
* [Be mindful of spelling and
grammar.](http://julipedia.meroh.net/2013/06/readability-mind-your-typos-and-grammar.html)
Mistakes of this kind are enough of a reason to return a pull request.
* Use proper punctuation for all sentences. Always start with a capital letter
and terminate with a period.
* Respect lexicographical sorting wherever possible.
* Lines must not be over 80 characters.
* No trailing whitespace.
* Two spaces after end-of-sentence periods.
* Two blank lines between functions. If there are two blank lines among code
blocks, they usually exist for a reason: keep them.
* In C++ code, prefix all C identifiers (those coming from `extern "C"`
includes) with `::`.
* Getter functions/methods only need to be documented via `\return`. A
redundant summary is not necessary.
# This is the list of people who have agreed to one of the CLAs and can
# contribute patches to the Kyua project.
#
# The AUTHORS file lists the copyright holders; this file lists people.
# For example: Google employees are listed here but not in AUTHORS
# because Google holds the copyright.
#
# See the following links for details on the CLA:
#
# https://developers.google.com/open-source/cla/individual
# https://developers.google.com/open-source/cla/corporate
#
# Names are sorted by last name and should be added as:
#
# * Name <email address>
* Sergey Bronnikov <sergeyb@openvz.org>
* Enji Cooper <yaneurabeya@gmail.com>
* Julio Merino <jmmv@google.com>
* Craig Rodrigues <rodrigc@crodrigues.org>
# Copyright 2010 The Kyua Authors.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of Google Inc. nor the names of its contributors
# may be used to endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
BUILTIN_STL_SUPPORT = YES
ENABLE_PREPROCESSING = YES
EXCLUDE_SYMBOLS = "ATF_TC*"
EXTRACT_ANON_NSPACES = YES
EXTRACT_LOCAL_CLASSES = YES
EXTRACT_PRIVATE = YES
EXTRACT_STATIC = YES
EXPAND_ONLY_PREDEF = YES
EXTENSION_MAPPING = ipp = C++
FILE_PATTERNS = *.c *.h *.cpp *.hpp *.ipp
GENERATE_LATEX = NO
GENERATE_TAGFILE = @top_builddir@/api-docs/api-docs.tag
HIDE_FRIEND_COMPOUNDS = YES
INPUT = @top_srcdir@
INPUT_ENCODING = ISO-8859-1
JAVADOC_AUTOBRIEF = YES
MACRO_EXPANSION = YES
OUTPUT_DIRECTORY = @top_builddir@/api-docs
OUTPUT_LANGUAGE = English
PREDEFINED += "KYUA_DEFS_NORETURN="
PREDEFINED += "KYUA_DEFS_FORMAT_PRINTF(x, y)="
PROJECT_NAME = "@PACKAGE_NAME@"
PROJECT_NUMBER = @VERSION@
QUIET = YES
RECURSIVE = YES
SORT_BY_SCOPE_NAME = YES
SORT_MEMBERS_CTORS_1ST = YES
WARN_IF_DOC_ERROR = YES
WARN_IF_UNDOCUMENTED = YES
WARN_NO_PARAMDOC = YES
WARNINGS = YES
Installation instructions
=========================
Kyua uses the GNU Automake, GNU Autoconf and GNU Libtool utilities as
its build system. These are used only when compiling the application
from the source code package. If you want to install Kyua from a binary
package, you do not need to read this document.
For the impatient:
$ ./configure
$ make
$ make check
Gain root privileges
# make install
Drop root privileges
$ make installcheck
Or alternatively, install as a regular user into your home directory:
$ ./configure --prefix ~/local
$ make
$ make check
$ make install
$ make installcheck
Dependencies
------------
To build and use Kyua successfully you need:
* A standards-compliant C and C++ complier.
* Lutok 0.4.
* pkg-config.
* SQLite 3.6.22.
To build the Kyua tests, you optionally need:
* The Automated Testing Framework (ATF), version 0.15 or greater. This
is required if you want to create a distribution file.
If you are building Kyua from the code on the repository, you will also
need the following tools:
* GNU Autoconf.
* GNU Automake.
* GNU Libtool.
Regenerating the build system
-----------------------------
This is not necessary if you are building from a formal release
distribution file.
On the other hand, if you are building Kyua from code extracted from the
repository, you must first regenerate the files used by the build
system. You will also need to do this if you modify `configure.ac`,
`Makefile.am` or any of the other build system files. To do this, simply
run:
$ autoreconf -i -s
If ATF is installed in a different prefix than Autoconf, you will also
need to tell autoreconf where the ATF M4 macros are located. Otherwise,
the configure script will be incomplete and will show confusing syntax
errors mentioning, for example, `ATF_CHECK_SH`. To fix this, you have
to run autoreconf in the following manner, replacing `<atf-prefix>` with
the appropriate path:
$ autoreconf -i -s -I <atf-prefix>/share/aclocal
General build procedure
-----------------------
To build and install the source package, you must follow these steps:
1. Configure the sources to adapt to your operating system. This is
done using the `configure` script located on the sources' top
directory, and it is usually invoked without arguments unless you
want to change the installation prefix. More details on this
procedure are given on a later section.
2. Build the sources to generate the binaries and scripts. Simply run
`make` on the sources' top directory after configuring them. No
problems should arise.
3. Check that the built programs work by running `make check`. You do
not need to be root to do this, but if you are not, some checks will
be skipped.
4. Install the program by running `make install`. You may need to
become root to issue this step.
5. Issue any manual installation steps that may be required. These are
described later in their own section.
6. Check that the installed programs work by running `make
installcheck`. You do not need to be root to do this, but if you are
not, some checks will be skipped.
Configuration flags
-------------------
The most common, standard flags given to `configure` are:
* `--prefix=directory`:
**Possible values:** Any path.
**Default:** `/usr/local`.
Specifies where the program (binaries and all associated files) will
be installed.
* `--sysconfdir=directory`:
**Possible values:** Any path.
**Default:** `/usr/local/etc`.
Specifies where the installed programs will look for configuration
files. `/kyua` will be appended to the given path unless
`KYUA_CONFSUBDIR` is redefined as explained later on.
* `--help`:
Shows information about all available flags and exits immediately,
without running any configuration tasks.
The following environment variables are specific to Kyua's `configure`
script:
* `GDB`:
**Possible values:** empty, absolute path to GNU GDB.
**Default:** empty.
Specifies the path to the GNU GDB binary that Kyua will use to gather a
stack trace of a crashing test program. If empty, the configure script
will try to find a suitable binary for you and, if not found, Kyua will
attempt to do the search at run time.
* `KYUA_ARCHITECTURE`:
**Possible values:** name of a CPU architecture (e.g. `x86_64`, `powerpc`).
**Default:** autodetected; typically the output of `uname -p`.
Specifies the name of the CPU architecture on which Kyua will run.
This value is used at run-time to determine tests that are not
applicable to the host system.
* `KYUA_CONFSUBDIR`:
**Possible values:** empty, a relative path.
**Default:** `kyua`.
Specifies the subdirectory of the configuration directory (given by
the `--sysconfdir` argument) under which Kyua will search for its
configuration files.
* `KYUA_CONFIG_FILE_FOR_CHECK`:
**Possible values:** none, an absolute path to an existing file.
**Default:** none.
Specifies the `kyua.conf` configuration file to use when running any
of the `check`, `installcheck` or `distcheck` targets on this source
tree. This setting is exclusively used to customize the test runs of
Kyua itself and has no effect whatsoever on the built product.
* `KYUA_MACHINE`:
**Possible values:** name of a machine type (e.g. `amd64`, `macppc`).
**Default:** autodetected; typically the output of `uname -m`.
Specifies the name of the machine type on which Kyua will run. This
value is used at run-time to determine tests that are not applicable
to the host system.
* `KYUA_TMPDIR`:
**Possible values:** an absolute path to a temporary directory.
**Default:** `/tmp`.
Specifies the path that Kyua will use to create temporary directories
in by default.
The following flags are specific to Kyua's `configure` script:
* `--enable-developer`:
**Possible values:** `yes`, `no`.
**Default:** `yes` in Git `HEAD` builds; `no` in formal releases.
Enables several features useful for development, such as the inclusion
of debugging symbols in all objects or the enforcement of compilation
warnings.
The compiler will be executed with an exhaustive collection of warning
detection features regardless of the value of this flag. However, such
warnings are only fatal when `--enable-developer` is `yes`.
* `--with-atf`:
**Possible values:** `yes`, `no`, `auto`.
**Default:** `auto`.
Enables usage of ATF to build (and later install) the tests.
Setting this to `yes` causes the configure script to look for ATF
unconditionally and abort if not found. Setting this to `auto` lets
configure perform the best decision based on availability of ATF.
Setting this to `no` explicitly disables ATF usage.
When support for tests is enabled, the build process will generate the
test programs and will later install them into the tests tree.
Running `make check` or `make installcheck` from within the source
directory will cause these tests to be run with Kyua.
* `--with-doxygen`:
**Possible values:** `yes`, `no`, `auto` or a path.
**Default:** `auto`.
Enables usage of Doxygen to generate documentation for internal APIs.
This documentation is *not* installed and is only provided to help the
developer of this package. Therefore, enabling or disabling Doxygen
causes absolutely no differences on the files installed by this
package.
Setting this to `yes` causes the configure script to look for Doxygen
unconditionally and abort if not found. Setting this to `auto` lets
configure perform the best decision based on availability of Doxygen.
Setting this to `no` explicitly disables Doxygen usage. And, lastly,
setting this to a path forces configure to use a specific Doxygen
binary, which must exist.
Post-installation steps
-----------------------
Copy the `Kyuafile.top` file installed in the examples directory to the
root of your tests hierarchy and name it `Kyuafile`. For example:
# cp /usr/local/share/kyua/examples/Kyuafile.top \
/usr/local/tests/Kyuafile
This will allow you to simply go into `/usr/tests` and run the tests
from there.
Run the tests!
--------------
Lastly, after a successful installation, you should periodically run the
tests from the final location to ensure things remain stable. Do so as
follows:
$ cd /usr/local/kyua && kyua test
The following configuration variables are specific to the 'kyua' test
suite and can be given to Kyua with arguments of the form
`-v test_suites.kyua.<variable_name>=<value>`:
* `run_coredump_tests`:
**Possible values:** `true` or `false`.
**Default:** `true`.
Avoids running tests that crash subprocesses on purpose to make them
dump core. Such tests are particularly slow on macOS, and it is
sometimes handy to disable them for quicker development iteration.
If you see any tests fail, do not hesitate to report them in:
https://github.com/jmmv/kyua/issues/
Thank you!
syntax(2)
test_suite("kyua")
include("bootstrap/Kyuafile")
include("cli/Kyuafile")
if fs.exists("doc/Kyuafile") then
-- The tests for the docs are not installed because they only cover the
-- build-time process of the manual pages.
include("doc/Kyuafile")
end
include("drivers/Kyuafile")
include("engine/Kyuafile")
include("examples/Kyuafile")
include("integration/Kyuafile")
include("model/Kyuafile")
include("store/Kyuafile")
include("utils/Kyuafile")
Copyright 2010-2015 The Kyua Authors.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of Google Inc. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Makefile.am 0 → 100644