teuthology.suite package

Submodules

teuthology.suite.build_matrix module

teuthology.suite.build_matrix.build_matrix(path, subset=None)

Return a list of items descibed by path such that if the list of items is chunked into mincyclicity pieces, each piece is still a good subset of the suite.

A good subset of a product ensures that each facet member appears at least once. A good subset of a sum ensures that the subset of each sub collection reflected in the subset is a good subset.

A mincyclicity of 0 does not attempt to enforce the good subset property.

The input is just a path. The output is an array of (description, [file list]) tuples.

For a normal file we generate a new item for the result list.

For a directory, we (recursively) generate a new item for each file/dir.

For a directory with a magic ‘+’ file, we generate a single item that concatenates all files/subdirs (A Sum).

For a directory with a magic ‘%’ file, we generate a result set for each item in the directory, and then do a product to generate a result list with all combinations (A Product).

The final description (after recursion) for each item will look like a relative path. If there was a % product, that path component will appear as a file with braces listing the selection of chosen subitems.

Parameters:
  • path – The path to search for yaml fragments
  • subset – (index, outof)
teuthology.suite.build_matrix.combine_path(left, right)

os.path.join(a, b) doesn’t like it when b is None

teuthology.suite.build_matrix.generate_combinations(path, mat, generate_from, generate_to)

Return a list of items describe by path

The input is just a path. The output is an array of (description, [file list]) tuples.

For a normal file we generate a new item for the result list.

For a directory, we (recursively) generate a new item for each file/dir.

For a directory with a magic ‘+’ file, we generate a single item that concatenates all files/subdirs.

For a directory with a magic ‘%’ file, we generate a result set for each item in the directory, and then do a product to generate a result list with all combinations.

The final description (after recursion) for each item will look like a relative path. If there was a % product, that path component will appear as a file with braces listing the selection of chosen subitems.

teuthology.suite.matrix module

class teuthology.suite.matrix.Base(item)

Bases: teuthology.suite.matrix.Matrix

Just a single item.

index(i)
minscanlen()
size()
tostr(depth)
class teuthology.suite.matrix.Concat(item, submats)

Bases: teuthology.suite.matrix.Matrix

Concatenates all items in child matrices

index(i)
minscanlen()
size()
tostr(depth)
class teuthology.suite.matrix.Cycle(num, mat)

Bases: teuthology.suite.matrix.Matrix

Run a matrix multiple times

index(i)
minscanlen()
size()
tostr(depth)
class teuthology.suite.matrix.Matrix

Interface for sets

cyclicity()

A cyclicity of N means that the set represented by the Matrix can be chopped into N good subsets of sequential indices.

index(i)

index() should return a recursive structure represending the paths to concatenate for index i:

Result :: (PathSegment, Result) | {Result} Path :: string

{Result} is a frozen_set of Results indicating that the set of paths resulting from each of the contained Results should be concatenated. (PathSegment, Result) indicates that PathSegment should be prepended to the paths resulting from Result.

minscanlen()

min run require to get a good sample

size()
tostr(depth)
class teuthology.suite.matrix.Product(item, _submats)

Bases: teuthology.suite.matrix.Matrix

Builds items by taking one item from each submatrix. Contiguous subsequences should move through all dimensions.

index(i)
minscanlen()
size()
tostr(depth)
class teuthology.suite.matrix.Sum(item, _submats)

Bases: teuthology.suite.matrix.Matrix

We want to mix the subsequences proportionately to their size.

The intuition is that we map all of the subsequences uniformly onto rational numbers in [0, 1). The ith subsequence with length l will have index k map onto i*<epsilon> + k*(1/l). i*<epsilon> ensures that no two subsequences have an index which shares a mapping in [0, 1) as long as <epsilon> is chosen to be small enough.

Rather than actually dealing with rational numbers, however, we’ll instead map onto whole numbers in [0, pseudo_size) where pseudo_size is the lcm of the subsequence lengths * the number of subsequences. Including the number of subsequences in the product allows us to use 1 as <epsilon>. For each subsequence, we designate an offset (position in input list) and a multiple (pseudo_size / size) such that the psuedo_index for index i is <offset> + i*<multiple>.

I don’t have a good way to map index to pseudo index, so we’ll precompute a mapping in the constructor (self._i_so_sis) from index to (subset_index, subset).

index(i)
minscanlen()
pi_to_sis(pi, (offset, multiple))

max(i) s.t. offset + i*multiple <= pi

pseudo_index_to_index(pi)

Count all pseudoindex values <= pi with corresponding subset indices

size()
tostr(depth)
teuthology.suite.matrix.generate_desc(joinf, result)

Generates the text description of the test represented by result

teuthology.suite.matrix.generate_lists(result)

Generates a set of tuples representing paths to concatenate

teuthology.suite.matrix.generate_paths(path, result, joinf=<function join>)

Generates from the result set a list of sorted paths to concatenate

teuthology.suite.matrix.lcm(a, b)
teuthology.suite.matrix.lcml(l)

teuthology.suite.placeholder module

class teuthology.suite.placeholder.Placeholder(name)

Bases: object

A placeholder for use with substitute_placeholders. Simply has a ‘name’ attribute.

teuthology.suite.placeholder.substitute_placeholders(input_dict, values_dict)

Replace any Placeholder instances with values named in values_dict. In the case of None values, the key is omitted from the result.

Searches through nested dicts.

Parameters:
  • input_dict – A dict which may contain one or more Placeholder instances as values.
  • values_dict – A dict, with keys matching the ‘name’ attributes of all of the Placeholder instances in the input_dict, and values to be substituted.
Returns:

The modified input_dict

teuthology.suite.run module

class teuthology.suite.run.Run(args)

Bases: object

WAIT_MAX_JOB_TIME = 1800
WAIT_PAUSE = 300
args
base_args
base_config
base_yaml_paths
build_base_args()
build_base_config()
ceph_repo_name
choose_ceph_hash()

Get the ceph hash: if –sha1/-S is supplied, use it if it is valid, and just keep the ceph_branch around. Otherwise use the current git branch tip.

choose_ceph_version(ceph_hash)
choose_kernel()
choose_suite_branch()
choose_suite_hash(suite_branch)
choose_teuthology_branch()
collect_jobs(arch, configs, newest=False)
config_input
create_initial_config()

Put together the config file used as the basis for each job in the run. Grabs hashes for the latest ceph, kernel and teuthology versions in the branches specified and specifies them so we know exactly what we’re testing.

Returns:A JobConfig object
kernel_dict
make_run_name()
Generate a run name. A run name looks like:
teuthology-2014-06-23_19:00:37-rados-dumpling-testing-basic-plana
name
package_versions
prepare_and_schedule()

Puts together some “base arguments” with which to execute teuthology-schedule for each job, then passes them and other parameters to schedule_suite(). Finally, schedules a “last-in-suite” job that sends an email to the specified address (if one is configured).

schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
schedule_suite()

Schedule the suite-run. Returns the number of jobs scheduled.

suite_repo_name
suite_repo_path

teuthology.suite.util module

teuthology.suite.util.fetch_repos(branch, test_name)

Fetch the suite repo (and also the teuthology repo) so that we can use it to build jobs. Repos are stored in ~/src/.

The reason the teuthology repo is also fetched is that currently we use subprocess to call teuthology-schedule to schedule jobs so we need to make sure it is up-to-date. For that reason we always fetch the master branch for test scheduling, regardless of what teuthology branch is requested for testing.

Returns:The path to the suite repo on disk
teuthology.suite.util.find_git_parent(project, sha1)
teuthology.suite.util.get_arch(machine_type)

Based on a given machine_type, return its architecture by querying the lock server.

Returns:A string or None
teuthology.suite.util.get_branch_info(project, branch, project_owner='ceph')

NOTE: This is currently not being used because of GitHub’s API rate limiting. We use github_branch_exists() instead.

Use the GitHub API to query a project’s branch. Returns:
{u’object’: {u’sha’: <a_sha_string>,
u’type’: <string>, u’url’: <url_to_commit>},

u’ref’: u’refs/heads/<branch>’, u’url’: <url_to_branch>}

We mainly use this to check if a branch exists.

teuthology.suite.util.get_distro_defaults(distro, machine_type)
Given a distro (e.g. ‘ubuntu’) and machine type, return:
(arch, release, pkg_type)
This is used to default to:
(‘x86_64’, ‘trusty’, ‘deb’) when passed ‘ubuntu’ and ‘plana’ (‘armv7l’, ‘saucy’, ‘deb’) when passed ‘ubuntu’ and ‘saya’ (‘x86_64’, ‘wheezy’, ‘deb’) when passed ‘debian’ (‘x86_64’, ‘fedora20’, ‘rpm’) when passed ‘fedora’

And (‘x86_64’, ‘centos7’, ‘rpm’) when passed anything else

teuthology.suite.util.get_gitbuilder_hash(project=None, branch=None, flavor=None, machine_type=None, distro=None, distro_version=None)

Find the hash representing the head of the project’s repository via querying a gitbuilder repo.

Will return None in the case of a 404 or any other HTTP error.

teuthology.suite.util.get_install_task_flavor(job_config)

Pokes through the install task’s configuration (including its overrides) to figure out which flavor it will want to install.

Only looks at the first instance of the install task in job_config.

teuthology.suite.util.get_package_versions(sha1, os_type, os_version, flavor, package_versions=None)

Will retrieve the package versions for the given sha1, os_type/version, and flavor from gitbuilder.

Optionally, a package_versions dict can be provided from previous calls to this function to avoid calling gitbuilder for information we’ve already retrieved.

The package_versions dict will be in the following format:

{
    "sha1": {
        "ubuntu": {
            "14.04": {
                "basic": "version",
            }
            "15.04": {
                "notcmalloc": "version",
            }
        }
        "rhel": {
            "basic": "version",
            }
    },
    "another-sha1": {
        "ubuntu": {
            "basic": "version",
            }
    }
}
Parameters:
  • sha1 – The sha1 hash of the ceph version.
  • os_type – The distro we want to get packages for, given the ceph sha1. Ex. ‘ubuntu’, ‘rhel’, etc.
  • os_version – The distro’s version, e.g. ‘14.04’, ‘7.0’
  • flavor – Package flavor (‘testing’, ‘notcmalloc’, etc.)
  • package_versions – Use this optionally to use cached results of previous calls to gitbuilder.
Returns:

A dict of package versions. Will return versions for all hashes/distros/vers, not just for the given hash/distro/ver.

teuthology.suite.util.get_worker(machine_type)

Map a given machine_type to a beanstalkd worker. If machine_type mentions multiple machine types - e.g. ‘plana,mira’, then this returns ‘multi’. Otherwise it returns what was passed.

teuthology.suite.util.git_branch_exists(project_or_url, branch, project_owner='ceph')

Query the git repository to check the existence of a project’s branch

Parameters:
  • project_or_url – Either a project name or a full URL
  • branch – The branch to query
  • project_owner – The GitHub project owner. Only used when a project name is passed; not when a URL is passed
teuthology.suite.util.git_ls_remote(project_or_url, branch, project_owner='ceph')

Find the latest sha1 for a given project’s branch.

Parameters:
  • project_or_url – Either a project name or a full URL
  • branch – The branch to query
  • project_owner – The GitHub project owner. Only used when a project name is passed; not when a URL is passed
Returns:

The sha1 if found; else None

teuthology.suite.util.git_validate_sha1(project, sha1, project_owner='ceph')

Use http to validate that project contains sha1 I can’t find a way to do this with git, period, so we have specific urls to HEAD for github and git.ceph.com/gitweb for now

teuthology.suite.util.has_packages_for_distro(sha1, os_type, os_version, flavor, package_versions=None)

Checks to see if gitbuilder has packages for the given sha1, os_type and kernel_flavor.

See above for package_versions description.

Parameters:
  • sha1 – The sha1 hash of the ceph version.
  • os_type – The distro we want to get packages for, given the ceph sha1. Ex. ‘ubuntu’, ‘rhel’, etc.
  • kernel_flavor – The kernel flavor
  • package_versions – Use this optionally to use cached results of previous calls to gitbuilder.
Returns:

True, if packages are found. False otherwise.

teuthology.suite.util.package_version_for_hash(hash, kernel_flavor='basic', distro='rhel', distro_version='7.0', machine_type='smithi')

Does what it says on the tin. Uses gitbuilder repos.

Returns:a string.
teuthology.suite.util.schedule_fail(message, name='')

If an email address has been specified anywhere, send an alert there. Then raise a ScheduleFailError.

teuthology.suite.util.strip_fragment_path(original_path)

Given a path, remove the text before ‘/suites/’. Part of the fix for http://tracker.ceph.com/issues/15470

teuthology.suite.util.teuthology_schedule(args, verbose, dry_run, log_prefix='')

Run teuthology-schedule to schedule individual jobs.

If –dry-run has been passed but –verbose has been passed just once, don’t actually run the command - only print what would be executed.

If –dry-run has been passed and –verbose has been passed multiple times, do both.

Module contents

exception teuthology.suite.WaitException

Bases: exceptions.Exception

teuthology.suite.get_rerun_filters(name, statuses)
teuthology.suite.main(args)
teuthology.suite.normalize_suite_name(name)
teuthology.suite.process_args(args)
teuthology.suite.wait(name, max_job_time, upload_url)