teuthology package

Subpackages

Submodules

teuthology.beanstalk module

class teuthology.beanstalk.JobDeleter(pattern)

Bases: teuthology.beanstalk.JobProcessor

add_job(job_id, job_config, job_obj=None)
process_job(job_id)
class teuthology.beanstalk.JobPrinter(show_desc=False, full=False)

Bases: teuthology.beanstalk.JobProcessor

process_job(job_id)
class teuthology.beanstalk.JobProcessor

Bases: object

add_job(job_id, job_config, job_obj=None)
complete()
process_job(job_id)
class teuthology.beanstalk.RunPrinter

Bases: teuthology.beanstalk.JobProcessor

process_job(job_id)
teuthology.beanstalk.connect()
teuthology.beanstalk.end_progress()
teuthology.beanstalk.main(args)
teuthology.beanstalk.pause_tube(connection, tube, duration)
teuthology.beanstalk.print_progress(index, total, message=None)
teuthology.beanstalk.stats_tube(connection, tube)
teuthology.beanstalk.walk_jobs(connection, tube_name, processor, pattern=None)

def callback(jobs_dict)

teuthology.beanstalk.watch_tube(connection, tube_name)

Watch a given tube, potentially correcting to ‘multi’ if necessary. Returns the tube_name that was actually used.

teuthology.config module

class teuthology.config.FakeNamespace(config_dict=None)

Bases: teuthology.config.YamlConfig

This class is meant to behave like a argparse Namespace

We’ll use this as a stop-gap as we refactor commands but allow the tasks to still be passed a single namespace object for the time being.

class teuthology.config.JobConfig(yaml_path=None)

Bases: teuthology.config.YamlConfig

class teuthology.config.TeuthologyConfig(yaml_path=None)

Bases: teuthology.config.YamlConfig

This class is intended to unify teuthology’s many configuration files and objects. Currently it serves as a convenient interface to ~/.teuthology.yaml and nothing else.

get_ceph_cm_ansible_git_url()
get_ceph_git_url()
get_ceph_qa_suite_git_url()
yaml_path = '/home/jenkins-build/.teuthology.yaml'
class teuthology.config.YamlConfig(yaml_path=None)

Bases: _abcoll.MutableMapping

A configuration object populated by parsing a yaml file, with optional default values.

Note that modifying the _defaults attribute of an instance can potentially yield confusing results; if you need to do modify defaults, use the class variable or create a subclass.

classmethod from_dict(in_dict)

Build a config object from a dict.

Parameters:in_dict – The dict to use
Returns:The config object
classmethod from_str(in_str)

Build a config object from a string or yaml stream.

Parameters:in_str – The stream or string
Returns:The config object
get(k[, d]) → D[k] if k in D, else d. d defaults to None.
load(conf=None)
to_dict()
Returns:A shallow copy of the configuration as a dict
to_str()
Returns:str(self)
update(in_dict)

Update an existing configuration using dict.update()

Parameters:in_dict – The dict to use to update
teuthology.config.init_logging()
teuthology.config.set_config_attr(obj)

Set obj.teuthology_config, mimicking the old behavior of misc.read_config

teuthology.contextutil module

teuthology.contextutil.nested(*args, **kwds)

Like contextlib.nested but takes callables returning context managers, to avoid the major reason why contextlib.nested was deprecated.

This version also logs any exceptions early, much like run_tasks, to ease debugging. TODO combine nested and run_tasks.

class teuthology.contextutil.safe_while(sleep=6, increment=0, tries=10, action=None, _raise=True, _sleeper=None)

Bases: object

A context manager to remove boiler plate code that deals with while loops that need a given number of tries and some seconds to sleep between each one of those tries.

The most simple example possible will try 10 times sleeping for 6 seconds:

>>> from teuthology.contexutil import safe_while
>>> with safe_while() as proceed:
...    while proceed():
...        # repetitive code here
...        print("hello world")
...
Traceback (most recent call last):
...
MaxWhileTries: reached maximum tries (5) after waiting for 75 seconds

Yes, this adds yet another level of indentation but it allows you to implement while loops exactly the same as before with just 1 more indentation level and one extra call. Everything else stays the same, code-wise. So adding this helper to existing code is simpler.

Parameters:
  • sleep – The amount of time to sleep between tries. Default 6
  • increment – The amount to add to the sleep value on each try. Default 0.
  • tries – The amount of tries before giving up. Default 10.
  • action – The name of the action being attempted. Default none.
  • _raise – Whether to raise an exception (or log a warning). Default True.
  • _sleeper – The function to use to sleep. Only used for testing. Default time.sleep

teuthology.coverage module

teuthology.coverage.analyze(test_dir, cov_tools_dir, lcov_output, html_output, skip_init)
teuthology.coverage.connect_to_db()
teuthology.coverage.log = <logging.Logger object>

The coverage database can be created in mysql with:

CREATE TABLE coverage (
run_id bigint(20) NOT NULL AUTO_INCREMENT, rev char(40) NOT NULL, test varchar(255) NOT NULL, suite varchar(255) NOT NULL, lines int(10) unsigned NOT NULL, line_cov float unsigned NOT NULL, functions int(10) unsigned NOT NULL, function_cov float unsigned NOT NULL, branches int(10) unsigned NOT NULL, branch_cov float unsigned NOT NULL, run_date timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (run_id)

) ENGINE=InnoDB DEFAULT CHARSET=utf8

teuthology.coverage.main(args)
teuthology.coverage.read_coverage(output)
teuthology.coverage.store_coverage(test_coverage, rev, suite)

teuthology.describe_tests module

teuthology.describe_tests.describe_suite(suite_dir, fields, include_facet, output_format)

Describe a suite listing each subdirectory and file once as a separate row.

Returns a tuple of (headers, rows) where both elements are lists of strings.

teuthology.describe_tests.describe_tests(args)
teuthology.describe_tests.extract_info(file_name, fields)

Read a yaml file and return a dictionary mapping the fields to the values of those fields in the file.

The returned dictionary will always contain all the provided fields, mapping any non-existent ones to ‘’.

Assumes fields are set in a format of:

{‘meta’: [{‘field’ : value, ‘field2’ : value2}]

or in yaml:

meta: - field: value

field2: value2

If ‘meta’ is present but not in this format, prints an error message and raises ParseError.

teuthology.describe_tests.get_combinations(suite_dir, fields, subset, limit, filter_in, filter_out, include_facet)

Describes the combinations of a suite, optionally limiting or filtering output based on the given parameters. Includes columns for the subsuite and facets when include_facet is True.

Returns a tuple of (headers, rows) where both elements are lists of strings.

teuthology.describe_tests.main(args)
teuthology.describe_tests.output_results(headers, rows, output_format, hrule)

Write the headers and rows given in the specified output format to stdout.

teuthology.describe_tests.path_relative_to_suites(path)

Attempt to trim the ceph-qa-suite root directory from the beginning of a path.

teuthology.describe_tests.tree_with_info(cur_dir, fields, include_facet, prefix, rows, output_format='plain')

Gather fields from all files and directories in cur_dir. Returns a list of strings for each path containing:

  1. the path relative to ceph-qa-suite/suites (or the basename with
    a /usr/bin/tree-like prefix if output_format is plain)
  2. the facet containing the path (if include_facet is True)
  3. the values of the provided fields in the path (‘’ is used for missing values) in the same order as the provided fields

teuthology.exceptions module

exception teuthology.exceptions.AnsibleFailedError(failures)

Bases: exceptions.Exception

Exception thrown when an ansible playbook fails

exception teuthology.exceptions.BootstrapError

Bases: exceptions.RuntimeError

exception teuthology.exceptions.BranchNotFoundError(branch, repo=None)

Bases: exceptions.ValueError

exception teuthology.exceptions.CommandCrashedError(command)

Bases: exceptions.Exception

Exception thrown on crash

exception teuthology.exceptions.CommandFailedError(command, exitstatus, node=None, label=None)

Bases: exceptions.Exception

Exception thrown on command failure

exception teuthology.exceptions.CommitNotFoundError(commit, repo=None)

Bases: exceptions.ValueError

exception teuthology.exceptions.ConfigError

Bases: exceptions.RuntimeError

Meant to be used when an invalid config entry is found.

exception teuthology.exceptions.ConnectionLostError(command, node=None)

Bases: exceptions.Exception

Exception thrown when the connection is lost

exception teuthology.exceptions.ConsoleError

Bases: exceptions.Exception

exception teuthology.exceptions.GitError

Bases: exceptions.RuntimeError

exception teuthology.exceptions.MaxWhileTries

Bases: exceptions.Exception

exception teuthology.exceptions.NoRemoteError

Bases: exceptions.Exception

message = 'This operation requires a remote'
exception teuthology.exceptions.ParseError

Bases: exceptions.Exception

exception teuthology.exceptions.QuotaExceededError(message)

Bases: exceptions.Exception

exception teuthology.exceptions.SELinuxError(node, denials)

Bases: exceptions.Exception

exception teuthology.exceptions.ScheduleFailError(message, name=None)

Bases: exceptions.RuntimeError

exception teuthology.exceptions.SkipJob

Bases: exceptions.Exception

Used by teuthology.worker when it notices that a job is broken and should be skipped.

exception teuthology.exceptions.UnsupportedPackageTypeError(node)

Bases: exceptions.Exception

exception teuthology.exceptions.VersionNotFoundError(url)

Bases: exceptions.Exception

teuthology.exit module

class teuthology.exit.Exiter

Bases: object

A helper to manage any signal handlers we need to call upon receiving a given signal

add_handler(signals, func)

Adds a handler function to be called when any of the given signals are received.

The handler function should have a signature like:

my_handler(signal, frame)
default_handler(signal_, frame)
class teuthology.exit.Handler(exiter, func, signals)

Bases: object

remove()

teuthology.job_status module

teuthology.job_status.get_status(summary)
Parameters:summary – The job summary dict. Normally ctx.summary
Returns:A status string like ‘pass’, ‘fail’, or ‘dead’
teuthology.job_status.set_status(summary, status)

Sets summary[‘status’] to status, and summary[‘success’] to True if status is ‘pass’. If status is not ‘pass’, then ‘success’ is False.

If status is None, do nothing.

Parameters:
  • summary – The job summary dict. Normally ctx.summary
  • status – The job status, e.g. ‘pass’, ‘fail’, ‘dead’

teuthology.kill module

teuthology.kill.find_pids(run_name)
teuthology.kill.find_run_info(serializer, run_name)
teuthology.kill.find_targets(run_name, owner)
teuthology.kill.kill_job(run_name, job_id, archive_base=None, owner=None)
teuthology.kill.kill_processes(run_name, pids=None)
teuthology.kill.kill_run(run_name, archive_base=None, owner=None, machine_type=None, preserve_queue=False)
teuthology.kill.main(args)
teuthology.kill.nuke_targets(targets_dict, owner)
teuthology.kill.process_matches_run(pid, run_name)
teuthology.kill.remove_beanstalk_jobs(run_name, tube_name)
teuthology.kill.remove_paddles_jobs(run_name)

teuthology.ls module

teuthology.ls.get_jobs(archive_dir)
teuthology.ls.ls(archive_dir, verbose)
teuthology.ls.main(args)
teuthology.ls.print_debug_info(job, job_dir, archive_dir)

teuthology.misc module

Miscellaneous teuthology functions. Used by other modules, but mostly called from tasks.

class teuthology.misc.MergeConfig(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: argparse.Action

Used by scripts to mergeg configurations. (nuke, run, and schedule, for example)

teuthology.misc.all_roles(cluster)

Generator of role values. Each call returns another role.

Parameters:cluster – Cluster extracted from the ctx.
teuthology.misc.all_roles_of_type(cluster, type_)

Generator of role values. Each call returns another role of the type specified.

Parameters:
  • cluster – Cluster extracted from the ctx.
  • type – role type
teuthology.misc.append_lines_to_file(remote, path, lines, sudo=False)

Append lines to a file. An intermediate file is used in the same manner as in Remove_lines_from_list.

teuthology.misc.canonicalize_hostname(hostname, user='ubuntu')
teuthology.misc.ceph_role(role)

Return the ceph name for the role, without any cluster prefix, e.g. osd.0.

teuthology.misc.cluster_roles_of_type(roles_for_host, type_, cluster)

Generator of roles.

Each call returns the next possible role of the type specified. :param roles_for_host: list of roles possible :param type_: type of role :param cluster: cluster name

teuthology.misc.config_file(string)

Create a config file

Parameters:string – name of yaml file used for config.
Returns:Dictionary of configuration information.
teuthology.misc.copy_file(from_remote, from_path, to_remote, to_path=None)

Copies a file from one remote to another.

teuthology.misc.create_file(remote, path, data='', permissions='644', sudo=False)

Create a file on the remote host.

teuthology.misc.create_simple_monmap(ctx, remote, conf, path=None, mon_bind_addrvec=False)

Writes a simple monmap based on current ceph.conf into path, or <testdir>/monmap by default.

Assumes ceph_conf is up to date.

Assumes mon sections are named “mon.*”, with the dot.

:return the FSID (as a string) of the newly created monmap

teuthology.misc.decanonicalize_hostname(hostname)
teuthology.misc.deep_merge(a, b)

Deep Merge. If a and b are both lists, all elements in b are added into a. If a and b are both dictionaries, elements in b are recursively added to a. :param a: object items will be merged into :param b: object items will be merged from

teuthology.misc.delete_file(remote, path, sudo=False, force=False, check=True)

rm a file on a remote site. Use force=True if the call should succeed even if the file is absent or rm path would otherwise fail.

teuthology.misc.feed_many_stdins(fp, processes)
Parameters:
  • fp – input file
  • processes – list of processes to be written to.
teuthology.misc.feed_many_stdins_and_close(fp, processes)

Feed many and then close processes.

Parameters:
  • fp – input file
  • processes – list of processes to be written to.
teuthology.misc.get_archive_dir(ctx)
Returns:archive directory (a subdirectory of the test directory)
teuthology.misc.get_ceph_binary_url(package=None, branch=None, tag=None, sha1=None, dist=None, flavor=None, format=None, arch=None)

return the url of the ceph binary found on gitbuildder.

teuthology.misc.get_clients(ctx, roles)

return all remote roles that are clients.

teuthology.misc.get_distro(ctx)

Get the name of the distro that we are using (usually the os_type).

teuthology.misc.get_distro_version(ctx)

Get the verstion of the distro that we are using (release number).

teuthology.misc.get_file(remote, path, sudo=False, dest_dir='/tmp')

Get the contents of a remote file. Do not use for large files; use Remote.get_file() instead.

teuthology.misc.get_first_mon(ctx, config, cluster='ceph')

return the “first” mon role (alphanumerically, for lack of anything better)

teuthology.misc.get_http_log_path(archive_dir, job_id=None)
Parameters:
  • archive_dir – directory to be searched
  • job_id – id of job that terminates the name of the log path
Returns:

http log path

teuthology.misc.get_mon_names(ctx, cluster='ceph')
Returns:a list of monitor names
teuthology.misc.get_mons(roles, ips, mon_bind_msgr2=False, mon_bind_addrvec=False)

Get monitors and their associated addresses

teuthology.misc.get_multi_machine_types(machinetype)

Converts machine type string to list based on common deliminators

teuthology.misc.get_pkg_type(os_type)
teuthology.misc.get_results_url(run_name, job_id=None)
Parameters:
  • run_name – The name of the test run
  • job_id – The job_id of the job. Optional.
Returns:

URL to the run (or job, if job_id is passed) in the results web UI. For example, Inktank uses Pulpito.

teuthology.misc.get_scratch_devices(remote)

Read the scratch disk list from remote host

teuthology.misc.get_system_type(remote, distro=False, version=False)

If distro, return distro. If version, return version (lsb_release -rs) If both, return both. If neither, return ‘deb’ or ‘rpm’ if distro is known to be one of those Finally, if unknown, return the unfiltered distro (from lsb_release -is)

teuthology.misc.get_test_user(ctx=None)
Parameters:ctx – Unused; accepted for compatibility
Returns:str – the user to run tests as on remote hosts
teuthology.misc.get_testdir(ctx=None)
Parameters:ctx – Unused; accepted for compatibility
Returns:A test directory
teuthology.misc.get_user()

Return the username in the format user@host.

teuthology.misc.get_valgrind_args(testdir, name, preamble, v)

Build a command line for running valgrind.

testdir - test results directory name - name of daemon (for naming hte log file) preamble - stuff we should run before valgrind v - valgrind arguments

teuthology.misc.get_wwn_id_map(remote, devs)
teuthology.misc.host_shortname(hostname)
teuthology.misc.is_arm(x)
teuthology.misc.is_in_dict(searchkey, searchval, d)

Test if searchkey/searchval are in dictionary. searchval may itself be a dict, in which case, recurse. searchval may be a subset at any nesting level (that is, all subkeys in searchval must be found in d at the same level/nest position, but searchval is not required to fully comprise d[searchkey]).

>>> is_in_dict('a', 'foo', {'a':'foo', 'b':'bar'})
True
>>> is_in_dict(
...     'a',
...     {'sub1':'key1', 'sub2':'key2'},
...     {'a':{'sub1':'key1', 'sub2':'key2', 'sub3':'key3'}}
... )
True
>>> is_in_dict('a', 'foo', {'a':'bar', 'b':'foo'})
False
>>> is_in_dict('a', 'foo', {'a':{'a': 'foo'}})
False
teuthology.misc.is_type(type_, cluster=None)

Returns a matcher function for whether role is of type given.

Parameters:cluster – cluster name to check in matcher (default to no check for cluster)
teuthology.misc.merge_configs(config_paths)

Takes one or many paths to yaml config files and merges them together, returning the result.

teuthology.misc.move_file(remote, from_path, to_path, sudo=False, preserve_perms=True)

Move a file from one path to another on a remote site

If preserve_perms is true, the contents of the destination file (to_path, which must already exist in this case) are replaced with the contents of the source file (from_path) and the permissions of to_path are preserved. If preserve_perms is false, to_path does not need to exist, and is simply clobbered if it does.

teuthology.misc.num_instances_of_type(cluster, type_, ceph_cluster='ceph')

Total the number of instances of the role type specified in all remotes.

Parameters:
  • cluster – Cluster extracted from ctx.
  • type – role
  • ceph_cluster – filter for ceph cluster name
teuthology.misc.prepend_lines_to_file(remote, path, lines, sudo=False)

Prepend lines to a file. An intermediate file is used in the same manner as in Remove_lines_from_list.

teuthology.misc.pull_directory(remote, remotedir, localdir)

Copy a remote directory to a local directory.

teuthology.misc.pull_directory_tarball(remote, remotedir, localfile)

Copy a remote directory to a local tarball.

teuthology.misc.reboot(node, timeout=300, interval=30)

Reboots a given system, then waits for it to come back up and re-establishes the ssh connection.

Parameters:
  • node – The teuthology.orchestra.remote.Remote object of the node
  • timeout – The amount of time, in seconds, after which to give up waiting for the node to return
  • interval – The amount of time, in seconds, to wait between attempts to re-establish with the node. This should not be set to less than maybe 10, to make sure the node actually goes down first.
teuthology.misc.reconnect(ctx, timeout, remotes=None)

Connect to all the machines in ctx.cluster.

Presumably, some of them won’t be up. Handle this by waiting for them, unless the wait time exceeds the specified timeout.

ctx needs to contain the cluster of machines you wish it to try and connect to, as well as a config holding the ssh keys for each of them. As long as it contains this data, you can construct a context that is a subset of your full cluster.

teuthology.misc.remove_lines_from_file(remote, path, line_is_valid_test, string_to_test_for)

Remove lines from a file. This involves reading the file in, removing the appropriate lines, saving the file, and then replacing the original file with the new file. Intermediate files are used to prevent data loss on when the main site goes up and down.

teuthology.misc.replace_all_with_clients(cluster, config)

Converts a dict containing a key all to one mapping all clients to the value of config[‘all’]

teuthology.misc.roles_of_type(roles_for_host, type_)

Generator of ids.

Each call returns the next possible role of the type specified. :param roles_for_host: list of roles possible :param type_: type of role

teuthology.misc.sh(command, log_limit=1024, cwd=None, env=None)

Run the shell command and return the output in ascii (stderr and stdout). If the command fails, raise an exception. The command and its output are logged, on success and on error.

teuthology.misc.skeleton_config(ctx, roles, ips, cluster='ceph', mon_bind_msgr2=False, mon_bind_addrvec=False)

Returns a ConfigObj that is prefilled with a skeleton config.

Use conf[section][key]=value or conf.merge to change it.

Use conf.write to write it out, override .filename first if you want.

teuthology.misc.split_role(role)

Return a tuple of cluster, type, and id If no cluster is included in the role, the default cluster, ‘ceph’, is used

teuthology.misc.ssh_keyscan(hostnames, _raise=True)

Fetch the SSH public key of one or more hosts

Parameters:
  • hostnames – A list of hostnames, or a dict keyed by hostname
  • _raise – Whether to raise an exception if not all keys are retrieved
Returns:

A dict keyed by hostname, with the host keys as values

teuthology.misc.ssh_keyscan_wait(hostname)

Run ssh-keyscan against a host, return True if it succeeds, False otherwise. Try again if ssh-keyscan timesout. :param hostname: on which ssh-keyscan is run

teuthology.misc.stop_daemons_of_type(ctx, type_, cluster='ceph')
Parameters:type – type of daemons to be stopped.
teuthology.misc.sudo_write_file(remote, path, data, perms=None, owner=None)

Write data to a remote file as super user

Parameters:
  • remote – Remote site.
  • path – Path on the remote being written to.
  • data – Data to be written.
  • perms – Permissions on the file being written
  • owner – Owner for the file being written

Both perms and owner are passed directly to chmod.

teuthology.misc.wait_until_healthy(ctx, remote, ceph_cluster='ceph', use_sudo=False)

Wait until a Ceph cluster is healthy. Give up after 15min.

teuthology.misc.wait_until_osds_up(ctx, cluster, remote, ceph_cluster='ceph')

Wait until all Ceph OSDs are booted.

teuthology.misc.write_file(remote, path, data)

Write data to a remote file

Parameters:
  • remote – Remote site.
  • path – Path on the remote being written to.
  • data – Data to be written.

teuthology.packaging module

class teuthology.packaging.GitbuilderProject(project, job_config, ctx=None, remote=None)

Bases: object

Represents a project that is built by gitbuilder.

base_url

The base url that points at this project on gitbuilder.

For example:

http://gitbuilder.ceph.com/ceph-deb-raring-x86_64-basic/ref/master
Returns:A string of the base url for this project
install_repo()

Install the .repo file or sources.list fragment on self.remote if there is one. If not, raises an exception

remove_repo()

Remove the .repo file or sources.list fragment on self.remote if there is one. If not, raises an exception

rpm_release = '1-0'
sha1

Performs a call to gitbuilder to retrieve the sha1 if not provided in the job_config. The returned value is cached so that this call only happens once.

Returns:The sha1 of the project as a string.
uri_reference

The URI reference that identifies what build of the project we’d like to use.

For example, the following could be returned:

ref/<branch>
sha1/<sha1>
ref/<tag>
Returns:The uri_reference as a string.
version

Performs a call to gitubilder to retrieve the version number for the project. The returned value is cached so that this call only happens once.

Returns:The version number of the project as a string.
class teuthology.packaging.ShamanProject(project, job_config, ctx=None, remote=None)

Bases: teuthology.packaging.GitbuilderProject

assert_result()
repo_url
scm_version
teuthology.packaging.get_builder_project()

Depending on whether config.use_shaman is True or False, return GitbuilderProject or ShamanProject (the class, not an instance).

teuthology.packaging.get_koji_build_info(build_id, remote, ctx)

Queries kojihub and retrieves information about the given build_id. The package, koji, must be installed on the remote for this command to work.

We need a remote here because koji can only be installed on rpm based machines and teuthology runs on Ubuntu.

Here is an example of the build info returned:

{‘owner_name’: ‘kdreyer’, ‘package_name’: ‘ceph’,
‘task_id’: 8534149, ‘completion_ts’: 1421278726.1171, ‘creation_event_id’: 10486804, ‘creation_time’: ‘2015-01-14 18:15:17.003134’, ‘epoch’: None, ‘nvr’: ‘ceph-0.80.5-4.el7ost’, ‘name’: ‘ceph’, ‘completion_time’: ‘2015-01-14 18:38:46.1171’, ‘state’: 1, ‘version’: ‘0.80.5’, ‘volume_name’: ‘DEFAULT’, ‘release’: ‘4.el7ost’, ‘creation_ts’: 1421277317.00313, ‘package_id’: 34590, ‘id’: 412677, ‘volume_id’: 0, ‘owner_id’: 2826

}

Parameters:
  • build_id – The koji build_id we want to retrieve info on.
  • remote – The remote to run the koji command on.
  • ctx – The ctx from the current run, used to provide a failure_reason and status if the koji command fails.
Returns:

A python dict containing info about the build.

teuthology.packaging.get_koji_package_name(package, build_info, arch='x86_64')

Builds the package name for a brew rpm.

Parameters:
  • package – The name of the package
  • build_info – A dict of koji build information, possibly retrieved from get_brew_build_info.
  • arch – The arch you want to download rpms for.
Returns:

A string representing the file name for the requested package in koji.

teuthology.packaging.get_koji_task_result(task_id, remote, ctx)

Queries kojihub and retrieves information about the given task_id. The package, koji, must be installed on the remote for this command to work.

We need a remote here because koji can only be installed on rpm based machines and teuthology runs on Ubuntu.

The results of the given task are returned. For example:

{

‘brootid’: 3303567, ‘srpms’: [], ‘rpms’: [

‘tasks/6745/9666745/kernel-4.1.0-0.rc2.git2.1.fc23.x86_64.rpm’, ‘tasks/6745/9666745/kernel-modules-4.1.0-0.rc2.git2.1.fc23.x86_64.rpm’,

],

‘logs’: []

}

Parameters:
  • task_id – The koji task_id we want to retrieve results for.
  • remote – The remote to run the koji command on.
  • ctx – The ctx from the current run, used to provide a failure_reason and status if the koji command fails.
Returns:

A python dict containing info about the task results.

teuthology.packaging.get_koji_task_rpm_info(package, task_rpms)

Extracts information about a given package from the provided rpm results of a koji task.

For example, if trying to retrieve the package ‘kernel’ from the results of a task, the output would look like this:

{
‘base_url’: ‘https://kojipkgs.fedoraproject.org/work/tasks/6745/9666745/’, ‘rpm_name’: ‘kernel-4.1.0-0.rc2.git2.1.fc23.x86_64.rpm’, ‘package_name’: ‘kernel’, ‘version’: ‘4.1.0-0.rc2.git2.1.fc23.x86_64’,

}

Parameters:
  • task_rpms – A list of rpms from a tasks reusults.
  • package – The name of the package to retrieve.
Returns:

A python dict containing info about the package.

teuthology.packaging.get_kojiroot_base_url(build_info, arch='x86_64')

Builds the base download url for kojiroot given the current build information.

Parameters:
  • build_info – A dict of koji build information, possibly retrieved from get_koji_build_info.
  • arch – The arch you want to download rpms for.
Returns:

The base_url to use when downloading rpms from brew.

teuthology.packaging.get_package_name(pkg, rem)

Find the remote-specific name of the generic ‘pkg’

teuthology.packaging.get_package_version(remote, package)
teuthology.packaging.get_service_name(service, rem)

Find the remote-specific name of the generic ‘service’

teuthology.packaging.install_package(package, remote)

Install ‘package’ on ‘remote’ Assumes repo has already been set up (perhaps with install_repo)

teuthology.packaging.log = <logging.Logger object>

Map ‘generic’ package name to ‘flavor-specific’ package name. If entry is None, either the package isn’t known here, or it’s known but should not be installed on remotes of this flavor

teuthology.packaging.remove_package(package, remote)

Remove package from remote

teuthology.parallel module

class teuthology.parallel.ExceptionHolder(exc_info)

Bases: object

teuthology.parallel.capture_traceback(func, *args, **kwargs)

Utility function to capture tracebacks of any exception func raises.

class teuthology.parallel.parallel

Bases: object

This class is a context manager for running functions in parallel.

You add functions to be run with the spawn method:

with parallel() as p:
    for foo in bar:
        p.spawn(quux, foo, baz=True)

You can iterate over the results (which are in arbitrary order):

with parallel() as p:
    for foo in bar:
        p.spawn(quux, foo, baz=True)
    for result in p:
        print(result)

If one of the spawned functions throws an exception, it will be thrown when iterating over the results, or when the with block ends.

At the end of the with block, the main thread waits until all spawned functions have completed, or, if one exited with an exception, kills the rest and raises the exception.

next()
spawn(func, *args, **kwargs)
teuthology.parallel.resurrect_traceback(exc)

teuthology.prune module

teuthology.prune.is_old_enough(file_name, days)
Returns:True if the file’s modification date is earlier than the amount of days specified
teuthology.prune.listdir(path)
teuthology.prune.main(args)

Main function; parses args and calls prune_archive()

teuthology.prune.maybe_compress_logs(run_dir, days, dry_run=False)
teuthology.prune.maybe_remove_jobs(run_dir, pass_days, fail_days, dry_run=False)

Remove entire job log directories if they are old enough and the job passed

teuthology.prune.maybe_remove_remotes(run_dir, days, dry_run=False)

Remove remote logs (not teuthology logs) from job directories if they are old enough

teuthology.prune.prune_archive(archive_dir, pass_days, fail_days, remotes_days, compress_days, dry_run=False)

Walk through the archive_dir, calling the cleanup functions to process directories that might be old enough

teuthology.prune.remove(path)

Attempt to recursively remove a directory. If an OSError is encountered, log it and continue.

teuthology.prune.should_preserve(dir_name)

Should the directory be preserved?

Returns:True if the directory contains a file named ‘.preserve’; False otherwise

teuthology.repo_utils module

teuthology.repo_utils.bootstrap_teuthology(dest_path)
teuthology.repo_utils.build_git_url(project, project_owner='ceph')

Return the git URL to clone the project

teuthology.repo_utils.clone_repo(repo_url, dest_path, branch, shallow=True)

Clone a repo into a path

Parameters:
  • repo_url – The full URL to the repo (not including the branch)
  • dest_path – The full path to the destination directory
  • branch – The branch.
  • shallow – Whether to perform a shallow clone (–depth 1)
Raises:

BranchNotFoundError if the branch is not found; GitError for other errors

teuthology.repo_utils.clone_repo_ref(repo_url, dest_path, ref)
teuthology.repo_utils.enforce_repo_state(repo_url, dest_path, branch, remove_on_error=True)

Use git to either clone or update a given repo, forcing it to switch to the specified branch.

Parameters:
  • repo_url – The full URL to the repo (not including the branch)
  • dest_path – The full path to the destination directory
  • branch – The branch.
  • remove – Whether or not to remove dest_dir when an error occurs
Raises:

BranchNotFoundError if the branch is not found; GitError for other errors

teuthology.repo_utils.fetch(repo_path)

Call “git fetch -p origin”

Parameters:repo_path – The full path to the repository
Raises:GitError if the operation fails
teuthology.repo_utils.fetch_branch(repo_path, branch, shallow=True)

Call “git fetch -p origin <branch>”

Parameters:
  • repo_path – The full path to the repository on-disk
  • branch – The branch.
  • shallow – Whether to perform a shallow fetch (–depth 1)
Raises:

BranchNotFoundError if the branch is not found; GitError for other errors

teuthology.repo_utils.fetch_qa_suite(branch, lock=True)

Make sure ceph-qa-suite is checked out.

Parameters:branch – The branch to fetch
Returns:The destination path
teuthology.repo_utils.fetch_refspec(ref)
teuthology.repo_utils.fetch_repo(url, branch, bootstrap=None, lock=True)

Make sure we have a given project’s repo checked out and up-to-date with the current branch requested

Parameters:
  • url – The URL to the repo
  • bootstrap – An optional callback function to execute. Gets passed a dest_dir argument: the path to the repo on-disk.
  • branch – The branch we want
Returns:

The destination path

teuthology.repo_utils.fetch_teuthology(branch, lock=True)

Make sure we have the correct teuthology branch checked out and up-to-date

Parameters:branch – The branch we want
Returns:The destination path
teuthology.repo_utils.is_fresh(path)

Has this file been modified in the last FRESHNESS_INTERVAL seconds?

Returns False if the file does not exist

teuthology.repo_utils.local_branch_from_ref(ref)
teuthology.repo_utils.ls_remote(url, ref)

Return the current sha1 for a given repository and ref

Returns:The sha1 if found; else None
teuthology.repo_utils.lsstrip(s, prefix)
teuthology.repo_utils.ref_to_dirname(branch)
teuthology.repo_utils.remote_ref_from_ref(ref, remote='origin')
teuthology.repo_utils.remove_pyc_files(dest_path)
teuthology.repo_utils.reset_repo(repo_url, dest_path, branch)
Parameters:
  • repo_url – The full URL to the repo (not including the branch)
  • dest_path – The full path to the destination directory
  • branch – The branch.
Raises:

BranchNotFoundError if the branch is not found; GitError for other errors

teuthology.repo_utils.rsstrip(s, suffix)
teuthology.repo_utils.set_remote(repo_path, repo_url)

Call “git remote set-url origin <repo_url>”

Parameters:
  • repo_url – The full URL to the repo (not including the branch)
  • repo_path – The full path to the repository
Raises:

GitError if the operation fails

teuthology.repo_utils.touch_file(path)
teuthology.repo_utils.url_to_dirname(url)

Given a URL, returns a string that’s safe to use as a directory name. Examples:

git://git.ceph.com/ceph-qa-suite.git -> git.ceph.com_ceph-qa-suite https://github.com/ceph/ceph -> github.com_ceph_ceph https://github.com/liewegas/ceph.git -> github.com_liewegas_ceph file:///my/dir/has/ceph.git -> my_dir_has_ceph
teuthology.repo_utils.validate_branch(branch)

teuthology.report module

class teuthology.report.ResultsReporter(archive_base=None, base_uri=None, save=False, refresh=False, log=None)

Bases: object

delete_job(run_name, job_id)

Delete a job from the results server.

Parameters:
  • run_name – The name of the run
  • job_id – The job’s id
delete_jobs(run_name, job_ids)

Delete multiple jobs from the results server.

Parameters:
  • run_name – The name of the run
  • job_ids – A list of job ids
delete_run(run_name)

Delete a run from the results server.

Parameters:run_name – The name of the run
get_jobs(run_name, job_id=None, fields=None)

Query the results server for jobs in a run

Parameters:
  • run_name – The name of the run
  • job_id – Optionally get a single job instead of all
  • fields – Optional. A list of fields to include in the result. Defaults to returning all fields.
get_rerun_conf(run_name)
get_run(run_name, fields=None)

Query the results server for a run

Parameters:
  • run_name – The name of the run
  • fields – Optional. A list of fields to include in the result. Defaults to returning all fields.
last_run

The last run to be successfully reported.

last_run_file = 'last_successful_run'
report_all_runs()

Report all runs in self.archive_dir to the results server.

report_job(run_name, job_id, job_info=None, dead=False)

Report a single job to the results server.

Parameters:
  • run_name – The name of the run. The run must already exist.
  • job_id – The job’s id
  • job_info – The job’s info dict. Optional - if not present, we look at the archive.
report_jobs(run_name, job_ids, dead=False)

Report several jobs to the results server.

Parameters:
  • run_name – The name of the run.
  • job_ids – The jobs’ ids
report_run(run_name, dead=False)

Report a single run to the results server.

Parameters:run_name – The name of the run.
Returns:The number of jobs reported.
report_runs(run_names)

Report several runs to the results server.

Parameters:run_names – The names of the runs.
class teuthology.report.ResultsSerializer(archive_base, log=None)

Bases: object

This class exists to poke around in the archive directory doing things like assembling lists of test runs, lists of their jobs, and merging sets of job YAML files together to form JSON objects.

all_runs

Look in the base archive directory for all test runs. Return a list of their names.

job_info(run_name, job_id, pretty=False, simple=False)

Given a run name and job id, merge the job’s YAML files together.

Parameters:
  • run_name – The name of the run.
  • job_id – The job’s id.
  • simple(bool) – Read less data for speed (only orig.config.yaml/info.yaml)
Returns:

A dict.

jobs_for_run(run_name)

Given a run name, look on the filesystem for directories containing job information, and return a dict mapping job IDs to job directories.

Parameters:run_name – The name of the run.
Returns:A dict like: {‘1’: ‘/path/to/1’, ‘2’: ‘path/to/2’}
json_for_job(run_name, job_id, pretty=False)

Given a run name and job id, merge the job’s YAML files together to create a JSON object.

Parameters:
  • run_name – The name of the run.
  • job_id – The job’s id.
Returns:

A JSON object.

running_jobs_for_run(run_name)

Like jobs_for_run(), but only returns jobs with no summary.yaml

Parameters:run_name – The name of the run.
Returns:A dict like: {‘1’: ‘/path/to/1’, ‘2’: ‘path/to/2’}
yamls = ('orig.config.yaml', 'config.yaml', 'info.yaml', 'summary.yaml')
teuthology.report.init_logging()

Set up logging for the module

Returns:a logger
teuthology.report.main(args)
teuthology.report.push_job_info(run_name, job_id, job_info, base_uri=None)

Push a job’s info (example: ctx.config) to the results server.

Parameters:
  • run_name – The name of the run.
  • job_id – The job’s id
  • job_info – A dict containing the job’s information.
  • base_uri – The endpoint of the results server. If you leave it out ResultsReporter will ask teuthology.config.
teuthology.report.try_delete_jobs(run_name, job_ids, delete_empty_run=True)

Using the same error checking and retry mechanism as try_push_job_info(), delete one or more jobs

Parameters:
  • run_name – The name of the run.
  • job_ids – Either a single job_id, or a list of job_ids
  • delete_empty_run – If this would empty the run, delete it.
teuthology.report.try_mark_run_dead(run_name)

Using the same error checking and retry mechanism as try_push_job_info(), mark any unfinished runs as dead.

Parameters:run_name – The name of the run.
teuthology.report.try_push_job_info(job_config, extra_info=None)
Wrap push_job_info, gracefully doing nothing if:
Anything inheriting from requests.exceptions.RequestException is raised A socket.error is raised config.results_server is not set config[‘job_id’] is not present or is None
Parameters:
  • job_config – The ctx.config object to push
  • extra_info – Optional second dict to push

teuthology.results module

teuthology.results.build_email_body(name, _reporter=None)
teuthology.results.email_results(subject, from_, to, body)
teuthology.results.format_job(run_name, job)
teuthology.results.generate_coverage(archive_dir, name)
teuthology.results.main(args)
teuthology.results.note_rerun_params(subset, seed)
teuthology.results.results(archive_dir, name, email, timeout, dry_run)
teuthology.results.seconds_to_hms(seconds)

teuthology.run module

teuthology.run.fetch_tasks_if_needed(job_config)

Fetch the suite repo (and include it in sys.path) so that we can use its tasks.

Returns the suite_path. The existing suite_path will be returned if the tasks can be imported, if not a new suite_path will try to be determined.

teuthology.run.get_initial_tasks(lock, config, machine_type)
teuthology.run.get_machine_type(machine_type, config)

If no machine_type is given, find the appropriate machine_type from the given config.

teuthology.run.get_summary(owner, description)
teuthology.run.get_teuthology_command(args)

Rebuilds the teuthology command used to run this job and returns it as a string.

teuthology.run.main(args)
teuthology.run.report_outcome(config, archive, summary, fake_ctx)

Reports on the final outcome of the command.

teuthology.run.set_up_logging(verbose, archive)
teuthology.run.setup_config(config_paths)

Takes a list of config yaml files and combines them into a single dictionary. Processes / validates the dictionary and then returns it.

teuthology.run.validate_tasks(config)

Ensures that config tasks is a list and doesn’t include ‘kernel’.

Returns the original tasks key if found. If not, returns an empty list.

teuthology.run.write_initial_metadata(archive, config, name, description, owner)

teuthology.run_tasks module

teuthology.run_tasks.get_task(name)
teuthology.run_tasks.run_one_task(taskname, **kwargs)
teuthology.run_tasks.run_tasks(tasks, ctx)

teuthology.safepath module

teuthology.safepath.makedirs(root, path)

os.makedirs gets confused if the path contains ‘..’, and root might.

This relies on the fact that path has been normalized by munge().

teuthology.safepath.munge(path)

Munge a potentially hostile path name to be safe to use.

This very definitely changes the meaning of the path, but it only does that for unsafe paths.

teuthology.schedule module

teuthology.schedule.build_config(args)

Given a dict of arguments, build a job config

teuthology.schedule.main(args)
teuthology.schedule.schedule_job(job_config, num=1)

Schedule a job.

Parameters:
  • job_config – The complete job dict
  • num – The number of times to schedule the job

teuthology.sentry module

teuthology.sentry.get_client()

teuthology.timer module

class teuthology.timer.Timer(path=None, sync=False)

Bases: object

A class that records timing data.

It was created in order to record time intervals between the execution of different tasks’ enter and exit methods.

data

Return an object similar to:

{'start': '2016-02-02_23:19:51',
 'elapsed': 10.65,
 'end': '2016-02-02_23:20:01',
 'marks': [
     {'message': 'event 1', 'interval': 0.0},
     {'message': 'event 2', 'interval': 8.58},
     {'message': 'event 3', 'interval': 10.65}
 ],
 }

‘start’ and ‘end’ times are in UTC.

datetime_format = '%Y-%m-%d_%H:%M:%S'
get_datetime_string(time)

Return a human-readable timestamp in UTC

Parameters:time – Time in seconds; like from time.time()
mark(message='')

Create a time mark

If necessary, call self._mark_start() to begin time-keeping. Then, create a new entry in self.marks with the message provided, along with the time elapsed in seconds since time-keeping began.

precision = 3
write()

teuthology.worker module

teuthology.worker.load_config(ctx=None)
teuthology.worker.main(ctx)
teuthology.worker.prep_job(job_config, log_file_path, archive_dir)
teuthology.worker.restart()
teuthology.worker.run_job(job_config, teuth_bin_path, archive_dir, verbose)
teuthology.worker.run_with_watchdog(process, job_config)
teuthology.worker.sentinel(path)
teuthology.worker.stop()

Module contents

teuthology.install_except_hook()

Install an exception hook that first logs any uncaught exception, then raises it.

teuthology.patch_gevent_hub_error_handler()
teuthology.setup_log_file(log_path)