Code snippets

This is Python code snippets I often reuse between different software. One day, maybe parts of this could be merged in the standard library or at least shipped in a reusable library?

Code documentation

This is the automatically generated documentation for the Python code.

ecdysis.argparse

improvements to the standard :module:`argparse` module

class ecdysis.argparse.NegateAction(option_strings, *args, **kwargs)

add a toggle flag to argparse

this is similar to ‘store_true’ or ‘store_false’, but allows arguments prefixed with –no to disable the default. the default is set depending on the first argument - if it starts with the negative form (defined by default as ‘–no’), the default is False, otherwise True.

originally written for the stressant project.

@deprecated use the BooleanOptionalAction from Python 3.9 instead, although it doesn’t have the default override we implemented here.

negative = '--no'

class ecdysis.argparse.ConfigAction(*args, **kwargs)

add configuration file to current defaults.

a list of default config files can be specified and will be parsed when added by ConfigArgumentParser.

it was reported this might not work well with subparsers, patches to fix that are welcome.

parse_config(path: str) → dict

abstract implementation of config file parsing, should be overridden in subclasses

class ecdysis.argparse.YamlConfigAction(*args, **kwargs)

YAML config file parser action

parse_config(path: str) → dict

This doesn’t handle errors around open() and others, callers should probably catch FileNotFoundError at least.

class ecdysis.argparse.ConfigArgumentParser(*args, **kwargs)

argument parser which supports parsing extra config files

Config files specified on the commandline through the YamlConfigAction arguments modify the default values on the spot. If a default is specified when adding an argument, it also gets immediately loaded.

This will typically be used in a subclass, like this:

self.add_argument(’–config’, action=YamlConfigAction, default=self.default_config())

This shows how the configuration file overrides the default value for an option:

>>> from tempfile import NamedTemporaryFile
>>> c = NamedTemporaryFile()
>>> c.write(b"foo: delayed\n")
13
>>> c.flush()
>>> parser = ConfigArgumentParser()
>>> a = parser.add_argument('--foo', default='bar')
>>> a = parser.add_argument('--config', action=YamlConfigAction, default=[c.name])
>>> args = parser.parse_args([])
>>> args.config == [c.name]
True
>>> args.foo
'delayed'
>>> args = parser.parse_args(['--foo', 'quux'])
>>> args.foo
'quux'

This is the same test, but with –config called earlier, which should still work:

>>> from tempfile import NamedTemporaryFile
>>> c = NamedTemporaryFile()
>>> c.write(b"foo: quux\n")
10
>>> c.flush()
>>> parser = ConfigArgumentParser()
>>> a = parser.add_argument('--config', action=YamlConfigAction, default=[c.name])
>>> a = parser.add_argument('--foo', default='bar')
>>> args = parser.parse_args([])
>>> args.config == [c.name]
True
>>> args.foo
'quux'
>>> args = parser.parse_args(['--foo', 'baz'])
>>> args.foo
'baz'

This tests that you can override the config file defaults altogether:

>>> parser = ConfigArgumentParser()
>>> a = parser.add_argument('--config', action=YamlConfigAction, default=[c.name])
>>> a = parser.add_argument('--foo', default='bar')
>>> args = parser.parse_args(['--config', '/dev/null'])
>>> args.foo
'bar'
>>> args = parser.parse_args(['--config', '/dev/null', '--foo', 'baz'])
>>> args.foo
'baz'

This tests multiple search paths, first one should be loaded:

>>> from tempfile import NamedTemporaryFile
>>> d = NamedTemporaryFile()
>>> d.write(b"foo: argh\n")
10
>>> d.flush()
>>> parser = ConfigArgumentParser()
>>> a = parser.add_argument('--config', action=YamlConfigAction, default=[d.name, c.name])
>>> a = parser.add_argument('--foo', default='bar')
>>> args = parser.parse_args([])
>>> args.foo
'argh'
>>> c.close()
>>> d.close()

There are actually many other implementations of this we might want to consider instead of maintaining our own:

https://github.com/omni-us/jsonargparse https://github.com/bw2/ConfigArgParse https://github.com/omry/omegaconf

See this comment for a quick review:

https://github.com/borgbackup/borg/issues/6551#issuecomment-1094104453

parse_args(args: Sequence[str] | None = None, namespace: Namespace | None = None) → Namespace

default_config() → Iterable[str]

handy shortcut to detect commonly used config paths

This list is processed as a FIFO: if a file is found in there, it will be parsed and the remaining ones will be ignored.

class ecdysis.argparse.LoggingAction(*args, **kwargs)

change log level on the fly

The logging system should be initialized before this, using basicConfig.

Example usage:

logging.basicConfig(level=”WARNING”, format=”%(message)s”) parser.add_argument(

“-v”, “–verbose”, action=LoggingAction, const=”INFO”, help=”enable verbose messages”,

) parser.add_argument(

“-d”, “–debug”, action=LoggingAction, const=”DEBUG”, help=”enable debugging messages”,

)

Or, if you want to default to “verbose” (AKA “INFO”):

logging.basicConfig(format=”%(message)s”) # INFO is default parser.add_argument(

“-q”, “–quiet”, action=LoggingAction, const=”WARNING”, help=”silence messages except warnings and errors”,

) parser.add_argument(

“-d”, “–debug”, action=LoggingAction, const=”DEBUG”, help=”enable debugging messages”,

)

ecdysis.cli

various commandline tools

class ecdysis.cli.throbber(factor=0, stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, symbol='.', fmt='{}', i=1)

weird logarithmic “progress bar”

when a throbber object is called, will display progress using the provided “symbol”

the throbber will print the symbol every time it’s called until it crosses a logarithmic threshold (the “factor”), at which point the factor is increased.

this is useful to display progress on large datasets that have an unknown size (so we can’t guess completion time and we can’t reasonably guess the progress/display ratio).

originally from the code I wrote for the Euler project

this function requires Python 3.3 at least, because it uses print(flush=True)

Other progress bars include:

Rich: https://rich.readthedocs.io/en/stable/progress.html tqdm: https://github.com/tqdm/tqdm progress: https://pypi.org/project/progress/ progressbar: https://pypi.org/project/progressbar/

class ecdysis.cli.Prompter

Set of prompt utilities.

This is untested. It mostly comes from Monkeysign, but was rewritten for notmuch-sync-flagged and in doing so, was significantly refactored without further tests.

This could possibly be replaced with:

https://github.com/prompt-toolkit/python-prompt-toolkit https://github.com/Mckinsey666/bullet

yes_no(prompt, default='y', choices=['y', 'n'])

This will show the given prompt, check if it matches the given choices, and return True if it matches the first choice provided. If some “false” string (e.g. empty string which happens when you just hit “enter”) is provided, the default value (which should be a boolean) is returned.

For unit testing, the input function can be overridden with input_func.

>>> prompter = Prompter()
>>> prompter.input = lambda x: 'y'
>>> prompter.yes_no('foo')
True
>>> prompter.input = lambda x: 'n'
>>> prompter.yes_no('foo')
False
>>> prompter.input = lambda x: ''
>>> prompter.yes_no('foo', default='y')
True
>>> prompter.yes_no('foo', default='n')
False

pick(prompt, default, choices)

acknowledge(prompt=None)

Just wait for the user to hit enter and return.

input(prompt)

Wrapper around python’s input function, to ease testing.

input_pass(prompt)

Input without showing the typed characters on the terminal.

ecdysis.logging

• similarly, bup-cron has this GlobalLogger and a Singleton concept that may be useful elsewhere? it certainly does a nice job at setting up all sorts of handlers and stuff. stressant also has a setup_logging function that also supports colors and SMTP mailers. debmans has a neat log_warnings hook as well.

• monkeysign also has facilities to (ab)use the logging handlers to send stuff to the GTK framework (GTKLoggingHandler) and a error handler in GTK (in msg_exception.py)

ecdysis.logging.logging_args(parser)

>>> from pprint import pprint
>>> parser = argparse.ArgumentParser()
>>> logging_args(parser)
>>> pprint(sorted(parser.parse_args(['--verbose']).__dict__.items()))
[('email', None),
 ('logfile', None),
 ('loglevel', 'INFO'),
 ('smtppass', None),
 ('smtpserver', None),
 ('smtpuser', None),
 ('syslog', None)]
>>> pprint(sorted(parser.parse_args(['--verbose', '--debug']).__dict__.items()))
[('email', None),
 ('logfile', None),
 ('loglevel', 'DEBUG'),
 ('smtppass', None),
 ('smtpserver', None),
 ('smtpuser', None),
 ('syslog', None)]
>>> pprint(sorted(parser.parse_args(['--verbose', '--syslog']).__dict__.items()))
[('email', None),
 ('logfile', None),
 ('loglevel', 'INFO'),
 ('smtppass', None),
 ('smtpserver', None),
 ('smtpuser', None),
 ('syslog', 'INFO')]

ecdysis.logging.advancedConfig(level='warning', stream=None, syslog=False, prog=None, email=False, smtpparams=None, logfile=None, logFormat='%(levelname)s: %(message)s', **kwargs)

setup standard Python logging facilities

this was taken from the debmans and stressant loggers, although it lacks stressant’s color support

Parameters:

• level (str) – logging level, usually one of levels

• stream (file) – stream to send logging events to, or None to

use the logging default (usually stderr)

Parameters:

syslog (str) – send log events to syslog at the specified

level. defaults to False, which doesn’t send syslog events

Parameters:

prog (str) – the program name to use in syslog lines, defaults

to .__prog__

Parameters:

email (str) – send logs by email to the given email address

using the BufferedSMTPHandler

Parameters:

smtpparams (dict) – parameters to use when sending

email. expected fields are:

• fromaddr (defaults to $USER@$FQDN)

• subject (defaults to ‘’)

• mailhost (defaults to the last part of the destination email)

• user (to authenticate against the SMTP server, defaults to no auth)

• pass (password to use, prompted using getpass otherwise)

Parameters:

logfile (str) – filename to pass to the FileHandler to log

directly to a file

Parameters:

logFormat (str) – logformat to use for the FileHandler and

BufferedSMTPHandler

class ecdysis.logging.BufferedSMTPHandler(mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None, capacity=5000, flushLevel=40, retries=1)

A handler class which sends records only when the buffer reaches capacity. The object is constructed with the arguments from SMTPHandler and MemoryHandler and basically behaves as a merge between the two classes.

The SMTPHandler.emit() implementation was copy-pasted here because it is not flexible enough to be overridden. We could possibly override the format() function to instead look at the internal buffer, but that would have possibly undesirable side-effects.

emit(record)

buffer the record in the MemoryHandler

flush()

Flush all records.

Format the records and send it to the specified addressees.

The only change from SMTPHandler here is the way the email body is created.

ecdysis.os

various overrides to the builtin os library

ecdysis.os.make_dirs_helper(path)

Create the directory if it does not exist

Return True if the directory was created, false if it was already present, throw an OSError exception if it cannot be created

>>> import tempfile
>>> import os
>>> import os.path as p
>>> d = tempfile.mkdtemp()
>>> make_dirs_helper(p.join(d, 'foo'))
True
>>> make_dirs_helper(p.join(d, 'foo'))
False
>>> make_dirs_helper(p.join('/dev/null', 'foo')) 
Traceback (most recent call last):
    ...
NotADirectoryError: [Errno 20] Not a directory: ...
>>> os.rmdir(p.join(d, 'foo'))
>>> os.rmdir(d)
>>>

ecdysis.packaging

ecdysis.packaging.find_parent_module()

find the name of a the first module calling this module

if we cannot find it, we return the current module’s name (__name__) instead.

ecdysis.packaging.find_static_file(path, module=None)

locate a file in the distribution

this will look in the shipped files in the package

this assumes the files are at the root of the package or the source tree (if not packaged)

this does not check if the file actually exists.

Parameters:

• path (str) – path for the file, relative to the source tree root

• module (str) – name of the module to find the find in. if None, guessed with find_parent_module()

Returns:

the absolute path to the file

ecdysis.strings

ecdysis.time

Indices and tables

• Index

• Module Index

• Search Page