plastid.util.io.filters module

Utility classes, analagous to Unix-style pipes, for filtering or processing input or output streams, such as file objects.

These filters may be composed by wrapping one around another, to perform multiple operations in line on, for example, a text stream, before finally converting it to another type of data.

Various readers are included:

AbstractReader

Base class for all Readers. To create a Reader, subclass this and override the filter() method.

CommentReader

read through text documents, skipping over commented lines

SkipBlankReader

read through text documents, skipping over blank lines

FunctionReader

apply an arbitrary function to each unit of input data

TeeReader

similar to Unix/Linux shell command tee. Allows input streams to be sent to an arbitrary number of listeners

And various writers:

AbstractWriter

Base class for all writers. To create a Writer, subclass this and override the filter() method.

ColorWriter

Enable ANSI coloring of text to output streams that support color. For streams that do not support color, text is not colored.

NameDateWriter

Prepend timestamps to each line of string input before writing

CommentWriter

Filter out commented lines from text stream before writing

And one convenience function:

colored()

Colorize text (via termcolor.colored()) if and only if color is supported by sys.stderr

Examples

Open a file, skipping comments and blank lines:

>>> my_reader = CommentReader(SkipBlankReader(open("some_file.txt")))
>>> for line in my_reader:
>>>     pass # do something with each line, now that comments are removed

Open a file, applying a function foo_func to each line of input (note, foo_func can return any data type, not just a string):

>>> my_reader = FunctionReader(open("some_file.txt"),foo_func)
>>> for data_unit in my_reader:
>>>     pass # do something with each `foo`ed data unit

Write to stdout, prepending name and date:

>>> import sys
>>> my_writer = NameDateWriter(stream=sys.stdout)
>>> my_writer.write(some_text)

class plastid.util.io.filters.BackwardReader(stream)

Bases: plastid.util.io.filters.AbstractReader

Reverses each line of a text stream character-wise.

Attributes

closed

Methods

close()	Close stream
fileno()	Returns underlying file descriptor if one exists.
filter(line)	Return next non-commented line of text
flush(/)	Flush write buffers, if applicable.
isatty()	Return whether this is an 'interactive' stream.
read()	Similar to file.read().
readable()	Return whether object was opened for reading.
readline()	Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines()	Similar to file.readlines().
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
writelines(lines, /)	Write a list of lines to stream.

next

close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

filter(line)

Return next non-commented line of text

Parameters

linestr

Line of text

Returns

str

Reversed line of text

flush(/)

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next()

read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns

str

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns

object

a unit of processed data

readlines()

Similar to file.readlines().

Returns

list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.ColorWriter(stream=None)

Bases: plastid.util.io.filters.AbstractWriter

Detect whether output stream supports color, and enable/disable colored output

Parameters

streamfile-like

Stream to write to (Default: sys.stderr)

Attributes

closed

Methods

:meth:`color`

Color text. Delegates to termcolor.colored() if color is supported. Otherwise, returns uncolored text.

close()

flush and close self.stream

color(text, **kwargs)

Color text with attributes specified in kwargs if stream supports ANSI color.

See termcolor.colored() for usage

Returns

str

text, colored as indicated, if color is supported

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

abstract filter(data)

Method that filters or processes each unit of data. Override this in subclasses

Parameters

dataunit of data

Whatever data to filter/format. Often string, but not necessarily

Returns

object

formatted data. Often string, but not necessary

flush()

Flush self.stream

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline(size=- 1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=- 1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

write(data)

Write data to self.stream

Parameters

dataunit of data

Whatever data to filter/format. Often string, but not necessary

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.CommentReader(stream)

Bases: plastid.util.io.filters.AbstractReader

Ignore lines beginning with ‘#’, optionally preceded by whitespace from a text stream. Comments beginning mid line are left in-place

Attributes

closed

Methods

close()	Close stream
fileno()	Returns underlying file descriptor if one exists.
filter(line)	Return next non-commented line of text
flush(/)	Flush write buffers, if applicable.
get_comments()	Return all of the comments that have been found so far.
isatty()	Return whether this is an 'interactive' stream.
read()	Similar to file.read().
readable()	Return whether object was opened for reading.
readline()	Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines()	Similar to file.readlines().
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
writelines(lines, /)	Write a list of lines to stream.

next

close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

filter(line)

Return next non-commented line of text

Parameters

linestr

Line of text

Returns

str

flush(/)

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

get_comments()

Return all of the comments that have been found so far. Does not remove comments from the internal list.

Returns

list

Comments found in text

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next()

read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns

str

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns

object

a unit of processed data

readlines()

Similar to file.readlines().

Returns

list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.CommentWriter(stream)

Bases: plastid.util.io.filters.AbstractWriter

Filter out lines beginning with ‘#’ from data written to a stream

Attributes

closed

Methods

close()	flush and close self.stream
fileno()	Returns underlying file descriptor if one exists.
filter(line)	Method that filters or processes each unit of data.
flush()	Flush self.stream
isatty()	Return whether this is an 'interactive' stream.
readable()	Return whether object was opened for reading.
readline([size])	Read and return a line from the stream.
readlines([hint])	Return a list of lines from the stream.
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
write(data)	Write data to self.stream
writelines(lines, /)	Write a list of lines to stream.

close()

flush and close self.stream

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

filter(line)

Method that filters or processes each unit of data. Override this in subclasses

Parameters

dataunit of data

Whatever data to filter/format. Often string, but not necessarily

Returns

object

formatted data. Often string, but not necessary

flush()

Flush self.stream

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline(size=- 1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=- 1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

write(data)

Write data to self.stream

Parameters

dataunit of data

Whatever data to filter/format. Often string, but not necessary

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.FunctionReader(stream, func)

Bases: plastid.util.io.filters.AbstractReader

Apply a function to each line in an input stream

Parameters

streamfile-like

Input stream

funcfunction

Function to apply to each unit of input in stream

Attributes

closed

Methods

close()	Close stream
fileno()	Returns underlying file descriptor if one exists.
filter(data)	Method that filters or processes each unit of data.
flush(/)	Flush write buffers, if applicable.
isatty()	Return whether this is an 'interactive' stream.
read()	Similar to file.read().
readable()	Return whether object was opened for reading.
readline()	Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines()	Similar to file.readlines().
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
writelines(lines, /)	Write a list of lines to stream.

next

close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

abstract filter(data)

Method that filters or processes each unit of data. Override this in subclasses

Parameters

dataunit of data

Whatever data to filter/format. Often string, but not necessary

Returns

object

formatted data. Often string, but not necessarily

flush(/)

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next()

read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns

str

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns

object

a unit of processed data

readlines()

Similar to file.readlines().

Returns

list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.NameDateWriter(name, line_delimiter='\n', stream=None)

Bases: plastid.util.io.filters.ColorWriter

Prepend program name, date, and time to each line of output

Attributes

closed

Methods

__call__(line)	Call self as a function.
close()	flush and close self.stream
color(text, **kwargs)	Color text with attributes specified in kwargs if stream supports ANSI color.
fileno()	Returns underlying file descriptor if one exists.
filter(line)	Prepend date and time to each line of input
flush()	Flush self.stream
isatty()	Return whether this is an 'interactive' stream.
readable()	Return whether object was opened for reading.
readline([size])	Read and return a line from the stream.
readlines([hint])	Return a list of lines from the stream.
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
write(data)	Write data to self.stream
writelines(lines, /)	Write a list of lines to stream.

close()

flush and close self.stream

color(text, **kwargs)

Color text with attributes specified in kwargs if stream supports ANSI color.

See termcolor.colored() for usage

Returns

str

text, colored as indicated, if color is supported

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

filter(line)

Prepend date and time to each line of input

Parameters

linestr

Input

Returns

strInput with date and time prepended

flush()

Flush self.stream

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline(size=- 1, /)

Read and return a line from the stream.

If size is specified, at most size bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines(hint=- 1, /)

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

write(data)

Write data to self.stream

Parameters

dataunit of data

Whatever data to filter/format. Often string, but not necessary

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

plastid.util.io.filters.Printer

alias of plastid.util.io.filters.NameDateWriter

class plastid.util.io.filters.SkipBlankReader(stream)

Bases: plastid.util.io.filters.AbstractReader

Ignores blank/whitespace-only lines in a text stream

Attributes

closed

Methods

close()	Close stream
fileno()	Returns underlying file descriptor if one exists.
filter(line)	Return next non-blank line of text
flush(/)	Flush write buffers, if applicable.
isatty()	Return whether this is an 'interactive' stream.
read()	Similar to file.read().
readable()	Return whether object was opened for reading.
readline()	Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines()	Similar to file.readlines().
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
writelines(lines, /)	Write a list of lines to stream.

next

close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

filter(line)

Return next non-blank line of text

Parameters

linestr

Line of text

Returns

str

flush(/)

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next()

read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns

str

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns

object

a unit of processed data

readlines()

Similar to file.readlines().

Returns

list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.TeeListener

Bases: object

Listener class for TeeFilter. Listeners if registered with a TeeReader will receive and process each unit of input via its alert() method

Methods

alert(data)

Process input from a TeeReader.

abstract alert(data)

Process input from a TeeReader. Override this method to perform the appropriate behavior.

Parameters

dataobject

A unit of data

class plastid.util.io.filters.TeeReader(stream)

Bases: plastid.util.io.filters.AbstractReader

Tee an input stream to listeners that register themselves with the TeeReader via the add_listener() method. Similar to shell command tee

Each listener must implement an alert() method, in order to receive the data. If alert() is not implemented, errors are suppressed.

See also

TeeListener

an example of a listener class

Attributes

closed

Methods

add_listener(listener)	Register a single listener with this reader
add_listeners(*many_listeners)	Register one or more listeners with this reader
close()	Close stream
fileno()	Returns underlying file descriptor if one exists.
filter(line)	Sends each line to each listener.
flush(/)	Flush write buffers, if applicable.
isatty()	Return whether this is an 'interactive' stream.
read()	Similar to file.read().
readable()	Return whether object was opened for reading.
readline()	Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines()	Similar to file.readlines().
seek	Change stream position.
seekable()	Return whether object supports random access.
tell(/)	Return current stream position.
truncate	Truncate file to size bytes.
writable()	Return whether object was opened for writing.
writelines(lines, /)	Write a list of lines to stream.

next

add_listener(listener)

Register a single listener with this reader

Parameters

listenerTeeListener-like

add_listeners(*many_listeners)

Register one or more listeners with this reader

Parameters

many_listenersone or more TeeListener-like

close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

OSError is raised if the IO object does not use a file descriptor.

filter(line)

Sends each line to each listener. Complains if listener cannot listen!

Parameters

linea unit of input data

Returns

input data

flush(/)

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next()

read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns

str

readable()

Return whether object was opened for reading.

If False, read() will raise OSError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns

object

a unit of processed data

readlines()

Similar to file.readlines().

Returns

list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

• 0 – start of stream (the default); offset should be zero or positive

• 1 – current stream position; offset may be negative

• 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise OSError. This method may need to do a test seek().

tell(/)

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, write() will raise OSError.

writelines(lines, /)

Write a list of lines to stream.

Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end.

closed

class plastid.util.io.filters.TestTeeListener

Bases: plastid.util.io.filters.TeeListener

Example of a TeeListener

Methods

alert(line)

Process input from a TeeReader.

alert(line)

Process input from a TeeReader. Override this method to perform the appropriate behavior.

Parameters

dataobject

A unit of data

plastid.util.io.filters.colored(x, **kwargs)