plastid.util.io.filters module

Utility classes, analagous to Unix-style pipes, for filtering or processing input or output streams, such as file objects.

These filters may be composed by wrapping one around another, to perform multiple operations in line on, for example, a text stream, before finally converting it to another type of data.

Various readers are included:

AbstractReader
Base class for all Readers. To create a Reader, subclass this and override the filter() method.
CommentReader
read through text documents, skipping over commented lines
SkipBlankReader
read through text documents, skipping over blank lines
FunctionReader
apply an arbitrary function to each unit of input data
TeeReader
similar to Unix/Linux shell command tee. Allows input streams to be sent to an arbitrary number of listeners

And various writers:

AbstractWriter
Base class for all writers. To create a Writer, subclass this and override the filter() method.
ColorWriter
Enable ANSI coloring of text to output streams that support color. For streams that do not support color, text is not colored.
NameDateWriter
Prepend timestamps to each line of string input before writing
CommentWriter
Filter out commented lines from text stream before writing

And one convenience function:

colored()
Colorize text (via termcolor.colored()) if and only if color is supported by sys.stderr

Examples

Open a file, skipping comments and blank lines:

>>> my_reader = CommentReader(SkipBlankReader(open("some_file.txt")))
>>> for line in my_reader:
>>>     pass # do something with each line, now that comments are removed

Open a file, applying a function foo_func to each line of input (note, foo_func can return any data type, not just a string):

>>> my_reader = FunctionReader(open("some_file.txt"),foo_func)
>>> for data_unit in my_reader:
>>>     pass # do something with each `foo`ed data unit

Write to stdout, prepending name and date:

>>> import sys
>>> my_writer = NameDateWriter(stream=sys.stdout)
>>> my_writer.write(some_text)
class plastid.util.io.filters.BackwardReader(stream)[source]

Bases: plastid.util.io.filters.AbstractReader

Reverses each line of a text stream character-wise.

Attributes:
closed

Methods

close() Close stream
filter(line) Return next non-commented line of text
flush Flush write buffers, if applicable.
read() Similar to file.read().
readline() Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines() Similar to file.readlines().
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
fileno  
isatty  
next  
readable  
seekable  
writable  
writelines  
close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(line)[source]

Return next non-commented line of text

Parameters:
line : str

Line of text

Returns:
str

Reversed line of text

flush()

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next() → the next value, or raise StopIteration
read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns:
str
readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns:
object

a unit of processed data

readlines()

Similar to file.readlines().

Returns:
list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

writelines()
closed
class plastid.util.io.filters.ColorWriter(stream=None)[source]

Bases: plastid.util.io.filters.AbstractWriter

Detect whether output stream supports color, and enable/disable colored output

Parameters:
stream : file-like

Stream to write to (Default: sys.stderr)

Attributes:
closed

Methods

:meth:`color` Color text. Delegates to termcolor.colored() if color is supported. Otherwise, returns uncolored text.
close()

flush and close self.stream

color(text, **kwargs)[source]

Color text with attributes specified in kwargs if stream supports ANSI color.

See termcolor.colored() for usage

Returns:
str

text, colored as indicated, if color is supported

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(data)

Method that filters or processes each unit of data. Override this in subclasses

Parameters:
data : unit of data

Whatever data to filter/format. Often string, but not necessarily

Returns:
object

formatted data. Often string, but not necessary

flush()

Flush self.stream

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Read and return a line from the stream.

If limit is specified, at most limit bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines()

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

write(data)

Write data to self.stream

Parameters:
data : unit of data

Whatever data to filter/format. Often string, but not necessary

writelines()
closed
next
class plastid.util.io.filters.CommentReader(stream)[source]

Bases: plastid.util.io.filters.AbstractReader

Ignore lines beginning with ‘#’, optionally preceded by whitespace from a text stream. Comments beginning mid line are left in-place

Attributes:
closed

Methods

close() Close stream
filter(line) Return next non-commented line of text
flush Flush write buffers, if applicable.
get_comments() Return all of the comments that have been found so far.
read() Similar to file.read().
readline() Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines() Similar to file.readlines().
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
fileno  
isatty  
next  
readable  
seekable  
writable  
writelines  
close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(line)[source]

Return next non-commented line of text

Parameters:
line : str

Line of text

Returns:
str
flush()

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

get_comments()[source]

Return all of the comments that have been found so far. Does not remove comments from the internal list.

Returns:
list

Comments found in text

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next() → the next value, or raise StopIteration
read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns:
str
readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns:
object

a unit of processed data

readlines()

Similar to file.readlines().

Returns:
list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

writelines()
closed
class plastid.util.io.filters.CommentWriter(stream)[source]

Bases: plastid.util.io.filters.AbstractWriter

Filter out lines beginning with ‘#’ from data written to a stream

Attributes:
closed

Methods

close() flush and close self.stream
flush() Flush self.stream
next
readline Read and return a line from the stream.
readlines Return a list of lines from the stream.
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
write(data) Write data to self.stream
fileno  
filter  
isatty  
readable  
seekable  
writable  
writelines  
close()

flush and close self.stream

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(line)[source]

Method that filters or processes each unit of data. Override this in subclasses

Parameters:
data : unit of data

Whatever data to filter/format. Often string, but not necessarily

Returns:
object

formatted data. Often string, but not necessary

flush()

Flush self.stream

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Read and return a line from the stream.

If limit is specified, at most limit bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines()

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

write(data)

Write data to self.stream

Parameters:
data : unit of data

Whatever data to filter/format. Often string, but not necessary

writelines()
closed
next
class plastid.util.io.filters.FunctionReader(stream, func)[source]

Bases: plastid.util.io.filters.AbstractReader

Apply a function to each line in an input stream

Parameters:
stream : file-like

Input stream

func : function

Function to apply to each unit of input in stream

Attributes:
closed

Methods

close() Close stream
filter(data) Method that filters or processes each unit of data.
flush Flush write buffers, if applicable.
read() Similar to file.read().
readline() Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines() Similar to file.readlines().
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
fileno  
isatty  
next  
readable  
seekable  
writable  
writelines  
close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(data)

Method that filters or processes each unit of data. Override this in subclasses

Parameters:
data : unit of data

Whatever data to filter/format. Often string, but not necessary

Returns:
object

formatted data. Often string, but not necessarily

flush()

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next() → the next value, or raise StopIteration
read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns:
str
readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns:
object

a unit of processed data

readlines()

Similar to file.readlines().

Returns:
list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

writelines()
closed
class plastid.util.io.filters.NameDateWriter(name, line_delimiter='n', stream=None)[source]

Bases: plastid.util.io.filters.ColorWriter

Prepend program name, date, and time to each line of output

Attributes:
closed

Methods

__call__(…) <==> x(…)
close() flush and close self.stream
color(text, **kwargs) Color text with attributes specified in kwargs if stream supports ANSI color.
filter(line) Prepend date and time to each line of input
flush() Flush self.stream
next
readline Read and return a line from the stream.
readlines Return a list of lines from the stream.
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
write(data) Write data to self.stream
fileno  
isatty  
readable  
seekable  
writable  
writelines  
close()

flush and close self.stream

color(text, **kwargs)

Color text with attributes specified in kwargs if stream supports ANSI color.

See termcolor.colored() for usage

Returns:
str

text, colored as indicated, if color is supported

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(line)[source]

Prepend date and time to each line of input

Parameters:
line : str

Input

Returns:
str : Input with date and time prepended
flush()

Flush self.stream

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Read and return a line from the stream.

If limit is specified, at most limit bytes will be read.

The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.

readlines()

Return a list of lines from the stream.

hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

write(data)

Write data to self.stream

Parameters:
data : unit of data

Whatever data to filter/format. Often string, but not necessary

writelines()
closed
next
plastid.util.io.filters.Printer

alias of plastid.util.io.filters.NameDateWriter

class plastid.util.io.filters.SkipBlankReader(stream)[source]

Bases: plastid.util.io.filters.AbstractReader

Ignores blank/whitespace-only lines in a text stream

Attributes:
closed

Methods

close() Close stream
filter(line) Return next non-blank line of text
flush Flush write buffers, if applicable.
read() Similar to file.read().
readline() Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines() Similar to file.readlines().
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
fileno  
isatty  
next  
readable  
seekable  
writable  
writelines  
close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(line)[source]

Return next non-blank line of text

Parameters:
line : str

Line of text

Returns:
str
flush()

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next() → the next value, or raise StopIteration
read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns:
str
readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns:
object

a unit of processed data

readlines()

Similar to file.readlines().

Returns:
list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

writelines()
closed
class plastid.util.io.filters.TeeListener[source]

Bases: object

Listener class for TeeFilter. Listeners if registered with a TeeReader will receive and process each unit of input via its alert() method

Methods

alert(data) Process input from a TeeReader.
alert(data)[source]

Process input from a TeeReader. Override this method to perform the appropriate behavior.

Parameters:
data : object

A unit of data

class plastid.util.io.filters.TeeReader(stream)[source]

Bases: plastid.util.io.filters.AbstractReader

Tee an input stream to listeners that register themselves with the TeeReader via the add_listener() method. Similar to shell command tee

Each listener must implement an alert() method, in order to receive the data. If alert() is not implemented, errors are suppressed.

See also

TeeListener
an example of a listener class
Attributes:
closed

Methods

add_listener(listener) Register a single listener with this reader
add_listeners(*many_listeners) Register one or more listeners with this reader
close() Close stream
filter(line) Sends each line to each listener.
flush Flush write buffers, if applicable.
read() Similar to file.read().
readline() Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.
readlines() Similar to file.readlines().
seek Change stream position.
tell Return current stream position.
truncate Truncate file to size bytes.
fileno  
isatty  
next  
readable  
seekable  
writable  
writelines  
add_listener(listener)[source]

Register a single listener with this reader

Parameters:
listener : TeeListener-like
add_listeners(*many_listeners)[source]

Register one or more listeners with this reader

Parameters:
many_listeners : one or more TeeListener-like
close()

Close stream

fileno()

Returns underlying file descriptor if one exists.

An IOError is raised if the IO object does not use a file descriptor.

filter(line)[source]

Sends each line to each listener. Complains if listener cannot listen!

Parameters:
line : a unit of input data
Returns:
input data
flush()

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

isatty()

Return whether this is an ‘interactive’ stream.

Return False if it can’t be determined.

next() → the next value, or raise StopIteration
read()

Similar to file.read(). Process all units of data, assuming it is string-like

Returns:
str
readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readline()

Process a single line of data, assuming it is string-like next(self) is more likely to behave as expected.

Returns:
object

a unit of processed data

readlines()

Similar to file.readlines().

Returns:
list

processed data

seek()

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive
  • 1 – current stream position; offset may be negative
  • 2 – end of stream; offset is usually negative

Return the new absolute position.

seekable()

Return whether object supports random access.

If False, seek(), tell() and truncate() will raise IOError. This method may need to do a test seek().

tell()

Return current stream position.

truncate()

Truncate file to size bytes.

File pointer is left unchanged. Size defaults to the current IO position as reported by tell(). Returns the new size.

writable()

Return whether object was opened for writing.

If False, read() will raise IOError.

writelines()
closed
class plastid.util.io.filters.TestTeeListener[source]

Bases: plastid.util.io.filters.TeeListener

Example of a TeeListener

Methods

alert  
alert(line)[source]

Process input from a TeeReader. Override this method to perform the appropriate behavior.

Parameters:
data : object

A unit of data

plastid.util.io.filters.colored(x, **kwargs)