Attributes

---

Parameter

A wrapper for pipeline parameters that can be loaded from and saved to YAML files. This class is meant to be used to create class-level attributes of your Pipe subclass (see example below). When used in this way, Parameter objects can be treated syntactically like the value that they contain.

Examples:

Example

class A(Pipe):
    p = Parameter('pname', default=0)

    def run(self) -> None:
        # `p` can be used like a regular class attribute. 
        print(self.p)
        self.p += 1
        print(self.p)

__init__(self, name, **kwargs) special

Parameters:

Name	Type	Description	Default
name	str	Identifier for the parameter. If loading the value from a YAML config file, this needs to match the corresponding field name.	required
default		(Optional[Any]) The default value that will be used for the parameter if no config file is specified. If not specified, the value of the parameter will be name unless overridden by a config file. The type of this value must be serializable to a YAML file -- it cannot be a function or a class. An error will be raised if the type is not allowed.	required

---

Product

A wrapper for the inputs/outputs of Pipe subclasses. The saving/loading behavior is altered by passing various flags to __init__.

By default, products passed into @produces will be pickled to the WARP cache location associated with the active session at runtime. Similarly, products passed into @dependencies will be unpickled

Custom saving/loading functions can be attached to a Product if pickling is not desired.

Note

The topology of the Graph is implicitly defined by passing Product instances to the @dependencies and @produces decorators in the Pipe declaration.

Tip

Product provides the custom syntax <<<, which is shorthand for assigning a value to the Product.value attribute.

Example

p = Product('p')
p << 0
p.value = 0  # equivalent

Info

Product contains the following class-level attributes that change the behavior of Workspace.build.

• __source__: The name of the pipe that produced this product.
• __static__: Specifies whether or not to cache the product statically.
• __external__: Specifies whether or not to cache the product externally.
• __save__: Specifies whether to cache the product or to leave it in working memory.

path: str property readonly

The correctly resolved path to the Product based on the state of the __external__ and __static__ attributes.

Warning

The correct WARP cache location for the current session must still be resolved, which can be done via path.format(warp.globals.product_dir()).

__init__(self, path, *, static=False, external=False, save=None, load=None) special

Parameters:

Name	Type	Description	Default
path	str	The relative path to save the product value to after the run() function of a Pipe subclass finishes. By default, this path will be appended to the path corresponding to the current WARP session directory in the WARP cache. Both static and external modify this behavior. If static=True, path will instead be appended to the static_products/ subdirectory of the WARP cache. If external=True, then the path will remain unaltered regardless of the static flag.	required
static	bool	Flag to mark this product as static, meaning that this product will be visible to other sessions. Unless external=True is specified, this will cause path to be appended to the static_products/ subdirectory of the WARP cache.	False
external	bool	Flag to mark this product as external, meaning that its path will left unmodified. Info This flag is used by the implementation of Source to allow the ingestion of external data artifacts into the pipeline.	False
save	Union[bool, function]	None: Cache the Product value by calling pickle. FunctionType: A user-defined callable implementing custom saving behavior. The function must have the signature (path: str, value: Any) -> None. The path argument to this function will always be the fully resolved file location. Passing a custom callable to save requires also passing a custom callable to load. Example The default save function is implemented in WARP as def save(path: str, value: Any) -> None: warp.utils.pickle(path, value) Warning This callable must create a file specified by path, otherwise WARP will detect this pipe as unbuilt and trigger rebuilds. False: Do not cache the product -- rather, leave it in working memory. Warning During backfill calls, products that are not cached in this manner will trigger rebuilds of the upstream pipes that produced them if said upstream pipes are direct ancestors of the target pipe.	None
load	Optional[function]	None: Call unpickle on the specified path. FunctionType: A user-defined callable implementing custom loading behavior. The callable must have the signature (path: str) -> Any. The path argument to this function will always be the fully resolved file location. Passing a custom callable to load requires also passing a custom callable to save. Example The default load function is implemented in WARP as def load(path: str) -> Any: return warp.utils.unpickle(path)	None

---

ParameterFile

A wrapper for declaring a config file to load Parameter values from. This config file should contain at least one field that corresponds to a parameter of the pipe in which this config is used.

This should be used as a class-level attribute of your Pipe subclass.

Examples:

class A(Pipe):
    # Must have a field `p_field` e.g. `p_field: 0`
    config = ParameterFile('../configs/A.yml')

    p = Parameter('p_field')

    def run(self) -> None:
        # The value from the config file. 
        # In this case, the output will be `p value 0`.
        print('p value: ', p)

__init__(self, path, multi_use=False) special

Parameters:

Name	Type	Description	Default
path	str	File path to a valid YAML file with fields matching Parameter names.	required
multi_use	bool	Flag to allow a config file to be used by more than just this pipe. In such cases, each pipe declaring this config file must use this flag.	False

---

AbstractPipeObject

TODO