Module flowcon.nn

Sub-modules

flowcon.nn.nde
flowcon.nn.nets

Classes

class CLipSwish

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class CLipSwish(nn.Module):

    def __init__(self):
        super(CLipSwish, self).__init__()
        self.swish = Swish()
        self._does_concat = True

    def forward(self, x):
        x = torch.cat((x, -x), 1)
        return self.swish(x).div_(1.004)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class CSin (w0=1)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class CSin(nn.Module):
    def __init__(self, w0=1):
        super(CSin, self).__init__()
        self.w0 = w0
        self._does_concat = True

    def forward(self, x):
        x = torch.cat((x, -x), 1)
        return torch.sin(x * self.w0) / (self.w0 * math.sqrt(2))

    def build_clone(self):
        return copy.deepcopy(self)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def build_clone(self)
def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class ConvResidualNet (in_channels, out_channels, hidden_channels, context_channels=None, num_blocks=2, activation=<function relu>, dropout_probability=0.0, use_batch_norm=False)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class ConvResidualNet(nn.Module):
    def __init__(
        self,
        in_channels,
        out_channels,
        hidden_channels,
        context_channels=None,
        num_blocks=2,
        activation=F.relu,
        dropout_probability=0.0,
        use_batch_norm=False,
    ):
        super().__init__()
        self.context_channels = context_channels
        self.hidden_channels = hidden_channels
        if context_channels is not None:
            self.initial_layer = nn.Conv2d(
                in_channels=in_channels + context_channels,
                out_channels=hidden_channels,
                kernel_size=1,
                padding=0,
            )
        else:
            self.initial_layer = nn.Conv2d(
                in_channels=in_channels,
                out_channels=hidden_channels,
                kernel_size=1,
                padding=0,
            )
        self.blocks = nn.ModuleList(
            [
                ConvResidualBlock(
                    channels=hidden_channels,
                    context_channels=context_channels,
                    activation=activation,
                    dropout_probability=dropout_probability,
                    use_batch_norm=use_batch_norm,
                )
                for _ in range(num_blocks)
            ]
        )
        self.final_layer = nn.Conv2d(
            hidden_channels, out_channels, kernel_size=1, padding=0
        )

    def forward(self, inputs, context=None):
        if context is None:
            temps = self.initial_layer(inputs)
        else:
            temps = self.initial_layer(torch.cat((inputs, context), dim=1))
        for block in self.blocks:
            temps = block(temps, context)
        outputs = self.final_layer(temps)
        return outputs

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, inputs, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class DenseNet (dimension, densenet_depth: int = 2, densenet_growth: int = 16, activation_function: Union[str, Callable] = 'CLipSwish', lip_coeff: float = 0.98, n_lipschitz_iters: int = 5, **kwargs)

Provides a Lipschitz contiuous network g(x) with a fixed lipschitz constant.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class DenseNet(_DenseNet):
    """
    Provides a Lipschitz contiuous network g(x)  with a fixed lipschitz constant.
    """

    def __init__(self,
                 dimension,
                 densenet_depth: int = 2,
                 densenet_growth: int = 16,
                 activation_function: Union[str, Callable] = "CLipSwish",
                 lip_coeff: float = 0.98,
                 n_lipschitz_iters: int = 5,
                 **kwargs):
        super().__init__(dimension=dimension,
                         densenet_depth=densenet_depth,
                         densenet_growth=densenet_growth,
                         activation_function=activation_function,
                         lip_coeff=lip_coeff,
                         n_lipschitz_iters=n_lipschitz_iters)

        if len(kwargs) > 0:
            logger.warning("Unused kwargs for {}: {}".format(self.__class__.__name__, pformat(kwargs)))
        self.dense_net, self.densenet_final_layer_dim = self.build_densenet(self.dimension,
                                                                            self.densenet_depth,
                                                                            self.densenet_growth)

    def forward(self, x, context=None):
        assert context is None, "Context not supported for this Class."
        return self.dense_net(x)

Ancestors

  • flowcon.nn.nets.invertible_densenet._DenseNet
  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class FCBlock (in_shape, out_shape, hidden_sizes, activation='tanh', activate_output=False, **kwargs)

Fully Connected Block, that also supports sine activations (they need a specific initialization)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class FCBlock(torch.nn.Module):
    """
    Fully Connected Block, that also supports sine activations (they need a specific initialization)
    """

    def __init__(self,
                 in_shape,
                 out_shape,
                 hidden_sizes,
                 activation="tanh",
                 activate_output=False,
                 **kwargs):
        super().__init__()

        self._in_shape = torch.Size(in_shape)
        self._in_prod = self._in_shape.numel()
        self._out_shape = torch.Size(out_shape)
        self._out_prod = self._out_shape.numel()
        self._hidden_sizes = hidden_sizes
        self._activation = activation
        self._activate_output = activate_output

        if len(hidden_sizes) == 0:
            raise ValueError("List of hidden sizes can't be empty.")

        nls_and_inits = {'sine': (Sine(kwargs.get("sine_frequency", 7)),
                                  gen_sine_init(kwargs.get("sine_frequency", 7)),
                                  first_layer_sine_init),
                         'relu': (nn.ReLU(inplace=True), init_weights_normal, None),
                         'sigmoid': (nn.Sigmoid(), init_weights_xavier, None),
                         'tanh': (nn.Tanh(), init_weights_xavier, None),
                         'selu': (nn.SELU(inplace=True), init_weights_selu, None),
                         'softplus': (nn.Softplus(), init_weights_normal, None),
                         'elu': (nn.ELU(inplace=True), init_weights_elu, None)}

        nl, self.weight_init, first_layer_init = nls_and_inits[activation]

        net = self.build_net(hidden_sizes, in_shape, nl, out_shape)
        self.net = torch.nn.Sequential(*net)
        self.initialize_weights(first_layer_init)

    def build_net(self, hidden_sizes, in_shape, nl, out_shape):
        net = []
        net.append(nn.Linear(self._in_prod, hidden_sizes[0]))
        for in_size, out_size in zip(hidden_sizes[:-1], hidden_sizes[1:]):
            net.append(nl)
            net.append(nn.Linear(in_size, out_size))
        net.append(nl)
        net.append(nn.Linear(hidden_sizes[-1], self._out_prod))
        if self._activate_output:
            net.append(nl)
        return net

    def initialize_weights(self, first_layer_init):
        self.net.apply(self.weight_init)
        if first_layer_init is not None:  # Apply special initialization to first layer, if applicable.
            self.net[0].apply(first_layer_init)

    def forward(self, inputs):
        return self.net(inputs.view(-1, self._in_prod)).view(-1, *self._out_shape)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def build_net(self, hidden_sizes, in_shape, nl, out_shape)
def forward(self, inputs) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

def initialize_weights(self, first_layer_init)
class FullSort (*args, **kwargs)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class FullSort(nn.Module):

    def forward(self, x):
        return torch.sort(x, 1)[0]

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class InputConditionalDenseNet (dimension, context_features, densenet_depth, densenet_growth=16, c_embed_hidden_sizes=(128, 128, 10), activation_function=flowcon.nn.nets.activations.Swish, lip_coeff=0.98, n_lipschitz_iters=5, **kwargs)

Provides a Lipschitz contiuous network g(x;t) = h(concat[x, f(c)]) with a fixed lipschitz constant. The network h(x;t) is a DenseNet and Lipschitz Continuous w.r.t. x. the network f provides an embedding of the context.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class InputConditionalDenseNet(_DenseNet):
    """
    Provides a Lipschitz contiuous network g(x;t) = h(concat[x, f(c)]) with a fixed lipschitz constant.
    The network h(x;t) is a DenseNet and Lipschitz Continuous w.r.t. x.
    the network f provides an embedding of the context.
    """

    def __init__(self, dimension, context_features,
                 densenet_depth,
                 densenet_growth=16,
                 c_embed_hidden_sizes=(128, 128, 10),
                 activation_function=activations.Swish,
                 lip_coeff=0.98,
                 n_lipschitz_iters=5,
                 **kwargs
                 ):
        super().__init__(dimension=dimension,
                         densenet_depth=densenet_depth,
                         densenet_growth=densenet_growth,
                         activation_function=activation_function,
                         lip_coeff=lip_coeff,
                         n_lipschitz_iters=n_lipschitz_iters)
        if len(kwargs) > 0:
            logger.warning("Unused kwargs for class '{}': \n {}".format(self.__class__.__name__, pformat(kwargs)))

        self.context_features = context_features
        self.c_embed_hidden_sizes = c_embed_hidden_sizes

        self.bn = torch.nn.BatchNorm1d(self.context_features)
        self.dense_net, self.densenet_final_layer_dim = self.build_densenet(
            total_in_channels=self.dimension + self.c_embed_hidden_sizes[-1],
            densenet_growth=densenet_growth,
            densenet_depth=densenet_depth,
            include_last_layer=True)

        self.context_embedding_net = MLP((self.context_features,), (self.c_embed_hidden_sizes[-1],),
                                         hidden_sizes=self.c_embed_hidden_sizes,
                                         activation=torch.nn.SiLU())

    def forward(self, inputs, context=None):
        context = self.bn(context)
        context_embedding = self.context_embedding_net(context)
        concat_inputs = torch.cat([inputs, context_embedding], -1)
        outputs = self.dense_net(concat_inputs)
        return outputs

Ancestors

  • flowcon.nn.nets.invertible_densenet._DenseNet
  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, inputs, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class LastLayerConditionalDenseNet (dimension, context_features, densenet_depth, densenet_growth=16, last_layer_hidden_sizes=(64, 64), activation_function=flowcon.nn.nets.activations.Swish, lip_coeff=0.98, n_lipschitz_iters=5, **kwargs)

Provides a Lipschitz contiuous network g(x;c) = h(x;t) with a fixed lipschitz constant. The network h(x;c) is a DenseNet. Let xϵR^d. For a unconditional DenseNet the last layer is given by Az, where zϵR^k with k>d is a higher dimensional embedding and AϵR^(d imes k) is a matrix with spectral norm <1. For making this last layer conditional, we instead parameterize A(c) with a hypernetwork. We further pass each row through a softmax (making A row stochastic), such that the Lipschitz constant of remains unchanged.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class LastLayerConditionalDenseNet(_DenseNet):
    """
    Provides a Lipschitz contiuous network g(x;c) = h(x;t) with a fixed lipschitz constant.
    The network h(x;c) is a DenseNet.
    Let xϵR^d.
    For a unconditional DenseNet the last layer is given by Az, where zϵR^k with k>d is a
    higher dimensional embedding and AϵR^(d\times k) is a matrix with spectral norm <1.
    For making this last layer conditional, we instead parameterize A(c) with a hypernetwork.
    We further pass each row through a softmax (making A row stochastic), such that the Lipschitz constant of remains unchanged.
    """

    def __init__(self, dimension, context_features,
                 densenet_depth,
                 densenet_growth=16,
                 last_layer_hidden_sizes=(64, 64),
                 activation_function=activations.Swish,
                 lip_coeff=0.98,
                 n_lipschitz_iters=5,
                 **kwargs
                 ):
        super().__init__(dimension=dimension,
                         densenet_depth=densenet_depth,
                         densenet_growth=densenet_growth,
                         activation_function=activation_function,
                         lip_coeff=lip_coeff,
                         n_lipschitz_iters=n_lipschitz_iters)
        if len(kwargs) > 0:
            logger.warning("Unused kwargs for class '{}': \n {}".format(self.__class__.__name__, pformat(kwargs)))

        self.context_features = context_features
        self.last_layer_hidden_sizes = last_layer_hidden_sizes

        self.bn = torch.nn.BatchNorm1d(self.context_features)
        self.dense_net, self.densenet_final_layer_dim = self.build_densenet(total_in_channels=self.dimension,
                                                                            densenet_growth=densenet_growth,
                                                                            densenet_depth=densenet_depth,
                                                                            include_last_layer=False)

        self.custom_attention = LastLayerAttention(dimension=self.dimension,
                                                   context_features=self.context_features,
                                                   value_dim=self.densenet_final_layer_dim,
                                                   hidden_sizes=self.last_layer_hidden_sizes)

    def forward(self, inputs, context=None):
        context = self.bn(context)
        values_weights = self.dense_net(inputs).unsqueeze(-1)
        weights = self.custom_attention.attention(context, values_weights)
        return weights

Ancestors

  • flowcon.nn.nets.invertible_densenet._DenseNet
  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, inputs, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class LeakyLSwish

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class LeakyLSwish(nn.Module):

    def __init__(self):
        super(LeakyLSwish, self).__init__()
        self.alpha = nn.Parameter(torch.tensor([-3.]))
        self.beta = nn.Parameter(torch.tensor([0.5]))

    def forward(self, x):
        alpha = torch.sigmoid(self.alpha)
        return alpha * x + (1 - alpha) * (x * torch.sigmoid_(x * F.softplus(self.beta))).div_(1.1)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class LipSwish

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class LipSwish(nn.Module):

    def __init__(self):
        super(LipSwish, self).__init__()
        self.swish = Swish()

    def forward(self, x):
        return self.swish(x).div_(1.004)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class LipschitzCube (*args, **kwargs)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class LipschitzCube(nn.Module):

    def forward(self, x):
        return (x >= 1).to(x) * (x - 2 / 3) + (x <= -1).to(x) * (x + 2 / 3) + ((x > -1) * (x < 1)).to(x) * x ** 3 / 3

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class MLP (in_shape, out_shape, hidden_sizes, activation=ReLU(), activate_output=False)

A standard multi-layer perceptron.

Args

in_shape
tuple, list or torch.Size, the shape of the input.
out_shape
tuple, list or torch.Size, the shape of the output.
hidden_sizes
iterable of ints, the hidden-layer sizes.
activation
callable, the activation function.
activate_output
bool, whether to apply the activation to the output.
Expand source code 
class MLP(nn.Module):
    """A standard multi-layer perceptron."""

    def __init__(
            self,
            in_shape,
            out_shape,
            hidden_sizes,
            activation=torch.nn.ReLU(),
            activate_output=False,
    ):
        """
        Args:
            in_shape: tuple, list or torch.Size, the shape of the input.
            out_shape: tuple, list or torch.Size, the shape of the output.
            hidden_sizes: iterable of ints, the hidden-layer sizes.
            activation: callable, the activation function.
            activate_output: bool, whether to apply the activation to the output.
        """
        super().__init__()
        self._in_shape = torch.Size(in_shape)
        self._in_prod = self._in_shape.numel()
        self._out_shape = torch.Size(out_shape)
        self._hidden_sizes = hidden_sizes
        self._activation = activation
        self._activate_output = activate_output

        if len(hidden_sizes) == 0:
            raise ValueError("List of hidden sizes can't be empty.")

        net = self.build_net(hidden_sizes, in_shape, nl=activation, out_shape=out_shape)
        self.net = torch.nn.Sequential(*net)

        # self.net = torch.jit.script(self.net)

    def build_net(self, hidden_sizes, in_shape, nl, out_shape):
        net = []
        net.append(nn.Linear(self._in_shape.numel(), hidden_sizes[0]))
        for in_size, out_size in zip(hidden_sizes[:-1], hidden_sizes[1:]):
            net.append(nl)
            net.append(nn.Linear(in_size, out_size))
        net.append(nl)
        net.append(nn.Linear(hidden_sizes[-1], self._out_shape.numel()))
        if self._activate_output:
            net.append(nl)
        return net

    def initialize_weights(self, first_layer_init):
        self.net.apply(self.weight_init)
        if first_layer_init is not None:  # Apply special initialization to first layer, if applicable.
            self.net[0].apply(first_layer_init)

    def forward(self, inputs):
        return self.net(inputs.view(-1, self._in_prod)).view(-1, *self._out_shape)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def build_net(self, hidden_sizes, in_shape, nl, out_shape)
def forward(self, inputs) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

def initialize_weights(self, first_layer_init)
class MaxMin (*args, **kwargs)

Module that computes max and min values of input tensor.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class MaxMin(nn.Module):
    """
    Module that computes max and min values of input tensor.
    """
    def forward(self, x):
        b, d = x.shape
        max_vals = torch.max(x.view(b, d // 2, 2), 2)[0]
        min_vals = torch.min(x.view(b, d // 2, 2), 2)[0]
        return torch.cat([max_vals, min_vals], 1)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class MixedConditionalDenseNet (dimension, context_features, densenet_depth, densenet_growth=16, last_layer_hidden_sizes=(64, 64), c_embed_hidden_sizes=(32, 32, 10), activation_function=flowcon.nn.nets.activations.Swish, lip_coeff=0.98, n_lipschitz_iters=5, **kwargs)

Combination of LastLayerConditionalDenseNet and InputConditionalDenseNet, i.e. both the first and last layer of the densenet are conditional.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class MixedConditionalDenseNet(_DenseNet):
    """
    Combination of LastLayerConditionalDenseNet and InputConditionalDenseNet,
    i.e. both the first and last layer of the densenet are conditional.
    """

    def __init__(self, dimension, context_features,
                 densenet_depth,
                 densenet_growth=16,
                 last_layer_hidden_sizes=(64, 64),
                 c_embed_hidden_sizes=(32, 32, 10),
                 activation_function=activations.Swish,
                 lip_coeff=0.98,
                 n_lipschitz_iters=5,
                 **kwargs
                 ):
        super().__init__(dimension=dimension,
                         densenet_depth=densenet_depth,
                         densenet_growth=densenet_growth,
                         activation_function=activation_function,
                         lip_coeff=lip_coeff,
                         n_lipschitz_iters=n_lipschitz_iters)

        if len(kwargs) > 0:
            logger.warning("Unused kwargs for class '{}': \n {}".format(self.__class__.__name__, pformat(kwargs)))

        self.context_features = context_features
        self.c_embed_hidden_sizes = c_embed_hidden_sizes
        self.last_layer_hidden_sizes = last_layer_hidden_sizes

        self.output_channels = self.calc_output_channels(self.activation,
                                                         self.densenet_growth)
        self.bn = torch.nn.BatchNorm1d(self.context_features)
        self.dense_net, self.densenet_final_layer_dim = self.build_densenet(
            total_in_channels=self.dimension + self.c_embed_hidden_sizes[-1],
            densenet_growth=densenet_growth,
            densenet_depth=densenet_depth,
            include_last_layer=False)

        self.custom_attention = LastLayerAttention(dimension=self.dimension,
                                                   context_features=self.context_features,
                                                   value_dim=self.densenet_final_layer_dim,
                                                   hidden_sizes=self.last_layer_hidden_sizes)

        self.context_embedding_net = MLP((self.context_features,), (self.c_embed_hidden_sizes[-1],),
                                         hidden_sizes=self.c_embed_hidden_sizes,
                                         activation=torch.nn.SiLU())

    def forward(self, inputs, context=None):
        context = self.bn(context)
        context_embedding = self.context_embedding_net(context)
        concat_inputs = torch.cat([inputs, context_embedding], -1)
        values_weights = self.dense_net(concat_inputs).unsqueeze(-1)
        weights = self.custom_attention.attention(context, values_weights)
        return weights

Ancestors

  • flowcon.nn.nets.invertible_densenet._DenseNet
  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, inputs, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class MultiplicativeAndInputConditionalDenseNet (dimension, context_features, densenet_depth, densenet_growth=16, c_embed_hidden_sizes=(128, 128, 10), m_embed_hidden_sizes=(128, 128), activation_function=flowcon.nn.nets.activations.Swish, lip_coeff=0.98, n_lipschitz_iters=5, **kwargs)

Provides a Lipschitz contiuous network g(x;c) = φ(c) h(x;t) with a fixed lipschitz constant. The network h(x;c) is an InputConditionalDenseNet and Lipschitz Continuous w.r.t. x, and φ(c) \in (0,1), making g Lipschitz Continuous w.r.t. x.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class MultiplicativeAndInputConditionalDenseNet(_DenseNet):
    """
    Provides a Lipschitz contiuous network g(x;c) = φ(c) h(x;t) with a fixed lipschitz constant.
    The network h(x;c) is an InputConditionalDenseNet and Lipschitz Continuous w.r.t. x, and φ(c) \in (0,1), making g Lipschitz Continuous w.r.t. x.
    """

    def __init__(self, dimension, context_features,
                 densenet_depth,
                 densenet_growth=16,
                 c_embed_hidden_sizes=(128, 128, 10),
                 m_embed_hidden_sizes=(128, 128),
                 activation_function=activations.Swish,
                 lip_coeff=0.98,
                 n_lipschitz_iters=5,
                 **kwargs
                 ):
        super().__init__(dimension=dimension,
                         densenet_depth=densenet_depth,
                         densenet_growth=densenet_growth,
                         activation_function=activation_function,
                         lip_coeff=lip_coeff,
                         n_lipschitz_iters=n_lipschitz_iters)
        if len(kwargs) > 0:
            logger.warning("Unused kwargs for class '{}': \n {}".format(self.__class__.__name__, pformat(kwargs)))

        self.context_features = context_features
        self.c_embed_hidden_sizes = c_embed_hidden_sizes
        self.m_embed_hidden_sizes = m_embed_hidden_sizes

        self.bn = torch.nn.BatchNorm1d(self.context_features)
        self.dense_net, self.densenet_final_layer_dim = self.build_densenet(
            total_in_channels=self.dimension + self.c_embed_hidden_sizes[-1],
            densenet_growth=densenet_growth,
            densenet_depth=densenet_depth,
            include_last_layer=True)

        self.factor_net = MLP((self.context_features,), (1,),
                              hidden_sizes=self.m_embed_hidden_sizes,
                              activation=torch.nn.SiLU())

        self.embedding = MLP((self.context_features,), (self.c_embed_hidden_sizes[-1],),
                             hidden_sizes=self.c_embed_hidden_sizes,
                             activation=torch.nn.SiLU())

    def forward(self, inputs, context=None):
        context = self.bn(context)
        factor = self.factor_net(context)
        context_embedding = self.embedding(context)
        concat_inputs = torch.cat([inputs, context_embedding], -1)
        outputs = torch.nn.functional.tanh(factor) * self.dense_net(concat_inputs)
        return outputs

Ancestors

  • flowcon.nn.nets.invertible_densenet._DenseNet
  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, inputs, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class ResidualNet (in_features, out_features, hidden_features, context_features=None, num_blocks=2, activation=ReLU(), dropout_probability=0.0, use_batch_norm=False)

A general-purpose residual network. Works only with 1-dim inputs.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class ResidualNet(nn.Module):
    """A general-purpose residual network. Works only with 1-dim inputs."""

    def __init__(
        self,
        in_features,
        out_features,
        hidden_features,
        context_features=None,
        num_blocks=2,
        activation=torch.nn.ReLU(),
        dropout_probability=0.0,
        use_batch_norm=False,
    ):
        super().__init__()
        self.hidden_features = hidden_features
        self.context_features = context_features
        if context_features is not None:
            self.initial_layer = nn.Linear(
                in_features + context_features, hidden_features
            )
        else:
            self.initial_layer = nn.Linear(in_features, hidden_features)
        self.blocks = nn.ModuleList(
            [
                ResidualBlock(
                    features=hidden_features,
                    context_features=context_features,
                    activation=activation,
                    dropout_probability=dropout_probability,
                    use_batch_norm=use_batch_norm,
                )
                for _ in range(num_blocks)
            ]
        )
        self.final_layer = nn.Linear(hidden_features, out_features)

    def forward(self, inputs, context=None):
        if context is None:
            temps = self.initial_layer(inputs)
        else:
            temps = self.initial_layer(torch.cat((inputs, context), dim=1))
        for block in self.blocks:
            temps = block(temps, context=context)
        outputs = self.final_layer(temps)
        return outputs

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, inputs, context=None) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class Sin (w0=1)

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class Sin(nn.Module):
    def __init__(self, w0=1):
        super(Sin, self).__init__()
        self.w0 = w0

    def forward(self, x):
        return torch.sin(x * self.w0) / self.w0

    def build_clone(self):
        return copy.deepcopy(self)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def build_clone(self)
def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class Swish

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes::

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call :meth:to, etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

:ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Expand source code 
class Swish(nn.Module):

    def __init__(self):
        super(Swish, self).__init__()
        self.beta = nn.Parameter(torch.tensor([0.5]))

    def forward(self, x):
        return (x * torch.sigmoid_(x * F.softplus(self.beta))).div_(1.1)

Ancestors

  • torch.nn.modules.module.Module

Class variables

var call_super_init : bool
var dump_patches : bool
var training : bool

Methods

def forward(self, x) ‑> Callable[..., Any]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the :class:Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.