Convolutional LSTM for spatial forecasting

This put up is the primary in a free collection exploring forecasting of spatially-determined knowledge over time. By spatially-determined I imply that regardless of the portions we’re attempting to foretell – be they univariate or multivariate time collection, of spatial dimensionality or not – the enter knowledge are given on a spatial grid.

For instance, the enter might be atmospheric measurements, comparable to sea floor temperature or strain, given at some set of latitudes and longitudes. The goal to be predicted might then span that very same (or one other) grid. Alternatively, it might be a univariate time collection, like a meteorological index.

However wait a second, it’s possible you’ll be pondering. For time-series prediction, we’ve that time-honored set of recurrent architectures (e.g., LSTM, GRU), proper? Proper. We do; however, as soon as we feed spatial knowledge to an RNN, treating completely different places as completely different enter options, we lose a necessary structural relationship. Importantly, we have to function in each house and time. We wish each: recurrence relations and convolutional filters. Enter convolutional RNNs.

What to anticipate from this put up

In the present day, we received’t soar into real-world purposes simply but. As a substitute, we’ll take our time to construct a convolutional LSTM (henceforth: convLSTM) in torch. For one, we’ve to – there is no such thing as a official PyTorch implementation.

What’s extra, this put up can function an introduction to constructing your personal modules. That is one thing it’s possible you’ll be aware of from Keras or not – relying on whether or not you’ve used customized fashions or fairly, most popular the declarative outline -> compile -> match fashion. (Sure, I’m implying there’s some switch happening if one involves torch from Keras customized coaching. Syntactic and semantic particulars could also be completely different, however each share the object-oriented fashion that permits for nice flexibility and management.)

Final however not least, we’ll additionally use this as a hands-on expertise with RNN architectures (the LSTM, particularly). Whereas the overall idea of recurrence could also be straightforward to know, it isn’t essentially self-evident how these architectures ought to, or might, be coded. Personally, I discover that unbiased of the framework used, RNN-related documentation leaves me confused. What precisely is being returned from calling an LSTM, or a GRU? (In Keras this is determined by the way you’ve outlined the layer in query.) I think that when we’ve determined what we need to return, the precise code received’t be that difficult. Consequently, we’ll take a detour clarifying what it’s that torch and Keras are giving us. Implementing our convLSTM will likely be much more simple thereafter.

A torch convLSTM

The code mentioned right here could also be discovered on GitHub. (Relying on once you’re studying this, the code in that repository might have developed although.)

My place to begin was one of many PyTorch implementations discovered on the web, specifically, this one. For those who seek for “PyTorch convGRU” or “PyTorch convLSTM”, one can find beautiful discrepancies in how these are realized – discrepancies not simply in syntax and/or engineering ambition, however on the semantic degree, proper on the heart of what the architectures could also be anticipated to do. As they are saying, let the customer beware. (Concerning the implementation I ended up porting, I’m assured that whereas quite a few optimizations will likely be attainable, the essential mechanism matches my expectations.)

What do I count on? Let’s method this job in a top-down manner.

Enter and output

The convLSTM’s enter will likely be a time collection of spatial knowledge, every statement being of dimension (time steps, channels, top, width).

Evaluate this with the same old RNN enter format, be it in torch or Keras. In each frameworks, RNNs count on tensors of dimension (timesteps, input_dim). input_dim is (1) for univariate time collection and larger than (1) for multivariate ones. Conceptually, we might match this to convLSTM’s channels dimension: There might be a single channel, for temperature, say – or there might be a number of, comparable to for strain, temperature, and humidity. The 2 extra dimensions present in convLSTM, top and width, are spatial indexes into the information.

In sum, we would like to have the ability to cross knowledge that:

• include a number of options,

• evolve in time, and

• are listed in two spatial dimensions.

How in regards to the output? We wish to have the ability to return forecasts for as many time steps as we’ve within the enter sequence. That is one thing that torch RNNs do by default, whereas Keras equivalents don’t. (It’s important to cross return_sequences = TRUE to acquire that impact.) If we’re keen on predictions for only a single cut-off date, we are able to at all times choose the final time step within the output tensor.

Nonetheless, with RNNs, it isn’t all about outputs. RNN architectures additionally carry by hidden states.

What are hidden states? I rigorously phrased that sentence to be as normal as attainable – intentionally circling across the confusion that, for my part, typically arises at this level. We’ll try to clear up a few of that confusion in a second, however let’s first end our high-level necessities specification.

We wish our convLSTM to be usable in numerous contexts and purposes. Varied architectures exist that make use of hidden states, most prominently maybe, encoder-decoder architectures. Thus, we would like our convLSTM to return these as effectively. Once more, that is one thing a torch LSTM does by default, whereas in Keras it’s achieved utilizing return_state = TRUE.

Now although, it truly is time for that interlude. We’ll kind out the methods issues are referred to as by each torch and Keras, and examine what you get again from their respective GRUs and LSTMs.

Interlude: Outputs, states, hidden values … what’s what?

For this to stay an interlude, I summarize findings on a excessive degree. The code snippets within the appendix present learn how to arrive at these outcomes. Closely commented, they probe return values from each Keras and torch GRUs and LSTMs. Working these will make the upcoming summaries appear loads much less summary.

First, let’s have a look at the methods you create an LSTM in each frameworks. (I’ll typically use LSTM because the “prototypical RNN instance”, and simply point out GRUs when there are variations vital within the context in query.)

In Keras, to create an LSTM it’s possible you’ll write one thing like this:

lstm <- layer_lstm(models = 1)

The torch equal can be:

lstm <- nn_lstm(
  input_size = 2, # variety of enter options
  hidden_size = 1 # variety of hidden (and output!) options
)

Don’t concentrate on torch‘s input_size parameter for this dialogue. (It’s the variety of options within the enter tensor.) The parallel happens between Keras’ models and torch’s hidden_size. For those who’ve been utilizing Keras, you’re in all probability pondering of models because the factor that determines output dimension (equivalently, the variety of options within the output). So when torch lets us arrive on the similar end result utilizing hidden_size, what does that imply? It signifies that one way or the other we’re specifying the identical factor, utilizing completely different terminology. And it does make sense, since at each time step present enter and former hidden state are added:

[
mathbf{h}_t = mathbf{W}_{x}mathbf{x}_t + mathbf{W}_{h}mathbf{h}_{t-1}
]

Now, about these hidden states.

When a Keras LSTM is outlined with return_state = TRUE, its return worth is a construction of three entities referred to as output, reminiscence state, and carry state. In torch, the identical entities are known as output, hidden state, and cell state. (In torch, we at all times get all of them.)

So are we coping with three various kinds of entities? We aren’t.

The cell, or carry state is that particular factor that units aside LSTMs from GRUs deemed liable for the “lengthy” in “lengthy short-term reminiscence”. Technically, it might be reported to the consumer in any respect time limits; as we’ll see shortly although, it isn’t.

What about outputs and hidden, or reminiscence states? Confusingly, these actually are the identical factor. Recall that for every enter merchandise within the enter sequence, we’re combining it with the earlier state, leading to a brand new state, to be made used of within the subsequent step:

[
mathbf{h}_t = mathbf{W}_{x}mathbf{x}_t + mathbf{W}_{h}mathbf{h}_{t-1}
]

Now, say that we’re keen on taking a look at simply the ultimate time step – that’s, the default output of a Keras LSTM. From that perspective, we are able to contemplate these intermediate computations as “hidden”. Seen like that, output and hidden states really feel completely different.

Nonetheless, we are able to additionally request to see the outputs for each time step. If we achieve this, there is no such thing as a distinction – the outputs (plural) equal the hidden states. This may be verified utilizing the code within the appendix.

Thus, of the three issues returned by an LSTM, two are actually the identical. How in regards to the GRU, then? As there is no such thing as a “cell state”, we actually have only one kind of factor left over – name it outputs or hidden states.

Let’s summarize this in a desk.

Now, about that public vs. non-public distinction. In each frameworks, we are able to get hold of outputs (hidden states) for each time step. The cell state, nevertheless, we are able to entry just for the final time step. That is purely an implementation choice. As we’ll see when constructing our personal recurrent module, there aren’t any obstacles inherent in protecting monitor of cell states and passing them again to the consumer.

For those who dislike the pragmatism of this distinction, you may at all times go along with the mathematics. When a brand new cell state has been computed (primarily based on prior cell state, enter, neglect, and cell gates – the specifics of which we aren’t going to get into right here), it’s reworked to the hidden (a.ok.a. output) state making use of yet one more, specifically, the output gate:

[
h_t = o_t odot tanh(c_t)
]

Undoubtedly, then, hidden state (output, resp.) builds on cell state, including extra modeling energy.

Now it’s time to get again to our unique purpose and construct that convLSTM. First although, let’s summarize the return values obtainable from torch and Keras.

convLSTM, the plan

In each torch and Keras RNN architectures, single time steps are processed by corresponding Cell courses: There’s an LSTM Cell matching the LSTM, a GRU Cell matching the GRU, and so forth. We do the identical for ConvLSTM. In convlstm_cell(), we first outline what ought to occur to a single statement; then in convlstm(), we construct up the recurrence logic.

As soon as we’re completed, we create a dummy dataset, as reduced-to-the-essentials as could be. With extra complicated datasets, even synthetic ones, likelihood is that if we don’t see any coaching progress, there are tons of of attainable explanations. We wish a sanity examine that, if failed, leaves no excuses. Lifelike purposes are left to future posts.

A single step: convlstm_cell

Our convlstm_cell’s constructor takes arguments input_dim , hidden_dim, and bias, similar to a torch LSTM Cell.

However we’re processing two-dimensional enter knowledge. As a substitute of the same old affine mixture of latest enter and former state, we use a convolution of kernel dimension kernel_size. Inside convlstm_cell, it’s self$conv that takes care of this.

Notice how the channels dimension, which within the unique enter knowledge would correspond to completely different variables, is creatively used to consolidate 4 convolutions into one: Every channel output will likely be handed to only one of many 4 cell gates. As soon as in possession of the convolution output, ahead() applies the gate logic, ensuing within the two varieties of states it must ship again to the caller.

library(torch)
library(zeallot)

convlstm_cell <- nn_module(
  
  initialize = perform(input_dim, hidden_dim, kernel_size, bias) {
    
    self$hidden_dim <- hidden_dim
    
    padding <- kernel_size %/% 2
    
    self$conv <- nn_conv2d(
      in_channels = input_dim + self$hidden_dim,
      # for every of enter, neglect, output, and cell gates
      out_channels = 4 * self$hidden_dim,
      kernel_size = kernel_size,
      padding = padding,
      bias = bias
    )
  },
  
  ahead = perform(x, prev_states) {

    c(h_prev, c_prev) %<-% prev_states
    
    mixed <- torch_cat(record(x, h_prev), dim = 2)  # concatenate alongside channel axis
    combined_conv <- self$conv(mixed)
    c(cc_i, cc_f, cc_o, cc_g) %<-% torch_split(combined_conv, self$hidden_dim, dim = 2)
    
    # enter, neglect, output, and cell gates (equivalent to torch's LSTM)
    i <- torch_sigmoid(cc_i)
    f <- torch_sigmoid(cc_f)
    o <- torch_sigmoid(cc_o)
    g <- torch_tanh(cc_g)
    
    # cell state
    c_next <- f * c_prev + i * g
    # hidden state
    h_next <- o * torch_tanh(c_next)
    
    record(h_next, c_next)
  },
  
  init_hidden = perform(batch_size, top, width) {
    
    record(
      torch_zeros(batch_size, self$hidden_dim, top, width, gadget = self$conv$weight$gadget),
      torch_zeros(batch_size, self$hidden_dim, top, width, gadget = self$conv$weight$gadget))
  }
)

Now convlstm_cell must be referred to as for each time step. That is completed by convlstm.

Iteration over time steps: convlstm

A convlstm might include a number of layers, similar to a torch LSTM. For every layer, we’re in a position to specify hidden and kernel sizes individually.

Throughout initialization, every layer will get its personal convlstm_cell. On name, convlstm executes two loops. The outer one iterates over layers. On the finish of every iteration, we retailer the ultimate pair (hidden state, cell state) for later reporting. The internal loop runs over enter sequences, calling convlstm_cell at every time step.

We additionally hold monitor of intermediate outputs, so we’ll have the ability to return the entire record of hidden_states seen in the course of the course of. In contrast to a torch LSTM, we do that for each layer.

convlstm <- nn_module(
  
  # hidden_dims and kernel_sizes are vectors, with one factor for every layer in n_layers
  initialize = perform(input_dim, hidden_dims, kernel_sizes, n_layers, bias = TRUE) {
 
    self$n_layers <- n_layers
    
    self$cell_list <- nn_module_list()
    
    for (i in 1:n_layers) {
      cur_input_dim <- if (i == 1) input_dim else hidden_dims[i - 1]
      self$cell_list$append(convlstm_cell(cur_input_dim, hidden_dims[i], kernel_sizes[i], bias))
    }
  },
  
  # we at all times assume batch-first
  ahead = perform(x) {
    
    c(batch_size, seq_len, num_channels, top, width) %<-% x$dimension()
   
    # initialize hidden states
    init_hidden <- vector(mode = "record", size = self$n_layers)
    for (i in 1:self$n_layers) {
      init_hidden[[i]] <- self$cell_list[[i]]$init_hidden(batch_size, top, width)
    }
    
    # record containing the outputs, of size seq_len, for every layer
    # this is similar as h, at every step within the sequence
    layer_output_list <- vector(mode = "record", size = self$n_layers)
    
    # record containing the final states (h, c) for every layer
    layer_state_list <- vector(mode = "record", size = self$n_layers)

    cur_layer_input <- x
    hidden_states <- init_hidden
    
    # loop over layers
    for (i in 1:self$n_layers) {
      
      # each layer's hidden state begins from 0 (non-stateful)
      c(h, c) %<-% hidden_states[[i]]
      # outputs, of size seq_len, for this layer
      # equivalently, record of h states for every time step
      output_sequence <- vector(mode = "record", size = seq_len)
      
      # loop over time steps
      for (t in 1:seq_len) {
        c(h, c) %<-% self$cell_list[[i]](cur_layer_input[ , t, , , ], record(h, c))
        # hold monitor of output (h) for each time step
        # h has dim (batch_size, hidden_size, top, width)
        output_sequence[[t]] <- h
      }

      # stack hs forever steps over seq_len dimension
      # stacked_outputs has dim (batch_size, seq_len, hidden_size, top, width)
      # similar as enter to ahead (x)
      stacked_outputs <- torch_stack(output_sequence, dim = 2)
      
      # cross the record of outputs (hs) to subsequent layer
      cur_layer_input <- stacked_outputs
      
      # hold monitor of record of outputs or this layer
      layer_output_list[[i]] <- stacked_outputs
      # hold monitor of final state for this layer
      layer_state_list[[i]] <- record(h, c)
    }
 
    record(layer_output_list, layer_state_list)
  }
    
)

Calling the convlstm

Let’s see the enter format anticipated by convlstm, and learn how to entry its completely different outputs.

Right here is an appropriate enter tensor.

# batch_size, seq_len, channels, top, width
x <- torch_rand(c(2, 4, 3, 16, 16))

First we make use of a single layer.

mannequin <- convlstm(input_dim = 3, hidden_dims = 5, kernel_sizes = 3, n_layers = 1)

c(layer_outputs, layer_last_states) %<-% mannequin(x)

We get again a listing of size two, which we instantly break up up into the 2 varieties of output returned: intermediate outputs from all layers, and remaining states (of each varieties) for the final layer.

With only a single layer, layer_outputs[[1]]holds all the layer’s intermediate outputs, stacked on dimension two.

dim(layer_outputs[[1]])
# [1]  2  4  5 16 16

layer_last_states[[1]]is a listing of tensors, the primary of which holds the only layer’s remaining hidden state, and the second, its remaining cell state.

dim(layer_last_states[[1]][[1]])
# [1]  2  5 16 16
dim(layer_last_states[[1]][[2]])
# [1]  2  5 16 16

For comparability, that is how return values search for a multi-layer structure.

mannequin <- convlstm(input_dim = 3, hidden_dims = c(5, 5, 1), kernel_sizes = rep(3, 3), n_layers = 3)
c(layer_outputs, layer_last_states) %<-% mannequin(x)

# for every layer, tensor of dimension (batch_size, seq_len, hidden_size, top, width)
dim(layer_outputs[[1]])
# 2  4  5 16 16
dim(layer_outputs[[3]])
# 2  4  1 16 16

# record of two tensors for every layer
str(layer_last_states)
# Checklist of three
#  $ :Checklist of two
#   ..$ :Float [1:2, 1:5, 1:16, 1:16]
#   ..$ :Float [1:2, 1:5, 1:16, 1:16]
#  $ :Checklist of two
#   ..$ :Float [1:2, 1:5, 1:16, 1:16]
#   ..$ :Float [1:2, 1:5, 1:16, 1:16]
#  $ :Checklist of two
#   ..$ :Float [1:2, 1:1, 1:16, 1:16]
#   ..$ :Float [1:2, 1:1, 1:16, 1:16]

# h, of dimension (batch_size, hidden_size, top, width)
dim(layer_last_states[[3]][[1]])
# 2  1 16 16

# c, of dimension (batch_size, hidden_size, top, width)
dim(layer_last_states[[3]][[2]])
# 2  1 16 16

Now we need to sanity-check this module with the simplest-possible dummy knowledge.

Sanity-checking the convlstm

We generate black-and-white “motion pictures” of diagonal beams successively translated in house.

Every sequence consists of six time steps, and every beam of six pixels. Only a single sequence is created manually. To create that one sequence, we begin from a single beam:

library(torchvision)

beams <- vector(mode = "record", size = 6)
beam <- torch_eye(6) %>% nnf_pad(c(6, 12, 12, 6)) # left, proper, high, backside
beams[[1]] <- beam

Utilizing torch_roll() , we create a sample the place this beam strikes up diagonally, and stack the person tensors alongside the timesteps dimension.

for (i in 2:6) {
  beams[[i]] <- torch_roll(beam, c(-(i-1),i-1), c(1, 2))
}

init_sequence <- torch_stack(beams, dim = 1)

That’s a single sequence. Due to torchvision::transform_random_affine(), we virtually effortlessly produce a dataset of 100 sequences. Transferring beams begin at random factors within the spatial body, however all of them share that upward-diagonal movement.

sequences <- vector(mode = "record", size = 100)
sequences[[1]] <- init_sequence

for (i in 2:100) {
  sequences[[i]] <- transform_random_affine(init_sequence, levels = 0, translate = c(0.5, 0.5))
}

enter <- torch_stack(sequences, dim = 1)

# add channels dimension
enter <- enter$unsqueeze(3)
dim(enter)
# [1] 100   6  1  24  24

That’s it for the uncooked knowledge. Now we nonetheless want a dataset and a dataloader. Of the six time steps, we use the primary 5 as enter and attempt to predict the final one.

dummy_ds <- dataset(
  
  initialize = perform(knowledge) {
    self$knowledge <- knowledge
  },
  
  .getitem = perform(i) {
    record(x = self$knowledge[i, 1:5, ..], y = self$knowledge[i, 6, ..])
  },
  
  .size = perform() {
    nrow(self$knowledge)
  }
)

ds <- dummy_ds(enter)
dl <- dataloader(ds, batch_size = 100)

Here’s a tiny-ish convLSTM, educated for movement prediction:

mannequin <- convlstm(input_dim = 1, hidden_dims = c(64, 1), kernel_sizes = c(3, 3), n_layers = 2)

optimizer <- optim_adam(mannequin$parameters)

num_epochs <- 100

for (epoch in 1:num_epochs) {
  
  mannequin$practice()
  batch_losses <- c()
  
  for (b in enumerate(dl)) {
    
    optimizer$zero_grad()
    
    # last-time-step output from final layer
    preds <- mannequin(b$x)[[2]][[2]][[1]]
  
    loss <- nnf_mse_loss(preds, b$y)
    batch_losses <- c(batch_losses, loss$merchandise())
    
    loss$backward()
    optimizer$step()
  }
  
  if (epoch %% 10 == 0)
    cat(sprintf("nEpoch %d, coaching loss:%3fn", epoch, imply(batch_losses)))
}

Epoch 10, coaching loss:0.008522

Epoch 20, coaching loss:0.008079

Epoch 30, coaching loss:0.006187

Epoch 40, coaching loss:0.003828

Epoch 50, coaching loss:0.002322

Epoch 60, coaching loss:0.001594

Epoch 70, coaching loss:0.001376

Epoch 80, coaching loss:0.001258

Epoch 90, coaching loss:0.001218

Epoch 100, coaching loss:0.001171

Loss decreases, however that in itself isn’t a assure the mannequin has discovered something. Has it? Let’s examine its forecast for the very first sequence and see.

For printing, I’m zooming in on the related area within the 24×24-pixel body. Right here is the bottom reality for time step six:

0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0
0  0  1  0  0  0  0  0  0  0
0  0  0  1  0  0  0  0  0  0
0  0  0  0  1  0  0  0  0  0
0  0  0  0  0  1  0  0  0  0
0  0  0  0  0  0  1  0  0  0
0  0  0  0  0  0  0  1  0  0
0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0

And right here is the forecast. This doesn’t look dangerous in any respect, given there was neither experimentation nor tuning concerned.

       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9] [,10]
 [1,]  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00     0
 [2,] -0.02  0.36  0.01  0.06  0.00  0.00  0.00  0.00  0.00     0
 [3,]  0.00 -0.01  0.71  0.01  0.06  0.00  0.00  0.00  0.00     0
 [4,] -0.01  0.04  0.00  0.75  0.01  0.06  0.00  0.00  0.00     0
 [5,]  0.00 -0.01 -0.01 -0.01  0.75  0.01  0.06  0.00  0.00     0
 [6,]  0.00  0.01  0.00 -0.07 -0.01  0.75  0.01  0.06  0.00     0
 [7,]  0.00  0.01 -0.01 -0.01 -0.07 -0.01  0.75  0.01  0.06     0
 [8,]  0.00  0.00  0.01  0.00  0.00 -0.01  0.00  0.71  0.00     0
 [9,]  0.00  0.00  0.00  0.01  0.01  0.00  0.03 -0.01  0.37     0
[10,]  0.00  0.00  0.00  0.00  0.00  0.00 -0.01 -0.01 -0.01     0

This could suffice for a sanity examine. For those who made it until the top, thanks on your endurance! In the very best case, you’ll have the ability to apply this structure (or the same one) to your personal knowledge – however even when not, I hope you’ve loved studying about torch mannequin coding and/or RNN weirdness 😉

I, for one, am definitely trying ahead to exploring convLSTMs on real-world issues within the close to future. Thanks for studying!

Appendix

This appendix incorporates the code used to create tables 1 and a pair of above.

Keras

LSTM

library(keras)

# batch of three, with 4 time steps every and a single function
enter <- k_random_normal(form = c(3L, 4L, 1L))
enter

# default args
# return form = (batch_size, models)
lstm <- layer_lstm(
  models = 1,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
lstm(enter)

# return_sequences = TRUE
# return form = (batch_size, time steps, models)
#
# word how for every merchandise within the batch, the worth for time step 4 equals that obtained above
lstm <- layer_lstm(
  models = 1,
  return_sequences = TRUE,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
  # bias is by default initialized to 0
)
lstm(enter)

# return_state = TRUE
# return form = record of:
#                - outputs, of form: (batch_size, models)
#                - "reminiscence states" for the final time step, of form: (batch_size, models)
#                - "carry states" for the final time step, of form: (batch_size, models)
#
# word how the primary and second record objects are similar!
lstm <- layer_lstm(
  models = 1,
  return_state = TRUE,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
lstm(enter)

# return_state = TRUE, return_sequences = TRUE
# return form = record of:
#                - outputs, of form: (batch_size, time steps, models)
#                - "reminiscence" states for the final time step, of form: (batch_size, models)
#                - "carry states" for the final time step, of form: (batch_size, models)
#
# word how once more, the "reminiscence" state present in record merchandise 2 matches the final-time step outputs reported in merchandise 1
lstm <- layer_lstm(
  models = 1,
  return_sequences = TRUE,
  return_state = TRUE,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
lstm(enter)

GRU

# default args
# return form = (batch_size, models)
gru <- layer_gru(
  models = 1,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
gru(enter)

# return_sequences = TRUE
# return form = (batch_size, time steps, models)
#
# word how for every merchandise within the batch, the worth for time step 4 equals that obtained above
gru <- layer_gru(
  models = 1,
  return_sequences = TRUE,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
gru(enter)

# return_state = TRUE
# return form = record of:
#    - outputs, of form: (batch_size, models)
#    - "reminiscence" states for the final time step, of form: (batch_size, models)
#
# word how the record objects are similar!
gru <- layer_gru(
  models = 1,
  return_state = TRUE,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
gru(enter)

# return_state = TRUE, return_sequences = TRUE
# return form = record of:
#    - outputs, of form: (batch_size, time steps, models)
#    - "reminiscence states" for the final time step, of form: (batch_size, models)
#
# word how once more, the "reminiscence state" present in record merchandise 2 matches the final-time-step outputs reported in merchandise 1
gru <- layer_gru(
  models = 1,
  return_sequences = TRUE,
  return_state = TRUE,
  kernel_initializer = initializer_constant(worth = 1),
  recurrent_initializer = initializer_constant(worth = 1)
)
gru(enter)

torch

LSTM (non-stacked structure)

library(torch)

# batch of three, with 4 time steps every and a single function
# we'll specify batch_first = TRUE when creating the LSTM
enter <- torch_randn(c(3, 4, 1))
enter

# default args
# return form = (batch_size, models)
#
# word: there's an extra argument num_layers that we might use to specify a stacked LSTM - successfully composing two LSTM modules
# default for num_layers is 1 although 
lstm <- nn_lstm(
  input_size = 1, # variety of enter options
  hidden_size = 1, # variety of hidden (and output!) options
  batch_first = TRUE # for simple comparability with Keras
)

nn_init_constant_(lstm$weight_ih_l1, 1)
nn_init_constant_(lstm$weight_hh_l1, 1)
nn_init_constant_(lstm$bias_ih_l1, 0)
nn_init_constant_(lstm$bias_hh_l1, 0)

# returns a listing of size 2, specifically
#   - outputs, of form (batch_size, time steps, hidden_size) - given we specified batch_first
#       Notice 1: If it is a stacked LSTM, these are the outputs from the final layer solely.
#               For our present function, that is irrelevant, as we're proscribing ourselves to single-layer LSTMs.
#       Notice 2: hidden_size right here is equal to models in Keras - each specify variety of options
#  - record of:
#    - hidden state for the final time step, of form (num_layers, batch_size, hidden_size)
#    - cell state for the final time step, of form (num_layers, batch_size, hidden_size)
#      Notice 3: For a single-layer LSTM, the hidden states are already offered within the first record merchandise.

lstm(enter)

GRU (non-stacked structure)

# default args
# return form = (batch_size, models)
#
# word: there's an extra argument num_layers that we might use to specify a stacked GRU - successfully composing two GRU modules
# default for num_layers is 1 although 
gru <- nn_gru(
  input_size = 1, # variety of enter options
  hidden_size = 1, # variety of hidden (and output!) options
  batch_first = TRUE # for simple comparability with Keras
)

nn_init_constant_(gru$weight_ih_l1, 1)
nn_init_constant_(gru$weight_hh_l1, 1)
nn_init_constant_(gru$bias_ih_l1, 0)
nn_init_constant_(gru$bias_hh_l1, 0)

# returns a listing of size 2, specifically
#   - outputs, of form (batch_size, time steps, hidden_size) - given we specified batch_first
#       Notice 1: If it is a stacked GRU, these are the outputs from the final layer solely.
#               For our present function, that is irrelevant, as we're proscribing ourselves to single-layer GRUs.
#       Notice 2: hidden_size right here is equal to models in Keras - each specify variety of options
#  - record of:
#    - hidden state for the final time step, of form (num_layers, batch_size, hidden_size)
#    - cell state for the final time step, of form (num_layers, batch_size, hidden_size)
#       Notice 3: For a single-layer GRU, these values are already offered within the first record merchandise.
gru(enter)