DL4S 0.1.4

DL4S 0.1.4

×

Maintained by Palle Klewitz.

DL4S 0.1.4

  • By
  • Palle Klewitz

DL4S

CocoaPods CocoaPods license

This framework contains an implementation of reverse mode automatic differentiation, vectorized implementations of common matrix and vector operators and high level neural network operations, such as convolution, recurrent units, and more.

Overview

  1. Installation
  2. Features
    1. Layers
    2. Optimizers
    3. Losses
    4. Tensor Operations
    5. Engines
    6. Architectures
  3. Examples
    1. Convolutional Networks
    2. Recurrent Network (LSTM)
    3. Generative Adversarial Network

Installation

CocoaPods

target 'Your-App-Name' do
    use_frameworks!
    pod 'DL4S', '~> 0.1.0'
end

Swift Package Manager

Add the dependency to your Package.swift file:

.package(url: "https://github.com/palle-k/DL4S.git", .branch("master"))

Then add DL4S as a dependency to your target:

.target(name: "MyPackage", dependencies: ["DL4S"])

Features

Layers

  • Convolution
  • Dense/Linear/Fully Connected
  • LSTM
  • Gated Recurrent Unit (GRU)
  • Vanilla RNN
  • Bidirectional RNNs
  • Max Pooling
  • Average Pooling
  • Relu
  • Tanh
  • Sigmoid
  • Softmax
  • Embedding
  • Batch Norm
  • Layer Norm
  • Lambda
  • Sequential
  • Dropout

Optimizers

  • SGD
  • Momentum
  • Adam
  • AdaGrad
  • AdaDelta
  • RMSProp

Losses

  • Binary Cross-Entropy
  • Categorical Cross-Entropy
  • MSE
  • L1 & L2 regularization

Tensor Operations

  • broadcast-add
  • broadcast-sub
  • broadcast-mul
  • broadcast-div
  • matmul
  • neg
  • exp
  • pow
  • log
  • sqrt
  • sin
  • cos
  • tan
  • tanh
  • l1 norm
  • l2 norm
  • sum
  • max
  • relu
  • leaky relu
  • reduce sum
  • reduce max
  • conv2d
  • max pool
  • avg pool
  • subscript
  • subscript range
  • transpose
  • axis permute
  • reverse
  • im2col
  • col2im
  • stack / concat

Engines

  • CPU (Accelerate framework)
  • GPU (Metal)

Architectures

Default implementations are provided for the following architectures:

  • ResNet (currently only ResNet-18)
  • VGG
  • AlexNet

Examples

Arithmetic & Differentiation

let a = Tensor<Float, CPU>([[1,2],[3,4],[5,6]], requiresGradient: true)
let prod = mmul(a.T, a)
let s = sum(prod)
let l = log(s)
print(l) // 5.1873856


// Backpropagate
l.backwards()

print(a.gradientDescription!)
/*
[[0.03351955, 0.03351955],
 [0.07821229, 0.07821229],
 [0.12290502, 0.12290502]]
*/

Convolutional Networks

Example for MNIST classification

// Input must be 1x28x28
let model = Sequential<Float, CPU>(
    Conv2D(inputChannels: 1, outputChannels: 6, kernelSize: 5, padding: 0).asAny(), // 4x24x24
    Relu().asAny(),
    MaxPool2D(windowSize: 2, stride: 2).asAny(), // 4x12x12
    Conv2D(inputChannels: 6, outputChannels: 16, kernelSize: 5, padding: 0).asAny(), // 16x8x8
    Relu().asAny(),
    MaxPool2D(windowSize: 2, stride: 2).asAny(), // 16x4x4
    Flatten().asAny(), // 256
    Dense(inputFeatures: 256, outputFeatures: 120).asAny(),
    Relu().asAny(),
    Dense(inputFeatures: 120, outputFeatures: 10).asAny(),
    Softmax().asAny()
)

let optimizer = Adam(parameters: model.trainableParameters, learningRate: 0.001)

// Single iteration of minibatch gradient descent
optimizer.zeroGradient()

let batch: Tensor<Float, CPU> = ... // shape: [batchSize, 28, 28]
let y_true: Tensor<Int32, CPU> = ... // shape: [batchSize]

let pred = model.forward(batch)
let loss = categoricalCrossEntropy(expected: y_true, actual: pred)

loss.backwards()
optimizer.step()

Recurrent Networks

Example for MNIST classification

The LSTM scans the image from top to bottom and uses the final hidden state for classification.

let model = Sequential<Float, CPU>(
    LSTM(inputSize: 28, hiddenSize: 128).asAny(),
    Dense(inputFeatures: 128, outputFeatures: 10).asAny(),
    Softmax().asAny()
)

let optimizer = Adam(parameters: model.trainableParameters, learningRate: 0.001)

// Single iteration of minibatch gradient descent
optimizer.zeroGradient()

let batch: Tensor<Float, CPU> = ... // shape: [batchSize, 28, 28]
let y_true: Tensor<Int32, CPU> = ... // shape: [batchSize]

let x = batch.permuted(to: 1, 0, 2) // Swap first and second axis
let pred = model.forward(x)
let loss = categoricalCrossEntropy(expected: y_true, actual: pred)

loss.backwards()
optimizer.step()

Generative Adversarial Networks

Example to generate random images similar to those in MNIST

let images: Tensor<Float, CPU> = ... // shape [numImages x 28 x 28]

let d1 = Dropout<Float, CPU>(rate: 0.5)
let d2 = Dropout<Float, CPU>(rate: 0.5)

let generator = Sequential<Float, CPU>(
    Dense(inputFeatures: 20, outputFeatures: 200).asAny(),
    Tanh().asAny(),
    d1.asAny(),
    Dense(inputFeatures: 200, outputFeatures: 800).asAny(),
    Tanh().asAny(),
    d2.asAny(),
    Dense(inputFeatures: 800, outputFeatures: 28 * 28).asAny(),
    Sigmoid().asAny(),
    Reshape(shape: 28, 28).asAny()
)

let discriminator = Sequential<Float, CPU>(
    Flatten().asAny(),
    Dense(inputFeatures: 28 * 28, outputFeatures: 400).asAny(),
    Tanh().asAny(),
    Dense(inputFeatures: 400, outputFeatures: 100).asAny(),
    Tanh().asAny(),
    Dense(inputFeatures: 100, outputFeatures: 1).asAny(),
    Sigmoid().asAny()
)

let network = Sequential(generator.asAny(), discriminator.asAny())


let optimGen = Adam(parameters: generator.trainableParameters, learningRate: 0.0003)
let optimDis = Adam(parameters: discriminator.trainableParameters, learningRate: 0.0003)

let batchSize = 32
let epochs = 10_000
let regularization: Float = 0.001

let genInputs = Tensor<Float, CPU>(repeating: 0, shape: batchSize, 20)

for epoch in 1 ... epochs {
    optimDis.zeroGradient()

    let real = Random.minibatch(from: images, count: batchSize)
    Random.fillNormal(genInputs)

    let realResult = discriminator.forward(real)
    let fakeResult = network.forward(genInputs)

    let dRegLoss = optimDis.parameters.map {l2loss($0, loss: regularization)}.reduce(0, +)
    let discriminatorLoss = -mean(log(realResult)) - mean(log(1 - fakeResult)) + dRegLoss

    discriminatorLoss.backwards()
    optimDis.step()

    var generatorLoss = Tensor<Float, CPU>(0)

    for _ in 0 ..< 4 {
        optimGen.zeroGradient()
        Random.fillNormal(genInputs)

        let genResult = network.forward(genInputs)

        let gRegLoss = optimGen.parameters.map {l2loss($0, loss: regularization)}.reduce(0, +)
        generatorLoss = -0.5 * mean(log(genResult)) + gRegLoss // heuristic non-saturating loss

        generatorLoss.backwards()
        optimGen.step()
    }

    if epoch % 100 == 0 {
        print(" [\(epoch)/\(epochs)] loss d: \(discriminatorLoss.item), g: \(generatorLoss.item)")
    }
}

Random.fillNormal(genInputs)
let genResult = generator.forward(genInputs)

for i in 0 ..< batchSize {
    let slice = genResult[i].T.unsqueeze(at: 0)
    guard let image = NSImage(slice), let imgData = image.tiffRepresentation else {
        continue
    }
    guard let rep = NSBitmapImageRep.init(data: imgData) else {
        continue
    }
    let png = rep.representation(using: .png, properties: [:])
    try? png?.write(to: URL(fileURLWithPath: "generated_\(i).png"))
}