MadDiff and MadIPM UniformBatch: Recent advances in implicit differentiation and batch optimization on GPU

Michael Klamkin – GaTech

MadIPM UniformBatch is a new feature in the MadIPM linear programming solver, enabling the resolution of “uniform” batches of instances in parallel on GPU. MadIPM leverages the uniform batch property to reuse the symbolic factorization, as well as a new batch automatic differentiation API, delivering considerable speedup over sequential solves as validated on large-scale Optimal Power Flow instances.

MadDiff is a GPU-accelerated implicit differentiation library for linear, quadratic, and nonlinear programming. MadDiff’s API is oriented around use-cases such as deep learning where the full Jacobian is not needed as vector-products suffice. Leveraging this fact, a new sparse parametric automatic differentiation interface, and the ability to re-use the KKT factorization from the forward solve, MadDiff delivers substantial speedup compared to existing solutions as validated on large-scale Optimal Power Flow instances. MadDiff also features an experimental integration with MadIPM UniformBatch, paving the way for simple and performant implementations of advanced deep learning architectures such as optimization layers and Sobolev loss functions.