NetFlux: Component Oriented Flow Based Programming Language

In this blog, I present a programming language that I design for generic, parallel, pipelined, asynchronous, stateless applications. The language is a DSL built within the confinements of Python programming language. You can find this project on Github.

Before moving on to the technical details of the language, let’s see the application areas.

• Deep Learning / Machine Learning
• Digital Signal Processing
• Image/Video Processing
• Text/Stream Processing
• Digital Circuit Simulation
• Queueing Network Simulation
• Embedded Systems
• Super-loop Event Handling system
• Map Reduce Paradigm for Big Data
• Internet of Things
• Web Server/DB programming
• UNIX style pipe and filter
• Audio Processing/Sound Enhancement
• Open and Closed Loop System

Now, that we have seen the applications it should motivate us to see how NetFlux accomplishes all of these.

At the core of the system, there are a bunch of classes. They are Microprocess, Component, ComponentBuilder, Scheduler, Transporter, Linker, Executor, Flux, and NetFlux. Understanding these objects will help us write NetFlux programs. Here is a class diagram of NetFlux’s core.

So, what is Component Oriented Flow Based Programming Paradigm? Let’s break it down. Just as Objects are the key concept used in Object Oriented Programming, in NetFlux, Components are central to everything. Components have ports. Ports are buffering slots for Input, Output, Error and Control Signals. Components interact by sending and receiving data through ports. Components are Microprocess. A Microprocess has an execution context and a run method. There is a special type of Microprocess called Transporter that delivers data. One Transporter is required for delivery form a source port to all that attached sink ports.

That was the first part. Now the second part. Flow Based Programming. Flow Based Programming is a programming paradigm in which the operational/executional model is based on the flow of data between loosely coupled component. In other words, it is your data-flow diagram written in code (implementation). Straight from white board to implementation as it is. Awesome!

Why did I name it as NetFlux? Flux stands for the flow that happens between the components. And NetFlux is the network of fluxes (flows). Also its a tongue in cheek for NetFlix 🙂

Why is NetFlux awesome? Typically software is developed with a data flow diagram on white board with bunch of component (boxes) and links (lines) and the way the data flows dynamically (over time). It starts with those boxes being separate modules, interacting minimally (decoupled) with other boxes. But when translated to implementation, the code base is typically written in Object oriented paradigm, with bunch of object, heavily depending on each other finally the software becomes something completely different from what was initially envisioned on the white board. NetFlux comes to the rescue. Its results in clean code straight from its original form on the white board to implementation.

For a moment, let’s assume that we want to write/build an HTTP Server with a Database in its back end. Roughly the white board sketch would look as shown in the figure below.

There are a bunch of components Request Validator, Response Formatter, Logger, Query Generator, Network I/O. The green path is a error free scenario. The orange links, indicate errors to be logged. The Network I/O component is the main interface with the external world. To implement and deploy something like this, the NetFlux’s approach has a 3 steps.

1. Build the components with ComponentBuilder
2. Define the flow diagram as graph in NetFlux DSL
3. Link, Schedule and Execute with Linker, Scheduler and Executor

Each component comes with IN, OUT and ERR port. We can add more ports if needed. A component is a abstraction over a function. In programming, functions have inputs, return output and may generate errors. So, each component is implemented separately as a python function and component-ified with ComponentBuilder. Refer NetFlux project on ithub for the details.

The classes Flux and NetFlux, implement a mini DSL in the python, so that the definition of the flow graph becomes easier. The Linker uses the NetFlux graph and initializes the links. The Scheduler is a component, that maintains a active queue so that each component in the graph is activated in a round robin fashion. The Executor component is the runtime engine of NetFlux. It can be made to run for a specified number of process clock tick or can be run indefinitely forever.

Happy exploring! Hope you give a try to NetFlux.