PraxisLIVE

OPENRNDR in PraxisLIVE

Wed, 07 May 2025 00:00:00 +0000

A lot of focus in recent releases of PraxisLIVE has been on support for custom roots and base components. This allows for building out a range of components to use with third-party libraries, all within the context of an individual project. And this being PraxisLIVE, all aspects being live recodeable as you do so!

Over the last few weeks, I’ve been exploring OPENRNDR - a fantastic open source creative coding framework that offers an alternative to the built-in support for Processing inside PraxisLIVE.

OPENRNDR maps well to a node graph based on similar principles to the core Processing support, and makes use of a number of libraries already in PraxisCORE. At the same time, it is written in Kotlin so throws up some interesting challenges for using it from a Java context (the page on Calling Kotlin from Java is very useful!).

I’ve uploaded a repository with the example code discussed here, that can also be used as a template for exploration. Aside from the logo image, everything is contained within the two source files for the project and root.

https://github.com/codelerity/pl-openrndr/

All seems to be working well on Linux and Windows at the moment. I’m working on a fix for the image nodes that seem to be causing issues on macOS. This is also using OPENRNDR 0.4.4 for now, due to another issue with the latest alphas.

Project definition

The first file in the project is the project.pxp file. In the IDE, this file is edited via the project properties. There are two key changes for the hub and libraries. The hub definition is changed to load all roots in a separate JVM launched with the same necessary Java options for the OpenGL support as in the built-in Processing (root:video) support.

java-options {
  -XstartOnFirstThread
  -XX:+IgnoreUnrecognizedVMOptions
  -Djava.awt.headless=true
}

The libraries section adds three PackageURLs for OPENRNDR artefacts on Maven Central. These will automatically be resolved and downloaded when you run the project, using the MIMA support built in to PraxisCORE. The various LWJGL dependencies required by OPENRNDR are already in the runtime and will be resolved from there.

libraries {
  "pkg:maven/org.jetbrains.kotlin/kotlin-stdlib@2.1.10"
  "pkg:maven/org.openrndr/openrndr-application-jvm@0.4.4"
  "pkg:maven/org.openrndr/openrndr-gl3-jvm@0.4.4"
}

Within the IDE, libraries can be added to a project using the project properties. See more at https://docs.praxislive.org/coding-libraries/

Custom root graph

The second file is the openrndr.pxr root graph. All of the integration code to work with OPENRNDR is in this file. Within the IDE, look at the Shared Code files, the root code (Edit Code in the graph background popup), as well as the code of individual nodes.

I’m not going to go into every aspect of the integration, but some key things to look for include -

Driver proxy

A key integration point between OPENRNDR and PraxisCORE happens in the root code at -

@Driver Consumer<Boolean> processor = this::process;

private void process(boolean active) {
    if (!active) {
        tell(self("stop"), "");
        return;
    }
    // call graph and render to window
    RenderTarget rt = getRenderTarget();
    rt = sink.process(rt);
    program.getDrawer().image(rt.colorBuffer(0));
    releaseRenderTarget(rt);
}

The @Driver annotation works similarly to @Proxy but can only be applied to an interface implementation in a root. PraxisCORE uses a kind of wrapper injection to wrap the field value in a proxy implementation which drives the root graph every time a call is made. This way the OPENRNDR rendering thread can drive the component graph.

The processor wrapper (now the value of this field) is passed into the very simple SHARED.RNDRProgram implementation of Program and called on every render. This interface could have been Runnable except that we also want to call it when the rendering exits due to the window closing. The RNDRProgram is passed to the MainThread executor, ensuring it runs on the startup thread of the JVM, required for OpenGL rendering on some platforms.

Data pipes

The next key integration point is the use of Data Pipes to pass OPENRNDR’s RenderTarget surfaces through the node graph. The root code includes -

@Ref.Publish
@Inject Ref<Data.Pipe<RenderTarget>> input;

@Inject Data.Sink<RenderTarget> sink;

public void init() {
    input.init(Data::identity);
    sink.onCreate(rt -> {
        return getRenderTarget();
    });
    sink.onClear(rt -> {
        rt.clearColor(0, CLEAR);
        return rt;
    });
    sink.onAccumulate((dst, src) -> {
        DrawerKt.isolatedWithTarget(program.getDrawer(), dst, unit(d -> {
            d.getDrawStyle().setBlendMode(BlendMode.ADD);
            d.image(src.colorBuffer(0));
        }));
        return dst;
    });
    sink.onDispose(rt -> {
        releaseRenderTarget(rt);
    });
    Data.link(input.get(), sink.input());
}

We use a Ref to publish an input that can be picked up and connected to by our output nodes. This in turn is connected to a Data.Sink configured to create, clear and add together render surfaces. Aside from the cache of RenderTarget held elsewhere in the code, this is all that is required to enable a data graph that includes support for adding multiple inputs and feedback paths.

The image in the header was created by adding a feedback path through a mix component on the mouse driven circle -

Custom component registration

The various custom component types defined in Shared Code are registered inside the root code -

@SupportedTypes(custom = {
    @CustomType(type = "openrndr:draw", base = RNDRDraw.class,
            template = RNDRDraw.TEMPLATE),
    @CustomType(type = "openrndr:composite", base = RNDRComposite.class,
            template = RNDRComposite.TEMPLATE),
    @CustomType(type = "openrndr:image", base = RNDRImage.class),
    @CustomType(type = "openrndr:mix", base = RNDRMix.class),
    @CustomType(type = "openrndr:out", base = RNDROut.class)
})
public void init() { ...

All the custom component types are defined in Shared Code, and so are fully live recodeable. The components that are designed to be extended with custom behaviour have a defined code template as a starting point.

A cut down version of the RNDRDraw type shows how these are defined as subtypes of CodeCodeDelegate. You can see how the data pipe configuration is handled within the base type, allowing for user nodes to just draw on the surface with the provided Drawer.

public class RNDRDraw extends CoreCodeDelegate {

    public static final String TEMPLATE =
            = """
              import org.openrndr.*;
              import org.openrndr.draw.*;
              import static SHARED.OPENRNDR.*;

                  @Override
                  public void draw(Drawer d) {
                     \s
                  }
              """;

    @In(1) Data.In<RenderTarget> in;
    @Out(1) Data.Out<RenderTarget> out;

    @Ref.Subscribe
    @Inject Ref<Program> programRef;

    public Program program;

    @Override
    public void init() {
        Data.link(in, Data.with(this::process), out);
    }

    private void process(RenderTarget frame) {
        program = programRef.get();
        DrawerKt.isolatedWithTarget(program.getDrawer(), frame, unit(d -> {
                    draw(d);
        });
    }

    public abstract void draw(Drawer d);

}

Also note the SHARED.OPENRNDR utilities that are statically imported into the template. These are used for simple, directly callable functions where OPENRNDR’s API is complex to use from Java. It is also used to simplify other aspects of the Kotlin interaction - eg. unit(..) to wrap consumers that need to return Kotlin’s Unit.INSTANCE.

Next steps

Currently I’m looking at stabilising some aspects of this and then adding it to the standard available project templates. In many ways, custom integrations like this, doing more within projects while keeping the core lightweight, is the future direction of PraxisLIVE.

I’m also looking at a different custom library integration with Avaje Jex and HTMX, as a possible replacement GUI … or even the basis of a fully recodeable editor.

Any thoughts or questions, please do jump in on the PraxisLIVE discussions at https://github.com/orgs/praxis-live/discussions, or reach out some other way.

Thanks for reading!

On Watch

Wed, 02 Apr 2025 00:00:00 +0000

PraxisLIVE v6.1 saw the first introduction of Watch functions into PraxisCORE, along with basic UI support in the PraxisLIVE graph editor. These saw further enhancement in the v6.2 release, and still more is planned.

A Watch function is a function control for accessing information about a component. They are somewhat similar to read-only properties, but for data that should only be calculated on demand, and perhaps asynchronously. They also support extended metadata, and the option of query arguments to request specific information. Watch functions are somewhat analogous to watch expressions in a debugger, but within the context of an active real-time re-codeable system. They can be rewritten as necessary to explore and expose the state of the running system, while having minimal overhead when not in use.

The only required metadata for a Watch function is the media (MIME) type of the returned data, which is used to select a suitable viewer. The function should also respond with the data type specified by the media type - usually String for text-based formats, and PBytes for binary ones. The graph component inside PraxisLIVE currently provides viewers for PNG and SVG images, as well as plain text.

Defining Watches

Watch functions can be defined in code using the @FN.Watch annotation. This works similarly to the generic @FN annotation (and under the hood uses the same control binding), with additional fields for defining the necessary metadata.

A simple text watch could be defined using -

@FN.Watch(mime = MIME_TEXT)
public String text() {
    return "Hello Watcher!";
}

- although as this returns the same text each time, it’s not the most useful of watches! The watch can be exposed on the graph editor in the IDE using the Expose sub-menu in the component popup. The expose feature also supports other types of control, such as properties. To expose any watch or control by default, use the @Config.Expose annotation.

Another supported media type for watches is SVG. The SVG data should be returned from the control as a String, and in a format suitable for embedding in HTML5 (the IDE doesn’t use HTML, but other future contexts will). We can write a component that highlights the state of a boolean property, and relate the watch to the port, using -

@P boolean status;

@Config.Expose("watch")
@Override
public void init() {
}

@FN.Watch(mime = MIME_SVG, relatedPort = "status")
String watch() {
    return """
        <svg width="64" height="64">
           <circle cx="32" cy="32" r="32" fill="COLOUR" />
        </svg>
        """.replace("COLOUR", status ? "green" : "red");
}

Async responses

Also added in v6.1 was support for asynchronous returns from function controls, which are very useful for Watches. Asynchronous values in component code are represented using the Async<T> type. An Async is somewhat similar to a standard Java Future, however it is neither thread-safe nor blocking. It is not possible to wait on the value, which will be completed by a future actor call. Function response calls will automatically be generated and sent when the returned Async completes.

We might use the standard ask() function to send a call to another control and return its result, or use async() to generate a response, or manage internally using Async or Async.Queue fields.

In the code below we create a video:custom component that can be used to watch video frames moving through the pipeline. This is a reduced version of the code in the built-in video:composite. In the watch function itself we return an Async<PBytes> that has been wrapped by the timeout function - this will complete the Async with error if nothing else does so within the specified time. We cannot initiate the encoding of the input itself until draw() is called. During the drawing method, a check is made to see if a watch result is pending. A scaled down PNG encoding of the input is started using write() which also returns an Async value. The pending Async is bound to the PNG result and then the field cleared.

@In PImage in;

@Persist Async<PBytes> watchInResponse;

@Override
public void draw() {
    if (watchInResponse != null) {
        double scale = 0.25;
        Async<PBytes> png = write(MIME_PNG, in, scale);
        Async.bind(png, watchInResponse);
        watchInResponse = null;
    }
    copy(in);
    release(in);
}

@FN.Watch(mime = MIME_PNG, relatedPort = "in")
Async<PBytes> watchIn() {
    if (watchInResponse == null || watchInResponse.failed()) {
        watchInResponse = timeout(1, new Async<>());
    }
    return watchInResponse;
}

Chaining, queries and future moldability

Watch functions can accept optional or required queries in the form of maps, although this feature is not yet exposed in the IDE interface.

For example, the first text example might be changed to -

@FN.Watch(mime = MIME_TEXT)
public String text(PMap query) {
    return "Hello " + query.getString("name", "<UNKNOWN>");
}

The Output window in the IDE can be used to call this function -

> /root/component.text [map name John]
--- Hello John

Queries will be used in future to support a range of feature such as data ranges and other formats.

Another future plan is to support chaining watches, using re-codeable watches in other components or embedded within the IDE to transform data, to highlight certain features or adapt media types. Watches are also designed to be used in other contexts away from the PraxisLIVE IDE, such as in a HTML-based UI or notebook-like interface.

All in all, Watches provide an open-ended way to explore, understand and represent what is going on inside your projects as they are running.

PraxisCORE / PraxisLIVE v6

Wed, 27 Nov 2024 00:00:00 +0000

I’m pleased to announce that this month saw the first version 6 release of PraxisCORE and PraxisLIVE. Somewhat longer in gestation than originally planned (due to things unrelated to the project), version 6 is a substantial rewrite of some key aspects of how PraxisCORE works. Expect some rough edges! These will improve as version 6 update releases happen over the coming months.

Key changes

Many deprecated and legacy features have been removed, removing things that were hindering development of some key new features. With existing projects, make sure to check for and correct use of deprecated features and code in the latest version 5 release before attempting to open a project in version 6. Also check the following section about removed root types.

The IDE and core now require minimum Java 21. Installers with bundled JDK will now ship with the latest JDK version as standard. Networked hubs now use Netty and Ion to provide a more robust and structured communication protocol than was possible with the previous OSC implementation. Much of the project saving and execution has moved directly into PraxisCORE, which means that much more of the IDE functionality works by calling Pcl commands. This also opens up the possibility of alternative editors, or even one day the IDE will be a PraxisCORE project!

Other libraries have been updated, including our libP5X updated to Processing 4. Runtime dependency management is now based on MiMa rather than Ivy.

There are a bunch of other useful changes. Custom roots can now specify use of table editing rather than the graph editor (we can now make fully recodeable MIDI editors, etc.) Containers can filter allowed child types and even specify custom types. The weight parameter on annotations is now optional - eg. @P int x,y,z; is now legal syntax! The ordering is in alphabetical order by default. Examples and custom components are being updated and added showing the new features.

The most obvious visual change is the new vector logo, window background actions, and revamped dashboard. The background actions and dashboard support were contributed to Apache NetBeans a few months ago, but were really written for PraxisLIVE!

Finally, the IDE build system has been moved to Maven to match PraxisCORE. Most of the IDE modules are now published to the central repository as well, allowing use of PraxisCORE and PraxisLIVE as a platform for other projects. One of which is currently in the works.

Root templates .. or where’s my OSC, MIDI, TinkerForge support?

Version 6 sees the removal of the built-in support for OSC, MIDI and TinkerForge roots. These features will be supported by using custom root templates and (where necessary) external libraries in future. A MIDI root template is already included in the custom components repository. OSC and TinkerForge examples will be added soon. Automatic migration from old root types is not currently available.

To use the MIDI template, make sure to install custom components from the dashboard display. Then use the popup menu on the project, select New / Other..., and find the MIDI template in the Roots category.

If you have any audio, video or other root graph that you want to use in other projects, you can also use the Save as Template... action on any .pxr file.

The custom MIDI component also offers a good demonstration of the new annotations for registering table editing and custom components - in particular here, where ControllerComponent is a subclass of CoreCodeDelegate in the shared code -

@SupportedTypes(system = false,
        custom = {
            @CustomType(type = "midi:controller", base = ControllerComponent.class)
        }
)
@DisplayTable(properties = {"channel", "controller", "binding"})
public void init() { ...

Towards introspection and moldability

Over the next few months I’ll be working on various improvements within PraxisCORE and PraxisLIVE related to introspection, and improved visualization and control over what is happening in the underlying system. This in part comes from an interest in exploring how aspects of moldable development and live moldable notebooks can work within the cyber-physical coding context of PraxisLIVE. Watch this space …

In the meantime, download PraxisLIVE v6 and get discovering!

ClassLoader trees

Mon, 14 Jun 2021 00:00:00 +0000

…exploring ClassLoader trees in the forest of actors…

PraxisCORE v5.2 added the ability to share code and types across components (actors) in a graph. This is a feature that’s been requested for a while, but took some thinking to work out how to implement. And it’s a key ingredient in how Codelerity is supporting general purpose programming usage of the PraxisLIVE suite of tools.

It’s been possible to add compiled libraries to PraxisCORE projects for a while, even at runtime, and v5.1 added the ability to include dependencies from Maven Central. However, such pre-compiled code is not editable. With the shared code feature we want to allow components to share features and types, while still allowing that code to be rewritable safely at runtime like the code of individual components.

Here we have a simple example of shared code usage in PraxisLIVE v5.2 (click to enlarge). We have a shared ID type passed between two components, id-source and id-consumer using data ports. This is somewhat contrived as data ports are usually used for packets of continuous data. We also have a shared Utils class for utility methods that can be static imported into our component code.

The problem to be solved was allowing the shared code to be edited and recompiled at any point, while ensuring that at any instant, all components in the graph are linked to the same shared type. We also want to ensure that any changes to shared code are always valid - not removing types or methods in use by any component.

Part of addressing these concerns is the limitation that shared code is only shared across the components within a single graph / root. As within our forest of actors architecture each root is guaranteed to be only active on a single thread at any time, we can ensure atomic replacement. It also shields us from having to deal with the problem of some roots being local and others remote. Shared code is currently stored as a property of the root component, although in theory we could also support shared code on sub-containers.

Implementing shared code involved some rethinking of the ClassLoader hierarchy used for dynamic code in components, as well as significant changes to the compiler service and other services used in the updating of component code.

A ClassLoader tree

In versions of PraxisCORE prior to v5.2, we have the following (truncated) hierarchy of ClassLoaders for each component.

System ClassLoader(s)
Library ClassLoader
Code delegate ClassLoaders

The system ClassLoaders are obviously the JDK and modules that make up the PraxisCORE runtime, on modulepath or classpath. The library ClassLoader is used for adding pre-compiled libraries to a project, or strictly to the RootHub - this is currently a subclass of URLClassLoader that allows for dynamic addition (not removal), and there is a single instance that exists for the lifetime of each hub.

Finally, the code delegate ClassLoaders are lightweight in-memory ClassLoaders that contain a single iteration of one component’s code. Every code update involves replacing this ClassLoader and the classes it contains. There can be many active code delegate ClassLoaders at any one time - one for every component in the graph that has customized code.

For more on the use of code delegates to support real-time programming / code hotswap see the related post on Just-in-Time Programming systems in Java.

From v5.2 this ClassLoader tree has changed - we now have a shared code ClassLoader inserted in the hierarchy.

System ClassLoader(s)
Library ClassLoader
Shared code ClassLoader
Code delegate ClassLoaders

That is, some component ClassLoaders may now have the shared code ClassLoader as a parent. As mentioned, shared code is currently attached to each root, so we now have a ClassLoader tree that roughly follows our forest of actors hierarchy - system (process) → library (root hub) → shared code (root) → code delegate (component).

Like the code delegate ClassLoaders, the shared code ClassLoader is lightweight and in-memory (in fact the same type, wrapping a PMap of PByte). Whenever any shared code type is updated, we have to recompile all the shared code types, and create a new ClassLoader. This also means that we need to recompile and create new ClassLoaders for every dependent code delegate - we cannot just reload the delegate bytecode in a new ClassLoader or we run the risk of NoSuchMethodError and other problems when the shared code changes incompatibly.

Because of the overhead of recompilation and atomic replacement, the shared code ClassLoader is kept as an optional element of the hierarchy - only components that require linking to shared code types have this ClassLoader as a parent. Likewise, because the code delegate ClassLoader remains separate and has the shared code ClassLoader as a parent, we can update component code without needing to recompile and reload any shared code. So, the shared code addition has minimal impact on runtime code updates that were possible prior to v5.2.

Compiler and context service rewrites

PraxisCORE has the concept of Protocols, akin to interfaces exposed by actors, and Services that are Protocol implementations that can be registered and located by other actors. There are two services involved in runtime code updates - a service that can create new CodeContexts (wrappers for CodeDelegate instances loaded from the above mentioned ClassLoaders) and a service that exposes the Java compiler. Both work with source and bytecode in memory rather than on disk.

The context service deals with actual Java references so must be in the same process (so one will exist in each PraxisCORE process across a distributed hub). The compiler service can run in a separate process, or even on a separate machine, so there will usually only be one.

Prior to v5.2, the compiler service accepted a single Java class body (ie. no imports, class definition, etc.) and the class name of a type that could be used to assist in wrapping the class body into valid Java source for compilation. A legacy of an earlier iteration of compilation where everything was handled in-process, this had the limitation of not working for full Java sources, or multiple classes. For v5.2 the compiler service was extended to accept a map of Java sources to compile. It can also accept an optional map of pre-existing classes to compile against.

The context service had to be rewritten to handle the wrapping of class bodies itself, as well as extended to provide a shared code context service that also handles recompilation and reloading of all dependent component code. This also required changes to how classes are identified. Assume a component with the following action -

@T(1) void logClass() {
    log(INFO, "Class : " + getClass().getName());
}

In versions prior to v5.2, if you trigger that action you will see -

INFO : /data/custom
Class : $

From v5.2 you will see something like -

INFO : /data/custom
Class : CODEa255.data.custom.code.$

Every component with custom code prior to v5.2 was backed by a delegate class named $ in the default package. This was fine, because a class is actually identified by both its name and the ClassLoader that loaded it. However, now we want to support compiling multiple components’ code at the same time, along with our shared code, we have a problem. Instead, we switch to a package name based on the component address, with a partially randomized root package (a255 here is random hex).

Going back to the basic patch shown in our image above, whenever the .code property on id-source or id-consumer receives a new iteration of code, that code will be sent to the code context service actor. That actor will wrap the code in its context, adding default imports, package location (eg. package CODExxxx.data.id_source.code;), and class definition (eg. class $ extends CoreCodeDelegate). This source will then be passed to the compiler service, along with the previously compiled bytecode for the SHARED package. On successful response, the compiled code delegate is created, wrapped, and passed back to the component.

On the other hand, when the code of either ID or Utils is updated, the .shared-code property will collect all shared code dependent components (in this case both id-source and id-consumer) and pass their source along in a task for the code context service. The service will wrap all component code as before, and add to a source code map along with all shared code, for passing to the compiler. On successful receipt, the returned bytecode map is partitioned to create the shared code parent ClassLoader and each individual delegate ClassLoader. Delegates are created, wrapped, and passed back in one message for atomic replacement. As the process is asynchronous, the response is validated against the current graph before each component is updated.

Whether or not a component is compiled and loaded against code from the SHARED package is currently triggered by a search for the String "SHARED." in the source code. This obviously has the possibility for false positives - it might change in future!

A note on Data

Data ports allow for continuous data packets to be passed from one component to another. They are typed by their generic signature, and if that changes then the configuration is invalidated and all connections are removed. Because class identity is actually a product of name and ClassLoader, this would have caused all ID ports to be disconnected on every shared code change. In v5.2 data ports have been updated to retain connections where the type is in the SHARED package and has the same name - atomic code replacement ensures that an ID.class reference is always identical across all components at any one time.

Inside the PraxisLIVE IDE

The PraxisLIVE IDE has been updated to support shared code editing - right-click on the graph editor to show / add shared types. PraxisLIVE uses Apache NetBeans’ memory file system to allow editing of shared types in the same way as used for component code. The shared code memory file system is separate to the ones used for components, but added in to the IDE’s class path support in a similar hierarchy as used in PraxisCORE for class loading. One difference is that shared code is always included in the class path, so that completion and code navigation works, whether the code currently references shared code or not. Imports will be correctly added on code completion if required, and CTRL-click on shared types or their members will open them in the editor.

Next steps

As you might gather from the length of this post, finding the right way to implement shared code was complicated and took time to get right. However, it really opens some interesting options for the future as we expand general purpose coding support in PraxisCORE.

One initial thing currently being looked at is using shared code to support component prototypes - effectively a way for multiple components to share the same code, but keeping some ability to override at a component level.

Keep an eye on the docs for shared code as these features evolve.

If you’re interested in the actual implementations of the processes outlined above, check out CodeProperty, SharedCodeContext, SharedCodeProperty, DefaultCodeFactoryService, and DefaultCompilerService.

Thanks for reading!

PraxisCORE / PraxisLIVE v5

Fri, 02 Oct 2020 00:00:00 +0000

PraxisLIVE and PraxisCORE v5.0 were finally released on September 21st, after a number of earlier beta releases. This was a few months later than originally scheduled, but then 2020 really hasn’t been a great year for best laid plans!

Radical changes

Version 5 is the most radical change to PraxisLIVE since the move to full re-codeability between v1 and v2. The extra months afforded by the 2020 lockdown and postponement of some work have been used to fully implement and extend the changes discussed in the 2020 Vision post.

There has been a major rewrite of both the core runtime and the IDE. Much legacy and deprecated functionality has been removed. All of PraxisCORE has now been rewritten around the new base actor system introduced in part in 4.4.0. PraxisCORE is now built on top of the Java module system, rather than the NetBeans platform (the PraxisLIVE IDE is still based on Apache NetBeans). The build for PraxisCORE has been moved to Maven, and individual modules are deployed to central for re-use from other projects.

Both PraxisLIVE and PraxisCORE now require at least Java 11, and fully embrace newer Java features. An open-source Java 11 JDK from AdoptOpenJDK is now included in all binary bundles, including in the new AppImage for Linux, although the standalone zip is still available.

PraxisLIVE now supports running multiple projects at once. Distributed hub support has moved from the IDE into PraxisCORE, and is configured separately for each project - the default configuration will run audio and video roots in separate Java processes. Additional processes are now started when required rather than automatically. The master/slave terminology has been removed, both because it’s problematic and because it no longer fully reflects the technical roles different processes can play. Read more about configuring project hubs in the docs.

The PraxisCORE runtime is now included in the praxiscore folder inside the IDE installation. It can be copied out and run from the CLI for networking or project running, or embedded inside a project to make it standalone.

Video / OpenGL visualization is now based on libp5x, our modularized fork of Processing that uses LWJGL for rendering. Audio support has now moved to our Pipes v2 library, which includes all the unit generators previously in PraxisLIVE itself. Both of these libraries are published separately via Maven for use elsewhere.

Rethinking coding

One thing you might notice on the website and elsewhere is a new tagline for the overall project - rethinking general purpose and creative coding. PraxisLIVE will always aim to be a great programming system for creative coding, but one key aspect of founding Codelerity to support PraxisLIVE’s development is to explore opportunities for use outside of this sector.

PraxisCORE’s move to Java modules and the publishing of individual modules is a part of this. Version 5 also lays the groundwork for a range of additional features to come over the next months that will benefit everyone. These include proper dependency management for improving third-party library usage (such as Maven dependencies), support for shared (and still editable) code across components, better support for working with embedded boards and electronics (and Raspberry Pi), and improving CLI and code editing support away from the IDE.

By version 6 it may be that all specific features (audio, video, etc.) get automatically downloaded only when required. But throughout version 5 expect additions that concentrate on what PraxisCORE and PraxisLIVE do best - rethinking the coding process, and bringing the best aspects of Erlang, Smalltalk and Extempore into the Java world.

Get it while it’s hot - download PraxisLIVE v5 now!

2020 Vision

Thu, 09 Jan 2020 00:00:00 +0000

So, with a pun too obvious to ignore (if ironic given my eyesight!), let’s take a look ahead at 2020 plans for PraxisLIVE and PraxisCORE. As well as talk a little about the founding of Codelerity Ltd.

Objects in the rear view mirror

In some ways, 2019 seemed a quiet year on the PraxisLIVE front. We had three minor releases which brought some useful new features, including support for Java 11. But there were no major changes, at least on the surface. Behind the scenes, however, some big changes have been afoot as preparations have been made for the upcoming v5.

2019 had some enjoyable PraxisLIVE related talks and workshops too, including trips to ICLC in Madrid, FOSDEM in Brussels, and LGM in Saarbrucken; JUG talks in Birmingham, Manchester, Dublin and London; and the really fun Code2Create workshop in Andover.

During 2019 I was also involved as a PMC member of the Apache NetBeans community, fleshing out the new quarterly release plan, and release managing two Apache NetBeans releases (11.1 and 11.2). NetBeans is an essential part of PraxisLIVE, which is now based on Apache NetBeans 11.1 - an update that has allowed us to run on and support Java 11+.

Jump to light speed

Aside from occasional paid projects and workshops, PraxisLIVE has mostly been a spare-time (if sometimes all consuming) pursuit. In the decade leading up to 2017, most of my paid work was, like many others, in web applications and services. Since 2017 I’ve been trying to concentrate more on work related directly to PraxisLIVE in order to spend more time on it, the majority of which to-date has been related to GStreamer.

In mid-2019 I set up Codelerity Ltd.. We will be offering consultancy and development work around real-time systems, real-time programming, media, embedded and tools on the JVM. This will expand on the recent consultancy work I have been doing, as well as investing in other use cases for aspects of PraxisCORE and PraxisLIVE. For example, we’ve recently been looking at low-latency video in machine learning, as well as development for embedded devices. If you think we might be able to help with something, get in touch!

Donations to support PraxisLIVE development are, and will remain, much appreciated. However, they are not enough to sustain the amount of time I’d like to be putting in. Assuming all goes to plan, Codelerity will allow me to spend more time investing in PraxisLIVE development, while ensuring it remains free, and continues with its creative and innovative approach to coding.

Number 5’s alive(-ish)

PraxisLIVE, or more specifically the PraxisCORE runtime, is undergoing some radical restructuring for v5. So radical we’ve switched to a completely new repo! We’re moving away from the NetBeans module system and fully embracing Java modules, and Java 11 will be the baseline supported Java version. We’re also moving to the Maven build system, and all modules that make up PraxisCORE will be published to Maven Central for reuse in other projects.

A new, simplified actor and component base was added in v4.4.0, initially just for data roots and some of the built-in Hub services. These will now form the basis for other modules, allowing a large amount of legacy code from the v1 days to be removed. The code is also being restructured to remove unwanted dependencies between modules, remove the need for Processing except in the OpenGL renderer, and replace all NetBeans usage in PraxisCORE. Many of these changes support Codelerity’s plans for other use cases for these modules, while also offering key benefits in PraxisLIVE.

Unfortunately, Processing still has some issues with Java 11, as does JOGL, so I’m currently working on forking Processing and tidying up the LWJGL-based OpenGL renderer. These changes should give us a better performing OpenGL pipeline, and the forked modules will also be published to Maven Central for other projects to make use of. Of course, I’d prefer to see those changes accepted upstream, but we shall see.

Changes in the PraxisLIVE IDE for v5 will not be as radical, but there will be some key changes to support changes in the runtime. Projects will now run in separate processes from the IDE, and there will be a Hub per project. This will allow multiple projects to be run independently, removing a confusion for new users. Distributed hub settings will also be moved to each project, and also be taken into account when the project is run from the command line or as a standalone application. Support for exporting standalone applications will also be improved.

PraxisLIVE v5 should be released sometime in the next few months, although as with all these things we’ll release when it’s ready! Not every change is decided yet - if you have things you’d like to see then open an issue, or join in the conversation.

Just-in-Time Programming

Mon, 31 Dec 2018 00:00:00 +0000

So, a few thoughts on the trials and tribulations of building a just-in-time programming system in Java. But wait, isn’t Java a JIT-compiled language anyway? Job done … next post … no, wait.

Firstly, let’s be clear, Java isn’t a JIT-compiled language, otherwise what’s javac for?! It (usually) has JIT-compiled bytecode, and that makes it a useful tool for this endeavour, but that doesn’t get us from source code to running code. And anyway, there’s far more to a useful just-in-time programming system than just getting code running. Let’s be clear about some requirements here.

we want a system that allows us to write and rewrite code as it’s running, at any point in the lifecycle (no development vs production distinction)
we want a system that allows us to replace code atomically at the right time in order to provide glitch-free updates during real-time processing
we want a system that allows us to easily and correctly manage resources added or removed at runtime
we want a system that handles errors gracefully and carries on running while we fix the problem
and, we want a system that can persist its current state so that we can carry on running from where we left off at at a later date.

Code as Data

PraxisCORE is a forest-of-actors runtime for real-time programming, designed to bring aspects of Erlang, SmallTalk and Extempore into the Java world. Like Erlang, it allows for the code of individual actors to be hot replaced at runtime. Unlike Erlang, and as far as I know unique amongst at least Java actor frameworks, is that actors accept their new behaviour as a message containing Java source code. This is great for providing SmallTalk like introspection and persistence, ensuring formatting and comments remain part of the persisted image. It also gives us a form of code as data - not full-on Lisp like homoiconicity, but still giving us powerful metaprogramming capabilities.

Got code, now what? No right of REPL.

Some people who’ve seen PraxisLIVE in action assume the runtime uses a form of hot code reloading like the JVM’s built-in agent based HotSwap mechanism; others that it’s using JShell, the Java REPL. The truth is that neither are suitable or can meet all of the requirements outlined above.

Java’s built-in HotSwap mechanism allows for reloading classes at runtime. However, it is limited to changes to code inside method bodies, without allowing for structural changes such as adding/removing methods, fields, or (often) lambdas. There are a number of projects that remove some of those limitations, such as JRebel and HotSwapAgent/DCEVM, but they’re a bit hacky, and in the latter case require a patched JVM. They offer some utility in development, but not the level of stability we’re looking for. And there’s part of me that thinks that if it’s not stable enough for production use, it’s not really stable enough for development.

Reloading classes with these HotSwap mechanisms has other problems that don’t meet our requirements above. Firstly we need to control exactly when the code change occurs (not in the middle of processing something), and hook in to the code change to correctly manage resources that are added or removed. But another problem is handling our requirement for a system that can persist its current state. It is too easy with this technique to remove code that is required, such as code that initializes a field. You get to a point where the project is running great, but when you stop and re-run it, it all goes boom!

So, the PraxisCORE runtime eschews class reloading, in fact for something much simpler. It’s actually not hard to compile and execute new bytecode in a running Java application by using a compiler interface and a custom classloader, and there are various applications that do this. A (perceived) downside of this approach is that a class in Java is defined not only by its name, but also the classloader that loads it. Using this approach we are not changing an existing class but creating a whole new one. But by careful use of our actor framework, and other common Java patterns like dependency injection, we can turn this limitation to our advantage in delivering our requirements.

The JShell Java REPL also bears some similarity with how PraxisCORE works, including similar contextual wrapping of Java source code in a class context with additional imports. However, PraxisCORE does not use this either, and not only because it predates JShell! A problem with REPLs is that they embed an iterative design process, and potentially the only way to get back to a guaranteed state is to rerun the entire code history. The PraxisCORE runtime and API is designed such that code replacement is transactional, where the code and data at any point forms the entire picture.

Behaviour | State | Identity

A standard Java object generally combines three things - behaviour, state and identity. A re-codeable actor in PraxisCORE needs to breaks these things apart, with a separate class for each.

CodeDelegate is the base class that the user’s code (automatically) extends, providing the behaviour aspect of actor. There are different CodeDelegate base classes for most base actor types, providing access to useful functions and fields, as well as optional or required method hooks (eg. init() and update()).
CodeContext is the class that manages state for our actor. There are currently different CodeContext subtypes for each CodeDelegate subtype, and an instance of a CodeContext wraps each CodeDelegate instance. The CodeContext connects annotated fields and methods in the CodeDelegate to the wider actor system, injecting state and managing resources across context changes and for persistence. NB. this means that although the CodeContext manages state, the schema for that state along with internal updates are controlled by the user’s code.
CodeComponent is the simplest of the three, and provides the identity functionality. It is the one class that exists throughout the lifecycle of an actor. It wraps a CodeContext and a mechanism for replacing it, but otherwise delegates all aspects of the component (actor) interface to the installed context. Unlike the other classes, there is currently only one type of CodeComponent used for all actors.
CodeConnector is an auxillary class, again currently one per CodeDelegate type, that separates out the functionality of introspecting on the user’s code and writing the “recipe” for the CodeContext.

The above allows a basic core:custom actor to accept the following simple snippet of code and transform itself into an actor capable of adding a value to numbers it receives, as well as automatically persisting that value -

@P(1) double value;
@Out(1) Output out;

@In(1) void in(double x) {
    out.send(x + value);
}

Full information on writing custom components (actors), and all the base types and annotations available, is at https://docs.praxislive.org/coding/

Compiler as a service

The PraxisCORE actor framework provides a service discovery mechanism, by which actors can request the addresses of other actors providing particular services. When an actor receives new source code, it does not handle turning this into a CodeContext itself. This is definitely not a real-time safe operation! Instead it looks up a service that can instantiate a CodeContext for it, based on the provided source code and base type.

If the CodeContext service does not have previously compiled bytecode for this source and base type, it may look up another service that provides access to the Java compiler. The compiler service was split out from the CodeContext service in PraxisLIVE v3 so that the compiler part can optionally be run in a separate process, or even on a different machine, using the distributed hubs functionality. This is particularly useful where garbage collection is an issue for low latency pipelines.

The CodeContext service handles all introspection on the user’s code, and a possible previous iteration, which may include instantiating necessary resources or creating the instructions to migrate from one iteration of code to the next. A key benefit of not changing the existing class (eg. HotSwap) is that we can achieve all this in the background while the existing code continues to execute.

When the CodeContext is ready, it is returned within an actor system message. The context is already in such a state that it can be atomically replaced inside the CodeComponent with minimal overhead, even in a low-latency pipeline (eg. real-time DSP). And despite the various different services involved, possibly across processes, the efficiency of the messaging system and the use of a memory-based filesystem for the compiler (source and bytecode never on disk) means that the time between enacting a change in the editor and that change being apparent is almost imperceptible.

Conclusion

Hopefully this has been a useful brief(ish) exposition of how PraxisCORE achieves its goal of being a just-in-time programming runtime. If you’ve got any questions, jump on the mailing list, chat or otherwise send me a message.

Otherwise, go have fun with it or come get involved …

… and donations (Donorbox with PayPal or card) are always appreciated!

A Forest of Actors

Wed, 01 Aug 2018 00:00:00 +0000

The recent release of PraxisLIVE v4 saw a focus on relicensing the PraxisCORE runtime, and promoting its standalone usage as a platform for media processing, data visualisation, sensors, robotics, IoT, and lots more. PraxisCORE is a modular runtime for cyberphysical programming, supporting the real-time coding of real-time systems. Absolutely essential to this is its asynchronous, shared-nothing message-passing system - a forest-of-actors evolution of the standard actor model with the needs of real-time processing in mind.

What’s the problem?

While the PraxisCORE architecture can be used for processing any type of data, it was initially designed with the needs for working with multiple real-time media streams in mind. For example, my creative work often involves working with audio and video together, or a low frame-rate camera doing motion detection and controlling a high frame-rate projection. Sometimes you might be working with video working at 60 fps and audio processing at 750 fps (PraxisCORE uses an internal 64 sample buffer). How do we process events in those pipelines and allow them to talk to each other without interfering with each other? Now add in the principle of edit everything live where we want to be able to insert new components, live compile code, or load resources, all without pausing or glitching.

Solving this problem was probably my initial reason for developing PraxisCORE. There’s some great software out there for working with audio or with graphics / video. However, there’s not a lot that I consider works well with both at the same time. For example, I think Processing is a great project - the live video coding API in PraxisCORE is based on it for a reason! But as a system for working with multiple media it is frustratingly lacking. This is not due to a lack of libraries for audio, etc. but because the underlying architecture is fundamentally tied to a single frame-rate based model.

What’s wrong with synchronization? … Everything!

Working with different media in this way is analogous to working with threads. In fact, in most cases this is exactly what we’re doing. Now a common way of handling communication between threads in a Java program is to use a synchronized block, or other form of locking. Unfortunately, these are terrible ways of dealing with this problem.

Time waits for nothing

There’s a great article by Ross Bencina (author of AudioMulch) called Real-time audio programming 101: time waits for nothing. It discusses in some detail the reason not to use mutexes and other common threading constructs when programming real-time audio. I’d consider it a must-read for anyone doing audio coding, and useful for anyone working with real-time requirements.

Remember your media thread is for processing media! You need to be aware of all the code that is running in that thread, and the maximum potential execution time of that code, to ensure your data is processed in time. If you start adding in locks, then besides the overhead of synchronization and putting your code at the mercy of the scheduler, you add in the possibility of contention with other code that is now effectively running in your real-time thread but might not be able to meet your real-time deadlines.

Optimization

There’s another problem with locks, and that is working out how coarse-grained they should be – how much code they should cover. You might expect that for best performance you’d want to keep the lock as fine as possible, only covering the minimal required code in an individual component. Besides being a bad idea as mentioned above, it also inhibits a more important performance optimization that happens in the PraxisCORE code. Audio, video and data pipelines switch off processing of sections of the graph where they can prove the output is not required. To achieve this, they need to be sure that changes can only happen at a particular point of each processing cycle. However, to achieve this using mutexes would massively increase the chances of contention.

Atomicity

Another problem with synchronization is one of misapprehension, but one I’ve seen on multiple occasions. Imagine for a moment you have an audio sample player type, and on that type is a synchronized play() method. From your control thread (maybe Processing’s draw() method) you call play, and the sample plays fine and all is well. Now imagine you decide to play 4 samples together, so you call play() 4 times in succession and all is well .. actually, no it isn’t, because there’s no guarantee that those 4 samples will play together. This is another issue with using locks – how can the user ensure that a series of events happen atomically?

The PraxisCORE architecture

One (probably the) way to deal with the issues above is to pass events into your media process using a non-blocking queue or ring-buffer. I’ve written a variety of code in the past that used variations on these methods, and I’ve used a variety of other “non-blocking” concepts (such as Futures) for dealing with resource loading, etc. However, mixing different techniques for different requirements can be problematic! PraxisCORE began life as an experiment to build an architecture based entirely around a single, uniform message-passing model. It is inspired by the actor model, with some notable divergences.

A forest not a tree

Components (actors) in PraxisCORE are arranged in trees. Each component has an address in a familiar slash-delimited format (eg. /audio/filter ). There is no single root - / has no meaning. Instead, the first element of the address refers to the root component, of which there can be many - a hub of roots creating a forest of actors. Roots generally, but not always, encompass a thread of execution.

Diverging from the typical actor model, components do not maintain their own queue for incoming messages. Mailbox functionality is handled by each root, with external messages being delivered through a single lock-free queue, putting the root component in charge of all scheduling. Messages also have a time stamp. Each root maintains a clock relative to an overall hub clock, ensuring messages are dispatched at the correct time.

Each component has zero or more controls, which act as end points for messages and are addressed in a form that echoes function calls (eg. /audio/filter.frequency). Controls may wrap simple properties and actions, or complex functions that set off a chain of further messages. PraxisCORE adheres to a principle of no shared state between components, and control call communication uses a simple immutable type system. Even the code of each component is backed by a control - source code received as data to a component sets off a chain of messages that compile the code and inject the new behaviour atomically.

As well as controls, components can have input and output ports, sometimes reflecting the same data. Ports provide a lightweight way for sibling components to communicate, and it is port connections that are reflected in the visual dataflow editor. Port communication itself is synchronous, although components can create or delay onward port messages, or use them to trigger asynchronous control calls. Port communication uses the same immutable type system, but extends this with the ability to pass mutable data such as audio buffers and video frames. This provides the ability to process real-time data in a lightweight and lock-free way across a series of components, while maintaining the strong encapsulation of the actor model.

Here you can see an example of an audio:player component from within the PraxisLIVE editor. Ports can be seen on the graph node itself, and controls within the open editor dialog. Note the correlation between them. Manipulating values within the editor dialog sends control messages to the component. Note also the ready and error ports - because loading a sample file is not real-time safe, the sample port triggers an asynchronous control message, with port messaging continuing on response.

At your service

Alongside an unlimited number of user root components, there are various system-defined roots. Here you can see the Hub Manager inside the PraxisLIVE IDE with the system roots toggle active. Some of these roots provide services such as the compiler or resource loader. The runtime is modularized and decoupled, and a service discovery mechanism is used by components to find the addresses of services they require. Another of these system roots might be the PraxisLIVE IDE itself. The IDE can only communicate with the runtime via message passing, and other components have no knowledge of the IDE, making it easy to disconnect it, or swap in another editor that represents the underlying model in a different way.

Distributed root hubs

A key advantage of this architecture is its ability to work transparently with roots running in separate processes, literally at the flick of a switch. This makes it possible to edit code and dataflow across multiple machines. It also allows a project to run across multiple local processes, perhaps to improve real-time performance by mitigating garbage collection effects across pipelines. Or to improve resilience by automatically restarting a root in case of failure.

In a Pcl

While it is perfectly possible to control the PraxisCORE system from Java code using callbacks, in a continuation passing style, the recommended way to interact with the system is via the PraxisLIVE IDE. This provides live visual editing of the actor graph, as well as strong introspection facilities. And it supports the editing of code fragments stored in controls on components, with full syntax highlighting and completion (based on top of the Apache NetBeans IDE).

PraxisCORE also has a simple text based command language. The syntax of this is loosely based on Tcl - and if that is pronounced tickle then Pcl must be Pickle! It is the language used inside the Terminal component in the PraxisLIVE IDE, and also the syntax of all the files used in projects.

Two common commands, with symbolic aliases, are @ (at) and ~ (connect). @ takes a component address, an optional component type (used to construct a new component) and a script (that will be run within the context of the address, allowing nested hierarchies. This simple language hides the complexity of working within the PraxisCORE message-passing system, the interpreter automatically handling the sending and receiving of messages.

This is a simple script used to create a simple graph identical to the Hello World example in the examples download – it is similar to the the contents of the video.pxr file from the graph editor. There are actually 5 different roots (threads) involved in the execution of this script, potentially split across up to 3 processes!

@ /video root:video {
    .width 400
    .height 300
    .fps 20

    @ ./noise video:source:noise {}

    @ ./image video:still {
        .image [file "resources/hello world.png"]
    }

    @ ./window video:output {
        .title "Hello World!"
    }

    ~ ./noise!out ./image!in
    ~ ./image!out ./window!in
}

/video.start

In practice in Praxis

Praxis (from Ancient Greek: πρᾶξις, translit. praxis) is the process by which a theory, lesson, or skill is enacted, embodied, or realized. (Wikipedia)

What began life as an experiment into a message-passing actor system for real-time use has now matured into a robust and resilient runtime. The initial complexity of building a general purpose system in this way has paid dividends over the years in the ease of working with almost any library on the JVM. And with lock-free messaging and strong encapsulation it has proven to be an ideal and natural architecture for supporting real-time hot code reloading in Java.

Come get involved in the next exciting steps for PraxisCORE, bringing the best aspects of Erlang, Smalltalk and Extempore into the Java world.

This post is based on a much earlier 2012 blog post at The Influence of the Actor Model, updated (and abridged, honestly!) with snippets from my ICLC conference paper on PraxisLIVE and other more recent aspects of PraxisCORE.

Welcome!

Fri, 22 Jun 2018 00:00:00 +0000

Welcome to the new PraxisLIVE blog, now hosted as part of a revamped website. The old WordPress blog will be left as an archive but no new posts added - some old but still relevant information will be updated and reposted here.

This coincides with the recent release of PraxisLIVE v4, and the restructuring and relicensing of the PraxisCORE runtime, which now has its own dedicated page. Separate pages have been added for downloads and community, with links to the relatively new Gitter chat and DonorBox donations. And the documentation has been re-hosted and given a new UI at https://docs.praxislive.org.

There are lots of exciting things in the pipeline for PraxisLIVE over the summer months so please keep an eye here for updates!

PraxisLIVE

OPENRNDR in PraxisLIVE

Project definition

Custom root graph

Driver proxy

Data pipes

Custom component registration

Next steps

On Watch

Defining Watches

Async responses

Chaining, queries and future moldability

PraxisCORE / PraxisLIVE v6

Key changes

Root templates .. or where’s my OSC, MIDI, TinkerForge support?

Towards introspection and moldability

ClassLoader trees

Sharing code

A ClassLoader tree

Compiler and context service rewrites

A note on Data

Inside the PraxisLIVE IDE

Next steps

PraxisCORE / PraxisLIVE v5

Radical changes

Rethinking coding

2020 Vision

Objects in the rear view mirror

Jump to light speed

Number 5’s alive(-ish)

Just-in-Time Programming

Code as Data

Got code, now what? No right of REPL.

Behaviour | State | Identity

Compiler as a service

Conclusion

A Forest of Actors

What’s the problem?

What’s wrong with synchronization? … Everything!

Time waits for nothing

Optimization

Atomicity

The PraxisCORE architecture

A forest not a tree

At your service

Distributed root hubs

In a Pcl

In practice in Praxis

Welcome!