Creating animation process toolset with generative AI

Animation
I developed a process for creating animations, using Claude 3.7 to code the tools. It worked well, but now I’ve thought of a better way to do it.
Published

March 28, 2025

I created the animation above using the techniques described below

TL;DR

  • I used generative AI to create a number of tools to create animations from my drawings. The main animation technique was to record mouse path movement.
  • It worked! (see above)
  • One of my biggest realizations in doing this was that using Generative AI to create interfaces (React forms) to enter data is a bit of a waste of time as you need to keep modifying the forms every time you want to make a change. Having a good YAML configuration file eliminates the need for an interface.
  • Using nodejs to render video frames is neat. Before I was doing it with Python, but nodejs makes much more sense and is much faster. I think there are a lot of possibilities for doing cool stuff with this.
  • The animator tool would be better with two config files, one that defines the assets, and the other that defines the animation.
  • Having done this, I have thought of a better and more flexible way to do it, so that will be my version two.

Background

On this website you can see that various of my writing have illustrations that I have drawn myself. I have developed my own techniques to do this (basically drawing the line work on paper, scanning and coloring them with Procreate, and then a bit of tweaking with Photoshop and clop for the final result). Now I want to create animated versions of these drawings.

I tried to do this with Procreate Dreams, but found it frustrating to use. So I spent a bit of time looking at [other tools]{2D animation techniques}, but I didn’t find anything that does what I want. I want something where I can define the animation in a script or editable file, so that I can use Generative AI to help create the movements or generate procedural movements. I also want tools to help me position things and do animations by recording mouse movement. So the obvious solution is to make my own tools. That’s the great thing about coding with AI, you can attempt to do things that would take too long otherwise.

Programming with AI works best if you break down what you want to do into steps, and then get the AI to address each step individually. In this case, I think I want three different tools. Here is a breakdown of my process and the required tools in bold:

  1. Create the line art with pen and paper.
  2. Photograph it, neaten it up in Photoshop, cut up in to layers with each layer being a part that can be separately animated. Group layers that will move together (e.g. the forearm, upper arm and hand will form part of a group, which is part of a larger group “person” containing all the other bits).
  3. Colour the parts using Procreate and export as a Photoshop file.
  4. In Photoshop again, arrange all the parts how I want the to appear in the animation. Make sure the layer and group names are unique and include the center point (for rotation) where appropriate, e.g. “Cloud1 801,275”
  5. Convert the PSD with a Python script, that creates a folder of PNG images for each layer, plus a JSON file containing the names, layer order, and rotation points. python create-json.py "./input"
  6. Record mouse paths with the mouse path recorder.
  7. Use the animation configuration tool to modify the output of the Python script and add the mouse path movements.
  8. See the results in the animation player.
  9. Convert the animation into frames for video using the animation exporter.

Let’s look at each step individually.

The line work

For the test animation I wanted a scene with various elements I could animate. This is what I drew:

Creating the Photoshop file

I then edited the photographed image in Photoshop, and imported it into Procreate to colour it on my morning commute:

Procreate can export as PSD, so I could then open this in Photoshop and sort out the Photoshop file to make it suitable for export. You can read about those two steps here:

The mouse path recorder

My drawings are obviously hand-drawn, a little bit crappy, but with their own appeal (I hope). I also want the animation style to have this slightly crappy, hand-done feel. I don’t want to do movement algorithmically, I want to do it “by hand”. So I want a tool that does a crude form of motion capture — the movement of the mouse. Then I can record multiple paths, and use these in different ways (for example, for position, but also for scale and rotation, or even opacity or whatever).

Requirements

  1. I should be able to configure:
    • Recording length (default 20 seconds).
    • The height and width of the stage (default 1920 by 1080).
    • The sampling rate (default 30 frames per second).
    • Displayed grid size (default 100).
    • Displayed grid divisions (default 5).
  2. Whilst recording the path, the recorded points should be displayed.
  3. Once the recording is finished, it should ask for a filename, and then save the path in an appropriate format.

After some coding with Claude 3.7 Sonnet I ended up with this:

View this in a new window.

Fairly neat I think. Alter the settings on the RHS panel, the press space to start recording.

The animator

So our initial data is coming from a JSON config file, so we can use react-jsonschema-form for the form interface, as that is a very established and active project. Firstly I made a JSON file with all the fields I expect to use in the animation tool. I used the output we created in the previous step, and add the missing fields to one of the elements. This is what I ended up with (shown here is the :


[
  {
    "group": "Solar system",
    "offsetX": 0,
    "offsetY": 0,
    "centerX": 3000,
    "centerY": 3000,
    "rotation": 0,
    "scale": 1,
    "movementPath": "",
    "ScalePath": "",
    "rotationPath": ""
    "children": [
      {
        "group": "Planet and moon",
        "centerX": 954,
        "centerY": 527,
        "children": [
          {
            "image": "moon.png",
            "centerX": 593,
            "centerY": 413
          },
          {
            "image": "planet.png",
            "centerX": 593,
            "centerY": 413
          }
        ]
      },
      {
        "image": "Sun.png",
        "centerX": 954,
        "centerY": 527
      }
    ]
  },
  {
    "image": "stars.png",
    "offsetX": 0,
    "offsetY": 0,
    "centerX": 0,
    "centerY": 0,
    "rotation": 0,
    "scale": 1,
    "movementPath": "",
    "ScalePath": "",
    "rotationPath": ""
  }
]

After some messing around I ended up with the below. This is really far too large to display in this page, so click the link below to see it in a new window. Then click the “load demo” button to see it in action.

View this in a new window.

Having got this far I realized that having the form interface to edit the config file was a hinderance, because it meant that each time I wanted to add a feature, the form needed to be updated, and it was much quicker and easier for me to just edit the JSON config file directly. So the final thing I did was to create a stand-alone animation tool that just displayed the animation, and another that uses nodejs to write each frame as a PNG file. I could then join the frames together into a video using the functionality built-in to the Quicktime player.

Overall this was very satisfying, and I’ve learnt some important lessons:

  • Getting AI to create React forms is very easy, but ultimately you end up with a huge amount of code that doesn’t really contribute much to your end goal. Having a big complex form to edit a JSON config file is a bit pointless when you can edit the file directly.
  • Using Nodejs to render frames is great! Lots of possibilities here. I need to experiment more.
  • The animator tool would be better with two config files, one that defines the assets, and the other that defines the animation.