Table of contents
September 8, 2024

Using D3.js and LiveView

Scheduler Utilization

In this article, I would like to explain how to use D3.js and LiveView to display the utilization of the Erlang schedulers.

Load testing are particularly useful for servers in order to test the non-functional requirements (memory consumption, CPU utilization, network utilization) for specific and selected use cases.

A heavy one-sided load is generated on the server and the CPU load and/or IO throughut is monitored. For network-loaded applications, low CPU utilization and high output would be optimal. The metric selected depends heavily on the application itself. But basically the concept of the load testing remains the same.

The utilization of the scheduler can be used to record the CPU utilization. At the same time, IO values can be queried. The Erlang-BEAM provides these information.

Normally you would use the observer to monitor the utilization of the scheduler. To do this, the applications :wx, :observer would have to be started. In some environments this is not possible. There it would be possible to establish a connection via the network and start the Observer locally. However, this is also not possible in some environments for configuration or security reasons.

What can we do?

We can enable the LiveDashboard. The LiveDashboard includes a similar range of functions as the Observer but unfortunately lacks scheduler utilization in real-time.

We can extend the LiveDashboard or write a simple LiveView to display the values. In this article, we will use D3.js and LiveView to display the utilization and IO activities:

  • Collecting the data
  • Updating the LiveView
  • Displaying the data with D3.js

Collecting the data

We can use the :scheduler.sample_all/0 and :erlang.statistics/1 functions to collect the data as follows:

def statistics(sample_duration \\ 1_000) do
  sample = :scheduler.sample_all()
  Process.sleep(sample_duration)
  utilization = :scheduler.utilization(sample, :scheduler.sample_all())

  io =
    case :erlang.statistics(:io) do
      {{:input, input}, {:output, output}} ->
        %{input: input, output: output}

      _other ->
        %{input: 0, output: 0}
    end

  %{io: io, utilization: utilization}
end

The result will look like:

%{
  io: %{input: 91010087025, output: 75252498056},
  utilization: [
    {:total, 0.030731203908744342, ~c"3.1%"},
    {:weighted, 0.061462407817488685, ~c"6.1%"},
    {:normal, 1, 0.06239245657016214, ~c"6.2%"},
    {:normal, 2, 0.0681790067151196, ~c"6.8%"},
    {:normal, 3, 0.06160054551952141, ~c"6.2%"},
    {:normal, 4, 0.053677618626724946, ~c"5.4%"},
    {:cpu, 5, 0.0, ~c"0.0%"},
    {:cpu, 6, 0.0, ~c"0.0%"},
    {:cpu, 7, 0.0, ~c"0.0%"},
    {:cpu, 8, 0.0, ~c"0.0%"},
    {:io, 9, 0.0, ~c"0.0%"},
    {:io, 10, 0.0, ~c"0.0%"},
    {:io, 11, 0.0, ~c"0.0%"},
    {:io, 12, 0.0, ~c"0.0%"},
    {:io, 13, 0.0, ~c"0.0%"},
    {:io, 14, 0.0, ~c"0.0%"},
    {:io, 15, 0.0, ~c"0.0%"},
    {:io, 16, 0.0, ~c"0.0%"},
    {:io, 17, 0.0, ~c"0.0%"},
    {:io, 18, 0.0022569613371039242, ~c"0.2%"}
  ]
}

The utilization contains a list of different tuples. The tuples containing :normal and :cpu are the interesting values. The :normal contains the utilization of one scheduler and the :cpu contains the utilization of a dirty scheduler. Each scheduler has an ID and this is the second value of the tuple.

Data flow

Before we look at the code fragments, I would like to discuss the data flow first.

A separate process queries the values for IO and utilization in an interval of 200ms and sends this via Phoenix.PubSub.broadcast(:superlist_pubsub, "scheduler", {"scheduler", ts, statistics}). The LiveView subscribes to the topic schedulerand implements the handle_info/2 function. There the data is converted into a JSON packet and sent to the LiveView or the JavaScript implementation, which then updates the SVG.

As the IO values are counters, we want to display the rate of change per interval and therefore create the difference between the previous and the current IO value. This is done using the diff_io/2 function.

The handle_info/2 function looks like this:

def handle_info({"scheduler", ts, %{utilization: utilization, io: io}}, socket) do

  dirty_schedulers =
    utilization
    |> Enum.filter(fn
      {type, _id, _value, _ignored} -> type == :cpu
      _other -> false
    end)
    |> Enum.map(fn {_type, id, value, _ignored} -> {id, value * 100.0} end)
    |> Map.new()

  normal_schedulers =
    utilization
    |> Enum.filter(fn
     {type, _id, _value, _ignored} -> type == :normal
      _other -> false
    end)
    |> Enum.map(fn {_type, id, value, _ignored} -> {id, value * 100.0} end)
    |> Map.new()

  socket =
    socket
    |> push_event("add-utilization", %{ts: ts, normals: normal_schedulers, dirties: dirty_schedulers, io: diff_io(socket, io)})
    |> assign(:io, io)

  {:noreply, socket}
end

The diff_io/2 function:

defp diff_io(%{assigns: %{io: nil}}, _io) do
  nil
end

defp diff_io(%{assigns: %{io: io}}, %{input: input, output: output}) do
  %{input: input - io.input, output: output - io.output}
end

Now let’s take a look at the JavaScript implementation. The complete implementation can be found in this Gist.

The hook is first defined for communication between the LiveView process and the JavaScript implementation.

const Utilization = {
    mounted() {
        this.utilization = init(this);
        this.io = initIO(this);
        this.handleEvent("add-utilization", (values) => {
            this.io(values);
            this.utilization(values);
        })
    },
    updated() {
    ...
    }
};

export default Utilization;

The HTML part looks like this:

<div class="border-2 p-2" id="statistics" phx-hook="Utilization">
  <svg style="height: 600px" id="utilization-graph">
  </svg>
  <svg class="mt-3" style="height: 600px" id="io-graph">
  </svg>
</div>

If we now call up the page in our browser, the hook is found and the mounted() function is called automatically.

In this function we initialize the SVG element and the D3.js graph. The init(this) returns a function which is used to update the graph if we receive new values. To receive new values we add a new event handler for add-utilization.

The code fragment shows only the parts for the normal schedulers. The code is identical for the dirty-schedulers.

We define the dimensions of the scaling functions for x and y.


function init(self) {
  let svg = d3.select("#utilization-graph");
  const width = 900;
  const height = 400;
  const marginRight = 30;
  const marginLeft = 60;
  const marginTop = 20;
  const marginBottom = 80;

  const x = d3.scaleTime(extentTime(), [marginLeft, width - marginRight]);
  const y = d3.scaleLinear([0, 100], [height - marginBottom, marginTop]);

We store the current values in the normals map and define the line generator:

  // utilization values for the normal schedulers
  let normals = {};

  // Declare the line generator.
  const line = d3.line()
   .x(d => x(d.ts))
   .y(d => y(d.utilization))
   .curve(d3.curveCatmullRom.alpha(0.5));

Then we create the SVG container, followed by the x- and y-axis:

  // Create the SVG container.
  svg = svg
   .attr("width", width)
   .attr("height", height)
   .attr("viewBox", [0, 0, width, height])
   .attr("style", "max-width: 100%; height: auto; height: intrinsic;");

  // Add the x-axis.
  svg.append("g")
   .attr("class", "x-axis")
   .attr("transform", `translate(0, ${height - marginBottom})`)
   .call(d3.axisBottom(x).ticks(width / 80).tickSizeOuter(0));

  // Add the y-axis, remove the domain line, add grid lines and a label.
  svg.append("g")
   .attr("transform", `translate(${marginLeft},0)`)
   .call(d3.axisLeft(y).ticks(height / 40))
   .call(g => g.select(".domain").remove())
   .call(g => g.selectAll(".tick line").clone()
     .attr("x2", width - marginLeft - marginRight)
     .attr("stroke-opacity", 0.1))
   .call(g => g.append("text")
     .attr("x", -marginLeft)
     .attr("y", 10)
     .attr("fill", "currentColor")
     .attr("text-anchor", "start")
     .text("Utilization in %"));

We add a group for displaying a simple legend, containing colored boxes for each scheduler:

  // add the group for the legend normal schedulers
  let normalLegend = svg.append("g").attr("transform", `translate(${marginLeft}, ${height - 50})`);

  // Append a path for the utilization lines.
  let graphView = svg.append("g").attr("class", "graph-view");

This function adds new values for the normals. We remove values older than one minute from normals. By changing the normals object we animate the SVG graph. For each scheduler the normals contains an array of the utilization values. The filterDomain() function removes old values from the array.

  function addUtilization(data, ts, values, domain) {
    if(values != null) {
      for (const [key, value] of Object.entries(values)) {
        let e = {ts: ts, utilization: value};
        if(data[key] === undefined) {
          data[key] = [e]
        }
        else {
          let normalValues = data[key];
          normalValues.push(e);
          data[key] = filterDomain(normalValues, domain[0], domain[1]);
        } // else
      }
    }
  }

Now we define the update function which is the result of the init method. This function creates a new domain for the x-axis and forces a redraw. Then we map the values of the normals object to a simple array normalSchedulerValues. We use the array to update the lines.

  function update(values) {
    let ts = new Date(values.ts);
    let domain = extentTime(new Date(values.ts));
    addUtilization(normals, ts, values.normals, domain);

    svg.select(".x-axis").call(d3.axisBottom(x).ticks(width / 80).tickSizeOuter(0));

    let i = 0;
    let normalSchedulerValues = [];
    for (const [key, values] of Object.entries(normals)) {
      normalSchedulerValues.push({index: i, id: key, values: values})
      i += 1;
    }

The join method accepts the three functions for adding, updating and removing values. enter is used to add new values. Here, we add a new path using the values of the schedulers. Since the scheduler won’t change, it is done only once.

The update function updates the d attribute with the new values from the array. This automatically animates the graph.

The exit function describes what to do when the graph is removed. In our case, this will not happen. For the sake of order, we define that the graph should be removed.

    graphView.selectAll('.normal-scheduler').data(normalSchedulerValues, function (d) { return d.id; })
    .join(
      enter => enter.append('path')
                  .attr('class', 'normal-scheduler')
                  .attr("fill", "none")
                  .attr("stroke", (d) => normColors[d.id % maxColors])
                  .attr("stroke-width", 0.5)
                  .attr('d', (d) => {return line(d.values);}),
      update => update.attr('d', (d) => {return line(d.values);}),
      exit => exit.remove()
     );

    normalLegend.selectAll('rect').data(normalSchedulerValues, function (d) { return d.id; })
      .join(
          enter => {
            enter.append('rect')
              .attr("fill", (d) => normColors[d.index % maxColors])
              .attr("x", (d) => d.index * 40)
              .attr("y", 0)
              .attr("width", 15)
              .attr("height", 15);
              enter.append("text")
                .attr("x", (d) => d.index * 40 + 18)
                .attr("y", 12)
                .attr("class", "text-xs")
                .text(function(d) { return d.id; });
              },
          update => update,
          exit => exit.remove()
         );
  }

  return function(data) {return update(data)};
}

The init function now returns a function, which in turn calls the local update function with the new data.

How are the SVG elements found for certain data? We can use the ID of the scheduler for this. The data function ensures that we tell D3.js how the elements should be identified.

graphView.selectAll('.normal-scheduler').data(normalSchedulerValues, function (d) { return d.id; })

Let’s go back to the JavaScript hook. If the LiveView sends new data using push_event(socket, "add-utilization",...), we call the update function stored in this.utilization with the new data. The update function appends the new values and maybe removes old values. Then the graph is updated.

const Utilization = {
    mounted() {
        this.utilization = init(this);
        this.handleEvent("add-utilization", (values) => {
            this.utilization(values);
        })
    },
    ...
};

Conclusion

We have learnt how to connect D3.js and of course other JavaScript packages with LiveView via JavaScript interoperability.

The implementation of hooks in particular makes it easy to organize communication between client and server with many options.