⌛ History

Some time back, in 2019, I published a small blogpost (in Dutch) about QBoxNext solution and how this was developed and deployed on Azure. However, since I got a new Smart Meter, this solution did not work anymore so I needed to find a new way of reading data from the Smart Meter and process and store this data. This blogpost describes the steps I’ve taken and technical challenges encountered when investigating a new solution and building a prototype using .NET.

1️⃣ P1

Here in the Netherlands, all Smart Meters are equipped with a P1 serial port. The port connector type is RJ-12 (“phone connector” with 6 positions and 6 contacts) port and the signal is inverted.

The table below shows the pin-reference:

Pin #Signal nameDescriptionRemark
1+5V+5V power supplyPower supply line
2Data RequestData RequestInput
3Data GNDData ground
4n.c.Not connected
5Data DataLine OutputOpen collector
6Power GNDPower groundPower supply line

The baud rate for the newer generation Smart Meters is 115200.

To read data using the P1, you need a USB Serial converter cable. Several ready made cables are available and look like:

Smart Meter USB Cable

2️⃣ Reading data from Serial Port in .NET

There are several solutions available in other programming languages (e.g. Python), but I wanted to build a prototype using in C# .NET because using .NET (Core) should work on Windows and Linux (including Linux ARM which is used by the Raspberry PI 4B, more on that later in this blogpost).

The first step is accessing the serial port (via the USB Serial Device converter) using C#.

There are some custom projects (e.g. RJCP.DLL.SerialPortStream) which I investigated, but in the end I just used the ’new’ System.IO.Ports to open a serial port and receive data.

Get all available serial ports on the system

var portNames = SerialPort.GetPortNames();

Open a port with a specific baud rate

Example code on a Windows system:

var serialPort = new SerialPort("COM5", 115200);
serialPort.Open();

On a Linux system, the same NuGet (System.IO.Ports) can be used, but the port name differs:

var serialPort = new SerialPort("/dev/ttyUSB0", 115200);
serialPort.Open();

Reading data

The data which is sent by the P1 Smart Meter follows a specific protocol (more on that later) where each request line contains data.
And multiple lines combine to a Telegram.

Reading a single line using the SerialPort-class is straightforward, just use the ReadLine() method:

while (serialPort.ReadLine() is { } line)
{
    _logger.LogDebug(line);

    if (line.StartsWith('/'))
    {
        // handle the start from the Telegram
    }

    if (line.StartsWith('!'))
    {
        // handle the end from the Telegram
    }
}

ℹ Note that the ReadLine() method actually removes the LF (\n) character when the line is read, so when combining all the lines to a valid DSMR Telegram, I need to append an extra \n to each line.

3️⃣ Dutch Smart Meter Requirements Protocol

The protocol is based on NEN-EN-IEC 62056-21 Mode D.
Data transfer is requested with a request line and automatically initiated every second until request line is released.

Protocol

The protocol contains 3 main parts: protocol

  1. The start character (/) + Smart Meter identification id (e.g. CTA5ZIV-METER) and this line ends with two CRLF-characters.
  2. The Data Lines: multiple COSEM (Companion Specification for Energy Metering) objects are encoded using OBIS (Object Identification System) and each line ends with a CRLF-character.
  3. The end character (!) + CRC16
    The CRC16 value calculated over the preceding characters in the data message (from / to !) and is using the polynomial calculation: x16 + x15 + x2 + 1).
    This CRC16 uses no XOR in, no XOR out and is computed with least significant bit first. The value is represented as 4 hexadecimal characters (MSB first).
    And this line does also end with the CRLF-character.

The transfer speed must be a fixed to 115200 baud. And the format of transmitted data must be defined as 8N1.

Namely:

  • 1 start bit
  • 8 data bits
  • no parity bit
  • 1 stop bit

Example message (Telegram) looks like this:

/CTA5ZIV-METER

1-3:0.2.8(50)
0-0:1.0.0(221228194418W)
0-0:96.1.1(**********************************)
1-0:1.8.1(000106.240*kWh)
1-0:1.8.2(000071.890*kWh)
1-0:2.8.1(000000.198*kWh)
1-0:2.8.2(000003.657*kWh)
0-0:96.14.0(0002)
1-0:1.7.0(01.867*kW)
1-0:2.7.0(00.000*kW)
0-0:96.7.21(00015)
0-0:96.7.9(00008)
1-0:99.97.0(1)(0-0:96.7.19)(220420171713S)(0000000000*s)
1-0:32.32.0(00000)
1-0:52.32.0(00000)
1-0:72.32.0(00000)
1-0:32.36.0(00005)
1-0:52.36.0(00003)
1-0:72.36.0(00004)
0-0:96.13.0()
1-0:32.7.0(231.0*V)
1-0:52.7.0(229.0*V)
1-0:72.7.0(230.0*V)
1-0:31.7.0(000*A)
1-0:51.7.0(000*A)
1-0:71.7.0(008*A)
1-0:21.7.0(00.000*kW)
1-0:41.7.0(00.074*kW)
1-0:61.7.0(01.792*kW)
1-0:22.7.0(00.000*kW)
1-0:42.7.0(00.000*kW)
1-0:62.7.0(00.000*kW)
0-1:24.1.0(003)
0-1:96.1.0(**********************************)
0-1:24.2.1(221228194000W)(00097.867*m3)
!4E60

ℹ Note that in the Telegram displayed above, the equipment identifiers 0-0:96.1.* are obscured because of privacy reasons.
More details on each OBIS field can be found in the specification.

4️⃣ Parsing the Telegram

For parsing the Telegram (which follows the COSEM definition and is encoded using OBIS) I’ve used a helper library: DSMRParser.Net.

Parse the Telegram (all lines)

var lines = "*** Example Telegram from above ***";
var parser = new DSMRTelegramParser();
if (parser.TryParse(lines, out var telegram))
{
    // Do something with the valid Telegram (e.g. trace, store this on disk / DB, etc).
}
else
{
    // In case the parsing fails, log this and take action accordingly.
}

For all properties which are defined in the Telegram Model, see the C# class.

5️⃣ Producer - Consumer pattern

In order to separate the reading of the data using the System.IO.Ports and parsing the Telegram using DSMRParser.Net I decided to prototype my solution using the Producer - Consumer pattern which really fits in this scenario.

The technology used to implement a Producer and Consumer is System.Threading.Channels.

A very simple implementation from a Producer - Consumer pattern (not related to P1 data or a Telegram) can be like:

class Program
{
    static async Task Main(string[] args)
    {
        var channel = Channel.CreateUnbounded<string>(); // 1

        var producer = new Producer(channel.Writer); // 2

        var consumer = new Consumer(channel.Reader); // 3
    }
}

class Producer
{
    private readonly ChannelWriter<string> _channelWriter;

    public Producer(ChannelWriter<string> channelWriter)
    {
        _channelWriter = channelWriter;
    }
}

class Consumer
{
    private readonly ChannelReader<string> _channelReader;

    public Consumer(ChannelReader<string> channelReader)
    {
        _channelReader = channelReader;
    }
}

In the above simple example a main method is defined to show you how the creation of the writer/reader happen 1.
The important things to note are:

  • for the Producer, I’ve passed it only a ChannelWriter, so it can only do write operations 2
  • for the Consumer, a ChannelReader is passed, so it can only read data 3

However this simple example cannot be used for reading P1 data and parsing Telegrams because the data is constantly send by the P1 Smart Meter and should be processed (read) line by line from the serial port. And when a complete Telegram is found, this Telegram should be verified and parsed and eventually stored (out of scope).

So the required building blocks for this solution are:

  • Producer
  • Consumer
  • Processor (which combines the Producer and the Consumer)

Producer : P1Reader

A class which needs to:

  • Read lines from the serial port
  • Assemble complete Telegram using these lines
  • Use the ChannelWriter<string> to write the Telegram to the Channel

Reference implementation for this P1Reader

internal class P1Reader : IP1Reader
{
    public async Task StartReadingAsync(CancellationToken cancellationToken = default)
    {
        var serialPort = _factory.CreateUsingFirstAvailableUSBSerialPort();

        serialPort.Open();

        await Task.Run(async () =>
        {
            while (!cancellationToken.IsCancellationRequested && serialPort.ReadLine() is { } line)
            {
                // Use a StringBuilder to combine all lines to a Telegram
                // If the line starts with '/', clear the StringBuilder
                // Else append the line (including an extra '\n') to the StringBuilder
                // If the line starts with '!', the Telegram has ended, write the Telegram to the write Channel
            }

            // Some housekeeping to complete the writer, close and dispose the serialPort.
        }, cancellationToken);
    }
}

For the complete file, see P1Reader.cs.

Consumer : TelegramParser

A class which needs to:

  • Keep reading from the ChannelReader<string> Channel
  • Verify and parse the Telegram
  • Forward the valid Telegram to a store (out of scope)

Reference implementation for this TelegramParser

The ReadAllAsync() method used in the reference code, as the name suggests, allows reading all the items from the channel. This returns an IAsyncEnumerable<string>, which means the items from the channel will be streamed / processed asynchronously.

internal class TelegramParser : ITelegramParser
{
    public async Task StartProcessingAsync(CancellationToken cancellationToken)
    {
        await foreach (var message in _reader.ReadAllAsync(cancellationToken))
        {
            if (_parserProxy.TryParse(message, out var telegram))
            {
                // Do something with the valid Telegram (e.g. trace, store this on disk / DB).
            }
            else
            {
                // In case the parsing fails, log this and take action accordingly.
            }
        }
    }
}

For the complete file, see TelegramParser.cs.

Combine : Processor

A class which needs to:

  • Create in instance of the P1Reader (using Dependency Injection)
  • Create in instance of the TelegramParser (using Dependency Injection)
  • Start reading using the P1Reader in a “Async Task - Fire and Forget” way
  • Also start processing the data using the TelegramParser in a “Async Task - Fire and Forget” way

Async Task - Fire and Forget

This means that the main process should start a new Task but should not wait on the result because the P1 port lines should be continuously read and also the data (Telegram) should be parsed continuously. However when exceptions occur (USB port does exists or reading serial data fails) these exceptions should be logged and if required, the total process should be stopped. To implement this requirement I’ve used a NuGet packaged called AsyncAwaitBestPractices. This package contains a useful extension method named SafeFireAndForget.

See example code below how this method is used.

Reference implementation for this Processor

internal class Processor : IProcessor
{
    public void Run(CancellationToken cancellationToken)
    {
        _reader.StartReadingAsync(cancellationToken).SafeFireAndForget(e =>
        {
            Log.Logger.Fatal(e, "Unable to read.");
            throw new Exception("Unable to read.", e);
        });

        _parser.StartProcessingAsync(cancellationToken).SafeFireAndForget(e =>
        {
            Log.Logger.Fatal(e, "Unable to parse.");
            throw new Exception("Unable to parse.", e);
        });
    }
}

For the complete class, see Processor.cs.

💻 Demo

Windows

Compiling and running the .NET 7 console application on Windows is straightforward, in the screenshot below you can see that COM5 is the detected USB Serial Device port. windows

ℹ Note that also the ThreadID is logged, so it’s visible which Consumer Thread handles the data received via the Channel.

WSL (Windows Subsystem for Linux)

Running the example application on WSL needs some more steps because by default the USB (serial) ports are not attached to the WSL instance.
Follow these steps:

  1. Install the latest version from usbipd on your Windows system

  2. To get a list from all the USB devices, run the command:

    1

    C:\Windows\System32>usbipd list

    On my system, the USB Serial Device converter adapter was located at bus id 1-4.

  3. Now attach that specific USB device to the running WSL instance:

    1

    C:\Windows\System32>usbipd wsl attach --busid=1-4

When all is configured correctly, you can just do a sudo dotnet run to start the example application.
Note that you need sudo because the USB port is accessed. wsl

ℹ Note that when you want to run the console application in Windows again, you need to detach the USB port from the WSL instance. See this page for more details.

Linux - ARM (Raspberry PI 4B)

It’s even possible to run the console app on a Raspberry PI 4B. You only need to install .NET SDK on the Raspberry Pi (link). And when that’s done correctly, just plug-in the USB serial adapter and run the example console application. pi

My complete example GitHub project can be found here.

💡 Conclusion

This blog post explains some details on the P1 Smart Meter protocol and describes my journey on how to build a prototype console application to read and parse data from the P1 Smart Meter using .NET and implementing the Producer - Consumer pattern using Channels.

❔ Feedback or questions

Do you have any feedback or questions regarding this or one of my other blogs? Feel free to contact me!

📚 Additional resources

comments powered by Disqus