Peter H. Thomas

DICOM and HL7 Packet Adventures

Published Jan. 29, 2025, 9:54 p.m. in 'Tech'

I work on a wide array of network-related (or network-connected) things for a lot of different types of customers. Our company has a number of medical practices as customers and I occasionally work with vendors on DICOM and HL7 issues. I normally don't do any of the actual work with those protocols; I am just responsible for the traffic getting where it needs to go. In this instance, though, the vendor wasn't helping and was pointing the finger at the network so I had to dig in.

Background

At this particular customer with multiple sites, we have a central server where SSH traffic comes in from a cloud-based EHR system and the EHR vendor makes tunnels to the PACS (medical imaging archive) servers at different locations to send HL7 traffic when patient appointments are created or canceled or patients have arrived for appointments. The PACS server vendor then processes the HL7 to generate a DICOM worklist for the x-ray and MRI systems so they don't have to key in patient and appointment information manually for each procedure. It was at the DICOM generation step where things fell apart.

The bug causing this issue had apparently been around for a while, but what uncovered it was the purchase of a new x-ray machine. This machine had worklist functionality like all the others at this customer, but at random intervals, the worklist would show up empty on this new machine. The other machines (MRI and other x-ray systems) would pull the list and search it without problems.

DICOM Digging

I worked with the x-ray equipment vendor who was working with Agfa technicians. My learning about DICOM starts here. Agfa support asked us for packet captures when running worklist queries from the x-ray machine. These queries are known as C-FIND requests. Wireshark has a decoder for DICOM packets in it and it places some information about the data in a packet in the Info column by default for us:

We can use this column to filter our packets by using something like _ws.col.info contains "C-FIND-RQ" to find C-FIND request packets. Otherwise, it's somewhat difficult to filter DICOM packets (we'll get to that in a bit).

Agfa asked us for more information from the packet captures because the queries looked correct. How did they know? In this instance, the C-FIND-RQ is accompanied by a C-FIND-RQ-DATA with the search terms included. In this case, the only search terms were for a "CR" modality type and studies scheduled for the date of 2024-09-16.

Side note: The numbers on the left in parentheses (e.g. (0008,0060)) are called tags. These numbers are stored directly in the packet data, so we can use this for filtering...sort of.

With some more information from Agfa, I was able to find the packets causing the problems. Agfa reported that they had seen this before at another customer with this same PACS vendor and they had documented what they had seen at that time, but the bug was still present on the PACS vendor side. The filter I used to identify these packets used the TCP payload field and a pattern of hex digits:

The empty "Scheduled Procedure Step Sequence" list (tag (0040,0100)) is the problem for the Agfa machines. The tcp.payload filter actually operates on the raw bytes contained in the packet rather than any interpretation of the data that Wireshark might do. You can see here that the filter box at the top correlates to the packet bytes of the selected empty Scheduled Procedure Step Sequence item. It has the tag numbers, though the bytes are transposed in the raw data (40:00:00:01 instead of as displayed in the packet details pane).

This tag value should have a date and time listed for the study appointment. All of the provider and patient information is listed above this in the redacted areas. Here is what this segment should look like:

Making progress?

Once we had found these items and confirmed with Agfa that this was similar behavior that another of their customers was experiencing, the task of working with the PACS and EHR vendors began. There didn't seem to be any pattern to when these issues arose. It just seemed random at first.

The PACS vendor had us bring in the EHR vendor to review HL7 messages that were sent in for a few good appointments and a few bad appointments. The PACS vendor took this information and basically just said that there weren't any problems on their end so there must be issues with something else. Since I controlled the SSH server that handles the HL7 traffic, I started a packet capture there logging all of the HL7 traffic by day.

For this I used tcpdump. You can have it write a date- or time-stamped file using the -G option with the -w option. The argument for the -G option is a number of seconds, how often to rotate the file, and the argument for -w is a file name, but it can also take strftime values for date formatting, so you could use something like

tcpdump -i enp0s1 -G 86400 -w 'hl7-%Y%m%d.pcap' 'port 5555'

to rotate every 24 hours generating a filename that looks like hl7-20240916.pcap, containing all traffic on port 5555.

HL7 Hefting

The HL7 messages that the EHR vendor was sending us over this interface are SIU_S12 (Create), SIU_S14 (Update), and SIU_S15 (Delete) messages. I had seen HL7 messages prior to this, but I also ended up being a liaison between the EHR vendor and the PACS vendor on this conversation because, even with my limited understanding, I could pick up on and correct mistakes that were being made. Back to learning...

HL7 messages are structured in segments, each basically a line in a text file. This isn't completely correct, but the details are mundane. Each segment has a type. These are the ones I cared about.

Segments are made up of fields. I was happy to find out that Wireshark can actually match on segments and fields to an extent without relying on tcp.payload filters. One bullet dodged...

So, here's where everything comes together. We take the patient ID from the DICOM worklist mesage where we have an empty Scheduled Procedure Step Sequence field, we'll use "331122" as an example. Then, in our HL7 capture file, we search for that value in one of the fields. We'll use hl7 && hl7.field == "331122" as our filter. At this point, we can see all the HL7 packets having to do with a particular patient ID.

With this filter, we can see four packets. We see two S14 messages and two S12 messages. The S14 messages are just updates to existing appointments. Looking more closely, in this case they are showing us that the patient was marked as ARRIVED and COMPLETED at the beginning and end of their appointment today.

The S12 messages are interesting, though. S12 messages are creating appointment records. The first is creating a follow-up appointment for 9/30/2024 (SCH Field 12).

This looks fine, but why is there another one the following morning at 6:00am? That's not in the practice's business hours. There shouldn't be any appointments created at that time of day.

This is for the same appointment date and time. And, looking at the MSH segment, field 10, we see that number is the same too. This is significant because that is the Message Control ID field. That is basically a serial number for that message. This message appears to have been resent. Based on prior experience, I know that HL7 responds and acknowledges receipt of these messages. If it's resent, that typically means that an error occurred and the message is being retried. Let's filter based on the Message Control ID field to see what shows up. The filter we can use for this is hl7 && hl7.field == "83469931", filtering on that field value. We can't filter on a specific field but this will work for what we need.

So, we have four packets that show up. The frame number for the first one, 2462, matches up with our initial SIU S12 that we saw above. We can see from the above screenshot why this was retried later. This was rejected because of the date/time format in the SCH. This is the bug. The PACS vendor's HL7 engine periodically rejects the date/time format. Seemingly at random.

Based on everything I've seen in other HL7 messages here, the format they say should be used, HH:MM:SS is invalid in HL7. The date/time is always presented in YYYYmmddHHMMSS format. The next morning the message is automatically retried by the EHR vendor with the exact same data as today except for the message timestamp and it is accepted (MSA, field 2 = AA).

End of the Story

I never got an answer from the PACS vendor as to why this was happening. I may not have understood the answer anyway, but I am still really curious. There was some more rigamarole we had to go through after this to get data cleaned up, but since this bug was fixed in the PACS vendor's HL7 engine, there have been no further complaints about this not working.

This all happened over the course of about 3 months and was very frustrating to deal with (for me and the customer), but I'm glad I was able to find the answers in the network traffic. I'm ever-curious about these kinds of things and it's rather unsatisfying to just hear "the bug was fixed." I'm really interested in the cause of the bug in the first place, but I guess we don't always get the answers we want.

Stay curious!