Cellular to Redundant Rockwell ControlLogix

Can RSLinx access a redundant pair via cellular?

Background
A Pair of Rockwell Automation ControlLogix racks with SRM Module and dual ENBT's will share a pair of IP addresses. One IP address is "the primary", and the other IP is "the backup" (if/when it is online). When the ENBTs switch role, they will issue the requisite Gratuitous ARPs to cause other local Ethernet devices (like a Digi cellular gateway) to update their ARP cache, thus comprehending that the IP-to-MAC address mapping has changed. Thus a NAT/Router forwarding to the primary IP should handle the fail-over with only modest bumps.

Config Details
A user desiring RSLinx (and RSLogix or RSView etc) to access a remote Rockwell ControlLogix (any RA/AB PLC) will be doing what the industry calls "Mobile-Terminated" access. The user needs to arrange a cell plan which offers either a fixed IP address to target, or at least a Dynamic DNS name to target (like tk101.iatips.com or panel2.digi.com). This is NOT what you obtain with an iPhone or personal air-card data plan. Those will have private IPs which only permit outgoing connections - called "Mobile-Originated". That was a buzzword lesson - expect to be asked about those two terms when you ask about cellular data plans!

So once you can arrange your targetable IP or DNS name, you need a cellular router such as one of the Digi Connect WAN family products. My favorite model today is the ConnectPort X4, but that has large memory for Python programming, wireless mesh and other goodies you won't need to link up your Rockwell PLC (Hey, I said it was MY favorite - doesn't mean it has to be yours!)

Note that contrary to folklore or urban legend, all cellular devices need certification to work on a system - even GSM devices. Many small suppliers get around this by including fine-print that say the device buyer is responsible to arrange such legalities, and since you (the buyer) don't read such fine print the salesperson will just say "Heck, it's GSM - so it is allowed everywhere world wide!" Deal with this issue as you see fit, but Digi has more formal certs in more countries worldwide than any of the other industrial players.

But back on subject, when the gateway comes up, it is assigned your known IP or DNS name. This is exactly how your home or business DSL/T-line line works. Yet when RSLinx tries to talk to the gateway on Ethernet/IP's well-known TCP port of 44818, the gateway will reject the connection as a weird attempt at hacking. You need to instruct the gateway:

1. to not reject the Ethernet/IP traffic on TCP port 44818

1. to instead forward it to a local IP on the Ethernet - which would be the IP of your ControlLogix ENBT (or the primary IP of the redundant pair)

The details of how this man-in-the-middle fake-out works is fascinating (to me), but quite a pile of text. If you are interested, this older blog entry goes through the NAT/Router details blow by blow. But bottom-line, your ENBT receives the RSLinx packet and needs to have its own Gateway IP set to the Digi gateway's local Ethernet IP address. Free hint: 9 out of 10 guys who call saying "Why can't RSLinx see my PLC through your Digi gateway?" have failed to set the correct Gateway IP in the PLC/ENBT.

So assuming your gateway and PLC are setup correctly, then targeting the RSLinx "Ethernet Devices" driver (with timeouts slowed down to 30 seconds) will cause your PLC's little icon to show up. With RSLinx running, you will create up to 200MB of billable cell traffic per month doing absolutely nothing - so don't leave it active. Note that the "Ethernet/IP Driver" won't work as it requires UDP broadcast, which can't be routed over the Internet.

At this point you'll say "Cool, now can I see my backup ControlLogix or a second PLC?", and the simple answer is "No." One of the realities of RSLinx and AB PLC is that the Ethernet/IP protocol is hard fixed to only the TCP/IP port 44818, and the NAT/Router can only forward TCP port 44818 to a single local PLC. The easy fix would be for Rockwell to change RSLinx to enable adding both an IP/DNS name and TCP port number - then the NAT/Router could forward TCP port 44818 as 44818 to the primary ENBT, TCP 44819 fixed-up to 44818 to the secondary ENBT, TCP 17256 (a random number :-]) fixed up to 44818 to an RSView panel and so on. Because the NAT/Router can restore all traffic to 44818 on the local Ethernet, RSLinx is the only tool needing to change.

Will Rockwell ever do this? It would take a programmer half a day to do - then a few weeks to test - then a few months to document and forestall support headaches. So who knows. They might. They might not.

But bottomline is a simple cellular NAT/Router can be used to talk to a pair of ControlLogix running in a redundant configuration - you will just be limited to seeing only the primary ENBT and the primary IP address.

Evolution of Data Plan Billing

Summary: the big three have moved away from unlimited data, towards limited data.

It is interesting - I once (as in last year) had a talk with a potential partner who'd been at some European conference and was convinced the world was on the verge of low-cost (sub-$20/month) unlimited cellular data plans. We were discussing the creation of report-by-exception tools to reduce SCADA costs, and this partner's strong faith in this belief caused them to eventually bail out of the talks, saying "In a year or two, no SCADA company will care about how much cellular data they use."

Yet as of the summer of 2008 the world of cellular data is moving in the opposite direction. Last year the big three (AT&T/Sprint/Verizon) offered "Unlimited Data" for personal users with the Service Terms listing a VERY narrow list of permitted activities - mainly email and web browsing, with many common things like file download/upload, media-streaming prohibited. So when ever one of the big three would cut off a user for moving too much data on an "unlimited plan", the service provider would fall back on the "You are doing prohibitted things, thus impacting our network, thus take your business elsewhere". What a way to cause bad feelings, eh? Note that this change is CONSUMER plans - machine-to-machine have always been limited, priced by the MB/month without rollover, plus with charges for data overages.

Now all three have dropped the price from the $80/month range down to $60/month range ... but added a hard limit of 5GB per month. Isn't free & vigorous market competition wonderful?

Sounds reasonable - 5,000 megabytes of data is a lot, yet this doesn't mean 5GB of data transfer. It means 5GB of metered activity, with many activities I've studied including up to 95% overhead. Thus someone only moving 20-30MB of real data in small packets per month might hit pretty close to their 5GB limit! My experience with normal wide-area-network traffic hints that a real PC user doing simple email and web-browsing once a day would probably move 1-2GB of data before hitting the 5GB total activity limit.

To paraphrase the wireless data service terms for all three:

• Data transport is always measured in full kilobytes
• Actual transport is always rounded up to next full-kilobyte at "end of session"
• Network overhead and resend requests caused by network errors can increase measured kilobytes.
• 2 of 3 mention always rounding up to nearest kilobyte every hour period.
• All warn that you will NOT receive an itemized detail of how your charges are calculated; you will NOT see which services were used or during which time periods the charges were inccurred under.
  

So if I send a single 50 byte UDP/IP packet, is that a full session and billed as 1024 bytes? Could be under this language since UDP is 'sessionless'.

Hmm, the term session is pretty ambiguous. Perhaps it means per "time you enable your PC-based cellular data card." That seems likely - plus if you left your device on twenty-four hours a day then the once per hour round-up would catch you.

I'm afraid I haven't offered any new answer here, other than to suggest you understand that low-cost unlimited data plans ARE NOT just around the corner ... at best we left them behind last year and I don't foresee them ever returning. I suppose all three now understand that huge new profits are to be made with these 5GB limits, which will cause many "super-salesman" using their cellular data plan daily to spend an extra $50 to $500 in monthly overage charges.

Rockwell PLC and TCP Headers

I have started running some tests of standard Rockwell protocols querying off-the-shelf Allen-Bradley PLC, with the goal to create a series of "estimators" for traffic. A user would enter the data to poll and the tool will estimates the data byte load contributed by this poll pattern.

The Mystery 17% Cost Increase:
Last night I ran a test polling ten words once a minute from an Allen-Bradley SLC5/05C's N7 file over GSM. This is nothing exotic - I ran similar tests a few months ago and had preconceived ideas of what to expect ... beep ... wrong! In between Then and Now, some unknown application changed my Windows XP system registry, enabling the "RFC 1323 Timestamp and Window Scale TCP options". The end result was an unexpected 16.51% increase in data byte traffic with no perceived value.

I have no clue which tool did this; and unfortunately Windows (at least 2K and XP) use a single global setting for the entire TCP stack. I could change it back ... but would that break this other mystery application? Will this other mystery application just change it back? Will I launch a mini cold-war race as this mystery application tries to keep RFC 1323 enabled and my test tools try to keep it disabled?

The Byte Counts with and without RFC1323:
Here is an exact accounting of the change in byte counts - remember, cellular is basically a mobile-IP tunnel which moves TCP/IP or UDP/IP as pure data payload. So you pay for both the IP and TCP headers, plus any data-less TCP Acknowledge or Keepalive packets.

I'll ignore the opening and closing of the socket, plus TCP Keepalive since I'm polling fairly steady-state once per minute. The PLC includes the TCP ACK in the response, so at least we avoid 1-of-2 data-less TCP Acknowledgments.

	no RFC1323	with RFC1323
Request: IP header	20	20
Request: TCP header	20	32
Request: CSPv4 Packet	42	42
Response: IP header	20	20
Response: TCP header	20	32
Response: CSPv4 Packet	56	56
Client ACK: IP header	20	20
Client ACK: TCP header	20	32
Client ACK: (no data)	0	0

	no RFC1323	with RFC1323
Total Bytes per Poll	218	254
Total Bytes per Hour	13,080	15,240
Total Bytes per Day	313,920	365,760
Total Bytes per Month	9,417,600	10,972,800

So this means a user doing 1 read of 10 words per minute would magically see a 16.51 % increase in data traffic ... just because they (or the IT department or even Microsoft Windows Update) changes a hidden registry setting. This is yet another example of both how hard it is to keep tight control on your cellular data costs; plus adds to my belief that using off-the-shelf host applications over cost sensitive IP networks is a losing battle. At some point you'll need a tool or device which is 100% "under-control" when it come to packet creation.

Windows Registry Details:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts

Tcp1323Opts
Key: Tcpip\Parameters
Value Type: REG_DWORD—number (flags)
Valid Range: 0, 1, 2, 3

• 0 (disable RFC 1323 options)
• 1 (window scaling enabled only)
• 2 (timestamps enabled only)
• 3 (both options enabled)

Default: No value. The default behavior is as follows: do not use the Timestamp and Window Scale options when initiating TCP connections but use them if the TCP peer that is initiating communication includes them in the SYN segment.

Description: This parameter controls the use of RFC 1323 Timestamp and Window Scale TCP options. Explicit settings for timestamps and window scaling are manipulated with flag bits. Bit 0 controls window scaling, and bit 1 controls timestamps.

Do Users Really Want Industrial Ethernet?

(For those impatent to read this to the end - I'm not saying don't use Ethernet ... I am just saying be careful you understand what your customers expect and what functionality they will assume you include *for free* when you add Ethernet)

My last post created some interesting feedback. But I want to emphasize a topic from that post more fully. For the last 15 years I've been involved in the "multi-vendor interface" business - linking multiple vendors' equipment by data comms. First I worked in RS-232 and 485, then fiber optics, then Ethernet, and now by virtually every technology that moves TCP/IP.

From time to time I am contacted by some pretty desperate customers - for example I had one customer who had piloted some Ethernet-based temperature sensors. Things worked fine in the lab with their lab computer, so they bought 50 ... only to find out they couldn't use them. It seems these sensors really were "just Ethernet" - they talked by Ethernet broadcast and direct MAC-layer packets. They didn't support TCP/IP and therefore could NOT be routed by any standard network infrastructure. The user could not talk to any of the sensors they had intended to install in panels around the plant because the "Computer Room" wasn't on the same physical Ethernet segment as the "floor". There was no way to broadcast or unicast MAC-level between the systems. This customer hoped I knew of some magic box to act as gateway between TCP/IP nodes and pure Ethernet nodes; I didn't.

So this brings me back to the concept of the true cost to implement "Ethernet". Customers who ask for Ethernet are not really asking for Ethernet hardware or an Ethernet media bus. They have the expectation that they can interface your "Ethernet Devices" with the wide variety of other equipment they have - including WiFi, routed Ethernet, fiber optics, wide-area networks, and so on. They also expect (at least in a future firmware rev) web pages for configuration, SNMP for remote management, strong encryption, and so on.

So the term "Ethernet" has taken on a life of its own - remember when 802.11 was called "Wireless Ethernet". Well, there is absolutely NOTHING Ethernet about 802.11, yet it was a useful PR move to link the two. No doubt it helped spread the acceptance of WiFi as we now call it. Interestingly enough, the current PCI verse PCI-Express adapters you buy for a PC are using the same PR trick - linking a new, unknown technology to an old established technology that merely accomplish the same function by very different means. Maybe Sony should have called Beta-Max VHS-Max instead ... but then I'm showing my age by even knowing that a consumer-oriented video standard other than VHS even existed.

But back to Ethernet. If you are a small device maker and have yet to start making Ethernet-based products, just be aware that customers who ask for "Ethernet products" aren't really asking for ... err, products with Ethernet. They are asking for products which integrate into (at a minimum) the wide family of TCP/IP based technologies out there. I am not even talking about should you support Modbus/TCP or ODVA Ethernet/IP or ProfiNet yet. I am just saying customers will expect your "Ethernet products" to be able to hold a raw TCP/IP or UDP/IP conversation with all of the other equipment they are investing in daily.

So the cost to add an Ethernet port is just a small part of your cost to "add Ethernet". That is why companies like Digi can sell Ethernet-to-Serial converters or sell "async Ethernet driver chips" like the Digi Connect ME which links to your CPU's serial UART. These devices of course cost more than $9.95 or even the cost of a few new hardware chips, but that higher cost is paying for TCP/IP, web servers, SNMP servers, strong AES encryption and all of the other things your customers expect when the buy "Ethernet products".

So to digress a bit, I suffer this "Oh, don't worry ... it's Ethernet" on a daily basis. So far I have to say at least 95% of the off-the-shelf software applications I test supporting TCP/IP don't work well with technologies other than direct Ethernet. This includes problems not only when extremely different media like satellite or cellular, but even when WiFi is used. So that is part of my mission in this blog - what you want is NOT to Ethernet-enable your products. Instead you need to "IP-enable" your products by way of an Ethernet interface.