Working at Realtime Robotics

April 2021 - August 2023, Boston

I won’t give too much detail about the content of work. Most things will be abstract description.

Create and maintain driver for controlling industrial robot over Ethernet at high cycle time.
Each brand of industrial robot uses vastly different control scheme and protocol, the driver allowed brand/model agnostic for upper stack.
Ensures timing of all component on real-time required code path.
Debug various network issue when controlling multiple industrial robot at the same time. Got familiar with networking up to layer 4 of ISO model (transport layer), specially good at looking into UDP and TCP issues. Learned to use all kinds of network debug tool like wireshark/tshark, tcpdump, arp-scan, nmap, etc.
Created a robot simulator that mimic actual network traffic talking to a real robot. Allowed realistic testing environment without needing an actual robot
Designed fully automated integration and stress test for our product. Found numerous issue as soon as test is online. The test is made to be very flexible and autonomous. It can adapt to use various robot model and test data automatically.
Deploy the automated test onto gitlab-CI for fully autonomous test running on a daily schedule. Shorten time for finding quality issue and reduced manual testing work load.

Realtime Robotics is a software company with the product for controlling industrial robot. For a proper, accurate, and detailed explanation, visiting rtr.ai and talk to a sales would be a lot better. Generally speaking, industrial robot is simple minded. They are excellent on their electrical-mechanical aspect: fast, accurate, powerful. However they are simple minded, no awareness of other robot or the environment. What Realtime Robotics does is plan out all robot’s motion accordingly and take over the detail control over exactly how robot moves.

I work at Realtime Robotics as a robotics software engineer under robot control team. We take care of commanding robots to execute the exact per-calculated spline. Industrial robot all have very well designed low level control, and those are not exposed to users. Thus we only stream over the expected joint position at each time stamp. The driver we create talk to the robot over network, usually a combination of UDP and TCP sockets, and send over one set of joint position at rate specified by robot’s protocol. Our software run on normal Linux distribution, and we are able to push driver to deliver each network packet with the timing accuracy down to 1ms.

Debugging network issue is very different from normal program debugging. A network packet needs to traverse through the Linux Kernel, network device, Ethernet cable, router/switch, then finally arrived at the robot. Each of these component could become a bottle neck on through put, or create intermittent package lost. Tools for debugging are also very different, usually wireshark is the tool of choice. However there are many bug that doesn’t show itself until hours of run time. Thus tshark/tcpdump are often used in scripts to run along side any test for capturing the critical moments. Furthermore, some robot use TCP and the means to deliver delay sensitive information, which made things a lot worse since the self error correction in TCP from kernel will throw the timing all over places.

The simulator I created mainly focus on mimic a realistic network traffic and IO stress for our machine. Industrial robot does come with their own simulator, however those are designed for robot programmer to visualize motion and simulate robot program (script run on the robot). They are horrible on their timing design and none of them can keep up with the timing requirement of their own protocol (real robot have real time operating system that handles these timing perfectly). Thus I created a simulation tool that run on an external device and focus on testing the timing of the communication from our controller. The simulator doesn’t do bit to bit accurate replication of the real robot. Instead, it uses a different scheme that transmit similar amount of data through similar protocol, thus generate a realistic network traffic just like controlling a real robot. One use case of the simulator is to test our product when controller large amount of robots. We won’t be able to fit 20-30 industrial robot in the same office, but we can still test how well all the components behave during this. We even found issue in network switches using these simulator.