Sometimes we have to learn lessons the hard (long) way…
I’ve had a few calls over the past week in which connectivity issues ruled with an iron fist. We spent over an hour each time trying to troubleshoot an issue that we couldn’t resolve: Why couldn’t ShareFile connect to a customer’s StorageZone? In both cases, customers were already Citrix customers with a NetScaler in place that they wanted to use to load balance ShareFile.
During our calls, we checked the server configuration, IIS settings and the NetScaler configuration. Everything looked correct – but what’s missing? Connectivity, that’s what. Requests to the server consistently timed out, SSL checker responses showed no connection and/or no certificate found. We knew there was a cert, and we knew we had configured the routes; but still, no connection.
In interacting with the system admins I was working with, we were losing something in our dialogue – words seemed to not translate properly, and a small language barrier was less than helpful. To fix this, we brought in a firewall/networking admin and discussed traffic flows, ran traces and still did not have a solution.
With the second customer a few days later, we thought we had it figured out. We (thought that we) set up a firewall rule to open port 443 on the firewall. When he applied the rule, however, we were no longer flowing through their NetScaler. We back tracked a little and while I was chatting with the network engineer from this firm, I caught my first glimpse of the issue – in his explanation, I noticed we used different verbiage. Eureka!!
This network engineer explained the rule he was applying to the firewall at the request of myself and the sysadmins. We requested he open port 443…. This is where things went awry. He took the request to open port 443 as a request to forward port 443 traffic coming in over a certain IP. When he created the forwarding rule, he took the external IP and forwarded traffic from the external firewall, through the DMZ and directly to the internal IP of the server in question.
I understand the difference between forwarding and opening ports, something I had learned when coming onto this team (really, though, I learned about it while troubleshooting my Xbox Live connection while one of my buddies was living with me. But that’s a story for another day). It’s worth noting that opening a port allows all traffic in and out of that port; whereas port forwarding is a way to open the port to traffic coming in over a single IP or IP range and forwards that traffic through the firewall to an IP in either the DMZ or internal network.
This is where I learned a new lesson: verbiage matters; especially when discussing networking.
When working with the sysadmins, we were using the term open when discussing the firewall. Unknowingly to us, the network engineer was taking the open and turning into forward, thereby sending traffic around our configuration. While the setup worked, it was not ideal or the desired behavior – why do all that configuration and have these products for security and networking if we don’t use them?
Once I realized what was happening – that we were forwarding traffic to a new IP, thereby NATing around our setup – we were able to quickly reconcile the issue. Simple as it may seem, we needed a new DMZ IP to broker the traffic between external and internal traffic. Thus, we reconfigured the content switch on the NetScaler to no longer use the external IP and intercept traffic. Rather, we set up the port forwarding rule on the external firewall to NAT traffic into the DMZ, where it is picked up by the NetScaler and directed accordingly. It worked!!!
On to the next challenge…