CORBA, Routing Tables, and Firewalls, oh my!

June 1, 2006 by Isaac

This past week I created a Python script to test a certain CORBA component. The component uses callbacks to pass data back to the caller. So, in theory, all one has to do is register a callback with the CORBA object, call the Start() method, and let the data start coming.

Now, once I got over the weirdness of CORBA, this was actually pretty straightforward. My script worked and I started getting the callbacks I expected. Well, I moved the script from one machine on the network to another -- they are plugged into the exact same network switch -- but as soon as I moved the script it stopped receiving callbacks.

After struggling for a few days, here's essentially what I learned and how I attempted to fix the problem:

• My firewall blocks all ports by default, which means I have to explicitly open a port for connection. The problem is that CORBA callbacks come in on pretty much any high port that they want. I "fixed" this, but it's not pretty. Since iptables cannot filter at a layer 5 level, which is needed to detect CORBA packets, I couldn't simply add a rule that said "allow incoming CORBA connections." Instead, I opened up a huge range of ports.
• Once I opened up the ports I was still not getting callbacks. The problem is that my new machine was multi-homed -- it had three network cards, each plugged into a different network. While this setup was good for the intended purpose of the machine, it was bad for CORBA.

• • There was one network on the public intranet that was leaking 10.* packets. Someone somewhere did not have their private subnet configured correctly.
  • One of the other three NIC's was on a private 10.* network.
  • Since there were 10.* packets coming in on the public network interface, and 10.* packets on one of the other interfaces, the computer seemed to get confused and would temporarily rewrite the routing table.
  • This caused problems for me trying to send/receive CORBA calls.
• So, after adding some firewall rules to block 10.* packets from the public interface I tried again. No luck.
• I dug deeper into iptables and found this flow diagram of how it works. I was adding rules to the INPUT and OUTPUT chains of the filter table. However, these rules are not applied until after the packet passes the routing table. So, I moved my rules to the PREROUTING chain of the mangle table. I did manage to try one more time before the network engineer here decided that my Test Server didn't need to be multi-homed anymore. Still no luck getting the callbacks to work.
• I still think I was on the right trail and that I could have figured it out, but now I'll never know. My Test Server is now lonely, as it's only on one network. The upside of this is that once the transition happened callbacks magically started working again.

I figure not all blog posts have to be success stories, right? Maybe someone else can pick up where I left off and figure out this problem.

If I do get a chance in the future, I plan on investigating the L-7 filter patch to iptables. I believe this will allow me to let GIOP/IIOP packets through without opening all those ports. Heck, I might have to dream up some application just so that I can try these filters out. It looks to be a really cool project.

Does anyone else have any experience with this sort of thing? Anyone ever used the L-7 filter?

Posted in Administration, Linux