Juniper SRX210 Sudden Back-To-Factory Defaults

Earlier this week I configured an Juniper SRX210 for testing. The configuration consisted of several security zones, IDP, UAC (layer3 enforcement) and Application Firewall and Identification. The Junos version I used was JUNOS 12.1X46-D15.3.

This setup worked until today. Today, the SRX was unresponsive. No ICMP reply, no SSH access, nothing. Accessing the SRX via the serial console showed me the Amnesiac login. This means that the configuration is gone. At least the configuration I created was reset to the factory defaults config. A typical WTF!!! moment.

Fortunately, I had configured logging to an external source (Splunk). So I went to investigate. Turned out that the SRX stopped sending syslog messages around 01:30PM. Further investigation showed that the config was actually reset (UI_FACTORY_OPERATION event), and checking the event-codes, it was (probably) done by pressing the reset button on the device.

(the following logging should be read bottom-top)

May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.808+02:00 srx210 mgd 6211 UI_COMMIT_PROGRESS [junos@2636.1.1.1.2.36 message="finished copying juniper.db to juniper.data+"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.369+02:00 srx210 mgd 6211 UI_COMMIT_PROGRESS [junos@2636.1.1.1.2.36 message="copying juniper.db to juniper.data+"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.369+02:00 srx210 mgd 6211 UI_COMMIT_PROGRESS [junos@2636.1.1.1.2.36 message="finished loading commit script changes"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.368+02:00 srx210 mgd 6211 UI_COMMIT_PROGRESS [junos@2636.1.1.1.2.36 message="no transient commit script changes"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.368+02:00 srx210 mgd 6211 UI_COMMIT_PROGRESS [junos@2636.1.1.1.2.36 message="no commit script changes"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.367+02:00 srx210 mgd 6211 UI_COMMIT_PROGRESS [junos@2636.1.1.1.2.36 message="start loading commit script changes"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.133+02:00 srx210 rmopd 1350 PING_TEST_COMPLETED [junos@2636.1.1.1.2.36 test-owner="XS4ALL" test-name="testsvr"]
May 28 13:28:30 10.0.0.1 1 2014-05-28T13:28:30.355+02:00 srx210 mgd 6211 - - auto-snapshot is not configured
May 28 13:28:28 10.0.0.1 1 2014-05-28T13:28:28.814+02:00 srx210 mgd 6211 UI_LOAD_JUNOS_DEFAULT_FILE_EVENT [junos@2636.1.1.1.2.36 pathname="/etc/config//srx210h-defaults.conf"]
May 28 13:28:27 10.0.0.1 1 2014-05-28T13:28:27.823+02:00 srx210 mgd 6211 UI_LOAD_JUNOS_DEFAULT_FILE_EVENT [junos@2636.1.1.1.2.36 pathname="/etc/config//jsrxsme-series-defaults.conf"]  
May 28 13:28:27 10.0.0.1 1 2014-05-28T13:28:27.217+02:00 srx210 mgd 6211 UI_LOAD_JUNOS_DEFAULT_FILE_EVENT [junos@2636.1.1.1.2.36 pathname="/etc/config//junos-defaults.conf"]   
May 28 13:28:26 10.0.0.1 1 2014-05-28T13:28:26.808+02:00 srx210 mgd 6211 - - WARNING: activating factory configuration
May 28 13:28:25 10.0.0.1 1 2014-05-28T13:28:25.378+02:00 srx210 mgd 6211 - - WARNING: removing all configurations
May 28 13:28:25 10.0.0.1 1 2014-05-28T13:28:25.232+02:00 srx210 mgd 6211 UI_FACTORY_OPERATION -
May 28 13:28:25 10.0.0.1 1 2014-05-28T13:27:35.000+02:00 srx210 rmopd 1350 PING_TEST_COMPLETED [junos@2636.1.1.1.2.36 test-owner="XS4ALL" test-name="testsvr"]
May 28 13:27:19 10.0.0.1 1 2014-05-28T13:26:39.941+02:00 srx210 - - - - last message repeated 11 times
May 28 13:17:19 10.0.0.1 1 2014-05-28T13:16:34.309+02:00 srx210 - - - - last message repeated 11 times
May 28 13:07:19 10.0.0.1 1 2014-05-28T13:06:28.346+02:00 srx210 rmopd 1350 PING_TEST_COMPLETED [junos@2636.1.1.1.2.36 test-owner="XS4ALL" test-name="testsvr"]
May 28 13:05:33 10.0.0.1 1 2014-05-28T13:05:33.297+02:00 srx210 rmopd 1350 PING_TEST_COMPLETED [junos@2636.1.1.1.2.36 test-owner="XS4ALL" test-name="testsvr"]
May 28 13:04:38 10.0.0.1 1 2014-05-28T13:04:38.248+02:00 srx210 rmopd 1350 PING_TEST_COMPLETED [junos@2636.1.1.1.2.36 test-owner="XS4ALL" test-name="testsvr"]
May 28 13:04:19 10.0.0.1 1 2014-05-28T13:04:19.548+02:00 srx210 utmd 1380 AV_PATTERN_UPDATED [junos@2636.1.1.1.2.36 version="05/28/2014 12:36 GMT, virus records: 522178" file-size="18635751"]
May 28 13:03:42 10.0.0.1 1 2014-05-28T13:03:42.934+02:00 srx210 rmopd 1350 PING_TEST_COMPLETED [junos@2636.1.1.1.2.36 test-owner="XS4ALL" test-name="testsvr"]

This is strange, since there was no one around at the time. So it must have been some sort of bug that caused this.

Thankfully I had a backup of the config, so the device was up and running again within 10 minutes. So now I have to keep an out out for this. Especially the next couple of days.

UPDATE: There's is definitely something wrong with the hardware. I tried different Junos versions (also stable recommended versions), and different configs, but for some reason the device detect a 'Config button pressed' event and goes back to the default factory config. This happens within 12 hours.

The device keeps going back to the factory default config. Today I changed the behavior of the reset button. The reset button doesn't react to physical interactions when adding the following line to the config:

root@srx210# set chassis config-button no-clear no-rescue

Let's see if that helps. If it does, it means that the hardware reset button (mechanism) is malfunctioning.

UPDATE 2: looks like the config button config did the trick (so far). The device is still up-and-running for nearly 24 hours. -keepingfingerscrossed-

UPDATE 3: and we have a winner. The SRX210 is still operational. Just need to remember to add the command when I reconfigure it. Perhaps a sticker on top as a visual reminder :-)

Domain User Membership check via LDAP

When you are using LDAP to determine Windows Active Directory group membership, and the group you are aiming for is the Domain Users group, than you're in for a surprise. It turns out that the LDAP interface doesn't have the Domain Users group listed for a user. It's missing the memberOf attribute for Domain Users. Just compare the following screenshots. The first screenshot shows the Active Directory user interface for the user Administrator, and the second shows the LDAP equivalent of that same user.

View fullsize

Active Directory group memberships

View fullsize

LDAP group memberships

The LDAP output doesn't show a 'memberOf: CN=Domain Users, CN=Users, DC=testdomain, DC=local' attribute.

The reason is that Active Directory has a so-called Primary Group attribute, and this is by default the Domain Users group. With that piece of information you might see a LDAP attribute called 'primaryGroupID' with a number. That number represents the Domain Users group.

So if you need to check for Domain User membership with LDAP, you should check the value of the primaryGroupID attribute. This value is (for as far as I know) always the same (513).

So if you're using Certificate based authentication on a Juniper Pulse Access Gateway or Pulse Access Control Service, and you need to check Windows Domain User group membership the primaryGroupID is the way to go.

View fullsize

B.t.w., if you're looking for a good cross-platform LDAP browser, I can recommend the Apache Directory Studio. It's intuitive, has a good interface and just works (oh... and it's free).

No EAP Protocol Was Agreed On

Having the opportunity to experiment with some Juniper security products at home has its (dis)advantages. Juniper offers a (limited) virtual appliance version for both the Unified Access Control appliance (aka the Infranet Controller or Pulse Access Control Gateway), and the SSL VPN solution (aka Secure Access or Pulse Secure Access Gateway).

The limited parts are:

• SSL is limited to 3 concurrent users
• UAC is limited to 5 concurrent users
• You cannot add additional licenses
• The UAC has no IF-MAP server capabilities, since that requires at least a 50 user license (and you cannot add additionel licenses).

Max. 3 concurrent SSL VPN users

Max. 5 concurrent UAC users

So yes, it's crippled, but still very nice to play with in a lab or home/study environment.

Anyway, I have both the UAC and the SSL VPN running at home. Both running in  VMWare Fusion on a MAC OSX server (Mac Mini).

A couple of months ago, Juniper released a new major version for the software (v5 for the UAC, and v8 for the SSL VPN), so I wanted to upgrade the VM's to the latstes software (also because of the Heartbleed bug in OpenSSL). This was no problem for the SSL VPN. The upgrade went smooth. However, the UAC was a different story. For some reason, the upgrade package was corrupt or invalid (even though it could be used to do a clean install), so upgrading was out of the question.

So I tried to do a clean install and see if I could import the old config of the existing UAC (v4.4) in the new version 5. Something that didn't work in the older versions of both the SSL VPN and UAC. Importing a software version meant that you needed the correct software version on the device first.

Anyway, importing the system config seemed to work, because all visible settings were correct. The XML import (other configuration settings regarding authentication servers, realms, user roles, etc.) also imported correctly (or so it seemed).
I compared the two configs side by side, and everything checked out. That was until I tried to authenticate on a switch with 802.1x. That didn't work as it should.

The logging of the UAC showed numerous 'No EAP Protocol Was Agreed On' errors. This was weird, because everything worked correctly on the older version.
Since the EAP protocol relies (for a part) on the SSL certificate on the device, I swapped that one for a new certificate from my personal PKI service.

After having checked, and double checked everything (I even tried authenticating against the older UAC version... which still worked), I decided to do a clean install (back to factory settings), and reconfigure the entire UAC by hand instead of the import.

Guess what, everything worked great after I had copied everything by hand.

So I guess that the import of a XML file belonging to a earlier software version still doesn't work. Only difference is that in the old days I got a warning/error.

So if you're getting the 'No EAP Protocol Was Agreed On' error in your events logging, and you did a recent upgrade, you might want to try a fresh install and configure things by hand.

I have no idea if this is applicable to the 'normal' hardware appliances with the software.

Expired SSL Certificates

When I'm doing my thing on customer projects, and there's some SSL stuff involved, I always keep reminding them to make sure that they renew their certificates in time. Why, because they almost always forget it the renew them in time, and after the expire stuff stops working, and they call us.

Guess what? My certificates expired this evening, so I got numerous warnings and errors in several applications that use those SSL certificates. Thankfully I run my own CA (XCA), and I documented where I use them, and how to replace them, so I was back in business in 10 minutes.

Lesson(s) learned: make a notification in my calendar to replace them ahead of time.

Juniper SRX IDP Attack Log Investigation

Last week I fiddled with the IDP functionality on my SRX100H. I eventually installed a modified version of the so-called "Recommended" IDP policy. The full Recommended policy consumes to much memory on the SRX100H, so I had to remove some rules.

After installing the IDP policy I 'configured' Splunk to run reports on IDP Attack Events (IDP Attacks in the last 24 hours). This way I have a nice overview of the detected attacks.

Today I checked the logging, and to my surprise I found several IDP_ATTACK_LOG_EVENT entries in the log. This triggered my curiosity a bit.

View fullsize

Thankfully I use an NTP server for syncing all the devices around the house, so working backward shouldn't be too hard. 

The IDP event was triggered by my iMac which had the 192.168.1.109 IP address at the time (hurray for DHCP logging). The reported exploit (HTTP:XSS:HTML-SCRIPT-IN-URL-VAR) is basically website related, so I started to dig through the browser history. Turned out that I visited the Victorinox website at the time looking for winter coats. During my visit at the website I tried to find a local store where they sell these items, but for some reason the 'Store Locator' thingy on the website didn't work. Now the logging explained why it didn't work.

It looks like they use a map-service (hosted on the destination address 199.16.46.10) in combination with some cross-site-scripting to deliver the functionality.

View fullsize

I tested this again, and indeed, every time I go to that website and try to locate a store, the same IDP_ATTACK_LOG_EVENT occurs.

In this case the cross-site-scripting (XSS) is relatively harmless. The use it to display a map with possible stores based on your query. Unfortunately, there are numerous other scenario's where this (XSS) isn't harmless.