Upgrading Cisco ISE from 3.1 to 3.2

Conscias Senior Nettverkskonsulent, William Lindberg, viser deg hvordan du går fram når du oppgraderer Cisco ISE fra versjon 3.1 til versjon 3.2.

Upgrading to from 3.1 to 3.2 sounds like a small step at first. It turns out however, that there are more to it than you would think.

Reasons for upgrading to 3.2

As I see it, there are 2 main reasons one would upgrade to 3.2:

  1. EAP-TLS and TEAP authorization support when using Microsoft Azure AD as external identity source.
  2. Version 3.2 is currently the gold star release and will most likely be a long-term supported version.

For a complete list of new features, check Cisco’s release notes.

The Deployment

The ISE deployment is based on the same UCS 3595 appliance

 

 

All the regular steps from the last guide I wrote about upgrading from 2.7 to 3.1 are still valid:

  • Check release notes for known caveats.
  • Install the latest patch on current version.
  • Obtain a backup of the ISE configuration and operational data.
  • Obtain a backup of the system logs.
  • Disable scheduled backups.
  • Export the certificates and private keys.
  • Configure a repository. Download the upgrade bundle and place it in the repository.
  • Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable.
  • Make a note of DNS, NTP & AD settings
  • Purge the operational data to improve upgrade performance.
  • Run the URT = Upgrade Readiness Test

In 3.1 however, there are some additional steps.

New steps in ISE 3.1

There is one known caveat when performing the URT tool on ISE 3.1; it simply doesn’t work, CSCwe24589.

It might have something to do with this new notification you get when you perform the upgrade command in CLI:

ise-psn02/admin# application upgrade prepare ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz Example-Repository

Be sure that all your software is working stable, check your system on UI page (Administration > System > Health Checks)
Type yes once confirmed that health of the system is good to proceed:  (yes/no) [yes] ?

Run the Health Check

The Health check is very similar to the upgrade readiness tool, but in a GUI format. For me it almost checks out fine, but something about the certificate store isn’t right.

 

The log:

Certificate Services OCSP Responder – ise-mnt01i#00005 Enabled Cert expired
Certificate Services OCSP Responder – ise-pan01#00004 Enabled Cert expired
Certificate Services Node CA – ise-pan01#00002 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-pan01#00003 Enabled Cert expired
Certificate Services OCSP Responder – ise-psn02#00007 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-psn02#00008 Enabled Cert expired
Certificate Services OCSP Responder – ise-psn01#00009 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-psn01#00010 Enabled Cert expired
Certificate Services OCSP Responder – ise-mnt02#00014 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-mnt02#00015 Enabled Cert expired
Certificate Services OCSP Responder – ise-psn04#00018 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-psn04#00019 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-mnt01i#00006 Enabled Cert expired
Certificate Services OCSP Responder – ise-mnt01i#00022 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-mnt01i#00023 Enabled Cert expired
Certificate Services OCSP Responder – ise-mnt02#00025 Enabled Cert expired
Certificate Services Endpoint Sub CA – ise-mnt02#00026 Enabled Cert expired

Looks like I am dealing with expired internal certificates in the CA store.

Troubleshooting the CA store

Apparently in ISE 3.0, there have been some changes to how the internal certificates are handled. The primary PAN = Policy Administration Node (Or Primary Administration Node, depending what documentation you read) needs to have the Certificate Authority function activated.

Source: https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine-30/217161-ca-service-and-est-service-on-ise.pdf

Start the Internal CA Service

Go to Administration > System > Deployment. Mark the Primary Policy Admin Node and press edit.

Activate Persona Policy Service and Enable Session Services underneath. The Node Group are the PSN nodes.

Verification

In CLI:

Check that Certificate Authority Service and EST Service are running on the PAN.

ise-pan01/admin# show application status ise

ISE PROCESS NAME STATE PROCESS ID
Database Listener running  4457
Database Server running 129 PROCESSES
Application Server running 317256
Profiler Database running 16383
ISE Indexing Engine running 305153
AD Connector running 27325
M&T Session Database disabled
M&T Log Processor disabled
Certificate Authority Service running 305935
EST Service running 328163
SXP Engine Service disabled
TC-NAC Service disabled
PassiveID WMI Service disabled
PassiveID Syslog Service disabled
PassiveID API Service disabled
PassiveID Agent Service disabled
PassiveID Endpoint Service disabled
PassiveID SPAN Service disabled
DHCP Server (dhcpd) disabled
DNS Server (named) disabled
ISE Messaging Service running 12463
ISE API Gateway Database Service running 307182
ISE API Gateway Service running 312253
Segmentation Policy Service disabled
REST Auth Service disabled
SSE Connector disabled
Hermes (pxGrid Cloud Agent) disabled

 

In GUI:

Administration > System > Certificates > Certificate Authority > Internal CA Settings

Source:

Renew the ISE OCSP Responder Certificates

Then you need to Renew the ISE OCSP Responder Certificates. To do that go to:

Administration > System > Certificates > Certificate Management > Certificate Signing Requests > Generate New Certificate Signing Request (CSR).

Choose Renew the ISE OCSP Responder Certificates in the list and press the button with the same name.

 

 

Source: https://community.cisco.com/t5/network-access-control/ise-ocsp-responder-certificate-expiring/td-p/4427030

After renewing the ISE OCSP Responder Certificates, you get redirected to Certificate Authority > Certificate Authority Certificates. Here you can indeed see that you have valid certificates now. However, you still need to get rid of all the expired certificates, which you can do from the same place.

Note: When deleting the expired certificates, a big warning window about endpoints being unreachable and other scary stuff shows up. Just make sure you are deleting the expired certificates and not the new ones and it should be fine. Even if you make a mistake, try perform a new renewal.

Note: Some of the old certificates related to the PAN didn’t show up for me and I had to delete them from Certificate Management > Trusted Certificates. Also, I was unable to remove some related stale certificates but it seems it didn’t affect the health check in any way.

After renewing and cleaning the certificate store, the health check turns all green.

 

 

 

From here I can proceed with upgrade as described in my previous guide.

Appendix

Confusion about patch requirements

If you read the release notes, you might discover that it says following:

“We recommend that you upgrade to the latest patch in the existing version before starting the upgrade.

Cisco ISE, Release 3.2, has parity with the Cisco ISE patch release: …​ and 3.1 Patch 3, and earlier patches.”

Earlier? A bit of a contradiction there, and they are only one sentence apart. There is also the Cisco ISE Upgrade Matrix you can refer to, and it says you should upgrade from the latest patch.

The latest patch for 3.1 at the moment is patch 7. I can confirm that upgrading from this patch works fine on a UCS 3595 deployment.

File Transfer Error

I got some strange error when trying to download the upgrade image to the first node:

ise-psn02/admin# application upgrade prepare ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz Example-Repository

Be sure that all your software is working stable, check your system on UI page (Administration > System > Health Checks)
Type yes once confirmed that health of the system is good to proceed:  (yes/no) [yes] ? yes

Getting bundle to local machine…
Unbundling Application Package…
% Unable to unbundle the package. It should be in tar.gz file format

Application upgrade preparation Failed

 

It is obviously in the tar.gz file format. It seems that somewhere down the line there was a corruption in the file transfer:

$ md5sum ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

7fe175c179a72510c4152e5645eb5abb  ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

 

That number doesn’t match what is documented in Cisco’s software center. This time I check that the MD5 checksum is correct after downloading the file.

In Windows:

C:\TFTP-Root\ISE-3595 v3-2>certutil -hashfile ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz MD5

MD5 hash of ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz:
ac89dee5fa86377836a497c981d7480c
CertUtil: -hashfile command completed successfully.

 

Seems to match so far with the information from Ciscos software center. Next check is after transferring it to the Linux repository:

$ md5sum ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

ac89dee5fa86377836a497c981d7480c  ise-upgradebundle-2.7.x-3.1.x-to-3.2.0.542b.SPA.x86_64.tar.gz

 

Looks good and now the upgrade is working.

Re-imaging instead of Upgrade

Some might prefer to re-image the nodes instead of upgrading. It might be a feasible choice in VM environments when you only have 2 nodes. There is a good baseline guide on that here:

https://www.lookingpoint.com/blog/cisco-ise-3.0-major-upgrade

And here is one for 6 – 8 nodes:

https://www.wiresandwi.fi/blog/cisco-ise-general-steps-for-upgrades-using-backup-and-restore-method

Just make sure that you are on the latest patch on the current version before exporting the configuration data. I have experienced problems when failing to do so.

Regardless if you are reimaging from a hypervisor or a UCS appliance, make sure to read the additional installation information from Cisco. Here you can find information about how to make a bootable USB, install with virtual media from the UCS KVM interface, VM requirements and more.

  • If you are using Virtual Media with HTML KVM interface: For me it took 4 hours to transfer the installation file from my PC to the appliance, and I was on-site with at least 1Gbps link. Also, use a supported browser. I used at first the Brave Browser and the KVM interface just kept timing out after 30min. I have heard that even Microsoft Edge browser have issues. Use either Firefox or Chrome. I can attest that at least Firefox works.
  • If you are using a USB stick, read the instructions carefully about how to make a bootable USB; because you need to edit some files for the installation to work properly, although I was never able to get a USB boot to work.
  • Don’t have a bootable USB stick inserted to the appliance at the same time you are performing KVM installation with virtual media. You will get grub error.

Reconsider if you really need to re-image. Based on my personal experience, I would only consider a re-image of appliances if Cisco TAC tells me to do it.