I recently upgraded an 8 node ISE appliance cluster (or “cube” as Cisco calls it) from 2.7 to 3.1. Below is a step-by-step guide to great success. Note that these instructions should be best practice for any version, regardless of being a VM or appliance box.

Summary

The Deployment

New Licensing Model

The Licensing model in 3.x changes from:

• Base
• Plus
• Apex

To:

• Essential
• Advantage
• Premier

The benefit of that is that you will only use either Essential, Advantage or Premier license per session, instead of having to use up to 3 licenses, like in the old model.

The drawback is that they moved SGT provisioning to Advantage Level, when it used to only require a base license before. For more information on how to migrate licenses, or switch to smart licensing, follow the link below the picture above.

Note: I will make a separate post about registering ISE to a Cisco Smart Software Manager On-Prem later.

Upgrade Preparations

These will be covered in more detail further down in the post:

• Obtain a backup of the ISE configuration and operational data.
• Obtain a backup of the system logs.
• Disable scheduled backups. Reconfigure the backup schedules after deployment upgrade is complete.
• Export the certificates and private keys.
• Configure a repository. Download the upgrade bundle and place it in the repository.
• Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable. You need this information to connect to Active Directory or RSA SecurID server after upgrade.
• Make a note of DNS, NTP & AD settings
• Purge the operational data to improve upgrade performance.
• Run the URT = Upgrade Readiness Test

From https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/upgrade_guide/b_ise_upgrade_guide_24/b_ise_upgrade_guide_24_chapter_010.html#id_15402

In addition, you should read the ISE 3.1 release notes to check that your current version is compatible with the ISE 3.1 upgrade.

Also make sure that you have some kind of console access to the ISE nodes, either through KVM terminal or Serial over LAN.

Choose the upgrade sequence

1. By default, the Secondary Administration Node is listed first in the upgrade sequence. After upgrade, this node becomes the Primary Administration Node in the new deployment.
2. The Primary Monitoring Node is the next one in the sequence to be upgraded to the new deployment.
3. Select the Policy Service Nodes and move them to the new deployment. You can alter the sequence in which the Policy Service Nodes are upgraded.
   You can upgrade the Policy Service Nodes in sequence or in parallel. You can select a set of Policy Service Nodes and upgrade them in parallel.
4. Select the Secondary Monitoring Node and move it to the new deployment.
5. Finally, select the Primary Administration Node and move it to the new deployment.

From https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/upgrade_guide/b_ise_upgrade_guide_24/b_ise_upgrade_guide_24_chapter_010.html#id_15492

Detailed Step-by-step Preparations

Check Release notes for ISE 3.1

You might want to look for Supported Hardware, Upgrade Information and Known Limitations and Workarounds

Supported Hardware

Cisco ISE 3.1 can be installed on the following platforms:

• SNS-3595-K9
• SNS-3615-K9
• SNS-3655-K9
• SNS-3695-K9

Upgrade Information

You can directly upgrade to Release 3.1 from the following Cisco ISE releases:

2.6, 2.7 or 3.0

If you are on a version earlier than Cisco ISE, Release 2.6, you must first upgrade to one of the releases listed above, and then upgrade to Release 3.1.

This is from Cisco: “We recommend that you upgrade to the latest patch in the existing version before starting the upgrade.

Cisco ISE 3.1 has parity with 2.6 Patch 9, 2.7 Patch 4​, and 3.0 Patch 2.”

Known Limitations and Workarounds

This section provides information about the various known limitations and the corresponding workarounds.

Authentication Might Fail for SNMP Users After Upgrade due to Wrong Hash Value

If you are upgrading from Cisco ISE 2.7 or earlier release to Cisco ISE 3.1, you must reconfigure the settings for SNMP users after the upgrade. Use the following commands to reconfigure the settings for SNMPv3 users:

https://www.cisco.com/c/en/us/td/docs/security/ise/3-1/release_notes/b_ise_31_RN.html

Setup an SFTP Linux Repository

ISE Configuration

From the ISE GUI:

Step 1: Navigate to Administration > System > Maintenance > Repository

Step 2: Create a new SFTP Repository

Note: After creating the repository, it will show up in the running configuration on all the ISE nodes:

This suggests that it should be possible to configure from CLI aswell. However, if configured from CLI, you have to add the repository manually to all nodes, and it won’t be visible in the GUI, therefore not recommended.

Step 3: Generate a keypair

Click on Generate Key pairs. When that is done you can click Export Public Key. That key will be used on the repository under Authorized_Keys

Note: You can generate keypair and export keys from the CLI but you need to specify a repository to do it. I realize that it looks like a catch-22 problem, if it is your first repository.

Step 4: Add the Linux Repository host key on the ISE nodes.

From the CLI, on all the ISE nodes, add the host_key for the linux Repository DNS and optionally an IP:

Linux Repository Configuration

Step 1: Remove the # infront of the lines in this file:

Step 2: Edit the Authorized Keys in your home directory. Insert the exported key from ISE and copy it for every node

Backup & Restore

Backup

Backed up data is only used in case something bad happens during the upgrade. If you are lucky, you won’t be needing this data later.

From the PAN node:
First, save the running configuration

Backing up Configuration data

From the M&T node:
Backing up Operational Data

On all the nodes:
Backup System Logs

Backing up Certificates

1. From the GUI, go to Administration > System > Certificates. Choose the certificates you want to backup and click Export.
2. Export both Certificate and Private Key and set at passphrase.

Note: The interesting certificates are those issued from an internal or public CA. The self-singed certificates can just be replaced with new ones.

Note: One reason that URT can fail, is if the deployment have expired certificates in it’s store. I recommend cleaning up expired certificates before exporting.

Backing up CA Store

From the PAN node:

For importing CA store, choose 8 instead.

Restore

This part should not be necessary if upgrade works fine. But if you are really unlucky, and you need to reimage your whole deployment from scratch:

Importing configuration data

Importing configuration data with ADE-OS config (initial configuration like IP and DNS)

Importing operational data (same command as configuration)

Note: If you have the Primary Administration Node (PAN) auto-failover configuration enabled in your deployment, disable this configuration before you restore a backup. You can enable the PAN auto-failover configuration after the restore is complete.

Disable scheduled Backups

From GUI:

Administration > System > Backup & Restore > Turn schedule status for configuration and operational data to disabled.

Purge Operational Data

On the M&T Nodes:

Disable PAN auto-failover

Administration > System >Deployment > PAN Failover > Uncheck PAN auto failover

Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable.

You may need this information to connect to Active Directory or RSA SecurID server after upgrade. Usually it reconnects fine after upgrade.

Make a note of DNS, NTP & AD settings

You can find these settings with a “show running-config” from CLI.

Run the Upgrade Readiness Test

Step 1: Go to the Cisco website, or search for ISE 3.1 download, and download the the upgrade readiness tool. In the same page where you can download ISE 3.1, scroll down to the bottom:

Step 2: Add the URT to the Repository

Step 3: Run the URT command from CLI on the Secondary PAN node

Upgrade Procedure

After URT checks out OK, you can proceed with the upgrade. You can upgrade from both GUI and the CLI. However, In my experience, upgrading from CLI is more stable.

Upgrading from GUI:

• Pros: Automatic upgrade in a user-defined sequence
• Cons: Sometimes doesn’t work

Upgrading from CLI:

• Pros: More control, and more stable
• Cons: have to start upgrade manually on each node

Upgrade from CLI

On each node:

In case something wrong happened and you need to try again:

Upgrade commands:

Note: These commands are recommended to be entered from a console connection

In my experience, upgrading a node takes around 1 – 2 hours, depending on what kind of persona it runs.

Upgrading the Secondary PAN and the Primary PAN nodes

Typically you will start upgrading the Secondary PAN node first, that will become the Primary PAN in the new deployment. It will also become the primary M&T node for the new deployment, until the Primary M&T node has been upgraded

Once the Primary M&T node has been upgraded, you can disable the M&T persona on the upgraded PAN node:

Administration > System > Deployment. Click on the PAN node and disable Monitoring

Upgrading the PSN nodes

When the new deployment have at least one PAN and one M&T node, PSN nodes can be added. If you don’t have that much load on the network, you can upgrade half the nodes at the same time, saving some time.

Note: Once the new PSN nodes comes up, they will start authenticating for the new deployment. Make sure that the first batch of PSN nodes are able to authenticate users and machines before proceeding with the next batch.

Upgrading the Secondary M&T node and the Primary PAN node

When you reach the point when it is only the secondary M&T and Primary PAN node left, upgrade cannot proceed on the M&T node, because there won’t be any M&T node left in the old deployment. Therefore you have to enable the M&T persona on the Primary PAN:

Administration > System > Deployment. Click on the PAN node and enable Monitoring

Then, you can proceed with upgrading the Secondary M&T node, and finish with the primary PAN.

Patching via CLI

I actually tried at first to patch via GUI, but it failed with an error message. So my stance is still that CLI is the best choice. Fortunately, patching takes way less time than upgrading and you can patch many nodes in parallel.

Patching Order

Patching order is a bit different from upgrade, although I’m not sure if it matters that much. My preferred order is as follows:

1. Primary PAN node (which is the Original Secondary PAN, unless you have changed it back already)
2. Primary M&T node and The first batch of the PSN nodes
3. The second batch of the PSN nodes, The Secondary M&T node and the Secondary PAN node.

Cleanup

When everything is complete, remember to:

• Activate PAN Auto-Failover
• Activate Scheduled Backups
• Activate Primary PAN node.

Manually Promote Secondary PAN to Primary

Step 1: Log in to the secondary PAN GUI.

Step 2: Go to Administration > System > Deployment.

Step 3: In the Edit Node window, click Promote to Primary.

Step 4: Click Save.

From https://www.cisco.com/c/en/us/td/docs/security/ise/3-0/admin_guide/b_ISE_admin_3_0/b_ISE_admin_30_deployment.html#reference_4E8632DBFC474C7787EC3AE18A940677

The Next Steps…

Migrate to new licensing model

Now we have to migrate to use the new 3.x licenses, at least within the next 90 days. I recommend this link from Cisco for instructions on how to do that.

Upgrade CIMC

Since this upgrade was performed on hardware appliance boxes, it is recommended to also upgrade firmware and BIOS on the CIMC = Cisco Integrated Management Controller interface. That procedure will be covered in a later post