Conscias Senior Nettverkskonsulent – William Lindberg – har blogget om hvordan du enklest kan oppgradere Cisco ISE fra versjon 2.7 til versjon 3.1
I recently upgraded an 8 node ISE appliance cluster (or “cube” as Cisco calls it) from 2.7 to 3.1. Below is a step-by-step guide to great success. Note that these instructions should be best practice for any version, regardless of being a VM or appliance box.
New Licensing Model
The Licensing model in 3.x changes from:
The benefit of that is that you will only use either Essential, Advantage or Premier license per session, instead of having to use up to 3 licenses, like in the old model.
The drawback is that they moved SGT provisioning to Advantage Level, when it used to only require a base license before. For more information on how to migrate licenses, or switch to smart licensing, follow the link below the picture above.
Note: I will make a separate post about registering ISE to a Cisco Smart Software Manager On-Prem later.
These will be covered in more detail further down in the post:
- Obtain a backup of the ISE configuration and operational data.
- Obtain a backup of the system logs.
- Disable scheduled backups. Reconfigure the backup schedules after deployment upgrade is complete.
- Export the certificates and private keys.
- Configure a repository. Download the upgrade bundle and place it in the repository.
- Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable. You need this information to connect to Active Directory or RSA SecurID server after upgrade.
- Make a note of DNS, NTP & AD settings
- Purge the operational data to improve upgrade performance.
- Run the URT = Upgrade Readiness Test
In addition, you should read the ISE 3.1 release notes to check that your current version is compatible with the ISE 3.1 upgrade.
Also make sure that you have some kind of console access to the ISE nodes, either through KVM terminal or Serial over LAN.
Choose the upgrade sequence
- By default, the Secondary Administration Node is listed first in the upgrade sequence. After upgrade, this node becomes the Primary Administration Node in the new deployment.
- The Primary Monitoring Node is the next one in the sequence to be upgraded to the new deployment.
- Select the Policy Service Nodes and move them to the new deployment. You can alter the sequence in which the Policy Service Nodes are upgraded.
You can upgrade the Policy Service Nodes in sequence or in parallel. You can select a set of Policy Service Nodes and upgrade them in parallel.
- Select the Secondary Monitoring Node and move it to the new deployment.
- Finally, select the Primary Administration Node and move it to the new deployment.
Detailed Step-by-step Preparations
Check Release notes for ISE 3.1
You might want to look for Supported Hardware, Upgrade Information and Known Limitations and Workarounds
Cisco ISE 3.1 can be installed on the following platforms:
You can directly upgrade to Release 3.1 from the following Cisco ISE releases:
2.6, 2.7 or 3.0
If you are on a version earlier than Cisco ISE, Release 2.6, you must first upgrade to one of the releases listed above, and then upgrade to Release 3.1.
This is from Cisco: “We recommend that you upgrade to the latest patch in the existing version before starting the upgrade.
Cisco ISE 3.1 has parity with 2.6 Patch 9, 2.7 Patch 4, and 3.0 Patch 2.”
Known Limitations and Workarounds
This section provides information about the various known limitations and the corresponding workarounds.
Authentication Might Fail for SNMP Users After Upgrade due to Wrong Hash Value
If you are upgrading from Cisco ISE 2.7 or earlier release to Cisco ISE 3.1, you must reconfigure the settings for SNMP users after the upgrade. Use the following commands to reconfigure the settings for SNMPv3 users:
|no snmp-server user <snmp user> <snmp version> <auth password> <priv password>|
|snmp-server user <snmp user> <snmp version> <auth password> <priv password>|
Setup an SFTP Linux Repository
From the ISE GUI:
Step 1: Navigate to Administration > System > Maintenance > Repository
Step 2: Create a new SFTP Repository
|Repository Name: Example-Repo
Server Name: example-repo.com
Enable PKI authentication: No
User Name: myusername
Note: After creating the repository, it will show up in the running configuration on all the ISE nodes:
user myusername password hash **********
This suggests that it should be possible to configure from CLI aswell. However, if configured from CLI, you have to add the repository manually to all nodes, and it won’t be visible in the GUI, therefore not recommended.
Step 3: Generate a keypair
Click on Generate Key pairs. When that is done you can click Export Public Key. That key will be used on the repository under Authorized_Keys
Note: You can generate keypair and export keys from the CLI but you need to specify a repository to do it. I realize that it looks like a catch-22 problem, if it is your first repository.
Step 4: Add the Linux Repository host key on the ISE nodes.
From the CLI, on all the ISE nodes, add the host_key for the linux Repository DNS and optionally an IP:
|crypto host_key add host example-repo.com
crypto host_key add host 2001:DB8:1234:ABCD::50
copy running-config startup-config
Linux Repository Configuration
Step 1: Remove the # infront of the lines in this file:
|sudo nano /etc/ssh/sshd_config|
RSAAuthentication yes # if it exists
|sudo service sshd restart|
Step 2: Edit the Authorized Keys in your home directory. Insert the exported key from ISE and copy it for every node
|sudo nano ~/.ssh/authorized_keys|
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
[email protected]n01 <— Hostname of the ISE node
Backup & Restore
Backed up data is only used in case something bad happens during the upgrade. If you are lucky, you won’t be needing this data later.
From the PAN node:
First, save the running configuration
|copy running-configuration startup-configuration|
Backing up Configuration data
|backup ISE_ConfigBackup_20221213 repository Example-Repo ise-config encryption-key plain XXXXXXXX|
From the M&T node:
Backing up Operational Data
|backup ISE_OperationalBackup_20221213 repository Example-Repo ise-operational encryption-key plain XXXXXXXX|
On all the nodes:
Backup System Logs
|backup-logs ISE_Logs20221213_bkxif240i repository Example-Repo encryption-key plain XXXXXXXX|
Backing up Certificates
- From the GUI, go to Administration > System > Certificates. Choose the certificates you want to backup and click Export.
- Export both Certificate and Private Key and set at passphrase.
Note: The interesting certificates are those issued from an internal or public CA. The self-singed certificates can just be replaced with new ones.
Note: One reason that URT can fail, is if the deployment have expired certificates in it’s store. I recommend cleaning up expired certificates before exporting.
Backing up CA Store
From the PAN node:
|application configure ise
Selection configuration option
For importing CA store, choose 8 instead.
This part should not be necessary if upgrade works fine. But if you are really unlucky, and you need to reimage your whole deployment from scratch:
Importing configuration data
|restore ISE_ConfigBackup_20221213 repository Example-Repo encryption-key plain XXXXXXXX|
Importing configuration data with ADE-OS config (initial configuration like IP and DNS)
|restore ISE_ConfigBackup_20221213 repository Example-Repo encryption-key plain XXXXXXXX include-adeos|
Importing operational data (same command as configuration)
|restore ISE_OperationalBackup_20221213 repository Example-Repo encryption-key plain XXXXXXXX|
Note: If you have the Primary Administration Node (PAN) auto-failover configuration enabled in your deployment, disable this configuration before you restore a backup. You can enable the PAN auto-failover configuration after the restore is complete.
Disable scheduled Backups
Administration > System > Backup & Restore > Turn schedule status for configuration and operational data to disabled.
Purge Operational Data
On the M&T Nodes:
|application configure ise
Selection configuration option
Enter number of days to be retained in purging MnT Operational data [between 1 to 90 days]
Disable PAN auto-failover
Administration > System >Deployment > PAN Failover > Uncheck PAN auto failover
Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable.
You may need this information to connect to Active Directory or RSA SecurID server after upgrade. Usually it reconnects fine after upgrade.
Make a note of DNS, NTP & AD settings
You can find these settings with a “show running-config” from CLI.
Run the Upgrade Readiness Test
Step 1: Go to the Cisco website, or search for ISE 3.1 download, and download the the upgrade readiness tool. In the same page where you can download ISE 3.1, scroll down to the bottom:
Step 2: Add the URT to the Repository
Step 3: Run the URT command from CLI on the Secondary PAN node
|application remove urt <– Cleans up old URT config.
application install ise-urtbundle-184.108.40.2068-1.0.0.SPA.x86_64.tar.gz Example-Repo
After URT checks out OK, you can proceed with the upgrade. You can upgrade from both GUI and the CLI. However, In my experience, upgrading from CLI is more stable.
Upgrading from GUI:
- Pros: Automatic upgrade in a user-defined sequence
- Cons: Sometimes doesn’t work
Upgrading from CLI:
- Pros: More control, and more stable
- Cons: have to start upgrade manually on each node
Upgrade from CLI
On each node:
In case something wrong happened and you need to try again:
|application upgrade cleanup|
|application upgrade prepare ise-upgradebundle-2.6.x-3.0.x-to-220.127.116.118b.SPA.x86_64.tar.gz Example-Repo
application upgrade proceed
Note: These commands are recommended to be entered from a console connection
In my experience, upgrading a node takes around 1 – 2 hours, depending on what kind of persona it runs.
Upgrading the Secondary PAN and the Primary PAN nodes
Typically you will start upgrading the Secondary PAN node first, that will become the Primary PAN in the new deployment. It will also become the primary M&T node for the new deployment, until the Primary M&T node has been upgraded
Once the Primary M&T node has been upgraded, you can disable the M&T persona on the upgraded PAN node:
Administration > System > Deployment. Click on the PAN node and disable Monitoring
Upgrading the PSN nodes
When the new deployment have at least one PAN and one M&T node, PSN nodes can be added. If you don’t have that much load on the network, you can upgrade half the nodes at the same time, saving some time.
Note: Once the new PSN nodes comes up, they will start authenticating for the new deployment. Make sure that the first batch of PSN nodes are able to authenticate users and machines before proceeding with the next batch.
Upgrading the Secondary M&T node and the Primary PAN node
When you reach the point when it is only the secondary M&T and Primary PAN node left, upgrade cannot proceed on the M&T node, because there won’t be any M&T node left in the old deployment. Therefore you have to enable the M&T persona on the Primary PAN:
Administration > System > Deployment. Click on the PAN node and enable Monitoring
Then, you can proceed with upgrading the Secondary M&T node, and finish with the primary PAN.
Patching via CLI
I actually tried at first to patch via GUI, but it failed with an error message. So my stance is still that CLI is the best choice. Fortunately, patching takes way less time than upgrading and you can patch many nodes in parallel.
|patch install ise-patchbundle-18.104.22.1688-Patch4-22091704.SPA.x86_64.tar.gz Example-Repo|
Patching order is a bit different from upgrade, although I’m not sure if it matters that much. My preferred order is as follows:
- Primary PAN node (which is the Original Secondary PAN, unless you have changed it back already)
- Primary M&T node and The first batch of the PSN nodes
- The second batch of the PSN nodes, The Secondary M&T node and the Secondary PAN node.
When everything is complete, remember to:
- Activate PAN Auto-Failover
- Activate Scheduled Backups
- Activate Primary PAN node.
Manually Promote Secondary PAN to Primary
Step 1: Log in to the secondary PAN GUI.
Step 2: Go to Administration > System > Deployment.
Step 3: In the Edit Node window, click Promote to Primary.
Step 4: Click Save.
The Next Steps…
Migrate to new licensing model
Now we have to migrate to use the new 3.x licenses, at least within the next 90 days. I recommend this link from Cisco for instructions on how to do that.
Since this upgrade was performed on hardware appliance boxes, it is recommended to also upgrade firmware and BIOS on the CIMC = Cisco Integrated Management Controller interface. That procedure will be covered in a later post