Common Cisco ISE Issues

Recently I decided to deploy a second ISE node in my home lab and run them in a Primary/Secondary configuration. Once i''d configured the secondary node and added it to the deployment, I decided to promote it to the primary (Mostly because the trial license on my Primary ISE node was about to run out) and found that there were some issues once I'd failed over. I actually ended up having to reboot both nodes a couple of times just to get the deployment and all ISE process to start running correctly. Anyway, back to my blog post. Once i'd had both nodes running for a while and confirmed all devices were authenticating and everything looked good, i logged in again and noticed that nothing was populating in the dashboard. I proceeded to reboot services again just to see if that fixed it but it turns out it didn't. I also found that I was having a large number of Queue Link errors in the alarms dashboard. This post will run through the process I used to resolve both of those issues. 

Context Visibility issue

I'll start with the context visibility issue which turns out is quite a common issue, and a simple one to fix. If like me, you're ISE dashboard looks like this, then your issue is with the Context Visibility database and it will need 

Log into the CLI of your ISE Primary and Secondary node as this will need to be done on both. Use the command show application statis ise to verify that all the processes are running. Before proceeding further, not that the next steps need to be run on your ISE Secondary node first. 

WRMEMISE01/admin#show application status ise

ISE PROCESS NAME                       STATE            PROCESS ID  
--------------------------------------------------------------------
Database Listener                      running          155981      
Database Server                        running          79 PROCESSES
Application Server                     running          171791      
Profiler Database                      running          163110      
ISE Indexing Engine                    running          172977      
AD Connector                           running          175264      
M&T Session Database                   running          168765      
M&T Log Processor                      running          172024      
Certificate Authority Service          running          3153608     
EST Service                            running          3158662     
SXP Engine Service                     disabled                     
TC-NAC Service                         disabled        
PassiveID WMI Service                  disabled                     
PassiveID Syslog Service               disabled                     
PassiveID API Service                  disabled                     
PassiveID Agent Service                disabled                     
PassiveID Endpoint Service             disabled                     
PassiveID SPAN Service                 disabled                     
DHCP Server (dhcpd)                    disabled                     
DNS Server (named)                     disabled                     
ISE Messaging Service                  running          3173769     
ISE API Gateway Database Service       running          162077      
ISE API Gateway Service                running          167499      
ISE pxGrid Direct Service              disabled                     
Segmentation Policy Service            disabled                     
REST Auth Service                      disabled                     
SSE Connector                          disabled                     
Hermes (pxGrid Cloud Agent)            disabled                     
McTrust (Meraki Sync Service)          disabled                     
ISE Node Exporter                      running          176959      
ISE Prometheus Service                 running          178921      
ISE Grafana Service                    disabled                     
ISE MNT LogAnalytics Elasticsearch     disabled                     
ISE Logstash Service                   disabled                     
ISE Kibana Service                     disabled                   

Once you've confirmed all of the ISE processes are running, use the comand application configure ise to start the reset process. (Again, make sure this is the secondary node first) When prompted, select option 20 for Reset Context Visibility and press y when prompted to proceed. 

WRMEMISE01/admin#application configure ise

Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data
[4]Reset M&T Database
[5]Refresh Database Statistics
[6]Display Profiler Statistics
[7]Export Internal CA Store
[8]Import Internal CA Store
[9]Create Missing Config Indexes
[10]Create Missing M&T Indexes
[12]Generate Daily KPM Stats
[13]Generate KPM Stats for last 8 Weeks
[14]Enable/Disable Counter Attribute Collection
[15]View Admin Users
[16]Get all Endpoints
[19]Establish Trust with controller
[20]Reset Context Visibility
[21]Synchronize Context Visibility With Database
[22]Generate Heap Dump
[23]Generate Thread Dump
[24]Force Backup Cancellation
[25]CleanUp ESR 5921 IOS Crash Info Files
[26]Recreate undotablespace
[27]Reset Upgrade Tables
[28]Recreate Temp tablespace
[29]Clear Sysaux tablespace
[30]Fetch SGA/PGA Memory usage
[31]Generate Self-Signed Admin Certificate
[32]View Certificates in NSSDB or CA_NSSDB
[33]Enable/Disable/Current_status of RSA_PSS signature for EAP-TLS
[34]Check and Repair Filesystem
[35]Enable/Disable/Current_status of Audit-Session-ID Uniqueness
[36]Localised ISE Install
[0]Exit

20
This will remove all data from Context Visibility. Do you want to proceed [y/n]: y

This process will run through and reset the database that is used for this service. Once the process has finished, you will see a message saying to proceed with running the reset on the Primary Admin Node before proceeding. Stop here and log into the CLI of the Primary node and continue to the next step. 

Checking ISE persona
  - Done
Stopping monit services
  - Done
Stopping ISE indexing engine
  - Done
Unzip and making changes to vcs jar
  - Done
Modifying ISE indexing engine configuration
  - Done
Removing ISE indexing engine data folder
  - Done
Starting ISE indexing engine
  - Done
Performing reset of ISE indexing engine
  - Done
Stopping ISE indexing engine
  - Done
Removing backup of vcs jar
  - Done
Reverting changes to ISE indexing engine configuration
  - Done
Please proceed with running reset indexing engine on Primary Admin Node(WRMEMISE02) now. Once reset finishes on Primary Admin Node, please come back and press Y here
Is reset indexing engine done on Primary Adming Node(WRMEMISE02) [ Y/N ] :

Once on the primary node, run the application configure ise command and select option 20 to reset Context Visibility and press Y when prompted with the are you sure message. 

WRMEMISE02/admin#application configure ise

Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data
[4]Reset M&T Database
[5]Refresh Database Statistics
[6]Display Profiler Statistics
[7]Export Internal CA Store
[8]Import Internal CA Store
[9]Create Missing Config Indexes
[10]Create Missing M&T Indexes
[12]Generate Daily KPM Stats
[13]Generate KPM Stats for last 8 Weeks
[14]Enable/Disable Counter Attribute Collection
[15]View Admin Users
[16]Get all Endpoints
[19]Establish Trust with controller
[20]Reset Context Visibility
[21]Synchronize Context Visibility With Database
[22]Generate Heap Dump
[23]Generate Thread Dump
[24]Force Backup Cancellation
[25]CleanUp ESR 5921 IOS Crash Info Files
[26]Recreate undotablespace
[27]Reset Upgrade Tables
[28]Recreate Temp tablespace
[29]Clear Sysaux tablespace
[30]Fetch SGA/PGA Memory usage
[31]Generate Self-Signed Admin Certificate
[32]View Certificates in NSSDB or CA_NSSDB
[33]Enable/Disable/Current_status of RSA_PSS signature for EAP-TLS
[34]Check and Repair Filesystem
[35]Enable/Disable/Current_status of Audit-Session-ID Uniqueness
[36]Localised ISE Install
[0]Exit

20
This will remove all data from Context Visibility. Do you want to proceed [y/n]: y

Once the process starts, you will be prompted with a message that asks if the reset has been started on the secondary node, press Y to continue and let the process finish running. Once it's done, you will see a message saying that you can continue the process on the secondary node. 

Checking ISE persona
  - Done
Reset of indexing engine on this node needs to be run only after running it on Secondary Admin Node(WRMEMISE01)
Is reset indexing engine started on Secondary Admin Node(WRMEMISE01) [ Y/N ] :y
Verifying ISE indexing engine services on Secondary Admin Node
  - Done
Stopping monit services
  - Done
Stopping ISE indexing engine
  - Done
Unzip and making changes to vcs jar
  - Done
Modifying ISE indexing engine configuration
  - Done
Removing ISE indexing engine data folder
  - Done
Starting ISE indexing engine
  - Done
Performing reset of ISE indexing engine
  - Done
Stopping ISE indexing engine
  - Done
Removing backup of vcs jar
  - Done
Reverting changes to ISE indexing engine configuration
  - Done
Starting ISE indexing engine
  - Done
Starting monit services
  - Done
Reset of indexing engine on this node is complete. Please proceed with remaining steps on Secondary Admin Node(WRMEMISE01)
Reset of Context Visibility is successful on this node

Log back into your secondary ISE node and press Y to finish the reset process. 

Is reset indexing engine done on Primary Adming Node(WRMEMISE02) [ Y/N ] :y
Verifying ISE indexing engine services on Primary Admin Node
  - Done
Starting ISE indexing engine
  - Done
Starting monit services
  - Done
Reset of Context Visibility is successful on this node

Once that's completed, you should now be able to log back into your ISE Web GUI and your context visibility dashboard should display correctly. 

Queue Link Error

The second issue i found was that I was receiving a number of these queue link errors related to certificates after my second ISE node deployment. In the alarms section on the main dashboard I received an error regarding Queue Link. When i clicked on the error to see the details, it showed the below messages. This issue turns out to be related to invalid certificates for the ISE nodes which makes sense as I replaced one of my nodes in the deployment. 

Once again, this turned out to be a pretty common issue and an easy fix. To resolve the issue, you need to replace the ISE Messaging Service certificates that are used by the ISE nodes. To do that, navigate to Administration -> System -> Certificates

On the certificates page, navigate to Certificate Management -> Certificate Signing Requests and then click Generate Certificate Signing Requests (CSR)

On the Certificate Signing Requests page under the Usage dropdown, select ISE Messaging Service, once selected, you will be prompted to select the nodes that you wish to regenerate the certificates for. Make sure you select both nodes here and click OK. 

About 50% of the time, that's all that you need to do and you should no longer see the queue link errors in the alarms dashboard. However sometimes you need to take it a step further and also regenerate the ISE node self signed CA certificates as well. To do that, go back into the Generate Certificate Signing Requests page, and from the Usage drop down, select ISE Root CA and then select Regenerate Root CA For All Nodes and click OK. 

Once the CA certificates have been regenerated, re-create the ISE Message Service certificates once more and that should resolve the Queue Link Error issues. If you've noticed anything missing or have any issues with this post or just want to say Hi, please leave a comment and let me know.