Cisco ISE M&T Database Reset
Recently had an issue with failed backups on my Cisco ISE server which turned out to be due to high disk usage. The alarm was initially saying the disk usage was above 70-80+%. I was able to verify this by logging into the CLI of the Cisco ISE server and using the command show disk. (I've removed some output for simplicity sake)
server/admin# show disk
Internal filesystems:
/opt : 77% used ( 425813348 of 588293360)I would like to note that this wasn't my home Lab server and this server has TAC support which while wasn't required to resolve the issue, they did help me confirm the cause and provide the resolution. TAC support found that the culprit of the high disk usage, was a result of excessive oracle database logs that the server had collected over the years.
There are two things you can do here to try and resolve the issue. The first is to try a Purge M&T Operational Data and the second is to perform a Reset M&T Database. The Purge is non impacting on your ISE server and will not cause an outage however the Reset will cause the all of the ISE services to be restarted on the node and thus cause an outage on that server. I decided to first try the Purge M&T Operational Data option as the second option would require an outage and therefore a change request which would take time. To perform either of these steps though, you will need to be logged into the CLI of your Cisco ISE server. Once logged in, use the command application configure ise.
server/admin# application configure ise
Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data
[4]Reset M&T Database
[5]Refresh Database Statistics
[6]Display Profiler Statistics
[7]Export Internal CA Store
[8]Import Internal CA Store
[9]Create Missing Config Indexes
[10]Create Missing M&T Indexes
[11]Enable/Disable ACS Migration
[12]Generate Daily KPM Stats
[13]Generate KPM Stats for last 8 Weeks
[14]Enable/Disable Counter Attribute Collection
[15]View Admin Users
[16]Get all Endpoints
[19]Establish Trust with controller
[20]Reset Context Visibility
[21]Synchronize Context Visibility With Database
[22]Generate Heap Dump
[23]Generate Thread Dump
[24]Force Backup Cancellation
[25]CleanUp ESR 5921 IOS Crash Info Files
[26]Recreate undotablespace
[27]Configure TCP params
[28]Fetch SGA/PGA Memory usage
[0]ExitIn the above menu, select option 3 to perform the Purge. You will then be prompted with how many days of data to keep, I selected 7 days here but you can select any number you want between 1 and 90. Once done, you will then be prompted to confirm the purge. Type Y and hit enter to begin the process.
Enter number of days to be retained in purging MnT Operational data [between 1 to 90 days]
For instance, Entering 20 will purge MnT Operational data older than 20 days
Enter 'exit' to return to the main menu without purging
Enter days to be retained: 7
You are about to purge M&T data older than 7 from your database.
Are you sure you want to proceed? y/n [n]: y
M&T Operational data older than 7 is getting removed from database
Once that's done, check your disk usage by running the command show disk from the ISE CLI once again.
server/admin# show disk
Internal filesystems:
/opt : 77% used ( 425813348 of 588293360)As you can see, for me, the Purge did nothing to resolve the disk usage issue and it required me to perform a Reset of the M&T Database. To do the Database Reset, once again run the command application configure ise from the CLI and this time select option 4 to run the Reset M&T Database.
server/admin# application configure ise
Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data
[4]Reset M&T Database
[5]Refresh Database Statistics
[6]Display Profiler Statistics
[7]Export Internal CA Store
[8]Import Internal CA Store
[9]Create Missing Config Indexes
[10]Create Missing M&T Indexes
[11]Enable/Disable ACS Migration
[12]Generate Daily KPM Stats
[13]Generate KPM Stats for last 8 Weeks
[14]Enable/Disable Counter Attribute Collection
[15]View Admin Users
[16]Get all Endpoints
[19]Establish Trust with controller
[20]Reset Context Visibility
[21]Synchronize Context Visibility With Database
[22]Generate Heap Dump
[23]Generate Thread Dump
[24]Force Backup Cancellation
[25]CleanUp ESR 5921 IOS Crash Info Files
[26]Recreate undotablespace
[27]Configure TCP params
[28]Fetch SGA/PGA Memory usage
[0]Exit
4
You will then be prompted with an are you sure message warning that the application will be restarted so once you are ready to proceed, type Y and hit enter to begin the reset. Remember this will cause an outage for the ISE server you are performing this on.
You are about to reset the M&T database. Following this operation, application will be restarted.
Are you sure you want to proceed? y/n [n]: yThis process took about 10 minutes for me but that will obviously depend on your ISE node specs and configuration etc.
Stopping application
Stopping ISE Monitoring & Troubleshooting Log Processor...
PassiveID WMI Service is disabled
PassiveID Syslog Service is disabled
PassiveID API Service is disabled
PassiveID Agent Service is disabled
PassiveID Endpoint Service is disabled
PassiveID SPAN Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server...
Stopping ISE Certificate Authority Service...
Stopping ISE EST Service...
ISE Sxp Engine Service is disabled
Stopping TC-NAC Service ...
VA Service is not running
ISE VA Database is not running
Segmentation Policy Service is disabled
REST Auth Service is disabled
Stopping ISE Messaging Service...
Stopping ISE API Gateway Service...
Stopping ISE API Gateway Database Service...
Stopping docker daemon...
Stopping ISE Profiler Database...
Stopping ISE Indexing Engine...
Stopping ISE Monitoring & Troubleshooting Session Database...
Stopping ISE AD Connector...
Stopping ISE Database processes...
Starting Database only
Creating ISE M&T database tables...
Restarting application
M&T Log Processor is not running
PassiveID WMI Service is disabled
PassiveID Syslog Service is disabled
PassiveID API Service is disabled
PassiveID Agent Service is disabled
PassiveID Endpoint Service is disabled
PassiveID SPAN Service is disabled
ISE pxGrid processes are disabled
ISE Application Server process is not running
Certificate Authority Service is not running
EST Service is not running
ISE Sxp Engine Service is disabled
ISE TC-NAC Service is disabled
Segmentation Policy Service is disabled
REST Auth Service is disabled
ISE Messaging Service is not running.
ISE API Gateway Service is not running.
ISE API Gateway Database Service is not running.
docker daemon is not running
ISE Profiler Database is not running
ISE Indexing Engine is not running
M&T Session Database is not running
ISE AD Connector is not running
Stopping ISE Database processes...
ISE Database processes already running, PID: 1824
Stopping ISE Database processes...
Starting docker daemon ...
Starting ISE Messaging Service...
Starting ISE API Gateway Database Service...
Starting ISE Monitoring & Troubleshooting Session Database...
Starting ISE Profiler Database...
Starting ISE API Gateway Service...
Starting ISE Application Server...
Starting ISE Monitoring & Troubleshooting Log Processor...
Starting ISE Indexing Engine...
Starting ISE Certificate Authority Service...
NSS database for CA Service is ready
ISE EST service is already running, PID: 12758
Starting ISE AD Connector...
Note: ISE Processes are initializing. Use 'show application status ise'
CLI to verify all processes are in running state. Once the reset has completed, you will be back at the application configure ise prompt. Type 0 and hit enter to exit the menu.
Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data
[4]Reset M&T Database
[5]Refresh Database Statistics
[6]Display Profiler Statistics
[7]Export Internal CA Store
[8]Import Internal CA Store
[9]Create Missing Config Indexes
[10]Create Missing M&T Indexes
[11]Enable/Disable ACS Migration
[12]Generate Daily KPM Stats
[13]Generate KPM Stats for last 8 Weeks
[14]Enable/Disable Counter Attribute Collection
[15]View Admin Users
[16]Get all Endpoints
[19]Establish Trust with controller
[20]Reset Context Visibility
[21]Synchronize Context Visibility With Database
[22]Generate Heap Dump
[23]Generate Thread Dump
[24]Force Backup Cancellation
[25]CleanUp ESR 5921 IOS Crash Info Files
[26]Recreate undotablespace
[27]Configure TCP params
[28]Fetch SGA/PGA Memory usage
[0]Exit
0As per the last line in the output of the Reset M&T Database process, verify that the ISE application services have started by using the command show application status ise.
server/admin# show application status ise
ISE PROCESS NAME STATE PROCESS ID
--------------------------------------------------------------------
Database Listener running 4464
Database Server running 114 PROCESSES
Application Server initializing
Profiler Database running 10114
ISE Indexing Engine running 16701
AD Connector running 17820
M&T Session Database running 9897
M&T Log Processor running 14854
Certificate Authority Service running 17670
EST Service running 19222
SXP Engine Service disabled
Docker Daemon running 6121
TC-NAC Service disabled
pxGrid Infrastructure Service disabled
pxGrid Publisher Subscriber Service disabled
pxGrid Connection Manager disabled
pxGrid Controller disabled
PassiveID WMI Service disabled
PassiveID Syslog Service disabled
PassiveID API Service disabled
PassiveID Agent Service disabled
PassiveID Endpoint Service disabled
PassiveID SPAN Service disabled
DHCP Server (dhcpd) disabled
DNS Server (named) disabled
ISE Messaging Service running 7242
ISE API Gateway Database Service running 9134
ISE API Gateway Service running 12648
Segmentation Policy Service disabled
REST Auth Service disabled
SSE Connector disabled Once the reset has completed and the application services have all started again, you can verify your disk usage to confirm that the disk space has been cleared up by using the show disk command.
server/admin# show disk
Internal filesystems:
/opt : 24% used ( 131580440 of 588293360)And that's it, my backup issue was now resolved and my ISE server status was back to normal. Thanks for checking out my blog. If you've noticed anything missing or have any issues setting this up, please leave a comment and let me know.
Add new comment