ACLs and TCAM

Recently at work we have run into an issue with TCAM space on some of our switches due to the large number of ACL's. So i decided to do some digging into TCAM and how ACL's use the space etc. 

What is TCAM

TCAM stands for Ternary Content Addressable Memory and is basically an extension of CAM and is used by switches when processing things like ACL's in order to be able to filter traffic at line rates. TCAM does this by matching on 3 different states unlike the CAM, which only matches on a 1 or a 0. TCAM allows for a third state which is the 'I don't care' or x bit. For example, if a binary value is 0111XX10, then the TCAM will match any of the following binary values:

01110010
01110110
01111010
01111110

This X bit, or don't care bit comes in play when specifying a subnet mask in an ACE for example. Each TCAM entry has 3 parts, the Value, Mask and the Result. The Value and Mask fields are 134 bits in length. These fields relate directly to each other in that if a bit in the Mask field is set, then the corresponding bit in the Value field must match and if a Mask bit is not set, then the corresponding Value bit doesn't matter. The results field in a TCAM entry is used to determine the action taken. With TCAM, this action isn't just a permit or a deny. It can be a QoS policer or a next hop value as well. 

When a switch populates the TCAM, it will organise the entries by the mask. Each unique mask will have up to 8 source/destination value pairs associated with it. If a mask source/destination pair has more than 8 values, a new mask pair is created allowing for another 8 unique entries. This organisation is what allows for the really fast lookup of the longest matching pair of source/destination values in TCAM.

Let's take a look at an example ACL

ip access-list extended TCAM-EXAMPLE
permit tcp any 10.20.0.0 0.0.0.255 eq 123
permit tcp 192.168.1.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 80
permit tcp 10.12.18.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 443
permit tcp 10.28.0.0 0.0.127.255 192.168.2.0 0.0.0.255 eq 443
 

When the switch Feature Manager goes to import these into the TCAM, it first looks at how many unique value and mask pairs there are. There are 3 unique pairs in the above ACL. 

1. Match 24 bits of the destination address. The any keyword in the source gets a don't care value (line 1)
2. Match 24 bits of the source address AND 24 bits of the destination address (line 2 and 3)
3. Match 17 bits of the source address AND 24 bits of the detination address (line 4)

These masks are then sorted and stored in the TCAM. Remember this is a very simplified example. The following would be the mark and value pairs for these entries.

Mask 1:

Match any of the source bits, and match the first 24 of the destination bits. 

• IP Protocol: TCP
• Source IP: 0.0.0.0 (any)
• Destination IP: 10.20.0.0
• Destination Port: 123
• Action: Permit

Mask 2:

Match 24 bits of the source address bits and match 24 bits of the destination address bits.

• IP Protocol: TCP
• Source IP: 192.168.1.0
• Destination IP: 10.20.0.0
• Destination Port: 80
• Action: Permit

And also

• IP Protocol: TCP
• Source IP: 10.12.18.0
• Destination IP: 10.20.0.0
• Destination Port: 443
• Action: Permit

Mask 3:

Match 17 of the source bits and match 24 of the destination bits.

• IP Protocol: TCP
• Source IP: 10.28.0.0
• Destination IP: 192.168.2.0
• Destination Port: 443
• Action: Permit

As you can see when you have ACL entries with similar mask pairs, this makes for an efficient lookup process when entered into the TCAM. 

When a switch receives a frame, the first 200 bytes of that frame are copied to the Forwarding Controller. The Forwarding Controller is responsible for the forwarding and processing of frames in relation to VLANS, QoS, ACLs etc. The first 200 bytes of an ethernet frame contain all of the required information to be able to make those decisions. When making a forwarding decision, depending on the switch architecture, there will be 3 TCAM lookups.

1. L2 Lookup
2. QoS and ACL lookup
3. L2/L3 forwarding lookup

There is of course more steps to the whole process but those are the steps relating to TCAM. 

TCAM Allocation

You can define on the switch how the TCAM is configured with regard to what resources are allocated and to where by using the switches SDM (Switching Database Manager) function. Because TCAM space is a finite resource, the switch will use a "template" in order to determine how much of each resource to allocate for each TCAM function. There are several SDM templates to choose from and this will vary depending on the switch model. Changing the SDM template is a simple process that requires a single command and a device reboot. There are 5 main TCAM areas which are listed below. 

• L2
• L3
• QoS ACEs
• Security ACEs
• IPv6

As mentioned earlier, Cisco switch give you the ability to choose the way you carve up your TCAM by using various templates. Usually the default template that is configured will work for most deployments, however depending on your switch function you may want to change it. To change the SDM template that the switch is using, use the command ⁠⁠⁠⁠⁠⁠⁠sdm prefer <template>. 

The following are the available templates on a Catalyst 3560x switch.

 access                                              Access bias
 default                                             Default bias
 dual-ipv4-and-ipv6                         Support both IPv4 and IPv6
 indirect-ipv4-and-ipv6-routing       Supports more V4 and V6 Indirect Routes
 routing                                             Unicast bias
 vlan                                                  VLAN bias

Once you have configured the SDM template, you will need to reload the switch for the changes to take effect. 

You can also view your TCAM utilisation by using the command  ⁠⁠⁠⁠⁠⁠⁠show platform tcam utilization

SW01#sh plat tcam ut CAM Utilization for ASIC# 0                      Max            Used                                             Masks/Values    Masks/values Unicast mac addresses:                       6364/6364         78/78 IPv4 IGMP groups + multicast routes:         1072/1072          5/5 IPv4 unicast directly-connected routes:      6144/6144          1/1 IPv4 unicast indirectly-connected routes:    2048/2048         61/61 IPv6 Multicast groups:                        112/112          17/17 IPv6 unicast directly-connected routes:        74/74            2/2 IPv6 unicast indirectly-connected routes:      32/32            6/6 IPv4 policy based routing aces:               412/412          12/12 IPv4 qos aces:                                512/512           6/6 IPv4 security aces:                           924/924          44/44 IPv6 policy based routing aces:                18/18            8/8 IPv6 qos aces:                                 22/22            5/5 IPv6 security aces:                            60/60           17/17 Note: Allocation of TCAM entries per feature uses a complex algorithm. The above information is meant to provide an abstract view of the current TCAM utilization