Fortinet SD-WAN still requires proper routing. For small deployments, static routing is sufficient. For bigger deployments, OSPF or BGP can be used. For large-scale deployments, the recommended option is BGP because of its better scalability and other less obvious benefits.
In the previous blog post, I described SD-WAN operations for traffic initiated from branches/spokes to data centers/hubs – https://letsnet.eu/fortinet-sd-wan-deep-dive/. But what about traffic initiated from data centers to branches? Which path will be used and how can we ensure the best path is selected? We cannot use the same logic as we do for branches. Imagine an infrastructure with 1000 branches, each location having two connections to the datacenter. In this scenario, it would require 2000 Performance SLA monitors configured on the HUBs. Such an approach doesn’t scale well, is difficult to manage, and will generate a lot of unnecessary traffic.
How to leverage BGP?
BGP communities allow tagging or labeling certain routes to convey additional information to other BGP routers in the network. This mechanism is used to group routes and influence routing decisions without modifying the actual route attributes. In our case, SPOKES using BGP communities will notify the HUB which routes are meeting SLA requirements. Based on this information, the HUB will prioritize certain routes.
Let’s elaborate on the configuration in detail. In my scenario, I use 2 connections from SPOKES to the HUB: HUB1-VPN1 uses MPLS, and HUB1-VPN2-2 uses LTE.
SPOKE
1. Routes maps to send specific bgp community to bgp neighbors:
config router route-map
edit "Priority_1"
config rule
edit 1
set set-community "65100:1"
next
end
next
edit "Priority_2"
config rule
edit 1
set set-community "65100:2"
next
end
next
edit "Priority_4"
config rule
edit 1
set set-community "65100:4"
next
end
end2. Assign route maps to BGP neighbors. When link quality is satisfactory, the spoke will send a route map defined under set route-map-out-preferable. If not, the route map set route-map-out will be sent:
config router bgp
config neighbor
edit "172.30.249.125"
set advertisement-interval 1
set capability-graceful-restart enable
set link-down-failover enable
set soft-reconfiguration enable
set description "HUB1-VPN2-2"
set interface "HUB1-VPN2-2"
set remote-as 65100
set route-map-out "Priority_4"
set connect-timer 10
set route-map-out-preferable "Priority_2"
next
edit "172.30.249.61"
set advertisement-interval 1
set capability-graceful-restart enable
set link-down-failover enable
set soft-reconfiguration enable
set description "HUB1-VPN1"
set interface "HUB1-VPN1"
set remote-as 65100
set route-map-out "Priority_4"
set connect-timer 10
set route-map-out-preferable "Priority_1"
next
endFortiGate will send route maps according to SLA outputs:
MPLS meets SLA -> Route map Priority_1 (set-community “65100:1”)
MPLS out of SLA -> Route map Priority_4 (set-community “65100:4”)
LTE meets SLA -> Route map Priority_2 (set-community “65100:2”)
LTE out of SLA -> Route map Priority_4 (set-community “65100:4”)
When both connections meets SLA, I want to use the MPLS line for traffic initiated from the data center. This is why I use different route maps Priority_1 and Priority_2
3. Under SD-WAN define which BPG neighbour to monitor, assign proper interface and health-check:
config system sdwan
config neighbor
edit "172.30.249.125"
set health-check "HUB1_HC"
set sla-id 1
set member 6
next
edit "172.30.249.61"
set health-check "HUB1_HC"
set sla-id 1
set member 5
next
endHUB
1. Community-list to match bgp community received from spokes:
config router community-list
edit "Priority_1"
config rule
edit 1
set action permit
set match "65100:1"
next
end
next
edit "Priority_2"
config rule
edit 1
set action permit
set match "65100:2"
next
end
next
edit "Priority_4"
config rule
edit 1
set action permit
set match "65100:4"
next
end
end
2. Transfer matched communities into route maps and set route priorities:
config router route-map
edit "RM-VPN-Priority"
config rule
edit 1
set match-community "Priority_1"
set set-priority 1
next
edit 2
set match-community "Priority_2"
set set-priority 2
next
edit 4
set match-community "Priority_4"
set set-priority 4
next
end
endTesting in the lab
Unfortunately, only the LTE connection is working, so I can show changes for routes based on the LTE connection. Looking into IPsec HUB1-VPN2-2, the tested parameters like packet loss, latency, and jitter meet SLA requirements
Spoke should send route map Priority_2 and HUB should set priority to 2 for that route, see HUB routing table (priority value is represented in bracket at the end):
HUB1 # get router info routing-table bgp
Routing table for VRF=0
B 10.223.43.0/24 [200/0] via 172.30.249.193 (recursive is directly connected, VPN2-2), 02:16:34, [2/0]Let’s simulate bad quality connection, I have changed SLA to different value, now the same latency doesn’t meet SLA.
Spoke should send route map Priority_4 and HUB should set priority to 4:
HUB1 # get router info routing-table bgp
Routing table for VRF=0
B 10.223.43.0/24 [200/0] via 172.30.249.193 (recursive is directly connected, VPN2-2), 00:00:05, [4/0]Lower priority values are better. Unfortunately, in my lab, only one connection is working. However, if there is another connection with a lower priority than 4, FortiGate will use that one. If there are multiple routes with the same priority, FortiGate will balance traffic across different links – for example, in my case, all connections are out of SLA with priority 4.
Summary
Using BGP communities allows you to easily notify central locations about the best quality routes without any additional configurations and probes from the HUB perspective. For large-scale implementations, this approach is much better and recommended due to better scalability and easier maintenance. Similar outcome can be achieved leveregin BGP tags, but in my opinion it is more complicate. Result is the same, so like always, I’m a fan of simpler solution.