Table of contents
Abbreviations
Chapter 1: Introduction
1.1 Thesis structure
Chapter 2: SD-WAN
2.1 General SD-WAN Architecture and Components
2.1.1 SD-WAN Edge
2.1.2 SD-WAN Controller
2.1.3 Service Orchestrator
2.2 Benefits of using SD-WAN
2.2.1 Overlay Networks
2.2.2 Hybrid WAN
2.2.3 Dynamic Path Selection and Increased Bandwidth
2.2.4 Zero Touch Provisioning
2.2.5 Easy access to the Cloud
2.2.6 Security
2.2.7 Per Application Routing
2.3 SD-WAN Architecture Types
2.3.1 On-Prem-Only
2.3.2 Cloud-Enabled
2.3.3 Cloud-Enabled plus Backbone
2.4 SD-WAN Deployment Models
Chapter 3: Used Technologies
3.1 gRPC
3.2 VxLAN
3.3 VRF-lite
3.4 STUN
3.5 Etherws
Chapter 4: EveryWAN
4.1 EveryWAN Architecture
4.2 EveryWAN Services
4.3 EveryEdge
4.4 EveryController
4.5 EveryGUI
4.6 Authentication and Registration Procedure
4.7 Tenant Management
4.8 EveryEdge Management
4.9 Overlay Management
Chapter 5: Southbound
5.1 VxLAN Implementation in Linux
5.2 VRF Implementation in Linux
5.3 Southbound API
5.3.1 Device API
5.3.2 Configuration API
5.4 NAT traversal with VxLAN
5.5 Management Tunnels
Chapter 6: Northbound
6.1 Northbound API
6.1.1 Edge API
6.1.2 Overlay API
6.1.3 Tenant API
6.2 Create Overlay Algorithm
6.3 Remove Overlay Algorithm
6.4 Generic Resource Manager
Chapter 7: Persistence with MongoDB
7.1 Tenants
7.2 Devices
7.3 Overlays
7.4 Configuration
Chapter 8: Emulation Environment and Demo
8.1 Emulation Environment
8.1.1 Open and NATed Access
8.2 Emulated Network Topology
8.3 Demo
Chapter 9: Conclusion and Future Work
9.1 Future Improvements
Bibliography
Acknowledgements
Abbildung in dieser Leseprobe nicht enthalten
Generally, enterprises use WAN connectivity to connect branch office to central office, or to connect datacentres separated by distance. In the past, WAN connections often use technology that require special proprietary hardware for each different type of connections and providers. In this way we had a monolith CPE model, that contrasted innovations, introducing HW and SW incompatibilities when new functions arrive, and making also difficult the network management for companies with several branch offices. This new technology, called SD-WAN that stands for Software Defined – Wide Area Network, is come out to solve many problems that affect the traditional WAN.
Abbildung in dieser Leseprobe nicht enthalten
Figure 1 - Traditional WAN and SD-WAN 1
The SD-WAN technology is born from vendors, and nowadays is adopted by the major of them. The goal is to reduce the CAPEX and OPEX for the network company through the simplification of branch office setup, supporting centralized control and network service delivery and leveraging multiple connection types on the same router to improve application performance and reliability. SD-WAN is an architecture that leverages SDN principles, and was aiming at simplify the management and operation of the networks (with a particular focus on WAN scenarios) by decoupling the networking hardware from its control programs and using software and open APIs to abstract the infrastructure and manage the connectivity and the services. At the time of research and work began of this thesis, there was no complete open source SD-WAN solutions. The only open source solution present is the one proposed by FlexiWAN 2. With the work of this thesis we want to implement a new open source SD-WAN called EveryWAN, exploiting the new concepts of Network Slicing and Overlays. In EveryWAN the different types of service are segmented through virtual independent Overlay networks that abstract the underlying network. In so doing we want to facilitate the use of multiple WAN connections and the assignment of different QoS and security for different services. We start form an SRv6 SDN solution that was extended to an SD-WAN which implements multiple Overlays to create End-to-End Slices among multiple sites. The work has focused on design the high-level open source SD-WAN architecture, design and implement the vCPE called EveryEdge able to support ZTP, design and implement the SD-WAN Controller/Orchestrator called EveryWAN that is able to control and program the EveryEdges and deploy services. The services are based on L3 tunnels that interconnects multiple customer sites transporting traffic of a specific applications. The EveryController was developed based on SDN and NFV principles and made it capable to automate and orchestrate e2e services giving to the users to control their sites. EveryWAN is based completely on Linux platform and in the implementation we assume that there is an IaaS that provides the computing workloads. The final part of the work consists in implement an emulated geographical network in which deploy and test the EveryWAN and related network service.
In Chapter 2 is described the SD-WAN state-of-the-art, taking information from SD-WAN vendors (like Cisco, Juniper and Oracle) form MEF standardization body and from technological portals like SDxCentral. In that chapter we are going to describe the general aspects of this technology, the benefits it brings, the architecture, the main components, and the deployment models. In Chapter 3 we are going to provide a brief view of the main Open Source technologies used to build our Open Source SD-WAN Solution. Following, in Chapter 4 a full description of EveryWAN Architecture is provided together with the management mechanisms supported. In Chapter 5 and Chapter 6 we are going more in deep in our Open Source SD-WAN to describe respectively the Southbound API and Northbound API exposed and the implemented algorithms. In Chapter 7 we will talk about the data persistence and how the collections ware organized in the Database to save SD-WAN network configuration and status. Chapter 8 will provide a description of the emulated environment and the network topology that was used to realize a demo of EveryWAN. Finally, in Chapter 9 we will discuss the obtained results compared to the state-of-the-art and the possible improvements and future works that can be done starting from our SD-WAN.
The Software-Defined Wide Area Network (SD-WAN) is a specific application of the Software-Defined Networking (SDN) technology applied to the WAN networks such as MPLS, fixed line broadband XDSL/Fiber, mobile broadband 4G/5G and satellite link 3.
The SD-WAN technology helps organizations connect branch office and central office sites spread over vast geographic areas with remote data centers, and multiple cloud environments. By decoupling the hardware from the software using cloud-based technologies and specialized software to create an abstraction layer. In this way organizations have greater flexibility and control over how data transfers take place. In addition, SD-WANs help businesses boost agility, availability, and performance, while potentially lowering costs by optimizing the use of resources in a multisite configuration. This technology is becoming so interesting due to its ability to incorporate the best networking technology available for a specific location thus transforming the access networks in intelligent and dynamic platforms. SD-WAN also alleviates some of the challenges related to traditional WANs and Multiprotocol Label Switching (MPLS) lines, like: 4
- scalability
- bandwidth demand increase
- vendor lock-in
- support applications running outside the enterprise environment
- flexibility in the architecture
In this section a generic SD-WAN architecture will be described briefly, based on the information provided by the MEF standardization body. The main components of SD-WAN architecture are SD-WAN Edge, SD-WAN Controller, and Service Orchestrator. The SD-WAN Edge component is the perimeter device that provide access for customers to an already existing networks. SD-WAN Controller and Service Orchestrator together provide centralized control and management to the SD-WAN. An additional component is the Subscriber Web Portal, where authorized users can activate the scenario, communicate with the Controller/Orchestrator to set up services, security policies, QoS parameters and monitor the SD-WAN Edges and the traffic between them. All these configuration tasks are done using a GUI and no longer a CLI. In the following subsections are described the fundamental functionality of SD-WAN Edge, SD-WAN Controller and Service Orchestrator.
Abbildung in dieser Leseprobe nicht enthalten
Figure 2- Generic SD-WAN architecture 5
The SD-WAN Edges are the components deployed in the enterprise sites that you want to interconnect with the SD-WAN technology. SD-WAN Edge functionality can be provided by both physical CPE device and virtual-CPE (vCPE). SD-WAN brings the new concept of vCPE or uCPE, in this case the Edge functionality are implemented as a software-based VNF. vCPE can be run on general compute platform, therefore can be deployed both at the customer premises and in cloud environments. SD-WAN Edge creates and terminates secured tunnels over different wired or wireless underlay networks. SD-WAN Edge perform application routing over one or more WAN links, performance measurements, and WAN optimization functions such as compression, packet reordering, packet deduplication and forward error correction. SD-WAN Edge also performs security policy enforcement and application-based QoS 6.
SD-WAN Controller manages physical or virtual SD-WAN Edge devices that are associated with him. It is responsible for authentication and activation, IP address management, tunnels and policies settings onto SD-WAN Edge. SD-WAN Controller is always connected with SD-WAN Edge to monitor their status and that of the tunnels. The information about the QoS performance metrics of each tunnel are collected to be used by the Service Orchestrator. SD-WAN Controller use the northbound APIs to communicate with the Service Orchestrator, and the southbound APIs to control and configure the SD-WAN Edges 6.
The Service Orchestrator manages the SD-WAN service lifecycle. It’s responsible for service fulfilment, control the entire SD-WAN environment, collect data and do analysis, guarantee performance and security. For example, the Service Orchestrator configure the end-to-end SD-WAN services between SD-WAN Edges over multiple WAN links, based on application-aware security and QoS. Service Orchestrator use the northbound APIs to communicate with the applications and the southbound to communicate with the SD-WAN Controller 6. In some SD-WAN implementations SD-WAN Controller and Service Orchestrator are combined together in the same entity, that can be instantiated on-premises of the enterprise or in the cloud (SaaS Orchestration).
In this section will be shown and discussed all the advantages that can be obtained by the remote locations of an enterprise in using the SD-WAN technology. All the advantages listed below are general, and a specific solution of a vendor or open source can also implement only a part of them.
As described in the MEF standard, SD-WAN operate over Underlay Connectivity Services (UCS). Underlay Connectivity Services are network services offered by network providers in order to create connectivity between the subscriber sites. The access to the UCS can be through different networking technology, such as LTE, WiFi, Ethernet, and also the transport can be based on different technology like MPLS, IP Routing, and Ethernet switching.
SD-WAN can be deployed over multiple and different UCS that can have different performance and cost characteristics that will be used to provide benefits in term of cost and resilience 5.
The UCS is provided through Underlay Networks that is the physical network and it is responsible to deliver the packets across networks. The Underlay Network consist of physical devices (Router and Switches) and physical connections, it has some limitation due to static topology, limited scalability and long time to deploy services and functions.
Through the use of encapsulation such as GRE, IPsec, DMVPN, VXLAN, or a proprietary tunnelling technology. SD-WAN is able to create Overlay Networks on top of heterogeneous Underlay Network, also from different providers, keeping the addressing plane.
Abbildung in dieser Leseprobe nicht enthalten
Figure 3 – Overlay networks
The Overlay Network created by SD-WAN allow to have a dynamic topology (full-mash/hub-and-spoke) constituted by logical links and virtual devices. In this way is possible to have a more scalable network with the ability to deploy functions faster than in the Underlay Network. Although additional overhead is introduced due to packet encapsulation, Overlay Network will enable the new paradigms application-aware, policy-driven and orchestrated connectivity between SD-WAN users 5.
Compared to other technologies an SD-WAN solution is able to support multiple WAN connection concurrently. The various WANs can have different performance and costs, as it is for example in the cases of Internet, MPLS, 4G/5G etc. The ability to use multiple WANs leads to the concept of transport independence 7, which overcome many constraints introduced in the use of classic WANs and bring flexibility at the application level.
Abbildung in dieser Leseprobe nicht enthalten
Figure 4 - Hybrid WAN 8
Such a system is Fault Tolerant: in the case a WAN link breaks, the SD-WAN Edge device is able to redirect the traffic to the other WANs dynamically without the user or application noticing. A specific link, for example a mobile broadband connection, can be dedicated as a backup link if there is a possibility that all WAN links break, thus guaranteeing continuous availability.
An SD-WAN solution is able to move traffic form one path to another on the fly in order to guarantee a given QoS. This is possible thanks to the ability to measure the performance of each available link. In this way, latency sensitive application traffic can be routed to the lowest latency link available, while the traffic that is latency tolerant and require more bandwidth can leverage the higher bandwidth pipe.
In so doing, the traffic traversing on SD-WAN network can be aggregated on all available links, thus having an available bandwidth that is the sum of the bandwidth of all the lines 9. This translates into the ability to facilitate the band increase respect other technologies.
Another important SD-WAN capability is Zero Touch Provisioning (ZTP), it’s a benefit that allow the systems to be provisioned and configured automatically, reducing cost and limiting manual interventions 10.
Once the CPE connected to the SD-WAN Controller was authenticated, for example though a token, receive an initial device configuration that is pushed by the SD-WAN Controller. The initial device configuration may contain: NIC settings, network configurations, network policy, software update instructions and other additional option. ZTP simplifies the configuration of the CPEs making it centralized, and increases speed in adding a new remote site by removing the need for on-site configuration or remote assistance 11.
Abbildung in dieser Leseprobe nicht enthalten
Figure 5 - Zero Touch Provisioning
An SD-WAN solution can deploy gateways in Infrastructure–as-a-Service (IaaS) and Platform-as-a-Service (PaaS) cloud environment, these gateways are software entity. In this way not only the branch office and the central office are connected in an efficient way, bat also the connectivity to the cloud infrastructure is simplified. SD-WAN gives an enhanced and automated connectivity to Software-as-a-Service (SaaS) applications, such as Salesforce Office 365 or Dropbox, optimizing it by choosing the best available path. Path selection is based on performance measurement obtained from all the available path. In the case is observed a performance degradation on a given path, the traffic will be moved dynamically to a more optimal path 12. SD-WAN guarantee scalable and with appropriate QoS connections to SaaS, and since applications generally reside in public cloud security is also guaranteed, generally through IPsec 7.
Abbildung in dieser Leseprobe nicht enthalten
Figure 6 - SD-WAN cloud access 7
Another important aspect is the security, SD-WAN must guarantee security for both Edge-to-Edge and Edge-to-Controller communications. All the element belonging to the same SD-WAN must be authenticated and authorized at the first access to the network, this is achieved with the use of digital certificates. The certificates are used to identify each element and to establish secure Transport Layer Security or Datagram Transport Layer Security (TLS/DTLS) control channels between the SD-WAN Edge device and the SD-WAN Controller (control plane). In this way all the configuration and networking information that comes from the controller can be propagated through the secure encrypted channel 12.
The data plane, i.e. the part of the network that carry user traffic from one edge to the other, require both traffic isolation and firewalling in addition to the encryption. In order to guarantee confidentiality, integrity and authentication, IPsec is used in most cases. Compared to the traditional case, where the encryption keys are generated by the edge device using the process known as Internet Key Exchange (IKE). SD-WAN introduces automation and save computational resources by generating the encryption keys in the SD-WAN Controller and distributing them via the control plane to the SD-WAN Edge devices 12.
Abbildung in dieser Leseprobe nicht enthalten
Figure 7 - IPsec key exchange in SD-WAN architecture 12
Traffic segmentation is performed by the edge device directing each incoming traffic over a VPN, thus providing also routing table isolation and firewall policy based on segmentation. The firewall, that could be set up in terms of VNF in edge devices, perform access control and restricts user traffic.
Per application routing is another important feature of SD-WAN, which introduce the ability to collect data plane information to compute the optimal path for a given application traffic. The collected information from the available links are: packet loss, latency, jitter, bandwidth and load. The ability to consider these factors in the paths selection for the application flows, compared to those use by standard routing protocols (such as route prefixes, link-state information, metrics etc.) brings some advantages. These advantages are in terms of application’s SLA fulfilment, increased application performance and reduced network cost thanks to a more efficient traffic load-balancing. Per application routing is achieved through the combined use of three elements 13:
- Identification: need to identify the application of interest, by matching the Layer 3 and Layer 4 headers in the packets, in order to be able to create a centralized data policy that maps every application to specific SLA requirements.
- WAN Monitoring: the performance of each tunnel between the SD-WAN Edge devices mast be monitored continuously to appropriately route every application and to detect performance degradation.
- Map applications to tunnels: to map an application’s data traffic to a tunnel with appropriate performances, is used Deep Packet Inspection (DPI) that is able to classify network flow data in real time. The best path/tunnel is selected based on the measurements and on the constraints specified in the policy in the SD-WAN Controller.
A practical example may be the one in which there are available two types of WAN links, an MPLS and a fixed broadband network. A critical traffic like a videoconference, which require high performance, is routed over the MPLS network as it is reliable and has a very low jitter and delay. Instead the non-critical traffic such as downloading or web browsing is routed over the fixed broadband network, which has lower performance than the MPLS network but it cost less.
According to 14, here are 3 main SD-WAN architectures that bring different benefits to enterprises. Which architecture is the best depends on the number of sites that the company has and the applications that are accessed through the WAN.
In the On-Prem-Only architecture, each site of the enterprise has an SD-WAN Edge, essentially a plug&play router. This type of architecture is used to efficiently connect the various enterprise sites with a full-mash or hub-and-spoke connectivity. In this case the company isn’t using cloud application, but all the applications are hosted in-house. A common configuration involves an MPLS network for real time traffic and a public internet for non-critic traffic 14.
Abbildung in dieser Leseprobe nicht enthalten
Figure 8 - SD-WAN Hub and Spoke configuration 15
Abbildung in dieser Leseprobe nicht enthalten
Figure 9 - SD-WAN Full-Mash configuration 15
In Cloud-Enabled SD-WAN architecture is offered an SD-WAN Edge, placed in the enterprise site, which is also able to connect to a cloud gateway (virtual). With this solution is possible to get the benefits of an on-prem-only architecture, plus the increased reliability and performance in accessing cloud applications. A common configuration in this architecture is to have in-house real-time applications that exploit a small MPLS network and cloud applications and other traffic that use public internet 14.
Cloud-Enabled plus Backbone can be considered as a next level of Cloud-Enabled architecture. This SD-WAN architecture offer an SD-WAN Edge connecting the enterprise site to the SD-WAN provider’s nearest network, where the traffic goes through the SD-WAN provider’s private backbone network (optic fiber). Using SD-WAN provider’s private backbone guarantee performance in terms of jitter, packet loss and latency. The backbone is also directly connected to the major cloud application providers, as in the previous case the access to cloud applications is optimized. A possible implementation is to use together fixed broadband with private backbone, using the private backbone for real-time application, removing the need for an expensive MPLS network. Currently there are not many vendors that offer this type or architecture, it can be adopted by Internet Service Providers, since they already have a backbone infrastructure 14.
A company that intends to adopt the SD-WAN technology must make a choice, whether to adopt a Do-It-Yourself or Managed Service deployment model. In the Do-It-Yourself model the end user install software, manage physical and virtual appliances and configure network for the deployment and maintenance of an SD-WAN solution. In the other case is the vendor/provider that is responsible for the installation and for the support of the deployment, in this case the SD-WAN is provided to the enterprises as a service. Decide the model to adopt depends on whether the enterprise has an IT staff in-house, able to manage the deployment, installation and configuration of SD-WAN for each remote sites. Do-It-Yourself deployment provide visibility and control over the entire environment, instead Managed deployment allows an easier and faster adoption of the SD-WAN 16.
In this chapter is given a brief explanation about the main open source technologies used to implement the EveryWAN solution. In particular, an overview is given about the functionality and terminology of gRPC, VxLAN, VRF-lite, STUN protocol and Etherws.
gRPC is an open source remote procedure calls (RPC) which allows a client application to call methods of a server application on a different machine as it was a local object. Thus allowing to create a distributed application and services, giving the possibility to define the client and the server with different programming languages. gRPC use HTTP/2 for transport and use the protocol buffer as both its Interface Definition Language and as its underlying message interchange format. In gRPC the service is defined by specifying methods that can be called remotely with their parameters and return types. On the server side, the server implements the methods and runs a gRPC server to handle the client calls. The server gets the incoming request, execute the service methods and send service response. On the client side, the client has a Stub that provides the same methods as the server, these methods can be called on the local object passing the parameters for the call in the appropriate protocol buffer. gRPC has SSL/TLS integration that provide authentication and guarantees integrity and confidentiality of data exchanged between Client and Server 17.
Abbildung in dieser Leseprobe nicht enthalten
Figure 10 - gRPC Server and Client 17
Virtual Extensible LAN (VxLAN) is an encapsulation protocol for running an Overlay network on existing L3 infrastructure (underlay), it attempts to address the scalability problems associated with large cloud computing. Respect to other tunnelling technique it’s a 1 to N network, not just point to point. The VxLAN encapsulation technique encapsulates L2 Ethernet frames within L4 UDP datagram. In the figure below is shown the encapsulation packet format, VxLAN adds a fixed overhead of 50 bytes 18.
Abbildung in dieser Leseprobe nicht enthalten
Figure 11 - VxLAN Encapsulation Packet Format 19
The two key concepts in the operation of the VxLAN are VNI and VTEP. VxLAN Network Identifier (VNI) is the identifier of the VxLAN segment. Similar to the VLAN tag, traffic can be isolated using different VNI. The VNI is 24-bits long, this allow to have up to 16M VxLAN segments in the same administrative domain 18.
The VxLAN Tunnel Endpoint (VTEP) is the function that handle the encapsulation and decapsulation of VxLAN packets. A VTEP has two interfaces, one interface towards the local LAN segments and the other to IP transport network (underlay network). The VTEP can be a software or hardware entity, which has an IP address. Another important task of VTEPs is learning that is similar to the learning bridge, any VTEP know all the remote VTEPS and learn the VTEP MAC to remote address mapping to be able to send the VxLAN packets on the right tunnel 18.
Abbildung in dieser Leseprobe nicht enthalten
Figure 12 - VxLAN unicast forwarding 19
Virtual Routing and Forwarding (VRF) is a technology of a MPLS network that allow to distinguish different traffic flows because they are connected to different routing table. A subsystem is VRF-Lite that does not use MPLS but provides the functionality of a virtual router allowing the creation of multiple logical router tables in the same physical router. This allow a network path to be segmented without using multiple devices, in this way we have traffic isolation at L3 for routing. Fundamentally, a VRF use a single routing table and require a forwarding table that designates the next hop for each data packet. The network devices are associated with the VRF by enslaving the devices to the VRF. The network addresses assigned to the devices are local to the VRF, and the visible routes are those of the routing table associated to the VRF 20.
Abbildung in dieser Leseprobe nicht enthalten
Figure 13 - VRF 21
Session Traversal Utilities for NAT (STUN) 22 is a standardized tool for hosts to discover the presence and the type of NAT. STUN discover also the public IP address and the port with which the UDP application is exposed when it exit NAT.
Abbildung in dieser Leseprobe nicht enthalten
Figure 14 - STUN test 23
STUN is a client-server protocol, where typically the client operates inside a private network, while the STUN server is located on the public internet. The client sends a binding request to the server. The server responds with a success response containing the IP address and the UPD port of the client, seen by the server prospective. A series of request and response are exchanged based on what the client receive as a response. The sever must have at least two public IP address to test the NAT by sending response with different IP address. Looking at the responses the client is able to determine which type of NAT there is between it and the server. The types of NAT detectable:
- Open: no presence of NAT or firewall
- Full-cone: The port is permanently open; any external host can send data to the LAN. Maps a public IP address and port to an internal IP and port. The only requirement is that the connections come in a specific port. It’s also known as port-forwarding
- Restricted-cone: Works in the same way as a Full-cone but applies additional restrictions based on an IP address. The connection must start form the internal client, and only the external IP address contacted can reply by setting the destination port the outbound NATed source port.
- Restricted-port: Acts as restricted-cone but apply restriction also to the source port. Packet form external client in addition to the requirements of restricted-cone must set the source port equal to the destination port set by internal client.
- Symmetric: Apply the same restrictions as restricted-port but the NAT does not do port reservation. The source port set by internal client is randomly changed by NAT. In this way there is a unique mapping for each connection. In addition to the requirements of restricted-port the external client must set the destination port equal to the source port randomly chosen by NAT.
- UDP Firewall: Firewall that allow UDP out. The external client must respond to the client who sent the request, like symmetric NAT but without translation.
- Blocked: Firewall that block UDP
Etherws is a python library for the implementation of software switch with the Ethernet over WebSocket tunnel. The main components of Etherws are Ether SW , Ether CTL and TAP. Ether SW is a simple virtual Ethernet switch that is controlled by Ether CTL. To the Ether SW can be added the TAP device and port to existing network interface. The TAP device is the virtual network interface that has an IP address, and it’s responsible for passing the Ethernet packets to the WebSocket connection simulating a Ethernet device 24.
[...]
