Framework and Requirements for GMPLS-based Control of Flexible Ethernet Network

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 .

Ethernet starts from 10M, then evolves to 100M, 1000M, 10G, etc. As the line rate goes higher, it is more and more difficult for manufacturing technology to support this 10-times-based evolution, and also it takes more time to develop new standard. For example, IEEE started standardization work on 100G Ethernet in December, 2007, and actually the initial discussion on this started about two year earlier, finished the first part of 100G standard in 802.3ba in June, 2010. As of today, it is almost 10 years since the beginning, the work on 100G for more models is still ongoing. The work on 400G Ethernet started from March, 2013, and it is expected to be finished by the end of 2017 which can covver a distance of 10km. It will take some more years before the 400G module is commonly available at a reasonable price in the market. There is no consensus yet what is going to be the next beyond, 800G or 1T. If operators want to use the next generation higher speed Ethernet interface, e.g. for Inter-DC connections where high speed interfaces are desired due to large traffic volume and rapid increase of traffic, they will need to wait for some years for the new interface modules . A possible mitigation is to use some bonding technology, such as 802.1AX link aggregation, but with the hash issue -- traffic may not be evenly distributed over multiple links. FlexE is a good solution for this problem, traffic is distributed over multiple physical links by 64/66B blocks in a round-robin manner. This is one of the key reasons that FlexE is invented. Besides bonding, FlexE also provides sub rate and channelization capability. In FlexE, a 100G physical link can be divided into 20 slots, and each slot is 5G. A subset of these 20 slots can be grouped together and provide a virtual link with bandwidth of 5G*N. FlexE also allows slots over multiple physical links be grouped together and provide bandwidth which is not integral multiple of a physical link, such as 150G bandwidth over two 100G physical links. This is called as channelization.

FlexE Group FlexE Group is a set of physical links which can be bonded together between two network devices. A FlexE group can have minimally one, and up to 254 physical links. .The links may be between two adjacent network devices, or be connected over transport network. If links are connected by transport network, for the purpose of skew control, FlexE traffic over different physical links should traverse through same route in the transport network. In some cases, links between two adjacent network devices cannot be added into a same FlexE group due to some constraints, such as the links spread across multiple line cards where skew may be a problem. FlexE group may be created automatically via signaling, which is not yet covered by the FlexE specification. The maintenance of FlexE Group, add / remove PHYs, may also be automated which requires further study. FlexE Group ID There can be multiple FlexE groups in a network device. In order to distinguish these groups, a FlexE Group ID, 20bits in length in , is assigned for each FlexE group. FlexE Group ID is carried in the overhead over each physical link in the FlexE Group, and the two end points of a FlexE connection should be using a same FlexE Group Number in each direction. The assignment and auto negotiation of FlexE Group ID may be performed via signaling, which needs further study. FlexE Group Member PHY Each Physical Link in the FlexE Group is a member PHY. The current FlexE specification only supports bonding of physical links of a same rate. In the future, FlexE may support links with rates other than 100G, such as 400G which is being standardized by IEEE. PHY Number Each PHY in a FlexE group is assigned with a PHY number, and a PHY number will be carried in the FlexE overhead over each PHY. FlexE specification requires that each end of the FlexE Group should assigned a same PHY number to a PHY. The PHY Number will also be used for FlexE mux and demux, FlexE traffic will be distributed to FlexE slots which may span over multiple PHYs in a round robin manner, which means traffic is distributed to PHYs with PHY number in ascending sequence. FlexE Calendar FlexE calendar is the representation of FlexE slot resource of a FlexE group. It is the combination of sub calendar of each physical link in a FlexE group. FlexE Sub Calendar FlexE sub calendar is the representation of the 20 slots of a physical link. Each FlexE sub calendar is carried in the FlexE overhead over the corresponding physical link when communicating with the other end. FlexE Client A FlexE client can be Ethernet packet flow, with the rates of 10G, 40G, 100G, and in the future 25G, 50G, 200G and 400G, according to section 7.2.2 of . It can also be some type of traffic other than Ethernet, CBR or VBR, such as fiber channel traffic. If FlexE is intended for network virtualization and network slicing in the future, or to support traffic with rates not listed above, it is better to support any N*5G rate. FlexE Client Channel The channel supporting a FlexE client between FlexE shim layer of two nodes is a FlexE client channel. This channel can be a group FlexE slots between two adjacent nodes; an OTN OCh, a WDM wavelength or a fiber in the FlexE unaware case; an ODUk or ODUflex; or the combination of above, etc. The forwarding method used in the channel maybe L0, L1, L2 or above. FlexE Client ID FlexE client ID is a 16bit integer, carried in the FlexE calendar in the overhead, representing the owner of a FlexE slot; each slot in use is related to a FlexE client ID. Receiver end of a FlexE group need to parse the FlexE calendar to find out which slots belongs to a FlexE client based on the mapping of FlexE client ID and slots, and then perform traffic mux. FlexE Shim The FlexE shim is the layer maps or demaps FlexE client to or from a set FlexE slots over a FlexE group. According to , the FlexE shim layer is within the 802.3 PCS layer, right below the 64/66B encode / decode module.

According to section 82, 83 of and , Ethernet and FlexE layering is shown in the figure below.

There are three typical FlexE transport use cases as depicted in , and correspondingly there are several network layering and mapping options.

In this mode, the FlexE traffic will be treated as bit stream on physical link basis, rather than at FlexE group or FlexE client basis. FlexE encapsulation and signaling is transparent to the transport network, The transport network will not try to interpret the bit stream. If the transport network covers a very long distance, skew might be a problem for FlexE, a large skew value should be considered. If there are multiple physical links in a FlexE group, it may be necessary to consider carrying the traffic of the various link along the same transport network path so as to mitigate skew issue.

+-------------------------+ | PMA | | | +-------------------------+ | | | PMD | | | +-------------------------+ | | | | MDI | | +--------------------+ +-------------------------+ | MEDIUM | | OTN G.709 | +--------------------+ +-------------------------+ ]]> According to section 17.7.5.1 and Annex E of , the data stream at the interface between PCS and PMA should be carried over OTN, as shown in the above figure. There is an efficiency problem in this mode. Because the transport network will not be able to know which FlexE slots are in use and which are not, then the transport network will have to carry all the traffic in a FlexE group. For example, if a FlexE group consists of two 100G links, and the configured FlexE bandwidth is 150G: 100G over one link and 50G over the other. But the transport network has to carry the total 200G traffic, unable to remove the unused 50G slots.

The transport network can understand the FlexE protocol, and will remove the unused slots before carrying FlexE traffic over the transport network. At the egress point of transport network, FlexE traffic will be mapped to the same number slots, while leaving some slots in the FlexE Group unused if necessary. This will save some bandwidth of the transport network. In this mode, the transport network interwork with FlexE below the FlexE layer, assuming FlexE is L1.5, most of the FlexE overhead will be transported over the transport network. The session management channel in the FlexE overhead will be terminated and replaced with idle control block, as specified in section 8.3 of . The data stream to be transported is above L1 and below FlexE shim (L1.5), which is called as L1.25 data stream.

+---------------------+ | | De/scramble | | | | | +---------------------+ | | | | | AM Add / Remove | | | | | | block Distribution | | | | | +---------------------+ | | | +-------------------------+ | | | PMA | | | +-------------------------+ | | | PMD | | | +-------------------------+ | | | | MDI | | +--------------------+ +-------------------------+ | MEDIUM | | OTN G.709 | +--------------------+ +-------------------------+ ]]> The FlexE traffic is interleaved before transporting, so there will be no skew issue; and OTN will use BGMP algorithm to map FlexE traffic into ODUk or ODUflex, as specified in section 17.11 of . [Note: The case when FlexE traffic is greater than the rate of a single link or a WMD wavelength is not yet considered by for the time being. This mode is usually used for a point to point path, rather than a P2MP or MP2MP case. The traffic stream is the valid traffic of a whole FlexE group.

This is also called FlexE termination transport. The FlexE traffic over multiple slot will be aggregated into a 64/66B stream, and the FlexE overhead will be removed before FlexE traffic is carried over the transport network. At the egress of the transport network, FlexE overhead will be added when the traffic is converted back into FlexE mode.

+--------------------------+ | | FlexE Shim | | | | | +---------------------+ | | | | | De/scramble | | | | | +---------------------+ | | | | | AM Add / Remove | | | | | | block Distribution | | | | | +---------------------+ | | | +-------------------------+ | | | PMA | | | +-------------------------+ | | | PMD | | | +-------------------------+ | | | | MDI | | +--------------------+ +--------------------------+ | MEDIUM | | OTN G.709 | +--------------------+ +--------------------------+ ]]> This mode does not have the skew issue because the FlexE overhead is removed and the concept of FlexE slots does not exist when the traffic is in the transport network, alignment between slot is not necessary. The traffic in a FlexE group can be divided into multiple flows in the transport network, each can be identified by FlexE client ID, or FlexE client ID plus FlexE group number. These different flows can be routed through different ODUk or ODUflex and consequently may traverse over different link path to different end station. As shown in , the FlexE overhead is removed from the FlexE traffic and the same number of 64/66B idle blocks are inserted at the IPG position between Ethernet frames when FlexE payload is Ethernet. It may be a different case if the FlexE payload is not Ethernet. Then the traffic is in the form of 64/66B block stream on a per FlexE client basis, and it is CBR stream. The mapping of this CBR stream over OTN uses IMP algorithm as specified in section 17.11 of .

This is another type of FlexE termination transport, FlexE overhead will be terminated and traffic will be converted into L2/L2.5/L3 packets streams on a per FlexE client basis, which is VBR stream, rather than 64/66B blocks in the above case. This will enable some new application scenario, such as transport network may be used to provide VPLS-like VPN service, or to support network virtualization and network slicing. A FlexE client can be modeled in a system as a virtual interface, a set of virtual interfaces can construct a L2 or L3 forwarding instance.

+--------------------------+ | RECONCILIATION | | | +------------------+ | | | | MII | | +-------------------------+ | | | PCS | | | | +---------------------+ | +--------------------------+ | | 64/66B En/Decode | | | 64/66B Encode / Decode | | +---------------------+ | +--------------------------+ | | FlexE Shim | | | Idle Add / Remove | | +---------------------+ | +--------------------------+ | | De/scramble | | | | | +---------------------+ | | | | | AM Add / Remove | | | | | | block Distribution | | | | | +---------------------+ | | | +-------------------------+ | | | PMA | | | +-------------------------+ | | | PMD | | | +-------------------------+ | | | | MDI | | +--------------------+ +--------------------------+ | MEDIUM | | OTN G.709 | +--------------------+ +--------------------------+ ]]>

+--------------------------+ | RECONCILIATION | | | +------------------+ | | | | MII | | +-------------------------+ +--------------------------+ | PCS | | GFP-F | | +---------------------+ | +--------------------------+ | | 64/66B En/Decode | | | | | +---------------------+ | | | | | FlexE Shim | | | | | +---------------------+ | | | | | De/scramble | | | | | +---------------------+ | | | | | AM Add / Remove | | | | | | block Distribution | | | | | +---------------------+ | | | +-------------------------+ | | | PMA | | | +-------------------------+ | | | PMD | | | +-------------------------+ | | | | MDI | | +--------------------+ +--------------------------+ | MEDIUM | | OTN G.709 | +--------------------+ +--------------------------+ ]]> provides two methods to map packet client signal into OPUk as specified in section 17.10 and 17.11 of , firs map packet flow into GFP-F encapsulation then into OPUk or OPUflex, or map packet flow into FlexE client signal in the form of 64/66B block, then into OPUflex using Idle Mapping Procedure (IMP) as specified in section 17.11 of .

The following parameters are applicable to FlexE Aware L1.25 Relay, FlexE Termination L1.5 Relay, and FlexE Termination L2/L2.5/L3 Relay. SENDER_TSPEC object: Class = 12 , C-Type = TBD. FLOWSPEC object: Class = 9, , C-Type = TBD. Traffic Parameters will be carried in the SENDER_TSPEC and FLOWSPEC object in Path Message to specify the traffic characteristics of a flow. section 4.2 of already provides a traffic parameter definition. Switching Type

Generalized Label Object is carried in Resv Message. . Section 3.1 of proivdes a label definition for FlexE which can support future links other than 100G and possible new granularities, also provides support to heterogeneous links in a FlexE group. If a simplified version is desired, the one in section 3.2 of can be considered.

This is actually a traditional mode, listed here for the purpose of completeness only. [Note: to consider carrying traffic of multiple links in a FlexE group along a same transport network path? TBD] LSP Encoding Type, Switching Type, G-PID will be carried in the Generalized Label Request Object. . LSP Encoding Type

Switching Type

Generalized PID (G-PID)

A new G-PID type is required, although this is not FlexE related. Label Format will be carried in the Generalized Label Object.

A new traffic parameter definition for FlexE unaware mode may not be necessary since the transport network is not supposed to know FlexE. A possible option is to reuse the traffic parameter definition in section 3.2.2 of .

LSP Encoding Type

Generalized PID (G-PID)

OTN will use BGMP algorithm to map the FlexE signal over transport network.

LSP Encoding Type

Generalized PID (G-PID)

LSP Encoding Type for FlexE client (packet).

Generalized PID (G-PID)

TBD.

SENDER_TSPEC object: Class = 12 , C-Type = TBD. FLOWSPEC object: Class = 9 , C-Type = TBD. LSP Encoding Type for FlexE aware group

LSP Encoding Type for FlexE client (L1.5).

LSP Encoding Type for FlexE client (packet).

Switching Type for FlexE.

Generalized PID (G-PID) for 100GE PCS

Generalized PID (G-PID) for FlexE aware transport.

Generalized PID (G-PID) for FlexE Client (L1.5).

Generalized PID (G-PID) for FlexE client (packet).

TBD.