2 ICN strives to automate the process of installing the local cluster
3 controller to the greatest degree possible – "zero touch
4 installation". Most of the work is done simply by booting up the jump
5 server (Local Controller). Once booted, the controller is fully
6 provisioned and begins to inspect and provision the bare metal
7 servers, until the cluster is entirely configured. This document shows
8 step-by-step how to configure the network and deployment architecture
14 # Deployment Architecture
15 The Local Controller is provisioned with the Metal3 Baremetal Operator
16 and Ironic, which enable provisioning of bare metal servers. The
17 controller has three network connections to the bare metal servers:
18 network A connects bare metal servers, network B is a private network
19 used for provisioning the bare metal servers and network C is the IPMI
20 network, used for control during provisioning. In addition, the
21 bare metal servers connect to the network D, the SRIOV network.
23 ![Figure 1](figure-1.png)*Figure 1: Deployment Architecture*
25 - Net A -- Bare metal network, lab networking for ssh. It is used as
26 the control plane for k8s, used by OVN and Flannel for the overlay
28 - Net B (internal network) -- Provisioning network used by Ironic to
30 - Net C (internal network) -- IPMI LAN to do IPMI protocol for the OS
31 provisioning. The NICs support IPMI. Use IPMI tool to set the static
33 - Net D (internal network) -- Data plane network for the Akraino
34 application. Using the SR-IOV networking and fiber cables. Intel
35 25GB and 40GB FLV NICs.
37 In some deployment models, you can combine Net C and Net A to be the
38 same networks, but the developer should take care of IP address
39 management between Net A and IPMI address of the server.
41 # Pre-installation Requirements
42 There are two main components in ICN Infra Local Controller - Local
43 Controller and k8s compute cluster.
46 The Local Controller will reside in the jump server to run the Metal3
47 operator, Binary Provisioning Agent (BPA) operator and BPA REST API
50 ### k8s Compute Cluster
51 The k8s compute cluster will actually run the workloads and is
52 installed on bare metal servers.
54 ## Hardware Requirements
56 ### Minimum Hardware Requirement
57 All-in-one VM based deployment requires servers with at least 32 GB
60 ### Recommended Hardware Requirements
61 Recommended hardware requirements are servers with 64GB Memory, 32
62 CPUs and SRIOV network cards.
64 ## Software Prerequisites
65 The jump server is required to be pre-installed with Ubuntu 18.04.
67 ## Database Prerequisites
68 No prerequisites for ICN blueprint.
70 ## Other Installation Requirements
72 ### Jump Server Requirements
74 #### Jump Server Hardware Requirements
75 - Local Controller: at least three network interfaces.
76 - Bare metal servers: four network interfaces, including one IPMI interface.
77 - Four or more hubs, with cabling, to connect four networks.
79 Hostname | CPU Model | Memory | Storage | 1GbE: NIC#, VLAN, (Connected extreme 480 switch) | 10GbE: NIC# VLAN, Network (Connected with IZ1 switch)
80 ---------|-----------|--------|---------|--------------------------------------------------|------------------------------------------------------
81 jump0 | Intel 2xE5-2699 | 64GB | 3TB (Sata)<br/>180 (SSD) | eth0: VLAN 110 (DMZ)<br/>eno1: VLAN 111 (Admin) | eno2: VLAN 112 (Private) VLAN 114 (Management)<br/>IF3: VLAN 113 (Storage) VLAN 1115 (Public)
83 #### Jump Server Software Requirements
84 ICN supports Ubuntu 18.04. The ICN blueprint installs all required
85 software during `make install`.
87 ### Network Requirements
88 Please refer to figure 1 for all the network requirements of the ICN
91 Please make sure you have 3 distinguished networks - Net A, Net B and
92 Net C as mentioned in figure 1. Local Controller uses the Net B and
93 Net C to provision the bare metal servers to do the OS provisioning.
95 ### Bare Metal Server Requirements
97 ### k8s Compute Cluster
99 #### Compute Server Hardware Requirements
101 Hostname | CPU Model | Memory | Storage | 1GbE: NIC#, VLAN, (Connected extreme 480 switch) | 10GbE: NIC# VLAN, Network (Connected with IZ1 switch)
102 ---------|-----------|--------|---------|--------------------------------------------------|------------------------------------------------------
103 node1 | Intel 2xE5-2699 | 64GB | 3TB (Sata)<br/>180 (SSD) | eth0: VLAN 110 (DMZ)<br/>eno1: VLAN 111 (Admin) | eno2: VLAN 112 (Private) VLAN 114 (Management)<br/>IF3: VLAN 113 (Storage) VLAN 1115 (Public)
104 node2 | Intel 2xE5-2699 | 64GB | 3TB (Sata)<br/>180 (SSD) | eth0: VLAN 110 (DMZ)<br/>eno1: VLAN 111 (Admin) | eno2: VLAN 112 (Private) VLAN 114 (Management)<br/>IF3: VLAN 113 (Storage) VLAN 1115 (Public)
105 node3 | Intel 2xE5-2699 | 64GB | 3TB (Sata)<br/>180 (SSD) | eth0: VLAN 110 (DMZ)<br/>eno1: VLAN 111 (Admin) | eno2: VLAN 112 (Private) VLAN 114 (Management)<br/>IF3: VLAN 113 (Storage) VLAN 1115 (Public)
107 #### Compute Server Software Requirements
108 The Local Controller will install all the software in compute servers
109 from the OS to the software required to bring up the k8s cluster.
111 ### Execution Requirements (Bare Metal Only)
112 The ICN blueprint checks all the precondition and execution
113 requirements for bare metal.
115 # Installation High-Level Overview
116 Installation is two-step process and everything starts with one
117 command `make install`.
118 - Installation of the Local Controller.
119 - Installation of compute cluster.
121 ## Bare Metal Deployment Guide
123 ### Install Bare Metal Jump Server
125 #### Creating a Node Inventory File
127 ##### Preconfiguration for the Local Controller in Jump Server
128 The user is required to provide the IPMI information of the servers
129 they connect to the Local Controller by editing node JSON sample file
130 in the directory icn/deploy/metal3/scripts/nodes.json.sample as
131 below. This example only shows 2 servers. If you want to increase
132 servers, just add another array.
139 "ipmi_driver_info": {
142 "address": "10.10.10.11"
145 "image_name": "bionic-server-cloudimg-amd64.img",
146 "username": "ubuntu",
147 "password": "mypasswd"
152 "ipmi_driver_info": {
155 "address": "10.10.10.12"
158 "image_name": "bionic-server-cloudimg-amd64.img",
159 "username": "ubuntu",
160 "password": "mypasswd"
166 ##### Local Controller Metal3 Configuration Reference
167 - *node*: The array of nodes required to add to Local Controller.
168 - *name*: This will be the hostname for the machine, once it is
169 provisioned by Metal3.
170 - *ipmi_driver_info*: IPMI driver info is a json field. It currently
171 holds the IPMI information required for Ironic to send the IPMI tool
173 - *username*: BMC username required to be provided for Ironic.
174 - *password*: BMC password required to be provided for Ironic.
175 - *address*: BMC server IPMI LAN IP address.
176 - *os*: Bare metal machine OS information is a json field. It
177 currently holds the image name to be provisioned, username and
178 password for the login.
179 - *image_name*: Images name should be in qcow2 format.
180 - *username*: Login username for the OS provisioned.
181 - *password*: Login password for the OS provisioned.
183 #### Creating the Settings Files
185 ##### Local Controller Network Configuration Reference
186 The user will find the network configuration file named as
187 "user_config.sh" in the ICN parent directory.
193 #Local Controller - Bootstrap cluster DHCP connection
194 #BS_DHCP_INTERFACE defines the interfaces, to which ICN DHCP deployment will bind
195 #e.g. export BS_DHCP_INTERFACE="ens513f0"
196 export BS_DHCP_INTERFACE=
198 #BS_DHCP_INTERFACE_IP defines the IPAM for the ICN DHCP to be managed.
199 #e.g. export BS_DHCP_INTERFACE_IP="172.31.1.1/24"
200 export BS_DHCP_INTERFACE_IP=
202 #Edge Location Provider Network configuration
203 #Net A - Provider Network
204 #If provider having specific Gateway and DNS server details in the edge location
205 #export PROVIDER_NETWORK_GATEWAY="10.10.110.1"
206 export PROVIDER_NETWORK_GATEWAY=
207 #export PROVIDER_NETWORK_DNS="8.8.8.8"
208 export PROVIDER_NETWORK_DNS=
210 #Ironic Metal3 settings for provisioning network
211 #Interface to which Ironic provision network to be connected
212 #Net B - Provisioning Network
213 #e.g. export IRONIC_INTERFACE="eno1"
214 export IRONIC_INTERFACE=
216 #Ironic Metal3 setting for IPMI LAN Network
217 #Interface to which Ironic IPMI LAN should bind
218 #Net C - IPMI LAN Network
219 #e.g. export IRONIC_IPMI_INTERFACE="eno2"
220 export IRONIC_IPMI_INTERFACE=
222 #Interface IP for the IPMI LAN, ICN verfiy the LAN Connection is active or not
223 #e.g. export IRONIC_IPMI_INTERFACE_IP="10.10.10.10"
224 #Net C - IPMI LAN Network
225 export IRONIC_IPMI_INTERFACE_IP=
229 After configuring the node inventory file and network configuration
230 files, please run `make install` from the ICN parent directory as
234 root@pod11-jump:# git clone "https://gerrit.akraino.org/r/icn"
235 Cloning into 'icn'...
236 remote: Counting objects: 69, done
237 remote: Finding sources: 100% (69/69)
238 remote: Total 4248 (delta 13), reused 4221 (delta 13)
239 Receiving objects: 100% (4248/4248), 7.74 MiB | 21.84 MiB/s, done.
240 Resolving deltas: 100% (1078/1078), done.
241 root@pod11-jump:# cd icn/
242 root@pod11-jump:# vim Makefile
243 root@pod11-jump:# make install
246 The following steps occurs once the `make install` command is given.
247 1. All the software required to run the bootstrap cluster is
248 downloaded and installed.
249 2. k8s cluster to maintain the bootstrap cluster and all the servers
250 in the edge location is installed.
251 3. Metal3 specific network configuration such as local DHCP server
252 networking for each edge location, Ironic networking for both
253 provisioning network and IPMI LAN network are identified and
255 4. Metal3 is launched with IPMI configuration as configured in
256 "user_config.sh" and provisions the bare metal servers using IPMI
257 LAN network. For more information refer to the [Debugging
258 Failures](#debugging-failures) section.
259 5. Metal3 launch verification runs with a timeout of 60 mins by
260 checking the status of all the servers being provisioned or not.
261 1. All servers are provisioned in parallel. For example, if your
262 deployment is having 10 servers in the edge location, all the 10
263 servers are provisioned at the same time.
264 2. Metal3 launch verification takes care of checking all the
265 servers are provisioned, the network interfaces are up and
266 provisioned with a provider network gateway and DNS server.
267 3. Metal3 launch verification checks the status of all servers
268 given in user_config.sh to make sure all the servers are
269 provisioned. For example, if 8 servers are provisioned and 2
270 servers are not provisioned, launch verification makes sure all
271 servers are provisioned before launch k8s clusters on those
273 6. BPA bare metal components are invoked with the MAC address of the
274 servers provisioned by Metal3, BPA bare metal components decide the
275 cluster size and also the number of clusters required in the edge
277 7. BPA bare metal runs the containerized Kuberenetes Reference
278 Deployment (KUD) as a job for each cluster. KUD installs the k8s
279 cluster on the slice of servers and install ONAP4K8S and all other
280 default plugins such as Multus, OVN, OVN4NFV, NFD, Virtlet and
282 8. BPA REST API agent installed in the bootstrap cluster or jump
283 server, and this install rest-api, rook/ceph, MinIO as the cloud
284 storage. This provides a way for user to upload their own software,
285 container images or OS image to jump server.
287 ## Virtual Deployment Guide
289 ### Standard Deployment Overview
290 ![Figure 2](figure-2.png)*Figure 2: Virtual Deployment Architecture*
292 Virtual deployment is used for the development environment using
293 Metal3 virtual deployment to create VM with PXE boot. VM Ansible
294 scripts the node inventory file in /opt/ironic. No setting is required
295 from the user to deploy the virtual deployment.
297 ### Snapshot Deployment Overview
298 No snapshot is implemented in ICN R2.
300 ### Special Requirements for Virtual Deployment
302 #### Install Jump Server
303 Jump server is required to be installed with Ubuntu 18.04. This will
304 install all the VMs and install the k8s clusters. Same as bare metal
305 deployment, use `make vm_install` to install virtual deployment.
307 #### Verifying the Setup - VMs
308 `make verify_all` installs two VMs with name master-0 and worker-0
309 with 8GB RAM and 8 vCPUs and installs k8s cluster on the VMs using the
310 ICN BPA operator and install the ICN BPA REST API verifier. BPA
311 operator installs the multi-cluster KUD to bring up k8s with all
314 # Verifying the Setup
315 ICN blueprint checks all the setup in both bare metal and VM
316 deployment. Verify script will check that Metal3 provisioned the OS in
317 each bare metal servers by checking with a timeout period of 60 sec
318 and interval of 30. BPA operator verifier will check whether the KUD
319 installation is complete by doing plain curl command to the k8s
320 cluster installed in bare metal and VM setup.
322 **Bare Metal Verifier**: Run the `make bm_verifer`, it will verify the
323 bare-metal deployment.
325 **Verifier**: Run the `make vm_verifier`, it will verify the virtual
328 # Developer Guide and Troubleshooting
329 For development uses the virtual deployment, it take up to 10 mins to
330 bring up the virtual BMC VMs with PXE boot. Virtual deployment works
331 well for the BPA operator development for Metal3 installation scripts.
333 ## Utilization of Images
334 No images provided in this ICN release.
336 ## Post-deployment Configuration
337 No post-deployment configuration required in this ICN release.
339 ## Debugging Failures
340 * For first time installation enable KVM console in the trial or lab
341 servers using Raritan console or use Intel web BMC console.
343 ![Figure 3](figure-3.png)
344 * Deprovision state will result in Ironic agent sleeping before next
345 heartbeat - it is not an error. It results in bare metal server
346 without OS and installed with ramdisk.
347 * Deprovision in Metal3 is not straight forward - Metal3 follows
348 various stages from provisioned, deprovisioning and ready. ICN
349 blueprint take care navigating the deprovisioning states and
350 removing the BareMetalHost (BMH) custom resouce in case of cleaning.
351 * Manual BMH cleaning of BMH or force cleaning of BMH resource result
352 in hang state - use `make bmh_clean` to remove the BMH state.
353 * Logs of Ironic, openstack baremetal command to see the state of the
355 * Logs of baremetal operator gives failure related to images or images
357 * It is not possible to change the state from provision to deprovision
358 or deprovision to provision without completing that state. All the
359 issues are handled in ICN scripts.
360 * k8s cluster failure can be debugged by KUD Pod logs.
363 Required Linux Foundation ID to launch bug in ICN:
364 https://jira.akraino.org/projects/ICN/issues
368 ## Bare Metal deployment
369 The command `make clean_all` uninstalls all the components installed by
371 * It de-provision all the servers provisioned and removes them from
373 * Baremetal operator is deleted followed by Ironic database and
375 * Network configuration such internal DHCP server, provisioning
376 interfaces and IPMI LAN interfaces are deleted.
377 * docker images built during the `make install` are deleted, such as
378 all Ironic, baremetal operator, BPA operator and KUD images.
379 * KUD will reset the bootstrap cluster - k8s cluster is torn down in
380 the jump server and all the associated docker images are removed.
381 * All software packages installed by `make install_all` are removed,
382 such as Ironic, openstack utility tool, docker packages and basic
383 prerequisite packages.
385 ## Virtual deployment
386 The command `make vm_clean_all` uninstalls all the components for the
391 ## Error Message Guide
392 The error message is explicit, all messages are captured in log
397 ## Blueprint Package Maintenance
398 No packages are maintained in ICN.
400 ## Software maintenance
403 ## Hardware maintenance
406 ## BluePrint Deployment Maintenance
409 # Frequently Asked Questions
410 **How to setup IPMI?**
412 First, make sure the IPMI tool is installed in your servers, if not
413 install them using `apt install ipmitool`. Then, check for the
414 ipmitool information of each servers using the command `ipmitool lan
415 print 1`. If the above command doesn't show the IPMI information, then
416 setup the IPMI static IP address using the following instructions:
417 - Mostl easy way to set up IPMI topology in your lab setup is by
420 https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool
421 - IPMI information can be considered during the BIOS setting as well.
423 **BMC web console URL is not working?**
425 It is hard to find issues or reason. Check the ipmitool bmc info to
426 find the issues, if the URL is not available.
428 **No change in BMH state - provisioning state is for more than 40min?**
430 Generally, Metal3 provision for bare metal takes 20 - 30 mins. Look at
431 the Ironic logs and baremetal operator to look at the state of
432 servers. Openstack baremetal node shows all state of the server right
435 **Why provide network is required?**
437 Generally, provider network DHCP servers in lab provide the router and
438 DNS server details. In some lab setup DHCP server don't provide this
445 * Copyright 2019 Intel Corporation, Inc
447 * Licensed under the Apache License, Version 2.0 (the "License");
448 * you may not use this file except in compliance with the License.
449 * You may obtain a copy of the License at
451 * http://www.apache.org/licenses/LICENSE-2.0
453 * Unless required by applicable law or agreed to in writing, software
454 * distributed under the License is distributed on an "AS IS" BASIS,
455 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
456 * See the License for the specific language governing permissions and
457 * limitations under the License.
463 # Definitions, acronyms and abbreviations