CNM Bureau Farm

CNM Bureau Farm (formerly known as CNM EndUser Farm; hereinafter, the Farm) is the CNM farm that hosts CNM Social, CNM Talk, and CNM Venture.

Technically, the Farm is a collection of software. End-users work with #End-user applications that are installed in #Virtual environments (VE) that are, consequently, installed on #Node OS, which is node-root-level operating system (OS), that is installed on the #Infrastructure, which consists of infrastructure-level #Bridges and hardware, which includes #Backup server and #Bare-metal servers.

To eliminate a single point of failure, the Farm includes three Nodes, which are coordinated by #High availability (HA) tools. Any Farm's node (hereinafter, the Node) is one of those hardware servers with all of software installed on the top of it.

End-user applications

The Farm's end-user applications (hereinafter, the Apps) are those with which end-users of the Farm interact. The Apps can be deployed utilizing two models:

1. Using containers; they already contain operating systems tailored specifically to the needs of the App.
2. In virtual machines (VM), without containers. In that model, the App is installed on the operating system of its VM.

HumHub

CNM Social, which is the end-user instance of CNM HumHub.

Odoo

CNM Venture, which is the end-user instance of CNM Odoo.

Jitsi

CNM Talk, which is the end-user instance of CNM Jitsi.

Virtual environments (VE)

CNMCyber Team uses virtualization to divide excessive hardware resources of #Bare-metal servers in smaller containers and virtual machines (VMs), which are created in virtual environments (VEs).

As its software for VEs, the Farm utilizes CNM ProxmoxVE. Every instance of CNM ProxmoxVE is installed on #Node OS, which require "physical" #Bare-metal servers. The Farm's CNM ProxmoxVE also utilizes #Storage platform as its storage.

Choice of VE COTS

CNMCyber Team has tried OpenStack and VirtualBox as its virtualization tools. The trials suggested that OpenStack required more hardware resources and VirtualBox didn't allow for required sophistication in comparison with the chosen COTS, ProxmoxVE.

Node OS

The interaction between CNM ProxmoxVE instances and the #Infrastructure is carried out by Debian operating system that comes in the same "box" as ProxmoxVE and is specially configured for that interaction.

Storage platform

To make objects, blocks, and files immediately available for the Apps' operations, the Farm uses CNM Ceph. This common distributed cluster foundation orchestrates storage spaces of the individual Nodes.

At the Farm, CNM Ceph is deployed at every Node. Every of #Bare-metal servers features double hard disks. Physically, CNM ProxmoxVE is installed on one disk of each Node; CNM Ceph uses three "second" disks. Since every disk is 512 GB, the Farm's CNM Ceph capacity is about 3 * 512 GB = 1.536 GB

High availability (HA)

High availability (HA) of the Farm assumes that no failure of any App or its database management system (DBMS) can cause the failure of the Farm as a whole. HA tools are based on:

• A principle of redundancy; that is why the Farm is built on three Nodes, not one. Every App is installed at least twice on different Nodes as described in the #HA at the App level section. Every object, block, or file is stored at least twice on different Nodes as described in the #HA at the DBMS level section.
• Management of redundant resources as described in the #HA management section. In plain English, the Farm needs to put into operations those and only those resources that are in operational shapes.

Generally speaking, HA comes with significant costs. So does HA of the Farm. For example, running three Nodes is more expensive than running one. The cost cannot exceed the benefit, so high availability cannot be absolute availability.

HA at the App level

When one App fails, its work continues its sister App installed on the second Node. If another App fails, its work continues its sister App installed on the third Node. If the third App fails, the Farm cannot provide its users with the App services any longer.

To ensure that, the Farm utilizes tools that comes with ProxmoxVE. Every virtual machine (VM) or container is kept on at least two Nodes. When the operational resource, VM or container, fails, CNM ProxmoxVE activates another resource and creates the third resource as a reserve. As a result, VM or container "migrates" from one Node to another Node.

HA at the DBMS level

When one DBMS fails, its work continues its sister DBMS installed on the second Node. When another DBMS fails, its work continues its sister DBMS installed on the third Node. If the third DBMS fails, the Farm can no longer provide the App with the data it requires to properly work.

To ensure that, the Farm utilizes the #Storage platform. Every object, block, or file is kept on at least two Nodes. When any stored resource fails, #Storage platform activates another resource and creates the third resource as a reserve. As a result, any stored resource "migrates" from one Node to another Node.

HA management

To manage redundant resources, the Farm:

• Monitors its resources to identify whether they are operational or failed as described in the #Monitoring section.
• Fences those resources that are identified as failed. As a result, non-operational resources are withdrawn from the list of available.
• Restores those resources that are fenced. The #Backup and recovery supports that feature, while constantly creating snapshots and reserve copies of the Farm and its parts in order to make them available for restoring when needed.

DNS architecture

Communication channels

The Farm's communication channels are built on the Nodes and #Bridges. Currently, the Farm uses three communication channels, each of which serves one of the network as follows:

1. Public network that uses external, public IPv4 addresses to integrate the Farm into the Internet. The public network is described in the #DNS entry point section of this wikipage.
2. Node network that uses internal, private IPv6 addresses to integrate the Nodes into one network cluster. This network cluster is described in the #Virtual environments section of this wikipage.
3. Storage network that uses internal, private IPv6 addresses to integrate storage spaces of the Nodes into one storage cluster. This storage cluster is described in the #Storage platform section of this wikipage.

The Farm's usage of IP addresses is best described in the #IP addresses section.

DNS intermediaries

For the purposes of this wikipage, "DNS intermediaries" refer the Farm's load balancer and reverse proxy (hereinafter, the Intermediaries). They can be compared to a front desk in some office, where somebody (a) takes external client's requests, (b) dispatches those requests to internal resources, (c) gets internal responses, and (d) returns them back to the client in the outside world.

The Intermediaries (a) receive requests that #Web servers, which the servers have got from the world outside of the Farm and processed, (b) dispatch those requests to an appropriate resource of a particular Node that the intermediaries have selected, (c) get internal responses, and (d) return those responses to the #Web servers to send them back.

The Intermediaries are responsible for dispatching external requests to those and only to those internal resources that the Farm's #Monitoring has identified as operational. To be accessible to more clients, the Intermediaries utilize public IPv4 addresses.

DNS zone

To locate the Farm's public resources in the Internet, the following resource records are created in the Farm's DNS zone:

Resource record	Type	Data	Comment (not a part of the records)
pm1.bskol.com	AAAA record	2a01:4f8:10a:439b::2	Node 1
pm2.bskol.com	AAAA record	2a01:4f8:10a:1791::2	Node 2
pm3.bskol.com	AAAA record	2a01:4f8:10b:cdb::2	Node 3
pbs.bskol.com	AAAA record	2a01:4f8:fff0:53::6	Backup server
pf.bskol.com	AAAA record	2a01:4f8:fff0:53::6	pfsense
pf.bskol.com	A record	88.99.71.85
npm1.bskol.com	A record	88.99.218.172	Node 1 Nginx
npm2.bskol.com	A record	88.99.71.85	Node 2 Nginx
npm3.bskol.com	A record	94.130.8.161	Node 3 Nginx
talk.cnmcyber.com	A record	2a01:4f8:fff0:53::2	CNM Talk (Jitsi)
corp.cnmcyber.com	A record	2a01:4f8:fff0:53::3	CNM Corp (Odoo)
social.cnmcyber.com	A record	2a01:4f8:fff0:53::4	CNM Social (HumHub)

IP addresses

To locate its resources in the #Communication channels, the Farm uses three types of IP addresses:

1. To access #Virtual environments (VE) of various Nodes from the outside world, the Farm features public IPv6 addresses. One address is assigned to each Node. Since there are three Nodes, three addresses of that type are created.
2. For an internal network of three Nodes, which is assembled on the internal Bridge, a private IP address is used. This network is not accessible from the Internet and not included in the Farm's DNS zone. For instance, the #Storage platform utilizes this network to synchronize its data. For this network, an address with the type "/24" is selected.
3. For an external network of three Nodes, which is assembled on the external Bridge, the Farm features public IPv4 addresses. They are handled by #DNS intermediaries.

Web servers

To communicate with the outside world via HTTP, the Farm deploys two web servers. Nginx handles requests initially and Apache HTTP Server handles those requests that haven't handled by Nginx.

Security tools

Monitoring

Сейчас не используется специальные функции.

Предложения кандидатов:

1. Стек -- prometheus + node-exporter + grafana
2. Prometheus to monitor VMs, Influx to monitor Pve nodes , Grafana for Dashbord
3. (M) grafana + influxdb + telegraf, а также zabbix. Для мониторинга веб-сайта использовать uptimerobot

Firewalls

iptables as a firewall

For security, we use Fail2ban because it operates by monitoring log files (e.g. /var/log/auth.log, /var/log/apache/access.log, etc.) for selected entries and running scripts based on them. Most commonly this is used to block selected IP addresses that may belong to hosts that are trying to breach the system's security. It can ban any host IP address that makes too many login attempts or performs any other unwanted action within a time frame defined by the administrator. Includes support for both IPv4 and IPv6.

Backup and recovery

OpenZFS или RAID создаёт резервные копии и может быть задействовано для восстановления данных Железа в случае аварий. Жёсткие диски каждого Железа сдвоены, как, например, 2x SSD SATA 512 GB. RAID или OpenZFS копирует данные основного диска Железа на резервный диск. Если основной диск теряет данные из-за сбоя, резервный диск будет использован для восстановления данных на основной диск. RAID или OpenZFS устанавливается непосредственно на Железо.

Accesses

End-user access

End-users of the Farm (hereinafter, the Patrons) access the Apps and the Apps only. Those users cannot access to the #Bare-metal servers, #Backup server, #Virtual environments (VEs), as well as #Security tools.

The Patrons access the Apps via those IPv4 addresses that are associated with the particular App. Opplet.net provides the Patrons with access automatically or, by bureaucrats or other power-users, manually.

Power-user access

Power-users of the Farm (hereinafter, the Admins) are those users who have authorized to access more resources of the Farm than a regular Patron.

1. Hardware-level admin. Administrative access to #Bare-metal servers and #Backup server is carried out without any IP addresses, through the administrative panel and administrative consoles that #Service provider grants to CNMCyber Customer. The customer grants hardware-level admin access personally.
2. VE-level admin. Administrative access to #Virtual environments (VEs) and #Security tools is carried out through IPv6 addresses linked to those tools. Access credentials are classified and securely stored in CNM Lab.
3. App-level admin. Administrative access to the Apps is carried out through the IPv4 addresses associated with the particular App. At the moment, those accesses are provided by other Admins manually.

Infrastructure

The infrastructure of the Farm consists of hardware, #Bare-metal servers and #Backup server, as well as #Bridges rented from the #Service provider.

Service provider

Hetzner has been serving as CNMCyber Team's Internet service provider (ISP) and lessor of the #Infrastructure since 2016. Offers from other potential providers, specifically, Contabo and DigitalOcean, have been periodically reviewed, but no one else has offered any better quality/price rate on a long-term basis.

Choice of bare-metal

Due to the lower cost, #Bare-metal servers were purchased via #Service provider's auction -- https://www.hetzner.com/sb?hdd_from=500&hdd_to=1000 -- based on the following assumptions:

• Number: ProxmoxVE normally requires three nodes. The third node is needed to provide quorum; however, it shall not necessarily run applications. At the same time, Ceph requires three nodes at least.
• Hard drives:
  1. The hard drive storage capacity for any Node shall be 512Gb at least.
  2. Because Ceph is selected to power the #Storage platform, any hard-drive of the Farm shall be both SSD and NVMe.
• Processors:
  1. The processor frequency for two Nodes of the Farm shall be 32Gb at least. Processor frequency requirements to the third Node may be lower because of ProxmoxVE's characteristics.
  2. Those servers that deploys Intel Xeon E3-1275v5 processors are preferable over those servers that deploys Intel Core i7-7700 ones.
• Location: At least two Nodes shall be located in the same data center. Although the #Service provider does not charge for internal traffic, this circumstance increases the speed of the whole Farm. If no nodes are available in the same data center, they shall be looked for in the same geographic location.

The hardware characteristics of the chosen Nodes are presented in #Bare-metal servers.

Bridges

Сеть каждого Узла использует мост по выбираемой по умолчанию в Network Configuration модели.

Hetzner vSwitches (hereinafter, the Bridges) serve as bridges for #Communication channels to connect the Nodes in networks and switch from one Node to another one. The #Service provider provides CNMCyber Team with the Bridges; the team can order up to 5 connectors to be connected to one Node. The Bridges come with the lease of the Nodes.

The Farm utilizes two Bridges:

1. Internal Bridge serves as the hub for node and storage networks. It is located on an internal, private IPv6 address to provide for data transfer between the Nodes and their storage spaces.
2. External Bridge serves as the hub for the public network, the Internet. It is located on external, public IPv4 address to provide for data transfer between the Farm's publicly-available and other Internet resources.

The Farm cannot support high availability of the Bridges. Resiliency of the Bridges is the courtesy of their owner, #Service provider.

Backup server

A Proxmox Backup Server is deployed on a 1 TB, unlimited traffic storage box BX-11 that has been rented for that purpose.

• Basic features: 10 concurrent connections, 100 sub-accounts, 10 snapshots, 10 automated snapshots, FTP, FTPS, SFTP, SCP, Samba/CIFS, BorgBackup, Restic, Rclone, rsync via SSH, HTTPS, WebDAV, Usable as network drive
• #Service provider's description: Storage Boxes provide you with safe and convenient online storage for your data. Score a Storage Box from one of Hetzner Online's German or Finnish data centers! With Hetzner Online Storage Boxes, you can access your data on the go wherever you have internet access. Storage Boxes can be used like an additional storage drive that you can conveniently access from your home PC, your smartphone, or your tablet. Hetzner Online Storage Boxes are available with various standard protocols which all support a wide array of apps. We have an assortment of diverse packages, so you can choose the storage capacity that best fits your individual needs. And upgrading or downgrading your choice at any time is hassle-free!

Bare-metal servers

The #Virtual environments (VEs) are deployed on three bare-metal servers. As the result of #Choice of bare-metal, #Node 1 hardware, #Node 2 hardware, and #Node 3 hardware have been rented for that purpose.

Node 1 hardware

1 x Dedicated Root Server "Server Auction"

• Intel Xeon E3-1275v5
• 2x SSD M.2 NVMe 512 GB
• 4x RAM 16384 MB DDR4 ECC
• NIC 1 Gbit Intel I219-LM
• Location: FSN1-DC1
• Rescue system (English)
• 1 x Primary IPv4

Node 2 hardware

1 x Dedicated Root Server "Server Auction"

• Intel Xeon E3-1275v5
• 2x SSD M.2 NVMe 512 GB
• 4x RAM 16384 MB DDR4 ECC
• NIC 1 Gbit Intel I219-LM
• Location: FSN1-DC1
• Rescue system (English)
• 1 x Primary IPv4

Node 3 hardware

1 x Dedicated Root Server "Server Auction"

• Intel Xeon E3-1275v5
• 2x SSD M.2 NVMe 512 GB
• 4x RAM 16384 MB DDR4 ECC
• NIC 1 Gbit Intel I219-LM
• Location: FSN1-DC1
• Rescue system (English)

See also

Related lectures

• What CNM Farms Are.

Useful recommendations

• https://www.informaticar.net/how-to-setup-proxmox-cluster-ha/ (using Ceph without Hetzner vSwitch)
• https://community.hetzner.com/tutorials/hyperconverged-proxmox-cloud (using Ceph with Hetzner vSwitch)
• https://pve.proxmox.com/wiki/High_Availability (general ProxmoxVE HA functionality)
• https://docs.hetzner.com/robot/dedicated-server/network/vswitch/ (general Hetzner vSwitch functionality)

Used terms

Lexicon

On this wiki page, the following terms are used for common concepts:

• A record. The DNS record that specifies the IPv4 address that corresponds to the domain name. The letter "A" in the name of the record, the so-called record type, came into the name from the first letter of the English word "address" (address).
• AAAA record. The DNS record that specifies the IPv6 address that corresponds to the domain name. The four "A"s in this record type represent the fact that the maximum number of IPv6 addresses (128 bits) is four times the maximum number of IPv4 addresses (32 bits).
• IP address. The address of a computing device or another web resource in the Internet that corresponds to the conventional Internet protocol (IP). When a World Wide Web visitor types in a domain name, such as "cnmcyber.com", the web browser looks in the DNS zone for the IP address to which the domain name is bound. IPv4 addresses correspond to IPv4 or IP, version 4. IPv6 addresses correspond to IPv6 or IP, version 6. Internet service providers (ISPs) lease public addresses to hosting providers and/or domain name registrars for re-selling, or, directly, to end customers.
• IPv4 address. The IP address that corresponds to the IPv4 Internet protocol. These addresses are 4 groups of numbers separated by dots. For example, 88.99.71.85 is one of the addresses of CNMCyber. Some addresses are reserved for private networks and cannot appear on the Internet. The number of IPv4 addresses is limited to 4.3 trillion, which at the time of development seemed like a sufficient number. IPv4 was developed in 1981. To solve the limitation problem, IPv6 was developed in 1995, but as of summer 2022, 62% of the Internet continues to use IPv4. In DNS zones, this address is specified in A records.
• IPv6 address. The IP address that corresponds to the IPv6 Internet protocol. These addresses are several groups of numbers and letters separated by colons. Some groups may be empty. For example, 2a01:4f8:fff0:53::2 is one of CNMCyber addresses and the group between double colons is empty. In DNS zones, this address is specified in AAAA records.
• DNS (Domain Name System). A hierarchical and decentralized domain name system that was originally created to link human-recognizable domain names to machine-processed Internet protocol addresses (IP addresses), and later came to be used to determine other data of these names and addresses. For example, a public key to sign mail can be added to text records. DNS records are contained in so-called DNS zones, which Internet service providers (ISPs) store.

• DNS record (resource record). Привязка стандартизованных данных к конкретному доменному имени. Запись состоит из типа (type), например , "AAAA" в AAAА записи, названия (resource record), например, jitsi.bskol.com, и привязанных к названию данных (data). Вместе, записи составляют DNS зону. Linking standardized data to a specific domain name. A record consists of a type (type), for example, "AAAA" in an AAAA record, a name (resource record), for example, jitsi.bskol.com, and data associated with the name (data). Together, the records make up a DNS zone.
• DNS zone. Ta часть системы доменных имён (DNS), которая управляется отвечающим в системе за конкретное доменное имя поставщиком услуг Интернета (Internet service provider или ISP) и которая определяет данные, связанные с этим доменным именем. Эти данные представлены в виде DNS записей, таких, как A запись или AAAA запись. That part of the Domain Name System (DNS) that is managed by the Internet service provider (ISP) responsible for a particular domain name in the system and that defines the data associated with that domain name. This data is in the form of DNS records, such as an A record or an AAAA record.

• Virtual machine (VM). A virtual computing device that simulates a computer and is created by a virtual environment. Similar to a regular computer, an operating system is installed on a VM, usually out of the box, and, on it, user applications.
• High availability (HA). The property of a system to have a higher uptime than an identical system that does not use high availability tools and techniques. No system and no part of a system can be completely protected from the threat of abnormal operation or an emergency. High availability can be described as the continued provision of services by the system at some "healthy" level when a certain part of it fails, while simultaneously recovering the very part that suffered from the failure. High-availability tools include redundant parts that are ready to take over the role of primary parts, monitoring devices to detect failures, and control devices that fencing non-working parts and redirect requests to working ones. The requirement for a “good”, albeit emergency, state distinguishes high availability from the concept of failure tolerance, which seeks to ensure that the average user of the system does not notice the failure of part of it.

• Domain name (hostname). Воспринимаемое людьми название веб-сайта или иного ресурса, особенно в сети Интернет, например, "bskol.com". Веб-просмотрщики и другие устройства работают с IP адресами, но эти адреса трудны для запоминания и воспроизведения людьми; для них, созданы доменные имена. В зонах DNS, доменные имена привязаны либо к IPv4 адресу, либо к IPv6 адресу, либо к обоим. The perceived name of a website or other resource, especially on the Internet, such as "bskol.com". Web browsers and other devices work with IP addresses, but these addresses are difficult for humans to remember and reproduce; for them, created domain names. In DNS zones, domain names are bound to either an IPv4 address, or an IPv6 address, or both.
• Container. Виртуальное компьютерное устройство, имитирующее компьютер с установленной операционной системой и пользовательскими приложениями, создаваемое виртуальной средой. Как правило, контейнеры задействуют облегчённую операционную систему, заточенную исключительно под работу установленных приложений. A virtual computing device that simulates a computer with an installed operating system and user applications, created by a virtual environment. As a rule, containers use a lightweight operating system, tailored exclusively for running installed applications.
• Operating system (OS). Программное обеспечение, которое, с одной стороны, взаимодействует либо с железным, либо с виртуальным компьютерным устройством и, с другой стороны, может взаимодействовать с пользовательскими приложениями. Software that, on the one hand, interacts with either a hardware or virtual computing device and, on the other hand, can interact with user applications.
• Failure tolerance -- это концепция такой работы системы, в которой конечный пользователь системы не может заметить отказа её части от штатной работы. Некоторые инструменты и методики отказоустойчивости аналогичны инструментам и методикам высокой доступности (high availability), которые способствуют предоставлению услуг системой при сбое её определённой части с одновременным восстановлением той самой части, которая пострадала от сбоя. Однако никакой набор не гарантирует, что любое восстановление будет моментальным и 100% полным. Потому "отказоустойчивость" -- это всё же концепция, к которой можно стремиться, но не конечная точка, которую можно достичь. Failure tolerance is the concept of such a system operation in which the end user of the system cannot notice the failure of part of it from regular work. Some fault-tolerance tools and techniques are similar to high availability tools and techniques, which facilitate the provision of services by a system when a certain part of it fails while recovering the very part that suffered the failure. However, no set guarantees that any recovery will be instant and 100% complete. So "fault tolerance" is still a concept to strive for, but not an end point to be reached.

• Internet service provider (ISP). An organization authorized by the Internet administration to provide domain names, store DNS zones, and return public requests with their data. With some exceptions, ISPs provide network access directly to end users or resellers. Many ISPs are also hosting providers.

Специальные термины

На данной вики-странице, используются следующие термины, которые специфичны для этой страницы:

• Bare-metal server. "Физический, железный" сервер, арендуемый у поставщика услуг размещения и описанный в Инфраструктуре.
• End-user application. Одна из установленных на Ферме деловых прилад.
• Поставщик услуг размещения. Поставщик услуг Интернета (Internet service provider или ISP), предоставляющий свои подключённые к сети Интернет "железные" сервера в аренду для размещения Фермы.
• Соединитель. Коммутационное устройство предоставляемое поставщиком услуг размещения Фермы и описанное в Соединителях.
• Virtual environment (VE). Виртуальнaя среда на базе программного обеспечения ProxmoxVE, описанная в Виртуальных средах.
• Узел (node). Комбинация одного Железа и установленного на нём программного обеспечения, представленная в сети и описанная в Узлах Фермы.
• Ферма. Делова Ферма, для описания которой предназначена данная вики-страница.
• Хранилище. Система для хранения объектов, блоков и файлов, которые Ферма либо обрабатывает, либо предоставляет пользователям без обработки. Термины "хранилище Узла" или, во множественном числе, "хранилища", подразумевают системы хранения на отдельном Узле. Система описана в Хранилищах Узлов.