Difference between revisions of "CNM Bureau Farm"

From CNM Wiki
Jump to: navigation, search
(The Cluster)
 
(491 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[CNM Bureau Farm]] (formerly known as [[CNM EndUser Farm]]; hereinafter, the ''Farm'') is the [[CNM Farms|CNM farm]] that hosts [[CNM Social]], [[CNM Talk]], and [[CNM Venture]] (hereinafter, the ''Apps''). The ''Apps'' are described in [[#The Apps]] section of this wikipage.
+
[[CNM Bureau Farm]] (formerly known as [[CNM EndUser Farm]]; hereinafter, [[#The Farm]]) is the segment of [[Opplet]] that is responsible for providing users with [[CNM Corp]] and [[CNMCyber.com]]. Most likely, [[#The Farm]] will also handle the applications that [[CNM Fed Farm]] currently provides.
  
Technically, the ''Farm'' is a collection of [[commercial off-the-shelf]] ([[commercial off-the-shelf|COTS]]) software. The ''Farm's'' [[end-user]]s (hereinafter, the ''Patrons'') work with the ''Apps'' that are installed in [[#The Cluster]] that is, consequently, installed on the [[#Infrastructure]] that consists of [[#The Bridges]] and hardware, which includes [[#Backup box]] and [[#Bare-metal servers]].
+
[[CNMCyber Team]] (hereinafter, [[#The Team]]) develops and administers [[#The Farm]]. To sustain [[#The Farm]] online, [[#The Team]] rents [[Bureau Infrastructure]].
  
To eliminate a [[single point of failure]], the ''Farm'' is built on not just one, but three [[#Bare-metal servers]]. Each of those hardware servers with all of software installed on the top of it (hereinafter, the ''Node'') is self-sufficient to host the ''Apps''. [[#High availability (HA)]] tools orchestrate coordination between the ''Nodes''.
 
  
 +
==In the nutshell==
 +
For the purposes of this very wikipage, [[end-user]]s are called [[#The End-Users]]. Collectively, [[end-user application]]s are called [[#The User Apps]].
  
==The Apps==
+
===Architecture===
Literally, the ''Apps'' are those [[end-user application]]s with which the ''Patrons'' interact. The ''Apps'' can be deployed utilizing two models:
+
: While using [[#The Farm]], [[#The End-Users]] work with [[#The User Apps]] that are installed in [[#The Cluster]] that is, consequently, hosted by [[#The Infrastructure]]. [[#The Cluster]] consists of [[#The Storage]], [[#The Environment]], [[#The Gateway]]. [[#The Infrastructure]] includes [[#The Bridges]] and [[#The Metal]], which is [[#The Farm]]'s hardware.
# Using [[container]]s; they already contain [[operating system]]s tailored specifically to the needs of the ''App''.
+
 
# In [[virtual machine]]s ([[virtual machine|VM]]) and without [[container]]s. In that model, the ''App'' is installed on the [[operating system]] of its [[virtual machine|VM]].
+
===Cluster-based===
 +
: To mitigate a [[single point of failure]] ([[single point of failure|SPOF]]), [[#The Farm]] is built on not just one, but three [[bare-metal server]]s. Each of those hardware servers with all of software installed on the top of it (hereinafter, [[#The Node]]) is self-sufficient to host [[#The User Apps]]. Various solutions such as [[#The uptime tools]] orchestrate coordination between [[#The Node]]s.
 +
 
 +
===COTS-powered===
 +
: [[#The Farm]]'s software is a collection of [[commercial off-the-shelf]] ([[commercial off-the-shelf|COTS]]) packages (hereinafter, [[#The COTS]]). In plain English, [[#The COTS]] is the software that is already available on the market. No single line of programming code is written specifically for [[#The Farm]]. Only time-tested market-proven solutions have been used. [[#The User Apps]] use [[HumHub]], [[Jitsi]], and [[Odoo]] instances. [[#The Cluster]] uses [[Ceph]], [[Proxmox]], and [[pfSense]].
 +
 
 +
==Addressing the needs==
 +
Development and sustenance of [[#The Farm]] address two [[business need]]s of [[CNMCyber Team]] (hereinafter, [[#The Team]]). For [[#The Team]], [[#The Farm]] shall serve as both [[#Tech side of the Apps]] and [[#Worksite]].
 +
 
 +
===Tech side of the Apps===
 +
: [[#The Team]] needs to provide [[#The End-Users]] with the services of [[#The User Apps]] 24 hours a day, 7 days a week. [[#The Farm]] shall power [[#The User Apps]] technologically; their content is outside of [[#The Farm]]'s scope.
 +
 
 +
===Worksite===
 +
: [[#The Team]] needs to provide those incumbents of [[CNMCyber practice]]s who work with [[#The Farm]]'s technologies with their worksite.
 +
 
 +
==Farm users==
 +
For the purposes of this very wikipage, a ''farm user'' refers to any user of [[#The Farm]]. The ''farm user's'' [[authentication]] and [[authorization]] administration is a part of [[identity and access management]] ([[Identity and access management|IAM]]).
 +
 
 +
===The End-Users===
 +
: For the purposes of this very wikipage, the ''Patron'' refers to an [[end-user]] of [[#The User Apps]]. They access [[#The User Apps]] via the [[graphic user interface]]s ([[graphic user interface|GUI]]s) that are associated with the particular application. At [[#The Farm]], those [[#User interfaces (UIs)]] are located at [[IPv4 address]]es. [[Opplet.net]] provides the ''Patrons'' with access automatically or, by [[#The Power-Users]], manually. The ''Patrons'' can access [[#The User Apps]] and [[#The User Apps]] without administrative tools only. The ''Patrons'' may or may not be members of [[#The Team]].
 +
 
 +
===The Power-Users===
 +
: For the purposes of this very wikipage, the ''Power-User'' refers to a [[power-user]] of [[#The User Apps]]. By definition, the ''Power-Users'' are those members of [[#The Team]] who have authorized to access more of [[#The User Apps]]' resources than [[#The End-Users]]. Normally, those resources are administrative tools that allow the ''Power-Users'' administer one or more of [[#The User Apps]].
 +
 
 +
: While having administrative access, the ''Power-Users'' serve as application-level administrators. They access [[#The User Apps]] via the same [[graphic user interface]]s ([[graphic user interface|GUI]]s) as [[#The End-Users]], but administrative-level ''UIs'' display administration tools in addition to those tools that are available to non-privileged [[end-user]]s.
 +
 
 +
: At the moment, [[#The Sysadmins]] can grant the ''Power-User'' rights to one or more of [[#The End-Users]] manually. Administrative access credentials are classified and securely stored in the application project spaces of [[CNM Lab]]. Those incumbents of [[CNMCyber practice]] who work with [[#The Farm]]'s technologies may or may not be the ''Power-Users''.
 +
 
 +
===The Sysadmins===
 +
: For the purposes of this very wikipage, the ''Sysadmin'' refers to a [[system administrator]] of [[#The Cluster]]. By definition, the ''Sysadmins'' are those members of [[#The Team]] who have authorized to access at least some of [[#The Cluster]]'s resources:
 +
:* [[#The Environment]] access is carried via [[#UI for the Environment]]. That access also includes root-level access to all of the applications that [[#The Environment]] hosts. While having such an access, the ''Sysadmins'' are able to delete or re-install applications such as [[#The User Apps]].
 +
:* [[#The Storage]] access is carried via [[#UI for the Storage]].
 +
:* [[#The Gateway]] access is carried via [[#UI for the Gateway]].
 +
 
 +
: [[#The Superusers]] can grant the ''Sysadmin'' rights to one or more of [[#The End-Users]] manually. Administrative access credentials are classified and securely stored in the cluster project spaces of [[CNM Lab]]. No incumbent of [[CNMCyber practice]] can be the ''Sysadmin''; only those who work in [[CNMCyber Office]] can.
 +
 
 +
===The Superusers===
 +
: For the purposes of this very wikipage, the ''Superuser'' refers to a [[superuser]] of [[#The Farm]]. By definition, the ''Superusers'' are those members of [[#The Team]] who have authorized to access any part of [[#The Farm]] starting with its hardware. [[#The Metal]]'s access is carried via [[#UI for the Metal]].
 +
 
 +
: [[#The Provider]] grants the ''Superuser'' rights to [[CNMCyber Customer]], who can manually pass it to one or more members of [[CNMCyber Office]]. The ''Superuser'' credentials are not stored in [[CNM Lab]].
 +
 
 +
==User interfaces (UIs)==
 +
For the purposes of this very wikipage, a [[user interface]] ([[user interface|UI]]) refers to [[#The COTS]]' feature that allows [[#The COTS]]' instance and its users to interact. ''UIs'' of [[#The User Apps]] are described in the wikipages that are dedicated to particular applications. The other ''UIs'' of [[#The Farm]] are described in the [[#UI of the Environment]], [[#UI of the Storage]], [[#UI of the Gateway]], and [[#UI for the Metal]] sections of this very wikipage.
 +
 
 +
===Dashboards===
 +
: For the purposes of this very wikipage, a ''Dashboard'' refers to a [[graphic user interface]] ([[graphic user interface|GUI]]) that either belongs to any [[#The COTS]] package installed in [[#The Farm]] or is provider by [[#The Provider]]. This screen-based interface allows [[#Farm users]] to interact with [[#The User Apps]] and other software through graphical buttons, icons, or hyperlinked texts rather than typed commands in [[command line interface]]s ([[command line interface|CLI]]s).
 +
 
 +
===Third-party UIs===
 +
: For the purposes of this very wikipage, a ''Third-party UI'' refers to any [[user interface]] ([[user interface|UI]]) that neither belongs to any [[#The COTS]] package installed in [[#The Farm]] nor is provider by [[#The Provider]]. [[#The Sysadmins]] and [[#The Suparusers]] may use access tools such as [[PuTTy]] and [[Midnight Commander]] to access [[#UI for the Metal]].
 +
 
 +
==The User Apps==
 +
For the purposes of this very wikipage, the ''User Apps'' refer to those [[end-user application]]s with which [[#The End-Users]] interact.
  
 
===HumHub===
 
===HumHub===
: [[CNM Social]], which is the end-user instance of [[CNM HumHub]].
+
: [[CNMCyber.com]], which is the end-user instance of [[Educaship HumHub]].
 +
 
 +
===Jitsi===
 +
: [[CNM Talk]], which is the end-user instance of [[Educaship Jitsi]].
  
 
===Odoo===
 
===Odoo===
: [[CNM Venture]], which is the end-user instance of [[CNM Odoo]].
+
: [[CNM Corp]], which is the end-user instance of [[Educaship Odoo]].
  
===Jitsi===
+
==The Cluster==
: [[CNM Talk]], which is the end-user instance of [[CNM Jitsi]].
+
For the purposes of this very wikipage, the ''Cluster'' refers to all of the software between [[#The User Apps]] and [[#The Infrastructure]].
 +
 
 +
===Cluster components===
 +
: [[#The Cluster]] consists of [[#The Node]]s and their management tools. The following components compose [[#The Cluster]]:
 +
:# '''[[#The Storage]]''' that is powered by [[Educaship Ceph]] to provide [[#The Environment]] with stored objects, blocks, and files. Three storage spaces of [[#The Node]]s create one distributed storage foundation.
 +
:# '''[[#The Environment]]''' that is powered by [[Educaship Proxmox]] to make [[container]]s and [[virtual machine]]s ([[virtual machine|VM]]s) available to [[#The User Apps]], so [[#The User Apps]] can function properly. Each of [[#The Node]]s features its own environment; [[#The uptime tools]] orchestrate them all.
 +
:# '''[[#The Gateway]]''' that is powered by [[Educaship pfSense]] to create a gateway between [[#The Environment]] and the outside world. There is only one gateway; if it fails, [[#The Farm]] fails.
 +
 
 +
===Choice of COTS===
 +
: While building [[#The Farm]] generally and [[#Cluster components]] specifically, [[#The Team]] utilized only [[#The COTS]] that is both [[open-source]] and free of charge. Other considerations for the choice are stated in the [[#COTS for the Environment]], [[#COTS for the Storage]], [[#COTS for the Gateway]], [[#COTS for backups]] sections of this very wikipage.
 +
 
 +
===Cluster provisioning===
 +
: Provisioning of [[#The Cluster]] occurs in the following sequence:
 +
:# Since [[#The Environment]] is installed on the top of [[#The Infrastructure]], the [[#Environment provisions]] shall be accommodated first.
 +
:# Since [[#The Storage]] is a part of [[#The Environment]], [[#Storage provisions]] shall be accommodated second.
 +
:# Since [[#The Gateway]] is installed in [[#The Environment]], [[#Gateway provisions]] shall be accommodated third.
 +
 
 +
===Cluster monitoring===
 +
: Monitoring features are to be identified. Previously, various candidates offered three options:
 +
:# Stack -- [[prometheus]] + node-exporter + [[grafana]]
 +
:# [[Prometheus]] to monitor VMs, Influx to monitor Pve nodes, [[Grafana]] for Dashbord
 +
:# [[grafana]] + [[influxdb]] + [[telegraf]], as well as [[zabbix]]. To monitor websites, use [[uptimerobot]]
 +
 
 +
Observability vs. APM vs. Monitoring
 +
 
 +
application performance management
  
==The Cluster==
+
===Cluster recovery===
For the purposes of this very wikipage, the ''Cluster'' refers to all of the software between [[#The Apps]] and [[#Infrastructure]]. the cluster of three ''Nodes'' that [[#High availability (HA)]] tools orchestrate.
+
==High availability (HA)==
 +
Generally speaking, [[high availability]] ([[High availability|HA]]) of any system assumes its higher uptime in comparison with a similar system without higher uptime ability. [[High availability|HA]] of [[#The Farm]] assumes its higher uptime in comparison with a similar farm built on one of [[#The Node]]s. Before [[#The uptime tools]] were deployed, [[#The Farm]] functioned only on one of [[#The Node]]s and, when it failed, services of [[#The User Apps]] were no longer available until the failure was fixed. Now, until at least one of [[#The Node]]s is operational, [[#The Farm]] is operational.
  
===Virtual environment (VE)===
+
===The uptime tools===
: [[CNMCyber Team]] uses [[virtualization]] to divide excessive hardware resources of [[#Bare-metal servers]] in smaller [[container]]s and [[virtual machine]]s ([[virtual machine|VM]]s), which are created in [[virtual environment]]s ([[Virtual environment|VE]]s).
+
: Both [[#The Environment]] and [[#The Storage]] feature advanced tools for [[#High availability (HA)]].
 +
:* '''The [[Educaship Proxmox]] instances'''. With regards to [[#The Farm]]'s applications, when any application fails, its work continues its sister application installed on the second of [[#The Node]]s. If another application fails, its work continues its sister application installed on the third of [[#The Node]]s. If the third application fails, [[#The Farm]] can no longer provide [[#The End-Users]] with [[#The Farm]]'s services in full. To ensure that, [[#The Farm]] utilizes tools that come with [[ProxmoxVE]]. Every [[virtual machine]] ([[virtual machine|VM]]) or [[container]] is kept on at least two of [[#The Node]]s. When the operational resource, [[virtual machine|VM]] or [[container]], fails on one instance, the second [[Educaship Proxmox]] instance activates its own resource and requests the third instance to create the third resource as a reserve. As a result, [[virtual machine|VM]] or [[container]] "migrates" from one of [[#The Node]]s to another.
 +
:* '''The [[Educaship Ceph]] instance'''. Because of distributed nature of [[Ceph]], [[#High availability (HA)]] is the native feature of [[#The Storage]]. When one [[database management system|DBMS]] fails, its work continues its sister [[database management system|DBMS]] installed on the second of [[#The Node]]s. When another [[database management system|DBMS]] fails, its work continues its sister [[database management system|DBMS]] installed on the third of [[#The Node]]s. If the third [[database management system|DBMS]] fails, [[#The Farm]] can no longer provide [[#The User Apps]] with the data it requires to properly work.
  
: As its software for [[Virtual environment|VE]]s, the ''Farm'' utilizes [[CNM ProxmoxVE]]. Every instance of [[CNM ProxmoxVE]] is installed on [[#Node OS]], which require "physical" [[#Bare-metal server]]s. The ''Farm's'' [[CNM ProxmoxVE]] also utilizes [[#Storage platform]] as its storage.
+
===Uptime limitations===
 +
: Generally speaking, [[High availability|HA]] comes with significant costs. So does [[#The uptime tools]]. At very least, running three of [[#The Node]]s is more expensive than running one. The cost cannot exceed the benefit, so [[high availability]] ([[high availability|HA]]) cannot be equal to [[failure tolerance]].
  
===COTS for VE===
+
===Uptime management===
: [[CNMCyber Team]] has tried [[OpenStack]] and [[VirtualBox]] as its virtualization tools. The trials suggested that [[OpenStack]] required more hardware resources and [[VirtualBox]] didn't allow for required sophistication in comparison with [[ProxmoxVE]], which has been chosen as [[Commercial off-the-shelf|COTS]] for the ''Farm's'' virtualization.
+
: To manage redundant resources, [[#The uptime tools]]:
 +
:* '''Monitor''' its resources to identify whether they are operational or failed as described in the [[#Monitoring]] section of this very wikipage.
 +
:* '''Fence''' those resources that are identified as failed. As a result, non-operational resources are withdrawn from the list of available.
 +
:* '''Restore''' those resources that are fenced. The [[#Recovery]] supports that feature, while constantly creating snapshots and reserve copies of [[#The Farm]] and its parts in order to make them available for restoring when needed.
  
===VE provisioning===
+
===Uptime principles===
: The interaction between [[CNM ProxmoxVE]] instances and the [[#Infrastructure]] is carried out by [[Debian]] [[operating system]] ([[operating system|OS]]) that comes in the same [[commercial off-the-shelf|COTS]] "box" as [[ProxmoxVE]] and is specially configured for that interaction.
+
: Principally, [[#High availability (HA)]] of [[#The Cluster]] is based on:
 +
:* A principle of redundancy. Each of [[#The User Apps]], as well as every object, block, or file that [[#The User Apps]] may use is stored at least twice on different hardware servers of [[#The Node]]s as [[#The uptime tools]] and [[#Uptime of the Storage]] sections describe.
 +
:* Management of redundant resources. [[#The Cluster]] needs to put into operations those and only those resources that are in a good standing and operational shape as described in the [[#Uptime management]] section.
  
==Storage platform==
+
==The Environment==
To make objects, blocks, and files immediately available for the ''Apps<nowiki>'</nowiki>'' operations, the ''Farm'' uses [[CNM Ceph]] as its [[storage]]. This common distributed cluster foundation orchestrates storage spaces of the individual ''Nodes''.
+
For the purposes of this very wikipage, the ''Environment'' refers to the [[virtual environment]] ([[virtual environment|VE]]) of [[#The Cluster]] or, allegorically, to the environment where [[#The User Apps]] "live".
  
===COTS for storage===
+
===COTS for the Environment===
: [[CNMCyber Team]] has tried [[OpenZFS]] and [[RAID]] as the ''Farm's'' storage. Initially, the original cluster developer proposed using [[Ceph]]. Later, the team substituted one node with another with higher hard disk, but without [[SSD]] and [[NVMe]]; as a result, the ''Farm's'' storage collapsed. The substituted node was disconnected (today, it serves as hardware for [[CNM Lab Farm]]), a new [[bare-metal server]] was purchased (today, it is the [[#Node 3 hardware]]) and [[Ceph]] restored.
+
: As [[#The COTS]] for the ''Environment'', [[#The Team]] utilizes [[Educaship Proxmox]]. For a while, [[#The Team]] has also tried [[OpenStack]] and [[VirtualBox]] as its virtualization tools. The trials suggested that [[OpenStack]] required more hardware resources and [[VirtualBox]] didn't allow for required sophistication in comparison with [[ProxmoxVE]], which has been chosen as [[#The COTS]] for [[#The Farm]]'s virtualization.
  
: As [[Commercial off-the-shelf|COTS]], [[ProxmoxVE]] comes with [[OpenZFS]]. [[CNMCyber Team]] has deployed the combination of both in its [[CNM Lab Farm]].
+
===Environment features===
 +
: [[#The Team]] uses [[virtualization]] to divide hardware resources of [[#The Node]]'s [[bare-metal server]]s in smaller [[container]]s and [[virtual machine]]s ([[virtual machine|VM]]s), which are created in the ''Environment'' to run [[#The User Apps]], [[#The Gateway]], and other applications. In [[#The Farm]], the [[ProxmoxVE]] instance tightly integrates [[KVM hypervisor]], [[LXC container]]s, [[Educaship Ceph]] as software-defined storage, as well as networking functionality on a single virtualization platform.
  
===Deployment model===
+
===Environment functions===
: At the ''Farm'', [[CNM Ceph]] is deployed at every ''Node''. Every of [[#Bare-metal servers]] features doubled hard disks. Physically, [[CNM ProxmoxVE]] is installed on one disk of each ''Node''; [[CNM Ceph]] uses three "second" disks. Since every disk is 512 GB, the ''Farm's'' [[CNM Ceph]] capacity is about 3 * 512 GB = 1.536 GB.
+
: [[#The Environment]] executes four major functions. It:
 +
:# '''Runs''' [[#The User Apps]], [[#The Gateway]], and other applications. They can be deployed utilizing two models:<ol type="a"><li>Using [[container]]s; they already contain [[operating system]]s tailored specifically to the needs of the ''App''.</li><li>In [[virtual machine]]s ([[virtual machine|VM]]) and without [[container]]s. In that model, the ''App'' is installed on the [[operating system]] of its [[virtual machine|VM]].</li></ol>
 +
:# '''Hosts''' [[#The Storage]] and [[#Backup box]]
 +
:# '''Connects''' the applications it runs and the storages it hosts to each other and to [[#The Bridges]], while creating networks.
 +
:# '''Creates''' backups and accommodates its own recovery when requested.
  
: While experimenting with [[OpenZFS]] and [[RAID]], [[CNMCyber Team]] has also tried another model. The second disks then served as reserve copies of the first ones.
+
===Environment provisions===
 +
: Every instance of [[ProxmoxVE]] requires one "physical" [[bare-metal server]]. The interaction between [[ProxmoxVE]] instances and [[#The Infrastructure]] is carried out by [[Debian]] [[operating system]] ([[operating system|OS]]) that comes in the same "box" of [[#The COTS]] as [[ProxmoxVE]] and is specifically configured for that interaction. [[#The Farm]]'s [[ProxmoxVE]] also hosts [[#The Storage]] as its [[storage]].
  
===Storage provisioning===
+
===UI of the Environment===
 +
: With regards to [[#User interfaces (UIs)]], [[#The Environment]] features its administrative interface, which belongs to [[#Dashboards]].
  
==High availability (HA)==
+
==The Storage==
[[High availability]] ([[High availability|HA]]) of the ''Farm'' assumes that no failure of any ''App'' or its [[database management system]] ([[database management system|DBMS]]) can cause the failure of the ''Farm'' as a whole. [[High availability|HA]] tools are based on:
+
For the purposes of this very wikipage, the ''Storage'' refers to the storage platform or [[database management system]] ([[database management system|DBMS]]) that provides [[#The User Apps]] with the [[storage]] they need to operate. Thus, the ''Storage'' supports [[#The Environment]]'s non-emergency operations and differs from the [[#Backup box]] that comes into play in emergencies.
* A principle of redundancy; once again, that is why the ''Farm'' is built on three ''Nodes'', not one. As a part of [[#Virtual environment (VE)]], every ''App'' is installed at least twice on different ''Nodes'' as described in the [[#HA at the App level]] section. As a part of [[#Storage platform]], every object, block, or file is stored at least twice on different ''Nodes'' as described in the [[#HA at the DBMS level]] section.
 
* Management of redundant resources as described in the [[#HA management]] section. In plain English, the ''Farm'' needs to put into operations those and only those resources that are in operational shapes.
 
Generally speaking, [[High availability|HA]] comes with significant costs. So does [[High availability|HA]] of the ''Farm''. At very least, running three ''Nodes'' is more expensive than running one. The cost cannot exceed the benefit, so [[high availability]] cannot be equal to [[failure tolerance]].
 
  
===HA at the App level===
+
===COTS for the Storage===
: When one ''App'' fails, its work continues its sister ''App'' installed on the second ''Node''. If another ''App'' fails, its work continues its sister ''App'' installed on the third ''Node''. If the third ''App'' fails, the ''Farm'' cannot provide the ''Patrons'' with the ''App'' services any longer.
+
: As [[#The COTS]] for [[#The Storage]], [[#The Team]] utilizes [[Educaship Ceph]]. Any [[ProxmoxVE]] instance requires some [[storage]] to operate.
  
: To ensure that, the ''Farm'' utilizes tools that come with [[ProxmoxVE]]. Every [[virtual machine]] ([[virtual machine|VM]]) or [[container]] is kept on at least two ''Nodes''. When the operational resource, [[virtual machine|VM]] or [[container]], fails, [[CNM ProxmoxVE]] activates another resource and creates the third resource as a reserve. As a result, [[virtual machine|VM]] or [[container]] "migrates" from one ''Node'' to another ''Node''.
+
: Before deploying [[#The uptime tools]], [[#The Team]] used [[RAID]] to make the double hard disks redundant. So, the [[ProxmoxVE]] instance was just installed on the top of one disk and replicated to the other disk automatically. Flexibly, [[ProxmoxVE]] allows for better usage of hard disks. [[ProxmoxVE]] can be configured to host as [[#The COTS]], many storage-type software packages such as [[ZFS]], [[NFS]], [[GlusterFS]], and so on.
  
===HA at the DBMS level===
+
: Initially, the cluster developer proposed using [[Ceph]]. Later, [[#The Team]] substituted one node with another with higher hard disk, but without [[SSD]] and [[NVMe]]; as a result, [[#The Farm]]'s storage collapsed. The substituted node was disconnected (today, it serves as hardware for [[CNM Lab Farm]]), a new [[bare-metal server]] was purchased (today, it is the [[#Node 3 hardware]]) and [[Ceph]] restored.
: When one [[database management system|DBMS]] fails, its work continues its sister [[database management system|DBMS]] installed on the second ''Node''. When another [[database management system|DBMS]] fails, its work continues its sister [[database management system|DBMS]] installed on the third ''Node''. If the third [[database management system|DBMS]] fails, the ''Farm'' can no longer provide the ''App'' with the data it requires to properly work.
 
  
: To ensure that, the ''Farm'' utilizes the [[#Storage platform]]. Every object, block, or file is kept on at least two ''Nodes''. When any stored resource fails, [[#Storage platform]] activates another resource and creates the third resource as a reserve. As a result, any stored resource "migrates" from one ''Node'' to another ''Node''.
+
: As [[#The COTS]], [[ProxmoxVE]] comes with [[OpenZFS]]. [[#The Team]] has deployed the combination of both in its [[CNM Lab Farm]].
  
===HA management===
+
===Storage features===
: To manage redundant resources, the ''Farm'':
+
: [[#The Storage]] features are:
:* '''Monitors''' its resources to identify whether they are operational or failed as described in the [[#Monitoring]] section.
+
:* File system
:* '''Fences''' those resources that are identified as failed. As a result, non-operational resources are withdrawn from the list of available.
+
:* Distributed
:* '''Restores''' those resources that are fenced. The [[#Recovery]] supports that feature, while constantly creating snapshots and reserve copies of the ''Farm'' and its parts in order to make them available for restoring when needed.
+
:* Fault tolerant
 +
:* Object storage
 +
:* Block device
  
==Web architecture==
+
===Storage functions===
For the purposes of this wikipage, "web architecture" refers to the ''Farm's'' outline of [[DNS record]]s and [[IP address]]es.
+
: To make objects, blocks, and files immediately available for [[#The User Apps]]' operations, [[#The Cluster]] uses a common distributed cluster foundation that orchestrates storage spaces of [[#The Node]]s.
  
===Channels and networks===
+
===Storage provisions===
: The ''Farm's'' communication channels are built on the [[#Bare-metal servers]] and [[#The Bridges]]. Currently, the ''Farm'' uses three communication channels, each of which serves one of the network as follows:
+
: Since [[#The Storage]] is installed on the top of [[#The Environment]], the ''Storage'' provisioning entails configuring a [[ProxmoxVE]] instance to work with a [[Educaship Ceph]] instance.
:# '''[[wide area network|WAN]]''' ([[wide area network]]), which is the ''Farm's'' public network that uses external, public [[IPv4 address]]es to integrate the [[#The Gateway]] into the [[Internet]]. The public network is described in the [[#LAN gateway]] section of this wikipage.
 
:# '''[[local area network|LAN]]''' ([[local area network]]), which is the ''Farm's'' private network that uses internal, private [[IPv6 address]]es to integrate [[#The Gateway]] and the ''Nodes'' into one network cluster. This network cluster is described in the [[#Virtual environment (VE)]] section of this wikipage.
 
:# '''[[storage area network|SAN]]''' ([[storage area network]]), which is the ''Farm's'' private network that uses internal, private [[IPv6 address]]es to integrate storage spaces of the ''Nodes'' into one storage cluster. This storage cluster is described in the [[#Storage platform]] section of this wikipage.
 
: The ''Farm's'' usage of [[IP address]]es is best described in the [[#IP addresses]] section.
 
  
===DNS zone===
+
: At [[#The Farm]], [[Educaship Ceph]] is deployed at all of [[#The Node]]s. Each of [[#The Node]]'s servers features doubled hard disks. Physically, a [[ProxmoxVE]] instance is installed on one disk of each of [[#The Node]]s; [[Educaship Ceph]] uses three "second" disks. So, [[#The Farm]] features three instances of [[ProxmoxVE]] and one instance of [[Ceph]].
: To locate the ''Farm's'' public resources in the [[Internet]], the following [[DNS record]]s are created in the ''Farm's'' [[DNS zone]]:
 
:{|class="wikitable"
 
!Field!!Type!!Data!!Comment (not a part of the records)
 
|-
 
|pm1.bskol.com||[[AAAA record]]||2a01:4f8:10a:439b::2||Node 1
 
|-
 
|pm2.bskol.com||[[AAAA record]]||2a01:4f8:10a:1791::2||Node 2
 
|-
 
|pm3.bskol.com||[[AAAA record]]||2a01:4f8:10b:cdb::2||Node 3
 
|-
 
|pf.bskol.com||[[A record]]||88.99.71.85||[[CNM pfSense]]
 
|-
 
|npm1.bskol.com||[[A record]]||88.99.218.172||Node 1 [[Nginx]]
 
|-
 
|npm2.bskol.com||[[A record]]||88.99.71.85||Node 2 [[Nginx]]
 
|-
 
|npm3.bskol.com||[[A record]]||94.130.8.161||Node 3 [[Nginx]]
 
|-
 
|talk.cnmcyber.com||[[A record]]||2a01:4f8:fff0:53::2||[[CNM Talk]] ([[CNM Jitsi]])
 
|-
 
|venture.cnmcyber.com||[[A record]]||2a01:4f8:fff0:53::3||[[CNM Venture]] ([[CNM Odoo]])
 
|-
 
|social.cnmcyber.com||[[A record]]||2a01:4f8:fff0:53::4||[[CNM Social]] ([[CNM HumHub]])
 
|}
 
  
===IP addresses===
+
: While experimenting with [[OpenZFS]] and [[RAID]], [[#The Team]] has also tried another model. The second disks then served as reserve copies of the first ones. Since every disk is just 512 GB, that model shrank [[#The Farm]]'s capacity in a half since both [[#The User Apps]] and their storage needed to fit the 512 GB limitation together.  
: To locate its resources in the [[#Communication channels]], the ''Farm'' uses three types of [[IP address]]es:
 
:# To access [[#Virtual environment (VE)]] of various ''Nodes'' from the outside world, the ''Farm'' features public [[IPv6 address]]es. One address is assigned to each ''Node''. Since there are three ''Nodes'', three addresses of that type are created.
 
:# For an internal network of three ''Nodes'', which is assembled on the internal ''Bridge'', a private [[IP address]] is used. This network is not accessible from the Internet and not included in the ''Farm's'' [[DNS zone]]. For instance, the [[#Storage platform]] utilizes this network to synchronize its data. For this network, an address with the type "/24" is selected.
 
:# For an external network of three ''Nodes'', which is assembled on the external ''Bridge'', the ''Farm'' features public [[IPv4 address]]es. They are handled by [[#Web intermediaries]].
 
  
==The Gateway==
+
: In the current model, [[#The User Apps]] shouldn't share their 512 GB with the storage. On another hand, [[#The Farm]]'s [[Educaship Ceph]] capacity is about 3 * 512 GB = 1.536 GB.
For the purposes of this wikipage, "gateway"  (hereinafter, the ''Gateway'') refers to the composition of software that is built on the external ''Bridge''. The ''Gateway'' is the hub for both ''Farm's'' [[wide area network]] ([[wide area network|WAN]]) and [[local area network]] ([[local area network|LAN]]). To power the ''Gateway'', [[CNM pfSense]] is deployed.
 
  
===COTS for Gateway===
+
===UI of the Storage===
: [[CNMCyber Team]] has tried [[iptables]] as a [[firewall]] and [[Fail2ban]], which operates by monitoring log files (e.g. /var/log/auth.log, /var/log/apache/access.log, etc.) for selected entries and running scripts based on them. Most commonly this is used to block selected IP addresses that may belong to hosts that are trying to breach the system's security. It can ban any host IP address that makes too many login attempts or performs any other unwanted action within a time frame defined by the administrator. Includes support for both [[IPv4]] and [[IPv6]].
+
: With regards to [[#User interfaces (UIs)]], [[#The Storage]] features its administrative interface, which belongs to [[#Dashboards]].
  
===Functions===
+
==The Gateway==
: The ''Gateway'' can be compared to an executive secretary, who (a) takes external client's requests, (b) serves as a [[gatekeeper]], while checking validity of those requests, (c) when the request is valid, selects to which internal resource to dispatch it, (d) dispatches those requests to the selected resource, (e) gets internal responses, and (f) returns them back to the client in the outside world.
+
For the purposes of this very wikipage, the ''Gateway'' refers to the composition of software that is built on the external ''Bridge''. The ''Gateway'' is the hub for both [[#The Farm]]'s [[wide area network]] ([[wide area network|WAN]]) and [[local area network]] ([[local area network|LAN]]). To power the ''Gateway'', [[Educaship pfSense]] is deployed.
  
: Thus, the ''Gateway'' (a) receives requests from the world outside of the ''Farm'', (b) serves as a [[firewall]], while checking validity of those requests, (c) when the request is valid, selects to which ''Node'' to dispatch it, (d) dispatches those requests to the selected ''Node'', (e) gets internal responses, and (f) returns those responses to the outside world.
+
The composition of software such as a [[load balancer]] or [[reverse proxy]] that is built on the [[#External Bridge]].
  
: The ''Gateway'' is responsible for dispatching external requests to those and only to those internal resources that the ''Farm's'' [[#Monitoring]] has identified as operational. To be more accessible to its clients, the ''Gateway'' utilizes public [[IPv4 address]]es.
+
===COTS for the Gateway===
 +
: As [[#The COTS]] for the ''Gateway'', [[#The Team]] utilizes [[Educaship pfSense]]. For a while, [[#The Team]] has also tried [[iptables]] as a [[firewall]] and [[Fail2ban]], which operates by monitoring log files (e.g. /var/log/auth.log, /var/log/apache/access.log, etc.) for selected entries and running scripts based on them. Most commonly this is used to block selected IP addresses that may belong to hosts that are trying to breach the system's security. It can ban any host IP address that makes too many login attempts or performs any other unwanted action within a time frame defined by the administrator. Includes support for both [[IPv4]] and [[IPv6]].
  
==Gateway components==
+
===Gateway features===
The ''Gateway'' includes [[#Firewall and router]], [[#Load balancer]], and [[#Webserver]].
 
  
===Firewall and router===
+
===Gateway functions===
: [[CNM pfSense]] plays roles of [[firewall]], [[reverse proxy]], and platform to which [[#Load balancer]] and [[#Webserver]] are attached.
+
[[FreeBSD]], HA, VPN, LDAP, backups, CARP VIP
  
===Load balancer===
+
: [[#The Gateway]] can be compared to an executive secretary, who (a) takes external client's requests, (b) serves as a [[gatekeeper]], while checking validity of those requests, (c) when the request is valid, selects to which internal resource to dispatch it, (d) dispatches those requests to the selected resource, (e) gets internal responses, and (f) returns them back to the client in the outside world.
: As a [[load balancer]], [[CNM pfSense]] uses the select version of [[HAProxy]] that is specifically configured as [[HAProxy]]'s add-on. As of summer of 2023, no full [[HAProxy Manager]] exists in the ''Farm''. As of summer of 2023, a [[round robin]] model is activated for load balancing.
 
  
===Webserver===
+
: Thus, [[#The Gateway]]:
: As its [[webserver]], [[pfSense]] utilizes [[lighttpd]]. Prior to deployment of [[CNM pfSense]], [[CNMCyber Team]] utilized two [[web server]]s to communicate with the outside world via [[HTTP]]. [[Nginx]] handled requests initially and [[Apache HTTP Server]] handled those requests that hadn't handled by [[Nginx]].
+
:# (constantly) '''Is monitoring''' state of internal resources of [[#The Farm]].
 +
:# '''Receives''' requests from the world outside of [[#The Farm]].
 +
:# '''Checks''' validity of external requests, while serving as a [[firewall]].
 +
:# When the request is valid, '''selects''' to which of [[#The Node]]s to dispatch it. [[#The Gateway]] is responsible for dispatching external requests to those and only to those internal resources that [[#Cluster monitoring]] has identified as operational.
 +
:# '''Dispatches''' those requests to [[#The Node]] that was selected.
 +
:# '''Collects''' internal responses.
 +
:# '''Returns''' those responses to the outside world.
  
==Emergency tools==
+
: To be more accessible to its clients, [[#The Gateway]] utilizes public [[IPv4 address]]es.
===Monitoring===
 
: Monitoring features are to be identified. Previously, various candidates offered three options:
 
:# Stack -- [[prometheus]] + node-exporter + [[grafana]]
 
:# [[Prometheus]] to monitor VMs, Influx to monitor Pve nodes, [[Grafana]] for Dashbord
 
:# [[grafana]] + [[influxdb]] + [[telegraf]], as well as [[zabbix]]. To monitor websites, use [[uptimerobot]]
 
  
===Recovery===
+
===Gateway provisions===
 +
: [[#The Gateway]] is deployed in a [[virtual machine]] ([[virtual machine|VM]]) of [[#The Environment]].
  
==Accesses==
+
===UI of the Gateway===
===End-user access===
+
: With regards to [[#User interfaces (UIs)]], [[#The Gateway]] features its administrative interface, which belongs to [[#Dashboards]].
: The ''Patrons'' access the ''Apps'' and the ''Apps'' only. Those users cannot access to the [[#Bare-metal servers]], [[#Backup box]], [[#Virtual environment (VE)]], [[#The Gateway]], as well as [[#Emergency tools]].
 
  
: The ''Patrons'' access the ''Apps'' via those [[IPv4 address]]es that are associated with the particular ''App''. [[Opplet.net]] provides the ''Patrons'' with access automatically or, by bureaucrats or other [[power-user]]s, manually.
+
==Gateway components==
 +
[[#The Gateway]] includes [[#Firewall and router]], [[#Load balancer]], and [[#Web server]].
  
===Power-user access===
+
===Firewall and router===
: [[Power-user]]s of the ''Farm'' (hereinafter, the ''Admins'') are those users who have authorized to access more resources of the ''Farm'' than a regular ''Patron''.
+
: [[Educaship pfSense]] plays roles of [[firewall]], [[reverse proxy]], and platform to which [[#Load balancer]] and [[#Web server]] are attached.
:# '''Hardware-level admin'''. Administrative access to [[#Bare-metal servers]] and [[#Backup box]] is carried out without any [[IP address]]es, through the administrative panel and administrative consoles that [[#Service provider]] grants to [[CNMCyber Customer]]. The customer grants hardware-level admin access personally.
 
:# '''VE-level admin'''. Administrative access to [[#Virtual environment (VE)]], [[#The Gateway]], and [[#Emergency tools]] is carried out through [[IPv6 address]]es linked to those tools. Access credentials are classified and securely stored in [[CNM Lab]].
 
:# '''App-level admin'''. Administrative access to the ''Apps'' is carried out through the [[IPv4 address]]es associated with the particular ''App''. At the moment, those accesses are provided by other ''Admins'' manually.
 
  
==Infrastructure==
+
===Load balancer===
The infrastructure of the ''Farm'' consists of hardware, [[#Bare-metal servers]] and [[#Backup box]], as well as [[#The Bridges]] rented from the [[#Service provider]].
+
: As a [[load balancer]], [[Educaship pfSense]] uses the select version of [[HAProxy]] that is specifically configured as [[HAProxy]]'s add-on. As of summer of 2023, no full [[HAProxy Manager]] exists in [[#The Farm]]. As of summer of 2023, a [[round robin]] model is activated for load balancing.
  
===Service provider===
+
===Web server===
: [[Hetzner]] has been serving as [[CNMCyber Team]]'s [[Internet service provider]] ([[Internet service provider|ISP]]) and lessor of the [[#Infrastructure]] since 2016. Offers from other potential providers, specifically, [[Contabo]] and [[DigitalOcean]], have been periodically reviewed, but no one else has offered any better quality/price rate on a long-term basis.
+
: As its [[web server]], [[pfSense]] utilizes [[lighttpd]]. Prior to deployment of [[Educaship pfSense]], [[#The Team]] utilized two [[web server]]s to communicate with the outside world via [[HTTP]]. [[Nginx]] handled requests initially and [[Apache HTTP Server]] handled those requests that hadn't handled by [[Nginx]].
  
===Choosing bare-metal===
+
==Web architecture==
: Due to the lower cost, [[#Bare-metal servers]] were purchased via [[#Service provider]]'s auction -- https://www.hetzner.com/sb?hdd_from=500&hdd_to=1000 -- based on the following assumptions:
+
For the purposes of this wikipage, "web architecture" refers to [[#The Farm]]'s outline of [[DNS record]]s and [[IP address]]es.
:* '''Number''': [[ProxmoxVE]] normally requires three nodes. The third node is needed to provide quorum; however, it shall not necessarily run applications. At the same time, [[Ceph]] requires three nodes at least.
 
:* '''Hard drives''':
 
:*# The hard drive storage capacity for any ''Node'' shall be 512Gb at least.
 
:*# Because [[Ceph]] is selected to power the [[#Storage platform]], any hard-drive of the ''Farm'' shall be both [[SSD]] and [[NVMe]].
 
:* '''Processors''':
 
:*# The processor frequency for two ''Nodes'' of the ''Farm'' shall be 32Gb at least. Processor frequency requirements to the third ''Node'' may be lower because of [[ProxmoxVE]]'s characteristics.
 
:*# Those servers that deploys Intel Xeon E3-1275v5 processors are preferable over those servers that deploys Intel Core i7-7700 ones.
 
:* '''Location''': At least two ''Nodes'' shall be located in the same data center. Although the [[#Service provider]] does not charge for internal traffic, this circumstance increases the speed of the whole ''Farm''. If no nodes are available in the same data center, they shall be looked for in the same geographic location.
 
  
: The hardware characteristics of the chosen ''Nodes'' are presented in [[#Bare-metal servers]].
+
===Channels and networks===
 +
: [[#The Farm]]'s communication channels are built on [[#The Metal]] and [[#The Bridges]]. Currently, [[#The Cluster]] uses three communication channels, each of which serves one of the network as follows:
 +
:# '''[[wide area network|WAN]]''' ([[wide area network]]), which is [[#The Farm]]'s public network that uses external, public [[IPv4 address]]es to integrate the [[#The Gateway]] into the [[Internet]]. The public network is described in the [[#The Gateway]] section of this wikipage.
 +
:# '''[[local area network|LAN]]''' ([[local area network]]), which is [[#The Farm]]'s private network that uses internal, private [[IPv6 address]]es to integrate [[#The Gateway]] and [[#The Node]]s into one network cluster. This network cluster is described in [[#The Environment]] section of this very wikipage.
 +
:# '''[[storage area network|SAN]]''' ([[storage area network]]), which is [[#The Farm]]'s private network that uses internal, private [[IPv6 address]]es to integrate storage spaces of [[#The Node]]s into one storage cluster. This storage cluster is described in [[#The Storage]] section of this wikipage.
 +
: [[#The Farm]]'s usage of [[IP address]]es is best described in the [[#IP addresses]] section.
  
==The Bridges==
+
===DNS zone===
Сеть каждого ''Узла'' использует мост по выбираемой по умолчанию в [https://pve.proxmox.com/wiki/Network_Configuration#_default_configuration_using_a_bridge Network Configuration] модели.
+
: To locate [[#The Farm]]'s public resources in the [[Internet]], the following [[DNS record]]s are created in [[#The Farm]]'s [[DNS zone]]:
 +
:{|class="wikitable"
 +
!Field!!Type!!Data!!Comment (not a part of the records)||Review
 +
|-
 +
|pm1.bskol.com||[[AAAA record]]||2a01:4f8:10a:439b::2||Node 1||No data
 +
|-
 +
|pm2.bskol.com||[[AAAA record]]||2a01:4f8:10a:1791::2||Node 2||No data
 +
|-
 +
|pm?.bskol.com||[[AAAA record]]||&nbsp;||Node ?||No data
 +
|-
 +
|pf.bskol.com||[[A record]]||88.99.71.85||[[Educaship pfSense]]||Record is not operational
 +
|-
 +
|talk.cnmcyber.com||[[A record]]||188.34.147.106||[[CNM Talk]] ([[Educaship Jitsi]])||Passed
 +
|-
 +
|corp.cnmcyber.com||[[A record]]||188.34.147.106||[[CNM Corp]] ([[Educaship Odoo]])||Passed
 +
|-
 +
|social.cnmcyber.com||[[A record]]||188.34.147.106||[[CNMCyber.com]] ([[Educaship HumHub]])||Passed
 +
|-
 +
|portainer.cnmcyber.com||[[A record]]||188.34.147.107||Docker server, dockers are used for all monitoring||Passed
 +
|-
 +
|dash-status.cnmcyber.com||[[A record]]||188.34.147.107||Dashboard for monitoring status powered by [[Uptime Kuma]]||Passed
 +
|-
 +
|status.cnmcyber.com||[[A record]]||188.34.147.107||||Passed
 +
|-
 +
|influxdb.cnmcyber.com||[[A record]]||188.34.147.107||[[InfluxDB]]||Passed
 +
|-
 +
|monitor.cnmcyber.com||[[A record]]||188.34.147.107||[[Grafana]]||Passed
 +
|-
 +
|npm.cnmcyber.com||[[A record]]||188.34.147.107||[[Nginx Proxy Manager]]||Passed
 +
|-
 +
|pass.cnmcyber.com||[[A record]]||188.34.147.107||[[Passbolt]]||Passed
 +
|}
  
[[Hetzner vSwitch]]es (hereinafter, the ''Bridges'') serve as bridges for [[#Communication channels]] to connect the ''Nodes'' in networks and switch from one ''Node'' to another one. The [[#Service provider]] provides [[CNMCyber Team]] with the ''Bridges''; the team can order up to 5 connectors to be connected to one ''Node''.
+
==Web server files==
  
The ''Farm'' cannot support [[high availability]] of the ''Bridges''. Resiliency of the ''Bridges'' is the courtesy of their owner, [[#Service provider]].
+
==Legacy==
  
The ''Farm'' utilizes two ''Bridges'': They come with the lease of the ''Nodes''.
+
haproxy.bskol.com 86400 A 0 185.213.25.206
 +
influx.bskol.com 86400 A 0 49.12.5.41
 +
monitor.bskol.com 86400 A 0 49.12.5.41
 +
pbs.bskol.com 86400 A 0 88.99.214.92
 +
pf.bskol.com 86400 A 0 88.99.71.85
 +
pm1.bskol.com 86400 A 0 88.99.218.172
 +
pm2.bskol.com 86400 A 0 88.99.71.85
 +
pm3.bskol.com 86400 A 0 88.99.214.92
 +
zabbix.bskol.com 86400 A 0 167.235.255.244
 +
pbs.bskol.com 86400 AAAA 0 2a01:4f8:10a:3f60::2
 +
pf.bskol.com 86400 AAAA 0 2a01:4f8:fff0:53::6
  
===Internal Bridge===
+
===IP addresses===
: '''Internal''' ''Bridge'' serves as the hub for node and storage networks. It is located on an internal, private [[IPv6 address]] to provide for data transfer between the ''Nodes'' and their storage spaces.
+
: To locate its resources in the [[#Communication channels]], [[#The Farm]] uses three types of [[IP address]]es:
 +
:# To access [[#The Environment]] from the outside world, [[#The Farm]] features public [[IPv6 address]]es. One address is assigned to each of [[#The Node]]s. Since there are three of them, three addresses of that type are created.
 +
:# For an internal network of [[#The Node]]s, which is assembled on the [[#Internal Bridge]], a private [[IP address]] is used. This network is not accessible from the Internet and not included in [[#The Farm]]'s [[DNS zone]]. For instance, [[#The Storage]] utilizes this network to synchronize its data. For this network, an address with the type "/24" is selected.
 +
:# For an external network of three ''Nodes'', which is assembled on the [[#External Bridge]], [[#The Farm]] features public [[IPv4 address]]es. They are handled by [[#Web intermediaries]].
  
===External Bridge===
+
===SSL certificates===
: '''External''' ''Bridge'' serves as the hub for the public network, the [[Internet]]. It is located on external, public [[IPv4 address]] to provide for data transfer between the ''Farm's'' publicly-available and other Internet resources.
 
  
 
==Backup box==
 
==Backup box==
 
A backup box is deployed on a 1 TB, unlimited traffic storage box BX-11 that has been rented for that purpose.
 
A backup box is deployed on a 1 TB, unlimited traffic storage box BX-11 that has been rented for that purpose.
  
===Basic features===
+
===COTS for backups===
 +
: [[#The Team]] utilizes no additional software beyond [[ProxmoxVE]] for backups. Initially, [[Proxmox Backup Server]] was used. However, it consumed the active storage. As a result, the storage box was just attached to [[#The Environment]]. And backup to that storage goes directly from and to [[#The Environment]].
 +
 
 +
===Box features===
 
: 10 concurrent connections, 100 sub-accounts, 10 snapshots, 10 automated snapshots, FTP, FTPS, SFTP, SCP, Samba/CIFS, BorgBackup,    Restic, Rclone, rsync via SSH, HTTPS, WebDAV, Usable as network drive
 
: 10 concurrent connections, 100 sub-accounts, 10 snapshots, 10 automated snapshots, FTP, FTPS, SFTP, SCP, Samba/CIFS, BorgBackup,    Restic, Rclone, rsync via SSH, HTTPS, WebDAV, Usable as network drive
  
===COTS for backups===
+
===Box functions===
: Initially, [[Proxmox Backup Server]] was used. However, it consumed the active storage. As a result, no additional [[COTS]] was deployed beyond Hetzner tools. The storage box was attached it on the ''Cluster'' as a storage. And backup to that storage goes directly from the ''Cluster''.
+
: [[#The Provider]]'s description: Storage Boxes provide you with safe and convenient online storage for your data. Score a Storage Box from one of Hetzner Online's German or Finnish data centers! With Hetzner Online Storage Boxes, you can access your data on the go wherever you have internet access. Storage Boxes can be used like an additional storage drive that you can conveniently access from your home PC, your smartphone, or your tablet. Hetzner Online Storage Boxes are available with various standard protocols which all support a wide array of apps. We have an assortment of diverse packages, so you can choose the storage capacity that best fits your individual needs. And upgrading or downgrading your choice at any time is hassle-free!
 +
 
 +
===Box provisions===
 +
 
 +
===UI for backups===
 +
: [[#User interfaces (UIs)]]
  
===Description===
+
==Used terms==
: [[#Service provider]]'s description: Storage Boxes provide you with safe and convenient online storage for your data. Score a Storage Box from one of Hetzner Online's German or Finnish data centers! With Hetzner Online Storage Boxes, you can access your data on the go wherever you have internet access. Storage Boxes can be used like an additional storage drive that you can conveniently access from your home PC, your smartphone, or your tablet. Hetzner Online Storage Boxes are available with various standard protocols which all support a wide array of apps. We have an assortment of diverse packages, so you can choose the storage capacity that best fits your individual needs. And upgrading or downgrading your choice at any time is hassle-free!
+
On this very wikipage, a few abbreviations and terms are commonly used.
 +
:* ''Bridge''. Any of two [[Hetzner vSwitch]]es that [[#The Farm]] utilizes.
 +
:* ''UI''. A [[user interface]], which is the ''COTS'' feature that allows a ''COTS'' instance and its users to interact.
  
==Bare-metal servers==
+
===The COTS===
The [[#Virtual environment (VE)]] are deployed on three [[bare-metal server]]s. As the result of [[#Choice of bare-metal]], [[#Node 1 hardware]], [[#Node 2 hardware]], and [[#Node 3 hardware]] have been rented for that purpose.
+
: [[Commercial off-the-shelf]] ([[commercial off-the-shelf|COTS]]) software.
  
===Node 1 hardware===
+
===The Farm===
:1 x Dedicated Root Server "Server Auction"
+
CNM Bureau Farm, this very wikipage describes it.
:* Intel Xeon E3-1275v5
 
:* 2x SSD M.2 NVMe 512 GB
 
:* 4x RAM 16384 MB DDR4 ECC
 
:* NIC 1 Gbit Intel I219-LM
 
:* Location: FSN1-DC1
 
:* Rescue system (English)
 
:* 1 x Primary IPv4
 
  
===Node 2 hardware===
+
===The Node===
:1 x Dedicated Root Server "Server Auction"
+
One hardware, [[bare-metal server]] of [[#The Infrastructure]] with all of software installed on the top of it.
:* Intel Xeon E3-1275v5
 
:* 2x SSD M.2 NVMe 512 GB
 
:* 4x RAM 16384 MB DDR4 ECC
 
:* NIC 1 Gbit Intel I219-LM
 
:* Location: FSN1-DC1
 
:* Rescue system (English)
 
:* 1 x Primary IPv4
 
  
===Node 3 hardware===
+
===The Team===
:1 x Dedicated Root Server "Server Auction"
+
[[CNMCyber Team]].
:* Intel Xeon E3-1275v5
 
:* 2x SSD M.2 NVMe 512 GB
 
:* 4x RAM 16384 MB DDR4 ECC
 
:* NIC 1 Gbit Intel I219-LM
 
:* Location: FSN1-DC1
 
:* Rescue system (English)
 
  
 
==See also==
 
==See also==
Line 241: Line 330:
 
===Related lectures===
 
===Related lectures===
 
:* [[What CNM Farms Are]].
 
:* [[What CNM Farms Are]].
 
===Special terms===
 
:On this wiki page, the following terms are used:
 
:* ''Admin''. A [[power-user]] of the ''Farm'' or any user who has authorized to access more resources of the ''Farm'' than a regular ''Patron''.
 
:* ''App''. Any of three [[end-user application]]s with which [[end-user]]s of the ''Farm'' interact.
 
:* ''Bridge''. Any of two [[Hetzner vSwitch]]e that the ''Farm'' utilizes.
 
:* ''Cluster''.
 
:* ''Farm''. CNM Bureau Farm, this very wikipage describes it.
 
:* ''Gateway''. The composition of software such as a [[load balancer]] or [[reverse proxy]] that is built on the [[#External Bridge]].
 
:* ''Node''. One ''Farm's'' hardware server with all of software installed on the top of it.
 
:* ''Patron''. An [[end-user]] of the ''Farm''.
 
  
 
===Useful recommendations===
 
===Useful recommendations===
Line 258: Line 336:
 
:* https://pve.proxmox.com/wiki/High_Availability (general [[ProxmoxVE]] [[HA]] functionality)
 
:* https://pve.proxmox.com/wiki/High_Availability (general [[ProxmoxVE]] [[HA]] functionality)
 
:* https://docs.hetzner.com/robot/dedicated-server/network/vswitch/ (general [[Hetzner vSwitch]] functionality)
 
:* https://docs.hetzner.com/robot/dedicated-server/network/vswitch/ (general [[Hetzner vSwitch]] functionality)
 +
 +
Jitsi/Odoo/HumHub, DNS, [[Educaship pfSense]], monitoring -- [[Telegraf]] + [[InfluxDB]] + [[Grafana]], [[Uptime Kuma]], [[Passbolt]], mail server, LDAP, DNS zone, IPv4, what else to do? HA test, alpha testing, DNS
  
 
[[Category:CNM Cloud products]][[Category: CNM Cyber Orientation]][[Category: Articles]]
 
[[Category:CNM Cloud products]][[Category: CNM Cyber Orientation]][[Category: Articles]]

Latest revision as of 15:57, 14 April 2024

CNM Bureau Farm (formerly known as CNM EndUser Farm; hereinafter, #The Farm) is the segment of Opplet that is responsible for providing users with CNM Corp and CNMCyber.com. Most likely, #The Farm will also handle the applications that CNM Fed Farm currently provides.

CNMCyber Team (hereinafter, #The Team) develops and administers #The Farm. To sustain #The Farm online, #The Team rents Bureau Infrastructure.


In the nutshell

For the purposes of this very wikipage, end-users are called #The End-Users. Collectively, end-user applications are called #The User Apps.

Architecture

While using #The Farm, #The End-Users work with #The User Apps that are installed in #The Cluster that is, consequently, hosted by #The Infrastructure. #The Cluster consists of #The Storage, #The Environment, #The Gateway. #The Infrastructure includes #The Bridges and #The Metal, which is #The Farm's hardware.

Cluster-based

To mitigate a single point of failure (SPOF), #The Farm is built on not just one, but three bare-metal servers. Each of those hardware servers with all of software installed on the top of it (hereinafter, #The Node) is self-sufficient to host #The User Apps. Various solutions such as #The uptime tools orchestrate coordination between #The Nodes.

COTS-powered

#The Farm's software is a collection of commercial off-the-shelf (COTS) packages (hereinafter, #The COTS). In plain English, #The COTS is the software that is already available on the market. No single line of programming code is written specifically for #The Farm. Only time-tested market-proven solutions have been used. #The User Apps use HumHub, Jitsi, and Odoo instances. #The Cluster uses Ceph, Proxmox, and pfSense.

Addressing the needs

Development and sustenance of #The Farm address two business needs of CNMCyber Team (hereinafter, #The Team). For #The Team, #The Farm shall serve as both #Tech side of the Apps and #Worksite.

Tech side of the Apps

#The Team needs to provide #The End-Users with the services of #The User Apps 24 hours a day, 7 days a week. #The Farm shall power #The User Apps technologically; their content is outside of #The Farm's scope.

Worksite

#The Team needs to provide those incumbents of CNMCyber practices who work with #The Farm's technologies with their worksite.

Farm users

For the purposes of this very wikipage, a farm user refers to any user of #The Farm. The farm user's authentication and authorization administration is a part of identity and access management (IAM).

The End-Users

For the purposes of this very wikipage, the Patron refers to an end-user of #The User Apps. They access #The User Apps via the graphic user interfaces (GUIs) that are associated with the particular application. At #The Farm, those #User interfaces (UIs) are located at IPv4 addresses. Opplet.net provides the Patrons with access automatically or, by #The Power-Users, manually. The Patrons can access #The User Apps and #The User Apps without administrative tools only. The Patrons may or may not be members of #The Team.

The Power-Users

For the purposes of this very wikipage, the Power-User refers to a power-user of #The User Apps. By definition, the Power-Users are those members of #The Team who have authorized to access more of #The User Apps' resources than #The End-Users. Normally, those resources are administrative tools that allow the Power-Users administer one or more of #The User Apps.
While having administrative access, the Power-Users serve as application-level administrators. They access #The User Apps via the same graphic user interfaces (GUIs) as #The End-Users, but administrative-level UIs display administration tools in addition to those tools that are available to non-privileged end-users.
At the moment, #The Sysadmins can grant the Power-User rights to one or more of #The End-Users manually. Administrative access credentials are classified and securely stored in the application project spaces of CNM Lab. Those incumbents of CNMCyber practice who work with #The Farm's technologies may or may not be the Power-Users.

The Sysadmins

For the purposes of this very wikipage, the Sysadmin refers to a system administrator of #The Cluster. By definition, the Sysadmins are those members of #The Team who have authorized to access at least some of #The Cluster's resources:
#The Superusers can grant the Sysadmin rights to one or more of #The End-Users manually. Administrative access credentials are classified and securely stored in the cluster project spaces of CNM Lab. No incumbent of CNMCyber practice can be the Sysadmin; only those who work in CNMCyber Office can.

The Superusers

For the purposes of this very wikipage, the Superuser refers to a superuser of #The Farm. By definition, the Superusers are those members of #The Team who have authorized to access any part of #The Farm starting with its hardware. #The Metal's access is carried via #UI for the Metal.
#The Provider grants the Superuser rights to CNMCyber Customer, who can manually pass it to one or more members of CNMCyber Office. The Superuser credentials are not stored in CNM Lab.

User interfaces (UIs)

For the purposes of this very wikipage, a user interface (UI) refers to #The COTS' feature that allows #The COTS' instance and its users to interact. UIs of #The User Apps are described in the wikipages that are dedicated to particular applications. The other UIs of #The Farm are described in the #UI of the Environment, #UI of the Storage, #UI of the Gateway, and #UI for the Metal sections of this very wikipage.

Dashboards

For the purposes of this very wikipage, a Dashboard refers to a graphic user interface (GUI) that either belongs to any #The COTS package installed in #The Farm or is provider by #The Provider. This screen-based interface allows #Farm users to interact with #The User Apps and other software through graphical buttons, icons, or hyperlinked texts rather than typed commands in command line interfaces (CLIs).

Third-party UIs

For the purposes of this very wikipage, a Third-party UI refers to any user interface (UI) that neither belongs to any #The COTS package installed in #The Farm nor is provider by #The Provider. #The Sysadmins and #The Suparusers may use access tools such as PuTTy and Midnight Commander to access #UI for the Metal.

The User Apps

For the purposes of this very wikipage, the User Apps refer to those end-user applications with which #The End-Users interact.

HumHub

CNMCyber.com, which is the end-user instance of Educaship HumHub.

Jitsi

CNM Talk, which is the end-user instance of Educaship Jitsi.

Odoo

CNM Corp, which is the end-user instance of Educaship Odoo.

The Cluster

For the purposes of this very wikipage, the Cluster refers to all of the software between #The User Apps and #The Infrastructure.

Cluster components

#The Cluster consists of #The Nodes and their management tools. The following components compose #The Cluster:
  1. #The Storage that is powered by Educaship Ceph to provide #The Environment with stored objects, blocks, and files. Three storage spaces of #The Nodes create one distributed storage foundation.
  2. #The Environment that is powered by Educaship Proxmox to make containers and virtual machines (VMs) available to #The User Apps, so #The User Apps can function properly. Each of #The Nodes features its own environment; #The uptime tools orchestrate them all.
  3. #The Gateway that is powered by Educaship pfSense to create a gateway between #The Environment and the outside world. There is only one gateway; if it fails, #The Farm fails.

Choice of COTS

While building #The Farm generally and #Cluster components specifically, #The Team utilized only #The COTS that is both open-source and free of charge. Other considerations for the choice are stated in the #COTS for the Environment, #COTS for the Storage, #COTS for the Gateway, #COTS for backups sections of this very wikipage.

Cluster provisioning

Provisioning of #The Cluster occurs in the following sequence:
  1. Since #The Environment is installed on the top of #The Infrastructure, the #Environment provisions shall be accommodated first.
  2. Since #The Storage is a part of #The Environment, #Storage provisions shall be accommodated second.
  3. Since #The Gateway is installed in #The Environment, #Gateway provisions shall be accommodated third.

Cluster monitoring

Monitoring features are to be identified. Previously, various candidates offered three options:
  1. Stack -- prometheus + node-exporter + grafana
  2. Prometheus to monitor VMs, Influx to monitor Pve nodes, Grafana for Dashbord
  3. grafana + influxdb + telegraf, as well as zabbix. To monitor websites, use uptimerobot

Observability vs. APM vs. Monitoring

application performance management

Cluster recovery

High availability (HA)

Generally speaking, high availability (HA) of any system assumes its higher uptime in comparison with a similar system without higher uptime ability. HA of #The Farm assumes its higher uptime in comparison with a similar farm built on one of #The Nodes. Before #The uptime tools were deployed, #The Farm functioned only on one of #The Nodes and, when it failed, services of #The User Apps were no longer available until the failure was fixed. Now, until at least one of #The Nodes is operational, #The Farm is operational.

The uptime tools

Both #The Environment and #The Storage feature advanced tools for #High availability (HA).

Uptime limitations

Generally speaking, HA comes with significant costs. So does #The uptime tools. At very least, running three of #The Nodes is more expensive than running one. The cost cannot exceed the benefit, so high availability (HA) cannot be equal to failure tolerance.

Uptime management

To manage redundant resources, #The uptime tools:
  • Monitor its resources to identify whether they are operational or failed as described in the #Monitoring section of this very wikipage.
  • Fence those resources that are identified as failed. As a result, non-operational resources are withdrawn from the list of available.
  • Restore those resources that are fenced. The #Recovery supports that feature, while constantly creating snapshots and reserve copies of #The Farm and its parts in order to make them available for restoring when needed.

Uptime principles

Principally, #High availability (HA) of #The Cluster is based on:

The Environment

For the purposes of this very wikipage, the Environment refers to the virtual environment (VE) of #The Cluster or, allegorically, to the environment where #The User Apps "live".

COTS for the Environment

As #The COTS for the Environment, #The Team utilizes Educaship Proxmox. For a while, #The Team has also tried OpenStack and VirtualBox as its virtualization tools. The trials suggested that OpenStack required more hardware resources and VirtualBox didn't allow for required sophistication in comparison with ProxmoxVE, which has been chosen as #The COTS for #The Farm's virtualization.

Environment features

#The Team uses virtualization to divide hardware resources of #The Node's bare-metal servers in smaller containers and virtual machines (VMs), which are created in the Environment to run #The User Apps, #The Gateway, and other applications. In #The Farm, the ProxmoxVE instance tightly integrates KVM hypervisor, LXC containers, Educaship Ceph as software-defined storage, as well as networking functionality on a single virtualization platform.

Environment functions

#The Environment executes four major functions. It:
  1. Runs #The User Apps, #The Gateway, and other applications. They can be deployed utilizing two models:
    1. Using containers; they already contain operating systems tailored specifically to the needs of the App.
    2. In virtual machines (VM) and without containers. In that model, the App is installed on the operating system of its VM.
  2. Hosts #The Storage and #Backup box
  3. Connects the applications it runs and the storages it hosts to each other and to #The Bridges, while creating networks.
  4. Creates backups and accommodates its own recovery when requested.

Environment provisions

Every instance of ProxmoxVE requires one "physical" bare-metal server. The interaction between ProxmoxVE instances and #The Infrastructure is carried out by Debian operating system (OS) that comes in the same "box" of #The COTS as ProxmoxVE and is specifically configured for that interaction. #The Farm's ProxmoxVE also hosts #The Storage as its storage.

UI of the Environment

With regards to #User interfaces (UIs), #The Environment features its administrative interface, which belongs to #Dashboards.

The Storage

For the purposes of this very wikipage, the Storage refers to the storage platform or database management system (DBMS) that provides #The User Apps with the storage they need to operate. Thus, the Storage supports #The Environment's non-emergency operations and differs from the #Backup box that comes into play in emergencies.

COTS for the Storage

As #The COTS for #The Storage, #The Team utilizes Educaship Ceph. Any ProxmoxVE instance requires some storage to operate.
Before deploying #The uptime tools, #The Team used RAID to make the double hard disks redundant. So, the ProxmoxVE instance was just installed on the top of one disk and replicated to the other disk automatically. Flexibly, ProxmoxVE allows for better usage of hard disks. ProxmoxVE can be configured to host as #The COTS, many storage-type software packages such as ZFS, NFS, GlusterFS, and so on.
Initially, the cluster developer proposed using Ceph. Later, #The Team substituted one node with another with higher hard disk, but without SSD and NVMe; as a result, #The Farm's storage collapsed. The substituted node was disconnected (today, it serves as hardware for CNM Lab Farm), a new bare-metal server was purchased (today, it is the #Node 3 hardware) and Ceph restored.
As #The COTS, ProxmoxVE comes with OpenZFS. #The Team has deployed the combination of both in its CNM Lab Farm.

Storage features

#The Storage features are:
  • File system
  • Distributed
  • Fault tolerant
  • Object storage
  • Block device

Storage functions

To make objects, blocks, and files immediately available for #The User Apps' operations, #The Cluster uses a common distributed cluster foundation that orchestrates storage spaces of #The Nodes.

Storage provisions

Since #The Storage is installed on the top of #The Environment, the Storage provisioning entails configuring a ProxmoxVE instance to work with a Educaship Ceph instance.
At #The Farm, Educaship Ceph is deployed at all of #The Nodes. Each of #The Node's servers features doubled hard disks. Physically, a ProxmoxVE instance is installed on one disk of each of #The Nodes; Educaship Ceph uses three "second" disks. So, #The Farm features three instances of ProxmoxVE and one instance of Ceph.
While experimenting with OpenZFS and RAID, #The Team has also tried another model. The second disks then served as reserve copies of the first ones. Since every disk is just 512 GB, that model shrank #The Farm's capacity in a half since both #The User Apps and their storage needed to fit the 512 GB limitation together.
In the current model, #The User Apps shouldn't share their 512 GB with the storage. On another hand, #The Farm's Educaship Ceph capacity is about 3 * 512 GB = 1.536 GB.

UI of the Storage

With regards to #User interfaces (UIs), #The Storage features its administrative interface, which belongs to #Dashboards.

The Gateway

For the purposes of this very wikipage, the Gateway refers to the composition of software that is built on the external Bridge. The Gateway is the hub for both #The Farm's wide area network (WAN) and local area network (LAN). To power the Gateway, Educaship pfSense is deployed.

The composition of software such as a load balancer or reverse proxy that is built on the #External Bridge.

COTS for the Gateway

As #The COTS for the Gateway, #The Team utilizes Educaship pfSense. For a while, #The Team has also tried iptables as a firewall and Fail2ban, which operates by monitoring log files (e.g. /var/log/auth.log, /var/log/apache/access.log, etc.) for selected entries and running scripts based on them. Most commonly this is used to block selected IP addresses that may belong to hosts that are trying to breach the system's security. It can ban any host IP address that makes too many login attempts or performs any other unwanted action within a time frame defined by the administrator. Includes support for both IPv4 and IPv6.

Gateway features

Gateway functions

FreeBSD, HA, VPN, LDAP, backups, CARP VIP

#The Gateway can be compared to an executive secretary, who (a) takes external client's requests, (b) serves as a gatekeeper, while checking validity of those requests, (c) when the request is valid, selects to which internal resource to dispatch it, (d) dispatches those requests to the selected resource, (e) gets internal responses, and (f) returns them back to the client in the outside world.
Thus, #The Gateway:
  1. (constantly) Is monitoring state of internal resources of #The Farm.
  2. Receives requests from the world outside of #The Farm.
  3. Checks validity of external requests, while serving as a firewall.
  4. When the request is valid, selects to which of #The Nodes to dispatch it. #The Gateway is responsible for dispatching external requests to those and only to those internal resources that #Cluster monitoring has identified as operational.
  5. Dispatches those requests to #The Node that was selected.
  6. Collects internal responses.
  7. Returns those responses to the outside world.
To be more accessible to its clients, #The Gateway utilizes public IPv4 addresses.

Gateway provisions

#The Gateway is deployed in a virtual machine (VM) of #The Environment.

UI of the Gateway

With regards to #User interfaces (UIs), #The Gateway features its administrative interface, which belongs to #Dashboards.

Gateway components

#The Gateway includes #Firewall and router, #Load balancer, and #Web server.

Firewall and router

Educaship pfSense plays roles of firewall, reverse proxy, and platform to which #Load balancer and #Web server are attached.

Load balancer

As a load balancer, Educaship pfSense uses the select version of HAProxy that is specifically configured as HAProxy's add-on. As of summer of 2023, no full HAProxy Manager exists in #The Farm. As of summer of 2023, a round robin model is activated for load balancing.

Web server

As its web server, pfSense utilizes lighttpd. Prior to deployment of Educaship pfSense, #The Team utilized two web servers to communicate with the outside world via HTTP. Nginx handled requests initially and Apache HTTP Server handled those requests that hadn't handled by Nginx.

Web architecture

For the purposes of this wikipage, "web architecture" refers to #The Farm's outline of DNS records and IP addresses.

Channels and networks

#The Farm's communication channels are built on #The Metal and #The Bridges. Currently, #The Cluster uses three communication channels, each of which serves one of the network as follows:
  1. WAN (wide area network), which is #The Farm's public network that uses external, public IPv4 addresses to integrate the #The Gateway into the Internet. The public network is described in the #The Gateway section of this wikipage.
  2. LAN (local area network), which is #The Farm's private network that uses internal, private IPv6 addresses to integrate #The Gateway and #The Nodes into one network cluster. This network cluster is described in #The Environment section of this very wikipage.
  3. SAN (storage area network), which is #The Farm's private network that uses internal, private IPv6 addresses to integrate storage spaces of #The Nodes into one storage cluster. This storage cluster is described in #The Storage section of this wikipage.
#The Farm's usage of IP addresses is best described in the #IP addresses section.

DNS zone

To locate #The Farm's public resources in the Internet, the following DNS records are created in #The Farm's DNS zone:
Field Type Data Comment (not a part of the records) Review
pm1.bskol.com AAAA record 2a01:4f8:10a:439b::2 Node 1 No data
pm2.bskol.com AAAA record 2a01:4f8:10a:1791::2 Node 2 No data
pm?.bskol.com AAAA record   Node ? No data
pf.bskol.com A record 88.99.71.85 Educaship pfSense Record is not operational
talk.cnmcyber.com A record 188.34.147.106 CNM Talk (Educaship Jitsi) Passed
corp.cnmcyber.com A record 188.34.147.106 CNM Corp (Educaship Odoo) Passed
social.cnmcyber.com A record 188.34.147.106 CNMCyber.com (Educaship HumHub) Passed
portainer.cnmcyber.com A record 188.34.147.107 Docker server, dockers are used for all monitoring Passed
dash-status.cnmcyber.com A record 188.34.147.107 Dashboard for monitoring status powered by Uptime Kuma Passed
status.cnmcyber.com A record 188.34.147.107 Passed
influxdb.cnmcyber.com A record 188.34.147.107 InfluxDB Passed
monitor.cnmcyber.com A record 188.34.147.107 Grafana Passed
npm.cnmcyber.com A record 188.34.147.107 Nginx Proxy Manager Passed
pass.cnmcyber.com A record 188.34.147.107 Passbolt Passed

Web server files

Legacy

haproxy.bskol.com 86400 A 0 185.213.25.206 influx.bskol.com 86400 A 0 49.12.5.41 monitor.bskol.com 86400 A 0 49.12.5.41 pbs.bskol.com 86400 A 0 88.99.214.92 pf.bskol.com 86400 A 0 88.99.71.85 pm1.bskol.com 86400 A 0 88.99.218.172 pm2.bskol.com 86400 A 0 88.99.71.85 pm3.bskol.com 86400 A 0 88.99.214.92 zabbix.bskol.com 86400 A 0 167.235.255.244 pbs.bskol.com 86400 AAAA 0 2a01:4f8:10a:3f60::2 pf.bskol.com 86400 AAAA 0 2a01:4f8:fff0:53::6

IP addresses

To locate its resources in the #Communication channels, #The Farm uses three types of IP addresses:
  1. To access #The Environment from the outside world, #The Farm features public IPv6 addresses. One address is assigned to each of #The Nodes. Since there are three of them, three addresses of that type are created.
  2. For an internal network of #The Nodes, which is assembled on the #Internal Bridge, a private IP address is used. This network is not accessible from the Internet and not included in #The Farm's DNS zone. For instance, #The Storage utilizes this network to synchronize its data. For this network, an address with the type "/24" is selected.
  3. For an external network of three Nodes, which is assembled on the #External Bridge, #The Farm features public IPv4 addresses. They are handled by #Web intermediaries.

SSL certificates

Backup box

A backup box is deployed on a 1 TB, unlimited traffic storage box BX-11 that has been rented for that purpose.

COTS for backups

#The Team utilizes no additional software beyond ProxmoxVE for backups. Initially, Proxmox Backup Server was used. However, it consumed the active storage. As a result, the storage box was just attached to #The Environment. And backup to that storage goes directly from and to #The Environment.

Box features

10 concurrent connections, 100 sub-accounts, 10 snapshots, 10 automated snapshots, FTP, FTPS, SFTP, SCP, Samba/CIFS, BorgBackup, Restic, Rclone, rsync via SSH, HTTPS, WebDAV, Usable as network drive

Box functions

#The Provider's description: Storage Boxes provide you with safe and convenient online storage for your data. Score a Storage Box from one of Hetzner Online's German or Finnish data centers! With Hetzner Online Storage Boxes, you can access your data on the go wherever you have internet access. Storage Boxes can be used like an additional storage drive that you can conveniently access from your home PC, your smartphone, or your tablet. Hetzner Online Storage Boxes are available with various standard protocols which all support a wide array of apps. We have an assortment of diverse packages, so you can choose the storage capacity that best fits your individual needs. And upgrading or downgrading your choice at any time is hassle-free!

Box provisions

UI for backups

#User interfaces (UIs)

Used terms

On this very wikipage, a few abbreviations and terms are commonly used.

The COTS

Commercial off-the-shelf (COTS) software.

The Farm

CNM Bureau Farm, this very wikipage describes it.

The Node

One hardware, bare-metal server of #The Infrastructure with all of software installed on the top of it.

The Team

CNMCyber Team.

See also

Related lectures

Useful recommendations

Jitsi/Odoo/HumHub, DNS, Educaship pfSense, monitoring -- Telegraf + InfluxDB + Grafana, Uptime Kuma, Passbolt, mail server, LDAP, DNS zone, IPv4, what else to do? HA test, alpha testing, DNS