Обсуждение:Кампусна Ферма

Basing on Delova Farm

Since you have already had an HA cluster, theoretically you have two main options.

1. First you can add your storage capacity by adding all your proxmox nodes new harddisk/ssd/nvme, then you can add them to your current ceph storage system.
2. And other option, you can attach iscsi or Nas storage to your proxmox cluster.

For that, you should upgrade your server hardware on your contabo server. You can use isci, NAS, or upgrade ssd on each nodes.

Предложения относительно реализации

I.I.

For deployment, my recommendation is to go with docker.

3 VPS would be enough for this project.

The first example is on all VPS. (example from the picture)

• MariaDB With Galera Plugin
• Maxscale
• Moodle
• MediaWiki
• Nginx Proxy (for cloudflare certificates and custom ports.)

Another example,Two small VPS for MariaDB and one nano node for MariaDB Arbiter

Two VPS for Maxscale, Moodle, MediaWiki, and Nginx Proxy.

A total of 5 VPS.

For MariaDB HA - We will use the Galera plugin and Maxscale for routing.

For Real-Time Sync data, we would use lsyncd as needed, for some images or similar.

They would use CloudFlare to route to Nginx proxy and free certificates.

B.T.R.N.

1. How do you see the implementation of this task? (Methods and applications for implementation)

I will go with Centos 7 or Centos Stream 9. The first node will be setup with all the required softwares (MediaWiki, Moodle and MariaDB). HA will be setup using Corosync and Pacemaker packages. Finally, testing and handover.

2. Each node has two IP addresses: IPv4 and IPv6. Do we need to buy additional IP addresses?

This should be sufficient. At max, we might need a floating IP address which will be used to access the hosted services (MediaWiki and Moodle).

3. How will cluster monitoring be provided?

Clusters will be monitoring using a gui interface. I will also provide set of commands which can be used to query the cluster incase gui is not accessible.

4. How much will we spend for the cluster? (with all the additional costs)

We might need a shared storage like SAN. I will confirm if this is really necessary. Other than this, I do not see any more additional costs.

5. How much time do you need to implement the cluster with report documentation?

I expect the entire setup ü documentation to take somewhere between 50 to 60 hours.

6. "The first node will be setup with all the required softwares (MediaWiki, Moodle and MariaDB)." -- What will be setup on other nodes?

For HA, there are two things to take care. 1) Data and 2) Services. If we use NAS then data replication we dont need to worry. So other nodes will have similar setup but with their services down and bring them up in case of fail over switch. This is taken care by Pacemaker. In summary, other nodes will have almost similar setup but will slightly different setup and its services down.

Another approach is to setup load balancer using HAProxy and Gluster to replicate the shared volumes. You will have a easy to use web interface to manage the cluster.

M.B.Y.

Q1: For mediawiki and moodle we can use HAproxy

Avideo depend on how many users uses your app.

https://github.com/WWBN/AVideo/wiki/AVideo-Platform-hardware-requirements

Q2 : no

Q3 : I am using nagios for services and networks availability, alertra.com for layer7 services, grafana for logs goaccess.io a real-time web log analyzer.

Q4 : What you pay now + Alertra supervision costs.

Q5 : We need to agree about all tasks first.

I think it need a week fulltime.

"database (not very clear)" -- synchronized on three nodes (MariaDB Galera), this decision is not final, we will consider your suggestions. Can you give more details with HAProxy? (Which node will it be installed on? What happens if that node goes down? How many nodes are needed for HA? What solutions do you offer for databases?)

For HAProxy failover : ha-diagram-animated.gif. For MariaDB, Galera is widely used, and also we can configure HAProxy for load balancing.

"For HAProxy failover : ha-diagram-animated.gif" -- Do you need a floating IP for this? Where will HAProxy be installed? On one of the nodes? If the node with HAProxy goes down?

For a well desined architecture, we need, at least 2 HAProxy nodes. If a HAProxy node goes down, the second one will be activated ( floating IP role ) HAProxy need a separate node for security and scalabity reasons with big app. But we can use it in single node (with or without failover) and with other app (for test or smale app) depends on users number and SLA agreement. HAProxy is very reliable.

K.S.

1.For this project, I suggest using the InnoDB cluster with ProxySql as a middleware.

2.Also, we can use CheckMK or PMM for health monitoring.

3.It doesn't require additional IP, but it's recommended that we'll use a private network for the cluster.

4.All components will be open-source software. It'll be zero cost. But it would help if you have a minimum of three servers for the database node ( one primary and two secondary servers). ProxySQL can be installed on your application server.

5.In idol conditions, It'll take 6-8 hours for everything, including documentation.

Why do you suggest using InnoDB? Our applications use MariaDB.

MariaDB Galera cluster is also a good choice, but the MySQL InnoDB cluster is much more scalable. Also, the MySQL InnoDB and MariaDB Galera clusters are interchangeable for most use cases( 95%). If you want, I can use the MariaDB Galera cluster. It'll give you some performance as an InnoDB cluster for 4-5 node clusters. Let me share one of my experiences with the Galera cluster. We had Galera Cluster with six dedicated servers on OVH. It was a Master-Master cluster with the 2-way write lock. But the issue was that when we added a new database node, it increased the time for Inserts/Replace queries. So, we build a new cluster with asynchronous replication, and it can handle 4x more Queries Per Second(QPS) than the previous cluster.

Programs and databases are currently functioning. We don't mind considering your option, but how do we migrate to InnoDB? Do you have any ideas about this?

Yes, I can help you with migration because dump both MySQL and MariaDB databases are compatible. Hence, we can migrate it easily. But still, it's had to verify everything, including the database and users after migration. Similar way, I do have a sufficient experience with the MariaDB Galera cluster too.

If we stick with MariaDB, what will allow us to get HA for our applications?

MariaDB Xpand ( MariaDB Maxscale) or ProxySQL I'll still suggest ProxySQL because of its vast number of functions. https://proxysql.com/, https://mariadb.com/products/enterprise/xpand/.

J.A.I.

1. I will be using percona's XTRADB cluster (proven effective) Documentation available on its website. Basically its MYSQL cluster developed by percona. I have already deployed several clusters like these.

2. No. Depending on the setup, I assume this would be available locally? LAN address are free. I would also need to review the current network structure your VPS are currently on

3. I think this is a separate job. There are available cluster monitoring tools or monitoring tools from third party providers like zabbix/grafana etc.

4. depending on the scope, I also include fine tuning the setup, like adjusting kernel parameters, network parameters, configurations on the cluster to suite your current resources, Overall, I am happy with at least $300 ($100 for each node) and a centralized IP address. For additional work like monitoring, depends on what type you would like to have

5. Documentation is a strenuous process. Implementing the cluster itself would take me at most 1 months work since I am not available all the time. With Documentation, I would say at least 2 months

in addition to no.1 I would also be implementing heartbeat since it would be the best approach, no need to change IP's of your clients since we will have a dedicated IP. however it comes with limitations

A.M.

A.M.:

I would like to help you achieve this project of configuring a haproxy in front of your 3 applications.

Please confirm that you want a proxy to load balance visitors or users of the 3 apps in front of the apps. and not between the apps and the database ?

If it is in front of the apps, let me propose you this:

• install an haproxy in front of the apps, on one another VPS (linux + haproxy).
• configure ip firewalling if any between the haproxy and the apps nodes.
• configure the 3 apps nodes as back-end of haproxy.
• configure the haproxy to get the right port of incomming visitors/users requests.
• configure ssl certificates of 3 apps on the haproxy if any.
• test the hole settings.

Those are my ideas. Note that i prefer to do this as a fixed-price project.

V.:

We haven't decided what it should look like yet. We have three VPS with working programs, but we may choose to take a new three VPS for this project. We want to know more about your implementation method.

Can you answer a few questions:

1 "install an haproxy in front of the apps, on one another VPS (linux + haproxy)." - What will happen if the node with HAProxy is disabled? Will applications be available?

2 How will cluster monitoring be provided?

3 Each node has two IP addresses: IPv4 and IPv6. Do we need to buy additional IP addresses?

4. How much time do you need to implement a cluster with report documentation?

5. How much will the cluster cost us with all the additional costs?

A.M.:

1. "install an haproxy in front of the apps, on one another VPS (linux + haproxy)." - What will happen if the node with HAProxy is disabled? Will applications be available?

--> The node with the haproxy will be the entry point of the apps, if it is disabled for any reason, it would need to get back online ASAP. The emergency measure would be to point the DNS names on the IP of apps directly while waiting for the haproxy. Of course, having a second backup haproxy would be a more secure option.

2. How will cluster monitoring be provided?

--> The best solution is to have another vps just for that, external of the apps and the haproxy. It might just be a ping or http requests. It might also be a paid solution from an online services.

3. Each node has two IP addresses: IPv4 and IPv6. Do we need to buy additional IP addresses?

--> Yes, just one, for the haproxy. It will be used as entry point for the apps for the haproxy to send to the backend (here the apps server).

4. How much time do you need to implement a cluster with report documentation?

--> I estimate this about 72h.

5. How much will the cluster cost us with all the additional costs?

--> I estimate this arround $330.

V.:

We've changed the ad a bit. Hope this makes the task easier. A few more questions:

1. "Of course, having a second backup haproxy would be a more secure option.", "The best solution is to have another vps just for that, external of the apps and the haproxy." -- How many VPS was optimal? What will be on each VPS?

2. What do you think about databases? How will they be implemented?

A.M.:

image.

V.:

1. Do we need database synchronization? If so, what will synchronize them?

2. You have not removed AVideo from your solution. Wouldn't that be a problem?

3. Will your solution pass our test?

If students are taking an exam on our Moodle course. And at this moment the node will fall. Will your decision allow you to continue taking the exam?

4. "The best solution is to have another vps just for that, external of the apps and the haproxy." -- What will it give us? Why is it the best monitoring solution?

A.M.:

1. Do we need database synchronization? If so, what will synchronize them?

--> yes, it will be synchronized with galera/mysql cluster.

2. You have not removed AVideo from your solution. Wouldn't that be a problem?

--> I can remove it from the final solution.

3. Will your solution pass our test?

--> It will, the tests consists of shutting down two nodes and test if the apps are still available. The haproxy will then send the incomming trafic to the remaining node. The monitoring node will email you or slack you at that time.

The second test is:

During documentation testing, we will erase the software from one, implement the rescue, and one expert will try to restore the software using your documentation ---> This means, i will install and configure all apps on all nodes. Which is beyond the haproxy config.

I will need to re-estimate the budget if so.

Please confirm if all apps are already available or need to be installed and configured along with the haproxy.

4. "The best solution is to have another vps just for that, external of the apps and the haproxy." -- What will it give us? Why is it the best monitoring solution?

--> It will have an external ability to check all apps from the outside of the 3 VPS of apps.

V.:

The manager wants to consider your offer regarding the price with the program installed and configured. We want to add our site (WordPress) to the cluster. Questions from the manager: 1. What about cluster security? What firewall is provided? 2. Is there a software solution for monitoring?

A.M.:

Please find my answers below:

1. What about cluster security? What firewall is provided?

--> We can use iptables on the haproxy node to filter trafic.

The documentation will include a part on how to block IP or allow specific one to access the cluster.

2. Is there a software solution for monitoring?

--> For now on, i can provide a bash script that will check the status of all services on all nodes:

- script to check the galera mysql cluster.

- script to check if the haproxy is alive.

- script to check if the http and php servers are available on all nodes.

- script to check cpu load, disk space on the nodes.

All scripts will alert to slack if you have one or email. Those will be included in the project without extra fee. Please provide or confirm the softwares informations below:

- databases use mysql server ?

- web servers on all nodes are nginx or apache or express node js ?

- the load balancer will be HAProxy.

- php backend for php driven apps like moodle or wordpress ? Which php version ?

I would plan this as following:

- check all configurations of all nodes to know the apps structure (folder path, process, ports, IP, gateway, ssh access between each node for monitoring).

- backup all databases for all apps.

- check and configure the sql cluster on all sql node.

- test sql clueter to see if data are well replicated.

- check the haproxy node to see version and confirm the configuration, ACL and front/backend config.

- check the haproxy node to see if firewall is installed and online to be configured.

- check the route from the haproxy to all nodes. Can it join all node without too much TTL.

- configure the front end on the haproxy.

- add the first backend on the haproxy to point to the first node.

- test the configuration by calling apps URL using haproxy IP.

- add another backend and test by calling the apps URL via haproxy IP.

- test to remove one node to see if calling the apps URL still working.

- if not, debug. If yes, add another node to test.

- from here, tell the client (You) to test the configuration. Open Apps with haproxy URL.

- implement monitoring script on the monitoring node.

After the tests with the client are successfull. Wait for production GO. If yes: Configure the DNS name of each app to point to the haproxy IP to get the cluster live. Send the documentation to the client and wait for the feedback. Please node that i will need root access to all node while configuring. You will need to change it when you will be satisfied about the configuration.

V.:

"databases use mysql server ?" -- MariaDB.

"web servers on all nodes are nginx or apache or express node js ?" -- What will be better for us? Why?

"the load balancer will be HAProxy" -- We are not against considering other offers.

"php backend for php driven apps like moodle or wordpress ? Which php version ?" -- 7.4

A.M.:

"databases use mysql server ?" -- MariaDB. --> ok.

"web servers on all nodes are nginx or apache or express node js ?" -- What will be better for us? Why? --> i think both can handle it without issue, thougth there is a laege debate on which one to use;

i master apache better than nginx, so i suggest apache.

"the load balancer will be HAProxy" -- We are not against considering other offers. --> ok.

"php backend for php driven apps like moodle or wordpress ? Which php version ?" -- 7.4 --> ok.

V.:

If we use Nginx will it be Nginx Open Source or Nginx Plus? More precisely: Is Nginx Open Source enough for us?

A.M.:

i think nginx opensource will do the job.

Реакция на предложения

Все аббревиатуры, все технологии, которые упоминали кандидаты, должны появиться на Правке. Я добавил в задание Вордпресс, так как он также на Кампусной Ферме. Ранее, я планировал делать его отдельным проектом, но раз мы вынесли AVideo, есть смысл вернуть Вордпресс.

По поводу архитектуры, нравится подход A.M. -- 5 VPS, из которых одна оставлена на вход, другая -- на мониторинг.

1. Вход (распределитель запросов на общественном веб-адресе). Есть HAProxy и Nginx Plus. Что-то ещё? Код Nginx Plus закрыт? Если да, то Nginx Plus нас на этом проекте не интересует. Если код открыт, тогда мы воспользуемся им для Оплёта.
2. Синхронизация данных. У нас есть хорошо работающая Galera. Расширение Galera в сторону MaxScale и/или xPand -- что мы получим и сколько это будет стоить?
3. Мониторинг. Не вижу дельных предложений.
4. Защитные стены (firewall). Не вижу дельных предложений.

Кстати, у нас есть 30 приглашений, которыми мы пока не воспользовались. И ещё 30 от объявления на железный кластер :)

Работа по контракту

Заказчик

Cool! I am separately sending access details of the new VPS, as well as of those VMs that Natalia created that you would need to configure the

cluster.

Подрядчик

Thank you for the information about the VPS. I have tested it and i have a successful access to it; VMs are working well.

Заказчик

That's great! Good luck with your work and let us know whether there is anything we can be helpful with.

Подрядчик

Finished:

• DONE 1. check and configure the sql cluster on all sql node.
• DONE 3. check the haproxy node to see version and confirm the configuration, ACL and front/backend config.
• DONE 4. check the haproxy node to see if firewall is installed and online to be configured.
• DONE 5. check the route from the haproxy to all nodes. Can it join all node without too much TTL.
• DONE 6. configure the front end on the haproxy.

Pending:

• PENDING 2. test sql cluster to see if data are well replicated.
• PENDING 7. add the first backend on the haproxy to point to the first node.
• PENDING 8. test the configuration by calling apps URL using haproxy IP.
• PENDING 9. add another backend and test by calling the apps URL via haproxy IP.
• PENDING 10. test to remove one node to see if calling the apps URL still working.
• PENDING 11. if not, debug. If yes, add another node to test.
• PENDING 12. write the most complete documentation

Заказчик

Thanks for the encouraging update! We are looking forward to working with you long term!

Подрядчик

I need the haproxy to be able to join the 3 servers on port 80. in private lan would be best if possible

• Would you mind asking Contabo : how do one connect to the 3 ubuntu VMs on port 80 ?
• Or can the haproxy VPS can join three VMs the private ip of each ubuntu?

Заказчик

• Contabo offers "private networking" for its servers within the same region, which is not our case. I am not sure that you realize that VMs are not on Contabo; they are on Hetzner.
• I am not an expert, but I believe that we need to try something like SDN or VPN to solve that without Contabo. However, I cannot guarantee success.
• In addition, Natalia will open ports 80 of Hetzner VPSes to the "world" (will make them accessible from outside) today.

Подрядчик

• Ok, i will see when the port 80 on the VPSes are open and will get back to you if i have any question.

Заказчик

I have just sent the port details separately.