Обсуждение:Делова Ферма

Работа по первому контракту (HA cluster)

Подрядчик

Your HA proxmox was ready. I have tested it, and it running well.. You can do trial and testing meanwhile I am writing its documentation ... However, Not all users could access IPv6, you must have ISP ipv6 supported, modem ipv6 supported and devices ipv6 supported. ... That's why, I changed my ISP to ISP ipv6 supported. And now, I can access IPv6 and IPv4

Заказчик

To test, I am going to assign a domain name and install a HumHub instance. How would you recommend installing that? What address shall I use for the domain name? ... I didn't get what I should do in regards to your IPv6 concerns. Do you want me to create one more vSwitch and buy IPv4 for that?

Подрядчик

Currently our proxmox is running well on ipv6, I put 2 domains using ipv6. First pm1.seethisnow.site, and second wordpress403.ntbprov.go.id. ... Can you ping to both of them? ... If not, you are the owner of these all services and can't reach your own assets. So, how can we sell these services?, Meanwhile not all of our team members could access it. That means, ipv6 only for specific target, and it is not common for all users out there yet for now. FYI, wordpress currently doesn't support ipv6 yet, so you can't install wordpress to server that only using ipv6. ... to sum up, my point is our servers is ipv6 ready and everything goes well on ipv6. I just think about your market target? Did they all ipv6 supported already? If not, I think you should consider about ipv4, I know it is quite expensive. But for now, ipv4's scope area much larger than ipv6. ... If I only thinking about my self, I would say "we should use ipv6". However, I cant be like that, I believe money is easy thing to find. And my satisfaction is when I can help you to reach your goal. And last but not least, decision is yours.

Заказчик

1. I succeeded in pinging both pm1.seethisnow.site and wordpress403.ntbprov.go.id
2. At the moment, we plan to have HumHub, Odoo, SuiteCRM, and, possibly, Jitsi there. I purchased a separate Cloudflare service for Wordpress. We may try to install all and ask our team members whether they can access.
3. What if we make both IPv4 and IPv6?
4. It looks like we could save money if we purchased IPv6 only servers, couldn't we?

P.S. I asked you two questions that you didn't address --

• How do you recommend installing our applications?
• What address shall I use for the domain name? 2a01:4f8:fff0:53::2 ?

Подрядчик

for better experience, we can use lxc containers for each apps. If lxc container not enough, we can upgrade to vm. ... then you can decide, which apps need to using HA feature. ... If you want easier to manage, you can use one ip for each apps, and lxc container very light (because we have so many ipv6, but if you want to use ipv6 yes we can use only 1 ip address). how many users will use jitsi? ... I can make all those apps behind pfsense firewall, so only need 1 public ip to all those vms or containers, except jitsi. Jitsi better use public ip directly. ... if you want add public ipv4, please add to vswitch with public ip. so we can use ipv4 and ipv6 simultaneously

Заказчик

1. I have researched the IPv6 topic more and decided not to add IPv4. According to Google (who else would know better?), 38% of the Internet is already IPv6. May you include a possibility of adding IPv4 to vSwitch with public IP into documentation?
2. Starting with lxc containers for each app sounds reasonable. All of the apps need to using HA feature.
3. I like your one-ip-for-each-app idea. So, should I do 2a01:4f8:fff0:53::2, 2a01:4f8:fff0:53::3, etc.?
4. I cannot estimate the number of Jitsi users. There were fewer than 10 before. I guess that we can target 50 as a max for now.
5. I am not familiar with the pfsense firewall, but am open to it. When you wrote, "so only need 1 public ip to all those vms or containers," did you mean IPv4?

Подрядчик

1. Sure, I can include a possibility of adding IPv4 to vSwitch with public IP into documentation.
2. All of the apps will be lxc containers using HA feature.
3. 2a01:4f8:fff0:53::2, 2a01:4f8:fff0:53::3, etc. addresses will work.
4. With regards to Jitsi, we can create a vm or container with 8 cpu core for maximum 50 participants. Or if you have a small groups, you can use talk from nextcloud. Or if you think, you need a video conference for student or engineer, you can use bigbluebutton.
5. Pfsense is a firewall OS, you can using ipv4 only, ipv6 only, or both of them. I just suggest if we use ipv4, so that could help to make more efficient in using ipv4. All vps behind firewall will using a private network, or network on vSwitch with non-public IP. And all internet connection will through pfsense interface.

Заказчик

• I added 4 AAAA records as follows -- DNS зона
• We planned to use nextcloud, but not on this cluster and not HA for sure.
• I haven't used BigBlueButton for years. Have they fixed their mobile access troubles?
• Okay, let's try pfsense. It sounds promising.

Подрядчик

• What version did you use bigbluebutton last time? It's version 2.6 now. And using ubuntu 20.04
• Have you point domain name for your proxmox server?
• May I know, how many employee did you have? I need this information to decide server specs for Odoo

Заказчик

• Last time, I used BigBlueButton somewhen in 2017, if not 2016. It used Flash then and didn't work on mobile devices. I looked at their specs now and see that now it can be run on Safari and Chrome, which is an improvement. You didn't respond to my direct question whether they fixed their mobile access troubles :)
• Besides 4 AAA records, I haven't done anything. What exactly do you want me to do? Should I assign different names to every node? every IPv4 address available? Anything else?
• We are in startup mode yet. Let's plan for 10 employees.

Подрядчик

• you can use following names:
  1. pm1.bskol.com A 88.99.218.172, AAAA 2a01:4f8:10a:439b::2
  2. pm2.bskol.com A 88.99.71.85, AAAA 2a01:4f8:10a:1791::2
  3. pm3.bskol.com A 94.130.8.161, AAAA 2a01:4f8:10b:cdb::2
• and if you want, you can give a subdomain for pfsense -- pf.bskol.com AAAA 2a01:4f8:fff0:53::6
• also, would you like to have backup server for your proxmox? if you do, I can Install proxmox backup server
• Sorry I missed that part, now they're using html5 and nodejs. bbb support mobile browsers since they migrated to html5.

Заказчик

1. I created A and AAAA records for pm1., pm2., and pm3., as well as AAAA for pf.bskol.com. That is funny. We (Natalia and myself) are relatively new to system administration. Someone advised us not to use our primary IP addresses in DNS, so we did so. Do you have any idea why? :)
2. The proxmox backup server is awesome -- yes, for sure.
3. I got your BBB point. I will discuss with Natalia how we can revisit that software.

Подрядчик

• PM1, PM2, PM3 are using your primary ip from hetzner. ... your IP from hetzner will we use as management ip for Proxmox. ... other hosted apps will use ip from vswitch, because with this method we can do HA.
• I think we stay with jitsi for now, I will build scale up jitsi.

Заказчик

Our IPv6 approach yet concerns me. What if we decide to expand, let say, to Pakistan? For now, each PVE instance has two addresses -- IPv4 and IPv6, but we really need only one, AAAA. May we use IPv4 addresses to point directly to our apps? To the best of my understanding, they will lack HA features, but they will still be accessible via IPv4.

Подрядчик

• Did you mean, we are using our only one pve ipv4 address for apps?
• If that so, that means we will disable HA feature. Why dont you say it from begining? So, I will not set up proxmox in HA mode. In HA mode, you only can use 1/3 capacity harddisk totally. So, if one server has 512GB, in HA we only can use 512GB totally. In fact, HA mode uses 3 servers, that means we able use 512GBx3 at maximum on clustering only mode.

Заказчик

I meant nothing you said; I am sorry for not being more clear. Let's try from another side. We don't really need IPv4 addresses, do we? If I delete A records, the cluster would still work, right?

Подрядчик

• Yes, absolutely. but I never tried to separate ipv4 and ipv6 which is given by hetzner. You want to try that method? Let's do it, please point ipv4 from pm1. to jitsi site, and ipv4 from pm2 to pfsense, and all others apps point to pfsense ipv4.
• lets try this, inform me if you have updated the records

Заказчик

I decided to do that step-by-step. For now, I have just deleted all of the "A" records. I am going to sleep now, wake up tomorrow and check. If it works, then, we will play with directing IPv4 directly to apps. What web server do you use? Nginx? Apache? Neither? Both?

Подрядчик

• some of apps will use apache and the others will use nginx
• actually, we only use ipv6 from begining. it's why I tried using my own domain with AAAA record only. pm1.seethisnow.site

Заказчик

Cool! I added 4 "A" records -- jitsi1 to access Jitsi, sprava2 to access Odoo, setka2 to access HumHub, svazka2 to access SuiteCRM ... If this attempt is not successful, we will discontinue it and concentrate on IPv6 only.

Подрядчик

npm.bskol.com online now. That means, I can deliver your requested ipv4. However, Your requirements is at advance level, and I need to re-install all these three servers.

Заказчик

Your news are encouraging! By the way, my idea worked, didn't it? If seriously, I really like our collaboration and am interested in its continuation.

Подрядчик

• https://setka2.bskol.com is online now ... https://jitsi1.bskol.com is online now ... Please do some test on this jitsi
• Can you ask support request to hetzner for remote console? I need remote console access to PM3 or Server Auction #1788549 . That would be great if you can provide that, I need to checking up PM3. Pm3 state was down on cluster

Заказчик

• With regard to the console, would you like us to have a conference call, so we can do whatever is needed together, or something else?
• I also got that message from Hetzner -- "Your assigned power button press for your server Server Auction #1788549 (94.130.8.161) has just been initiated."

Подрядчик

Yes, for now I need that remote console, as long as I remember on hetzner we can request for remote console. And it is free for first 3 hours everytime we need remote console. On pm3 proxmox, I have installed backup server for all nodes. If remote console doesn't free any more, please inform me and I will reinstall the server. However, it would be great If I can do troubleshoot first, so I can get information what happened to that server.

Заказчик

I got now what you need and requested the KVM console.

Подрядчик

• Odoo v.15 url: https://sprava2.bskol.com
• Ok, I think I should reinstall it ... it said, error: no such partition. ... I am not sure what happened to its partition, But when I am checking there is no partition on it
• I just afraid that you need these server to ready In next two or three days. So I need finish these earlier ,because I need do some trial test before we use it as production server.

Заказчик

• No worries; the scope of the project increased, so should its schedule.
• This server is bought via auction; some of them are simply not good. I don't know whether you checked them. Do you want me to buy another server?

Подрядчик

• No, I have reinstalled it. Lets see in next few days.
• Can you create a dns record for pbs.bskol.com to AAAA 2a01:4f8:fff0:53::6 ... pbs =proxmox backup server
• please check SuiteCRM 7.11 url: https://svazka2.bskol.com
• all Proxmox servers and vms already online ... I will write the documentation

Заказчик

• The record for pbs. is created.
• I will ask Natalia to start testing and am looking forward to the docs

Подрядчик

• you can access backup server from each proxmox nodes or you access it by visiting this url: https://pbs.bskol.com:8007
• do you have your own email server? ... IF you don't have email server and having plan to build email server. I have a little suggestion, you can use cyberpane (https://cyberpanel.net)l or cwp pro (https://control-webpanel.com/cwppro).

Подрядчик

Hopefully Natalia satisfy with those apps installation.

Заказчик

I can promise that we will not ask you to re-install. Let's complete that project and go to another.

Подрядчик

It's okay if you think it's needed to reinstall, as long as not ask for reinstall HA proxmox.

Заказчик

Absolutely. You talked about the apps -- I precisely meant them, not PVE, PVE backup or Nginx

Приёмка изделий по первому контракту

Подрядчик

It's ready. Please inform me, if there is something I missed. We can communicate more to avoid miscommunication. We need create a chanel like slack or etc. Don't worry, I still can fix it

Заказчик

Almost there. Give me a day more.

Подрядчик

Is Natalia able to create a new vms/containers and make them connected to Internet?

Заказчик

Yes, she tried and, when I asked your question, she couldn't stop talking for a few minutes :)

• Natalia's questions:
  1. VM access. Natalia couldn't access VMs with applications.
  2. Mapping. We cannot find anything that resembles iptables. Does the pfsense carry mapping? Where can we find it?
  3. Nginx. The link, https://npm.bskol.com/, is down now. However, I clearly recall that I could get to Nginx initially.
  4. pfsense. We couldn't find documents related to pfsense. We also have no expertise in that piece of software.
  5. UDP. Your cluster doesn't experience any problems with UDP port blocking. How did you succeed in doing so?
• My questions:
  1. Physical replication. What did you use for bare-metal-level replication -- RAID or ZFS? Why or why not?
  2. IPv6 app DNS. My idea for DNS was presented in its DNS zone. However, the AAAA records for applications are not resolvable. Probably, we misunderstood each other.
  3. Conference. We would probably need to have a conference -- if so, I propose to test your Jitsi instance 🙂

Подрядчик

1. VM access. You can't access any vms while npm state is down. Npm has mapping to each vms behind it. Npm has role as reverse proxy to each vms on pm2.
2. Mapping. Rpc bind, as long as I remember it was fixed on proxmox 7. However, I forgot do I have configured to accept rpc bind connection from internal network only or not? Usually I configured to allow rpc bind for internal network only. So it couldn't be accessed from external network like internet.
3. Nginx. I have fixed npm.bskol.com
4. pfsense. I decide to not install pfsense because of lack ipv4. We don't have any ipv4 left, so I decide to use NPM.
5. UDP. No clear answer.
6. Physical replication. Due to our limited storage, I decided to do some breakthrough, including how to make your data safe. So I create pbs to backup all vms on cluster. I know that's not enough, but quite good for short term. I will suggest you to subscribe storage box on hetzner, you can starting from 1 TB storagebox. ... Nevertheless, for cluster we already have ceph with 3 nodes replication. FYI, I can add storage box from hetzner to proxmox as your backup storage, then we can remove pbs. Furhtermore, we can create second ceph pool storage for cluster. ... And I can create script to make it auto remount after server restarted.
7. IPv6 app DNS. I thought you didn't want to use it. However, I can add NIC interface in next few minutes. Setka.bskol.com online now.
8. Conference. No clear answer.

I will continue to fixing the others tomorrow

Заказчик

That's great! Thanks for the update. Please let me know whether anything is needed on my side.

Подрядчик

If you want, you can add npm1.bskol.com with jitsi1.bskol.com ipv4, and npm3.bskol.com with pm3.bskol.com ipv4 ... And you can change http://npm.bskol.com to http://npm2.bskol.com, so each node has its nginx proxy manager

Заказчик

I have just done right that.

Подрядчик

I see, Ok. I just want to inform you that all your nodes state has been online, except jitsi.bskol.com and pfsense. we still find out, how to make jitsi work with two different network interfaces. FYI, we have decided to cancel pfsense installation.

Заказчик

I am glad to hear from you! I got your pfsense info. Please leave any network interface for Jitsi -- it looks like we need to move Jitsi to the next project. Let's close this project and move to the next one.

Подрядчик

By the way, I had feeling that you and team a quite confuse, how to create a vps or container correctly so that accessible from Internet. Am I correct? If that so, I can explain it on vicon meeting. We can arrange time if needed.

Обсуждение второго контракта

Заказчик

Here are new questions as follows:

1. Security -- what can you offer?
2. Replication -- you mentioned two options: to add a storage box as NAS or to add a storage box for CEPH -- right? Which is the best?
3. Monitoring -- what can you offer?
4. Outside the cluster you assembled, we use a bigger server (4Tb) for VMs. May we substitute the third server in your cluster with a bigger server that will host a few projects without HA capacity? We plan to use its second disk for ZFS, so working CEPH in that case must be on the first disk, right? Is that possible? At that bigger server, we consider adding a function of automatic creation of VMs for training participants, most likely, using Terraform and Ansible. Would it be a problem?

Подрядчик

1. Security -- what do you need? For cluster, I think firewall plus fail2ban is quite good enough.
2. Replication -- I said a storage box for backup server. So we can use it as backup server. We can't use storagebox as active storage, because its speed is not so fast.
3. Monitoring -- I can do build and monitoring using grafana + influxdb+telegraf , and also I can use zabbix to. For website monitoring, we can use uptimerobot I think.
4. Substituting -- I think as long as we can mount it as storage on cluster, we can use it as Vm disk. However, I hope the connection speed is quite fast enough. If not, it will take time when loading data.

Заказчик

Okay, let's start with 3 items:

1. Security. Let's do firewall plus fail2ban. Separately, we will order security for various applications.
2. Replication. We will add a 1Tb storage box for the backup server. Now, may you explain your idea here, "Furhtermore, we can create second ceph pool storage for cluster. ... And I can create script to make it auto remount after server restarted"? What is this? Where would that second ceph pool storage be located? What should I do to get started?
3. Monitoring. Please build grafana + influxdb+telegraf , and also I can use zabbix to. For website monitoring, use uptimerobot. What do you need on my side to get started? Should I buy a VPS from hetzner? Anything else?

For your situational awareness,

• We purchased an additional bare metal server with 128Gb RAM and 2Tb disk, but not an SSD one at the same location. Natalia will create a ZFS replication there, so one disk would be replicated to another disk, so CEPH should be located together with applications. Then, she will try to add it to the cluster and, further, we would delete the third node.
• We plan to re-do IPv6/IPv4 entrance, switching from vSwitch to HAProxy, for which we will buy a separate VPS.

Once again, please start working on that and let me know what is needed on my side.

Подрядчик

I mea, If you attached a storagebox to cluster, I can remove pbs as a backup server. Then I can remount all second drive on cluster, to create a new ceph storage. So it will be 2 ceph storage on all cluster. As its effect, all "local-storage0*" (please see on proxmox) on each nodes will be removed. What do you think? If okay with this, I will create it.

Заказчик

1. To order the storage box, should I make any "Additional comments"?
2. I don't understand "remount" second drives in the contest. Right now, CEPH is located on second drives. Would you disconnect CEPH from first drives? Where would that second ceph pool storage be located?
3. With regard to monitoring, you didn't respond to this -- "What do you need on my side to get started? Should I buy a VPS from hetzner? Anything else?" I am looking at CX11 (1 Intel, RAM 2 GB, Disk 20 GB), CPX11 (2 AMD, RAM 2 GB, Disk 40 GB), CX21 (2 Intel, RAM 4 GB, Disk 40 GB), or CPX21 (3 AMD, RAM 4 GB Disk 80 GB). Am I right? Would a VPS work?

Подрядчик

1. No need, just order 1 TB.
2. Yes, For now ceph using second drives. I mean I will recreate the storage, from local-storage to became our second cluster ceph. But you should think your needs, did you need 2 storages ceph on cluster? Because create second ceph will decrease the storage size totally, from 400GB each nodes to became 400GB for all clusters (3 nodes).
3. Actually we can use a vps from our clusters, but if you think we need a separate server for monitoring. Vps with 4 GB is enough for now.

Заказчик

1. I have just ordered a storage box and am sending you its details separately.
2. Let's leave CEPH as it is for now.
3. I am sure that a separate VPS is better than using one of the servers that needs to be monitored. Now, what image? Ubuntu 20.04? Next, networking. We don't need IPv4, do we? Public IPv6 and private networks? Should we use Hetzner's firewall or set up our own?

Подрядчик

As per IPv4, as long as all servers we monitored are ipv6, we don't need ipv4. For firewall, I would prefer to use our own firewall if possible.

Заказчик

Now, what image? Ubuntu 20.04? Next, networking. Do we need private networks in addition to Public IPv6?

Подрядчик

• Correct, we're using Ubuntu 20.04.
• For private network, actually I am not sure how we can connect to all servers using private network. I have no experience in subscribing vps on Hetzner, So I have no information does vps able to connect to hetzner switch or not? However, if you want a dedicated servers one, please use the auction server.

Заказчик

No, no hetzner VPS is able to connect to the hetzner vswitch. Can we create something like SDN or VPN?

Подрядчик

• Did you mean, we create a vpn to all servers?? I think it's not efficient method.
• How if we try create a vps on current proxmox first, then we tried which way does your need? Or which way the most efficient one?

Заказчик

• Why isn't vpn to all servers efficient? We will buy two VPS from Hetzner, use one for monitoring and another one for HAProxy and will retire the vSwitch with Public IPv6 -- why isn't this idea great? :)
• We don't have current needs -- instead, I would like to build a secure, productive and affordable cluster.

Подрядчик

If that so, please purchase one ipv4 public, on pm2 or on each node. So I can install pfsense, then create a vpn server.

Заказчик

• I would like to buy 2 CX11 vCPU 1 Intel RAM 2 GB Disk space 20 GB at https://www.hetzner.com/cloud -- one is with IPv4 (for HAProxy), one is without (for monitoring). I guess we can order private networks to link the two.
• I don't understand your "on pm2 or on each node" part. Please clarify.

Подрядчик

Sorry I thought you will understand. Pm2 I mean pm2.bskol.com, each node I refer to pm1, pm2 and pm3. Why we're not create a pfsense on dedicated vps?, because I would prefer on vm that has direct connection to our others vm or servers, this scheme will influence the connection speed. I believe if we place pfsense on our server network and directly connected (it could be LAN connected or inside same virtual server), the response will be faster and help our server more secure when we access it.

Заказчик

Wait, I am telling you about 2 CX11 vCPU 1 Intel RAM 2 GB Disk space 20 GB at https://www.hetzner.com/cloud -- one is for HAProxy, another is for monitoring. You are telling me about pm2 and pfsense. May you just say whether my idea will work?

Подрядчик

• Sorry, I just explain about planning to having vpn server (pfsense), you just need to purchase one ip address then I can install a vpn server.
• Continue to your topic, at the moment We can't buy a vps for haproxy, I need the draw plan or topology plan or anything that relate to that. So I can Understand where is our haproxy will be placed, we need to draw the network scheme or topology plan first I think. So we can see the function of that haproxy clearly.
• However, for monitoring yes you can buy that one today, and please buy vps that ipv4 included.

Заказчик

Aha, I didn't realize two things --

1. pfSense has some VPN capabilities -- https://docs.netgate.com/pfsense/en/latest/vpn/common-deployments.html
2. You cannot evaluate immediately whether HAProxy can work with our nodes.

Now, I am still willing to buy 2 VPSes with public IPv4 -- we would use one for monitoring and another one for pfSense and, possibly, for HAProxy. Earlier, you indicated that HAProxy works well with pfSense. Is that okay?

Подрядчик

Ah I see... Yes, you're right.. but I never use haproxy using wan connection to load balance host(servers). I usually use it on LAN connection. The scheme is Internet -> pfsense/ha proxy -> Lan connection -> servers

Заказчик

So, what do you want me to do?

Подрядчик

• If you want your server like on node pm1,pm2.and pm3.bskol.com use pfsense as their firewall and haproxy as their load balancer, you need add single ipv4.
• But if you want to buy a vps for haproxy and there is no lan connection to our servers, I think the firewall on pfsense will not function properly. And also I think we need extra effort to make all our servers connect to vpn server on pfsense. and also I have no experience on implementing haproxy via wan connection. So I needs time to learn from best practice on internet if any.

Заказчик

• I am sure that locating monitoring tools on the resource that needs to be monitored is not a permanent solution. Okay, I will add public IPv4 to pm2, what if it fails? Why bother with temporary solutions if we are not in a rush?
• Yesterday, I sent an offer to somebody who promised to implement HAProxy on our VPSes located in different regions. Should we wait to see whether he does that? We can move to another project, let say, databases, in that case.

Подрядчик

To sum up,

1. As long as I remember for monitoring I said you can buy a vps, and please choose the vps that has public ipv4. I agree that we need buy a vps for monitoring and monitoring case closed now. Yes you can buy a vps for monitoring
2. lets continue talk to second case, HAproxy, I didn't say it can't using WAN connection but I need time on implementing it. Because I never have experience on implementing it using WAN connection between ha proxy and host or server. I just had experienc on LAN connection.

so I have experience on this method: Internet -> haproxy --- LAN connection ---> hosts/servers. I never had the experience on this method: Internet -> haproxy --- WAN connection ---> hosts/servers, But I have experience in using Nginx to that method. the things that I think couldn't be happen on WAN connection is when we use pfsense as firewall, I said here pfsense as firewall not haproxy. and last, impelementing vpn to pfsense that using WAN connection to server ( internet --> pfsense(vpn server) ---- WAN/Internet connection ----> hosts/server) is quite hard I think. Yes you can buy a vps for haproxy, but, no you can't buy a vps for firewall and vpn server at the moment. I am so sorry, sometime I need a picture to imagine what solutions that I can use to solve a problem.

1. When can I start to add the storagebox and other tasks?

Заказчик

Let's start the second contract, while moving HAProxy out of it. As I wrote, I awarded a contract to someone to design a WAN/HAProxy solution. The deadline is October 23. I believe that it is feasible to wait to get new expertise and then act. Let's postpone our WAN/HAProxy project therefore.

Работа по второму контракту (storage, firewall, monitoring)

Подрядчик

• I already added the storagebox and make it as backup storage to proxmox cluster. I attached the documentation via email.
• Ok, next I will secure cluster using fail2ban.