Intro
ping 127.0.0.1
There are some interests which do not change! I used to write html and css in college days. I have had shared hosting accounts, VPS and so on! I still remember creating a small site for college workshop using WordPress. I learnt web development and managing VPS through youtube. I have always wanted to host on my own server and somehow have managed to do it this year. Also the recent development in smart home devices have compelled me to try it. I will share my experiences in creating a homelab here hoping it might be useful for someone, especially people from India.
Huge thanks to these reddit communities for inspiration.
The Hardware
My HomeLab Configuration.
So you need bare metal first to do everything. I searched in google for about 3 months and decided to buy a used HP/Dell Server. Some other options were going on budget and getting a NUC, Mini PC from aliexpress etc., Finally decided to purchase used server mainly because i wanted to get the feel of it. Server & Rack:
I purchased a used Dell R210II from zaco computers. Came to know about them from some posts i found in quora. The server had quad core xeon processor and 8GB of RAM without hard disk. The total cost was around Rs.32k including taxes. I then purchased a 120GB SSD from amazon and installed in it. I would highly recommend zaco computers for purchasing used servers. I contacted them through email and got prompt response. They shipped through fedex and server properly packed.
At the time of purchase i had no idea of what an iDRAC is! but they have put an iDRAC enterprise card in it without me specifically mentioning it. It was a surprise for me actually. Later i found it very useful for controlling noisy dell fans. So if you are going with Dell, make sure you have iDRAC card installed before you purchase.
The only reason for going with Dell R210 was that the length of server would fit into a network rack! I have installed my server and switch in a network rack (jumbo version) purchased from mass racks. I would also recommend mass racks if you need any network racks. Get a cantilever tray for switches. One 6U jumbo rack with cantilever tray costed around 2.7k.
If you are reading this, never ever install a server in a network rack! It was my idea and later i realized that it was very bad. First, the server is very heavy and coupled with rack and switches the entire arrangement is so heavy to lift and mount it on wall. Even if you mount on wall the small 4-corner support will not hold it. I have supported in front by using two extra anchor bolts from top. Second, cable management is very difficult. If you are from India i feel, It is cheap to assemble a wooden rack and rack mount angles with carpenter help. Dimensions can easily be found online. Mass racks also has wheel mounted server racks.
Switch, AP and Misc
I had a relative coming from US and he was kind enough to carry a used Dell 16-port switch ($40 ebay) and a refurbished Tp-link EAP225 ($50 amazon).
So now i have a complete setup with a modem (i had already), a wifi AP, a switch and a server!
Now let’s spin up ubuntu…
Virtualisation with ESXi
After getting my server up and running, i installed ubuntu server 18.04 on bare metal. Few days passed and already it had more than basic nginx and php. Also I felt the server hardware was not used to its full potential by using it just as a webserver. I was searching for a solution and came across VMware ESXI and bumped into the world of virtualisation. It is bare metal hypervisor. Having individual virtual machines would be easy for managing different set of software.
I installed free version of ESXI 6.7 and after installing i created 4 VMs (pfsense, webserver, homeassistant and TIG stack).
One port on my server is directly connected to modem and other one two switch. I created two virtual switches, named them WAN and LAN. Only pfsense VM had two ports attached to it, and all other VMs are on LAN vswitch. pfsense had fully fledged firewall rules with SNORT and pfblockerNG enabled. It also had openVPN server running, so that i could check from remote.
TIG stack is obvious, it had few grafana dashboards which looked awesome and telegraf collected metrics from pfsense and other devices using SNMP.
Then I installed home-assistant and played with it for a while. It is a great piece of software for home automation lovers.
The webserver VM had two static sites, one my personal and other one for my uncle. They were working well with dynamic DNS and i off-loaded all the images to cloudinary. Only html,css and js loads from my server. (Still i have a 512 kbps upload speed on my ADSL connection)
This was going on for about three months. It was a great experience to install, configure and troubleshoot all the softwares i have mentioned above, especially pfsense and home-assistant which have some learning curve.
Then I felt that something was missing in ESXI, and came across proxmox. After seeing a couple of videos on youtube, I decided to get rid of ESXI. Backed up some important configurations such as nginx, pfsense, home-assistant and deleted ESXI!
Switching to Proxmox
Problem With ESXi
There are lot of people who use VMWare ESXi for virtualization. But most of them manage through Vsphere and have multiple servers VSAN clusters. The vsphere license is not free. The main limitation i faced with free version of ESXi, is that there is no proper backup solution. I had to shutdown VMs and copy manually to NAS. I tried some solutions found in reddit communities and third party software, none of which served my purpose properly. Ideally i would love to take automated full backups of all my VMs, which was missing in free ESXi. Also for people with 2-3 servers in small homelab, clustering with free ESXi is not possible.
Enter into Proxmox
I went through couple of youtube videos which walked through Proxmox configuration. I then manually backed up all my important configurations in ESXi and formatted the disk and installed proxmox. After playing with proxmox I can say that the following features are enough for switching to proxmox from ESXi.
-
Auto backup: I have added my NAS backup directory as NFS mount point directly inside proxmox GUI and it takes automated backup of all my VMs every Sunday night while I sleep! such a relief.
-
Clustering: Although i don’t use it is great feature for people with multiple servers on their homelab. You can also create a HA cluster!!
-
Containers: You can add LXC containers directly through GUI, such a neat feature. i use some small services like reverse proxy, wireguard etc., in separate containers. Some also use docker with portainer.
-
Storage: You can create a ZFS pool (ah! those freenas times) inside proxmox and also you can ceph storage. I don’t use it, but many find it very useful.
I am using it for more than 3 months and proxmox seems to be pretty stable. No issues so far and I would highly recommend to anyone. The only problem i faced was some routing issue in default vmbr0 bridge. It was also solved after some help from reddit and a full restart.
Typical Indian Homelab Struggle
The biggest issue i was facing initially are those powercuts. I am not sure about the rest of the world, but in India we are facing frequent power cuts ranging from few minutes to hours sometimes. I had a APC Backups 600VA, which is enough for my single server and NAS but the issue was my server would reboot sometimes during power cut. I really had a nightmare when my server boots each and everytime during power cut.
Online UPS
After doing some research, i decided to buy online-ups. The 3k-5k UPS which we normally purchase is a line-interactive UPS and they take milli-seconds for switching over. Hence sometimes if the delay more than 50 or 100 mill seconds my server starts to reboot. Online UPS is costly and getting them online is little difficult in India. I could find only this store selling APC online UPS at a good price.
I had few contacts in chennai, and purchased a 1kVA online UPS. It is working well for the past 6 months and it costs Rs.15k. If anyone wants details send me a email. As of writing this post, my proxmox shows uptime of 157 days!
So in short, if you are getting a server for homelab, get an online UPS along with it.
Portforward alternative
Getting a High Speed FTTH
For people in India, FTTH is still a dream especially for those who are in small cities. People in metro cities like chennai have a plenty of options and providers to choose from, whereas people in tier-2 cities have less choice and even there were no one providing in my town till 1 year back. So i finally see a 100Mbps speed on my screen, thanks to cherrinet. Although BSNL is providing FTTH in recent years, i couldn’t get in my place, since mine was few kilometers away from town.
Anyone remember those noise from dial up modem ? (I’ll write a post someday on how i updated my IE for Rs.3k!)
Speed with Cons
Ok, now fire stick streams 1080p without any lag but that high connection came with a problem. By default i cannot port forward. They provide a CGNAT service, so all i get is private IP on WAN interface (I cannot open standard ports like 80 443 for incoming traffic, without which my site will not be accessible from outside world.) The ONT devices don’t have any configuration and they connect using PPPoE. If i would like to portforward, i should get a static IP which will cost, around 5k per year. So after googling, i decided to create a VPN connection between my pfSense and a remote VPS where i could point my DNS and open common ports to the internet. I created a droplet in digitalocean and tried configuring openvpn in ubuntu. I faced lot of issues related to routing. (not an expert in linux and routing).
pfSense Site-to-Site VPN
Then i came across a site to site VPN tutorial in pfsense docs, and immediately thought of installing pfSense in a cloud VPS. Luckily vultr has native support to pfsense and you can do a 1 click install. So after installing i had little issues with the configuration and fixed it by googling and doing some research.
update: moved to linode (referral link) since they have a datacenter in Mumbai, ping is 40ms!!
For pointing domains to internal IP address, i need a reverse proxy, for with there is haproxy addon for pfsense. Now i have couple of services like this blog, nextcloud etc., exposed to the world. Basically this is how it all works…
Cloudflare DNS –> Cloudflare Firewall –> pfsense in cloud –> SNORT filtering and blocking –> HAProxy –> Open VPN Tunnel –> pfsense in homelab –> Individual services (www, nextcloud etc.,)
One real advantage is SNORT filters all the unwanted traffic in cloud and allows (hopefully :) ) only required traffic to pass through vpn tunnel.
Useful Links
https://www.techhelpguides.com/2017/06/12/ultimate-pfsense-openvpn-guide/
https://nguvu.org/pfsense/pfsense-inbound_vpn/
https://blog.devita.co/pfsense-to-proxy-traffic-for-websites-using-pfsense/