Got extra compute power and want to help research scientists create a cure for the virus? Folding@Home is a distributed computing network where we donate our spare compute power by running a client which processes work units which are like puzzle pieces for the bigger compute dataset. They've already grown the total compute power to larger than the world's 7 fastest super computers combined.
How do you know if you are helping research the cure? Check this link and look at your project ID. I've stood up an 8 computer F@H farm and plan on adding another 2 computers once I get parts delivered. 4 of these nodes are actually unraid VM's which each have a GPU assigned even though they are physically one server. You can see a picture of the unraid server build for the GPUs here. The 5 CPU only nodes are below minus my laptop. These are all the systems I replaced with the single unraid server.
I've used the advanced control tool to add in all of the node members. You need to set up each one enabling remote access as it's locked down by default. When they are all added, you can see your total Points Per Day for all nodes on the bottom:
I found that running 4 GPU's + all VM vcores draws > 1,100 watts and overloads my UPS. GPUs run much faster than the CPU cores, so I've prioritized GPUs for the unraid server which is why the 4 VM nodes don't have CPU slots (saving just enough juice to not overload the UPS). The 6 other machines are CPU only as they are using onboard video since I pulled the GPU's for the unraid server build. So in total, the unraid server pulls about 900 watts while 4 of the 6 CPU only systems are currently pulling about 500 watts.
Update: You'll want to disable Spectre & Meltdown microcode mitigation as it slows down intel based systems by up to 30%.
What I've found is that the back end servers are having bandwidth/connectivity issues which causes the client to end up spending a long time waiting for work. I create a few different scripts for forcing the client to attempt a more frequent download. Here's the script I run locally on my main system when it gets stuck in waiting for work.
I created this script which runs on startup for pure folding machines which has loop logic to restart when CPU & GPU usage is low.
Bryan Vine
Automation scripts, server admin tips and tricks, How-To's, and various tech goodness.
About Me
My Links
Wednesday, March 25, 2020
Tuesday, February 25, 2020
UNRAID: My venture into virtualized desktops
Unraid is a paid distribution of Linux by Lime Technology Inc which uses the KVM hypervisor, Docker, and it's own proprietary parity storage system to provide an all in one server platform that can do just about anything. Inspired by Linus Tech Tip's videos , I decided to do a proof of concept (PoC) by virtualizing my own system.
After a bit of a learning curve, I worked out the issues and successfully got a VM running with SSD, GPU, and USB pass-through working flawlessly. I then decided pull the trigger after doing a few weeks worth of research to combine 5 systems into a new Unraid server build. 4 desktop gaming VMs, each with dedicated a USB port and GPU. I decided to use virtual disks for their drives on my fastest M.2 NVMe drive.
In all it's glory (with the front tinted glass removed to see better):
The server also runs a dozen docker containers and a couple of other server VMs. It literally replaces all of the computers in the house with a single system. Here are the specs:
It's a beast and by far my most expensive build, coming in around $5k. I didn't buy all of the parts at once, the GPUs were pulled from their respective systems as well as the disks. I didn't need all of the SSD's, but I figured since the motherboard has two M.2 slots and 8 SATA ports, I might as well hook them all up. The main components for the build I had to buy new cost just under $2k.
Here's what the main page looks like and how I've decided to allocate my drives. The M.2 drives get VM's and Dockers, one SSD for caching, the others are for game library shares. As you can see, the main array is 24TB in size, one of the drives is used entirely for parity. This the main feature of unraid. You can have up to two parity drives if needed per array. Everything is formatted BTRFS except the 240gb SSD which is XFS for swapfile (which hasn't been needed yet). The NTFS drive will be formatted to BTRFS and will become a secondary game library (yes, we have over 1TB of games just on Steam alone)
One of the cool things you can do when you have 4 gaming systems that share a single piece of hardware, is use networked shared drives for game storage. That way you only have to install the game one time and all users can access it and you don't have to have duplicate copies of the installed files taking up extra space. The game libraries are shared using virtual 100GB ethernet adapters on each of the VMs so load times are lightning fast. Steam natively supports network share drives for installation, Epic Games has to be "tricked", and Blizzard games don't work at all, but I have another technique for saving disk space, using ref-linked virtual disks.
Looking at the VMs, you can see the breakdown of resources. All CPUs are actually hyper-threaded vcores which is why there are 32 in total. All of the allocated vcores are isolated and dedicated for each VM. The kids each get 4 vcores, 8gb ram, and a AMD based GPU. The missus gets 6 cores, 12gb ram, and the GTX 1060. I've found that the 8 vcores and 20gb ram with the GTX 1080Ti about on part with my old Intel i7 8700K 6 core CPU. A good general rule of thumb when sizing VMs is to part 1 vcore per 2gb of ram. I might up my VM to 10 vcores, but right now, the remaining vcores are reserved for the unraid system and docker containers which really need more CPU I've found.
I was able to get a nice overclock thanks to the water cooling from 3.5ghz to 4.3ghz all cores. The main dash board page looks like this:
I had to do some custom modding to get 4 GPUs to fit. The RX 550 only gets a x1 PCIe lane which is surprisingly enough bandwidth for it to function at max speed and 1080p gaming is fine. I had to modify the GTX 1060 by removing some of the plastic from the housing so the PCIe extension card would fit next to it.
I also had to remove the PCIe bracket and install the USB controller card internally and used the case's 4 front ports to connect to the card's USB headers. I covered the back and sides of the card with tape to prevent it from shorting out by the card touching the case. You can also see the 4 x 8TB drives and 4 SSD's are just jammed in with all the cable management (mostly PCIe power connectors and SATA cables).
I had to mount the RX 550 GPU upside down using the horizontal GPU kit installed backwards with the PCIe extension hidden just out of view.
Finally, I installed VESA TV mounting hardware (rated for 150 pounds) and anchored the mounting rails into two studs using 4 x 3" drywall screws for extra security. The whole unit with glass weighs around 60 pounds.
So far, I'm extremely happy with how things turned out for this build. Overall, I think there is some room of optimization. I know I can push the overclock to at least 4.5 ghz while losing some power efficiency; I'd rather save on power and heat and have a solid system. I would have preferred a TRX40 Thread-ripper base, but that would have increased costs by at least $1500. And I don't really need more cores, just more PCIe lanes. That said, I do feel like the 4 GPUs aren't starved for bandwidth; the more powerful nvidia's get x8 each and the RX 570 gets x2 while the RX 550 still runs fine with x1.
A larger E-ATX server board would have saved me from using PCIe extensions, but again, cost. I would have also preferred to go with 128gb ram, but I'm only using 90% of the ram now. And I could also leverage the swap file disk, but I'm not over allocating ram. Audio over HDMI did require some tweaking (MSI interrupts) so the audio doesn't get choppy, but I figured that out. I also had to adjust the windows VM real time virtual clock settings in the config to stop it from using more CPU then the VM was actually utilizing:
The PCIe USB card is a huge pain in that sometimes only 2 or 3 of the ports show up and a reboot or two is required to bring back the missing ports. It only happens upon reboot, so that's at least tolerable. With everything powered on and running at idle, the system uses only around 200 watts surprisingly, and about 850+ when all four of us play games.
Another weird issue I'm having is using a docker to backup the array to an online backup service provider. The docker invokes many threads in the underlining core OS which can cause latency issues on my main VM. I've found a workaround for now (cron job restarting the docker every few hours), but I'd like to figure out how to prevent it.
I don't think I'll ever go back to separate gaming computers especially since I've cabled up the house for the 4 workstations. I absolutely love all the RGB and wall mounted case; it's a wonderful show piece. It also saves power (almost $600/year by my estimates), allows for some upgrades (I will likely swap out GPUs and eventually go to 128gb ram). I would love to add water cooling to my GPU, but it's a low priority. I'll probably do it when I swap it out with my next upgrade.
More importantly, this was the first time I changed out my storage OS to a purely linux solution. The old VM/storage server was running Windows Server 2016 with HyperV VM's and Storage Spaces for the array. Unraid is definitely faster in terms of disk I/O and Dockers are more flexible and easier to configure than VMs. I also really like how recoverable data is if the array breaks, which happens sometimes.
If you are a hard core enthusiast like myself, I recommend this project. It's not for the faint of heart, but it's really not as bad as trying to build systems in early days of bitcoin/altcoin mining. Unraid is a wonderful OS and the forum support is pretty great. Lots of helpful videos from SpaceInvaderOne.
After a bit of a learning curve, I worked out the issues and successfully got a VM running with SSD, GPU, and USB pass-through working flawlessly. I then decided pull the trigger after doing a few weeks worth of research to combine 5 systems into a new Unraid server build. 4 desktop gaming VMs, each with dedicated a USB port and GPU. I decided to use virtual disks for their drives on my fastest M.2 NVMe drive.
In all it's glory (with the front tinted glass removed to see better):
The server also runs a dozen docker containers and a couple of other server VMs. It literally replaces all of the computers in the house with a single system. Here are the specs:
- AMD Ryzen 9 3950X 3.5ghz 16 Core
- Enermax AIO 360mm liquid III cooler
- Asus ROG Strix X570-E Gaming Motherboard
- 64gb of Timetec Hynix PC3600mhz DDR4 16x4 DIMMs
- Rosewill TOKAMAK 1200watt Titanium PSU
- Thermaltake Core P5 wall mounted case
- FebSmart 4 channel 8 port USB card (quad USB chips for pass through)
- Storage:
- 2 x M.2 NVMe Drives (1.5TB)
- 4 x SSD (2.75TB)
- 4 x 8TB Seagate green HDDs (32TB)
- GPUs:
- EVGA SC2 nVidia GTX 1080 Ti 11gb
- EVGA SC nVidia GTX 1060 6gb
- XFX AMD RX 570 8gb
- Gigabyte AMD RX 550 2gb
- 2 x PCIe x1 to x16 USB extensions (one for RX 550 GPU and one for USB card)
- 3 x 50ft HDMI 2.0 cables
- 3 x 50ft USB 2.0 cables w/ power repeaters in the middle
- 4 USB hubs
- 200mm RGB cooling fan
- RAM cooler
It's a beast and by far my most expensive build, coming in around $5k. I didn't buy all of the parts at once, the GPUs were pulled from their respective systems as well as the disks. I didn't need all of the SSD's, but I figured since the motherboard has two M.2 slots and 8 SATA ports, I might as well hook them all up. The main components for the build I had to buy new cost just under $2k.
Here's what the main page looks like and how I've decided to allocate my drives. The M.2 drives get VM's and Dockers, one SSD for caching, the others are for game library shares. As you can see, the main array is 24TB in size, one of the drives is used entirely for parity. This the main feature of unraid. You can have up to two parity drives if needed per array. Everything is formatted BTRFS except the 240gb SSD which is XFS for swapfile (which hasn't been needed yet). The NTFS drive will be formatted to BTRFS and will become a secondary game library (yes, we have over 1TB of games just on Steam alone)
One of the cool things you can do when you have 4 gaming systems that share a single piece of hardware, is use networked shared drives for game storage. That way you only have to install the game one time and all users can access it and you don't have to have duplicate copies of the installed files taking up extra space. The game libraries are shared using virtual 100GB ethernet adapters on each of the VMs so load times are lightning fast. Steam natively supports network share drives for installation, Epic Games has to be "tricked", and Blizzard games don't work at all, but I have another technique for saving disk space, using ref-linked virtual disks.
Looking at the VMs, you can see the breakdown of resources. All CPUs are actually hyper-threaded vcores which is why there are 32 in total. All of the allocated vcores are isolated and dedicated for each VM. The kids each get 4 vcores, 8gb ram, and a AMD based GPU. The missus gets 6 cores, 12gb ram, and the GTX 1060. I've found that the 8 vcores and 20gb ram with the GTX 1080Ti about on part with my old Intel i7 8700K 6 core CPU. A good general rule of thumb when sizing VMs is to part 1 vcore per 2gb of ram. I might up my VM to 10 vcores, but right now, the remaining vcores are reserved for the unraid system and docker containers which really need more CPU I've found.
I was able to get a nice overclock thanks to the water cooling from 3.5ghz to 4.3ghz all cores. The main dash board page looks like this:
I had to do some custom modding to get 4 GPUs to fit. The RX 550 only gets a x1 PCIe lane which is surprisingly enough bandwidth for it to function at max speed and 1080p gaming is fine. I had to modify the GTX 1060 by removing some of the plastic from the housing so the PCIe extension card would fit next to it.
I also had to remove the PCIe bracket and install the USB controller card internally and used the case's 4 front ports to connect to the card's USB headers. I covered the back and sides of the card with tape to prevent it from shorting out by the card touching the case. You can also see the 4 x 8TB drives and 4 SSD's are just jammed in with all the cable management (mostly PCIe power connectors and SATA cables).
I had to mount the RX 550 GPU upside down using the horizontal GPU kit installed backwards with the PCIe extension hidden just out of view.
Finally, I installed VESA TV mounting hardware (rated for 150 pounds) and anchored the mounting rails into two studs using 4 x 3" drywall screws for extra security. The whole unit with glass weighs around 60 pounds.
So far, I'm extremely happy with how things turned out for this build. Overall, I think there is some room of optimization. I know I can push the overclock to at least 4.5 ghz while losing some power efficiency; I'd rather save on power and heat and have a solid system. I would have preferred a TRX40 Thread-ripper base, but that would have increased costs by at least $1500. And I don't really need more cores, just more PCIe lanes. That said, I do feel like the 4 GPUs aren't starved for bandwidth; the more powerful nvidia's get x8 each and the RX 570 gets x2 while the RX 550 still runs fine with x1.
A larger E-ATX server board would have saved me from using PCIe extensions, but again, cost. I would have also preferred to go with 128gb ram, but I'm only using 90% of the ram now. And I could also leverage the swap file disk, but I'm not over allocating ram. Audio over HDMI did require some tweaking (MSI interrupts) so the audio doesn't get choppy, but I figured that out. I also had to adjust the windows VM real time virtual clock settings in the config to stop it from using more CPU then the VM was actually utilizing:
The PCIe USB card is a huge pain in that sometimes only 2 or 3 of the ports show up and a reboot or two is required to bring back the missing ports. It only happens upon reboot, so that's at least tolerable. With everything powered on and running at idle, the system uses only around 200 watts surprisingly, and about 850+ when all four of us play games.
Another weird issue I'm having is using a docker to backup the array to an online backup service provider. The docker invokes many threads in the underlining core OS which can cause latency issues on my main VM. I've found a workaround for now (cron job restarting the docker every few hours), but I'd like to figure out how to prevent it.
I don't think I'll ever go back to separate gaming computers especially since I've cabled up the house for the 4 workstations. I absolutely love all the RGB and wall mounted case; it's a wonderful show piece. It also saves power (almost $600/year by my estimates), allows for some upgrades (I will likely swap out GPUs and eventually go to 128gb ram). I would love to add water cooling to my GPU, but it's a low priority. I'll probably do it when I swap it out with my next upgrade.
More importantly, this was the first time I changed out my storage OS to a purely linux solution. The old VM/storage server was running Windows Server 2016 with HyperV VM's and Storage Spaces for the array. Unraid is definitely faster in terms of disk I/O and Dockers are more flexible and easier to configure than VMs. I also really like how recoverable data is if the array breaks, which happens sometimes.
If you are a hard core enthusiast like myself, I recommend this project. It's not for the faint of heart, but it's really not as bad as trying to build systems in early days of bitcoin/altcoin mining. Unraid is a wonderful OS and the forum support is pretty great. Lots of helpful videos from SpaceInvaderOne.
Sunday, May 12, 2019
Powershell Script - MassDownloader - Efficient, Automated, Fault Tolerant, idempotent downloader with real time metrics
So it's been exactly 1 year since I published here. Life has been beyond busy. I'm hoping to get back on the bandwagon of updating my blog, I have dozens of functions and scripts I'd like to publish that I've been building for various needs. I'll also hopefully get some how-to's posted about running windows server 2019 VM host with NAS & other VMs.
Recently, I needed to download a ton of files and I found trying to do it with a browser or wget wasn't going to work. I wanted a robust, lightweight, efficient tool which I could easily add custom logic URL parsing logic to. I wanted it to be idempotent while supporting resumable downloads that run in the background. I decided to leverage the BITS platform which is really the best of breed downloading method on windows.
I also wanted to see real time statistics as it churned through hundreds of downloads. I also wanted the function to run independent of the scrapper function which was populating the download list file with all the URLs to download. This way I had a means of throttling downloads that efficiently took advantage of my full bandwidth without over saturating it with excess overhead and packet loss.
The solution I came up with leaves out some more advanced features like auto throttling based on tcp statistics and I took a stab at calculating estimated total time of completion and I found without better accounting, the results were garbage. I have a few other ideas I might implement in a future version of this, but I figured I'd just publish what I used to download hundreds of files, over 200gb total successfully.
The script also includes a way to stop all downloads should you need to. Just run "Stop-DownloadFiles"
Ran from ISE:
Ran from standard prompt:
Here's the source code:
Recently, I needed to download a ton of files and I found trying to do it with a browser or wget wasn't going to work. I wanted a robust, lightweight, efficient tool which I could easily add custom logic URL parsing logic to. I wanted it to be idempotent while supporting resumable downloads that run in the background. I decided to leverage the BITS platform which is really the best of breed downloading method on windows.
I also wanted to see real time statistics as it churned through hundreds of downloads. I also wanted the function to run independent of the scrapper function which was populating the download list file with all the URLs to download. This way I had a means of throttling downloads that efficiently took advantage of my full bandwidth without over saturating it with excess overhead and packet loss.
The solution I came up with leaves out some more advanced features like auto throttling based on tcp statistics and I took a stab at calculating estimated total time of completion and I found without better accounting, the results were garbage. I have a few other ideas I might implement in a future version of this, but I figured I'd just publish what I used to download hundreds of files, over 200gb total successfully.
The script also includes a way to stop all downloads should you need to. Just run "Stop-DownloadFiles"
Ran from ISE:
Ran from standard prompt:
Here's the source code:
Saturday, May 12, 2018
Powershell Script - Invoke-RemoteShellCommand - Run remote Linux bash commands with output wrapper
Recently, I've been working much more with linux servers and I even challenged myself to run Ubuntu on my primary personal laptop while still doing mostly powershell development. I needed a way to quickly scale powershell core deployment out to servers, so I came up with a little wrapper function which simplifies this task leveraging putty's plink ssh client. Her'e's what I came up with.
Powershell Script - 7zip/unzip powershell native replacement
Last year, I was tasked with creating a replacement for unzip executables in our environment with native powershell/.net extraction function. Because our environment used to heavily rely on both 7z.exe and unzip.exe (code has been updated so it's not anymore), I made a series of functions which mimic a limited set of parameter behavior for these executables. Here's what I came up with.
Thursday, December 14, 2017
State of Net Neutrality - End of Internet Freedom?
Today is a very dark day for internet freedom. With the Federal Communications Commission voting today to repeal the Obama era protection from two years ago, an assault has begun on our freedom of speech. It won't be long before we'll start to see ISPs and telecoms throttle various sites and services.
As time goes on, new campaigns to get customers to switch over to tiered plans will begin and seem enticing at first, offering internet for cheaper prices. But there will be catches, and they will get worse with time. More restrictions and higher priced tiers will become normal, just offering the same internet we currently have now.
Eventually, everyone will be forced to switch over to these new tiered plans, ultimately with ISPs and telecoms making billions more off the public and businesses. This is especially true for the providers who have market monopolies, like the big four cellular networks and the cable companies in the US.
At the same time, censorship will become more obvious, as emboldened and greedy providers become like mobsters, collecting "safety" money from services, websites, and content providers just so their site or service isn't censored or throttled. Free VoIP or chat providers will have to pay ransom money to the providers or no one will be able to get to their services, so they will disappear.
The darkweb as we know it will likely be censored and blocked entirely. A cat and mouse game will begin with hackers and activists trying to circumvent an ever growing firewall controlled by the providers (and likely the government as well) meant maximize profit while allowing propaganda to find it's way into everything we do online.
While this future is very dark indeed, there will be resistance. First, write and call your congressman, they can overturn this vote. The Electronic Frontier Foundation will be the spear tip, with it's staff of digital expert attorneys suing the FCC. Also, I expect decentralized wireless mesh networks groups to see a surge in growth, with many new nodes coming online as the resistance and knowledge spreads.
While EFF and other groups fight the FCC, most people won't have the skills or will be too geographically far to participate in a wireless mesh networking group, there is something that almost everyone can do to combat throttling and censorship by their ISP and wireless provider. They can use a VPN provider to tunnel their data so the providers can't see it. This also prevents them from monitoring your usage (which is something they've been able to do, even before the vote today).
I signed up for PureVPN, 3 years for $69, which is a really good deal. You get to use 5 devices simultaneously, but if you configure your home router, all devices connected to it will be secured and it only counts as one device. The Android and iOS apps works great from my testing, but the best speed is had by the in-browser plug-ins for both chrome and firefox. They have plenty of servers in the US and abroad in other countries too.
So I encourage everyone who believes in net neutrality to donate to EFF, pick a VPN provider and start using it to connect to the internet, and if you are more savvy and have the know-how, join a local wireless mesh networking group and set up your own node. If you have the resources, getting your HAM license will let you operate a HAMnet, which will allow you to legally use radio frequencies that can carry wireless connections hundreds of miles.
I predict it will have to get worse, before it gets better. Meanwhile, the stocks of ISPs & wireless providers will likely grow significantly as they are the real winners today.
As time goes on, new campaigns to get customers to switch over to tiered plans will begin and seem enticing at first, offering internet for cheaper prices. But there will be catches, and they will get worse with time. More restrictions and higher priced tiers will become normal, just offering the same internet we currently have now.
Eventually, everyone will be forced to switch over to these new tiered plans, ultimately with ISPs and telecoms making billions more off the public and businesses. This is especially true for the providers who have market monopolies, like the big four cellular networks and the cable companies in the US.
At the same time, censorship will become more obvious, as emboldened and greedy providers become like mobsters, collecting "safety" money from services, websites, and content providers just so their site or service isn't censored or throttled. Free VoIP or chat providers will have to pay ransom money to the providers or no one will be able to get to their services, so they will disappear.
The darkweb as we know it will likely be censored and blocked entirely. A cat and mouse game will begin with hackers and activists trying to circumvent an ever growing firewall controlled by the providers (and likely the government as well) meant maximize profit while allowing propaganda to find it's way into everything we do online.
While this future is very dark indeed, there will be resistance. First, write and call your congressman, they can overturn this vote. The Electronic Frontier Foundation will be the spear tip, with it's staff of digital expert attorneys suing the FCC. Also, I expect decentralized wireless mesh networks groups to see a surge in growth, with many new nodes coming online as the resistance and knowledge spreads.
While EFF and other groups fight the FCC, most people won't have the skills or will be too geographically far to participate in a wireless mesh networking group, there is something that almost everyone can do to combat throttling and censorship by their ISP and wireless provider. They can use a VPN provider to tunnel their data so the providers can't see it. This also prevents them from monitoring your usage (which is something they've been able to do, even before the vote today).
I signed up for PureVPN, 3 years for $69, which is a really good deal. You get to use 5 devices simultaneously, but if you configure your home router, all devices connected to it will be secured and it only counts as one device. The Android and iOS apps works great from my testing, but the best speed is had by the in-browser plug-ins for both chrome and firefox. They have plenty of servers in the US and abroad in other countries too.
So I encourage everyone who believes in net neutrality to donate to EFF, pick a VPN provider and start using it to connect to the internet, and if you are more savvy and have the know-how, join a local wireless mesh networking group and set up your own node. If you have the resources, getting your HAM license will let you operate a HAMnet, which will allow you to legally use radio frequencies that can carry wireless connections hundreds of miles.
I predict it will have to get worse, before it gets better. Meanwhile, the stocks of ISPs & wireless providers will likely grow significantly as they are the real winners today.
Wednesday, May 17, 2017
Powershell Video - Jeffrey Snover - State of the Union
Very much worth watching, but TLDR; learning Powershell will make you successful, even if you are a linux engineer.
Wednesday, February 1, 2017
Beginner's Guide - Anonymity and Privacy (Part 1)
Given the current political climate, I feel like it's a good time to share some important information on privacy and anonymity. Many parts of the world don't have net neutrality and more corporations and governments are pushing for a more regulated and censored internet. This high level beginner's guide will cover a broad set of technologies and is only cursory, but should be a good starting point for those looking to protect their privacy and anonymity. Convenience vs Anonymity, you lose one for the other. There are three levels of anonymity which I'll cover which are aimed who you wish to remain anonymous from:
- Internet Service Providers (ISP)
- Corporations & Individuals
- Governments
Before I dive into some real tools hackers use, however I'm obliged to state the obvious.
Disclaimer: All information in this post is for academic/informational purposes only. There is no such thing as true anonymity online, only layers of obfuscation. I do not condone any illegal activities online and utilizing these tools will not prevent you from being caught. The dark web is already under surveillance by various private and governmental entities and most activities are closely monitored. Do not try to do any of the following activities on the darkweb (or online in general): buy/sell illegal drugs, weapons, explosives, porn, assassinations, etc. At best, you'll be scammed, at worse, you will get caught and go to jail. Lastly, do not harass, spam, dox, or cyber bully. Just because you can be anonymous, doesn't mean you can't be caught, so don't be a jerk. Lastly, I'm not responsible for anything that happens to you or your systems as a result of using these tools.
With that out of the way, let's talk about what you should use some of these tools for. Getting around censorships, whistleblowing, communicating with political activists, expressing yourself freely in public forums without the fear of being targeted. Really, freedom of expression is my key reason for writing this. An ideal use of anonymity tools would be for someone who works for a government or corporation and wishes to be politically active but isn't allowed to be due to fear of retaliation from their employer or government. I'll include some more use cases as I break down the different levels of anonymity.
ISP
Be it your home cable/DSL or your cellular provider, your ISP can see all of the network traffic you send and receive from the web. This gives them great power and can watch what you do, censor you and parts of the web from you, etc. Do you watch porn online? Your ISP knows every kind of fetish you have.
VPN
Fortunately, it's relatively easy to block your ISP from seeing your internet traffic by using a VPN (Virtual Private Network) provider. These paid services are usually only a few bucks a month and let you secure you encrypt your internet traffic so your ISP can't see what you are doing online.
VPN
Fortunately, it's relatively easy to block your ISP from seeing your internet traffic by using a VPN (Virtual Private Network) provider. These paid services are usually only a few bucks a month and let you secure you encrypt your internet traffic so your ISP can't see what you are doing online.
Using a VPN isn't considered "deep web" since it's just encrypting standard web traffic over single connection. Of course, the VPN provider can see your decrypted traffic however, so really it's shifting trust from your ISP to the VPN provider. They also have a light to moderate impact on your broadband performance as the encryption overhead and extra point of relay adds latency and can affect throughput performance. Generally, it's not noticeable with most online activities like web surfing, video streaming and gaming sometimes are impacted.
VPNs can be configured in two ways: on your device (PC/tablet/phone/etc) or on your router which gives all of your devices access to the VPN. Setting up the later takes some extra know-how; same with setting up phones & tablets. If you are going to configure your router, it's best to configure rules for games and video streaming providers to not use the VPN service. PCs have special software the VPN providers make which makes setting them up a breeze. Each provider has different configurations, pricing, performance throughput, features, etc. Some VPN providers strive to protect your privacy, while others are ran by the NSA directly and give the government even more direct insight into your personal lives!
Here's a few VPN reviews. I know that Private Internet Access works directly with the NSA so they are pretty much a no go. StrongVPN has really great service, but again, they are a US based company and likely is being tapped by the NSA as well. Another note regarding VPNs, they are great at circumventing corporate and even national censorship firewalls. For example, there are providers who specialize in getting around the Great Chinese Firewall. I recommend using a VPN if you do not trust your ISP or need to circumvent censorship.
Corporations & Individuals
Most individuals who target others online to do reconnaissance (aka online stalking) usually get their intel from public records and corporation data collection sources like Spokeo. It's nearly impossible to hide your public records especially if you are a homeowner. Here's a pretty decent article on how to limit your online public record exposure. You can also limit your online exposure from malicious individuals by locking down your social media profiles.
Search Engines
Perhaps the biggest culprit of tracking and logging your internet activities is your search engine. Most of the world uses Google, followed closely by Apple, Yahoo, and Bing (Microsoft). Aside from these companies being compelled to work with the NSA, they also keep logs of your activities and profile you to better display more relevant ads and sell your information to third party companies. If you wish to hide your search engine terms, you need to switch to a privacy commited search engine provider like DuckDuckGo.
Cookies
Hiding your identity from corporations is difficult because most websites use cookies to tag your computer and track you. Search engines, social networks, online shopping, even just viewing an information page like this sends you a cookie. In addition to cookies, simply loading an ad or external resource (like an embedded video, ad, image, etc) gives your IP address and browser information to third party sites which can track you.
If you'd like to stop this, you can do several things to block the loading of external ads and acceptance of cookies. Disabling cookies is one of them, most browsers support this. Using Adblock plus is highly recommended. And the must have is Privacy Badger which stops many forms of tracking.
Proxies
Even if all these measures are taken, without using a VPN, your source IP address is still being presented to sites you surf or services you use and can be tracked and even geolocated. You could use a public proxy server service (free or paid) which will mask your IP, but the proxy provider now can see all of your traffic like VPN providers can, but it's worse since proxies act like the man-in-the-middle and can decrypt secure SSL connections even so I discourage the usage of public proxies and only use my own Privoxy and Squid Proxy servers. The advantage to running your own private servers locally is that you can limit tracking and ads for your whole network. There are also routers you can buy which have such services built-in like AdTrap. To be clear though, running private proxies will not mask your IP address.
Javascript
Another major security hole built into virtually all browsers is Javascript (JS). This is code that runs on your browser and can easily be used to identify you. Unfortunately, many pages require JS to load correctly, so disabling it breaks many pages. That's why I like to use a browser plug-in that has a quick on/off switch and allows for temporary access for a single particular page that I trust.
Deep Web & Tor
If you are trying to hide your IP address without using a VPN provider (which are still very traceable), you need to connect through the an encrypted obfuscation mesh network (aka anonymous network) within the deep web. Tor is the most common and widely used. Installing the Tor browser and connecting to the network greatly slows down web surfing but adds multiple layers of encryption and masking relaying of usually 3-6 servers often in different countries. This will hide your ip from any site you wish to visit, but to be clear, it's very possible to be traced back to your IP given sufficient resources. This is why it's generally accepted that Tor is not robust enough to prevent governments from tracing and intercepting your "anonymized" surfing to standard websites. But this is generally good enough to stop most corporations and individuals from identifying you, assuming you don't give yourself away through your actions.
Deanonymization
Of course I must now talk about deanonymization which can occur in many ways by not following strict rules of surfing anonymously. You lose your anonymity if you log into any of your known public accounts, like facebook, webmail, twitter, google, youtube, apple, etc. Any time you even enter in your username that is tied to any public accounts, you risk exposing your identity. For this reason, once you log into tor, you should always create new accounts and only log into those accounts while connected to tor.
Those account names should never share the same handles or usernames as your other public accounts. You never should use your real email address. If a service requires you to enter in an email address, you can first buy prepaid credit cards (with cash in person) or use bitcoin, and then buy anonymous email addresses from a secure provider like Lavabit, then use that address to register your new masked identity social network accounts. If you need a phone number to receive SMS (text messages), you can also use prepaid cards to purchase an online accessible number. There are many free no signup required providers which you can use as well, most of them have embedded cookies and malware in the ads though, so be careful.
Perhaps most importantly, never ever give away your real name, phone number, address, age, family, friends, hobbies, schools attended, places of work, places visited, even what you drive. Any identifying information can easily be used against you to narrow the search to identify you. Imagine every post you make on any forum, chatroom, or social network is being looked at by a team of investigators trying to figure out who you are. If you leave no clues, you can assume a relative high level of anonymity.
Tails
Even using the Tor browser with JS disabled being smart about what information you share, it's still possible to be tracked. Your Operating System (OS) and computer hardware can give you away and render your system easily traceable. Ultimately, if you are paranoid, the real answer is to change operating systems. Not permanently, but using a live boot OS from a USB stick or DVD that any computer can temporarily load without removing your existing OS and files. The gold standard of anonymous operating systems is Tails. This linux based custom tuned OS leaves no traces once it's rebooted and renders any PC virtually untrackable. It has Tor browser and has a ton of security features built in and enabled by default. Here's a quick video of how to install tails correctly. Note it requires two USB sticks, at least 4gb each.
Conclusion
Once you've figured out your needs for anonymity and privacy based on the kind of activities you wish to hide from whom, selecting adequate technologies to utilize should hopefully be easier now that you've had this crash course. It is giving up convenience for privacy and anonymity. My opinion as of 2/1/2017 is that utilization of anonymous networks for political activism in the US might be a bit overkill, but might advantageous for certain scenarios and offer added peace of mind.
In part 2 I will cover more in depth scenarios, especially around secure communication, physical security, asset protection, wireless networks, political activism, and using non-PC devices like tablets and phones.
Javascript
Another major security hole built into virtually all browsers is Javascript (JS). This is code that runs on your browser and can easily be used to identify you. Unfortunately, many pages require JS to load correctly, so disabling it breaks many pages. That's why I like to use a browser plug-in that has a quick on/off switch and allows for temporary access for a single particular page that I trust.
Deep Web & Tor
If you are trying to hide your IP address without using a VPN provider (which are still very traceable), you need to connect through the an encrypted obfuscation mesh network (aka anonymous network) within the deep web. Tor is the most common and widely used. Installing the Tor browser and connecting to the network greatly slows down web surfing but adds multiple layers of encryption and masking relaying of usually 3-6 servers often in different countries. This will hide your ip from any site you wish to visit, but to be clear, it's very possible to be traced back to your IP given sufficient resources. This is why it's generally accepted that Tor is not robust enough to prevent governments from tracing and intercepting your "anonymized" surfing to standard websites. But this is generally good enough to stop most corporations and individuals from identifying you, assuming you don't give yourself away through your actions.
Deanonymization
Of course I must now talk about deanonymization which can occur in many ways by not following strict rules of surfing anonymously. You lose your anonymity if you log into any of your known public accounts, like facebook, webmail, twitter, google, youtube, apple, etc. Any time you even enter in your username that is tied to any public accounts, you risk exposing your identity. For this reason, once you log into tor, you should always create new accounts and only log into those accounts while connected to tor.
Those account names should never share the same handles or usernames as your other public accounts. You never should use your real email address. If a service requires you to enter in an email address, you can first buy prepaid credit cards (with cash in person) or use bitcoin, and then buy anonymous email addresses from a secure provider like Lavabit, then use that address to register your new masked identity social network accounts. If you need a phone number to receive SMS (text messages), you can also use prepaid cards to purchase an online accessible number. There are many free no signup required providers which you can use as well, most of them have embedded cookies and malware in the ads though, so be careful.
Perhaps most importantly, never ever give away your real name, phone number, address, age, family, friends, hobbies, schools attended, places of work, places visited, even what you drive. Any identifying information can easily be used against you to narrow the search to identify you. Imagine every post you make on any forum, chatroom, or social network is being looked at by a team of investigators trying to figure out who you are. If you leave no clues, you can assume a relative high level of anonymity.
Tails
Even using the Tor browser with JS disabled being smart about what information you share, it's still possible to be tracked. Your Operating System (OS) and computer hardware can give you away and render your system easily traceable. Ultimately, if you are paranoid, the real answer is to change operating systems. Not permanently, but using a live boot OS from a USB stick or DVD that any computer can temporarily load without removing your existing OS and files. The gold standard of anonymous operating systems is Tails. This linux based custom tuned OS leaves no traces once it's rebooted and renders any PC virtually untrackable. It has Tor browser and has a ton of security features built in and enabled by default. Here's a quick video of how to install tails correctly. Note it requires two USB sticks, at least 4gb each.
If you run Tails, connect to Tor, and follow the rules of protecting your anonymity, you can remain undiscoverable to most of the world, minus governments (and potentially very large mega corporations). There are a few more gotchas regarding tails usage.
Governments
For the Edward Snowdens and other whistleblowers of the world, even sticking to just the deep web isn't enough. Communicating using the dark web is the ultimate way remain anonymous. The dark web isn't like the normal web in that you can't access normal sites. Tor is only part dark web as you can still surf normal (surface web) sites; it's technically a hybrid web. The Tor protocol and network has been shown to be hackable by those entities with enough resources such including nation states and mega corporations. Therefore those most paranoid use a more advanced protocol/network that is purely dark web.
I2P
Built with clear advantages over Tor, i2p is what most hackers use for many activities. While it has many clear advantages, it takes time to connect and really works better as a persistently connected dedicated system. It used to be bundled into Tails, but was disabled in more recent builds frankly because most users of tails don't need to be on it. You can turn it on however during the boot up of tails.
Hidden dark web sites on both Tor and i2p are likely monitored and indexed by private security research firms and governments, so really, only direct messaging and secure encrypted email are your two ways of communicating completely securely. Forums, IRC, and various chat rooms are also often logged and monitored, so unless you trust a particular forum or IRC server (like one you are running yourself), assume someone is able to read everything you type.
Freenet
There are three anonymous networks; we already covered Tor and i2p, the last is Freenet. It's the oldest and has it's advantages. It's best used to combat censorship for publishing information that would be potentially fatal to the author if they could be identified.
For these reasons, I do not recommend novices run either i2p nor Freenet until they are experienced with Tor and know what they are doing; browsing them is not for the faint of heart. Despite the high level of monitoring, terrorist organizations and hacktivist groups mostly reside on these more advanced anonymous networks and it's obviously better not to get mixed up with either.
Conclusion
Once you've figured out your needs for anonymity and privacy based on the kind of activities you wish to hide from whom, selecting adequate technologies to utilize should hopefully be easier now that you've had this crash course. It is giving up convenience for privacy and anonymity. My opinion as of 2/1/2017 is that utilization of anonymous networks for political activism in the US might be a bit overkill, but might advantageous for certain scenarios and offer added peace of mind.
In part 2 I will cover more in depth scenarios, especially around secure communication, physical security, asset protection, wireless networks, political activism, and using non-PC devices like tablets and phones.
Friday, February 26, 2016
Powershell Quick Script - Wrap any powershell script into a batch file
It's been a number of months since I've written anything here which is unfortunate since I've been writing lots of nifty things in powershell for work. So this should be the first of many scripts which need sharing.
I recently ran into an issue where I had to make a single file script which could easily be ran by a simple user who could just double-click on it and it would just work, regardless of which version of powershell they had or if their Execution Policy was set correctly. The solution I went for was the embed a powershell script inside of a batch file which is more universally accepted on legacy systems and by most windows admins.
There are several solutions I found which involve encoding the powershell script into a long base64 string and feeding it into powershell.exe, but this has a size limitation which larger scripts easily hit. Another solution I saw was to strip away special formatting, comments, certain characters, and then wrap it in curly brackets "{}" and again feed it to powershell.exe as a command. This too also suffers from the max length problem as well as requires special editing to make it work.
My solution is much simpler, suffers from no length constraints and really has one drawback which matters if you are watching the error output stream. The solution is very simple. Just add the following line before your powershell script:
Then add the following after your script:
Finally, save your script as a .cmd or .bat file extension. What this will do is cause your code to be executed as batch, which will then copy itself to the temp location and add the extension ps1 and then feed that into powershell.exe while bypassing the execution policy ofthe machine. Once the powershell code finishes, it will exit, returning back to the batch wrapper which will then clean up the temp ps1 file and return the error code from powershell exit. I highly suggest your powershell code has it's own built in exits. Most automation systems are specifically looking for exit codes and you want exit with 0 if it's successful or another number if it's not. If you do don't want the script to close the window when it's done, replace the last line "exit %el%" with "pause". Here's an example of the complete file:
I recently ran into an issue where I had to make a single file script which could easily be ran by a simple user who could just double-click on it and it would just work, regardless of which version of powershell they had or if their Execution Policy was set correctly. The solution I went for was the embed a powershell script inside of a batch file which is more universally accepted on legacy systems and by most windows admins.
There are several solutions I found which involve encoding the powershell script into a long base64 string and feeding it into powershell.exe, but this has a size limitation which larger scripts easily hit. Another solution I saw was to strip away special formatting, comments, certain characters, and then wrap it in curly brackets "{}" and again feed it to powershell.exe as a command. This too also suffers from the max length problem as well as requires special editing to make it work.
My solution is much simpler, suffers from no length constraints and really has one drawback which matters if you are watching the error output stream. The solution is very simple. Just add the following line before your powershell script:
goto ExecutePowershell cls
Then add the following after your script:
exit #end of powershell code - batch code now: :ExecutePowershell echo off set filename=%temp%\Tempscript.ps1 copy /y %0 %filename% echo NOTE - you will see error output in the error stream about 'goto' - this is expected and can be ignored. cls powershell -ExecutionPolicy unrestricted -file %filename% %* set /a el=%errorlevel% del %filename% exit %el%
Finally, save your script as a .cmd or .bat file extension. What this will do is cause your code to be executed as batch, which will then copy itself to the temp location and add the extension ps1 and then feed that into powershell.exe while bypassing the execution policy ofthe machine. Once the powershell code finishes, it will exit, returning back to the batch wrapper which will then clean up the temp ps1 file and return the error code from powershell exit. I highly suggest your powershell code has it's own built in exits. Most automation systems are specifically looking for exit codes and you want exit with 0 if it's successful or another number if it's not. If you do don't want the script to close the window when it's done, replace the last line "exit %el%" with "pause". Here's an example of the complete file:
Monday, October 12, 2015
Powershell Script - Set-UserPassword - Remotely sets local account passwords
Here's a great script to change passwords in bulk on many servers. I've added verbose and error output for logging purposes as well as time/date stamping for when actual password setting occurs.
PS C:\> 'server1','server2','Badserver' | Set-UserPassword -Username 'test' -Password 'pass123' -Verbose VERBOSE: Processing server 'server1'... VERBOSE: Connected to server 'server1'... VERBOSE: Retrieved user objects from server 'server1'... VERBOSE: Found user 'test' from server 'server1'... 10/12/2015 14:41:31 - Successfully changed password for user 'test' on server 'server1' VERBOSE: Processing server 'server2'... VERBOSE: Connected to server 'server2'... VERBOSE: Retrieved user objects from server 'server2'... WARNING: ERROR: No user 'test' on server 'server2' VERBOSE: Processing server 'Badserver'... WARNING: ERROR: Failed to connect to server 'Badserver' PS C:\>
Subscribe to:
Posts (Atom)