Subject: Using serial console as a poor mans IP kvm?

From: Jarle Aase <[email-masked]>
Date: Thu, 8 Sep 2016 22:26:59 +0300

I want to set up a few servers at home. Unfortunately, as I live in 
Bulgaria at the moment, the electric power is gone pretty often for 
longer periods than my UPS'es can deal with. So my servers will have to 
be started at least a few times every quarter.

Another challenge with living in Bulgaria is that there is no law or 
order. The Police is just a branch of the Mafia. I need to protect the 
data on the servers with full disk encryption in case they are stolen.

That means that I need to reboot the servers relatively often, and 
provide the luks passwords every time. Some times I am far away when 
this happens. I have been considering Supermicro motherboards with built 
in support for remote management - or old KVM IP switches from Ebay. The 
problem with Supermicro is that it's expensive and difficult to get the 
RAM required for their recent Skylake boards. The problem with Ebay is 
that few suppliers ships to Bulgaria, and getting anything trough the 
custom's here takes a whole day. Then there is the question if the 
device works at all...

So I'm thinking about serial consoles. My gateway router will reboot 
after an outage, and it can act as a VPN endpoint. So I can access IP 
devices. With a rasberry pi and some relays, I can probably trigger a 
cold reboot whenever I need to. If I could log on to the grub console on 
the servers over a serial link, that's all I need, really.

Does anyone here have any experience with remote control with Debian 
boxes over serial? Will it work reliable?

Thanks in advance.

Jarle

---

From: Jarle Aase <[email-masked]>
Date: Fri, 9 Sep 2016 21:03:33 +0300

Hi,

I was just about to order some usb2serial hardware when I read this. 
Your suggestion will give fewer "moving parts" and is actually very 
simple to implement. I will loose the ability to do a cold boot, but it 
will probably not matter too much in my case, at least not for now.

I'll try it when I get the first server assembled. Thanks a lot!

Jarle

Den 09. sep. 2016 10:31, skrev [email-masked]:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Thu, Sep 08, 2016 at 10:26:59PM +0300, Jarle Aase wrote:
>> longer periods than my UPS'es can deal with. So my servers will have
>> to be started at least a few times every quarter.
> [...]
>
>> this happens [...]
> An interesting alternative to the serial console thing is baking
> in an SSH server into the initramfs. There are small SSH servers
> built for that, like Dropbear.
>
> Upside is that you don't need any additional hardware and it's
> pretty well integrated into Debian. Downside is that you need
> BIOS, the bootloader and initramfs working (with the serial you
> at least get a chance to fix the bootloader remotely).
>
> https://packages.debian.org/sid/dropbear-initramfs
> https://wiki.debian.org/RescueInitramfs
> https://projectgus.com/2013/05/encrypted-rootfs-over-ssh-with-debian-wheezy/
>
> Might be worth a try.
>
> Regards
> - -- t
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iEYEARECAAYFAlfSZVMACgkQBcgs9XrR2kZGNgCfZhrMlouUceQSVJgzimE+b2YG
> GokAn0PpEqnw2lgmFiGTu554OQtpt9Wa
> =AKQd
> -----END PGP SIGNATURE-----
>

---

From: Lars Noodén <[email-masked]>
Date: Fri, 9 Sep 2016 08:46:11 +0300

On 09/08/2016 10:26 PM, Jarle Aase wrote:
>...

Quite a while back (Etch) I had some Debian machines running via serial
console.  As far as I know everything should still work just as nicely
over serial console.  From what I recall, you'll have to set console
settings several places in the system to cover all contingencies for
booting and recovery.

I've used USB-to-serial adapters with the Prolific chipset.  They've
worked fine for me, in various models.  (I haven't tried FTDI and am
suspicious of them.)  There are also specialized PCI and PCIe serial
console servers which add 4 or 8 extra serial ports to a machine.  But
if you're going to run everything off of a single rpi then a
USB-to-serial adapter is the way to go.  There are ones that go USB to 4
or 8 serial ports, but they are hard to find affordably any more.

About the power relays, I did that before and had a lot of help to make
some custom ones, nothing being on the market back then.  I found
someone with skill to build a custom setup that worked over GPIO.
However, nowadays there are several devices that look interesting.  One
pre-made series that caught my eye a few weeks ago was this one:

	https://unipi.technology/shop/

However, I have not evaluated any units so that is just to point to
what's on the market and not any endorsement.  You'll need to wire plugs
and such, too, and I can't see any fuses on those units.

Regards,
Lars

---

From: Tixy <[email-masked]>
Date: Fri, 09 Sep 2016 08:15:37 +0100

On Fri, 2016-09-09 at 08:46 +0300, Lars Noodén wrote:
> suspicious of them.)

And my experience is the opposite. I have genuine (there's apparently a
lot of fakes) FTDI devices in pretty much daily use for many years
without problems. This is using ser2net on a local network for accessing
serial consoles on ARM based development boards. ser2net will be
insecure telnet or raw port forwarding but if it's not exposed to the
internet and you can ssh tunnel into the local network then that's a lot
better. I've done that method for carrying on working with my boards
whilst across the other side of the world. Of course, a means of power
cycling devices is also essential.

-- 
Tixy

---

From: Eike Lantzsch <[email-masked]>
Date: Fri, 09 Sep 2016 08:44:51 -0400

> cycling devices is also essential.

I second that. I had many weird problems with PL2303 but never any on any OS* 
with FTDI FT232 chips.

[*] Debian-Linux, OpenBSD, OSX and MS-Windows 
-- 
Eike Lantzsch ZP6CGE

---

From: Don Armstrong <[email-masked]>
Date: Thu, 8 Sep 2016 17:37:55 -0500

On Thu, 08 Sep 2016, Jarle Aase wrote:
> Does anyone here have any experience with remote control with Debian boxes
> over serial? Will it work reliable?

It's fairly reliable; I actually prefer it to using KVM in almost all
cases. You just need to get it configured properly in grub, the bios,
and the kernel, and you should be fine.

That said, providing LUKS input over the wire is always going to be
problematic unless you have known secured links to the terminal. [But
maybe you'll know if the government has done this to you.]

-- 
Don Armstrong                      https://www.donarmstrong.com

Whatever you do will be insignificant, but it is very important that
you do it.
 -- Mohandas Karamchand Gandhi

---

From: "Neal P. Murphy" <[email-masked]>
Date: Thu, 8 Sep 2016 16:38:03 -0400

On Thu, 8 Sep 2016 22:26:59 +0300

> boxes over serial? Will it work reliable?

Generally speaking....

I haven't used a serial console on Debian in particular. I do it on a Linux-based system I maintain; serial console works very well, provided you remember the main differences: it's not a VESA console, it doesn't know about CTRL-ALT-DEL, and you may see no output until grub starts (unless the BIOS/firmware can do serial console). The system's terminal type may not match the emulator's type and the display may be somewhat garbled; the use of serial ports for interactive use has declined greatly over the years, as has conformance to serial terminal protocols. Also, a <BREAK> starts SysRq.

I suspect grub should work with a serial console; I've never tried it with the new grub. I wrote a 6-line patch for grub legacy (which I use for my system) that allows one to use either VESA or serial console, or choose one with a keystroke. I've been using it for 3-4 years now without trouble.

To tell linux to use a serial console, connect a terminal (or emulator) to ttyS0 (COMa) on the server and set it to 115200-8-N-1, let grub start, then edit the boot entry and add the option "console=ttys0,115200" to the kernel command line, and boot. 

Your main problem will be operating the servers' reset switches should it be necessary. I think this is usually done by having the DCD, DSR, RI, or CTS line of the TIA-232 port close-then-open a relay that acts as the reset switch. (But your idea of using an rPI would work, too.)

Of course, the encryption key reader has to work on a serial port as well.

What would be interesting is if there were a 'VGA scanner' for the rPI so it could send you the screen changes, say, ten times a second. (And there are some USB devices that may work.) And if the rPI had a client USB port (so it could act like a keyboard) you would be able to see the 'monitor' and type on the 'keyboard'.

---

From: Miles Fidelman <[email-masked]>
Date: Thu, 8 Sep 2016 16:23:45 -0400

On 9/8/16 3:26 PM, Jarle Aase wrote:

> this happens. I have been considering Supermicro motherboards with 
> built in support for remote management - or old KVM IP switches from 
> Ebay. The problem with Supermicro is that it's expensive and difficult 
> to get the RAM required for their recent Skylake boards. The problem 
> with Ebay is that few suppliers ships to Bulgaria, and getting 
> anything trough the custom's here takes a whole day. Then there is the 
> question if the device works at all...
>
> cold reboot whenever I need to. If I could log on to the grub console 
> on the servers over a serial link, that's all I need, really.
>

It sort of works.

I've done this two ways:

1.  External serial-to-ethernet box.  The external box turns out to be 
somewhat flakey, and a security hole (unpatched embedded linux with some 
vulnerabilities, and it needs to be rekeyed annually, but that doesn't 
actually work very smoothly).

2. Supermicro IPMI board:  Sometimes works, sometimes simply doesn't 
respond - usually when one needs it most.

In both cases, unless you layer a VPN on top of them, they are really 
nasty security holes.  I've ended up resorting to the old "call the data 
center and have a human push the button" - but that doesn't sound like 
it applies to your situation.

Good luck finding a solution.

Miles Fidelman

-- 
In theory, there is no difference between theory and practice.
In practice, there is.  .... Yogi Berra

---

From: Dan Ritter <[email-masked]>
Date: Thu, 8 Sep 2016 15:52:43 -0400

On Thu, Sep 08, 2016 at 10:26:59PM +0300, Jarle Aase wrote:
> I want to set up a few servers at home. Unfortunately, as I live in Bulgaria
> at the moment, the electric power is gone pretty often for longer periods
> than my UPS'es can deal with. So my servers will have to be started at least
> a few times every quarter.
> 
> Another challenge with living in Bulgaria is that there is no law or order.
> The Police is just a branch of the Mafia. I need to protect the data on the
> servers with full disk encryption in case they are stolen.
> 
> That means that I need to reboot the servers relatively often, and provide
> the luks passwords every time. Some times I am far away when this happens. I
> have been considering Supermicro motherboards with built in support for
> remote management - or old KVM IP switches from Ebay. The problem with
> Supermicro is that it's expensive and difficult to get the RAM required for
> their recent Skylake boards. The problem with Ebay is that few suppliers
> ships to Bulgaria, and getting anything trough the custom's here takes a
> whole day. Then there is the question if the device works at all...
> 
> So I'm thinking about serial consoles. My gateway router will reboot after
> an outage, and it can act as a VPN endpoint. So I can access IP devices.
> With a rasberry pi and some relays, I can probably trigger a cold reboot
> whenever I need to. If I could log on to the grub console on the servers
> over a serial link, that's all I need, really.
> 

We use serial consoles on Debian and Oracle Linux boxes all the
time, and have done so for more than a decade. They are more
dependable than anything else -- once you have set them up and
tested them through a full reboot cycle.

There are relatively expensive, but compact, devices that will
do both serial access and power switching. 8 of each is a common
configuration, as is 16. We like WTI boxes.

I am somewhat suspicious of USB-to-serial adapters in general,
but they are cheap and you can hook up lots at once through a
USB hub. You will probably want to test several brands in order
to find something reliable.

Incidentally, there are very few applications in which a Skylake
processor will be notably faster than the previous generation of 
Broadwells -- and Broadwells can use DDR3. You might save a lot 
of money and get built-in KVMs that way.

-dsr-

---

From: Glenn English <[email-masked]>
Date: Thu, 8 Sep 2016 15:43:31 -0600

For remote access, the RPi sounds like a good idea to me. I've had one on the 'Net for several years, doing things not requiring major CPU power. It's on my UPS, and it's had no reliability problems.

A relatively small dedicated UPS would likely keep your border router and an RPi going for quite a while.

An RPi is just a small Debian box, so you could reliably get to it over a VPN. Then, with a little innovative routing, you could look around the net(s) and see what's going on. 

With a relay board on the RPi's GPIO connector, you could turn things back on, if necessary. No RS-232, new motherboards, or KVM anythings required.

Assuming you can write a little Python...

No?

-- 
Glenn English

---

From: "Neal P. Murphy" <[email-masked]>
Date: Thu, 8 Sep 2016 17:58:23 -0400

On Thu, 8 Sep 2016 15:43:31 -0600

> With a relay board on the RPi's GPIO connector, you could turn things back on, if necessary. No RS-232, new motherboards, or KVM anythings required.

Except for entering the encryption keys, and seeing when to enter them....