# Developing over SSH

With the previous part's VM still running, let's try connecting to our machine over SSH.

## Network addresses, loopback and IP nets

Normally, to connect to a machine, you'd find its IP address. On Linux, a decade ago, you would've used ifconfig. Nowadays you can use ip addr:

The loopback interface (lo) is local, so it's not useful to reach the box from the outside: you can see it can be accessed over IPv4 at address 127.0.0.1 but not just! What we're reading here is 127.0.0.1/8, which corresponds to the range 127.0.0.1 - 127.255.255.255

This is useful, as it lets us have services that listen on different loopback IPv4 addresses, and apply different routing rules to different 127.0.0.1/8 addresses.

For IPv6 however, ::1/128 indicates a single address, ::1 (which is the short form of 0:0:0:0:0:0:0:1):

We can use any of these addresses to connect to our VM over SSH, from our VM.

...but that's not very useful. We already had terminal access there, through the VirtualBox GUI.

What we'd like is to connect to it from the outside. We saw earlier that we had another network interface, enp0s3. These used to be called something like eth0 (for ethernet), but now the scheme is:

• en = EtherNet
• p0 = Bus number 0
• s3 = Slot number 3

You can read more about the new naming scheme if you're curious.

My VM had an IPv4 address of 10.0.2.15/24 (with a range of 10.0.2.1-10.0.2.254, according to an IP subnet calculator such as this one), and I am able to ping that address from the VM (from the "guest"):

But not from the host (still Windows 11, here with the wonderful Windows Terminal and the latest PowerShell 7):

And that is working as designed. The guest is on its own, separate network. In fact, my other VM also has an address of 10.0.2.15 on its own network.

That's why we set up port forwarding in the previous part: it has VirtualBox listen on port 2223, and forward connections to the VM's private network.

We can make sure it's set up correctly with PowerShell (this is Windows-specific, but there's equivalents on Linux and macOS that are a web search away):

And sure enough, it accepts TCP connections on that port:

## Connecting over SSH

We can connect to it with ssh (again, it's an optional Windows component now, it ships with macOS by default, and look to your Linux distribution's package manager search functionality) like so:

Shell session
$ssh amos@127.0.0.1 -p 2223 The authenticity of host '[127.0.0.1]:2223 ([127.0.0.1]:2223)' can't be established. ED25519 key fingerprint is SHA256:zwxa3nLGjzTOLg2m3+jN91fpMH7BWVJkow89tYcygtE. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])?  It asks us whether we want to trust that server key, and we do, so we can type "yes" and press enter: Shell session $ yes
Warning: Permanently added '[127.0.0.1]:2223' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-53-generic x86_64)

* Documentation:  https://help.ubuntu.com
* Management:     https://landscape.canonical.com

System information as of Wed Nov 23 10:48:00 AM UTC 2022

Usage of /:   14.3% of 47.93GB   Users logged in:         1
Memory usage: 1%                 IPv4 address for enp0s3: 10.0.2.15
Swap usage:   0%

* Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
just raised the bar for easy, resilient and secure K8s cluster deployment.

https://ubuntu.com/engage/secure-kubernetes-at-the-edge

0 updates can be applied immediately.

Last login: Wed Nov 23 09:59:30 2022 from ::1
amos@miles:~$whoami amos  (Press Ctrl-D to exit out of the SSH session. This indicates "end of file"). However, it's not very convenient to have to type that whole ssh amos@127.0.0.1 -p 2223 command every time, so we can add an entry to the host's ~/.ssh/config config file instead. Cool bear's hot tip ~ denotes your home directory, and it'll be familiar to you if you're on Linux or macOS. It also works in PowerShell, which means you can do something like code ~/.ssh/config to edit it. If the ~/.ssh directory doesn't exist yet, mkdir ~/.ssh should work on all three OSes, again — there's a Windows alias for it. The config section looks like: # in ~/.ssh/config Host miles HostName 127.0.0.1 Port 2223 User amos ForwardAgent yes  Mhh what's that ForwardAgent thingy doing here? Well, you know how, from our host operating system, we have an SSH keypair, added to an SSH agent, that we use to authenticate with GitHub and connect to our VM? Yes? I mean... oh, yes okay, Windows ships with ssh-agent too nowadays, or you can use 1Password for this, so yes I guess so? Yes it does: Shell session $ echo $env:OS Windows_NT$ ssh-add -l
2048 SHA256:4txMCM8iFJaOmrB7qVAMNwSdy7KUbVvqMrcBdLd/VXo teleport:fasterthanlime (RSA-CERT)
4096 SHA256:r8YfVEk6CVCO9S4ykJZew2qM+cSR/nFWLs8Ovul6hMk amos@tails (RSA)


Well look at this: if we use the VirtualBox GUI to access the VM and try to talk to GitHub, it doesn't know who we are:

But if we connect from our host to our guest, then from that SSH session, try to talk to GitHub, it does!

Shell session
$~ ❯ ssh miles Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-53-generic x86_64) (cut: spammy Ubuntu banner) Last login: Wed Nov 23 11:00:26 2022 from 10.0.2.2 amos@miles:~$ ssh -T git@github.com
Hi fasterthanlime! You've successfully authenticated, but GitHub does not provide shell access.
amos@miles:~$cat hello.txt I was made inside the guest  And then copy it from the guest to the host. This is run from the host: Shell session ~ ❯ scp miles:~/hello.txt . hello.txt 100% 28 0.0KB/s 00:00 ~ took 4s ❯ cat hello.txt I was made inside the guest  That's it. scp stands for "secure copy" and uses ssh for data transfer, uses the same authentication mechanisms, etc. You can also copy files to the guest, with scp ./some-host-file.txt miles:~/path-on-guest. ## Installing a Rust toolchain From the guest, let's install a rust toolchain with rustup: Shell session amos@miles:~$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Welcome to Rust!

programming language, and its package manager, Cargo.

(cut)

Current installation options:

default host triple: x86_64-unknown-linux-gnu
default toolchain: stable (default)
profile: default
modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2022-11-03, rust version 1.65.0 (897e37553 2022-11-02)
(cut)
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

stable-x86_64-unknown-linux-gnu installed - rustc 1.65.0 (897e37553 2022-11-02)

Rust is installed now. Great!

To get started you may need to restart your current shell.
Cargo's bin directory ($HOME/.cargo/bin). To configure your current shell, run: source "$HOME/.cargo/env"


Like the installer just said, we can source ~/.cargo/env to get "cargo" in our $PATH. Mh? Well, it currently isn't: Shell session amos@miles:~$ cargo
sudo snap install rustup  # version 1.24.3, or
sudo apt  install cargo   # version 0.60.0ubuntu1-0ubuntu1~22.04.1
See 'snap info rustup' for additional versions.
amos@miles:~$which cargo amos@miles:~$


But if we source ~/.cargo/env, then it is!

Shell session
amos@miles:~$source ~/.cargo/env amos@miles:~$ which cargo
/home/amos/.cargo/bin/cargo
amos@miles:~$cargo -vV cargo 1.65.0 (4bc8f24d3 2022-10-20) release: 1.65.0 commit-hash: 4bc8f24d3e899462e43621aab981f6383a370365 commit-date: 2022-10-20 host: x86_64-unknown-linux-gnu libgit2: 1.5.0 (sys:0.15.0 vendored) libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q) os: Ubuntu 22.04 (jammy) [64-bit]  ## Installing a web server Now, we can use cargo to install a web server, like, say, sfz: Shell session amos@miles:~$ cargo install sfz
Updating crates.io index
Fetch [===>                     ]  16.54%, 11.34MiB/s


...because it's the first time we need the crates.io index, it has to download the whole thing. This might take a while, even with fast internet.

Wait, it has to download an index of all crates ever published on crates.io?

For now, yes. There's a sparse_index RFC I like a lot, but we're on a stable version of cargo right now, so we don't get to use it, yet.

Eventually, our command... fails with this:

Shell session
  Downloaded hyper v0.14.23
Downloaded 130 crates (9.0 MB) in 0.67s (largest was brotli at 1.4 MB)
Compiling version_check v0.9.4
Compiling proc-macro2 v1.0.47
error: linker cc not found
|
= note: No such file or directory (os error 2)

error: could not compile proc-macro2 due to previous error
error: failed to compile sfz v0.7.1, intermediate artifacts can be found at /tmp/cargo-installsTSzh7


Oh no.

Never fear, we just need to ask our friendly neighborhood package manager to install a couple things.

It says cc but really, we want gcc here.

Shell session
amos@miles:~$sudo apt install gcc [sudo] password for amos: Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages were automatically installed and are no longer required: libflashrom1 libftdi1-2 Use 'sudo apt autoremove' to remove them. The following additional packages will be installed: cpp cpp-11 fontconfig-config fonts-dejavu-core gcc-11 gcc-11-base libasan6 libatomic1 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0 libfontconfig1 libgcc-11-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0 libmpc3 libnsl-dev libquadmath0 libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev manpages-dev rpcsvc-proto Suggested packages: cpp-doc gcc-11-locales gcc-multilib make autoconf automake libtool flex bison gdb gcc-doc gcc-11-multilib gcc-11-doc glibc-doc libgd-tools The following NEW packages will be installed: cpp cpp-11 fontconfig-config fonts-dejavu-core gcc gcc-11 gcc-11-base libasan6 libatomic1 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0 libfontconfig1 libgcc-11-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0 libmpc3 libnsl-dev libquadmath0 libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev manpages-dev rpcsvc-proto 0 upgraded, 37 newly installed, 0 to remove and 4 not upgraded. Need to get 48.6 MB of archives. After this operation, 152 MB of additional disk space will be used. Do you want to continue? [Y/n]  Since Y (yes) is capitalized here, we can simply press "Enter" to continue. And just like that we have a cc command in $PATH:

Shell session
amos@miles:~$which cc /usr/bin/cc  It's.. a symbolic link: Shell session amos@miles:~$ ls -lhA /usr/bin/cc
lrwxrwxrwx 1 root root 20 Nov 23 11:19 /usr/bin/cc -> /etc/alternatives/cc


...which is managed by update-alternatives, a very Debian thing:

Shell session
$update-alternatives --list cc /usr/bin/gcc  And, yeah. It's just gcc. Let's try again! Shell session $ cargo install sfz
(cut)
Compiling qstring v0.7.2
Compiling tokio-util v0.7.4
Compiling sfz v0.7.1
Finished release [optimized] target(s) in 2m 02s
Installing /home/amos/.cargo/bin/sfz
Installed package sfz v0.7.1 (executable sfz)


And this time it works!

It's in $PATH, too, because we sourced ~/.cargo/env earlier: Shell session amos@miles:~$ which sfz
/home/amos/.cargo/bin/sfz


Let's run it:

Shell session
amos@miles:~$sfz Files served on http://127.0.0.1:5000  Cool bear's hot tip You can exit out of this with Ctrl+C, which sends an interrupt signal (SIGINT) to sfz. The difference with "exiting out of an SSH session" is that sfz isn't reading the standard input, so it won't detect an EOF there. But the default signal handler will quit the (otherwise long-running) program. And, from another terminal also in the guest, we can see that it does work: Shell session amos@miles:~$ curl -I 0:5000
HTTP/1.1 200 OK
server: sfz/0.7.1
accept-ranges: bytes
content-type: text/html; charset=utf-8
content-length: 2166
date: Wed, 23 Nov 2022 11:26:46 GMT


## Accessing our web server from the host

However, the question is... how are we going to access all this from the host?

Well, for starters, it's listening on 127.0.0.1, which is definitely not accessible from the outside.

You're right — we can have sfz listen on 0.0.0.0 instead, which means "all network interfaces".

Shell session
amos@miles:~$sfz -b 0.0.0.0 Files served on http://0.0.0.0:5000  ...but it won't help much. Remember: the VM's enp0s3 interface is connected to a network that's private to this VM. We can't reach it even from the host: Shell session $ curl -v -I http://10.0.2.15:5000
*   Trying 10.0.2.15:5000...
* connect to 10.0.2.15 port 5000 failed: Timed out
* Failed to connect to 10.0.2.15 port 5000 after 21001 ms: Timed out
* Closing connection 0
curl: (28) Failed to connect to 10.0.2.15 port 5000 after 21001 ms: Timed out


There's a couple options here: we could add another port forwarding rule in the VirtualBox network configuration. Or we could forward that port over SSH!

Let's exit out of sfz with Ctrl+C and out of our current SSH session with Ctrl+D:

Shell session
amos@miles:~$sfz -b 0.0.0.0 Files served on http://0.0.0.0:5000 ^C amos@miles:~$
logout
Connection to 127.0.0.1 closed.



And then log back into our VM, but asking SSH to forward something for us:

Shell session
$ssh -L 5000:localhost:5000 miles (cut) Last login: Wed Nov 23 11:26:31 2022 from 10.0.2.2 amos@miles:~$


This has the ssh client listen on port 5000 on the host, and whenever it accepts connections on there, it forwards them to localhost:5000 from the perspective of the guest.

Cool bear's hot tip

It seems odd at first that we have to specify localhost in there, because for simple scenarios, it's what we want. But imagine we're connecting to some host that's part of a private network, and we're simply using it as a relay to reach a third host.

We might end up doing something like ssh -L 5000:third-host:5000 relay-host.

Now, we can run sfz again:

Shell session
$amos@miles:~$ sfz
Files served on http://127.0.0.1:5000



And access it from our.. host OS!

It's confusing because "host" means "some device on a network" and also "the operating system that runs the VM hypervisor, in which the guest OS is executed".

In this case, I mean the latter.

Shell session
~
❯ curl -I http://localhost:5000
HTTP/1.1 200 OK
server: sfz/0.7.1
accept-ranges: bytes
content-type: text/html; charset=utf-8
content-length: 2166
date: Wed, 23 Nov 2022 11:42:27 GMT



In fact, we can open it in a browser:

In fact, here's our hello.txt file. Hello to you too!

## VSCode's Remote SSH feature

You now know enough to develop server applications remotely on your Linux VM.

However, because this is my series and I do what I want, I'll show you why it's nice to use VSCode for this.

First off, you'll need to download / install VSCode, through the official website or some package manager.

Cool bear's hot tip

There's open-source distributions of VSCode like VSCodium (similar to Chrome -> Chromium), but they might not have the extensions we want, so, if you pick that, you're on your own.

Make sure you have the "Remote - SSH" extension installed (from the command menu, pick "Install extensions", or click on the icon from the leftmost bar that looks like a package: a grid of four squares, one of them trying to break free):

(There's also "Remote Development" group that includes "Remote - SSH", "Remote - WSL" and "Dev Containers", if you're interested in that).

Once the extension is installed, you can open the command palette (F1 on Linux/Windows, Shift+Cmd+P on macOS), start typing "Connec...", and pick "Remote-SSH: Connect to host"

Next up, VSCode will read your ~/.ssh/config and suggest picking one of the hosts it found there:

Picking "miles" opens a new window, with another prompt asking us to select the platform, let's pick Linux:

VSCode will then download its server component on the remote machine (our guest):

And we're good to go!

We can open the built-in terminal with CmdOrCtrl+Backquote, or clicking this little icon, then selecting the Terminal tab:

And tada, we have a terminal on our guest VM:

From there, we can run sfz again:

Cool bear's hot tip

If you get "Server error: error creating server listener: Address already in use (os error 98)", that probably means you still have sfz running in another terminal session.

Make sure to exit out of it first! (If everything else fails, you can try killall sfz).

Shell session
amos@miles:~\$ sfz
Files served on http://127.0.0.1:5000



Now let's try opening it in a browser...

And it works!

Wait, didn't we forget something?

Forget something? Like what?

Well, when we did it with SSH in command line, we had to forward port 5000 from the host to the guest. Is that still active somehow?

Oh, no, VSCode just detected that something inside the VM is listening on port 5000, and it set up port forwarding for us, over the same SSH connection.

## Making VSCode all comfy and nice

I use the excellent Iosevka font, paired with either of "GitHub Light" / "Github Dark" (following the time of day), and these extensions:

• "rust-analyzer" is an absolute must-have: it provides code intelligence, completion and refactoring for Rust.
• "Vim" provides me with Vim keybindings inside of VS Code. It's not quite the same, but I've been using that for years (along with a couple custom bindings to switch/close tabs, also hover, jump to definition, and move to the next/previous diagnostic) and I'm very happy with it.
• "Error Lens" shows diagnostics inline. I'll occasionally still have to expand the "Problems" pane/tab to see a bigger version, or simply run cargo clippy in the integrated terminal, but it's good for the 90% portion of silly mistakes.
• "Better TOML" provides syntax highlighting when editing Cargo.toml, .cargo/config.toml, etc.

I have a lot more extensions, some minor QoL things (like "FontSize Shortcuts", "TODO Highlight" or "vscode-position"), some language-specific ("x86 and x86_64 Assembly", "Nix", "systemd-unit-file"), and some bigger, like "Git Blame", "GitHub Pull Requests and Issues".