Sockets

Sockets#

Once an internet connection has been established, MicroPython uses sockets to access resources on the network, just like CPython (and pretty much all programming languages).

Sockets are quite low level; frequently higher level libraries can be used instead. But if you want to write your own webserver, for example, you likely will use sockets.

Examples presented are adapted from the MicroPython github repository. Check them out for additional information.

http Client#

The code below first looks up the ip address of the server (google.com). It then creates a socket, connects to it at port 80 and downloads 2000 bytes.

import socket

ai = socket.getaddrinfo('google.com', 80)
print("Address information:", ai)
addr = ai[0][-1]

s = socket.socket()
s.connect(addr)
s.write(b"GET / HTTP/1.0\r\n\r\n")

print("\nResponse:")
print(s.read(2000).decode(), "...")

s.close()

Show code cell output Hide code cell output

Connected to esp32 @ serial:///dev/ttyUSB0
Address information: [(2, 1, 0, 'google.com', ('142.250.191.46', 80))]

Response:
HTTP/1.0 200 OK
Date: Thu, 09 Dec 2021 20:56:17 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2021-12-09-20; expires=Sat, 08-Jan-2022 20:56:17 GMT; path=/; domain=.google.com; Secure
Set-Cookie: NID=511=EtAZ6YCY123hQF0O5EzPwAwG_dy5oAVQD0ph_rSNLdxsxROCEABHQiSqm4eQexkyIejMn_-9NJ6LgOr5Y-Zv_bwnqXZ47UojX7-TWcK2gcqBTf9tyiD7nsN8BoH6vA2VJPCBOfZPF3uJXN-dnd4Do35y8F6L9jri2ASzEYLoWPE; expires=Fri, 10-Jun-2022 20:56:17 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-6752733080595605-cst.gif" itemprop="image"><meta content="Seasonal Holidays 2021" property="twitter:title"><meta content="Seasonal Holidays 2021 #GoogleDoodle" property="twitter:description"><meta content="Seasonal Holidays 2021 #GoogleDoodle" property="og:description"><meta content="summary_large_image" property="twitter:card"><meta content="@GoogleDoodles" property="twitter:site"><meta content="https://www.google.com/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-2xa.gif" property="twitter:image"><meta content="https://www.google.com/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-2xa.gif" property="og:image"><meta content="1000" property="og:image:width"><meta content="400" property="og:image:height"><meta content="https://www.google.com/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-2xa ...

The response is quite wordy with embedded graphics meant for visualization in a browser, not parse by a microcontroller. Some sites, e.g. for weather data, can produce simpler responses optimized for parsing by machines.

http Server#

Let’s do the opposite and create a simple webserver.

%connect esp32 -q

import socket, network


CONTENT = b"""\
HTTP/1.0 200 OK

Hello #{} from MicroPython!
"""

PORT = 8080


def webserver():
    my_ip = network.WLAN(network.STA_IF).ifconfig()[0]
    s = socket.socket()

    # Binding to all interfaces - server will be accessible to other hosts!
    ai = socket.getaddrinfo("0.0.0.0", PORT)
    addr = ai[0][-1]

    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(addr)
    s.listen(5)
    print("Listening, connect your browser to http://{}:{}/".format(my_ip, PORT))

    try:
        counter = 0
        while True:
            client_sock, client_addr = s.accept()

            print("Request from", client_addr)
            req = client_sock.readline()
            print("\nRequest:")
            print(req)
            while True:
                h = client_sock.readline()
                if h == b"" or h == b"\r\n":
                    break
                print(h)
            client_sock.write(CONTENT.format(counter))

            client_sock.close()
            counter += 1
            print()
    finally:
        s.close()

webserver()

Show code cell output Hide code cell output

Listening, connect your browser to http://10.39.40.168:8080/
Request from ('10.39.40.114', 50287)

Request:
b'GET / HTTP/1.1\r\n'
b'Host: 10.39.40.168:8080\r\n'
b'Connection: keep-alive\r\n'
b'DNT: 1\r\n'
b'Upgrade-Insecure-Requests: 1\r\n'
b'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36\r\n'
b'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\n'
b'Accept-Encoding: gzip, deflate\r\n'
b'Accept-Language: en-US,en;q=0.9,de-CH;q=0.8,de;q=0.7,fr-FR;q=0.6,fr;q=0.5,zh-HK;q=0.4,zh-CN;q=0.3,zh-TW;q=0.2,zh;q=0.1\r\n'

Request from ('10.39.40.114', 50288)

Request:
b'GET /favicon.ico HTTP/1.1\r\n'
b'Host: 10.39.40.168:8080\r\n'
b'Connection: keep-alive\r\n'
b'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36\r\n'
b'DNT: 1\r\n'
b'Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8\r\n'
b'Referer: http://10.39.40.168:8080/\r\n'
b'Accept-Encoding: gzip, deflate\r\n'
b'Accept-Language: en-US,en;q=0.9,de-CH;q=0.8,de;q=0.7,fr-FR;q=0.6,fr;q=0.5,zh-HK;q=0.4,zh-CN;q=0.3,zh-TW;q=0.2,zh;q=0.1\r\n'

Request from ('10.39.40.114', 50289)

Request:
b'GET / HTTP/1.1\r\n'
b'Host: 10.39.40.168:8080\r\n'
b'Connection: keep-alive\r\n'
b'Cache-Control: max-age=0\r\n'
b'DNT: 1\r\n'
b'Upgrade-Insecure-Requests: 1\r\n'
b'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36\r\n'
b'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\n'
b'Accept-Encoding: gzip, deflate\r\n'
b'Accept-Language: en-US,en;q=0.9,de-CH;q=0.8,de;q=0.7,fr-FR;q=0.6,fr;q=0.5,zh-HK;q=0.4,zh-CN;q=0.3,zh-TW;q=0.2,zh;q=0.1\r\n'

Request from ('10.39.40.114', 50290)

Request:
b'GET /favicon.ico HTTP/1.1\r\n'
b'Host: 10.39.40.168:8080\r\n'
b'Connection: keep-alive\r\n'
b'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36\r\n'
b'DNT: 1\r\n'
b'Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8\r\n'
b'Referer: http://10.39.40.168:8080/\r\n'
b'Accept-Encoding: gzip, deflate\r\n'
b'Accept-Language: en-US,en;q=0.9,de-CH;q=0.8,de;q=0.7,fr-FR;q=0.6,fr;q=0.5,zh-HK;q=0.4,zh-CN;q=0.3,zh-TW;q=0.2,zh;q=0.1\r\n'

Request from ('10.39.40.200', 57106)

Request:
b'GET / HTTP/1.1\r\n'
b'Host: 10.39.40.168:8080\r\n'
b'User-Agent: curl/7.68.0\r\n'
b'Accept: */*\r\n'

Interrupted

Click the http link above to open a browser window. Notice two things:

The hello counter increases by two everytime you refresh the page in the browser. The reason is that the browser (at least mine) make two requests each time it loads the page.
The browser sends lots of data with each request. The kind of browser, the languages it speaks, etc. That’s helpful for marketers to track users, but it’s a bit over the top for small microcontrollers with limited memory and processing power.

We’ll check out more efficient means for microcontrollers to communicate over the internet.

Secure Client#

The secure, https, client is almost the same except that the port has been changed from 80 to 443 and the line s = ssl.wrap_socket(s) been added.

import socket, ssl

ai = socket.getaddrinfo("google.com", 443)
print("Address information:", ai)
addr = ai[0][-1]

s = socket.socket()
s.connect(addr)
s = ssl.wrap_socket(s)

s.write(b"GET / HTTP/1.0\r\n\r\n")

print("\nResponse:")
print(s.read(2000).decode())

s.close()

Show code cell output Hide code cell output

Address information: [(2, 1, 0, 'google.com', ('142.250.191.46', 443))]

Response:
HTTP/1.0 200 OK
Date: Thu, 09 Dec 2021 20:58:14 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2021-12-09-20; expires=Sat, 08-Jan-2022 20:58:14 GMT; path=/; domain=.google.com; Secure
Set-Cookie: NID=511=M45oCsiHFWSgRWfrX2VB--jEwDhmAKvh7tQl-sR3fbhVrwK3321ryOGfg2NVQaw2ONcmhsrMwPBYcvxIsHFJcTnixzciFARp6xCxofjeWzS8QJ0ll1URT0VM3_iTpasCOyLFvtCfwieSRmaLicLdnVQIiUXRlsAknUMpL5ScCbY; expires=Fri, 10-Jun-2022 20:58:14 GMT; path=/; domain=.google.com; HttpOnly
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Accept-Ranges: none
Vary: Accept-Encoding

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-6752733080595605-cst.gif" itemprop="image"><meta content="Seasonal Holidays 2021" property="twitter:title"><meta content="Seasonal Holidays 2021 #GoogleDoodle" property="twitter:description"><meta content="Seasonal Holidays 2021 #GoogleDoodle" property="og:description"><meta content="summary_large_image" property="twitter:card"><meta content="@GoogleDoodles" property="twitter:site"><meta content="https://www.google.com/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-2xa.gif" property="twitter:image"><meta content="https://www.google.com/logos/doodles/2021/seasonal-holidays-2021-6753651837109324-2xa.gif" property="og:image"><meta content="1000" 

Secure Server#

Secure webservers use a certificate and a private key to encrypt data and “prove” their identity to the web client (e.g. browser).

The certificate contains the domain name of the server. Since our ESP32 does not have a domain name we’ll use the IP address instead.

The steps below are a somewhat contorted way to get the IP address from the ESP32 to the host without copy and paste (which would work also and be much simpler but not as “clever”).

import network
my_ip = network.WLAN(network.STA_IF).ifconfig()[0]
print(my_ip)
%store my_ip

Connected to esp32 @ serial:///dev/ttyUSB0
10.39.40.168

Retrieve the IP on the host and assign it to shell environment variable my_ip:

%%host

%store -r my_ip
import os
os.environ["my_ip"] = my_ip

Verify it’s correct:

%%bash
echo "microcontroller IP:" $my_ip

microcontroller IP: 10.39.40.168

Now we are ready to create a certificate.

Let’s collect all relevant information in a configuration file from which the certificate and key will be generated. Change the values in the [req_distinguished_name] section if you wish. They are just for information. Note that we embed the IP address from the microcontroller in the certificate specification. This is necessary so the client can verify the server.

I found this guide helpful.

%%bash

# create a folder for the certificate
cd $IOT_PROJECTS/internet
mkdir -p ssl
cd ssl

# write the certificate spec
cat << EOF >cert.conf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = CA
L = San Francisco
O = MicroPython Webserver
OU = iot49-$my_ip
CN = iot49-$my_ip
[v3_req]
keyUsage = critical, digitalSignature, keyAgreement
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = $my_ip
IP.1  = $my_ip
EOF

# create the certificate and private key
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout cert.key -outform PEM -out cert.crt -config cert.conf

Generating a RSA private key
..........................................................................+++++
................................+++++
writing new private key to 'cert.key'
-----

The certificate and private key are now in folder ssl:

!cat $IOT_PROJECTS/internet/ssl/cert.key
!cat $IOT_PROJECTS/internet/ssl/cert.crt

Show code cell output Hide code cell output

Let’s copy them to the ESP32, without the first and last lines:

%%bash

cd $IOT_PROJECTS/internet

mkdir -p code/ssl

sed '1d; $d' ssl/cert.key > code/ssl/cert.key
sed '1d; $d' ssl/cert.crt > code/ssl/cert.crt

Verify that the certificate works with CPython https server:

%%host

import http.server, ssl, socket

PORT = 4443

with http.server.HTTPServer(('0.0.0.0', PORT), http.server.SimpleHTTPRequestHandler) as httpd:
    httpd.socket = ssl.wrap_socket(httpd.socket,
                                   server_side=True,
                                   keyfile='ssl/cert.key',
                                   certfile='ssl/cert.crt',
                                   ssl_version=ssl.PROTOCOL_TLS)

    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
        s.connect(('0.0.0.1', 80))
        my_ip = s.getsockname()[0]

    print(f"serving at https://{my_ip}:{PORT}")

    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        print("so long ...")

serving at https://10.39.40.200:4443

10.39.40.114 - - [12/Dec/2021 18:13:11] "GET / HTTP/1.1" 200 -

so long ...

Finally run the server with code adapted from a micropython example.

%rsync
%softreset

import ubinascii as binascii
import network

try:
    import usocket as socket
except:
    import socket
import ussl as ssl

# Read the certificate and key, convert to binary

with open('/ssl/cert.key') as f:
    key  = binascii.a2b_base64(f.read())

with open('/ssl/cert.crt') as f:
    cert = binascii.a2b_base64(f.read())


CONTENT = b"""\
HTTP/1.0 200 OK

Hello #%d from MicroPython!
"""

def main():
    s = socket.socket()

    # Binding to all interfaces - server will be accessible to other hosts!
    ai = socket.getaddrinfo("0.0.0.0", 8443)
    addr = ai[0][-1]

    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(addr)
    s.listen(5)

    ip = network.WLAN(network.STA_IF).ifconfig()[0]
    print("Listening at https://{}:8443/".format(ip))

    counter = 1
    while True:
        client_s, client_addr = s.accept()
        # CPython uses key keyfile/certfile arguments, but MicroPython uses key/cert
        try:
            client_s = ssl.wrap_socket(client_s, server_side=True, key=key, cert=cert)
            print("\n----- Request", counter)
            # Both CPython and MicroPython SSLSocket objects support read() and
            # write() methods.
            # Browsers are prone to terminate SSL connection abruptly if they
            # see unknown certificate, etc. We must continue in such case -
            # next request they issue will likely be more well-behaving and
            # will succeed.
            req = client_s.readline()
            print(req.decode(), end="")
            while True:
                h = client_s.readline()
                if h == b"" or h == b"\r\n":
                    break
                print(h.decode(), end="")
            if req:
                client_s.write(CONTENT % counter)
            counter += 1
            print()
        except OSError as e:
            print("OSError", e)
        finally:
            client_s.close()            

main()

You’ll get a warning that the connection is not private. That’s because of the self-signed certificate - the browser does not know us and hence can ascertain that the certificate is legitimate. If you add it to the browser, the error disappears.

Running (substitute the correct IP address)

curl --insecure -v https://10.39.40.168:8443/

from the command line prints information about the certificate (and avoids the OSError). The --insecure flag is needed since curl doesn’t trust us in our capacity as “Certificate Authority”.

Wireshark#

But how do we know that the transmitted data is indeed encrypted and secure? We need a spy on the internet that intercepts and views packets.

Wireshark is a program that does just that and it’s preinstalled in ide49. Go to the home screen and click on the blue shark fin.

Highlight the wlan (wlp3s0) and enter port 8080 as the capture filter. Then press the shark fin in the toolbar to start the capture. For a first test, start the unencrypted http server and run curl http://10.39.40.168:8080/ (update the ip address). Wireshark will list all the captured packets and you can read Hello from MicroPython in the clear. Anyone with access to the network can read your message.

Now stop Wireshark (press first the red square and then the tool with the cross) and change the capture filter port 8443 and start the capture. Start the https server and run curl --insecure -v https://10.39.40.168:8443/. Now all packets are encrypted:

Sockets

Contents

Sockets#

http Client#

http Server#

Secure Client#

Secure Server#

Wireshark#