Magics

The IoT Kernel is derived from the “standard” IPython kernel in Jupyter. Normally, code entered into a cell is sent to the currently connected microcontroller for execution. The IoT kernel also processes magics. These differ from those in the IPython kernel.

If a cell stars with %%host, all content is instead sent to the IPython kernel: code is evaluated by the Python interpreter on the host (the device on which the Jupyter server is running). Likewise, magics that follow %%host are handled by IPython rather than the IoT Kernel.

IPython Kernel

The Jupyter Python kernel is provided by IPython. It extends the Python syntax with additional features e.g. to interact with the Linux shell, to profile and debug Python code, and run code written in different languages such as Javascript. These extensions are accessed through magics, commands preceeded by one or two percent signs.

With the IoT Kernel these features are available in cells marked with the %%host magic.

For example:

%%host

%lsmagic

Show code cell output Hide code cell output

Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

%%host

%%timeit

a = 0
for i in range(10):
    a += i**i

10.5 µs ± 11.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

IPython magics have a number of features for exchanging data with Python. These are available only in %%host cell magic sections.

Shell to Python

To pass shell outputs to Python use

%%host

folder = !ls
print(folder)

['bin', 'ide49', 'iot-device', 'iot-kernel', 'iot49-dev', 'iot49.org', 'micropython', 'scratch']

With %%bash use the following syntax:

%%host
%%bash --out output --err error
echo -n "hi, stdout"
echo -n "hello, stderr" >&2

%%host
print(f"out = {output} & err = {error}")

out = hi, stdout & err = hello, stderr

Python to shell

It is also possible to pass the values of python variables to the shell:

%%host
%%bash -s "{__doc__}"
echo I got this from Python: $1

I got this from Python: Automatically created module for IPython interactive environment

%%host
%%timeit?

Show code cell output Hide code cell output

Docstring:
Time execution of a Python statement or expression

Usage, in line mode:
  %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement
or in cell mode:
  %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code
  code
  code...

Time execution of a Python statement or expression using the timeit
module.  This function can be used both as a line and cell magic:

- In line mode you can time a single-line statement (though multiple
  ones can be chained with using semicolons).

- In cell mode, the statement in the first line is used as setup code
  (executed but not timed) and the body of the cell is timed.  The cell
  body has access to any variables created in the setup code.

Options:
-n<N>: execute the given statement <N> times in a loop. If <N> is not
provided, <N> is determined so as to get sufficient accuracy.

-r<R>: number of repeats <R>, each consisting of <N> loops, and take the
best result.
Default: 7

-t: use time.time to measure the time, which is the default on Unix.
This function measures wall time.

-c: use time.clock to measure the time, which is the default on
Windows and measures wall time. On Unix, resource.getrusage is used
instead and returns the CPU user time.

-p<P>: use a precision of <P> digits to display the timing result.
Default: 3

-q: Quiet, do not print result.

-o: return a TimeitResult that can be stored in a variable to inspect
    the result in more details.

.. versionchanged:: 7.3
    User variables are no longer expanded,
    the magic line is always left unmodified.

Examples
--------
::

  In [1]: %timeit pass
  8.26 ns ± 0.12 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)

  In [2]: u = None

  In [3]: %timeit u is None
  29.9 ns ± 0.643 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

  In [4]: %timeit -r 4 u == None

  In [5]: import time

  In [6]: %timeit -n1 time.sleep(2)


The times reported by %timeit will be slightly higher than those
reported by the timeit.py script when variables are accessed. This is
due to the fact that %timeit executes the statement in the namespace
of the shell, compared with timeit.py, which uses a single setup
statement to import function or create variables. Generally, the bias
does not matter as long as results from timeit.py are not mixed with
those from %timeit.
File:      /usr/local/lib/python3.8/site-packages/IPython/core/magics/execution.py

Help

%%host

%alias?

Show code cell output Hide code cell output

Docstring:
Define an alias for a system command.

'%alias alias_name cmd' defines 'alias_name' as an alias for 'cmd'

Then, typing 'alias_name params' will execute the system command 'cmd
params' (from your underlying operating system).

Aliases have lower precedence than magic functions and Python normal
variables, so if 'foo' is both a Python variable and an alias, the
alias can not be executed until 'del foo' removes the Python variable.

You can use the %l specifier in an alias definition to represent the
whole line when the alias is called.  For example::

  In [2]: alias bracket echo "Input in brackets: <%l>"
  In [3]: bracket hello world
  Input in brackets: <hello world>

You can also define aliases with parameters using %s specifiers (one
per parameter)::

  In [1]: alias parts echo first %s second %s
  In [2]: %parts A B
  first A second B
  In [3]: %parts A
  Incorrect number of arguments: 2 expected.
  parts is an alias to: 'echo first %s second %s'

Note that %l and %s are mutually exclusive.  You can only use one or
the other in your aliases.

Aliases expand Python variables just like system calls using ! or !!
do: all expressions prefixed with '$' get expanded.  For details of
the semantic rules, see PEP-215:
http://www.python.org/peps/pep-0215.html.  This is the library used by
IPython for variable expansion.  If you want to access a true shell
variable, an extra $ is necessary to prevent its expansion by
IPython::

  In [6]: alias show echo
  In [7]: PATH='A Python string'
  In [8]: show $PATH
  A Python string
  In [9]: show $$PATH
  /usr/local/lf9560/bin:/usr/local/intel/compiler70/ia32/bin:...

You can use the alias facility to access all of $PATH.  See the %rehashx
function, which automatically creates aliases for the contents of your
$PATH.

If called with no parameters, %alias prints the current alias table
for your system.  For posix systems, the default aliases are 'cat',
'cp', 'mv', 'rm', 'rmdir', and 'mkdir', and other platform-specific
aliases are added.  For windows-based systems, the default aliases are
'copy', 'ddir', 'echo', 'ls', 'ldir', 'mkdir', 'ren', and 'rmdir'.

You can see the definition of alias by adding a question mark in the
end::

  In [1]: cat?
  Repr: <alias cat for 'cat'>
File:      /usr/local/lib/python3.8/site-packages/IPython/core/magics/osm.py

%%host

import sys

sys?

Show code cell output Hide code cell output

Type:        module
String form: <module 'sys' (built-in)>
Docstring:  
This module provides access to some objects used or maintained by the
interpreter and to functions that interact strongly with the interpreter.

Dynamic objects:

argv -- command line arguments; argv[0] is the script pathname if known
path -- module search path; path[0] is the script directory, else ''
modules -- dictionary of loaded modules

displayhook -- called to show results in an interactive session
excepthook -- called to handle any uncaught exception other than SystemExit
  To customize printing in an interactive session or to install a custom
  top-level exception handler, assign other functions to replace these.

stdin -- standard input file object; used by input()
stdout -- standard output file object; used by print()
stderr -- standard error object; used for error messages
  By assigning other file objects (or objects that behave like files)
  to these, it is possible to redirect all of the interpreter's I/O.

last_type -- type of last uncaught exception
last_value -- value of last uncaught exception
last_traceback -- traceback of last uncaught exception
  These three are only available in an interactive session after a
  traceback has been printed.

Static objects:

builtin_module_names -- tuple of module names built into this interpreter
copyright -- copyright notice pertaining to this interpreter
exec_prefix -- prefix used to find the machine-specific Python library
executable -- absolute path of the executable binary of the Python interpreter
float_info -- a named tuple with information about the float implementation.
float_repr_style -- string indicating the style of repr() output for floats
hash_info -- a named tuple with information about the hash algorithm.
hexversion -- version information encoded as a single integer
implementation -- Python implementation information.
int_info -- a named tuple with information about the int implementation.
maxsize -- the largest supported length of containers.
maxunicode -- the value of the largest Unicode code point
platform -- platform identifier
prefix -- prefix used to find the Python library
thread_info -- a named tuple with information about the thread implementation.
version -- the version of this interpreter as a string
version_info -- version information as a named tuple
__stdin__ -- the original stdin; don't touch!
__stdout__ -- the original stdout; don't touch!
__stderr__ -- the original stderr; don't touch!
__displayhook__ -- the original displayhook; don't touch!
__excepthook__ -- the original excepthook; don't touch!

Functions:

displayhook() -- print an object to the screen, and save it in builtins._
excepthook() -- print an exception and its traceback to sys.stderr
exc_info() -- return thread-safe information about the current exception
exit() -- exit the interpreter by raising SystemExit
getdlopenflags() -- returns flags to be used for dlopen() calls
getprofile() -- get the global profiling function
getrefcount() -- return the reference count for an object (plus one :-)
getrecursionlimit() -- return the max recursion depth for the interpreter
getsizeof() -- return the size of an object in bytes
gettrace() -- get the global debug tracing function
setcheckinterval() -- control how often the interpreter checks for events
setdlopenflags() -- set the flags to be used for dlopen() calls
setprofile() -- set the global profiling function
setrecursionlimit() -- set the max recursion depth for the interpreter
settrace() -- set the global debug tracing function

%%host

*get*?

get_ipython
getattr

%%host

getattr?

Show code cell output Hide code cell output

Docstring:
getattr(object, name[, default]) -> value

Get a named attribute from an object; getattr(x, 'y') is equivalent to x.y.
When a default argument is given, it is returned when the attribute doesn't
exist; without it, an exception is raised in that case.
Type:      builtin_function_or_method

Consult the IPython documentation for additional information.

IoT Kernel

Cells that do not begin with %%host are handled by the IoT kernel. Code is sent to the currently connected microcontroller for evaluation, or, if no controller is connected, the kernel tries to connect to the last device the notebook was connected to.

The IoT Kernel implements its own set of magics and rules for using them. Although there is overlap, the %%bash cell magic for example is available both in the IoT and the IPython kernel, in general they are different.

%lsmagic is also available in both kernels and return a different set of magics. Running magics from the “wrong” kernel results in errors:

%%host
%timeit 2**32
%discover

40.5 ns ± 0.145 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

UsageError: Line magic function `%discover` not found.

%discover
%timeit 2**32

pico  serial:///dev/ttyUSB0  

Line magic timeit not defined

The descriptions below apply to magics in the IoT Kernel.

%lsmagic

Shows a brief synopsis of all magics available in the IoT Kernel:

%lsmagic

Line Magic:    -h shows help (e.g. %discover -h)
  %cat         Output contents of file stored on microcontroller
  %cd          Change current working directory on host
  %connect     Connect to device by name, uid, or url
  %cp          Copy files between host and microcontroller
  %discover    Discover available devices
  %gettime     Query microcontroller time
  %info        Summary about connected device
  %loglevel    Set logging level.
  %lsmagic     List all magic functions
  %mkdirs      Create all directories on the microcontroller, as needed (similar to Linux mkdir -p)
  %name        Name of currently connected microcontroller
  %pip         Install packages from PyPi
  %platform    sys.platform of currently connected device
  %rdiff       Show differences between microcontroller and host directories
  %register    Register device
  %rlist       List files on microcontroller
  %rm          Remove files on microcontroller
  %rsync       Synchronize microcontroller to host directories
  %softreset   Reset microcontroller. Similar to pressing the reset button.
  %store       Copy variables between microcontroller and storage
  %synctime    Synchronize microcontroller time to host
  %uid         UID of currently connected microcontroller
  %unregister  Unregister device
  %upip        Install MicroPython packages
  %url         URL of currently connected microcontroller
  !            Pass line to bash shell for evaluation

Cell Magic:    -h shows help (e.g. %%connect -h)
  %%bash       Pass cell to bash shell for evaluation
  %%connect    Generalization of %connect to run code on several devices sequentially
  %%service    Send code to bash in named container for execution
  %%ssh        Pass cell body to ssh.
  %%writefile  Write cell contents to file

Help

Use built-in help for usage information:

%%service -h

usage: %%service [-h] [--err ERR] [--out OUT] [-s SHELL] [-u USER] container

Send code to bash in named container for execution

positional arguments:
  container           name of container to ssh into

optional arguments:
  -h, --help          show this help message and exit
  --err ERR           store stderr in shell environment variable
  --out OUT           store stdout in shell environment variable
  -s SHELL, --shell SHELL
                      Shell to use in target container. Default: /bin/bash
  -u USER, --user USER
                      Username or UID (format: <name|uid>[:<group|gid>])

This specifically supports docker/balena apps.

Before running the instructions, the file .init_${container}.sh
is sourced, if it exists. ${container} is the name of the service given.

Example:
    %%service esp-idf
    printenv | grep BALENA_SERVICE_NAME
    which idf.py

    # lookup mdns address and assign to $OUT
    %%service host --out OUT
    ping -c 2 pi4server.local | awk -F"[()]" '{print $2}'

Differences

Mostly IoT and IPython magics differ, but there is some overlap.

%%bash

The IoT Kernel implementation of %%bash incrementally prints output, whereas the IPython version waits for shell execution to end before printing output. The former behavior is useful for long running commands such as compilation.

On the other hand, the IPython implementation permits passing in Python variable. The IoT Kernel does not support this feature.

%%host
some_variable = "some value"

%%host
%%bash -s "{some_variable}"
echo some variable = $1

some variable = some value

! (shell)

The IPython kernel support assigning shell output to a Python variable. The IoT Kernel does not support this feature.

%%host
folder = !ls /

%%host
print(folder)

['bin', 'boot', 'dev', 'etc', 'home', 'host', 'lib', 'lib64', 'media', 'mnt', 'opt', 'proc', 'root', 'run', 'sbin', 'service-config', 'srv', 'sys', 'tmp', 'usr', 'var']