Plaintext
PROUDLY INDEPENDENT SINCE 2013
PIs IN SPACE!
The Raspberry
Pi projects
orbiting the
August 2016 www.linuxvoice.com earth
BUILD A WEBSITE PUBLISHING PARANOIA
HUGO EBOOKS STEGANOGRAPHY
Generate a static site Get your words out there Hide data where no-one will
without faffing around into the eternal library look: your holiday photos
{ KERNEL {
INSIDE THE
LINUX
Discover the power at the heart of your Linux box
Seize full control over your machine’s hardware
Take your first steps as a kernel hacker
BURSTING
WITH
GHOST Blog quickly and cleanly with Free Software AWESOME
TAILS Protect your privacy the easy way – install this distro! TUTORIALS
LIBREOFFICE Inside the project that’s taking FOSS to the masses
OPEN RIGHTS GROUP RAIN, RAIN, GO AWAY
Augus t 2016 £5.99 Printe d in the UK
JIM KILLOCK RASPBERRY PI
On spying, oversight, Build a virtual weather
and how the man with a Pi, some
government wants motors and a bit of
to watch us all coloured cardboard
ELIXIR › RSA ENCRYPTION › KRITA & MORE!
FOSSTALK LIVE
2016
A free evening of live Linux Podcasts
Saturday 6 August 2016
Plus Stuart Langridge and Dave MegaSlippers
http://www.fosstalk.com/tickets
The Harrison, 28 Harrison Street, Kings Cross, London, WC1H 8JF
Doors 5pm
ISSUE 29 WELCOME
TO THE KERNEL AND BEYOND
The August issue What’s hot in LV#029
ANDREW GREGORY
The landlord of my local pub
hates Windows 10, and I’ve
managed to persuade him to
make the switch to Linux. I’ll take
this issue with me next time I go
for a pint so he can choose his
BEN EVERARD distro from the group test. p50
Long-term Linux user and
best-selling author Ben is GRAHAM MORRISON
usually found knee-deep in GCC is one of those tools that I
either Python code or a use all the time, but I don’t know
tangle of wires. too much about what goes on
inside it. This month, I’ve
W
particularly enjoyed the FAQ,
hat is Linux? That’s the question we’re tackling this issue. where Mike sheds a little light on
We already know that it’s a kernel – the heart of the this ubiquitous compiler. p32
operating system – but what does this mean? How is it
organised and what does it do? Despite using Linux for well over a MIKE SAUNDERS
decade and a half, I really only had a slight idea about what went on I use public and private keys all
inside the kernel, so when we got Valentine’s article in, I was the time, but they just seemed like
particularly excited to read it. No only is this information interesting, mysterious numbers to me until I
it’s also important to know. When running a Linux system – or any read John Lane’s tutorial. Now I
OS for that matter – the more you understand, the more power you know exactly what I need to do to
have. One of the great things about using an open source OS is that keep my data safe when
transferring it on the internet. p84
we can delve down into the depths and really see what’s going on.
Armed with this new knowledge, I feel I’m better equipped to deal
with any problems that may pop up in the future.
Ben Everard
Editor, Linux Voice
Linux Voice is different.
THE LINUX VOICE TEAM
Linux Voice is special. SUBSCRIBE
Editor Ben Everard
ben@linuxvoice.com Here’s why… ON PAGE 56
Deputy editor Andrew Gregory
1 At the end of each financial year we’ll
andrew@linuxvoice.com
give 50% of our profits to a selection of
Editor in hiding Graham Morrison organisations that support free
graham@linuxvoice.com software, decided by a vote among our
Editor at large Mike Saunders readers (that’s you).
mike@linuxvoice.com
Games editor Michel Loubet-Jambert 2 No later than nine months after first
michel@linuxvoice.com publication, we will relicense all of our
Creative director Stacey Black content under the Creative Commons
stacey@linuxvoice.com CC-BY-SA licence, so that old content
can still be useful, and can live on even
Malign puppetmaster Nick Veitch
after the magazine has come off the
nick@linuxvoice.com
shelves
Editorial contributors:
Mark Crutch, Juliet Kemp, 3 We’re a small company, so we don’t
Vincent Mealing, Simon Phipps, have a board of directors or a bunch of
Les Pounder, Mayank Sharma, shareholders in the City of London to
Amit Saha, Valentine Sinitsyn keep happy. The only people that matter
to us are the readers.
www.linuxvoice.com 3
CONTENTS ISSUE 29 AUGUST 2016
Contents
Here on the plains of the Serengeti, a troupe of monkeys is playing…
Regulars Cover Feature
News 06
Android is spreading its mechanical
tentacles to the Raspberry Pi 3 and Chrome 14
OS, OwnCloud has forked into a new project,
and the Krita project is now €37,000 richer.
Distrohopper 08
This month featuring a gateway distro for
Windows users and a destination distro for
the security-obsessed.
Speak your brains 10
On the importance of facial foliage in
matters Unixy, Linuxy and geeky in general.
Subscribe! 12/56
Never again be the sad-faced Victorian
urchin child, face pressed up against the
glass of the newsagent having missed out
on the last copy of Linux Voice.
FOSSPicks 58
What is it that makes Linux Linux? How does your distro speak to your
Free range software, organically grown and hardware? Why, it’s the Linux kernel of course, laid bare on page 14.
allowed enough space and light to grow
naturally into tasty morsels for us to install
on our Linux machines. Interview Feature
Core Tech 94
Look under the hood of Linux and find
out what’s making that funny noise. This 34 inside 22
month: processes, threads and Systemtap.
Geek Desktop 98
Nick’s looking back at the olden days, which
is usually our sign that we should take his
Jameson’s away from him.
Jim Killock Inside LibreOffice
SUBSCRIBE The Executive director of the Open Rights How a humble word processor became the figurehead of
ON PAGE 56 Group on spying, privacy and GCHQ. resistance to the global proprietary software empire.
FAQ Group Test
GCC 32 Beginners’ distros 50
The little compiler that could – Help your friends who haven’t
without it, everything else would found Linux yet by recommending
be just a nice idea. Thanks GCC! one of these beginner OSes.
SECRETS OF AMAROK
TURN TO PAGE 26
4 www.linuxvoice.com
ISSUE 29 AUGUST 2016 CONTENTS
Feature Tutorials
28
Steganography 66
Hide data inside innocent-looking image
files to keep your secrets safe from all who
would do you harm (yes, we’re paranoid).
Pi projects in space Hugo
Generate static webites without messing
68
Dip your toe in some of the code running on the ISS right now. about with your own themes, CSS and
HTML – Hugo does it all for you.
Reviews Publishing with FOSS 74
Write, edit and publish in EPUB format, then
sit back and watch as the public acclaim/
Tails 42 deafening silence floods in.
Love the idea of privacy Raspberry Pi 78
online, but don’t know Turn a stream of data from the interwebs
where to start? Start into a physical display, with Python,
here – Tails bundles cardboard and some LEDs.
everything you need into
one easy to install Linux Coding
distribution.
Krita 3.0 43 Ghost 44 Lumo 45
For drawing, animation and If you have a blog and you want Isometric puzzle gaming for
aimless doodles, Krita is the best to keep it simple, try this elegant, the connoisseurs – turn on and
Free application by a mile. easy to use solution. pretend it’s still 1984.
RSA encryption 82
Cryptography is hard – find out how hard
with this mind-melting advanced tutorial.
Gaming on Linux 46 Books 48
The nights are light, the days are long and full of When the power fails, you need something to do. Elixir 88
possibility – so stay in, pull up a chair and refine Why not read some books? Here are two good ones Develop a web application that’s reliable,
your reflexes with some fine games. filled with useful information. fault-tolerant and highly available.
www.linuxvoice.com 5
NEWS ANALYSIS
NEWSANALYSIS
The Linux Voice view on what’s going on in the world of Free Software.
Opinion
Liberating Java
Legal shenanigans in California make their weight felt all around the programming universe.
Simon Phipps
is ex-president of the The full Java API is now available licensed
Open Source Initiative
and a board member under the GPL, and you can use all of or any
of the Open Rights
Group and of Open part of it subject to those terms
Source for America.
– unaltered. That means you can like “can Sun (now renamed Oracle
implement Java programs in any America) prevent anyone using parts of
language. It also means that should you OpenJDK for purposes of which they don’t
T
he long-running lawsuit Oracle borrow just a few of the class files from approve” (the answer is “no”). You’ll find
started against Google back in 2010 OpenJDK, they can be linked with code the FAQ in the Internet Archive via wmk.
reached another milestone recently under other licences. me/Java-FAQ since Oracle deleted it in
with a jury decision that Google’s use of the In particular from both of those 2011.
Java language in Android constitutes fair observations, the full Java API is now Most importantly, there is now a large
use. I spent a good part of May in San available licensed under the GPL and you community of developers able to maintain
Francisco and testified in court there as to can use all of or any part of it subject to Java regardless of Oracle’s strategy.
how, in 2006–7, Sun Microsystems released those terms. Indeed, Red Hat maintains on an official
the Java platform under the GNU General Every source file in OpenJDK includes basis several of the versions that would
Public Licence. comprehensive comments that document otherwise have been abandoned.
I testified because of my role as Sun’s the specification for each class. The In the lawsuit, OpenJDK played a key role
head of open source at the time. Releasing JavaDocs tool is able to harvest those in proving to the jury that Google should not
Java as OpenJDK was an excellent thing to comments to build a specification for the owe Oracle $9 billion for using the Java
do for software freedom – Richard Stallman Java platform. Since all those comments programming language in Android, even
even agreed to say so on camera in a Sun are licensed under the GPL, it is possible before they started using it for Android N.
promotional video. Among the reasons why: to build a specification licensed only under But more relevantly for the rest of us, Java
The full source code for the Java platform the GPL, which places no restrictions on remains a platform that’s suitable for
is now available as genuine free software reimplementation. open source use because
under the GPL. That basic fact is itself a From the moment we announced Java OpenJDK was
huge benefit. would be open source, there were many, genuinely liberated
When we (the Sun Java and open source many questions in the community. We and not just a
teams) released OpenJDK in 2006–7, we collated them all and then worked with corporate facade.
were concerned that the open source Sun’s business and legal teams to write
community might not trust our motives, approved answers to all the questions we
so as well as using the unaltered GPL, we could find. The result was a magnum opus;
also applied the licence exception that the a comprehensive and authoritative FAQ
GNU Projects’s team implementing a free that leaves no doubt about Sun’s intent.
Java used – the Classpath Exception This includes answers to key questions
Releasing Java as OpenJDK was an excellent
thing to do for software freedom – Richard It’s OK to
Stallman even agreed to say so on camera use Java in
Android. Phew!
6 www.linuxvoice.com
ANALYSIS NEWS
Systemd • Google • Linux • Android • OwnCloud • Krita • LibreOffice
CATCHUP Summarised: the biggest news
stories from the last month
Systemd kills background Google brings Android Linux kernel 4.6 released
1 processes by default 2 support to Raspberry Pi 3 Yes, kernel 4.6 is here with
Systemd has made some There are already plenty a bunch of improvements:
pretty controversial changes to Linux; of operating systems to run on OrangeFS distributed filesystem
now it will automatically terminate your Raspberry Pi: a zillion Linux support, USB 3.1 SuperSpeed,
processes when you log out, so you distributions, NetBSD, and even NVIDIA GeForce GTX 900
need to be aware of it if you run a Windows 10. Now Google is bringing Maxwell support along with
multiplexer like Screen or Tmux. Some a version of Android to the Pi – and Dell XPS 13 Skylake.
argue that this change makes sense – specifically, the Raspberry Pi 3. In www.kernelnewbies.org
that things shouldn’t carry on running in Google’s Android Open Source Project
the background unless you specifically (AOSP) a new device tree has popped
say they should – whereas others up for the single-board computer, so Pi
argue that it’s yet another change that owners should soon be able to run the
goes against the decade-long Unix vast number of apps that have been
philosophy. released for Android.
Linux: the largest Android apps are coming OwnCloud gets forked:
4 software project on earth 5 to Chrome OS 6 say hello to Nextcloud
Kernel developer Greg Plenty of questions have been Forks in FOSS are often ugly
Kroah-Hartman has delivered a raised about the relationship between but sometimes necessary. OwnCloud
presentation explaining that the Linux Google’s two mobile operating systems: founder Frank Karlitschek and a bunch
kernel is the largest development will Android and Chrome OS eventually of long-time OwnCloud developers
project on the planet. Kernel 4.5 merge? Or will the company kill one of have left the project to start a new one:
contains a whopping 21 million lines of them off? Well, now it looks like Chrome Nextcloud. The reasons aren’t entirely
source code, and in the last year the OS will soon be able to run Android clear, but it appears that OwnCloud Inc.
kernel project received contributions applications – so more than a million was becoming too vulnerable to the
from around 4,000 developers in at apps and games from the Play Store. whims of corporate backers. Nextcloud
least 440 different countries. And to This feature is only available in the aims to be more accountable to the
think that in the 1990s, many people Chrome OS developer channel right community with a better long-term
said open source software development now, but could make its way onto vision and future.
was simply not sustainable… Chromebooks in the future. www.nextcloud.com
Krita achieves €37,000 LibreOffice 5.2 beta has
7 Kickstarter success 8 been released
Image editor Krita is having a Due for release in August,
fabulous time. The developers behind LibreOffice 5.2 will bring a bunch of
it have run a Kickstarter crowdfunding improvements across the suite: a single
campaign to improve the text and toolbar mode for Writer (ideal for low-
vector tools, setting a target of €30,000. resolution displays), new drawing tools,
In the end, the final sum of donations a selection filter in the Cross References
was another €7,000 on top of that, dialog, better keyboard shortcuts in
thanks to over 1,000 backers from the Calc, and new spreadsheet functions
community. Now the developers will galore. If you want to try a beta release,
be able to work on a better interface and report any bugs you find to make
for adding and manipulating text, along the final version really shine, grab it
with a better workflow for from the website here:
vector objects. http://tinyurl.com/h6bsemy
www.linuxvoice.com 7
DISTROHOPPER LINUX DISTROS
DISTROHOPPER
What’s hot and happening in the world of Linux distros (and BSD!).
ChaletOS 16.04
Welcome, newcomers!
C
halet is a well put together Ubuntu-
based distro aimed at capturing
Windows users and keeping them on
Linux, rather than scaring them off back to
their proprietary comfort zone. One of the
most notable things it does in this regard is
take some liberties with the concepts of free
software by including non-free media
codecs and the like by default, sparing those
trying Linux for the first time the faff of not
understanding why their MP3 files don’t play.
Otherwise, Chalet is more about
aesthetics than anything else, since it
targets those looking for a simple and
familiar experience, dispelling the myth that
Linux is all about headaches and terminals. ChaletOS’ eye candy, speed and simplicity woo users away from Windows.
As such, it offers extremely cohesive and
consistent themes, including one that It also does well to use Xfce considering the system is “Star Point”, an educational/
closely resembles Windows 10. To aid in this its responsiveness without being as tutorial application to teach newcomers
endeavour, the distribution comes with its bare-bones as other DEs, since speed and about Linux and how to use it. Though there
own “ChaletOS Style Changer” and at first decent boot times are other factors that are are many distros out there aiming to grab
glance, an experienced Linux user would attractive to Linux newcomers. Similarly, Windows users, none really hit all the
struggle to tell that it is the Xfce desktop, having the stability of an LTS release will requirements as well as ChaletOS, and it’s
being used since the themes are so well mean that this new user won’t be put off by certainly one to consider when introducing
implemented. any unexpected quirks. A new addition to others to Linux.
Simplicity Linux 16.04
Lightweight, but with functionality.
S
implicity Linux – a bloat-free even more bare-bones version) and Media
distribution derived from Puppy Linux (with XBMC pre-installed) seem to have been
with some differences – has released discontinued, with the latter presumably
version 16.04, since Puppy itself is now since it’s not a lot of effort to install XMBC on
based on the Ubuntu release cycle. The two the Mini build. The standard Simplicity desktop looks rather
main releases of Simplicity – which both Though Simplicity is not ready to be used good for a lightweight distro.
use LXDE – are called Mini (previously as a day-to-day desktop, its current aims are
Netbook) and Desktop, where the former to make it more appealing to those crossing in itself, as it includes the likes of Gimp,
has the bare essentials and the latter has a over from the world of Windows, and while Firefox, Libreoffice and Wine. Simplicity is one
full-blown desktop experience. Besides this is a laudable aim, for now these efforts to keep an eye on, as the project seems to
those, there’s the X release, an experimental consist mostly of some UI tweaks and be at a crossroads, also having dropped
sandbox release where the developers try cramming in an absurd amount of software 64-bit support in the two main releases, but
out new ideas. The previous Obsidian (an into a small space. This is an impressive feat heading in interesting directions.
8 www.linuxvoice.com
LINUX DISTROS DISTROHOPPER
News from the *BSD camps
What’s going on in the world of FreeBSD, NetBSD and OpenBSD.
T
here’s been a lot of progress across
the BSD camps with regard to more
modern hardware support, as well
as more up-to-date features. The first of this
progress comes from DrangonFlyBSD,
where Wayland/Weston support has been
moving forward to the point where
applications can run on the display server. It
isn’t so straightforward yet and requires
some technical knowledge to pull off,
however the person responsible for getting it
up and running claims that even at this
stage, it feels faster than X.org. There have
also been advancements with drivers, with
the i915 DRM driver being ported over from
the Linux 4.3 kernel, replacing the outdated
one from the 3.x series, thus improving
stability and providing proper support for DragonFlyBSD can now run the Wayland compositor and relevant applications, though it’s not quite
Intel Skylake CPUs. ready yet for most desktop users (photo: Distrowatch).
Meanwhile, CoreCLR (the open source
implementation of the .NET framework) has support for audio mixing – before, sound as additional hardware support such as for
been ported to NetBSD, though for the time was played on a first-come, first-served the Freescale i.MX7. These are some pretty
being it requires assemblies to be cross- basis with only one application playing audio positive steps, as NetBSD has been lagging
compiled from Linux. A lot more at a time. There is also a wide range of behind the other BSDs for some time now.
functionality in other areas is also being platform improvements being made to Version 0.9.0 of PC-BSD’s Lumina Desktop
added to NetBSD, most notably progress on amd64, x86 and ARM architectures, as well has also been released, adding support for
compositing effects like shadows and
Version 0.9.0 of PC-BSD’s Lumina desktop transparency, as well as a new plain text
editor written in Qt 5. The Fluxbox-based
has been released, adding support for effects desktop is making good progress leading up
like shadows and transparency to its 1.0.0 release, which is expected to be
released at the same time as FreeBSD 11.
Qubes OS 3.1
Qubes is an operating system focused on security, and is hard to describe as
being strictly a Linux system by design. The Linux kernel itself never interacts
with the hardware directly, with the system instead making use of the Xen
hypervisor to do that, using a microkernel design to run multiple instances of
differing operating systems. One can see it as running a virtual machine directly
on hardware, rather than running a virtual machine through an installed
operating system, and this security by compartmentalisation approach is
mostly what Qubes is about. The basic premise is to use separate isolated VMs
for different tasks – such as one for interactions where credit card details are
required and another for general browsing – to avoid cross-contamination.
Despite this seeming like a mess, the creation of the virtual machines is
handled through a small and simple VM manager, which allows the allocation of
labels such as “work” or “personal” and to assign each window running within
that VM coloured borders, making them easily identifiable so as not to create
security flaws through human error. The windows are made to be as unintrusive Qubes running two VMs, one of which is running two windows.
as possible, making the whole experience seamless rather than clunky, and
even copy-pasting between VMs is possible through keyboard shortcuts.
However, there are many drawbacks, since not only is running multiple VMs like those hurdles, things become straightforward. Aside from the ease of managing
this resource intensive, but Qubes runs only on a limited range of hardware. It’s VMs, for the ordinary user Qubes functions exactly like a regular Linux desktop,
also 64-bit only, though given the memory requirements, this shouldn’t be using the familiar KDE’s Plasma desktop and Fedora as the template for its
surprising. Actually installing the OS is also challenging, but if all you clear VMs, though any Xen-supported OS is possible.
www.linuxvoice.com 9
MAIL YOUR LETTERS
YOUR LETTERS
Got an idea for the magazine? Or a great discovery? Email us: letters@linuxvoice.com
ANGELS WITH HAIRY FACES
Ben – when I look at the photos of you and your three
colleagues on LV’s Welcome to Linux page, I have just
realised that I can improve my Linux skills at a stroke… by
STAR growing a beard!
LETTER Godfrey Green, Cardigan, Wales
Ben says: Forget about us: look at Richard Stallman
and Alan Cox, or go further back to the Unix days
of beard glory and look at Brian Kernighan, Dennis
Ritchie and Ken Thompson – magnificent beasts
all of them. We’ve got a long way to go. But we’re
confident we can make it. Except Mike.
Let’s all just ignore the sexism implied in the notion that you
need to have a beard to be a leet Linux user…
BOOTSTRAP
Bootstrap (p78 issue 28), with its ‘grid-based layout because the web is relatively new, there’s a million
system,’ represents best practice in the first decade of this ways of doing things that are all the ‘right’ way, and
century and such systems remain very popular with CMS things will keep changing depending on whatever
vendors and professional developers who learned their technology is in vogue this week. We do know for
trade in the earlier part of this century. certain though that, thanks to the web’s inherent
But Apple, Mozilla and Opera proposed a flow-process openess, the barriers to entry will remain low,
approach, which was adopted – albeit in the face of great so anyone can be an expert. And of course, Free
resistance from traditionalists – in 2011. Software means we’ll always have lots of tools to
With it I have developed two websites each with a single play around with – such as Hugo, which we explore
CSS file, which enables content to be displayed on on page 68.
anything from a 19-inch Apple monitor to an Xperia2
smartphone in configurations suitable for each screen.
For example, three images in a page display one above
each other on the smartphone and side by side on the
Apple because the content flows to suit the dimensions of
the viewport.
There is no need for multiple CSS files and the recently
introduced v unit makes it easy to ensure that images
adjust their size to the available viewport.
There is a reasonably comprehensive account of how I
have done this at http://heatholdboys.org.uk/content/
HOBA_website_documentation.pdf, and it might be worth
considering an article on the pros and cons of these
different approaches to website design.
John
Whether you prefer grid- or flow-based design, Linux is our
Ben says: Thanks John. It seems to me that favourite platform for web and app design.
10
YOUR LETTERS MAIL
CONVERSION
I just thought I’d drop you a line about my most recent
Linux convert. My mum has an aged PC in the corner that
she uses to watch YouTube videos. She has loads of
precious vinyl records in the garage in tattered plastic
bags that never get played because she’d rather have the
convenience of playing music through the computer,
including bands that seemingly only she can remember.
This machine has been staggering along on Windows XP
and taking increasingly long to do anything as a decade of Mint no longer
ships with all the
malware has slowed it to a crawl. Anyway she finally
media codecs
trusted me enough to put Linux Mint on it, and it now
included, but it’s
takes a hammering every Friday night when she’s had a still easy to set up
few and wants to relive the glory nights of Motown. a drunken jukebox
Another happy customer. for family and
Sarah Barnes, County Durham friends.
UBLOATU
One of the things that drew me to Linux was how
streamlined it is. There’s no need to install things you don’t
need, wasting time and bandwidth. I was an early Ubuntu
user, and benefitted from one of the free CDs that
Canonical used to post out to help with Ubuntu adoption.
It also makes sense to keep a Linux distro small as they
are often used in rural areas where bandwidth isn’t as
good as it is in the developed world. So why on earth is
Ubuntu making its next release 2GB? It’s going to be as
big as Debian at this rate, so what’s the point?
David Wilkes, Shrewsbury
Ubuntu, we still
it, but maybe Ubuntu isn’t the best choice for you;
love you. Warty
Andrew says: As projects change, so their aims try Xubuntu instead and go from there. If you want
warts and all.
change. Ubuntu made its first release 12 years a user-friendly distribution that makes sensible
ago, so it’s not surprising that it’s shifted from its default choices on behalf of the the users, Ubuntu
original goals. If you want light, you can still get is still a good bet.
www.linuxvoice.com 11
SUBSCRIBE
Subscribe
shop.linuxvoice.com
Get your regular dose
of Linux Voice, the
magazine that:
Gives 50% of its profits
back to Free Software
Licenses its content
SUBSCRIBE TO CC-BY-SA within 9 months
US/Canada subs prices
TODAY! 1-year print & digital: £95
12-month digital only: £38
Get many pages Access our Save money on
of tutorials, rapidly growing the shop price
features, interviews back-issues archive and get each issue
and reviews – all DRM-free and delivered to
every month ready to download your door
Payment is in Pounds Sterling. 12-month subscribers will receive 12 issues of Linux Voice a year. 7-month
subscribers will receive 7 issue of Linux Voice. If you are dissatisfied in any way you can write to us to cancel your
subscription at subscriptions@linuxvoice.com and we will refund you for all unmailed issues.
12 www.linuxvoice.com
SUBSCRIBE
All subscribers get access to every
single digital back issue –
that’s about 1,000,000 words of
tutorials, reviews and free software
hackery at your fingertips
Overseas subs prices
12-month print & digital:
Europe: £85
US/Canada: £95
Rest of world: £99 DIGITAL
SUBSCRIPTION*
ONLY
*
£38
WHEREVER IN THE WORLD YOU
ARE – IT’S DIGITAL, SO THERE ARE
NO POSTAGE COSTS
www.linuxvoice.com 13
FEATURE INSIDE THE LINUX KERNEL
{ KERNEL {
INSIDE THE
LINUX
As any sufficiently advanced technology is indistinguishable from
magic, Dr Valentine Sinitsyn takes his wand and a cloak to show
you around the dark corridors of the Linux kernel dungeons.
W
henever you say: “I’m a Linux user,” Richard and does many other things. For us, computers
Stallman raises his finger and rectifies: “just work”: we open a browser then switch to a
“GNU/Linux.” His choice of nomenclature word processor, then back again. We in plug a
gives the GNU project credit for creating the majority flash memory card and upload our holiday snaps
of the user-level components we run every day. Bash? somewhere. It’s intuitive, and we hardly ever think that
It’s from GNU. GCC? The same. Think of a Linux there are complex algorithms in place to make things
command, and check its origin – most likely, you use run smoothly.
a GNU variant. You probably know that Linux runs on many
What’s Linux, then? It’s a kernel – “a computer platforms. This includes a whole range from
program that constitutes the central core of a embedded MIPS or ARM processors through
computer’s operating system”. The kernel isn’t commodity x86 and PowerPC to heavyweights like
something you interact with directly as a user. It is IBM mainframes. Linux supports all this diversity from
what gives the operating system its identity. Running a single codebase. To make this possible, the kernel
Bash on Windows doesn’t make it Linux. Even the carefully separates generic and architecture-specific
latest Windows Subsystem for Linux just emulates parts. For example, the process scheduling algorithm
the Linux kernel on top of NT one. It doesn’t matter if is generic. The code to switch process contexts
you prefer Ubuntu, Fedora, SUSE or Arch: they may naturally is not, as each architecture has its own set of
feel different on the surface, yet they are all Linuxes. registers. So, the scheduler chooses the next process
FreeBSD or Illumos may look familiar as they are Unix to run, then calls the architecture-specific code to
variants. Still, they are separate from Linux and each apply the change. Generic parts are in C; low-level
other, as they build on different kernels. architecture-specific operations use Assembler.
The kernel’s job is often invisible. It schedules C is a procedural language. Yet kernel developers
processes, manages memory and peripheral devices, employ object-oriented or even a functional style
The Linux kernel carefully separates
generic and architecture specific parts
14 www.linuxvoice.com
INSIDE THE LINUX KERNEL FEATURE
across the codebase. Many kernel- kernel is also preemptive (many other
level concepts are represented as C kernels aren’t). This means that after
structures with embedded function the interrupt, the scheduler may decide
pointers. Effectively, they act as objects to return control to some other code,
with virtual methods that you find not yours. What if that code accesses
in languages such as C++ or Java. the object you were working with?
Consider the filp (“file pointer”) usage These are only a few examples of what
below: a kernel hacker should keep in mind
long vfs_ioctl(struct file *filp, unsigned when writing kernel-level code. Having
int cmd, unsigned long arg) in-kernel synchronisation the wrong
{ way is disastrous.
int error = -ENOTTY;
if (!filp->f_op->unlocked_ioctl) We are the kernel
goto out; The Linux kernel comprises various
error = filp->f_op->unlocked_ subsystems corresponding to different
ioctl(filp, cmd, arg); features it provides. Subsystems are
if (error == -ENOIOCTLCMD) logically separated from each other
error = -ENOTTY; and typically have dedicated directories The Linux Kernel Archives website is the official kernel
out: in the source tree. However, as Linux homepage. Here you get “vanilla” kernel sources.
return error; is the monolithic kernel, they aren’t
} isolated from each other when the present. Memory management lives in
Programming in the kernel is still kernel is running. mm and arch/.../mm.
rather different from userspace. Stack It’s difficult to identify a single “most Perhaps the most populated
space is limited, and you can’t do important” subsystem, but we’d vote directory in the kernel sources is
floating point math easily. Any bug in for the process scheduler. It selects drivers/. Unsurprisingly, this stores
your code affects the whole system. processes to run and dictates for how device drivers for peripherals that Linux
You can’t expect the kernel to deliver long they will own the CPU. Scheduler supports (there are quite a few). Having
you SIGSEGV for a NULL pointer error directly affects both the system a driver in-tree isn’t a requirement,
in your code, because you are now the performance and responsiveness (or as one can also wrap it as a kernel
kernel. Many issues you don’t even latency). Scheduler sources are found module. Support for PCI, USB, and
think about in userspace code become in the kernel/sched directory. other buses also comes through the
your responsibility in the kernel. Then comes the memory. The kernel drivers subsystem.
Perhaps the most important one is does two main things to manage your There are many other kernel
concurrency. In userspace, it’s mostly gigabytes. First, it allocates chunks subsystems, such as networking. Many
a concern for multithreaded programs. called pages for the userspace and for of them are worth a book on their own.
The synchronous code runs line by itself. When a process wants some We aren’t going to dig that deep today,
line. You are the only owner of the more RAM, it issues brk(2) or mmap(2) so turn over the page and let’s look into
data unless you share it with other system calls to increase the heap process scheduler operation.
processes somehow. (LV018). Second, the kernel configures
That’s not the case in the kernel. hardware-dependent mechanisms to KERNEL VERSIONING
It’s asynchronous by its very nature. provide each process an isolated virtual
Imagine an interrupt occurs while the address space. This includes swapping
CPU is executing your code. What out pages to disk and reading them Like any self-respecting piece of software, the Linux kernel
happens if a driver decides to update back when necessary so that a process has a version number. You can check what version your
current kernel is with the following command:
the buffer you were reading? The Linux may use more memory than physically
$ uname -r
4.5.4-1-ARCH
The kernel version is first three dotted numbers (major,
release and patch level). Everything after the dash is what
your distribution adds.
Linux 2.6 was released in December 2003. The 2.6
series lasted for more than seven years. In May 2011, Linus
decided 3.0.0 would follow then-current 2.6.39. The reasons
weren’t technical; officially, it was to celebrate the 20th
anniversary of Linux. Yet we all know the real reason was
that Linus can’t count up to 40 (proof: https://lkml.org/
lkml/2011/5/29/204). Linux 4.0 was released in April 2015.
This was also a non-technical bump. Linux Kernel Newbies
suggests that “the less you think about it, the better”
(http://kernelnewbies.org/Linux_4.0).
You can learn more about Linux kernel versions at
https://www.kernel.org/category/releases.html.
Linux Kernel Newbies is a site aimed at those who wants to start hacking the kernel.
www.linuxvoice.com 15
FEATURE INSIDE THE LINUX KERNEL
PROCESSES AND SCHEDULING
Computers are a means to run programs. Here’s how the kernel makes it happen.
(struct task_struct), defined in linux/
sched.h.
The kernel defines the current macro,
which returns the descriptor for the
currently running process. This is a
popular operation, and it must be fast.
For example, the x86 architecture uses
a per-CPU variable. This makes the
descriptor accessible at the known
offset from the %gs or %fs segment.
Most other architectures put a pointer
to the current process descriptor in the
bottom of the kernel stack.
Every process has a userspace stack,
typically located at the top of userspace
memory. It is used for function calls
htop displays kernel threads alongside userspace ones. Note they are all ktrheadd descendants. and local variables. The kernel mode
stack serves the same purposes,
A process is an instance of a computer the kernel executes some operations but it is used when a process runs
program that is being executed. A on behalf of the process. There are, in kernel mode (say, during a system
typical Linux system hosts many more however, processes (better said, call). The user mode stack is relatively
processes than there are CPU cores threads) that run exclusively in the large (several megabytes) and grows
available, so arithmetic dictates that the kernel space. They are all kthreadd dynamically as needed. The kernel
kernel must share computing resources children, and they serve various mode stack is small (8KB for x86 and
among these processes somehow. purposes. For example, ksoftirqd helps 16KB for x86-64) and static. There are
Even on a uniprocessor system, servicing hardware interrupts; kswapd also auxiliary kernel stacks for interrupt
everything looks as if processes run in swaps out memory pages; and jbd2 and exception handlers, but you can
parallel. flushes the filesystem journal to the ignore these details for now.
Many operating systems permit disk. The kernel also distinguishes
multiple threads of execution within between process states. A process can
one process. Linux does this as well, Process descriptors be runnable, meaning a scheduler can
yet it doesn’t distinguish between Each process has resources that the pick it up for execution. Or, it can be
threads and processes at the kernel kernel needs to track. So, the kernel sleeping (blocked), either interruptible
level. Threads are just processes that creates a so-called process descriptor or uninterruptible. The difference is that
have common memory space, file for each process in a system. It stores interruptible sleep can be terminated
descriptors, and a few other things. the process address space, open with a signal. When you can’t stop a
Linux creates processes with so- files and pending signals, along with program with Ctrl+C or kill, it means
called “forking”. One process (a parent) internal kernel bits like the process the process is in uninterruptible sleep
creates an exact copy of itself (a child), state and flags. Technically, the process somewhere in the kernel. ps marks
which then runs independently. The descriptor is a lengthy C structure such processes with D.
clone() system call implements this
procedure both for processes and for pid: 344
threads. vruntime 100 ms
Cloning (or forking) a process Leftmost runs first
implies copying all of its memory. This
doesn’t sound lightweight. Moreover,
pid: 128 pid: 1032
it’s often useless, as a typical child
vruntime 75 ms vruntime 150 ms
immediately issues execve() to run
another executable. For optimisation,
Linux employs the copy-on-write (COW)
Empty Empty Empty Empty
technique and doesn’t copy memory
unless necessary.
Most processes run in userspace. Red-black tree is a self-balancing data structure that keeps its height low. CFS uses it to
When they do a system call like read(), find the next process to run quickly.
16 www.linuxvoice.com
INSIDE THE LINUX KERNEL FEATURE
When it’s time to reschedule, the
kernel selects a process to run next and }
schedule();
OTHER KERNEL RESOURCES
does a “context switch”. That is, it refills __set_current_state(TASK_RUNNING);
the CPU registers with new values and remove_wait_queue(&devp->hd_ If this month’s cover feature has whetted your appetite, and
sets up a new address space. When the waitqueue, &wait); you want to learn more, there are numerous good resources.
switch completes, a new process thinks } For starters, consider the already-mentioned Linux
it just returned from a kernel function The hpet_read() method implements Kernel Newbies, available at http://kernelnewbies.org.
It’s a community website, perhaps best-known for its
call and continues normally. the read() system call for /dev/hpet.
no-nonsense kernel changelogs. Also have a look at Kernel
When an HPET interrupt occurs, its Planet (http://planet.kernel.org), which gathers blogs from
Calling for the scheduler handler updates hd_irqdata and calls many kernel developers. Many new kernel features are being
How does the kernel know it’s time to wake_up_interruptible(&devp->hd_ discussed at Linux Weekly News (http://lwn.net). Note that
reschedule? Every few milliseconds, waitqueue);. these two resources aren’t aimed at beginners, though.
There are also several good kernel books. The problem
it calls the scheduler_tick() function. Many kernels don’t reschedule
is that most of them are at least five years old. They cover
It checks if the current process had a process when it is in the kernel Linux 2.6, which is not much different from now-current
already been running for too long. mode. Linux would do this unless 4.x, as you already know. Ultimately, their age is not a big
If this is the case, a special flag kernel preemption was temporarily deal, as fundamental parts of the kernel rarely change. But
TIF_NEED_RESCHED is set on the disabled with preempt_disable(). of course there will be some differences between what you
read and up-to-date code.
process descriptor. Later, when the Kernel preemption occurs when Linux
For the first read, consider the Linux Kernel Development,
kernel returns control to the userspace finishes servicing a hardware interrupt 3rd Edition by Robert Love. It’s relatively short and high-level
after servicing an interrupt or a system and returns control to the kernel mode enough to build a complete picture without digging too
call, this flag is examined. If it is set, code. It may also happen when the much into details. You may also consider Professional Linux
schedule() is called. kernel enables preemption back with Kernel Architecture by Wolfgang Mauerer, which is a bit older,
a bit thicker and more in-depth.
schedule() is the scheduler’s main preempt_enable().
entry point. Many kernel functions call
it explicitly when they need current Schedulers galore
to sleep. A common reason is that a Linux sports different scheduling also sets some maximum value. Then,
process wants to read data that isn’t algorithms often called classes. it introduces the concept of virtual
available at the time. Completely Fair Scheduler (CFS) is the running time (vruntime), which is a
First, a process descriptor is added default. There are also two real-time process runtime weighted by its priority
to the wait queue. Then, current is classes for higher priority processes (see nice(2)). On an ideal multitasking
rescheduled. Later, some other code and the Earliest Deadline First (EDF) CPU, all processes of the same priority
The kernel creates a so-called process
descriptor for each process in a system
wakes up processes on the queue. For scheduler for real-time processes would have the same vruntime. So,
a blocked process, this looks like the with timing constraints. You can set a CFS picks a process with a minimum
schedule() function has returned. This scheduler class for the process with vruntime and runs it. Real-time
is how it happens in /dev/hpet (a high- the sched_setscheduler() system call. scheduling classes are a bit simpler.
precision timer) device driver: EDF processes take precedence over SCHED_FIFO processes run until they
static ssize_t real time processes, which run before decide to relinquish the CPU. SCHED_
hpet_read(struct file *file, char __user normal ones. RR processes are given a timeslice and
*buf, size_t count, loff_t * ppos) CFS builds on a simple idea. Imagine are scheduled in round-robin fashion.
{ “an ideal multitasking CPU” that When a process has consumed its
DECLARE_WAITQUEUE(wait, current); precisely shares its 100% physical timeslice, it is refilled, and the process is
struct hpet_dev *devp; power between running tasks. This added to the end of the queue.
CPU would execute two tasks really The deadline scheduler (EDF) is a
devp = file->private_data; in parallel, devoting 50% power to new guy on the block. Introduced with
add_wait_queue(&devp->hd_waitqueue, each. CFS models this on top of real Linux 3.14, it ensures that a process is
&wait); hardware. given a predefined time to run within
for ( ; ; ) { A real CPU can run only one process each accounting period. Say, you may
set_current_state(TASK_ per core at time. Context switches are want a process to run for 20ms every
INTERRUPTIBLE); costly, so CFS sets a minimum time 100ms. This is important for time-
data = devp->hd_irqdata; a process can execute. If a process sensitive applications. The algorithm
if (data) doesn’t yield the CPU for too long, the chooses the process with the earliest
break; system becomes unresponsive, so it deadline, hence the name.
www.linuxvoice.com 17
FEATURE INSIDE THE LINUX KERNEL
MEMORY MANAGER
Take care of your memory – all 640KB of it.
Most architectures today provide a page size is 4KB, it takes a few percent from the standard C library, which is a
Memory Management Unit (MMU). You of the memory available. This doesn’t user-level thing. So, the kernel needs its
may think of the MMU as a chip that seem to be a prohibitive price to pay for own memory allocator.
translates memory addresses and a useful metadata struct page. At the lowest level in the kernel lies
enforces memory protection. This Pages are organised into zones. The the page allocator. This shouldn’t come
paves the way for process-isolated reason for this is that not all pages as a surprise, as the page is the basic
virtual address spaces. MMU operates are born equal. Old ISA devices, for memory management unit. Yet a page
on pages. Typically, a page is 4KB, instance, can only use memory in the is relatively big chunk of data, and
although it’s possible to create huge lowest 16MB. 32-bit devices can’t see no one goes to ask for 4KB if all they
pages spanning megabytes or even memory above 4GB. Naturally, one need is 128 bytes. So, there are other
gigabytes. The idea is to increase doesn’t allocate “normal” memory there mechanisms built on top of the page
granularity to minimise costs. If the unless absolutely necessary. Zones are allocator as well.
MMU wants just one bit of metadata what facilitates making such decisions A memory allocator often has two
per memory byte, this means 12.5% in the kernel. main goals. First, it needs to keep
overhead with a byte-level granularity, A 64-bit PC has most of its memory memory fragmentation low in the long
but only 0.003% with 4K pages. in the Normal and DMA32 zones. A 32- run. Otherwise, it wouldn’t be possible
Linux also needs to track memory bit system is likely to have an additional to allocate a large chunk of memory,
metadata, and it also does it with page- HighMem zone. The kernel address even if it is formally available. Imagine
level granularity. Each physical page space is typically 1GB on 32-bit hosts, memory where all even pages are used,
has an associated struct page structure and HighMem includes memory above and all odd pages are free. One can’t
in the kernel. It tells if the page is free that. On 64-bit systems, the address allocate a block larger than one page,
or not, or if it is dirty (that is, has some space is a huge 64TB, and everything even if half the memory is empty.
data not on the disk yet). In short, struct fits in Normal. To prevent this situation, you can
page is like struct task_struct in that allocate and free up memory in large
it stores all the information the kernel Meet your buddies blocks. And here comes another goal:
needs to manage a memory page. All C programmers know malloc(). You the allocator shouldn’t waste too much
Having a structure per single page tell this function how many bytes of memory. Linux addresses both points
may seem like a waste. If struct page memory you want, and it finds a big with a variation of algorithm known as
is about 100 bytes (actually less), and enough chunk for you. malloc() comes a buddy system. It allocates pages in
Kernel image User page User page User page Physical memory (x86-64)
0x1000000 (16M)
0x3FF0000 (+512M)
Userspace Kernel space
Hole
Hole Virtual memory (x86-64)
0xffffc7ffffffffff
0x00007fffffffffff
0xffff800000000000
0xffff880000000000
Physical memory map
(64 TB)
The kernel builds virtual address spaces for userspace processes and for itself. We discussed it in LV018, now online.
18 www.linuxvoice.com
INSIDE THE LINUX KERNEL FEATURE
power-of-two chunks: 1, 2, 4, 8 and so corresponding kernel functions carry ti = page_address(alloc_kmem_pages_
on. Each size has a separate free list, the kmem_cache_ prefix. An object’s node(node, THREADINFO_GFP, THREAD_
or list of pages (struct page), which are memory comes from page-sized SIZE_ORDER));
available. If someone asks for 3 pages, blocks. They are called “slabs”, and are if (!ti)
the allocator looks up the 4-page list. If allocated via a buddy system. When return NULL;
it’s empty, the allocator splits an 8-page the request comes, and there are no ...
block in two halves called “buddies” free objects in a cache, a new page is }
(hence the name). Then it returns the allocated and initialised. The kernel also Here, struct task_struct comes from
first half to the caller and adds the aligns objects in a slab the way they a slab; struct thread_info comes from
second to the 4-page list. Later, when don’t step on each other’s toe in CPU the buddy system. Finally, there is the
the first buddy is freed, the allocator hardware caches. kmalloc() function to allocate arbitrary-
detects it and promotes a combined SLAB is a classic implementation of sized buffers. To facilitate it, the kernel
8-page block back to the original list. this idea that first appeared in Solaris. has several caches (kmalloc-N) for
cat /proc/buddyinfo tells how many SLOB is its low-footprint variant. It power-of-two sized blocks smaller than
free blocks are currently available in works best for smaller systems but two pages. For larger blocks, pages are
each of your system’s zones. The isn’t that good on larger ones. SLUB allocated via a buddy system.
When the kernel forks a process, it allocates a
new process descriptor and kernel stack space
number of pages in the block (also introduced some optimisations to So far, we’ve spoken of physical
called an “order”) increases from left to original SLAB, which makes it the memory. However, programs and the
right: 0, 1, 2 and so on. default in recent Linux kernels. kernel itself run in a virtual address
When the kernel forks a process, it space, so the same address may
SLAB, SLOB, and SLUB allocates a new process descriptor refer to different memory locations in
The kernel also provides a way to and kernel stack space. This occurs in userspace processes (the kernel has
allocate specific objects, such as struct dup_task_struct(): a single virtual address space). When
task_struct. They are sometimes small static struct task_struct *dup_task_ MMU detects the program tries to
and short-lived, which warrants them struct(struct task_struct *orig) touch an address which is not mapped,
special treatment. To this end, the { it calls the page fault handler in the
kernel has three related algorithms: struct task_struct *tsk; kernel. The latter decides whether the
SLAB, SLUB and SLOB. struct thread_info *ti; fault should result in system panic (if
The idea is to keep “ready to tsk = kmem_cache_alloc_node(task_ it occurred within the kernel), or just
consume” objects and return them struct_cachep, GFP_KERNEL, node); SIGSEGV. Sometimes, the fault is not a
quickly without allocating any memory. if (!tsk) problem at all, as the page in question
This also serves as a cache, hence return NULL; has been just swapped out.
STEP BY STEP: PEEK INTO KERNEL CACHES WITH SLABINFO
BUILD THE TOOL GET SOME STATS PLOT SOME GRAPHS
slabinfo comes together with the Linux kernel. Grab Now run the tool as ./slabinfo (root permissions are A picture is worth a thousand words. Build some
the latest version from www.kernel.org (it’s big), required). It will produce a lengthy table containing totals with while [ 1 ]; do ./slabinfo -X >> stats;
untar, cd into linux-X.Y.Z/tools/vm, then do make all slab caches found in your system along with sleep 1; done (wait a dozen of seconds). Then run
slabinfo. You’ll need build-essentials or the similar some stats. As an exercise, try to find the kmalloc or bash ./slabinfo-gnuplot.sh stats to build graphs.
package from your Linux distribution. task_struct caches. How many objects are in them? Green is slab size. Red is loss (how much space the
What’s the object size? slab wastes).
www.linuxvoice.com 19
FEATURE INSIDE THE LINUX KERNEL
SYSTEM CALLS
The kernel is here so you can focus on your job, not managing the hardware.
wraps the system call of the same
name. This doesn’t have to be the
case in all Unixes, though. uname(3) is
POSIX standard, uname(2) is, in theory,
Linux-specific.
Not all C library functions are simple
wrappers, of course. Linux defines a
few hundred system calls, and many
of them are rather low-level. Others
multiplex several userspace visible
functions. There are system calls that
are specific to hardware architecture.
Often, they carry the arch_ prefix. There
are system calls that have no C library
wrappers, such as futex(2) or native
asynchronous I/O family (LV026). No
cscope is a venerable source code navigation tool. It’s old enough to remembers the days of PDP-11, wrappers are a fat hint that you don’t
and it’s still useful enough to be relevant in 2016. Good job, cscope! want to use these system calls in your
programs directly. Yet it is still possible
Up until now, we saw the kernel as interface are carefully chosen, and to issue them the indirect way via
something transparent, sitting between arguments you pass are thoroughly syscall(), as the C library does.
our programs and the hardware, and validated. So, rare-yet-possible bugs In fact, Linux distinguishes system
providing some useful abstractions to aside, system calls provide a safe way calls not by names, but by numbers.
the former. From time to time, the to call into kernel functions. SYS_uname is a macro that expands
programs want to request some to 63 on 64-bit x86 machines. This
service from the kernel explicitly. The role of libc number is also architecture-dependent:
Without that, it won’t be possible to As a programmer, you almost never it’s 122 for 32-bit x86, for instance.
read and write files, or exchange data deal with system calls directly. The Internally, the kernel recognises this
over a network. standard C library (nine out of ten system call as “new uname”. There
For security reasons, the kernel is times, Glibc) wraps them to provide a are also “old uname” and even “old old
isolated from the rest of the system. standard API such as POSIX. Imagine uname.” It’s not uncommon to have
This happens at hardware level (think you want to get some information such a convoluted history in the world
MMU again), and this means you can’t about the system you run on. POSIX of system calls. From time to time,
You can think of a sysytem call as an interrupt
that you generate to draw the kernel’s attention
simply call a kernel function from defines the uname(3) function and kernel developers make incompatible
userspace. Instead, one uses a system associated structure definitions for this changes to data structures but leave
call interface as a well-defined gateway purpose. It’s available on many Unixes, the semantics unchanged. It seems
to the kernel. and its Linux implementation may be reasonable to declare an earlier
This isn’t the only means to switch to as simple as this: implementation as “old”. Luckily, a
the kernel mode: hardware interrupts int uname(struct utsname *uts) standard C library hides all these
incur this as well, but they are outside { nuances from mere mortals.
the programmer’s control. You may return syscall(SYS_uname, uts); The exact method of making a
think of a system call as an interrupt } system call, that is, switching to kernel
that you, not hardware, generate to This code comes from Musl (https:// mode, is also architecture-specific.
draw the kernel’s attention. Services www.musl-libc.org), an alternative C On x86 computers, this used to be
available through the system call library for Linux. As you see, it simply a software interrupt, int $0x80. You
20 www.linuxvoice.com
INSIDE THE LINUX KERNEL FEATURE
may still find it in older manuals and
tutorials. This method is still supported,
a shared library, vDSO defines several
symbols (think functions). One of
FINDING YOUR WAY
yet rarely used. The reason is that these functions executes the preferred
interrupts are quite costly. As programs instruction for the system call. As If you are wondering how Linux works, the sources are the
do system calls very often, this may the kernel builds and maps vDSO on ultimate answer. However, the kernel is a large program.
hurt the performance. So newer CPUs its own, it has complete control of Without dedicated navigation tools, it’s very easy to get lost
introduced dedicated instructions what this instruction would be. Other in its 50K+ source files.
You need to look up identifiers, like variables or function
for fast switching to the kernel mode vDSO symbols may provide optimised
names, and search for raw text strings. You’d want to jump
and returning the control back. Intel versions for selected system calls (like directly to locations where these identifiers are defined, but
processors implement sysenter/ gettimeofday(2)), which don’t incur also list locations where they occur in the code. Quite often,
sysexit instructions and AMD chips switch to the kernel at all. you’d also want to know where some variable gets its value.
have syscall/sysret. There are also Either way, when a switch to the Luckily, there are several free tools to help you with all
of these. A de-facto standard one is Cscope. It wasn’t built
some peculiarities related to 64-bit and kernel mode occurs, a system call
specifically for the Linux kernel. In fact, it’s almost as old
32-bit modes. How does the poor C dispatcher in the kernel ultimately runs. as Unix. Born in Bell Labs, it was a part of AT&T. Santa
library account for all these specifics It analyses the system call number and Cruz Operations, a predecessor to the ill-fated SCO Group,
when issuing system calls? calls an appropriate implementation. released Cscope under the BSD licence in 2000.
By convention, it carries the sys_ prefix. Linux provides a dedicated Makefile target to generate
the Cscope database. Running make cscope && cscope
Nuts and bolts For our good old friend uname(2), this
-d will bring you a text-based interface. Use the arrow keys
In short, it doesn’t. Otherwise, it will be sys_newuname(). This function to choose what you’re looking for (say, a C symbol), type
won’t be possible to introduce a new lives in kernel/sys.c: the search terms and hit Enter. Results are displayed in
switching method in the kernel without SYSCALL_DEFINE1(newuname, struct the upper half of the screen. Press Tab to switch between
touching every C library implementation new_utsname __user *, name) halves. Select the result with Up and Down keys, or press a
single-letter hotkey to open the location in $EDITOR. If your
in the world. This doesn’t scale well. {
search yielded many results, use Space to turn pages. To
Instead, the kernel maps a “vDSO” int errno = 0; exit, press Ctrl+D. The ? key brings the help page.
(virtual dynamic shared object) in every down_read(&uts_sem); For a friendlier alternative, try LXR (Linux Cross
process address space: if (copy_to_user(name, Referencer). It’s a web application built specifically for Linux,
$ cat /proc/self/maps utsname(), sizeof *name)) yet found its usages in other projects (Mozilla). You can find
LXR online here: http://lxr.free-electrons.com.
... errno = -EFAULT;
First, select the kernel version. This defaults to the latest
7ffd86721000-7ffd86742000 rw-p up_read(&uts_sem); mainline. You can navigate the sources manually in the
00000000 00:00 0 [stack] return errno; Source Navigation tab. If you want to trace the origins of
7ffd867e2000-7ffd867e4000 r-xp } some log message, LXR should be your first stop.
00000000 00:00 0 [vdso] As before, we simplified the
ffffffffff600000-ffffffffff601000 r-xp implementation a bit to highlight the
00000000 00:00 0 [vsyscall] main points. The sys_ prefix is added
A vDSO is much like a shared library in the SYSCALL_DEFINE1 macro. do_something() kernel function that
which is always present. You may The code calls the utsname() kernel does all heavy lifting. A semaphore,
think it’s a waste of RAM. Again, it’s function, which gathers the required uts_sem, protects the structure from
not, because Linux keeps only one data from an internal kernel structure, concurrent access (didn’t I tell you
copy of a non-writable page (note the and copies bytes to the userspace that kernel programming is always
r-xp permission bits) in memory. As memory. Sometimes, there is a asynchronous?). Also note that
negative return values indicate an
error. The convention in userspace is
different, so the C library detects it and
sets the errno variable appropriately.
We hope you enjoyed your trip
under the Linux kernel surface. The
kernel is huge, and there are many
books devoted to single subsystems
of it. They cover many nuances which
are essential if you are serious about
kernel programing. Yet the kernel is no
magic, just different from what we have
in a comfy userspace. It also implies
a good understanding of computer
architecture, even for generic pieces
like the memory allocator. Not many
of us write kernel code (if you have a
patch accepted, please let us know!),
With LXR, you can navigate kernel sources straight from you web browser. It comes in but it’s always fun to learn how gears fit
handy when you read things like this web page. together.
www.linuxvoice.com 21
FEATURE INSIDE LIBREOFFICE
inside
Along with GNU/Linux and Firefox, LibreOffice is
one of the biggest success stories in Free Software.
Find out where it came from, and how it’s developed.
A
sk a Linux user to name some really on office suites could at last be broken
exciting things going on in the Free thanks to contenders like LibreOffice.
Software world right now, and they And the reason is simple: it’s a much
will probably mention Gnome or KDE easier job to switch an office suite than it
updates, or perhaps some flashy new is to migrate an entire operating system.
graphical effects promised by Wayland, or Consider how much success Firefox had in
maybe some awesome new games in the its early days; Microsoft’s Internet Explorer
pipeline. Even new programming languages, utterly dominated the browser market, and
container technology and Firefox updates many said it could never be beaten. But
can be fascinating. But office suites? Surely Firefox’s better performance, security and
that’s the least exciting thing on this featureset brought it up to 30% marketshare,
wonderful green planet – right? while Internet Explorer usage dropped quickly
Well, no. It’s true that the humble office (especially thanks to competition from enough, while some bemoan LibreOffice’s
suite is a mere tool for getting things done, Google’s Chrome). “old-style” interface, many users of earlier
a quotidian aid for productivity rather than Microsoft Office versions prefer switching to
something to wow and entertain you. But Reach for the prize LibreOffice than having to use the ribbon that
this is a topic that’s well worth keeping an We could see the same with LibreOffice. Microsoft is so proud of.
eye on, as it has the potential to completely Sure, switching office suites is a bigger job So, tens of millions of people use
change the computing landscape. We won’t than changing browsers, but it’s certainly LibreOffice each day, the project is growing,
see Linux completely dominate the desktop possible. The French government has and it looks to have a healthy future. But
market in the next few years – the days migrated hundreds of thousands of where did it come from, how does it work
when we’d get excited about the year of computers from Microsoft Office to internally, and what’s in the pipeline for the
Linux on the DesktopTM are over – but there’s LibreOffice, and the Italian military is also future? Over the next few pages, we’ll give
a chance that Microsoft’s de facto monopoly undergoing a major transition. And funnily you all the details.
Tens of millions of people use LibreOffice each day, the
project is growing, and it looks to have a healthy future
22 www.linuxvoice.com
INSIDE LIBREOFFICE FEATURE
FROM HUMBLE BEGINNINGS
Many of us cut our teeth on Linux back in the late
1990s, as it started to develop from a hacker’s
plaything into a viable operating system for servers
and home desktops. Around the time, many computer
magazines started featuring Linux distributions on
their coverdiscs, along with a certain office suite
called StarOffice. This suite had a strange look and
feel: it tried to ape the Windows 95 desktop with a
taskbar and Start-like button, and managed to
maintain this design across the various platforms on
which it ran (Windows, Solaris, Linux).
StarOffice came into the world in 1985 as StarWriter,
a word processor developed by a German student
for the 8-bit home computers at the time. In the
following years, more components of the office suite
were added and it was ported to MS-DOS, IBM’s OS/2
and Microsoft Windows. StarOffice was a small but
important player in the early 1990s, but something
happened that suddenly elevated it into the spotlight.
Unix giant Sun Microsystems (which is today
owned by Oracle) was looking to install Microsoft LibreOffice’s codebase
Office on its 42,000 workstations. That’s a hefty One of the goals of LibreOffice was to be as dates back to StarOffice
licensing cost – so Sun decided to buy StarOffice independent from control of a single company as in the 1980s (image
outright in 1999 as a cost-saving measure. StarOffice possible. To that end, the team behind it set up The credit: http://tinyurl.com/
was also made available as a free download, so Document Foundation (TDF), a non-profit entity heuv5q8).
even while it was seen as a bit lacking and clunky in registered in Berlin and with members across the
comparison to Microsoft’s offering, many people used planet. This was a bold move, especially for such a
it as a cost-cutting measure alone. fledgling project, but it established a structure and
But then Sun did something drastic: the company statutes for the project to ensure democracy and
open sourced StarOffice, creating OpenOffice.org, transparency throughout. TDF is comprised of several
which was a monumental effort. The suite was built bodies, including:
from millions of lines of source code, many of which The board of directors The main administration of
still contained German comments right from the early
days of the suite, and getting the FOSS community on
board was a difficult task. Early releases of OpenOffice. Early releases of OpenOffice.org were slow to use
org were slow to use and glitchy in places but provided
Linux distributions with a powerful office suite that
and glitchy in places, but provided Linux distros
abided by open source principles. with a powerful open source office suite
While Sun maintained control of the OpenOffice.org
project, as the 2000s progressed a small community
of independent developers, supporters and marketers LibreOffice’s projects and teams.
built up around it. There was some concern that Sun The membership committee Administers
had too much sway in the project, and these fears membership applications, and oversees election of
only got worse when Oracle snapped up Sun in 2010 the board of directors.
– what would Oracle, hardly the biggest champions of The advisory board For companies and
open source in the world, do with the software now? organisations that support LibreOffice to provide
ideas and advice.
A fork in the road The statutes of TDF state the following: “The board
So a team of OpenOffice.org developers and of directors is therefore obliged to ensure, that the
supporters from the community forked the code and board of directors itself, the membership committee,
created LibreOffice in late 2010. This was one of the and the advisory board, at maximum have one third of
biggest splits in FOSS history, and with few developers their members being employed by a single company,
left working on OpenOffice.org, Oracle decided to hand organisation, entity or their respective affiliates”. This
it over to The Apache Foundation (where it still is a smart move that largely eliminates the problems
survives today as Apache Open Office, albeit with very of the past – too much control from a single company
little development work underway). like Sun or Oracle.
www.linuxvoice.com 23
FEATURE INSIDE LIBREOFFICE
THE NOW AND THE FUTURE
Florian Effenberger is the
executive director at The So LibreOffice has established itself, in part thanks to Major releases such as 4.4 or 5.0 are issued every
Document Foundation, the efforts of The Document Foundation, as a healthy six months, and each of those releases receives a
managing the team that and vibrant followup to OpenOffice.org. But how number of bugfix updates (such as 5.0.1). In addition,
works on administration, exactly does the project work? Who’s in control? Well, one release branch is named “Fresh”, meaning that it’s
documentation and TDF is supported by donations, the majority of which the newest codebase which may have some features
marketing.
come from end users who choose to give a bit of that need more testing; the other branch is called “Still”
money when downloading the software. Many and is recommended for large-scale deployments in
onlookers were sceptical, in the early days, that such a governments and businesses. At the time of writing,
large project (with over 7 million lines of source code) the Fresh branch is 5.1, while Still is 5.0.
could survive off donations – but TDF has shown that
it can work. Long-term support
Indeed, TDF employs a small team to help further Speaking of deployments, while The Document
LibreOffice. These aren’t primarily developers, but Foundation strives to support each major release of
staff working on other supporting aspects of the LibreOffice for several months, it’s limited in its
project, such as infrastructure, developer mentoring, resources and recommends that companies use
documentation and marketing. Most of the Certified Developers (www.documentfoundation.org/
development effort comes from hackers working at gethelp/developers) for long-term support. This is a
companies that use LibreOffice in their products – bit like the Red Hat Enterprise Linux model, where you
get the core product (in the form of CentOS) for free
with some community support, but if you want to roll
The Document Foundation is supported by it out across 20,000 servers or workstations, it’s
probably wise to pay for commercial support as well.
donations, the majority of which come from users As LibreOffice moves towards a new release,
who give a bit when downloading the software various teams inside the project hold meetings
and post minutes on mailing lists. This includes
the design team, engineering steering committee,
such as Red Hat, Canonical and Collabora (see our documentation team, marketing team and others.
interview with Michael Meeks in issue 5). Plus, of The person coordinating this effort, and managing
course, there are contributions from other developers the staff at TDF, is the executive director, Florian
in the community as well. Effenberger. We spoke to Florian way back in issue 1
In order to get releases out of the door – and for our first interview – so if you’ve been reading Linux
avoid Debian-esque “we’ll release when it’s ready” Voice right from the start, dig out that issue and enjoy
delays – TDF adopted a time-based release schedule. a look back at the state of LibreOffice back then!
24 www.linuxvoice.com
INSIDE LIBREOFFICE FEATURE
Because the LibreOffice codebase is so vast, it
can be difficult for new developers to get involved.
In recent years, LibreOffice developers have done an
impressive job tidying up the codebase, making it
easier to navigate and build, and removing lots of
those ancient German source code comments. In
addition, because LibreOffice is designed to run on
many platforms, there are large levels of abstraction
in the suite, which have caused problems (especially
with performance) in the past. Collabora’s Michael
Meeks has spent a lot of time wrestling with the
internals of the software to remove old bottlenecks
and make it more responsive – see http://tinyurl.
com/q72bctv for a (very technical) description.
When a new release is available, TDF doesn’t
simply chuck it onto the downloads page and wait
LibreOffice is heading
for mirrors to propagate. No, there’s an increasing But why is a cloud version so important? The big
to the cloud, thanks to
need to market the software professionally, making issue is administration. Imagine you’re responsible the work of Collabora
sure that end-users and journalists are fully aware for IT in a company with 10,000 desktop PCs. When and developers in the
of the new features and changes. In addition, TDF a new version of LibreOffice is available, you have to community.
holds a yearly conference where developers, users roll it out on every computer, make sure it’s all working
and supporters meet up – this year it will be held from correctly, and that nobody has messed up their
7–9 September in Brno, Czech Republic, so visit installation with some custom settings or packages.
https://conference.libreoffice.org if you’re interested. There are ways to handle this in a more automated
fashion, but it’s tough.
What’s coming up?
The next major release of LibreOffice, version 5.2, is
due in early August 2016. This will include interface While the standalone desktop version of
improvements in Writer (such as a single toolbar
mode for low-resolution displays), new spreadsheet LibreOffice is coming along well, there’s increasing
functions in Calc, along with fixes and enhancements demand for a cloud version
in other areas of the suite. For companies working
with sensitive data, document signing has been
improved as well. And, as with every release, a lot of With a cloud version, the software is run on a server
work has been done to make the Microsoft Office file and rendered in users’ web browsers – so there’s only
format filters even more exact. one version to worry about. Admins can upgrade and
While the standalone desktop version of LibreOffice customise that single version on the server, and all
is coming along well, there’s increasing demand end users will receive the changes via their browsers.
(especially from businesses that want to switch to the So it makes the admin’s life much easier, and also
software) for a cloud version. Progress is being made means that end users can connect to the server from
in this area: Collabora, one of the major LibreOffice different machines and locations, always interacting
contributors, has announced version 1.0 of its with the same LibreOffice instance.
Collabora Online suite (www.collaboraoffice.com), a So there’s plenty to look forward to in LibreOffice,
trimmed-down version of LibreOffice that’s accessible and the project looks to have a healthy future. There’s
in a web browser. While Collabora aims to make even the possibility that Thunderbird, Mozilla’s email
money providing support for its version, the company client, could be integrated with LibreOffice some today
is donating code back to the main LibreOffice source to provide a complete productivity and collaboration
tree, so a community-supported version should follow. solution to compete with MS Office and Outlook
GET INVOLVED!
Because LibreOffice is such a large project, the idea of getting The best place to start is www.libreoffice.org. Click the
involved may seem incredibly daunting at first. And indeed, up Community menu on the top-right and you’ll see different areas
until a few years ago, the barriers to entry were rather high. But of the project: design, development, documentation, marketing,
the LibreOffice team has done a lot of grunt work to simplify the native-lang projects (for translations) and testing. So even if
build process and introduce new developers to the codebase, you’re not a coder, there are plenty of ways to get involved.
so there has never been a better time to contribute. Plus, saying Even if you can only spare half an hour a week to update some
that you’ve been involved in a major Free Software project like documentation, translate a few interface strings or confirm a
LibreOffice is mightily good for your CV! handful of bugs, all participation is greatly appreciated.
www.linuxvoice.com 25
SECRETS AMAROK
SECRETS OF
AMAROK
Get the most out of KDE’s in-house music player.
M
usic touches a part of our souls that rational thought can’t Something that’s so essential to the human condition deserves to
reach. It can makes us smile when we’re sad, help us be treated as a priority on our digital setups. Mastering your music
remember event’s we’d forgotten and compel us to dance collection probably won’t make you any more productive, or make
even when we’re tired. It’s a cultural universal, and every group of your computer run faster, but it may well make you happier, and we
humans on earth has some type of music, though it comes in many think that’s more important. Let’s find out how to get the most out of
different form. Amarok, one of the most popular music players for Linux.
01
02
stations or lyric-streaming services. To
manage your scripts go to Settings >
Configure Amarok > Scripts.
03
Dynamic playlists
If you’ve got a lot of music,
it can be hard to find the
tracks you want to listen to. We don’t
mean locate a particular track, but
01 02
Manage your tracks Scripts decide the right selection of music for
If you’re you’re like us and Amarok has tons of the moment. With Amarok’s Dynamic
have a mass of half-sorted features, but it doesn’t have Playlists, you can specify a set of rules
MP3 files from two decades of ripping everything you could possibly want. If and the software will find a selection of
CDs and downloading from various there’s a particular thing that you want tracks for you to listen to. If you want
sources (all legal of course), you’ll find to do that’s not yet possible, you can 80s rock (and why wouldn’t you?), just
Amarok’s music management features use JavaScript and the Qt bindings to set up the Dynamic Playlist, put on your
useful. It can shift your tracks into a add more functionality. There’s also a mullet wig and ripped jeans, then
properly structured directory system large library of scripts already available head-bang the night away. Amarok will
with just a couple of clicks. to do things like add internet radio keep your playlist topped up with more
music from your collection so you won’t
With Amarok’s Dynamic Playlists, you can run out.
specify a set of rules and the software will find
04
Remote control
While it isn’t a core part of
a selection of tracks for you to listen to Amarok, there’s additional
26 www.linuxvoice.com
AMAROK SECRETS
03 04
05
06
software you can use to control your the ability to get all the latest music
Amarok playback while you’re away legally. If you’re after something a little
from the desktop, including an Android less commercial, you can use Jamendo
app (AmaroKontrol http://bit. to get music by independent artists,
ly/1TaEO8Q). Amarok is probably a bit and if you’re more interested in spoken
too heavyweight to make it useful as an
embedded music player, but this
word than tunes, there’s integration
with Podcast Directory and Librevox.
07
feature enables you to use your PC’s
07
sound system as a general music Bookmarking
player without having to switch back to If you’re listening to a
the mouse or keyboard regularly. podcast, audio book or long
piece of music, you might want to
08
05
File tracking return to a particular point. This could
Your music player builds up be because you want to restart in the
a lot of information about same place, or there’s a particularly
the music files you have – how much poignant or interesting section you
you play them, whether you favourite want to return to. Amarok enables you
them, etc. This is great until you move to bookmark places in your tracks so
your collection onto a different hard you can easily return without having to
drive and suddenly all this information memorise the time position in the track.
is gone. Well, not with Amarok: it can still
08
link your information to your music if it’s Konfiguration
in a different location, so you never lose Amarok is the KDE music
this metadata. player, so as you would
expect, it’s highly customiseable. In fact, music player enables you to connect to
06
Buy music just about every aspect of the an external MySQL database? All this
Want to listen to some new application can be tweaked to your configuration works equally well on
music? Amarok can desires, even to the extent of selecting Qt- and GTK-based desktops, so you
integrate with Amazon, Magnatune and the position on the screen that don’t have to be using KDE to reap the
MP3tunes (through plugins) to give you notifications will appear. What other full benefits.
www.linuxvoice.com 27
FEATURE PIs IN SPACE
Les ‘Spaceman’ Pounder calls occupants of interplanetary craft to find out what’s
been going on with Linux and computing in low Earth orbit.
O
n December 15th 2015, astronauts children. Tim’s aspect of the mission was comes with a plethora of sensors used to
from NASA, Russian Space Agency enhanced with the use of a ubiquitous gather data; using the Sense HAT we can
and the European Space Agency platform: the Raspberry Pi, which made its measure temperature, humidity, pressure,
blasted off from Baikonur Cosmodrome, way to space because UK Space Agency acceleration, orientation and magnetic field
Kazakhstan as part of the Principia mission. wanted to engage with the public via a strength and direction. Also present on the
Principia, or Philosophiæ Naturalis Principia series of experiments and agreed that a board is a miniature joystick for basic input
Mathematica , Latin for “Mathematical harmonised platform – the Raspberry Pi and an 8x8 grid of multi-colour LEDs, which
Principles of Natural Philosophy” was the – would be beneficial to children, offering can be used as a method of output.
mission name chosen by Flight Engineer 4, as it would a unique opportunity to use the Children from around the United Kingdom
former army major and were asked to submit
test pilot Tim Peake. The
goal of the Principia
Using the Sense HAT we can measure their project ideas for
consideration, and from
mission is to work on temperature, humidity, acceleration, the many hundreds of
experiments that cannot applications seven were
be conducted on Earth; pressure, orientation and magnetic field chosen. We’re going to
these include physics, look at three that piqued
biology and technology demonstrations. The same hardware as an astronaut in space. our interest and which can be easily
location for these experiments being the After passing every test to ensure that replicated at home.
International Space Station (ISS) in orbit the equipment was ready for flight final Due to the length of these projects we
around Earth. certification for use aboard the ISS was have cherry-picked interesting parts of
In 2009, Tim Peake was appointed an awarded. The Astro Pi project was ready for the code, but we have included a link to
ambassador for UK science and space- launch. the project page which will provide all of
based careers and worked with the The hardware behind Astro Pi is an the code that you will need to replicate the
various agencies to promote science and add-on board for the Raspberry Pi, projects on planet Earth.
NASA
engineering as career possibilities for school called Sense HAT. The Sense HAT board Get ready for blast off!
28 www.linuxvoice.com
PIs IN SPACE FEATURE
PROJECT 1 – WATCHDOG
Keep astronauts safe and sound with a Pi and a mere 2,000-odd lines of code.
This project, created by Cottenham Village College, is Top: Watchdog is a clever
a backup environmental system monitor designed to project that monitors the
cross-check the ISS’s own environmental control crew’s status and alerts
systems. Watchdog works by using the sensors them to any life-support
present on the Sense HAT, chiefly the temperature, issues. Including recording
pressure and humidity. Any changes outside of data to a black box for
retrieval.
mission parameters will cause the alarm to trigger,
Bottom: Using some
alerting the crew to an incident. advanced calculations to
As this project is huge, totalling 2,681 lines of code, plot the course of the ISS
we shall take a look at sections of the code; for the full and detect which country
code, please visit https://astro-pi.org/competition/ it is over, Flags is a cross
winners/\#watchdog. The first thing that impressed curricular learning project.
us was the diligent use of comments to create a “table
of contents” used to identify what sections of code
control the various aspects of functionality. Here is a
snippet to illustrate their use.
# 1 # CREDITS [165 - 168]
# 2 # IMPORT MODULES [171 - 180]
# 3 # SETTING UP PROGRAM [183 - 206]
# SETS ASTROPI MODULES AS FRIENDLY NAME [185 - 187]
# SETTING UP RASPBERRYPI FOR FLIGHT BUTTONS TO USE
GPIO PINS [189 - 192]
# ASSIGNING FRIENDLY NAMES FOR GPIO PINS [195 - 202] further investigation.
# FORCING PROGRAM TO RUN PROGRAM WITHIN WHILE LOOP file = open(‘log/’+(str(tmstmp))+’ watchdog-log.csv’, ‘w’)
[204 - 206] file.write(“\”Time\”,\”Display\”,\”Temperature\”,\”Temp_
After the comments the code starts by importing Reading\”,\”Temp_Alarm\”,\”Temp_
a series of libraries to enhance the project. We start Snapshot\”,\”Humidity\”,\”Hum_Reading\”,\”Hum_
by importing the RPi.GPIO library, this enables the Alarm\”,\”Hum_Snapshot\”,\”Pressure\”,\”PSI_Reading\”,\”PSI_
ISS crew to control the project using the joystick Alarm\”,\”PSI_Snapshot\”,\”Pitch\”,\”Roll\”,\”Yaw\”\n”)
and buttons present on the Astro Pi units. Next we Going from lines 2094–2153 we see a function
import the time-logging library, which will be used used to detect and react to user action, such as
as a counter. We import the time library and the pressing the joystick right to show the air pressure on
sleep function to control the pace of the project. The the LED matrix. On line 2155–2157 we see a for loop,
asctime function is used to convert the time into a cleverly configuring each of the GPIO pins used for
human-readable format. The other imports cover user input.
using the filesystem, the Sense HAT and the camera. for pin in [UP, DOWN, LEFT, RIGHT, A, B]:## SETUP GPIP PIN
import RPi.GPIO as GPIO VALUES
import time, logging GPIO.setup(pin, GPIO.IN, pull_up_down=GPIO.PUD_UP)
from time import sleep, asctime GPIO.add_event_detect(pin, GPIO.FALLING, callback=button_
import datetime pressed, bouncetime=500)
import sys, os From lines 2185–2294 we see classes created.
import astro_pi The first class, AstroPiContinuous(AstroPi), contains
from astro_pi import AstroPi configuration for all of the sensors including the
import picamera camera. The second class, CPUTemp, is used to
Going down the code to line 212, we can see a time configure, read and convert the temperature taken
stamp used to keep an accurate reference point for from the temperature sensor.
data collection. From lines 233 to 248 we can see From line 2299 to the end we see a try, finally
variables used to store default values for the various construction used to handle reading the sensor data,
sensors present on the Sense HAT. Using these write it to an external file and take pictures using the
values the team cleverly bypass an issue where the Raspberry Pi camera. There are sections of code that
temperature sensor returns a higher than normal can mute the alarms and use the Watchdog project as
value, largely due to being placed over the Pi’s CPU. a blackbox flight recorder. Once the project ends the
The class logged all of the data to an external CSV code cleans up and exits to the command line
file, which can later be imported into a spreadsheet for A great piece of code!
www.linuxvoice.com 29
FEATURE PIs IN SPACE
PROJECT 2 – FLAGS
Where in the world is our intrepid team of space adventurers?
delimiter and using that as a pattern. Finally the new
strings are converted to float values.
Lines 54–1507 comprise a series of conditional
tests used to check the longitude and latitude of the
ISS against the values held for a particular country.
Two variables, X and O, are used to create a list, with
a layout of 8x8, to represent the LED matrix on the
Sense HAT. By positioning the X and O in the correct
place we can create the correct flag for a country.
With the list created the pixels are then updated to
show the flag. The code then pauses for six seconds
before scrolling two messages across the LED matrix,
in the local language for this country. Here is the
Right: The Astro Pi
website, https://astro- Team Space-Byrds from Thirsk School have created a example for the United Kingdom.
pi.org, contains more great project to track the ISS. Typically we only hear if (lati[0] <= 53 and lati[0]>= 52) and (longt[0] >= -4 and
information about the about NORAD at Christmas, thanks to a long-running longt[0]<= -1):
competition winners, Santa Tracker, which originated via a joke print “United Kingdom”
the goal of the Principia communiqué in 1948 which continues to this day. But X = [255, 0, 0] # Red
mission, and regular blog NORAD provides telemetry data for satellites in orbit, O = [255, 255, 255] # White
posts from the ISS. and this project uses this data to track the ISS without UK = [
the need for an internet connection. The location data O, O, O, X, X, O, O, O,
is then used to display the flag for the country over O, O, O, X, X, O, O, O,
which the ISS is orbiting, and then scroll a phrase in O, O, O, X, X, O, O, O,
the local language for that country. X, X, X, X, X, X, X, X,
All of the code for this project can be found at X, X, X, X, X, X, X, X,
https://astro-pi.org/competition/winners/\#flags. O, O, O, X, X, O, O, O,
What first struck us about this project was the use O, O, O, X, X, O, O, O,
of the ephem library. This library provides a means to O, O, O, X, X, O, O, O
perform high-precision astronomical computations. ]
On lines 17 and 18 we see two lines of numbers; ap.set_pixels(UK)
this is satellite data for the ISS, which we shall later time.sleep(6)
convert to a format for use with ephem on lines 33 ap.show_message(“Hello ISS, you are over the UK”)
and 34. Inside of a while True loop, lines 27 to 30, ap.show_message(“Hello ISS. How are you!”, text_
we see three lines of code that capture the current colour=[255, 0, 0])
temperature, pressure and orientation of the Sense At the end of the conditional test we have an else
HAT. condition, line 1509, which is used to quickly scroll
.... the current temperature. It then uses a while loop
temp = str(ap.get_temperature()) that will loop and count down from 5000 to 0. While
Below: The Astro Pi pressure = str(ap.get_pressure()) counting down random integers are chosen for the
units are safely installed orientation = ap.get_orientation_degrees() x,y co-ordinates of pixels on the LED matrix. The mix
aboard the ISS and are .... of coloured light is chosen at random, giving us the
being routinely used by From lines 36 to 48 we see the latitude and illusion of blinking LEDs computing a problem.
Tim Peake who is running
longitude calculated and converted from a string while FLASH > 0:
code written by UK school
children. of characters, splitting the string by identifying a x = randint(0, 7)
y = randint(0, 7)
r = randint(0, 0)
g = randint(0, 100)
b = randint(0, 255)
ap.set_pixel(x, y, r, g, b)
FLASH = FLASH - 1
The code then advises the user that their location is
being computed, ready to display the correct flag. The
pressure is calculated and shown on the LED matrix
before the matrix runs the computer animation.
30 www.linuxvoice.com
PIs IN SPACE FEATURE
PROJECT 3 – SPACECRAFT
Lego, on a computer, in space…
Hannah Belshaw from Cumnor House Girls School Top: As well as Astro
has created a great way of visualising sensor data by Pi, the crew of the ISS
using Minecraft. This project has many different also run other science
aspects; we’ll focus on the data logger and the experiments – here we
Minecraft data playback scripts. All of the code for this see blood being taken fron
project can be found at https://astro-pi.org/ Tim Peake by Commander
Timothy Kopra.
competition/winners/\#spacecraft.
Bottom: The International
The data logger code, astropidatalogger.py, is Space Station is huge –
rather short as it imports a lot of configuration from measuring 109 metres by
an user-created external library. The data logger 73 metres it is the largest
process is called on lines 29 and 32. With line 32 the man-made object in Earth
logger starts to gather data, saving the output to an orbit.
external CSV file. The library contains all of the raw
code that interacts with the sensors and formats the
data inclusion in a CSV file. In the external library,
astropidata.py, on line 52 we see the creation of a
class designed to log the sensor data. There are a
number of functions starting from line 93, which
handle printing verbose information to the terminal.
On line 188 we see the creation of a dictionary, used to
hold specific information that can be referenced via a
key, typically the name of the data that we wish to apr.get_cpu_temperature(),
save. The names of the fields for the data spans all apr.get_temperature(),
the way to line 231. apr.get_humidity(),
datarow = {} apr.get_pressure(),
datarow[DATETIME] = datetime.now() apr.get_orientation(),
datarow[TIME] = time() apr.get_joystick())
datarow[CPU_TEMP] = cpu_temp.get_temperature() From line 113 we see the creation of another
datarow[HUMIDITY] = ap.get_humidity() class. This time it handles a command line interface
From lines 248–362 we see another class that enabling the user to interact with the data via a shell
reads the data from the sensors and then stores interface. Commands entered into the shell can be
it in the appropriate section of the dictionary. Here passed arguments, extra information or configuration,
is an excerpt from line 344–347 showing the in this case it is the name of the file where the data is
accelerometer data being stored. stored.
def get_gyroscope_raw(self): def __init__(self):
return {“x”: float(self.data[GYRO_RAW_X]), Cmd.__init__(self)
“y”: float(self.data[GYRO_RAW_Y]), self.prompt = “SpaceCRAFT $ “
“z”: float(self.data[GYRO_RAW_Z])} self.intro = “Welcome to SpaceCRAFT data playback. Type
help or ? to list commands.\n”
Minecraft data playback self.playback = None
Playback of the data captured is handled via From lines 181–194 we see a function that handles
mcastroplayback.py and this code visualises the data the playback speed of the visualisation, either real
using Minecraft and its many different block types. On time, x1 or the value can be increased doubling each
line 39 we see the raw CSV data being imported into time all the way to x16 speed. The code for playback
the code. speed also incorporates error handling, in case the
apr = AstroPiDataReader(self.filename) user provides an incorrect speed.
Further down the code, from line 45 onwards, see At the end of the code we see two lines of code
the beginnings of the Minecraft code; on line 50 we get that handle running the project and calling the various
the position of the player so that any visualisations are classes that make up the code.
created near to their location. From lines 65–73 we if __name__ == “__main__”:
see the code that will update the ISS data visualisation PlaybackCommands().cmdloop()
with the latest information. This robust project encompasses many different
isstowerdisplay.update( aspects of Python, data structures and provides an
apr.get_time(), entertaining method of understanding data.
www.linuxvoice.com 31
FAQ GCC
MIKE SAUNDERS
GCC
The GNU Compiler Collection may be the most important piece of free software
ever developed – even more than the Linux kernel. Here’s why.
Linus Torvalds managed to erase all
traces of its source code from the world
FOSS compilers existed – but they
were very primitive in comparison. GCC
Whoa, it says up there that and retired to a desert island, we had was simply years ahead, and was so
GCC is more important than other options. Most of the FOSS dominant in the Free Software world
the Linux kernel! Can I have some of computing world could have switched that many programs used GCC-specific
what you’re smoking? to FreeBSD, for instance, which does a extensions in their source code.
No, we’re serious. Yes, the Linux similar job to GNU/Linux as a
kernel is an incredible workstation and server Unix flavour. OK, so GCC was a cornerstone
accomplishment, a vast body of code But all of these other operating of FOSS development back
and effort from thousands of talented systems had something in common: then. But what about today?
developers. It’s arguably the best they were all built with GCC, the GNU Well, it’s still a hugely significant
operating system kernel in the world. Compiler Collection (formerly the GNU piece of software, and the default
But in order to understand the C Compiler, but the name was changed compiler on pretty much every GNU/
importance of GCC, we have to step as it evolved to support more Linux distribution, but some
back in time a bit. Don’t worry though programming languages). Had GCC competition is emerging in the form of
– this isn’t going to be a drawn-out disappeared or been made illegal back LLVM/Clang. This is a compiler
history lesson. then, the Free Software community toolchain for C, C++ and Objective C(++)
Picture the scene in the late 1990s would have been up a very unpleasant that’s released under a more permissive
and early 2000s. There were several creek without even the slightest hint of licence than the GNU GPL used by GCC
free software operating systems a paddle. – so in other words, LLVM/Clang is
making good progress: GNU/Linux, We had various kernels, text editors, more appealing to proprietary software
FreeBSD, OpenBSD and NetBSD. Had libraries, window managers and other companies. Apple has been one of the
the Linux kernel – for some reason tools that could be swapped out, but leading developers of LLVM/Clang, and
– been made completely illegal, or GCC was one of a kind. Sure, other other closed source companies have
gotten involved as well.
GCC is still a hugely significant piece of Right now, GCC and LLVM/Clang are
very close in terms of performance
software, and the default compiler on pretty (both in compile times and generated
code), with each project claiming a lead
much every GNU/Linux distribution in certain areas. But LLVM/Clang has
32 www.linuxvoice.com
GCC FAQ
shown enough maturity to be the ideal world, both projects will provide
default compiler in FreeBSD 10 healthy competition and result in even
onwards – although GCC is still better compilers.
available as an option. By the end of
2014, LLVM/Clang was able to compile OK, so let’s get a wee bit
almost 95% of the Debian software technical. How exactly does
archive without problems, which is GCC work?
mightily impressive considering that’s Compilers are incredibly
over 40,000 packages. complicated pieces of software,
and warrant entire books about their
So is that the end for GCC? Is design – but we can summarise the
LLVM/Clang the future? key components here. GCC doesn’t
That remains to be seen. One simply read a piece of code like
thing that makes LLVM/Clang so puts(“Hello world”); and determine the
appealing to developers is its right CPU instructions to do the job;
modularity. The toolchain can be neatly instead, it’s made up of multiple
split up so that the separate parts can components that go through the code
be used in an IDE (integrated in a stage-by-stage basis.
development environment), providing To start off, the “front end” part of the
useful debugging and error messages toolchain looks at the human-written
along the compilation and linking chain. source code, parses it, checks for errors
In comparison, GCC makes it harder for and generates a “syntax tree” (a tree-like
IDEs to access the code during the representation of the code). This syntax
intermediate compilation steps, tree is not specific to any particular
providing hurdles for seamless language – so you have front-ends for Protip: despite what the GCC logo may have you believe,
integration. C, C++, Ada and other languages that all gnus are not born in eggs, nor do they hatch out with
GNU founder Richard Stallman has generate a syntax tree in the same fully-grown antlers.
said this is a feature and not a bug, and format for the next step of the
essential to keep GCC away from being compilation process. has mountains of information; in
wrapped up in proprietary IDEs. But at This middle stage performs particular, the page for new contributors
the end of the day, it’s down to the old optimisation on the syntax tree, at https://gcc.gnu.org/contribute.html
battle of idealism vs pragmatism. If creating an intermediate language that is a good starting point. GCC is a large
LLVM/Clang ends up producing faster looks a bit like assembly language but and mature project with some of the
and smaller code while working much is still CPU-independent. Finally, the most experienced hackers on the
better with IDEs, many hackers may be “back end” takes the intermediate planet, so it may take a while to get
willing to forgive its more permissive language and converts it into assembly yourself familiar with the codebase and
licence and just get their work done. language for a specific architecture. infrastructure, but any contributions are
That code is then assembled and welcome. And as with almost every
But surely GCC is making linked, resulting in a binary executable Free and open source software project,
progress, right? I heard that that you can run. source code is just one slice of the
there was a new 6.1 version Because compilation is split up into development pie – the team also
released just recently… these separate stages, developers can welcome bug reports, documentation
Yes, GCC is still going strong and work on the parts they like without updates and other non-hacking
has a lot of hugely talented having to know everything about every contributions as well.
people working on it. Version 6.1 was a language and CPU. Someone with If you want to get involved but don’t
major upgrade, using the C++14 extensive knowledge of C++, for know where to start, or you’re short of
standard by default (from 2014, as example, can hack on the C++ front-end ideas, see https://gcc.gnu.org/projects
opposed to previous versions of GCC without having to know anything about – and especially, the “projects for
which defaulted to the 1998 standard). x86 or ARM processors. Similarly, a low- beginner GCC hackers” link. This
Support for very old architectures is level coder who’s ace at micro- contains a list of newbie-friendly
being dropped, new optimisations are optimisations on x86 chips can work on hacking tasks, such as splitting up giant
being rolled into the code base, and the the back end without having to deal source code files and removing
runtime library is being worked on too. with the complexities of C++. duplicated code. These jobs may sound
It’s also important to note that GCC trivial in the grand scheme of things,
supports a wide range of CPUs and Ooh, that actually sounds like but they help to make GCC tidier,
languages, and is being updated for fun. I did a bit of x86 assembly cleaner and more accessible for other
relatively new languages such as Go. language once, thanks to a certain contributors. Plus, being able to say
So we don’t have any deep concerns tutorial series in Linux Voice… that you contributed to a compiler is yet
about GCC’s future, even with LLVM/ Then get involved! The GCC another thing to shout about on your
Clang wooing lots of developers. In an website at https://gcc.gnu.org geek business card…
www.linuxvoice.com 33
INTERVIEW JIM KILLOCK
C K
LO ndo
n to
IL
r y Lo
o sca ered.
t
re both
K
t u
d ven ld be
ve rar shou
E e
M
B en hy w
I
a nd nd w
y
or ce, a
J
e g
w Gr eillan
e v
A ndr , sur
m ice edom
n t ry t fre
u u
Co k abo
tal
of
rd
h ea dies
n’t f bo
have ful o lic n
u d pub r
yo an wh
p: if f a h ising p up ry of
u o , ra
Gro ne cro ine e
hts is o MPs that ach or th e
ig G g s m e th
n R OR yin ice the nc d, d
O pe t up. lobb njust inst veilla g fiel goo .
he ook i d to ing i aga y, sur layin d of nse
T
e t p a e
it, l icat figh bs u rivac vel p a lo on s lock…
d nd
d s a rld ata e a aki om m Kil
e r u p le ng m
es wo s d us , m f c , Ji
a ren dern er it’ s to here ful o ector
aw mo heth firm p is t poon e dir
e
the te. W onlin Grou big s cutiv
sta t of ghts ith a s exe
h ’
rig en Ri nts w ORG
p
O um e t e o
k
arg spo
W e
34 www.linuxvoice.com
JIM KILLOCK INTERVIEW
“It doesn’t make me very comfortable that my personal
internet activities are being assessed for how suspicious
they look. That places me under suspicion; it places you
under suspicion, it places your readers under suspicion.”
www.linuxvoice.com 35
INTERVIEW JIM KILLOCK
Let’s get down to business – That’s the Norwegian model, of transparency in the EU process, and
what is the Open Rights Group isn’t it? it’s probably less when it’s civil servants
working on right now? JK: It is. And the thing with that of drafting SIs.
Jim Killock: We’re working on mobile course if that that might not be very
data and how mobile phone companies politically palatable. So you’ve got a So who, in the UK government,
are collecting and using people’s data. hard choice between the economically gets it, do you think? Tom
Both data protection and privacy are easy route and the politically tenable Watson and David Davies are the
going to big debates in Europe over the route. I think it’s going to be quite hard obvious two that spring out. It
six months, particularly privacy. to leave Europe and then say “well seems to me that, there’s an
actually we’re going to keep everything increasingly authoritarian streak in
That’s if we’re still in Europe the same and keep adopting European
– after the referendum we
might have left…
laws’, because that’s going to make no
sense. It’s very hard to know which of
GCHQ are scoring everyone
JK: I mean… yeah… ORG is not pro- or these tendencies wins out. on the basis of how much
anti-Europe as such. But assuming we I wonder how the UK parliament
did leave, there are really big questions would cope with doing this sort of risk they think we are
for digital rights. How do you do all of legislation They’re very detailed,
the telecoms regulation? What do you technically intricate and not very mainstream politics, and there are
do with data protection? What do you interesting to voters most of the time, only a few people who understand
do with e-privacy? Do we stay in the and I don’t think that our MPs are going that spying on people is A Bad
digital single market? And the curious to have a lot of time or interest for these Thing.
thing there is that if you stay in the reasons. JK People in government in the cabinet
single market then you have to keep all What would happen in practice is are always are under a lot of pressure
these laws, which means the situation that our civil servants would do all to play to the gallery, and therefore even
simply doesn’t change. So leaving of this and it would all go through as the people who might understand the
Europe might not make a great deal of statutory instruments. So you’d still problems don’t often vocalise them.
difference; it might well be that a lot of have a democratic deficit, possibly a In parliament at the moment we do
these laws stay exactly the same worse one, because there is some kind have a shortage of attention to these
Jim tells us that it’s hard to get the public
interested in TTIP (how do you campaign on
something that’s happening in secret?), but it’s
most probably not going to happen anyway.
36 www.linuxvoice.com
JIM KILLOCK INTERVIEW
“The first thing you’ve got to
remember is that politicians are not
overly familiar with all of the detail
of surveillance law or what might
constitute mass intrusion”.
issues. We do have some interesting oppose it more effectively at a later to see how something that’s entirely
MPs who were elected on the Labour stage. Do you think that’s a winning indiscriminate could be viewed as
backbenches who are sort of making a strategy? proportionate. But we haven’t seen
bit of progress, but none of them have JK I don’t think it is particularly effective it tested in the courts in quite these
had time to grapple with this agenda. right now, but I understand why they did terms, and of course the opposing
I think we miss Julian Huppert it. Everybody has known for a while that argument is kind of “How are we
[former Liberal Democrat MP for these laws need updating; that’s partly meant to to do this? How are we meant
Cambridge]; he was very good, and because of Snowden – you can see to achieve the results we need if we
it’s a pity that we don’t have him in that the law is out of kilter with what’s aren’t sweeping it all up?”
parliament. Within the Conservative happened, what the practice is; and it’s So Labour are trying to push for a
ranks there are a number of people who partly because in the 10 or 15 years review of the powers to see whether
are very good. Obviously David Davis, since they last legislated, the world has they’re justified or not. They’re trying
but at least one or two of them have changed. I think we can understand to in a sense get somebody else to tell
ended up in the cabinet, where they that legislation is needed, and if you them whether this is legitimate or not.
effectively get shut up, so you don’t start from that assumption, it’s going to
hear from them. feel like a much better option to try to So what could the effects of
I think the other thing with the improve what has been proposed the IP bill be for the average
Conservatives right now is that rather than try to oppose it wholesale. law abiding citizen who doesn’t have
obviously they’re preoccupied with The other thing is that Labour anything to hide?
Europe – that’s the problem with the themselves are quite torn about how to JK: The thing you have to remember as
Investigatory Powers bill right now. A approach this. Part of the Labour party an individual is that… GCHQ are literally
lot of the people you’d expect to have at is very concerned about civil liberties scoring everybody on the basis of how
least some kind of interest in this have and wants to see it do more about much risk they think we are. You can
not been paying attention, because these issue, and part of it still very choose to ignore it, but the fact is that
they’re putting all of their energies into afraid of being painted as being weak we are all being evaluated. They’re
the referendum. And yet these next few on crime, weak on terror and so on, and piecing together information per
weeks are also the critical time for the they want to portray themselves as individual, trying to correlate it with
Investigatory Powers bill, making the supporting the secret services as much other information about individuals,
public side of campaigning very very as possible. So they have a problem. then deciding whether we appear to be
difficult. Attention is elsewhere. Then there are some really close to certain sorts of patterns.
fundamental issues that we don’t Those of us who do match or seem
Labour’s rationale for understand. One of the biggest is bulk close to certain patterns then get
abstaining on the first vote powers. Are bulk powers justifiable? looked at more closely. But the fact is
was that they would be able to From our point of view it’s very hard that machines are evaluating all of us.
www.linuxvoice.com 37
INTERVIEW JIM KILLOCK
Before joining ORG, Jim was
the external communications
chap at the Green Party.
GCHQ’s argument is that this process going to make people that little bit more people from crime, terrorism, and that
of evaluation, correlation, matching of conformist and that little but less risk- individuals have a right to life. That’s
data doesn’t matter, because it’s only taking. also a powerful argument. That’s when
machines that are doing this, and What happens with the journalist you end up saying that you have to
none of this is being looked at by an who finds it harder to persuade a have some surveillance. And that’s why
individual. But it doesn’t make me very whistleblower to hand evidence over? we end up in this horrible area around
comfortable that my personal internet Or what happens with an individual who the techniques that GCHQ use that
activities are being assessed for how needs legal protection and that legal really work. Because you have to ask
suspicious they look. That places me protection is known by the authorities; whether they are (as GCHQ claim) the
under suspicion; it places you under do they not seek legal advice? Those only way to reach the conclusions that
suspicion, it places your readers under sorts of things are bad for us all. If the they’re reaching, the only way to
suspicion. rule of law, or the ability of journalists to develop these and the only way to get
Does using Tor place you more under do their work, is limited, then the result to these people.
suspicion? It kind of does, it’s very likely is that we have a less democratic, less
to. Is using a VPN likely to give you a free society. We borrowed a similar point
couple more suspicion points? Yes. If somebody ends up in jail a little bit from Bruce Schneier a couple
Does moving around in certain parts longer, because they’ve been afraid to
of town put you at greater suspicion?
Yes. Each of these things, your flight
talk to a lawyer, that undermines the
rule of law for all of us. It changes our
You don’t want everyone
records, passport history, whatever it society. I think those are the reasons to feel like they’re under
happens to be, all these things build a we try to restrain surveillance; we don’t
profile. And that means that sometimes try to just say “looks, we’ll deal with surveillance. That’s wrong
some people are going to come and it when it goes wrong”, which is kind
start asking questions. Should I read of the approach that the government of issues ago. Even with a filter
this website that provides information has when they say they’ll fix it with that’s 99.99% accurate, if you run
about extremists? Maybe not – maybe oversight and checks and balances. that on 100m people that’s a huge
I don’t want the police or GCHQ to be You limit surveillance because you don’t number of false positives.
knowing that about me. Even as an want everyone to feel like they’re under JK: You’ve got a false positives
individual there are serious concerns surveillance. That’s wrong. problem, but you’ve also got a huge
that people’s behaviour online changes. need to sift everything. If your
Effectively your free expression, your Fair enough. algorithms and sorting processes
right to impart and receive information, JK: It’s not hard it is, talking to requires each and everybody’s
it is limited because of your fear of people who already agree? The problem metadata to be sorted and then to find
surveillance. We don’t know the full is that the counterview is that the ways to assess all of that metadata,
effect of that yet, but it is inevitably government has a duty to defend and to make it searchable, so you’re
38 www.linuxvoice.com
JIM KILLOCK INTERVIEW
creating a huge amount of machinery vague thing that’s going on. It’s the these principles have been debated in
just to deal with irrelevant information. point at which Theresa May [the Home the past, and the limit online has to be
Maybe it’s more easily solvable that Secretary] stands up in parliament and the same as offline. We have the same
I’m giving credit for, but at the very says “You’ve got to vote for this” and privacy rights, we have the same free
least you’ve got to question that use of someone from the other side stands expression rights.
resources. up and says “what you’re proposing is The dominant metaphor in the mind
totalitarian”. That is the point that public of a lot of politicians is the internet as
Did you do the Putin posters start to understand that this is really publisher, so the internet should behave
with the slogans over the top happening and they need to take a side. itself like a newspaper would, or the BBC
of them? should. So they expect the individuals
JK: We did that with the other Don’t I want to ask you about Myles online to behave as though they were a
Spy On Us groups – we raised about Jackman [who has recently BBC journalist; not saying outrageous
£15,000 to help publish those and we joined the ORG as legal director]. things, not speaking as they would in
also got some grants in to do that. And How’s he getting on? public… But at at the same time, most
we worked with an ad agency who JK: Miles is great. He’s very interesting people are talking pretty much as they
wishes to remain anonymous. The little because he’s been a campaign lawyer. would to their friends, so the metaphor
bit I did, apart from a bit of logistics, He thinks like a campaigner, he’s at work in most people’s minds when
was to help them find a CC image they someone who wants to see the law they say and do things online is that
could use. change and is capable of pinpointing of common everyday speech, not
the problems when they go wrong for editorialised content, and that’s quite
I was wondering where that individuals. I think he’ll be a very hard for politicians to grasp. You see
photo came from… effective member of our team. that also in the police when they react to
JK: It was kindly published by the More widely for ORG I think the content that is offensive.
Kremlin. So if you look on the posters legal dimension to our work is going
they all say “Image CC Kremlin.ru”. to become more important over time, The chap who was arrested for
The big challenge is trying to get the because it’s one thing to argue these having a Nazi pug, for example.
public interested.I don’t think it’s that things out with bureaucrats who’ve got JK: The Hitler pug video is puerile, and
the public is not interested; it’s just that an excuse for everything, but if we are I’m not going to claim that it’s a great
there are so many competing problems prepared to take these things to court example of tasteful humour or anything
an the EU referendum right now is that will remind politicians that if they like that, but the question really has to
overshadowing literally everything. We step over the limit in these areas then be if this man is inciting racial hatred,
did the Putin posters to remind people they aren’t going to get their way. why isn’t he being prosecuted under
that this issue is still there, but what you In this area there’s been a sort of racial hatred laws? The reason is, I
actually need is the politicians arguing assumption that because digital is new think, because the police reckon they
it out and explaining to the public that politicians have a kind of carte wouldn’t get away with it. He’s very
why they’re taking certain positions. blanche to do whatever they like and clear about his intentions; he says in
Until that happens, the issue isn’t a can define the boundaries as they like. the video that he’s trying to annoy his
real decision for people. It’s just some And that ignores the fact that a lot of girlfriend and that he’s not actually
advocating Hitlerian politics, and
therefore a court would find it difficult
“It is actually quite difficult at the
to convict as an incitement to racial
moment for Europe to compete in
hatred. So they threatened him with
a market with the USA because of
different privacy laws and standards.” offences under the Communications
Act, because ‘grossly offensive’ is
essentially an ‘I don’t like it very much’
test that drags in anything where you
can get a lot of people to shout ‘I don’t
like this.’ That’s not a sufficient test for
prosecuting someone.
That doesn’t mean you have to like
the video; but just because I think what
he did wasn’t an appropriate thing to do,
doesn’t mean that I have the right to tell
him he has to go to jail. But of course
for politicians a lot of the time it’s easier
to tap into the offence that is generated
rather than try to work out whether the
lines are properly drawn. I guess that’s
why we have courts and it’s why we
have ORG.
www.linuxvoice.com 39
MUGS AND T-SHIRTS!
You can drink
coffee, tea or
champagne.
It’s all about
choice.
This mug
respects your
freedom. Look,
it even says
GNU/Linux!
shop.linuxvoice.com
INTRO REVIEWS
REVIEWS
The latest software and hardware, rigorously bashed against a wall by our crack team.
On test this issue . . .
Tails 42 Krita 43
Tor is essential for maintaining your privacy – and New features and even more polish make KDE’s
Andrew Gregory Tails helps you set it up. Simple. drawing app a must-try for artists.
Is grateful to the brewers of Hebden Bridge for
this month’s creativity juice.
T
here’s a bit of theme running
through this month’s reviews:
neo-Luddism. We’ve shunned
the latest AAA first-person shooters to
review Lumo. It’s simple, it’s wonderful
and we like it a lot.
In the world of distributions there are Ghost 44 Lumo 45
all sorts of advanced features, but Bring the stark elegance of Adrian Frutiger to your The isometric gameplay of the 1980s coupled with
we’ve chosen to review one that has online musings with this blogging engine. the graphics performace of today. Noiiiice!
simplicity as its raison d’etre – Tail,
which makes it simple to use the Tor
anonymising software. And Ben’s been
playing with Ghost, a blogging platform
that trades on its simplicity.
Group test and books
Simplicity is a feature
The reason we like simplicity isn’t that
we’re lazy or thick: we like it because
we’re got better things to do that learn a
whole new CMS when all we want to do
is put up some allotment photographs.
We don’t want to have to learn a new
control system when the WASD keys
and Space bar can give us immediate
feedback. And frankly, if someone’s
gone to the bother of configuring the
Tor Browser for us, why on earth would Booooooooooooooks!!!! 48 Group test – beginner distros 50
you bother to do it for yourself? Presumably privacy advice for girls is the same as Emancipate yourself from mental slavery, put
Immediacy is attractive and rewarding. it is for boys. Likewise the fundamentals of Bitcoin Windows down and install one of these fine
andrew@linuxvoice.com don’t change according to your gender. beginner-friendly distributions.
www.linuxvoice.com 41
REVIEWS LINUX DISTRIBUTION
Tails 2.4
Browse the web completely anonymously. Well, sort-of.
I
Web https://tails.boum.org s there such a thing as total anonymity on the Tails in VirtualBox, it complains that your anonymity
Platforms x86 internet? Well, if you’re willing to travel by foot to a could be reduced (ie the virtual machine could still
Price Free
foreign country with a stolen laptop, hack be monitoring your keypresses). So for maximum
someone’s Wi-Fi and do you work that way – quite anonymity it’s best to run it on native hardware.
possibly. But for most people, Tor provides a certain Along with the Tor Browser – a modification of
level of anonymity by routing your internet traffic Firefox – Tails includes other software in its 1GB DVD
through a complex network of servers, making it download ISO, such as the Pidgin instant messenger,
difficult for the end resource that you access to Icedove (a rebranded Thunderbird), Gobby (a document
determine where the original request came from. collaboration tool) and other useful bits and pieces.
Tails (The Amnesic Incognito Live System) is a Yes, this all bulks up the download size of the distro,
Debian-based live distro that includes everything you but we appreciate having a versatile toolset with all
need to get started with Torout of the box. Just boot it network traffic going through Tor.
up and start browsing – as simple as that. If you boot Tails 2.4 features updates for all its major software
components, and the Gnome Tweak Tool has been
removed. This is a controversial change for some,
given how non-configurable Gnome 3 is; on the other
hand, we understand that it makes support much
easier when end users can’t pull apart their desktop
beyond recognition. Another change is the removal of
#tails on IRC as a method of communication, as the
Tails team recommends using XMPP chat instead.
In all, it’s a solid upgrade for the distro and we give
credit to the Tails team for maintaining its focus and
consistently delivering a good experience. It doesn’t
guarantee 100% anonymity, but it’s a big step in the
right direction.
While the Tor Browser is
In a world where government (and corporate)
the main feature in Tails, spying is everywhere, Tails is a valuable tool.
plenty of other software is
included as well.
42 www.linuxvoice.com
KRITA REVIEWS
Krita 3.0
Graham Morrison finds his own version of the adult colouring book phenomenon.
B
efore we look at what’s inside this major Web https://krita.org/
Developer The Krita team
update to our favourite drawing tool, it’s worth Licence GPLv2+
looking at what’s been going on outside the
code. In particular, we think the Krita team is doing an
amazing job at managing the project, both by getting
great support for its crowdfunding efforts, and via the
publicity it generates through its website and media
channels. We highly recommend our readers take a
The new layer UI and
look at some of the time-lapse design creation videos, multi-selection makes
example files, brush presets and tutorials, as they multi-layer blender a very
provide a brilliant overview of what Krita is capable of. creative and rewarding
process.
Let’s get squiggling!
Animation alone is worthy of the 3.0 release moniker, Intel CPU with 8GB of RAM, Intel graphics and a very
literarily adding an entire new dimension to Krita. With high resolution display. Everything behaved brilliantly.
the new animation and timeline ‘docker’ panes, you This release is the result of a year’s work, and of the
can easily start making your drawings move, adding great many other additions, our favourites include the
one frame after another and using onion skin overlays grids and guides overhaul – guides are now saved
to preview previous and next frames. It reminds us of with your work and the unified pane makes them easy
the ancient Deluxe Paint on the Amiga, and while Krita’s to access – the ability to move and manipulate more
implementation is obviously capable of serious than one layer at once and the new popup palette. The
results, it’s just as fun. It’s probably a good thing that new update, thanks to its speed and design, feels like
the OpenGL 3.0 (and Qt 5!) performance a genuine professional application capable of
enhancements have made it into this release too, as stunning results in the right hands.
creating animations requires a lot of fast switching
between both rasterised and vector layers, but we An absolutely brilliant drawing app, whether
you’re a beginner or a professional.
experienced great performance and no stability
issued with the latest release on a modest 2.2GHz
www.linuxvoice.com 43
REVIEWS BLOGGING PLATFORM
Ghost 0.8
There are ghouls on the web, but Ben Everard ain’t afraid of no Ghost.
G
Web ghost.org host has all the features you’d expect from a software up is a little more involved than many web
Developer Ghost Foundation blogging platform – you can create posts, apps because it uses Node.js and has to be routed
Licence MIT
manage them, and publish them on the web. through a proxy (such as Nginx). There are detailed
Many people can work together to create a single blog instructions on the project’s web page (https://
with each publishing under their own name. The thing support.ghost.org/developers). Because of the
that sets Ghost apart from the other options is the technologies used, you’ll hit some slight complications
user experience. Blogs are written in Markdown, and if you plan on running Ghost on the same box you use
there’s a live preview of the output on the right-hand for Apache/PHP web apps.
side of the screen. The interface is uncluttered, and The hosted options start from $19 per month for
everything is stripped away to force you to focus on up to 25,000 page views. The cost means that Ghost
what’s important – the content. The Ghost software is more suited to professional writing than someone
processes the Markdown and combines it with your looking to write the occasional piece about a hobby.
selected theme to produce a great-looking blog. When productivity is important, ease of use is a
There are two options for running Ghost – you can feature. It’s difficult to make an ugly post on Ghost
host it yourself (the software is open source) or you (when compared to HTML-based blogs that give you
can use the hosted option at ghost.io. Setting the enough flexibility to create horrid-looking pages).
When blogging is something that you tend to put
off, these two factors may make you more likely to
actually post something.
Some blogging platforms are flexible to the point
that they’re really content management systems
that can be bent into any website. Ghost is not one of
these – it’s a blogging platform for blogging, and by
focusing on one thing, it performs the task excellently.
The cost might be a little high for hobbyist bloggers,
but for professionals, Ghost has a lot to offer.
The marketplace contains
a bewildering array of
A blog platform for bloggers, but the hosted
themes (both free and options are pricey.
paid-for) to suit almost
any taste.
44 www.linuxvoice.com
LUMO REVIEWS
Lumo
Graham Morrison falls head over heels in love (again) with 2.5 dimensions.
L
umo is an isometric puzzle game. If that means Web http://triple-eh.tumblr.com
Developer Gareth Noyce/Triple
anything to you, you’re going to love Lumo. It Eh Games
brilliantly recreates the challenges and Price £14.99 (Steam)
addictive gameplay of its forebears, rendered in a
modern graphics engine running from Steam on your
Controls can be configured
Linux box. If ‘isometric puzzler’ means nothing to you, to find the most natural fit
you should note that Lumo isn’t really isometric, nor for the view on-screen,
much of a puzzler. Isometric is a reference to the way though they never make it
games designers built the graphics for their games easy.
back in the 1980s, placing pixels at the north west,
north east, south east and south west points adjacent comes from the limitations of the display, and the
to another pixel. This technique limited what would language of the isometric view is a fundamental part
now be called jaggies or aliasing, the stepped of the challenge. Without any instructions or
appearance of a line when it’s drawn across a low prompting, the player is able to make sense of their
resolution, and the arrival of these games on home surroundings and work their way from one room to
computers caused a revolution back in 1984. 256 × the next. There are hundreds of rooms grouped into
192 pixels never looked the same after Ultimate Play levels, and getting through the game is going to be a
The Game released Knight Lore for the ZX Spectrum. serious quest. Success depends upon precision
timing and enough mental dexterity to transpose the
Isometric alienation on-screen angles into keyboard or game controller
Lumo comes with a huge chunk of nostalgia, and it movements, sometimes reversed, always under
does a skillful job at weaving in references from the pressure. It’s this unforgiving environment that makes
80s into its gameplay, from Oliver Frey to Shahid the game so compelling and utterly addictive. We
Ahmad, and retains an authentically British Bedrooms loved it. Just like its 80s counterparts.
to Billions aesthetic (a great film if you’ve not seen it).
But more importantly, Lumo revisits a gameplay
Two thumbs up. Inane grin.
mechanic that’s been lost in the era of 25GB game
patches and tenth generation GPUs. This mechanic
www.linuxvoice.com 45
REVIEWS GAMING
GAMING ON LINUX
The tastiest brain candy to relax those tired neurons
STAR TREKKIN’ Stellaris
Intergalactic grand strategy.
Web http://store.steampowered.com/app/281990
Price £34.99
T
his game got a lot of people excited
when it was first announced – it’s
reminiscent of Master of Orion and is
brought to us be the people behind Crusader
Michel Loubet-Jambert is our Games
Editor. He hasn’t had a decent night’s Kings and Europa Universalis (two of the finest
sleep since Steam came out on Linux. strategy games on Linux), so it should be a
seemed like a match made in heaven. The game’s high level of detail enables us to see
F
ollowing last month’s coverage Stellaris certainly does a lot right. The familiar space battles and planets from up close.
of how the Vive VR is yet to interface, ability to create and customise galactic
have Linux support, the first
empires, a sensible non-Hollywood approach to small quests and the excitement of discovering
steps to this being amended have
been taken, albeit by crafty science, a good mix of well executed mechanics neighbouring empires, followed by a strong
community workarounds rather than and story events that provide intrigue and mid-game with the imminent threat of war and
by Valve, which plans to add support mystery to the galaxy all serve to provide an strong factions, the game becomes a bit of a
at a later date. Those who bought a extremely polished game. On top of this, Stellaris slog in the later stages and the new technologies
Vive to use with a Windows dual-boot
does well to remain grounded in sticking and mechanics steadily unlocked throughout the
should check out this guide (http://
tinyurl.com/z4qtgv7) on how to get it conventionally understood concepts like game do little to remedy this.
up and running. For those looking to terraforming and FTL travel, instead of getting
buy a Vive to use exclusively on into the outlandish realms of force crystals and Bigger and better
Linux, it’s best to wait for official magical powers, which can break immersion if Nevertheless, strategy games of the last 10 or so
Linux support.
not done right and make decision-making years have rarely been perfect from the start,
Something briefly touched upon in
Distrohopper last month was arbitrary in a strategy game. often improving greatly with expansions.
progress with running Linux on a Where the game falls short is with a problem Fortunately, a tonne of free content has been
PS4. These efforts have now begun that is all-too-common in strategy games, and announced for the coming months. Stellaris is
to seriously pay off, and an install of that’s in the late game. After an incredible still certainly worth getting now, with the
Arch Linux on a PS4 has been able to
exploration phase in the early game filled with knowledge that it will get even better over time.
run Steam in Big Picture mode. To
make this feat even more impressive,
the modder responsible also
managed to install Radeon drivers to
make full use of the console’s AMD
graphics. As a demonstration, a video
has been making the rounds of the hit
indie game Bastion running natively
on the hardware. Needless to say,
now with far more native Linux
games around, the possibilities for
console Linux distros are far more
exciting.
On the development side of things,
the Unity Engine is to add Vulkan
support very soon, adding a major
advance to one of the most
widely-used game engines out there.
The developers are also working on
some SDL improvements, which In Stellaris, the Fermi paradox
should allow for Wayland and Mir has been answered as the Milky
support. Way is densely inhabited.
46 www.linuxvoice.com
GAMING REVIEWS
Shadwen ALSO RELEASED…
A stylish medieval stealth game.
Web http://store.steampowered.com/app/425210 simply evading them using craftable
Price £12.99
tranquillisers and traps adds a lot of player
choice and replay value.
S
hadwen is a stealth game that The story holds it all together nicely and
breaks a few of the genre’s the inclusion of Lily – a young girl rescued
conventions. Other than the female by Shadwen – adds more dimensions to a
protagonist and companion, the game single-objective plot. Unfortunately,
also makes use of time pausing and Shadwen feels wonky in areas with some
Overfall
rewinding. These elements mix things up weird physics and collision detection at This RPG with turn-based combat has it all: a
a bit and make the game feel far more times. Similarly, at the time of writing, strong story, pleasing art and addictive
stealth-like than in those where the player controller support is completely absent on gameplay. What makes Overfall unique is the
can be caught repeatedly. Similarly, giving Linux, which is a shame since this kind of way in which the story is told, having Rogue-
the choice between killing the guards or game greatly benefits from it. like permadeath elements, it occurs through its
multiple runs rather than in a linear manner. It
Overfall provides the fleshed-out world and
characters of a traditional RPG.
http://store.steampowered.com/app/402310
Join Shadwen on her mission Blueprint Tycoon
to kill the King while jumping Fresh out of Early Access, Blueprint Tycoon is a
about everywhere. game which – despite the aesthetics – has
very little to do with blueprints. It can be seen
as more of a business sim game, where the
objective is to gather raw materials and
The Mims Beginning
transform them into goods to be sold at a
profit. It’s pretty challenging, though also
rather addictive, and involves using meticulous
A contemporary alternative to Populous. planning to achieve optimum efficiency.
http://store.steampowered.com/app/454060
Web http://store.steampowered.com/app/337820
Price £12.99
A
fter some time in Early Access,
The Mims Beginning has now seen
a full release, and promises the
ability to “re-live the golden age of god
games”. In The Mims, the player is tasked
with overseeing a group of aliens who The Mims has a classic feel while using
have settled on floating islands in space, modern graphics to its advantage.
attending to their needs and ensuring Lumo
Lumo brings back isometric platforming after a
their survival. Both the premise and the though the game doesn’t have a steep
long absence, but now rendered in real 3D
graphical style is reminiscent of this learning curve. rather than hampered by the graphical
golden age, while the added benefit of One very unusual omission in the limitations of the 1990s. This does a lot to
hindsight ensures that the best aspects current build of the game is the lack of a offer extra charm to its already pretty – though
are taken from these classics. save feature during missions, and while minimalist – levels and character, and fuses
modern lighting effects and the like with
The game has pretty much every the rest of the game is polished and
some classic gameplay. There are also plenty
mechanic you’d expect, a decent story bug-free, this is something that really of fun minigames and secrets in this hugely
mode, lots of buildings and a good should have made its way into the final enjoyable game (see our review p45).
interface to manage it all. There’s lots to version. Aside from this, The Mims delivers http://store.steampowered.com/app/345480
do, so expect to get many hours out of it what it promises.
www.linuxvoice.com 47
REVIEWS BOOKS
The Smart Girl’s Guide To Privacy
Ben Everard is neither smart nor a girl, but he does protect his privacy.
Author Violet Blue
Publisher Digita Publications Privacy
Price £3.59
ISBN None
V
iolet Blue, blogger at Tiny Nibbles: women want to have a social life online, and
Open Source Sex, isn’t a typical this may include things like internet dating.
privacy advocate. Of all her Rather than keeping your privacy by turning
publications, this is the only one that could off your computer, this book goes through
be considered safe for work (the others are the particular risks and potential mitigations
all of an intimate nature). As a woman who for being an active member of the online
speaks about sexuality online, she has had community. It gives advice for what to do
to learn how to stay safe on the internet. should you lose control of information such
Keeping our private information private as through doxxing or revenge porn.
can be difficult in the digital world, but it’s a This book is an important read if
challenge that we should all take seriously. you’re a woman online, and it’s essential
The Smart Girl’s Guide To Privacy is a great reading if you’re a woman online who (for
guide for women who usually find privacy whatever reason) is particularly at risk from
advice too technical or dry to digest, and harassment.
also gives real-world examples of what
can go wrong if you don’t take necessary The best source of online privacy advice
for women we know.
precautions. The Smart Girl’s Guide To Privacy Smart, sensible and accessible advice for
is both practical and realistic. It accepts that staying safe online.
Bitcoin For The Befuddled
Money money money, must be funny, in a Bitcoin world.
Author Conrad Barski and Chris Wilmer
Publisher No Starch Press;
Price £16.50
ISBN 978-1593275730
L
et’s face it: Bitcoin is complicated all the storms and seems to be as strong as
in a huge number of ways, and if it ever. Bitcoin is fascinating for its own sake,
doesn’t befuddle you a bit, you’re but the principals behind it (such as using a
either an expert or not really thinking about blockchain as a distributed ledger) may well
it. There’s the underlying processes of how find applications in far more areas. Even if
a decentralised digital currency can even the actual currency of Bitcoin goes away, the
work, there’s the practicalities of how to buy technical ideas behind it won’t, so it’s well
and spend it and there’s the philosophical worth making the effort to understand it.
reasons for why we’d want such a thing in Bitcoin For The Befuddled is a great read
the first place. for programmers and other technical people
All this is without getting started on what who want to understand what’s going
money actually is. Bitcoin For The Befuddled on with cryptocurrencies. Non-technical
tackles these areas and even goes further to readers won’t get quite as much out of it,
show you what to expect in the future and but should still be able to follow most of the
how to incorporate the currency into your ideas in the book.
software.
Bitcoin has attracted more than its fair A wide-reaching guidebook to the most popular
digital currency.
share of controversy over the years, and has Tired of endlessly mashing money into your
had wildly varying prices, but it’s weathered screen? Then this book is for you.
48 www.linuxvoice.com
GROUP TEST BEGINNER DISTROS
GROUP TEST
Mayank Sharma helps solve the one question put to every geek – what’s the best Linux
distribution for those just starting their journey into Free Software?
On test Beginner distros
M
y niece recently got a still presents too many choices that
Antergos laptop on her birthday can potentially confuse someone
URL www.antergos.com and wondered if she who’s just starting out. They also
Licence GPL and others could run Linux on it. For a second I look almost too polished and lack
Latest release 2016.04.22 was tempted to suggest one of the vibrancy and attractiveness
What’s a rolling release doing in a either Linux Mint or Ubuntu. These that would catch the fancy of a new
Group Test of distros for newbies? distros make it easy to flesh out the user. After all, Linux’s legacy as a
installation with plugins to work distro for uber geeks or the server
Deepin with all sorts of online services and room needs to be buried under a
URL www.deepin.org play all types of content. Ubuntu thick coat of lively interfaces and
Licence GPL and others also works with a number of user-friendly design.
Latest release 15.1 popular software vendors, and you In this Group Test we’ll look at
Are the Chinese as proficient in building can rest assured that if there’s a distros designed to cater to the
distros as they are with everything else? Linux-compatible piece of code out needs (and whims) of the new Linux
there, it’ll surely work on Ubuntu. user. We’ll go off-piste and look past
Elementary OS But are these considerations
enough? If anything they just make
the usual list of desktop distros
that are often pitched to the new
URL https://elementary.io user. Our collection of distros are
Ubuntu an ideal base for a beginner
Licence GPL and others
distro. designed from the ground-up to be
Latest release 0.3.2
Arguably the most well known newbie A distro that’s just easy to use attractive and useful to beginners.
distro – does it trump the competition? isn’t always the best bet for helping We’ll analyse the unique aspects of
new users traverse the open source these distros and how they
KaOS ocean (see box below). For starters, help them convince a user to
URL https://kaosx.us the average Linux desktop distro indulge in something new and bold.
Licence GPL and others
Latest release 2016.04
KDE for the new users? You gotta
A distro that’s easy to use isn’t
be kidding me! always the best bet for helping new
Pinguy OS users traverse the open source ocean
URL www.pinguyos.com
Licence GPL and others What makes a beginner-friendly distro?
Latest release 14.04.4-1 It’s a mistake to assume that a environments before they even get to
It looks appealing but is that enough to distribution designed for the average a download image. These projects also
top the rest? desktop user would be well suited for take the time to tweak the components
someone who’s just starting out with within the distro to make them easier for
Linux. A new Linux user needs to be new users to approach them. Some even
Solus cajoled and handheld through regular modify the code of popular apps to write
everyday computing tasks as they get customised versions of the apps that are
URL www.solus-project.com
familiar with the lay of the land. designed to be operated by newbies.
Licence GPL and others
To begin with, these distributions help No matter how they customise their
Latest release 1.1 users by making choices on their behalf. offering, Linux distributions aimed at the
Is this the solution to end all desktop Linux and FOSS are all about choice, but inexperienced user are far easier to get
geekery? it’d be a cumbersome exercise for a new started with and use than your typical
user to sift through the myriad desktop desktop distro.
50 www.linuxvoice.com
BEGINNER DISTROS GROUP TEST
Hardware requirements
Do they burden your silicon?
W
hat’s the cost of all the glow and custom-built resource-friendly compositing That said, we should keep in mind that
glitter you get with these graphical environments. Furthermore, many some distros are able to tune down the
beginner-friendly distros? Do they of these distros bundle some software that glitter automatically based on the hardware
all need state-of-the-art hardware? The you wouldn’t normally find installed by resources at their disposal, so you might be
different distros on test here have different default inside a desktop distribution, able to run them on a machine that’s been
hardware requirements. The primary factor primarily because of the lack of their sitting on your desk for a couple of years
is the choice of their desktop environment. intended audiences’ ability to find these for now. However, to enjoy these distros to the
Some extend the ease of use by extending themselves. Some of these specialised hilt and take full advantage of their beginner-
the conveniences offered by full-blown applications can be quite demanding and friendly usability, we’ll advise you to run
desktop environments, while others rely on may take a toll on your computer. them on a well stocked machine.
Deepin
Is beauty really just skin deep?
G
ive Deepin a try and you’ll be amazed
by its reimagined desktop
experience. The Debian-based distro
has customised all aspects of the distro and
that has had a pleasantly positive effect on
the user experience.
By default, Deepin’s installer will take up
all the space on the selected disk, but you
can manually configure the partitioning
by clicking Expert Mode. After installation
you get to witness one of the major factors
that sets the distro apart from other Debian
derivatives; the distro’s home-grown The entire distro is designed with ease of use in mind, and is well-suited for touch devices as well.
desktop – Deepin Desktop Environment
(DDE). It’s based on HTML 5 and WebKit, and At first glance, the Deepin Store appears uses the same parlance as Windows and
uses a mix of QML and Go for its various to be a clone of the Ubuntu Software Centre. involves right-clicking an application and
components. DDE has a clean and clutter- However, it’s better organised and offers selecting the Uninstall option. In another
free interface and tries its best to replicate a much neater experience. Besides the departure from convention, the Control
the usability and aesthetics of Mac OS X. desktop apps, the Store also offers several Centre is integrated into the desktop itself,
Besides the desktop itself, notable web apps for installation, such as Google instead of being offered as a distinct
homebrewed Deepin components include Drive, Hangout, Pixlr editor, and more, application. The bottom-right hot corner
the application launcher, dock and control courtesy of the distro’s collaboration with opens the pop-out side panel that houses
centre. The launcher is elegant and blazingly the Intel Crosswalk project. the various settings under different sections.
fast and it will either display the installed You can manage all aspects of the desktop,
apps in alphabetical order or separate them Kitchen-sink approach from the boot manager to the desktop
under categories. In addition to the customised desktop, theme from the controls here. It also helps
The desktop has hot corners and the top- Deepin includes several custom-built apps, keep track of and apply any updates. The
left corner opens the full-screen application such as Deepin Boot Maker for creating a Control Centre also enables you to access
launcher. To search for an application you bootable USB, Deepin Music, Deepin Store, the Deepin Manual as well as the Remote
start typing the letters and the launcher will and more. Besides the custom apps the Assistance feature, which uses Chrome
offer all matching results. The dock at the distro also bundles several proprietary apps Remote Desktop to share your desktop with
bottom of the screen has several views/ such as Google Chrome, the Steam client, other Deepin users.
modes. Besides the default stylish Fashion WPS Office, and CrossOver Linux.
mode, there’s the Efficient mode, which The Deepin developers have gone to VERDICT
places icons on the desktop, as well as great lengths to make using the distro A well designed and
elegant distro with a host
the Classic mode, which is similar to the familiar to those coming from proprietary of custom apps.
Efficient mode but with smaller icons. environments. For instance, removing apps
www.linuxvoice.com 51
GROUP TEST BEGINNER DISTROS
KaOS
Chaos?
K
aOS isn’t your typical distro Continuing with the KDE theme of the
designed for new users: it’s distro, there’s the Calligra office suite,
designed primarily for people the Qupzilla web browser, Clementine
who are at home with the KDE desktop. audio player, KDE Telepathy for instant
What makes KaOS beginner-friendly messaging, Quassel for IRC, SMPlayer
is its tightly integrated streamlined and MPV media players, and many
design. Unlike most other Linux distros more including apps to stream
that burden users with ample choices, YouTube and record the screen and
KaOS has consciously decided to keep the webcam. To help users flesh out
the options limited. The installable live their installer, KaOS uses the Octopi
distro takes a conservative approach to graphical application installer, which KaOS has a simple repository layout that helps newbie
package management, offering only is a front-end to the powerful Pacman wrap their heads around the open source universe.
KDE as the desktop, and is available for package manager. To maintain its
64-bit machines with its repositories tight integration, KaOS doesn’t use business simple and straightforward.
replete with only x86_64 packages. any upstream repositories, choosing Also helping its cause is the new KDE
Another aspect that will help new instead to build each of the packages Plasma 5, which is pretty enough to
users is the use of the distro-agnostic in its three repositories from scratch rope in new users who wouldn’t even
Calamares installer. One of its highlights specifically for KaOS. realise that they’re using a distro based
is the advanced partitioning feature, on one of the geekiest flavours of Linux.
which offers several partitioning KDE for newbies
options including the ability to carve out KDE is perhaps not the easiest desktop VERDICT
partitions to your liking. environment to get to grip with, but Its conservative approach
and tight integration give
The distro includes all the apps KaOS is so well packaged that it it real appeal.
you’d need for everyday desktop tasks. makes conducting everyday desktop
Antergos
Enter the dragon.
H
ere’s another Arch-based and more. The installer then downloads
distro that’s developed with the components you have selected,
simplicity in mind. Antergos which might take some time depending
provides a fully configured environment on your internet connection.
that can be used right out of the box.
The live CD boots to a customised Search for applications
Gnome-based desktop featuring a dock Besides the apps offered in the
ripped out from the Activities Overview installation screen, if you choose to
using the Dash to Dock extension. install the default Gnome desktop,
Antergos uses a custom installer the distro also includes the Gnome Like KaOS, Antergos will help instill the ways of a rolling
called Cnchi, which is easy to navigate Music and Video apps, Pidgin, and the release without intimidating a beginner.
and use despite its beta status. Its Chromium browser. Antergos ships with
partitioning step also offers several a graphical package manager called Also, while the email category lists just
options that will appeal to both new and Pamac. It’s easy to navigate, and just one client, a search turns up several.
experienced users. like KaOS’s Octopi, Pacmac might not However, once you’ve got it set up,
Despite its goal, Antergos passes be pretty to look at but it gets the job Antergos is a pleasure to use and
on the burden of selecting core done. However we’d suggest using doesn’t pose any debilitating issues
components to the user. One the search function to find packages that are beyond the grasp of ordinary
screen asks you to select a desktop instead of relying on the available desktop users.
environment from either Cinnamon, software categories, which are not
Gnome, KDE, Mate, Openbox or Xfce. In always intuitive. For instance, Firefox is VERDICT
The software selection
the next you’re asked to select optional available under the browser category, screen rules it out for
components from a list that includes while add-ons for the browser are absolute Linux newbies.
Firefox, LibreOffice, Steam, PlayOnLinux housed under a different category.
52 www.linuxvoice.com
BEGINNER DISTROS GROUP TEST
Pinguy OS Attack of the clones
The pinup guy.
Missing Windows already?
Z
orin OS is an unusual distro that, like the
others in this group test, is aimed at
easing Windows users into a Linux
distro. However, what makes it unique is its
Gnome desktop that’s modified to resemble
Windows 7. Zorin is based on Ubuntu, and its
goal is to package the goodness offered by
that distro into a system that “anyone can use
without learning anything new thanks to its
familiar interface.”
To that end, the home-brewed Zorin Look
Changer app lets you customise the distro
to match different versions of Windows.
The default desktop behaves pretty much
like Windows 7 and although it’s not an
Pinguy OS also includes Pinguy Builder, which is a fork of the popular Remastersys tool to help exact replica, it does provide a comfortable
create custom ISO images from your Pinguy OS installation. experience for users who only have experience
with using Windows (you can also use the
T
he Pinguy OS distro gives you the Spotify and Steam for Linux. There’s Look Changer to make the desktop resemble
option to either explore the live also Wine, which you can manage with the older Windows XP, if you’re feeling
environment or jump straight into the bundled PlayOnLinux front-end. For nostalgic). Besides the applications to modify
the installer. The distro is based on the customisers, the distro includes the its desktop layout, Zorin OS also includes a
Ubuntu LTS releases and relies on the Gnome and Ubuntu Tweak Tools. custom app to easily modify the theme.
Ubuntu installer to anchor the distro to The distro is available in four different
your computer. Shiny as a new pin versions: the Core and Lite editions are free,
The distro uses a customised Gnome If you need more software, the distro while the Business and Ultimate flavours
desktop featuring a semi-transparent offers the Ubuntu Software Centre as well cost €8.99 (about £7) and €9.99 (about £8)
panel at the top and a dock at the bottom as the Synaptic package manager. The respectively. The paid versions come with
with a Conky-powered applet applet on the package managers are equipped with support and a few extra features, such as the
right-hand side of the screen displaying a large number of repositories, many of option of using interfaces that mimic Mac
vital information about the computer. The which are enabled by default, including OS X and Windows 2000. The Core edition
panel houses the Gno-Menu extension, those for Linux Mint and Ubuntu. There includes all the apps you’d find in a normal
which serves as the default application are PPAs for themes and apps such as desktop Linux distribution, while the Lite
launcher. It contains a categorised list Clementine, VLC and Gnome. Pinguy also edition is designed for older machines and
of software and also keeps track of includes the Y PPA Manager to keep track the Ultimate edition is chock-full of all kinds of
frequently used apps and recently viewed of all the PPAs. apps and games.
files. You can also bring up the default Pinguy’s developer has cherry-picked
Gnome 3 Activities application launcher the components to offer a convenient
by pressing the Win key. Move the mouse user experience. The distro uses the Nemo
to the left-hand edge of the screen to file manager along with a bunch of plugins
reveal several home folders. to integrate Dropbox, Samba and the file
Pinguy is chock full of general purpose archiver. The file manager is also equipped
and specialised apps including LibreOffice, with scripts such as Torrent-Video-Player,
OpenShot video editor, Boot Repair, Devede, so you can right-click on a .torrent file
Empathy, Handbrake, Shotwell, and more. and use the Torrent-Video-Player option
For video playback, besides VLC, it’s also to stream it without downloading.
got the Plex Media server. The Clementine There’s also a script to help organise your
music player also comes equipped with downloaded TV shows and movies into
access to over a dozen online music folders with cover arts and subtitles.
services including Grooveshark, Last.fm,
Spotify, Jamendo and SoundCloud. VERDICT
An elegant distro that’s
Besides the open source apps, Pinguy OS easy to use but needs a You can use the Zorin web browser manager to
also includes several popular proprietary decent machine. pick and install any of the four popular browsers.
ones, such as Dropbox, Skype, TeamViewer,
www.linuxvoice.com 53
GROUP TEST BEGINNER DISTROS
Elementary OS vs Solus
Custom desktop faceoff.
E
lementary OS has made a name
for itself as an elegant and
user-friendly distro. Its custom
desktop, Pantheon, takes cues from the
Mac OS X desktop, and has its own
Mutter-based window manager called
Gala. The desktop nicely integrates the
various other important elements, such
as the Plank dock, the top panel (called
Wingpanel) and the Slingshot application
launcher, to present a smooth, unified,
user experience. Nearly all actions on
the desktop are subtly animated.
One of the strangest things about
Elementary is that it comes with
an unusual set of applications. The
distro supplies a number of custom
tools, such as the Geary mail client,
Scratch text editor and Audience Elementary OS can be used on older hardware as well because of its low system requirements.
video player, which are designed to
assist inexperienced users. However, The distro boots into a Live and has a search box for finding apps
Elementary OS doesn’t offer many session, and as with most of the core without having to navigate the menus.
apps out of the box and doesn’t include components, its installer too is custom The one strange aspect of the menu
proprietary codecs or ship any non-GTK built. While it’s fairly intuitive and easy is that it keeps reorganising the apps
apps, which is why it doesn’t include on the eyes, it lacks an automated within the categories depending on
the likes of LibreOffice and Firefox. partitioner. This is something that the how often they are used. While this
Instead it uses the Midori web browser. developers will have to correct very ensures that the frequently used apps
For package management the distro soon, because asking first-timers to are always at the top of each menu, it
uses the Ubuntu’s Software Centre fire up GParted and manually craft might seem confusing to not find the
and it pulls software from the Ubuntu partitions won’t get them very far. app at their usual places.
repositories as well as PPAs of its own. Solus uses a custom desktop called Another unique aspect of Solus is
Budgie, which is based on Gnome the all-in-one applet, notification and
Another solution libraries. The Budgie desktop tries its customisation centre called Raven.
Unlike some of its peers, Solus is not best to replicate the classic Gnome This houses all the information in two
based on another distro, which gives its look and feel while offering several tabs: all the notifications are tracked
developers manoeuvrability to mould all conveniences of the modern Gnome in one tab, while the Applets tab
aspects of the distro to their vision, desktop. Budgie’s application menu displays the calendar, sound volume
including the user experience. displays a categorised list of apps and sound devices. All items on the
panel, including the application menu
and clock etc, are applets, and you can
place and reposition them.
Solus has plenty of apps for
regular desktop use, including Firefox,
Thunderbird and VLC, but lacks an
office suite and any sort of games.
The distro also has a custom package
manager that’s designed intelligently
for inexperienced campaigners with a
minimal, unimposing interface.
VERDICT
ELEMENTARY OS It’s SOLUS The easy-to-
pleasing to the eyes use experience is
but its defaults aren’t hindered by the lack of
the best for a newbie. an automated installer.
The Budgie desktop is a wonderful piece of software that blends old and new functionality.
54 www.linuxvoice.com
BEGINNER DISTROS GROUP TEST
OUR VERDICT
Beginner distros
R
ecommending a distro to a the window manager up to its apps
new Linux user depends is crafted to adhere to its design
strongly on their proficiency principles. We’d recommend it over
and comfort level with the Solus, but only just.
computer. That said, we hesitate to The top two distros for new
recommend Antergos, because it Linux users are Pinguy OS and Thanks to its focus on design and aesthetics, Deepin is easily the
presents too many choices to the Deepin. Both distros offer a good new benchmark for newbie-friendly distros.
user who is probably not well mix of form and function and their
equipped to make the correct pleasing desktop environments
decision. Then there’s KaOS, which give access to its vast number of
presents an intriguing option applications. However Pinguy’s 1st Deepin
because of its tight integration, but usability comes at a cost. All of its
isn’t designed specifically for new customisations consume a lot of Killer feature Elegant-looking custom desktop and tools.
users. However, those with some resources, and you’ll only be able to URL www.deepin.org
mileage under their belt will surely enjoy Pinguy OS on a machine that Full of customisations to serve the new user well.
appreciate the consistency of the has at least 4GB of RAM.
desktop and the coherent design. Deepin too isn’t without issues.
It’s a close call between Solus Designed to be user friendly, the 2nd PinguyOS
and Elementary OS for a position distro is ideal for beginners but
on the podium. Solus is by no doesn’t offer much for advanced Killer feature Customised Gnome desktop and apps galore.
means just another newbie-friendly users. So while its installer enables URL http://pinguyos.com
Designed to pamper the new user, it only narrowly loses out.
distro. It’s a project with solid you to perform all the usual
foundations and leadership that partitioning operations, Deepin
has a clear vision and experience
to realise it. Once the issue with the
doesn’t support LVM, and you
can’t create an encrypted home
3rd Elementary
installer have been ironed out, the folder, which is something that
distro should be ready to compete most distros support. It’s a pretty Killer feature Pantheon desktop and pure GTK experience.
URL https://elementary.io
with the established players. close battle between Pinguy OS
It’s aesthetically pleasing, but its default software selection
Elementary OS has also put in and Deepin but in the end Pinguy’s necessitates a visit to the package manager.
quite an effort into building custom stringent hardware requirements
tools and libraries. Everything from help clinch the deal for Deepin.
4th Solus OS
Solus OS has solid foundations and
leadership that has a clear vision Killer feature The Budgie desktop environment.
URL www.solus-project.com
and the experience to realise it Still early days for the distro that needs to simplify its installer.
Other user-friendly distros 5th KaOS
There are several other projects that desktop built atop the lightweight
produce distros that are easy to use and Enlightenment desktop. Like its peers, Killer feature Well integrated KDE experience.
can also double up as beginner-friendly Bodhi takes the pain out of cumbersome URL https://kaosx.us
distros. There’s PCLinuxOS, which began processes like fleshing out the distro by The KDE-centric distro isn’t specifically designed for beginners
life as a repository for improving a stock using easy-to-operate custom apps. but still does a good job.
Mandrake release and then forked into a You might also want to take at friendly
distro of its own in 2003. It uses a rolling versions of popular mainstream releases.
release model with ISO releases every Korora, for example, produces versions
now and then to assist new users take based on the official Fedora releases. 6th Antergos
the plunge. Then there’s the Arch-based Manjaro
Another useful distro is Bodhi Linux, Linux, which produces several flavours,
whose minimalistic nature is in contrast and the Ubuntu-based Netrunner. These Killer feature Makes Arch Linux accessible to the masses.
to the usual approach of cramming the distros lower the learning curve of using URL www.antergos.com
distro with apps. Bodhi uses the Moksha their parent distros. For a newbie-friendly distro, it offers too many choices.
www.linuxvoice.com 55
SUBSCRIBE
Subscribe
shop.linuxvoice.com
Introducing Linux Voice,
the magazine that:
Gives 50% of its profits
back to Free Software
Licenses its content
CC-BY-SA within 9 months
12-month subs prices
UK – £55
Europe – £85
US/Canada – £95
ROW – £99
7-month subs prices DIGITAL
UK – £38 SUBSCRIPTION
Europe – £53
US/Canada – £57
ONLY £38
ROW – £60
Get 114 pages Access our Save money on
of tutorials, rapidly growing the shop price
features, interviews back-issues archive and get each issue
and reviews – all DRM-free and delivered to
every month ready to download your door
Payment is in Pounds Sterling. 12-month subscribers will receive 12 issues of Linux Voice a year. 7-month
subscribers will receive 7 issue of Linux Voice. If you are dissatisfied in any way you can write to us to cancel your
subscription at subscriptions@linuxvoice.com and we will refund you for all unmailed issues.
56 www.linuxvoice.com
NEXT MONTH
NEXT MONTH IN
ON SALE EVEN MORE AWESOME!
THURSDAY
Firefox
21 JULY For so long the
darling of Free
Software, Firefox has
had its ups and
downs. Is it
yesterday’s web
browser now, or can
it take back the web?
ZFS
The Ferrari of
filesystems has been
on our radar for a
while now, but we’ve
never been brave
enough to take the
plunge. Well now we
are – let’s upgrade!
Wayland
The graphical
SAVE TIME FOR THE WIN
revolution has been
a long time coming.
Now it’s time to set
It’s summer – so speed up your computer up, configure and
with our clutch of tips and tricks and spend use Wayland. How is
it going to make our
the time saved doing awesome things. lives better than X?
LINUX VOICE IS BROUGHT TO YOU BY
Editor Graham Morrison Editorial consultant Nick Veitch through the use of advice in this magazine. Copyright Linux is a trademark of Linus
graham@linuxvoice.com nick@linuxvoice.com Experiment with Linux at your own risk! Torvalds, and is used with permission.
Deputy editor Andrew Gregory Distributed by Marketforce (UK) Ltd, 2nd Anything in this magazine may not be
andrew@linuxvoice.com All code printed in this magazine is licensed Floor, 5 Churchill Place, Canary Wharf, reproduced without permission of the editor,
Technical editor Ben Everard under the GNU GPLv3 London, E14 5HU until March 2017 when all content (including
ben@linuxvoice.com Tel: +44 (0) 20 3148 3300 our images) is re-licensed CC-BY-SA.
Editor at large Mike Saunders Printed in the UK by ©Linux Voice Ltd 2015
mike@linuxvoice.com Acorn Web Offset Ltd Circulation Marketing by Intermedia Brand ISSN 2054-3778
Creative director Stacey Black Marketing Ltd, registered office North Quay
stacey@linuxvoice.com Disclaimer We accept no liability for any House, Sutton Harbour, Plymouth PL4 0RA Subscribe: shop.linuxvoice.com
loss of data or damage to your hardware Tel: 01737 852166 subscriptions@linuxvoice.com
www.linuxvoice.com 57
FOSSPICKS
FOSSpicks Sparkling gems and new
releases from the world of
Free and Open Source Software
Our benevolent editorial overlord Graham Morrison tears himself away
from updating Arch Linux to search for the best new free software.
Storage pruner
QDirStat 1.0
O
ne of our favourite Linux visualisation was an area chart, it or operating system with a modern
utilities is called Filelight. also included a tree view of your Qt port.
This is a KDE tool, now files and folders, just like a Unlike Filelight, which really only
ported to other environments, that traditional file manager. Having this served one function, QDirStat gives
visualises the size of files stored on functionality built right into the you lots more information and
your drive using a multilevel pie desktop, so you could launch the control over how you look at your
chart (called a sunburst chart, application as soon as you got your files and folder. The tree view lists
apparently). With a simple glance, first ‘Out of storage space’ the percentage of your storage that
you can easily see which files were warnings, saved our storage bacon a file or folder is taking up, along
taking up the most space and on several nervous occasions. with specific sizes, and a
where they were located. It’s still the customisable colour can be defined
best way we’ve found of freeing up KDE Kontinues for different file types, making them
space by finding forgotten ISOs and QDirStat is a continuation and port very easy to identify and filter from
games. Many of these features of KDirStat (80% of the old code has both the tree view and the area map
could also be found in KDE 3’s been rewritten), unlocking it from (called a ‘tree map’ by QDirStat). You
KDirStat, often included in older that old KDE dependency and can even create rules for ignoring
distributions, and while its size making it available to any desktop files you don’t want listed.
Clicking on a file or folder nicely
animates a transition in the other
view so that everything stays in
sync, and the same right-click
4 5 context menu can be used to
perform actions on your selected
3 files and folders. Along with the
expected, and dangerous, ‘delete’
option, you can open a terminal or a
6
file manager, zoom in to expand a
folder, and perform other actions
like make clean and git clean to
8 remove old build material from
7 folders of source code. You can
2
create your own cleanup actions
1
from the Settings pane, and it’s as
easy as entering the command with
some simple variables representing
the path, file and directory name.
This deceptively powerful feature
makes QDirStat the most powerful
application we’ve found for
visualising space and folders and
1 Tree map The area of each block corresponds to a file size, while the colour is for file type. 2 Custom cleaning
Developers can remove unwanted build and git files, and you can add your own. 3 File tree The traditional unfoldable file
safely cleaning up your storage.
view is also available. 4 Percentages Easily see exactly which files and folders are taking the most space.
5 Custom columns You can add and remove columns from the main view. 6 View rules Ignore files you don’t want Project website
touched. 7 MIME types Set colours and create your own types for better visualisation. https://github.com/shundhammer/qdirstat
58 www.linuxvoice.com
FOSSPICKS
Bash Google
Googler 2.4.1
I
t could be a temporary trend, whether it’s for scripts or for your
but we’re using the command own convenience. Type googler
line more and more each followed by your search terms and
month. Its enforced minimalism is the results are delivered directly to
the opposite of GUI bloat because your terminal.
command line programmers are We’re happy to report that
forced to think carefully about how development has been prolific, and
functions are implemented and the recent 2.4.0 upgrade bundles
how those functions presented to several significant new features.
the user, whether that’s through a The output looks much cleaner,
command’s arguments, or through especially for news results, and you
a curses-based interface. can click on links to open them
This doesn’t seem to happen from your browser.
with most desktops and graphical There are keyboard shortcuts too, We had thought that by using the command line we’d save
user interfaces, where there always and you can output the results with ourselves from Google distractions. We were wrong.
seems to be room to add another JSON formatting. This is excellent if
feature, or where usability testing you want to script your searches applications, and it’s this feature we
gets forgotten in all the excitement and integrate the results with other think will be the most helpful,
of adding new things. Googler is especially for analysis such as
another step forward in our
command line conversion, and it’s
Googler is a tool that lets you customised ranking reports.
one we’ve mentioned before. It’s a interact with Google from Project website
tool that lets you interact with
Google from the command line, the command line https://github.com/jarun/googler/releases/
tag/v2.4.1
Search utility
FSearch 0.1alpha
N
ow that our hard drives are time you ran a search, but FSearch
full and we’ve forgotten side-steps this issue by being
half the things on there, incredibly fast. It indexed our home
local search has become as folder, containing 136,250 items, in
important as online search. And yet under 2 seconds, and that’s with a
we’ve yet to find the ideal solution. PC from five years ago.
There are lots of pervasive search It delivers results just as quickly,
tools – Ubuntu’s works well, as filling the results table as you type.
does KDE’s – but we’re slightly It doesn’t search within files, but
distrustful of their pervasive nature. that’s not what FSearch is for. It
KDE’s Nepomuk process often helps you find your files instantly,
seems to take up huge slices of whether that’s via a regular We used to think desktop integrated search was what we were
CPU and its database size can be expression or some vague looking for, but FSearch does it better and more quickly.
huge. FSearch offers the same Proustian remembrance of what
pervasive search functions, but the file is called. support Qt 5; it’s tiny and takes no
constrains them to a single FSearch is built using GTK 3, time to build. This is in huge
application where you have although the author wants to also contrast to many of the other
complete control over what is search tools available, and despite
indexed, where the index lives and
when you need to run it.
FSearch indexed our home this being a very early alpha, we had
no issues with stability or usability.
This size and scope would folder, containing 136,250
ordinarily make it slow, as you’d
need to rebuild the database each items, in under 2 seconds Project website
https://cboxdoerfer.github.io/fsearch/
www.linuxvoice.com 59
FOSSPICKS
C++ interpreter
Cling
T
here’s a huge difference with the creation of interpreters that
between interpreted can transform what was previously
programming languages only compilable code into an
and those that need to be compiled executable instruction that runs
and run as binary objects. This immediately.
difference is apparent in both the This on the fly interpretation is
complexity of running your exactly what Cling does, using both
applications and in the approaches the LLVM and Clang libraries. You
taken by the syntax within the type in C++ and it runs each line
languages. But the speed and independently, building a virtual
capabilities of modern hardware is machine of your code. For those of Run C++ like it’s embedded into your Commodore 64.
definitely bringing both camps us with a long history of C and C++,
closer from the perspectives both it feels like black magic. But it’s also you to write good code, and any
of end users and also developers. liberating, letting you try things out errors or ambiguities appear
Python and Go, for example, are or run ‘scripts’ without the tedium of immediately.
interpreted languages that are used having to run make at every step to If you’ve ever wanted to try C++
in a phenomenal number of check your code. It’s also a great but have always been put off by the
mission critical applications, educational tool, because it forces complexity of building, linking and
previously the domain of languages running executables, the world of
like C++. C++ itself is an old
traditional language that’s seen as
Modern hardware is bringing interpreted programming is right
here waiting for you.
poles apart from Python and Go. compiled and interpreted
However, remarkably, C++ is also
benefiting from modern hardware languages closer together Project website
https://root.cern.ch/cling
Developer package manager
Qpm
W
e’ve done some Qt available already are unlikely to be
programming in the available through a normal package
past, and one of the manager, and really help with both
things that can be annoying is Qt and QML/QtQuick development.
trying to track down the various bits It also keeps your packages and
and pieces of Qt-related dependencies up to date,
dependencies that you may need to independent from your distribution
pool together to make a single package updates, and that’s
application. This is a problem that important when you need control
Qpm attempts to solve rather over exactly what your applications At least your typical developer won’t be put off by using curl to
elegantly. are linked to. install the package.
Unlike packages for your chosen
distribution, Qpm’s packages are Developers! future, as the qpm command itself
tailored specifically for your project, At the time of writing, there are only makes it easy to publish your own.
enabling you to use them and 34 packages being served (qpm list The whole command itself is easy
incorporate them into your code tells you this), but we’re hoping to use – it’s installed via a simple
without any extra hassle. This is many more are included in the curl command and typing qpm tells
different from installing them you all you need to know about
through a package manager, as searching, installing, installing and
you’re never certain which versions Qpm keeps your packages creating packages.
are installed, and that’s if there’s a and dependencies up to date
package available. It’s still early
independent from your distro
Project website
days for Qpm, but the packages www.qpm.io
60 www.linuxvoice.com
FOSSPICKS
Emulator front-end
AQemu 0.9.2
E
mulating old hardware is for example, you can specify the PC
definitely fun, even if most of hardware you’d like to conjure up by
the fun comes from getting its age – a PC from 1990–95, for
old games running rather than example, which is a great way of
playing them. Too often we go back playing games from the golden era
to our old favourites to find that of DOS. Sadly, the GUI lacks the
without our 14-year-old reflexes, equivalent ‘Turbo’ button, but we
we’re just cannon fodder. Emulating may make a feature request for a
modern hardware isn’t always fun, fully authentic 1995 experience.
but it is very practical. It’s the best
way of experimenting with a new Run Linux on Linux
distribution, and for deploying a For the less nostalgic, AQemu will
thousand servers across the cloud also create machines using a Linux,
(sort of). Windows or OS X template, Why is it that a PC with a CPU speed of 66MHz and only 64MB of
Back on the desktop, QEmu is one complete with storage and RAM booted faster than the futuristic machine we’re typing on?
of our favourite tools, and while it’s networking. As with all of the
great from the command line, options, you still need to provide
there’s nothing wrong with your own installation medium, and go through the installation
accepting a little help from the GUI. procedure, even if this is from a disk
AQemu is an alternative GUI that
provides lots of help, from locating
AQemu is an alternative GUI image of Doom taken from a
3.5-inch floppy disk.
the location of the binary at startup for the QEmu emulator that
to helping you create your first
virtual hardware. We like the way, provides lots of help Project website
https://sourceforge.net/projects/aqemu
Disk manager
KDE Partition Manager 2.2.0
A
ll of us have been saved by using a GTK-based desktop, and
the perennial GParted. It’s offers almost all the same features
the one application you can as GParted.
rely on when you need to dance But this release is significant to
that dangerous partitioning dance. us for one major reason – KDE
When there’s delicate manoeuvring Partition Manager 2.2.0 adds a
to be done, we’d even prefer it over feature we’ve wanted/needed from
the command line equivalents GParted for ages – the ability to
because sometimes you just want resize encrypted LUKS partitions.
to see that your data is safe. This is important because more
GParted has never failed us, and more of us are using LUKS
despite our own actions failing us partitions to store our Linux and
on more than one occasion. KDE data installations.
Partition Manager is very much like
GParted, only with what we’d Killer feature As you can see, with five
describe as a more modern user LUKS partitions stop people getting hard drives and up to partition rather than creating a new
interface. Having the choice, and access to your data when they have seven partition on each, one and copying over the data is
the competition, can only be a good access to your hardware, but we desperately needed going to save us lots of trouble.
thing for such an important task, because of the way LUKS hides a good partition For that feature alone, thank you
and we’re suckers for a nice KDE Linux filesystems within their manager. KDE Partition Manager!
application. random noise of encryption,
Project website
KDE Partition Manager looks resizing has been a tricky problem, https://stikonas.eu/wordpress/2016/05/27/
fantastic, especially if you’re not and being able to resize your kde-partition-manager-2-2-0
www.linuxvoice.com 61
FOSSPICKS
Webmail
Roundcube 1.2.0
T
his has been a month of has native support for encryption.
turmoil for one of the of the This is significant because it’s both
largest server-based open a brilliant feature and a brilliant
source projects we rely on. The show of support for what should be
OwnCloud project has been forked part of every email client and email
into NextCloud, and we genuinely exchange.
hope this doesn’t affect the
development or momentum of Keep it secret…
what we think is one of the most Roundcube accomplishes
important open source and Free encryption either via a Firefox plugin
Software projects available. called Mailvelope, or on the server
It shouldn’t, as most of the via the Enigma plugin and GnuPG. Finally, there’s an open source webmail solution that has
developers from the old project From the perspective of a user, encryption support baked in.
have moved to the new project, but Mailvelope is easy to install and get
we don’t want to imagine a future to grips with, and makes grabbing into your web-based keychain so
without a genuine alternative to the remote PGG keys for decryption you can decrypt messages. This is
proprietary cloud-based behemoths simple – when it detects a public exactly what you need, so there’s
of Google, Microsoft and Facebook. key, it will automatically import this even less of an excuse to start
Another cloud project we used signing and encrypting emails. Oh,
has also celebrated this month, and
that’s Roundcube, with a new
When it detects a public key, and we love the way you can now
finally search between dates!
release of its brilliant webmail it will import this into your Project website
application. The major feature for
this update is that Roundcube now web-based keychain /news/2016/05/22/roundcube-webmail-1.2.0-
released
Spectrum analyser
Cava (from git)
W
e have Kurruptor on our going into remote computers and
IRC channel to thank for even servers.
this particular find. It’s a It feels weird SSHing into a box
simple but useful and slightly and then watching the little
compulsive discovery. Cava is a frequency bars bouncing around,
spectrum analyser for the but it’s also surprisingly useful. This
command line, rendering the could be a visual baby monitor, for
various frequencies of an audio example, or a way of seeing if a
signal using Curses onto your conversation has finished without
console. Because it’s for the having to listen to the content. If Now Hollywood film directors have another Linux command to use
command line, it’s both tiny and you’re processing audio, it’s also a when portraying hackers on screen.
quick, and being quick is a great great way to see the range of
asset if you’re measuring audio in frequencies you’re recording, or listening on the right input, and we’d
real time. have recorded. recommend the excellent
On a practical level, it can be a By default, it works easily with Pavucontrol for this. A few simple
useful way of monitoring your PulseAudio. Just run the command key commands can change the
surroundings before a recording. and make sure PulseAudio is scale and colours, but that
Higher and lower frequencies may overcomplicates things. Just let
be inaudible to our old ears and yet yourself get mesmerised by the
be visible on the screen. But It’s a great way to see the look of music on your console.
because this is also running from range of frequencies you’re
the command line, you can use it to
recording, or have recorded
Project website
snoop, at least visually, on the audio https://github.com/karlstav/cava
62 www.linuxvoice.com
FOSSPICKS
Data rescue
TestDisk 7.0
P
icture the scene: you’re
browsing an online shopping
emporium and see that
there’s a special offer on 1TB hard
drives. This is brilliant. You can now
buy several and potentially put
them to good use as part of a RAID
array, or just have lots of cheap
storage. This is what we did a
couple of years ago, except we
never got around to building a RAID
configuration and instead used the
drives within our main PC where the
drives quickly absorbed gigabytes
of Kylie Ann Minogue rarities.
Two years later, we’re updating Never start a partitioning operation without having photorec and testdisk handy (and a backup, of course).
Arch and installing Ubuntu, when
we repartition one of those drives, – and we don’t want to promise just photos, as their names imply).
certain that the drive we chose was miracles – there’s a chance your Dozens of file types are supported,
correct. Except, because it’s old table can be resurrected and and this major update adds more,
identical to two more 1TB drives, it rewritten to your drive, reinstating including .kra Krita files. When
isn’t. It’s the one filled with treasured your data immediately. detected, you can rip out the raw
sounds. Denial. Anger. Bargaining. If this happens to you, first make data into a usable file again, though
Depression. Acceptance. sure you don’t touch your drive at you’ll often lose the filename and
all. Instead, reboot to a rescue USB any folder organisation you had.
Lazarus raised stick or run TestDisk from a distro This is a last resort rather than a
Thankfully, there’s TestDisk. This that doesn’t touch your drive. fix, but it might mean you’re able to
amazing utility has saved our raw There’s a good chance it will be able keep your precious photos, and
data more times than we can think to restore your old configuration both TestDisk and PhotoRec are two
of. It’s particularly brilliant when and your data. the best tools to have on hand. That
you’ve accidentally repartitioned a If this doesn’t work, there’s they’re still being developed and
drive already full of data, a situation another part of the TestDisk suite passing major milestones like a 7.0
that initially looks lamentably grim. that can help, and that’s PhotoRec release is something we’re all
But because you’ve only written a and its GUI counterpart, QPhotoRec. grateful for. Thanks TestDisk!
new partition table to a special part These scan the raw blocks of your
of the drive, and not overwritten drive looking for the telltale Project website
your actual data, there’s a chance signatures of known file types (not www.cgsecurity.org
STEP BY STEP: RESTORE YOUR OLD PARTITION
1
Run testdisk with the drive you want to 2
TestDisk will guess the partition type, but 3
TestDisk will look for the remnants of a
scan as the single argument. You’ll also you need to make sure it’s guessed partition table. When it finds one, select
need sudo or root privileges. correctly. Then select Analyse. Write – then donate to TestDisk!
www.linuxvoice.com 63
INTRO TUTORIALS
TUTORIALS
Warning: excessive Linux knowledge may lead to fun and more efficient computing.
In this issue . . .
66 68
Hide data in images with Build fast, clean websites
Mike Saunders
Makes a bomb shipping computers from Steganography with Hugo
Barnard’s Star to Sol.
Conceal secret messages or other information Does the world need yet another website creation
inside files with Ben Everard’s sneaky guide to the engine? Well yes, when it’s as awesome as Hugo, as
art of steganography. Amit Saha explains.
H
ave I mentioned recently how
much I love Vim? It has been a
while since I’ve written about it 74 78
in Linux Voice, and if it weren’t for Ben’s
wise editorial decisions I’d do a 32-page
feature every month about its raw
awesomeness. But seriously, learning a
powerful text editor like Vim was the
best decision I ever made (in my
computing life, at least. Combining
bacon with Sriracha sauce was the Publish with Display data on a
best decision ever).
Vim scares many people away; it
Free Software physical interface
feels like some weird relic from the Yes, it’s possible to make ebooks from scratch – Les Pounder mixes together a Raspberry Pi Zero,
1970s at first (and indeed, its origins lie Andrew Conway has all the tips and tricks you need stepper motors and external data sources for
way back in early Unix releases). But to do just that. maximum awesomeness.
don’t just think about it like an editor –
it’s more of a machine and a language
for manipulating text. Just like a Coding
programming language, you can build
up useful routines and methods for
Get access to ev
doing things, and repeat them to save ery
Linux Voice tuto
you heaps of time. Yes, Vim may be rial ever
published in ou
overkill for taking a few notes, but for r digital
library of back-i
anything else it’s superb. Check out the ssues
available exclus
video I made back in 2014 to learn to ively to
subscribers – tu
really love Vim: www.youtube.com/ rn to
page p56 to join
watch?v=rfl9KQb_HVk – and drop me Encryption 82 Elixir 88 .
a line and let me know if you become a Understand how the encryption Mihalis Tsoukalos gets you
convert too! that secures the web works. started with this language and
mike@linuxvoice.com John Lane introduces RSA. shows you some of its secrets.
www.linuxvoice.com 65
TUTORIAL STEGANOGRAPHY
HIDE CONFIDENTIAL DATA
WITH STEGANOGRAPHY
Hide encrypted data in the one place the spooks will never look for it – your photos.
E
BEN EVERARD ncryption is a great way to keep private In this tutorial we're going to look at steganography,
information secret; if used properly, there's no which is the process for hiding data inside media files.
Why do this? way for an attacker to break modern Typically, this is done with image files. The technique
• Guard against data encryption. However, sometimes we don't want an works by subtly changing some details of the image
theft. attacker to even know we have information that's in a way that's barely perceptible to the human eye. To
• Communicate encrypted. An extreme case for this is that an attacker the rest of the world, it looks like you have an
securely. could blackmail you into giving them the password, obsession with images of fluffy kittens, but really
• Combine your but it can be useful for any occasion when we simply you're safely guarding your secret family recipe that
backups with your
family portraits to want to keep our encryption private. makes the fluffiest Yorkshire puddings.
save space.
STEP BY STEP: CONCEAL ENCRYPTED DATA
1
Get the software 2
Get an image
There's a wide range of steganography software Any image should work, but some images work better
available. We've opted to use OpenStego, because it's than others. OpenStego can read most image formats,
written in Java so should work on most systems and but the output will always be in PNG because it has to
is easy to use. You can download the this software be in a lossless format – the type of compression in
from https://github.com/syvaidya/openstego/ JPEGs, for example, could destroy the data stored in
releases – there's a Deb file for Debian-based the image file. This formatting of the image can lead
systems or a Zip file for other Linuxes (and you should to some images being viewed with more suspicion
be able to run this on any system with Java). To run than others. Photographs are rarely stored as PNGs,
OpenStego from the Zip, just extract the contents and so an alert eavesdropper may be suspicious if they
run the openstego.sh script. There's also a Windows come across a large number of photos stored in PNG
release if you need to share your secrets with a format. Screenshots, on the other hand, are regularly
Windows machine. kept as PNGs, as JPEGs struggle with text.
You'll need to make sure you have Java installed. OpenStego will, by default, store three bits per colour
OpenStego runs on either Oracle's version of Java or channel per pixel, so you can store one byte per pixel
the OpenJDK. We'll also use the gpg command line (it's technically 9 bits, but the maths is easier if you
tool for encrypting data, and you should find this in use 1 byte, and you rarely need to fill an image). You
your distro's repositories if it's not installed by default. can split a large amount of data across several
images if needed.
66 www.linuxvoice.com
STEGANOGRAPHY TUTORIAL
3
Encrypt your data 4
Hide your secrets
Steganography hides your data within images, which Now you have your image and your encrypted file, it's
is good, but not perfect – if an adversary finds out time to combine the two into our steganographic
you've used steganography then there's a good secret. Start OpenStego and you will see the Hide Data
chance they'll be able to recover the data. OpenStego and Extract Data options on the left of the window.
does enable you to password-protect your data, but it Make sure Hide Data is selected, then enter the GPG
does this using the outdated DES encryption file in the Message File box, the image file in the Cover
algorithm. If you're going to the trouble of hiding your File box (here you can select multiple images if you've
data in images, the data is probably important enough got a large message file), and a name for the output
to be worth encrypting using the best available file. There's no need to enter a password, as we've
encryption, so we'll first secure it using the gpg already encrypted the files and an additional layer of
command. This is done with the following (enter a security won't add anything.
password when prompted): The original image file will remain untouched, and
gpg --cipher-algo AES256 -c <inputfile> the data will be hidden inside the file you created in
This will create a new file with the same name as the Output File option. You can open this up in an
inputfile, but with the additional suffix .gpg. It's this image viewer and it should look identical (or almost
encrypted file that you will use with OpenStego. identical) to the original file despite having a secret
message stored inside it.
5
Share your secrets 6
Decrypt your secrets
Now you have your secrets hidden inside an image, The final step is to extract the secrets from the image.
it's time to do something with them. What, exactly, this Just like hiding them, this is a two-stage process: first
step entails depends on what your secrets are and we have to get the encrypted data out of the image,
why you hid them inside a file. then we have to decrypt the data. Extracting the
If you just did it to ensure your data doesn't fall encrypted data is done using OpenStego. This time
victim to any hackers that get into your computer, you need to select Extract Data and enter the image
then the only thing to do here is get rid of the other file and the output file. You don't need to enter the
files containing those secrets. If you did this to enable password as we'll decrypt it next. Make sure the
you to share your secrets then you can now send output file has the .gpg suffix.
them to other people. If you post these pictures on a Once you've got the output as a GPG file, you can
social network such as Facebook, they'll be re- decrypt the data using the command
formatted and will probably lose the information gpg <encryptedfile>
stored within them. You have to share them as files GPG will automatically detect the cipher type and
rather than as images, so anything like email prompt you to enter the password. You should now
attachments or DropBox should work fine. have recovered your secrets from the image. As
You'll also need to make sure that anyone who you've seen, steganography is a simple way of adding
should be able to read them has the password. an extra layer of protection to your most valuable
secrets. Now go out and take loads of photos to mask
your secret snaps.
www.linuxvoice.com 67
TUTORIAL HUGO
SET UP YOUR NEXT
BLOG WITH HUGO
Generate hugely configurable static websites at lightning speed!
W
AMIT SAHA ith static website generators, we can write
the content of your site as formatted text
Why do this? files in our favourite text editor, convert
• Get content online, them to HTML files, copy those files to a web host and
fast, without having we're done. The number of static site generators
to wrestle with the available today is staggering; some of the most
complexities of a
CMS. popular ones are Jekyll, Octopress, Pelican, Nikola and
• It's another excuse to of course, Hugo.
mess around with Go! Hugo is written in Golang, so we first have to make
sure we have Golang installed and have the GOPATH
environment variable on your system. If you don't
have the go tools (compiler and other tools) installed,
you can either use the distro's package manager
to install them or download the Linux binary and
follow the instructions on the install page at https:// The site we will be building in this article – you
golang.org/doc/install. Once the installation steps can see it live at https://amitsaha.github.io/
are completed, open your favourite terminal emulator, linux_voice_4.
type go version and it should print a message similar
to below: github.com/fsnotify/fsnotify (download)
$ go version ..
go version go1.6 linux/amd64 ..
We next need to set up our Golang workspace. At this stage, we should have Hugo built and the
If you already have GOPATH set up, you may skip binary placed in the $GOPATH/bin/ sub-directory. You
ahead. Create a sub-directory golang in your home can print the version of Hugo using hugo version:
directory (/home/<user>) and two sub-directories, src $ $GOPATH/bin/hugo version
and bin inside it. The directory tree for your workspace Hugo Static Site Generator v0.16-DEV BuildDate:
should look as follows: 2016-04-27T18:29:22+10:00
$ tree -L 1 ~/golang If we execute $GOPATH/bin/hugo --help, it will print
bin a brief overview of the various flags and commands:
src $ $GOPATH/bin/hugo --help
hugo is the main command, used to build your Hugo site.
Today I Learned Hugo is a Fast and Flexible Static Site Generator
The go compiler and other tools expect the GOPATH built with love by spf13 and friends in Go.
environment variable to point to the workspace Complete documentation is available at http://gohugo.io/.
directory, so set the following to your .bashrc or the Usage:
file relevant to your shell, so that it is always set when hugo [flags]
you start a new terminal session (Replace <user> with hugo [command]
your username): ..
export GOPATH=/home/<user>/golang As the help message shows, Hugo's functionality
Once you have set the above, start a new terminal is available via various subcommands and flags. For
session and type go env GOPATH: example, the hugo new site command will be used
$ go env GOPATH to create a new site and hugo server will start a local
/home/<user>/golang server to serve our site's content. Appending --help
Next, we will get the source for Hugo and build it: to to a Hugo sub-command (eg $GOPATH/bin/hugo
$ go get -v github.com/spf13/hugo new --help) will display help message for the sub-
github.com/spf13/hugo (download) command as well.
68 www.linuxvoice.com
HUGO TUTORIAL
We will create a site called "Today I Learned". The Posts with categories –
content of the site, including the posts, pages and the generated in double-quick
configuration, will live in a single sub-directory in our time..
filesystem. Let's say we want to create our site in a
subdirectory today-i-learned; we'll use the hugo new
site today-i-learned command. This will create a new
sub-directory today-i-learned in the directory you
executed the command:
$ $GOPATH/bin/hugo new site today-i-learned
Congratulations! Your new Hugo site is created in "/
home/amit/today-i-learned".
Just a few more steps and you're ready to go: Your rendered home page is blank: /index.html is
1. Download a theme into the same-named folder. Choose zero-length
a theme from https://themes.gohugo.io or * Did you specify a theme on the command-line or in
create your own with the "hugo new theme your
<THEMENAME>" command "config.toml" file? (Current theme: "")
2. Perhaps you want to add some content. You can add * For more debugging information, run "hugo -v"
single files with "hugo new <SECTIONNAME>/<FILENAM =================================================
E>.<FORMAT>" ============
3. Start the built-in live server via "hugo server" 0 draft content
For more information read the documentation at https:// 0 future content
gohugo.io. 0 pages created
At this stage, Hugo has created a scaffold for our site: 0 non-page files copied
$ tree today-i-learned 0 paginator pages created
. 0 tags created
archetypes 0 categories created
config.toml in 32 ms
content Watching for changes in /home/amit/today-i-learned/
data {data,content,layouts,static}
layouts Serving pages from memory
static Web Server is available at http://localhost:1313/ (bind
themes address 127.0.0.1)
The config.toml file is our site configuration Press Ctrl+C to stop
formatted as a TOML file. The subdirectories created A web server has been started for us and you
above are all empty and they serve different purposes: can visit the URL from your browser, but you will be
archetypes When we create new content, Hugo greeted with a blank page – which of course isn't
pre-fills the content's front matter with metadata surprising. Keep the server running, switch to a new
such as title and date. Using archetypes, we can terminal window to create our first blog post:
customise the metadata we want to be pre-filled. $ cd today-i-learned/
content This is where all the content will live $ $GOPATH/bin/hugo new post/hello-world.md
data This directory can be used to load custom data /home/amit/today-i-learned/content/post/hello-world.
from a YAML-, JSON- or TOML-formatted file and md created
make it available at site-generation time. You can A new subdirectory, post, has been created under the
think of it as like a file based data store for your site. content directory with a file hello-world.md under it:
layouts This is where the layout of the site can be $ tree content
customised using templates. content
static This is where, we should place any custom post
CSS, JavaScript, images or any files we want to be a hello-world.md
part of our site. At this stage, it's worth discussing the concept of
themes We will store our theme here. "sections" in Hugo. Hugo lets you organise your site's
At this stage we have our site structure ready, but content into any structure you please. So, for example
we are missing content and a theme. As important as (as we will in this article) we can categorise our site's
they are, let's ignore them for the time being and go content into being a "post" and "page", or you could
ahead and start a server to serve our site: just call them something else entirely.
$ cd today-i-learned/ We can now write our blog post content in the
$ $GOPATH/bin/hugo server hello-world.md file. It currently will contain the
Started building site following:
================================================= +++
============ date = "2016-05-12T08:05:42+10:00"
www.linuxvoice.com 69
TUTORIAL HUGO
post
hello-world.md
data
layouts
static
themes
hyde-x
LICENSE
README.md
archetypes
images
layouts
static
theme.toml
Our next step is to tell Hugo that we want to use
the "hyde-x" theme. We do so by changing our config.
Figure 4: Syntax
draft = true toml to be the following:
highlighting code
demonstration. title = "hello world" baseurl = "http://replace-this-with-your-hugo-site.com/"
+++ languageCode = "en-us"
The content within the +++ is referred to as the title = "Today I Learned"
front matter, and is the metadata. Open the file in your theme = "hyde-x"
favourite text editor and change the content to the Note that we also changed the title of our site
following: and specified the theme. Now, if we go back to the
+++ terminal where we left the "hugo server" running, we
date = "2016-05-12T08:05:42+10:00" will see messages like:
title = "hello world" Change detected, rebuilding site
+++ 2016-05-12 08:16 +1000
Hello World! This is the first blog post in my brand Config file changed: /home/amit/today-i-learned/config.
new blog "Today I Learned". I will be using this blog as toml
a place to post my notes on everyday things I learn. Now, if we go back to http://localhost:1313 on your
We have removed the draft = true line from browser, you will see your site with your first post.
the header so that our post will be available for If you click on the post title, you are led to the entire
consumption via our site. We now have some content post which is available at http://localhost:1313/post/
for our site, but we don't have a theme yet and so hello-world. If you now look at your site's directory,
we still can't view our site content. http://themes. you will see no traces of HTML files anywhere. This is
gohugo.io is a showcase of a number of Hugo because Hugo serves your pages from memory.
themes. The choice of theme will determine how Let's add a new post to our blog while keeping our
your site's content is structured and how it looks. server running:
In addition, a theme may have additional features $ pwd
already available. For our site, we will use the "hyde-x" /home/amit/today-i-learned
theme (https://github.com/zyro/hyde-x). It looks $ GOPATH/bin/hugo new --editor emacs post/hugo-live-
nice and makes adding social links really easy. Instead reload.md
of cloning the Git repository for the theme, we will Passing the option --editor followed by the path
download a Zip archive of the theme from to your editor will open your editor with the new blog
https://github.com/zyro/hyde-x/archive/master. post front-matter pre-filled in and ready for you to type
zip, unzip it and place it in the themes sub-directory in your post. Note how the front-matter has changed
created for us: from our first post? This is because the hyde-x theme
PRO TIP $ cd themes defines an archetype in its archetypes subdirectory,
The [params] section in the config.toml $ wget https://github.com/zyro/hyde-x/ which overrides the default archetype. We will type in
file are made available by Hugo to be used archive/master.zip a couple of sentences:
in templates. In our case the templates $ unzip master.zip Hugo has a live reload feature, which means the
that will be used to render our homepage
are part of the theme we are using and $ mv hyde-x-master hyde-x moment we save this page, not only our my site
hence any theme specific parameters will $ rm master.zip content regenerated, but the browser also reloads the
be specified int he "params" section. At this stage, our site structure looks site. This means, we can see our new post without
like this: needing to refresh the site.
$ tree -L 3 As I write in the blog post above, you will see
. that the site you had opened on your browser has
archetypes automatically been live reloaded. This is a unique
config.toml feature of Hugo, which it achieves by utilising web
content sockets, and is certainly great when you're working on
70 www.linuxvoice.com
HUGO TUTORIAL
Figure 3: Side bar showing
Resources the social links and RSS
Feed
• Hugo showcase https://gohugo.io/showcase
• Archetypes https://gohugo.io/content/archetypes/
• Sections https://gohugo.io/content/sections/
• Configuration https://gohugo.io/overview/configuration/
• Shortcodes https://gohugo.io/extras/shortcodes/
• Syntax Highlighting https://gohugo.io/extras/highlighting
• Templates https://gohugo.io/templates/homepage/
• hyde-x theme https://github.com/zyro/hyde-x
• Datafileshttps://gohugo.io/extras/datafiles#the-data-
folder
• Static sites with docker http://ilkka.io/blog/static-sites-
with-docker
• Automated site deployments https://gohugo.io/tutorials/
automated-deployments
• Hugo tools https://gohugo.io/tools file so that it looks as follows:
baseurl = "http://replace-this-with-your-hugo-site.com/"
languageCode = "en-us"
your site's content. To disable the live reload, we can title = "Today I Learned"
pass the --disableLiveReload option to "hugo server". theme = "hyde-x"
[[menu.main]]
Adding pages to our blog name = "About"
To add a new page to our blog, we will once again use url = "/page/about/"
the "hugo new" command. From within the today-i- weight = 2
learned directory, execute the following command, Now, we will see the link to the About page on your
which will once again open a specified editor to edit homepage. The section [[menu.main]] indicates that
the page we want to create: we are adding this menu to the main menu, has the
$ $GOPATH/bin/hugo new --editor emacs page/about.md name "About", which is the text we see, the URL it
Type in something that you would want to be in points to and its weight. Weight decides the order of
the page, save the file and exit. You will see that a file the menu items.
about.md has been created in the page subdirectory
under the content directory. Categories and tags
$ tree content/ With Hugo you can classify your site's content into
content/ categories and tags. It refers to these as "taxonomies".
page We first have to define them in the site's configuration
about.md before we can classify our content using these. Here's
post our config.toml after defining the categories and tags:
hello-world.md baseurl = "http://replace-this-with-your-hugo-site.com/"
hugo-live-reload.md languageCode = "en-us"
On the terminal you have "hugo server" running, you title = "Today I Learned"
will see messages such as: theme = "hyde-x"
adding created directory to watchlist /home/amit/ [[menu.main]]
today-i-learned/content/page name = "About"
Change detected, rebuilding site url = "/page/about/"
2016-05-13 07:58 +1000 weight = 2
0 draft content [taxonomies]
0 future content tag = "tags"
2 pages created category = "categories"
1 non-page files copied The "[taxonomies]" section in the site configuration
1 paginator pages created defines the tag and category for our site.
0 tags created Now, we can add categories and tags to our
0 categories created existing posts. The first post will now look like this:
in 7 ms $ cat content/post/hello-world.md
Change detected, rebuilding site +++
2016-05-13 07:58 +1000 date = "2016-05-12T08:05:42+10:00"
However, if you go to your browser window, you title = "hello world"
will not see a link to the page that we just added. The categories = ["updates"]
reason is that menus in Hugo have to be explicitly +++
configured. hyde-x already shows us a link to the site Hello World! This is the first blog post in my brand
home, but we have to do some work to get new pages new blog "Today I Learned". I will be using this blog as
to be visible in the menu. We will edit our config.toml a place to post my notes on everyday things I learn.
www.linuxvoice.com 71
TUTORIAL HUGO
Figure 5: Demonstration of options when it comes to syntax highlighting - server
using the "gist" shortcut to side and client side.
embed a GitHub gist This is an example of client side highlighting. You can
read all about it
[here](http://gohugo.io/extras/highlighting/).
~~~python
# A line of Python
print('Hello world')
~~~
~~~go
// A simple program in Golang
The second post is updated to the following: package main
+++ import (
date = "2016-05-12T09:20:03+10:00" "fmt"
title = "Hugo Live Reload" )
categories = ["blogging", "hugo"] func main() {
tags = ["golang"] fmt.Println("Hello world")
+++ }
Hugo has a live reload feature, which means the ~~~
moment I save this page, not only is my site content In addition, we will have to select a syntax
regenerated, but the browser also reloads the site. highlighting scheme using the highlight key in the
This means, I can see my new post without needing params section:
to refresh the site myself. highlight = "zenburn"
Now, we can see that our posts have categories We will see that the post has syntax highlighted
assigned (Figure 2), and if you click on a category code (Figure 4). There are various highlighting
label, you will find all posts in that category. schemes available with hyde-x, which you can see in
the themes/hyde-x/static/css/highlight sub-directory.
Social, RSS feeds and other customisation
The "hyde-x" theme makes it easy to add links to your Shortcodes
social profiles such as your GitHub and Twitter Shortcodes in Hugo enable you to do common things
profiles. To do so, we will add a new section "params" which Markdown doesn't allow (Figure 5). Their
to our site configuration and add the following: syntax is usually {{< short-code parameter1
[params] parameter2 >}}. For example, we link to another post
github = "https://github.com/amitsaha" or page in our blog, we will use the ref shortcode:
twitter = "https://twitter.com/echorand" [post]({{< ref "post/hello-world.md" >}})
rss = true [page]({{< ref "page/about.md" >}})
You will see that there are links to the specified There are various other useful shortcodes
GitHub, Twitter profiles and the RSS feed for your blog – for example to embed a GitHub gist with ID
contents (Figure 3). The theme also has support for 9864ec0475dd9b68c4a38be37726e552 we will use
various other social profiles. The RSS feed is for the the gist shortcode:
entire site; if you want to refer to category-specific {{< gist amitsaha 9864ec0475dd9b68c4a38be37726e552
feeds, you can find them at http://localhost:1313/ >}}
categories/<category>/index.xml. The first parameter to the gist shortcode is the GitHub
There's a good chance you will have code in your username, and the second parameter is the Gist ID.
blog posts, and you want it to be syntax highlighted.
Hugo has support for two kinds of syntax highlighting Hosting your content on GitHub pages
– server side or render time (using Pygments) and A site is perhaps never done, but I think we are at a
client-side using JavaScript. We will see an example of point where we are ready to go live. We will deploy our
the latter. Let's create a new post with this content: static site using GitHub pages, because it's free and
$ cat content/post/syntax-highlighting-code.md easy to set up. GitHub pages allow two kinds of sites
+++ – one of the form <your-username>.github.io or
categories = ["golang", "code", "python"] <your-username>.github.io/<repo_name>. The first
date = "2016-05-13T17:02:38+10:00" step is to create a repository on GitHub named
description = "" linux_voice_4 and create a branch gh-pages from the
keywords = [] repository page. Next, create a local clone of the
title = "Syntax highlighting code" repository:
+++ $ git clone git@github.com:amitsaha/linux_voice_4.git
I learned how to post syntax highlighted code in a post. $ cd linux_voice_4
Hugo has two $ git branch
72 www.linuxvoice.com
HUGO TUTORIAL
Figure 6: Hugo comes with
master in-built Disqus integration
We will use the master branch of the repository to and the hyde-x theme lists
keep a copy of our site's "source", and the gh-pages the number of comments
branch will have only the generated files. Our next against each post.
step is to simply move the entire directory tree of
today-i-learned to the linux_voice_4 subdirectory so
that it looks like this:
$ tree -L 2 linux_voice_4
linux_voice_4
LICENSE
README.md
archetypes We'll get back our master branch and add the
config.toml above. The config.toml file now looks like this:
content baseurl = "http://<your-github-username>.github.io/
page linux_voice_4/"
post languageCode = "en-us"
data title = "Today I Learned"
layouts theme = "hyde-x
static disqusShortname = "hugo-todayilearned"
themes googleAnalytics = "UA-77766553-1"
hyde-x [[menu.main]]
It's time to now modify our config.toml file to add name = "About"
our base URL: url = "page/about/"
baseurl = "http://<your-github-username>.github.io/ weight = 2
linux_voice_4/" [taxonomies]
Our next step is to generate the HTML files for our tag = "tags"
content and add everything to the master branch: category = "categories"
$ $GOPATH/bin/hugo [params]
$ git add -A . highlight = "zenburn"
$ git commit -m "Initial version" home = "Home"
At this stage, we have our generated site in the github = "https://github.com/amitsaha"
public subdirectory. We now want to move the twitter = "https://twitter.com/echorand"
contents of that directory to our gh-pages branch. We rss = true
will do it using a straightforward but admittedly naive Once we re-publish the site by building the site from
approach – we will copy the contents to a directory master and pushing to our gh-pages branch, the
/tmp/hugo_public, checkout the gh-pages branch posts should have the Disqus commenting system
and copy the content from /tmp/hugo_public, then and Google Analytics enabled on all the pages.
commit everything to the gh-pages branch and finally
push both the branches: Conclusion
$ cp -r public/ /tmp/hugo_public As I explored Hugo while working on the article, I was
$ git checkout gh-pages initially overwhelmed by the features and the need to
$ cp -r /tmp/hugo_public/* . configure even the smallest of things. However, I have
$ git add -A . no hesitation in saying that Hugo stands out with its
$ git commit -m "New build" huge number of built-in features, and the enormous
$ git push origin master gh-pages configurability is a good thing. It includes sensible
If you now visit https://<your-github-user-name>. features by default and puts the content creator in
github.io/linux_voice_4, you should see your site's control of how the site should be written, structured
content. Now that our site is live, we want to integrate and appear.
Google Analytics with it for tracking and Disqus to add You can find the source for the "Today I Learned"
the ability to comment on our blog posts (Figure 6). site at https://github.com/amitsaha/linux_voice_4
in addition to a set of resources to explore next. The
Google Analytics and Disqus integration copy of hyde-x in my Git repository is slightly changed
To add Google Analytics tracking to our page, we to create the links correctly. I will investigate it further
create a property on Google Analytics, get our tracking and post updates here: https://github.com/spf13/
ID and add it as googleAnalytics = "UA-77766553-1". hugo/issues/2147.
To add Disqus integration to our site, we have to first
create an account on Disqus, get the shortname for
Amit Saha is the author of Doing Math with Python (No Starch Press)
this site and simply add disqusShortname = "<you- and a software engineer. He blogs at https://echorand.me, tweets
short-name>" in the config.toml file. @echorand and can be reached via email at amitsaha.in@gmail.com
www.linuxvoice.com 73
TUTORIAL PUBLISHING
MAKE EBOOKS FROM
SCRATCH WITH FOSS
Birth your literature upon the world the best way possible – the Free way!
I
ANDREW CONWAY t was on a dark and stormy night when, suddenly, your work using a web browser and simulate
the velvet drapes shimmered, the candle flickered something close to what might appear on an eReader,
Why do this? and you were overcome with a desire to write a app or even in the print version.
• Write an eBook book. But how best to do it? You could use LibreOffice The third and most important advantage of HTML
with simplicity and Writer, or try Latex, but perhaps markdown and HTML and CSS is that it's very easy to transform it into
control is an option. Then there's the question of how to almost any format. In fact, one of the most common
• Write with FOSS tools publish: you like those old, dusty tomes piled in the electronic formats – the EPUB – is a Zip file that
on Linux
corner of the study, but then the Raven quoth includes XHTML and CSS files. Our aim in this article
• Prepare for any
format: eBook, web "Nevermore!", so an eBook it will be. But how to is to build an EPUB manually.
or print produce a book for these new fangled eReaders? You could use LibreOffice Writer to write and format
Read on, but hurry! lest the Cthulhu devour you before your text and then export HTML and CSS, but that
you commit your thoughts to ePrint. way you will sacrifice both simplicity and control. You
It is now the norm for documents to be stored in at do not need to be a web designer to use the HTML
least two distinct parts: one contains the content, and CSS needed to make a book; it's much simpler
such as the words in this article; and separate from than putting together a website.
that is information on style, such as what font to use
for the headings. The most obvious example of this is Markdown
the use of HTML for web content, and CSS to tell a You can write HTML directly. However, after a while
web browser how the content should be displayed. you'll start to learn why Markdown was invented.
We're going to use HTML and CSS as the basis for Opening every paragraph with <p> and closing it again
writing our book for a few reasons. First, it means you with </p> gets irritating after you've done it a hundred
can use any text editor to write your content, and you times, and it's prone to error as it's almost inevitable
can keep style at arm's length while you toil over your that you'll forget to close such tags at some point.
choice of words and sculpt your prose to perfection. You're spoilt for choice when it comes to converting
Also, because proofreading is usefully done using Markdown to HTML. I use the Python markdown_py;
something closer to the final format, you can preview snother good choice would be Pandoc, which makes
Markdown quick start
Markdown has become a favourite format for HTML, which can look quite jumble. browser (Firefox). If Markdown doesn't support the
coders who tire of having to close every tag of The screenshot below gives you a quick formatting you want, you can always use HTML
HTML. It was created in 2004 by John Gruber with introduction to the basics of Markdown. On the tags. See Marco Fioretti's excellent introduction to
help from Aaron Swartz. In addition to being easy left is the markdown in a text editor (Kate), and on Markdown in Linux Voice issue 10 for more detail
to write, it's designed to be readable as-is, unlike the left is HTML generated from it viewed in a web and also a list of handy cheatsheets on its syntax.
74 www.linuxvoice.com
PUBLISHING TUTORIAL
a worthy claim to being the Swiss Army knife of text about this via a podcast associated with a computing
format converters. magazine, Minix Vocals I think it was.) First install it if
There is however one drawback with Markdown: you haven't got it:
there is no standard. Over the years different sudo apt-get install entr
implementations have created different syntaxes. Now enter the following in a terminal:
The first problem this creates is that a web search ls test.markdown | entr markdown_py test.markdown -f
on syntax might throw up results that aren't correct test.html
for the markdown implementation you are using. The This says: keep an eye on the file test.markdown; if
second problem is that Markdown has portability it changes, run the command after entr which, in our
issues: a file put through two different Markdown case, will create the test.html file. Now make an edit
converters might produce different results. to test.markdown, save it and then reload the browser
The best to way to avoid these problems is to and you'll see your edits are rendered.
stick with one Markdown converter and keep your But we're only half way to our goal. We want the
formatting as simple as possible. If you do encounter browser to reload automatically when we save. There
any peculiarities of the Markdown converter you're
using, you can keep your document portable by using
HTML tags instead of the offending syntax.
You can write HTML directly. However,
after a while you'll start to learn why
From laziness comes efficiency
So let's get writing. Enter some Markdown into a text Markdown was invented
editor. Now save it as test.markdown (you can use
.md if you prefer, but older text editors may think are two ways to do this. The simplest method is to
you're writing Modula-2 code and highlight it use Midori. No, not the alcoholic liqueur of green hue,
incorrectly) and generate the HTML with this on the but the Midori web browser, which you will likely find
command line in your distro's repositories, though it is included in
markdown_py test.markdown some distributions, notably Raspbian. It has the handy
You'll then see the HTML produced as output to the feature that you can instruct it to reload its current tab
terminal. We'd rather it went to a file, so do this from the command line. To get this working save the
markdown_py test.markdown -f test.html following text into a file called refresh.sh:
The -f option tells markdown_py to send its output markdown_py test.markdown -f test.html
into the file test.html. You could use the > symbol to midori -e Reload
send the standard output to a file, but this will cause Before running this script, open up test.html in
problems for something we'll want to do later. Midori manually.
Now open the file test.html. You can do this via We can now use entr to run the Bash script
your web browser's menu, though as browsers are refresh.sh whenever we change test.markdown:
increasingly intent on hiding menus from you, you ls test.markdown | entr sh refresh.sh
might want to type something like this into the URL Try editing the Markdown file and, when you save
bar file:///home/jim/somedir/test.html – where jim it, you should see what's shown in Midori update
is the username and somedir is the directory with accordingly.
test.html in it. Alternatively you could try dragging If you'd rather not use Midori, perhaps because you
test.html from your file manager and dropping it onto don't want to install it for this one purpose, then you
your browser. can achieve the same thing with almost any web
You'll now see your masterpiece rendered with browser. To do this, go to entr's website entrproject.
formatting. It's a bit ugly – we'll come to styling CSS org, download its reload-browser script and use
later. For now we wish to concentrate on content. it in place of the midori -e Reload command. The
Make some edit to test.markdown, save it, run script works by faking a Ctrl+R press in your browser
markdown_py on it again and then hit reload on your window using the xdotool command. Bear in mind
browser to see the results. that it doesn't work in all window managers, and
Now imagine doing that several hundred times a you might need to install xdotool from your distro's
day. And count the number of times your hand goes repositories.
from keyboard to mouse and back again. Yes, you can One reason we use Midori is so we can dedicate the
use a few keyboard shortcuts, or even avoid a mouse browser to this one task and keep it separate from all
completely with a tiling window manager, but wouldn't other web browsing. If you use your main browser for
it be nice to just hit Save and see the result appear the job and leave it on the wrong tab and save your
instantly in your web browser? Let's go WYSIWYG. Markdown file, then the reload-browser script will
Not so much What You *See* Is What You Get but merrily reload that tab. The result may be harmless, or
What You *Save* Is What You Get. it might be irritating, but it might cause a more serious
We're going to save our wrists from repetitive strain problem if you're half way through entering a long
injury by using the nifty command entr. (I first heard form with important information like bank details.
www.linuxvoice.com 75
TUTORIAL PUBLISHING
<body>
Anatomy of an EPUB <h1 class="centre">My Book</h1>
An EPUB is a Zip file with a prescribed set <p class="centre">A.N. Author</p>
of files and directories, as shown here. The <p class="centre">Copyright 2016 A.N. Author </p>
mimetype must be the first file in the archive This contains the tags needed at the start of a valid
and must not be compressed. This enables XHTML file, which we'll need for the EPUB later, and
applications to quickly determine that the
file is an EPUB. Most of the files in the EPUB
it specifies the character encoding as UTF-8 and the
are XML files, with exceptions being the CSS CSS file as mybook.css in which the centre class
file and content files such as images, audio we use in head.html is defined. Then the body of the
or video. The main content of the book is HTML starts and a basic title page is created with a
supplied in one or more XHTML files using title, author and copyright information.
HTML 5 syntax. You can have one file for an
entire book, but it's more usual to split it into a
The secret of writing the CSS file for an e-document
file per chapter, or even per section. is to keep it simple. Here's a CSS file that's all you need
The container.xml's main purpose is to for simple but decent eBook:
point to content.opf (though it can be given body {margin: 3em 3em 3em 3em;}
any name) which contains a manifest of all h2 {page-break-before: always;}
files in the EPUB, along with the order in which
h3 {page-break-after: avoid;}
the main parts of the book are meant to be read, and references to navigation elements. In the
current EPUB 3 format, navigation menus in applications and eReaders will use a file that's .centre {text-align: center;}
usually called toc.xhtml. This is a change from EPUB 2, which used an XML .ncx file to specify p {text-indent: 1em;}
the navigation structure. As many eReaders and applications still use EPUB 2, it's still a good First notice that all sizes are in units of em, where
idea to include an NCX .ncx, but an EPUB 3 reader will ignore it and expect to find toc.xhtml. 1em is equal to the font size, and 3em means three
The full EPUB specification can be found at www.idpf.org/epub.
times the font size. Using pixel size px would be
very naughty in this context and is likely to cause
unexpected weirdness on some eReaders and
Now let's pull together all the bits and pieces we unpredictable results when printed. Next, we say that
need to make a book. There's not space to include all there should always be a page break before the h2
the lines of XML needed in this article so we omitted style, because it will be used for chapter headings.
the most boring ones, but you can find complete For headings within chapters we use h3, and we want
versions of all files mentioned at github.com/mcnalu/ to avoid breaks after them if possible. This will avoid
linuxvoice-publishing. First, we need to augment our having headings near the bottom of a page. These
script so that we create a proper HTML document page-break styles will have no effect in a web browser,
with CSS styling, and also handle multiple chapters. but will in an eReader or if you print the HTML from
Here we've just included two to demonstrate the your browser.
principle: chapter1.markdown and chapter2. The .centre defines a class that was used in head.
markdown. Create a text file called make_book.sh html to centre text on the title page. The text-indent
containing this: for p specifies that each paragraph should start with a
#Insert beginning of HTML file and timestamp small indent, as is conventional in print and eBooks.
cat head.html > mybook.html We can generate a new preview of our book when
echo "<p class="centre">Generated: " >> mybook.html any part of it is saved using the entr command:
date >> mybook.html ls *.markdown mybook.css | entr sh make_book.sh
echo "</p>" >> mybook.html This triggers a rebuild of the book if any of the
#Create one big markdown file markdown files change or the CSS file changes.
echo "##Contents" > mybook.markdown At this point, assuming you've written some text for
echo "[TOC]" >> mybook.markdown chapters 1 and 2, you will have a book that you can
cat chapter1.markdown >> mybook.markdown view in a web browser. Next we'll show how to wrap
cat chapter2.markdown >> mybook.markdown this up in a valid eBook format.
#Create the html with a table of contents
markdown_py -x toc mybook.markdown >> mybook.html Prepare to publish
#Finish off the HTML file Of the very many eReader formats, probably the most
echo "</body></html>" >> mybook.html widely used is EPUB, though arguably the dominance
midori -e Reload of Amazon's Kindle means that many people are
The head.html should look something like this: using the AZW format without knowing it. AZW is
<?xml version="1.0" encoding="utf-8"?> based on the older MOBI format, but since 2011
<html xmlns="http://www.w3.org/1999/xhtml" Amazon has been using its newer Kindle Format 8
xmlns:epub="http://www.idpf.org/2007/ops"> (.fd8 or .azw3), which is based on HTML 5 and CSS.
<head> We'll concentrate on how to manually create a
<title>My Book</title> version 3 EPUB. Have a look at the boxout on the
<link rel="stylesheet" type="text/css" href="mybook. anatomy of EPUB for an overview before reading on.
css"/> First, in an empty directory, create the basic
</head> structure of what needs to go into the EPUB file:
76 www.linuxvoice.com
PUBLISHING TUTORIAL
echo -n "application/epub+zip" > mimetype used to enable eReaders to offer a navigation menu
mkdir META-INF OEBPS no matter where you are in the book. For an eBook,
cp /some/path/mybook.html OEBPS/mybook.xhtml there's probably no reason to have both, but it does no
cp /some/path/mybook.css OEBPS harm to leave it in and most eReaders will respond to
Notice we've used -n, which tells echo not to add the links as expected.
a newline character. This is important because the Now that all the files are in place we can create the
mimetype file must not have more than one line. Also, EPUB file as follows:
notice that we renamed the mybook.html to mybook. zip -0X mybook.epub mimetype
xhtml, which is conventional in EPUB 3. zip -Xr9D mybook.epub META-INF/ OEBPS/
Now create a file called container.xml inside the The first zip command is performed separately
META-INF directory that contains this: to ensure that mimetype is the first file in the Zip
<?xml version="1.0" encoding="UTF-8" ?> archive, and the -0 option (that's a zero) tells Zip
<container version="1.0" xmlns="urn:oasis:names:tc:ope not to compress that file. This is important so that
ndocument:xmlns:container"> applications can easily find and read the mimetype.
<rootfiles> The second command compresses and stores all
<rootfile full-path="OEBPS/content.opf" media- other directories and files in the archive. The X stops
type="application/oebps-package+xml"/> extra file attributes being included in the archive; r
</rootfiles> tells it to include files recursively in sub-directories; 9
</container> means maximum compression and D omits separate
The main task of container.xml is to point to the entries for directories.
content.opf file. This is also an XML file, and its You can now open the EPUB in any application
important lines are: and check that it's displaying correctly. If you want to
<manifest> check you've structured your EPUB correctly, you can
<item id="toc" properties="nav" href="toc.xhtml" download the EpubCheck tool from https://github.
media-type="application/xhtml+xml" /> com/IDPF/epubcheck.
<item id="template_css" href="mybook.css" media-
type="text/css" /> Epilogue
<item id="mybook" href="mybook.xhtml" media- Hopefully this has given you an understanding of how
type="application/xhtml+xml" /> an electronic document is structured and that by
</manifest> using Markdown and keeping a light touch with CSS
<spine> styling, you can keep your writing process simple
<itemref idref="mybook" /> while staying in control of the finished product. That
</spine> said, the process of creating an EPUB manually is
The manifest lists the contents of the EPUB file: rather fiddly. You can automate it further by writing
the table of contents (toc), the CSS file and the main your own scripts to copy files into the directory
content file. The spine is used to list the linear reading structure and zip up the EPUB, but another option is to
order of the document, though in our case we've use the Calibre application to generate the EPUB. We'll
opted for only one big XHTML file. If we had one describe Calibre and various ways you can publish an
.xhtml file per chapter, the spine section would need eBook and a print version in a follow-on article.
an itemref for each one.
The final file we'll need is toc.xhtml. The important
A real example
bit of this is this:
<nav id="toc" epub:type="toc"> The techniques in this article are not just
some theoretical musings, but those I have
<h1 class="frontmatter">Table of Contents</h1>
used to produce a real book called An Active
<ol class="contents"> Citizen's Guide to Scotland (see activecitizen.
<li><a href="mybook.xhtml#contents">Contents</ scot if you're interested). All the text for
a></li> this book was written in Markdown in KDE's
<li><a href="mybook.xhtml#chapter-1">Chapter 1</ Kate text editor, and tables and figures were
mostly generated using LibreOffice Calc with
a></li>
a few being made with Gnuplot. Although the
<li><a href="mybook.xhtml#chapter-2">Chapter Bash script used to construct the book was a
2</a></li> little more complex than the one shown here,
</ol> it is still pretty short at 43 lines.
</nav> Some publishers may insist on their
authors providing a particular format,
You can save yourself some time by looking too often .docx, but, if you make it as
inside mybook.xhtml for the table of contents that a bankable, best-selling author like JK
markdown_py generated. Those <li> items can be Rowling, you could, if you so wished, submit
copy and pasted into toc.xhtml, although you'll have your manuscript scrawled by hand in purple ink on the back of a thousand fag packets.
to add in mybook.xhtml before the # in each href link.
The toc.xhtml is separate to the table of contents Andrew Conway watches the stars from his wood-panelled study.
we've already put at the start of the book and will be He likes open data and what you can do with it using Free Software.
www.linuxvoice.com 77
TUTORIAL LINUX FOR LEARNERS
RASPBERRY PI DISPLAY
DATA PHYSICALLY
Turn input from the internet into something you can hold in your hand.
S
LES POUNDER ummer time is upon us in the northern
hemisphere, and the weather is always a keenly
Why do this? discussed topic of conversation. Is it warm
• Make the most out of enough to get the barbecue out? Being the chief
GPIO Zero question. Using a little GPIO Zero code, a stepper
• Control a stepper motor and LEDS plus a Python weather module, we
motor can build ourselves a cool weather frame that will
• Work with external keep us up to date. There are two parts to the
data sources
hardware build. First we have the components, chiefly
our stepper motor and eight LEDs. Our stepper motor
has a driver board using a ULN2003 chip. The board
requires 5V of power from our Raspberry Pi and a
You will need Ground (GND) connection. We can then connect the Our completed project sits inside its picture frame home
• Any model of other “input” pins of the board to our Raspberry Pi. We ready to inform us of the next change in weather.
Raspberry Pi can also attach the LEDs to the corresponding pins of
• Wi-Fi connection our Raspberry Pi via a 220Ω resistor. We start by opening a Terminal window, the icon for
• Stepper motor The second part of the build is the physical frame. which is found in the top-left of the screen and looks
• LEDs We chose to use a cheap picture frame and replaced like a black screen.
• 220Ω resistor the glass with a piece of cardboard that fit into place Our first task is to install the Python library for
(RED-RED-BROWN) with a gentle push. We then stuck some coloured OpenWeatherMap. In the terminal type the following
• Breadboard card to the cardboard to hide the backing. Next we and press Enter.
• Male–female measure out a dial using an old CD and then split $ sudo pip3 install pyowm
jumper wire
the circumference of the dial into segments for our With installation complete we now need to sign up
• Female–female
jumper wire temperature range. To attach the stepper motor to to the OpenWeatherMap service via its website. In a
• Power for your the cardboard we used some machine screws and browser visit https://home.openweathermap.org/
Raspberry Pi nuts. But to attach the dial to our stepper we used a users/sign_up and create a new account. This will
lollipop stick into which we cut a notch that matched generate an API key, which will enable our project to
the stepper motor spindle. By attaching the lollipop request weather data. Keep this key handy, as we will
stick to our dial using tape and then sliding on to the need it later in the project. Also note that this key is
The OpenWeatherMap
website has a great suite spindle, we now have a reliable method of precise private and linked to your account, so do not share
of tools for you to research rotational movement. We then attached another dial your API key.
historical and forecast to the frame, but this time with no stepper motor. In
future weather. this dial we inserted 8 LEDs. Coding the project
We start coding the project by opening the Python 3
application, found in the Programming menu. With
Python 3 open click on File > File New to open a new
blank document. Immediately save your work as
Weather_Frame.py. This will enable quick saving as
we work through the code.
Our first section of Python code is a series of
imports. First we import two classes from GPIO
Zero, namely LED to handle our LED indicators, and
OutputDevice, a class enables us to directly control
GPIO pins.
from gpiozero import LED, OutputDevice
import pyowm
78 www.linuxvoice.com
LINUX FOR LEARNERS TUTORIAL
import time
We next create variables that will be used to
identify the LEDs for the different weather statuses
that we wish to identify. The GPIO Zero LED class is
remarkably easy to use and requires little code for
configuration.
Sunny = LED(17)
Bluesky = LED(27)
Snow = LED(22)
Cloudy = LED(10)
Thunder = LED(9)
Shower = LED(11)
Rain = LED(5)
Fog = LED(6)
Next we use another class, OutputDevice, from
GPIO Zero. The class enables control of any GPIO pin.
Stepper motors normally
Here we use it to create four pins that will be used to Inside this function we have an if condition that
come as just a motor,
control our stepper motor. checks the name of the stepper motor to be used. In but we picked these up
Temp_IN1 = OutputDevice(23) this case it is Temp for our temperature gauge. This cheaply from eBay, which
Temp_IN2 = OutputDevice(24) condition is indented to show that it is part of the come with their own
Temp_IN3 = OutputDevice(25) function. controller board.
Temp_IN4 = OutputDevice(8) if stepper == "Temp":
Finally we create a new variable called delay that is Next we create another indentation, which we use
used to control the stepper motor speed. This is the to construct a for loop. This loop will pulse the GPIO
optimum time to run the stepper smoothly. pins connected to our stepper motor in the correct
delay = 0.01 sequence to drive the stepper clockwise. At each time
We now create a function that will control our one pin is turned on while the rest are off, causing the
stepper motor. This function is called cw, short for stepper to move one step. By changing the pins in
clockwise, and it takes three arguments: the number quick succession we can create a smooth rotation. In
of steps to move; the delay between each step; and the code we only partially show the sequence, which
the stepper motor to control. This function can control can be seen in the in the project files downloadable
multiple stepper motors, but for this project we use from our GitHub page.
just the one. for i in range(steps):
def cw(steps,delay,stepper): Temp_IN1.on()
Temp_IN2.off()
Temp_IN3.off()
Work with remote data: OpenWeatherMap Temp_IN4.off()
time.sleep(delay)
Python has many methods of working with external data,
Temp_IN1.off()
one of which is OpenWeatherMap.
OpenWeatherMap is a free resource of worldwide Temp_IN2.on()
weather data that has an extensive API for many Temp_IN3.off()
languages.The OpenWeatherMap website details the many Temp_IN4.off()
uses of this project. It can be used as a simple website to time.sleep(delay)
query historical and forecasted weather data, but by using
….
an API for Python, it enables our Raspberry Pi to receive
remote data that will ultimately control our appliance, With that we end this function by making a new line
creating a physical appliance that can show us the data under the function, and ensuring that our cursor is to
in an unusual way. By merging remote data with physical the left of the window.
computing we can create new appliances and even use data We now create a new function, this time called
to drive art installations. In our tutorial we used pyowm
ccw, Counter Clockwise. This function has the same
from https://github.com/csparpa/pyowm, which has now
been packaged ready for use with the Python package arguments as cw, but reverses the sequence to force
manager pip3. the stepper to rotate counter clockwise. Again, not all
There are many more modules that can pull data from of the code is shown due to its length:
other services, for example eBay and newsfeeds. Another def ccw(steps,delay,stepper):
source of external data is IFTTT, short for “If This, Then
if stepper == "Temp":
That”. IFTTT is a trigger- and event-based system that can
link into many web services and your mobile device. You for i in range(steps):
can even create a location-based trigger that will turn on Temp_IN1.off()
your TV ready for when you get home. You can learn more Temp_IN2.off()
about IFTTT at https://ifttt.com. Temp_IN3.off()
Temp_IN4.on()
www.linuxvoice.com 79
TUTORIAL LINUX FOR LEARNERS
the key, which in turn will return the correct weather
condition as a string. A dictionary uses { } to contain
the data. Our keys are identified by a colon, :, and our
data are strings containing weather status.
codes = {
211:"thunderstorm",
313:"shower rain and drizzle",
321:"shower drizzle",
500:"light rain",
…
}
Next we create a variable called w, which is used to
temporarily store weather data.
w = observation.get_weather()
From this variable we now obtain the temperature
from our chosen location, which we then store in a
new data structure called a The new data structure
is a dictionary and we're looking for the key temp,
as in temperature. On the next line of code we wrap
the extracted data in a function that will convert the
data into an integer. All of this is then contained in a
variable called a.
a = w.get_temperature('celsius')
a = int(a['temp'])
Our circuit involves quite a
time.sleep(delay) Next we create a variable called b, and in there we
few wires, for best results
build and test segments of Temp_IN1.off() store the weather code that matches the current
the circuit as you go. It is Temp_IN2.off() weather at our chosen location. This will be used later
easier to debug a hardware Temp_IN3.on() with our weather codes dictionary.
issue. Temp_IN4.off() b = w.get_weather_code()
time.sleep(delay) To confirm that the data stored in variables a and
…. b is correct we print the contents to the Python shell.
We now close this function and create a new This step can be removed once the data has been
function called get_weather. This function uses the confirmed correct.
OpenWeatherMap API (Application Programming print(a)
Interface) via a Python module. Using this module print(b)
we can check the weather for any location across the
globe. This function takes one argument, n, which Else, if
represents the location where we would like to know Still inside our function, we now create a series of
conditional tests that will check the value of the
variable a, which is the temperature of our chosen
Using the OpenWeatherMap API via location. If the temperature matches one of the tests,
Python we can check the weather for then it it is considered True, and the code that relates
to that condition is executed. Our first test uses if and
any location across the globe checks the value of a against the value -5 as in -5C. If
the weather.
def get_weather(n):
Indented in our function we create a variable called
owm, and in here we store output of connecting to the
OpenWeatherMap API using the Python module and
our secret API key. We then create a variable called
observation, which will get the weather data for your
chosen location, via the n argument.
owm = pyowm.OWM("YOUR SECRET API KEY")
observation = owm.weather_at_place((n))
Still in the get_weather function we now create
a dictionary, a Python data structure that can store We used the weather codes from http://bugs.
data with associated keys. In our project we use openweathermap.org/projects/api/wiki/Weather_
a dictionary to store the numerical code given by Condition_Codes to generate our dictionary of weather
OpenWeatherMap for different weather conditions as conditions.
80 www.linuxvoice.com
LINUX FOR LEARNERS TUTORIAL
This project can be built
Working with motors with any Raspberry Pi. We
There are many different types of motors on the market.
chose to use the Pi 3 as it
The first motors we typically come across are simple DC has WIFI built in, but this
motors. These are really cheap and easy to use but not very project could be built using
precise. Next we have micro gear metal motors, which are a Pi Zero.
typically geared to a set ratio – in other words the lower the
ratio, the faster they spin. A high-ratio motor produces slow
movement and plenty of pulling power for larger chassis.
Stepper motors are slow motors, but they are very precise,
in fact they are commonly used in DVD drives, scanners and
printers, devices which require precision movement.
Every motor needs some form of controller, and should
never be connected to the GPIO directly (this can cause
damage to your Raspberry Pi). For the stepper motor we
bought two units from eBay. These units came with their
own controller boards built around the ULN2003, a high-
voltage and high-current controller. The ULN2003 receives
power from the Raspberry Pi and then provides it to the
stepper motor at a higher rate of current, much more than
the Pi can normally supply. Typically you can only power
one stepper motor from the Raspberry Pi; if you need
to power more, an external 5V power supply is required. variables for each of the LEDs using the LED class
You could use a 4 x AA battery box or hack a USB lead from GPIO Zero. So now we can turn an LED on or off
from a USB battery to provide the necessary current. Just
by calling its name followed by on or off. For our first
remember to connect the GND of your battery to the GND of
your Raspberry Pi condition, if the weather code returns “211” then the
LED to indicate a Thunderstorm will illuminate for five
seconds before turning off.
that condition is true then we call the ccw function if b == 211:
with the arguments denoting 11 steps, delay variable, Thunder.on()
and control our “Temp” stepper motor. This will cause time.sleep(5)
the stepper to spin and rotate the temperature dial for Thunder.off()
11 steps, taking us to -5C on the dial. Then after five The code continues with a series of elif statements,
seconds we call the cw function to rotate the stepper each testing the value of b. Our final test is else, and
motor back to its original position. we use this to indicate that it is sunny, as every other
if a == -5: test has evaluated as False, so Else must be True.
ccw(11,delay,"Temp") elif b == 804:
time.sleep(5) Cloudy.on()
cw(11,delay,"Temp") time.sleep(5)
Cloudy.off()
Define a temperature range Else:
We cover a temperature range of -5C to +40C and for Sunny.on()
each temperature we require a conditional test. For time.sleep(5)
subsequent tests we will use “else if”, shortened to Sunny.off()
elif in Python. If the first condition is not true, then the With the functions completed we now go to our
tests will continue until a test evaluates as True, or if main body of code. Here we use a While True loop
none of the tests work then the else condition must to constantly run our code. Inside the loop we call
be True, in which case it forces the code to wait for our get_weather function with our location as an
five seconds before repeating the process. Here we argument. In my case that's “Blackpool, UK”. This calls
see a snippet of the code. all of the code that we have written previously. Once
…. the functions have run, the loop sleeps for 15 minutes
elif a == 40: before repeating the process.
ccw(506,delay,"Temp") while True:
time.sleep(5) get_weather("Blackpool, UK")
cw(506,delay,"Temp") time.sleep(900)
else: With the code and hardware complete, save your
time.sleep(5) work and click on the Run menu, then click on Run
This ends the conditional tests against the Module to run the code. You will see the stepper
temperature data. Now we create a new series of motor come to life and after a few seconds the LED
tests that will check the weather code saved in the weather indicator will illuminate the current status.
variable b against the values hard-coded in our tests.
Again we start with an if condition, but this time we Les Pounder makes things, breaks things, and spends the rest of his
control our LEDs. At the start of the code we created time teaching teachers about the new IT curriculum.
www.linuxvoice.com 81
CODING ENCRYPTION
ROLL YOUR OWN
ENCRYPTION WITH RSA
Write your own crypto tool and understand how the web stays secure.
Y
JOHN LANE ou are protected by encryption whenever you The first thing to understand is that cryptography
access a secure website on the internet, and is a mathematical problem; a numbers game. When
Why do this? you may know this as SSL. Most secure web we talk about a message we mean a number. A
• Deepen your faith sessions begin with a handshake that relies on an potentially large number with hundreds of digits,
that the crypto people asymmetric cryptographic algorithm called RSA. It's but a number nonetheless. To convey a message of
know what they're also used for other things, but most people will words, it first needs to be converted into a number.
doing.
encounter it through their web browsing activities. Fortunately, computers are good at that and we have
• Show off with some
hard maths. We explored how SSL works in issue eight (you codes like ASCII for specifically that purpose. So we'll
can download it from https://www.linuxvoice. put words to one side until our final example at the
com/issues/008/ssl.pdf) and how it begins with an end and concentrate on the numbers for now. And
encrypted key exchange. RSA is used to perform that we'll begin by using small ones to keep things simple.
exchange securely, and how it does that is the subject RSA provides a formula to convert one number (the
of this tutorial. plaintext) into another (the ciphertext). It looks like
RSA was invented in 1977 by Ron Rivest, Adi this:
Shamir and Leonard Adleman – hence RSA. It is the y Ξ xe (mod n)
Finding factors isn't the
way to crack RSA: it most popular asymmetric cryptosystem in use today. Another formula is used to decrypt. It is very similar:
doesn't take a very long When we say that it's asymmetric, we mean that it x Ξ yd (mod n)
key for the time to factor it uses different keys to encrypt and decrypt a message. And that's it. In fact, they're the same function that
to be noticeable. We call these keys public and private. we can represent in code (our examples are in Ruby):
45,000
40,000
35,000
30,000
Time (seconds)
25,000
20,000
15,000
10,000
5,000
0
0 10 20 30 40 50 60 70 80
Number size (bits)
82 www.linuxvoice.com
ENCRYPTION CODING
Listing 1: the algorithms "rsa.rb"
0 #!/usr/bin/ruby 18 end
1 def gcd(a,b) 19 end
2 (r = a % b) == 0 ? b : gcd(b,r) 20 def key(e, *args)
3 end 21 case args.length
4 def phi(*args) 22 when 1
5 case args.length 23 n = args[0]
6 when 2 24 t = phi(n)
7 (args[0]-1)*(args[1]-1) 25 when 2
8 when 1 26 p, q = *args
9 n = args[0] 27 n = p*q
10 (1..n).reduce(0) { |p,i| gcd(n,i)==1 ? p+1 : p } 28 t = phi(p,q)
11 end 29 end
12 end 30 d = eea(e,t).first % t
13 def eea(a,b) 31 [d, n]
14 return b==0 ? [1,b] : begin 32 end
15 q, r = a.divmod b 33 def crypt(m, e, n)
16 s, t = eea(b,r) 34 m**e % n
17 [t, s - q * t] 35 end
def crypt(m,e,n) How can we do that? Well, our formula is very
m**e % n similar to what is known as Euler's Theorem, which
end looks like this:
You could now encrypt a plaintext message x into xφ(n)Ξ 1 (mod n)
ciphertext y if you know what e and n are. If you sent where φ(n) (the Greek character φ) is Euler's Totient
that ciphertext y to someone else and they knew d Function. We'll talk more about that later but, for now,
and n then they would be able to decrypt y to recover it's interesting because we can use it to write an
the plaintext message x. But we don't yet know what equation to help solve d. If we multiply both sides by x
those values are; this is what they mean: we have
e is the public exponent and it's part of the public xφ(n)x = xφ(n)+1 = xed n Ξ x (mod n)
key, along with n. Now, if you consider that the only variable is the
d is the private exponent and, along with n, forms exponent, you can say that
the private key. ed Ξ φ (n) +1 (mod φ(n))
n is called the modulus, and the mod in the Note how the modulus becomes φ(n), because
formulae indicate they're using modular arithmetic
(see the box if you need that explained).
We are free to choose what n is, but our choice
All our calculations are done mod n,
defines the biggest number that can be encrypted. which means that we can encrypt
This is because all our calculations are done mod n,
which means that we can encrypt between 0 and n-1. between 0 and n-1
We use the size of n in binary digits (bits) to describe
the key size. Real-world keys should be at least that's the modulus of exponents in a modulo n
2048 bits to be considered secure (that's a 617-digit formula (Euler'sTheorem proves this if you are
number). interested). We can replace the modulus with some
Before choosing e and d, think about the encryption multiple, k, of φ(n):
and decryption formulae and that decryption undoes ed=kφ(n)+1
encryption. This implies that you should be able to ed + kφ(n) = 1
combine them like this: We now have a formula in the style of Bézout's
x Ξ (xe)d (mod n) Identity. This is another theorem in the elementary
which is theory of numbers and it states that, for two integers
x Ξ xed (mod n) a and b, and their greatest common divisor c, there
And you can, and you get ed=1. But it doesn't follow exist integers x and y such that:
that d=1/e. No, if it were that simple then we wouldn't ax+by=c
have encryption. And it isn't that simple because of and x and y are called the Bézout Coefficients of a and
the modulus, and there are no fractions in modular b and, if those are e and φ(n)then x will be the private
arithmetic. exponent, d, that we seek.
So we need to find an integer value for d that gives To make this work out for us, the theorem requires
1 when multiplied by e, and we say that value d is the that the greatest common divisor, c, of e and φ(n) is 1
modular multiplicative inverse of e. or, in other words, that they are coprime. While we're
www.linuxvoice.com 83
CODING ENCRYPTION
Modular arithmetic
Modular arithmetic is way of counting integers where they
0 0
wrap around upon reaching a value that we call the "Modulus".
Time and, more specifically, clocks provide an easy to +4h
understand real-world example of everyday use. 9 3 9 3
Modular formulas are suffixed with (mod n) to indicate
they are modulo n and the triple-bar congruence (Ξ) symbol
indicates modular equivalence, which is like equals (=) in the 6 6
modulo arithmetic world.
A congruence like 15 Ξ pmod 12 can be written as
12k+15=3 where k is any positive integer; any multiple (k) of Time is modulo 12 making one o'clock four hours later
the modulus (12) is congruent (equivalent) to 0 (mod 12). So than nine: 9+4 Ξ PMOD:12
(CC BY-SA): https://en.wikipedia.org/wiki/File:Clock_group.svg
the integers 3, 15, 27 and so-on are congruent in modulo 12.
Another way to look at it is as the remainder in a whole-
number division, so 9+4=13 and 13 / 12=1 with a remainder of
1. There's a modulus operator that takes two arguments and Consider that 4 Ξ 14 mod 10 but 4/2 Ξ / 14/2 mod 10. Modular
returns the remainder like 15 (mod 12) = 3. Note that the mod arithmetic is constrained to integers – there are no fractions.
operator's result is an equality (=) rather than a congruence Another thing to watch out for with modular reduction is
(Ξ). Many programming languages use a % operator for this: that exponents must be reduced by a different modulus: the.
15 % 12 = 3. The algorithms used by RSA are based in modular arithmetic,
While we're talking division – watch out: modular arithmetic albeit with a very large modulus!
doesn't have division but, instead, has modular reduction.
free to use any value for e within that constraint, most We say it's a "hard" thing to do, not because the
real-world implementations choose one of the so- algorithm is difficult, but because it would require
called Fermat Primes (3, 5, 17, 257 and 65,537) with a disproportionate amount of time (and computer
65,537 being the most popular. power) to complete.
So, all we have to do to find d is to solve Bézout's But there are numbers where iteration can be
Identity but, before we can do that, we need to find avoided: prime numbers. Primes cannot be divided, so
φ(n). everything less than a prime is coprime to it. In other
words, if n is prime then φ(n) = n-1 (which is what
The Phi Factor Fermat had said originally).
In 1640, Pierre de Fermat stated Fermat's Little Still, anyone knowing n (remember that n forms part
Theorem for prime numbers. A hundred years later, a of the public key) would be able to quickly determine
Swiss mathematician called Leonhard Euler the private key if n was prime. So we really don't want
n to be prime; we need a composite number.
You can find the greatest common We can use another property of the totient
to efficiently find it for a composite number: it's
divisor using an algorithm that was multiplicative. This means, for two numbers p and q,
that:
documented over 2000 years ago φ(pq)=φ(p)φ(q)
Now, if p and q are prime then the totient of pq
(pronounced Oiler) generalised Fermat's theorem so will be (p-1)(q-1). This means that you can quickly
that it could also be applied to non-prime (composite) calculate the totient of a number if you can find its
numbers. Euler's Theorem defines the "totient" of a prime factors. But finding factors of large numbers
number n to be the count of the positive integers that is also hard. So we choose the prime numbers and
are less-than n and are coprime with n. multiply them to make n. We keep them secret (as
The totient can be derived by iterating part of the private key) and that's what keeps your
over the numbers between 1 and n online banking account secure.
PRO TIP
and increasing a total for any number So we can now find the totient, φ(n), either
A United States patent for RSA was
found to be coprime with n – that is any iteratively for small values of n or, for any value that is
granted in 1983 and expired in 2000. You
can read it at https://www.google.com/ number that has a greatest common composed as the product of known prime numbers. If
patents/US4405829. divisor (gcd) of 1 with n. You can find we only use two primes then it'll be even more difficult
the gcd using an algorithm that was to find them by factoring n.
documented over two thousand years We'll call our two prime factors p and q, and can
ago by the Greek mathematician, Euclid (see the obtain them from any suitable source such as
ancient algorithms box). OpenSSL, which includes a command-line tool that
The totient is the key to the security of the can generate 16-bit or larger primes: "openssl prime
RSA algorithm because computing the totient -generate -bits 20".
iteratively would take too long for large numbers. The Wolfram Alpha computational knowledge
84 www.linuxvoice.com
ENCRYPTION CODING
engine (wolframalpha.com); a request like
"randomprime(1<<2047)" will produce a 2048-bit
prime.
It doesn't matter where the factors come from, as
long as they are prime. The bit length of n is the sum
of the bit lengths of p and q; you can use half the
desired bit length of n for the factors (so two 1024-bit
factors would give a 2048-bit key).
Now that we have e and n (or its prime factors p
and q), we can find the totient t. Listing 1 presents
everything we need:
gcd (lines 1–3) uses the Euclidean Algorithm to find
the greatest common divisor of two numbers a and
b, and
phi (lines 4–12) returns the totient of one number
by iteration or of two numbers (assumed prime)
using the totient formula.
The key method (lines 20–32) generates a private
OpenSSL provides tools that can create and verify standard key files from the parameters
key from a public exponent, e, and a modulus that
generated by our test program.
can be either a number, n, or its prime factors p,q. It
first uses phi to get the totient, t, of the modulus and
then uses a variant of Euclid's algorithm, the Extended The tests demonstrate that the algorithm is sound,
Euclidean Algorithm, to compute the private key. but think about large numbers for a moment. Raising
You don't need to know how the Extended numbers to large exponents is likely to exceed the
Euclidean Algorithm (eea, Listing 1, lines 13–19) or the capabilities of your math library, even with a big
other algorithms work to understand how RSA uses number library (we found this happened
them, but there are many well explained resources on when n was around 21 bits). We need
the web if you want to know more. another way and, because we want the PRO TIP
The key method passes, on line 30, the public result modulo n, modular arithmetic can The current factoring record was set
exponent, e, and the totient, t, into eea and receives provide the answer: a large exponent in 2009 when the factors of a 768-bit
number were found. It took two years to
their Bézout Coefficients - the first one is the private can be reduced to a simpler problem factor the 232-digit number.
key. That's taken modulo t to ensure it's a positive and one way to do this is known as the
number (if it's negative, it'll have a positive congruence binary or squaring method. Here is one
modulo t). way to implement it:
As an example, take e=17 and n=26. Those def powmod(base, exponent, modulus)
numbers are small enough to iteratively compute t or return modulus==1 ? 0 : begin
factorise n into p=2 and q=13 and then obtain t=(2-1) result = 1
(13-1)=12. The algorithm will return the private key base = base % modulus
d=5. We have our keys – we can now encrypt!
Encryption is easy. Take a message m=20 and it's while exponent > 0
just a matter of ne mod n = 2017 mod 26 = 24. And result = result*base%modulus if exponent%2 == 1
decryption is just as easy: 245 mod 26 = 20. Both use exponent = exponent >> 1
the same formula so a single crypt method (lines base = base*base%modulus
33–35) provides the implementation.
Listing 2 presents a small test that exercises the end
algorithms in Listing 1. We can use it to demonstrate result
that the message, m, is encrypted to ciphertext, c with end
public key e, which is decrypted to mm by private key end
d: Add this new powermod method into rsa.rb and
$ ./testrsa.rb 20 17 26 change crypt (Listing 1, line 34) to use it:
m:20 e:17 args:26 powmod(m,e,n)
n:26 d:5 c:24 mm:20 This allows key sizes to be increased beyond 21
You can also test with larger numbers and use bits. You can try to generate a key with 2048 bits if
prime factors: you feel brave enough (you will need to use two 1024-
$ ./testrsa.rb 29384 65537 199 617 bit primes). Something like this :
m:29384 e:65537 args:199,617 $ p=$(openssl prime -generate -bits 1024)
n:122783 d:78209 c:17392 mm:29384 $ q=$(openssl prime -generate -bits 1024)
$ ./testrsa.rb 29384 65537 $p $q
Exponential Efficiencies The binary method only needs to perform
www.linuxvoice.com 85
CODING ENCRYPTION
Listing 2: Testing "testrsa.rb"
0 #!/usr/bin/ruby 10 m, c, *params = rsa(*ARGV.map{|i| i.to_i})
1 require_relative 'rsa' 11 puts "asn1=SEQUENCE:rsa_key\n\n[rsa_key]"
2 def rsa(m, e, *args) 12 %w(modulus pubExp privExp p q e1 e2 coeff).each do |i|
3 puts "m:#{m} e:#{e} args:#{args.join(',')}" 13 break if params.empty?
4 d, n, *crt = key(e,*args) 14 puts "#{i}=INTEGER:#{params.shift}"
5 mm = crypt(c = crypt(m,e,n),d,n) 15 end
6 puts "n:#{n} d:#{d} c:#{c} mm:#{mm}" 16 end
7 [m,c,n,e,d,*crt] If given no arguments this test program it will continuously
8 end generate input data to soak-test the algorithm.
9 if ARGV.length > 0
multiplicative operations for each 1 bit in the the private key and let them become part of it. The
exponent, so we can make further efficiencies by values are two new exponents and a modular inverse
having as few as possible. The usual choice of 65537 and we extend the key method in Listing 1 (replace
for the public exponent e has only two 1 bits but the line 31 with the below) to compute them:
private exponent d will be a large number with a lot if defined?(p) && defined?(q)
of 1 bits, and this can make decrypting a time-wise dp = d % (p-1)
expensive operation. dq = d % (q-1)
We can employ techniques to minimise this: we can qinv = eea(q,p).first % p
use Carmichael Numbers to find a smaller value for d [d,n,p,q,dp,dq,qinv]
and then use the Chinese Remainder Theorem to split else
the decrypting exponent operation into two smaller [d, n]
(and more efficient) modular exponent calculations. end
Then, provide a new decrypt method that can be
We'll use our RSA implementation in used with those values to decrypt a message:
def decrypt(c,p,q,dp,dq,qinv)
the real world to generate a key and m1 = powmod(c,dp,p)
use it with OpenSSL
m2 = powmod(c,dq,q)
h = qinv*(m1-m2) % p
m = m2 + h*q
The Carmichael Function (sometimes called the end
Reduced Totient), usually denoted as λ (lambda), If we have the CRT values in our private key then we
is a drop-in replacement for Euler's Totient that can can decrypt a message more efficiently by using:
be used when the prime factors p and q are known. decrypt(c,p,q,dp,dq,qinv)
It results in private keys that are smaller. Smaller instead of the equally valid but less efficient
numbers have fewer bits and we've seen that fewer crypt(c,d,n)
bits are less work when computing exponents. To use
it, add its definition to the code: Hello, World!
def lam(p,q) Those optimisations complete the RSA
(p-1).lcm(q-1) implementation. We'll now show it is compatible with
end real-world implementations by generating a key and
and then change the key method (Listing 1, line 28) to using it with OpenSSL. We'll also use a real text
use it. Change it so that lam is used instead of phi to message instead of a number to demonstrate that
obtain the totent value t: real words can be encrypted.
t = lam(p,q) First of all we will use the test program to create
The rest of the code will use the value as if it is φ; some key parameters (the message isn't important
nothing else needs to be changed. Private keys will because were only interested in the key; choose any
now be smaller! public exponent and prime factors that you want -
You can test with e=3, p=17 and q=23. If we used we've used 32 bits to keep them readable but 1024 bit
φ, we'd get d=235 but using λ it's d=59. What's primes will work too):
interesting is that this reveals a little-known fact – $ ./testrsa.rb 12345 65537 3273361529 4176726557
there is more than one private key for any public key: Put them in a temporary text file formatted like this:
both values of d will decrypt a message that was asn1=SEQUENCE:rsa_key
encrypted with the public exponent e=3 and modulus version=INTEGER:0
n=17*23=391. modulus=INTEGER:13671936028836425653
The second optimisation uses the Chinese pubExp=INTEGER:65537
Remainder Theorem to split the computation in two. privExp=INTEGER:531444176792982513
We compute three additional values when we create p=INTEGER:3273361529
86 www.linuxvoice.com
ENCRYPTION CODING
q=INTEGER:4176726557 ciphertext). We can encrypt with OpenSSL and
e1=INTEGER:1295519345 decrypt with our test:
e2=INTEGER:20202669 $ echo -n 'HiWorld!' > message.txt
coeff=INTEGER:1526950891 $ openssl rsautl -pubin -inkey public.pem -in message.
You may have noticed that the test program also txt -encrypt -raw -out ciphertext.txt
outputs key information in this format; now you know $ ./crypt.rb < ciphertext.txt 531444176792982513
why. Save the temporary file (with a recognisable 13671936028836425653
name, say key.cnf), and run it through OpenSSL to HiWorld!
create a standard .pem formatted key files (we have First we place our message into a text file. We're
to first create a DER-formatted key and then change using "raw" mode with OpenSSL, which means that
it to PEM): it won't apply the padding that it would normally use
$ openssl asn1parse -genconf key.cnf -out key.der to make the message the same size as the modulus
$ openssl rsa -in key.der -inform der -check -noout and also more secure. Padding is external to the RSA
RSA key ok algorithm and space prohibits covering it now, so
using -raw disables it. Note, however that OpenSSL
$ openssl rsa -in key.der -inform der -out private.pem then insists that the input message is the same size
writing RSA key as the modulus – hence our test message is exactly
$ openssl rsa -in key.der -inform der -out public.pem eight characters to match our 64-bit key.
-pubout With the message taken care of, we call OpenSSL
writing RSA key to get an encrypted ciphertext.txt file, which we pass
That gives us valid keys in text files private. as input to our test program. We pass our private
pem and public.pem that we'll now use to perform exponent and modulus as arguments and receive our
some encryption. We'll need another very small test original text as output. Finally, reversing the order, we
program for that (crypt.rb): can encrypt with our test, passing our public exponent
#!/usr/bin/ruby this time, and decrypt with OpenSSL:
require_relative 'rsa' $ ./crypt.rb < message.txt 65537 13671936028836425653 >
m, mm = 0, '' ciphertext.txt
STDIN.read.each_byte { |c| m=(m<<8)+c }
m = crypt(m,*ARGV.map{|i| i.to_i})
OpenSSL insists that the input message
while m>0 do is the same saze as the modulus –
hence our test message is 8 characters
mm.prepend (m & 255).chr
m = m >> 8
end
print mm $ openssl rsautl -inkey private.pem -in ciphertext.txt
This encrypts or decrypts a message read from -decrypt -raw
standard input. First it converts the message into HiWorld!
an integer as required by the algorithm. It then Everything works as expected, proving our RSA
calls the crypt method, passing in the command- implementation works as it should. We'll leave it as
line arguments, where the exponent and modulus an exercise for you to write a decrypter that uses the
should be given. The integer returned by crypt is then Chinese Remainder Theorem to decrypt the message
returned to a string and output (print is used so that (hint: use the decrypt method instead of crypt).
a newline isn't appended because this will corrupt
Final words
We've shown how to write your own RSA
cryptosystem to demonstrate how it works by
Ancient algorithms revealing the centuries-old algorithms that it relies
RSA depends on some algorithms that have stood the test upon. We've created keys and then used them with
of time. Euler's theorems, which are the foundation for OpenSSL to demonstrate that they are compatible.
RSA, were penned in the 17th century but aren't the oldest. However, we haven't covered everything, and there are
The Chinese Remainder Theorem, which can speed up
ways of using RSA that are less secure than others.
decryption, was evident as far back as the 3rd century.
But it was around 300BC when the Greek mathematician Real-world implementations are aware of such
Euclid documented his Euclidean Algorithm for the greatest potential weaknesses and apply techniques such as
common divisor (or gcd) of two numbers – the largest the padding schemes that we mentioned to mitigate
number that evenly divides both of them. He discovered, them. Don't rely on a homebrew cryptosystem to
given two numbers a and b, that gcd(a,b) is the same as
protect your secrets!
gcd(b, a mod b) and that this fact can be applied recursively
until a mod b Ξ 0; we expressed this succinctly in code
(Listing 1, line 2). John Lane provides technical solutions to business problems.
He has yet to find something that Linux can't solve.
www.linuxvoice.com 87
CODING ELIXIR
USE ELIXIR TO DEVELOP
A WEB APPLICATION
Elixir can make your software reliable, fault-tolerant and highly available.
E
MIHALIS lixir is a functional programming language sudo dpkg -i erlang-solutions_1.0_all.deb
TSOUKALOS created by José Valim built on top of the Now you can grab the packages you need:
Erlang Virtual Machine. Elixir tries to improve sudo apt-get update
Why do this?
the complicated parts of Erlang while keeping the sudo apt install esl-erlang
• Develop better web
applications
good parts intact. Elixir helps you write cleaner sudo apt install elixir
• Use Elixir to develop
programs using less code than Erlang. This means For other distros, the process will be a little different;
fault-tolerant web two things: first, that you can better understand what the details are at http://elixir-land.org/install.html.
applications an Elixir program tries to implement; and second, that You can find the version of Elixir you are using by
• Learn how to develop you can maintain an Elixir program more easily than running elixir --version; at the time of writing this
a website in Elixir
an equivalent Erlang program. Both Elixir and Erlang tutorial, the latest stable version of Elixir is 1.2.4.
are well suited for writing reliable server software. The following code is the Elixir version of the "Hello
Elixir enables you to very easily create powerful World!" program:
software, as we will demonstrate. After finishing this $ cat helloWorld.ex
tutorial you will be able to program your own highly defmodule LinuxVoice do
available, fault-tolerant web application in Elixir. def hello do
You'll need to install both Erlang and Elixir. If you're IO.puts "Hello World!"
on a Debian-based system, you'll need to add the end
repositories for Erlang Solutions, like so: end
wget http://packages.erlang-solutions.com/erlang-
solutions_1.0_all.deb LinuxVoice.hello
The reason for needing a module and a function
for such a simple program is that all Elixir code must
be organised in modules and functions. The last
command is for automatically calling the desired
function. The .ex extension is used for files that
contain Elixir code, whereas the .exs extension is used
for Elixir scripts.
You can execute helloWorld.ex using the interactive
Elixir shell (iex) or compile it and execute it using the
Elixir compiler (elixirc). Alternatively, you can use the
Elixir script runner, named elixir, which is similar to iex
but automatically exits when the Elixir script finishes.
So, you can run helloWorld.ex as follows:
$ elixir helloWorld.ex
Hello World!
$ elixirc helloWorld.ex
Hello World!
$ ls -l Elixir.LinuxVoice.beam
-rw-r--r-- 1 mtsouk mtsouk 1348 Apr 25 16:44 Elixir.
LinuxVoice.beam
$ file Elixir.LinuxVoice.beam
Elixir.LinuxVoice.beam: Erlang BEAM file
The mix utility is the Elixir
build tool, that can help $ iex
you create, compile, test iex(1)> c("helloWorld.ex")
and manage your projects'
dependencies. Hello World!
88 www.linuxvoice.com
ELIXIR CODING
Here's what will happen
[LinuxVoice] mix --help. If you want to find more information about
if you try to start another
iex(2)> LinuxVoice.hello the new command, for example, you can execute mix instance of the same web
Hello World! help new – this works for every mix command. application.
:ok Processes in Elixir are identified by a unique
As you can see, elixirc automatically generates a process ID (PID). A PID has the following form and its
BEAM file as does the Elixir shell after processing the own data type, which means that you cannot manage
c("helloWorld.ex") command. This happens because a process ID as if it were a string:
when you compile an Elixir program, the compiler #PID<0.185.0>
converts the code into a BEAM (Bodgan's Erlang The project for the web application will be created
Abstract Machine) file, used by Erlang. The Elixir script with the help of the mix tool. Run mix new web --sup
runner doesn't generate a BEAM file which can be very in the new directory to build the necessary files (see
convenient when experimenting with Elixir. figure 1). The --sup option that was used is optional;
its purpose is generating an OTP application skeleton
Process Power
Elixir processes are very lightweight and isolated from
each other. An Elixir program is implemented as a An Elixir program is implemented as
large number of small processes that do simple tasks
and communicate with each other using code that
a large number of small processes that
has no side effects. The biggest difference between perform simple tasks
Elixir and Erlang is that in Elixir the value of a variable
can change after its initial assignment whereas in including a supervision tree (see boxout). You'll learn
Erlang this is not allowed. more about supervisors later on in this tutorial.
Elixir offers a tool for creating new projects called Other useful mix commands include mix compile,
mix. To get a list of all available mix options, execute which compiles the source files of the current project,
mix clean, which erases the generated application
files and mix run, which is used for executing your
What is OTP? project – running mix run on an empty project
OTP stands for Open Telecom Platform, and is Erlang’s generates no output. The mix test command runs the
collection of open source libraries and tools designed for tests of a project; when you create a new project, mix
developing big projects. OTP is about taking all the generic automatically generates some dummy tests – it is the
components, putting them into libraries, making sure they
job of the developer to create real tests.The purpose
work fine and reliably and then reusing that code as often
as possible. The programmer needs only to deal with things of the mix.exs file is to configure your project. The
that change from application to application. code for the project can be found at ./lib/web.ex and
OTP enables you to supervise existing Elixir and Erlang ./lib/web/router.ex – these are the files that you are
code. In order to supervise an existing module, you will to going to edit.
write additional Elixir code, but you will not need to make
You can start the web application by executing the
any changes to the module you want to supervise! If you
are going to write real-world Elixir or Erlang software, you following command:
will eventually have to learn the OTP Framework. $ iex -S mix
iex(1)> :application.which_applications
www.linuxvoice.com 89
CODING ELIXIR
the application function is used for describing the
application itself. Finally, the deps function is used
for listing the dependencies of the project. The
application function states the name of the Elixir
module that will be used for developing the application
– in this case the name of the module is Web, based
on the argument of the mix new command. However,
the name of the application is :web, as defined in the
project function – you will use this name to start and
stop the application. When you start the application
using mix, the Web.start/2 function is automatically
called. The /2 means that the Web.start() function
requires two arguments.
The final version of mix.exs without any comments
is the following:
defmodule Web.Mixfile do
use Mix.Project
def project do
[app: :web,
version: "0.1.1",
elixir: "~> 1.2",
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
deps: deps]
end
def application do
[applications:
The various URLs
[{:web, 'web', '0.0.1'}, {:logger, 'logger', '1.2.3'}, {:mix, 'mix', [:logger, :cowboy, :plug],
supported by the web
'1.2.3'}, mod: {Web, []}
application. You can add
as many URLs as you want! {:iex, 'iex', '1.2.3'}, {:elixir, 'elixir', '1.2.3'}, ]
{:compiler, 'ERTS CXC 138 10', '6.0.3'}, {:stdlib, 'ERTS CXC end
138 10', '2.8'},
{:kernel, 'ERTS CXC 138 10', '4.2'}] defp deps do
The previous output shows that the web application [{:cowboy, "~> 1.0.3"},
is running along with many other applications {:ranch, "1.2.1"},
including mix, logger, iex, elixir, compiler, stdlib and {:plug, "~> 1.1.2"}]
kernel. end
end
For such a simple app, the main task of As you can see from the definition of the deps
function, the project uses three external components
the developer is defining the supported named Plug, Ranch and Cowboy. After defining that
you want to use Plug, Ranch and Cowboy, mix will
URLS using pattern matching get all the modules for you or give you instructions on
how to install them and any other additional module
In this section you're going to make the web dependencies when you try to run or compile the
application support multiple web pages. There exist project. As you can understand from the definition
many web frameworks that can make your life of the application function, you also need to tell mix
easier, including the famous Phoenix web framework. which modules your web application will actually use.
However, the simplicity of our application does not As Ranch is not directly used by :web, you don't need
justify the use of Phoenix . to include it in the list.
For such a simple application, the main task of The Elixir code of the ./lib/web.ex file is:
the developer is defining the supported URLs using defmodule Web do
pattern matching. With pattern matching in functional use Application
programming languages, you need to provide a route
or a case that matches the URLs that cannot be def start(_type, _args) do
matched by the other routes. import Supervisor.Spec, warn: false
The mix.exs file has three parts.: the project IO.puts "Starting Web Router!"
function is used for describing the project, whereas children = [
90 www.linuxvoice.com
ELIXIR CODING
end
Plug.Adapters.Cowboy.child_spec(:http, Web.
Router, []) get "/" do
] conn
|> send_resp(200, "ok")
opts = [strategy: :one_for_one, name: Web.Supervisor] |> halt
Supervisor.start_link(children, opts) end
end
get "/bug" do
end raise "WAT"
You will need to create a new module inside the lib end
directory that will serve all HTTP connections with
the help of the Plug module – the name of the new get "/hello/:name" do
module will be Web.Router and its code can be found conn
inside .lib/web/router.ex, where you can define as |> send_resp(200, "Hello #{name}!")
many routes as you want. |> halt
defmodule Web.Router do end
use Plug.Router
use Plug.Debugger match _ do
conn
plug Plug.Logger |> send_resp(404, "Not Found")
plug :match |> halt
plug :dispatch end
end
def start_server do The match _ do part is the case that matches what
end is left. Please note that the match all case must be the
last one in your code because only the first match is
def init(options) do executed.
options You can start the web application with the help of
end the Elixir interactive shell as follows:
$ iex -S min
def start_link do You can also stop the web server from running by
start_server exiting the Elixir shell. First, you should press Ctrl+C,
which will display the message:
The Erlang philosophy iex(2)>
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
The design of Erlang follows six rules. The first rule
is Isolation, which means that Erlang processes are (v)ersion (k)ill (D)b-tables (d)istribution
isolated; therefore each Erlang process has its own stack Select a to end the server.
and heap and is separately garbage collected. Also, Figure 2 shows the output you will get when you
processes cannot see the memory of other processes and try to execute another instance of the same web
consequently cannot harm other processes. The second
application. This happens because a TCP port, in
rule is Concurrency, which means that Erlang processes are
concurrent by design. So, in theory, all processes can run in this case port 4000, can only be used by a single
parallel. This is an excellent property now that computers application.
have multi-core processors, because processes can be Figure 3 shows the web application in action. The
spread over the available cores. /bug address uses Plug.Debugger to show a very
The third rule is Failure Detection. As failure cannot be
informative web page that can be used for debugging
avoided, Erlang processes can detect failures. You can
also create a link between two processes; therefore, when purposes.
a process dies for some reason, some other process can
be informed about the failure of the first process. So, when Under supervision
something fails, you can let someone else fix the problem. A supervisor has a very specific responsibility:
The fourth rule is Live Code Upgrade. Put simply, Erlang
supervising other processes. If something bad
can be modified as it runs! Applications can be upgraded
while running without downtime! The fifth rule is Fault happens to the supervised process, the supervisor will
Identification. This means that when a process fails, the notice and start a new process.
error signal contains additional data provided by the Erlang The good news is that you do not need to write any
runtime system that tells exactly why the process has extra code to create and use a supervisor for your
failed. The sixth rule is Stable Storage, which is not done in
web application, as this has already be handled by mix
Erlang but in third-party libraries. You can use Mnesia, Riak,
or another supported databases for storing data. Every when you executed mix with the --sup command line
process can access the data of a database because the parameter. All processes included in the "children" list
data is shared among Erlang processes. will be supervised. The :one_for_one option inside the
start function of the web.ex file tells the Supervisor to
www.linuxvoice.com 91
CODING ELIXIR
As all Elixir processes communicate with
messages, it does not matter whether all processes
are running on the same machine, because they will
be able to communicate with each other if needed.
This section will show you how two different Elixir
shells can communicate with each other. Although
you will need at least two Linux machines to run a
distributed application, you can simulate the process
on one Linux machine. The first step includes starting
two Elixir shells with different names on two different
terminals on the same Linux machine. In the first shell
you should execute the following command:
$ iex --sname n1
In the second shell, you should execute the
following command:
$ iex --sname n2
As you can see in the image, left, the two different
Elixir shells have different prompts, depending on the
value of the --sname command line option, which
helps you differentiate between them. Then, you
define a new module, named WantToCommunicate,
in shell n2 that just contains the sendMessage/0
function. The key point here is that shell n1 knows
nothing about the WantToCommunicate module and
its sendMessage/0 function, which is verified by the
following output:
Look! Two Elixir shells iex(n1@mail)1> WantToCommunicate.sendMessage
communicating with each start one process if one process crashes – this is the ** (UndefinedFunctionError) undefined function
other! most important part of the Elixir code related to the WantToCommunicate.sendMessage/0 (module
supervisor. WantToCommunicate is not available)
In order to see what this really means, you should WantToCommunicate.sendMessage()
first find the process ID of the current process and kill However, the n1 shell can use the
it. The steps are the following: WantToCommunicate.sendMessage/0 function and
iex(5)> children = Supervisor.which_children(Web. get its output as follows:
Supervisor) iex(n1@mail)1> Node.spawn_link :n2@mail, fn ->
[{{:ranch_listener_sup, Web.Router.HTTP}, #PID<0.279.0>, WantToCommunicate.sendMessage end
:supervisor, Message sent!
[:ranch_listener_sup]}] #PID<9332.79.0>
iex(6)> [{_, pid, _, _}] = children So, we called a function from a different shell on
[{{:ranch_listener_sup, Web.Router.HTTP}, #PID<0.279.0>, the same computer, which could have been a remote
:supervisor, computer, without writing any extra code!
[:ranch_listener_sup]}] We hope that this tutorial helped you realise the
iex(7)> pid advantages of Elixir and OTP – if you're developing a
#PID<0.279.0> web application, you should definitely consider writing
iex(8)> Process.exit(pid, :kill) it in Elixir!
true
iex(9)> children = Supervisor.which_children(Web.
Supervisor)
Bibliography
[{{:ranch_listener_sup, Web.Router.HTTP}, #PID<0.393.0>, You can learn more about Elixir by reading Programming
:supervisor, Elixir 1.2: Functional, Concurrent, Pragmatic, Fun from
Pragmatic Bookshelf. Two other excellent books about
[:ranch_listener_sup]}]
Elixir are Elixir in Action from Manning Publications and
iex(10)> [{_, pid, _, _}] = children Seven More Languages in Seven Weeks from Pragmatic
[{{:ranch_listener_sup, Web.Router.HTTP}, #PID<0.393.0>, Bookshelf, which contains a chapter on Elixir.
:supervisor, If you're interested in Erlang you'll find both Programming
[:ranch_listener_sup]}] Erlang, 2nd edition from Pragmatic Bookshelf and Learn
You Some Erlang for Great Good! from No Starch Press
iex(11)> pid
very useful. Finally, there's Erlang and OTP in Action from
#PID<0.393.0> Manning Publications, which talks about OTP.
The Supervisor automatically started a new
process with pid #PID<0.393.0> after you killed the Mihalis Tsoukalos is a Unix administrator, programmer and
old process with pid #PID<0.279.0>. mathematician who also enjoys writing technical articles.
92 www.linuxvoice.com
ELIXIR CODING
www.linuxvoice.com 93
CORETECHNOLOGY NETWORKING
CORE
Valentine Sinitsyn develops
high-loaded services and
teaches students completely
unrelated subjects. He also has
a KDE developer account that
he’s never really used.
TECHNOLOGY
Prise the back off Linux and find out what really makes it tick.
POSIX Threads
M
ost general-purpose OSes today run threads. This way, the main UI thread is kept ready for
processes. They serve as units of isolation: user-generated events (say, mouse clicks). Overall,
processes have distinct address spaces, the application remains responsive, even if it does
and if one of them crashes, the others remain some complex maths in the background. It's also
unaffected. Processes are also units of parallelism: an common to spawn threads for heavy I/O tasks, yet
OS can execute many of them simultaneously. On a asynchronous approaches (LV016 and LV028) may
uniprocessor system, this is just an illusion carefully serve you better. Like processes, threads come at a
preserved by the process scheduler. Most PCs today price. CPU-bound tasks are usually worth it, while for
are multicore, though, and processes can really run in I/O bound tasks, there could be cheaper alternatives.
parallel on different cores. To sum up, processes are a Different operating systems provide various
useful yet costly abstraction. threading APIs. For Unix (Linux included), POSIX
Sometimes, you don't need isolation. You just want Threads are the standard. IEEE POSIX 1003.1c
to run some tasks in parallel. This could make a good defines them, and the implementation is usually
case for "lightweight processes", or threads. In this available as libpthread which comes with the
Core Tech, we'll look at POSIX Threads (Pthreads), and standard C library (libc). To enable threading support
how Linux implements them. in your code, pass the -pthread switch to GCC.
Creating a thread is simple:
Hello, threads! #include <pthread.h>
As a programmer, you may think of a thread as a static void * thread_fn(void *unused)
function that executes in parallel with main(). For {
Linux, threads are just processes that share some int done = 0;
resources. This includes memory, file descriptors, while (!done) {
htop displays threads in a
and signal handlers. Each thread has its own stack /* Do the work */
different colour, and with a
custom name, settable via and CPU context, so the kernel can schedule them }
pthread_setname_np(3). independently. return NULL;
Note this four-core CPU A typical desktop application may offload time- }
runs 314 threads in total. consuming jobs, such as software rendering, to int main()
{
pthread_t thread_id;
void *retval;
int err;
err = pthread_create(&thread_id, NULL, thread_fn,
NULL);
if (err) { /* Handle it */ };
err = pthread_join(thread_id, &retval);
...
return 0;
}
First, we define a "thread function", thread_fn.
That's the code our thread will execute. Thread
functions receive a sole void * argument. When the
function returns, or calls pthread_exit(), the thread
is terminated. So, it is a common pattern to loop in a
94 www.linuxvoice.com
NETWORKING CORETECHNOLOGY
thread function until some condition is fulfilled.
Next, we actually spawn a thread in main() with
pthread_create(). thread_id is like a handle to the
thread. Don't confuse it with TID (see gettid(2)), which
is a Linux process' property. pthread_t is just some
opaque value used as a reference to a specific thread
across Pthreads functions. For example, you can
cancel a thread with pthread_cancel(thread_id).
Races manifest
The second argument to pthread_create() defines but it's rarely useful. The ultimate goal is to have just
themselves in many
the thread attributes. They are useful for fine-tuning, enough threads to process incoming data with no different ways. When a
like setting a thread stack size, but that's beyond pauses, and no threads idling for too long. writer scrolls the buffer
the scope of this Core Tech. The third argument is a halfway behind the reader's
pointer to the thread function, and the last one is the Taming concurrency back, you get two pluses
value that thread_fn accepts. Now, imagine a thread that accepts a fixed-size buffer in a row.
Quite often, you just want to spawn a thread to and continuously fills it with some symbols. That's a
do some background processing, then forget about poor man's imitation of a complex rendering task:
it. In this case, you should mark it as "detached" #include <pthread.h>
with pthread_detach(). When a detached thread struct shared_buf {
terminates, the system reclaims its resources char *data;
automatically. long size;
If you want the result, use pthread_join(). This };
call waits for the thread to terminate, then grabs the static void *render(void *arg)
void * value passed to return/pthread_exit(). There {
isn't much sense in calling pthread_join() just after struct shared_buf *sbuf = (struct shared_buf *)arg;
pthread_create(), as we do here. But if the thread has char *buf = sbuf->data, c;
already finished, pthread_join() collects its exit value int i;
and returns immediately. for (;;) {
It is important to call either pthread_detach() or c = buf[sbuf->size - 1];
pthread_join(). By default, all threads are joinable, and for (i = sbuf->size - 2; i >= 0; i--) {
if you forget to collect the result, you'll get a "zombie buf[i + 1] = buf[i];
thread" wasting the system's resources. This being }
said, all threads typically lasts only until their parent buf[0] = c;
process terminates. If you want children to outlive }
their parents, you probably want processes, not return NULL;
threads (LV019). }
How many threads to spawn depends on the The main() function initialises data in the buffer and
task's nature. Some do best with one thread per CPU dumps it to stdout, acting as a viewer:
core. sysconf(_SC_NPROCESSORS_ONLN) reports #include <stdio.h>
how many processors are currently online. Yet this #include <stdlib.h>
doesn't guarantee that each thread will really run on #include <time.h>
a dedicated core. pthread_setaffinity_np() may help, #define LINE_SIZE 24
int main()
{
Threads and signals struct timespec t = {0, 100000000L};
As you know, you can send a signal to a Unix process. struct shared_buf sbuf;
What happens if you send a signal to a multithreaded pthread_t thread_id;
application? Recall that signal handlers are shared among char buf[LINE_SIZE + 1];
all threads in a process. So, the kernel delivers the signal to
int i;
a random thread. It is possible, however, to set per-thread
signal masks with pthread_sigmask(3). This way, you buf[LINE_SIZE] = '\0';
can choose which threads are going to respond to which for (i = 0; i < LINE_SIZE; i++) {
signals. Say, if your program uses SIGHUP to reload a buf[i] = (i % 3 == 0) ? '+' : '-';
config, you may want to handle it in the main thread. Just }
block SIGHUP in all worker threads, and you're done. The
sbuf.data = buf;
kill command will deliver this signal to the main thread.
It is also possible to send a signal to specific thread. the sbuf.size = LINE_SIZE;
tgkill(2) system call does this. Libc usually wraps it, or its pthread_create(&thread_id, NULL, render, &sbuf);
older variant, tkill(2), as the pthread_kill(3) function. For pthread_detach(thread_id);
instance, when you cancel a thread with pthread_cancel(), for (;;) {
the target thread gets a SIGCANCEL signal. It is usually the
printf("%s\n", sbuf.data);
first POSIX real-time signal (32). This means, SIGCANCEL is
delivered before any other real-time signal pending. nanosleep(&t, NULL);
printf("\033[1A");
www.linuxvoice.com 95
CORETECHNOLOGY NETWORKING
Non-atomic comparisons pthread_mutex_lock(&sbuf->mutex);
may result in two lock = false lock = false /* The buffer is updated */
threads holding a "lock" pthread_mutex_unlock(&sbuf->mutex);
simultaneously. Use }
dedicated primitives, Luke. ...
lock == false?
}
lock == false? int main()
Time {
No
...
lock = true
sbuf.data = buf;
lock = true sbuf.size = LINE_SIZE;
pthread_mutex_init(&sbuf.mutex, NULL);
...
Both threads "hold" the lock now for (;;) {
pthread_mutex_lock(&sbuf.mutex);
printf("%s\n", sbuf.data);
pthread_mutex_unlock(&sbuf.mutex);
} ...
return 0; }
} pthread_mutex_destroy(&sbuf.mutex);
The \033[1A ANSI sequence moves the cursor return 0;
one line up. It's often used to animate progress bars }
in Linux console tools. Perhaps I wanted this code Both threads now grab the mutex before doing
to produce a creeping line effect. And it really does anything to the buffer and release it after that. Keeping
this, but more often than not, the animation appears locks for too long isn't a good idea, as it prevents other
garbled (see the screenshot). What's wrong? threads' progress. Locks can become performance
Shared data is the culprit. There is no bottlenecks when contended, so one should design
synchronisation between threads. So, the reader may synchronisation very carefully. The code path between
get a half-baked frame, and the writer may change the lock being acquired and released is often called a
the buffer while it's being read. This brings us a whole "critical section", as only one thread can execute it at
new set of issues related to atomicity and races. It's the given time. This being said, it's better to think of
complex topic, but let us scratch the surface. mutexes as a way to protect specific data, not code.
We need to ensure that only one thread accesses So we embedded the mutex in the buffer structure.
the buffer at given moment. Pthreads has many Having no synchronisation for shared data is bad,
"synchronisation primitives" to achieve this. Perhaps but having it wrong is worse. Imagine you have two
the simplest one is a mutex. mutexes, A and B. Thread 1 acquires mutex A and
Mutex stands for "mutual exclusion". It's something waits for B. Thread 2 does the opposite. Neither one
a thread can lock and unlock. Many threads may
try to lock a mutex simultaneously, but only one will
Green thread
succeed. The others will be blocked until the mutex is
released. A mutex is like a flag: You may also come across the term "green threads". These
int locked = 0;
are threads that exist completely in userspace, in some
language runtime (think Java or Erlang). The name may
if (!locked) {
read as if they are lightweight, which is often the case. But
locked = 1; in fact, it came from the Green Team at Sun Microsystems,
/* Proceed safely */ which did the threading in the Java VM.
} Green threads aren't real threads as we've discussed;
However, mutex operations are atomic, while this usually, they don't exploit multicore CPUs. They offer
cooperative multitasking. This means that the thread itself
code is not. A potential race is depicted on the figure. yields control to the scheduler when it thinks it's time.
Races are often non-trivial to reproduce and debug. Quite often, this is transparent to the programmer. This
So, please use dedicated primitives, not homebrew simplifies things a bit: don't make critical sections; simply
flags, in your Pthreads code. don't reschedule when you are not ready to. Yet it also adds
Fixing a race in our program is straightforward: responsibilities: if a thread keeps running for too long, it
makes the others starving. If it blocks on I/O, it blocks the
struct shared_buf {
whole program. This sounds similar to how asynchronous
... I/O works (LV016). In fact, green threads are often bound to
pthread_mutex_t mutex; event loops internally.
}; Some languages, like Java or Erlang, provide green
static void *render(void *arg)
threads out of the box. Others may have special libraries,
such as Python's Gevent or Greenlet. Green threads aren't
{
a substitute for real ones, but they do have their usages. At
... least, they run even if the OS lacks threading support.
for (;;) {
96 www.linuxvoice.com
NETWORKING CORETECHNOLOGY
is going to proceed, and they will remain in this state Nothing lasts forever,
forever. This situation is called a deadlock. To avoid it, except two threads waiting
always acquire multiple locks in the same order. Thread 1 for each other to release
a lock.
More primitives Lock A Lock B
Quite often, you have threads that only write shared
data and threads that only read it. Any number of
threads may read data simultaneously, but any writer Thread 2
must have exclusive access to the resource. A mutex
is a bad choice here, as it won't allow multiple readers. Lock B Lock A
Read-write locks (or "rwlocks" for short) are the
answer. Pthreads implements them as pthread_lock_t
and have a bunch of functions (pthread_rwlock_*()) It is also possible (yet rather difficult) to write
to operate on them. Readers call `pthread_rwlock_ multithreaded programs lock-free. More formally,
rdlock()/pthread_rwlock_rdunlock(), so no writer the code is lock-free if it keeps at least one thread
may change the data while it's being read. Writers use progressing towards the end result. That's not the
pthread_rwlock_wrlock() to obtain exclusive access. case with locks. Imagine the kernel decides to re-
Some may say that rwlock is just a specialisation of schedule a userspace thread holding the lock. No
another primitive, a conditional variable (or "condvar"). other thread can make progress in this case until the
An rwlock is basically something with two counters kernel resumes the thread and the lock is released.
and a mutex to protect them. To get a read lock, you Lock-free programming is made possible with low-
acquire the mutex and check if the number of writers level atomic operations, among other things. Usually,
is zero. If yes, you increase the readers count, release reading and writing a naturally-aligned simple type
the mutex and proceed. If not, you release the mutex (such as a 4-byte aligned 32-bit integer) is atomic.
and block until the writers count reaches zero. There are also ways to increment and decrement an
That's exactly how a conditional variable works: it integer, or exchange two values atomically. Finally,
is always associated with a mutex. You grab it and many CPUs provide atomic compare-and-swap (CAS)
check the condition. If it's false, you call pthread_ operation. On x86, it's lock cmpxchg. That is, it's
cond_wait() to atomically release the mutex and put a possible to check a target value against the expected
thread to sleep. When the condition changes, another one and update it if they match without any other code
thread signals an event to one or all waiters with intervening. This could be used to update shared data
pthread_cond_signal() or pthread_cond_broadcast(), in a lock-free fashion.
respectively. In waiting threads, pthread_cond_wait() Multithread programming is hard, and lock-free
grabs the mutex again and returns. The thread programming is even harder – you don't do it on a
should now re-evaluate the condition and either whim. The principle that it delivers better performance
proceed or go back to sleep. The main guarantee isn't true in all cases either. So, you should evaluate
that a conditional variables provide is again atomicity. carefully what your performance bottlenecks are – if
Releasing a mutex and putting a thread to sleep these are locks, you may try to re-implement parts
occurs as a single operation. There's no chance for a of your algorithm lock-free. How to do it properly is
wake-up signal to get lost in between. beyond the scope of this Core Tech, however.
Command of the month: stap
In multithreaded programs, the order of statements in contended locks. SystemTap comes with many
the sources is different from the order of execution. examples found in /usr/share/doc/systemtap/
So, merely looking at the code doesn't reveal the examples. One of these is the process/futexes.stp
sequence of the program's operations. You want script. It traces the futex syscall and dumps how
some tools for live introspection, also called dynamic many times the thread had to sleep waiting for the
code analysis. lock, and for how long. Lower values are generally
SystemTap is one of such tools, and stap is its main better. Here's what I got on my Fedora 23 box for our
executable. SystemTap enables you to write small sample program:
scripts called probes, and attach them to various $ sudo stap -c ./sample futex.stp
events happening in userspace and in the kernel. ...
It was designed to be safe for use on production sample[3006] lock 0x7ffcc7301fe0 contended 79 times, 26
systems. So, you can study non-trivial situations such avg us
as deadlocks, in the wild. Make sure you have the kernel debug info installed.
While SystemTap can do many things, the reason If you want SystemTap covered in one of the future
we mention it here is that it helps to determine Core Teches, please drop us a line.
www.linuxvoice.com 97
/DEV/RANDOM/ FINAL THOUGHTS
Final thoughts, musings and reflections
MY LINUX SETUP
Nick Veitch
was the original editor
of Linux Format, a
SEND US YOUR
role he played until he
got bored and went
to work at Canonical
instead. Splitter!
M
any lifetimes ago, it seems to me,
I was the editor of a magazine
called Amiga Format. It was a
DESKTOPS!
magazine that was so stratospherically
successful that we could do pretty much
anything. The kind of problems about
appeasing advertisers, compromises over
paper quality, penny-pinching over the best
writers and a myriad more that were the
constant juggling act of most magazines
simply didn’t apply any more. We made
something brilliant, and that enabled us to
remain making something brilliant. It is a
situation few people in a commercial
environment ever find themselves in, and I
was grateful to be a part of it.
It is precisely this environment that drives
1
some of the success of open source. Make
something good = people want it. More
3
users generally leads to the ability to do even
more good stuff. The trick of turning open
source into a revenue-generating business
seems to be adding on the money-spinning Take a photo of your desk.
in a way that doesn’t bring everything
tumbling down. And answer these awesome
The exit of Frank Karlitschek and other
2
questions.
developers from OwnCloud tends to suggest
this is a balancing act that isn’t always easy
to navigate. Though nobody has given
specific reasons for most of the developers > What version of Linux are
upping sticks, the comments that have been you currently using?
made (eg http://goo.gl/kFYUvr) suggest Tell us a little about the > What desktop are you using
things we can see.
that ‘investors’ were felt to have undue at the moment?
influence over the roadmap. The rapid > What was the first Linux
emergence of Frank’s new project, setup you ever used?
NextCloud, and the employment of pretty > What Free Software/open
much anyone from OwnCloud who wanted Then send your photos and text to: source can’t you live without?
to come underlines where the value of an > What do other people love
open source business really lies. geekdesktop@linuxvoice.com but you can’t get on with?
98 www.linuxvoice.com
This is what we’ve done in the last 24 issues.
Subscribe to the next 12 from just £38.
Every subscription includes access to every PDF, ePub and audio edition we’ve ever published.
shop.linuxvoice.com