Plaintext
PROUDLY INDEPENDENT SINCE 2013
Virtualisation
Run a virtual
system inside
your Linux
September 2016 www.linuxvoice.com machine
ARDUINO
HARDWARE
HACKING
P28
SPEED UP YOUR
INTERFACE!
ULTIMATE
SPEEDUPS SPEED UP
YOUR INTERWEB!
Get maximum performance
from your Linux machine!
SPEED UP
PUBLISH A YOUR SERVER!
MASTERPIECE
WITH CALIBRE
P22 REVIEW:
FEDORA 24
WORKSTATION
P42
BURSTING
ANSIBLE Harden many servers at once the lazy way
WITH AWESOME
APACHE Find out what’s going on with log files and lots of graphs TUTORIALS!
ELASTIC BEANSTALK Deploy a web app to Amazon Web Services
FREEDOM FOR FORMATS! ONE OF US! ONE OF US!
S e pte mb e r 2016 £5.9 9 Pr inte d in the UK
FRIDRICH STRBA HELLO, LINUX
The Document New to Linux? Start
Liberation Project – here – our introduction
making sure you can to the finest operating
read your own files. system in the world.
RETROPIE › XATTRS › DIGIKAM 5 & MORE!
FOSSTALK LIVE
2016
A free evening of live Linux Podcasts
Saturday 6 August 2016
Plus Stuart Langridge and Dave MegaSlippers
http://www.fosstalk.com/tickets
The Harrison, 28 Harrison Street, Kings Cross, London, WC1H 8JF
Doors 5pm
ISSUE 30 WELCOME
FULL SPEED AHEAD!
The September issue What’s hot in LV#030
ANDREW GREGORY
This month’s group test on
desktop virtualisation has helped
me turn my PC into hundreds of
little PCs. Why run one distro
when you can run all of them at
the same time?
BEN EVERARD p50
Long-term Linux user and
best-selling author Ben is GRAHAM MORRISON
usually found knee-deep in I’m planning on writing a book on
either Python code or a the finer points of Belgian beer
tangle of wires. making with the help of Andrew
Conway’s tutorial. Now I just
W
need to finish a few more batches
hen Les Pounder sent in the Arduino feature (page 28), he of golden brew to perfect the art.
included the note, “I forgot how much I enjoyed hacking p68
with Arduino.” I played with one of my Arduinos this
month, and had a great time as well. Personally, my enjoyment MIKE SAUNDERS
comes from the simplicity of the device: there’s little extraneous I’m planning the next step for my
hardware, and instead of an operating system, there’s a bootloader. If own operating system, MikeOS,
you need any more features, you have to build them yourself. By where it takes over the world of
being stripped bare, the Arduino forces you to think about exactly web servers. I’ve been reading
what you need your computer to do. about log file analysis to see
Don’t worry, I’m still a Linux user at heart. This month I’ve been what the competitors can do.
p82
tuning my machines for maximum performance, and that means
getting to know exactly how it’s using the resources it has – it’s a
little like programming an Arduino.
Ben Everard
Editor, Linux Voice
Linux Voice is different.
THE LINUX VOICE TEAM
Linux Voice is special. SUBSCRIBE
Editor Ben Everard
ben@linuxvoice.com Here’s why… ON PAGE 56
Deputy editor Andrew Gregory
1 At the end of each financial year we’ll
andrew@linuxvoice.com
give 50% of our profits to a selection of
Editor in hiding Graham Morrison organisations that support free
graham@linuxvoice.com software, decided by a vote among our
Editor at large Mike Saunders readers (that’s you).
mike@linuxvoice.com
Games editor Michel Loubet-Jambert 2 No later than nine months after first
michel@linuxvoice.com publication, we will relicense all of our
Creative director Stacey Black content under the Creative Commons
stacey@linuxvoice.com CC-BY-SA licence, so that old content
can still be useful, and can live on even
Malign puppetmaster Nick Veitch
after the magazine has come off the
nick@linuxvoice.com
shelves
Editorial contributors: Andrew
Conway, Mark Crutch, Sebastian 3 We’re a small company, so we don’t
Göttschkes, Vincent Mealing, Simon have a board of directors or a bunch of
Phipps, Les Pounder, Mayank Sharma, shareholders in the City of London to
Amit Saha, Valentine Sinitsyn keep happy. The only people that matter
to us are the readers.
www.linuxvoice.com 3
CONTENTS ISSUE 30 SEPTEMBER 2016
Contents
If at first you don’t succeed, Google the problem, then go back to using Debian.
Regulars Cover Feature
News 06
PS3 users in the US get a payout, there are
new releases of KDE and Digikam, Linux 14
desktop use passes 2%, and we’re taking
over Microsoft’s Azure cloud platform too.
Distrohopper 08
Featuring one of our favourites at the
moment – Manjaro 16.06 (aka the
excellence of Arch without the hassle).
Speak your brains 10
What’s going on a the heart of government
(in software terms, that is), an indexing
solution, love for Fedora and vote of thanks.
Subscribe! 12/56
Never again fell the gut-wrenching pain
of missing out on your favourite Linux
magazine. Subscribe today!
FOSSPicks 58
Your bog-standard laptop can be a supercomputer if you just clean away
Free-range, grass-fed software, slaughtered the cruft and make a few simple tweaks – here’s how…
at 18 months old and hung for 35 days,
cooked on the bone and tasted thoroughly
by Graham Morrison. Interview Feature
Core Tech 94
Files contain metadata including
permissions – but what if you could add 34 22
your own metadata to make files even more
useful? With xattrs, you can!
Linux inside 98
The SpaceX Falcon 9 rocket – bringing the
cost of space missions down with the help
of the Linux kernel.
Fridrich Strba Welcome to Linux
SUBSCRIBE How the Document Liberation Project is What is this crazy little thing called Linux? It’s only the
ON PAGE 56 working to free your data. safest, smartest and most secure way to run a computer!
FAQ Group Test
Flatpak 32 Virtualisation apps 50
Distro-independent packaging Test new distros, write code in a
has long been the dream for Linux clean environment or just mess
– is Flatpak the chosen one? about with a virtual machine.
SECRETS OF CALIBRE
TURN TO PAGE 26
4 www.linuxvoice.com
ISSUE 30 SEPTEMBER 2016 CONTENTS
Feature Tutorials
28
Calibre 66
Edit your own personal online newspaper to
get the news you want without any of the
rubbish Game Of Thrones memes.
Build all the things! Publish books
… and staying with Calibre, why not write
68
How a humble microcontroller kick-started the march of the makers. your own fantasy epic with free software
and publish it yourself?
Reviews Raspberry Pi 74
Use a barcode scanner and little bit of
Python to control characters in Minecraft.
Fedora 24 42 it’s like Pokemon Go but better!
Our verdict on one of Ansible 78
the big beasts of the Keep on top of server security, updates
Linux desktop – the and configuration by controlling loads of
community-run derivative machines all at once.
of Red Hat, by the people,
for the people. Coding
RetroPie 43 FritzBox 44 Digikam 45
Play retro games on a humble Replace your ISP’s rubbish router Photographers are spoiled for
Raspberry Pi with this fabulous with something that’ll give you choice on Linux – here’s one of
emulation platform. more control over your network. our favourite apps for snappers.
Amazon Electric Beanstalk 82
Deploy a web application with Amazon’s
super-scalable hosting platform.
Gaming on Linux 46 Books 48
To prepare you for the book reviews page, here’s When zombies attack and the world is in flames Apache log files 86
a quick review of a zombie survival game – sadly, you’ll be glad you learned how to make little Find out what’s going on on with all those
Arduinos are not included in the gameplay. Arduino- and Raspberry Pi-powered devices. blinking lights and whirring hard drives.
www.linuxvoice.com 5
NEWS ANALYSIS
NEWSANALYSIS
The Linux Voice view on what’s going on in the world of Free Software.
Opinion
Seperate yet united
Open Source developers aren’t working for free – they’re working for themselves.
Simon Phipps
is ex-president of the Open Source is thus what happens when
Open Source Initiative
and a board member people choose to work together on the same
of the Open Rights
Group and of Open codebase rather than working separately
Source for America.
design, they are not “working for free” or community norms. But beyond those
“donating their work” so much as they are organisational essentials, an open source
“participating in co-development”. community is inherently neither a non-profit
S
o you’re thinking of devising an open Open source is thus what happens when or a for-profit organisation; profit is an
source strategy for your business. several different people choose to work orthogonal concept.
The most important concept to together on the same codebase rather than Some of the contributors might be
understand when devising an open source working separately, liberated to do so by the present for direct profit from the code, but
strategy is the separation of community four freedoms that the licence protects. the community as a whole is actually a
interests and commercial interests. What Each of them is there for their own reasons. mesh of different participants, all with their
does that mean? There is no pooling of funds to pay for work own motivational models and all paying their
Open source can be defined as the to be done, because everyone is solely own way to achieve them outside the
co-development of software by a community responsible for their own costs. The only context of the community. Communities do
of people who choose to align a fragment of funding pool a project needs is to cover the not have business models. If the
their self-interest in order to do so. The costs no-one should bear alone, such as motivational model of some participants
commons in which they work contains project infrastructure and administration or involves business, that harms no-one. But
software free from usage restrictions with for tendering for paid work that is in the the community itself is about the liberty to
guaranteed freedoms to use, study, modify interests of all. align interests, not about the presence or
and distribute it – “free software” – because absence of profit – that is purely the domain
of an OSI-approved copyright licence (an It’s not about the money of the participants privately.
“open source licence”). All such licences As a consequence of this connected-but- Thus in a healthy open source community,
grant permission to anyone to use, improve separate status, there is no fiscal power that I’m free to maintain my privacy around my
and share the licensed source code and the any contributor holds over others – no-one motivations and how I’m funding my
object code that it produces. The licence has the right to tell the others what to do. involvement if I wish. On the other hand, I’m
creates a safe space for collaboration, where There will sometimes be a non-profit able to work in an environment of
everyone has permission in advance to organisation for administrative reasons, and transparency where all the code is known, all
innovate however they want. often a technical co-ordinating group to its origins are known, all its defects are
The community members each work at make sure releases can be scheduled and potentially known.
their own expense in order to achieve a duplication can be avoided. To maintain That combination of transparency with
shared outcome that benefits all, including trust, enable development transparency and privacy is, in my opinion, a primary
themselves. When they create an permit individual privacy, it’s reasonable to characteristic of an open source community.
enhancement, fix a defect, participate in a devise and apply governance that asserts Communities without the seminal Apache
Software Foundation rule “if it didn’t happen
There is no fiscal power that any open source as a matter of open record, it didn’t happen”
are closed, regardless of the software
contributor holds over others – no-one has license. Open source is about transparency
the right to tell the others what to do at the community level but also about the
privacy of the individuals involved.
6 www.linuxvoice.com
ANALYSIS NEWS
KDE • Linux Mint • Slackware • Digikam • Desktop • 64-bit Linux • PS3 • Azure
CATCHUP Summarised: the biggest news
stories from the last month
KDE Plasma 5.7 released So long, 32-bit distros… Linux Mint 18 “Sarah”
1 KDE Plasma 5.7 was released 2 If you’re running a PC or 3 released
on 5 July 2016, and brings laptop bought within the Based on Ubuntu 16.04,
about extended Jump List Actions in last five years, chances are that it Linux Mint 18 is a long term support
KRunner for quicker access to certain has a 64-bit chip inside. Now some (LTS) release that will be supported
tasks within an application. Also, the distro makers, such as Ubuntu and until 2021. The Mate edition includes
Agenda view in the Calendar is back, OpenSUSE, are considering dropping version 1.14 of the Mate desktop,
while many improvements have been support for 32-bit processors. As which sports better GTK 3 support and
made to the Volume Control applet OpenSUSE Chairman Richard Brown better management of Python Caja
(such as the ability to control volume on says: “32-bit support doubles our extensions.
a per-application basis). testing burden (actually, more so – do
Accessibility is better in the new you know how hard it is to find 32-bit
version as well: Breeze icons within hardware these days?)”. Of course,
applications are now tinted depending some smaller distros will continue to
on the colour scheme. support the older hardware.
Linux finally reaches Slackware 14.2 released Sony pays out to angry
4 over 2% of desktop 5 We like Slackware. It’s very 6 PS3 owners
market share Unix-like at its heart, eschews At launch, the PlayStation 3
We’ve all dreamt about the “year of complicated package management let you install Linux, which was a selling
Linux on the desktop” for as long as systems for a simple tarball-based point for a small but vocal number of
we can remember, but things are approach, and provides an alternative users. Ridiculously, Sony later issued a
starting to go in the right direction at init and process management system firmware update removing this feature
last. As of early July 2016, according to to Systemd (not that we hate Systemd, – and you needed to install this update
W3Counter stats, Linux now accounts but we appreciate diversity). This new to play newer games.
for 2.8% of computer users accessing release includes 4.4.14 and GCC 5.3, Some Linux fans took Sony to court
the web. It may not sound like much, along with Perl 5.22.2, Python 2.7.11, because of this move, and now it looks
but it’s still way better than the ~1% Ruby 2.2.5, Subversion 1.9.4, Git 2.9.0 like Sony will pay out $55 to each
around which it was hovering for many and KDE 4.14.21. To download it or buy user who had Linux installed and then
years – fingers crossed there’s more a boxed set and support the project, see couldn’t use it. Not much then, but at
succcess to come. www.slackware.com. least it’s something.
Digikam 5.0.0 has been A third of Azure VMs are
7 released 8 now running Linux
It has been two years in the Microsoft and Linux really are
making, and a huge amount of work, making for interesting bedfellows these
but Digikam 5.0.0 is finally here. Most days. Previously, only a quarter of virtual
notably, this photo management tool machines on Microsoft’s Azure cloud
has been updated to Qt 5, with all Qt infrastructure were running Linux, but
4 and KDE 4-related code removed. In now that has gone up to a third. Cynics
addition, the app uses more Qt libraries would say that Microsoft is simply
and fewer KDE ones, making it easier going where the money is, and Linux is
to port to other OSes. Digikam 5.0.0 is huge in the cloud, but we’re happy to
also considerably faster than previous see the company being a bit more open
releases, thanks to optimisation work to other platforms now. It’s a lot better
done by a Google Summer of Code- than the terrible Steve Ballmer days.
sponsored developer. Sunlit uplands are ahead!
www.linuxvoice.com 7
DISTROHOPPER LINUX DISTROS
DISTROHOPPER
What’s hot and happening in the world of Linux distros (and BSD!).
Linux Lite 3.0
No fuss, no nonsense.
T
his no-nonsense distro has recently
released version 3.0, based on the
Ubuntu 16.04 LTS codebase for
added stability. There’s been some
considerable updates since 2.8, including
new themes, a new login manager and an
overhaul of the very sleek and easy-to-use
“Lite Software” software installation
manager, along with the 16.04 package
updates. While it clearly brands itself as a
light distribution and does earn its name in
this regard, it is perhaps better to look at
Linux Lite as an extremely complete desktop
environment, just without all the bulky and
useless bits. This is made immediately Linux Lite’s approach seems to have struck a chord, as it is growing in popularity.
evident through the use of Xfce, which takes
a similar approach in trimming things down thus has an appeal that most lightweight issue with. As such, this fits in nicely with the
without having a negative effect on usability distros lack, offering no bloatware without idea of a system for people who just want to
and appearance. sacrificing functionality. get things done, with the distro staying well
While Linux Lite does run on very modest The distro is also extremely easy to set up out of the way, not being demanding in
hardware, the minimum being a 700MHz and to maintain, with the ability to get going terms of configuration or resources. While
processor and 512MB of RAM, it is also well in minutes and things like updates, software Linux Lite doesn’t break the mould in any
suited to those with high-end hardware who installation, version upgrades and driver meaningful way, it’s hard to think of a better
simply want to use the extra horsepower for installations done through a simple unified alternative for those looking for a solid
things other than desktop effects. Linux Lite GUI that entry-level users would have no workhorse distro.
Manjaro 16.06
Arch for human beings.
M
anjaro has, for a long time, been Pacman package manager, a new “Manjaro
the distro of choice for those Settings Manager” to install and remove
looking to delve into the world of different kernels should 4.4 not suffice and,
Arch Linux without feeling overwhelmed. for the KDE version, a KCM module to
This latest stable version of Manjaro, integrate the settings manager into Plasma The Manjaro developers have described the
dubbed “Daniella”, ships with Xfce 4.12, and 5’s native settings, as well as updating Xfce flavour as the “flagship” edition.
while this desktop environment remains the Plasma itself to version 5.6.
default choice for this distribution, there is It remains an extremely attractive distro, once installed. While maintaining Manjaro
also the KDE Plasma flavour, as well as the not only for its rolling release model (for requires a bit more knowledge than using,
Net edition, which comes with no those who prefer it) and “Arch made easy” say, Ubuntu, and wouldn’t be worth
preinstalled desktop environment. philosophy, but also for a myriad of other recommending to someone taking their first
The biggest changes from 15.12 include reasons such as its easy installation, steps into Linux, non-technical users could
an update to kernel 4.4 LTS, updates to the user-friendliness and “out of the box” usage use it without issues.
8 www.linuxvoice.com
LINUX DISTROS DISTROHOPPER
News from the *BSD camps
What’s going on in the world of FreeBSD, NetBSD and OpenBSD.
N
etBSD is celebrating the 50th
release of the pkgsrc package
management system with a series
of developer interviews to mark the event,
which can be found on the project blog:
blog.netbsd.org/tnf. It was initially released
in 1997 when it was forked from the
FreeBSD Ports collection, and both of these
remain the default package management
systems on each operating system. At the
time of writing, pkgsrc contains over 17,000
packages and is available on multiple
systems. NetBSD itself saw the first bugfix
release of the 7.x series in the form of
version 7.0.1. Meanwhile, the system’s
firewall software pfSense has seen a major
point release in the 2.3 series, incorporating
over 100 changes. A ZFS partitioning option on the FreeBSD 11 installer, alongside other options.
Meanwhile, FreeBSD has now joined Linux
as the latest operating system to be has offered BSD (and Linux) for a long time be supporting the BSD clients directly so as
included alongside Windows in Microsoft’s now, this step shows Microsoft recognising to “remove the burden” from the FreeBSD
Azure cloud computing service. While that, in terms of market share, Windows Foundation, to which it will be sending code.
previously customers could upload custom simply can’t compete with Unix-like Also within the FreeBSD world, the ZFS
images, FreeBSD 10.3 is now one of the operating systems in these applications. Fault Management Daemon has been
available images pre-built by Microsoft. The company, which has been making a integrated into FreeBSD 11, adding
While Amazon’s competing Web Services series of similar moves in recent years, will increased functionality for the filesystem in
what seems to be a growing trend towards
FreeBSD has joined Linux as the latest its support and adoption. On the DragonFly
BSD fork, its kernel now supports NVMe
OS to be included alongside Windows in (Non-Volatile Memory Express), meaning
Microsoft’s Azure cloud computing service that users can now make use of this modern
storage standard.
Remix OS for PC
Linux has now dominated pretty much every other space out
there, but still only manages third place on the desktop. So how
about reversing that by getting a mobile distribution that almost
everyone is familiar with and bringing it to PCs? Remix OS does
just that, by bringing Android to the desktop.
Remix’s performance on x86 leaves ARM devices in the dust,
and the system itself is pretty seamless, running Android
applications on the desktop. While using things like office
applications is far more productive than on a touchscreen,
seasoned Linux users may find it less convoluted to install a
lightweight Linux distro with a vast array of available software
specifically designed for the desktop.
Though Remix OS is unlikely to be the one to achieve
mass-market desktop Linux glory, as it’s becoming more likely
that Chrome OS and Android are to become one and the same in
the future, either case would be a bit of a hollow victory. While
technically, the Linux kernel would be running on a lot of
machines if such a thing were to gain traction, it would be at the
cost of many of the values and freedoms associated with GNU/
Linux. Remix OS itself is free but not libre, while the Google
experience would also be somewhat of a walled garden. Remix OS can run native Android applications on the desktop.
www.linuxvoice.com 9
MAIL YOUR LETTERS
YOUR LETTERS
Got an idea for the magazine? Or a great discovery? Email us: letters@linuxvoice.com
LINUX.GOV
Again a great magazine! heard anything from them since. Rather than
Some time ago I heard about Collabora doing a speculate here, I think the best thing to do is to get
government office suite. Is that live now? What about the in touch with our chums at Collabora and find out
STAR ODF requirement? from them what’s going on. One thing that we can
LETTER www.collaboraoffice.com/solutions/collabora-govoffice be sure of, however, is that progress will be slow:
Richmond Makerlabs, Ham United Group Free Software implies a cultural shift that’s at least
as important as any financial saving, and as the
Andrew says: Collabora did indeed sign an Cabinet Office has now started using Google Docs
agreement with the UK government in October (!) it looks like we won’t be ‘taking back control’ any
2015 to provide office software, but we haven’t time soon.
Free software for the public
sector makes so much
sense; it’s a shame it has to
happen so slowly.
#INCLUDE LV_FULSOME_PRAISE
In his letter, Ken Riley asked for an index of Linux Voice
articles. Recoll works very well on a downloaded collection
of .pdf files, including other formats. Recoll builds a
full-text index that you can query; it includes word
proximity and stemming, and has an easy-to-use GUI. You
can set Recoll for periodic automatic updates of the index.
Recoll is available in the Debian repository and it has good
help documentation.
Though good, Recoll can’t overcome typos and dodgy
copy editing.
Regards, Andrew Shead
Andrew says: Thanks Andrew. It sounds like this
might make a decent stopgap solution, which, to be Sadly, Recoll is not a platform puzzle game set on Mars. But it
honest, is what we’re all about. can help you make an index out of PDF files.
10
YOUR LETTERS MAIL
SO LAST YEAR
We get it: you guys like Ubuntu. It’s the best thing since
sliced bread. But the world has moved on. Unity is broken,
and one of the biggest features of the last release was
that it doesn’t spy on you anymore! As if we should even
need to be told that!
Fedora is and always has been the One True Way. It’s a
proper desktop distro for people who want to get things
done (sorry Arch users, but most of us have better things
to do than read the wiki before every piffling little update).
It respects your freedom, and it Just Works. Carry on.
Iain McAllister
Andrew says: It chills me to the bone when
Graham goes offline when he’s updating Arch; we
just don’t know when we’ll ever see him again,
or what mental state he’ll be in when he does Remember kids:
resurface. Ubuntu has never done that to me, and those readers who haven’t tried it in a while, for its spying is bad.
if it did, I know that a quick Google search would excellent community as much as anything (which Unless you’re the
bring up the answers to any problems instantly. incidentally is by far the best feature of Ubuntu). Home Secretary.
It’s worth looking into Fedora for the benefit of Watch this space for more Fedora action!
FREEDOM ISN’T FREE
Thanks for the interview with Jim Killock; I didn’t agree James Olssen, Michigan
with everything he said, but it was nice to get a few proper,
thought-out reasons why surveillance is bad. My own Ben says: Thanks James. In these bonkers, vitriolic
thoughts are that it wouldn’t matter a damn whether the times, it’s reassuring that we can all get along
security services can read my email if I get blown up by a despite our differences of opinion. Every time
terrorist, and I’m really not interesting enough to spy on, someone earnestly compares the UK with North
but I do appreciate seeing the counter argument Korea I sigh inwardly, because it’s just not that
presented in a sensible manner. I haven’t changed my simple. The Open Rights Group is making the case
mind, but I have at least thought about the implications, in a proper, nuanced manner, and it deserves our
and that’s to your and his credit. support.
www.linuxvoice.com 11
SUBSCRIBE
Subscribe
shop.linuxvoice.com
Get your regular dose
of Linux Voice, the
magazine that:
Gives 50% of its profits
back to Free Software
Licenses its content
SUBSCRIBE TO CC-BY-SA within 9 months
US/Canada subs prices
TODAY! 1-year print & digital: £95
12-month digital only: £38
Get many pages Access our Save money on
of tutorials, rapidly growing the shop price
features, interviews back-issues archive and get each issue
and reviews – all DRM-free and delivered to
every month ready to download your door
Payment is in Pounds Sterling. 12-month subscribers will receive 12 issues of Linux Voice a year. 7-month
subscribers will receive 7 issue of Linux Voice. If you are dissatisfied in any way you can write to us to cancel your
subscription at subscriptions@linuxvoice.com and we will refund you for all unmailed issues.
12 www.linuxvoice.com
SUBSCRIBE
All subscribers get access to every
single digital back issue –
that’s about 1,000,000 words of
tutorials, reviews and free software
hackery at your fingertips
Overseas subs prices
12-month print & digital:
Europe: £85
US/Canada: £95
Rest of world: £99 DIGITAL
SUBSCRIPTION*
ONLY
*
£38
WHEREVER IN THE WORLD YOU
ARE – IT’S DIGITAL, SO THERE ARE
NO POSTAGE COSTS
www.linuxvoice.com 13
FEATURE ULTIMATE SPEEDUPS
ULTIMATE
SPEEDUPS
Get ready to push your Linux
machine into overdrive with
Ben Everard’s guide to blazing
fast performance.
H
aving a fast machine isn’t all about spending Just like all machines, computers need a tune-up
money on the latest hardware: it’s about every now and again for them to perform at their
tuning the software to get the most out of best. There’s no set list of things that you should do
the hardware you’ve got, whether that’s a Raspberry Pi to make your computer run faster – instead, there
or a high-powered server. Even the most humble of are things you can investigate, and trade-offs you can
modern computers is vastly more powerful than the make to balance performance with user experience.
machines of a decade ago, and we could quite happily Let’s banish waiting time to the annals of
run Linux at the turn of the century, so if things are computing history and make our computers as fast
starting to run like a tortoise in treacle, it can’t be the as they can be, with a new, leaner, faster environment.
hardware to blame. Strap yourself in!
Just like all machines, computers need a tune-up
every now and then for them to perform at their best
14 www.linuxvoice.com
ULTIMATE SPEEDUPS FEATURE
SPEED TEST YOUR MACHINE
If you don’t know how fast you’re going, you won’t know if you’re going faster.
The first step in trying to increase the
speed of your system is to find out how
fast it’s currently running. Once you
start monitoring the speed, you can see
what the effects of a particular change
are, and you can roll back anything that
makes things slower.
There are two basic ways of testing
a computer’s speed: benchmarking
and performance monitoring.
Benchmarking is where you run a
repeatable test under different setups
and see how the computer performed;
while performance monitoring is seeing
how your computer is performing as
you’re using it normally. The advantage
of benchmarking is that it is a perfectly
fair test, because exactly the same
thing is run every time. The advantage
of performance monitoring is that
the data is more relevant because it’s Gnome-system-monitor lets you see what’s going on with a clickable user interface.
running the exact tasks that you run
normally. the performance across different In Ubuntu, the configuration file is
The simplest benchmarking tool is systems or setups. There are a huge /etc/default/sysstat. Open this file and
GNU Time, which measures how long number of benchmarks available, change the line:
other commands take to execute. For and you can easily spend more time ENABLED=”false”
example, you can use it to test how benchmarking your system than you to
long it takes to unzip a file (which tests
processor and disk performance) with:
time tar xvf myfile.tar.gz The best benchmarking tool for Linux by far is
Using time, you can benchmark
specific tasks that you do regularly the Phoronix Test Suite (PTS)
rather than relying on general
benchmarks. could possibly save by increasing ENABLED=”true”
If you want to really push your performance, but if you’re a Then restart the service with:
benchmarking as far as you can, the performance geek, there’s really no sudo service sysstat restart
best tool for Linux is the Phoronix Test other tool that has the breadth of In the normal setup, Sysstat will keep
Suite (PTS). This is a framework for benchmarks that PTS has. logs of the last 28 days in your log
running benchmarks and comparing directory (usually /var/log/sysstat).
Real-world performance These are binary logs, so you can’t view
Our go-to tool for performance them using the usual commandline
monitoring is Sysstat (available in tools (such as grep and sed), instead
Quick Speedup: almost all distros’ repositories). Sysstat they have to be interpreted by software
Boot faster collects and stores statistics, so you that understands them. The most usual
Do you waste valuable seconds can go back and look at past tool for the job is sar.
every time you start your computer performance issues. Since Sysstat If you just enter sar at the command
with the boot menu? You can reduce
is a monitoring tool rather than a line, you’ll see the CPU usage for
the amount of time grub waits before
starting by editing the /etc/default/ benchmarking tool, it can take the current day broken down into
grub file. Change the GRUB_TIMEOUT more work to understand the 10-minute averages, but the tool
option to something smaller such as 2 output. However, it can potentially enables you to drill down much further
(seconds). You’ll need to save this tell you a lot about the real-world than this.
file and run update-grub for the
performance of your machines. You can get more up-to-the-minute
option to be picked up.
Once you’ve installed Sysstat, you data with the command:
need to make sure it’s running properly. sar 10
www.linuxvoice.com 15
FEATURE ULTIMATE SPEEDUPS
RAM, but it also slows down the
machine significantly. If you’ve ever
experienced a machine running very
slowly as you switch from one
application to another, this is likely due
to the process of changing the bits of
data between RAM and swap. In
performance monitoring, it’s important
to look at both memory utilisation and
swap usage.
Passing the -r flag to sar will output
memory statistics. The most important
column here is the %memused, which
will will tell you how much of the
memory is currently in use. You can
see the swap statistics with the -S flag.
Using swap isn’t inherently bad, but if
your system’s running slowly and you
have consistently high amounts of
swap usage, you should consider using
applications that use less memory,
You can see how your resources are being used over time with gnome-system-monitor.
or look at upgrading the amount of
memory you have.
up your machine. However, you may Monitoring your system enables
find that you still have CPU cycles to you to see how it’s performing without
Quick fingers spare when running slowly because having to push it to its limits (as a
A significant amount of time, there are many things that can affect benchmark would). For example, you
the thing slowing down the computer the speed of a computer. might have a routine task that you run
is the person sitting in front of it. periodically; by monitoring the system
In general, the more you use the
mouse, the slower it is to perform
Not just the CPU resources during its execution, you can
actions, so learning keyboard shortcuts Your computer has RAM for storing the tell if it’s using more or less resources
for your desktop and most data and code of running programs – than usual. You can also use it to see
commonly used programs can this is the fastest data store on your exactly which resources are being
give you a significant machine. It also has a hard drive to stretched when a system is reaching
speedup.
store data that’s not currently in use, its limits. Depending on your setup, the
which is much larger than RAM but limiting factor could be the CPU, the
also much slower. The link between network port, the amount of memory
these is know as swap – this is a bit of or the disk space. By monitoring the
the hard drive that acts like RAM. If you resources used, you can find the area
which will continuously output the data have too much data in RAM, your that’s limiting performance, and after
every 10 seconds. machine will put some of the contents you’ve discovered where the problem
The default sar output will aggregate of RAM in swap. This enables you to lies, you can take steps to avoid the
the usage of all CPU cores, which run more software than you can fit in problem in the future.
gives a good overview of the system,
but it can hide some problems. For
example, if you have a single-threaded
application maxing out one CPU core
of a quad-core system, it would appear
as though the CPU was 75% idle, but
your computer performance would still
be bound by the CPU. You can view the
statistics for all cores separately
sar -P ALL 10
Keeping an eye on CPU usage will
give you an idea of how much of your
CPU time you’re using. If you get close
to 100% usage on one or all of the
cores when your machine is running
slowly, then you need to focus on Having trouble understanding the output of Sar? You can use Ksar to produce graphs
reducing the CPU load in order to speed showing how the usage changes over time.
16 www.linuxvoice.com
ULTIMATE SPEEDUPS FEATURE
SUPERCHARGE YOUR DESKTOP
Get your Linux PC running at full throttle.
By far the biggest thing you can do to
speed up a desktop Linux computer is
to select an appropriate desktop
environment. The two key factors to
consider are the amount of CPU time a
desktop uses and the amount of
memory it uses. You can see how
much of a problem these are for you by
monitoring your machine’s resources.
Memory usage of the desktop will be
less of a problem when you first start
the machine, but as you open more
applications (particularly memory-
hungry software such as office suites
or web browsers), you may notice that
the machine slows down as it has to
shuffle data in and out of swap space.
In general, if you want ultimate
performance, or if you’re using a slow
machine, you should avoid the big three
desktops – KDE, Gnome and Unity If you use the Phoronix Test Suite, you can compare your machines performance to thousands of
– as they’re all designed with a lot of others on openbenchmarking.org
features that help the user, but which
also use valuable system resources. parts to work quickly including logging disk caches. This memory isn’t really
Instead, try a lighter desktop, such as in and starting the file manager. There free because it’s in use, but it isn’t
LXQt, Openbox, Moksha or Mate, all
If you want performance you should avoid
of which offer a functional, working
environment without wasting your PC’s
resources on fluff.
The first simple test we do to check using the big three – KDE, Gnome and Unity
the speed of a desktop is a timed
exercise of going from the login isn’t an easy way to automate this really in use because it can be emptied
manager to opening the file manager test, so you’ll need to use a stopwatch, instantly and nothing is lost (the caches
to opening a specific file in a text editor. and the performance will differ hugely can be repopulated from disk when
This simple benchmark will test the depending on your machine. they’re needed). Some applications
snappiness of the desktop because A second, slightly more scientific that output memory statistics include
the faster the desktop performs, the test is to check the amount of memory the caches in the used memory values
quicker you’ll be able to perform all used after first booting the system. (such as the top command) while
these tasks. It requires a few different You have to check this as soon as
you’ve booted and logged in, because
otherwise the software you’re running
(other than the desktop environment)
will affect the figures. To check the
Terminal velocity amount of free memory, you can use
Graphical applications almost always sar or open a terminal and enter
take longer to load and use more free. The value you’re looking for
memory than their command line
is in the mem line under the used
equivalents. We’re not
recommending that you ditch column. The lower this is, the less
your desktop entirely, but the memory your desktop is using,
more command line skills you so the more memory there is
have, the faster you’ll be able available for other processes.
to use your machine.
The reporting of memory on Linux
can be a little confusing, because the Systemd-ui gives you a graphical front-end to services
kernel will use any spare memory for (and other init-related options) for distros using Systemd.
www.linuxvoice.com 17
FEATURE ULTIMATE SPEEDUPS
others don’t. If you’re unsure, always
check the output of the free command,
as this gives a detailed breakdown
including used memory and caches.
Beyond the desktop
The speed of a desktop is about far
more than just the environment itself.
The selection of all the different bits of
software can have a huge impact of the
speed of your system.
For example, LibreOffice is a hugely
powerful piece of software, but do
you need the full power? Editing the
document for this article in LibreOffice
uses 1.8GB of memory. In AbiWord it Plunge into the world of benchmarking and discover the exact speed your machine runs
uses 1GB. In Gedit it uses 600K and at with the Phoronix Test Suite.
in Nano it needs just 30K. Of course,
not all the pieces of software have it’s soffice (a hangover from the
the same functionality, so as with the previous name of StarOffice).
desktop environment, the task for the This will output a line for every
user is to pick the best application for running process that matches that line,
the purpose taking into account the one of which will be the grep command
Parallel lives
There are some alternatives to
amount of memory the software needs. you used to search for the process,
common software that utilise multi- The process for checking the amount so you need to pick the right one. The
core machines better than of memory a process is using is a first column on the line for the process
the traditional tools. For example, little convoluted. First you need to find will be the numeric process ID (or PID).
pigz is a parallel implementation out the process ID of the application. You can then use the top command to
of gzip that’s faster at compressing
files on multi-core machines. You
You do this by opening a terminal and find out more information about this
can also use Gnu Parallel to split running : process with:
a single command over more ps -x | grep <application name> top -p <process ID>
than one core. Here, <application name> needs to This will display various details about
be the command used to launch the the specified process, including the
software. This is often the software current CPU usage. The three columns
name in lower case, but for LibreOffice of data that are most interesting to
us are VIRT (total amount of virtual
memory used, including swap), RES
(amount of memory currently in RAM
but not including swap) and SHR (total
amount of shared memory used).
This final column brings us onto
another factor to consider when
picking desktop software: the amount
of shared memory they use. This
basically comes down to the number
of libraries an application uses, and the
important thing to consider here is that
libraries can be shared among different
applications yet only take up a single
slot of memory.
In desktop Linux terms, this means
that if you use a Qt-based desktop
(such as KDE or LXQt), all the Qt
software can use the libraries already
loaded by the desktop environment,
but GTK software will have to load
the libraries into their own chunk of
memory. In other words, if you’re
short on memory, there is an
Looking for a desktop that’s fairly lightweight, but also stylish and easy to use? We recommend Mate. advantage to sticking with software
18 www.linuxvoice.com
ULTIMATE SPEEDUPS FEATURE
designed specifically for your desktop
environment.
When it comes to memory, the big
thing that slows down your machine
is shuffling data between RAM and
swap. The Linux kernel decides what
to transfer from RAM to swap by
looking at the amount of free RAM, how
recently a bit of memory was last used,
and the swappiness setting.
Tuning swap
This last aspect – the swappiness –
can be changed to make your system
more or less ready to use swap. The
benefit of having a high amount of
swappiness is that, with more of the
data in swap, there’s more empty space
in RAM that can be used quickly if
needed. Any spare RAM is used by the
disk cache, so a high swappiness can
lead to faster disk access. The benefit When it comes to eye candy without overly taxing your machine, the Moksha desktop is hard to beat.
of low swappiness is that data isn’t put
in swap so readily so your system can more RAM and higher values better on running and which aren’t. In general, it’s
be more responsive when switching systems with less, but the performance a bad idea to stop a service unless you
programs. implications are complicated and will are sure that it’s not needed, because
There isn’t a definite answer for what depend on exactly how you use your some of them may provide behind-the-
the best swappiness value is, and the machine, so the best advice we have scenes functionality to other software.
results of different values can vary is to try changing it and see what However, you may find software that
depending on the amount of RAM on happens. you installed for a project that you no
the system and the speed of your hard longer need. You can stop a service
drive. At your service from running with:
You can see your current swappiness Your Linux system has many pieces of service <name> stop
setting with: software running in the background Doing this will only stop it in the
cat /proc/sys/vm/swappiness quietly getting on with their jobs. These current session. If you restart your
By default, this is 60 on Ubuntu and services are started automatically machine, the service will restart. You
many other distros. when you boot your computer and can stop services from starting with:
You can change the swappiness by usually keep running silently until you service <name> disable
opening the file /etc/sysctl.conf as root
Your Linux system has many pieces of
(such as with sudo nano /etc/sysctl.
conf). The swappiness is set on the line:
software running quietly in the background
vm.swappiness = 60
Change 60 to any value between 1
and 100. In general, lower values are
likely to work better on systems with turn it off. Some of the services you You can reverse this with service
have running will be important, but <name> start, and service <name>
some may just be wasting resources enable if you decide you need the
providing functionality that you don’t services in the future.
need. The method for controlling Most services are quite efficient if
Add more power services varies between distributions, they’re not under any heavy load, so
If you’ve got a large command but on most modern distros, you disabling an unused web server isn’t
that you need to run on a low can see what’s running with: going to transform your machine, but
power computer, but you also
service --status-all it should speed up your boot times
have more computers available, you
can share the processing load among If that doesn’t work, you and reduce the amount of memory
them. For example, distcc shares the should consult your distro’s used when running. As well as the
work of compiling software between documentation for information on performance increase, disabling
many computers, and Gnu Parallel can running services. unused services will also improve
split a shell script between
The output will depend on the the security of your machine, since
many machines at once.
distro you’re running, but it should there will less running software for an
make it clear exactly what services are attacker to probe.
www.linuxvoice.com 19
FEATURE ULTIMATE SPEEDUPS
SYSTEM CALLS
Upgrade your server performance without upgrading your hardware
anything you need it to. However, the
downside is that it’s memory and CPU
intensive. The main culprits in the stack
are Apache and PHP.
The Apache web server is powerful,
but do you need all the power? If you’re
trying to get the best performance out
of a server – whether this is because
you’re running a hugely popular website
that’s struggling to cope with the load
or because you’re trying to run on low-
powered hardware – then an alternative
may offer better performance, such as
the Lighttp and NginX web servers.
Update your software
Web frameworks can be complicated
and it’s not usually possible to switch
Htop provides a more comprehensive overview of your machine than the standard top utility. between different options easily to
save a few MB of memory. There are,
has to invoke PHP, which then talks to however, often tweaks you can perform
the database – often multiple times to increase performance. In many
– before creating the HTML for the cases, frameworks run on interpreted
Disable previews page that is sent to the user. This all languages such as PHP, Python or
Many file managers will show a
happens very quickly, but if a lot of Ruby, and there are changes you can
preview of the image files in a directory,
but in large folders, this can put a people are requesting pages then this make to the setup of the interpreter that
significant extra strain on constant processing of PHP and SQL will instantly boost its speed.
your computer. If you don’t need can cause performance problems. Where possible, you should always
previews, you can disable them. The However, a lot of web pages don’t run the latest stable version of
process differs a little between file
change very frequently so there’s no the language. Most programming
managers, but in Nautilus, go to
Edit > Preferences > Preview need to repeatedly calculate the HTML languages are continually updated with
and change the settings. for a web page. Instead you can performance (and security) features,
perform the processing once and then so keeping up with the latest version
send the HTML output every time a should give you the best performance.
user requests the page. You only need The same principal applies to the web
to update this cached page each time framework you’re running.
the website changes. For example, in some cases the
The easiest way to increase the HipHop Virtual Machine can run
performance of a server is to decrease Getting picky PHP faster than the standard PHP
its workload. That might sound like a Just as software selection on the interpreter. Using non-standard
strange thing to say, but it’s often desktop can make a huge difference software can cause issues with some
possible to perform the same function to the performance, choosing the right frameworks, so you’ll need to check
while at the same time reducing the server software can significantly speed carefully, but the reward can be a
significantly faster server.
Server performance can be hard
The latest version of PHP should give your to understand, and therefore hard to
web server the best performance
optimise. Only by keeping a close eye
on what’s going on will you be able
to see what’s causing problems and
amount the server has to do by judicial up a server as well. The most common what could lead to speedups. The
use of caching. Take, for example, a Linux web server stack is LAMP (Linux, performance monitoring tools we
web server hosting a PHP web app Apache, MySQL and PHP) is popular looked at earlier in this article are a
such as WordPress. Every time because it’s fairly straightforward to set good start, but you’ll also need specific
someone visits the site, the web server up and is capable of running almost tools for the environment you run.
20 www.linuxvoice.com
ULTIMATE SPEEDUPS FEATURE
WEB SPEEDUPS
Computing can be as much about the speed of the web as it is the speed of your PC.
There’s little to choose between the
performance of any of the major web
browsers any more. There is, however,
one thing that you can do to speed up
your web browsing regardless of which
software you choose: use an ad blocker.
The exact speedup you get from this
varies significantly depending on which
sites you look at, but halving page load
times is realistic for most browsing. In
addition to reducing page load times,
blocking adverts can significantly
reduce the CPU and memory usage of
the browser, which should lead to a
better desktop experience. There are a
few different ways of blocking adverts:
Install a browser addon
This is the simplest method for
blocking adverts and probably the right
option for most people. It’s easy to set
up and will work on your computer
whatever network you’re connected to.
The Brave web browser delivers a fast, ad-free experience by default.
Use an external ad blocker
You can also use an additional network The Brave web browser has an in-built desktop down. If you don’t need it at all,
device to block adverts. This method ad blocker that means you don’t need disabling this plugin can speed up your
can run better on machines with limited to perform any extra configuration. machine, otherwise you can set it to
resources and will automatically block Future versions of the software will ‘click to activate’ so that you can control
all the devices on a particular network. also enable you to pay websites to exactly when it’s used.
However, this method will only work on compensate for their lost ad revenue. Your Linux box should now be fully
a single network, so it’s not appropriate Finally, the plugins you have installed tuned for performance, so the only
for portable devices such as laptops. can make a significant difference thing left is to decide what to do with all
to browser performance. Flash, the time you’ll save now you don’t have
Use a browser that blocks adverts in particular, can slow your entire to wait around for your PC.
MOORE’S LAW
Every year, computer manufacturers find a way newer machines hasn’t held up to the law. There Despite Moore’s law, our computers don’t
to get more and more performance out of bits are still other avenues that chip manufacturers feel any faster than they did 20 years ago. Even
of silicon. There are a few reasons for this, but can explore to find extra processor speed, so an old Raspberry Pi has millions of cycles per
the primary driver is that chip engineers work computers are likely to keep getting faster, but second to use.
out how to make the transistors ever smaller. at a much slower rate than they have done in Eben Upton: It’s got a lot to do with Parkinson’s
Smaller transistors mean more of them can fit the past. law too – work expands to the time available.
on a chip, they can be placed closer together It seems to us that this exponential increase So this is kind of the CPU version of Parkinson’s
and they can run at lower voltages – all things in computer speed should have left concerns law. One of the things we’ve done with the Pi
that come together to mean that the chips can about performance back in the 1990s, but it is refuse to accept that 700 million cycles-per-
be more powerful with smaller sizes. All this hasn’t, and the experience of using a computer second in the processor is slow. I just won’t
is usually summed up by Moore’s law (named day-to-day doesn’t seem to have got faster in accept it.
after Gordon E Moore, co-founder of Intel) line with the processing power. We put this And people kept saying, “It’s such a slow
which states that the number of transistors on question to the man responsible for more processor.” and I’m saying “It can do 700 million
a chip doubles every two years. This is more of low-powered computers than anyone else, the things a second! Your high-definition screen
a rule-of-thumb than an actual law, but it held Raspberry Pi foundation’s Eben Upton back has only got two million pixels. You can do
true from the mid 70s (when it was proposed) when the Raspberry Pi 2 came out. This is what 350 things to each pixel on your screen every
until around 2012. Since then, the speed of he had to say: second. How is that slow?”
www.linuxvoice.com 21
FEATURE LINUX FOR EVERYONE
THE LINUX
NEWBIE GUIDE
Are you new to Linux? Or do you want to help your friends and colleagues
make the switch? Our guide explains all.
T
here’s nothing wrong with being a working over the internet. But an increasing full control over your computer. With
Linux newbie. We were all there amount of work comes from companies Windows, Mac OS or iOS, you can never be
once. But if you’ve picked up your such as IBM, Intel, Red Hat and Canonical. sure what the software is doing – you can’t
first copy of Linux Voice looking for an easy They don’t make money from selling the get the source code, and you can’t fix it
entry into the operating system, or you’ve operating system, but they generate revenue yourself, or pay anyone else to do it.
been using it for a while but still feel unsure by offering support contracts, services,
about some things, we’re here to help. documentation and other benefits. 3
It’s super reliable
To start, what is Linux? It’s an operating Configured correctly, a Linux system simply
system, much like Windows and Mac OS X. 2
It’s open to everyone won’t crash unless something is wrong with
It runs on your computer, acting as a middle Because Linux is open source, anyone can your hardware. We know people who’ve
man between your hardware and your study its inner workings. You can download been running Linux servers for several years
applications. It manages your computer’s the source code (the original human- without a single reboot. Linux is designed in
memory, helps different programs to run readable recipe) of Linux, change it, and such a way that its various components are
together (without stepping on each other’s recompile it to run on your computer. Now, well isolated from one another, so if there’s
toes), and has drivers for your hardware. few people have the technical nous to do an issue with one part of the operating
Historically, Linux had a reputation for being this, but it’s essential nonetheless: you have system (such as the graphical user
difficult to use, but that’s far from the truth interface), the rest of it carries on chugging
now. In fact, you can install Linux and be away – for years and years.
browsing the web, editing documents and
playing games in 15 minutes. Linux experts: 4
It works with your files
But what makes Linux so awesome? Here this guide is Creative Although Linux is a different operating
are the four biggest selling points: system to Windows and Mac OS, and
Commons (BY-SA) doesn’t run all of the same programs, it’s the
1
It saves you money licensed, so cut it out most compatible OS in existence. You can
You don’t have to pay a penny to use Linux. or photocopy it and open your Microsoft Office documents in
But how is such a large body of software LibreOffice, you can play all your videos and
completely free – who pays for its share it. Help others to music in VLC, and there are Linux
development? Much of the work on Linux is convert! equivalents for pretty much every
done by volunteers around the world, application on Windows and Mac OS X.
22 www.linuxvoice.com
LINUX FOR EVERYONE FEATURE
CHOOSING A DISTRO
Linux comes in many flavours – find the best one for you.
Before you get started on your Linux journey, targeted at newbies, some at power users, Now, there are hundreds of Linux
you need to choose a distribution – a bundle some at security, some at low-end devices distributions out there, which can seem
of Linux and related software. This might (such as old netbooks), and so forth. They all baffling at first. But the vast majority are
seem like an extra hassle when compared to share the same core software, but include simply based on another one, so in actuality,
the Windows or Mac OS approach, but it other features and add-on software to be the there’s only a handful of unique distributions.
makes sense. Some Linux distributions are most suitable for a specific scenario. Let’s look at a few of the most notable…
Ubuntu Fedora OpenSUSE Debian Arch Linux Linux Mint
www.ubuntu.com www.getfedora.org www.opensuse.org www.debian.org www.archlinux.org www.linuxmint.com
Ubuntu Linux is Fedora is a community OpenSUSE started life in Debian is well known Instead of having Linux Mint is based on
primarily geared towards supported distribution the mid 90s. It’s popular for its stability and is big updates every Ubuntu, but provides a
desktops and laptops, known for incorporating with intermediate Linux therefore used on tens six months or every different interface and
although it’s making cutting-edge users, sporting an of millions of servers year like most other set of default software.
gains on tablets and technologies, and makes excellent configuration around the world. distros, Arch constantly It’s popular amongst
phones as well. With new releases every six tool called Yast that lets Debian is a community changing with the new users and has a
Ubuntu, you can get months. Like Ubuntu, you tweak all aspects project and provides the latest software. This is very helpful supporting
a modern, well-tested Fedora focuses on of your system from a basis for many other great for power users, community.
version of Linux with having an attractive and single point. distributions, such as although it can cause
just a few mouse clicks. versatile interface. Ubuntu and its spin-offs. some problems .
Making the big decision community on the web (eg www.askubuntu. box, so we think it’s the best way to start.
So after all this, we recommend going with com). The Ubuntu team puts a lot of effort After a few months of working with Ubuntu,
Ubuntu. It’s the best known distro, is very into its interface and into making sure that you’ll be confident enough in Linux to try
polished, and has a huge supporting the operating system works well out of the other distributions.
GIVING CREDIT TO GNU
Here’s an important history note: what we call back to the early 1980s, and is called GNU, for GNU’s everyone could install and use. So the GNU project
“Linux” today is the work of multiple projects that Not Unix (recursive acronym glory). played a huge role, which is why you sometimes
have been running since the 1980s, all of which have GNU developed a lot of software to help create see Linux referred to as GNU/Linux, and today the
worked together to create a free, open and shareable a fully free operating system, and was paired up operating system has hundreds of thousands of
computing platform. One of these projects dates with the Linux kernel in 1991 to create something developers around the globe.
www.linuxvoice.com 23
FEATURE LINUX FOR EVERYONE
SYSTEM
REQUIREMENTS
INSTALLING UBUNTU LINUX • 1GHz Intel/AMD CPU
• 2GB RAM
• 10GB drive space
Follow our step-by-step guide and get using Linux in 15 minutes…
1
Get Ubuntu 2
Start your PC
Point your browser at www.ubuntu.com/download/desktop and get Next, you need to boot up your PC from the DVD-R or USB key; you
the latest version (eg 16.04). You will download an ISO file, which is a normally need to press a key on your keyboard when your computer
disc image that can be burned to a DVD-R using your regular disc starts to do this, so consult your PC’s documentation to find out
burning software (you can also create a bootable USB stick). how. After a few moments, the screen above will appear.
3
The installer 4
Splitting up the disk
Ubuntu’s installer will check that your PC has sufficient space to In the next step, choose where to install Linux on your hard drive.
install Linux. If you are connected to the internet, you can download You can install it alongside Windows, and have a menu when you
updates, extra drivers and media file codecs during installation; click start your PC to choose your operating system, or you can dedicate
the icon in the top-right to set up Wi-Fi connection if you need one. the whole hard drive to it.
5
Create a user account 6
Almost finished...
The Linux files will be copied to your hard drive, and you’ll be asked Grab a cup of tea, and when all the files have been copied over, the
to set your location and keyboard layout. You will also be prompted installer will prompt you to reboot the machine. Click on Restart Now
to set up a user account so that you can identify yourself to the and remove the DVD or USB key once the PC restarts, then choose
operating system and log in – don’t forget your password! Ubuntu from the boot menu that appears.
24 www.linuxvoice.com
LINUX FOR EVERYONE FEATURE
USING LINUX
Now you’re ready to work (and play!) in your new Linux installation.
When Ubuntu first starts, a window will appear listing
some common keyboard shortcuts that are worth
memorising to make you work more quickly, so once
you’ve glanced through them, click the X button in the
top-left. Next, click on the Ubuntu button in the top of
the bar on the left. This is similar to the Start button in
Windows: it lets you browse included software (go to
the Applications button at the bottom after clicking it,
and then Installed to see what’s included by default).
You can also type to search for files or to run
programs directly.
Other buttons in the panel on the left are shortcuts
to useful programs. When you start a new program,
its icon will appear on this bar; right-click it and choose
“Lock to Launcher” to keep it there after closing the
app. In this way, it’s a bit like the Windows taskbar –
but much more flexible.
When you’re running applications, you can click and
Ubuntu’s Unity desktop is
drag the titlebars to move them, and use the edges with from Windows or Mac OS. Firefox is arguably the
friendly for newbies and
to resize them. Ubuntu has a global menu bar, like best browser out there, combining good performance liked by many power-users
in Mac OS X; when using an application, move your and thousands of extensions with excellent privacy as well.
mouse pointer to the top bar to show menu entries. settings. And underneath Firefox you’ll see three
Also in the bar at the top you’ll find icons for audio icons for LibreOffice, opening the word processor,
levels, power management and networking. Click the spreadsheet and presentation tool respectively.
cog icon on the far-right to log out or shut down the LibreOffice is the flagship office suite on Linux,
machine when you’re finished with your work. and is tremendously capable, having seen decades
To change settings with your installation, click the of development in its previous incarnations as
cog-and-spanner icon in the panel on the left. This OpenOffice and StarOffice. LibreOffice does a very good
opens up the Systems Settings window, from which job of opening Microsoft Office documents – although
you can configure your installation, manage hardware, there can be slight formatting issues with some very
and add new user accounts. If multiple people will be complicated documents. Still, if you open an Office
using Linux, give them all separate accounts so they document from one version of the suite in a different
can have their own desktop and software settings. version, you’ll likely experience the same thing, so this
is something even Microsoft struggles with!
Included software For email, click the Ubuntu button and search for
To access your personal files, click on the drawer Thunderbird. This is an email client from the makers
button underneath the Ubuntu button on the left-hand of Firefox, and is mature and very stable. Other pre-
panel. Your “home” directory is like My Documents in installed software worth exploring is Rhythmbox (a
Windows – it’s where your personal files are stored. If music player), Empathy (for instant messaging) and
you insert a DVD or plug in a USB key, a window will Shotwell (a photo manager). Of course, you’ll find
pop up showing its contents, and on the left-hand plenty of small tools such as a calculator and text
panel of the file manager, you can also access editor as well.
resources on the network. So, those are the basics – have fun taking it
Underneath the drawer button you’ll see an icon from here! If you need any help, visit our forums at
for Firefox, a web browser you’re probably familiar www.linuxvoice.com
ADDING MORE SOFTWARE
Your Ubuntu installation comes pre-installed Software Centre to explore programs available to Audacity (for editing audio files), OpenShot (a movie
with many top-class applications, and there are download – most of them free and open source. You maker), VLC (a media player that handles virtually
thousands more available. Click on the Ubuntu can browse categories down the left. Some of our every format under the sun) and HomeBank (a
button, type “software” and choose the Ubuntu recommendations include Gimp (an image editor), personal finance tool).
www.linuxvoice.com 25
SECRETS CALIBRE
SECRETS OF
CALIBRE
Master your eBooks and create a flexible personal library.
T
he technology used to make them has changed hugely from and view books when they’re reduced to data files, but eBooks also
hand-written papyrus to printed paper to computers and the allow instant access to a mind boggling number of books.
very latest eInk screens. Books have not only survived all this When it comes to Ebooks on Linux, there’s one tool that stands
change but they’ve grown ever more popular with each technological out above the others: Calibre. It’s our application of choice for
advance. The latest incarnation of this ancient format – eBooks reading, managing and even creating eBooks. It’s full of features, but
– provides some new challenges to the reader, such as how to store can be confusing to new users. Here are its eight best features…
01 02
Rights Management (DRM) on their
downloads, which limits what you can
do with your eBooks. Calibre will not
convert these books unless you use a
DRM-remover tool first (which may be
illegal in some countries).
03
Editing
As well as reading eBooks,
Calibre helps you create
01
Syncing storage and a better backup system), your own. The powerful in-built editor
While you can read eBooks and just keep the books you’re currently enables you to tweak existing books or
on your PC, you’ll usually get reading on your devices. write new ones from scratch. You’ll
a better experience on an eReader, a need to understand HTML to make the
02
phones or a tablet. After all, it’s nicer to Convert most of it, but for most eBooks, a
relax in a comfortable chair with an There are a number of simple layout is best so you only need
eReader than to sit at your desk. Calibre different eBook formats, basic HTML. It’s our tool of choice for
enables you to send books to devices and not all eReaders can support all creating the Linux Voice digital editions.
that are connected (either physically or formats. Fear not: Calibre can step in to
04
in some cases via a local network). This convert your library from almost any Web server
way you can keep your full library on format to almost any format. However, Calibre includes a built in
your PC (where you probably have more some eBook stores include Digital web server. Go to
Preferences > Change Calibre
If your Calibre machine is available on the public Behaviour > Sharing Over The Net to set
it up. Once it’s running, you can access
internet, you can get access to your library from your books through the web interface.
This makes it easy to transfer books
anywhere in the world onto devices without the need to
26 www.linuxvoice.com
CALIBRE SECRETS
03 04
06
05
07
physically connect them to the Command line interface
machine. If your Calibre machine is While Calibre is primarily a
available on the public internet, you can graphical application, you
get access to your library from can use it from the terminal. This could
anywhere in the world. be particularly useful if you wanted to
run it on a headless server to enable
05 07
Online news you to access your eBooks and
Calibre will automatically scheduled downloads when your main
download online news PC is switched off. You can also use
sources and convert them into eBook this interface to convert eBooks
formats that you can browse offline. between formats or to launch the
You can schedule these downloads to viewer – both features that could be
happen at a certain time, for example, useful if integrating Calibre in scripts or
first thing in the morning, so you have other bits of software.
something to read on the train to work.
08
If coupled with the web server (secret Tweaks and plugins
4), this could give you your own private The out-of-the-box Calibre
stash of downloadable reading matter. setup is good for most
people, but you can add features to 08
06
Library management make it fit your personal workflow.
If you’re an avid reader, you Plugins enable you to add new bits of
could quickly find yourself functionality, such as the ability to
swamped with more books than you access online stores or convert to an
can easily manage. Calibre helps you obscure file format.
filter your library by author, series, rating, Tweaks enable you to change minor
format, tags and other options. Used bits of Calibre behaviour by changing
properly, this filtering should mean that some of the underlying Python code.
you never have to spend long looking Between these two sets of tools, you
for what you want regardless of how can make Calibre work exactly how you
many books you have. want it to.
www.linuxvoice.com 27
FEATURE ARDUINO
BUILT WITH
Les Pounder looks into the little machine that's powering
a quiet revolution – the march of the makers!
T
he Arduino microcontroller has changed the into their work bringing forth the merger of arts and
world. It’s now part of the larger Maker technology. In the mid 2000s the maker
ecosystem, which it helped found and later community as we know it was still in its infancy. The
shape. The Internet of Things, a network of connected Interaction Design Institute Ivrea (IDII) in Italy, a
appliances and devices, Physical Computing, where centre specialising in how users interact with
computer science concepts are used with hardware in computers, was working on creating a
new and interesting ways: without the Arduino we development platform with a supporting
would not have these technologies and communities hardware device, called Wiring, which formed
and technologies as we know it. part of Hernando Barragán’s master’s thesis in
But how did the Arduino come to life? Well, it all 2003. The first prototype Wiring project used the
started as a way for artists to integrate technology Parallax Javelin Stamp microcontroller, but this
The Arduino started as a way for artists to
integrate technology into their work, bringing
forth the merger of arts and technology
28 www.linuxvoice.com
ARDUINO FEATURE
microcontroller required proprietary tools to work with
the board; Hernando wanted to make an open source
project, and so later prototypes used Atmel ARM-
based AT91R4008 microcontrollers, starting the
popular combination of Arduino and Atmel hardware.
Hernando later went on to be the only student to
graduate with distinction in 2004. At this time
Massimo Banzi and Casey Reas were supervisors
for Hernando’s project and took part in a
four-week project called “Strangely
Familiar” that taught physical
computing to 22 students.
The goal of Wiring, and the earlier
Processing project created by Casey Reas
and Benjamin Fry, was to enable non-
programmers, typically artists, to program
using an easy-to-understand language and a
simple open source prototyping platform. For
Processing, the term “sketches” was used to identify
the code of a project which would create on screen
visualisations. This term has been retained in the
The Arduino Uno is the
Arduino community. In 2005 the Arduino team formed
most common of the
and the first Arduino-branded board was released.
range and is used to
The Serial Arduino came with an ATmega8 chip and great effect as a beginner
ran at 16MHz. To connect the Serial Arduino to a to intermediate project
computer a DE-9 9 pin serial connection was required, platform.
a port that is no longer found on many computers.
The Arduino USB was the first board to feature a USB
interface and this was handled via the FTDI FT232BM, something that the Raspberry Pi does not feature.
which enabled serial data and power connections to These analog pins can be used with potentiometers
the host computer. to create delicate forms of input for precise control of
an output device, such as a motor or LED.
Number 1 Over time the Arduino project went on to become
The most famous Arduino is the one that everyone the de facto standard for makers and hackers, and
starts with: the Uno. The Arduino Uno is a humble many books, projects and accessories were produced.
platform that serves as your first step to becoming a The Arduino’s popularity has also seen the creation
maker. Coming with an Atmel ATmega328P of “Arduino Day”, which for 2016 took place on 2 April
microcontroller running at 16MHz and 32kB of flash The Arduino has been used by makers and artists for
memory, the Arduino Uno has 14 digital pins, with six installation pieces such as Minimaforms’ Petting Zoo,
of these pins also coming with pulse width which looked at how environments can interact with
modulation (PWM), which useful for controlling users to shape the architecture of the future. The low
motors with a variable speed. The Arduino Uno, as price and the ease of use is what incites makers and
well as other Arduinos, comes with six analog pins, hackers to choose the Arduino for their projects.
SEND IN THE CLONES!
Being an open source platform, the Arduino has spawned many world of physical computing. Once you become confident with
clone devices. Due to rules put in place early in the life of the the platform and start creating multiple projects, it would be
Arduino, no clones may be called an “Arduino”; rather they are financially prudent to purchase the clones; just make sure that
“Arduino Compatible” they are of good quality. Open source hardware is generally good
These Arduino-compatible boards are just as good as their quality, but sometimes a rogue board slips through the net.
official counterparts, and in most cases are significantly cheaper. One Arduino compatible that is worth attention is The
We managed to source an Arduino Uno clone for £1.68 including Shrimp project, based in Morecambe and named for the famous
postage from AliExpress, and we found an Arduino Mega (the shrimping farming community. The Shrimp is a build-it-yourself
larger board with more pins) for a mere £4.08. So should we buy bare-bones Arduino that comes as a kit and is assembled on a
these boards in favour of the official boards? From our personal breadboard. They work exactly like an Arduino Uno, and retail for
opinion purchasing an official board as your first device will £10. These kits help makers understand the parts that make up
provide you with the confidence that it has been built and tested these powerful prototyping boards. More information from
to a high standard, ensuring that you have a good start in the http://start.shrimping.it.
www.linuxvoice.com 29
FEATURE ARDUINO
The open source hardware
In recent years we have seen an explosion of which is now in its fourth year of production having
of the Arduino can be
single-board computers taking on the dominance of seen a number of iterative releases that have added
broken down to its
constituent elements and the microcontroller-based Arduino. The Arduino has more to the board for the same price of $35. Not
used on a breadboard,
such as the ShrimpingIt
project.
In recent years we have seen an explosion of
single-board computers taking on the dominance
of the microcontroller-based Arduino
been with us since the mid 2000s and has driven the resting on their laurels, the Arduino team have also
uptake of physical computing and powered the maker released new boards. In 2013 we saw the release of
Arduinos turn up in the movement, and this is long before the Raspberry Pi, the Arduino Yun, powered by an ATmega32U4 and
most unlikely of places.
Microbit etc. The Arduino’s position as the leading including an Atheros AR9331 Wi-Fi System on a Chip
Here we see a Picade
joystick controller and platform for hardware hacking was threatened by (SoC) enabling the Yun to use Wi-Fi in projects. The
audio amplifier. This the release of the Raspberry Pi, Yun also provides a 400MHz processor and 64MB of
uses an Arduino RAM to use with Linino, a derivative of the OpenWrt
to provide an Linux distribution. The Yun was not alone in heralding
interface for new features to the Arduino platform: in late 2013 the
arcade controls. Intel Galileo was released, which was the first Arduino-
compatible board to feature an Intel processor. The
Galileo was designed to be hardware- and software-
compatible with Arduino Uno projects while offering
greater processing power, thanks largely to a 400MHz
Intel Quark SoC X1000, a 32-bit single-core processor
that offered the same processing power as an Intel
Pentium from the mid 2000s.
The Intel Galileo also offered a greater number
of ports and connectors for the growing Internet of
Things movement, namely Ethernet, PCI Express,
micro SD cards and USB 2.0. The board also offered
the ACPI power-saving functionality to efficiently
30 www.linuxvoice.com
ARDUINO FEATURE
manage the power of your project. The Galileo later As of May 2015 the Arduino is also known in
spawned its successor, the Galileo Gen 2, which some sales territories as Genuino, a trademark
offered the same functionality but in a slightly more created by four of the original five founding members
refined platform. who initially formed Arduino LLC, a company that
The Internet of Things is something that we have would hold the trademarks for the brand and license
There are many clone
mentioned throughout this feature, and the Arduino the manufacture and sale of boards to external
Arduino boards on the
product line has seen its latest board, the MKR1000, companies. This occurred as a result of the Arduino
market. Some match the
offer a new and unique platform. Powered by an ARM trademark being secretly registered in Italy by a fifth Arduino specification,
Cortex M0 and featuring a low-power Wi-Fi chip, the member of the team. Subsequent negotiations failed whereas others can be
MKR1000 is seen as an IoT platform to power the to unite the brand, forcing the Arduino LLC team to directly inserted into a
next generation of smart devices. The board can still create the Genuino brand for use outside of the breadboard.
be programmed using the familiar Arduino editor. United States of America.
A SIMPLE TUTORIAL TO CREATE A SENSOR
THAT SHOWS DISTANCE USING LEDS
There is no better way to understand how easy the pulse, to an input which will receive the echo. Please You will need
Arduino is to use than by getting hands-on with it. In refer to the diagram for more information. • An Arduino Uno
this project we introduce the Arduino platform by • Four LEDs
creating a distance sensor that uses an ultrasonic Software setup • Four 220Ω resistors
sensor to measure distances using a pulse of sound. Our first step is installing Arduino on our machine. The (red-red-brown-gold)
This is then processed and output via a series of Arduino homepage has complete instructions on how • A breadboard
LEDs. to do this for all operating systems: https://www. • Male–male jumper
wires
arduino.cc/en/Guide/HomePage.
• All of the code for
Hardware setup With the Arduino software installed, launch the this project as well
We start the project by constructing the circuit. We application and you will immediately see a blank as a high-resolution
used an HC-SR04 ultrasonic sensor that we screen with today’s date as a suggested filename. In diagram can be
downloaded from
purchased from eBay. This requires four connections: here we shall write our code. https://github.
to 5V power; ground (GND); the trigger, which sends a We start by creating two variables that are constant com/lesp/Arduino-
pulse; and an echo, which receives the reflected pulse. integers; in other words these values do not change. Distance-Sensor/
archive/master.zip
As you will see in the diagram we have opted to use These variables are pingPin, used to state which pin
the same pin (pin 12) on the Arduino. Using code we our Trigger and Echo pins are connected to, and time,
can switch the pin from an output, which will send the which is used to store a default time value to control
www.linuxvoice.com 31
FEATURE ARDUINO
cm = microsecondsToCentimeters(duration);
Our next section of code is a method of debug. We
print the distances to the Serial Monitor, accessed via
the Tools menu, so that we can see the values and
check that they are correct.
Serial.print(inches);
Serial.print(“in, “);
Serial.print(cm);
Serial.print(“cm”);
Serial.println();
We now start a conditional test that will check the
distance returned, and use that to control our LEDs.
We will show a portion of this code, the whole of
which can be downloaded from our GitHub page.
Our first test is to see if an object is less than 5cm
away; if that is True, we print “Less than 5CM” to the
serial monitor. We turn pin 11 on thus lighting our LED.
Pins 10,9 and 8 remain turned off. We then delay for
10 microseconds using the time variable we created
The Arduino IDE works
the pace of the code. earlier.
with all versions of the
Arduino boards. If a const int pingPin = 12; if (cm < 5){
board is not listed, it can const int time = 10; Serial.print(“Less than 5CM”);
be added via the Boards In our next block of code we set up the pins that digitalWrite(11, HIGH);
Manager option. This will be used for our LEDs. These pins are 8–11 and digitalWrite(10, LOW);
includes the ESP8266- each one is an output. We also start a serial interface digitalWrite(9, LOW);
based boards. at 9600 baud (bits per second). We will use the serial digitalWrite(8, LOW);
monitor to check that our code works. delay(time);
void setup() { }
Serial.begin(9600); Our second test handles distances less than 10cm
pinMode(8, OUTPUT); and greater than 5cm. If that is the case then all pins
pinMode(9, OUTPUT); except pin 10 are turned off, thus illuminating a new
pinMode(10, OUTPUT); LED. We use else if for this condition. If you are used
pinMode(11, OUTPUT); to Python, this is referred to as elif.
} else if (cm < 10 && cm > 5){
We now move to the main loop of our code. Serial.print(“Less than 10CM”);
This loop will continue forever. We start the loop digitalWrite(11, LOW);
by declaring long variables, used to contain long digitalWrite(10, HIGH);
numbers, for the duration, and lengths in inches and digitalWrite(9, LOW);
cm. Next we turn our pingPin, pin 12, into an output digitalWrite(8, LOW);
before we ensure that the pin is turned off. We then delay(time);
pause for 2 microseconds before turning the pin on
for 5 microseconds, which is just enough time to send
a pulse. We then turn the pingPin off.
void loop() {
long duration, inches, cm;
pinMode(pingPin, OUTPUT);
digitalWrite(pingPin, LOW);
delayMicroseconds(2);
digitalWrite(pingPin, HIGH);
delayMicroseconds(5);
digitalWrite(pingPin, LOW);
Still inside the loop we now change our pingPin
from an output to an input, ready to receive the echo
ping. We then store the duration of time taken in a
variable that is then used to calculate the distance in
inches and centimetres. The conversion process is
handled later in the code.
pinMode(pingPin, INPUT); There are few differences between clones and the original
duration = pulseIn(pingPin, HIGH); Ardiuno (chiefly they cannot use the Arduino trademark),
inches = microsecondsToInches(duration); but the clones can share the same layout as the originals.
32 www.linuxvoice.com
ARDUINO FEATURE
The Arduino IDE is the
default way to work with
your board. It comes with
a series of examples and
tutorials designed to help
new users get to grips with
the Arduino.
} digitalWrite(10, LOW);
There are three more conditions to test, which you digitalWrite(9, LOW);
can see in the code download for this tutorial. Our final digitalWrite(8, LOW);
condition handles when the distance is greater than delay(time);
30cm. All of the LEDs are turned off and we print a }
message to the serial monitor. We now close the loop after delaying the code for
else { 100 microseconds.
Serial.print(“Greater than 30CM”); delay(100);
digitalWrite(11, LOW); }
Our final lines of code are long variables that we use
to store the distances when calculated as inches or
centimetres.
long microsecondsToInches(long microseconds) {
return microseconds / 74 / 2;
}
long microsecondsToCentimeters(long microseconds) {
return microseconds / 29 / 2;
}
With the code completed, save your work. Ensure
that your Arduino is connected to your computer and
then click on the Upload button (an arrow pointing
Building the circuit for our project is relatively simple; right). Once it's done uploading, the Arduino will reboot
just take it step by step and follow each wire from point and the script will run. Place an object in the path of
to point. the sensor to trigger the LEDs.
SHIELDS UP CAPTAIN?
The Arduino platform has an extensive ecosystem of peripherals and SparkFun MP3 Shield, which provide audio output for your
and components that can be added to your project, from simple projects – imagine a sensor-triggered scare device for this year's
components such as LEDs and buzzers to more bespoke add-ons Halloween party. Arduino shields come from a variety of sources
such as GPS and 3G data. The Arduino uses a common add-on and retail from around £5 to £50 and as with other elements of
system called shields. These shields are placed on top of the the Arduino community, many of these shields can be picked up
Arduino and provide extra functionality. cheaper as clones but with varying levels of quality.
For example, the Ethernet shield provides a simple Ethernet When starting out with the Arduino you may be tempted to
interface, enabling our Arduino to work as a low-powered web splash out on shields, but before you do, get used to the Arduino
server, albeit not one that you would want to power your site. The as it is. You will be amazed as to how much can be done with
main use for the Ethernet Shield is to send data to the web, so such frugal resources. We don’t need a GHz CPU and gigabytes
sensor data can be gathered and sent to a remote computer for of RAM to run a sensor-powered data collection device; rather we
processing. Other notable shields are the Adafruit Wave Shield need a dependable and power efficient platform to work from.
www.linuxvoice.com 33
FAQ FLATPAK
Flatpak
Hurrah! Distro-independent packaging is finally here, after so many years of waiting.
Or is it?
MIKE SAUNDERS That’s true. If you’re using a distro Flatpak aims to fix this by making
with large package repositories, cross-distro packages possible. So
Oh great, yet another such as Debian, Fedora, OpenSUSE and regardless of what distro you’re using,
packaging format! Is that what their derivatives, you’ve always had a when FooApp 2.0 is released, you can
the world really needs? wealth of software to choose from. But install it straight away, without having
We know what you’re thinking. there’s one major problem here: what to wait for your distro to package it up,
We love the flexibility and choice happens when you want to install or compile it from source, or do any
inherent in free software, but there’s so something that’s not in your distro’s other kind of technical gymnastics just
much duplication of effort out there, repos? What happens if your distro only to try out a new program.
and sometimes it would make a lot has FooApp 1.1, but the developers of
more sense if developers worked FooApp have released 2.0 with a Wait a sec – hasn’t this been
together on a grand project rather than tramload of new features and goodies tried before? I remember
having countless half-baked apps and to play with? hearing about Autopackage…
standards doing the rounds. Well, you’re up a certain creek with a Yes, there have been numerous
So if Flatpak were just yet another tiny paddle. Yes, maybe the FooApp attempts to do something like
packaging format a la Deb or RPM, our developers have spent ages creating this. It’s a tough nut to crack. But
shoulders would be aching from new packages for every major distro. Or Flatpak is rapidly maturing, it has the
shrugging by now. But it’s not. Flatpak perhaps there’s some kind of backports backing of major distros, and the
promises to fix something that has repository where you can find it. Or technical structure on which it sits is
been deeply wrong with GNU/Linux for maybe you’re running a rolling-release well thought-out. Autopackage used all
many years: the almost unbreakable tie distro, but they have their drawbacks as manner of tricks and cludges to just
between packages and distribution well (everything is a moving target). So about make a binary executable file
releases. what do you do? Most people who don’t work across multiple distros, regardless
have the time or nous to get the new of the libraries installed, but it never
But how is that a problem? I’ve version will just wait for the next round really took off.
been using $DISTRO for years of distro updates – even if it’s another Flatpak, on the other hand, is ready to
and always get the software I need! six months away. use today. There are already some
big-name open source applications
such as LibreOffice available in Flatpak
Flatpak promises to fix something that has format, and the range is growing.
Flatpak was originally designed by
been wrong with GNU/Linux for many years: Alexander Larsson of Red Hat, and
while it’s true that Red Hat is a major
the tie between packages and distro releases backer of the technology, developers
34 www.linuxvoice.com
FLATPAK FAQ
from many other projects and distros
are involved as well. So we expect it to
stick around for a while.
OK, this sounds pretty good.
So how does it work?
Flatpak builds upon various
technologies that are starting to
emerge or become mature in the Linux
world. For starters, it uses cgroups and
Linux namespaces for “sandboxing” –
that is, keeping each Flatpak program
separate, so they can’t interfere with
one another. The goal here is to ensure
that a malicious Flatpak can’t do too
much damage to the system, or mess
with the workings of other software.
Now, you may think this is a bit
paranoid, as Linux has barely been Flatpak apps are sandboxed from one another, but use shared runtimes for dependencies.
affected by malware so far in its life. But
that’s because most of us install What Flatpak does is this: it uses a libhardlyused, the distro maintainers
software from a vetted and well- system of “runtimes” which provide a will patch it up, issue an updated
maintained distribution repository, set of base libraries that apps can version, and that small bunch of
where we can be pretty much certain depend on. These runtimes are the programs that use it will benefit from
that the software hasn’t had dodgy same across distros – so you don’t the update.
backdoors injected. Repositories are need to worry about minor version With Flatpak, each app that depends
not perfect, but they’ve done a great job changes messing everything up. If you on libhardlyused will have its own
over the years. install a Flatpak app that uses Gnome version. So when a security vulnerability
With Flatpak, it will be easier to libraries, Flatpak will first check whether comes along, every app using
download and install a random you have the Gnome runtime installed libhardlyused will need to be updated
program from a random website. More (and if not, install it). There are various separately. Some apps may receive
users will start doing this, so it’s runtimes providing a reliable set of updates quicker than others – some
essential to avoid the problems that dependencies, and while they may eat may not be updated at all. So the
plague Windows when people double up a chunk of disk space, especially if classical distro approach with its
click every .exe file they see (or get sent you have runtimes for multiple Gnome zillions of packages provides better
in spam emails). So with Flatpak, versions installed, it makes it easier for security in some cases, but Flatpak
applications are isolated from one package maintainers to know what’s strives for convenience as well.
another, and a malicious app is available in a distro.
restricted in the harm it can do. OK, so I guess time will tell if it
And these runtimes include really takes off…
OK, but what about the absolutely every library under Exactly. The pace of change in
dependency problem? Isn’t the sun? the Linux world is extremely
that the biggest issue of all with Well, no – that’d be bonkers. rapid, and we love that, but it still takes
packaging on Linux right now? There has to be a trade-off a long time for new technologies to
Yes, it is. Say you want to install somewhere. Runtimes include major really propagate into mainstream
FooApp 2.0 as mentioned before, libraries like GTK, but if you want to build distros. For how long have we been
but your package manager says you a Flatpak package that includes an waiting for Wayland, for instance? (OK,
need libfoobarbaz-12.0.1.3 whereas obscure library not provided by any there are still issues to fix, but it seems
your distribution only provides runtime, you should roll that library into like it has been ready for the next round
libfoobarbaz-12.0.1.2 (which turns out the Flatpak itself. This keeps the of distro releases for five years now.)
to be incompatible, because the Flatpak app nicely self-contained and Still, you can try it for yourself today
developers didn’t follow a sensible means that users don’t need to hunt by visiting http://flatpak.org and
version number scheme). So you decide down extras. following the installation instructions
to compile libfoobarbaz-12.0.1.3 from There is, of course, a downside to there. The website also includes a guide
source, install GCC and the whole this. In a normal Linux distribution, an to making your own Flatpak, so if you’ve
toolchain kaboodle, find out that you obscure library – let’s call it been working on your own Free
also need to compile and install libhardlyused – would be provided Software project but found it a pain to
liblolwut-0.2.5 and libohreally-9.3.6 separately in the package repositories, package it up for multiple distros, give it
and then just get angry and close your and used by a handful of programs. If a a try – it may save you a lot of time in
laptop and go out for a walk. security hole is discovered in the long run.
www.linuxvoice.com 35
INTERVIEW FRIDRICH STRBA
“
Micromanaging people is
not good in open source, so
when they want to do
something, they will do it –
you just give them the
frameworks and let them do
that they want with them
FRIDRICH
STRBA Ben Everard meets the man behind free access to file formats.
D
ata. Whatever you do with your information. Proprietary software isn’t documents was a major stumbling block
computer, data is going to be at always designed with this information for Linux adoption, but now, the vast
the heart of it, whether it’s an office sharing in mind. After all, if only one piece majority of open source office tools can
document, a chunk of program code, some of software can read your data, you have read and write (almost) perfectly. This
images, or any other set of 1s and 0s. The to keep using that bit of software – and change is due to the hard work of a lot of
important thing for this data is that you buying updates. people, and leading the charge has been
can access it from multiple programs so For many years, the inability of open the Document Liberation Project. We chat
that if there’s ever a problem with your source software to consistently read and to Fridrich Strba, co-founder of the DLP, to
chosen tool, you can still used the precious save data in the latest proprietary office find out what’s been going on.
36 www.linuxvoice.com
FRIDRICH STRBA INTERVIEW
www.linuxvoice.com 37
INTERVIEW FRIDRICH STRBA
Can you tell me a little bit that was not working, instead of saying look at it. That was how it started, with
about how the document ‘oh that commit doesn’t work’, the one the project itself and the legal
liberation project got started? who saw it fixed it and we advanced framework. The collaboration started in
Fridrich Strba: Actually, it got started quite quickly from there. 2011, and the project as a framework
by the LibreOffice community. At that started two years ago – 2 April 2014.
time, LibreOffice had good support for From this one file format you
quite a lot of proprietary file formats, decided to expand? You’ve supported several file
especially Microsoft Office formats, but FS: We became emboldened by this formats over several versions.
then there were parts of Microsoft result and we started to look towards In general, are they getting easier or
Office, like Visio, that were not other file formats. Valёk has quite a harder to work with?
supported. We were always trying to good experience of looking at files and FS: We have a theory about
support it because people were asking trying to find patterns and such things incremental reverse engineering,
for it. We realised that Valёk Filippov so we managed to work with Corel Draw because even if there are several
(who is now collaborating with me) and then Microsoft Publisher, and then versions of a file format, no company
found a way to reverse engineer the file we were going on and on. At a certain can completely re-write their software.
format. Because the file format is moment, we realised that the best way If you know a certain version, you can
binary compressed with a custom to make it not just depend on us two try to go lower because certain data
compression, it was not simple to was to create a project and make the structures may change, or maybe some
understand how the files look inside, code a little more modular so people representation of numbers may
and he managed to do it. can re-use the framework and let them change, but nobody’s going to
This helped us to put it as a Google do what they want to do. completely change the file format
Summer Of Code project, so maybe Generally micromanaging people is because that takes several years to get
some student would start to work on it, not good in open source: when they
and we managed to get a student. By want to do something, they will do it.
that time, I also had time not only to You will just give them the frameworks I don’t think anybody
mentor the project but also to code on
it – basically tandem coding with the
and let them do that they want with
them.
wants to make it harder
person. After three months we We gave it a legal foundation. It was to read their files…
managed to to get something kind of a constellation of libraries that
reasonably nice done, and it was gravitating around each other. We
encouraged us. We’re not extremely built the project so that it has some right. For example, with Corel Draw, we
good communicators – we were existence beyond the libraries so that if started with versions 7, 8 and 9, and
communicating through code – so we one of the libraries has a bug and the then we went lower and we support
got a method of work that was good for person who works on it doesn’t have everything from the beginning of Corel
us. For example, if there was a commit much time, you still feel a bit obliged to Draw. We then looked at the later
versions… there were some little
differences, but it’s not like you have to
completely reverse engineer a new file
format. You can just look at the
differences, and since we have tools
that can show you binary diffs of
chunks of the file, it was pretty easy. It
was still a chunk of work, but it was not
completely different.
Are there many new file
formats that you want to be
able to support?
FS: There are file formats that we’d like
to support and we don’t know really
how to support them because the file
format is really complicated, like for
example InDesign.
We can support Visio files in
everything that exists from Visio 1 to the
one that comes with the newest
Microsoft Office – that’s actually
Do you want to help open up closed file formats?
XML-based, but still the data structures
Pop by the #documentliberation-dev channel at
irc.freenode.net and say hello. are the same. With Corel Draw we
support everything from version 1.
38 www.linuxvoice.com
FRIDRICH STRBA INTERVIEW
Versions 1 and 2 are completely
Projects benefitting from Fridrich’s
different – OK, not completely different,
work include Scribus, Inkscape,
but the encapsulation was different.
Calligra and LibreOffice.
Now the last one we did was version 8,
which was released this year – we
realised that because of the way we
parsed the documents, we were also
able to parse the new documents with
the old code.
What, in your view, is the
toughest file format that you
support?
FS: InDesign is complicated because it’s
basically a database, and it changes
with each version. Even Adobe is
unable to save files in lower versions – were actually documented before, so I with other software as well?
it only has migration from the lower knew the frameworks we were using, FS: This application is used by
version to the current version and then and we evolved them into something Inkscape, it’s used by Scribus, it’s used
you can’t save it in the lower version self contained. I was working on by everybody who does something with
any more. something that imported Word Perfect importing the file format – it’s used by
Freehand is quite a tough file format files as my first open source project. It Calligra. If they use these files formats,
because you have to have all the was not my project, but I contributed to they use our libraries, because at a
records in order to be able to parse the project, and I suddenly became certain moment, we killed the market. It
them, because they don’t have size maintainer when the other maintainers doesn’t make sense for someone to
information. You have to know how to didn’t have much time to do it. re-invent the wheel in another
parse each record to be able to jump to Then I worked on Word Perfect framework because we tried to make
the next one. Graphic file format, and I tried to unify the framework very independent – we
the APIs to extract them from the don’t have any GTK structures or Qt
Have you ever come across libraries because the data structures structures, we have just our structures
anything that you feel has been could be the same. The callback that are good for the libraries. They’re
put in deliberately to make it hard to functions weren’t the same, but at a pure C++ and you can plug it easily, so
reverse engineer? certain moment, we were at a point there’s no real reason to do it differently.
FS: Frankly no. I think that people just where the data structures were from
try to dump their document in the files the Word Perfect library, the interface for Are you looking for more
somehow… There are some custom putting out the images was from the people to get involved?
compressions, but they didn’t do it Word Perfect Graphics library, so if you FS: Oh yes. It’s always good for people
because they want to obfuscate it, but wanted to import something from to come. We need people! Well, what
because they want to have it as a them, you had to have the Word Perfect we don’t need are talkers, but anybody
feature – considerably smaller files. I library and the Word Perfect Graphics who can make any meaningful
don’t think anyone wants to make it library. We decided that what can be contribution is welcome. If you want to
harder to read the files. common, we put into a single function import something, start to create your
and make the other things pluggable. library and we’ll integrate it in our
For you, personally, what got framework. You can do that and we’ll
you interested in this area? You mentioned that The make you famous. We can’t make you
FS: It’s the technical challenge. I’d Document Liberation Project rich because we haven’t made
worked on other file importers that came out of LibreOffice. Do you work ourselves rich!
OLETOY
If you want to start poking around inside files to see ./re-lab/oletoy/view.py
what’s going on, and potentially reverse engineer This will start a graphical application that enables
new file types, Oletoy is the tool of choice for the you to open files of many different types. You can
Document Liberation Project. As you would expect, see how the data’s arranged in the file in text and hex
it’s Free Software, and you can download it from format, and known structures will be broken down
GitHub with: to make them easier to follow. In the Edit menu,
git clone https://github.com/renyxa/re-lab.git you’ll find options to manipulate the file by adding or If you want to add support for a new file format,
Then you can start the application with: removing data. Oletoy can help you find out how it functions.
www.linuxvoice.com 39
MUGS AND T-SHIRTS!
You can drink
coffee, tea or
champagne.
It’s all about
choice.
This mug
respects your
freedom. Look,
it even says
GNU/Linux!
shop.linuxvoice.com
INTRO REVIEWS
REVIEWS
The latest software and hardware, rigorously bashed against a wall by our crack team.
On test this issue . . .
Fedora 24 Workstation 42 RetroPie 43
The flagship Gnome distro gets more polish, more Mike enjoyed playing with this emulation setup so
Andrew Gregory software and more excellent. Ben loves it. much that we haven’t heard from him in weeks.
It turns out that yoghurt pots half-filled with
Belgian trappist beer really do trap slugs.
I
n this issue’s news roundup Mike
reports that Linux has now reached
a whopping 2.8% market share in
desktop computer use. This is, quite
simply, amazing. Viewed in isolation,
2.8% seems piffling, but for most of the
last 10 years, the figure has been Fritzbox 44 Digikam 5 45
hovering around at just over 1%. I love It’s a router, Jim, but not as you know it: this little Photographers of Linux, fire up your package
statistics, so I’m choosing to interpret device will make the master of your home network. managers and download this wonderful tool.
that roughly 1.8% percentage point rise
as a 180% increase. Extrapolate that
over the next 10 years and Linux will
have achieved the utter domination of
running 999% of all desktop computers.
Group test and books
Though mathematically implausible, I
contend that this huge expansion will
be made possible by the huge increase
in the nuber of devices that we can
reasonably describe as a desktop
computer. Fridge? Baby monitor?
Television? Smart insulin pump? Trip
computer in your car? These are
desktops. Kind of.
Silliness aside, 2.8% is huge: it’s a
growing, commerically relevant slice of
the computer using market, and it
means, finally, that Linux can’t be Booooooooooooooks!!!! 48 Group test – virtualisation platforms 50
ignored. Just remember that you liked Penetration testing relies on a standard set of tools, When you’re sick of the whirring, chugging sound of
Linux before it was cool. but wouldn’t it be awsome if you could craft your a hard drive being overwritten, it’s time to virtualise
andrew@linuxvoice.com own in Python? Yes – yes it is awesome. your distro experimentation with one of these.
www.linuxvoice.com 41
REVIEWS LINUX DISTRO
Fedora 24 Workstation
Ben Everard doffs his hat to a cutting-edge Linux distro.
F
Web https://getfedora.org edora – the community Linux distribution it makes it much easier to get to know the software.
Developer Fedora Community sponsored by Red Hat – continues its endless Other updates include an improvement to the search
Licence Various free software
march to the future with version 24. As always, function in Files, which makes it easier to narrow
licences
this latest release comes jam-packed with the latest down the results by type of file and date the file was
Linux technology. The big new feature in 24 is support used, and the addition of simple editing to the Photos
for the Flatpak packaging format, which provides image viewer.
developers with a way of releasing their software in a
controlled, sandboxed environment. See the FAQ on All about choice
page 34 for more details. If you’d rather a different desktop, there are spins for
Fedora defaults to the Gnome desktop environment KDE Plasma, Xfce, LXDE, Mate, Cinnamon and Sugar
and this release comes with version 3.20 (named (an environment designed for children).
Delhi), which brings with it a host of improvements. It seems that with every Fedora release, we talk
Our favourite addition is the new shortcuts window. about Wayland, the next-generation display server.
Press Ctrl+? or Ctrl+F1 in any of the Gnome It’s still not quite ready for prime time, so by default,
applications and you’ll get a new window showing version 24 of Fedora will stay with the older X server
the shortcuts for that software. It’s a simple idea, but (adventurous users can install Wayland if they wish).
At the time of writing, the Fedora team are planning
to make the switch to Wayland with the next release
(25), however, this change has been delayed so
many times that we would caution against placing
a bet on it.
As well as the Workstation version that we’ve
reviewed here, there are server and cloud builds for
running in other environments. Fedora 24 delivers
exactly what we’ve come to expect from Fedora – the
latest Linux tech bundled up in an easy-to-use format.
The enhanced file search
Flatpak and Gnome 3.20 make this a useful
allows searching by date or release even though Wayland is delayed.
file type without resorting
to the terminal.
42 www.linuxvoice.com
PI GAMING DISTRO REVIEWS
RetroPie 3.8
Got a Raspberry Pi sitting around doing nothing? Install this.
W
e dread to think of how much time we the joypad. Some advanced options
spent playing games on the classic 8-bit still require command line fiddling,
and 16-bit consoles and computers in the but for the most part it’s a
80s and 90s. Actually, forget that – playing those beautifully elegant plug-and-play
games encouraged us to explore computers further, retro gaming solution.
start writing rubbish little shoot-em-ups in BASIC, and Even better, RetroPie lets you
finally move on to the black art of assembly language. map a combination of buttons
Today, nothing fazes us, and we can hex-edit to exit the emulators, so if you
filenames in initrd images with both hands tied behind have a few hundred Game
our backs. Boy games installed on the
RetroPie is a specialised Raspbian-based distro for SD card (which you obtained
the Raspberry Pi that focuses entirely on retro completely legally, of course) you easily
gaming. You write it to an SD card, connect your Pi to can go through them, enjoy the good ones, and
a telly, plug in a couple of USB joypads and voilà: you feel terribly sorry for the kids (or their parents) who If you’re a Game Boy fan,
you absolutely must play
can emulate games released for the SNES, Mega splashed out £30 per pop on the rubbish ones. Note
Zelda: Link’s Awakening. No
Drive (aka Genesis), Game Boy, Game Gear, NES, that if your USB joypad has shoulder buttons, you can
exceptions.
Master System, PC and other platforms. Indeed, with use these to quickly skip pages in the list of games,
a Pi 3 you can try emulating more high-spec consoles which saves a huge amount of time when you have
such as the PlayStation and N64, but performance lots to try out. Web https://retropie.org.uk/
and compatibility varies significantly. Stick with the So if you have a Pi sitting around doing nothing download
8-bit and 16-bit consoles and you’ll be fine. special, and you want to see if you can beat your Platforms Raspberry Pi
Price Free
When we first started using RetroPie a couple of old Super Mario Kart lap times or complete Streets of
years ago, it was impressive but rather fiddly; you had Rage II without losing a single life, there’s nothing
to attach a USB keyboard, exit out of the shiny better than this.
emulator front-end, and perform various tricks to get
USB joypads properly configured in all of the Retro gaming bliss – and even better that you
can easily take it round to a mate’s place.
emulators. This has been greatly simplified over time
– now you just need to hold down certain buttons on
www.linuxvoice.com 43
REVIEWS ROUTER
FRITZ!Box 7490 Firmware 6.5
Graham Morrison avoids begging BT not to send him a HomeHub 6.
W
Web https://en.avm.de e reviewed the FRITZ!Box 7490 wireless to the original router. One month and 247,057MB
Price £225 router back in issue 10, and we liked it. In of downloads later, we’ve not experienced a single
particular, it filled a difficult gap in the UK connection issue with the new firmware.
market for routers that can replace the generally
woeful hardware you get from your internet service The Firm
provider. More importantly, alongside ADSL, it also The other major addition for this firmware is a
supported VDSL, which made it the only router we’ve completely new web interface. The new design is
found that worked with BT’s consumer fibre network clearer and more responsive, and now works on small
(BT Infinity). screens. This is quite an accomplishment for a piece
The reason we’re looking at the 7490 again is of hardware that squeezes so many features into one
because there’s been a major firmware update – the place – there’s USB network attached storage, media
uninspiringly named 06.50 (now 06.52). This update streaming, DECT telephony with fax and answer
changes almost everything about the devices, from machine, home automation, guest Wi-Fi accounts,
a complete overhaul of the web interface and its quality of service and per-device online filtering, to list
functions, to the stability and speed of its wireless and just a few. All these functions are now easy to get to
internet connections. That last point was of particular without any supporting apps, and work perfectly.
interest to us as we had to stop using the old firmware You get the same detail about your internet
when BT started to degrade the performance of connection too, enabling you to change almost
our connection – a problem solved by going back everything about how your network is configured. This
is something you can’t do with the usually locked-
down routers provided by ISPs and gives the 7490 a
clear advantage.
This is still an expensive router. But it combines the
functionality of many devices, and the new firmware
feels like a hardware upgrade to a device that was
difficult to beat.
The hardware hasn’t
changed, but the new
Costly, but could replace several boxes with
firmware is so different one. And the new firmware is like a new device.
from the old, it may as well
be a new device.
44 www.linuxvoice.com
PHOTO ALBUM REVIEWS
Digikam 5.0
Graham Morrison finally finds an app to replace his beloved Kalbum.
D
Web https://www.digikam.org
igikam has always been brilliant, and has been
Licence GPLv2+
our default photo management application for
a decade. It handles all the formats we care
about, including camera specific RAW images, and
offers just the right balance of processing and editing
for almost of the tasks we need to perform. Version 5
is a major update, and the result of two years’ work by
the developers.
One of Digikam’s best
Most of this work has been concentrated on features is its metadata
rewriting almost the entire codebase to work with Qt editor, where you can
5. This was obviously a huge undertaking for an change almost any of the
application as wedded to the KDE 4 and Qt 4 data held on your images.
frameworks as Digikam was. Every KIO-slave instance
was removed, for example, which required the complexity of the user interface, your large photo
database code to be replaced with a more platform- libraries, the editor windows and previews all appear
agnostic multi-threaded implementation. Like several more quickly than before.
other applications from the KDE 3/4 era, this means The move to Qt 5 may not have added many new
Digikam is now more portable, with both Windows and features, but that’s probably a good thing – this is an
Mac versions being much easier to install, and there’s application that does almost everything we want it to
a solid plan to remove the remaining dependencies to anyway. Whether it’s playing with geolocation and the
make Digikam almost dependency free. integrated maps, editing image metadata, merging
But portability isn’t the only reason to move to Qt 5: bracketed images for HDR or the brilliant photo
it’s a much more modern platform, that’s more duplicate finder, you can’t be serious about Linux and
efficient and more flexible than it’s earlier revisions. It’s photography without adding this to your desktop.
the power behind KDE’s Plasma desktop, for example,
and helps to make Digikam better aligned with the An excellent photography app, whether you’re a
beginner or a professional. Digikam is ace.
Plasma desktop. Font rendering and theme
integration is better, for instance, and despite the
www.linuxvoice.com 45
REVIEWS GAMING
GAMING ON LINUX
The tastiest brain candy to relax those tired neurons
VULPINE CUNNING Hearts of Iron IV
Change the course of history.
Web http://store.steampowered.com/app/394360
Price £34.99
W
ith Hearts of Iron now appearing on
Linux with its latest installment,
Victoria is the only one of Paradox’s
series of grand strategy games to be missing on
Michel Loubet-Jambert is our Games
Editor. He hasn’t had a decent night’s Tux-powered machines. So if you’ve had a
sleep since Steam came out on Linux. chance to delve into some dark ages or medieval
strategy, or if those time periods aren’t your It is possible to play as any country existing in the
S
team Machines were highly thing, there’s some World War Two to get into. time period.
anticipated by the Linux Hearts of Iron has typically strayed from the
community, but didn’t manage
formula slightly, focusing more on warfare than On the positive side, the lack of an absurd
to make the splash many had hoped
for. Though they are far from dead, nation-building, and the latest installment in the amount of mechanics, and streamlining others
with vendors like Alienware releasing series is no exception. While there are political like politics and trade, means that the game is far
new models recently, it seems the and technological aspects to explore, these more approachable for newcomers than other
strategy is a long-term one which mostly serve the purpose of ramping up series. That isn’t to say that the game is
gambles on the Vulkan API becoming
mobilisation and improving weapons. While in suddenly less complicated than other strategy
the industry standard; Linux ports
should thus become easier to the other games it was possible to turn the likes of games like Civilization – the developers do
point where the platform can Ethiopia into a global empire, in Hearts of Iron IV recommend keeping the wiki open in a tab while
seriously compete with Windows in this would result in a pretty boring game unless playing the game, though it’s still far less
terms of titles. With seven of the 10 playing as one of the major Allied, Axis or intimidating than Europa Universalis IV. In this
most popular games on Steam having
Comintern nations. Though the possibility for sense, the game strikes a balance between
Linux support, this doesn’t seem like
an impossible feat. some interesting alternative history scenarios satisfying the existing hardcore fans without
While the HTC/Valve Vive virtual exist with some secondary powers, such as others writing it off as over complicated. Overall
reality headset has seen delays in winning the civil war as the Republicans in Spain, though, it’s hard not to feel like this is a step
coming to Linux, another VR headset for the most part, not being part of the main backwards in some regards, and Crusader Kings II
known as the HDK2 by Razer already
conflict leaves little to do. and Europa Universalis IV are more immersive.
supports Linux and is due to hit
shelves in July. While all the main VR
systems are still works in progress
since the technology is still in its
early days, the HDK2 isn’t just
exciting due to Linux support, but
also the $399 RRP, which is $200 and
$400 lower than the Oculus Rift and
Vive respectively.
What makes this headset even
more interesting is the use of the
Open Source Virtual Reality (OSVR)
ecosystem, an open standard for VR.
It also supports SteamVR, so
compatibility won’t be an issue
should OSVR not become the de facto
industry standard. The 90Hz refresh
rate and 2,160x1,200 resolution puts
it on par with the mainstream VR
specs, though some early reviews The game has a considerable
have found the visual quality lacking focus on battle plans and the
in comparison.
military-industrial complex.
46 www.linuxvoice.com
GAMING REVIEWS
Dead Island Definitive Edition ALSO RELEASED…
Back from the dead.
Web http://store.steampowered.com/app/383150 issues on Linux, the worst of these being
Price £11.99
the inability to craft weapons effectively
(one of the game’s main mechanics);
D
ead Island was released in 2011 however these are gone from this version
and we got it on Linux in 2014, but and performance is also decent with
now it’s returned using the far 60fps possible on mid-range hardware.
more modern game engine used for its As far as zombie games go, it’s still a
more mature cousin Dying Light. decent enough game, but after playing
Nation Red
The graphical overhaul is immediately Dying Light, it feels like a serious This top-down zombie shooter harks back to
obvious, and while not quite on par with downgrade since there isn’t really anything an era where the player stood around and
Dying Light, visually it does look like Dead Island does better. Still, at this price, it mowed down anything coming in their
something that could have been released could be worth picking up for the extra direction. Nation Red adds a lot to this classic
recently. The original game wasn’t without dose of zombie killing. formula, such as full 3D graphics and a decent
variety of weapons and game modes. The
ability to play something like this in online or
local multiplayer makes it a lot of fun.
http://store.steampowered.com/app/39800
The tropical island of Banoi Edna & Harvey: Harvey’s New Eyes
and its undead inhabitants look This charming point-and-click adventure does
better and deader than ever. extremely well with the genre’s staple
trappings. Its characters are very memorable,
the humour is top-notch and the puzzles are
entertaining, adding its own unique quirkiness
F1 2015
and visual style that sets the game apart from
the rest. The game’s bizarre and imaginative
world are tied up nicely with the story, making
Great racing but no campaign. this a must-have for adventure fans.
http://store.steampowered.com/app/219910
Web http://store.steampowered.com/app/286570
Price £39.99
F
1 2015 is the first game of the
franchise to be ported to Linux
and the latest racing game to be
released. With such a major racing title
coming out, we now finally have a decent
roster of racing games on Linux after a The graphics and racing are as good as it
considerable time with almost none. gets, adding far more realism.
F1 2015 does everything one would The Mean Greens – Plastic Warfare
Now ported to Linux, this third-person shooter
expect from a Formula 1 game, with the featured a career mode, allowing the
where the player controls toy soldiers brings a
ability to choose from teams, tweak car player to start in a smaller team, improve breath of fresh air through an interesting
settings and play practice races as well as their skills and be offered bigger contracts, aesthetic and lighthearted take on the shooter
qualifying and the main events. The in F1 2015 this is stripped down to the genre. There’s a good variety of maps and
physics and AI are solid, adding a great point where the single-player mode game modes, while matches mostly take the
form of five a side online multiplayer.
deal of challenge to the game. consists mostly of going through different
Unfortunately, there’s no longer a huge number
There’s also some impressive graphics seasons. The racing itself is among the of people online, but hopefully the Linux port
and online multiplayer, however, it’s the most enjoyable out there, though those should help remedy that.
single-player campaign where the game expecting a deeper campaign experience http://store.steampowered.com/app/360940
falls short. Where previous games have may be disappointed.
www.linuxvoice.com 47
REVIEWS BOOKS
Maker’s Guide To The Zombie Apocalypse.
When the apocalypse strikes, Ben Everard’s obscure electronics hoard will pay off.
Author Simon Monk
Publisher No Starch Press
Price £16.50
ISBN 978-1593276676
W
hen society inevitably collapses doesn’t strike during our lives, then it turns
into chaos and hoards of undead out that the skills you need to survive the
walk the earth, some people will zombie apocalypse are also useful if you’re
be more equipped to survive than others. interested in building your own electronics.
You don’t need to stock up on supplies The Maker’s Guide To The Zombie
– everything you need will be available Apocalypse is a good introduction to building
in abandoned shops – what you need is devices with the Arduino and Raspberry Pi.
knowledge. Without the internet, there’ll You’ll learn how to attach extra hardware,
be no YouTube instructional videos or sense the environment and make the two
Wikipedia pages, so you’ll only have what’s devices communicate wirelessly – all great
in your head or on paper books. skills whether you’re defending a base from
The Maker’s Guide To The Zombie zombie attack, making a smart home or
Apocalypse tells you how to build the taking your first forays into the world of
electronics you’ll need in the post- robotics. Stay safe out there.
apocalyptic world. You can learn how to
generate electricity, and use this power to A fun and useful introduction to physical
computing.
increase your chances of survival. If we live The information is useful in all post-apocalyptic
in fortunate times, and the zombie plague scenarios, not just those caused by the undead.
Black Hat Python: Python Programming
for Hackers and Pentesters
Ben Everard is slightly disapointed that this book isn’t about snake millinery.
Author Justin Seitz
Publisher No Starch Press
Price £15.66
ISBN 978-1593275907
W
hen it comes to ethical hacking, Black Hat Python helps people who
you can do a lot with the pre- understand Python and know the principals
made tools. The Metasploit of penetration testing bring those skills
Framework, the Burp Suite and others give together. You don’t have to be an expert to
you the capability to probe almost any get the most out of this book, but no time is
desktop, sever or network without having spent helping the reader get up to speed.
to enter a line of code. However, at a certain The book guides the reader through
point in your penetration testing, you will networking, attacking websites, building
come up against the limits of these. Perhaps a trojan horse, and then looks at some
you’re probing custom-made software Windows weaknesses. At the end of the
that doesn’t quite fall within the remit of book, the reader should have a good idea
the available tools; perhaps you’re testing what Python is capable of and how to use
for an exotic type of bug; or maybe you this language for software security.
just want to delve deeper and get a better
understanding of what’s going on – at that A flexible and powerful approach to penetration
testing.
point, there’s no option but to roll up your We strongly recommend that the information in
sleeves and write some code. this book is only used for white-hat hacking.
48 www.linuxvoice.com
GROUP TEST DESKTOP VIRTUALISATION APPS
GROUP TEST
He might not be able to Alohomora his way through doors, but Mayank Sharma can whip up
virtual machines out of thin air with little effort.
On test Desktop virtualisation apps
T
Gnome Boxes ruth be told, we’ve all got too a new piece of code on your main
URL https://wiki.gnome.org/Apps/Boxes much computing power at machine, tinker with it inside a VM.
Licence GNU GPL v3 our disposal. It’ll help you You can also use the VM to get a
Latest release 3.20.2 reduce the boot times and speed up feel for FreeBSD, Haiku or some
Is simplicity the way to go? application launches only up to a other esoteric operating system
certain extent. One of the best uses without jeopardising the partitions
of extra CPU cycles is to create and contents of the disk on the
Qemu virtual machines. This decades-old computer. Furthermore, there are
enterprise technology is now tame a couple of things you can do with
URL www.qemu.org
Licence GNU GPL v2 enough to be used by the average VMs that you can’t on a physical
Latest release 2.6.0 desktop user. The Gnome desktop computer. For example, moving
Has the oldest app in the group test kept up environment even bundles one with VMs from one computer to another
with the times? the standard desktop apps such as takes a lot less effort than backing
Gedit and Transmission. up and restoring a physical machine.
While they have clear advantages In this group test we’ll examine
VirtualBox for enterprise deployments, virtual some of the best apps that you can
URL www.virtualbox.org machines (or VMs) make a lot of use to virtualise machines inside
Licence GPL and PUEL sense on the desktop. You can use your home computer. Some have a
Latest release 5.0.20 them as fully functional computing very intuitive interface while others
Does the app from Oracle deserve all the environments that are isolated from offer more features and flexibility.
hoopla? your main computing environment We’ll highlight the strong suits
to test new software or even of the individual apps and help
complete operating systems. If you you find one that best suits your
Virtual Machine Manager don’t feel brave enough to compile requirements.
URL www.virt-manager.org
Licence GNU GPL v2+ You can use a virtual machine as
Latest release 1.3.2
Is Red Hat’s challenger to Oracle any good? a fully functional environment
isolated from your main computer
VMware Player
URL www.vmware.com/products/player Virtualisation vs emulation
Licence Freemium Virtualisation and emulation are two The main difference between
Latest release 12.0.0 similar technologies that are often virtualisation and emulation is that
Is this freeware better than the free mistaken for each other despite several while emulated environments require
software options? distinct differences. Emulation involves a software bridge to interact with the
making one system imitate another. hardware, virtualisation accesses the
The most popular use of emulation is to host’s hardware directly. Virtualisation
run software that’s designed for other involves simulating parts of a computer’s
VMware Workstation hardware such as running console- hardware, but most operations still occur
based games on a PC. You can also on the real hardware. Due to this reason,
URL www.vmware.com/products/
use emulation software to conjure virtualisation is usually always faster
workstation
up complex pieces of hardware. For than emulation. But unlike emulation, the
Licence Trialware example, Bochs is an emulator that can host system has to have an architecture
Latest release 12.1.1 emulate an entire processor in software. identical to the virtualised guest system.
Is it really worth all that money?
50 www.linuxvoice.com
DESKTOP VIRTUALISATION APPS GROUP TEST
Commonly used virtualisation jargon
Terms you should know.
Desktop virtualisation is just one aspect of the There are several types of hypervisors. A bare-metal namely para-virtualisation and operating system
much broader virtualisation realm. While you don’t hypervisor, such as XenServer, runs directly on assisted virtualisation. These are used in server
need to have an in-depth knowledge to create and the hardware, and unlike hosted hypervisors like and large-scale environments and help minimise
run VMs on your desktop, a familiarity with some VirtualBox don’t require a separate host operating the overhead of running and managing a virtual
of the most common terms will help you digest the system. Hypervisors rely on command-set extensions environment.
trends and follow developments and news related to in your computer’s processors to accelerate common Other terms you may come across include a
virtualisation with ease. virtualisation activities and boost performance. snapshot, which is an image of the state of a VM at
One of the most common terms you’ll come Intel-VT and AMD-V are the two sets of extensions a specific point in time; and virtual appliance, which
across is hypervisor, which is the piece of software for Intel and AMD processors respectively. There are is a virtual machine with a fully preinstalled and
that enables you to create and run virtual machines. other form of virtualisation besides full virtualisation, preconfigured operating system.
Virtual Machine
Manager
A very real overseer.
The kernel-based virtual machine (KVM) is
the virtualisation infrastructure built directly
into the Linux kernel, because of which it
performs exceptionally well. Note however
that KVM requires a processor with
hardware virtualisation support. Fire up a
terminal and enter
egrep -c ‘(svm|vmx)’ /proc/cpuinfo
The command will return 1 or more if you
have a processor compatible with KVM. Guests can use a couple of protocols to export their graphical framebuffers, including VNC and Spice.
The Virtual Machine Manager, commonly
referred to as virt-manager, is one of the VMware, there are no additional guest The CPU section also lets you define the
most popular hypervisors, which interacts additions or similar extensions that will maximum number of CPUs accessed by
with KVM via the libvirt API to create and enable such features. Virt-manager can also the VM, which can be greater than the
manage virtual machines (VMs). attach USB devices including removable number of CPUs allocated by default. This
Using the app is pretty straightforward. drives, webcams and Bluetooth devices enables you to hotplug additional CPUs as
Like other virtualisation apps, virt-manager found on the host and all work inside the supported by KVM to cope with additional
also employs a wizard to create new VMs. guest seamlessly without any issues. processing demands.
The five-step process begins with selecting Also while virt-manager primarily creates
an installation method. Virt-manager then Senior management KVM VMs, it can also manage Xen and LXC
asks you to assign memory, CPUs and Besides the usual slew of features, virt- containers. Since Virtuozzo 7 containers
storage to the VM. In the last screen you manager also includes a host of functions and VMs are managed via the libvirt API, it
select the network settings for the VM that’ll appeal to advanced users. For is possible to use virt-manager for the same
and can also tweak other virtual hardware starters, while the default virtual hardware purpose as well. Furthermore, virt-manager
settings before powering on the VM. Using settings will work for most users, virt- ships with a bunch of command line tools.
the app you can add and customise various manager offers fine-grained control over There’s virt-install for creating a VM, virt-clone
kinds of hardware and controllers. some pieces of hardware which will appeal for duplicating guests, and virt-viewer for
The app’s main interface displays a list to advanced users. For example, in the CPU displaying a minimal graphical console for
of all the VMs, and if one is running, it’ll also section you can manually specify a CPU the guest among others. One of the most
display its live resource utilisation statistics. model for the guest. The section lists a huge popular is virsh, which is a CLI interactive
Inside the VM, sound works out of the box number of CPU models such as Pentium 3, shell for managing all aspects of a VM.
and the display can also scale to full-screen. Opteron G5, Haswell, Westmere and more. If
You can copy and paste text between the virt-manager isn’t able to use the exact CPU An intuitive and feature-rich app that
VM and the host, but can’t move files in the model, libvirt automatically falls back to a works well for new users as well as the
more demanding ones.
same fashion. Also unlike other platform closest model supported by the hypervisor
virtualisation tools such as VirtualBox and while maintaining the list of CPU features.
www.linuxvoice.com 51
GROUP TEST DESKTOP VIRTUALISATION APPS
Gnome Boxes
The idiot’s box?
Gnome Boxes uses libvirt, which also Even when you want to manually
powers the Virtual Machine Manager and customise the settings for the VM
exposes just enough functionality to be Boxes offers only two customisable
usable while keeping it simple enough hardware parameters, namely number
to make it approachable by of CPUs and amount of memory.
virtualisation debutants.
It’s no surprise then that the app has Simple stuff
a simple user interface. When you wish Once the VM is up and running, Boxes
to create a new VM, the app gives you a behaves pretty decently. The VMs can
bunch of options. You can either enter switch to full-screen without issues and
the URL from which the app will fetch the sound works inside the VMs In addition to pointing it to an ISO image you can also
an ISO image and boot off it, or point to without any issues. You can also copy point Boxes to a VNC server or oVirt and LibVirt brokers.
an ISO, which is the most commonly and paste text between the host and
used option. If Boxes finds other VM, and it has support for taking supposed to – the webcam throws
libvirt-managed VMs on the computer snapshots. input/output errors and the Bluetooth
it also gives you the option to import Like the other tools on test here, device isn’t visible to other devices, for
them. However this option is a work in Boxes also has an option to connect the example. However, USB drives
progress and in our tests the imported webcam, Bluetooth, fingerprint reader connected to the host show up without
machines fail to boot. and other devices found on the host to making a fuss.
While Boxes claims to define the ideal the VM. When toggled, the devices
settings for a VM after it recognises disappear from the host and show up in The features are in line with its objective of
the ISO image you’ve pointing it to, the output of the dmesg command on simplicity, but this limits the app’s
usefulness in the long run.
the settings aren’t as ideal as it claims the guest like they should. However,
them to be, and are very conservative. none of them work as they are
VMware Player
Free as in cheap.
VMware Player is available as a free any additional drivers in the VM. When
download for non-commercial use. you enable the VM printer, VMware
According to its release notes, the latest Player configures a virtual serial port to
version improves performance while communicate with the host printers.
suspending and resuming encrypted Some of the useful features, such as
virtual machines and also support for the ability to move files and copy and
4K monitors. Player publishes a list of paste text, requires the installation of
officially supported distros that it can the VMware Tools package. However,
recognise and for which it can unlike with VirtualBox, installing VMware
configure appropriate VM settings. One Tools is an antiquated process – you You can access and download VMware’s library of virtual
of its unique features is the unattended have to manually extract the tools appliances from within VMware Player.
Easy Installation mechanism, available and then install them via a text-based
for a few distros including Ubuntu. interface. the host’s integrated webcam, the
Player’s VM creation wizard is the Some of the more interesting image on the guest appears distorted.
standard affair and very easy to follow. features provided by the add-on tools, Another minor usability irritant is that
You can also choose to customise such as dragging and dropping images shutting down a VM also exits the app.
the hardware at the end of the wizard between applications, work only However, Player’s biggest limitation is
before powering on the machine. between Windows hosts and guests. that it only lets you run one VM at a
Player can virtualise the usual slew The Unity function is one of these – it time, which is rather debilitating.
of hardware such as disks, network enables you to run Windows from
adapters, sound cards and more. One the guest on the host. However, like The free but proprietary app has a couple
interesting option is Printers, which many of its unique functions, this isn’t of interesting features that fail to make up
for its lacklustre performance.
enables the guest to print to any printer available for Linux guests and hosts.
connected to the host without installing What’s more, while Player can attach
52 www.linuxvoice.com
DESKTOP VIRTUALISATION APPS GROUP TEST
Qemu A virtualisation
Command and conquer.
distro
Create and manage VMs
remotely.
Proxmox Virtual Environment (VE) is open source
server virtualisation management software. Unlike apps
such as VirtualBox and Gnome Boxes, Proxmox VE is a
Debian-based Linux distro with a modified RHEL kernel.
With the distro you can deploy and manage virtual
machines. Proxmox VE offers the ability to manage
both container-based virtualisation with LXC and full
virtualisation with KVM.
The distro includes a simplified bare metal installer
that takes over the entire disk. Once it’s installed you
can access Proxmox via a web interface from any
computer on the network. The management interface
includes a VNC console and supports SSL, and you can
use it to create virtual servers as well as containers.
Front-ends like Qemu Launcher and the recently forked Aqemu do a good job of providing a For enhanced security, the interface supports multiple
graphical interface to many of Qemu’s advanced options. authentication methods and a role-based user and
permissions management.
Qemu is a very popular processor CLI utilities to create and define various Proxmox offers several storage models. The virtual
machine images can either be stored on one or several
emulator and virtualiser that uses aspects of the virtual machine isn’t local storage types such as LVM and ZFS as well as
something called dynamic translation to everyone’s cup of tea. on network shared storage like NFS and GlusterFS.
speed up its magic. When used as a Proxmox carries an integrated graphical backup tool
machine emulator, Qemu can run OSes For command-line lovers called vzdump, which creates snapshots of virtual
and apps made for one machine, such as Qemu supports various disk formats guests. The backup tool can do both scheduled backups
and live backups and creates a tarball of the VM that
ARM, on a different machine, such as your including qcow2, which is one of its most includes the virtual disks and all the configuration data.
x86 desktop. However, this dexterity feature-rich formats. This format boasts You can find lots of documentation and several video
comes at the price of performance. of capabilities such as the ability to take tutorials on the project’s website. Proxmox is developed
To overcome this limitation, Qemu is multiple VM snapshots, AES encryption by Proxmox Server Solutions in Austria and is released
often used together with the kernel’s KVM and zlib compression. You can present under the Affero GNU General Public Licence.
module. But there’s more to what you can multiple virtual drives to the guest system
do with Qemu/KVM than what’s exposed by attaching up to four image files. Best of
by VMM. You can also run Qemu without a all you can also loopback mount a qcow2
host kernel driver. When using KVM, Qemu image on the host for transferring files
can virtualise x86, server and embedded between the guest and the host. You can
PowerPC, and S390 guests, while plain also convert the image to the VDI format
Qemu (without KVM) can virtualise and use it with VirtualBox.
architectures like ARM and PowerPC. Qemu supports networking and can
Qemu also boasts of impressive emulate some popular network cards.
features. A couple of releases ago Qemu You can connect these virtual NICs to a
got a VirtIO-GPU driver for 2D graphics, Qemu VM using several different ways.
which boosted the graphics performance The easiest of these is the user mode
of the guest machines. In the latest networking, which creates a private virtual
release the VirtIO-GPU driver even enables network along with a firewall, a DHCP
the guest systems to use the OpenGL server, a DNS server and a Samba server.
acceleration provided by the host system. Qemu also includes an interface for
The cost of Qemu’s rich set of features tasks like attaching USB disks and taking
comes at the cost of usability. Qemu is screenshots that you find with other apps.
essentially a command-line utility and However, this interface is also CLI-based,
will typically install a huge subset of unlike the ones found in other apps.
qemu- prefixed tools, each of which refers You can download server appliance templates
to a specific hardware architecture you Boasts more functionality than the others, from within the Proxmox VE web interface as well
can emulate with Qemu. While it’s well but requires familiarity with its command
line tools.
as optimised appliances from the Turnkey Linux
documented and poses little trouble to project.
experienced campaigners, grappling with
www.linuxvoice.com 53
GROUP TEST DESKTOP VIRTUALISATION APPS
VirtualBox vs VMware Workstation
Feature-rich virtual machine builders.
The biggest difference between the two
well-known platform virtualisation apps
is cost. While VirtualBox costs naught, a
single copy of Workstation costs $250
(about £185) and comes with a
complimentary 30-day installation
support. Also VirtualBox is open source
for the most part and is available in the
repositories of the major desktop
distros. However for some (albeit
useful) functions VirtualBox requires the
proprietary guest additions extensions.
VirtualBox can be credited for making
virtualisation accessible to desktop
users. The app offers para-virtualisation
support, namely Hyper-V for Windows
and KVM for Linux, which boosts the
performance of the VM. VirtualBox
also offers disk image encryption You can run VirtualBox on a headless server and control it remotely either via third-party web-based
for improved security. However, this interfaces or via its own extensive command-line tools.
feature is only available if you install the
proprietary VirtualBox Extension Pack. virtualisation users. By default, VMs is the same as in Player, but with many
Some other features dependent on the are isolated from the network. But if more options to tweak the virtual
proprietary add-on are support for USB you’re running a server inside a VM, hardware. You can, for example, define
3.0 devices and bi-directional sharing. VirtualBox can set up port forwarding to the number of cores for each processor
Like other apps, VirtualBox takes you make sure the server is reachable from you add to the VM. Workstation also
through a wizard to create a VM. Once outside the VM. includes the Virtual Network Editor tool
you’ve created a VM you can power it that lets you create complex network
on or tweak the settings for its virtual Man your stations configurations. The app also lets you
hardware. The VM settings window VMware Workstation looks different encrypt VMs and restrict unauthorised
houses some useful options such as from its freely available sibling tested users from modifying the VM.
the ability to manually select a para- earlier. For one, Workstation lets you run Workstation’s latest version supports
virtualisation interface for the VM. You multiple VMs concurrently inside DirectX 10 and OpenGL 3.3, which
can also mark virtual disks as hot- separate tabs. Its UI also presents means it trumps the other apps in
pluggable devices. additional options such as an interface terms of rendering 3D apps and games.
In addition to the desktop centric to convert VMs into the OVF format and Unlike the other apps on test here,
features, VirtualBox also includes another to mount virtual disks on to the Workstation lets you allocate up to 2GB
several functions for advanced host. The wizard for creating a new VM of video memory to a VM. Workstation
also includes a command-line tool for
operating VMs from the CLI, but it’s
not as extensive as VirtualBox. Both
also have useful snapshot and cloning
features to preserve the state of a VM
and to duplicate a VM, respectively.
VirtualBox
Ships with enough features to satisfy both
the desktop user as well as the advanced
virtualisation campaigner.
VMware Workstation Pro
The expensive licence gets you features
that make more sense to an enterprise
user than on an everyday desktop.
You can run Workstation as a server to share virtual machines with others.
54 www.linuxvoice.com
DESKTOP VIRTUALISATION APPS GROUP TEST
OUR VERDICT
Desktop virtualisation tools
Unlike some of the other group tests, Qemu. Aqemu has recently been forked
this one was surprisingly easy to judge. and bears a new look, but you’ll have to
Our unending love for open source manually compile it. Virtual Machine Manager lets you add a
software has made us intolerant of the Next we eliminate Gnome Boxes. filesystem passthrough to share files between
tiniest of mistakes in proprietary It’s a nifty little app but the biggest the guest and the host.
software and for good reason. Why problem is that it is focused primarily
would you want to throw away money on simplicity. That’s not usually a bad
or your freedom over software that’s thing, especially when rating apps for 1 Virtual Machine Manager
inferior to free and open source the desktop, but in the case of Boxes
Killer feature Rich set of virtual hardware
options? It’s because of these reasons the modesty comes at the expense of customisations.
that both of VMware’s contenders lose several useful features. URL https://virt-manager.org
out. The free-of-cost Player product An open source app that’ll satiate the
loses out for failing to give us a Proprietary killer virtualisation needs of a large number of users.
compelling reason to recommend it The runners-up spot goes to VirtualBox.
over other options. Its biggest turn off is The recently released major version is
the inability to run multiple VMs at the a watershed release for the app which 2 VirtualBox
same time. Its big brother, Workstation, has managed to claw back onto the
Killer feature Intuitive interface.
fails to justify its cost for the radars of serious virtualisation users URL www.virtualbox.org
virtualisation needs of the average after incorporating para-virtualisation One of the most popular desktop virtualisation
desktop user. abilities. However, to enjoy all its app, which deserves all the accolades it gets.
We can also strike off the venerable capabilities you’ll have to rely on the
Qemu from the list of contenders proprietary guest editions.
because of its basic interface. The Why would you want to use 3 Gnome Boxes
comprehensive and robust CLI-based proprietary software when you can get
Killer feature Straightforward interface.
app involves a learning curve that’s a bit the same function with FOSS? Virtual URL https://wiki.gnome.org/Apps/Boxes
too much for a desktop user pampered Machine Manager, our group test winner, This no-nonsense app is a wonderful starting
by graphical interfaces. The third-party comes equipped with several features point for virtualisation newbies.
front-ends do a commendable job that you get with VirtualBox’s proprietary
of exposing some of its impressive add-ons. Virt-manager is pretty intuitive
capabilities, but many have failed to to use and scales well, which makes it 4 Qemu
keep pace with the development of ideal for a large demographic.
Killer feature The unfathomable list of
customisations.
Why would you want to throw away URL www.qemu.org
The very powerful CLI tool that can be of use
money or your freedom over software to desktop users only via one of its graphical
front-ends.
that’s inferior to FOSS options?
Virtualised servers 5 VMware Player
It seems ludicrous to put a server inside a attacker can cause is only limited to the Killer feature Available free of cost.
virtual machine – a mission-critical server virtual environment and cannot permeate to URL www.vmware.com/in/products/player
task requires a dedicated machine for the underlying physical machine. Also virtual The proprietary freeware doesn’t really offer
reliability reasons, right? But those of you machines are more malleable and portable any compelling reasons over the open source
who only need a server occasionally can host than physical servers. You can give it more alternatives.
them inside a VM instead of earmarking a storage space or RAM or even number-
physical machine for the task. For example, crunching prowess with only a couple of
you can host an instance of your favourite clicks. It’s also easier to export a virtual
web server to test code or even host hard disk and move it to another physical 6 VMware Workstation
websites for a limited audience by installing machine. Several hypervisors also enable
Killer feature Graphics support
lightweight web servers inside the VM. you to pause your servers and even take a
URL www.vmware.com/in/products/
There are several advantages of this snapshot of a healthy working state of the
approach. First up, in case your server is server that you can revert to in the event of
workstation
exploited in some way, the damage the a mishap.
It’s proprietary, expensive and is really designed
for the enterprise and power desktop user.
www.linuxvoice.com 55
SUBSCRIBE
Subscribe
shop.linuxvoice.com
Introducing Linux Voice,
the magazine that:
Gives 50% of its profits
back to Free Software
Licenses its content
CC-BY-SA within 9 months
12-month subs prices
UK – £55
Europe – £85
US/Canada – £95
ROW – £99
7-month subs prices DIGITAL
UK – £38 SUBSCRIPTION
Europe – £53
US/Canada – £57
ONLY £38
ROW – £60
Get 96 pages Access our Save money on
of tutorials, rapidly growing the shop price
features, interviews back-issues archive and get each issue
and reviews – all DRM-free and delivered to
every month ready to download your door
Payment is in Pounds Sterling. 12-month subscribers will receive 12 issues of Linux Voice a year. 7-month
subscribers will receive 7 issue of Linux Voice. If you are dissatisfied in any way you can write to us to cancel your
subscription at subscriptions@linuxvoice.com and we will refund you for all unmailed issues.
56 www.linuxvoice.com
NEXT MONTH
NEXT MONTH IN
ON SALE EVEN MORE AWESOME!
THURSDAY
RetroPie
25 AUGUST We’ve finally got it
working, so we
thought we’d share
the most epic way
on earth to play the
games of Mike’s
youth on negligibly
cheap hardware.
Ubuntu Snap
A new packaging
format hoves into
view, bearing a
bushel of promises
about dependencies,
security and
convenience. Were
the prophets right?
Linux.gov
Her Majesty’s
FEDORA: IT’S AWESOME
government is
switching some of
its staff from
Freedom, stability, a sense of community and Microsoft Office to
whole load of cutting-edge software – find Google Docs. Only
LibreOffice can save
out what makes Fedora so darned good. us now…
LINUX VOICE IS BROUGHT TO YOU BY
Editor Ben Everard Editorial consultant Nick Veitch through the use of advice in this magazine. Copyright Linux is a trademark of Linus
ben@linuxvoice.com nick@linuxvoice.com Experiment with Linux at your own risk! Torvalds, and is used with permission.
Deputy editor Andrew Gregory Distributed by Marketforce (UK) Ltd, 2nd Anything in this magazine may not be
andrew@linuxvoice.com All code printed in this magazine is licensed Floor, 5 Churchill Place, Canary Wharf, reproduced without permission of the
Editor at large Mike Saunders under the GNU GPLv3 London, E14 5HU editor, until February 2017 when all content
mike@linuxvoice.com Tel: +44 (0) 20 3148 3300 (including our images) is re-licensed CC-
Editor in hiding Graham Morrison Printed in the UK by BY-SA.
graham@linuxvoice.com Acorn Web Offset Ltd Circulation Marketing by Intermedia Brand ©Linux Voice Ltd 2016
Creative director Stacey Black Marketing Ltd, registered office North Quay ISSN 2054-3778
stacey@linuxvoice.com Disclaimer We accept no liability for any House, Sutton Harbour, Plymouth PL4 0RA
loss of data or damage to your hardware Tel: 01737 852166 Subscribe: shop.linuxvoice.com
www.linuxvoice.com 57
FOSSPICKS
FOSSpicks Sparkling gems and new
releases from the world of
Free and Open Source Software
Our benevolent editorial overlord Graham Morrison tears himself away
from updating Arch Linux to search for the best new free software.
Audio effects
Linux Studio Plugins Project 1.0.8
A
s regular readers will know, adjusting brightness or saturation The microphones further away will
we really enjoy using audio in an image, have been few and far obviously get the audio slightly later,
and music software with between in the world of open and this can cause phase problems
Linux. Individual programmers and source, especially when you need on playback as the offset energy in
small teams have created unique quality output. But we’re happy to one waveform cancels out or
software that can help musicians report that this is a gap that the emphasises the energy in another.
differentiate themselves from the Linux Studio Plugins Project (LSP) fits Using the delay compensator, you
huge mainstream of Ableton Live brilliantly. can make sure every waveform is in
virtual DJs with their default set of synchronisation, as if each mic had
presets and popular plugins. Plug and play been exactly the same distance
There’s open source software for LSP is a collection of audio plugins from the source.
creating notes from algorithms, that have been developed for The plugin even offers ‘distance’
open source software for processing audio. They can be used as a scale for the delay, so you don’t
constructing your own sound creatively, of course, but this is a need to perform calculations with
generators and sequencers, and a collection of plugins that acts the speed of sound on the back of
thousand different effects to make mostly as a toolkit. The entire suite an envelope. But it also includes
your music sound like infinite is brilliant at fixing problems and milliseconds and samples, which is
variations of an alien landscape. making your audio sound better. perfect if you want to compensate
But regardless of whether you’re Take the delay compensator, for for a slow DA converter or piece of
creating pop music or musique example. This will delay the audio external equipment, or even the
concrète, you still need a core of by a set period of time, and is various delays introduced by
effects and processors for essential if you’re recording software synthesizers and effects.
day-to-day editing. These kinds of something with several The most surprising inclusion is a
effects – the audio equivalent of microphones at different distances. very capable sampler. These are
essential for simple sound
triggering, both creatively as a
sound source, and as a general
production resource. They’re great
2
3 for backing tracks, for instance, or
sound effects during podcast
recording. The options won’t match
a professional sampler, but they’re
4
perfect for triggering sounds and
loops. And the latest release of the
plugin suite includes a trigger
1
6
sampler that will play back sounds
when it hears a specific frequency,
5 rather than waiting for some MIDI
input. This is a unique plugin, and
one well worth investigating, as is
1 Sampler Play up to eight sounds with your MIDI keyboard, or trigger effects and loops from a sequencer.
2 Delay compensator Essential when recording multiple inputs with multiple distances. the entire suite.
3 Spectrum analyser See audio frequencies you can’t hear. 4 Stereo/Mono Each effect is supplied as both stereo and
mono version. 5 Phase detector Eliminate that weird sweeping sound from your recordings. Project website
6 VST and LADSPA Native Linux VST, LADSPA and LV2 versions of the plugins can be installed.
http://lsp-plug.in
58 www.linuxvoice.com
FOSSPICKS
Movie utility
Govie
U
ntil XBMC and Kodi made obviously useful on its own
integrating film metadata because film fans can avoid loading
into an open source up a browser if they need some
application look easy, movie fans information, but it’s also possible to
were often left to random searching load up the IMDB page in your
and saved files from IMDB (the default browser with the -o option.
Internet Movie Database) to get It’s the scripting potential of Govie
their fix of movie memorabilia. If that we really like, as this command
only they’d had access to Govie! makes it easy to populate your own
Govie is a very simple command scripts and applications with the
line tool that queries the IMDB latest data from IMDB. In particular, Watch Blade Runner now, before Ridley Scott ruins it with a
servers and delivers results directly the -p option will download a poster nonsensical sequel.
to your command prompt. Typing image for the movie you’re
govie -d “Blade Runner”, for searching for. You can even list are interested in. We found this
instance, will return the year the more than one film in a single useful when filling in missing
movie was made, the IMBD review command, letting you quickly images from Kodi, and we could
score for the movie, the MetaCritic automate a collection of posters for easily automate the download of
score and a sentence or two on the any collection of movies you own or posters for movies we record off
plot, all delineated by ‘greater-than’ digital television in the same way
and ‘less-than’ symbols. If there’s
some ambiguity about the film
Govie queries the IMDB we used to for CD/LP album covers.
you’re looking for, you can search servers and delivers results Project website
for a film in a specific year with the
-y argument. The command is to your command prompt https://github.com/narenaryan/Govie
Theme editor
Plasma Theme Explorer
K
DE’s Plasma themes have images, requiring them to be edited
become difficult to in an application like Inkscape if you
understand. In old versions want them changed. The main
of KDE, you used to be able to problem is that none of this is
change many of the parameters obvious, and exploring the various
that were responsible for how a files is difficult. Which is exactly
theme looked, such as the amount where Plasma Theme Explorer can
of drop-shadow, or adjusting the help. It’s been around for a while,
blue glow that the desktop and it might be difficult to find a
defaulted to for windows. The package for your distro (it’s part of
modern desktop eschews these Arch’s Plasma packages, however).
options, and this is because the When installed, it previews and Dive into the details of how a Plasma theme is constructed, and
way themes are created has lists all the graphical elements for a change the parts you don’t like.
changed. Most themes are now a theme, so you can see exactly what
collection of scalable ‘SVG’ image they each do. If the theme is locally these elements in an editor or
files, slotted into pre-defined pieces installed, rather than installed adjust the colour palette. It’s a great
of the user-interface like a jigsaw. ‘system wide’, it also lets you open starting place for creating your own
You need an image for the themes, or adjusting elements of a
top-left corner of a window, for
instance, or for the middle part of
Plasma Theme Explorer is theme you’ve always wanted to
change but never found the option.
the desktop panel. Many effects, a great starting place for
such as any drop-shadow or glow,
are baked into these scalable creating your own themes Project website
www.kde.org
www.linuxvoice.com 59
FOSSPICKS
Desktop theme
Arc Dark
D
espite many studies particular theme is its breadth and
showing that black text on a consistancy. It includes separate
white background is the theme files for Chromium, Firefox,
best for readability, as this the desktop, the command line and
magazine proves, many developers even the Kicker app launcher and
seem to spend days in front of Yakuake drop-down terminal. The
various terminals with a reversed colours are dark but not black,
colour palette – bright text on a reducing the contrast.
dark background. This is because The only slight modification we
the brightness that surrounds the needed to make was brightening Dark themes don’t always look great in screenshots, but they can
text can end up being more the pale blue used for text, but we help tired eyes that spend all day infront of a screen.
fatiguing on your eyes than the really like the palette generally. On a
reversed text, even when the clarity terminal, it’s a cross between a dark Dark feels very modern and
of the text is slightly reduced or solarised theme and Ubuntu’s integrated, much like a dark version
leaves shadows on your retinas. Tango, and because the same of Google’s material design. The
This has spawned a plethora of colour is used for both a window’s only element missing is a
terminal, Vim and desktop themes background and for the window complementary icon theme, but
that attempt to better integrate decoration, such as the titlebar, Arc we’ve found the dark icons from
reversed palettes into an entire the ‘Papirus’ set integrate perfectly
working environment. Arc Dark is
one such theme, and it’s the best
Arc Dark feels modern and with the style and aesthetic of
Arc Dark.
we’ve found, especially for GTK- integrated… a dark version of
based desktops and KDE Plasma.
What we like most about this Google’s Material design Project website
https://github.com/horst3180/arc-theme
Programming
Chuck
I
t’s been possible to write code similar circuit boards, making it the
that’s interpreted live and perfect platform for music
generates music for a while, experiments. But most importantly,
and there are some electronic and because Chuck has been built
musicians that build an entire for audio and music, timing is part
performance out of this method. of its fabric. Create two functions
Instead of a DJ with a real or virtual that generate a beat, for example,
terminal, you can watch Emacs and run them both at the same
projected onto a large screen as time – Chuck will ensure they both
interweaving beats and melodies stay exactly in time over hours, with
are coded in real time. Lots of sample accuracy, using a forking This is the Chuck
different languages are capable of mechanic that Chuck calls a ‘spork’. equivalent of “Hello modulation synthesis, or at a note
doing this, but a language called You can spork many different World!”, the code level for Steve Reich-like interrelated
‘Chuck’ is the best we’ve found. processes all generating audio and required to generate a fragments of repeating melody.
Chuck has been written note data, and they’ll all keep exact sine wave. Chuck accomplishes this with an
specifically for audio and music timing with one another. You can easy to use JavaScript-like syntax
projects. It can talk to MIDI and OSC forget about the complexity of that works brilliantly with Linux
synthesizers and audio equipment threads, or of generating processes using both ALSA and PulseAudio
without any further libraries, and signals that stay in – or no audio at all if you’re after
incorporating all kinds of input synchronisation. You can simply that John Cage sound.
protocols for things like data gloves, get on with the job of creating
laser harps and even iPads. The sounds, whether that’s at a sample Project website
latest version talks to Arduinos and level for something like frequency http://chuck.cs.princeton.edu
60 www.linuxvoice.com
FOSSPICKS
Document conversion
Pandoc 1.17.0
P
andoc has been around for a
decade, but if you’ve never
taken a look at this brilliant
document conversion tool, now is a
great time to do so. It’s a command
line utility that deals in the dark art
of converting a document from one
type to another.
Recently the source format of
choice has come to be Markdown.
Markdown is a simple way of
marking text files to signal which
parts are titles and subheadings, for
instance, or items in a list or table
(along with other markings). It does
this in a way that doesn’t break the
readability of the original text file,
marking a heading by placing a line
of ==== symbols beneath, a sub
heading with ---- or an item within a
list with a preceding -.
When you’ve learnt this simple
The pandoc command
syntax, you can write in Markdown The only problem with all this requirements. You can
can turn one document
as quickly as with plain text. But power is that Pandoc’s array of into another, but it’s automatically word wrap, for
because Markdown started off as options can be bewildering, and you most useful for example, by adding --column=80.
an informal idea, rather than a need to have a good idea about the converting Markdown You can generate headers that are
corporate-sponsored specification, specifics of both your source into almost anything compatible with GitHub’s
lots of different people have been document and your potential else. markdown by adding --atx-
making lots of small changes. destination document to be able to headers. There’s even a specific
And this is where we’ve found choose the best command line module for GitHub-flavoured
Pandoc to be brilliant, because while arguments. Even then, we’ve nearly Markdown if you need an all-in-one
it can be used to convert always needed to resort to trial and solution, and getting
documents between re-usable text error. At its simplest, you call the documentation from source code
and word processors, it’s best when command and tell it which formats into GitHub is one of Pandoc’s
it’s converting between the broad you want to convert between: greatest features.
churches of Markdown, as well as pandoc --from=rst --to=markdown Output modules, such as the
letting you dive into lots of specific --output markdown.md input.rst Markdown one we’re using in this
arguments for each element you’d The power comes from being example, can have features turned
like converted and how you’d like able to augment these simple on and off, or strung together with a
them to appear in the end format. commands with your own series of + symbols. You can define
pipes as the separation character in
table, and define backticked code
blocks by following markdown with
+backtick_code_blocks+pipe_
tables, for instance, letting you
carve your perfect output in a
perfectly reproducible command,
and that’s without even looking
at the ePub, Latex, Docbook and
Docx formats that are also
There’s also a website
that will perform the supported by Pandoc.
same actions on your
documents without Project website
installing anything. http://pandoc.org
www.linuxvoice.com 61
FOSSPICKS
Unix utility belt
moreutils 0.59
T
he tagline for moreutils is “a descriptions; vipe inserts a text
growing collection of the editor into a pipe; lckdo runs a
Unix tools that nobody program that already has a lock;
thought to write 30 years ago,” and and ifdata is for grabbing a load of
we can’t do a better job at information on a network interface
summarising this excellent without going into ifconfig.
collection of tiny tools that do one
job each, but do that one job well. Let a thousand flowers bloom
There are currently 15 We also really liked the tiny but
commands in the package, immensely useful ts command.
including the dubiously named pee When running, this inserts a time
and the immensely useful sponge. stamp of anything pushed into its This is a collection of tools that adhere to the old Unix philosophy
Sponge is a great example of the input, whether that’s the time of a of only doing one job, but doing that job well.
kind of problems that these tools line in a script being executed or the
solve. It simply ‘soaks’ up the time some debugging output was utilities waiting to be judged worthy
standard input and places it in a file. delivered. The project has become of inclusion. We especially hope
It will attempt to update a file, rather so successful that there’s now a haschanged makes it, as this
than replace one, and will wait for all queue of other small but general creates a hash of a file when first
of the input before writing. This run and then checks whether the
makes it a good tool for pushing
parameters into configuration files.
“…a collection of the Unix hash has changed when run
subsequently.
Other commands are more tools that no-one thought
specific. errno, for example, will look
up the error number names and to write 30 years ago” Project website
https://joeyh.name/code/moreutils/
Google Drive client
GoSync 0.4
W
e’ve taken a step back need temporary access to. This has
from public cloud meant that the lack of an official
services like Dropbox, Linux client – long promised by
partly because of privacy concerns, Google – has affected its usability.
but also because we don’t want to GoSync is the latest third-party
become reliant on something that client we’ve spied, trying to fill this
may become expensive. hole. It’s built using Python 2.7, and
OwnCloud/NextCloud has been our we’d recommend using Python’s
drop-in replacement – it’s open pip tool to install it and its various
source, provides many of the same dependencies. If you’re still waiting for a GUI to Google Drive, try GoSync [image:
facilities, and can be run on a cheap The only thing you need to be Himanshu Chauhan].
VPS and even Amazon services for careful of is any conflict with a
very little money. Python 3 installation, as this is now need to go through the steps to
We’ve even had success running the default, but most distributions create your own clients+secrets.
the server off a modest Raspberry have a solution for running both json file, which means getting an
Pi 2 at home, although the storage (they may use pip2 as the install API key from Google. This is
access through USB does become command, for example). You also straightforward, although the
the bottleneck. But we must admit GoSync docs avoid telling you how
that we also use Google’s Drive to do this directly (although it does
service for convenience, especially The lack of an official Linux offer hints when you first launch it).
for those documents scanned with client for Google Drive has Project website
the mobile application and for large
affected its usability
https://github.com/hschauhan/gosync/
binary objects we know we’ll only releases/tag/v0.4
62 www.linuxvoice.com
FOSSPICKS
Two-factor authentication
FreeOTP+
W
e’d argue that two-factor you can install through FDroid. The We loved the original
authentication is now only problem with FreeOTP is that open source FreeOTP
essential, even if you’re you couldn’t easily back up your authenticator, and the
not obsessed with security. With so keys, which meant if you lost or simple features in this
many websites spewing so many broke your phone, you’d become new version make it
work the upgrade.
login details and so much personal locked out of the accounts you’ve
information, adding a second factor secured with FreeOTP. This is where
to the process of logging in is a a new fork of FreeOTP delivers –
good way of giving yourself more FreeOTP+. It adds what we’d argue
control and security. is an essential function – backing
One of the most common up your key database.
methods is through Google By default, it can save a JSON file
Authenticator, a tool that generates with these details to your Google
a one-time password (OTP) that Drive, which is probably wise, as
lasts only a short time. The you’re already likely to be securing
password is generated from a key your Google account with two-
that you add to the authenticator, factor authentication. You definitely
and while the application is now don’t want this file to be accessible,
proprietary, older versions were so encrypt it if you’re storing it
open source and have been forked locally, which is also an option.
into various new tools. Then, if the worst should happen,
Our favourite has always been you won’t need to beg your account
FreeOTP (now hosted by the Fedora holders to re-instate your access – Project website
Project), an open source just install FreeOTP+ and restore https://github.com/helloworld1/FreeOTPPlus
implementation for Android that your keys from the backup.
Open source maps
OsmAnd~
T
here can’t have been many open source, it’s available through The latest version of
Android users who haven’t F-Droid with no download limits or OsmAnd even includes
come across this wonderful restrictions. This is why there’s a turn-by-turn road
app. It’s a portal to the world of tilde ~ symbol in the project’s name naviation, complete with
open source maps and a great tool – this character is used to recorded or synthesized
voice.
for finding your way around, differentiate an open source build
whether on foot, on bike or in a car. from the commercial package,
But its best feature is that it’s open which adds a + to its name.
source, and so are the maps, Either way, the app is full of
mostly pooling resources from features, from the compass view
OpenStreetMap, but also sites such that changes according to the
as Wikipedia for other information. direction you’re pointing, the plugin
Unlike most other mapping system with view for skiing and
applications, all this data is sailing and the ability to turn on and
downloaded for offline using, which off the many OpenStreetMap
is perfect if you’re travelling in layers. You can even just download
locations with extortionate data the roads, if you’re sticking with the
fees. We’d highly recommend car, and this provision is one of the
paying for this app through through reasons we’d recommend paying
the Google Play store, which as the cost of the infrastructure
side-steps a download limit on the behind the map provision must be Project website
demo version. But as the app is also considerable. http://osmand.net/
www.linuxvoice.com 63
INTRO TUTORIALS
TUTORIALS
Warning: excessive Linux knowledge may lead to fun and more efficient computing.
In this issue . . .
66 68
Create your own news Publish with Free
Mike Saunders
Has almost finished porting Systemd to site with Calibre Software
MikeOS.
Ben Everard uses Calibre to condense the news, Publish to suit your style, use FOSS tools to free
so he doesn’t get too distracted by the rolling farce your creations, and transform text – also with
that is the United Kingdom these days. Calibre. Andrew Conway explains all.
S
omeone asked me the other day
what my first experience of
programming was. It took me a 74 78
while to remember exactly, but then my
neurons finally aligned: it was typing
out the BASIC source code for a
Breakout-like game from the ZX
Spectrum +2A manual (Bustout, at
http://tinyurl.com/jo2f65c). At the time
I had very little idea what the code did,
and I was only seven or eight years old, Raspberry Pi: Input data Harden many servers
but I remember throwing a massive
hissy-fit when the game wouldn’t work
with barcodes with Ansible
properly. Luckily, my brother went Les Pounder shows how the humble barcode can Sebastian Göttschkes takes you through Ansible,
through the code I’d typed in and found be used to do much more than tell us how much a a great way to prevent the most common attacks
the mistake. How right Linus was about can of beans costs. against web servers.
many eyeballs making bugs shallow…
People often deride BASIC for being a
bad starter language, and it’s true that Coding
ZX Spectrum BASIC with its line
numbers and GOTOs was an awful
Get access to ev
language that encouraged spaghetti ery
Linux Voice tuto
code. But it really encouraged me to go rial ever
published in ou
further, try new languages, and poke r digital
library of back-i
around inside the machine. I then ssues
available exclus
moved on to the Amiga (let me wipe a ively to
subscribers – tu
tear from my eye) and then Linux. Who rn to
page p56 to join
would’ve thought that I’d write my own Coding: log files 82 Amazon Beanstalk 86 .
BASIC interpreter in assembly language Use AWK and R to reveal Amit Saha hosts and scales
for my own OS 25 years later… information from your Apache a Golang web application on
mike@linuxvoice.com logs, with Mihalis Tsoukalos. Amazon Elastic Beanstalk.
www.linuxvoice.com 65
TUTORIAL CALIBRE
CREATE YOUR OWN NEWS
WEBSITE WITH CALIBRE
Wrap up the latest news as an eBook and share them on your personal website.
T
BEN EVERARD he web has made a phenomenal amount of downloads the latest news from whichever news sites
information available, much of it up-to-the- you want, packages these stories up as an ePub, then
Why do this? minute news. However, web browsers don’t serves these ePubs on your own website so you can
• Deprive Murdoch of always make good environments for reading the grab them from your portable devices.
advertising revenue news. There can be flashing adverts, pages can be A word of caution though: the more you focus on
• Save bandwidth slow to load and jump around as you read them, and what you want to read, the less you'll accidentally
• Avoid having to be you may want to read the news when you’re offline. In stumble across things that you don't want to read, but
confronted with this tutorial, we’re going to get around all these might find interesting. Beware of trapping yourself in
opinions that differ
from your own problems by building a system that automatically your own information bubble!
STEP BY STEP: PUBLISH YOUR PERSONAL NEWS
1
Install Calibre 2
Get an eReader
The first thing we’re going to need is the software to We’re going to convert the news sources into ePub
run our newspaper website. In this case, it’s just a format, so you’re going to need a way to read these.
single application that does everything: Calibre. If you This doesn’t have to be specialist hardware – there
run a server, that’s ideal; if not, this can be your home are plenty of eReaders available for Android and iOS
PC, but you’ll only be able to use the website when the devices as well. Cool Reader for Android is open
PC’s switched on. If you want to be able to access source (available via F-Droid or the Google Play store),
your website from outside of your LAN, you’ll need to and a capable reader for phones and tablets.
set up port forwarding on your router and dynamic Adjusting the font and background of the reader may
DNS – see your router’s documentation for details of make reading easier on your eyes.
how to do this. For the best reading experience, eReader hardware
You can use Calibre to manage your digital library as has eInk screens that put less strain on your eyes. Any
well as running your news website. If you’re a Linux of the eReaders that support ePubs will work with this
Voice subscriber, you can import our ePubs and make setup provided they have a web browser to enable
them available on all your devices. them to download the files.
66 www.linuxvoice.com
CALIBRE TUTORIAL
3
Set up news collection 4
Start the web server
Now you’ve got all the software you need, it’s time to Calibre should start the first news download shortly
configure Calibre to download the news you want. after it’s set up, so the next thing we need is the web
Websites aren’t usually designed to be converted into server running. You can do this in two ways, either
ePub format, so the software needs to know how the through the GUI or from the command line. In the GUI,
site is structured in order to get the latest information go to Preferences > Sharing Over The Net to open the
without clogging up your computer with the entire web server options window. The defaults should be
content from the site. Fortunately, Calibre comes with fine unless you want to password-protect your library,
over 1,500 news sources already configured, so or if you want to run on a different port because you
there’s a pretty good chance it knows how to already have a server on 8080. Press the Start Server
download the items you want. button to get online.
Click on the Fetch News button, and find the news You can also use the calibre-server command to
source you want in the list. Check the Schedule For launch the web server from the command line. It
Download box and set the download frequency (by doesn’t deamonise, so you may want to run it via
default, this will be every day). Once you hit Save, tmux or create an init/Systemd script.
Calibre will add that site to your schedule and start
grabbing the news.
5
Read and enjoy 6
(Optional) Create a custom news source
Your library is now online. Head to localhost:8080 in We’ve looked at adding a news source that Calibre
your web browser on the same machine Calibre runs already knows, but you can also add any site you
on to make sure everything’s set up. From other want. Click on the drop-down arrow next to the Fetch
machines on the same local network, you can access News button and select Add A Custom News Source.
your Calibre library via the IP address of the server. The easiest way to add a news source is via RSS. For
Enter the command ip addr in a terminal and you’ll get example, to add a news source for LinuxVoice.com,
lots of details about the machine's connection. Look give the recipe a title, then you just need to add a
for the block that details your network connection, and single feed. Add the feed URL https://www.
in that, you’ll see a line that starts with inet followed by linuxvoice.com/feed/ with another title, click on Add
four numbers separated by full stops. For example: Feed, then Save. Creating the feed won’t automatically
inet 192.168.0.19/24 schedule a download, so you’ll have to add your new
In this case, you can use the URL source as per step 3. If the source you want doesn’t
http://192.168.0.19:8080 to access your library from have RSS, you can still add it, but you’ll need to use
other machines on the same local network. Advanced Mode, which involves Python coding.
www.linuxvoice.com 67
TUTORIAL PUBLISHING
PUBLISH FREELY
WITH CALIBRE
Influence global ideas by self-publishing your scibbles on Free Software.
Y
ANDREW CONWAY ou can use one of many good word website somewhere. So have a look, and if they ask
processors to do your writing, or you can work for a stack of papyrus with hieroglyphs from a
Why do this? in plain text with formatting specified by particular Egyptian dynastic period, then that is what
• Publish to suit your markup (or down) languages and an array of free and you should send them. Thankfully, most publishers
style open source software tools. With these you can bang aren't that fussy (Linux Voice accepts submissions
• Use FOSS tools and out the words and create your very own magnum from any dynastic period) but most first-time authors
free your creations opus, but what good would that be if no-one ever are still asked to provide a chapter or two of their text
• Transform your texts reads it? printed on paper. The idea is that emailing an
with Calibre
There are two main routes to publishing these days: electronic copy is too easy and publishers like to erect
you need to either find a publisher or self-publish. some barriers to deter half-hearted submissions.
Either way, at some point you'll have to decide on a But, for whatever reason, it is likely that you'll have
format for your book. Even if you're aiming for a your text in one format and find that you need to
dead-tree version of your book, these days people will provide it in some other format. Last issue we went
Calibre is licensed under
expect there to be an eBook too, so it's something that through the process of creating text in markdown and
the GNU GPL v3 and is
available on Linux via your cannot be avoided. then turning it into HTML and CSS, and we showed
distro's repositories, but First off, if you're seeking a publisher, especially if how this could be manually turned into an EPUB file,
builds are also available for you're unpublished, you must submit exactly what a one of the most popular eBook formats.
Mac OS X and Windows. publisher asks for, which is usually specified on their In this article we'll look at two related things: how to
68 www.linuxvoice.com
PUBLISHING TUTORIAL
convert formats, and also some platforms on which the table of contents can be tricky to get right. To be
you can self-publish. We'll also pay some attention to fair, this is true of any ePublishing platform, but the
formatting for print because, as mentioned above, this downside of KDP making it very easy to publish is that
is still often required when seeking a publisher. it lulls authors into a false sense that all aspects are
easy, and there are quite a few books published using
Where to self-publish it where the author has not taken enough care to get
We were genuinely amazed at how many ePublishing the details right. That said, Amazon has made an
platforms there are these days. It seems that many an effort to help authors improve quality, such as an
investor has parted with money to fund entrepreneurs automatic spell-check of the entire text when you
and their start-ups that cater for every conceivable upload your book. Our main advice for KDP, and in fact
ePublication niche. We can't cover them all, so we've for all platforms out there, is to check that the finished
picked three that represent three quite distinct parts of
the spectrum. Notice that we use the word 'platform'
rather than 'publisher' at times – the distinction may
The technical process of publishing on
be a little pedantic, but part of the meaning of the Amazon is not too difficult, though it is
word publisher is to publicise your work. With
self-publishing, the onus is really on the author to push difficult to do well
their work out into the world, making best use of the
many self-publishing platforms out there, and, as product looks acceptable on as many eReading
most are not exclusive, there's nothing to stop you devices as you can, including phones and tablets.
selling on many simultaneously. To get started with KDP, go to kdp.amazon.com
and have a read of the documentation there. There's a
Amazon KDP 100 page Amazon Kindle Publishing Guidelines
Let's start with the most famous: Amazon Kindle document that's worth at least a skim. Unfortunately,
Direct Publishing (KDP). The most obvious reason to although there is a lot of very good information in that
go with this big corporate is that it is so well known. guide, and elsewhere, not all of it was up to date
The second reason is that it offers a higher (advice on the use of UTF-8 character sets was
percentage of royalties than a traditional publisher but, conflicting, for example). Nevertheless, if you've read
as we'll see, not the highest among ePublishing the basics, it is quite feasible to go from signing up to
platforms. (Royalties are the percentage of money KDP to having your book live and selling within 24
from sales that go to the book's author.) A book I've hours (not including writing the book of course!).
just published on Amazon gets 70% royalties, whereas One of the best ways to prepare a book for
one that I wrote over 10 years ago, prior to the uploading to KDP is to use Amazon's Kindlegen tool.
ePublishing revolution, gets just 10% royalties. But Happily, this tool is available for Linux via a binary
don't forget, with self-publishing you have to put in the tarball download, although the software itself is not
work that a traditional publisher might have done for released under a FOSS licence. We downloaded it and
you, otherwise you may end up with 70% of nothing. got it working without any dependency issues. If your
The technical process of publishing on Amazon is book is just in one file called book.html in directory
not too difficult, though it is difficult to do well. There /home/fred/books, then a command as simple as
are subtle aesthetics that can impact both sales, like a this can do the trick from the directory where the
poor cover, and the reading experience, such as layout kindlegen binary is located:
and appearance, and some technical details such as ./kindlegen /home/fred/books/book.html
Comparison of eBook platforms
Website Input formats Download formats Monies to author DRM Exclusive
kdp.amazon.com HTML,EPUB,MOB Via kindle app 70% royalties Up to author Up to author
leanpub.com markdown,DOCX EPUB,PDF,MOBI 90% minus $0.5 per transaction No No
unglue.it *no conversion* EPUB,PDF,MOBI 92% minus $0.25 per transaction No No
www.linuxvoice.com 69
TUTORIAL PUBLISHING
Calibre quick tour
Calibre's interface is divided into four main areas. For Flatland we have the downloaded EPUB and the panel, which brings up a context menu. At the
The main one in the middle shows you books in MOBI format that Calibre created for us. The button bottom-right you can see the Jobs indicator, which
your library, and you can use the left-hand panel to panel at the top enables you to perform common displays progress of time-consuming tasks. To the
filter them by author, tag, language and more. The tasks such as adding books and converting them left of that are some icons that control the layout of
panel on the right gives you some brief information between formats. Most of those functions are also Calibre's display, which toggle the visibilities of the
about the book including a list of available formats. available via a right-click on a book in the main three panes below the buttons.
This will output the file book.mobi, which you Document Licence, but neither have Amazon given
upload to the KDP website and publish. Kindlegen can clear answers on this point when asked. What is
convert books of much greater complexity from an crucial, and indeed important on any platform, is that
EPUB or from a directory or Zip of HTML and image you own the copyright for the text.
files. If you followed last issue's tutorial then you can
either give the EPUB file directly to the kindlegen Leanpub
command, or skip the EPUB creation and just give it Leanpub's strapline is "Publish early, publish often".
the content.opf file, from which it will find all HTML This may give you a clue as to the intended audience:
content files, image files, the CSS file (for styling) and, folk who like developing in the open. Unlike Amazon,
if present, .ncx or other files for generating the table of Leanpub is not just about publishing to a mass
contents. market and making the author many sponduliks;
If you've written your book using a word processor, rather it provides a platform for the author to write and
such as LibreOffice Writer or Microsoft Word, then you'll develop her or his book and, if they choose, to do so
need to export it as an HTML file first. We've not within a community. To this end Leanpub's platform is
tangled with this method, but there is some advice on designed to make it easy for readers to contact the
how to do it on the KDP website. author and make comments on books on dedicated
To our surprise, KDP places few restrictions on feedback pages. They also encourage authors to put
exclusivity. You can publish your book elsewhere and works in progress on Leanpub; in fact, that's part of its
at different prices, but Amazon warns you that they raison d'être.
may drop the price to match a lower one offered Unsurprisingly, Leanpub's catalogue contains many
elsewhere, and they also try to tempt you to be books on technical subjects related to writing code,
exclusive with enhanced promotional benefits. We especially in a FOSS context. Leanpub's very first
were also pleased to discover that the author gets to book, Startup Lessons Learned was written by one of
decide whether Digital Rights Management (DRM) is the proponents of the Lean startup model for fledgling
applied to their book. There's no obvious reason why companies, from which Leanpub got its name.
KDP's terms and conditions would prevent use of Leanpub is competitive with their royalties, offering
Creative Commons licences or the GNU Free authors 90% less 50 cents per transaction. Not only
70 www.linuxvoice.com
PUBLISHING TUTORIAL
Unglu.it offers three
payment models, so
you can choose the one
that fits best with your
commercial goals.
that, but you can set a recommended and minimum charged and anyone can read the book for free, and
price and then let potential readers pay whatever they remix it within the terms of the chosen licence. The
wish. The minimum price can even be zero with the second method, buy-to-unglue, is similar, except that
50 cents fee waived. We've used Leanpub and while instead of pledging, individuals buy copies of the
we saw some readers nab a book for free (fine, eBook and each purchase moves the book closer to
because we said they could!), we also saw others pay being unglued once the target is reached. Both of
over the recommended price. As Humble Bundle these methods are essentially variants on
demonstrated for games, and to a lesser extent for crowdfunding except that the book has to be
eBooks, allowing users to set their own prices does complete at the outset.
not necessarily lead to a crash in prices. The third method, thanks-for-ungluing, is different.
Plain text using Markdown is the preferred writing Here the author releases the book under a Creative
format, and they've created their own flavour of Commons licence first and then simply asks
Markdown called Markua. There's even a book on the supporters to make donations as a gesture of
Markua specification published on Leanpub. You can appreciation. Needless to say, whatever method is
either write using their online editor or else on your chosen, the end result is that the book becomes freely
computer using any text editor you like and use their available in electronic format without any DRM. EPUB,
integration with Dropbox or GitHub to synchronise PDF and MOBI formats are supported. Clear
your local copy with the one on their website. Either instructions for downloading books to your eReader
way, when you make a change to your book you can, are given, and if you sign up for an unglue.it account
at the press of a button, create PDF, EPUB and MOBI you can send books direct to your Kindle.
versions and peruse an online preview. Overall, if For each contribution that is made to a book
you're comfortable with Markdown, this is even easier (there's a minimum of $1), Unglue.it takes $0.25 plus
than using Amazon KDP's tools. Also, if Markdown is 8%, and the remainder goes to the creator of the work.
not to your taste and you prefer working with Writer or However, Unglue.it is part of a not-for-profit
Word, then you can opt to save a .docx file to a corporation called the Free Ebook Foundation, so the
Dropbox folder and have that transformed into the proceeds it collects will further the cause of freeing
above formats. eBooks. Overall, this is an excellent way to promote
As you might expect, Leanpub doesn't stipulate any
licensing and doesn't mind if you make your book
available elsewhere. Unglue.it aims to free books by getting
Unglue.it
them released under one of the
Unglue.it aims to free books by getting them released Creative Commons licences
under one of the Creative Commons licences – to free
them from being stuck in the "glue" of traditional freedom in creative works and in addition to any new
publishing and copyright restrictions. It serves the books that are released freely, it offers a new
three functions you might expect of an ePublishing electronic lease of life to a sea of out-of-print books
platform: it helps you to publicise your book, it enables that publishers have lost interest in.
people to download it, and you can make money. The
copyright remains with the author and you can't Many others
unglue a book without the copyright holder's consent. There are of course many more platforms for
There are three routes to ungluing your eBook. The launching your ebook, including offerings from big
first is pledge-to-unglue, in which individuals each players such as Apple with iBooks, Google Play, and
pledge an amount of money and once a target set by smaller but significant platforms such as
the author is reached, the pledgers' credit cards are Smashwords. In addition to eBooks there are some
www.linuxvoice.com 71
TUTORIAL PUBLISHING
that specialise in self-publishing to print, such as Format is MOBI. To do the conversion, click on the OK
Amazon's CreateSpace or Lulu. button at the bottom-right and after a few moments
Many of the services out there, including two of the of the 'Jobs' indicator at the bottom-right whirring
ones mentioned above, provide ways of converting to round, you'll have a MOBI edition of the book. You'll
common eBook formats. However, if you want more see that EPUB and MOBI are both listed as formats in
control of details – and as mentioned above, details the right-hand pane.
such as the cover and styling are important – then You can save the EPUB or MOBI or any other
you'll want to perform the conversion process format that's available via the right-click context menu,
yourself. If you're not averse to a bit of XML you can but you can also just go to the Calibre Library directory
craft an EPUB manually, but XML is, almost by design, in your file manager (or on the command line) and find
a human-unfriendly markup language, so most of us the .epub or .mobi files there.
will be more comfortable using an application that will If you've constructed a book from source in HTML
automate it, and the FOSS star on this stage is Calibre. then you can use Calibre to assemble it into a full
Calibre is an application that manages libraries of eBook in a format of your choosing. If you don't have
eBooks and enables you to convert between a wide such a book to hand, download the very simple
array of formats. There are also many plugins that mybook example (it's only got one page) from the
add to the basic functionality: these range from previous article from here: github.com/mcnalu/
statistics functions on wordcounts to plugins that can linuxvoice-publishing. Click the Add button and select
remove DRM from eBooks. the HTML file you wish to use, and Calibre will pull it
The latest versions of Calibre are built around the into its library.
Qt 5 framework and look slick as a result, though its At this point the book will only be held in ZIP format,
button-cum-menu interface might feel a little odd and if you try and view it, it'll just show you the files
at first sight. See the boxout for a quick tour of its within it. If you look closely, you'll notice that Calibre
main window. had spotted that a CSS file is specified and brought
Calibre comes pre-installed with one book: its own that into the library too. To turn the book into a proper
Quick Start Guide. Double-click on it in the middle eBook format, just hit the Convert button as described
above. Once this is done you'll see the new format
Most of us will be more comfortable (MOBI by default) listed on the right, and viewing the
book will now bring it up in Calibre's viewer.
using an application that will automate
Write on, commander!
the creation of our EPUB file With software such as Calibre, you can take charge of
your writing. You can alter the look and feel, edit the
pane and it will open in Calibre's eBook viewer. This is CSS and even take advantage of its 'heuristic
worth a few moments of your time, and at only 37 processing' feature to automate mundane tasks that
pages it doesn't take long to flick through. may take hours to do by hand, such as removing
Adding books to Calibre is easy enough. Click on the blank lines and dealing with unwanted wrapping of
Add Books button at the top-left and a file browser lines. It can also detect the structure of your book and
window will open. Select any book file that you generate a table of contents for it. This is all possible
happen to have lying around on your computer and with command-line-fu, but the beauty of Calibre is it
Calibre will pull it into its library. This copies book files gives some of this power to non-technical users.
into the Calibre Library directory, which is placed in Despite all the applications and services that allow
your home directory. If you don't have any book files, you to create and publish your book to the world, there
then we recommend downloading the book Flatland is still one bit that no computer can yet help you with:
from unglue.it – a book written about what it'd be like the creative process of writing. For most authors – at
to live in a two-dimensional world, written by Edwin least ones who aren't simply doing it for money – they
Abbott in 1884. write because there's something they feel they must
Converting eBooks is straightforward. We express. Learning how to perform some technical
downloaded the EPUB of Flatland for this example, but task, such as creating an eBook, is interesting in itself,
the same principle applies to any book and in any but for many of us it is not quite enough; we must also
format. Select the book in the main pane by clicking share the knowledge, be it in book, blog or forum post.
once on it, and then click on the Convert Books (or you It is said that everyone has a book in them. Whether
can select it via a right-click context menu). The that's true or not, it's certainly worth asking yourself if
conversion window opens and offers you a you have some knowledge you'd like to share. The art
bewildering array of things you can change, from the of good writing may remain as challenging as it every
cover image and metadata, all the way down to was, but the process of publishing a book has never
making search and replacements in the text with been easier.
regular expressions. For now, don't worry about any of
that, just notice that at the top-left it says the Input Andrew Conway watches the stars from his wood-panelled study.
Format is EPUB and at the top-right the Output He likes open data and what you can do with it using Free Software.
72 www.linuxvoice.com
PUBLISHING TUTORIAL
www.linuxvoice.com 73
TUTORIAL LINUX FOR LEARNERS
BARCODE INTERFACES
WITH EASYGUI
The humble barcode can do much more than tell us how much a tin of beans costs.
T
LES POUNDER he humble barcode is all around us. Our
parcels, beans and medicines are all
Why do this? catalogued using a series of thick and thin
• Add a new input lines. In issue 20 we used barcodes from store
method to your cupboard staples to create an adventure game similar
projects to Pokemon, you can find the code for this at https://
• Use lasers! github.com/lesp/LinuxVoice_Issue_20.
Barcodes are great fun, and they can also be used
You will need as a simple form of input. Using a barcode scanner
• Any model Raspberry purchased from Amazon for around £30 we can
Pi running the latest create a novel method of input that any child can
Raspbian release
operate. The scanner works as a true plug-and-play Playing Minecraft by scanning a barcode is just one of the
• A barcode scanner
device, requiring no installation or drivers; it simply many actions that we can trigger with our new toy.
• Female–male jumper
wire appears as a human interface device similar to a
• 3 x LED (red, amber, keyboard. Once a barcode is scanned the value is really easy thanks to online services such as http://
green) decoded and sent as standard input to the computer; www.barcode-generator.org . We chose to create
• A breadboard this is followed by a virtual Enter keypress that will CODE39 barcodes and then save them as SVG files.
• 3 x 220Ω resistor (red, enter the value. CODE39 barcodes are incredibly flexible and permit
red, brown)) There are many types of barcodes, including the the use of alphanumeric characters, enabling us
EAN13, CODE39 and ISBN variants. EAN13 is used to add text to our barcodes. With a little help from
in retail, so you will see this barcode on your tins of Inkscape, https://inkscape.org/en we created a series
The circuit for our flashing
LED code is relatively beans. CODE39 is used by parcel companies, and of A5-sized sheets that contained all of the barcodes
simple, requiring only a few ISBN is used by publishers to catalogue books, and for a particular game. We have included our cards as
cost-effective components is commonly used on Amazon to search its vast part of the downloads for this project.
to build. database of books. Creating your own barcodes is The barcodes created can be used to control
or trigger many different aspects of a project. For
example, we could have a barcode that triggers a
robot to drive around: the reader can be mounted on
top of the robot and it can scan the floor before it.
Once it detects a barcode it can react by turning in
another direction, playing a sound or flashing some
lights. Let's get started and build a new game using
our own custom barcodes and a little Python code.
Hardware
Our hardware installation is rather simple and
comprises two parts. First, we can plug in the USB
barcode scanner into a spare USB port on our
Raspberry Pi. Second, we'll need to build the circuit for
our flashing LEDs. This circuit is relatively complex,
requiring that we connect each of the LEDs to a GPIO
(General Purpose Input Output) pin. We attached the
long leg of each LED, red to pin 2, yellow to pin 3 and
green to pin 4 via a resistor to limit the current and
using male–female jumper wires. Please see the
diagram for a suggested layout – a high-resolution
74 www.linuxvoice.com
LINUX FOR LEARNERS TUTORIAL
version is available via the software download for this
project. Once the wiring is ready, attach all the
accessories required to use your Raspberry Pi and
boot to the Raspbian desktop.
Software
We start the software section of this project by
opening the Python 3 editor, which can be found in the
main menu at the top-left of the screen. In the menu
look for Programming and then Python 3. Once the EasyGUI produces dialog
editor is open, click on File > New and create a new function from the Random module. We shall use it to boxes that match the look
blank document. It is best practice to save often, so generate a random number in our code. and feel of the operating
we shall save the blank document to ensure that import subprocess system on which it is being
subsequent saves are quicker. Click on File > Save and import time used.
save your project as Barcode-project.py. import easygui as eg
This project is coded entirely in Python 3 and uses from gpiozero import TrafficLights
a number of modules, sometimes referred to as import pygame
“libraries”. The majority of the modules that we use are import pygame.mixer
installed by default on Raspbian, the only exception from random import randint
being the EasyGUI module, which we will need to Our next section of code contains four functions,
install manually. We shall open a terminal – you can the first of which is an audio player. This function
find the icon for this on the Raspbian desktop – at the takes one argument, the audio file to play. Inside the
top-left of the screen looking like a small computer function we initialise the audio mixer ready to start
monitor. Open the terminal and run the following playback. We then load the audio file, queuing it ready
command, press Enter to execute. for playback. Lastly we then play the audio file, with
$ sudo pip3 install easygui the number of times it is played bieng controlled by
Once this command has completed, you can close the value in brackets at the end of the line.
the terminal and return to the Python editor. def audio(file):
To work with modules we need to import them, pygame.mixer.init()
and we do that first of all. Our first module is called pygame.mixer.music.load(file)
subprocess, which is used to interact with the pygame.mixer.music.play(1)
underlying Raspbian operating system; we shall use it Our second function is a number game, where
to call commands via the Terminal. The next module a random number between one and five is chosen
is called Time, which is used to control the pace of using the randint function. The random number is
our project, for example setting delays for how long stored in a variable called number, and on the next
an LED should be on/off for. We import the EasyGUI line we create another variable called guess into
module but rename it to eg; this shortens the rather which we store a placeholder value. We now use a
long module name. Our next import is GPIO Zero, and while loop, this loop will repeat while a value is True.
from it we import the class TrafficLights. This class In this case it will keep repeating if the value of our
can be used with three LEDs and control them in a guess is not equal to the randomly chosen number.
similar manner to common traffic lights. Our next two Inside of this while loop we update the value stored
imports are Pygame and Pygame.mixer. Pygame is a in the variable guess; it is now used to store the
game/media creation framework for Python, and we answer to a question. To ask a question we use the
are using its mixer to play audio clips on demand. Our easygui module, specifically the enterbox function.
final import is the randint, short for random integer, This function produces a dialog that will wait for user
EasyGUI
In this tutorial we used the fantastic EasyGUI to create a user the following into the Python shell that we just opened.
interface. EasyGUI is a cross-platform module, working with eg.msgbox(title="Hello Reader",msg="Thanks for reading Linux
Windows, Mac and Linux. We’ve used this module in previous Voice")
tutorials as it provides an easy introduction to the various A little more advanced, but still easy to use, the File Open
dialogs and prompts that a computer uses to communicate dialog box is something that we see everyday. Using EasyGUI
with the user. Using EasyGUI is relatively straightforward, and we can create this dialog, which will capture the full path to
it comes with a series of dialog boxes. To see them all, open the file that you wish to open. This can then be saved to a
the Python 3 editor and in the Python shell type. variable. Here we use the dialog to capture the filename that
import easygui as eg we wish to process
eg.egdemo() filename = eg.fileopenbox(title="Open File",msg="Please open a file")
The most basic dialog is a message box, or msgbox. This is You can read more about EasyGUI on its documentation
used to advise the user. To create a basic message box type page at http://pythonhosted.org/easygui.
www.linuxvoice.com 75
TUTORIAL LINUX FOR LEARNERS
TrafficLights class that we have LEDs on GPIO pins
2, 3 and 4. Pin 2 is red, 3 is yellow and 4 is green. The
TrafficLights class expects the LEDS to be presented
in that order. We then turn the green LED on ready for
the sequence to start.
def flashing_LED():
lights = TrafficLights(2, 3, 4)
lights.green.on()
Still inside of the function we now use a for loop
that will loop twice before exiting. Inside the for loop
we create a sequence that will turn each of the LEDs
on and off according to the sequence in which a
traffic light controls traffic. To ensure that the LEDs
When the barcode is
input, and this is where our barcode scanner is used. are on and off for the correct amount of time we use
scanned the information
is automatically entered The enterbox function takes a number of arguments; time.sleep and pass the function the correct number
into the dialog box. It even for our project we use title and msg. Title creates of seconds to wait.
presses the Enter key for a title for the dialog, and msg is used to talk to the for i in range(2):
us. user. In the msg argument we use \n; this is a Python time.sleep(10)
instruction to create a new line, helping us to keep the lights.green.off()
lights.amber.on()
This project can be extended into an time.sleep(1)
lights.amber.off()
interactive media player that uses lights.red.on()
time.sleep(10)
barcodes to play videos lights.amber.on()
time.sleep(1)
dialog box tidy. At the very end of the msg we see lights.green.on()
…when doubled it is"+str((number*2))) lights.amber.off()
This is a method of connecting the number chosen, lights.red.off()
multiplying it by two and then converting the data Our final function is called quiz and as the name
type to a string, as only identical data types can be suggests it is a simple quiz that asks one question.
joined together in this manner. By doing this we can We start the function by creating a variable called Q
give the user a hint to solve the math problem and and in there we store the answer to a question asked
keep the code agile so that it requires no manual
updates. On the next line we convert the user's guess
into an integer; this ensures the user only provides
numerical values. As the while loop is still running, if
the guess and the random number do not match, we
call the audio function and instruct it to play a sound
indicating a wrong answer. In the last part of the
while loop, if the guess matches the random number
then the first condition is False, and means that the
Else condition is active, triggering the playback of the
correct audio file.
def numbergame():
number = randint(1,5)
guess = 0
while guess != number:
guess = eg.enterbox(title="I'm thinking of a number
between 1 and 10",msg="What number am I thinking of? \
nHere is a hint, when doubled it is"+str((number*2)))
guess = int(guess)
audio("/home/pi/wrong.wav")
else:
audio("/home/pi/correct.wav")
Our third function is used to flash the LEDs
attached to the GPIO. The sequence is a typical UK
traffic light. To control the LEDs we use GPIO Zero
and its built in TrafficLights class. We start the We imported our barcodes into Inkscape and made our
function by defining its name and then instructing the own project cards.
76 www.linuxvoice.com
LINUX FOR LEARNERS TUTORIAL
GPIO Zero
GPIO Zero was introduced in late 2015 as an alternative
resource to work with the Raspberry Pi GPIO. Typically a
user wishing to work with the GPIO would need to use the
RPi.GPIO module, which is still being actively developed
and used in projects. But this module is a little tricky for
those new to coding as it requires the user to know a little
electronics and understand how the GPIO pins work. This EasyGUI has many
can be complicated, and this is where GPIO Zero comes in. while loop and return to the main while True loop.
different dialog boxes and
GPIO Zero provides a number of classes and functions if code == "MINECRAFT": includes many advanced
that enable the user to just go ahead and build a project. print(code) features that can be
Using the LED class all we need to do is tell GPIO Zero
which pin we have connected the LED to, then we can use
subprocess.call(["minecraft-pi"]) dropped into your project.
functions to turn the LED on, off, fade or flash the LED break
with no need to use a loop. Other classes include working The next condition to test is handled using an else
with motors, including a class to build a robot controlled if statement, which in Python is shortened to elif. For
by a user with a keyboard (or for a more advanced project the first of the elif conditions we compare the value of
sensors can be used). Sensors includes a passive infrared
the code variable against the hard-coded value SONIC
(PIR) which detects movement based on body heat,
ultrasonics, which use ultrasonic pulses to judge distances PI. If this condition is true then the code will launch
and detect objects, and a light sensor, which can be used to the Sonic Pi application.
measure a light level and react accordingly. elif code == "SONIC PI":
print(code)
subprocess.call(["sonic-pi"])
by the EasyGUI Enterbox function. If the answer break
given is not the same as the hard-coded answer, in For the next three elif conditions we compare the
this case A, then we call the audio function that we scanned value against the hard-coded values for LED
created earlier and play the wrong answer audio file. If FLASH, NUMBER GAME and QUIZ.
the answer matches then the player hears the correct elif code == "LED FLASH":
answer audio file. print(code)
We now move on to the main body of code and the flashing_LED()
logic that controls our project. break
We start the main body with an infinite loop, while elif code == "NUMBER GAME":
True. This loop will constantly run until the project is numbergame()
PRO TIP
exited. inside the loop we create a new variable called break
All of the code for this project can be
found at https://github.com/lesp/LV30-
play, which is used to store the answer to a question, elif code == "QUIZ": BarcodeProject/archive/master.zip
again asked via the EasyGUI enterbox function. quiz()
while True: break
play = eg.enterbox(title="Would you like to play a We now break out of the if…elif conditional
game?", msg="Would you like to play a game?") statements and return to the main while True loop.
Now we create a new while loop, which will At the start of this loop we said that while the value of
continue while the value of the variable play is YES. the variable play is YES, run the indented code. But if
Inside this new loop we firstly play the 'correct answer' the player answers no to the question “Would you like
jingle, to indicate that the program is working. We to play a game?”, this condition is False and so the
next create a variable called code that will store the else condition is activated, causing the code to play
barcode scanned by the user. the 'wrong answer' audio clip and then exit the game.
audio("/home/pi/correct.wav") With the code completed ensure that it has been
code = eg.enterbox(title="Please scan a barcode to saved and that your cards are ready. Click on Run
start",msg="Please scan a barcode to start") > Run Module to start the code. The first question
Still inside of the while play == ”Yes loop we now asked will be if you would like to play the game, using
create a series of conditional statements. These are the barcode scanner and the Quiz card answer yes
tests that will check the contents of the code variable to continue. The next dialog will ask you to scan a
against a series of hard-coded values, the same barcode to choose a game. Pick any card and scan
values that are encoded into the barcodes. the barcode to interact with the game.
Our first condition to test is whether the contents This project can be extended into an interactive
of the code variable are the same as the value media player that uses barcodes to play videos.
MINECRAFT. If that is true then the code inside of Imagine encoding the barcode of a book into the code
that condition is executed. So for this condition we and using it to trigger scenes from a play or movie
print MINECRAFT to the Python shell, as a means based on the book!
of debugging the code. We next use the subprocess
module and call the minecraft-pi command as if we Les Pounder makes things, breaks things, and spends the rest of his
were using the terminal. Lastly we break out of the time teaching teachers about the new IT curriculum.
www.linuxvoice.com 77
TUTORIAL ANSIBLE
HARDEN YOUR SERVER
USING ANSIBLE
Create playbooks to be run on any number of servers to secure them.
I
SEBASTIAN f you ever set up a web server and paid attention This is where Ansible comes in. Ansible describes
GÖTTSCHKES to the iptables logs, you might have noticed that itself as a tool for “IT-Automation”, which means that
the server was getting packets on various ports after defining tasks (like running apt-get update on a
Why do this? minutes after it got online. This is especially true if Debian system), they can be executed on any remote
• Prevent the most your IP address belongs to one of the cloud providers server you have access to through SSH. The idea is
common attacks
against web servers such as AWS, DigitalOcean or a large data centre. that instead of running commands on your servers
• Run tasks on every It’s very likely that some of those packets were manually, they are all defined for Ansible and can
server without even sent from software looking for new servers on the be run multiple times, against multiple servers, to
logging in manually
internet. On each server the software finds, it tries produce the same outcome. Ansible is run from your
• Repeate the same
various attacks that are known to work on unpatched local machine and is agentless, meaning that there is
tasks on new servers
with a single line. software versions or for unsecure settings. nothing to do on the remote host in order to use it.
This means trouble for insecure servers, but it also To get started, you need to install Ansible on your
means that with an up-to-date system and some local machine. This can be done on Debian-based
basic security settings enabled, these attacks won’t systems with
be successful. There are already a lot of tutorials on apt-get install ansible
how to harden a server against the most common or on any other system through pip (Ansible is written
attacks. The caveat is that while most of them are in Python):
not very long, the steps still need to be executed on pip install ansible
every server you run. If there is an updated version To run Ansible, we need two things beside the
of a specific software (like SSH or the firewall), you software itself: a so-called inventory file telling Ansible
might need to update all those servers and maybe which servers to execute the tasks on, and a playbook
adjust the configuration as well. This can add up and which contains the task to be executed. The inventory
Ansible runs on your local ultimately lead to you not doing it at all. Keeping track file follows the syntax of ini-files and contains
machine and uses SSH of which configuration is in place on which server and hostnames, IP addresses and specific settings for
to execute commands on what was installed is also a problem if the number of each host. It is also used to cluster hosts into groups,
remote hosts. servers grows. which makes it easier to run tasks in only one group
of servers (eg web servers and database servers). An
example inventory file might look like this:
server1 ansible_host=127.0.0.101
server2 ansible_host=127.0.0.102 ansible_user=deploy
server3 ansible_host=127.0.0.103 ansible_port=30
[webserver]
server1
server2
[dbserver]
server3
In this example, we define three servers: the first
one is called server1 and has its IP address set to
127.0.0.101; for the second server, we define that
Ansible should log in using the user deploy; the third
server has a different SSH port set. There are many
other settings that one can define here, and all of
them are optional.
We also put the three servers into two groups, one
called webserver and one called dbserver. These
groups can be used to determine which tasks should
78 www.linuxvoice.com
ANSIBLE TUTORIAL
be run on each group of servers (eg only installing
nginx on servers from the group webserver). Servers
can be in more than one group and there are ways to
define subgroups as well.
With this inventory file, we can start using Ansible.
Let’s try to ping those servers with a so-called ad-hoc
task, which means we won’t write any playbook files
right now:
ansible all -i inventory -m ping
Running the ping module
This tells Ansible to run the ping module on all hosts to block IPs that try malicious stuff.
to get “pong” back for
specified in the file named inventory. A module is an We’ll go through those in a somewhat random order every host in the inventory
addition to the core Ansible software and offers ways to explore Ansible step by step. The final playbook will file.
to interact with common software. Instead of writing have all tasks in a specific order that makes sense
shell commands for every task, a module for eg apt (eg creating a different user before locking down SSH
offers a simple way to tell Ansible which software to access for root). This tutorial and the playbook will
install through apt. be targeting Ubuntu 14.04 servers, but with a little
If you run the command, you should get immediate adjustment the playbook can be run on any server.
feedback if everything is correct in your inventory Let’s start with setting up a playbook and using it to
file. If Ansible can’t find your host, cannot SSH into install fail2ban. This is what a basic playbook with one
it or isn’t able to run the task after the connection task which installs the fail2ban software through apt
is established, you need to revisit the inventory file looks like:
and see if the correct options are set. If Ansible can’t
execute tasks, make sure Python 2 is installed on the To run Ansible we need two things:
remote machine and that the user Ansible uses to log
in can use it. an inventory file and a playbook
These ad-hoc commands can be used to execute
one-off commands on all servers without manually
containing the tasks to be executed
logging into them one at a time. It can be used to
restart a specific process on all servers (or a group of ---
servers), inspect log files or ask for status reports of - hosts: all
various sorts. become: yes
Before we dive into playbooks and how to tell tasks:
Ansible what tasks to run, let’s figure out what steps - name: install fail2ban
we should perform in order to harden our server: apt:
For any server, the root account should have a good pkg=fail2ban
password and should not be used to work with state=latest
directly. Instead, a different user which can perform The syntax used is YAML, which is an easy to
root tasks using sudo should be used. use “language” better suitable for humans than for
This user should be able to log in with his SSH key example JSON. The hosts key holds the group of
instead of password. hosts for which the tasks should be applied. Using all
For SSH, we should only accept SSH keys to log in here means that all hosts specified in the inventory
and only allow our user to log in. file will be used. The become key tells Ansible that it
A firewall that locks down unused ports protects should become the root user for the tasks run within
the server against software that might run but does this playbook. As we're going to run a lot of tasks
not need to be accessed from the outside (like a which require sudo, which is easier than specifying it
database which is only accessed locally). for every task.
The fail2ban software uses log files from the server Each task has a name that's used when printing
the result to the shell. Below the name we specify
Ansible from source the name of the module to be used. Each module
has its own parameters depending on what it needs
As Ansible is written in Python, it can be installed from
to perform the task at hand. For apt, we need to
source pretty easy. Get the source code either as a
download from the website or using Git, install the specify the package we want to install using the pkg
dependencies and run the setup script. The Ansible docs parameter. The state can be either absent, which
contain a tutorial on running Ansible from source. means it’s uninstalled if currently installed, present to
The advantage is that you are able to run the latest install it once and if it’s installed do nothing or latest,
version and you can update to beta versions if needed. You
which updates the package if there is a new version.
can also apply patches, which might already work but are
not released yet to fix bugs you are experiencing. And of Ansible tasks should be idempotent, so instead of
course, adding your own patches is possible as well. specifying what should be done tasks specify the
state which should be achieved. In our first example,
www.linuxvoice.com 79
TUTORIAL ANSIBLE
notified. Handlers are used to restart services like
sshd to make sure the changes to the config have any
effect. We need to specify our handler not within the
list of tasks but add another list with the key handlers:
handlers:
- name: restart sshd
service:
name=sshd
state=restarted
Ansible groups all notify events and execute them
The Ansible website at the end of the playbook run, once for each handler.
contains a lot of useful the state we want to achieve is that the fail2ban This means you can call notify many times for one
content to learn Ansible. package is installed in the latest version. The module handler but it only gets executed once.
itself takes care of the steps needed to get there. We also said we needed a firewall to protect our
For apt to work correctly, we should also do an server. Given the complexity of iptables, ufw is usually
update: a better tool for the job. We need to install it, open up
- name: update apt cache the ports we really need (22 for SSH, 80 for HTTP and
apt: maybe 443 for HTTPS) and enable ufw. We also need
cache_valid_time=3600 to restart ufw after this.
update_cache=yes To run a task more than once with different
Again, we are using the apt module, but this time arguments we can use with_items. Ansible runs the
we specify the update_cache parameter as well as task for each item in the list and replaces {{ item }}
cache_valid_time to prevent apt updating every time with the current text. A task using with_items could
we run the playbook and instead skip this task if apt look like this:
was updated in the last hour (3600 seconds). - name: allow various ports in ufw
To secure sshd, we should adjust the configuration ufw:
file. We can use the copy module to copy a locally port={{ item }}
stored sshd_config file to our server. It’s best to copy proto=tcp
the current sshd_config from one of your servers rule=allow
using rsync: state=enabled
rsync username@ip:/etc/ssh/sshd_config . with_items:
You should find the sshd_config file in your - 22
- 80
Ansible has a lot for you, especially if - 443
notify: restart ufw
you want to manage more servers or Ansible executes this task three times, for “22”, “80”
as well as “443”. Only the port changes, the other
if the setup is getting more complex parameters stay the same. Afterwards, all three ports
are allowed within ufw.
current working directory. Make sure to change We didn’t change the password for root yet. Even
PermitRootLogin to no (and remove any “#” at the if root cannot SSH into the server any more, a strong
beginning of the line) and PasswordAuthentication to password is still recommended. The user module has
no as well. got us covered:
This prevents the most common attack, which is - name: change root password
trying to SSH into the server, using the root user and user:
various passwords. The task to copy this file to every
server would look like:
- name: change sshd config Different operating systems
copy: Ansible does not provide any layer on top of modules,
dest=/etc/ssh/sshd_config meaning that a playbook containing tasks to interact with
src=sshd_config apt-get will fail on a system that does not have apt-get
notify: restart sshd available, like Red Hat-based Linux systems. There are a
few ways to work around this which are out of the scope of
This will take the file at src and copy it over to the
this tutorial. The documentation offers some ideas on how
server, storing it at dest. We could also use the to get started if you need this.
template module, which would mean that Ansible If you want to run Ansible from different host systems,
would run the configuration file through the Jinja2 you need to install Ansible on all of them. This can be
templating engine. This lets you use variables inside challenging if Windows hosts are involved or if some hosts
are running an old version of Ansible. You can use Vagrant
the templates, which can be passed in from the
or Docker to put Ansible into a controlled environment and
playbook. The last line in our example specifies that run it from there.
if this task changes anything, a so-called handler is
80 www.linuxvoice.com
ANSIBLE TUTORIAL
My first 5/10 minutes on a server
This tutorial is loosely based on two blog posts named “My
First 5 Minutes On A Server; Or, Essential Security for Linux
Servers” (https://plusbryan.com/my-first-5-minutes-on-
a-server-or-essential-security-for-linux-servers) and “My
First 10 Minutes On a Server – Primer for Securing Ubuntu”
(www.codelitt.com/blog/my-first-10-minutes-on-a-
server-primer-for-securing-ubuntu). Both tutorials contain
additional steps which were not included in this tutorial
in order to keep it short. The reader can go ahead and
implement the missing steps using Ansible as shown above.
A word of caution: All these steps are basic first steps
and do not protect you against sophisticated attacks as
well as exploits of bugs in software you are using. While
these steps will prevent the most common attacks, hackers
might still be able to get into your system.
name=root was executed and if the task changed something on
Output from running an
password=encryptet_password the remote server, if it failed and so on.
Ansible playbook against
state=present The full playbook, containing all tasks outlined one remote server.
The value for the password cannot be plain text, above, together with the inventory file and an example
as this would be unsecure, given that playbooks are sshd_config file can be found at https://gist.github.
plain-text on your hard drive and usually live inside com/Sgoettschkes/f737a89b0481f741a39e1943c
a Git repository as well. To generate the encrypted e2dcd9b.
password, use the Ansible has a lot more for you, especially if you
mkpasswd --method=SHA-512 want to manage more servers or if the setup is
utility or refer to the Ansible FAQ at https://docs. getting more complex: variables can be used almost
ansible.com/ansible/faq.html#how-do-i-generate- anywhere in Ansible playbooks. They can be defined
crypted-passwords-for-the-user-module for other on many levels (within a playbook, for each group or
ways to generate the password. host and even on the command line when running
To create a new user, the same module and the Ansible playbook). This way, you can create
parameters can be used. Adding the parameter flexible playbooks that can be filled in with variables
groups and setting it to sudo as well as adding the depending on which host the task is executed on.
parameter bash set to /bin/bash makes sure the user So-called facts are also variables that Ansible fills
is created with sensitive settings. Your SSH key for with information from the servers against which you
logging in through SSH can be copied from ~/.ssh/ are currently running the playbook. This information
id_rsa.pub to ~/.ssh/authorized_keys on the remote includes the IP address, amount of RAM, network
server. information and much more. This way, you could set
Allowing the sudo group to run sudo without a up a ufw firewall to allow access to your database
password can be done by using the lineinfile module: servers only from the web servers in a dynamic way,
- name: make sudo group user access sudo without reading the IP addresses from all servers in the group
password web server and using these values to set up ufw allow
lineinfile: rules. If you change the inventory file later and run the
dest=/etc/sudoers playbook again, ufw gets adjusted automatically.
regexp="^%sudo" Splitting up your playbook into roles you can reuse
line="%sudo ALL=(ALL:ALL) NOPASSWD:ALL" your playbooks in different scenarios. One role could
Here we specify a regex which is used to find the be SSH and it could take care of sshd_config. This
line we want to replace with what we specified as role can then be used in many projects and you
the line argument. In our case, we are looking for only need to maintain it once. There is even a public
a line that starts with %sudo. If this line is found, it repository for roles, called Ansible Galaxy: https://
is replaced with our line. If it’s not found, the line is galaxy.ansible.com.
appended to the bottom. Using Ansible is a great way to not spend much
With the full setup in place, we can run the playbook time doing infrastructure work and knowing exactly
against our servers: what happened to each server. With the playbook
ansible-playbook -i inventory tasks.yml outlined above, your servers will be more secure and
We use the easy to set up in the future. And it makes a good base
ansible-playbook to add your own setup on top!
command and pass the inventory file we created as
well as the playbook containing the tasks we want to Sebastian Göttsckes is interested in Python, Dart, Ansible, PHP,
run. Ansible shows us, for each task, on which server it and a little thing called the Google Cloud Platform.
www.linuxvoice.com 81
CODING VISUALISE LOG FILES
PROCESS AND VISUALISE
APACHE LOGS FILES
Use AWK and R to reveal useful information from your Apache log files.
T
MIHALIS here are two main reasons for working with such as inserting data into a database or turning data
TSOUKALOS log files: the first one is for creating reports into a beautiful PDF report.
and summaries; and the second one is for The good thing with all presented command line
Why do this? discovering abnormal events, which can either be utilities is that none of them changes its input, which
• Process Apache log possible security threats or indications of a means that you can even work with active Apache log
files using AWK
misbehaving executable. files without the fear of changing or destroying them.
• Visualise the
This tutorial will tell you how to process log files
information found in
Apache logs that come from an Apache web server in order to The format of an Apache log entry
• Identify possible extract useful information using traditional Unix A common log entry in an Apache log file will be
security threats command line tools such as Grep and Awk. It will also similar to the following:
show you how to visualise the extracted information, 73.208.169.12 - - [28/Jun/2016:10:20:02 +0300] "GET /misc/
because a plot can help you focus on what really menu-leaf.png HTTP/1.1" 200 457 "http://www.
matters. Additionally, non-technical mtsoukalos.eu/Enable-Root-user-on-El-Capitan"
people feel more comfortable with plots "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)
PRO TIP than with text reports. We're going to AppleWebKit/601.6.17 (KHTML, like Gecko) Version/9.1.1
Apache log files are usually very large, use the R statistical package for data Safari/601.6.17" 604
especially if you have a popular website; visualisation, but you can also use the However, this is not the only format that you might
the trick is to test your scripts using
smaller log files before trying to process Julia programming language, Gnuplot, come up with, because Apache enables you to define
the actual log files. However, it is Matlab, Python, Perl, etc. Although your own. The right place for this is inside apache2.
considered a bad practice to experiment scripting languages such as Perl, Python conf, using lines beginning with LogFormat:
on production machines, so it's better to
transfer your log files on another machine. and Ruby can be used for processing $ grep -i logformat /etc/apache2/apache2.conf
log files, nothing beats the speed of LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}
Awk for relatively simple jobs. Scripting i\" \"%{User-Agent}i\"" vhost_combined
languages are more suitable for advanced things LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\"
\"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\"
\"%{User-Agent}i\" %D" myformat
The name of each log format is defined by the
last word of each line (vhost_combined, combined,
common and myformat). You can find more about
the various formatters used for creating custom log
formats at http://httpd.apache.org/docs/2.4/mod/
mod_log_config.html.
Should you wish to use any one of them, you should
write the following entry inside the definition file of a
website, which most often is a VirtualHost block:
CustomLog /srv/www/a_site/logs/access.log myformat
The rest of the tutorial will use log files that use the
myformat format.
The sed utility
Some of the text reports sed is another handy command line utility for
you can get when processing text input. Imagine that you have log
processing Apache log files entries that format dates and times as follows:
with Awk. [27/Jun/2016:22:49:03 +0300]
82 www.linuxvoice.com
VISUALISE LOG FILES CODING
The following sed command replaces Jun with
30000
June on a log file:
$ sed 's/-Jun-/-June-/g' access.log
The next sed command prints the first 1000 lines
of a file:
25000
8e+09-
$ sed -n 1,1000p access.log
The next command tells sed to print just the first
data $V2
line of the input:
$ sed '1!d' access.log
20000
The last example shows how to globally delete
4e+09-
a single character, in this case [, from a text stream
using sed:
$ cat text.file | sed 's/\[//g'
Sed is priceless when you want to clean up log files
15000
0e+09-
from control characters or perform global search and
replace operations, but it cannot do very complex
things because it is not a programming language. 0 10 20 30
data $V1 0 2 4 6
data $V1
Using Awk
Awk is a handy programming language for processing served by the web server per day of the week: This is the output of the
text data with many capabilities. As an example, you # 0 is Sunday, etc. plotData.R script, which
plots data acquired by
can find out the IPs of your top five clients with the function dayOfWeek(year, month, day) {
processing Apache log files
following command: day_of_week = 0;
using Awk.
$ cat access.log | awk {'print $1'} | sort | uniq -c | sort -nr | if (month <= 2)
head -5 {
Although this tutorial is not about Awk, it is worth month += 12;
explaining the previous command. The cat command year--;
gives the contents of access.log as input to the awk }
command, which prints the first field of each line that day_of_week = (day + month * 2 + int(((month + 1) * 6) /
is the IP address of the client. The sort command 10) + year + int(year / 4) - int(year / 100) + int(year / 400) +
sorts all lines and the uniq -c command both deletes 2);
repeated lines and precedes each output line with the day_of_week = day_of_week % 7;
count of the number of times the line occurred in the return ((day_of_week ? day_of_week : 7) - 1);
input. Then you sort the output numerically in reverse }
order before printing the first five lines. BEGIN {
Similarly, you can create a summary of all HTTP month["Jan"] = 1; month["Feb"] = 2;
status codes: month["Mar"] = 3;
$ cat access.log | awk {'print $9'} | sort | uniq -c | sort -nr month["Apr"] = 4; month["May"] = 5;
A very handy report is the one that counts the month["Jun"] = 6;
number of connections per hour of the day: month["Jul"] = 7; month["Aug"] = 8;
$ cat access.log | awk {'print $4'} | awk -F: {' print $2 '} | month["Sep"] = 9;
sort | uniq -c | awk {' print $2 "\t" $1 '} month["Oct"] = 10; month["Nov"] = 11;
As a day has 24 hours, you will get at most 24 month["Dec"] = 12;
lines of output. The following Awk script, saved as }
bytesDOW.awk, calculates the number of bytes {
split($4, left, ":");
split(left[1], desired, "/");
About Awk, sed and grep year = desired[3];
Awk is an interpreted programming language designed for myMonth = month[desired[2]];
easy, productive and fast text processing, data extraction day = substr(desired[1], 2);
and reporting created at Bell Labs back in 1970s; Linux currentDay = dayOfWeek(year, myMonth, day);
systems use the much improved GNU Awk version.
myData[currentDay] += $10;
Sed (Stream Editor) is a tool for performing global search
and replace operations on text files, whereas Grep is a }
command line tool for searching text files using regular END {
expressions. The Grep tool cannot change its input, which for (val in myData)
is the main reason for using utilities such as Awk and print val,"\t", myData[val];
Sed. Both Awk and Sed can replace Grep, but its simplicity
}
makes Grep unique. There exist many Grep variations
including pgrep, egrep and fgrep. As a week has seven days, the output will have
seven lines. It it necessary to know which field holds
www.linuxvoice.com 83
CODING VISUALISE LOG FILES
Drupal Hacking Attempts! The next report shows the number of connections
per day of the month, which gives you a pretty good
idea of what is going on on your website:
1500
$ cat access.log | awk {'print $4'} | awk -F/ {' print $1 '} |
sort | uniq -c | awk {' print $2 "\t" $1 '} | sed 's/\[//g'
The output from the previous command will be
used by R later on in this tutorial, so it is saved as
dayOfMonth.data. Note that the two fields of each
1000
line are separated by a tab character.
You can easily create analogous reports that show
the total number of connections per month, per year,
Count
per IP address, per web page, etc.
The following code prints a report about the various
500
versions of the HTTP protocol:
$ cat access.log | awk {'print $8'} | sort | uniq -c
406 HTTP/1.0"
3902 HTTP/1.1"
The last report that will be created will show the
number of unique IPs per day of the month:
9
0 2 4 6
Days of Week
$ cat access.log | awk {'print $1, $4'} | awk -F/ {'print $1'} |
awk -F\[ {'print $2, $1'} | sort | uniq | awk {'print $1'} | uniq
This is the output of the drupal.R script that reads a file named drupal.data and plots it. -c | awk {'print $2, $1'}
the number of bytes transferred – in this case it is the Using the Grep utility
10th field. The following Grep command finds all log entries that
You can also create a report that shows the number contain a given IP address, and counts them using wc:
of connections per day of the week using a similar $ grep 66\.249\.64\.139 ./access.log | wc
Awk script, which is saved as connectionsDOW.awk. 30 558 6835
The following output shows the differences between The next Grep command shows all entries that both
bytesDOW.awk and connectionsDOW.awk using the contain a given IP address and return a 404 status
diff utility: code:
$ diff connectionsDOW.awk bytesDOW.awk $ grep 66.249.64.139 /srv/www/www.mtsoukalos.eu/logs/
32c32 access.log | grep " 404 "
< myData[currentDay]++; The last example finds all log entries that contain
files with the .png or .PNG extension:
The first step to any successful data $ grep '\.png\|\.PNG' ./access.log | wc
1157 26062 308592
visualisation is knowing what kind of A variation of the previous command finds all
accesses to ZIP files:
information you're looking for $ grep '\.zip\|\.ZIP' ./access.log | wc
2 48 608
--- Please bear in mind that Grep does not allow you to
myData[currentDay] += $10; add logic to your searches; therefore, doing advanced
The way Awk finds out the day of the week needs operations such as comparisons requires tools like
some explanation: as the name of the day cannot be Awk, Perl or another scripting language.
found in the log file, we will have to find it on own own.
An Awk function named dayOfWeek does all the job Using R for visualisation
for us – as you can see, finding the day of the week R offers a plethora of plots and graphs that you can
from a date is not a trivial task. Additionally, you also use; it's your job to select the right kind of plot for
need to convert the name of the month to a number, presenting your data. It is now time to visualise both
which is done with the help of an associative array connectionsPerDayOfWeek.data and dayOfMonth.
named month. data using R.
Both connectionsDOW.awk and bytesDOW.awk The following R script, which is named plotData.R,
must be executed as follows: will plot both data files:
$ cat ./access.log | awk -f connectionsDOW.awk #!/usr/bin/env Rscript
$ cat ./access.log | awk -f bytesDOW.awk require(ggplot2)
The output of connectionsDOW.awk will data <- read.table("./connectionsPerDayOfWeek.data",
be used by R and therefore it is saved as header=FALSE)
connectionsPerDayOfWeek.data. outputfile <- paste("cDay", ".png", sep="")
84 www.linuxvoice.com
VISUALISE LOG FILES CODING
png(filename=outputfile, width=1200, height=1600) Joomla Hacking Attempts!
p <- ggplot(data, aes(data$V1, data$V2)) + geom_
point(size=10, colour="red")
print(p)
data <- read.table("./dayOfMonth.data", header=FALSE)
outputfile <- paste("cDayOfMonth", ".png", sep="")
png(filename=outputfile, width=1200, height=1600)
20000
q <- ggplot(data, aes(data$V1, data$V2)) + geom_
line(size=4, colour=data$V1)
print(q)
You can execute the R script as follows:
Count
$ chmod 755 time.R
10000
$ ./time.R
The script will produce two PNG files named cDay.
png and cDayOfMonth.png as defined in the code.
Visualising for security
The first step to a successful visualisation is knowing
0
what kind of information you're looking for, which
0 2 4 6
means that you should also know how your web Days of Week
application works. For a Drupal site you can start by
monitoring the GET /?q=node/add HTTP/1.1, GET The output of joomla.R shows the number of hacking attempts per day of the week
/?q=user/register HTTP/1.1, GET /?q=node/add (Sunday = 0, etc.).
HTTP/1.0 and GET /?q=user/register HTTP/1.0
requests that indicate direct hack attempts. The The output is saved as drupal.png.
easiest way to get the log entries that contain such The administration page of a Joomla site can be
requests is with the help of Grep. Then, you will found at the /administrator/ path, which means that
process the log entries using connectionsDOW.awk only an administrator has the right to visit this URL.
and save the data in a file named drupal.data: So, the first step is using Grep is to find out all paths
$ grep '"GET /?q=node/add HTTP/1.1" \| "GET /?q=user/ that contain /administrator or /administrator/ and
register HTTP/1.1" \| "GET /?q=node/add HTTP/1.0" \| then process the output using the connectionsDOW.
"GET /?q=user/register HTTP/1.0"' access.log | awk -f awk Awk script:
connectionsDOW.awk > drupal.data $ cat ./access.log | grep -i administrator | awk -f
Last, you will process drupal.data using the connectionsDOW.awk > joomla.data
following R script: The next R script plots the collected information:
#!/usr/bin/env Rscript #!/usr/bin/env Rscript
require(ggplot2) require(ggplot2)
data <- read.table("./drupal.data", header=FALSE) data <- read.table("./joomla.data", header=FALSE)
outputfile <- paste("drupal", ".png", sep="") outputfile <- paste("joomla", ".png", sep="")
png(filename=outputfile, width=1600, height=1200) png(filename=outputfile, width=1600, height=1200)
ggplot(data, aes(x = data$V1, y = data$V2)) + geom_ p <- ggplot(data, aes(x = data$V1, y = data$V2)) + geom_
bar(stat = "identity", colour="black") + ggtitle("Drupal bar(stat = "identity", fill = data$V2, colour="black")
Hacking Attempts!") + labs(x="Day of Week", y="Count") p <- p + ggtitle("Joomla Hacking
Attempts!") + labs(x="Day of Week",
PRO TIP
y="Count")
The cron tool is your friend because it
Why visualise? p <- p + theme(plot.title = element_ enables you to execute your scripts at
text(size = rel(3), colour = "black")) night or during weekends. In other words,
All web server administrators understand the importance do not execute heavy scripts when a Linux
of having a high-level view of their web traffic. Plots and print(p)
machine has more important things to do,
graphs enable you to have a quick overview of your web The generated plot is saved as like serving user requests.
server traffic, which is very difficult to watch otherwise joomla.png.
because web traffic is getting bigger and bigger.
As you can understand, the format of
However, do not forget that at the end of the day what
really matters is the actual data! In other words, even the the log file is not important; what is really important
most impressive plot or graph cannot save you from data is recsognising the data that matters and processing
that cannot reveal the truth; so, do not try to measure the it. Additionally, as long as you extract your data in a
performance of a web server during weekends or do not standard format, no other changes need to be made
search for hacking attempts during normal working hours,
to the rest of the code.
because web servers have less traffic during weekends and
hacking attempts usually take place late at night or during
holidays; use your common sense! Mihalis Tsoukalos is a Unix administrator, mathematician
and programmer who enjoys writing technical articles.
www.linuxvoice.com 85
CODING ELASTIC BEANSTALK
ELASTIC BEANSTALK:
DEPLOY A WEB APP
Host and scale a Golang web application on Amazon Elastic Beanstalk.
A
AMIT SAHA mazon Elastic Beanstalk is a Platform as a choose to install the Go tools using your distro's
service (PaaS) and is part of Amazon Web package manager. However, it's likely that it may be
Why do this? Services (AWS). Basically, this means that you lagging behind the upstream release. Hence, we will
• Get started with outsource your worry of managing servers including install it manually . First download the 1.6 Linux binary
Platform as a Service the operating system that is running on it to AWS. All from https://golang.org/dl/; then untar the package
(PaaS) you need to think about now is writing your web with sudo tar -C /usr/local -xzf go1.6.2.linux-amd64.
• Automatically scale application; as long as this starts correctly, Amazon tar.gz. Next, add /usr/local/go/bin to your PATH by
your latest web
deployment Elastic Beanstalk will take care of the rest. We get the adding export PATH=$PATH:/usr/local/go/bin in
• Add another advantage of auto scaling – automatically increasing your .bashrc.
buzzword to your CV and decreasing the number of instances of our The next step is to set up our Go workspace – a
application based on network traffic (for example), directory where all our Golang source code will live
and can upgrade to new versions of our application and an environment variable GOPATH whose value is
without downtime, as well as access to a whole this directory. My workspace is set up as the directory
bunch of other AWS services. The trade-off to using a golang in the directory $HOME/work (for the purpose
PaaS is, of course, that we give away our control over of this article, my $HOME is /home/vagrant):
the software (including the operating system) that $ mkdir $HOME/work/golang
runs on it. In this directory, we will create a src sub-directory:
In this article, we will write a simple web application $ mdkir $HOME/work/golang/src
in Golang and deploy it to Elastic Beanstalk. The final Next, in our .bashrc file or similar, we will add the
web application we will write is an Integer Obfuscation following:
service: pass an integer to it, and you will get a JSON export GOPATH=$HOME/work/golang
object with a random string back. An example request Now, if we open a new terminal session, the
Figure 1: Applications and response using the curl command looks as command go env GOPATH should print the above
hosted on Elastic
follows: path.
Beanstalk. AWS Resources
$ curl http://linux-voice-5.us-west-2.elasticbeanstalk.
include the Linux VM
instances, Auto Scaling com/?id=1 Our web application
groups, Elastic Load {"id":1,"obfuscated_id":"6d6p4M"} Now, we will write our web application. The current
Balancer, Security groups The first step to set up Golang on our system is to stable version of the language is 1.6, with 1.7 due out
and others. install the Go compiler and other tools. You may soon. But Elastic Beanstalk supports only 1.3 and 1.4
now. Hence for our web application, we are left with
AWS Resources AWS Resources AWS Resources AWS Resources two choices:
Don't use any Golang feature which is not present in
1.4.
Deploy our web app as a docker container using the
latest Golang features.
We will keep it simple first and attempt the first
Elastic Beanstalk option, and then adopt the second deployment option.
The next code listing shows our web application:
// Basic HTTP server listening on 5000
package main
import (
"net/http"
"fmt"
$ eb create $ eb deploy
)
func handleRequest(w http.ResponseWriter, r *http.
Local Development Environment
Request) {
86 www.linuxvoice.com
ELASTIC BEANSTALK CODING
fmt.Fprintf(w, "Hello World")
}
func main() {
http.HandleFunc("/", handleRequest)
http.ListenAndServe(":5000", nil)
}
The first line in the above program is a comment.
Next, we declare the package for our program.
We declare that this program is an executable by
specifying the package as main. Next, we import two
packages from the standard library net/http and fmt.
Then we write a function, handleRequest(), which
handles any request sent to our web application.
The first parameter is a variable, w of type http.
Figure 2: Creating a user to
ResponseWriter, which corresponds to the read end The first step for us is to create an Amazon AWS
access AWS services
of the client. Anything we write to this object forms account by going to https://aws.amazon.com/
our response to the client's request. The second free. While creating the account, we will be asked to
parameter, r is a variable of type http.Request and supply a credit card information. There is a 12-month
gives us access to the client request we are serving. introductory offer where we will not be charged if our
This includes the request type, the request body, usage of AWS resources do not exceed the limits of
headers and others. In this function, we write a string the free tier. On the account creation page, you can
"Hello World" to w, and this any client connecting to read the various terms and conditions associated with
our web application will get "Hello World" back. the free tier, the various services and usage limits and
Next, we have the main() function, which is this is also a good chance to have a quick read of all
where our program's execution starts. The first the various services that AWS provides.
statement tells us that any request made to the
root path "/" of our web application should be sent Signing up for an Amazon AWS account
to the handleRequest() function for processing. The user account that you just created is the "root"
We do so using the HandleFunc() function in the AWS account. Although we can start using AWS using
http package. To actually start the HTTP server, this account, we don't recommend you do so: instead,
we call the ListenAndServe() function with the first we'll create what is referred to as an AWS IAM user
argument being the address to listen on. :5000 means by going to https://console.aws.amazon.com/iam/
to listen on port 5000 on all interfaces. The reason home and clicking on "Create New Users" (Figure 2).
we use this port is because that's where Elastic Next, when prompted, download the credentials file
Beanstalk expects our web application to listen on. (Figure 3). This is a CSV file having your username,
Save the above program in a file $GOPATH/src/ AWS access key ID and secret access key.
github.com/amitsaha/linux_voice_5/application.go, Next, we need to give the user permissions for
navigate to the directory $GOPATH/src/github.com/ performing all the operations we need for working
amitsaha/linux_voice_5 and then run it using go run with Elastic Beanstalk. From the https://console.
application.go: aws.amazon.com/iam/home?#users page,
$ go run application.go click on the username we created above, go the
Our web application is now running on port 5000. "Permissions" tab, then click on "Attach Policy".
If we send a curl request from another terminal, we From the list of policies check "IAMFullAccess" and
should get a "Hello World" message back: "AWSElasticBeanstalkFullAccess" and click "Attach
$ curl 127.0.0.1:5000 Policy". Now, the "Permissions" tab for the user should
Hello World show these policies (Figure 4).
Now that our first web application is ready, let's host
it on Elastic Beanstalk. Use Ctrl + C in the terminal Setting up Elastic Beanstalk CLI
where you started the server to stop the server. The awsebcli Python package provides a command
First, we will initialise a Git repository for our web line interface to Elastic Beanstalk. It is a good idea to
application code: install it in its own virtual environment. If you do not
$ pwd have virtualenv installed, you can install it using your
/home/vagrant/work/golang/src/github.com/amitsaha/ distro's package manager (on Ubuntu 14.04 and
linux_voice_5 Fedora 23, you can install the package python-
$ git init virtualenv). Next, create a virtual environment in a
Initialized empty Git repository in /home/vagrant/work/ location of your choice. I will create a virtual
golang/src/github.com/amitsaha/linux_voice_5/.git/ environment in the ~/work/venvs directory:
Let's add the current application.go file and commit it: $ virtualenv ~/work/venvs/ebcli
$ git add application.go $ . ~/work/venvs/ebcli/bin/activate
$ git commit -m "First application version" Once we have created the virtual environment and
www.linuxvoice.com 87
CODING ELASTIC BEANSTALK
console and going to the individual AWS service's
page for the same region as your web application.
At this stage, our application is deployed.
Execute the command eb open while in the same
directory and you should see "Hello World" on a
web page. Congratulations! Your web application
is now deployed. Let's go back to the two pieces of
information eb create asked us for earlier:
Figure 3: Download activated it, we can install the awsebcli package using Enter Environment Name
credentials file. pip install awsebcli. Let's verify the install: (default is linux-voice-5-dev):
$ eb --version Enter DNS CNAME prefix
EB CLI 3.7.6 (Python 2.7.1) (default is linux-voice-5-dev):
At this stage the program eb is installed and ready First, we are asked to specify the environment
to use. The first step is to create a configuration file name. We can deploy your application into multiple
where we will put our AWS credentials. Create a file environments running the same or different versions
~/.aws/config and fill in the following: of our application. Later on, we will create a new
[profile eb-cli] environment that will be our production environment
aws_access_key_id = <YOUR ACCESS KEY> and another to act as our staging environment.
aws_secret_access_key = <YOUR SECRET ACCESS KEY > Next, we are asked to specify a DNS CNAME
The access key and the secret key are both prefix for our web application. The default is the
available in the credentials file we downloaded when environment name, which means our application will
creating the account for the new user. Next, as the be available at http://linux-voice-5-dev.us-west-2.
first step for deployment, we will initialise our Elastic elasticbeanstalk.com. I should mention here that you
Beanstalk application. Navigate to the directory where can use a custom domain for your application using
our code lives and run eb init: AWS's Route53 services.
We first select the AWS region where we plan to The "service role" that is created for our environment
host our application. The region you choose would be is how we give Elastic Beanstalk permission to carry
determined by various factors including geographical out the various operations on our behalf. This includes
closeness to your application's users', cost, creating and destroying AWS resources associated
compliance and the availability of the service in that with our environment.
region. For our case, we will simply select the default
(us-west-2) region. Inspecting your environment state
Next, we enter a name for our application. By The status sub-command displays the status of an
default eb init suggests us to use the same as the environment. If we don't specify an environment
current directory name. In the next two steps, we name, it displays the status of the current
select the platform and the platform version (Go and environment. The eb logs command displays the last
Go 1.4 respectively). When using the Go 1.4 platform, 100 lines of various log files from the instance running
Elastic Beanstalk expects the main program to be your application. This includes access logs, error logs
named as application.go, which is why we used that and activity logs. If we get any unexpected behaviour
specific filename earlier. We answer n when asked if from your web application, these are the logs we will
we want to set up SSH for our instance. look into. To download the entire logs, we have to
Based on our answers above, a configuration specify the --all switch:
file config.yml is created in the directory The events sub-command can be used to retrieve
.elasticbeanstalk. If you see the .gitignore file, you will a list of recent events – essentially a high-level
see that the Elastic Beanstalk configuration file is not summarised view of what has been happening in your
version controlled. This means that the configuration current environment. When this is used with the -f
file will not be uploaded when our web application is flag, we can follow events as they happen:
uploaded to AWS or if we push our code to a remote $ eb events -f
repository. INFO: createEnvironment is starting.
The next step is create the environment for our INFO: Using elasticbeanstalk-us-west-2-367082021788
web application. An "environment" encompasses as Amazon S3 storage bucket for environment data.
your application code and all the AWS resources INFO: Created security group named: sg-db62cdbd
associated with the running instance of your INFO: Environment health has transitioned to Pending.
application. These resources include EC2 instances, Initialization in progress (running for 22 seconds). There
load balancer, an auto scaling group, S3 buckets, are no instances.
Security Groups and IAM roles. To create an INFO: Created load balancer named: awseb-e-2-
environment, we use the eb create command: AWSEBLoa-FUF7TCRQW3H4
$ eb create INFO: Created security group named: awseb-e-
You can look at all the AWS resources that our web 23852cmkuv-stack-AWSEBSecurityGroup-
application is currently using by signing in on the AWS RHBOGD32SNE9
88 www.linuxvoice.com
ELASTIC BEANSTALK CODING
INFO: Created Auto Scaling launch configuration named:
awseb-e-23852cmkuv-stack-AWSEBAutoScalingLaunch
Configuration-16GFEWY8H7O53
INFO: Created Auto Scaling group named: awseb-e-
23852cmkuv-stack-AWSEBAutoScalingGroup-
1SKOJXCY6RFJN
INFO: Waiting for EC2 instances to launch. This may
take a few minutes.
...
We can specify a different environment name to the
command as eb events <environment-name>.
Deploying our web app as a Docker container
Docker (www.docker.com) is a software
containerisation solution that enables us to run the
Figure 4: Policies attached
software we want to. This means that we can run our func main() {
to the user.
own Linux distribution using our own version of the http.HandleFunc("/", handleRequest)
software we want to. Hence, using Docker we can use http.ListenAndServe(":5000", nil)
a more recent version of Golang for our web }
application. We import the runtime package from the standard
The instructions for installing the Docker engine library, which we then use to get the Go version using
on Linux are available at https://docs.docker.com/ the Version() function. Now, when a client connects to
engine/installation/linux. our application, we will respond with the Go version in
Once you have Docker engine installed, let's addition to the greeting. Let's build an image from our
write the Dockerfile that Elastic Beanstalk will use Dockerfile next:
to deploy our application. Save the following into $ sudo docker build -t amitsaha/linux_voice_5 .
a file Dockerfile in the same directory as our web ..
application: The -t option is used to specify an image tag and is
FROM golang usually of the form username/image_name. Once the
ADD application.go /go/src/github.com/amitsaha/linux_ image has been built, we will start a container with it:
voice_5/application.go $ sudo docker run -P amitsaha/linux_voice_5
EXPOSE 5000 From another terminal, if you do sudo docker ps,
CMD ["go", "run", "/go/src/github.com/amitsaha/linux_ you should see a number of columns in the output;
voice_5/application.go"] the PORTS column tells us that we have the port
The first statement in the Dockerfile states the base 32768 listening on our host, which maps to the port
image on which we will deploy our application. The 5000 in the container on which our web application is
golang image (https://hub.docker.com/_/golang) listening on. Hence, $ curl 127.0.0.1:32768 will return
is based on Debian and at the time of writing has us:
Go 1.6 installed. Next, we copy our web application $ curl 127.0.0.1:32768
source, application.go, to a path /go/src/github. Hello World. Running: go1.6.2
com/amitsaha/linux_voice_5/application.go in the This proves that our Dockerfile works. Let's stop the
image we will build. The statement EXPOSE 5000 running container using $ sudo docker stop <your-
makes our web application accessible from the host container-name>, where the name from the docker
operating system, and the final statement specifies ps command. Now, let's add everything to Git and
the command we want to run when the image is run. commit:
Before we go ahead with trying our our Dockerfile, $ git add -A .
let's first update our application.go file to the $ git commit -m "Dockerized version"
following: We are now ready to deploy. However, since we will
// Basic HTTP server listening on 5000 now use a different platform, we will first terminate
package main our current environment using eb terminate. Once the
import ( termination is complete, let's select the Docker 1.9.1
"fmt" platform:
"net/http" $ eb platform select
"runtime" It appears you are using Docker. Is this correct?
) (y/n): y
func handleRequest(w http.ResponseWriter, r *http. Select a platform version.
Request) { 1) Docker 1.9.1
fmt.Fprintf(w, "Hello World. Running: ") 2) Docker 1.7.1
fmt.Fprintf(w, runtime.Version()) 3) Docker 1.6.2
} (default is 1): 1
www.linuxvoice.com 89
CODING ELASTIC BEANSTALK
The function getHashId(), which performs the
obfuscation, is as follows:
func getHashId(id int) (string, error) {
hd := hashids.NewData()
hd.Salt = strconv.Itoa(rand.Int())
hd.MinLength = 6
h := hashids.NewWithData(hd)
return h.Encode([]int{id})
}
func getHashId(id int) (string, error) defines
a function getHashId() that takes in an integer
parameter id and returns two values – one of type
string and the other of type error. In the first three
statements we set up our hashids generator. The salt
value should be a string and is set to a random integer
converted to a string. Since the salt will be set to a
random string everytime we generate a hashid, the
obfuscated string will be different even when called
with the same integer. We also set the minimum
length of the obfuscated string to 6. This ensures
that the obfuscated string is at least six characters
in length. The last two statements generate the
obfuscated string and return it.
The handleRequest() function is changed to the
following:
func handleRequest(w http.ResponseWriter, r *http.
Figure 5: Monitoring Request) {
dashboard. Our platform has now been changed successfully, id, err := strconv.Atoi(r.URL.Query().Get("id"))
so we will now create a new environment with eb if err != nil {
create. Once the environment creation has been http.Error(w, "Bad id supplied", 400)
completed, if we visit http://linux-voice-5-dev. } else {
us-west-2.elasticbeanstalk.com, we should get the generatedId, err := getHashId(id)
response as "Hello World. Running: go1.6.2". This tells if err != nil {
us that our web application is now up and running in http.Error(w, "Error generating Id", 500)
the docker container we created with our Dockerfile. } else {
Next, we will modify our web application to do fmt.Fprintf(w, generatedId)
something slightly more useful. }
}
Integer obfuscation as a service }
Hashids (http://hashids.org) is a way of obfuscating The first statement in this function extracts the
integers (possible database identifiers) to a randomly value passed via the id query parameter using r.URL.
generated string. It can be useful in any scenario Query().Get("id") and converts it to an integer using
where you don't want to expose the database the function Atoi() from the strconv package. If a
identifier to your user. We will now update our web non-integral value was specified for id or it was not
application's source code to implement a Hash ID as a specified, the value of err is not nil and we return
service. When we pass in an integer to the service, we 400 HTTP error with the message "Bad id supplied".
will get a random string back. The request will be of If the passed-in value of id could be successfully
the form http://linux-voice-5.us-west-2. converted, we next call the getHashId() function
elasticbeanstalk.com/?id=1 and we will get a JSON with the converted value. If there was some problem
response back of the form {"id":1,"obfuscated_ in generating the obfuscated string, we send an
id":"BARw3W"}. error with a 500 HTTP response. Else, we write the
You can see the entire code at https://github.com/ generatedId to the ResponseWriter object, w, using
amitsaha/linux_voice_5/blob/master/application. the Fprintf() function.
go. Here I will discuss the main changes from our Overwrite your current application.go with the one
last version. At the beginning of the file we have a at https://github.com/amitsaha/linux_voice_5/tree/
bunch of additional imports. The package we use to ded80e07cd90278ba3501201632faa02de886157.
generate obfuscated strings is the third-party package Before we can run our application and try it out, we
github.com/speps/go-hashids. In addition, we also will fetch the go-hashids package by executing the
import two other standard library packages math/ following command while being in the $HOME/work/
rand and strconv. golang/src/github.com/amitsaha/linux_voice_5
90 www.linuxvoice.com
ELASTIC BEANSTALK CODING
directory: INFO: Deploying new version to instance(s).
$ go get . INFO: Environment health has transitioned from Ok to
.. Info. Application update in progress on 1 instance. 0 out
Now, we can start our web application as earlier: of 1 instance completed (running for 42 seconds).
$ go run application.go INFO: Successfully built aws_beanstalk/staging-app
From another terminal, let's try sending in a couple INFO: Docker container d4ee18f8cbe9 is running aws_
of requests: beanstalk/current-app.
$ curl 127.0.0.1:5000?id=1 INFO: New application version was deployed to running
dvNDnp EC2 instances.
$ curl 127.0.0.1:5000?id=abd INFO: Environment update completed successfully.
Bad id supplied If you now go to the URL http://linux-voice-5-
Our web application is now ready to be deployed. dev.us-west-2.elasticbeanstalk.com/?id=1 in your
But first, we will need to update our Dockerfile to the browser or using the curl command, you will see that
following: a string is returned.
FROM golang
ADD application.go /go/src/github.com/amitsaha/linux_ Setting up a production environment
voice_5/application.go We now have our web application running at http://
RUN cd /go/src/github.com/amitsaha/linux_voice_5/ && linux-voice-5-dev.us-west-2.elasticbeanstalk.com;
go get -d -v . we used the CNAME linux-voice-5-dev to indicate that
EXPOSE 5000 this is going to be our development instance of the
CMD ["go", "run", "/go/src/github.com/amitsaha/linux_ web application. We are convinced that we are happy
voice_5/application.go"] with the state of the application now, or in other words
The additional command RUN cd /go/src/github. production-ready. Now, we will set up our production
com/amitsaha/linux_voice_5/ && go get -d -v . is environment:
needed for fetching the go-hashids package as we $ eb create
did earlier. Enter Environment Name
(default is linux-voice-5-dev): linux-voice-5
Deploying the updated application Enter DNS CNAME prefix
Our web application is now updated and we have (default is linux-voice-5): linux-voice-5
verified that it works as we expect it to. We have also ...
updated the Dockerfile to reflect the changes. Let's ..
stage the changes to Git and create a new commit This new environment is called linux-voice-5
with our changes: and the web application will be available at linux-
$ git add -A . voice-5.us-west-2.elasticbeanstalk.com. Once
$ git commit -m "Obfuscation service" the environment has been successfully created, try
Now, to deploy our updated application, we will use making a few requests to it and make sure our web
the eb deploy command: application is behaving as expected.
$ eb deploy At this stage, we have two environments for our
Creating application version archive "app- application – one for dev and the other production.
ded8-160617_094945". We can list our current environments using eb list:
Uploading linux_voice_5/app-ded8-160617_094945.zip to $ eb list
S3. This may take a while. linux-voice-5
Upload Complete. * linux-voice-5-dev
INFO: Environment update is starting. The environment with a * indicates the current
Figure 6: Autoscaling
Policies.
www.linuxvoice.com 91
CODING ELASTIC BEANSTALK
default environment. This means that if you do will deploy to the development environment, linux-
not specify an environment name to a command voice-5-dev:
that accepts an environment name, this is the $ eb deploy linux-voice-5-dev
environment against which the operation will be ...
performed. Using eb deploy linux-voice-5-dev we deploy
the current version of our application to the linux-
Monitoring your application instances voice-5-dev environment. Once the deployment
We now have the first version of our web application has completed, we can make a request to see if our
up and running in two environments, and we would change is now live:
like to keep an eye on how the instances on which our $ curl http://linux-voice-5-dev.us-west-2.
applications are running are doing. We can do so elasticbeanstalk.com/?id=1
using the eb health command. It will show us the {"id":1,"obfuscated_id":"ePgm6r"}
platform we are running, the overall health of the OK, so things are working as expected in the
environment, instance specific requests/second, development environment. It's worth noting that
percentage of requests that were served with different this update happens in place on the instances. This
HTTP status codes and others. For an interactive means for a window of time, your web application
view, we can use the eb health --refresh command. won't be receiving any requests when the update
The command also takes a environment name is happening. In production, we don't want this to
as the parameter, hence eb health linux-voice-5 will happen, or in other words we want to aim for a zero-
show the health for the linux-voice-5 environment. In downtime deployment. We can achieve this using the
addition, the Web UI for Elastic Beanstalk (ttps://us- following approach:
west-2.console.aws.amazon.com/elasticbeanstalk) 1
Create a new environment with the updated version
has a "Monitoring" tab for each environment (Fig 5). of our application (we will call the environment
linux-voice-5-stg).
Blue-green deployment 2
Verify it works as expected.
At this stage, our web application is running in 3
Swap the DNS from the current production
production, and let's say that people are already environment to now point to the environment we
depending on our service. Chances are that before created in step 1.
long we would want to deploy a new version of our 4
Terminate the old production environment.
web application. Let's change our web application to This is usually referred to as "blue green"
now return the obfuscated number as a JSON deployment. First, we create the new environment
response rather than in plain text. The repository at linux-voice-5-stg:
https://github.com/amitsaha/linux_voice_5 has the $ eb create linux-voice-5-stg
modified source code for application.go. I present the ..
changes from our previous version below. Once the environment is ready, we can find the
The first change is to import the encoding/json CNAME of this environment as follows:
package from the standard library. The next change $ eb status linux-voice-5-stg
is to define a struct, Response: Environment details for: linux-voice-5-stg
type Response struct { Application name: linux_voice_5
Id int `json:"id"` Region: us-west-2
ObfuscatedId string `json:"obfuscated_id"` Deployed Version: app-7113-160620_231417
} Environment ID: e-am6ptn5sgc
We define a struct Response with two fields: Id of Platform: 64bit Amazon Linux 2016.03 v2.1.0 running
type int and ObfuscatedId of type string. For each we Docker 1.9.1
add a tag string in backticks, which tells the compiler Tier: WebServer-Standard
that when we encode this structure as a JSON object, CNAME: linux-voice-5-stg.f3tvjjma9v.us-west-2.
we want the field to appear as id and obfuscated_id elasticbeanstalk.com
respectively. If we don't do so the JSON object will Updated: 2016-06-20 13:19:30.832000+00:00
have the fields as Id and ObfuscatedId, which is not Status: Ready
usually the convention for JSON objects. Health: Green
r := Response{Id: id, ObfuscatedId: generatedId} The CNAME of this environment has been
w.Header().Set("Content-Type", "application/json") randomly assigned, and we can make sure our web
json.NewEncoder(w).Encode(r) application is behaving as we expect it to by making
In the first line of the above block, we create a a request such as linux-voice-5-stg.f3tvjjma9v.us-
variable of type Response object, r. Next, we set the west-2.elasticbeanstalk.com?id=1. This completes
"Content-type" header to application/json. Finally, Step 2 from above.
in the last line we create a new JSON encoder that Now, we want to switch our current production
writes to w, and write the response variable encoded environment (linux-voice-5) to switch to this one:
as JSON. Once the changes are committed to our $ eb swap linux-voice-5-stg -n linux-voice-5
repository, we're ready to deploy the update. First, we INFO: swapEnvironmentCNAMEs is starting.
92 www.linuxvoice.com
ELASTIC BEANSTALK CODING
INFO: Swapping CNAMEs for environments 'linux-voice-
5-stg' and 'linux-voice-5'.
INFO: 'linux-voice-5.us-west-2.elasticbeanstalk.com'
now points to 'awseb-e-a-AWSEBLoa-
EI8FCU3NGSQX-1272685286.us-west-2.elb.amazonaws.
com'.
INFO: Completed swapping CNAMEs for environments
'linux-voice-5-stg' and 'linux-voice-5'.
$ eb status linux-voice-5-stg
Environment details for: linux-voice-5-stg
Application name: linux_voice_5
Region: us-west-2
Deployed Version: app-7113-160620_231417
Environment ID: e-am6ptn5sgc
Platform: 64bit Amazon Linux 2016.03 v2.1.0 running
Docker 1.9.1
Tier: WebServer-Standard Figure 7: Maximum
CNAME: linux-voice-5.us-west-2.elasticbeanstalk.com duration of 300 seconds is referred to as the breach NetworkOut for our
Updated: 2016-06-21 00:32:03.618000+00:00 duration. Hence, the scale up policy here is triggered instance during a high-
Status: Ready if the the average total number of bytes exceeds traffic window
Health: Green 6000000 bytes (6MB) for more than 300 seconds
Unfortunately, AWS Elastic Beanstalk doesn't and alternatively the scale down policy is triggered
currently have a feature to rename environments, if the average total number of out bytes is less than
which means that we will need to mentally remember 2000000 (2MB) for more than 300 seconds.
which is currently our production environment using We will use the wrk (https://github.com/wg/wrk)
some convention. Our production web application is tool to simulate traffic to our web application. After a
now running in the linux-voice-5-stg environment, bit of trial and error, I found that the following would
so we can safely terminate our old production result in 6MB NetworkOut bytes for over five minutes:
environment now: $ wrk -t50 -c500 -d5m http://linux-voice-5.us-west-2.
$ eb terminate linux-voice-5 elasticbeanstalk.com/?id=1
We can see from Figure 7 that in the time window
Scaling the number of application instances between 11.19 and 11.35 the maximum NetworkOut
We have an auto scaling group se tup for our exceeded 6MB for more than five minutes. This
application in each environment, which enables us to triggered the scale up operation and a new instance
scale up or scale down the number of instances. An will be added to your environment. Once your traffic
auto scaling group specifies the minimum number of again reduces to less than 2MB for more than five
instances, maximum number of instances and the minutes the older instance in the environment will be
desired number of instances we want running at any automatically terminated.
given point of time. A scale up operation happens In most cases, a web application would be backed
when the current number of instances is less than the by a database. Amazon Relational Database Service
maximum number of instances and the specified (RDS), also part of AWS, allows a choice between
metric dimension exceeds the metric threshold. MySQL and PostgreSQL (among others). An RDS
Conversely, a scale down operation happens when the instance can be created as part of your Elastic
number of instances running is more than the desired Beanstalk application environment using eb create
number of instances and the specified metric --database. The drawback to this approach is that
dimension is less than the metric threshold. This will if you terminate your environment as part of a
be clearer when we consider the auto scaling group deployment as we saw in blue-green deployment, it
that Elastic Beanstalk automatically creates for us. brings down the RDS instance associated with it too.
When we go to https://us-west-2.console.aws. Hence, it is recommended that you manage your RDS
amazon.com/ec2/autoscaling/home (assuming instance separately. Once we have our RDS instance
us-west-2 is your AWS region) via your browser, you set up, we will use an appropriate driver listed at
will see two auto scaling groups, one for each of the https://github.com/golang/go/wiki/SQLDrivers to
two environments we currently have deployed our interface with the database server.
application to. When you click on any of the scaling The GitHub repository at https://github.com/
group, you will see the two policies defined, one for amitsaha/linux_voice_5 has the final web application
scaling up and the other for scaling down (Figure code along with resources to learn more about Elastic
5). The scaling down policy for the group states Beanstalk, Docker and Golang.
the execution policy as "NetworkOut < 2000000
Amit Saha is the author of Doing Math with Python (No Starch Press)
for 300 seconds" and the scaling up policy states and a software engineer. He blogs at https://echorand.me, tweets
"NetworkOut > 6000000 for 300 seconds". The @echorand and can be reached via email at amitsaha.in@gmail.com.
www.linuxvoice.com 93
CORETECHNOLOGY EXTENDED ATTRIBUTES
CORE
Valentine Sinitsyn develops
high-loaded services and
teaches students completely
unrelated subjects. He also has
a KDE developer account that
he’s never really used.
TECHNOLOGY
Prise the back off Linux and find out what really makes it tick.
Extended attributes & POSIX ACL
I
n Linux, pretty much everything is a file. This to have CAP_SYS_ADMIN capability to read or write
includes your documents and executable trusted extended attributes. This usually means it
programs, directories (or folders), devices and must run as root. The GlusterFS distributed filesystem
even IPC objects such as Unix sockets. Each file has a implements many of its features via trusted extended
name and a size, along with some timestamps, attributes.
access permissions, and other associated metadata. Finally, there is extended user attributes and user.*
Nine times out of ten, this is everything you namespace. This is where you put the metadata you
need. But imagine you want your own, custom bit want. Of course, this is also subject to permission
of metadata. This could be a tag or a URL you've checks; you can't write extended attributes if you
downloaded the file from. It could also be some can't write to a filesystem object, and you can't read
permission: "Alice and Bob can read this". Linux makes or list them unless you have the read permission.
this possible with extended attributes, or simply Moreover, only regular files and directories may
"xattrs". carry extended user attributes. They are commonly
employed to store the file's original URL, MIME
Extending file metadata type, or character set. Freedesktop.org defines
In a nutshell, extended attributes are key-value pairs some attributes (https://www.freedesktop.org/
associated with a filesystem object. Attribute names wiki/CommonExtendedAttributes) for conforming
(or keys) are strings. Values can be anything: at this applications to use. However, nothing prevents
point, they are just sequences of bytes. Moreover, keys you from creating your own ones; just think of a
are namespaced, and attribute names are always sufficiently unique name to prevent clashes. A popular
namespace.something. Namespaces are also called recommendation is to prefix your custom attribute
attribute classes. The kernel currently recognises four with the reversed domain name you own.
of them (see xattr(7)). Extended attributes naturally need some support
security.* is for kernel security modules. For at the filesystem's side. Linux has many filesystems,
example, SELinux stores file context here. File and the good news is that all major players in this
capabilities also rely on this mechanism. Then, there field provide xattrs. This includes both ext4 and btrfs,
is a system.* namespace. POSIX Access Control Lists which you likely have on your PC or server. Distributed
(ACL) live there, as we'll learn in a moment. Trusted filesystems, say HDFS or the aforementioned
extended attributes live in trusted.* are available only GlusterFS, implement them as well.
to trusted userspace processes. A process needs There are a couple of commands you may use
to work with extended attributes from the shell. In
Ubuntu, they come with the attr package, which you
are not likely to have by default. So, install it via apt-
get. In other major distributions, the package name is
the same, or similar.
To get extended attributes from a file, you use the
getfattr command. It understands a few options: -d
dumps all extended attributes that match the filter
Eiciel (https://rofi.
that -m sets. This defaults to ^user\\., or extended
roger-ferrer.org/eiciel)
is a graphical extended user attributes only. For anything else, you'd want to
attributes/ACL editor for adjust this regular expression accordingly. A typical
Gnome. It also appears as file doesn't have too many extended attributes, so you
a tab in Nautilus. may disable the filter altogether with -m -.
94 www.linuxvoice.com
EXTENDED ATTRIBUTES CORETECHNOLOGY
If you already know the name of the attribute you
are looking for, tell getfattr with -n. As extended
attributes are binary values in general, the command
encodes them when printing. By default, Base64 is
used, but you can force hexadecimal or text encoding
with -e. -R makes getfattr recursive. This way, you can
list extended attributes on everything below the given
directory. This is helpful sometimes, as find knows
nothing about extended attributes.
In the wild
By now, you may start wondering where you can find FreeDesktop has some
extended attributes in your system. Recall that some $ setfattr -n user.sha256sum -v "$(sha256sum -b recommendation for
programs already use them for various purposes. somefile)" somefile extended user attributes.
Everything we need is a list of candidates to hunt for. $ getfattr -n user.sha256sum --only-values somefile | Consult them before
Personally, I'm a Netscape, then Mozilla, then Firefox sha256sum -c reinventing the wheel.
user since the late 90s. The rest of the world seems to somefile: OK
opt for Google Chrome (or at least Chromium) in 2016. We need the double quotes with `setfattr` (LV027),
If you are from this camp, try this: as `sha256sum` output contains embedded spaces.
$ cd ~/Downloads Moreover, we supply the --only-values switch to
$ getfattr -d getfattr -d patch-4.6.xz dump the raw attribute value. This is a previous
# file: patch-4.6.xz sha256sum output in this case. If we used trusted
user.xdg.origin.url="https://cdn.kernel.org/pub/linux/ extended attributes instead of user ones, this could
kernel/v4.x/patch-4.6.xz" serve as a reliable integrity check method.
user.xdg.referrer.url="https://www.kernel.org/" However, not every Linux command preserves
Of course, you should supply getfattr something extended attributes when copying or moving your files
you downloaded recently instead of my patch-4.6.xz. around. Let's do some quick experiments:
You see that Chromium remembers the file's original $ setfattr -n user.attr -v 1 somefile
URL and the page it was downloaded from, or the $ cp somefile somefile_copy
referrer. Isn't it handy, especially if you can't remember $ mv somefile anotherfile
where you got this file? If you aren't a Chrome fan $ getfattr -n user.attr somefile_copy anotherfile
(hooray!), the curl command line tool can do much the somefile_copy: user.attr: No such attribute
same. You only need to send it an --xattr option: # file: anotherfile
$ curl -s -o patch-4.6.xz --xattr https://cdn.kernel.org/ user.attr="1"
pub/linux/kernel/v4.x/patch-4.6.xz You see that mv preserves extended attributes
$ getfattr patch-4.6.xz while cp doesn't – at least, that's the default
# file: patch-4.6.xz behaviour. tar and rsync are like cp in this sense. This
user.mime_type="application/x-xz" is because moving a file keeps its inode number the
user.xdg.origin.url="https://cdn.kernel.org/pub/linux/ same, and extended attributes are really associated
kernel/v4.x/patch-4.6.xz" with inodes. Even if mv works across filesystem
The attributes are slightly different now. curl doesn't boundaries, it has to emulate this behaviour.
know the page you get the link from, so it doesn't You may tell cp and friends to preserve extended
set the referrer. Still, they are "common extended attributes as well. For cp, --preserve=xattr does the
attributes", in the FreeDesktop specification's sense. trick. Other tools accept --xattr (rsync) or --xattrs
(tar), akin to curl. The bottom line is you shouldn't
Making your own treat extended attributes and the file as something
You can also create your own custom attributes with indivisible. An unintended cp may rip everything off.
setfattr tool. It's simple: Recall that extended user attributes aren't the only
$ setfattr -n user.tag -v todo somefile namespace. Here's what I have attached to a humble
$ getfattr -n user.tag somefile ping on my Fedora 23 box:
# file: somefile $ getfattr -m - -d /usr/bin/ping
user.tag = "todo" # file: usr/bin/ping
This way, you can implement file tags at the security.capability=0sAQAAAgAwAAAAAAAAAAAAAAA
filesystem level. Below is a quick way to find all files AAAA=
marked with the todo tag in your home directory: security.selinux="system_u:object_r:ping_exec_t:s0"
$ getfattr -R -n user.tag ~ | grep file: The first attribute, security.selinux, defines the
If this yields more than a handful of items, perhaps SELinux security context for the file. The second,
you want to rethink your time management strategy. security.capability, defines the file capabilites. We
You may also use extended attributes to store the discussed them back in LV023, but in case you
file's hash sum, such as MD5 or SHA256: missed that issue, capabilities are what give ping the
www.linuxvoice.com 95
CORETECHNOLOGY EXTENDED ATTRIBUTES
them together? POSIX ACL facilitates this with the
default ACL that you can assign to a directory. Later,
when you create something within that directory, it
will automatically get a copy of the default ACL to
start with. The directory's default ACL doesn't have
to coincide with its own ACL. Say, you may have
a directory executable for Bob, so he can list its
contents. This doesn't automatically mean that Bob
will be able to execute any file within the directory.
Files in the directory may also have an ACL other than
the default. It's adjustable with the commands we'll
see in a moment.
Every file you download in right to create raw network sockets. Otherwise, this is Finally, you want some notation to communicate
Chrome gets a couple of
a privileged operation available to the root user only. POSIX ACL to the system. There are two of them: the
extended attributes as a
You surely know how traditional file permissions long text form and the short text form. In either case,
bonus. Well done, Google!
work in Linux. There are three groups of bits the entry is represented as a colon-separated string. Its
responsible for read, write and execute permissions fields are the type, the qualifier (a user or group name
for the owner, group and everyone else, respectively. or ID), and permission bits. The latter use well-known
There are also some special bits like the sticky one, rwx notation. In long text form, each entry comes on
but they don't change the overall picture. a separate line. The hash mark (#) starts a comment.
Traditional Unix permissions do their job quite well. If there is an ACL_MASK entry present, a comment
But there are some rare configurations that you can't describes effective access rights. Consider this:
express in their terms. Say, Alice wants read and write user::rwx # ACL_USER_OBJ entry
permissions for herself and Bob, read permission for user:alice:rwx # effective: rw-
the group and nothing for the rest of the world. group::rw- # ACL_GROUP_OBJ entry
POSIX ACL comes to the rescue. If you have some mask::rw-
experience administering Microsoft Windows, you other::r--
already know the concepts. Each filesystem object Here, the owner has full permissions. Alice wants
has a set of associated users and groups, along with the same, but ACL_MASK disables the x bit for her.
their respective access rights. As there is no hard The group can read and write. If it also wanted to
division by owner, group and the world, ACL allows for execute, ACL_MASK would disallow it. Everyone else
much greater flexibility. This is hardly a feature you'll can read, and that's all.
use often, yet it may come up useful someday. The short text form is more compact. Entries are
More specifically, POSIX ACL is a set of entries comma-separated, and no comments are allowed.
consisting of a type, a qualifier, and a set of You may also abbreviate types as u, g, m and o. The
permissions (see acl(5)). There are six distinct long text form is intended to be human-readable; short
entry types. Three of them are the traditional owner, text form is mostly an interchange format.
group and others, known as ACL_USER_OBJ,
ACL_GROUP_OBJ and ACL_OTHER. Then there's Time to play
ACL_MASK, which plays a role similar to umask. Now, let us play with POSIX ACL a bit. To manipulate
That is, it contains maximum permissions for the file access rights, you'll need several command-line tools
or directory. If some permission bit is unset in ACL_ which come with the acl package. You should be
MASK, setting it for any user (other than the owner) or able to find one in your distribution's repositories. The
group will have no effect. Two other types, ACL_USER getfacl tool displays ACL for the filesystem object:
and ACL_GROUP, define access permissions for $ getfacl somefile
named users and groups. # file: path/to/somefile
A natural question now is how POSIX ACL and
traditional Unix permissions interplay. The answer
is they are always in sync. If you change file owner
Do it in Python
permissions, ACL_USER_OBJ entry is updated, and This month, we look at extended attributes and POSIX ACL
vice versa. Group permissions are little more tricky. from the shell angle. However, the commands we covered
are by no means the only interface to these features.
They are mapped to ACL_MASK, if it is present. If
For more complex scripting needs, consider two Python
not, they are mapped to ACL_GROUP_OBJ. World bindings: xattr (https://pypi.python.org/pypi/xattr) and
permissions are always synchronised with ACL_ pyxattr (http://pyxattr.k1024.org). The former claims to
OTHER. In other words, POSIX ACL is an extension support more platforms: Linux and Mac OX S plus FreeBSD
mechanism which complements traditional Unix file and Solaris in experimental status. The latter has a
somewhat cleaner API and is better documented. xattr also
permissions. You can think in terms of POSIX ACL
provides a command-line tool of the same name. You can
exclusively, yet it is rarely useful. use it to manage extended attributes much the same way
It is also natural for files within a single directory to you do with getfattr and setfattr.
share similar permissions. Otherwise, why do you put
96 www.linuxvoice.com
EXTENDED ATTRIBUTES CORETECHNOLOGY
# owner: val
# group: val
user::rw-
group::r--
other::r--
You see it uses the long text form. -t switches to
tabular format, and you can turn off comments with
-c. -d dumps the default ACL for an object. You can
say somefile doesn't have any POSIX ACL assigned
– otherwise, there would be more than three entries.
Let's fix this:
$ setfacl -m user:alice:rw somefile
$ getfacl -c somefile
user::rw-
KDE (well, the Dolphin file
user:alice:rw- POSIX ACL are really extended attributes that the
manager) understands
... kernel recognises and handles appropriately. Our good POSIX ACL and lets you
We granted Alice a write permission to somefile old friend getfattr proves it easily: manage them out of the
with setfacl. -m tells us that we want to modify $ getfattr -m - -d somefile box.
the existing access list. If we wanted to revoke a # file: somefile
permission, we'd use -x. It is also possible to create a system.posix_acl_access=0sAgAAAAEABgD/////...
default ACL for a directory with setfacl -d: The value doesn't say much by itself; that's why we
$ mkdir someplace use dedicated tools like getfacl. This also means you
$ setfacl -d -m user:alice:rw someplace may inadvertently rip off POSIX ACL with cp:
$ touch someplace/somefile $ cp somefile anotherfile
$ getfacl -c someplace/somefile $ ls -l anotherfile
user::rw- -rw-rw-r-- 1 val val 0 Jun 7 00:53 anotherfile
user:alice:rw- Note there is no +. Again, mv preserves POSIX ACL,
... unless the target filesystem doesn't provide support
Instead of providing permission entries at the for extended attributes.
command line, you can use the -M and -X options. Extended attributes and POSIX ACL are quite
They accept the names of files containing the ACL simple yet are somewhat a disguised feature. Their
you want to apply. You can easily tell when an object unpopularity is not a consequence of some design
has some POSIX ACL entries. ls -l marks those files or flaw; extended attributes are a low-level mechanism,
directories with a plus sign: so you don't always have to interface with it directly.
$ ls -l POSIX ACL is a bit too flexible for a general Linux
-rw-rw-r--+ 1 val val 0 Jun 7 00:29 somefile system. It is fun to play with, but if you use POSIX ACL
drwxrwxr-x+ 2 val val 4096 Jun 7 00:40 someplace in production, please drop us a note.
Command of the month: chattr and lsattr
Linux recognises quite a few attributes, but only a all supported attributes in chattr(1). Two commands
handful of them are seen in the wild. Many attributes exists to manage these attributes. First, there's chattr
are related to dynamic file compression. The data is to set and unset them. The syntax is simple. You
stored "zipped" on the disk and uncompressed on supply a single-letter attribute name prefixed with
the fly when you read it. There is an attribute to wipe + to enable the attribute, or - to disable it. = causes
deleted files with zeros. And there is also an attribute chattr to overwrite the current attributes with those
that makes file undeletion possible. you specified. Say, chattr +i somefile makes somefile
Traditional file attributes are case-sensitive single- immutable. The lsattr command lists attributes
letter values. For instance, A prescribes not to update for the files you specify, or everything in the current
the file's access time. This may save some power directory:
on disk I/O, especially on laptops with magnetic hard $ lsattr
disks. i makes a file immutable. No user, including -------------e-- ./somefile
root, can move or delete an immutable file, or create ----i--------e-- ./immutable
a hard link to it. Only a superuser or somebody with Here, both files are using filesystem extents, and
CAP_LINUX_IMMUTABLE capability can add this immutable is, well, immutable:
attribute to a file or remove it once added. $ sudo rm immutable
The e attribute is read-only, and it is a marker that rm: cannot remove ‘immutable’: Operation not permitted
the file is using filesystem extents. You'll find the list of Even the almighty root isn't that mighty at times.
www.linuxvoice.com 97
LINUX INSIDE
LINUX INSIDE:
THE FALCON 9
Free Software at the cutting edge
of space exploration
SpaceX’s Falcon 9 rocket – shown here launching a
Dragon spacecraft on a resupply mission to the
International Space Station – uses a combination
of liquid oxygen and kerosene to propel cargoes
into space. Unlike other commercial rockets, the
Falcon 9 is designed to be reusable, and on 8 April
2016, it became the first rocket to successfully land
at sea when it touched down on a robotic drone-
ship in the Atlantic. This reuse should enable
SpaceX to dramatically lower the cost of space
exploration, and it’s all powered by the Linux kernel.
98 www.linuxvoice.com
This is what we’ve done in the last 24 issues.
Subscribe to the next 12 from just £38.
Every subscription includes access to every PDF, ePub and audio edition we’ve ever published.
shop.linuxvoice.com