CELF 2013 Toybox talk - http://landley.net/talks/celf-2013.txt Video of presentation: https://www.youtube.com/watch?v=SGmtP5Lg_t0 What is Toybox - Fresh implementation of linux/android command line, BSD licensed. - Draws from Posix, LSB, LFS, toolbox, bash man page, etc. Why is toybox? http://landley.net/aboriginal/about.html#selfhost - The smartphone is replacing the PC. - mainframe -> minicomputer ->microcomptuer ->smartphone - disruptive technology (8->16->32->64 bits is sustaining technology) - Previous generation kicked up into "server space", accessed through new - becomes general purpose as matures (Dalvik is the new ROM basic) - successful tablet is big phone, not small PC. - USB hub, USB keyboard, USB mouse, USB->HDMI adapter: dev workstation http://www.lilliputuk.com/monitors/usb/um900/ - gigahertz processor, gigabyte of ram, gigabytes of storage. - kick off giant build job on Amazon ec2 cloud if you want. - Not independent until self hosting. - DOS cross compiled from minicomputer - outgrew ROM basic to turbo/gw/quick basic, to turbo/quick/visual C. - software: kernel, cmdline (toybox), libc (musl), toolchain (qcc?) - not yet solved problem, but solvable problem. - why not extend toolbox/bionic? - because there's nothing there. Both are stubs. - intentional locked down system. 1 billion unadministered broadband. - linux containers - compiler still a problem. - my dream - qcc (tinycc + qemu tcg, both bsd licensed) - not there yet, not time to do this until after toybox 1.0. - alternatives - LLVM is sponsored by Apple (serves iPhone first), C++, but there. - pcc, open64, tinycc... - gcc as mis-licensed blob, aftermarket install. - netscape browser, nvidia driver, flash plugin... - this was not a successful strategy for the desktop. - note: gui not our problem, leave that to android. - Android must win over iPhone or next generation locked down. - Apple is not hamstrung by its own incompetence. - Apple vs franklin decision created proprietary software in first place. - Put clones (power computing) out of business. Sue bloggers. QEMU can't emulate without signing key in keyboard controller (long before win8). - today's Apple does not see Linux as existential thread - but Steve Jobs is dead. - Network effects, S-curve, natural "standards", winner take all. - positive feedback creates de-facto standard (natural monopoly) - owners of standards resist commodification (corner market, moat) - windows refund day, beos killed by bootloader exclusion, UEFI... - Android not vanilla linux: oppose or accept? - Reasons vanilla linux won't succeed here - too late for new entrant (established leaders, winner take all) - third generation of mac vs PC. (Apple II vs S100) - We've had purists before - CP/M-16 or MP/M are the _real_ OS, DOS is inferior. - Use _real_ unix not a clone... - microvax (disruption goes upmarket, not down) - unity and windows 8 suck at _both_. - openmoko was our windows CE: both predated iPhone/android. - preinstalls (channel partners): claimed, serious latency - let Tandy be Tandy - Major: open source can't do UI, flaw in dev model (aesthetic issues). - result: 20th annual "year of the linux desktop", the boy who cried wolf still "feels like it was designed by a committee". - Is android better than iOS? Can we further open android later? - Android not copyleft: oppose or accept? - Copyleft is dying. - GPLv2 was category killer, synonymous with copyleft. - terminal node of a directed graph of license convertability - universal receiver - A license was either GPL-compatible or it wasn't. - GPLv3 broke "the" GPL into incompatible forks that can't share code. - Linux and Samba can't share code, implement 2 ends of same protocol. - QEMU caught between GPLv2 Linux drivers and GPLv3 binutils/gdb processor instruction set descriptions. Can't take code from both. - "GPLv2 or later" give to both but can't take code from _either_one_. - FSF expected universal compliance, but hijacked lifeboat clause when boat wasn't sinking. GPLv2 not rendered unenforceable, they just pulled a Darth Vader "I am altering the bargain, pray I don't alter it any further." And then shipped The Phantom Meanance. - This has poisoned copyleft. GPL use declining. Multiple GPLv3 successor projects (http://xkcd.com/927). - The one thing GPLv3 achieved was undermining GPLv2. - fragmentation only increasing http://www.freerepublic.com/focus/f-chat/2905643/posts https://lwn.net/Articles/538839/ - In the absence of universal receiver, go to universal donor (BSD/PD) - Programmers are not lawyers. - I learned ONE license, with 17 years testing, it went away, ok then. - Alternative is the "civil disobedience" (napster) approach: http://tieguy.org/blog/2013/01/27/taking-post-open-source-seriously-as-a-statement-about-copyright-law/ Considering copyright/patent/IP too dumb to live, not participating, causes huge headaches while IP system still there. - why not extend toolbox/bionic instead of replace? - because there's nothing there - intentional locked down system. 1 billion unadministered broadband. - just enough to run dalvik. (The new ROM BASIC.) - Must outgrow dalvik to run native code. - linux containers, union mounts - vm vs container - chroot on steroids. Own pids, mounts, network routing... - systematic nesting of unix permissions - problem: LXC is not ripe yet, and vanilla ahead of android. - vanilla's working on this - LXC is just one userspace control package for kernel infrastructure - containers: namespaces, cgroups, mount flags... - Not a fan of selinux, apparmor, capability bits - you don't get "watertight" by plugging the holes in a collander. - system's hard enough to administer already, so you make it _more_ complicated? Micromanaging apps. - Ken Thompson said a file is a sequence of bytes. If you need extended attributes, you're doing it wrong. - Containers (namespaces, cgroups) are unix way to do this. Nest the whole system so each container has its own root. - This is a judgement call, I could be argued out of it by users who really want this stuff and can make a good case for it. But signifcant complexity cost. And selinux is not a substitute for "nothing outside dalvik to exploit" minimalism. - So: BSD licensed command line implementation. - simple, readable, security auditable, minimize attack surface. - SPOT: single point of truth, only need to change it once. - code reuse - is good if it accomplishes these goals - is BAD if it makes you pull in code you don't need. - Not using trumps RE-using. - Code you didn't write is not cost-free. - #ifdef considered harmful (busybox is full of 'em now) http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful - good support for linux containers - mostly transparent from within - drop-in replacement for android toolbox - sufficient for self hosting development environment - standards compliant - no external dependencies - not even ncurses, zlib. - just libc and built-in code. - mulicall binary - statically linked and dropped on a system. - Tried shell script snippets, thought better of it. - two install codepaths, two search paths, implement in two languages... - portable between linux and linux - android/vanilla, distros, 4.x release. - maintains binary compatability, yet a.out went away anyway. - I'll care about macosx when i can download a test environment and boot it under qemu. - exit 0 has been success since 1970, EXIT_SUCCESS pointless. stdin/stdout/stderr are 0, 1, 2. - less obvious: permission bit positions are defined in posix. - note: make defconfig is maximum sane configuration, commands that are "done" - status.html So: What is toybox? - started in 2006: I could better than busybox by starting over. - mothballed ~2008: proved I could, not enough reason to continue - undermine designated successor, 10 year headstart, several years of my own work, good enough for most users... did other things. - being as good or slightly better than existing entrenched solution with giant headerstart not enough to convert users. - Push upstream. (Toybox "patch" in busybox, integrated config/help.) - relaunched Nov 2011: targeting android - Tim Bird's "bentobox" project, mirror of android upstreaming - I owned the copyrights, only 5 other ocntributors at the time, I could triage and relicense it. - not copyleft (android policy: no GPL in userspace) - busybox predates android. Waiting for them to "come around" is like waiting for the kernel to switch to GPLv3 or PC vendors to switch to Linux on the desktop. - Leveraging ~2 years work at the start, and another year since. What does toybox implement? http://landley.net/toybox/roadmap.html http://landley.net/toybox/status.html - posix-2008 (SUSv4, etc.) - big command list, best available standard, but not perfect. - lots of obsolete crap (sccs, ed, batch processing commands) - failure mode is to omit stuff (init, mount) - triage - LSB 4.1 - deeply crappy standard. - half of it is reverting deprecations in posix. - at least this specifies "mount". - but not init - specifies rpm: bad call. Ubuntu ain't RPM. - triage, take this one with a grain of salt. - android toolbox - lives in "android core" git. - triage - container support for security - unshare - examine lxc. (Fix chroot in kernel.) - google has epic "not invented here" syndrome - install toybox, put first in $PATH. Leave toybox alone. - Eliminating need for toolbox won't eliminate safety blanket. - development environment - I personally have domain expertise here. 10 years of Aboriginal Linux. - leverage existing work I've done, got me into busybox in first place - aboriginal linux (Ab Origine: latin, "from the beginning") http://landley.net/aboriginal https://speakerdeck.com/mirell/developing-for-non-x86-targets-using-qemu - smallest, simplest self hosting development environment - I built LFS, automated it, replaced gnu packages with busybox/uClibc - A linux that even the FSF can't stick a GNU/ on the front of. - I extended busybox so much the maintainer handed over the project. - LFS packages I replaced with busybox: - bzip2, coreutils, diffutils, e2fsprogs, file, findutils, gawk, - grep, inetutils, less, modutils, net-tools, patch, procps, sed, - shadow, sysklogd, sysvinit, tar, util-linux, and vim. ALMOST bash - seven packages total - linux, busybox, uClibc, binutils, gcc, make, bash - but I can get it down further. Not defconfig, select just some busybox commands. (baseconfig-busybox) - record-commands.sh - host-tools.sh - symlink a few known binaries from host - build the rest from sorurce. (Busybox, squashfs...) - work out minimal busybox .config to build itself and LFS 6.8 - then add obvious missing stuff like "ps", "vi", sh command history. - init script needs mount and such. - build Linux From Scratch - not what they _provide_ (tcl/expect?) but what they _need_ in $PATH. - deployed record-commands.sh on target to instrument native build. - busybox? - Already taken into account via aboriginal (baseconfig-busybox). - NOT using defconfig anymore. Trimmed list significantly. - busybox including something is not a reason for toybox to have it. - katamari damacy theme's been playing for a few years. - "A maintainer's job is to say no." - Alan Cox. - I wrote many busybox commands. Already ported lots of "provably my code". - many others easy to redo, I did them once already. (sed, mount...) - Other packages that do something similar, which we may be able to replace. - klibc, sash, sbase, beastiebox, s6/9base, nash... - http://xkcd.com/927 - Things we _should_ have: - containers (unshare, lxc) - still a moving target - designed by big iron cardpunch guys. - Elaborate administration, giant config files - failure of this implementation of the tool, not infrastructure - network administration stuff - route/ifconfig family - dhcp (client, server, relay, dumpleases) - filesystems - mkswap was trivial - mkfatfs is in android toolbox - as long as we've opened this can of worms: - mkisofs, mke2fs (ext3, ext4), squashfs - btrfs? flash filesystems? - (reiserfs died with Nina. Deal with it.) - treat them as archivers: create/extract from directory tree, streaming (mke2fs dir | gzip > out.img) - mdev - based on devtmpfs, kernel loads own firmware - init (big unsolved problem) - oneit, sysvinit, runit, android init, upstart - let systemd die with GNU-slash - shell that's a reasonable bash replacement - not everything, but {some,things} are <(useful) - circa bash 2.x really - pipefail copied by several other shells already, but not in posix/dash - no time to go into the dash mess - bash was first program linux ever ran - Linus's book "just for fun": term program extended to run bash - default shell of every linux distro from 0.0.1 - ubuntu: boot scripts too slow, changing per-file #!/blah too much work - https://wiki.ubuntu.com/DashAsBinSh - didn't fix anything, switched to upstart later anyway. - worst technical decision ubuntu ever made, never reverted. - some things are more complicated than they seem: - Man pages - 2005 I suggested _simple_ "man for busybox": http://lists.busybox.net/pipermail/busybox/2005-June/048842.html zcat /usr/man/some_compressed_file.gz | less OH THE BIKESHEDDING. - Doclifter (http://www.catb.org/~esr/doclifter/doclifter.html) - Note all the "macro packages" and "common idioms". What doclifter did was give man page maintainers a tool to convert their masters, and then generate man page markup from the output. Or html. Or PDF. Or asciidoc. (Almost nobody maintains man page masters anymore.) - Expecting troff to get you man is like expecting lexx and yacc to get you gcc. There's more to it. Producing 7 bit ascii output with simple highlighting should NOT be Tardis console levels of complicated, but leave bearded hermits alone in their caves for 15 years, you get tesseracts and wormholes. - ssh (dropbear exists) - really needs stunnel though - vncviewer (requires x11 dependency). - Blue sky "would be nice", dunno if it's worth it yet: - hex editor - 9p server (virtfs, simple protocol, useful. Excuse NOT to do nfs/samba.) - iotop in top - Linus's microemacs (posix specifies vi, lots of common code with less...) - netcat udp mode (for netconsole) - microterm, rsync, whois, kexec, screen, ntp/rdate - httpd, dnsd - mtd stuff? - Toolchain things that actually belong in qcc (development stuff) - make, m4, lex, yacc, c99 (posix!) http://landley.net/code/tinycc/qcc/commands.txt http://landley.net/code/tinycc/qcc/todo.txt - strace - expect (to kill tcl) - unifdef - NOP libtool (libtool makes non-elf look like elf. Does nothing on linux.) - Having nothing to do and screwing it up is a trademark of the FSF. - uboot mkimage - record-commands wrapper? CELF 2013 Toybox talk - http://landley.net/talks/celf-2013.txt Video of presentation: https://www.youtube.com/watch?v=SGmtP5Lg_t0