Registration is now open! Early-bird rates until Feb. 20.
Hotel reservations are open too! (off-site link) Conference rates until Feb. 20.
Financial aid is available! Application deadline is Feb. 11.
Dates to remember:
* Tutorials: Thursday, March 13, 2008
* Conference: Friday, March 14, through Sunday, March 16, 2008
* Sprints: Monday, March 17 through Thursday, March 20, 2008
The Python Software Foundation is proud to present PyCon 2008, the 6th annual Python community conference, this year in Chicago.
Come to PyCon 2008 in Chicago to:
* meet interesting people in the Python community
* learn about cool things others have done recently
* show off the cool things you've done recently
* learn about projects, tools & techniques
* advance open source projects
There's a lot happening at PyCon:
*
Tutorial Day: Thursday March 13
A chance for in-depth learning from the experts.
*
Conference Days: Friday March 14 to Sunday March 16.
o Keynote talks from luminaries.
o Scheduled talks on a myriad of subjects.
o Lightning talks: 5-minute impromptu talks, scheduled at the conference (come prepared to talk!).
o Open Space: plenty of rooms available for follow-up discussions after scheduled talks, group discussions of projects, unscheduled talks, demos, brainstorming...
o And the hallway track is always popular!
*
Development Sprints: Monday March 17 to Thursday March 20.
Up to four days of intensive learning about and development of open-source projects. We provide the space and the infrastructure (network, power, tables & chairs), you provide the enthusiasm and the brainpower. All experience levels welcome. The sprints are free!
*
Expo Hall: Friday March 14 & Saturday March 15.
Many sponsors and vendors will have booths in an expanded exhibitor area this year. Get a T-shirt, find a job, talk to the companies who participate in the Python community!
But most importantly,
*
The Community: PyCon is a community conference!
PyCon is organized and run by volunteers from the Python community. Please join us!
The conference venue is the Crowne Plaza Chicago O'Hare hotel. Reservations should be made here (off-site) in order to receive the conference rate.
Latest News (PyCon Blog)Subscribe to the RSS Feed
# Reminder: Financial Aid Application Deadline is Feb. 11th
2008-02-08 22:24:30
Just a quick reminder -- applications for financial aid for PyCon 2008 are due on Monday, February 11th! Please see the PyCon website for application ...
# Intro to Sprints sessions
2008-02-06 17:02:27
There will be two pre-sprint sessions that will be run on Sunday afternoon, after the end of the conference talks.Intro to Sprinting (60 minutes, at 3...
# Publicizing PyCon
2008-01-29 14:27:53
We need you to let people know about PyCon!PyCon has always relied on the community to get the word out. This year, we've put together a "Publicizing...
# PyCon 2008 Financial Aid Available
2008-01-22 20:07:29
The Python Software Foundation has allocated some funds to help people attend PyCon 2008. If you would like to come to PyCon but can't afford it, the ...
# PyCon 2008 Registration Open!
2008-01-21 01:17:12
I am pleased to announce that PyCon 2008 registration is now open! Early-bird registration is open until February 20, so there's one month to registe...
Monday, February 11, 2008
Sunday, February 10, 2008
Is Microsoft Office Adware?
Posted by Andrew Z at Saturday, February 9, 2008
Is Microsoft Office adware? Wikipedia defines adware as "any software package which automatically plays, displays, or downloads advertising material to a computer after the software is installed on it or while the application is being used."
In Microsoft Office Professional 2003's help, a search for "APA" (a popular documentation style) brings up two links labeled Microsoft Office Marketplace.
Microsoft Office Word 2003 help showing two marketplace ads
A click on the link opens a web page where the item is available for a fee. Also, there's a banner advertisement from Dell.
Screenshot of Microsoft Office Marketplace
A search in Office's help for "print" leads to the brief article "Print more than one copy" with two up-selling links for Office 2007.
Microsoft Office Word 2003 help showing two up-selling ads for Office 2007
By the way, clicking either promotion launches Internet Explorer even when Firefox is the default browser. The reason is the Office help runs an embedded Internet Explorer showing a page from office.microsoft.com. Despite the technical explanation, it can be confusing and inconvenient for the user to use a non-default browser.
On Microsoft.com, Sandi Hardmeier, MVP, concludes her adware definition, "Ads are not bad by themselves but they become a problem when they are unauthorized. Unfortunately, many adware programs do not give users enough notice or control." In Office, where is the "notice or control"? A workaround is to search the Offline Help instead of the default Microsoft Office Online.
Could Office be spyware? Microsoft defines spyware as "software that performs certain behaviors such as advertising." Microsoft continues:
That does not mean all software that provides ads or tracks your online activities is bad. For example, you might sign up for a free music service, but you "pay" for the service by agreeing to receive targeted ads. If you understand the terms and agree to them, you may have decided that it is a fair tradeoff. You might also agree to let the company track your online activities to determine which ads to show you.
Basically, spyware includes adware, but not all ads are bad. Are ads bad about after paying for an Office license?
Another part of spyware is tracking. The Office 2003 EULA (.PDF and .XPS wrapped in .EXE) states:
CONSENT TO USE OF DATA. You agree that Microsoft and its affiliates may collect and use technical information gathered as part of the product support services provided to you, if any, related to the Software. Microsoft may use this information solely to improve our products or to provide customized services or technologies to you and will not disclose this information in a form that personally identifies you.
While in Office the collected information is not personal, the EULA does not mention disabling collection.
The Office Privacy Statement admits the use of tracking cookies. While cookies are normal and generally harmless, it is unusual to require cookies or to use them in a desktop application. If cookies are disabled in Internet Explorer, the Office help fails.
Microsoft Office Word 2003 help: an error when cookies are disabled in Internet Explorer
In conclusion, Office 2003 does display ads, and certain parts require cookies. While these are a normal and healthy part of the web, it is, at least, unusual for a commercial desktop application.
Is Microsoft Office adware? Wikipedia defines adware as "any software package which automatically plays, displays, or downloads advertising material to a computer after the software is installed on it or while the application is being used."
In Microsoft Office Professional 2003's help, a search for "APA" (a popular documentation style) brings up two links labeled Microsoft Office Marketplace.
Microsoft Office Word 2003 help showing two marketplace ads
A click on the link opens a web page where the item is available for a fee. Also, there's a banner advertisement from Dell.
Screenshot of Microsoft Office Marketplace
A search in Office's help for "print" leads to the brief article "Print more than one copy" with two up-selling links for Office 2007.
Microsoft Office Word 2003 help showing two up-selling ads for Office 2007
By the way, clicking either promotion launches Internet Explorer even when Firefox is the default browser. The reason is the Office help runs an embedded Internet Explorer showing a page from office.microsoft.com. Despite the technical explanation, it can be confusing and inconvenient for the user to use a non-default browser.
On Microsoft.com, Sandi Hardmeier, MVP, concludes her adware definition, "Ads are not bad by themselves but they become a problem when they are unauthorized. Unfortunately, many adware programs do not give users enough notice or control." In Office, where is the "notice or control"? A workaround is to search the Offline Help instead of the default Microsoft Office Online.
Could Office be spyware? Microsoft defines spyware as "software that performs certain behaviors such as advertising." Microsoft continues:
That does not mean all software that provides ads or tracks your online activities is bad. For example, you might sign up for a free music service, but you "pay" for the service by agreeing to receive targeted ads. If you understand the terms and agree to them, you may have decided that it is a fair tradeoff. You might also agree to let the company track your online activities to determine which ads to show you.
Basically, spyware includes adware, but not all ads are bad. Are ads bad about after paying for an Office license?
Another part of spyware is tracking. The Office 2003 EULA (.PDF and .XPS wrapped in .EXE) states:
CONSENT TO USE OF DATA. You agree that Microsoft and its affiliates may collect and use technical information gathered as part of the product support services provided to you, if any, related to the Software. Microsoft may use this information solely to improve our products or to provide customized services or technologies to you and will not disclose this information in a form that personally identifies you.
While in Office the collected information is not personal, the EULA does not mention disabling collection.
The Office Privacy Statement admits the use of tracking cookies. While cookies are normal and generally harmless, it is unusual to require cookies or to use them in a desktop application. If cookies are disabled in Internet Explorer, the Office help fails.
Microsoft Office Word 2003 help: an error when cookies are disabled in Internet Explorer
In conclusion, Office 2003 does display ads, and certain parts require cookies. While these are a normal and healthy part of the web, it is, at least, unusual for a commercial desktop application.
Waving the flag: NetBSD developers speak about version 4.0
By Federico Biancuzzi | Published: January 30, 2008 - 11:53PM CT (Whole Article)
Introduction
The NetBSD community announced last month the official release of NetBSD 4.0, the latest version of the Unix-like open-source operating system. Version 4.0 includes significant new features like Bluetooth support, version 3 of the Xen virtual machine monitor, new device drivers, and improvements to the Veriexec file integrity subsystem. NetBSD, which is known for its high portability, is capable of running on 54 different system architectures and is suitable for use on a wide range of hardware, including desktops, servers, mobile devices, and even kitchen toasters.
Meet the developers
To commemorate the NetBSD 4.0 launch, enthusiast Federico Biancuzzi communicated with 21 developers to produce this expansive interview with loads of insightful information about the NetBSD 4.0 development process.
NetBSD Foundation secretary Christos Zoulas will discuss why sendmail has been removed and what goals the project has for the fundraising process. He wrote a lot of code, such as svr4 emulation, isapnp code, ptyfs, siginfo, ELF loader, mach-o loader, statvfs, pam integration, etc.
Liam J. Foy will explain the delay in the release engineering process.
Elad Efrat's area of interest in NetBSD is mostly enabling security technologies. From allowing flexible fine-grained security policies on the system, to actually writing them and making them easy to deploy, he is interested in minimizing the time it takes to construct a secure installation. His major contributions to NetBSD, if trying to chronologically order them, are Veriexec, fileassoc(9), kauth(9) (and the secmodel(9) derivatives), security model abstraction (bsd44, securelevel), PaX features (MPROTECT, Segvguard, ASLR), pw_policy(3). These topics were summarized in a paper published on SecurityFocus in late 2006 and later presented in EuroBSDCon 2006. Subsets were also presented in smaller academic and military venues in Israel.
Matt Fleming will describe how veriexecgen works.
Nicolas Joly will present the status of Linux binary compatibility.
Matthias Scheler is currently mostly working on "pkgsrc" and occasionally fixing bugs in the base system. He was the responsible release engineer who managed the NetBSD 3.0 release. He will talk about the X Window System and the new digital transfer mode in cdplay.
Joerg Sonnenberger contributed a lot of infrastructure work for pkgsrc and he is the responsible for modular Xorg in pkgsrc. His contributions in the NetBSD base system are ACPI and x86 improvements.
Jason Thorpe will describe proplib(3).
Manuel Bouyer is a NetBSD user since 0.8. His first contribution was porting the OpenBSD shared library support for NetBSD/pmax in 1995, then he started working on ATAPI devices support. In late 1996/early 1997, he also wrote the ext2fs support, starting from the ffs code (no GPL code in it). He got invited to becode a developer at this time to integrate his work on ATAPI and ext2fs. After that, he kept working in the ATA and SCSI areas (he added the primary DMA support for PCI ATA controllers for example). On occasion, he also worked on other parts of the kernel (Ethernet controller drivers, of dual-endian support to FFS, for example).
When Xen 2.0 was out, he started working on domain0 support so that he could build virtual servers not relying on Linux. He added support for Xen 3.0 when it got out, and became NetBSD/xen portmaster at about this time. He also became a releng member in early 2007 to give some manpower to the team. He will give an overview of NetBSD/Xen and the new 'no-emulation' eltorito boot method.
Phil Nelson installed and ran the first web server serving www.netbsd.org. (the service has since been taken over by project servers.) He still does some minor admin work for the web server.
He runs the WWU build cluster and improved the scripts for the build clusters so they could build releases faster. He wrote a program called xarcmp that compares the contents of archives without having to extract them. It is part of the standard tool now used by both build clusters.
He wrote the initial versions of menuc, msgc and sysinst. sysinst is the current install system and it uses menuc and msgc. He will talk about LFS (Log-structured File System).
Julio M. Merino Vidal has been a member of The NetBSD Foundation and a developer since November 2002. He started contributing to the pkgsrc project with the main goal of getting GNOME 2 to work under NetBSD. He then made other contributions to the core NetBSD operating system, the most relevant of which are the tmpfs file system and the automated testing framework (ATF). He will discuss the features of tmpfs.
Antti Kantee worked on pkgsrc, device drivers and hardware support for various platforms, file systems, and assorted kernel and userland bits. He maintains the file(1) utility in the NetBSD source tree and NetHack in pkgsrc. He will describe the relationship between puffs, FUSE, and ReFUSE.
Alistair Crooks is the president of The NetBSD Foundation, Core Team member, and founder of pkgsrc.
He contributed to pkgsrc, user management software - user(8), ReFUSE (BSD-licensed re-implementation of FUSE), numerous filesystems based on ReFUSE, and iSCSI target and initiator. He will make clear which parts of the iSCSI protocol are included in this release and how much "hackathons" are helpful.
Reinoud Zandijk will add details about the implementation of the Universal Disk Format (UDF).
Iain Hibbert's major contribution has been the Bluetooth protocol stack and associated drivers and utilities. He talks about Bluetooth support in NetBSD.
Arnaud Lacombe has been a developer for about one year now. At the beginning, he joined the project to fix bugs found by the Coverity scan of NetBSD. He is interested in porting NetBSD to new embedded platforms. OpenMoko was a really interesting way to do it, but he never managed to get hardware, so this project is stalled for the moment.
Martin Fouts, Noud de Brouwer, and Arnaud discuss the current support for the hardware included in the iPhone and OpenMoko (Neo1973).
Alan Ritter will comment on the possibility of using the NDIS wrapper to port binary drivers among platforms.
Yamamoto Takashi will describe what agr(4) is.
Jan Schaumann used to work a lot on the web site. He also ported pkgsrc to IRIX and did bulk-builds there. He is a member of the communication-exec team and managed NetBSD's participation in the first two Google Summer of Code. He will discuss their experience with the Google Summer of Code project.
Release engineering, Sendmail, and kauth
What happened to the Release Engineering process for 4.0?
Liam J. Foy: Basically, the release engineering was started as planned. However, after the release engineering started a lot of changes were made which would be too time consuming to pull up (pull up from current to the NetBSD 4branch). Thus, we started the process again with all the changes merged.
You may remember a few NetBSD hackathons which took place. Well, these are what caused the large number of changes.
Why did you remove sendmail?
Christos Zoulas: Sendmail has been, is, and will be a security accident waiting to happen (unless it is rewritten from the ground up with security consciousness). Performing character pointer gymnastics in 50-100 line loops does not create any warm and fuzzy feelings for me. To top this off, most sendmail security issues are marked as confidential, and we are prevented from fixing or mentioning the problem until the ban is lifted. The last time this happened, we said "enough" and removed it altogether before the ban for that particular security issue was lifted
Would you like to present kauth, the kernel authorization framework, included in this release?
Elad Efrat: Kernel authorization is really something cool that I'm happy many people will get a chance to test in this release.
Basically, the story goes like this: the traditional Unix security model, the one where root is almighty and everyone else is not, is slowly beginning to show signs of age as demands for finer-grained security policies arise. The problem becomes more apparent when you look at the kernel code and see that, probably as time went by, people used various ways to check whether a certain operation is allowed to be performed or not—most are variants of checking whether the effective user-id is 0 or using the suser() function—effectively embedding the security model to the kernel, in a way that the question "Can this user open a raw socket?" really became "Is this user root?"
Kauth(9) was originally designed by Apple, and it provides a dispatching model that allows us to really ask if a certain operation can be performed by specifying it along with the relevant context. This abstracts the security model from the various kernel subsystems. The system is divided to "scopes" that collect actions of the same nature (right now, NetBSD has scopes like "process", "network","machdep", etc.), and each scope has a set of "actions" that define the operation (the context provided depends on the operation). When the kernel wants to ask if a certain operation is allowed, it calls the authorization wrapper for the relevant scope, providing it the action and context, and receives back a binary response of "yes" or "no".
This upper layer, like I said, effectively abstracts the way these decisions are made, allowing one to "plug" (almost) any security model one can think of.
These security models are implemented by attaching "listeners" (kauth(9) terminology for callback functions) that receive the action and context and make the decision. In the future it is hoped that these listeners will be able to run on an entirely different machine, thus allowing a centralized security policy for an entire network to be controlled from a single host.
Unfortunately, in NetBSD 4.0, although the security model abstraction is almost complete, there is still not full control over all operations. That is—in some places, the question is still "Is this user root?", so implementing different security models will not be complete. However, it will provide an indication of how well kauth(9) works and how users react to this new ability to develop custom security models (for example, classic uses like restricting raw networking, or binding to certain low ports, to a few users). I'm hoping to be surprised by our user base's creativity.
I highly recommend spending a few minutes reading kauth(9) and secmodel(9) in the current man pages. There's a little taste of what's in the plans for NetBSD 5.0—like credential inheritance control—with the ultimate goal being to provide system administrators with security policy control that requires either zero or very little effort.
PaX, fileassoc, and Veriexec
What security features have you added to mprotect(2) from PaX?
Elad Efrat: First, the PaX project, for those who aren't familiar with it, is responsible for all the modern exploit mitigation technologies. It's where stuff like W^X and ASLR (Address Space Layout Randomization) were born, along with many, many other cool features. One of them, which I initially ported to NetBSD, is PaX MPROTECT.
PaX MPROTECT can be thought of as "strict W^X". If in a normal system, a program starts with no memory pages that are both writable and executable, but those pages can still be created using mprotect(2), systems or programs that run with PaX MPROTECT are "immune" to attacks where protection on pages is modified, often by trashing arguments to mprotect(2).
paxctl(8) is a tool that enables PaX MPROTECT on a per-program basis, if you don't want to enable it globally. I'm afraid that not much testing was done on various architectures using this feature, so users should first experiment with it...
How does the new fileassoc KPI (Kernel Programming Interface) work?
Elad Efrat: I'll begin by describing what fileassoc(9) is.
Some (newer) filesystems allow on to attach metadata to files by using extended attributes (in addition to the usual ones—permissions, timestamps, etc.) and such. A common use for them may be, for example, to store ACLs (Access Control List) for the file that allow finer granularity on access control.
The problem is that these extended attributes are filesystem-dependent and either don't exist on most filesystems or are interfaced differently. This is where fileassoc(9) comes into play: it allows you to attach metadata to files in a filesystem-independent way while storing the information in fast-access kernel tables. The advantage is that it really is filesystem-independent, so there's the potential of adding features and such to filesystems that lack them (for example, again, ACLs). The disadvantages are that the metadata is not really "attached" to the file in any way, and must be loaded in some form—from a database file or such—to the kernel. So that the dependency becomes on the OS itself, but that's cool, because it's aimed for NetBSDsystems.
The KPI works by storing data in hash tables. Metadata is identified by a key (a C string) and is a stream of bytes. A kernel subsystem can then use the fileassoc(9) KPI to either attach, query, or remove metadata from files, by specifying the key.
The fileassoc(9) KPI is implemented in the VFS (Virtual File System) layer, identifying each file using its "file handle" (these are supposed to be unique). So pretty much any filesystem is supported. An example (and the only, at the moment) consumer of fileassoc(9) is Veriexec, where a database file holds information that is parsed by a userland tool, then fed to fileassoc(9) using a special device.
Plans to write a generic interface for communicating with fileassoc(9) have come up in the past, but since we're not yet convinced of the necessity, these are just plans at the moment. Developers interested in feeding data to fileassoc(9) should also implement their own special device stub—at least for now.
Is there any news about Veriexec?
Elad Efrat: As with NetBSD 4.0, I feel Veriexec has gotten to a point where it should probably be used by most of our users.
Veriexec is NetBSD's integrity subsystem, which, in short, can guarantee the integrity of the programs, configuration files, etc., on the system.
In addition to the tons of improvements in performance and stability, and the many features added, NetBSD 4.0 introduces 'veriexecgen', which is a tool written by Matt Fleming (mjf@). Veriexecgen tremendously lowers the bar for fingerprint database generation to a point where it's possible to run a single command with no arguments after installation to have Veriexec set-up appropriately. I strongly recommend reading the veriexecgen(8) main page and giving it a try.
What is veriexecgen and how does it work?
Matt Fleming: veriexecgen is a program that runs against directories and generates a set of fingerprints for use with veriexec. This fingerprint database is usually stored in /etc/signatures.
XFree86, pkgsrc, proplib, and Xen
Do you still include Linux binary compatibility?
Nicolas Joly: I'm currently working on improving Linux binary compatibility on AMD64 for both 32- and 64-bit applications.
The amd64 compat linux is the first one that include NPTL (NativePOSIX Thread Library) emulation found on Linux 2.6 kernel. Likewise, compat linux32 is pretty new, and even if mostly identical to i386linux emulation needs to be modified.
For NetBSD 4.0, kernel support for compat linux is not enabled bydefault as this is not as stable as other ports. For this release, do not expect to run complicated linux applications yet, but basic ones should work. In the mean time, -current has made some progresses...
Which X Window System is included in NetBSD 4.0?
Matthias Scheler: XFree86 4.5.0.
The XFree86 Project has unfortunately lost a lot of momentum. NetBSD is currently in the process of switching to the X.org X11 distribution. We initially tried integrating the monolithic X.org distribution into our xsrc source tree. But it never reached a state where it worked on all platforms and supported cross builds via build.sh.
After the X.org project changed to a modular distribution, it was obvious that pkgsrc is the best way to integrate X11 in the future. There is ongoing work mostly by Joerg Sonnenberger to make modular X11 in pkgsrc cross-buildable. When that work has been finished and properly integrated with the system installation, the XFree86 distribution in xsrc will probably be retired.
What's new in pkgsrc?
Joerg Sonnenberger: I'm only aware of NetBSD 4 being a requirement to use the cross-compiling support. The updates to pkg_install will be in NetBSD4.1, so that is ruled out.
The cross-compiling support allows a small subset of pkgsrc to be built for any architecture running NetBSD using the output of build.sh. Currently this subset includes modular Xorg and a few applications like Xpdf as proof of concept. This is intended to replace the aging XFree86in xsrc, but needs some more polishing work on the pkgsrc side.
What is proplib(3) and what can we use it for?
Jason Thorpe: proplib is a library for manipulating property lists. Property lists are collections of properties, typically stored in a dictionary. A dictionary is an associative array, essentially key-valuepairs. The keys are strings, and the values are strings, numbers, opaque data, booleans, arrays, dictionaries, ...
This is handy in a variety of applications. For example, communicating structured data between userspace and the kernel, describing properties of devices, etc.
What can we do with NetBSD/Xen?
Manuel Bouyer: Well, we can do a lot of things, from merging different physical boxes to a single one (if you need different OSes for different tasks, for example), to easily create test systems (it's very convenient for kernel developement: a guest boots much faster than a regular PC). With the virtualization features of recent X86 CPUs (which are supported by Xen3 with NetBSD as dom0), you can also boot plain i386 OSes. This can be used to test install media, for example, or to load systems that can't be paravirtualized (e.g., Windows).
Filesystems
What is the status of LFS (Log-structured File System)?
Phil Nelson: LFS is in use on about 1/2 of the 23 nodes of the WWU build cluster. Granted, it is running an older 4.0-Beta kernel, but they have been running just fine for quite a while. Currently all the machines in the WWU build cluster have been up 78+ days and continuously building. They are all i386 machines.
Would you like to describe the features of tmpfs?
Julio M. Merino Vidal: tmpfs was born as a replacement for MFS—the memory file system included in all BSDs as far as I know—and as such it shares many functionality with it. It basically is an efficient, memory-based file system which means that it uses part of the system's virtual memory to store files in a way that is more space- and speed-efficient than MFS. It supports all of the features expected from a Unix-style filesystem, including hard links, symbolic links, devices, permissions, file flags and NFS exportability, but it currently does not support sparse files.
As opposed to MFS, tmpfs file systems can grow and shrink automatically depending on the available memory if they are configured as such; their memory consumption can also be upper-limited. And even though the end user cannot directly notice it, tmpfs's code is much simpler than MFS's one, which means that it is easier to audit, test and optimize.
As you just mentioned, tmpfs is a bit different than MFS, but why did MFS need a replacement in the first place?
Julio M. Merino Vidal: The main problem of MFS is that it performs poorly, uses too much memory, and the used memory cannot be reclaimed unless the file system is unmounted. To understand why this happens, we need to see how MFS works, and to see that, we first need to outline how FFS is designed.
FFS (or UFS), the traditional BSD file system, is conceptually split in two different layers. the upper layer handles all the logic from the file system, including the management of directories, the common file operations, the routines to map data to blocks, etc. The lower layer lays out the data on disk, allocating blocks and inodes as needed, etc. This seems like a very nice design, but in the end, it imposes restrictions on the way the lower layer operates (or at least that's the impression I gathered; I'm not too familiar on FFS's code).
MFS is just a replacement of FFS's lower layer to operate on virtual memory. The approach it takes is dead simple: it allocates a contiguous block of memory and treats that as if it were a disk, organizing the contents of such memory region as disk blocks. As you can expect, such approach has a lot of overhead because the file system uses an incorrect abstraction—disk blocks and cylinder groups—to manage memory.
tmpfs, on the other hand, is a much simpler filesystem which takes advantage of the fact that it is always memory-backed. It uses traditional structures, linked lists, arrays, memory pools and other sorts of data abstractions to represent the contents of the file system. These abstractions are easier to deal with and hence require less resources to manage.
It is also interesting to mention that tmpfs grew a custom regression testing suite that was also well received by developers. But the way it was created was suboptimal, which made me start the ATF (Automated Testing Framework) project this year. We'll see a better testing suite in NetBSD 5.0, or at least I hope so! Oh, and by the way, FreeBSD also imported tmpfs in its source tree.
What is the relationship between puffs, FUSE and ReFUSE?
Antti Kantee: puffs stands for Pass-to-Userspace Framework File System. It is an interface and a framework implementation for userspace filesystems on NetBSD. The userspace filesystem interface itself is heavily influenced by the kernel virtual filesystem interface. puffs can be thought to consist of two parts: the mechanism for transporting file system requests from the kernel to userspace and a userspace library, libpuffs, for interfacing with the kernel and writing the userspace filesystem implementation.
FUSE, or Filesystem in Userspace, is another API for implementing userspace filesystems. It is native to Linux, but has been widely ported to other operating systems such as FreeBSD and OpenSolaris. FUSE is considered the standard interface for implementing userspace filesystems with numerous filesystems readily available.
ReFUSE is the implementation of the FUSE API for NetBSD. It is implemented on top of puffs and completely done in userspace. Other non-Linux operating systems implement FUSE compatibility in the kernel, but we believe the ReFUSE approach is the right way to address the issue: export the kernel filesystem interface in the most natural way possible (puffs) and implement compatibility in userspace (ReFUSE).
A very important difference to note is API stability. Since the FUSE API is well-established, it is stable, and filesystems written for it will most likely continue to function for a long time. The puffs API is a "low-level" API which follows the NetBSD kernel virtual filesystem API fairly closely. As the virtual filesystem API on NetBSD evolves, so will the puffs API (and vice versa, if things go as planned). A benefit from this evolutionary incompatibility is the ability to use all the features provided by the kernel API at the best possible performance.
What is their status in NetBSD 4.0?
Antti Kantee: puffs is still under heavy development. The version of puffs found in 4.0 is a snapshot of what was in the tree at the time 4.0 was branched. This means that ReFUSE and therefore FUSE support is unfortunately not present. The most useful application for users is likely to be ssshfs, simple sshfs, which can be found in source form from the tree under src/share/examples/puffs/ssshfs. It implements sshfs functionality. This implementation was superceded by mount_psshfs(8) for NetBSD 5.0.
Technically there is no reason why support for FUSE could not be added to the NetBSD 4 branch. However, it requires backporting ReFUSE and some puffs features from the development branch and therefore a person with time and motivation to do the work, and, above all, test and support it.
Introduction
The NetBSD community announced last month the official release of NetBSD 4.0, the latest version of the Unix-like open-source operating system. Version 4.0 includes significant new features like Bluetooth support, version 3 of the Xen virtual machine monitor, new device drivers, and improvements to the Veriexec file integrity subsystem. NetBSD, which is known for its high portability, is capable of running on 54 different system architectures and is suitable for use on a wide range of hardware, including desktops, servers, mobile devices, and even kitchen toasters.
Meet the developers
To commemorate the NetBSD 4.0 launch, enthusiast Federico Biancuzzi communicated with 21 developers to produce this expansive interview with loads of insightful information about the NetBSD 4.0 development process.
NetBSD Foundation secretary Christos Zoulas will discuss why sendmail has been removed and what goals the project has for the fundraising process. He wrote a lot of code, such as svr4 emulation, isapnp code, ptyfs, siginfo, ELF loader, mach-o loader, statvfs, pam integration, etc.
Liam J. Foy will explain the delay in the release engineering process.
Elad Efrat's area of interest in NetBSD is mostly enabling security technologies. From allowing flexible fine-grained security policies on the system, to actually writing them and making them easy to deploy, he is interested in minimizing the time it takes to construct a secure installation. His major contributions to NetBSD, if trying to chronologically order them, are Veriexec, fileassoc(9), kauth(9) (and the secmodel(9) derivatives), security model abstraction (bsd44, securelevel), PaX features (MPROTECT, Segvguard, ASLR), pw_policy(3). These topics were summarized in a paper published on SecurityFocus in late 2006 and later presented in EuroBSDCon 2006. Subsets were also presented in smaller academic and military venues in Israel.
Matt Fleming will describe how veriexecgen works.
Nicolas Joly will present the status of Linux binary compatibility.
Matthias Scheler is currently mostly working on "pkgsrc" and occasionally fixing bugs in the base system. He was the responsible release engineer who managed the NetBSD 3.0 release. He will talk about the X Window System and the new digital transfer mode in cdplay.
Joerg Sonnenberger contributed a lot of infrastructure work for pkgsrc and he is the responsible for modular Xorg in pkgsrc. His contributions in the NetBSD base system are ACPI and x86 improvements.
Jason Thorpe will describe proplib(3).
Manuel Bouyer is a NetBSD user since 0.8. His first contribution was porting the OpenBSD shared library support for NetBSD/pmax in 1995, then he started working on ATAPI devices support. In late 1996/early 1997, he also wrote the ext2fs support, starting from the ffs code (no GPL code in it). He got invited to becode a developer at this time to integrate his work on ATAPI and ext2fs. After that, he kept working in the ATA and SCSI areas (he added the primary DMA support for PCI ATA controllers for example). On occasion, he also worked on other parts of the kernel (Ethernet controller drivers, of dual-endian support to FFS, for example).
When Xen 2.0 was out, he started working on domain0 support so that he could build virtual servers not relying on Linux. He added support for Xen 3.0 when it got out, and became NetBSD/xen portmaster at about this time. He also became a releng member in early 2007 to give some manpower to the team. He will give an overview of NetBSD/Xen and the new 'no-emulation' eltorito boot method.
Phil Nelson installed and ran the first web server serving www.netbsd.org. (the service has since been taken over by project servers.) He still does some minor admin work for the web server.
He runs the WWU build cluster and improved the scripts for the build clusters so they could build releases faster. He wrote a program called xarcmp that compares the contents of archives without having to extract them. It is part of the standard tool now used by both build clusters.
He wrote the initial versions of menuc, msgc and sysinst. sysinst is the current install system and it uses menuc and msgc. He will talk about LFS (Log-structured File System).
Julio M. Merino Vidal has been a member of The NetBSD Foundation and a developer since November 2002. He started contributing to the pkgsrc project with the main goal of getting GNOME 2 to work under NetBSD. He then made other contributions to the core NetBSD operating system, the most relevant of which are the tmpfs file system and the automated testing framework (ATF). He will discuss the features of tmpfs.
Antti Kantee worked on pkgsrc, device drivers and hardware support for various platforms, file systems, and assorted kernel and userland bits. He maintains the file(1) utility in the NetBSD source tree and NetHack in pkgsrc. He will describe the relationship between puffs, FUSE, and ReFUSE.
Alistair Crooks is the president of The NetBSD Foundation, Core Team member, and founder of pkgsrc.
He contributed to pkgsrc, user management software - user(8), ReFUSE (BSD-licensed re-implementation of FUSE), numerous filesystems based on ReFUSE, and iSCSI target and initiator. He will make clear which parts of the iSCSI protocol are included in this release and how much "hackathons" are helpful.
Reinoud Zandijk will add details about the implementation of the Universal Disk Format (UDF).
Iain Hibbert's major contribution has been the Bluetooth protocol stack and associated drivers and utilities. He talks about Bluetooth support in NetBSD.
Arnaud Lacombe has been a developer for about one year now. At the beginning, he joined the project to fix bugs found by the Coverity scan of NetBSD. He is interested in porting NetBSD to new embedded platforms. OpenMoko was a really interesting way to do it, but he never managed to get hardware, so this project is stalled for the moment.
Martin Fouts, Noud de Brouwer, and Arnaud discuss the current support for the hardware included in the iPhone and OpenMoko (Neo1973).
Alan Ritter will comment on the possibility of using the NDIS wrapper to port binary drivers among platforms.
Yamamoto Takashi will describe what agr(4) is.
Jan Schaumann used to work a lot on the web site. He also ported pkgsrc to IRIX and did bulk-builds there. He is a member of the communication-exec team and managed NetBSD's participation in the first two Google Summer of Code. He will discuss their experience with the Google Summer of Code project.
Release engineering, Sendmail, and kauth
What happened to the Release Engineering process for 4.0?
Liam J. Foy: Basically, the release engineering was started as planned. However, after the release engineering started a lot of changes were made which would be too time consuming to pull up (pull up from current to the NetBSD 4branch). Thus, we started the process again with all the changes merged.
You may remember a few NetBSD hackathons which took place. Well, these are what caused the large number of changes.
Why did you remove sendmail?
Christos Zoulas: Sendmail has been, is, and will be a security accident waiting to happen (unless it is rewritten from the ground up with security consciousness). Performing character pointer gymnastics in 50-100 line loops does not create any warm and fuzzy feelings for me. To top this off, most sendmail security issues are marked as confidential, and we are prevented from fixing or mentioning the problem until the ban is lifted. The last time this happened, we said "enough" and removed it altogether before the ban for that particular security issue was lifted
Would you like to present kauth, the kernel authorization framework, included in this release?
Elad Efrat: Kernel authorization is really something cool that I'm happy many people will get a chance to test in this release.
Basically, the story goes like this: the traditional Unix security model, the one where root is almighty and everyone else is not, is slowly beginning to show signs of age as demands for finer-grained security policies arise. The problem becomes more apparent when you look at the kernel code and see that, probably as time went by, people used various ways to check whether a certain operation is allowed to be performed or not—most are variants of checking whether the effective user-id is 0 or using the suser() function—effectively embedding the security model to the kernel, in a way that the question "Can this user open a raw socket?" really became "Is this user root?"
Kauth(9) was originally designed by Apple, and it provides a dispatching model that allows us to really ask if a certain operation can be performed by specifying it along with the relevant context. This abstracts the security model from the various kernel subsystems. The system is divided to "scopes" that collect actions of the same nature (right now, NetBSD has scopes like "process", "network","machdep", etc.), and each scope has a set of "actions" that define the operation (the context provided depends on the operation). When the kernel wants to ask if a certain operation is allowed, it calls the authorization wrapper for the relevant scope, providing it the action and context, and receives back a binary response of "yes" or "no".
This upper layer, like I said, effectively abstracts the way these decisions are made, allowing one to "plug" (almost) any security model one can think of.
These security models are implemented by attaching "listeners" (kauth(9) terminology for callback functions) that receive the action and context and make the decision. In the future it is hoped that these listeners will be able to run on an entirely different machine, thus allowing a centralized security policy for an entire network to be controlled from a single host.
Unfortunately, in NetBSD 4.0, although the security model abstraction is almost complete, there is still not full control over all operations. That is—in some places, the question is still "Is this user root?", so implementing different security models will not be complete. However, it will provide an indication of how well kauth(9) works and how users react to this new ability to develop custom security models (for example, classic uses like restricting raw networking, or binding to certain low ports, to a few users). I'm hoping to be surprised by our user base's creativity.
I highly recommend spending a few minutes reading kauth(9) and secmodel(9) in the current man pages. There's a little taste of what's in the plans for NetBSD 5.0—like credential inheritance control—with the ultimate goal being to provide system administrators with security policy control that requires either zero or very little effort.
PaX, fileassoc, and Veriexec
What security features have you added to mprotect(2) from PaX?
Elad Efrat: First, the PaX project, for those who aren't familiar with it, is responsible for all the modern exploit mitigation technologies. It's where stuff like W^X and ASLR (Address Space Layout Randomization) were born, along with many, many other cool features. One of them, which I initially ported to NetBSD, is PaX MPROTECT.
PaX MPROTECT can be thought of as "strict W^X". If in a normal system, a program starts with no memory pages that are both writable and executable, but those pages can still be created using mprotect(2), systems or programs that run with PaX MPROTECT are "immune" to attacks where protection on pages is modified, often by trashing arguments to mprotect(2).
paxctl(8) is a tool that enables PaX MPROTECT on a per-program basis, if you don't want to enable it globally. I'm afraid that not much testing was done on various architectures using this feature, so users should first experiment with it...
How does the new fileassoc KPI (Kernel Programming Interface) work?
Elad Efrat: I'll begin by describing what fileassoc(9) is.
Some (newer) filesystems allow on to attach metadata to files by using extended attributes (in addition to the usual ones—permissions, timestamps, etc.) and such. A common use for them may be, for example, to store ACLs (Access Control List) for the file that allow finer granularity on access control.
The problem is that these extended attributes are filesystem-dependent and either don't exist on most filesystems or are interfaced differently. This is where fileassoc(9) comes into play: it allows you to attach metadata to files in a filesystem-independent way while storing the information in fast-access kernel tables. The advantage is that it really is filesystem-independent, so there's the potential of adding features and such to filesystems that lack them (for example, again, ACLs). The disadvantages are that the metadata is not really "attached" to the file in any way, and must be loaded in some form—from a database file or such—to the kernel. So that the dependency becomes on the OS itself, but that's cool, because it's aimed for NetBSDsystems.
The KPI works by storing data in hash tables. Metadata is identified by a key (a C string) and is a stream of bytes. A kernel subsystem can then use the fileassoc(9) KPI to either attach, query, or remove metadata from files, by specifying the key.
The fileassoc(9) KPI is implemented in the VFS (Virtual File System) layer, identifying each file using its "file handle" (these are supposed to be unique). So pretty much any filesystem is supported. An example (and the only, at the moment) consumer of fileassoc(9) is Veriexec, where a database file holds information that is parsed by a userland tool, then fed to fileassoc(9) using a special device.
Plans to write a generic interface for communicating with fileassoc(9) have come up in the past, but since we're not yet convinced of the necessity, these are just plans at the moment. Developers interested in feeding data to fileassoc(9) should also implement their own special device stub—at least for now.
Is there any news about Veriexec?
Elad Efrat: As with NetBSD 4.0, I feel Veriexec has gotten to a point where it should probably be used by most of our users.
Veriexec is NetBSD's integrity subsystem, which, in short, can guarantee the integrity of the programs, configuration files, etc., on the system.
In addition to the tons of improvements in performance and stability, and the many features added, NetBSD 4.0 introduces 'veriexecgen', which is a tool written by Matt Fleming (mjf@). Veriexecgen tremendously lowers the bar for fingerprint database generation to a point where it's possible to run a single command with no arguments after installation to have Veriexec set-up appropriately. I strongly recommend reading the veriexecgen(8) main page and giving it a try.
What is veriexecgen and how does it work?
Matt Fleming: veriexecgen is a program that runs against directories and generates a set of fingerprints for use with veriexec. This fingerprint database is usually stored in /etc/signatures.
XFree86, pkgsrc, proplib, and Xen
Do you still include Linux binary compatibility?
Nicolas Joly: I'm currently working on improving Linux binary compatibility on AMD64 for both 32- and 64-bit applications.
The amd64 compat linux is the first one that include NPTL (NativePOSIX Thread Library) emulation found on Linux 2.6 kernel. Likewise, compat linux32 is pretty new, and even if mostly identical to i386linux emulation needs to be modified.
For NetBSD 4.0, kernel support for compat linux is not enabled bydefault as this is not as stable as other ports. For this release, do not expect to run complicated linux applications yet, but basic ones should work. In the mean time, -current has made some progresses...
Which X Window System is included in NetBSD 4.0?
Matthias Scheler: XFree86 4.5.0.
The XFree86 Project has unfortunately lost a lot of momentum. NetBSD is currently in the process of switching to the X.org X11 distribution. We initially tried integrating the monolithic X.org distribution into our xsrc source tree. But it never reached a state where it worked on all platforms and supported cross builds via build.sh.
After the X.org project changed to a modular distribution, it was obvious that pkgsrc is the best way to integrate X11 in the future. There is ongoing work mostly by Joerg Sonnenberger to make modular X11 in pkgsrc cross-buildable. When that work has been finished and properly integrated with the system installation, the XFree86 distribution in xsrc will probably be retired.
What's new in pkgsrc?
Joerg Sonnenberger: I'm only aware of NetBSD 4 being a requirement to use the cross-compiling support. The updates to pkg_install will be in NetBSD4.1, so that is ruled out.
The cross-compiling support allows a small subset of pkgsrc to be built for any architecture running NetBSD using the output of build.sh. Currently this subset includes modular Xorg and a few applications like Xpdf as proof of concept. This is intended to replace the aging XFree86in xsrc, but needs some more polishing work on the pkgsrc side.
What is proplib(3) and what can we use it for?
Jason Thorpe: proplib is a library for manipulating property lists. Property lists are collections of properties, typically stored in a dictionary. A dictionary is an associative array, essentially key-valuepairs. The keys are strings, and the values are strings, numbers, opaque data, booleans, arrays, dictionaries, ...
This is handy in a variety of applications. For example, communicating structured data between userspace and the kernel, describing properties of devices, etc.
What can we do with NetBSD/Xen?
Manuel Bouyer: Well, we can do a lot of things, from merging different physical boxes to a single one (if you need different OSes for different tasks, for example), to easily create test systems (it's very convenient for kernel developement: a guest boots much faster than a regular PC). With the virtualization features of recent X86 CPUs (which are supported by Xen3 with NetBSD as dom0), you can also boot plain i386 OSes. This can be used to test install media, for example, or to load systems that can't be paravirtualized (e.g., Windows).
Filesystems
What is the status of LFS (Log-structured File System)?
Phil Nelson: LFS is in use on about 1/2 of the 23 nodes of the WWU build cluster. Granted, it is running an older 4.0-Beta kernel, but they have been running just fine for quite a while. Currently all the machines in the WWU build cluster have been up 78+ days and continuously building. They are all i386 machines.
Would you like to describe the features of tmpfs?
Julio M. Merino Vidal: tmpfs was born as a replacement for MFS—the memory file system included in all BSDs as far as I know—and as such it shares many functionality with it. It basically is an efficient, memory-based file system which means that it uses part of the system's virtual memory to store files in a way that is more space- and speed-efficient than MFS. It supports all of the features expected from a Unix-style filesystem, including hard links, symbolic links, devices, permissions, file flags and NFS exportability, but it currently does not support sparse files.
As opposed to MFS, tmpfs file systems can grow and shrink automatically depending on the available memory if they are configured as such; their memory consumption can also be upper-limited. And even though the end user cannot directly notice it, tmpfs's code is much simpler than MFS's one, which means that it is easier to audit, test and optimize.
As you just mentioned, tmpfs is a bit different than MFS, but why did MFS need a replacement in the first place?
Julio M. Merino Vidal: The main problem of MFS is that it performs poorly, uses too much memory, and the used memory cannot be reclaimed unless the file system is unmounted. To understand why this happens, we need to see how MFS works, and to see that, we first need to outline how FFS is designed.
FFS (or UFS), the traditional BSD file system, is conceptually split in two different layers. the upper layer handles all the logic from the file system, including the management of directories, the common file operations, the routines to map data to blocks, etc. The lower layer lays out the data on disk, allocating blocks and inodes as needed, etc. This seems like a very nice design, but in the end, it imposes restrictions on the way the lower layer operates (or at least that's the impression I gathered; I'm not too familiar on FFS's code).
MFS is just a replacement of FFS's lower layer to operate on virtual memory. The approach it takes is dead simple: it allocates a contiguous block of memory and treats that as if it were a disk, organizing the contents of such memory region as disk blocks. As you can expect, such approach has a lot of overhead because the file system uses an incorrect abstraction—disk blocks and cylinder groups—to manage memory.
tmpfs, on the other hand, is a much simpler filesystem which takes advantage of the fact that it is always memory-backed. It uses traditional structures, linked lists, arrays, memory pools and other sorts of data abstractions to represent the contents of the file system. These abstractions are easier to deal with and hence require less resources to manage.
It is also interesting to mention that tmpfs grew a custom regression testing suite that was also well received by developers. But the way it was created was suboptimal, which made me start the ATF (Automated Testing Framework) project this year. We'll see a better testing suite in NetBSD 5.0, or at least I hope so! Oh, and by the way, FreeBSD also imported tmpfs in its source tree.
What is the relationship between puffs, FUSE and ReFUSE?
Antti Kantee: puffs stands for Pass-to-Userspace Framework File System. It is an interface and a framework implementation for userspace filesystems on NetBSD. The userspace filesystem interface itself is heavily influenced by the kernel virtual filesystem interface. puffs can be thought to consist of two parts: the mechanism for transporting file system requests from the kernel to userspace and a userspace library, libpuffs, for interfacing with the kernel and writing the userspace filesystem implementation.
FUSE, or Filesystem in Userspace, is another API for implementing userspace filesystems. It is native to Linux, but has been widely ported to other operating systems such as FreeBSD and OpenSolaris. FUSE is considered the standard interface for implementing userspace filesystems with numerous filesystems readily available.
ReFUSE is the implementation of the FUSE API for NetBSD. It is implemented on top of puffs and completely done in userspace. Other non-Linux operating systems implement FUSE compatibility in the kernel, but we believe the ReFUSE approach is the right way to address the issue: export the kernel filesystem interface in the most natural way possible (puffs) and implement compatibility in userspace (ReFUSE).
A very important difference to note is API stability. Since the FUSE API is well-established, it is stable, and filesystems written for it will most likely continue to function for a long time. The puffs API is a "low-level" API which follows the NetBSD kernel virtual filesystem API fairly closely. As the virtual filesystem API on NetBSD evolves, so will the puffs API (and vice versa, if things go as planned). A benefit from this evolutionary incompatibility is the ability to use all the features provided by the kernel API at the best possible performance.
What is their status in NetBSD 4.0?
Antti Kantee: puffs is still under heavy development. The version of puffs found in 4.0 is a snapshot of what was in the tree at the time 4.0 was branched. This means that ReFUSE and therefore FUSE support is unfortunately not present. The most useful application for users is likely to be ssshfs, simple sshfs, which can be found in source form from the tree under src/share/examples/puffs/ssshfs. It implements sshfs functionality. This implementation was superceded by mount_psshfs(8) for NetBSD 5.0.
Technically there is no reason why support for FUSE could not be added to the NetBSD 4 branch. However, it requires backporting ReFUSE and some puffs features from the development branch and therefore a person with time and motivation to do the work, and, above all, test and support it.
Friday, February 8, 2008
Autosave script for Keynote
Apple Tech Forums
As many of you may already realize there is no autosave feature that is native to Keynote for the Mac; so after searching around for a few I found this little script that should do the trick for you. Good Luck!!!
Hunter
Tired to press cmd + S ?
to autosave iWork's documents
paste this script in the Script Editor
menu > File > Save As…
uncheck screen at startup
check stay in background
save this script as an application
Install it as one of the applications launched at boot_time from the Accounts PreferencePane (tab Login Items)
Every ten minutes, if Pages or Keynote is in use, the open documents (already saved once) would be saved.
if Numbers is in use, the frontmost document will be saved.
If I made no mistake, if the document was never saved the "Save" dialog will ask you to save it. (OK, it works)
--SCRIPT autosave4iWork
property minutesBetweenSaves : 10
on idle
my auto4PK("Pages")
my auto4PK("Keynote")
my auto4Numbers()
return minutesBetweenSaves * 60
end idle
on auto4PK(theApp)
tell application "System Events" to set theAppIsRunning to (name of processes) contains "theApp"
if theAppIsRunning = true then
tell application theApp
repeat with aDoc in every document
if path of aDoc exists then
if modified of aDoc then save aDoc
end if
end repeat
end tell
end if
end auto4PK
on auto4Numbers()
set theApp to "Numbers"
tell application "System Events" to set theAppIsRunning to (name of processes) contains theApp
if theAppIsRunning = true then
tell application theApp to activate
tell application "System Events" to tell application process theApp to keystroke "s" using {command down}
end if
end auto4Numbers
--SCRIPT
As many of you may already realize there is no autosave feature that is native to Keynote for the Mac; so after searching around for a few I found this little script that should do the trick for you. Good Luck!!!
Hunter
Tired to press cmd + S ?
to autosave iWork's documents
paste this script in the Script Editor
menu > File > Save As…
uncheck screen at startup
check stay in background
save this script as an application
Install it as one of the applications launched at boot_time from the Accounts PreferencePane (tab Login Items)
Every ten minutes, if Pages or Keynote is in use, the open documents (already saved once) would be saved.
if Numbers is in use, the frontmost document will be saved.
If I made no mistake, if the document was never saved the "Save" dialog will ask you to save it. (OK, it works)
--SCRIPT autosave4iWork
property minutesBetweenSaves : 10
on idle
my auto4PK("Pages")
my auto4PK("Keynote")
my auto4Numbers()
return minutesBetweenSaves * 60
end idle
on auto4PK(theApp)
tell application "System Events" to set theAppIsRunning to (name of processes) contains "theApp"
if theAppIsRunning = true then
tell application theApp
repeat with aDoc in every document
if path of aDoc exists then
if modified of aDoc then save aDoc
end if
end repeat
end tell
end if
end auto4PK
on auto4Numbers()
set theApp to "Numbers"
tell application "System Events" to set theAppIsRunning to (name of processes) contains theApp
if theAppIsRunning = true then
tell application theApp to activate
tell application "System Events" to tell application process theApp to keystroke "s" using {command down}
end if
end auto4Numbers
--SCRIPT
Thursday, February 7, 2008
Three photo mosaic apps compared
By Nathan Willis on February 04, 2008 (4:00:00 PM)
Photo mosaics are recreations of one large image composed of tiny tiles of other smaller images. They can be a fun project and make good use of the hundreds of less-than-extraordinary photos on your hard drive. We compared three easy-to-use Linux-based utilities for generating photo mosaics -- Pixelize, Metapixel, and Imosaic -- on speed, quality, and other factors.
Pixelize
Pixelize is the oldest of the three apps; it uses GTK1 for its interface, which may make it stand out from your other desktop applications. You won't be able to use your favorite file selection widget, either, which can be frustrating -- the GTK1 file selector does not remember the last directory you visited, and it does not provide thumbnail previews of image files.
Pixelize in action
You can download source code from the project's Web page, but check your Linux distribution's package management system first -- many distros include Pixelize. The latest version is 0.9.2, and does not have any unusual package prerequisites.
The package includes two programs: the GUI front end pixelize, and the command-line back end make_db. You must run make_db at least once before you run pixelize in order to populate the database of source images from which the app will tile your mosaic. Just run make_db somedirectory/*, and make_db will index all of the image files within somedirectory. The utility cannot descend into other directories recursively, so you will have to execute make_db multiple times if you store your images in multiple locations.
Once you are ready, launch the GUI with pixelize &. Up will pop an empty window; choose File -> Open to select the picture from which you want to create a mosaic.
Choose Options -> Options to adjust the program's parameters. You can specify either the number of images to use in each row and column of the mosaic, or the size to make each mosaic tile -- changing either factor automatically adjusts the other. The only other option to worry about is the proximity of duplicates. If you have a very small set of source images, Pixelize will have to use each several times; this parameter allows you to space them further apart to make them less visible.
Once you are ready, choose Options -> Render, and Pixelize will generate your mosaic, drawing it on top of the original image in the main window. If you like what you see, choose File -> Save.
Metapixel
Metapixel is a command-line-only tool, but it offers more flexibility in mosaic creation than does Pixelize. The latest release is version 1.0.2, which you can download from the project's site. Source code as well as Fedora RPMs are available. But as with Pixelize, many modern Linux distros ship Metapixel, so see if you can use an official package if you are not interested in compiling your own binary. Metapixel requires Perl for its image preparation step.
Before you can create a mosaic, you must run metapixel-prepare to prepare a batch of source images. The complete syntax is metapixel-prepare --width=n --height=m --recurse sourcedirectory destinationdirectory. This will scan through sourcedirectory (recursively), and generate n-by-m tile images for each image it finds, saving them to destinationdirectory. The --recurse flag is optional, as are --width and --height; without them metapixel-prepare will create 128x128 tiles.
You can create a basic mosaic with metapixel --library=destinationdirectory --metapixel inputimage.jpg outputimage.jpg. The --library flag should designate a directory of tiles preprocessed by metapixel-prepare; the ability to maintain multiple such directories allows you more flexibility in building your mosaics. Metapixel can generate JPEG or PNG output, depending on the output file name you provide.
Metapixel has several optional flags you can use to tweak the output. You can scale the output file in relation to the input with --scale=x, or assign different relative weights to the different color channels in the pattern-matching step. Collage mode (invoked with --collage) generates a mosaic where the tiles overlap each other, rather than being in strict rows and columns. There are even two different algorithms for determining which tile best matches which part of the mosaic, so you can experiment and decide what produces the best results for your input and tile collection.
Imosaic
IMosaic in action
Imosaic is the only closed source app in this mix. It is a .Net program that the author has also packaged for Linux and Mac OS X to run using Mono. You can download the latest package from imosaic.net. Unpack the archive anywhere on your system and launch the app with mono IMosaic.exe &.
I had trouble getting the software to run using Ubuntu 7.10's Mono; even with all of the available mono* packages, Imosaic refused to run. I was able to get it working by downloading a vanilla Mono installer from the Mono Web site; luckily you can have more than one copy of Mono installed at any time.
Once you launch Imosaic, the first step to creating a mosaic is defining a collection of tile images. Choose Edit -> Images Collection from the menu, and in the Image Collection Editor, add as many files and directories as you need. To use your collection, you must explicitly choose Task -> Process images to prepare the tiles, then explicitly save the collection via File -> Save Collection. When you have a collection defined, you add it to the mosaic creation process under Imosaic's Image's Collections [sic] tab.
You select the image you want to transform into a mosaic with File -> Open Source Image. By default Imosaic displays grid lines over the image to show you where the mosaic tiles will fall, which is helpful. You can resize the tile dimensions in the Settings tab, as well as adjust factors like JPEG output quality and distance between duplicate tiles. When you are ready to convert, click the Process button and wait. The Statistics tab provides help info on the mosaic creation process.
The Linux and OS X port of Imosaic is recent, and that is reflected in UI quirks that take some getting used to. For instance, Windows naming conventions are used throughout the interface, such as My Computer and Personal -- the latter of which refers to your home folder. I also encountered a lot of sudden crashes when trying to build image collections; after every three or four "Add directory" steps the app would blink out of existence, erasing all changes I had made to my in-progress collection. Until the developer fixes the problem, consider yourself warned and save those collections often.
Triptych
Side by side, it is hard to pick a best program from among the three apps. Metapixel offers far more mosaic creation options, but its lack of a GUI means you have to use trial and error to get a good feel for what those options do. I like how simple Pixelize is and how you can see your mosaic being generated as it happens on screen, but GTK1 is ugly and almost unusable nowadays.
Imosaic bests both of the others in a few important areas. It is the only program that integrates preprocessing an image collection (or even informs you in the interface that you need to preprocess a collection), and is the only one that allows you to cancel mosaic creation mid-process. But the Linux build is so buggy for now that it is hard to build a large enough collection to produce good output, and several of its key features (e.g. the Sequences tab) remain undocumented and are thus of unknown value.
Left to right: output from Pixelize, Metapixel, and IMosaic
As far as output quality goes, I was happiest with Metapixel's results, using the --metric=wavelet and --search=global options. Pixelize tended to produce desaturated, nearly monochrome mosaics -- and I have thousands of color images, so the problem is likely with the program's algorithm. Imosaic produced respectable results, but the frequent crashes made it hard to build a large image collection with the variety needed to generate large mosaics.
A good GUI for Metapixel and some bug fixes for Imosaic would put them on approximately equal footing.
Finally, if you take an interest in photo mosaics, you might be interested to learn of the patent situation. Several patents on creating photo mosaics were granted to US-based Runaway Technology, beginning in 2000, despite apparent evidence of prior art dating back at least as far as 1993. It is unclear if the company has successfully defended its patents, but if your dreams include monetizing your mosaics, the inventions claimed in the patent are something you should look into.
Photo mosaics are recreations of one large image composed of tiny tiles of other smaller images. They can be a fun project and make good use of the hundreds of less-than-extraordinary photos on your hard drive. We compared three easy-to-use Linux-based utilities for generating photo mosaics -- Pixelize, Metapixel, and Imosaic -- on speed, quality, and other factors.
Pixelize
Pixelize is the oldest of the three apps; it uses GTK1 for its interface, which may make it stand out from your other desktop applications. You won't be able to use your favorite file selection widget, either, which can be frustrating -- the GTK1 file selector does not remember the last directory you visited, and it does not provide thumbnail previews of image files.
Pixelize in action
You can download source code from the project's Web page, but check your Linux distribution's package management system first -- many distros include Pixelize. The latest version is 0.9.2, and does not have any unusual package prerequisites.
The package includes two programs: the GUI front end pixelize, and the command-line back end make_db. You must run make_db at least once before you run pixelize in order to populate the database of source images from which the app will tile your mosaic. Just run make_db somedirectory/*, and make_db will index all of the image files within somedirectory. The utility cannot descend into other directories recursively, so you will have to execute make_db multiple times if you store your images in multiple locations.
Once you are ready, launch the GUI with pixelize &. Up will pop an empty window; choose File -> Open to select the picture from which you want to create a mosaic.
Choose Options -> Options to adjust the program's parameters. You can specify either the number of images to use in each row and column of the mosaic, or the size to make each mosaic tile -- changing either factor automatically adjusts the other. The only other option to worry about is the proximity of duplicates. If you have a very small set of source images, Pixelize will have to use each several times; this parameter allows you to space them further apart to make them less visible.
Once you are ready, choose Options -> Render, and Pixelize will generate your mosaic, drawing it on top of the original image in the main window. If you like what you see, choose File -> Save.
Metapixel
Metapixel is a command-line-only tool, but it offers more flexibility in mosaic creation than does Pixelize. The latest release is version 1.0.2, which you can download from the project's site. Source code as well as Fedora RPMs are available. But as with Pixelize, many modern Linux distros ship Metapixel, so see if you can use an official package if you are not interested in compiling your own binary. Metapixel requires Perl for its image preparation step.
Before you can create a mosaic, you must run metapixel-prepare to prepare a batch of source images. The complete syntax is metapixel-prepare --width=n --height=m --recurse sourcedirectory destinationdirectory. This will scan through sourcedirectory (recursively), and generate n-by-m tile images for each image it finds, saving them to destinationdirectory. The --recurse flag is optional, as are --width and --height; without them metapixel-prepare will create 128x128 tiles.
You can create a basic mosaic with metapixel --library=destinationdirectory --metapixel inputimage.jpg outputimage.jpg. The --library flag should designate a directory of tiles preprocessed by metapixel-prepare; the ability to maintain multiple such directories allows you more flexibility in building your mosaics. Metapixel can generate JPEG or PNG output, depending on the output file name you provide.
Metapixel has several optional flags you can use to tweak the output. You can scale the output file in relation to the input with --scale=x, or assign different relative weights to the different color channels in the pattern-matching step. Collage mode (invoked with --collage) generates a mosaic where the tiles overlap each other, rather than being in strict rows and columns. There are even two different algorithms for determining which tile best matches which part of the mosaic, so you can experiment and decide what produces the best results for your input and tile collection.
Imosaic
IMosaic in action
Imosaic is the only closed source app in this mix. It is a .Net program that the author has also packaged for Linux and Mac OS X to run using Mono. You can download the latest package from imosaic.net. Unpack the archive anywhere on your system and launch the app with mono IMosaic.exe &.
I had trouble getting the software to run using Ubuntu 7.10's Mono; even with all of the available mono* packages, Imosaic refused to run. I was able to get it working by downloading a vanilla Mono installer from the Mono Web site; luckily you can have more than one copy of Mono installed at any time.
Once you launch Imosaic, the first step to creating a mosaic is defining a collection of tile images. Choose Edit -> Images Collection from the menu, and in the Image Collection Editor, add as many files and directories as you need. To use your collection, you must explicitly choose Task -> Process images to prepare the tiles, then explicitly save the collection via File -> Save Collection. When you have a collection defined, you add it to the mosaic creation process under Imosaic's Image's Collections [sic] tab.
You select the image you want to transform into a mosaic with File -> Open Source Image. By default Imosaic displays grid lines over the image to show you where the mosaic tiles will fall, which is helpful. You can resize the tile dimensions in the Settings tab, as well as adjust factors like JPEG output quality and distance between duplicate tiles. When you are ready to convert, click the Process button and wait. The Statistics tab provides help info on the mosaic creation process.
The Linux and OS X port of Imosaic is recent, and that is reflected in UI quirks that take some getting used to. For instance, Windows naming conventions are used throughout the interface, such as My Computer and Personal -- the latter of which refers to your home folder. I also encountered a lot of sudden crashes when trying to build image collections; after every three or four "Add directory" steps the app would blink out of existence, erasing all changes I had made to my in-progress collection. Until the developer fixes the problem, consider yourself warned and save those collections often.
Triptych
Side by side, it is hard to pick a best program from among the three apps. Metapixel offers far more mosaic creation options, but its lack of a GUI means you have to use trial and error to get a good feel for what those options do. I like how simple Pixelize is and how you can see your mosaic being generated as it happens on screen, but GTK1 is ugly and almost unusable nowadays.
Imosaic bests both of the others in a few important areas. It is the only program that integrates preprocessing an image collection (or even informs you in the interface that you need to preprocess a collection), and is the only one that allows you to cancel mosaic creation mid-process. But the Linux build is so buggy for now that it is hard to build a large enough collection to produce good output, and several of its key features (e.g. the Sequences tab) remain undocumented and are thus of unknown value.
Left to right: output from Pixelize, Metapixel, and IMosaic
As far as output quality goes, I was happiest with Metapixel's results, using the --metric=wavelet and --search=global options. Pixelize tended to produce desaturated, nearly monochrome mosaics -- and I have thousands of color images, so the problem is likely with the program's algorithm. Imosaic produced respectable results, but the frequent crashes made it hard to build a large image collection with the variety needed to generate large mosaics.
A good GUI for Metapixel and some bug fixes for Imosaic would put them on approximately equal footing.
Finally, if you take an interest in photo mosaics, you might be interested to learn of the patent situation. Several patents on creating photo mosaics were granted to US-based Runaway Technology, beginning in 2000, despite apparent evidence of prior art dating back at least as far as 1993. It is unclear if the company has successfully defended its patents, but if your dreams include monetizing your mosaics, the inventions claimed in the patent are something you should look into.
Sunday, February 3, 2008
Efficient rsyncrypto hides remote sync data
By Ben Martin on February 01, 2008 (9:00:00 AM)
The rsync utility is smart enough to send only enough bytes of a changed file to a remote system to enable the remote file to become identical to the local file. When that information is sensitive, using rsync over SSH protects files while in transit.To protect the files when they are on the server you might first encrypt them with GPG. But the manner in which GPG encrypts slightly changed files foils rsync's efficiency.rsyncrypto allows you to encrypt your files while still allowing you to leverage the speed of rsync.
One of the aims of encryption is to try to make any change to the unencrypted file completely modify how the encrypted file appears. This means that somebody who has access to a series of encrypted files gets little information about how the little changes you might make to the unencrypted file are affecting the encrypted file over time. The downside is that such security causes the encryption program to modify most of the encrypted file. If you then use rsync to copy such a file to the remote server, it will have to send almost the complete file to the remote server each time.
The goal of rsyncrypto is to encrypt files in such a manner that only a slight and controlled amount of security is sacrificed in order to make rsync able to send the encrypted files much quicker. It aims to leak no more than 20 bits of aggregated information per 8KB of plaintext file.
For an example of an information leak, suppose you have an XML file and you use rsyncrypto to copy the file to a remote host. Then you change a single XML attribute and use rsyncrypto to copy the updates across. Now suppose an attacker captured the encrypted versions in transit, and thus has copies of both the encrypted file before the change and after the change. The first thing they learn is that only the first 8KB of the file changed, because that is all that was sent the second time. If they can speculate what sort of file the unencrypted file was (for example, an XML file) then they can try to use that guess in an attempt to recover information.
Rsyncrypto encrypts parts of the file independently, thus keeping any changes you make to a single block of the file local to that block in the encrypted version. If you're protecting a collection of personal files from a possible remote system compromise, such a tradeoff in security might be acceptable. On the other hand, if you cannot allow any information leaks, then you'll have to accept that the whole encrypted file will change radically each time you change the unencrypted file. If that's the case, using rsync on GnuPG-encrypted files might suit your needs.
On a Fedora 8 machine, you have to download both rsyncrypto and the dependency argtable2 and install them using the standard ./configure; make; make install combination, starting with argtable2.
rsyncrypto is designed to be used as a presync option to rsync. That is, you first use rsyncrypto on the plain unencrypted files to obtain an encrypted directory tree, and you then run rsync to send that encrypted tree to a remote system. The following command syntax shows a template for directory encryption and decryption:
# to encrypt rsyncrypto -r srcdir /tmp/encrypted srcdir.keys mykey.crt # to decrypt rsyncrypto -d -r /tmp/encrypted srcdir srcdir.keys mykey.crt
The keys and certificates referenced in these commands are generated by OpenSSL, as we'll see in a moment. In the commands, the srcdir is encrypted and sent to /tmp/encrypted with the individual keys used to encrypt each file in srcdir saved into srcdir.keys. The mykey.crt is a certificate that is used to protect all the keys in srcdir.keys. If you still have all the keys, you can use your certificate in the decryption operation to obtain the plaintext files again. If you lose srcdir.keys, all is not lost, but you must use the private key for mykey.crt to regain the encrypted keys that are also stored in /tmp/encrypted.
The following is a full one-way sync to a remote server using both rsyncrypto and rsync to obtain an encrypted backup on a remote machine. The example first generates a master key and certificate using OpenSSL, then makes an encrypted backup of ~/foodir onto the remote machine v8tsrv:
$ mkdir ~/rsyncrypto-keys $ cd ~/rsyncrypto-keys $ openssl req -nodes -newkey rsa:1536 -x509 -keyout rckey.key -out rckey.crt $ cd ~ $ mkdir foodir $ date >foodir/df1.txt $ date >foodir/df2.txt $ rsyncrypto -r foodir /tmp/encrypted foodir.keys ~/rsyncrypto-keys/rckey.crt $ rsync -av /tmp/encrypted ben@v8tsrv:~
In order to test the speed gain of using rsyncrypto as opposed to using other encryption with rsync, I used /dev/urandom to create a file of random bytes, encrypted it with both rsyncrypto and GnuPG, and rsynced both of these to a remote system using rsync. I then modified the plaintext file, encrypted the file again, and synced the encrypted file with the remote system. In this case, I modified 6KB of data at an offset of 17KB into the file using dd and left all the other data intact. The final rsync commands show that the rsyncrypto-encrypted tree only needed to send 58,102 bytes, whereas the GnuPG-encrypted file required the entire file to be sent to the remote system:
$ cd ~/foodir $ rm -f * $ dd if=/dev/urandom of=testfile.random bs=1024 count=500 512000 bytes (512 kB) copied, 0.088045 s, 5.8 MB/s $ cd ~ $ rsyncrypto -r foodir foodir.rcrypto foodir.keys ~/rsyncrypto-keys/rckey.crt $ ls -l foodir.rcrypto/ -rw-r--r-- ... 502K 2008-01-08 19:59 testfile.random $ mkdir foodir.gpg $ gpg --gen-key ... $ mkdir foodir.gpg $ gpg --output foodir.gpg/testfile.random.gpg -e foodir/testfile.random $ ls -l foodir.gpg -rw-r--r-- 1 ... 501K 2008-01-08 20:07 testfile.random.gpg $ rsync -av foodir.rcrypto ben@v8tsrv:~ sent 513356 bytes received 48 bytes 342269.33 bytes/sec $ rsync -av foodir.gpg ben@v8tsrv:~ sent 513026 bytes received 48 bytes 342049.33 bytes/sec # # modify the input file starting at 17KB into the file for 6KB # $ dd if=/dev/urandom of=~/foodir/testfile.random bs=1024 count=6 seek=17 conv=notrunc $ ls -l testfile.random -rw-r--r-- 1 ... 500K 2008-01-08 20:17 testfile.random $ rsyncrypto -r foodir foodir.rcrypto foodir.keys ~/rsyncrypto-keys/rckey.crt $ gpg --output foodir.gpg/testfile.random.gpg -e foodir/testfile.random # # See how much gets sent # $ rsync -av foodir.rcrypto ben@v8tsrv:~ sent 58102 bytes received 4368 bytes 124940.00 bytes/sec $ rsync -av foodir.gpg ben@v8tsrv:~ sent 513024 bytes received 4368 bytes 1034784.00 bytes/sec
Using rsyncrypto with rsync, you can protect the files that you send to a remote system while allowing modified files to be sent in a bandwidth-efficient manner. There is a slight loss of security using rsyncrypto, because changes to the unencrypted file do not propagate throughout the entire encrypted file. When this security trade-off is acceptable, you can get much quicker bandwidth-friendly network syncs and still achieve good encryption on the files stored on the remote server.
If you wish to hide file names as well as their content on the remote server, you can use the --name-encrypt=map option to rsyncrypto, which stores a mapping from the original file name to a garbled random file name in a mapping file, and outputs files using only their garbled random file name in the encrypted directory tree.
Ben Martin has been working on filesystems for more than 10 years. He completed his Ph.D. and now offers consulting services focused on libferris, filesystems, and search solutions.
The rsync utility is smart enough to send only enough bytes of a changed file to a remote system to enable the remote file to become identical to the local file. When that information is sensitive, using rsync over SSH protects files while in transit.To protect the files when they are on the server you might first encrypt them with GPG. But the manner in which GPG encrypts slightly changed files foils rsync's efficiency.rsyncrypto allows you to encrypt your files while still allowing you to leverage the speed of rsync.
One of the aims of encryption is to try to make any change to the unencrypted file completely modify how the encrypted file appears. This means that somebody who has access to a series of encrypted files gets little information about how the little changes you might make to the unencrypted file are affecting the encrypted file over time. The downside is that such security causes the encryption program to modify most of the encrypted file. If you then use rsync to copy such a file to the remote server, it will have to send almost the complete file to the remote server each time.
The goal of rsyncrypto is to encrypt files in such a manner that only a slight and controlled amount of security is sacrificed in order to make rsync able to send the encrypted files much quicker. It aims to leak no more than 20 bits of aggregated information per 8KB of plaintext file.
For an example of an information leak, suppose you have an XML file and you use rsyncrypto to copy the file to a remote host. Then you change a single XML attribute and use rsyncrypto to copy the updates across. Now suppose an attacker captured the encrypted versions in transit, and thus has copies of both the encrypted file before the change and after the change. The first thing they learn is that only the first 8KB of the file changed, because that is all that was sent the second time. If they can speculate what sort of file the unencrypted file was (for example, an XML file) then they can try to use that guess in an attempt to recover information.
Rsyncrypto encrypts parts of the file independently, thus keeping any changes you make to a single block of the file local to that block in the encrypted version. If you're protecting a collection of personal files from a possible remote system compromise, such a tradeoff in security might be acceptable. On the other hand, if you cannot allow any information leaks, then you'll have to accept that the whole encrypted file will change radically each time you change the unencrypted file. If that's the case, using rsync on GnuPG-encrypted files might suit your needs.
On a Fedora 8 machine, you have to download both rsyncrypto and the dependency argtable2 and install them using the standard ./configure; make; make install combination, starting with argtable2.
rsyncrypto is designed to be used as a presync option to rsync. That is, you first use rsyncrypto on the plain unencrypted files to obtain an encrypted directory tree, and you then run rsync to send that encrypted tree to a remote system. The following command syntax shows a template for directory encryption and decryption:
# to encrypt rsyncrypto -r srcdir /tmp/encrypted srcdir.keys mykey.crt # to decrypt rsyncrypto -d -r /tmp/encrypted srcdir srcdir.keys mykey.crt
The keys and certificates referenced in these commands are generated by OpenSSL, as we'll see in a moment. In the commands, the srcdir is encrypted and sent to /tmp/encrypted with the individual keys used to encrypt each file in srcdir saved into srcdir.keys. The mykey.crt is a certificate that is used to protect all the keys in srcdir.keys. If you still have all the keys, you can use your certificate in the decryption operation to obtain the plaintext files again. If you lose srcdir.keys, all is not lost, but you must use the private key for mykey.crt to regain the encrypted keys that are also stored in /tmp/encrypted.
The following is a full one-way sync to a remote server using both rsyncrypto and rsync to obtain an encrypted backup on a remote machine. The example first generates a master key and certificate using OpenSSL, then makes an encrypted backup of ~/foodir onto the remote machine v8tsrv:
$ mkdir ~/rsyncrypto-keys $ cd ~/rsyncrypto-keys $ openssl req -nodes -newkey rsa:1536 -x509 -keyout rckey.key -out rckey.crt $ cd ~ $ mkdir foodir $ date >foodir/df1.txt $ date >foodir/df2.txt $ rsyncrypto -r foodir /tmp/encrypted foodir.keys ~/rsyncrypto-keys/rckey.crt $ rsync -av /tmp/encrypted ben@v8tsrv:~
In order to test the speed gain of using rsyncrypto as opposed to using other encryption with rsync, I used /dev/urandom to create a file of random bytes, encrypted it with both rsyncrypto and GnuPG, and rsynced both of these to a remote system using rsync. I then modified the plaintext file, encrypted the file again, and synced the encrypted file with the remote system. In this case, I modified 6KB of data at an offset of 17KB into the file using dd and left all the other data intact. The final rsync commands show that the rsyncrypto-encrypted tree only needed to send 58,102 bytes, whereas the GnuPG-encrypted file required the entire file to be sent to the remote system:
$ cd ~/foodir $ rm -f * $ dd if=/dev/urandom of=testfile.random bs=1024 count=500 512000 bytes (512 kB) copied, 0.088045 s, 5.8 MB/s $ cd ~ $ rsyncrypto -r foodir foodir.rcrypto foodir.keys ~/rsyncrypto-keys/rckey.crt $ ls -l foodir.rcrypto/ -rw-r--r-- ... 502K 2008-01-08 19:59 testfile.random $ mkdir foodir.gpg $ gpg --gen-key ... $ mkdir foodir.gpg $ gpg --output foodir.gpg/testfile.random.gpg -e foodir/testfile.random $ ls -l foodir.gpg -rw-r--r-- 1 ... 501K 2008-01-08 20:07 testfile.random.gpg $ rsync -av foodir.rcrypto ben@v8tsrv:~ sent 513356 bytes received 48 bytes 342269.33 bytes/sec $ rsync -av foodir.gpg ben@v8tsrv:~ sent 513026 bytes received 48 bytes 342049.33 bytes/sec # # modify the input file starting at 17KB into the file for 6KB # $ dd if=/dev/urandom of=~/foodir/testfile.random bs=1024 count=6 seek=17 conv=notrunc $ ls -l testfile.random -rw-r--r-- 1 ... 500K 2008-01-08 20:17 testfile.random $ rsyncrypto -r foodir foodir.rcrypto foodir.keys ~/rsyncrypto-keys/rckey.crt $ gpg --output foodir.gpg/testfile.random.gpg -e foodir/testfile.random # # See how much gets sent # $ rsync -av foodir.rcrypto ben@v8tsrv:~ sent 58102 bytes received 4368 bytes 124940.00 bytes/sec $ rsync -av foodir.gpg ben@v8tsrv:~ sent 513024 bytes received 4368 bytes 1034784.00 bytes/sec
Using rsyncrypto with rsync, you can protect the files that you send to a remote system while allowing modified files to be sent in a bandwidth-efficient manner. There is a slight loss of security using rsyncrypto, because changes to the unencrypted file do not propagate throughout the entire encrypted file. When this security trade-off is acceptable, you can get much quicker bandwidth-friendly network syncs and still achieve good encryption on the files stored on the remote server.
If you wish to hide file names as well as their content on the remote server, you can use the --name-encrypt=map option to rsyncrypto, which stores a mapping from the original file name to a garbled random file name in a mapping file, and outputs files using only their garbled random file name in the encrypted directory tree.
Ben Martin has been working on filesystems for more than 10 years. He completed his Ph.D. and now offers consulting services focused on libferris, filesystems, and search solutions.
Is server or client processing better for charts and graphs?
By Colin Beckingham on February 01, 2008 (4:00:00 PM)
Webmasters are frequently required to serve up charts and graphs to clients. Part of the planning for such images involves a decision about whether to process the chart on the server or at the client end. Of course, it depends on the circumstances. There are costs and benefits to both approaches.
The generation of a chart at the server involves the creation of an image such as a .png file and then displaying this file as part of the delivered page. Prior to the image creation a script must set up the data points and the axis labels, switch on colours, create a legend, size the picture, and send it out.
Client-side processing sends the parameters and data points for processing by the client, often with JavaScript. However, actual delivery of the page involves sending not only the data and parameters, but also the required JavaScript and cascading stylesheet libraries to draw the final chart. The following table compares the two approaches.
Summary Comparison Server side Client side
Advantages
* Data remains confidential
* Chart can be copied and pasted as a separate unit
* If multiple copies of the chart are required, the image can be easily repeated
* Browser may be able to resize the image
* Images load progressively, visibly in the browser
* Data is available locally for reprocessing
* Once the .js and .css libraries are downloaded they can be re-used with no further download overhead
Disadvantages
* Identical data cannot be reprocessed without refreshing the page
* Heavier load on the server processor
* Server side image storage required
* Multiple identical images need to be separately drawn
* .js and .css libraries need to be part of the download package
* JavaScript must be available and activated on the client
* Taking a copy of a chart requires a screenshot
* Libraries load without screen activity, giving the appearance of slow loading
The open source world offers class libraries to provide either or both solutions. An example of a simple server-side library is libchart, and a client side library is Webfx Charting, both of which can be incorporated into a PHP script. Both projects provide samples of library output and example code. Each method probably requires approximately the same amount of coding effort. This article is not intended to compare the output of the two libraries, just to examine the relative advantages and disadvantages of the two approaches. Additionally, there are other methods such as Java which are not considered here.
The choice for webmasters comes down to two major components: the actual processing time, and the bandwidth required to deliver the final product. Generally the bandwidth component is easier to quantify. Some server managers place official quotas on bandwidth, but leave unofficial honour system quotas on processing time, reserving the right to reduce service to webmasters who place unfair burdens on processing time on shared servers. It is in the interests of webmasters to keep the server managers happy on both counts.
Download size
Focusing on the code necessary to draw the chart on the client machine, in the server side situation the code can be very short:
In the client side situation the minimum might be:
That's more than 800 characters, not counting spaces or data, about 450 of which is repeated for each chart to be displayed. Compared with 22 characters for the server side, that's not so trivial.
When it comes to data, the client-side is expecting an array of numbers. In the case that the array is short and consists of simple integers, the array is short and almost trivial. Now consider that the data might be an array of double-precision numbers such as [123456.7890,123456.7889,...,2.6]. A webmaster might be able to round down to integers before transmission, but this would lose some precision, which could be a problem when the data is to be re-used at the client end. Two hundred data points at 11 characters per number each plus the separation character gives an array character length of about 2,400.
In the case of one single chart we can expect something like:
Server side (bytes) Client side (bytes)
Basic html 22 Basic html 400
Libraries 0 Libraries (loaded once per page, if not cached) 66,000
Image 25,000 Image code 450
Data 0 Data (est., per image) 2,400
Total 25,022 Total 69,250
So the download for the client side will probably be larger for one image.
Now consider the case of five different charts on the same page:
Server side (bytes) Client side (bytes)
Basic html 100 Basic html 400
Libraries 0 Libraries (loaded once per page, if not cached) 66,000
Image 125,000 Image code 2,250
Data 0 Data (est., 5 images) 12,000
Total 125,100 Total 80,650
In the case of five charts, the client side download package is smaller. Clearly the issue of the size of the download depends entirely on the number of charts. While these numbers may appear insignificant, when a page is served up many thousands of times, the total becomes a threat to your bandwidth limit.
CPU load
Next we need to consider the issue of CPU load. The server-side class library uses the freetype and gd libraries. Some server managers consider these libraries to be processor-intensive, a critical issue in a shared server environment. You would expect that server-side processing would create a greater CPU load, but is that really so?
Using the top utility on a standalone single-processor server (2.8GHz Intel with hyperthreading and 2GB of RAM) I had the following experience. I set up a PHP script to generate a chart from 200 random data points using MySQL. One script output the charts as three separate 28KB .png images and the other drew three charts with JavaScript. Some of the CPU load can be attributed to MySQL, but since the database processing for each is the same, this should cancel out.
Results GD -> .png JavaScript
Page load seconds (*dialup) 15 3
CPU load Average increase of 10% for ~5 seconds Average increase of 3% for ~3 seconds
Download size (Source code + Other) 28K * 3 = 84K 8K + 66K = 74K
Conclusion
In summary, charting and graphing are important and informative assets. After all, a picture can be worth a thousand words. However, in a Web context, we need to balance whether it is better to send charts and graphs as pictures or the thousand words. Providing the data and then using JavaScript locally to draw the results not only gives a faster page load to the end user, but reduces the load on the server CPU, in this case by a factor of about three.
Colin Beckingham is a freelance programmer and writer in Eastern Ontario. He is currently working on a project that logs and charts the operation of a biomass burner using open source resources.
Webmasters are frequently required to serve up charts and graphs to clients. Part of the planning for such images involves a decision about whether to process the chart on the server or at the client end. Of course, it depends on the circumstances. There are costs and benefits to both approaches.
The generation of a chart at the server involves the creation of an image such as a .png file and then displaying this file as part of the delivered page. Prior to the image creation a script must set up the data points and the axis labels, switch on colours, create a legend, size the picture, and send it out.
Client-side processing sends the parameters and data points for processing by the client, often with JavaScript. However, actual delivery of the page involves sending not only the data and parameters, but also the required JavaScript and cascading stylesheet libraries to draw the final chart. The following table compares the two approaches.
Summary Comparison Server side Client side
Advantages
* Data remains confidential
* Chart can be copied and pasted as a separate unit
* If multiple copies of the chart are required, the image can be easily repeated
* Browser may be able to resize the image
* Images load progressively, visibly in the browser
* Data is available locally for reprocessing
* Once the .js and .css libraries are downloaded they can be re-used with no further download overhead
Disadvantages
* Identical data cannot be reprocessed without refreshing the page
* Heavier load on the server processor
* Server side image storage required
* Multiple identical images need to be separately drawn
* .js and .css libraries need to be part of the download package
* JavaScript must be available and activated on the client
* Taking a copy of a chart requires a screenshot
* Libraries load without screen activity, giving the appearance of slow loading
The open source world offers class libraries to provide either or both solutions. An example of a simple server-side library is libchart, and a client side library is Webfx Charting, both of which can be incorporated into a PHP script. Both projects provide samples of library output and example code. Each method probably requires approximately the same amount of coding effort. This article is not intended to compare the output of the two libraries, just to examine the relative advantages and disadvantages of the two approaches. Additionally, there are other methods such as Java which are not considered here.
The choice for webmasters comes down to two major components: the actual processing time, and the bandwidth required to deliver the final product. Generally the bandwidth component is easier to quantify. Some server managers place official quotas on bandwidth, but leave unofficial honour system quotas on processing time, reserving the right to reduce service to webmasters who place unfair burdens on processing time on shared servers. It is in the interests of webmasters to keep the server managers happy on both counts.
Download size
Focusing on the code necessary to draw the chart on the client machine, in the server side situation the code can be very short:
In the client side situation the minimum might be:
That's more than 800 characters, not counting spaces or data, about 450 of which is repeated for each chart to be displayed. Compared with 22 characters for the server side, that's not so trivial.
When it comes to data, the client-side is expecting an array of numbers. In the case that the array is short and consists of simple integers, the array is short and almost trivial. Now consider that the data might be an array of double-precision numbers such as [123456.7890,123456.7889,...,2.6]. A webmaster might be able to round down to integers before transmission, but this would lose some precision, which could be a problem when the data is to be re-used at the client end. Two hundred data points at 11 characters per number each plus the separation character gives an array character length of about 2,400.
In the case of one single chart we can expect something like:
Server side (bytes) Client side (bytes)
Basic html 22 Basic html 400
Libraries 0 Libraries (loaded once per page, if not cached) 66,000
Image 25,000 Image code 450
Data 0 Data (est., per image) 2,400
Total 25,022 Total 69,250
So the download for the client side will probably be larger for one image.
Now consider the case of five different charts on the same page:
Server side (bytes) Client side (bytes)
Basic html 100 Basic html 400
Libraries 0 Libraries (loaded once per page, if not cached) 66,000
Image 125,000 Image code 2,250
Data 0 Data (est., 5 images) 12,000
Total 125,100 Total 80,650
In the case of five charts, the client side download package is smaller. Clearly the issue of the size of the download depends entirely on the number of charts. While these numbers may appear insignificant, when a page is served up many thousands of times, the total becomes a threat to your bandwidth limit.
CPU load
Next we need to consider the issue of CPU load. The server-side class library uses the freetype and gd libraries. Some server managers consider these libraries to be processor-intensive, a critical issue in a shared server environment. You would expect that server-side processing would create a greater CPU load, but is that really so?
Using the top utility on a standalone single-processor server (2.8GHz Intel with hyperthreading and 2GB of RAM) I had the following experience. I set up a PHP script to generate a chart from 200 random data points using MySQL. One script output the charts as three separate 28KB .png images and the other drew three charts with JavaScript. Some of the CPU load can be attributed to MySQL, but since the database processing for each is the same, this should cancel out.
Results GD -> .png JavaScript
Page load seconds (*dialup) 15 3
CPU load Average increase of 10% for ~5 seconds Average increase of 3% for ~3 seconds
Download size (Source code + Other) 28K * 3 = 84K 8K + 66K = 74K
Conclusion
In summary, charting and graphing are important and informative assets. After all, a picture can be worth a thousand words. However, in a Web context, we need to balance whether it is better to send charts and graphs as pictures or the thousand words. Providing the data and then using JavaScript locally to draw the results not only gives a faster page load to the end user, but reduces the load on the server CPU, in this case by a factor of about three.
Colin Beckingham is a freelance programmer and writer in Eastern Ontario. He is currently working on a project that logs and charts the operation of a biomass burner using open source resources.
Album Cover Art Downloader makes iPod a touch nicer
By Tina Gasperson on February 01, 2008 (7:00:00 PM)
When ripping CDs from my own collection or (shh) my friends', I didn't always bother to include the cover art. Personally, I never considered album art for my iPod all that important. That has changed now that we have an iPod touch in the family. If I'm missing a lot of album art, the experience of virtually flipping through my music collection, something Apple calls "cover flow," is diminished. iTunes doesn't always offer art for albums I didn't purchase from iTunes. Thankfully, a little GPL-licensed application called Album Cover Art Downloader (ACAD) solved my problem.
The iPod touch's cover flow is an irresistible option to "flip" through your virtual album collection by sweeping a fingertip across a landscape-oriented screen. If the art is missing, iPod replaces it with a dull, dark gray square with boring musical notes. That's no fun! So, I went looking for a solution to the missing album art problem, and found a GPL program that has become my favorite for keeping my album art straight. ACAD runs on Windows or Linux, and it makes the process of finding and downloading cover art quick, easy, and fun.
Downloading and installing ACAD is no problem; there is a self-extracting executable for Windows, an .rpm for Red Hat, and a .deb for Debian. The source code is available for everyone else. you can configure ACAD to search for album art from a variety of sources, including Yahoo!, Amazon, and Buy.com, then tell ACAD if you'd like to save the files to be compatible with KDE and GNOME, in a generic image file format like .png, embed them in ID3v2 tags, or set them for use in Windows Media files. Finally, tell ACAD how you'd like it to "guess" which album art to look for, either by looking at the ID3v2 tags or examining the pathname %(artist)%(album), and you're set.
Once the program is installed and configured, select File -> Open from the menu to show ACAD where your music files are. From there, you can select individual albums or songs for which you'd like ACAD to find cover art, or you can just select everything -- though if your music collection is very large, ACAD has a tendency to bog down and even crash. I found it best to choose no more than three or four albums at a time, just to be safe.
ACAD then begins a search for your album art and presents you with choices. Highlight the album on the left, or a single track, if for some reason you don't want to put album art on the entire album (or if you only have one or two tracks from the album), select the artwork you want, and click Set as Cover. Make sure you have the correct album highlighted before you set the cover, or ACAD will happily embed the wrong artwork in your music files. If that happens, delete the cover and try again.
Once I located album art for the majority of my music, it only took a few minutes to re-import the files onto my iPod touch, and just like that, my cover flow was flowing much better, thank you.
Tina Gasperson writes about business and technology for some of the most respected publications in the industry. She's been freelancing since 1998.
When ripping CDs from my own collection or (shh) my friends', I didn't always bother to include the cover art. Personally, I never considered album art for my iPod all that important. That has changed now that we have an iPod touch in the family. If I'm missing a lot of album art, the experience of virtually flipping through my music collection, something Apple calls "cover flow," is diminished. iTunes doesn't always offer art for albums I didn't purchase from iTunes. Thankfully, a little GPL-licensed application called Album Cover Art Downloader (ACAD) solved my problem.
The iPod touch's cover flow is an irresistible option to "flip" through your virtual album collection by sweeping a fingertip across a landscape-oriented screen. If the art is missing, iPod replaces it with a dull, dark gray square with boring musical notes. That's no fun! So, I went looking for a solution to the missing album art problem, and found a GPL program that has become my favorite for keeping my album art straight. ACAD runs on Windows or Linux, and it makes the process of finding and downloading cover art quick, easy, and fun.
Downloading and installing ACAD is no problem; there is a self-extracting executable for Windows, an .rpm for Red Hat, and a .deb for Debian. The source code is available for everyone else. you can configure ACAD to search for album art from a variety of sources, including Yahoo!, Amazon, and Buy.com, then tell ACAD if you'd like to save the files to be compatible with KDE and GNOME, in a generic image file format like .png, embed them in ID3v2 tags, or set them for use in Windows Media files. Finally, tell ACAD how you'd like it to "guess" which album art to look for, either by looking at the ID3v2 tags or examining the pathname %(artist)%(album), and you're set.
Once the program is installed and configured, select File -> Open from the menu to show ACAD where your music files are. From there, you can select individual albums or songs for which you'd like ACAD to find cover art, or you can just select everything -- though if your music collection is very large, ACAD has a tendency to bog down and even crash. I found it best to choose no more than three or four albums at a time, just to be safe.
ACAD then begins a search for your album art and presents you with choices. Highlight the album on the left, or a single track, if for some reason you don't want to put album art on the entire album (or if you only have one or two tracks from the album), select the artwork you want, and click Set as Cover. Make sure you have the correct album highlighted before you set the cover, or ACAD will happily embed the wrong artwork in your music files. If that happens, delete the cover and try again.
Once I located album art for the majority of my music, it only took a few minutes to re-import the files onto my iPod touch, and just like that, my cover flow was flowing much better, thank you.
Tina Gasperson writes about business and technology for some of the most respected publications in the industry. She's been freelancing since 1998.
Finding the happy medium in FOSS
By Thomas King on February 02, 2008 (2:00:00 PM)
Last year, Dell began offering Ubuntu on non-corporate desktops and laptops, opening the door for other large computer companies to follow suit. With this offering came a lot of discussion over what Dell should include with each computer sold. In a recent iTWire article concerning Dell's inclusion of its re-worked Ubuntu 7.10 and LinDVD (a commercial Linux DVD player), comments ran the gamut from FOSS purity to legal questions to even questioning Dell's motives. Clearly the FOSS community is pulled in all directions trying to satisfy users. Is there any happy medium? Can the community balance the requests of purists and pragmatists and still release usable products?
Here are the four points, if you will, on which FOSS developers are pulled in different directions:
* Free as in freedom.
* Free as in beer.
* Legal and ethical.
* Usable and pragmatic.
The differences are more philosophical than nuts-and-bolts.
Free as in freedom
Richard M. Stallman back in the 1980s proclaimed that all software should be available to the common man, where anyone should be allowed to freely view the source code, learn from it, change it, and improve it. This was an answer to what he saw as commercialism overtaking what was once open academia. He laid out what he saw as the Four Freedoms that software should provide:
* The freedom to run the program as you wish.
* The freedom to study the program's source code and change it so the program does what you wish.
* The freedom to distribute exact copies to others, when you wish.
* The freedom to distribute copies of your modified versions to others, when you wish.
Since then, he "fathered" the GNU Public License (GPL), currently in its third version. In simplistic terms, source code released under the GPL is available to anyone at a nominal fee or a freely accessible location, and any derivatives must also be licensed under the GPL subject to the same terms.
These terms have caused companies interested in GPL code to balk at using it, claiming any derivatives would open their secrets to the world. However, there are plenty of examples of successful companies using open source code. Red Hat, for instance, says the following about using pure open source in their products:
All software is written with source code. With open source software, the code is protected by a special license that ensures everyone has access to that code. That means no one company can fully own it. Freedom means choice. Choice means power. That's why we believe open source is inevitable. It returns control to the customer. You can see the code, change it, learn from it. Bugs are more quickly found and fixed. And when customers don't like how one vendor is serving them, they can choose another without overhauling their infrastructure. No more technology lock-in. No more monopolies.
What does this mean for developers? What freedom exists in a license that demands that their work be freely available to anyone else? This was answered in a blog entry by numerodix initiated in part by the arguments between the GPL and the BSD camps:
Both license models make software free, but only GPL software is sustainably free. The BSD gives greater freedom, the GPL gives more freedom. Choose which one you value more.
I contend the cause of software freedom as it pertains to licensing is best answered by using the best tool for the job, and that is up to the software author and developer.
Free as in beer
Free as in beer software does not always provide developers a living. So how do they make money? By only selling support? Many programmers do not have the skills necessary to provide good support to end users. Their jargon and mannerisms are too different, and they are accustomed to mathematical logic. Throw in high emotion by end users and most geeks are lost. Others may not have the business sense to grow a successful business by themselves around this model. A few years ago, one anti-FOSS advocate, Daniel Wallace, even went so far as to sue the Free Software Foundation and others for endangering his livelihood. The case ended up being thrown out, but it brought about questions on how sustainable "free as in beer" can really be.
FOSS developers can make a living while still giving their code away. They may be able to work for a large company that pays developers to work on FOSS full time. This may not be practical, though, for those who want to work on their own projects. Many of these developers end up coding in their spare time, or are fortunate enough to work for a company that encourages them to work on their own projects during business hours. Other developers are indeed successful in selling support contracts and can even build up excellent businesses doing so. Some have even suggested using software bounties to include wanted features.
Since "necessity is the mother of invention," the FOSS community will continue to find ways to sustain its projects, developers, and users, and continue to be innovative and competitive. There are many choices, some of which require a change in priorities or process in order to keep from compromising one's ideals.
Legal and ethical
How does a developer become creative and innovative without stepping on toes? How can one learn from other programmers without eventually violating a law? Is it possible to be innovative and still be bound by legalities and ethical restraints? In the world's increasingly litigious society, it becomes more and more mandatory to keep one's p's and q's in order.
Many comments surrounding Dell's Ubuntu 7.10 release focused on the issue of libdvdcss, the "software library for accessing and unscrambling DVDs encrypted with the Content Scramble System (CSS)" offered in some distributions' repositories. Since its legality is still argued, LinDVD appeared to be the best answer for Dell to give their customers a better experience, yet it is seemingly in direct conflict with RMS and his camp's beliefs in utter software freedom. Distro managers must weigh their own risk versus their users' benefit when including proprietary codecs and drivers.
A recurring theme in kernel discussions is the idea of binary blobs and binary drivers being included with the Linux kernel, which is revolting to almost everyone involved. Distributing these drivers and blobs with Linux is thought to be illegal at worst and unethical at best. However, step above the kernel level and give users a chance to download their own binary drivers, and further divisions show up. Although free as in beer, these closed drivers are not free as in freedom and often raise the ethical rankles of developers since they cannot be maintained by the community at large. With these drivers, end users are at the mercy of the company that issues them to fix problems.
Consider the legal brouhaha when one Linux developer tried to change the BSD license on the madwifi driver to convert it to GPL. Once the legal issues were resolved, questions of ethics arose -- especially how ethical was it to suggest the BSD license allowed anyone to use BSD code and not return improvements on it. The letter of the BSD license could be followed, but the spirit of openness was violated and arguments ensued.
In addition to what is legal and ethical to include and distribute with GNU/Linux, developers must also deal with sabre-rattling by proprietary companies thinking they have their intellectual property (patents and/or copyrights) in FOSS projects. There are numerous examples: SCO and their dying litigation, OLPC being dragged into a lawsuit in Nigeria over keyboard design, and Microsoft generating FUD with its statement that FOSS violates 235 Microsoft patents. Though most of this is hot air, it still affects the way projects are run and the way developers must certify their work. Instead of focusing on simply being creative and collaborative, they must also be mindful of the legalities. Even a hint of a violation can bring harsh litigation.
Thankfully, developers are not without help -- witness the past year's GPL legal wranglings and the Software Freedom Law Center. Or take McAfee's statement that some of its software may contain GPL code, yet the company may not be abiding by the license agreement. Although McAfee now states this is "normal filing language," that fact may never have come to light without the GPL showing its legal strength.
Until all software is released under a set of compatible free licenses, developers will be hounded to ensure software is "legal and ethical," even if they are not prevented from making progress, and they do have help.
Usable and pragmatic
I consider myself more pragmatic than idealistic, and I'm always peeved when people are more concerned with form to the detriment of function. Make sure it works first, and then pretty it up!
In this respect, I see why distributions like Ubuntu are successful. They are attractive, but they have worked hard to continuously improve their functionality and usability. With a large influx of new users cutting their teeth on Ubuntu, GNU/Linux is becoming synonymous in some quarters with Ubuntu Linux!
The meat of the matter for software is this: human beings must be able to use it. Albert Einstein said, "Concern for man and his fate must always form the chief interest of all technical endeavors. Never forget this in the midst of your diagrams and equations."
This point may conflict with some of the others. Wholly following freedom means users may lose out on experiencing something on the Web or elsewhere. If users must make sure they use no closed source codecs or tools, they sometimes have to ignore what is practical in favor of what is idealistic. On the other hand, wholly following what is "usable and pragmatic" sometimes means ignoring your freedoms over whatever fits the bill. And ignoring "free as in beer" may undercut the community you want to use your software, but not paying a straight salary for programming may cause talented workers to bypass FOSS.
"Normal" users want their computers to just work, and this is pounded into the Linux community over and over again. Therefore, some suggest, developers should ease up on their idealistic ways and allow proprietary ideas, patents, licenses, and whatever compromise is needed in order to get warm seats using FOSS. This attitude would be enough to make RMS have a seizure and make FOSS veterans foam at the mouth!
Still, formats like Flash, MP3, and other codecs are in ubiquitous use on the Internet. "Normal" users want to surf to a Web site and watch, listen, or read whatever they want. So what if they lose some freedoms? So what if licensing a codec comes with a fee, as long as they don't pay it? So what if it's not legal or ethical to distribute these codecs?
There is little wonder that distributions such as Linux Mint and PCLinuxOS, which bundle proprietary software, are popular. However, in their pursuit of usability these distros come fairly close to with some of the points mentioned before. Ubuntu tries to walk that thin line in making it easy for users to install proprietary codecs and drivers, yet not actually roll them out with their distributions. Fedora makes sure everyone knows it is interested in freedom first but somewhat begrudgingly now gives its users an easy way to install add-on drivers and codecs. Other distributions strike their own balance in their own ways.
GNU/Linux is becoming more usable in its own right, but many users demand developers add usability and ignore other important tenets of FOSS -- yet the same developers have to appease FOSS veterans too.
So what is happy medium?
Can we reconcile these four points? Similar to the saying, "Cheap, fast, good -- pick any two," FOSS is in a constant struggle to balance these issues.
When I was in the Army, one of the exercises we used was called the Ranger Push up. Four soldiers would make a square, each soldier's feet on the upper-back of another soldier. All four would have to push up at the same time or they all would collapse. If one person failed to push up, the other three felt the pressure.
In much the same way, FOSS attempts to balance these four points. It makes for a mad collection of Catch-22s, perfect timing, critical mass, and compromise. FOSS is already a force to be reckoned with, and is perfectly usable in most situations. It is past making ripples and well on its way to making tsunamis in the software world. As it matures, can the FOSS community find the right balance?
Thomas King is a server technician for a large computer corporation. He has used computers for three decades, and is currently in the seventh year of his FOSS adventures.
Last year, Dell began offering Ubuntu on non-corporate desktops and laptops, opening the door for other large computer companies to follow suit. With this offering came a lot of discussion over what Dell should include with each computer sold. In a recent iTWire article concerning Dell's inclusion of its re-worked Ubuntu 7.10 and LinDVD (a commercial Linux DVD player), comments ran the gamut from FOSS purity to legal questions to even questioning Dell's motives. Clearly the FOSS community is pulled in all directions trying to satisfy users. Is there any happy medium? Can the community balance the requests of purists and pragmatists and still release usable products?
Here are the four points, if you will, on which FOSS developers are pulled in different directions:
* Free as in freedom.
* Free as in beer.
* Legal and ethical.
* Usable and pragmatic.
The differences are more philosophical than nuts-and-bolts.
Free as in freedom
Richard M. Stallman back in the 1980s proclaimed that all software should be available to the common man, where anyone should be allowed to freely view the source code, learn from it, change it, and improve it. This was an answer to what he saw as commercialism overtaking what was once open academia. He laid out what he saw as the Four Freedoms that software should provide:
* The freedom to run the program as you wish.
* The freedom to study the program's source code and change it so the program does what you wish.
* The freedom to distribute exact copies to others, when you wish.
* The freedom to distribute copies of your modified versions to others, when you wish.
Since then, he "fathered" the GNU Public License (GPL), currently in its third version. In simplistic terms, source code released under the GPL is available to anyone at a nominal fee or a freely accessible location, and any derivatives must also be licensed under the GPL subject to the same terms.
These terms have caused companies interested in GPL code to balk at using it, claiming any derivatives would open their secrets to the world. However, there are plenty of examples of successful companies using open source code. Red Hat, for instance, says the following about using pure open source in their products:
All software is written with source code. With open source software, the code is protected by a special license that ensures everyone has access to that code. That means no one company can fully own it. Freedom means choice. Choice means power. That's why we believe open source is inevitable. It returns control to the customer. You can see the code, change it, learn from it. Bugs are more quickly found and fixed. And when customers don't like how one vendor is serving them, they can choose another without overhauling their infrastructure. No more technology lock-in. No more monopolies.
What does this mean for developers? What freedom exists in a license that demands that their work be freely available to anyone else? This was answered in a blog entry by numerodix initiated in part by the arguments between the GPL and the BSD camps:
Both license models make software free, but only GPL software is sustainably free. The BSD gives greater freedom, the GPL gives more freedom. Choose which one you value more.
I contend the cause of software freedom as it pertains to licensing is best answered by using the best tool for the job, and that is up to the software author and developer.
Free as in beer
Free as in beer software does not always provide developers a living. So how do they make money? By only selling support? Many programmers do not have the skills necessary to provide good support to end users. Their jargon and mannerisms are too different, and they are accustomed to mathematical logic. Throw in high emotion by end users and most geeks are lost. Others may not have the business sense to grow a successful business by themselves around this model. A few years ago, one anti-FOSS advocate, Daniel Wallace, even went so far as to sue the Free Software Foundation and others for endangering his livelihood. The case ended up being thrown out, but it brought about questions on how sustainable "free as in beer" can really be.
FOSS developers can make a living while still giving their code away. They may be able to work for a large company that pays developers to work on FOSS full time. This may not be practical, though, for those who want to work on their own projects. Many of these developers end up coding in their spare time, or are fortunate enough to work for a company that encourages them to work on their own projects during business hours. Other developers are indeed successful in selling support contracts and can even build up excellent businesses doing so. Some have even suggested using software bounties to include wanted features.
Since "necessity is the mother of invention," the FOSS community will continue to find ways to sustain its projects, developers, and users, and continue to be innovative and competitive. There are many choices, some of which require a change in priorities or process in order to keep from compromising one's ideals.
Legal and ethical
How does a developer become creative and innovative without stepping on toes? How can one learn from other programmers without eventually violating a law? Is it possible to be innovative and still be bound by legalities and ethical restraints? In the world's increasingly litigious society, it becomes more and more mandatory to keep one's p's and q's in order.
Many comments surrounding Dell's Ubuntu 7.10 release focused on the issue of libdvdcss, the "software library for accessing and unscrambling DVDs encrypted with the Content Scramble System (CSS)" offered in some distributions' repositories. Since its legality is still argued, LinDVD appeared to be the best answer for Dell to give their customers a better experience, yet it is seemingly in direct conflict with RMS and his camp's beliefs in utter software freedom. Distro managers must weigh their own risk versus their users' benefit when including proprietary codecs and drivers.
A recurring theme in kernel discussions is the idea of binary blobs and binary drivers being included with the Linux kernel, which is revolting to almost everyone involved. Distributing these drivers and blobs with Linux is thought to be illegal at worst and unethical at best. However, step above the kernel level and give users a chance to download their own binary drivers, and further divisions show up. Although free as in beer, these closed drivers are not free as in freedom and often raise the ethical rankles of developers since they cannot be maintained by the community at large. With these drivers, end users are at the mercy of the company that issues them to fix problems.
Consider the legal brouhaha when one Linux developer tried to change the BSD license on the madwifi driver to convert it to GPL. Once the legal issues were resolved, questions of ethics arose -- especially how ethical was it to suggest the BSD license allowed anyone to use BSD code and not return improvements on it. The letter of the BSD license could be followed, but the spirit of openness was violated and arguments ensued.
In addition to what is legal and ethical to include and distribute with GNU/Linux, developers must also deal with sabre-rattling by proprietary companies thinking they have their intellectual property (patents and/or copyrights) in FOSS projects. There are numerous examples: SCO and their dying litigation, OLPC being dragged into a lawsuit in Nigeria over keyboard design, and Microsoft generating FUD with its statement that FOSS violates 235 Microsoft patents. Though most of this is hot air, it still affects the way projects are run and the way developers must certify their work. Instead of focusing on simply being creative and collaborative, they must also be mindful of the legalities. Even a hint of a violation can bring harsh litigation.
Thankfully, developers are not without help -- witness the past year's GPL legal wranglings and the Software Freedom Law Center. Or take McAfee's statement that some of its software may contain GPL code, yet the company may not be abiding by the license agreement. Although McAfee now states this is "normal filing language," that fact may never have come to light without the GPL showing its legal strength.
Until all software is released under a set of compatible free licenses, developers will be hounded to ensure software is "legal and ethical," even if they are not prevented from making progress, and they do have help.
Usable and pragmatic
I consider myself more pragmatic than idealistic, and I'm always peeved when people are more concerned with form to the detriment of function. Make sure it works first, and then pretty it up!
In this respect, I see why distributions like Ubuntu are successful. They are attractive, but they have worked hard to continuously improve their functionality and usability. With a large influx of new users cutting their teeth on Ubuntu, GNU/Linux is becoming synonymous in some quarters with Ubuntu Linux!
The meat of the matter for software is this: human beings must be able to use it. Albert Einstein said, "Concern for man and his fate must always form the chief interest of all technical endeavors. Never forget this in the midst of your diagrams and equations."
This point may conflict with some of the others. Wholly following freedom means users may lose out on experiencing something on the Web or elsewhere. If users must make sure they use no closed source codecs or tools, they sometimes have to ignore what is practical in favor of what is idealistic. On the other hand, wholly following what is "usable and pragmatic" sometimes means ignoring your freedoms over whatever fits the bill. And ignoring "free as in beer" may undercut the community you want to use your software, but not paying a straight salary for programming may cause talented workers to bypass FOSS.
"Normal" users want their computers to just work, and this is pounded into the Linux community over and over again. Therefore, some suggest, developers should ease up on their idealistic ways and allow proprietary ideas, patents, licenses, and whatever compromise is needed in order to get warm seats using FOSS. This attitude would be enough to make RMS have a seizure and make FOSS veterans foam at the mouth!
Still, formats like Flash, MP3, and other codecs are in ubiquitous use on the Internet. "Normal" users want to surf to a Web site and watch, listen, or read whatever they want. So what if they lose some freedoms? So what if licensing a codec comes with a fee, as long as they don't pay it? So what if it's not legal or ethical to distribute these codecs?
There is little wonder that distributions such as Linux Mint and PCLinuxOS, which bundle proprietary software, are popular. However, in their pursuit of usability these distros come fairly close to with some of the points mentioned before. Ubuntu tries to walk that thin line in making it easy for users to install proprietary codecs and drivers, yet not actually roll them out with their distributions. Fedora makes sure everyone knows it is interested in freedom first but somewhat begrudgingly now gives its users an easy way to install add-on drivers and codecs. Other distributions strike their own balance in their own ways.
GNU/Linux is becoming more usable in its own right, but many users demand developers add usability and ignore other important tenets of FOSS -- yet the same developers have to appease FOSS veterans too.
So what is happy medium?
Can we reconcile these four points? Similar to the saying, "Cheap, fast, good -- pick any two," FOSS is in a constant struggle to balance these issues.
When I was in the Army, one of the exercises we used was called the Ranger Push up. Four soldiers would make a square, each soldier's feet on the upper-back of another soldier. All four would have to push up at the same time or they all would collapse. If one person failed to push up, the other three felt the pressure.
In much the same way, FOSS attempts to balance these four points. It makes for a mad collection of Catch-22s, perfect timing, critical mass, and compromise. FOSS is already a force to be reckoned with, and is perfectly usable in most situations. It is past making ripples and well on its way to making tsunamis in the software world. As it matures, can the FOSS community find the right balance?
Thomas King is a server technician for a large computer corporation. He has used computers for three decades, and is currently in the seventh year of his FOSS adventures.
MIT researchers fight gridlock with Linux
Feb. 02, 2008
At the Massachusetts Institute of Technology (MIT), researchers are testing a Linux-based automotive telematics system intended to reduce traffic congestion. CarTel is a distributed, GPS-enabled mobile sensor network that uses WiFi "opportunistically" to exploit brief windows of coverage to update a central traffic analysis program.
The CarTel system is used to test a growing number of automotive-related research projects at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). In some ways, the system is similar to the recently announced Dash Express navigation system, in that it uses embedded Linux, GPS, and WiFi, and links up to a central route analysis system. Unlike Dash, however, CarTel currently lacks a display or real-time navigation and mapping features. Dash, meanwhile, offers additional location-based services and incorporates a cellular modem (850MHz triband GSM) to provide a continuous connection, switching to WiFi when available.
CarTel, open and with case
Some experiments using the CarTel design have used a cellular modem, says Associate Professor Sam Madden in an interview. Yet, the main goal is to find out how much can be achieved with an intermittent communications system. Cellular service would add significantly to the operating cost of the system.
CarTel leaders Sam Madden (left) and Hari Balakrishnan (right)
Madden, who leads the project along with Professor Hari Balakrishnan, says CarTel's goal is to establish a flexible, affordable platform for diverse automotive-related research projects. These include testing continuous-stream querying, exploring issues of mobile, intermittent WiFi, and researching fleet management applications, driver safety technology, and identifying pothole location for maintenance planning.
The main focus, however, is to develop sophisticated route-selection algorithms to address the problem of congested traffic. "Everyone agrees that traffic is a pain," says Madden, "so avoiding it is good. Hopefully this will have some direct benefit to users." Improving routing could have a significant impact on commute time, says Madden, which in turn would reduce fuel consumption and pollution.
As a test deployment, the CarTel team installed Linux-based computers and sensor kits in 30 limousines from a Cambridge, Mass.-based livery service called PlanetTran. For the last year, the hybrid-based limos have been driving around collecting data about Boston-area traffic conditions, as well as information about potholes, WiFi availability, and the health of the cars themselves.
CarTel's OBD-II interface
The CarTel computer is equipped with an OBD-II (on-board diagnostics II) interface to the car's sensor network. This lets it record speed, mileage, emissions, RPMs, and other information. In addition, the car computer hooks up to additional sensors, depending on the experiment, such as cameras or three-way accelerometers to measure and map potholes. Other possibilities include noise and pollution detectors.
Inside CarTel
The current hardware platform for CarTel is a box about 7 x 5 x 1 inches equipped with a Soekris net4801 single-board computer (SBC) running Linux 2.6. The computer has a "586-class" CPU running at 266MHz, with 128MB of RAM, and 1GB (or more) of flash. The CarTel box is equipped with two serial ports for collecting sensor data, plus a USB 1.1 port, and a WiFi card plugged into a miniPCI slot. A commodity GPS unit is connected via one USB port. A Bluetooth dongle connects to the other USB port, allowing connections from a mobile phone. For telematics data, the system uses an OBD-to-serial adapter from ScanTool.net.
CarTel's Soekris net4801 SBC
(Click to enlarge)
The MIT team has developed some innovative technology to get the most out of fleeting, off-and-on encounters with WiFi access points. First, it has developed a WiFi communications protocol called EasyWiFi that is optimized for brief encounters. EasyWiFi takes only about a hundred milliseconds to establish a mobile wireless connection, says Madden, instead of a more typical five to ten seconds. The modifications are only on the client side, so the protocols can work with any WiFi connection.
Other communication protocols handle variable and intermittent connectivity. A "dpipe" (delay-tolerant pipe) transport programming abstraction developed by postdoc Jakob Eriksson allows producer and consumer nodes to reliably transport data across an intermittently connected network. The client-server dpipe connection is optimized for start-and-stop delivery and for situations when the IP address of an end-point changes frequently.
The dpipe uses a modified delay that can maintain its status even when the session is dropped, says Madden. "When connectivity is availability again, it pops the data out the WiFi connection to the server where it stores it in memory," he adds.
A longer term store-and-forward technology, called CafNet, (carry and forward network) employs protocols that enable cars to serve as data mules, delivering data between nodes that are not connected via telecommunications. This technology could be used, for example, in military, mining, agricultural or scientific applications that are spread out over vast distances, delivering data from sensor networks deployed in the field to Internet servers. Each sensor would require only short-range WiFi connectivity, thereby lowering costs and easing maintenance.
Real-time queries and what-if route algorithms
Distributed sensor data is processed by a portal application that is built around a Linux- and SQL-based stream-processing query application called ICEDB. This delay-tolerant, continuous query processor enables applications running on the portal to issue queries via an API, or transfer a sequence of data between nodes using dpipe. Queries can specify what sensor data is needed and at what rate, and how the data should be sub-sampled, filtered, summarized, and prioritized.
Applications can also query the portal's relational database for analysis. These "snapshot" queries work from available data and don't need to wait synchronously for complete results. All this underlying complexity is shielded from the developer, says Madden, so it appears like a standard SQL relational database.
The ICEDB database was modified from a similar "TinyDB" database that Madden developed years ago for the TinyOS operating system. TinyDB was used to query continuous-feed data from sensor networks that used sensor motes from Crossbow Technologies.
"We made a conscious decision to move to Linux because TinyOS was not as easy to work with," says Madden. "With Linux, there are also a huge number of people developing device drivers, and our graduate students already know how to develop with it."
CarTel portal showing congested routes ranging from red (slow) to green (average)
(Click to enlarge)
The CarTel portal provides a geo-spatial data visualization system based on Google Maps that stores and marks sensor data using GPS coordinates. The portal organizes data as "traces," linear sets of sensor readings collected during a drive. Users can query the system using a graphical query interface to select traces, and then visualize the traces combined with various summaries, statistics, and maps. The portal allows queries both on the driver's own history, as well as the aggregate driving history of other drivers.
Users could include fleet directors, city planning officials, or the drivers themselves, logging in either from a WiFi-enabled laptop in the car or later at home. For example, drivers can see congested areas of their commutes marked in red, orange, and yellow segments, with greens indicating average speeds, and the rare blues indicating clear sailing.
The current focus of the project is in developing algorithms that run on top of the portal application to help drivers plot the best route at a given time. For example, the team's MyRoute project includes applications that model delays observed on road segments as statistical distributions. Various algorithms then use these to compute optimal routes for different times of the day.
"Instead of asking the shortest time or shortest distance from point A to point B, you ask what route should be taken, say, for the highest probability of getting to the airport by a certain time depending on the time selected," says Madden.
Meanwhile, additional research is underway using CarTel to improve fleet management and fine-tune street repair schedules based on identifying the deepest potholes that cars are hitting most frequently. Other research will investigate using telematics data to warn drivers of potential safety hazards or maintenance needs.
With the Soekris net4801 expected to reach end-of-life later this year, the CarTel group is planning to move to another Linux-based design, hoping to reduce the current over-$400 pricetag for the complete system. The plan is to expand the network to 1,000 vehicles, which will help improve the granularity of collected data for better route planning. Other potential lines of inquiry include peer-to-peer WiFi communications between cars to sense upcoming traffic congestion or to assist in onboard safety systems.
Last May, a U.S. government- and industry-led coalition called the Vehicle Infrastructure Integration Consortium (VII-C) was established, aiming to equip every car and roadside in America over the next decade with wirelessly connected Linux-based computers. The goal is to lower driver death rates, reduce traffic jams, and media-enable cars before 2017. The VII-C is funded by the U.S. Department of Transportation (DOT), and includes seven vehicle manufacturers already involved in the U.S. DOT's Intelligent Vehicle Initiative.
More information on the MIT CarTel project can be found here.
--by Eric Brown
At the Massachusetts Institute of Technology (MIT), researchers are testing a Linux-based automotive telematics system intended to reduce traffic congestion. CarTel is a distributed, GPS-enabled mobile sensor network that uses WiFi "opportunistically" to exploit brief windows of coverage to update a central traffic analysis program.
The CarTel system is used to test a growing number of automotive-related research projects at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). In some ways, the system is similar to the recently announced Dash Express navigation system, in that it uses embedded Linux, GPS, and WiFi, and links up to a central route analysis system. Unlike Dash, however, CarTel currently lacks a display or real-time navigation and mapping features. Dash, meanwhile, offers additional location-based services and incorporates a cellular modem (850MHz triband GSM) to provide a continuous connection, switching to WiFi when available.
CarTel, open and with case
Some experiments using the CarTel design have used a cellular modem, says Associate Professor Sam Madden in an interview. Yet, the main goal is to find out how much can be achieved with an intermittent communications system. Cellular service would add significantly to the operating cost of the system.
CarTel leaders Sam Madden (left) and Hari Balakrishnan (right)
Madden, who leads the project along with Professor Hari Balakrishnan, says CarTel's goal is to establish a flexible, affordable platform for diverse automotive-related research projects. These include testing continuous-stream querying, exploring issues of mobile, intermittent WiFi, and researching fleet management applications, driver safety technology, and identifying pothole location for maintenance planning.
The main focus, however, is to develop sophisticated route-selection algorithms to address the problem of congested traffic. "Everyone agrees that traffic is a pain," says Madden, "so avoiding it is good. Hopefully this will have some direct benefit to users." Improving routing could have a significant impact on commute time, says Madden, which in turn would reduce fuel consumption and pollution.
As a test deployment, the CarTel team installed Linux-based computers and sensor kits in 30 limousines from a Cambridge, Mass.-based livery service called PlanetTran. For the last year, the hybrid-based limos have been driving around collecting data about Boston-area traffic conditions, as well as information about potholes, WiFi availability, and the health of the cars themselves.
CarTel's OBD-II interface
The CarTel computer is equipped with an OBD-II (on-board diagnostics II) interface to the car's sensor network. This lets it record speed, mileage, emissions, RPMs, and other information. In addition, the car computer hooks up to additional sensors, depending on the experiment, such as cameras or three-way accelerometers to measure and map potholes. Other possibilities include noise and pollution detectors.
Inside CarTel
The current hardware platform for CarTel is a box about 7 x 5 x 1 inches equipped with a Soekris net4801 single-board computer (SBC) running Linux 2.6. The computer has a "586-class" CPU running at 266MHz, with 128MB of RAM, and 1GB (or more) of flash. The CarTel box is equipped with two serial ports for collecting sensor data, plus a USB 1.1 port, and a WiFi card plugged into a miniPCI slot. A commodity GPS unit is connected via one USB port. A Bluetooth dongle connects to the other USB port, allowing connections from a mobile phone. For telematics data, the system uses an OBD-to-serial adapter from ScanTool.net.
CarTel's Soekris net4801 SBC
(Click to enlarge)
The MIT team has developed some innovative technology to get the most out of fleeting, off-and-on encounters with WiFi access points. First, it has developed a WiFi communications protocol called EasyWiFi that is optimized for brief encounters. EasyWiFi takes only about a hundred milliseconds to establish a mobile wireless connection, says Madden, instead of a more typical five to ten seconds. The modifications are only on the client side, so the protocols can work with any WiFi connection.
Other communication protocols handle variable and intermittent connectivity. A "dpipe" (delay-tolerant pipe) transport programming abstraction developed by postdoc Jakob Eriksson allows producer and consumer nodes to reliably transport data across an intermittently connected network. The client-server dpipe connection is optimized for start-and-stop delivery and for situations when the IP address of an end-point changes frequently.
The dpipe uses a modified delay that can maintain its status even when the session is dropped, says Madden. "When connectivity is availability again, it pops the data out the WiFi connection to the server where it stores it in memory," he adds.
A longer term store-and-forward technology, called CafNet, (carry and forward network) employs protocols that enable cars to serve as data mules, delivering data between nodes that are not connected via telecommunications. This technology could be used, for example, in military, mining, agricultural or scientific applications that are spread out over vast distances, delivering data from sensor networks deployed in the field to Internet servers. Each sensor would require only short-range WiFi connectivity, thereby lowering costs and easing maintenance.
Real-time queries and what-if route algorithms
Distributed sensor data is processed by a portal application that is built around a Linux- and SQL-based stream-processing query application called ICEDB. This delay-tolerant, continuous query processor enables applications running on the portal to issue queries via an API, or transfer a sequence of data between nodes using dpipe. Queries can specify what sensor data is needed and at what rate, and how the data should be sub-sampled, filtered, summarized, and prioritized.
Applications can also query the portal's relational database for analysis. These "snapshot" queries work from available data and don't need to wait synchronously for complete results. All this underlying complexity is shielded from the developer, says Madden, so it appears like a standard SQL relational database.
The ICEDB database was modified from a similar "TinyDB" database that Madden developed years ago for the TinyOS operating system. TinyDB was used to query continuous-feed data from sensor networks that used sensor motes from Crossbow Technologies.
"We made a conscious decision to move to Linux because TinyOS was not as easy to work with," says Madden. "With Linux, there are also a huge number of people developing device drivers, and our graduate students already know how to develop with it."
CarTel portal showing congested routes ranging from red (slow) to green (average)
(Click to enlarge)
The CarTel portal provides a geo-spatial data visualization system based on Google Maps that stores and marks sensor data using GPS coordinates. The portal organizes data as "traces," linear sets of sensor readings collected during a drive. Users can query the system using a graphical query interface to select traces, and then visualize the traces combined with various summaries, statistics, and maps. The portal allows queries both on the driver's own history, as well as the aggregate driving history of other drivers.
Users could include fleet directors, city planning officials, or the drivers themselves, logging in either from a WiFi-enabled laptop in the car or later at home. For example, drivers can see congested areas of their commutes marked in red, orange, and yellow segments, with greens indicating average speeds, and the rare blues indicating clear sailing.
The current focus of the project is in developing algorithms that run on top of the portal application to help drivers plot the best route at a given time. For example, the team's MyRoute project includes applications that model delays observed on road segments as statistical distributions. Various algorithms then use these to compute optimal routes for different times of the day.
"Instead of asking the shortest time or shortest distance from point A to point B, you ask what route should be taken, say, for the highest probability of getting to the airport by a certain time depending on the time selected," says Madden.
Meanwhile, additional research is underway using CarTel to improve fleet management and fine-tune street repair schedules based on identifying the deepest potholes that cars are hitting most frequently. Other research will investigate using telematics data to warn drivers of potential safety hazards or maintenance needs.
With the Soekris net4801 expected to reach end-of-life later this year, the CarTel group is planning to move to another Linux-based design, hoping to reduce the current over-$400 pricetag for the complete system. The plan is to expand the network to 1,000 vehicles, which will help improve the granularity of collected data for better route planning. Other potential lines of inquiry include peer-to-peer WiFi communications between cars to sense upcoming traffic congestion or to assist in onboard safety systems.
Last May, a U.S. government- and industry-led coalition called the Vehicle Infrastructure Integration Consortium (VII-C) was established, aiming to equip every car and roadside in America over the next decade with wirelessly connected Linux-based computers. The goal is to lower driver death rates, reduce traffic jams, and media-enable cars before 2017. The VII-C is funded by the U.S. Department of Transportation (DOT), and includes seven vehicle manufacturers already involved in the U.S. DOT's Intelligent Vehicle Initiative.
More information on the MIT CarTel project can be found here.
--by Eric Brown
How to speed up Windows Vista: official and unofficial tips
By Tim Anderson, ITWriting
Strip, shutdown, repeat
Microsoft has published an article on speeding up Vista, aimed at general users.
It's not too bad. Here's the summary:
* Delete programs you never use
* Limit how many programs load at startup
* Defragment your hard drive
* Clean up your hard disk
* Run fewer programs at the same time
* Turn off visual effects
* Restart regularly
* Add more memory
* Check for viruses and spyware
* Disable services you don't need
Still, it's a bit scattergun. I prefer a two-stage approach to improving performance (same applies to a single application):
1. Find out what is slow
2. Speed it up, or leave it out
For example, the benefits of adding memory tail off after a certain point. Task Manager will tell you to what extent RAM is slowing down Vista. Further, adding memory beyond 3GB is pretty much wasted on 32-bit Vista, since the system can only address 3GB, and the BIOS will likely use a lot of the fourth gigabyte address space. That said, a system that is critically short of RAM (in other words, constantly swapping out memory to the hard drive) is in my opinion broken and unusable. Adding RAM in such cases delivers huge rewards.
Uninstalling programs gives little performance benefit if they are not running (unless disk space is limited). The aim is to reduce the number of running processes, not entries in the Start menu.
Vista defragments your drive regularly, by default. The benefits are often rather small, so it would be equally valid to suggest removing it from the schedule, or scheduling it to run less frequently.
The advice to restart regularly needs examination. Yes, a reboot can fix a sluggish machine. But it shouldn't be necessary, and I recall that keeping Vista always-on was intended to be a benefit of the operating system. Yes, here's a quote from a Power Management in Windows Vista presentation [and here's the PowerPoint]:
* Windows Vista promotes the use of sleep as the default off state
In the right circumstances, Vista can run for ages without any problem. I've actually had Media Center (Vista Ultimate) run for several months without any issues; though this kind of thing is not very green so that's another reason to do regular switch offs. Still, to my mind "restart regularly" is a symptom of some problem that should be fixed.
Turning off visual effects is reasonable advice, though once again it may not yield much benefit. I tried it on my system and was surprised how little difference it made. Reason: I am running with Aero and a decent-ish graphics card, and hardware acceleration seems to handle the visual effects rather easily. Once again, if it's not the thing slowing you down, then removing it won't speed you up. You can test this quite simply, though it is tedious. Try it both ways. Did it make a difference? Measure it if possible.
It really is worth using the built-in tools, like Task Manager and the Reliability and Performance Monitor, to see which processes are grabbing lots of RAM and CPU. One of the good things about Vista is that such tools are easy to find. Click Start, type "reliability", and click the link.
I'd also like to see mention of some favorite candidates for slowing down Vista:
1. Outlook 2007
2. The indexing service
3. Anti-virus software
4. Windows Defender
Hmmm, at least three of these are from Microsoft. Perhaps they are too embarrassing to mention.
Finally, I suspect disk performance is a big factor in real-world Vista speed. The reason is that many applications are very talkative when it comes to disk access. Here's something to try: go along to the Sysinternals site and download Process Monitor. This gives a good picture of what the actual processes on your Vista box are up to. Note how many events are occurring, how many of them involve file I/O, and which processes are responsible. You will also discover a large part of the reason why Outlook 2007 is so slow.
PS Another article, also just published, has good coverage of swap files and ReadyBoost.
This article originally appeared in ITWriting.
Copyright (c) 2007, ITWriting.com.
A freelance journalist since 1992, Tim Anderson specializes in programming and internet development topics. He has columns in Personal Computer World and IT Week, and also contributes regularly to The Register. He writes from time to time for other periodicals including Developer Network Journal Online, and Hardcopy.
Strip, shutdown, repeat
Microsoft has published an article on speeding up Vista, aimed at general users.
It's not too bad. Here's the summary:
* Delete programs you never use
* Limit how many programs load at startup
* Defragment your hard drive
* Clean up your hard disk
* Run fewer programs at the same time
* Turn off visual effects
* Restart regularly
* Add more memory
* Check for viruses and spyware
* Disable services you don't need
Still, it's a bit scattergun. I prefer a two-stage approach to improving performance (same applies to a single application):
1. Find out what is slow
2. Speed it up, or leave it out
For example, the benefits of adding memory tail off after a certain point. Task Manager will tell you to what extent RAM is slowing down Vista. Further, adding memory beyond 3GB is pretty much wasted on 32-bit Vista, since the system can only address 3GB, and the BIOS will likely use a lot of the fourth gigabyte address space. That said, a system that is critically short of RAM (in other words, constantly swapping out memory to the hard drive) is in my opinion broken and unusable. Adding RAM in such cases delivers huge rewards.
Uninstalling programs gives little performance benefit if they are not running (unless disk space is limited). The aim is to reduce the number of running processes, not entries in the Start menu.
Vista defragments your drive regularly, by default. The benefits are often rather small, so it would be equally valid to suggest removing it from the schedule, or scheduling it to run less frequently.
The advice to restart regularly needs examination. Yes, a reboot can fix a sluggish machine. But it shouldn't be necessary, and I recall that keeping Vista always-on was intended to be a benefit of the operating system. Yes, here's a quote from a Power Management in Windows Vista presentation [and here's the PowerPoint]:
* Windows Vista promotes the use of sleep as the default off state
In the right circumstances, Vista can run for ages without any problem. I've actually had Media Center (Vista Ultimate) run for several months without any issues; though this kind of thing is not very green so that's another reason to do regular switch offs. Still, to my mind "restart regularly" is a symptom of some problem that should be fixed.
Turning off visual effects is reasonable advice, though once again it may not yield much benefit. I tried it on my system and was surprised how little difference it made. Reason: I am running with Aero and a decent-ish graphics card, and hardware acceleration seems to handle the visual effects rather easily. Once again, if it's not the thing slowing you down, then removing it won't speed you up. You can test this quite simply, though it is tedious. Try it both ways. Did it make a difference? Measure it if possible.
It really is worth using the built-in tools, like Task Manager and the Reliability and Performance Monitor, to see which processes are grabbing lots of RAM and CPU. One of the good things about Vista is that such tools are easy to find. Click Start, type "reliability", and click the link.
I'd also like to see mention of some favorite candidates for slowing down Vista:
1. Outlook 2007
2. The indexing service
3. Anti-virus software
4. Windows Defender
Hmmm, at least three of these are from Microsoft. Perhaps they are too embarrassing to mention.
Finally, I suspect disk performance is a big factor in real-world Vista speed. The reason is that many applications are very talkative when it comes to disk access. Here's something to try: go along to the Sysinternals site and download Process Monitor. This gives a good picture of what the actual processes on your Vista box are up to. Note how many events are occurring, how many of them involve file I/O, and which processes are responsible. You will also discover a large part of the reason why Outlook 2007 is so slow.
PS Another article, also just published, has good coverage of swap files and ReadyBoost.
This article originally appeared in ITWriting.
Copyright (c) 2007, ITWriting.com.
A freelance journalist since 1992, Tim Anderson specializes in programming and internet development topics. He has columns in Personal Computer World and IT Week, and also contributes regularly to The Register. He writes from time to time for other periodicals including Developer Network Journal Online, and Hardcopy.
Build Your Own Jabber Server for Private Communication
Thursday, November 29 2007 @ 03:48 PM EST
XMPP is an open technology for instant messaging and presence information. It provides the opportunity to build an open source and free messaging server to handle many applications in varied environments. We are going to setup a XMPP server on a LAN to handle communications between LAN and WAN contacts. By running our own server we can gain some insight into how XMPP and Jabber work, create an efficient and easy to use internal communications setup, connect our server to other servers and services including WAN communications, and last, but not least, we can have our own private communications using SSL. Read on for more on XMPP and how we actually set it all up.
We will be using Openfire, a nice XMPP server written in Java and is one of the most feature-full Jabber servers out there. There are two versions, one licensed for commercial use, and one licensed under GNU. Accordingly, we will use the GNU version for our server. Some of the nice benefits we hope to achieve with this project are creating a private communications network using SSL, enable private file transfers, create an easy to use and manage communications forum for our LAN and have the ability to communicate with those outside of our LAN (even on separate jabber networks), and give us some insight into the inner workings of some of our everyday communications tools.
We will be installing Openfire on an Kubuntu 7.10 machine on our LAN. Our machine was pretty loaded spec-wise, but after some testing, it seemed to become apparent that a lower spec machine would probably do the job as well. I'll highlight the basic steps to the setup now and then we'll jump right in.
1- Obtain openfire for Linux
2- Prepare our machine for the openfire server and create an openfire user and group to run the server under
3- Setup our database
4- Install and configure openfire
5- Setup our server to start on boot
6- Configure our firewall to handle our server for external communications with the WAN
7- See our server in action
Lastly, we will discuss some caveats.
[1] First we want to grab Openfire here. We want to grab the tar.gz form of Openfire and while we are here we will also grab the Linux version of Spark. Spark is a nice client that we can use for the actual communications. Our server will allow connections from any jabber supported client such as pidgin(gaim), adium, etc., but I'll show you some of the workings with Spark as I have found it to be a nice cross-platform entity. Since we grabbed the tar.gz form of the Openfire server, it does not include Java. So in step 2 we will need to make sure we have Java 5 installed. Download the server and client into a writable directory such as your home or desktop.
[2] Now we need to prepare our machine for the server. Unarchive the server and move it to a good direcotry:
tar xzvf openfire_3_4_1.tar.gz
sudo mv openfire /usr/bin/
Next we will create a user to run a server under to keep things a little safer and tidier and give that user the ownership of our newly created openfire directory.
sudo groupadd ofire
sudo useradd -d /usr/bin/openfire -g ofire ofire
sudo passwd ofire
sudo chown -R ofire:ofire /usr/bin/openfire
We also now need to make sure Java is ready for Openfire.
sudo apt-get install java5-*
Follow the instructions for installing java...you will need to download the java5 documentation and put in /tmp according to the installer:
sudo chown root:root jdk-1_5_0-doc.zip
sudo mv jdk-1_5_0-doc.zip /tmp
[3] Openfire will use a database for operations and we will setup one using mysql. We will install mysql and then set up our openfire database, give it proper permissions, and use some information given in the openfire directory to setup the rest of the necesities.
sudo apt-get install mysql-server-5.0 mysql-client-5.0
mysql -u root -p [when prompted enter the password you setup when you installed mysql above]
CREATE DATABASE openfire;
GRANT all on openfire.* to ofire@localhost IDENTIFIED BY password(remember_this_for_setup_later_in_webgui);
cd /usr/bin/openfire/resources/database
cat openfire_mysql.sql | sudo msyql -p openfire ;
[4] Now it is finally time to start up our server and do the installation procedures by navigating to 9090 on the server.
su - ofire
cd /usr/bin/openfire/bin
./openfire start
Now if you are on the server itself, run:
http://localhost:9090
Else, if you are on your LAN, run:
http://IP.OF.YOUR.SERVER:9090
Follow the onscreen instructions and remember to use the password you used to setup the openfire database above and the user ofire when you get to the database page. And also remember that we are using MySQL. Now you are set! Configure your heart away.
[5] To get openfire to start on boot we will add a provided boot script to init.d and we will have to edit a line in the script.
sudo ln -s /usr/bin/openfire/bin/openfire /etc/init.d/
Now open with script and scroll to line 262 where the state nohup is and append the following the end of the line - DO NOT include the quotation marks, as they are here only in presentation:
">../logs/STDOUT.log >../logs/STDERR.log"
You can do this with nano, for example by doing the following:
sudo nano /usr/bin/openfire/bin/openfire
Now you can type Ctrl-W followed by nohup (searches for nohup line) and to verify this is the write line you can type Ctrl-C and see that it says you are currently at line 262. Then append that line with the line shown above.
It should now say on line 262 at the very end:
"-DopenfireHome=$app_home -Dopenfire.lib.dir=$app_home/lib >../logs/STDOUT.log 2>../logs/STDERR.log &"
Now we need to give it execute permissions and to update rc and we are set.
sudo chmod +x /etc/init.d/openfire
sudo update-rc.d openfire defaults
[6] Now we want to configure our firewall to allow for communications with the WAN. This will vary depending on your firewall. So I will tell you the ports and then please consult your firewall documentation on how to open them if you need that assistance. We use monowall so I will kinda give a quick overview with using that.
You will need to do this for each of the ports (for a range you can do at once):
Log into monowall and then choose NAT and add a new rule.
Type in the port for TCP from the list below and then choose your LAN internal IP of your openfire server and input the same port. Then give it a descriptive name and be sure to choose the box at the bottom to auto-add rule. Then click ok and apply the changes.
5222-5223 : for the basic client connections and for client ssl respectively
5269 : for server to server connection if you want to connect to those on other IM servers
7777 : for file transfers
[7] Now you can use your spark client to create an account and login or you can use clients like adium or pidgin. I won't go into specifics for each client, but if you need help feel free to shoot me a message or post a comment and I will help as best I can. Below you can see a few quick screen shots of Spark running.
Now for a few caveats. I won't provide workarounds here as that is always part of the fun of exploring new projects. A few I have fixed and others I haven't, so I am curious to see if any discussion below ensues as to some ideas.
-When on the LAN you will need to adjust your server to the IP of the openfire server (or host) and when you are afar accessing from the WAN then you need to redirect to your domain of your home. See one of our earlier tutorials when we discuss WAN IP addresses and so forth. So just remember that when you come home from work and can't figure out when you aren't connecting to your openfire server.
-There seems to be issues creating new users with some clients. Pidgin seems to be ok, but Adium and iChat don't seem to be working. However, they can connect just fine. So initially you can just create an account for those users yourself in your admin page [http://your.openfire.server.ip:9090]
So far that is all we have. There are a lot of configuration options in the actual openfire server as well as in the clients themselves. Explore and check the documentation. I'll post some more details as I come across new aspects and we may even explore some of the plugins at a later point such as VOIP via Asterisk and so forth. Like I said, feel free to comment or send me a message if you have any questions about our setup or run across any snags. The openfire forums are also a good stomping ground as well. Good luck and happy chatting on your new private server!
XMPP is an open technology for instant messaging and presence information. It provides the opportunity to build an open source and free messaging server to handle many applications in varied environments. We are going to setup a XMPP server on a LAN to handle communications between LAN and WAN contacts. By running our own server we can gain some insight into how XMPP and Jabber work, create an efficient and easy to use internal communications setup, connect our server to other servers and services including WAN communications, and last, but not least, we can have our own private communications using SSL. Read on for more on XMPP and how we actually set it all up.
We will be using Openfire, a nice XMPP server written in Java and is one of the most feature-full Jabber servers out there. There are two versions, one licensed for commercial use, and one licensed under GNU. Accordingly, we will use the GNU version for our server. Some of the nice benefits we hope to achieve with this project are creating a private communications network using SSL, enable private file transfers, create an easy to use and manage communications forum for our LAN and have the ability to communicate with those outside of our LAN (even on separate jabber networks), and give us some insight into the inner workings of some of our everyday communications tools.
We will be installing Openfire on an Kubuntu 7.10 machine on our LAN. Our machine was pretty loaded spec-wise, but after some testing, it seemed to become apparent that a lower spec machine would probably do the job as well. I'll highlight the basic steps to the setup now and then we'll jump right in.
1- Obtain openfire for Linux
2- Prepare our machine for the openfire server and create an openfire user and group to run the server under
3- Setup our database
4- Install and configure openfire
5- Setup our server to start on boot
6- Configure our firewall to handle our server for external communications with the WAN
7- See our server in action
Lastly, we will discuss some caveats.
[1] First we want to grab Openfire here. We want to grab the tar.gz form of Openfire and while we are here we will also grab the Linux version of Spark. Spark is a nice client that we can use for the actual communications. Our server will allow connections from any jabber supported client such as pidgin(gaim), adium, etc., but I'll show you some of the workings with Spark as I have found it to be a nice cross-platform entity. Since we grabbed the tar.gz form of the Openfire server, it does not include Java. So in step 2 we will need to make sure we have Java 5 installed. Download the server and client into a writable directory such as your home or desktop.
[2] Now we need to prepare our machine for the server. Unarchive the server and move it to a good direcotry:
tar xzvf openfire_3_4_1.tar.gz
sudo mv openfire /usr/bin/
Next we will create a user to run a server under to keep things a little safer and tidier and give that user the ownership of our newly created openfire directory.
sudo groupadd ofire
sudo useradd -d /usr/bin/openfire -g ofire ofire
sudo passwd ofire
sudo chown -R ofire:ofire /usr/bin/openfire
We also now need to make sure Java is ready for Openfire.
sudo apt-get install java5-*
Follow the instructions for installing java...you will need to download the java5 documentation and put in /tmp according to the installer:
sudo chown root:root jdk-1_5_0-doc.zip
sudo mv jdk-1_5_0-doc.zip /tmp
[3] Openfire will use a database for operations and we will setup one using mysql. We will install mysql and then set up our openfire database, give it proper permissions, and use some information given in the openfire directory to setup the rest of the necesities.
sudo apt-get install mysql-server-5.0 mysql-client-5.0
mysql -u root -p [when prompted enter the password you setup when you installed mysql above]
CREATE DATABASE openfire;
GRANT all on openfire.* to ofire@localhost IDENTIFIED BY password(remember_this_for_setup_later_in_webgui);
cd /usr/bin/openfire/resources/database
cat openfire_mysql.sql | sudo msyql -p openfire ;
[4] Now it is finally time to start up our server and do the installation procedures by navigating to 9090 on the server.
su - ofire
cd /usr/bin/openfire/bin
./openfire start
Now if you are on the server itself, run:
http://localhost:9090
Else, if you are on your LAN, run:
http://IP.OF.YOUR.SERVER:9090
Follow the onscreen instructions and remember to use the password you used to setup the openfire database above and the user ofire when you get to the database page. And also remember that we are using MySQL. Now you are set! Configure your heart away.
[5] To get openfire to start on boot we will add a provided boot script to init.d and we will have to edit a line in the script.
sudo ln -s /usr/bin/openfire/bin/openfire /etc/init.d/
Now open with script and scroll to line 262 where the state nohup is and append the following the end of the line - DO NOT include the quotation marks, as they are here only in presentation:
">../logs/STDOUT.log >../logs/STDERR.log"
You can do this with nano, for example by doing the following:
sudo nano /usr/bin/openfire/bin/openfire
Now you can type Ctrl-W followed by nohup (searches for nohup line) and to verify this is the write line you can type Ctrl-C and see that it says you are currently at line 262. Then append that line with the line shown above.
It should now say on line 262 at the very end:
"-DopenfireHome=$app_home -Dopenfire.lib.dir=$app_home/lib >../logs/STDOUT.log 2>../logs/STDERR.log &"
Now we need to give it execute permissions and to update rc and we are set.
sudo chmod +x /etc/init.d/openfire
sudo update-rc.d openfire defaults
[6] Now we want to configure our firewall to allow for communications with the WAN. This will vary depending on your firewall. So I will tell you the ports and then please consult your firewall documentation on how to open them if you need that assistance. We use monowall so I will kinda give a quick overview with using that.
You will need to do this for each of the ports (for a range you can do at once):
Log into monowall and then choose NAT and add a new rule.
Type in the port for TCP from the list below and then choose your LAN internal IP of your openfire server and input the same port. Then give it a descriptive name and be sure to choose the box at the bottom to auto-add rule. Then click ok and apply the changes.
5222-5223 : for the basic client connections and for client ssl respectively
5269 : for server to server connection if you want to connect to those on other IM servers
7777 : for file transfers
[7] Now you can use your spark client to create an account and login or you can use clients like adium or pidgin. I won't go into specifics for each client, but if you need help feel free to shoot me a message or post a comment and I will help as best I can. Below you can see a few quick screen shots of Spark running.
Now for a few caveats. I won't provide workarounds here as that is always part of the fun of exploring new projects. A few I have fixed and others I haven't, so I am curious to see if any discussion below ensues as to some ideas.
-When on the LAN you will need to adjust your server to the IP of the openfire server (or host) and when you are afar accessing from the WAN then you need to redirect to your domain of your home. See one of our earlier tutorials when we discuss WAN IP addresses and so forth. So just remember that when you come home from work and can't figure out when you aren't connecting to your openfire server.
-There seems to be issues creating new users with some clients. Pidgin seems to be ok, but Adium and iChat don't seem to be working. However, they can connect just fine. So initially you can just create an account for those users yourself in your admin page [http://your.openfire.server.ip:9090]
So far that is all we have. There are a lot of configuration options in the actual openfire server as well as in the clients themselves. Explore and check the documentation. I'll post some more details as I come across new aspects and we may even explore some of the plugins at a later point such as VOIP via Asterisk and so forth. Like I said, feel free to comment or send me a message if you have any questions about our setup or run across any snags. The openfire forums are also a good stomping ground as well. Good luck and happy chatting on your new private server!
Subscribe to:
Posts (Atom)