Automate posting links from Delicious to Blogger

As you can tell, I've been playing with a way to create real blog posts from Delicious, not just an RSS feed splice. The old feed splice was being done with Feedburner, and it usually worked pretty well, but it lacked the flexibility I wanted and furthermore, it wouldn't actually create a post here. It would only show up in the RSS feed.

Delicious has an "experimental" (as in, not maintained since 2008) "Blog Posting Tool" that simply refuses to work with Blogger, the platform we currently use. In fact, it's so arcane that I don't really know if anyone has ever really made it work on anything other than Wordpress.

I threw something together in PHP that uses the Simple HTML DOM library to parse the output of the Delicious API. One part of that library that broke pretty spectacularly at first was trying to grab the "tag" attribute of each element. Simple HTML DOM uses "tag" as a variable name all over the place, and it caused some major recursion. I used preg_replace to change the name of "tag" to something else to avoid this namespace collision.

It's written in PHP because I was originally planning on cooking up something web-based, but I went more lightweight instead. Who knows, maybe I'll re-write it later on. Perhaps perl, python, ruby or even newLISP would have been better for this. It's designed to be run manually from the command-line or (more specifically) a nightly cron job.

I just know that there aren't any easy-to-use solutions for posting Delicious links to Blogger, and a lot of people seem to want something like this. It's rough, has some vestigial debugging code left in it, and it's probably not easy enough for most folks. It does what I wanted it to do, though, and you'll probably be seeing a lot more frequent link posts because of it.

Source. Public Domain. Yadda yadda. Enjoy.

Network programming and distributed scripting with newLISP

I'm not sure why I never re-posted this here, but alas, better late than never.  I should have an article published in the Summer 2009 issue of 2600 as well, but I haven't seen it yet.

This initially showed up in 2600: The Hacker Quarterly Volume 23 Number 4 (Winter 2006-2007)

newLISP (www.newlisp.org) is a relative newcomer to the interpreted language arena in terms of popularity. While it had its humble beginnings back in 1991 when Lutz Mueller started working on it, only in the last 4 years has development been consistently active.

newLISP is everything that old-school LISP languages are, with a lot of modern features. First off, it's a scripting language that's extremely fast. It has networking ability that's powerful enough to write TCP or UDP client or server applications. Then, to top that off, it has a command called net-eval which makes newLISP stand out from the crowd by giving it the unique ability to easily distribute tasks to other nodes over a network connection.

Binaries (under 200 kilobytes) are available for Windows, BSD, Linux, Mac OS X, Solaris and a host of other platforms. It is released under the GPL. Performance is also second to none. newLISP has been topping the charts on script interpreter benchmarks in several categories thanks to it's small size (under 200 kilobytes) and efficient C code. It outruns php, perl, and even ruby.

newLISP also has some other tricks up its sleeve that make it an excellent system administration scripting language. It has decent filesystem support, so it can see if files or directories exist, determine if a file's permissions are acceptable for reading or writing, and it has very powerful text processing ability using PCRE (perl compatible regular expressions). Finally, it's also worth mentioning that newLISP can easily import whole functions from dynamic libraries such as libmysqlclient (instant MySQL access from within newLISP!), tcl/tk (for creating graphical applications in newLISP) and zlib (for compression and decompression) just to name a few. This makes newLISP one of the most robust and flexible languages around. As you can tell, newLISP is a formidable choice for hackers, geeks, network admins or security professionals wishing to create scripted programs to do network operations or distributed computing with minimal effort

I am lucky to have been able to work directly with Lutz, the founder and creator of newLISP. I got a few direct lessons from him, and from there, started tinkering with it on my own. With that, the first thing I did was create a makeshift port scanner. I learn easiest by example, so here is what I came up with.
Click to see port.lsp

The first part simply assigns the command line arguments into a list called params, then makes sure that 4 parameters were given (program name, host, begin port and ending port). If not, it displays a usage tip before exiting. The second part assigns elements of the list to appropriate variables, then uses a for loop to iterate through the ports, displaying open port numbers that are open. Note that on machines with packet filters that "drop" packets, this port scan will take a very long time. nmap is a much more robust port scanner, however this little script demonstrates the power of newLISP's network commands. We'll run this as a test just for fun:

$ ./port.lsp 192.168.0.105 1 200 

21 open 

22 open 

23 open 

25 open 

79 open 

111 open

Now, let's look into distributed computing, shall we? The core command behind newLISP's distributed computing power -- called "net-eval" -- operates on a list of lists (similar to a 3 dimensional array). The inner-most list is a list of host, port, and a string representing the command(s) you wish to run on the remote node. The outer-most list can contain as many host-port-command lists as your heart desires, allowing you to run many distributed processes at once, and get the results back all at the same time. Then, outside those lists is a timeout in milliseconds. If a result isn't returned in the timeout period, the operation returns "nil" (that is, false). To clarify, net-eval syntax is as follows: 

(net-eval (list (list "host" port-number command-string)) timeout) 

On each remote node, you must have a newLISP listener, which is simply started by running "newlisp -c -d port-number" from the command line. On UNIX environments, you may put an ampersand (&) at the end to launch it in the background, or you may even wish to use "set NOHUP" and log off to leave it running in the background indefinitely. In my example, I went to my Solaris box and launched it, listening on port 31337 as follows: 

$ newlisp -c -d 31337 &

I also launched newLISP listeners on various other machines on my home network, including a few OpenBSD machines, and my wife's MUD/BBS server running Windows Server 2003 with the "Services for UNIX" tools installed. 

Now, care must be taken. It is a bad idea to have a newLISP listener running on a public IP address, because commands like process or exec can launch shell processes on the newLISP node, which is just as good as giving away an unprotected shell account on your network. I advise using newLISP listener nodes only behind a NAT or firewall, or on a segregated network. 

Let's run a test script, shall we? In LISP, boolean and math operations are always performed by placing the operator first, followed by the symbols to apply it to. In addition, the symbols are numbers, but they could easily be strings or lists with some operations. Adding 1 + 2 in LISP is as simple as (+ 1 2) I will start by running a quick addition operation on 1 remote node with a 3000ms (3 second) timeout.

Click to see net-eval-test.lsp 

When we run it, we get the answer to this mind-boggling math problem: 

$ ./net-eval-test.lsp 

(3) 

Now, to expand this even more, I have added three other nodes into the mix, which shows more clearly how the nested list syntax of net-eval works, and I'll demonstrate remote command execution at the same time, using the "exec" command. Notice how the quotes around the command to be run is escaped with backslashes. This is needed to keep from confusing the interpreter. To put quotes inside a quoted string, you need to escape them. This is almost universal to all programming languages. On UNIX-like platforms, uname is used to get information about the operating system and architecture. uname -s -n -m will list the OS that's running, the hostname, and the machine architecture. 

Click to see uname.lsp 

The result is a newLISP list of strings, containing the results of running the command: 

$ ./uname.lsp 

(("SunOS sparky sun4u") ("OpenBSD compy386 i386") ("OpenBSD bouncer sparc") ("Windows mudbbs x86"))

The online documentation for newLISP is very extensive, and features a few rather advanced demonstration scripts, including a working web server written entirely in newLISP. While learning a new programming language is never easy, newLISP is more than mature enough in both implementation and documentation to make it a pretty easy language to add to your list.

Links:
NewLISP Website, full of demonstration newLISP programs, documentation, binaries for many platforms, and newLISP source code:

NewLISP.org

(newLISPer) is a journal, or blog, written by a guy who was just learning newLISP. It's turned into a bunch of newLISP tutorials with some philosophy tossed in as well:

Newlisper Blog ((now unbalanced-parentheses) (you see what I did there?)

Norman's code snippets is a website full of newLISP programs and snippets for Linux (not tested on other platforms). There is a lot of really interesting applications and widgets available to download:

Norman's code Snippets

Encryption Technologies to Avoid

This is intended to be a supplementary information resource on cryptographic best practices. In this article I will cover some cryptographic algorithms with known flaws and some alternatives.

=== ToC =============

1. Introduction

2. Known insecure encryption technologies

3. Alternatives

4. Informative resources

====================

1. Introduction

Security is not the black box on your desktop or network. It it isn't a graphic you put on your website saying that the site is secure because you have a firewall. Security is the constant dialog between you and the other people in your organization on how to best secure the information which you have been entrusted with.

To that end I would like to bring up some technologies which we should avoid or migrate away from at our earliest opportunity.

2. Known insecure encryption technologies.

• DES - DES has been broken for quite a while.
• MD5
  
• SHA1

DES stands for Digital Encryption Standard, it was developed by IBM in 1972. Originally the team at IBM was said to have specified a 128 bit algorithm but was later asked by the NSA to reduce it to 64bit and later 56. This is refuted by government probes, then the insecurity was verified by cryptanalysis and later interviews with the people involved in the project and in de-classified documents.

MD5 is a hashing algorithm designed by Professor Ronald Rivest of MIT. There was research which theorized collisions in the algorithm in 1993, more collisions in 1996 and finally an algorithm was created to generate collisions in 2005 using a large cluster and in 2006 the algorithm was perfected only requiring a single computer.

The SHA cryptographic functions were written by the NSA in 1992. In 2005 flaws were found in the SHA-1 algorithm, although these flaws are yet to be exploited it was decided to move on to a different algorithm.

Things that were never a good idea to begin with:

• Character replacement ex. ROT13, Caesar Cipher
  
• XOR encryption against a trivial key
  
• base64 encoding (or any other encoding)

Character replacement, encoding and XOR are trivial to decrypt/decode. In the case of encoding or replacement, these are easily detected because the structure of the message itself isn't changed. XOR is computationally simple so brute forcing, frequency analysis and other methods can be used to decipher it. Using XOR against a very complex non-repeating one-time pad can offer quite a bit of security as long as strict protocol is followed in handling cleartext and the pad itself.

3. Alternatives
DES users should consider using AES, preferably one of the versions with larger key sizes 256 or 512. MD5 and SHA1 users should consider using the SHA-2 algorithm which are considered to be safe for now. There is an on-going contest to decide what algorithm to use for the new SHA-3 standard.

Many SSL certificates are signed using a MD5 algorithm. This creates the possibility of forging ssl certificates. In addition there was a bad patch to the OpenSSH package in Debian Linux in 2008 which made certificates generated by the affected server insecure. There is a firefox extension which will warn you if the certificate is vulnerable.

4. Informative resources

Wikipedia.org (Accessed April 26, 2009)

• http://en.wikipedia.org/wiki/Advanced_Encryption_Standard
• http://en.wikipedia.org/wiki/Data_Encryption_Standard
• http://en.wikipedia.org/wiki/MD5
• http://en.wikipedia.org/wiki/SHA1

Anka, Márton. "SSL Black List." (Accessed April 26, 2009)
http://www.codefromthe70s.org/sslblacklist.aspx

See also:

• GPG Part 1
• GPG Part 2
• Rubber Hose Cryptanalysis
• Steganography
• Encrypted filesystem images
• Cryptography fun with newLISP
  

newLISP: Twitter Followers/Friends/Common

Here's what I came up with. It does the following:

1. Builds the HTTP Basic Auth header
2. Fetches all your followers (100 at a time until no more are found)
3. Fetches all your friends (again, 100 at a time)
4. Looks for each of your followers in your friends list
5. Prints each follower that you haven't marked as a friend
6. Adds commons (friends who follow you back) to a list
7. Prints each friend that's not following you
8. Prints the list of commons
   

I'm still a newLISP neophyte, and there are some things I know I could have done much more efficiently and "lispy" but this is about the best I could come up with. I also would normally use much longer lines, but I broke it up so that it would show here without breaking the format of the page too badly. Oh, and mega-props to Code2HTML for colorizing this.

#!/usr/bin/newlisp
# Build an HTTP Basic Auth header for get-url.
# Set 'user and 'pass manually
(set 'user "YourTwitterName" 'pass "NotMyPasswd")
(set 'hedr (append "Authorization: Basic "
(base64-enc (append user ":" pass)) "\r\n\r\n"))

# Grabs followers until results list has less than 100
(set 'page 0) (do-until (< numthese 100) (inc page)
(set 'xml (get-url (append
"http://twitter.com/statuses/followers.xml?page="(string page))
5000 hedr))
(set 'these (find-all "<screen_name>(.*)</screen_name>" xml $1))
(cond ((= page 1) (set 'fol these))
((> page 1) (set 'fol (append fol these))))
(set 'numthese (length these))
)

# Grabs friends until results list has less than 100
(set 'page 0) (do-until (< numthese 100) (inc page)
(set 'xml (get-url (append
"http://twitter.com/statuses/friends.xml?page="(string page))
5000 hedr))
(set 'these (find-all "<screen_name>(.*)</screen_name>" xml $1))
(cond ((= page 1) (set 'fri these))
((> page 1) (set 'fri (append fri these))))
(set 'numthese (length these)))

# Iterate followers against friends list
(dolist (follower fol)
(if (nil? (find follower fri))
(print (append follower " is not a friend\r\n"))
# If follower and friend, push to com list
(if (nil? com) (set 'com (list follower))
(push follower com))))

# Iterate friends against followers list
(dolist (friend fri)
(if (nil? (find friend fol))
(print (append friend " is not following\r\n")) (inc notfol)))

# Print list of common (friend & follow)
(dolist (common com)
(print (append common " is following back.\r\n")))
(exit)

Why'd I do this? Why not use Twitter Karma or some other online tool?

1. I didn't want to enter my username and password into someone else's web app.
2. I wanted a quick programming project that could take up part of my weekend.
3. It was fun.
   

Twitter Followers/Friends from the CLI

I started getting curious on twitter. I had two questions:

1. Who am I following that's not following me back? (i.e. can Martin Roesch hear me? The answer is no, he can't)
2. Who is following me that I'm not following back?

Already familiar enough with the Twitter API, I threw together some quick and ugly command-line foo, and @digitaljestin wanted to know how I did it. This is REALLY ugly, and could use a lot of refinement. That said:

I'll probably program a quick stand-alone newlisp or php tool for this over the weekend. Regardless, here's how I did it on the CLI.

First, Twitter will only hand you 100 friends and followers at once. If I were going to automate this, I would poll the followers_count and following_count attributes from http://twitter.com/users/show/username.xml to figure out how many "pages" I needed to fetch.

If you have 203 followers, you will have to do three requests for follower info. Same with friends (those whom you follow). I had over 200 (but less than 300) each. So I did 3 of each request.

I'm only interested in the screen_name attribute within the XML of each. Note that I'm doing a lot of cheap grep | awk crap here, so it just builds lists of screen names without any markup.

$ wget http://user:password@twitter.com/statuses/followers.xml \

| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' > followers.txt

$ wget http://user:password@twitter.com/statuses/followers.xml\?page=2 \

| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> followers.txt

$ wget http://user:password@twitter.com/statuses/followers.xml\?page=3 \

| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> followers.txt

$ wget http://user:password@twitter.com/statuses/friends.xml \
| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' > friends.txt

$ wget http://user:password@twitter.com/statuses/friends.xml\?page=2 \
| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> friends.txt

$ wget http://user:password@twitter.com/statuses/friends.xml\?page=3\
| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> friends.txt

Then, I just sorted them:
$ sort friends.txt > friends-sort.txt
$ sort followers.txt > followers-sort.txt

Using diff, it's easy to tell who is not following you, and who you aren't following.
The < shows lines that only appear only in the first file (ones you follow only). The > shows lines that only appear only in the second file (ones following you). Grepping for only lines that start with < and > avoids all the patch-file line offset stuff. Some diffs have varying syntax to do this, but letting grep filter it should work across more platforms.

$ diff friends.txt followers.txt | grep "[<>]" | sort
[excerpt]

< H_i_R
< Hak5
< KCWeather
< Scobleizer
< Veronica
< bacontwits
< beseKUre
< brightkite
< datalossdb
< hackadaydotcom
< ihacked
< ihackstuff
< kingpin_
< milw0rm
< mroesch
< obsessable
< om
< packetlife
< pauldotcom
< schneier
< textfiles
< wilw
< window
------------------ (split added by ax0n)
> BlackHatUSA
> Computersaurus
> HacClearwater
> HackersAlerts
> HackerspacesBot
> SOURCEBoston
> SecuritySatan
> quine
> reverz
> rsreese
> secureideas
> securitypro2009
> stopthemanga

IP Subnetting - more fun with newLISP

I decided it would be fun to try to make an IP subnet calculator with newLISP.

Thanks to Elica and Lutz on the newLISP discussion boards. I needed some help with the logic. There's probably a way to compact this code down to about 3 lines somehow, but I'm still a newLISP n00b. I stuck with the logic examples that I have a firm understanding of, but the discussion yielded some interesting results.

Here's what I came up with

#!/usr/bin/newlisp
# newLISP IP Address calculator by ax0n
# ax0n (at) h-i-r.net
(define (iptostr ip4)
# Converts an integer to an IP in dotted decimal notation
(string
  (mod (/ ip4 0x1000000) 0x100) "."
  (mod (/ ip4 0x10000) 0x100) "."
  (mod (/ ip4 0x100) 0x100) "."
  (mod ip4 0x100))
)

(define (iptonum ip4str)
# Converts an IP string to an integer
 (map set '(one two three four) (parse ip4str "."))
 (+ (* 0x1000000 (float one)) (* 0x10000 (float two))
  (* 0x100 (float three)) (float four))
)

(cond(
 (< (length (main-args)) 3)
 # Display usage if no args passed
 (println "usage: ipcalc.lsp ip-address/maskbits")
 (println "ex:    ipcalc.lsp 192.168.1.20/24")
 )
 (true
  (set 'ipstr (last(main-args)))
  (map set '(ipaddr bits) (parse ipstr "/"))
  (set 'binip (iptonum ipaddr))
  (set 'netmask (& 0xffffffff ( << 0xffffffff (- 32 (int bits)))))
  (set 'netaddr (& binip netmask))
  (set 'bcast (& 0xffffffff (| binip (~ netmask))))
  (println "host IP:    " ipaddr )
  (println "netmask:    " (iptostr netmask) )
  (println "network:    " (iptostr netaddr) )
  (println "broadcast:  " (iptostr bcast))
  (println "Host range: " (iptostr (+ netaddr 1))" - "(iptostr (- bcast 1)))
 )
)
(exit)

Running it by itself gives you a syntax help page.

-bash-3.2$ ./ipcalc.lsp
usage: ipcalc.lsp ip-address/maskbits
ex: ipcalc.lsp 192.168.1.20/24

You have to provide an IP Address and mask in CIDR Notation. It does the rest!

-bash-3.2$ ./ipcalc.lsp 192.168.0.49/24
host IP:    192.168.0.49
netmask:    255.255.255.0
network:    192.168.0.0
broadcast:  192.168.0.255
Host range: 192.168.0.1 - 192.168.0.254

You can download the script here:
http://stuff.h-i-r.net/blogstuff/ipcalc.lsp

newLISP OTP implementation

I had some free time, so I came up with something fun to play with.

I can't stress enough the importance of protocol when dealing with cryptography. Leaving no trace of the used portion of the pad nor the cleartext file is essential to maintaining the secrecy of your messages.

Some operating systems let you set flags or attributes on files so that they're automatically secure-erased when deleted. Other operating systems require a tool such as "wipe" or "shred" to completely destroy the files. The "leftover" pad can be re-used for future communications. You can make the pad as large or as small as you wish, but it must be larger than the file you wish to encrypt, and it's up to all parties to keep their pads "in sync" with the others. I didn't say OTP was practical, it's just really good when implemented correctly.

I advocate keeping ALL sensitive files (the pad and cleartext) on a USB flash drive along with a secure-erase tool. If the risk of compromise exists, 5-10 seconds in a microwave will render it and all contents completely useless. Although USB flash drives are solid state, given the nature of their construction, it's been proven that some information may be retrievable from them even after a secure erase. Always use your head.

Now that the formalities are out of the way, let's get started!

First, I created a 5kB file of pseudo-random data to use for my pad, and made a copy of the pad for the recipient.

Chimera:~/test ax0n$ dd if=/dev/urandom of=pad bs=1k count=5
5+0 records in
5+0 records out
5120 bytes transferred in 0.001332 secs (3843715 bytes/sec)

Chimera:~/test ax0n$ cp pad receiver-pad

Next, I launched my script. It requires 4 pieces of information. The pad file, the file to encrypt or decrypt, the output file, and the file to write the leftover portion of the pad to. After running it, you can see the script, the leftover pad, the original pad (which should be secure-erased immediately after use), the encrypted target file (in this case, my /etc/passwd file) and the pristine copy of the original pad which should have been handed (never transmitted!) to my recipient.

Chimera:~/test ax0n$ ./crypt.lsp pad /etc/passwd passwd.crypt leftover-pad

Chimera:~/test ax0n$ ls -la
total 56
drwxr-xr-x    7 ax0n  ax0n   238 Feb  7 22:36 .
drwx------   55 ax0n  ax0n  1870 Feb  7 22:33 ..
-rwxr-xr-x    1 ax0n  ax0n   675 Feb  7 22:33 crypt.lsp
-rw-r--r--    1 ax0n  ax0n  3188 Feb  7 22:36 leftover-pad
-rw-r--r--    1 ax0n  ax0n  5120 Feb  7 22:36 pad
-rw-r--r--    1 ax0n  ax0n  1932 Feb  7 22:36 passwd.crypt
-rw-r--r--    1 ax0n  ax0n  5120 Feb  7 22:36 receiver-pad

Here's part of the hexdump from the encrypted password file. It's very, very random.

Chimera:~/test ax0n$ hexdump -C passwd.crypt
00000000  d3 17 89 23 2f 51 bc 9c  c2 3d d3 b3 fb e0 5c 4b  |...#/Q...=....\K|
00000010  43 8e 4f e7 71 53 c7 fd  2b b2 ff 37 32 64 3a 2d  |C.O.qS..+..72d:-|
00000020  df 8e 19 fa 64 07 c9 8d  f2 36 f0 41 e6 ca 65 67  |....d....6.A..eg|
00000030  a3 a4 e6 b0 dc 48 14 08  12 0d c4 72 1a 18 c6 bc  |.....H.....r....|
00000040  0a cd 79 69 2b f2 62 15  63 48 da f5 3d 36 41 e8  |..yi+.b.cH..=6A.|

...

00000730  4c 71 17 04 13 50 d2 e0  ab 3a 2d 44 16 0d 72 cf  |Lq...P...:-D..r.|
00000740  60 47 16 43 8a 1f 73 03  e8 e9 b8 71 d6 ee fd 68  |`G.C..s....q...h|
00000750  09 dd 81 60 35 65 0b 8f  66 03 97 f4 96 39 78 44  |...`5e..f....9xD|
00000760  f8 24 99 bc 68 6e 28 14  f3 2f fc 0a a3 68 47 70  |.$..hn(../...hGp|
00000770  fd 41 fc 51 e2 c2 0c de  07 be 40 ce 6c a8 bb 5b  |.A.Q......@.l..[|
00000780  88 4b 52 50 a4 95 b5 37  71 12 12 76              |.KRP...7q..v|

On the receiving end, we use the receiver-pad file against the encrypted password file, then we can see the new files with ls.

Chimera:~/test ax0n$ ./crypt.lsp receiver-pad  passwd.crypt passwd.clear reciever-leftover-pad

Chimera:~/test ax0n$ ls -la
total 72
drwxr-xr-x    9 ax0n  ax0n   306 Feb  7 22:38 .
drwx------   55 ax0n  ax0n  1870 Feb  7 22:33 ..
-rwxr-xr-x    1 ax0n  ax0n   675 Feb  7 22:33 crypt.lsp
-rw-r--r--    1 ax0n  ax0n  3188 Feb  7 22:36 leftover-pad
-rw-r--r--    1 ax0n  ax0n  5120 Feb  7 22:36 pad
-rw-r--r--    1 ax0n  ax0n  1932 Feb  7 22:38 passwd.clear
-rw-r--r--    1 ax0n  ax0n  1932 Feb  7 22:36 passwd.crypt
-rw-r--r--    1 ax0n  ax0n  3188 Feb  7 22:38 receiver-leftover-pad
-rw-r--r--    1 ax0n  ax0n  5120 Feb  7 22:36 receiver-pad

Let's see if the decrypted file is legible...

Chimera:~/test ax0n$ cat passwd.clear
##
# User Database
#
# Note that this file is consulted when the system is running in single-user
# mode.  At other times this information is handled by one or more of:
# lookupd DirectoryServices
# By default, lookupd gets information from NetInfo, so this file will
# not be consulted unless you have changed lookupd's configuration.
# This file is used while in single user mode.
#
# To use this file for normal authentication, you may enable it with
# /Applications/Utilities/Directory Access.
##
nobody:*:-2:-2:Unprivileged User:/:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
daemon:*:1:1:System Services:/var/root:/usr/bin/false

...

tokend:*:91:91:Token Daemon:/var/empty:/usr/bin/false
securityagent:*:92:92:SecurityAgent:/var/empty:/usr/bin/false
unknown:*:99:99:Unknown User:/var/empty:/usr/bin/false

Yep, it worked!

Let's look at the newLISP code for this project. On the newLISP discussion boards, cormullion and Lutz (the founder of newLISP) reminded me of the beauty of the "cond" command, which works a little bit like "case" except it evaluates whole expressions, not just one variable. I added a bit of whitespace into the code to show how "cond" works. They also helped me re-factor my argument assignment statements down to one line with "map".

#!/usr/bin/newlisp
(cond(
  (< (length (main-args)) 5)
      (println "USAGE: crypt.lsp [pad] [file] [output] [pad-remainder]")
  )    
  (true
    (map set '(pad target output remainder) (rest (rest (main-args))))      
    (write-file output (encrypt (read-file target) (read-file pad))) 
    (write-file remainder (slice (read-file pad) (length (read-file target))))
  )
) 
(exit) 

Instead of taking the painstaking route of stepping through all the logic to show how XOR encryption works like I did in my cryptography example a few days ago, I am using as many newLISP shortcuts as I know how to while maintaining the feature set I want. For instance, newLISP has the "encrypt" function which is exactly what I was using in my original example. It runs a bitwise XOR of the contents of two variables, even if those variables contain the entire contents of files! There's no need to explode strings, map the char command, or anything like that. In the above example, newLISP essentially handles all the core logic in this single line:

(write-file output (encrypt (read-file target) (read-file pad)))

It single-handedly reads the pad and target file, XORs them with the encrypt function and writes the resulting garble to the output file.

The line below it figures the length of the target file, skips over that many bytes of the pad file and writes the remaining bytes in the pad out to the remainder file.

Look for more articles like this as I keep playing with newLISP and learning more about it.

Cryptanalysis and histograms

I've been interested in cryptanalysis and encryption for a long time. I can remember using PGP in the early 90's, my little Zenith laptop taking literally hours to create puny 256-bit keys, and being utterly fascinated by the whole process.

One way that cryptography is analyzed is letter frequency counting. There are certain characters that show up more frequently than others depending on what natural language is used. In letter substitution and other simple encryption schemes, creating a histogram of some random samples of that language (for my sake, we'll assume this is English), one can take that and lay it over the histogram of ciphertext, finding letters that are more common in the ciphertext and in the language sample. Analysts begin substituting letters that peak the charts, and eventually may come up with a partial decryption that's complete enough to go trial-and-error.

Histograms are just neat, though. Histograms of color data in photos are cool, too. Loosely speaking, histograms are bar graphs of frequency data.

Here is a histogram of the first chapter of Genesis (KJV):


On the far right are control characters such as carriage returns. The big vertical bar is the frequency count of spaces. To the left of that are numbers, then capital letters, punctuation, and finally lowercase letters toward the middle of the histogram. It's easy to tell that this is a text file from the histogram.

Next I'll show you a binary executable file, next to a GIF image. You can see these are binary files with a lot of "nulls" on the far right, and lots of high-bit characters on the left. They look nothing like a text file.



The beauty of a really good random one-time pad as I'd discussed in Wednesday's post becomes evident. On the left is a histogram of a really good random pad file. On the right is the first chapter of genesis (used in our first example) XORed against the random pad.



The entropy is evident!

I have an implementation of the newLISP one-time pad tool that I'll write about this weekend.

I used PHP/GD to create the histograms used in this article. Here's the quick-and-dirty histogram generator code I made:

<?php
####
# 256x256 byte frequency histogram maker
# by ax0n
###
header("Content-type: image/png"); // Tell browser to expect a PNG
if(! isset($_GET['file'])){
$filename="data";
}else{
$filename=$_GET['file'];
}
$im=imagecreate(256,256); // allocate image resource
$background_color = imagecolorallocate($im, 0, 0, 0);
$text_color = imagecolorallocate($im, 233, 14, 91);
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
$histarray=(count_chars($contents, 1)); // built the frequency array
foreach ($histarray as $i => $val) {
$graph=($val/max($histarray))*256; // Normalize the values
imageline($im,$i,1,$i,$graph,$text_color); // Draw chart
}
$im=imagerotate($im, 180, 0); // make bars go from down to up (cheap hack)
imagepng($im); // dump image to browser
imagedestroy($im); // release image resource
?>

Cryptography fun with newLISP

Some of you may have seen my article in the latest 2600 magazine about newLISP, a very fast scripting language based on LISP.

Well, the HiR crew has, for the better part of a year, been kicking around cryptography concepts. We keep coming back to one-time pad cryptography. It's fascinating in its simplicity. It's so computationally trivial that a human can quickly encrypt or decrypt a simple modular addition one-time pad scheme with nothing but a pencil and paper. The other thing is that, when implemented properly, OTP is perfectly secret and highly resistant to all forms of cryptanalysis. To this day, OTP is one of the most powerful and feared crypto schemes in existance.

Back here in computer world, we do a comparison on two numeric values assigned to the "cleartext" message. Since all ASCII characters have a decimal value in an 8-bit character space (256 distinct combinations), it's very easy to perform a bitwise Exclusive Or (XOR) on the cleartext and the key. XOR is simply "one or the other but not both". Let's try a cleartext of the letter A (binary 01000001) XOR against a key of "q" (binary 01110001). Comparing the bits vertically, the result will only have a "1" wherever either A or q have a differing bit at that location, but will have a "0" wherever both bits are the same.

"A"=01000001
"q"=01110001
-----00110000 (as it turns out, this is an ASCII uppercase "O")

newLISP, like all LISP based languages, handles lists and symbols very well. I just played around with this concept on the newLISP command line, and the result was fun, but not very practical as executed. I'll discuss ways to make this tinkering session a little more practical at the end of the article.

Ideally, the key would not be a string, but a very solid random set of characters known by the parties who are encrypting messages to one another. This doesn't even need to be a string, it may be binary random data...

A few things about this code. First, I'm a newbie at newLISP. Let me describe how some of this stuff works. The "explode" command turns a string variable into a list of characters. "char" turns a character into its ASCII decimal equivalent, or an ASCII decimal number back into a character. "map" simply takes the operation (such as "char") and uses that operation on a list. So (map char (explode cleartext)) would return the output of "char" for each individual character in the contents of the "cleartext" variable. Confusing enough? Okay, awesome!

---

newLISP code that I type will be in bold, red italics.
newLISP output will be in bold.
My comments will be in italics.

---

# newlisp
newLISP v.9.0 on OSX UTF-8, execute 'newlisp -h' for more info.

#first, I'll assign a key and convert it to a list of ascii numbers.
> (set 'key "Pa5$w0rd!")
"Pa5$w0rd!"

> (set 'kcharlist (map char (explode key)))
(80 97 53 36 119 48 114 100 33)

#Then, I'll assign a cleartext string and convert it to a list of ascii numbers as well.
> (set 'cleartext "HiR ownz.")
"HiR ownz."

> (set 'ccharlist (map char (explode cleartext)))
(72 105 82 32 111 119 110 122 46)

#In many languages, newLISP included, the carat "^" is the symbol for XOR. This line outputs the XOR values for each character compared between the cleartext and key)
> (set 'cryptcharlist (map ^ ccharlist kcharlist))
(24 8 103 4 24 71 28 30 15)

#Note that some of the characters from the XOR operation are non-printable, so their ASCII string notation /nnn is used instead.
> (set 'cryptostring (join (map char cryptcharlist)))
"\024\008g\004\024G\028\030\015"

#Now on the receiving end, we take the known key character list and we XOR the crypto character list. In reality we'd have to generate these again but we'd already set the variables above. No sense repeating the process here.
> (set 'decryptcharlist (map ^ cryptcharlist kcharlist))
(72 105 82 32 111 119 110 122 46)

#Finally, we take that string of numbers and join them after converting them from ASCII Decimal to characters again.
> (set 'decryptstring (join (map char decryptcharlist)))
"HiR ownz."

---

Now, as I'd mentioned before, in order to be really practical, you'd need an actual one-time pad that was very random. What I made above was a variant of Vernam cipher to give you a taste of how simple XOR based encryption is, not a genuine one-time pad. A few things that would make the above more practical:

• A lot of the steps could have been consolidated. I broke it down to show how things worked. I could have simply put all of the logic in one line, but it would have been confusing.
• The key and cleartext should be able to be read from a file. As implemented above, this should work with binary data (executables, images, etc) and with a very random binary key (pad) file.
• In the code above, the key needs to be the exact same number of characters as the cleartext. If implemented practically, the program should read the length of the cleartext and use that many bytes of the random pad.
• In a true one-time pad scheme, the sender and recipient both possess the same random key data, but the sender must destroy the part of the key that was used as soon as the encryption is performed. That way, only the recipient(s) may decrypt the message, as they have the only existing copy of the key. In turn, the recipient must destroy the part of the key used to decrypt the message so that in the event the encrypted message was compromised, there is no existing copy of the key available to aid in decryption.
• The above inconveniences have been the main downfall to widespread use of OTP.
• OTP's strength relies heavily on protocol, not technology.

As I get more free time, I will write a quick one-time pad newLISP script that can do most of the above things. I just got the craving to tinker and write a little bit, and thought I'd share it here.

--Ax0n