Macinstruct Code Mojo

Create Sitemaps with SiteOrbiter

mcone — Wed, 06 Feb 2008 21:44:56 -0700

Janet Fouts
Thursday, February 7, 2007

Websites are getting larger and larger, and keeping track of them as they evolve can be a hassle. If you're a web developer, you'll want to create what's called a site map, a special file that lists all of the pages of your website in an hierarchical order. This file can be submitted to search engines to help them index your website. But how the heck are you supposed to create one on your Mac?

Enter a native Mac application called SiteOrbiter. SiteOrbiter is freeware (registration required) and requires Mac OS 10.1 or later. This application will index your files, create an XML site map for Google and other search engines, and also create a visual sitemap that can be saved as a PDF.

Here's how to do it:

Download SiteOrbiter from http://www.siteorbiter.cc/index.html. To install the application, open the disk image and drag SiteOrbiter to your Applications folder.
When you open SiteOrbiter for the first time, you'll be greeted by the SiteOrbiter Assistant. Enter the URL of the website you'd like to index or, if it's located on your Mac's hard drive, click the Browse button and select the top level directory.
SiteOrbiter will start indexing your website. This could take a while, especially if you have a large website. You'll see files flashing by in the status display in the upper-left corner of the screen (below).
Once SiteOrbiter has finished indexing your website, you'll see a graphical representation of your website in the main window.
Save your sitemap for search engines by selecting Save from the File menu. You can also export the graphical representation of your website by selecting "Extract to PDF" from the File menu.
You can also export the data as a sitemap for Google by selecting "Generate Site Map" from the Reports and Maps menu.

Taking Another Look at the SiteOrbiter Assistant

Now that you've created your very own sitemap with SiteOrbiter, it's time to take a look under SiteOrbiter's hood. We'll start by reexamining the SiteOrbiter Assistant. What do all of those options actually do?

User Agent: This menu is especially interesting. You can appear to the server to be using any agent you like. If, for example, your site is designed to deliver different dynamic data to different browsers, you can generate a site map of what each of those browsers are being delivered and check for accuracy of your sniffer scripts.
Maximum Recursive Depth: This option allows you decide how deep to decend into the bowels of the website. If you're not sure, start with 8 or 10 and see if a lot of pages are missing. If so, you can adjust as necessary.
Copy Remote Site: You can also use SiteOrbiter to download a remote website to your Mac's hard drive. Use this cautiously. It can take a while to download, and if the site isn't yours please don't violate any copyrights!

Configuring SiteOrbiter's Preferences

The settings you configure in SiteOrbiter's preferences can have a huge effect on the size of your sitemap. Here are some of the options you need to know about:

Download Folder:Select the folder you want to use as an archive.
Note: If you download a site, and you move the files, SiteOrbiter will download the entire site again when you next request a download. If not, SiteOrbiter will only download files that have been modified.
Only Include:You can set this to include, for example, only HTML, PHP or maybe just SWF files.
Treat Links with Arguments: Each argument will be recorded as a seperate page. (I.e.: http://www.mysite.com/index.pl?today and http://www.mysite.com/index.pl?tomorrow would each be separate pages.)

Understanding SiteOrbiter's Toolbar

You know what we're talking about. What exactly is that unholy mass of buttons on SiteOrbiter's main window?

Index/Stop: Clicking this will re-index your website or, if SiteOrbiter is currently re-indexing, stop the re-indexing process.
Browser: Highlight a page and click Browser to display that page in a browser window.
Up, Left, Down, Right: Moves the map around in the window. You can also use the scrollbars.
Hide Map: For a large site, redrawing the map can be a processor hog, so you can turn it off to improve performance.
Download: Again, please be judicious. This can be a boon when you are taking over a website, and you'll have the site map as well. Once you've downloaded the site you can use this feature to keep your disk copy up to date. SiteOrbiter will only download differences after the initial download.
Transcript: This shows header information. Useful with dynamic pages and scripts.
Only HTML: Ignores non-HTML files in the map view.
Select Lock: You can lock a node to keep it open even as you click on other nodes. Think of it as a placeholder.
Names: Turned off by default, this will show the names associated with the nodes.
Selected: Shows just the info for the selected node.
Zoom: Zoom out for a macro view or in on a particular node.

Using the Map Window

SiteOrbiter's real power becomes apparent when you start manipulating your sitemap. There are several view options that will help you view your website.

You can view all of the objects in the map, or a subset. Subset is the default setting. To switch views, just click the display mode link in the lower right hand corner. You can select from Tree, Concentric Circles, Subset or lower left. You can also click and drag nodes on the map to better organize them.

Subset mode: Clicking on a link in the map window will select that link and you can then view any links related to that "node." This makes it simple to quickly see the relationships between objects in the site, and you'll see outbound and inbound links.
All Objects: This display mode can be quite daunting on a larger site, but it makes it easy to spot orphaned pages and get a very high level view of the site. The home page is always identified as the starting object, but you can use "show from" mode to select a particular node and show the site from that viewpoint.

Conclusion

That's pretty much it. All in all, SiteOrbiter is a useful tool for getting a grip on the loose ends that may be lolling around in your website, and it can also give you a lot of insight into who's linking to your site and what pages they are linking to. Combine that with your traffic stats and you'll have a much better picture of what your users are up to!

Meet Your Macinstructor

Janet Fouts is a web developer and designer who provides tech support for a bevy of clients and friends who need a little help with their Macs. In Code Mojo she will be sharing her web development tools and tips to make your life a bit easier. You can contact Janet at macinstruct@tatudigital.com.

Set Up a Wireless Captive Portal Server

mcone — Wed, 25 Jul 2007 01:42:21 -0600

David Miller
Wednesday, July 25, 2007

What does a captive portal server, also called a NAC (Network Access Control) do? It can sandbox any wireless connection until some form of authentication is provided. These servers are used in many cafes and public places that offer wireless internet. For example, when you try to connect to the wireless network at Starbucks, it will force your web browser to the same page - no matter what URL you enter. Until you authenticate, you can't go anywhere on the Internet.

There are some great commercial solutions out there that work really well, but implementing 802.1X port security can cost lots of money and time. Companies like Infoblox provide an appliance, and when coupled with solutions like Bigfix can provide a very robust and transparent network security model. Pitney Bowes has implemented a system were an unknown user can attach their laptop and get one of two services: Either simple port 80 access for 8 hours, or the option to register and automatically have their computer scanned patched/updated to comply with the companies policies. This is all transparent.

So what do you do if you don't have the resources or the money to buy one of these solutions? You can still implement some control over DHCP assignment. The goal is to eliminate the unknown on your network. In our case, we will use the Ethernet MAC address of the client's NIC as a way of authentication. Each network interface card has its own unique address, and it looks something like this: 00:30:65:88:01:93. When an unknown computer is placed on the network that asks for an IP through DHCP it will receive one that is restricted. No matter what web page the user tries, we will always show them a pre-defined page. When a computer that has its MAC address defined on the server asks for an IP, it will be given the correct TCP/IP settings to use the network.

Note: This is not secure, and will not prevent someone from stealing a working IP on your network. If the user is savvy enough they can find the TCP/IP info on another computer and use it. If you are looking for that kind of security, then this article is not enough for you. Check out Infoblox or another commercial solution.

Here's how to do it:

Make sure the Mac you are using has all os updates and security patches applied.
From the Apple menu, select System Preferences. Select Network. Turn off all unused configurations. The interface we will be using is the Built-in Ethernet.
Select Built-in Ethernet and click Duplicate. Click Rename and add your dummy subnet.
Edit the interfaces to have the correct subnets.
Make sure you are logged in as an admin to use the sudo command. You will need to use a text editor like VI, Pico, Emacs, or TextWrangler to create and edit the config files needed.
Download the latest version of DHCP from http://www.isc.org/index.pl?/sw/dhcp/. The latest version at the time of this article is 3.0.6. You will need to have the GCC compiler installed. Install the Xcode developer software that came with your Mac or latest Mac OS X install disks.
Once you have downloaded the DHCP source, double-click on the .gz file to unzip and untar the directory. Launch the Terminal application (located in /Applications/Utilities). Use the "cd" command to change to the dhcp directory you just expanded. The easiest way is to type "cd" and drag and drop the dhcp directory into the Terminal window. Here's an example:
cd /Users/dmc/Desktop/Downloads/dhcp-3.0.6/
Now type the following and hit return.
./configure
Type the following and hit return.
make
Type the following and hit return.
sudo make install

Setting Up DHCP

You now should have a working DHCPd package installed on the Mac. The next step is to set up the dhcpd.conf file that lives in /etc. The below example will do just that.

Here's how the thing works:

Hand out static IPs in the 10.0.1.0 range to hosts that we have the Ethernet hardware address of (MAC address).
Hand out dummy IPs in the 192.168.50.0 range that will force all host names to resolve back to our captive portal server. This will force any http requests to show the page we want people to see.

You should read the man page for dhcpd.conf to gain an understanding of the options and commands used here.

======================begin dhcpd.conf=========================== # dhcpd.conf # this file should be located in /etc default-lease-time 900; ddns-update-style none; deny client-updates;


shared-network "mynet" {

# This is the dummy network

subnet 192.168.50.0 netmask 255.255.255.0 {

range 192.168.50.2 192.168.50.253;

max-lease-time 7200;

option subnet-mask 255.255.255.0;

option broadcast-address 192.168.50.255;

option domain-name-servers 192.168.50.1;

option routers 192.168.50.1;

}
# This is the real network

subnet 10.0.1.0 netmask 255.255.255.0 {

max-lease-time 21600;

option subnet-mask 255.255.255.0;

option broadcast-address 10.0.1.255;

option routers 10.0.1.1;

option domain-name-servers 10.0.1.251;

option domain-name "example.com";

}

# This is a sample reserved host entry

host example1 {

hardware ethernet 00:30:65:88:01:93;

fixed-address 10.0.1.5;

}

host example2 { hardware ethernet 00:16:cb:91:16:93; fixed-address 10.0.1.6; } } ======================end dhcpd.conf===========================

Create the dhcpd.leases file using the following command:
sudo touch /var/db/dhcpd.leases
To test the the DHCP server, run the following command in a Terminal window:
sudo /usr/sbin/dhcpd -f -d en0

The above command runs DHCPd in the foreground using the -f option and in debug mode using the -d option. This allows you to see all actions the DHCP server is performing. Populate the dhcpd.conf with the MAC addresses of the hosts you want.
Alternatively, you can also assign random IPs from the real network if you don't care who gets what IP. To do this, leave out the the "fixed-address" line of the reserved host. in the real network section add a line to deny unknown clients and add a range of addresses to hand out like so:

} # This is the real network subnet 10.0.1.0 netmask 255.255.255.0 { max-lease-time 21600; range 10.0.1.2 10.0.1.250 deny unknown-clients; option subnet-mask 255.255.255.0; option broadcast-address 10.0.1.255; option routers 10.0.1.1; option domain-name-servers 10.0.1.251; option domain-name "example.com"; }
Here is a launchdaemon plist file you can use to launch dhcpd at boot. Place it in /Library/LaunchDaemons.
To load the file into launchd, type the following command into Terminal:

/bin/launchctl load -w /Library/LaunchDaemons/org.isc.dhcpd.plist

To unload and keep dhcpd from launching at boot, type the following command into Terminal:

/bin/launchctl unload -w /Library/LaunchDaemons/org.isc.dhcpd.plist

Set Up the Dummy DNS Server

First create a file in /etc called rndc.key. This file will keep BIND from complaining that it is missing. Since you will not be updating, the DNS we can use a generic file and not include anything in the named.conf.
======================begin rndc.key=========================== key "rndc-key" { algorithm hmac-md5; secret "8E48raKxqEVCtKYFxA+loQ=="; };
======================end rndc.key===========================
You will need to create the following named.conf file in /etc:
======================begin named.conf===========================
acl dummynets { 192.168.50.0/24; 192.168.51.0/24; }; options { directory "/var/named"; auth-nxdomain no; # conform to RFC1035 notify no; allow-query { dummynets; }; allow-recursion { none; }; allow-transfer { none; }; max-ncache-ttl 60; }; logging { category default { _default_log; }; channel _default_log { file "/Library/Logs/named.log"; severity info; print-time yes; }; }; zone "." IN { type master; file "db.fakeroot"; allow-query { dummynets; }; };
======================end named.conf===========================

The above says to use db.fakeroot as the root dns zone file. The "acl dummynets { 192.168.50.0/24; 192.168.51.0/24; };" tells that only our dummy subnet can request queries.
Now we need to create the db.fakeroot zone file in /var/named.
======================begin db.fakeroot===========================
$TTL 7200 @ IN SOA hostname.exapmle.com user.example.com ( 1 ; Serial 3600 ; Refresh every 1 hours 1800 ; Retry every 30 minutes 604800 ; Expire after 7 days 1 ) ; TTL 1 second IN NS 192.168.50.1 IN A 192.168.50.1 * IN A 192.168.50.1
======================end db.fakeroot===========================

The above zone file resolves all hostnames back to 192.168.50.1
Apple has included a launch daemon plist that can be used to start and stop the named service. Unfortunately, it doesn’t work, because it launches named before all the network stuff is loaded at boot. Until we get a fix, we will use the old style startup item. Download this file, unzip it and place the BIND folder in /Library/Startupitems/. This will make sure named is launched at startup.
Test the DNS service by typing the following command on a host computer that has been given one of the dummy IPs:
host www.cnn.com

It should resolve to 192.168.50.1

Set Up Apache to Redirect Missing Paths

The line you want to edit in your httpd.conf file looks like the following:

#ErrorDocument 404 /missing.html

Change it to:

ErrorDocument 404 /

Create a simple index.html page that tells the user to contact you for proper access and save it to /Library/webserver/Documents. Turn on personal websharing in the sharing system prefs pane to start Apache.

You can get pretty creative using CGI and have users fill out a form that gives you their information.

Meet Your Macinstructor

David Miller has been supporting Macs since the mid '90s. He has worked for large ISPs and DOE sites, and he's currently working as a Unix systems administrator. He has graciously allowed Macinstruct to reprint this tutorial, which was originally written for his personal website: http://www.sdmachelp.com/

Create a Web Development Environment With MAMP

mcone — Wed, 18 Jul 2007 02:30:16 -0600

Matthew Cone
Wednesday, July 18, 2007

Those of us who have websites know that we need a development environment to work on new features, make changes, and just experiment. If you're creating static HTML pages, you don't need much - you can create the pages on your Mac and preview them locally in your web browser or a with WYSIWYG application like iWeb. However, things are a bit more complicated if you're working with web applications that use PHP and MySQL.

In that case, you're going to want to use MAMP, a free web development environment for your Mac. MAMP comes with everything you need, including Apache, PHP, MySQL, and even the buzzword-compliant acronym. (MAMP, by the way, stands for Mac, Apache, MySQL, and PHP.) It's not suitable for a production web server, but it's perfect for your home Mac.

We'll show you how to install and configure MAMP on your Mac. To spice things up a little, we'll also install Drupal, an open source content management system (CMS) that powers the likes of The Onion, TWiT.tv, Mac|Life, and yes, even Macinstruct.

Install and Configure MAMP

Installing MAMP is a pretty straightforward process. The installer is graphical, and MAMP keeps itself contained to a single folder in your Applications folder, which is really nice. Here's how to install:

Download MAMP from living-e. It could take a bit longer to download than other applications - our MAMP download weighed in at a hefty 109 MB.
Double-click on the MAMP disk image to mount it on your Mac. Drag the MAMP folder to your Applications folder. Leave MAMP in your Applications folder and don't nest it in other folders. (The server applications require this.)
Launch MAMP by double-clicking the MAMP icon in the MAMP folder. Once the MAMP window appears, click Start Servers to get everything going. When you see green lights, you know you're good to go.
MAMP will open a web page in your default web browser. This web page contains important information! Do not close this web page until you have jotted down all of the information you need. The page also contains links to phpMyAdmin and SQLiteManager.
That's it! Now you can start installing your web site. (Applications > MAMP > htpdocs) Your website can be found at: http://localhost:8888.

Install and Configure Drupal

Now that we have MAMP working, we need to put it to use. You can host anything you want - content management systems such as WordPress will work perfectly. We're going to use Drupal as an example. Here's how you can install it, too:

Download Drupal from the official website.
Uncompress the zip file in: Applications > MAMP > htdocs. Cutting and pasting all of the Drupal files into the home directory is a good idea. To do this, take all of the files in the Drupal folder (ours is called "drupal-5.1") and drag them to the htdocs folder.
With your web browser, visit http://localhost:8888. If everything worked, you should see a web page like the one below.
Set the database type to mysql. In the Database name box, type test. We could create a new database, but it's easier just to use test - a database MAMP creates by default. The database username and password is root. If everything works, you'll see a Drupal installation complete message on the next page.
We'd like to say that you're finished, but you're actually just getting started. Customizing Drupal can be a long, painstaking task. Click here for a wonderful tutorial on using Drupal.

XAMPP: The MAMP Alternative

If you don't like MAMP, be sure to try XAMPP, another free Mac web development environment. We don't think XAMPP is quite as polished as MAMP, but it does have a number of features that MAMP doesn't, so if you're a hardcore tinkerer, you better check out XAMPP!

Meet Your Macinstructor

Matthew Cone is a technical writer living and working in Albuquerque, New Mexico. In his free time, he does the desert rat thing and hikes and road bikes around the Southwest. The rest of the time, he studies straw-bale houses, reads Anarchist philosophy, and pretends to not be working. You can email him at: matt@macinstruct.com

Software Piracy: Black Beard & Captain Kidd!

mcone — Wed, 07 Mar 2007 01:49:49 -0700

Joe Wilkins
Wednesday, March 7, 2007

In keeping with a recent article about piracy in the Runtime Revolution Newsletter, I've chosen to reprint an article I wrote in 2000 for the original Macinstruct website. It is still quite relevant and on target. Last week's Code Mojo article presented me with significant issues - more than I had originally thought - but I will be back next week reviewing some of the scripting that was done in the Coloring Book application.
-Joe

Real or fictional? Sinister or comical? Pirates are no joke! They do serious harm! An act of piracy is committed when someone takes anything that isn't theirs and uses it for their own purposes. It seems different than outright stealing; but, for the life of me, I don't see how or why! Once upon a time, such acts were performed only on the high seas; while today's pirates perform theirs over the Internet, not the ocean waves.

The modern pirate is fettered with neither eye patches nor peg-legs, though the eyes do blur at times and the legs go to sleep from lack of use. Using computer software without a proper, valid authorization, a license, is an act of piracy. Software pirates are every bit as bad as those who hold up banks or liquor stores at gun point. In some respects they are even worse, since they rarely get caught, are rarely prosecuted, yet do severe, real damage to the lives of other people.

Why Am I Talking About Piracy on Macinstruct?

Because new Mac users frequent this site to learn about their Macs, and the use of Mac software, and that is what I am going to discuss. Piracy is but one of several important considerations to which many users are never adequately exposed. At this time I will cover copyrights and piracy. New, as you may be to the world of Mac software, you have most likely been exposed to situations in which you should have read a license agreement before clicking on a product installer's "I Agree" button. If you are like most of us, you maybe read the first one you encountered, but have assumed that all of the others are "exactly" the same. That may not necessarily be the case.

Freeware

Though there is no need to Register this type of software, it is a good idea to do so, since you may get a break on new versions if the owner chooses to change to shareware or commercial releases. You may, at least, be notified when new, free versions are issued. This software's license probably comes with some limitations, so read the terms of the agreement. The usual limitations are that you may not sell it to anyone and, when you give it to someone, you must include all of the items that come with it - such as "Read Me" files. So, it is still possible to be guilty of pirating free software. You must obey the terms of your free license. In some cases you will not be allowed to include the product on software collection CDs that are sold for profit, but an email to the owner will usually get you permission to do so, as long as the proper recognition is provided and you send them a free copy of your CD. After all, "exposure and distribution" is the name of the game, and we all need as much of it as we can get.

Shareware

When you pay the shareware fee, usually less than $25US, you will be registered and licensed by the owner of the product's copyright, and often entitled to subsequent upgrades for free, or to minor upgrades for free, and major upgrades for a reduced fee. Again, read the terms of the agreement. Once more, even though you will actually be encouraged to pass the product along to your friends and associates, there are usually some limitations as to the manner in which you may do this. The same limitations I mentioned for freeware distribution normally apply to shareware as well. If you do not pay the shareware fee, and you use the product contrary to the licensing agreement, you have pirated the software. Most shareware permits you to use the product for some specified period of time to determine whether you wish to have a permanent license, or not. If you plan to leave it on your hard drive, you should pay the shareware fee - even if you only use it once in a blue moon, or on every February 29th.

Commercial Software

In most instances, you will pay for the product before you receive it and you may choose not to register with the owner if that is your pleasure, but I would again recommend that you do so.

It is pretty difficult to "accidentally" pirate this kind of software. If a friend "loans" you a copy of one of their software products; or, heaven forbid, "sells" you a copy without providing everything that they acquired when purchasing it, and you do anything more than try it out - which I think is reasonable, though it may be against the letter of the licensee's agreement, then you are using a pirated copy.

Almost any reasonable software publisher will want you to demo their products with a friend, even to the extent of "loaning" them a copy, so long as they are told in no uncertain terms that they must purchase their own license if they decide to keep the product on their hard drive. I have mixed feelings about someone keeping a copy around on some removable media - just in case they eventually decide to license it. From a strictly legal point of view, I believe that would be piracy as well, but I think you may have to search your own conscience on this one. My feeling - just feeling mind you - is that if you produce something a second time using a product, then you must license it regardless of how you have it "hanging around."

Copy Protection

First, the disclaimer. I am not an expert on copyright law, and have no training in that area. I do not pretend to offer legal advice on that or related subjects, though the reader may feel that I have and do. There are many views on these subjects. You might want to do a search for "copyrights" to check out the many diverse opinions regarding both copyrights and piracy. You will also find an interesting topic called "copylefts."

Regardless of how a product is distributed, all software was created by someone and is the intellectual property of the entity proclaiming the copyright on and/or in their software, or other work. It is not required that the item, claimed as the copyrighted property of that entity, be registered; but, in the case of disputes, registration makes the copyright entitlement much easier to prove, and subsequently enforced if need be. Registering a copyright does not need to be an expensive process handled by an attorney. You can do it yourself by completing the appropriate forms and paying the required, nominal fee.

In Conclusion

We like to think of the Macintosh community as somehow "better" than those who only use PCs. In many respects this has been demonstrated by the paucity of viruses to be found infecting our Macintoshes. For the most part, I think you will also find that Mac users are more inclined to pay their rightful shareware fees. I would like to think this to be true about the larger act of piracy as well - that we do it much less. It would be terrific if we didn't do it at all!

It is unfortunate when some software developers have to resort to exotic registration schemes and/or hardware "dongles" to protect their intellectual property - and it is still theirs. We are only allowed to borrow it for a specified period of time, and under very specific conditions. Taking these extra measures makes their software more difficult to use, and more expensive to produce for all of us.

Reward those who produce exceptional software at reasonable prices. Pay the shareware fees, register your acquisitions, adhere to the conditions of the licensing agreements, and provide the owner with intelligent feedback so that they may improve their products and make them even more useful for us. Promote the best software among your friends and associates; let them know when you've received excellent support. Be just as vocal when you find someone using unauthorized software. It can be done in such a manner as to point out the problem, without embarrassing them. Sometimes, a mere reference to the fact that you received a discount on your last upgrade, because you had registered yours, will stimulate them into doing the same. Always assume that it was an oversight on their part, but never ignore it completely. We all have a major investment in time and money in the best personal computer "system" to be found. Let's make sure that it stays healthy from top to bottom by doing our part. We may not like the law, but it is the law - for now.

Meet Your Macinstructor

Joe Wilkins is a licensed architect who produces all of his work on Macs (yes, even when all he had to work with was a Mac Plus, floppy disks and a wide carriage dot-matrix printer). He has produced his own "Picture" fonts, programmed 5 commercial applications, and chalked up more than twenty-two years of Mac experience - starting with the Lisa. He also authored the "University of HyperCard" on the original Macinstruct website. Email him at mepepetoo@macinstruct.com