|
ZPop
|
read my profile
sign my guestbook
Name: larry
Country: United States
State: Massachusetts
Interests: Sending jokes to Zach. Being nosey. Walking Brewster. Music.
Expertise: Being proud of my kids. It's a tough job, but someone's gotta do it.
Occupation: Research and development
Industry: Research
Message: message meEmail: email me
Website: visit my website
Member Since:
1/19/2002
|
|
| Platoons of Little White PeopleSome of the better off people in my neighborhood, like the guy who appears in a popular public TV production (and the hottie across and slightly down the street from him), can afford to hire "landscapers" to take care of their yards. Thus on a regular basis, Brewster and I, who spend quite a bit of time walking around the neighborhood, checking out the sights and smells, see platoons of little brown people scurrying around these people's lawns.
Yes, I know that sounds a bit racist. I also know that I have a son who has worked as a landscaper, so the stereotype isn't completely true. None-the-less, the vast majority of landscapers in my neighborhood are somewhat more swarthy than your average Anglo Saxon person, whose family has lived in the U.S. for half a dozen generations...or more. As nearly as I can tell, they aren't speaking English to each other either. I have no problem with that. That's just how things are in the suburbs north of Boston.
So it was quite interesting to me when YSOP assigned my mission trip group to be a landscape crew one day. After a long subway ride out from the center of Washington, D.C., and a bit of a walk, we found ourselves in front of a building managed by Mr. Barry. He has a last name, but I forget it. He likes to be called Barry, but, given that he's clearly the boss of his domain, he is "Mr. Barry" to us lesser folks.
Mr. Barry manages a transition home, Milestone Place, housing about 35 people for an organization called Community Family Life Service. He is quite strict with them: no loud noise; no drugs; no alcohol; no overnight visitors; clean up after yourself in common areas; etc. Mr. Barry doesn't take crap from anyone. If you leave your dirty dishes in the sink, figuring you'll do them later, you'll find them in the dumpster.
But, Mr. Barry had a problem. He doesn't have much of a budget for upkeep. If you don't keep a place up, the tenants soon get slack and things go to the dogs...so to speak. Mr. Barry, however, is a very smart man. He has found a solution to his problem: volunteers.
Yup, Mr. Barry gets volunteers to come out to his place and do maintenance of various sorts. Sometimes it's painting, sometimes repair, sometimes cleaning. For my group, mostly teenage girls, we were to do "yard work". We began with mowing the lawn and trimming the edges. We also did quite a bit of weeding and some general trash pick up in the parking lot. As I was pushing the lawn mower around, I suddenly realized how smart Mr. Barry was. Whereas my neighbors, like the public TV personality (and the hottie across and slightly down the street from him), have to pay good money to get little brown people to keep up their yards, Mr. Barry can get a platoon of little white people to keep up his yard for free.
He had something over 1700 volunteers last year, and expects to have even more this coming year to keep things neat and tidy for him. Don't take my word for it. Go visit Mr. Barry. You'll see a picture of the girls and me on his wall, along with pictures of the other several thousand volunteers he's had work for him over the past few years. Just don't forget to bring a rake, broom, or paint brush with you. Posted via email from Slothoughts Hypolite | | |
| Backing Up Your Server Using PHP by ngungo and lgpiper
Introduction
When the Ma.gnolia site went down a few weeks ago, most people's data were irretrievably lost. One can learn two lessons from this issue. One is that a robust, off-site back up plan is necessary. The other is that back-ups necessarily need be redundant. Thus, if one back-up becomes corrupt or incapacitated in any way, the data are still available via the second back up. We deal with the first of these two problems: developing a routine that ensures regular back up. Our concern is with moderate sized data bases. We are developing a few small web applications: Monpage.com and bookMarkR.us. They may well grow slowly, but they will never grow at all if we don't protect our users' data from the beginning.
Background
We wish to develop a generalized backup routine. The language of the back up is of little consequence. We have chosen to program our back-up routine in PHP. The core of the back up will be a function that looks something like the following:
function dirBK($from, $dest) {
// This function will recursively copy all files and sub directories
// from a single directory, $from, on a hosting server
// to a single directory, $dest, on a remote storage disk.
// rsync seems the best candidate for the file-copying protocol.
}
If there are a lot data in a huge number of directories--e.g. one directory per monpage account, or one directory per site that each account monitors--, using rsync is not without its own problems. It can tie up the i/o ports on the server, then websites--anything else for that matter--on the hosting server would become temporarily unavailable. All activity would freeze whilst waiting for rsync to finish. Depending on the amount of data, this shut-down could take up to several hours. One way to solve the problem is to run the back up in piecemeal fashion, backing up one small directory at a time, with a pause in between each back-up segment. Thus, the function dirBK() must incorporate a loop which spreads the many small back-ups over an extended period of time--e.g. hours, days....
Another problem with rsync arises where, under a non-ideal scenario such as broadband disruption, rsync hangs and does nothing. We must, therefore, find a solution for this problem too.
Given the above considerations, we now have three major tasks, each of which may include some subtasks:
- Developing the dirBK($from, $dest) routine.
- Developing a general driver to run this routine in a loop.
- Scheduling the driver (e.g. running the back-up routine regularly using cron.)
This article only deals with the first two tasks because the scheduling task is server specific and also because, once the general routine functions properly, automatic scheduling is relatively simple to implement.
Back-up Routine Development
Developing dirBK is rsync specific and secondary to the topic here, so we will treat it last. Just be aware that it can be disrupted and hold up the whole grand scheme based on our driver routine. We will first describe the driver in simple terms, than add detail as we go along.
We begin with the variable, $dirs, which is an array of all directories that need to be backed up. The driver routine then becomes:
foreach($dirs as $dir) {
$from = 'path/to/dir/'.$dir;
$dest = 'path/to/destination/'.$dir;
dirBK($from, $dest);
sleep(5);
}
This seems to be a good approach, backing up each directory, one at a time, and pausing few seconds in between. The pausing time could be a tuning parameter, depending on the efficiency of the i/o. What might still cause problems, however, is the situation in which dirBK() function halts in the middle of the file transfer process. One solution might be if we could somehow run each individual directory back up as a independent process. Thus, if one transfer breaks down, others will still be able to continue normally. To do this, we need to implement process spawning, using the pcntl_fork() function. Our modified routine now becomes:
foreach($dirs as $dir) {
$from = 'path/to/dir/'.$dir;
$dest = 'path/to/destination/'.$dir;
$pid = pcntl_fork(); // spawn child process
if (!$pid) { // a child process returns zero
dirBK($from, $dest);
exit;
}
sleep(5);
}
We further need to check 2 things. First, if we will want to exclude some directories. For example, in a normal directory scanning we should exclude the "dot" and "double dot" directories. Secondly, we must be sure that what we are backing up is, indeed, a directory. So the next version of the routine is:
foreach($dirs as $dir) {
if ($dir == '.') continue; //exclude '.' and '..' directories
if ($dir == '..') continue;
$from = 'path/to/dir/'.$dir;
if (is_dir($from)) { //make sure array element is a directory
$dest = 'path/to/destination/'.$dir;
$pid = pcntl_fork(); // spawn child process
if (!$pid) { // a child process returns zero
dirBK($from, $dest);
exit;
}
}
sleep(5);
}
At the end, the driver quits and all independent backup processes, running as the driver's children, also quit, including those that hung up for whatever reason. The command, sleep(5), gives some breathing room between each backup increment. Sometime, however, five seconds will not be long enough to ensure that the last cycle is complete. Thus, we give it a a little extra time, perhaps an extra 30 seconds.
Above, we specified an array of the directories to back up, $dirs. There are a number of ways to populate this array. If all of the directories are in the rootBK directory, we can use the scandir() function to populate the array. Our basic routine is now as follows:
<?php
$dirs = scandir('path/to/rootBK/'); //populate the array, $dirs
foreach($dirs as $dir) {
if ($dir == '.') continue; //exclude '.' and '..' directories
if ($dir == '..') continue;
$from = 'path/to/dir/'.$dir;
if (is_dir($from)) { //make sure array element is a directory
$dest = 'path/to/destination/'.$dir;
$pid = pcntl_fork(); // spawn child process
if (!$pid) { // a child process returns zero
dirBK($from, $dest);
exit;
}
}
sleep(5);
}
sleep(30);
?>
It now remains to develop the dirBK routine. The whole routine can just a system call to the rsync command:
rsync -r /path/to/existingsite/ username@newsite.com:/path/to/newsite/
It may be necessary to extend this with a few more parameters to specify a password file:
rsync -azq --delete -e "ssh -i /users/home/myaccount/.ssh/ss" /path/to/backups/ myaccount@myaccount.strongspace.com:/home/myaccount/backups
We won't go into the details of the rsync command, but you can find them on line at the following urls:
We will use the second method above, i.e. with the password file specified. To break up the command into more comprehensible components, we have,
$rsync = '/path/to/rsync -azq --delete -e "ssh -i /.ssh/ss"';
$from = '/path/to/backups/';
$remote = 'myaccount@myaccount.strongspace.com:/home/myaccount/backups';
Then, as mentioned above, we embed these variables into a php system call, remembering that we need a space between the three strings, $rsync, $from, and $remote:
system($rsync. ' ' .$from. ' ' .$remote);
Placing this call into the main scheme produces the final version:
<?php
$rsync = '/path/to/rsync -azq --delete -e "ssh -i /.ssh/ss"';
$remote = 'myaccount@myaccount.strongspace.com:/home/myaccount/backups';
$dirs = scandir('path/to/rootBK/');
foreach($dirs as $dir) {
if ($dir == '.') continue;
if ($dir == '..') continue;
$from = 'path/to/dir/'.$dir;
if (is_dir($from)) {
$dest = $remote.$dir;
$pid = pcntl_fork();
if (!$pid) {
system($rsync. ' ' .$from. ' ' .$dest);
// echo $dir;
exit;
}
}
sleep(5);
}
sleep(30);
?>
The echo $dir command was added for debugging purposes. To begin, one might wish to comment out the system call and un-comment the echo line. Then when the routine appears to be processing the expected directories properly, reverse the comments so that the full back-up is working.
If the name the file containing our routine backUp.php, we can run it manually from the command line, making sure to be in the proper working directory:
# php5 backUp.php
Sometimes it is necessary to specify the full path to the php command, and also the php.ini file. In that case the command might look as follows:
/usr/local/bin/php -c /users/home/userName/etc/php5/ /users/home/userName/backUp.php
The specifics, naturally, depend on one's particular server configuration.
Once the routine works as intended from the command line, then one should write a cron job to automate the process. That is a subject for another time. Posted via email from Slothoughts Hypolite | | |
| Finally ... A Real Psalm 'o MaticSeveral years ago, it occurred to me to create a Psalm-o-matic for Lent. I have no idea why this occurred to me, it just popped into my head one day and I couldn't forget it. In theory, we are supposed to be a bit more focused on spiritual things, like scripture and prayer, during Lent. Just because we're Congregationalists—UCCs actually—why shouldn't we attempt to bow to tradition, at least a bit. So to make the scripture part a little more palatable—or so I thought at the time—how 'bout if we each read a Psalm each day? Rather than expect people actually to hunt up a Bible, and then figure out where the Psalms lay in that book, why not just e-mail out a Psalm? If we're to read Psalms, why focus on the three or four we already know. How 'bout we read a randomly selected Psalm? That way, in theory at least, we would get through Lent with a representative selection of Psalms, the familiar and the not-to familiar.
Thus was born the idea of the Psalm o' matic. Now to figure out how to implement it. Of course, I thought it would be cool to write a computer program to do the work for me. I had a job at the time, and was a bit rusty on my programming, so I ended up with the coward's way. I used the random-number generator function in my spread sheet program to make up a list of numbers between 1 and 150. Then I copied the list over to a schedule, printed it out, and manually e-mailed out the Psalm scheduled for each day. Not all that o' matic, huh?
I did that for a couple of years, but this year, since I was unemployed, I figured that I should brush off my very rusty shell scripting and try to make a real, bone fide psalm-o-matic. The shell script would, in theory at least, randomly pick out a number from 1 to 150, select the Psalm that related to that number, and do the e-mailing for me. All while I slept.
The first thing I needed to do was come up with a method for generating random numbers from 1 to 150. Of course that's trivial if you are programming in C or C++, or using a spread sheet, but I was trying to do this in Unix shell scripting, the only way I knew how to send out e-mails automagically. It turned out that what I knew about sending out e-mails—learned in Intro to Unix back in the summer of 2000—was no longer valid, at least not on my system. But I didn't know that when I began this project.
Anyway, it turned out that generating random numbers from the unix shell was relatively easy. My server at Joyent runs bash-shell scripts, and the bash shell has a build-in shell variable called $RANDOM. $RANDOM returns a random number from 0 to 32767 whenever it is called. To pare this selection down to 1 to 150, I merely had to multiply the $RANDOM variable by 150, divide by 32767, and then add 1 to take care of the fact that Psalms begin with 1 not 0 (zero).
psalmNo=$((1+150*$RANDOM/32767))
and the variable, $psalmNo would indeed be a number between 1 and 150.
So far, so good. Now I just needed to select the appropriate Psalm and send it off. Selecting the appropriate Psalm was trivial, albeit tedious. I made up 150 text files with names like psalm_xxx.txt, where xxx was a number from 1 to 150. Then I just needed to combine three elements, the root of the text-file name, the Psalm number and the extension:
body="$filePath$fileRoot$psalmNo$fileExtension"
Oh, yeah, I forgot to mention, the $filePath thing tells the computer where to find the directory that has all the Psalms in it.
So now we get to what I assumed would be the easy part, e-mailing the Psalm. In olden days, we did something like the following:
mailx -s "e-mail subject" someone@somedomain.com <$body
That is, the file in $body is inserted in an e-mail message to someone with the subject given. Well, it didn't work. After thrashing around for quite some time, I asked for help and was told that mailx no longer worked because of security issues. I had to use sendmail -t instead.
I'd never heard about sendmail -t before. I guess it didn't work in the olden days. Or perhaps it just didn't work at UMass/Lowell, who tend to be a bit behind the times (when I took Java Script, I was told to be wary about things that caused problems with Netscape 2.0. WTF? we were in a post-Netscape era by the time I took that class.)
Anyway, I had to learn about sendmail. That wasn't overly difficult. I discovered that the -t option meant that the system used the header information embedded in the file. So eventually, I figured out that meant I had to make up the files I e-mailed out with stuff like "To: someone@somedomain.com" at the top of the file, and the text file containing the Psalm at the end. Well, crap, I didn't want to go through 150 files and put in all the header stuff first. What if I wanted to do something else with the files later on?
Fortunately, this wasn't a problem. I suddenly remembered about "appending" things to a file. To append things, you just did a double carat thingie, i.e.
cat $body<<$filePath$mailMsg
and you get the stuff in $body, i.e. the Psalm file, appended to $mailMsg, the thing you created with all the header info. So I build it up a bit at a time, starting with the "To: someone@somedomain.com" stuff, adding in the From, Reply-to, and Subject stuff and finally ending with the Psalm file.
So after all this ruminating, we ended up with
# !/usr/local/bin/bash
# psalm o' matic script
#
# This works as follows: me@myServer$ /usr/local/bin/bash psalm-0-matic
# If you don't precede with the "bash" it won't work.
# The fully qualified paths through out are necessary to make this work as a cron script
#
# This uses my random number generator script
#
# Why in the hell am I using vi?
# I'm not anymore -- using TextPad with sftpDrive (now ExpanDrive)
filePath="/users/home/myUserName/domains/myDomain.org/web/public/Psalms/"
fileRoot="Psalm_"
fileExtension=".txt"
# ***seed Random with a "random" number -- then select Psalm o' day
RANDOM=$RANDOM
psalmNo=$((1+150*$RANDOM/32767)) #all 150 -- Oh, yeah!
# *****************prepare the message ******************
mailMsg="MailMsg"
today=`/usr/xpg4/bin/date '+%A, %B %d, %Y'`
# ***************** prepare header header ******************
dateField="Date: "$today
#toField="To: me "
#later on it would be fccr-ace.
addressFile=pomList
toField="To: "`cat "$filePath$addressFile"` #pomList is comma delimited: name , ...
fromField="From: me "
replyToField="Reply-to: me@myOtherDomain.net" #later on it would be fccr-ace?
subjectField="Subject: Psalm for "$today"--Psalm "$psalmNo
# ***************** assemble message ******************
body="$filePath$fileRoot$psalmNo$fileExtension"
echo $dateField<$filePath$mailMsg
echo $toField<<$filePath$mailMsg
echo $fromField<<$filePath$mailMsg
echo $replyToField<<$filePath$mailMsg
echo $subjectField<<$filePath$mailMsg
echo " " <<$filePath$mailMsg
echo " " <<$filePath$mailMsg
cat $body<<$filePath$mailMsg
# *****************send the message ******************
/usr/local/bin/sendmail -t <$filePath$mailMsg
Ok, this worked. Getting it to send out automagically turned out to be more trouble than I realized, but I eventually got so I was sending myself Psalms on an hourly basis. Just in time for Lent.
Addendum:
The scheme worked like a charm and we lucked out in that Ps. 119 never came up. Not sure anyone would have been able to slog through that one in one sitting.
Addendum 2:
Formatting this was a real PITA. I'd have been better off just doing raw xhtml on my static site. Then again, I suppose that I needed to (re?)learn that'code' is not a block-level element. Posted via email from Slothoughts Hypolite | | |
| Ave Verum Corpus ...... That's what we sang at the Maundy Thursday service tonight. I have always loved this piece. I don't think we were so great, but then again, we probably weren't that bad. We recorded a passable version of it a few years back, which is reproduced here.
Because we're Congregationalists, we can sing in Latin. That seems weird, since our denominational roots are decidedly anti-Catholic (think Puritans). But for quite some time after Vatican II in the mid 60s, Catholics weren't allowed to do Latin. So you had to visit Congregationalists and Presbyterians and the like to hear things in Latin. I think that may have changed in the past few years.
The lyrics we sang:
Ave, ave, verum corpus
natum de Maria Virgine,
Vere passum immolatum
in Cruce pro homine,
Cujus latus perforatum
unda fluxit et sanguine,
Esto nobis praegustatum
in mortis examine.
Translation:
Hail,true body
born of the Virgin Mary,
Who truly suffered, sacrificed
on the Cross for man,
Whose pierced side overflowed
with water and blood,
Be for us a foretaste
In the test of death. Posted via email from Slothoughts Hypolite | | |
| I Put a Spell on You -- Still amazing after all these yearsOnce I was baking cookies with my older sister. We were listening to the radio while we baked. The songs were typical teen-ager kind of stuff for those days. Then, amazingly, the DJ played a song for which he wanted his listeners to provide feed back. The kitchen was suddenly filled with the sounds of I Put a Spell on You, by Screamin' Jay Hawkins. I guess the other listeners weren't so impressed as I was. I thought the song was amazing, but I never heard it again until I was an adult with kids of my own. One of my kids helped me find this copy of the song. Posted via email from Slothoughts Hypolite | | |
|
