Todays search engines rely on software packages called spiders or robots.  These automated tools are used to search the web to discover new pages.

A brief history of search crawlers

The first crawler was the World Wide Web Wander and it appeared in 1993.  It was developed by MIT and it's initial purpose was to measure the growth of the web.  Soon after, however, an index was generated from the results – effectively the first "search engine."

Since then, crawlers have evolved and developed.  Initially crawlers were simple creatures, only able to index specific bits of web page data such as meta tags.  Soon, however, search engines realized that a truly effective crawler needs to be able to index other information, including visible text, alt tags, images and even other non-HTML content such as PDF's word processor documents and more.

How a crawler works

Generally, the crawler gets a list of URL's to visit and store.  The crawler doesn't rank the pages, it only goes out and gets copies which it stores, or forwards to the search engine to later index and rank according to various aspects.

Search crawlers also are smart enough to follow links they find on pages.  They may follow these links as they find them, or they will store them and visit them later.

To date there are literally dozens of crawlers out regularly indexing the web.  Some are specialized crawlers – such as image indexers, while others are more general and therefore more well known.

Some of the most well known crawlers include Googlebot (from Google) MSNBot (from MSN) and Slurp (from Yahoo!).  There is also the Teoma crawler (from Ask Jeeves), as well as an assortment of crawlers from other engines, such as shopping engines, blog search engines and more.

Generally, when a crawler comes to visit a site, they request a file called "robots.txt." this file tells the search crawler which files it can request, and which files or directories it's not allowed to visit.

The file can also be used to limit specific spiders access to any or all of the site, and can also be used to control how many times the crawler visits the site, by limiting it's speed or the times when the crawler can visit.  (Yahoo!s Slurp and MSNBot both support the "Crawl Delay" directive which tells the crawlers to slow down on their crawling).

It's not imperative that a site have a robots.txt file however as a crawler will assume it is OK to index the site if there isn't such a file.

Generally, today's crawlers are stripped down versions of web browsers.  Some, like Googlebot, are built upon a text based web browser called Lynx.  Therefore one of the tools one can use to verify a site is the Lynx browser.  by loading the site in the browser you can see essentially what the crawlers "see." You can then look for errors in the pages as well as any navigation problems the crawler may come up against.

One other thing you may notice, as you view your web server log reports, is that some browsers come many different times and with many different configurations.

Yahoo!s Slurp, for example emulates many different hardware platforms – from Windows 98 to Windows XP, and many different browsers, from Internet Explorer to Mozilla. MSNbot also works like this – emulating different operating systems and browsers.

They do this to ensure compatibility – after all, the search engines want to be sure that the majority of their users find a site which they can use.  Therefore, as a design tip, you should test your site against various hardware platforms and browsers as well.  You don't have to use the variety that the search engines use, but you should test against Internet Explorer, Netscape and Firefox.  Also, you should try your site on other platforms such as a Mac or Linux just to ensure compatibility.

You may also notice, upon reviewing your reports, that crawlers like Googlebot will visit repeatedly and request the same page(s) repeatedly.  This is common as crawlers also want to be sure the site is stable and also to measure the page's change frequency.

If your site goes down temporarily when a crawler visits repeatedly like this, don't worry.  The crawlers are smart enough to leave and come back later and try again.  If, however, they continue to find the site down, or slow to respond, they may opt to stay away for longer periods, or index the site more slowly.  This can negatively impact your site's performance in the search engines.

As time goes on, we'd expect these spiders to become even more advanced.  As new authoring technology comes available, or new indexing options become available, then the search crawlers will be adapted.  Remember, the goal of all the search engines is to have the most complete index of files found on the web.  This means they want to be able to index more than just web pages.

So as you are designing your site, be sure to keep the crawlers in mind.  Don't build your site for crawlers – build it for users – but be sure to test it thoroughly so that the crawlers see what you want them to without hindrances or roadblocks.  Remember – the crawler is a site owners best friend.

Rob Sullivan - SEO Specialist and Internet Marketing Consultant. http://www.textlinkbrokers.com

First you’ve got to get into your car and drive to wherever the meeting is, wasting precious hours in traffic. And being on time just isn’t enough! You’d better arrive early so you can get a good parking spot, register, and get a halfway decent seat.

Then you can twiddle your thumbs or reread the agenda five or six times while you sit there and wait for the course to begin. And when it's over you've got to walk back to your car, despite inclement weather, and drive back to your office. So, now you’ve spent three or more hours of your busy day to attend a one-hour class.

Whether it’s making the leap from employee to entrepreneurship and starting your own business, excelling in your current career, seeking a better job, motivating your staff or managing your time and money -- there is a way to gain access to personal and professional knowledge, camaraderie, and one-to-one interaction simply by using your telephone. Teleclasses are a great way to effect positive change in your life. Telecommunication is the buzzword – and virtual classrooms, teleclasses, tele-seminars, tele-learning or conference call classes are the wave of the future!

Teleclasses are offered on a wide range of business and personal development topics and are an exciting, innovative way to attain valuable information, explore new ideas and gain new skills. You can examine your spirituality, inspire your life through the arts, learn a craft, dialogue on best places to travel, learn to create better relationships, learn to eat better or exercise to improve your health and fitness, or just simply have fun.

A teleclass is much like a regular class – except it brings the classroom to your doorstep! The technology is as simple as punching the digits on a telephone and the returns are immense! You simply dial into the conference call at a designated time; announce yourself to the other class participants, and the program begins.

The facilitator/teacher makes a presentation and there are usually questions and comments, followed by a lively and robust exchange of ideas. A major perk is that most teleclasses are offered at a variety of times and dates, allowing you to schedule them at your convenience.
Usually one-hour in length, and fully interactive, teleclasses can be accessed from your desk, your home, your car, a pay phone or even your hotel room.

Best of all, the cost is significantly less than a traditional class or seminar when you consider travel and time; making tele-learning an extremely affordable knowledge tool.

Teleclasses offer positive and essential education, using technology as a tool for communication with strong and consistently positive results. And when the program ends, you hang up the phone. And guess what, your one-hour class took exactly one hour!

One distinct advantage is that you can interact with other people that have different ability levels, skills and backgrounds, who can be from almost anywhere in the country or the world.

You can take a class just for yourself or you can set up a teleclass that everybody in your company can take simultaneously. The groups are usually small, 20 or less, so everyone gets individual attention through instruction that is both topical and focused.

Teleclasses are a hassle-free, money saving alternative to traditional “in person” seminars and classes. NO Travel – NO Parking fees and you get cutting-edge, innovative and customized learning in an informal setting with a select group of people who, like you, want to take command of their learning experience and aren’t afraid to take risks to increase their knowledge base and accelerate their development!

So the next time you contemplate taking a seminar or a class to improve your business, personal or interpersonal skills -- you might want to make it a teleclass.

Rick Sherréll is the founder of Pro Speakers Bureau an Atlanta-based agency that delivers high-quality speakers, trainers and subject matter experts as well as a variety of teleclasses, seminars and special events. For list of available teleclasses visit www.ProSpeakersBureau.com/Teleclas

While this scenario may seem more like a dream than reality to you right now, it’s entirely possible to achieve and it’s much easier to do than you might imagine. Besides starting each day with a big, Cheshire Cat grin, and dollar signs floating through your mind’s eye, there are numerous other benefits to having a product available on the web.

Positions you as an expert - Perception is everything on the internet and the creation of your own product results in positioning you as an expert and is a critical step in generating new business.

Builds your brand identity - Your product represents you and your business in the marketplace, and making it available on the Web is the first step toward getting your product into more hands, heads, and homes.

Reaches a global marketplace - Having a product available on the Web means that you've expanded your geographic marketplace from local neighborhoods where your product sits on a shelf, to the entire world via your website.

Creates a 24/7, passive revenue, profit machine - The Web never sleeps which means that you can literally turn your computer and website into a cash register around the clock, and many, if not all, of the processes can be automated.

Levels the playing field - In the past, the entrepreneur or small business owner was unable to compete with the larger companies which could afford to mass market products. The Web makes it possible for even the smallest of businesses to compete.

Instantly increases the effectiveness of your sales cycle - This is especially critical for service professionals. Often a consumer will opt to purchase a product as a trial before deciding to purchase your service. Having a quality product available on the Web allows them to get to know a bit more about you and what you have to offer.

In addition to the great benefits listed above, which apply to any online product, there are several more which apply specifically to information products and make them an even more attractive choice.

Greater cost-effectiveness – Since traditional production and distribution is unnecessary, your costs are significantly reduced and your margin for profit is significantly increased.

Increase your speed to market - While it might take months or years to get a hardbound book written, edited, published, produced, and distributed to book stores, you can deliver the same content in the format of an e-book and bring it into the world in a matter of days or weeks instead.

Content can be leveraged in several formats – The same content can be presented in several different formats; as an e-book, an ecourse, a teleclass, a home-study course, and the list goes on and on.

Opportunity for bold self-expression and learning – The Web allows you to get your message to millions simultaneously while at the same time offering you the opportunity to learn in action as you challenge yourself to create something totally unique.

© 2005 Michael Port & Associates LLC

Get more clients with Michael Port, expert marketing coach for small business owners and professional service providers. Free small business resources, networking opportunities, articles, advice and coaching on professional services marketing at http://www.michaelport.com . Receive a free chapter from ‘Book Yourself Solid’ at http://www.bookyourselfsolid.com

There’s an easy to measure, but often overlooked number that can tell you a great deal about the effectiveness of your website. The ‘short visit rate’ is the percentage of visitors that leave your website within 10 seconds.

The percentage itself is not really important. Every website, every audience and every industry is different. Even the most sophisticated websites can easily have 50 percent of visitors bailing early. What is important is what you’re doing to reduce this number. Why? Because there is no cheaper way to repair the spout that feeds your pipeline. While it may take months to test and correct your sales process, telemarketing or direct mail efforts, the Web allows for rapid and cost-effective testing. You can positively change your short visit rate in a few days, even a few hours.

The Impact
If visitors to your site don’t stick around, they can’t request more information, take the demo, sign up for the newsletter or do any of the things the site was designed to make them do. It took considerable time and money to bring these visitors to your site--each time one leaves, you’re squandering your general marketing and advertising efforts and dollars. Worse, those visitors leaving are likely seeking information or a solution elsewhere on the web. You’re helping create demand for your competition.

The Cause
Short visits are usually a combination of two things--the wrong people coming and your site’s inability to engage the visitor. This is particularly true when it comes to traffic from search engines. Because these visitors are looking for something specific (and usually in a hurry), it’s only logical that they would quickly bail if the site doesn’t satisfy their needs.

What To Do About It
Luckily, we’re dealing with the web, where the data is rich and the adjustments are easy. The first step is to see where you are. Ask your webmaster or web marketing person to report on the percentage of both all visitors, and search visitors that are leaving within 10 seconds. If they can’t get this information for you, enlist some help or get a simple web traffic analysis tool, as this is basic data every company should capture.

Once you get these numbers, don’t flip out. It’s just a baseline, a starting point from which to improve. Ask yourself, ‘what might be causing these people to leave?’. Visit your site in your prospect’s shoes. Might it be the non-differentiating or overly technical message? Maybe the site just doesn’t look very professional. Perhaps the majority of your prospects simply don’t like the color red. Most likely, it’s the lack of any compelling reason to stay. If you don’t have something to grab visitors’ attention and get them clicking for more, you can continue to expect poor retention rates.

Sometimes I find it helpful with my clients to look at this problem from a traditional point of view. What if this was a tradeshow booth? How could we get people to stay at the booth longer? The key here is to work together with your sales and marketing folks (notice we left the IT department out) to brainstorm and come to a consensus as to what you suspect the single biggest problem might be. Once you decide, have your webmaster make the necessary changes and measure the difference. If you don’t get a whole lot of visits, it may take a long time to come to a conclusion. This is where doing a quick pay-per-click advertising campaign can come in real handy if for nothing else than to get some inexpensive market research.

You’ve Got Nothing to Lose Except Business It’s easy to dismiss this entire problem; after all, you probably didn’t know you had a problem. And you can always add more sales reps and send out more mailers. However, the ‘problem’ could actually be an opportunity in disguise—if you take these first steps. You may be surprised how enthusiastic your web team and senior management are about wanting to improve in this area. It is fun, challenging and, unlike many other aspects of business, an effort that provides rapid and rich feedback. Hold contests! Give the person or team that comes up with the most dramatic improvement the corner office! He or she just made your company more efficient and profitable.

Short-visit syndrome is something most executives don’t like to mention, let alone talk about. Fortunately, confronting the issue head on and doing things to remedy it is far less painful than you think and the results are well worth the effort.

Todd Miechiels is a B2B search marketing consultant who helps companies that suffer from short-visit syndrome and other Internet marketing challenges. His website is www.miechiels.com.

I’ve been watching social networking for some time now. In fact I’m a member of various social networking sites including MySpace and LinkedIn just to name two.

I joined partly because I wanted to see what they were, but more importantly to see what impact social networking would have on SEM in the coming years.

I’ve been a member of various services for some time and the reach these sites have is incredible.

For example, from my LinkedIn network of seven people I have an expanded network of over 12,600 people.

Imagine that - I’m only a click or two away from close to 13,000 other people who share my similar interests ranging from what I like to watch on TV to work I could provide to them.

Through my connections and their connections, I’m connected to people ranging from the American Cancer Society, to Sun Microsystems to the University of Texas to Google.

But what marketing opportunities are there for Social Networking?

Well, let’s look at MySpace.

MySpace is one of the top sites on the Web today. It racked up 9.4 billion page-views in August 2005 (more than Google) and new users are signing up at a mind-boggling rate of 3.5 million a month.

MySpace is typical of where today’s 18-30 year old goes to manage their digital life. It allows users to post photos of themselves and their friends, create a blog, list their favorite bands, view and share videos, suggest things to do and lists a set of people they consider friends. It is on this “Friends List” where most of the opportunities lie.

All thirty million plus users of MySpace have a friend’s page that lists people that person considers their friend. This is a list of people that they are interested in talking to and about, as well as hearing from on a regular basis. Once you add someone as your friend they can send you emails, comment on your photos, read your blogs, as well as leave messages that you can then share with others. This is an opportunity for instant feedback about you.

The ability to add friends to your page is key for marketing to MySpace users. According to Courtney Holt, head of new media and strategic marketing at Interscope Records, “This generation is growing up without having ever watched programmed media.” “They don’t think in terms of the album, and they don’t think in terms of a TV schedule. They think in terms of TiVo, P2P, AOL, and of course MySpace.”

You can see how this could grow.

Let’s say you create a MySpace account to talk about your product or service. You blog about it and search for others that may share your interests.

You then invite them to be your friend. When they become a friend you start your “soft sell” pitching your product to them.

As they grow to appreciate it, they start blogging and sharing it. Soon hundreds or even thousand of people are talking about you and your product or service.

Don’t think this will work?

Let me give you some examples. There are many bands who have gotten their start on MySpace. Simply by hosting some of their music online and blogging about themselves they developed a following. Soon they had record deals and contracts lined up.

Of course to use services like MySpace you need to have something this target market needs. If you don’t then you probably shouldn’t put too much effort into MySpace.

But that doesn’t mean other social networking opportunities should be overlooked. As I mentioned above, LinkedIn is more of a professional introduction service. If your product or service fits here then by all means explore it further.

And there are others as well. Services such as Yahoo!s MyWeb, Flickr, del.icio.us and more.

So if you’ve ever wondered what other online opportunities could be out there, consider social networking. It could be the next great online marketing channel.

Rob Sullivan is a SEO Consultant and Writer for www.textlinkbrokers.com Textlinkbrokers is the trusted leader in building long term rankings through safe and effective link building.

Podcasting uses the same technology that most bloggers use to get exposure—RSS. RSS or “Rich Site Summary/Real Simple Syndication”(choose your poison) is essentially used to broadcast the blogging contents. The RSS is versatile enough to accommodate image, data, video or audio files. Essentially podcasting is an audio file in the RSS feed's enclosure tag. The RSS version 2.0 supports the current Podcasting specifications and is the widely used format.

The podcast audience gets the latest stuff of their interest on their media player. It’s time shifted and place shifted, so they can listen to it at the time and place of their choice. This beats radio and even streaming radio. The technology works with all contents-music as well as audio blogs. It’s the latter use which has garnered the most interest.

In general, the entry fee for the podcasting is low. All one needs is a decent quality microphone to ensure quality recording. Of course, one can get fancy, with audio editing software, sound effects, and such, but ultimately if the content is interesting/useful one can build a loyal audience. Think Craig’s list Vs. the-fancy-shmancy-flash-site.
So why bother with podcasting? Taking the blogging analogy ( www.insurgentjg.com ) further, the podcasters are a pioneering bunch. The main stream talk radio shows have not got into the act just yet, just like the main stream media does not use the blogging format(as much as it should)—leaving the field open for podcasting enthusiasts. While a lot of content may not be polished, some of us are looking for “Reality” version of things. The anonymous blogger writing from her basement can(and sometimes does) command a loyal readership that would make the editor of New York Times blanch. Similarly, the podcaster can have an audience rivaling political commentators and MTV DJ’s. It’s all about the value the podcaster provides to the audience. The meritocracy of internet ensures that only the cream rises to the top.
What is required to Publish Podcasts(Podcaster).
Some simple steps and you are on your way to be a podcaster:
1. Record your audio content in a file and post it to your website.
2. Generate an RSS feed containing a link to the audio file in the enclosure field of a RSS 2.0 compliant feed.
3. You are ready to promote your music/thoughts/ideas to the rest of the world.
What is required to listen to Podcasts (Podcastee):
Three simple steps for web surfers to listen to podcasts:
1. Download a News aggregator or RSS reader which is RSS 2.0 compliant. If you are blessed with an IPod or similar device, even that may not be necessary.
2. Insert the URL of the podcast feed into the news aggregator or podcast management software.
3. When new items appear in the category of your interest, you can analyze the narrative and listen to what you really find interesting.
Podcasting as a field is new enough that it’s still not that difficult to get exposure. A microphone, some diskspace on a server and talent is all you need to get going. Unfortunately the exponential growth of the medium and potential for marketing and advertisement ensures that the cacophony of spam may drown your podcasts. Now is the time to create that loyal audience. Happy podcasting!
For more information about podcasting visit http://www.podshock.com