"Dark Matter" © DigitalBlasphemy.com
The problem stated is one of vast distances - the time for even a fast traveling extraterrestrial asteroid from even the closest star system would take an exorbitant period of time. If the average speed of an asteroid in the main belt is around 47,000 mph, and the distance to Alpha Centauri is 4.4 light-years or 25,848,247,139.8 miles (Proxima Centauri is sometimes closer, but let's go with the main star), then it would take that asteroid about 101,304 years to reach Earth.
Hardiness of Bacteria
Recently, bacteria have been found buried deep in solid rock - bacteria with very slow metabolic states and are probably thousands of years old. Penn State scientists discovered in Kalaallit Nunaat (Greenland) dormant ultra-small bacteria (Chryseobacterium greenlandensis) trapped 2 miles deep in 120,000 year old ice core samples. The scientists were able to bring them back and found it needed few nutrients to live. The scientists figure their small size helped them so survive trapped so long in the ice. Some studies indicate some bacteria can live suspended in sediments, amber, and halite for millions of years. In 2007 a group of scientists published a paper provided evidence for bacteria surviving in some frozen permafrost samples up to a half a million years. We have seen from other posts the ability for some bacteria and viruses to survive the vacuum of space. In a previous post I reported that even the small, multicellular Water Bear can survive periods in space - including normally deadly doses of ionizing radiation (Water Bears in Space!).
With Alpha Centauri 101,304 years away (by asteroid) and bacteria that can lie dormant for at least 120,000 years (and especially those that can lie dormant for 1/2 million years), it seems that some unicellular life could theoretically make the trip. The red dwarf planetary system Gliese 581 (with one planet that is warm and terrestrial) is 20.5 light-years away, or around 471,984.5 years away at main belt asteroid velocities. Still within the range of some unicellular creatures. We have seen in earlier posts that there are scenarios where red dwarf systems can be hospitable to life.
Sol's Close Encounters
There is something else to consider as well. Our solar system is not a fixed point in the galaxy. Some researchers feel that the Sun had some close encounters with other solar systems during its 4.6 billion years of existence. The distance that a bacteria hosting extraterrestrial asteroid could have been even smaller than 4.4 light years. Some scientists feel that the Sun may have had a close encounter with another star 4 billion years ago - a very close encounter: the other star may have come as close as 14 to 19 billion miles (Neptune is 4.7 billion miles away from the Sun). That is close enough for major gravitational permutations of each solar system - including the possible exchange of an outer planet. That's right - our solar system could have, in its outermost reaches, a planet from another system left behind from this close encounter.
Let us say the other system already had life established on it. The close encounter with our Sun could cause asteroids to be jostled and sent colliding into the system's life-bearing planet, throwing up chunks of the planet into space. One of those chunks could have been captured by our solar system and eventually made it to Earth (just like some Martian "chunks" have made it to Earth, the result of some asteroid impact on the surface of Mars ejecting Martian rocks into space). Or the other system could already have asteroids with dormant unicellular life on them left over from collisions it had within itself before encountering the Sun, and one or more of those asteroids captured by our solar system.
In conclusion, with regards to panspermia, some stellar distances are not too vast for living organisms to be transverse and end up on Earth. Does not mean it has actually happened - the chances are still probably rather small. But the distances are not too vast, at least for stellar distances of 21 light-years or less.
Coghlan, Andy. "'Resurrection Bug' Revived after 120,000 Years." Life. New Scientist. 15 June 2009. Web. 30 July 2009.
"Did Our Sun Capture Alien Worlds? Close Encounter May Explain Some Objects Beyond Neptune." Science News. ScienceDaily. 9 Dec. 2004. Web. 30 July 2009. <http://www.sciencedaily.com/releases/2004/12/041208235835.htm>
Johnson, Sarah S., Martin B. Hebsgaardt, Torben R. Christensen, et. al. "Ancient Bacteria Show Evidence of DNA Repair." PNAS. Proceedings of the National Academy of Sciences of the United States of America. 25 July 2007. Web. 30 July 2009. <http://www.pnas.org/content/104/36/14401.full>