Unlike the always-about-to-happen flying car, self-driving cars are a sci-fi fantasy that really is just around the corner. Just a few years ago, adaptive cruise control was a feature reserved for flagship luxury sedans like the Mercedes S-Class. Today, you can get adaptive cruise control on a midsize family sedan, the Chrysler 200, and many cars come with lane-keeping assist technology that makes driving down the highway practically an autonomous affair. There are also a number of manufacturers like Volvo and Subaru that offer emergency braking technology to bring your car to a stop on its own. As mainstream vehicles gain more and more options, they’re beginning to feel so close to truly autonomous, it’s easy to imagine riding through town in a driverless taxi just a few years from now.

There are also plenty of companies that are developing completely autonomous vehicles. Google is famously working on being the first company to bring a fully autonomous car to market, with their prototypes logging hundreds of thousands of accident-free miles. Most recently, Mercedes introduced its F015, a self-driving concept car, at this year’s Consumer Electronics Show, but they’re far from the first automaker to work on a self-driving vehicle. In fact, most automakers are working on self-driving vehicles in some regard. With so much technology already developed and so many companies developing fully autonomous vehicles, surely it will only be a few years before the first one goes on sale, right?

Probably not.

The most obvious hurdle is, of course, the legal one. In ideal conditions, autonomous vehicles make fewer errors than human drivers, making the odds of an accident quite low. Eventually, an autonomous vehicle will get into an accident though, and when that happens, someone will have to be at fault. Will the vehicle owner be at fault? What if the owner lent the car to a friend? Would the primary passenger be at fault? Does the automaker bear some responsibility for the error in its product’s decision making? Is the automaker at fault at all?

Until these kinds of questions are answered, there’s no way that cars are going to be truly autonomous. Self-driving cars will probably go on sale before then, but it’s going to be a while before “self-drive” is legally allowed to be anything more than an incredibly advanced cruise control.

There are, however, two other hurdles to the implementation of fully autonomous vehicles, and figuring out how to handle them is going to be quite a challenge.

The first challenge is that our infrastructure isn’t set up well for fully autonomous vehicles, and a lot of it has to do with parking. If a car can’t pay for its own parking, the passenger is going to have to sit in the car until it finds a place to park, then pay. That may sound like a minor concern now, but part of the appeal of fully autonomous vehicles is that they should be able to drop their passengers at the door before parking themselves.

Additionally, a lot of parking changes and restrictions are displayed with hand-posted signs. Inevitably, a self-driving car is going to park itself in a spot that’s reserved for permit holders, in a bus stop, or on the wrong street, and it’s going to get towed, costing owners hundreds of dollars. Something tells me that those owners won’t be too happy to find out that their self-driving car got itself towed because it couldn’t read.

The closer cars get to being fully autonomous, the more likely it is that a better system for self-parking will be developed, but at least for now, it would be very difficult to trust a fully autonomous vehicle to park itself safely.

While self-parking will probably eventually be handled, automakers are also going to have to make sure these self-driving systems are just as functional during inclement weather as they are when it’s clear and sunny outside. A Google car is nearly perfect while driving itself around southern California, but on a clear day, there’s nothing to confuse the computers that make all the driving decisions.

What happens when it pours down rain though? What about during the winter in New England when the sensors get covered in ice and snow? Will the car still be able to function just as well? Will it calculate the risk of making an error is too high and refuse to turn on? Until automakers figure out what to do with snowstorms and other forces majeure, it’s going to be very difficult to sell a vehicle that you can only use eight months out of the year.

If automakers can figure out how to make fully autonomous vehicles work well enough and can get regulators to approve their use, it will lead to a total revolution in how people get around. Unfortunately, making that happen is going to take a lot more than just a few years. There’s a decent chance that the next generation Mercedes S-Class will have a cruise control option that allows it to practically drive itself, but don’t expect to see a production version of the F015 any time soon.