UP ON THE High Line, all seems still. The roar of Manhattan still crams your ears — the sirens, the coursing streets, those loud American voices. But even two storeys up from the sidewalk the full-grown trees, beautifully blooming weeds and buxom shrubbery seem to muffle everything.

Maybe it’s the imagination. Scampering up to this derelict railway viaduct from Tenth Avenue is like walking through Alice’s Looking Glass into the Secret Garden. You step from the chic grime of Chelsea, Hell’s Kitchen and Meatpacking, New York’s most feverishly gentrifying neighbourhoods, into another world, one and a half miles and 6.7 acres of wilderness Manhattan Eden, coursing through 22 blocks of prime real estate, left to rot and bloom in one of the densest cities on earth, where — round here — a square foot can lighten your pocket by $2,500 (£1,350). So much space for free seems obscene.

You’re not meant to be up here. This is private property. But in less gentrified times, when the razor wire was more forgiving, locals would sneak a trespass. A bevy of clubbers at 4am, maybe. Or, five years ago, perhaps Joshua David.

David, a freelance journalist, is co-founder of Friends of the High Line, set up with the artist Robert Hammond in 1999 to find a new future for the viaduct. “I lived here for 13 years and hardly noticed the thing. When you walk past, you just see little fragments. You get no idea of its enormousness,” he says.

The High Line has been part of the West Side’s landscape for decades, just another forgotten chunk of the city you take for granted but would miss like a limb if it were demolished. Every town’s got architectural detritus left by the eddying streams of progress. Even the city that never sleeps.

When it opened in 1934, the High Line was progress, built as a vastly expensive Depression-age programme designed to pump life back into the neighbourhood by speedily railroading freight straight into warehouses, putting an end to Death Avenue, the street-level railroad, so named for all the pedestrian fatalities it caused.

It came too late. Progress moved on. Railways were going bust across America, and factories were fleeing to the ‘burbs.

In the Fifties and Sixties it suffered death by a thousand Beeching-style cuts and, in 1980, ground to a halt; its last delivery, appropriately, frozen turkeys. This fragment survived simply because it was in what was until the late Nineties one of Manhattan’s less salubrious neighbourhoods.

Florent Morellet, a restaurateur and fellow High Line enthusiast, opened his French diner, Florent, on Gansevoort Street in 1985. “They thought I was craaaazy,” he says. “It was just meatpackers, truck drivers and gay clubs. Then the transgender prostitutes moved in. And the crack dealers.” And then the artists. And then, in gentrification’s usual domino effect, the real estate brokers.

Today, Morellet’s neighbourhood has a surreal air, with 25 meatpackers left shifting offal beside high-class fashion boutiques such as Jeffrey’s.

This is the city that invented gentrification back in the 1960s, when artists such as Warhol and Rauschenberg found beauty in the dirty dereliction of SoHo’s emptying industrial warehouses. Exactly the same is happening today with the High Line — once worthless, now “revalued” by Chelsea and Meatpacking’s incoming arty middle class, who see beauty in its shabby wilderness. It’s exotic in a shiny, overdeveloped city whose industrial past has all but disappeared.

Only to developers the High Line is an eyesore depressing land values, exactly what was thought of the now priceless loft apartments in SoHo. With real estate on the rise again in New York, there are big plans for the West Side, one of the least developed parts of the city. The question is the same one being played out at Ground Zero, the one that’s always played out on city streets: how to reconcile real estate and what Morellet calls “a sense of place”.

Back in the 1980s one spirited local railway enthusiast, Peter Obletz, tried to get the High Line’s trains running again. He backed the wrong economic horse. The new New York has nothing to do with industry; rather, it’s the consumerist playground of Carrie Bradshaw in Sex and the City.

People will pay for the kind of arty urban experience offered by a step up on to the High Line. It’s the so-called Bilbao effect. The architect Steven Holl got it right. Twenty years ago he proposed turning the High Line into a “bridge of houses”, an urban promenade. David and Hammond followed suit, holding, last year, an ideas competition for the High Line’s future, attracting 720 entries which proposed turning the viaduct into cow pastures, rollercoasters and one long swimming pool.

The “urban promenade” is all the rage these days, from the London Eye to Paris Plage, Mayor Bernard Delanoë’s astonishingly successful beach along the Seine. But in David and Hammond’s mind was another of the grands projets in Paris. In 1998 a derelict railway viaduct shadowing Avenue Daumesnil across the 12th arrondissement from the Place de la Bastille reopened as the Promenade Plantée, a three- mile (4.8 km) rooftop park, allowing you to indulge your nosyparkerness by peering into apartments.

The question was how to convince New York’s city authorities to back such fripperies. Gentrification was David’s argument. On the one hand it gave him powerful local friends, such as Hillary Clinton, Harvey Keitel, Cynthia Nixon of Sex and the City, Cindy Sherman and Glenn Close, all of whom attend a glamorous yearly fundraiser hosted by Edward Norton and the designer Diane von Furstenberg. But David also had to appeal to that age-old Manhattan quality, greed fused elegantly with civic altruism. The Friends handed the city a study showing how, by not demolishing the High Line but turning it into a public asset, property prices would rise in the neighbourhood.

Ground Zero might be all rancour but elsewhere in the city Michael Bloomberg, the mayor, has pursued a less confrontational approach to urban planning than Rudy Giuliani (who was all for the High Line’s demolition). He’s brought in young, zippy staff, such as the city head of planning, Amanda Burden, keen on loosening slightly the zoning rules that have segmented New York rigidly since 1916, to release land for developers — though without alienating Manhattan’s powerful community activists — to create a more “mixed use”, diverse city.

“The High Line is such a magical space,” says Burden, “ugly, but with a unique character. I always said that if ever I got into a position of power this would be my highest priority. And now I am.”

The trick is to make the magic work for everyone. “This is a real-estate-driven city, and developers are always looking for the bottom line. But then the mayor has always insisted that architecture is a great economic developer.”

The answer? With one hand the city plans to protect the hundreds of galleries that today line the High Line as an “art and culture” district by turning the viaduct into an urban promenade; the city will reward developers who give public access. With the other hand the surrounding area will be re-zoned from mainly industrial to residential, to please the developers by allowing 4,200 new apartments.

Owners of properties near the High Line will be able to sell “air rights” to new developers to build taller buildings along Tenth and Eleventh Avenues, while keeping roofs low along the line, protecting its surreal views through the cityscape. Solomon couldn’t have done better.

This summer the high-profile finalists in the High Line’s masterplan competition have been on display at the Centre for Architecture. I prefer the least interventionist of all, that of TerraGRAM, who want to keep — even accelerate — the High Line’s wilderness qualities, that Secret Garden. But, according to The New York Times last week, a team has been provisionally selected comprising the landscape designers Field Operations, architects Diller Scofidio & Renfro and the artist Olafur Eliasson, who created the phenomenally popular artificial sun, the Weather Project, at Tate Modern this year.

Their proposal balances man-made and “natural”, the heroic and the wilderness, most radically. They call it “agri-ecture”, “a post-industrial instrument of leisure life and growth”, which weaves hard landscape — undulating concrete boardwalk arching high, meandering low, over galleries, market places, public swimming pool, beach, and daringly cantilevered amphitheatre — into soft vegetation, here woods, marshland, wilderness, meadows, there preened municipal borders, ending by the Gansevoort Meat Market in a gallery and staircase with a glass wall that turns butchering into a spectator sport. It has “magic” in spades.

With the Friends confident about raising up to £100 million, work could begin in the autumn on a low-rise project whose subtlety and intelligence could, in the end, put Ground Zero’s Freedom Tower, all 1,776ft (541 metres) of it, in the shade.